1.1 Data Products, Event Semantics, and Contracts: Data Must Be Interpretable from the Moment It Is Generated

A column called success may describe an entire task or only one API attempt. Decide whether a retry creates another event or another task before calculating a rate.

Whether data can be analyzed begins with whether its semantics at generation are clearly defined. Cleaning can fix formatting, but it cannot reconstruct business meaning that was never recorded.

Learning Objectives

• Infer data requirements from decision-making problems;
• Distinguish between entities, events, snapshots, and metrics;
• Establish schemas, semantic definitions, and quality agreements for datasets;
• Handle event IDs, timestamps, duplicates, late arrivals, and schema evolution.

1. Start with the Question the Data Must Answer

Suppose a data team asks, "Has the completion rate for different task types declined over the past four weeks?" Before collecting data, it must define:

• What constitutes a unique task identity;
• Which states count as "completed";
• How cancellations, timeouts, and retries should be included in the denominator;
• Whether tasks are attributed to their creation date or completion date;
• Timezone and the exact point in time for the statistics;
• What level of data latency constitutes a complete and usable dataset.

Without these definitions, a SQL query may consistently return a number, but that number won’t accurately answer the original question.

Data collection follows the principle of purpose limitation: only gather fields that are explicitly needed for the intended use case, and always document the purpose, responsible party, and retention period. "May be useful in the future" is not a valid justification for collecting data indefinitely.

2. Entities, Events, and Snapshots

Three common record models:

Model	What one row represents	Typical fields
Current State	The known state of an object at a given moment	mission_id, status, updated_at
Event	A fact that occurred at a specific point in time	event_id, mission_id, event_type, occurred_at
Periodic Snapshot	A complete state observed at a specific point in time	snapshot_date, mission_id, status

Events are well-suited for tracking change history, but replay and out-of-order processing become more complex. Current state queries are simple, yet they risk losing the evolution of changes. Snapshots enable analysis of a system’s state at a given point in time, but they require significant storage and introduce challenges related to snapshot gaps.

The same pipeline can simultaneously retain both events and derived current states. The key is to avoid combining all three into a single table and relying on column names to infer meaning.

3. Time Has at Least Two Dimensions

occurred_at: the actual time when a business event occurred
ingested_at: the time when the platform received the record

A late event has an earlier occurred_at and a later ingested_at. Use event time for business windows and ingestion time for pipeline delay. With only one timestamp, a changed historical metric is difficult to explain.

It's also essential to clearly define:

• Timestamp timezone;
• Precision and acceptable tolerance;
• How to handle inaccurate device clocks;
• When a window closes;
• Whether late data should be retroactively applied to historical results.

4. Stable IDs Support Idempotency

Network retries, producer restarts, and consumer recoveries can all lead to duplicate deliveries. If a business event generates a new ID on every retry, downstream systems cannot reliably deduplicate events.

A message contract may include:

{
  "event_id": "evt-m1042-completed-3",
  "event_type": "mission.completed",
  "schema_version": 3,
  "mission_id": "M-1042",
  "occurred_at": "2026-08-03T09:15:00Z",
  "producer": "mission-service",
  "payload": {"result": "success"}
}

A stable event_id identifies repeated delivery, but the consumer still needs atomic deduplication and effect application. If it records the ID before updating the result and then crashes, a retry may skip unfinished work. If it updates first and records later, the retry may repeat the effect. Use a shared transaction or a documented equivalent; deduplication retention must cover the allowed replay horizon.

"Exactly once" should be broken down into specific scopes: is message transmission idempotent? Are side effects and business outcomes idempotent? Many systems rely on at-least-once delivery combined with idempotent consumers to achieve a single, consistent business outcome.

The producer has a similar boundary: committing a business transaction and publishing an event separately can lose the event between the two. An outbox written in the business transaction plus a retryable publisher addresses that gap; it does not remove the consumer’s deduplication requirement. Kafka’s delivery-semantics discussion also distinguishes log processing from external-system effects.

5. Data Contracts Are More Than Just Field Types

An executable contract typically includes:

• Field names, logical types, and nullability;
• Units, enumerated values, and value ranges;
• Primary keys or business-unique keys;
• Event and time semantics;
• Schema compatibility rules;
• Freshness, completeness, and availability targets;
• Data classification and access requirements;
• Ownership, consumer definitions, and change notification workflows.

For example, duration written as an integer is far insufficient: it must specify whether it represents milliseconds or seconds, whether it includes queueing time, whether failed tasks are allowed to be null, whether negative values might result from clock adjustments, and other such details.

6. Schema Evolution Requires Compatibility Strategies

Common changes:

• Adding optional fields;
• Expanding numeric ranges;
• Renaming or removing fields;
• Altering the meaning of enumerations;
• Splitting a single event into multiple event types.

Adding a field can also break consumers that read data by column position after using SELECT *. Compatibility depends on the serialization format and the consuming applications' actual behavior.

Safe workflow:

1. Release both old and new schema formats that consumers can understand;
2. Begin producers writing the new field;
3. Monitor old-consumer usage and the new field’s null rate separately;
4. Complete data backfilling or migration;
5. Remove the old field after a deprecation period.

A schema registry can validate structural compatibility but cannot guarantee that the business meaning of fields has not changed. Semantic changes require versioning, change notices, and consumer confirmation.

7. Metrics Also Need Contracts

Example of a metric definition:

Name: mission_completion_rate
Numerator: unique missions that entered completed within the statistical window
Denominator: unique missions that entered terminal state within the same cohort
Exclusions: test tenants, manually replayed events
Time: by terminal event occurred_at, UTC day
Maturity period: 24 hours after window closes
Owner: Operations Analytics

This denominator measures completion among tasks that reached a terminal state; it excludes unfinished work. If four created tasks contain two completed, one failed, and one pending task, the terminal-state rate is $2/3$, while completion in the creation cohort is $2/4$. Neither can substitute for the other. Reopening and later correction need a versioned rule for which terminal state counts.

A 24-hour maturity period is a declared reporting policy, not proof that every event arrived. Choose it from observed lateness and reconciliation evidence, label provisional periods, and state what happens when later corrections still arrive.

8. The Original Layer Is Not a "Forever Correct" Backup

Preserving immutable original inputs supports replay and auditing, but it may still contain:

• Incorrect values written by the source system;
• Failed or truncated data decoding;
• Personal or sensitive information;
• Duplicated or out-of-order events;
• Multiple versions of data with evolved schemas.

The original layer should record ingestion batches, checksums, source origins, schema versions, and access controls. Immutability does not imply indefinite retention; deletion obligations may require the concurrent removal of original, derived, backup, and index data.

Common Misconceptions

• "Collect all data first, then define metrics": Without semantic clarity from the start, data cleaning cannot restore meaningful context.
• "A single timestamp is sufficient": Event time and processing time answer fundamentally different questions.
• "Message queues eliminate duplication": Retry mechanisms and recovery still require idempotent design.
• "Schema compatibility equals business compatibility": Even if field types remain unchanged, the meaning of those fields may have evolved.

Exercise

1. Define the numerator, denominator, time attribution, and exclusion criteria for "task completion rate."
2. Design event IDs and retry semantics for task state transitions.
3. Identify a schema change that is type-compatible but semantically incompatible.
4. List at least five provenance fields for a raw events table.

Summary

The data lifecycle begins with defining facts, not with reading a CSV file. Event semantics, dual time, stable IDs, and data contracts determine whether downstream systems can replay, deduplicate, and interpret metrics correctly.

The next lesson builds the ingestion, storage, processing, publishing, archiving, and deletion workflow along this contract, turning each step into an operational state transition.