17.2 Publish/Subscribe, Stream Processing, and Event Pipeline Observability

Ten minutes after a tournament ends, the leaderboard still shows old scores. The on-call engineer sees a live broker but cannot tell whether the event is stuck in the Outbox, the consumer queue, or a projection update.

Event systems come in many forms. Notification-based publish-subscribe systems care about "who should respond after a fact occurs," while stream processing focuses on "how a continuous event sequence is transformed, aggregated, and materialized." Both use a broker, but they don't share the same semantics regarding state and time.

Publish-Subscribe and Persistent Streams

Dimension	Notification-type Pub/Sub	Persistent Event Stream
Primary Objective	Broadcast facts, trigger reactions	Preserve ordered records, continuous computation
Consumption Position	Typically Controlled by Queue Acknowledgment	Typically Controlled by Offset/Checkpoint
Replay	May be limited	Typically a core capability
State	Often maintained outside the messaging layer	Processors commonly maintain a state store
Typical Use Cases	Email notifications, cache invalidation	Ranking aggregation, real-time risk control, metrics pipelines

Specific products may support multiple semantics simultaneously; design should focus on business needs, not product labels.

Distinguish Event, Processing, and Ingestion Time

• Event time: the time when the fact occurred in the source system;
• Processing time: the clock time when the processor handles the event;
• Ingestion time: The time when the platform receives the event.

Network latency and offline clients can cause event ordering issues. When calculating a "score every five minutes" based on event time, a watermark estimates event-time progress. It does not prove that no older record can arrive. A Flink operator with multiple inputs is generally limited by the slowest active input watermark. An idle input can stall all windows unless idleness is handled correctly. Late-event policies can:

• Allow a certain amount of lateness and update the old window;
• Send the late event into the correction stream;
• Only add corrections, never silently discard;
• Expose how final the displayed results are.

For example, consider a Flink event-time window [10:00, 10:05) with two minutes of allowed lateness. Once the watermark reaches 10:05, it can emit an initial result. A new record timestamped 10:03 arriving while the watermark is 10:06 can still update that window. If the watermark has reached 10:08, that record instead needs a configured late-data side output or business correction path. These are watermark values, not elapsed wall-clock minutes.

Window updates can emit multiple aggregate results for the same window. The sink must update by (playerId, windowStart) or handle retractions; it must not add every result as fresh points. Flink defaults to zero allowed lateness and drops records beyond window retention by default. Configure the correction policy explicitly.

When a window is final remains a business decision as well as a framework setting.

Stateful processing requires recoverable checkpoints

A ranking aggregator must coordinate its state, input offset, and output when maintaining player scores:

Complete checkpoint: consistent input positions, operator state, and required sink commit metadata
Recover: restore state → replay from saved positions → commit or deduplicate at the sink

This is a recovery relationship, not a protocol obtained by writing four sequential calls. A framework coordinates the recovery boundary through snapshots or transactions, and the source must support replay from saved positions. Applying external output once also requires a sink that participates in the commit protocol or supports idempotent writes. Wrapping an ordinary HTTP call in a checkpoint does not make the receiver remember that it already charged the account.

Before upgrading the processing logic, you must assess: whether the old state is compatible with the new schema, whether you need to rebuild from the original stream, and whether such a rebuild could overwhelm downstream systems.

Choreography doesn't mean no process responsibility

When multiple consumers collaborate around an event, someone still needs end-to-end ownership. At least maintain:

• Flowcharts and event owners;
• SLA and failure semantics for each step;
• Correlation IDs and business status queries;
• Alerts, compensation, and manual operation interfaces.

If a critical process requires reading ten consumer source files to understand, introduce a process view, state projection, or refactor to explicit orchestration.

How to Trace an Event Chain

Synchronous calls also need instrumentation to propagate context; a call stack does not cross process boundaries. For asynchronous events, distinguish:

• Trace context: for example, the trace ID, parent span ID, and flags in W3C traceparent; a traceId alone cannot restore the full parent relationship;
• correlationId: Same business process;
• causationId: The command or event that directly caused the current event;
• eventId: The unique identity of the current event.

Producers inject context into message properties or agreed envelope fields. Consumers extract it and create processing spans. For batches or results caused by multiple messages, span links can represent multiple causal inputs; follow the conventions of the OpenTelemetry instrumentation in use.

Do not force every event into a single, never-ending trace. Long-running processes can be connected across multiple traces and visualized using a business timeline to show their state.

Key Metrics

Infrastructure online does not mean the event system is healthy. At least observe:

• Production rate, consumption rate, and lag;
• Age of the oldest unresolved event;
• End-to-end processing latency, rather than broker latency alone;
• Retry, dead-letter, duplicate, and out-of-order counts;
• Partition skew and hot key;
• Projected version, reconstruction progress, and business reconciliation variances.

Offset lag measures a position difference, not waiting time in seconds, and records can have very different processing costs. Measuring from occurredAt to completion also includes source clock skew; retain receipt times and stage durations too.

Alerts should be tied to user impact. For example, increasing lag that remains within business freshness targets might indicate a capacity warning; critical settlement events missed by deadline require immediate response.

Pre-launch fault exercise

• Consumers crash after the business commit but before acknowledgement;
• One partition contains a permanently invalid message;
• Events are late, duplicated, and mixed across versions;
• The broker is temporarily unavailable, causing outbox backlog;
• New events continue to enter during projection rebuilding;
• A hot aggregate key overwhelms a single partition.

These recovery paths let the team find and repair failures before an old leaderboard becomes the only evidence that something went wrong.

References

• Apache Kafka, Kafka Streams Architecture
• Apache Flink, Timely Stream Processing
• Apache Flink, Windows and allowed lateness
• OpenTelemetry, Messaging spans
• OpenTelemetry, Messaging semantic conventions