17.2 Publish/Subscribe, Stream Processing, and Event Chain Observation

After the event crosses multiple services, the on-call technician sees only that the final number is wrong but can't pinpoint which segment is missing, duplicated, or delayed.

Event systems come in many forms. Notification-based publish-subscribe systems care about "who should respond after a fact occurs," while stream processing focuses on "how a continuous event sequence is transformed, aggregated, and materialized." Both use a broker, but they don't share the same semantics regarding state and time.

Publish-Subscribe and Persistent Streams

Dimension	Notification-type Pub/Sub	Persistent Event Stream
Primary Objective	Broadcast facts, trigger reactions	Preserve ordered records, continuous computation
Consumption Position	Typically Controlled by Queue Acknowledgment	Typically Controlled by Offset/Checkpoint
Replay	May be limited	Typically a core capability
Status	Often maintained outside of consumers	Processor commonly maintains status storage
Typical Use Cases	Email notifications, cache invalidation	Ranking aggregation, real-time risk control, metrics pipelines

Specific products may support multiple semantics simultaneously; design should focus on business needs, not product labels.

Stream processing must distinguish between time

• Event time: the time when the fact occurred in the source system;
• Processing time: the actual time the processor takes to process;
• Ingestion time: The time when the platform receives the event.

Network latency and offline clients can cause event ordering issues. When calculating a "score every five minutes" based on event time, a watermark indicates where the system believes the data stream has progressed, and a late-event policy must be defined.

• Allow a certain amount of lateness and update the old window;
• Send the late event into the correction stream;
• Only add corrections, never silently discard;
• The level of finalization of results presented externally.

When window results are considered "final" is a business decision, rather than a framework parameter alone.

Stateful processing requires recoverable checkpoints

A ranking aggregator must coordinate its state, input offset, and output when maintaining player scores:

Read event → Update local state → Write output → Commit checkpoint

A single failure can lead to replay. While the processing framework might offer transactions or checkpointing mechanisms, external databases and HTTP side effects still require idempotent keys, an Outbox, or dedicated sinks to ensure safety.

Before upgrading the processing logic, you must assess: whether the old state is compatible with the new schema, whether you need to rebuild from the original stream, and whether such a rebuild could overwhelm downstream systems.

Choreography doesn't mean no process responsibility

When multiple consumers collaborate around an event, someone still needs end-to-end ownership. At least maintain:

• Flowcharts and event owners;
• SLA and failure semantics for each step;
• Query related ID and business status;
• Alarm, compensation, and manual operation interfaces.

If a critical process requires reading ten consumer source files to understand, introduce a process view, state projection, or refactor to explicit orchestration.

How to Trace an Event Chain

Synchronous call traces propagate through the call stack; asynchronous events span time and processes and require simultaneous retention:

• traceId: Current technology call chain;
• correlationId: Same business process;
• causationId: The command or event that directly caused the current event;
• eventId: The unique identity of the current event.

Don't force every event into a single, never-ending trace. Long-running processes can be connected across multiple traces and visualized using a business timeline to show their state.

Key Metrics

Infrastructure online does not mean the event system is healthy. At least observe:

• Production rate, consumption rate, and lag;
• Age of the oldest unresolved event;
• End-to-end processing latency, rather than broker latency alone;
• Retry, dead-letter, duplicate, and out-of-order counts;
• Partition skew and hot key;
• Projected version, reconstruction progress, and business reconciliation variances.

Alerts should be tied to user impact. For example, increasing lag that remains within business freshness targets might indicate a capacity warning; critical settlement events missed by deadline require immediate response.

Pre-launch fault exercise

• Consumers crash after submission but before confirmation;
• A single partition reports permanent failure;
• Events are late, duplicated, and mixed across versions;
• The broker is temporarily unavailable, causing outbox backlog;
• New events continue to enter during reprojecting;
• Hotspot aggregation key overwhelms a single partition.

It's not just about swapping synchronous errors for asynchronous silence, event-driven architectures can truly recover from these scenarios.

References

• Apache Kafka, Kafka Streams Architecture
• Apache Flink, Timely Stream Processing
• OpenTelemetry, Messaging semantic conventions