15.2 Outbox, Idempotent Consumption, and Recoverable Messages

A process crashes between committing the database transaction and sending the message, leaving the system with an order that will never notify downstream services.

Services often need to update databases and publish events simultaneously. If you commit the database first and then send the message, the process might crash between the two steps; if you send the message first and then commit the database, consumers might see facts that ultimately don't exist.

The Transactional Outbox writes business changes and pending messages into the same local transaction, then has an independent relay publish the messages.

Close Outbox Atomic Gap

BEGIN;

UPDATE registration
SET status = 'CONFIRMED'
WHERE id = :registration_id
  AND status = 'PENDING_PAYMENT';

INSERT INTO outbox (
    event_id, aggregate_id, aggregate_version,
    event_type, payload, occurred_at, publish_state
) VALUES (
    :event_id, :registration_id, :version,
    'RegistrationConfirmed', :payload, CURRENT_TIMESTAMP, 'PENDING'
);

COMMIT;

Relay can poll the Outbox or use database log CDC to detect changes. Regardless of the method, there's still a risk of failure during the "message sent, publication status not yet recorded" phase, leading to potential duplicate publications.

Outbox provides "business change and release intent atomic records," not automatic exactly-once end-to-end processing.

Consumers must be idempotent

Consumers can build an Inbox/deduplication table using an event ID:

BEGIN;

INSERT INTO consumed_message (consumer, event_id, consumed_at)
VALUES (:consumer, :event_id, CURRENT_TIMESTAMP)
ON CONFLICT DO NOTHING;

-- Execute business update only if insertion is successful
UPDATE reward_account
SET points = points + :delta
WHERE player_id = :player_id;

COMMIT;

The actual implementation must check the number of affected rows on insert and keep "record marked as consumed" and business updates within the same transaction. If deduplication is done only in an in-memory set, records will be lost upon restart. If consumption is marked before business updates, failed crashes will permanently miss processing.

Some operations are inherently idempotent, such as "set the state to the value of version 7"; others, like "add 10 points to the score," must rely on event IDs or business operation IDs to ensure uniqueness.

Order should follow business key definition

Message systems typically guarantee ordering only within a single partition. Using aggregate_id as a partition key ensures that events from the same registration or competition enter the same ordered stream; global ordering across different aggregates should not be assumed.

Consumers should also check the aggregated version:

Current version 5, receiving version 6 → Apply
Current version 6, received version 6 → duplicate, ignore
Current version 5, receiving version 8 → missing 6, 7; postpone or fix

Relying solely on arrival time is unreliable for handling replay and cross-partition lag.

Retries and Dead Letter Queues

First, distinguish the error:

• Transient error: Database temporarily unavailable, retry after backoff;
• Permanent error: schema incompatibility, missing required fields, duplicates won't self-heal;
• Business rejection: Should not pretend to be infrastructure retry;
• Unknown error: Preserve context and limit retry count.

A dead-letter queue isn't a trash bin. Messages that enter a dead-letter queue must:

• Original payload, headers, event ID, and failure reason;
• Alerts and clearly assigns ownership;
• Audited operations that are fixed, replayed, or discarded;
• Replay events while still adhering to idempotency and ordering rules.

Infinite retries will block partitions and incur costs; skipping them outright might compromise business invariants.

Operations Metrics

At least monitor:

• The age of the oldest pending message in the Outbox;
• Release and consumption delays;
• Repeated messages, idempotency hits, and version gaps;
• Retry count, dead letter queue size, and age of the oldest dead letter;
• Account reconciliation discrepancies, rather than whether the broker is online alone.

Next lesson: distinguishing CQRS from Event Sourcing, explaining when they solve read/write model and history reconstruction issues, and when they merely add complexity.

References

• Chris Richardson, Transactional Outbox
• Debezium, Outbox Event Router
• Apache Kafka, Design: Delivery Semantics