17.2 Confirming Boundaries, Idempotent Consumption, and Message Backlog Recovery

"The message is processed exactly once" sounds like a broker configuration, but in reality, it spans producers, brokers, consumers, and the business database. As long as there's any possibility that a message could be lost, the sender must make a decision between "possibly succeeded" and "possibly failed."

Three Delivery Semantics: First Define the Scope

• at-most-once: Messages may be lost, but no active retry is performed;
• at-least-once: Messages are not easily lost, but delivery may be repeated during fault recovery;
• exactly-once: Within a clearly defined boundary, the final outcome is observationally equivalent to a single delivery.

"Exactly-once" must explicitly define its scope. Kafka transactions can atomically write across multiple Kafka partitions and allow read_committed consumers to hide uncommitted records. However, this does not automatically include a single email, an HTTP call, or any external database within the same transaction boundary.

Producer Confirmation Still Introduces Uncertainty

If a producer times out after sending a message:

broker does not receive -> retry is necessary
broker has persisted the message but confirmation is lost -> retry may result in duplicates

Kafka's idempotent producer uses mechanisms like producer identity and partition sequence to eliminate duplicate retries within a session scope. Similarly, in RabbitMQ, if the publisher confirms a message and then loses connection, the producer cannot be certain whether the confirmation has reached the destination. As a result, business events should still rely on stable event_id or idempotent keys.

Consumer Ack Must Follow Business Processing

receive
validate schema
perform idempotent business transaction
record event_id / resulting state
ack or commit offset

Acknowledging the message before writing to the database risks permanently losing business processing if the process crashes between those two steps. Writing to the database before acknowledging introduces the possibility of message redelivery, but an idempotent business table can detect and ignore duplicates.

CREATE TABLE consumed_event (
  consumer_name text NOT NULL,
  event_id uuid NOT NULL,
  consumed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, event_id)
);

BEGIN;

INSERT INTO consumed_event (consumer_name, event_id)
VALUES ('inventory-projector', :event_id)
ON CONFLICT DO NOTHING;

-- Execute business update within the same transaction only if the first insert actually succeeds

COMMIT;

When implementing this, the system must read the number of affected rows from the insert operation to ensure that duplicate events do not trigger redundant updates to the business table. The retention period for idempotent records must cover both the longest possible replay window of the broker and the manual recovery window.

Outbox Solves the Dual Write Problem of Database Commit and Message Publication

If a business transaction first modifies the database and then publishes a message, the process might crash between these two steps. If it first publishes a message and then updates the database, consumers might receive facts that haven't yet been validated or committed.

The Outbox pattern writes both the business state and the pending events into the same local transaction:

BEGIN;

UPDATE orders
SET status = 'paid'
WHERE order_id = :order_id
  AND status = 'pending';

INSERT INTO outbox_event (
  event_id, aggregate_type, aggregate_id, event_type, payload
) VALUES (
  :event_id, 'order', :order_id, 'OrderPaid', :payload
);

COMMIT;

A separate relay or CDC component reads from the outbox to publish events. Relays may re-publish messages, so consumers must still be idempotent. While the Outbox ensures atomic creation of facts, it does not guarantee end-to-end exactly-once semantics across the entire pipeline.

Order Holds Only Within Selected Keys and Channels

If an order event must maintain sequence, it should use the same partition key/queue and carry the aggregate version:

{
  "event_id": "...",
  "aggregate_id": "order-9182",
  "aggregate_version": 7,
  "event_type": "OrderPaid"
}

Consumers can reject backward versions or temporarily hold gaps until resolved. There is no inherent global order across partitions; parallel processing, retry queues, and dead-letter replay can alter the final sequence.

Poison Message and Retry Budget

Serialization failures, permanent business constraint errors, and transient dependency faults cannot be handled with the same infinite retry loop.

Recommended categorization:

• transient: exponential backoff, jitter, and finite retry counts;
• rate-limited: respect the service's retry window and enforce rate limiting;
• permanent/schema: route to an isolation queue, preserve the original message, error details, and version;
• operator action: replay after manual correction, maintaining original key order.

Dead-letter queues are not trash bins. They must include alerts, ownership assignment, remediation procedures, replay tools, and a strategy for handling subsequent failures.

Backpressure and Accumulated Recovery

Queues can turn burst traffic into backlog, but they can't magically increase downstream capacity. You should also monitor:

• ingress rate versus sustainable processing rate;
• consumer lag or backlog age, rather than message count alone;
• Processing latency and failure rate per line;
• skew between partition and queue;
• Broker disk, replication, and retention quotas;
• Time required to recover from peak backlog to normal levels.

If 10,000 events per second enter and the system can only process 8,000, backlog will grow infinitely. Peak shaving works only if the long-term average consumption rate exceeds the long-term average production rate, or if the system can discard or aggregate some events.

RabbitMQ consumer prefetch controls the number of unacknowledged deliveries; in Kafka, max.poll.interval.ms, batch size, and processing thread model collectively influence rebalance and throughput. Parameters should be tuned around the cost of processing a single message and memory budget through benchmarking, not copied from fixed recommendations.

Schema Evolution

Once messages enter the long-retention log, both old consumers and historical replay processes will encounter them. Events must include versioning and compatibility strategies:

• Adding optional fields is generally backward compatible;
• Changing the meaning of a field is more dangerous than changing its name;
• Before removing a field, verify that all consumers and historical replay mechanisms are unaffected;
• Events should represent facts that have occurred, not be reused as remote procedure call command packages;
• In CI pipelines, validate schema compatibility and representativeness of older message samples.

References

• KafkaProducer: Idempotence and Transactions
• RabbitMQ: Consumer Acknowledgements and Publisher Confirms
• Debezium: Outbox Event Router

The next chapter will no longer compare product marketing claims, but instead establish a selection process that records facts, constraints, costs, and exit strategies.