17.1 Understanding Kafka, Pulsar, and RabbitMQ Through Message Lifecycle

Message systems are not simply "a buffer between two services." They define who persists messages, how messages are scoped and ordered, where consumption progress is stored, how retries are handled on failure, and who ultimately bears the cost of message backlog.

Draw the Complete Lifecycle

A reliable message must at least traverse the following steps:

producer business state
  -> serialize + choose destination/key
  -> broker receives
  -> broker persists/replicates
  -> producer gets confirmation
  -> consumer fetches/delivery
  -> consumer changes downstream state
  -> consumer records acknowledgement/offset
  -> retention or deletion

A power outage or network interruption between any two consecutive steps can result in an ambiguous state: "The other side has already processed it, but I haven't received confirmation." Reliable design cannot simply rely on calling send() and ack().

Queue and Log Consumption Positions Differ

Traditional queues are typically managed by a broker that tracks which deliveries have not yet been acknowledged. Once acknowledged, messages can be deleted or no longer delivered to that consumer. Multiple consumers can compete for work within the same queue.

Persistent logs append messages to ordered partitions, and consumers maintain their own position/offset. Messages are not immediately removed after being read by one consumer; other consumer groups can independently replay them until they exceed retention policies.

Both can support work distribution and publish-subscribe patterns, but their cost models differ:

• Queues excel at flexible routing, per-message acknowledgment, and reliable task delivery;
• Logs are optimized for high-throughput append operations, independent consumer progress tracking, and historical replay;
• Log replay requires downstream consumers to be idempotent and capable of handling older events;
• Queues suffer from memory overhead due to a large number of unacknowledged deliveries held by the broker and consumers.

Kafka: Partitioned Logs and Consumer Groups

A Kafka topic is split into partitions, each of which is an ordered append-only log with offsets. Ordering is guaranteed only within a single partition and does not hold across the entire topic by default.

Producers typically use keys to determine partitioning: if a business requirement demands message processing in order, a stable key should be used, such as order_id. However, popular keys can create hotspots in a single partition. Increasing the number of partitions may also alter the default partitioning mapping, so the evolution of key-to-partition mapping must be considered during design.

In the traditional consumer-group model, one partition is assigned to a single consumer at any given time. When the number of consumers exceeds the number of partitions, additional instances have no partitions to consume. Scaling up or down, or a consumer failure, triggers rebalancing, which may redistribute messages across consumers during processing. Kafka share groups use a different record-sharing and acknowledgment model and can have more consumers than partitions. The offset discussion below concerns traditional consumer groups.

A committed offset normally identifies the next record to read: after processing offset 40, commit 41. If 40 and 41 execute concurrently and only 41 finishes, committing 42 would skip unfinished record 40 after restart.

Offsets indicate "where the group has processed up to," not proof that business logic has completed. Committing offsets too early risks message loss; committing too late risks duplicate processing:

poll record
process business side effect
commit offset

If a consumer crashes after writing business logic but before committing the offset, the message will be retried. Therefore, consumers must remain idempotent.

Kafka's retention policy deletes old segments based on time or size. Log compaction, on the other hand, gradually removes older records per key while retaining the latest value of surviving keys. Compaction is not immediate deduplication; delete tombstones also have retention limits. Neither behaves like consume-and-delete queueing.

Pulsar: Separation of Broker and Persistent Log Storage

The Pulsar broker handles connections, topic ownership, scheduling, and message distribution to consumers. BookKeeper bookies are responsible for storing persistent message logs, while the metadata store manages coordination of metadata. Topic ownership can move between brokers without moving all stored data, and storage capacity can be independently scaled through the addition of bookies.

This separation does not eliminate coordination costs between storage and computation. Operations such as ledger creation, ensemble writes, acknowledgment quorums, cursors, and backlog management still directly influence write acknowledgments and recovery behavior. Choosing Pulsar should be driven by specific requirements (such as multi-tenancy, topic volume, storage scalability, or geo-replication) rather than simply because the architecture is "more advanced."

RabbitMQ: Routing from Exchange to Queue

In RabbitMQ, a producer publishes messages to an exchange, which then routes them to one or more queues based on bindings:

producer -> exchange --binding--> queue -> consumer

Common exchange types:

• Direct: Matches the routing key exactly;
• Topic: Uses pattern matching based on dot-separated keys;
• Fanout: Ignores the routing key and broadcasts the message to all bound queues;
• Headers: Routes messages based on header conditions.

The exchange is responsible for routing, not for storing messages. If no queue matches a message, whether it is returned or discarded depends on configurations such as mandatory and alternate exchange. It is not safe to assume that "publishing succeeds" means "someone will receive it."

Queue types and persistence settings influence fault semantics. Publisher confirms indicate that the broker has accepted responsibility for delivering the message to the specified queue type. Consumer acknowledgments mean that the consumer has successfully processed the message and is ready for deletion or advancement. These acknowledgments operate in opposite directions. Even an unroutable message can receive a publisher confirm; mandatory returns or another routing check must establish whether a target queue received it.

Select by Constraint, Not by Brand

Requirement	Initial Evaluation	Requires Validation
Event logs, replay capability, multiple independent consumer groups	Kafka / Pulsar	Number of partitions, retention period, disk backlog, and replay throughput
Complex routing, work queues, message-level ACKs	RabbitMQ	Queue type, prefetch settings, retry and dead-letter strategy
Large number of topics, independent broker and storage scaling	Pulsar	BookKeeper operations, ledger/backlog management, and multi-tenant isolation
Simple asynchronous tasks with an existing database	Evaluate database queues/outbox first	Lock contention, polling behavior, cleanup mechanisms, and scalability limits

Product comparisons must use fixed versions, deployment topologies, and reliability configurations. Throughput benchmarks derived from disabling replication and disk flushing cannot be used to guarantee performance under production-grade reliability.

References

• Apache Kafka Documentation
• Apache Pulsar: Architecture Overview
• RabbitMQ: Reliability Guide

The next section begins with failure points and explains why at-most-once, at-least-once, and exactly-once semantics must all specify their scope.