Chapter 17: Message Queue

---

Metadata Card

Attribute	Content
Difficulty	(Intermediate)
Prerequisites	Basic understanding of producer/consumer patterns
Keywords	Message queue, Kafka, RabbitMQ, Pulsar, pub-sub, broker, partition, offset
Code	Python (client examples)

---

Your Progress

"The conveyor belt system between warehouses — decoupling production and consumption, smoothing peaks and filling valleys."

Why Message Queues?

Problem	Solution
Tight coupling between services	Decouple producers and consumers
Traffic spikes overwhelm consumers	Buffer + back-pressure
Service temporarily unavailable	Persistent storage + replay
Broadcast to multiple services	Pub-sub pattern
Async processing	Fire-and-forget

Core Concepts

Concept	Description
Producer	Sends messages to the queue
Consumer	Reads messages from the queue
Broker	Server that manages topics/partitions
Topic	Category/feed name for messages
Partition	Ordered, immutable sequence of messages
Offset	Position of a message within a partition
Consumer Group	Multiple consumers coordinate to process a topic

Delivery Semantics

Guarantee	Meaning
At most once	Message may be lost, never duplicated
At least once	Message never lost, may be duplicated
Exactly once	Message delivered once and only once

Kafka

Design: Distributed commit log. Topics split into partitions, each partition is an ordered log.

Topic "orders"
┌──────────┬──────────┬──────────┬──────────┐
│Partition0│Partition1│Partition2│Partition3│
├──────────┼──────────┼──────────┼──────────┤
│ msg 0     │ msg 0     │ msg 0     │ msg 0     │
│ msg 1     │ msg 1     │ msg 1     │ msg 1     │
│ msg 2     │ msg 2     │ msg 2     │ msg 2     │
│ ...       │ ...       │ ...       │ ...       │
└──────────┴──────────┴──────────┴──────────┘

Key features: High throughput (sequential I/O), durable, replayable, scalable.

RabbitMQ

Design: Smart broker / dumb consumer model. Exchange routes messages to queues based on routing rules.

Exchange types: Direct, Topic, Fanout, Headers.

Features: AMQP protocol, flexible routing, dead letter queues.

Pulsar

Design: Separate serving and storage layers (Apache BookKeeper). Supports both queue and streaming semantics.

Key innovation: Unified messaging model — same system can be used as a message queue (exclusive consumers) or a streaming system (competing consumers).

Use Cases

• Event-driven microservices: Service A publishes events, Service B/C subscribe
• Stream processing: Kafka Streams, Flink, Spark Streaming
• Task queues: Celery, Bull, Sidekiq
• Log aggregation: Centralized log collection
• Metrics collection: Distributed metrics pipeline

---

Traveler's Notes

Message queues are the backbone of modern, loosely-coupled distributed systems. They allow services to communicate asynchronously, smooth out traffic spikes, and provide a durable buffer between producers and consumers. Understanding the trade-offs between Kafka (throughput), RabbitMQ (flexible routing), and Pulsar (unified model) helps you choose the right foundation for your architecture.