Metadata Card
- Prerequisites: Chapter 1 (Distributed Systems Fundamentals), Chapter 2 (RPC)
- Estimated Time: 50 minutes
- Core Difficulty: Advanced
- Reading Mode: High focus (read uninterrupted)
- Completion Milestone: Draw Raft's state transition diagram, explain Leader election and log replication flow, understand Paxos intuition, know ZooKeeper's leader election and config management
Your Progress
You can now coordinate communication between two outposts — message format is standardized, transmission channels are reliable. But your battle map has 5 forward stations, each with its own sentry logs. The problem now isn't "how to transmit data," it's "can 5 stations agree on the same thing."
General Lin's map has a red cross on the target position. He says: "Five squads must reach this position. Not four, not three — five. If one squad doesn't make it, the assault is invalid."
In distributed systems terms: consensus. Your Task
Master three layers of distributed consensus: Raft (Leader election, log replication, safety), Paxos intuition, ZAB basics.
Raft: Three independent sub-problems — Leader Election, Log Replication, Safety.
Roles: Follower → Candidate (election timeout) → Leader (majority votes) → Follower (higher term discovered).
Leader Election: Follower with no heartbeat for 150-300ms increments term, becomes Candidate, votes for itself, sends RequestVote RPC. Wins with majority.
Log Replication: Client writes to Leader → Leader appends to local log → Sends AppendEntries to Followers → Majority confirms → Commit → Apply to state machine.
Safety: Elected Leader must have all committed logs. Voting condition: Candidate's log must be at least as up-to-date as the voter's.
Paxos: Two-phase protocol (Prepare-Promise, Accept-Accepted). Multiple rounds. Harder to implement correctly.
ZAB: ZooKeeper's atomic broadcast protocol. Leader election includes a "discovery phase" where the new Leader synchronizes log with all Followers.
Common Pitfalls: Election timeout too tight for cross-region deployment, only 2 nodes (need majority = 2, one failure stops cluster), treating "replied to client" as "persisted" (data still in memory), testing only happy path.
Traveler's Notes
Raft solves distributed consensus with three independent mechanisms: Leader election, log replication, safety. Its understandability is the key engineering success. In production (etcd, Consul, ZooKeeper), you must understand every timeout and state transition — because consensus errors at runtime are fatal to consistency.
Next: Distributed Storage (Chapter 4).