Software Systems Atlas

English

中文

English

中文

Theme

VOL 07 / CH 03 / LESSON 02

3.2 Raft's Safe Reads, Membership Changes, and Snapshots ​

While log replication can be safely achieved, it does not guarantee linearizability for all reads, nor does it allow direct modification of configuration files to add or remove nodes. Production-grade Raft systems must also handle leader expiration, membership transitions, log compaction, disk space constraints, and latency.

Leader Local Reads May Return Stale Results ​

After a previous Leader is network-isolated, it might temporarily remain unaware that a new Leader has been elected with a higher term. If it directly reads its local state and responds, it may return stale results.

Common approaches for linearizable reads:

  • The read request first passes through log commit, ensuring safety but adding overhead to write paths;
  • ReadIndex / quorum confirmation: the Leader first establishes that an entry from its current term has committed, records a safe index for this read, confirms leadership with majority communication initiated after the read arrived, and waits until the state machine has applied through that index;
  • Lease read: relies on clock and lease assumptions and must strictly satisfy implementation conditions.

A newly elected Leader often commits a current-term no-op to establish the committed prefix. A successful response to an old heartbeat cannot prove leadership for a new read. Otherwise, evidence collected before isolation would be used to justify stale answers after isolation.

Follower reads are typically stale unless they first obtain a safe read index from the Leader for this read and catch up. APIs must explicitly define read consistency levels and should not treat arbitrary node GET operations as strongly consistent.

Leader Completeness Alone Does Not Make Reads Current ​

Raft's log safety guarantees that committed entries will not be overwritten by different values. However, client-visible linearizability requires additional assurances:

  • For retries of one logical command, reuse its request ID and retain executed IDs and results in replicated state and snapshots. Deduplication must survive leader changes; generating an ID alone prevents nothing;
  • Reads must follow a safe read protocol;
  • The state machine executes commands in the order they are committed and deterministically;
  • Snapshots must be restored to the same state as the log during recovery;
  • Clients must not bypass these read and write protocols to access replica state directly.

Member Changes Cannot Simply Replace the List ​

A direct switch from the old configuration {A,B,C} to the new configuration {C,D,E} allows {A,B} to form an old majority and {D,E} a new majority, with no shared member. Each could elect its own leader.

Raft's original paper proposes joint consensus: during the transition phase, decisions require agreement from both the old and new configurations' majorities before the new configuration is finally committed. Specific systems may implement variants such as adding or removing one member at a time, but they must strictly adhere to their protocol and operational sequence.

Suppose one of three voting members has failed. Directly adding a fourth voter that has not started changes the majority threshold from 2 to 3, while only two members remain healthy. etcd supports adding a non-voting learner and promoting it after catch-up, and checks for unsafe membership changes. Follow the product’s supported sequence for replacing failed members; another address in the list does not supply another healthy voter.

Before changing membership, check:

  • Has the new node caught up to a sufficiently recent log tail?
  • Are there already known failed members in the current cluster?
  • How much quorum margin remains during the transition?
  • Is node identity stable, and could an old data directory rejoin with incorrect identity?
  • Could automation concurrently execute multiple configuration changes?

Snapshots Compact Committed Prefixes ​

Log growth without bound slows down replay and fills disk space. A node can snapshot the state at an applied position together with that position’s index and term, then remove covered entries after the snapshot is durable. This index is the last entry included in the snapshot, which need not be the current log tail.

If a lagging Follower requires log entries that have been compacted, the Leader sends a snapshot via InstallSnapshot and resumes replication from the point after the snapshot.

Snapshots must:

  • Be atomically tied to a specific applied index;
  • Contain the state-machine data, deduplication state, and membership configuration corresponding to that position;
  • Be validated for integrity and safely replace the old state;
  • Be coordinated with concurrent log appends;
  • Not leave a gap (where logs are deleted but the snapshot is incomplete) during node crashes.

Internal snapshots used for log compaction advance with the cluster and may fail with the original disk. They cannot alone provide accidental-deletion recovery or history retention. Independently exported snapshots, retained elsewhere and verified as restorable, can serve as backups, as with etcd snapshot backups. Their usefulness still depends on retention, version compatibility, and recovery exercises.

Disk is part of the consensus path ​

Fast communication within a majority cannot compensate for slow disk fsync; commits still wait for durable acknowledgments. Full disk exhaustion, tail latency, firmware issues, and filesystem corruption can all render a node unable to participate in quorum.

Monitor the following:

  • Leader transitions and duration of leaderless periods;
  • Latency from propose to commit, and from commit to apply;
  • The amount each Follower lags behind in matchIndex;
  • fsync latency, disk space availability, and WAL errors;
  • Time taken to generate, transfer, and restore snapshots;
  • Membership changes and failure states.

The Usage Boundaries of Systems Like etcd ​

etcd serves relatively small, strongly consistent control-plane state, rather than large objects or high-throughput data streams. This product boundary is not a limit on Raft’s uses: sharded databases can replicate application data across multiple Raft groups. With etcd, respect its request-size, storage-capacity, compaction, watch, and backup limits.

Clients should use the transactional, revision, and watch semantics provided by the system itself, rather than inferring API reliability from the fact that the underlying protocol is Raft.

Relationship with Paxos and ZAB ​

Paxos family protocols and Raft both use majority intersection to achieve consensus, but they differ in abstraction and engineering structure; you cannot simply map Raft's roles and log rules onto every Paxos implementation.

ZooKeeper's ZAB is designed around leader-driven atomic broadcast and recovery phases, and its client semantics include sessions, watches, and ordering guarantees. Understanding a protocol builds a foundation, but the actual product behavior must be grounded in its own specifications.

Fault Testing ​

  • The leader crashes after appending locally but before replicating;
  • The Leader crashes after a majority of replicas have durably replicated, but before clients receive a response;
  • The old leader is isolated, a new leader is elected, and then the old leader recovers;
  • Followers' logs have conflicting suffixes;
  • A snapshot is being written halfway when the process terminates;
  • A node is lost during a membership change;
  • The disk fills up or fsync has a long tail.

The trustworthiness of consensus implementations comes from model checking, mature codebases, and systematic fault testing, not from a simplified election algorithm.

References ​

Built with VitePress | Software Systems Atlas

Copyright 2026 Steven Yang