3.2 Raft's Safe Reads, Membership Changes, and Snapshots

While log replication can be safely achieved, it does not guarantee linearizability for all reads, nor does it allow direct modification of configuration files to add or remove nodes. Production-grade Raft systems must also handle leader expiration, membership transitions, log compaction, disk space constraints, and latency.

Leader Local Reads May Return Stale Results

After a previous Leader is network-isolated, it might temporarily remain unaware that a new Leader has been elected with a higher term. If it directly reads its local state and responds, it may return stale results.

Common approaches for linearizable reads:

• The read request first passes through log commit, ensuring safety but adding overhead to write paths;
• ReadIndex / quorum confirmation: verifies that the current Leader still holds leadership and waits for the state machine to apply to a safe index;
• Lease read: relies on clock and lease assumptions and must strictly satisfy implementation conditions.

Follower reads are typically stale unless they first fetch a safe read index from the Leader and catch up. APIs must explicitly define read consistency levels and should not treat arbitrary node GET operations as strongly consistent.

Leader completeness does not equal leader never returning old data

Raft's log safety guarantees that committed entries will not be overwritten by different values. However, client-visible linear consistency requires additional assurances:

• Each command must have a unique request ID so that after a leader election, the command is not re-applied;
• Reads must use a secure protocol;
• The state machine executes commands in the order they are committed and deterministically;
• Snapshots must be restored to the same state as the log during recovery;
• Clients must not bypass the cluster to directly access individual replicas.

Member Changes Cannot Simply Replace the List

A direct switch from the old configuration {A,B,C} to the new configuration {C,D,E} risks having two disjoint majority factions, each selecting its own leader.

Raft's original paper proposes joint consensus: during the transition phase, decisions require agreement from both the old and new configurations' majorities before the new configuration is finally committed. Specific systems may implement variants such as adding or removing one member at a time, but they must strictly adhere to their protocol and operational sequence.

Before performing a member change, the following checks must be conducted:

• Has the new node caught up to a sufficiently recent log tail?
• Are there already known failed members in the current cluster?
• How much quorum margin remains during the transition?
• Is node identity stable, and could an old data directory rejoin with incorrect identity?
• Could automation concurrently execute multiple configuration changes?

Snapshot Compression of Committed Prefixes

Log growth without bound slows down replay and fills disk space. A node can save the applied state and the last index/term contained in the log as a snapshot, then delete the prior log entries.

If a lagging Follower requires logs that have been compressed, the Leader sends a snapshot via InstallSnapshot and resumes replication from the point after the snapshot.

Snapshots must:

• Be atomically tied to a specific applied index;
• Contain all data required to restore the state machine;
• Be validated for integrity and safely replace the old state;
• Be coordinated with concurrent log appends;
• Not leave a gap (where logs are deleted but the snapshot is incomplete) during node crashes.

Snapshots are not business backups. They typically represent only the recoverable internal state of the cluster and do not replace cross-region backups, accidental deletion recovery, or historical data retention.

Disk is part of the consensus path

The majority network converges quickly, but disk fsync operations are slow, this means commit latency remains high. Full disk exhaustion, tail latency, firmware issues, and filesystem corruption can all render a node unable to participate in quorum.

Monitor the following:

• Leader transitions and duration of leaderless periods;
• Latency from propose to commit, and from commit to apply;
• The amount each Follower lags behind in matchIndex;
• fsync latency, disk space availability, and WAL errors;
• Time taken to generate, transfer, and restore snapshots;
• Membership changes and failure states.

The Usage Boundaries of Systems Like etcd

Raft is well-suited for replicating small, strongly consistent control-plane states, but it does not mean that simply dumping large objects or high-throughput data streams directly into etcd automatically delivers free consistency. Developers must adhere to specific product limitations regarding object size, storage capacity, compression, watch capabilities, and backup requirements.

Clients should leverage the transactional, revision, and watch semantics provided by the system itself, rather than inferring API reliability from the fact that the underlying protocol is Raft.

Relationship with Paxos and ZAB

Paxos family protocols and Raft both use majority intersection to achieve consensus, but they differ in abstraction and engineering structure; you cannot simply map Raft's roles and log rules onto every Paxos implementation.

ZooKeeper's ZAB is designed around leader-driven atomic broadcast and recovery phases, and its client semantics include sessions, watches, and ordering guarantees. Understanding a protocol builds a foundation, but the actual product behavior must be grounded in its own specifications.

Fault Testing

• The leader crashes after appending locally but before replicating;
• The system crashes after a majority of replicas have replicated, but before clients receive a response;
• The old leader is isolated, a new leader is elected, and then the old leader recovers;
• Followers' logs have conflicting suffixes;
• A snapshot is being written halfway when the process terminates;
• A node is lost during a membership change;
• The disk fills up or fsync has a long tail.

The trustworthiness of consensus implementations comes from model checking, mature codebases, and systematic fault testing, not from a simplified election algorithm.

References

• Ongaro, Ousterhout, Raft dissertation
• etcd, Learning design
• ZooKeeper, Consistency Guarantees