15.2 CAP, Read-Write Consistency, and Degradation Boundaries
The CAP theorem is often depicted as a triangle, leading people to believe that databases can freely choose two out of three properties under normal conditions. In reality, the correct interpretation is that when a network partition occurs and messages across the partition cannot reach their intended destinations, a system cannot guarantee both linearizability for all reads and writes and eventual completion of every request reaching a non-faulty node. The theorem does not assign a millisecond deadline to eventual completion.
What Do the Three Letters Mean?
- Consistency: Here, this typically refers to linearizability, every operation appears to happen atomically between the call and the return, and all clients observe the same real-time order. This is not the "C" in ACID, nor does it mean "replicas eventually converge."
- Availability: Every request sent to a non-faulty node eventually receives a non-error response, but the response does not guarantee it contains the most recent write.
- Partition tolerance: The system continues to operate according to specification even when messages between nodes are arbitrarily lost or delayed.
As long as a system is deployed across a network, it's impossible to rely on "choosing CA" to eliminate partitions. In practice, the design must answer: when a partition occurs, which operations must maintain consistency and reject service, and which can return stale values or accept writes that will be merged later.
CP and AP Are Operational Strategies, Not Permanent Labels
Suppose an inventory replica group has three replicas, with two separated from the third by a network partition.
To maintain linearizable inventory deduction, one side must verify it still holds a valid majority, or a lease meeting its protocol's clock and fencing conditions; if it cannot confirm this, the replica refuses to write. This sacrifices availability during partitioning to prevent both sides from selling the last item.
The pair can form a majority; the isolated replica cannot. A group with only two voting replicas requires both for a majority, so neither isolated replica has majority write authority.
If "like" operations allow both sides to continue accepting requests during a partition, and the system can later merge counts or aggregate events upon recovery, availability is preserved to a greater extent. However, clients during the partition may not see the latest results from the other side.
The same system can apply a CP strategy to login permissions and an AP strategy to view counters. A product name cannot substitute for clear, semantic documentation of each API's behavior.
Consistency Goes Beyond Strong or Eventually Consistent
Common session or ordering guarantees in applications include:
- Read-your-writes: A user can read the value they just wrote;
- Monotonic reads: Within a single session, a user never sees a newer version and then reverts to an older one;
- Monotonic writes: Writes within a session occur in a strictly ordered sequence;
- Causal consistency: The observed order of causally related operations is consistent across all participants;
- Bounded staleness: Reads are guaranteed not to be more than a specified time or version lag behind the current state.
These guarantees can be less expensive than global linearizability while still offering a better user experience than unconstrained eventual consistency. Interfaces should explicitly define their scope: single key, single partition, within the same region, or global.
Quorum Intersection Is Just a Basic Condition
Within one fixed replica set for the same key, with N replicas, if a write waits for W responses and a read queries R replicas, R + W > N ensures that the write and read sets have at least one intersection. A sloppy quorum using substitute replicas may not satisfy this fixed-set premise; membership changes also need their own protocol. Intersection alone does not establish linearizability; mechanisms such as version selection, conflict resolution during concurrent writes, failure recovery, read repair, or consensus protocols are still required.
Cassandra's Consistency Level determines how many replicas the coordinator waits for before acknowledging a write; the write is still sent to the target replicas. QUORUM, LOCAL_QUORUM, and ONE may have different latencies and failure domains, especially across data centers, making it insufficient to rely solely on numerical values.
Beyond Latency and Partitioning: PACELC
Even without complete network partitioning, systems continually face trade-offs between latency and consistency: synchronously waiting for additional replicas enhances confirmation strength but increases tail latency and sensitivity to slow replicas. PACELC distinguishes the availability-versus-consistency choice during partitions from the latency-versus-consistency choice otherwise, highlighting that CAP only describes failure boundaries, not a complete decision framework.
Conflict Resolution Must Align with Business Algebra
"Last Write Wins" is simple, but treating physical or client timestamps as final arbiters can let clock skew and late writes override correct data. A more reliable approach depends on whether data can be merged:
- Collections can use deduplication events or collection CRDTs (Conflict-Free Replicated Data Types);
- Grow-only counts can keep monotonic per-replica components, merge by componentwise maximum, and read by summing those components;
- Individual fields in user profiles can be merged at the field level;
- Invariants like balances or unique usernames that cannot be arbitrarily merged typically require a single serialized point or a consensus transaction.
For example, grow-only counter states (2,0) and (0,3) merge to (2,3), whose value is 5. Receiving (0,3) again keeps it at 5; adding observed totals would count twice. This state-based CRDT merge needs commutativity, associativity, and idempotency. Operation-based replication may instead rely on causal ordering and deduplicated delivery; an arbitrary JSON merge does not supply either protocol.
Put Semantics into API Contracts
Every critical read-write interface should answer these questions:
- What does a successful response mean, how many replicas were written, and has it been durably persisted?
- From where should subsequent reads be made to satisfy read-your-writes?
- In the event of a network partition or consensus lacking a majority, does the system fail, time out, or accept pending writes for later reconciliation?
- Can retries result in duplicates? How long are idempotent keys retained?
- Who detects conflicts, how are they merged, and is the user notified?
These answers are more instructive than simply stating "our database is AP" when designing client implementations or conducting failure scenarios.
A Business-Categorized Example
| Data | Strategy During a Partition | Reason |
|---|---|---|
| Payment ledger | Reject on uncertain write authority | Prevents double-spending or arbitrary merging |
| Product descriptions | Accept local stale reads; writes go through authoritative regions | Stale read cost is manageable |
| Like events | Accept locally, deduplicate, asynchronously merge | Operations are commutative and idempotent |
| Access control policies | Default to rejection or use time-limited verified snapshots | Risk of unauthorized access is high |
"Degradation" must be designed and tested before a partition occurs, never decided reactively after a failure.
Section Checkpoint
- Can explain that CAP applies only under network partition conditions, constraining consistency and availability in such scenarios.
- Can distinguish between linearizability, ACID consistency, and eventual convergence.
- Will not assert system linearizability solely based on
R + W > N. - Can define the success, failure, and merge semantics for specific business operations under failure conditions.
References
The next chapter moves into the implementation layer: how data is sharded, how replica groups elect leaders, and how cross-shard transactions are committed.