16.2 Cross-Shard Transactions, Clocks, and Geographical Distribution

A single-shard write only requires ordering within a single replication group. Cross-shard transfers, however, involve multiple independent logs. Each shard may replicate healthily, yet the transaction might still only take effect in half the shards. Replication consensus and atomic commit address two distinct layers of challenges.

2PC Decides All or Nothing

Classic two-phase commit (2PC) consists of a coordinator and participants:

Prepare:
  coordinator -> participants: canCommit?
  participants: persist prepared state and reserve resources

Decision:
  all yes  -> persist COMMIT decision -> notify all
  any no   -> persist ABORT decision  -> notify all

After participants vote "yes," they cannot independently commit or roll back; they must wait for the final decision. As a result, traditional 2PC can block indefinitely in the face of coordinator failure or network partitioning, and may hold locks or intent states for extended periods. Modern databases address this by replicating coordination state, parallelizing intent writes, supporting recovery queries, or using consensus protocols to shorten the uncertainty window. However, the claim that "cross-shard transactions incur no coordination overhead" still does not hold.

Suppose the coordinator persists COMMIT, notifies A, and crashes while B remains prepared. B cannot choose ABORT on a timeout without contradicting A. Recovery must retrieve the original decision and notify B again. Cleanup can finish at different times on different shards; the read protocol must prevent exposing partial results as a complete transaction.

2PC guarantees an atomic decision; it does not ensure transaction isolation on its own. Systems must still rely on MVCC, locking, or timestamp-based protocols to manage concurrent reads and writes.

Transaction Records and Write Intent

A CockroachDB-style implementation writes provisional values or intents and associates them with a transaction record. The state names below describe that implementation family, not a universal 2PC state machine:

key A -> intent(value, txn-9)
key B -> intent(value, txn-9)
txn-9 -> PENDING / STAGING / COMMITTED / ABORTED

Other transactions encountering an intent can wait, increment their timestamp, trigger priority arbitration, or determine the original transaction's status through recovery. A client timeout is not proof of abort, particularly when STAGING requires checking in-flight writes. After commit, the intent is resolved into a regular MVCC version; upon rollback, it is removed.

These metadata entries make fault recovery an integral part of the protocol, but they also explain why long-running transactions and high-conflict cross-shard writes amplify tail latency.

Why Serializable Still Requires Client Retries

Distributed databases execute read and write operations in parallel to reduce latency, then terminate certain transactions when conflicts arise or timestamps cannot be maintained. A successfully committed Serializable transaction guarantees serializability in effect, but this does not mean every attempt will succeed.

Applications should handle retryable errors as specified by the product and reexecute the transaction from its entry point:

begin
  read all decision inputs
  write provisional results
  commit
on retryable conflict:
  rollback
  bounded backoff + jitter
  retry the whole transaction

The larger the cross-shard transaction, the more ranges it involves, the greater the network round-trips, and the higher the potential for conflicts. Prioritize aligning business transactions with shard keys, and keep unavoidable cross-shard operations as concise as possible.

Why Physical Clocks Can't Be Used Directly to Order Transactions

Machine clocks drift over time, and NTP can discipline clocks, but disconnection, clock adjustments, and faults can widen the error; it offers no unconditional global error bound. If we directly use the local now() as a global commit order, two regions might assign timestamps that reflect reversed actual timelines.

Common approaches include:

• Hybrid Logical Clocks: Combine physical time with logical counters to preserve causal monotonicity through message exchange; bounding wall-clock deviation also needs synchronization, monitoring, and a response to excessive skew;
• TrueTime-style intervals: Return an uncertainty interval for [earliest, latest], and allow protocols to wait until the uncertainty resolves before making external commitments;
• Centralized or sharded timestamp services: Use a coordination service to assign comparable versions across nodes;
• Consensus log positions: Provide a natural ordering within a single replication group.

Timestamp selection must be discussed in concert with the transaction protocol, MVCC, and fault model.

Spanner Needs More Than Atomic Clocks

TrueTime provides Spanner with time intervals that include error bounds. By combining read-write transactions with concurrency control, replication, and commit wait mechanisms, Spanner ensures that commit timestamps respect externally observable real-time order. MVCC enables consistent snapshots to be read at a specified timestamp.

External consistency goes beyond standard serializability by adding a real-time ordering constraint: if transaction T1 commits before transaction T2 begins, no observer may perceive T2 as occurring before T1.

This does not mean all global transactions are free from cross-region latency. Write latency still depends on replica placement, shard participation, and synchronization confirmation. Deployment decisions must balance write locality, read latency, and fault domain objectives.

CockroachDB's Range Perspective

CockroachDB divides its ordered key-value space into ranges, each replicated using Raft consensus. Typically, the leaseholder of a range is responsible for the most up-to-date consistent reads and writes. When a SQL transaction spans multiple ranges, the transaction layer coordinates write intents and transaction state to ensure global ACID semantics.

Older follower reads can be performed from local replicas at timestamps no later than the closed timestamp, once the local replica has applied the relevant state; this provides a clearly defined, time-stamped snapshot of historical consistency, rather than reading "approximately fresh" data from asynchronous replicas alone.

As product implementations continue to evolve, the course should focus on the layered architecture: SQL → transaction → distribution and routing → range replication → local storage, rather than memorizing the exact number of RPC calls in a specific version.

Data Locality Is Latency Design

Geographic distribution tables typically require choosing among:

• Single-region write leadership: local writes are fast, remote writes cross-region;
• Row or tenant-based regional assignment: local access is fast, but transactions across regional ownership are expensive;
• Global read replicas: reads are close, but more synchronized replicas mean slower writes;
• Bounded stale follower reads: reads are local and don’t block writes, but they return data from a historical point in time.

Design reviews should answer with actual transaction traces: which keys are accessed by a single request, where the leases and leaders for those keys reside, how many fault domains must be confirmed, and what is the worst-case cross-region round-trip time.

When Cross-Shard Transactions Should Not Be Used

If the business logic allows asynchronous completion, use local transactions combined with an outbox to publish facts, followed by idempotent consumers that update derived state. However, Saga or compensation mechanisms are not equivalent to ACID transactions: intermediate states become visible to the outside world, compensation may fail, and the business must explicitly define an observable state machine.

Irreversible constraints (such as balances, quotas, and uniqueness rules) should not be arbitrarily decomposed into eventually consistent messages simply for the sake of decoupling. First, validate that the business can tolerate intermediate states and the semantics of compensation.

References

• CockroachDB: Transaction Layer
• Cloud Spanner: TrueTime and External Consistency
• Cloud Spanner: Transactions

The next chapter continues along the boundary after transaction commit: how to reliably deliver changes to other systems and correctly handle duplicates, out-of-order events, and message backlog.