16.2 Cross-Shard Transactions, Clocks, and Geographical Distribution

A single-shard write only requires ordering within a single replication group. Cross-shard transfers, however, involve multiple independent logs, each shard may replicate healthily, yet the transaction might still only take effect in half the shards. Replication consensus and atomic commit address two distinct layers of challenges.

2PC Decides All or Nothing

Classic two-phase commit (2PC) consists of a coordinator and participants:

Prepare:
  coordinator -> participants: canCommit?
  participants: persist prepared state and reserve resources

Decision:
  all yes  -> persist COMMIT decision -> notify all
  any no   -> persist ABORT decision  -> notify all

After participants vote "yes," they cannot independently commit or rollback, they must wait for the final decision. As a result, traditional 2PC can block indefinitely in the face of coordinator failure or network partitioning, and may hold locks or intent states for extended periods. Modern databases address this by replicating coordination state, parallelizing intent writes, supporting recovery queries, or using consensus protocols to shorten the uncertainty window. However, the claim that "cross-shard transactions incur no coordination overhead" still does not hold.

2PC only guarantees atomicity of the decision, it does not ensure transaction isolation on its own. Systems must still rely on MVCC, locking, or timestamp-based protocols to manage concurrent reads and writes.

Transaction Records and Write Intent

A common implementation is to first write a provisional value/intent to the target key, pointing to the transaction record:

key A -> intent(value, txn-9)
key B -> intent(value, txn-9)
txn-9 -> PENDING / STAGING / COMMITTED / ABORTED

Other transactions encountering an intent can wait, increment their timestamp, trigger priority arbitration, or restore their state after confirming the original transaction has expired. After commit, the intent is resolved into a regular MVCC version; upon rollback, it is removed.

These metadata entries make fault recovery an integral part of the protocol, but they also explain why long-running transactions and high-conflict cross-shard writes amplify tail latency.

Why Serializable Still Requires Client Retries

Distributed databases execute read and write operations in parallel to reduce latency, then terminate certain transactions when conflicts arise or timestamps cannot be maintained. A successfully committed Serializable transaction guarantees serializability in effect, but this does not mean every attempt will succeed.

Applications should handle retryable errors as specified by the product and reexecute the transaction from its entry point:

begin
  read all decision inputs
  write provisional results
  commit
on retryable conflict:
  rollback
  bounded backoff + jitter
  retry the whole transaction

The larger the cross-shard transaction, the more ranges it involves, the greater the network round-trips, and the higher the potential for conflicts. Prioritize aligning business transactions with shard keys, and keep unavoidable cross-shard operations as concise as possible.

Why Physical Clocks Can't Be Used Directly to Order Transactions

Machine clocks drift over time, and NTP can only keep errors within a bounded range, it cannot guarantee that all nodes perceive the same instant. If we directly use the local now() as a global commit order, two regions might assign timestamps that reflect reversed actual timelines.

Common approaches include:

• Hybrid Logical Clocks: Combine physical time with logical counters to preserve causal monotonicity while bounding deviation from wall-clock time;
• TrueTime-style intervals: Return an uncertainty interval for [earliest, latest], and allow protocols to wait until the uncertainty resolves before making external commitments;
• Centralized or sharded timestamp services: Use a coordination service to assign comparable versions across nodes;
• Consensus log positions: Provide a natural ordering within a single-replica group.

Timestamp selection must be discussed in concert with the transaction protocol, MVCC, and fault model.

Spanner's key Goes Beyond "it uses atomic clocks"

TrueTime provides Spanner with time intervals that include error bounds. By combining read-write transactions with concurrency control, replication, and commit wait mechanisms, Spanner ensures that commit timestamps respect externally observable causality. MVCC enables consistent snapshots to be read at a specified timestamp.

External consistency goes beyond standard serializability by adding a real-time ordering constraint: if transaction T1 commits before transaction T2 begins, no observer may perceive T2 as occurring before T1.

This does not mean all global transactions are free from cross-region latency. Write latency still depends on replica placement, shard participation, and synchronization confirmation. Deployment decisions must balance write locality, read latency, and fault domain objectives.

CockroachDB's Range Perspective

CockroachDB divides its ordered key-value space into ranges, each replicated using Raft consensus. Typically, the leaseholder of a range is responsible for the most up-to-date consistent reads and writes. When a SQL transaction spans multiple ranges, the transaction layer coordinates write intents and transaction state to ensure global ACID semantics.

Older follower reads can be performed from local replicas as long as they occur before the closed timestamp; this provides a clearly defined, time-stamped snapshot of historical consistency, rather than reading "approximately fresh" data from asynchronous replicas alone.

As product implementations continue to evolve, the course should focus on the layered architecture: SQL → transaction → distribution and routing → range replication → local storage, rather than memorizing the exact number of RPC calls in a specific version.

Data Locality Is Delayed Design

Geographic distribution tables typically require choosing among:

• Single-region write leadership: local writes are fast, remote writes cross-region;
• Row or tenant-based regional assignment: local access is fast, but cross-tenant transactions are expensive;
• Global read replicas: reads are close, but more synchronized replicas mean slower writes;
• Bounded stale follower reads: reads are local and don’t block writes, but they return data from a historical point in time.

Design reviews should answer with actual transaction traces: which keys are accessed by a single request, where the leases and leaders for those keys reside, how many fault domains must be confirmed, and what is the worst-case cross-region round-trip time.

When Cross-Shard Transactions Should Not Be Used

If the business logic allows asynchronous completion, use local transactions combined with an outbox to publish facts, followed by idempotent consumers that update derived state. However, Saga or compensation mechanisms are not equivalent to ACID transactions: intermediate states become visible to the outside world, compensation may fail, and the business must explicitly define an observable state machine.

Irreversible constraints (such as balances, quotas, and uniqueness rules) should not be arbitrarily decomposed into eventually consistent messages simply for the sake of decoupling. First, validate that the business can tolerate intermediate states and the semantics of compensation.

References

• CockroachDB: Transaction Layer
• Cloud Spanner: TrueTime and External Consistency
• Cloud Spanner: Transactions

The next chapter continues along the boundary after transaction commit: how to reliably deliver changes to other systems and correctly handle duplicates, out-of-order events, and message backlog.