4.3 From HDFS to Spanner: Workload Determines Architecture
Both HDFS and Spanner store data across machines, but they serve different purposes: HDFS is designed for large files and high-throughput sequential access, while Spanner supports global sharding, synchronous replication, and externally consistent transactions under its default serializable isolation. Product scale cannot replace workload analysis.
Separate HDFS Metadata and Data Paths
The NameNode maintains the namespace, the mapping from files to blocks, and replica status; DataNodes store the actual blocks. Clients first query metadata, then transfer data directly to DataNodes. The diagram shows a replicated file’s write pipeline. Reads normally choose a suitable replica and retry another on failure; they do not traverse the entire pipeline.
Client ──metadata──> NameNode
│
└──block stream──> DataNode A → DataNode B → DataNode CLarge blocks reduce metadata overhead and addressing cost, pipelined replication increases write throughput, and rack-aware replica placement balances network cost against failure domain.
HDFS suits large files, batch processing, and sequential streaming I/O. A file normally has one writer and supports appends rather than arbitrary in-place modification. Numerous small files consume NameNode memory for file and block metadata. HDFS does not target low-latency random updates or full general-purpose POSIX semantics. Erasure-coded files also use a different layout from the replication pipeline shown here. Specific block sizes and replica counts are determined by deployment configurations and should not be treated as universal protocol rules.
Metadata high availability still requires coordination
An Active/Standby NameNode requires shared edit logs or a consistent logging mechanism, fault detection, and fencing. With Quorum Journal Manager, for example, the Active writes edits to a JournalNode majority, and the Standby catches up before safely taking over. When automatic failover is configured, ZKFC coordinates election through ZooKeeper. Detecting an unhealthy Active is insufficient: fencing must prevent the old instance from continuing unsafe service. "I am still alive" cannot be the two Actives’ only point of agreement.
DataNodes periodically report block status, and the NameNode schedules replication for under-replicated blocks and removal of excess replicas, accounting for failure-domain placement. Recovery traffic must be throttled to prevent replication storms triggered by node failures from impacting foreground jobs.
Spanner combines sharding, replication, and a time protocol
Spanner shards data by key range, and replica groups use consensus to acknowledge writes. Replicas need not be fully caught up at every instant. The transaction layer coordinates cross-shard reads and writes, while TrueTime exposes physical-time uncertainty. Together with mechanisms such as MVCC, these support external consistency and historical reads.
The key point isn't "we have GPS/atomic clocks so the database is fast," but rather:
- Replica groups determine write order through consensus;
- Transactions require concurrency control and cross-shard coordination;
- The time API exposes uncertainty as an interval;
- The protocol uses methods like waiting to ensure the commit timestamp respects the actual time order.
For example, after choosing commit timestamp s, commit wait delays external completion until TrueTime can establish that real time has passed s. Better clock synchronization alone cannot provide that ordering guarantee; cross-shard transactions also need atomic commit across participating groups.
These guarantees incur the relevant network, consensus, and commit-wait costs. The schema and key design must still avoid hot-spot ranges.
Define Access Patterns Before Choosing Storage
| Requirement | Preference |
|---|---|
| Ordered scanning of large files at PB scale, batch jobs | Distributed file/object storage |
| High write throughput, partition-key access, tunable consistency | Leaderless wide-column/key-value system |
| Global transactions, relational queries, external consistency | Distributed SQL with coordination overhead |
| Small-scale strongly consistent metadata | Consensus KV, no large objects |
Also specify object size, request rate, range queries, transaction scope, read freshness, region, RPO/RTO, cost, and operational capabilities.
Backups and Replicas Serve Different Purposes
Replicas propagate accidental deletions, incorrect writes, and application bugs. Backups need independent fault domains, retention periods, immutability, and recovery drills.
For any storage system, establish:
- Which point in time can be restored;
- How long recovery takes;
- Are the encryption key and schema recoverable together?
- How the recovery environment avoids overwriting production;
- When was the last full-scale drill?
References
- Apache Hadoop, HDFS Architecture
- Apache Hadoop, HDFS High Availability Using QJM
- Google Cloud, TrueTime and external consistency
- Google Cloud, Strict serializability and external consistency in Spanner