4.3 From HDFS to Spanner: Workload Determines Architecture
Both HDFS and Spanner store data across machines, but they serve different purposes: HDFS is designed for large files and high-throughput sequential access, while Spanner is built for global sharding, synchronous replication, and external consistency in transactions. Product scale cannot replace workloads analysis.
Separate HDFS Metadata and Data Paths
The NameNode maintains the namespace, the mapping from files to blocks, and replica status; DataNodes store the actual blocks. Clients first query metadata, then directly transfer data to DataNodes.
Client ──metadata──> NameNode
│
└──block stream──> DataNode A → DataNode B → DataNode CLarge blocks reduce metadata overhead and addressing cost, pipelined replication increases write throughput, and rack-aware replica placement balances network cost against failure domain.
HDFS is well-suited for large files, batch processing, and streaming read/write operations, but it's not ideal for massive numbers of small files, low-latency random updates, or general POSIX semantics. Specific block sizes and replica counts are determined by deployment configurations and should not be treated as universal protocol rules.
Metadata high availability still requires coordination
An Active/Standby NameNode requires shared edit logs or a consistent logging mechanism, fault detection, and fencing. Simply detecting an unhealthy Active node isn't enough, we must prevent the old Active from continuing to write to shared resources, otherwise we risk split-brain.
DataNodes periodically report block status, and the NameNode performs block replication based on missing replicas, excess replicas, and failure domains. Recovery traffic must be throttled to prevent replication storms triggered by node failures from impacting front-running jobs.
Spanner combines sharding, replication, and a time protocol
Spanner shards data by key range, with replicas of each shard synchronized via consensus; the transaction layer coordinates cross-shard reads and writes, and TrueTime exposes an uncertainty interval of physical time to support external consistency and historical time queries.
The key point isn't "we have GPS/atomic clocks so the database is fast," but rather:
- The copy set determines write order by consensus;
- Transactions require concurrency control and cross-shard coordination;
- The Time API exposes error as a range;
- The protocol uses methods like waiting to ensure the submission timestamp respects the actual time order.
These guarantees will cover the costs of cross-regional network, consensus, and submission wait times. The schema and key design must still avoid hot-spot ranges.
Choose Write-Through Storage Access Mode
| Requirement | Preference |
|---|---|
| Ordered scanning of large files at PB scale, batch jobs | Distributed file/object storage |
| High write throughput, partition-key access, tunable consistency | Leaderless wide-column/key-value system |
| Global affairs, relationship queries, external consistency | Distributed SQL with coordination overhead |
| Small-scale strongly consistent metadata | Consensus KV, no large objects |
Also clearly defined: object size, request rate, range queries, transaction scope, read freshness, region, RPO/RTO, cost, and operational capabilities.
Backups vs. Copies
Replicas will sync application misdeletions, erroneous writes, and application bugs. Backups need independent fault domains, retention periods, immutability, and recovery drills.
You must prove it for any storage system.
- To which point can it be restored;
- How long until recovery;
- Are the encryption key and schema recoverable together?
- How does restoring the environment prevent overwriting production;
- When was the last full-scale drill?
References
- Google Research, The Google File System
- Google Research, Spanner: Google's Globally-Distributed Database