5.2 Compaction, Amplification Effects, and LSM Operations Maintenance
As sorted files accumulate, reads merge more sources and old versions keep occupying space. Compaction selects files, merges their contents, removes versions that can be safely reclaimed, and installs the new file set. This lesson uses amplification, disk headroom, and backlog to check whether background work can keep pace with writes. Waiting for the background to be idle can be a long wait.
Why Compaction Is Necessary
If you flush data without compaction:
- The number of files and runs keeps growing;
- Point lookups and range merges must check more sources;
- Tombstones and overwritten values consume unnecessary space;
- Pressure on open files, metadata, and cache increases;
- Restart, recovery, and backup enumeration become significantly heavier.
Compaction transforms the cost of "updating in place" into a background merge operation, but the overall cost still falls on the device throughput and CPU capacity.
Leveled compaction
A typical leveled layout:
L0: overlapping flush files
L1: sorted files, key ranges non-overlapping within level
L2: larger total target, non-overlapping within level
...When a level exceeds its size/file target, the picker selects an input range and merges files that overlap with the next level. The advantages typically include fewer point lookup candidates at lower levels and controlled space amplification; the cost is that data may be rewritten multiple times across levels.
Level size multiplier, file size, and L0 triggers are configuration strategies, not fixed, version-agnostic defaults like "10x multiplier, 64 MB." These values are determined by the current engine options and live metadata.
Tiered / size-tiered / universal
This strategy allows multiple sorted runs to coexist within the same key space, merging them when size, age, or count thresholds are reached. It typically reduces redundant data movement and performs well under write-heavy workloads, but reads must traverse more runs. Older versions and tombstone entries may persist for longer periods.
"Tiered" is not a single, universally applied algorithm across all products. The compaction strategies in Cassandra's SizeTieredCompactionStrategy, RocksDB's Universal Compaction, and other engines use different selection and space allocation rules. Therefore, a threshold or configuration from one product cannot be directly applied to another.
FIFO and Time-Window
Time-series workloads may suit FIFO or time-window strategies, but their deletion conditions differ. FIFO can remove the oldest files to meet a total size limit, even when their business data has not expired. Time-window compaction combined with TTL may allow whole expired files to be reclaimed. Check whether that eviction policy is acceptable, then examine out-of-order writes and window boundaries.
Late-arriving data, cross-window range queries, updates to existing keys, and legal retention policies can all alter the design.
Three Types of Amplification
Write Amplification
physical bytes written by storage stack / logical user bytes writtenDefine the numerator explicitly: counting only SST compaction writes, adding WAL, or also including filesystem and device writes gives different ratios. Compression may reduce physical bytes below the logical payload, but this does not imply the absence of compaction-related CPU usage or read overhead.
Read Amplification
Can be defined in terms of files, blocks, probes, bytes, or I/O operations. The amplification for point negative lookups, point positive lookups, and range scans differs significantly, so a single "level" metric cannot universally represent all types of requests.
Space Amplification
physical live storage / logical latest live dataState whether physical storage in the numerator includes WAL, obsolete-but-not-deleted files, snapshots, tombstones, replicas, or backups. Reporting only a ratio without specifying the measurement scope renders the metric non-comparable.
These objectives are interdependent but do not follow a fixed triangular relationship. Workload characteristics, compression ratios, key distribution, retention policies, and the underlying storage device all influence the frontier between logical and physical storage.
Compaction Debt and Steady State
Let logical ingest be W bytes/s and the average compaction write amplification be A. The bandwidth demand solely for compaction scales as:
W × AHere A counts compaction output bytes divided by logical input bytes. With an assumed W = 20 MB/s and A = 6, compaction output alone needs 120 MB/s. This is a capacity calculation, excluding WAL, flushes, reads, and filesystem overhead. If the underlying storage device's sustained throughput for background operations falls below this demand, pending compaction bytes will accumulate and eventually lead to slowdown or stall.
Bursts can be temporarily buffered in memtables or L0 levels, but steady-state performance cannot be sustained by an ever-growing backlog. Capacity planning must focus on stable windows of several hours or days, rather than one-minute peak benchmarks alone.
Write stall is a protection mechanism
Common triggers:
- Too many immutable memtables;
- L0 file count or size exceeding thresholds;
- Pending compaction bytes surpassing limits;
- WAL or space limits reached;
- Background threads or I/O unable to keep up.
Slowdowns and stalls bound memory use and read amplification. The real issue lies in why compaction capacity falls short of ingestion rate: device saturation, CPU-bound compression, poor compaction configuration, oversized value sizes, snapshot retention policies, or contention over shared resources.
Applications must implement backpressure and deadlines. Unbounded retries will exacerbate stalls.
Tombstone and Snapshot Retention
Compaction can only remove tombstones when it is proven that older values are no longer visible to any relevant readers. Long-lived snapshots, transactions, iterators, replication lag, or backup pins can extend retention periods.
Symptoms:
- Space does not decrease after a logical delete;
- High read/write bytes during compaction;
- Point or range reads scan large numbers of old versions;
- The age of the oldest snapshot continues to increase.
Performing a manual full compaction immediately after deleting large amounts of data may trigger an I/O storm. First, verify the retention blockers and business window, then choose between range or manual compaction.
Bloom and cache cannot replace compaction
A Bloom filter can avoid some negative point reads, but it cannot:
- Remove overwritten values;
- Recycle tombstones;
- Reduce all costs associated with range merges;
- Decrease space occupied by obsolete files;
- Resolve compaction debt.
Block cache can hide some I/O overhead, but an excessive number of files or runs still consumes metadata, CPU, iterators, and cache space.
B+ Tree and LSM Are Not a "Read-Heavy vs. Write-Heavy" Binary Choice
When evaluating storage engines, at least the following dimensions should be compared:
| Dimension | Questions to Ask |
|---|---|
| Write | Sustained vs. burst write rate, update distribution, durability, value size |
| Point read | Positive-to-negative request ratio, tail latency, cache budget |
| Range read | Range width, sort order, snapshot support, merge sources |
| Space | Compression efficiency, TTL settings, tombstone handling, temporary compaction headroom |
| Consistency | Transaction model, secondary indexes, constraint enforcement |
| Operations | Backup capabilities, repair procedures, upgrade path, observability, stall behavior |
| Hardware | Local SSD, network block or object storage, IOPS, bandwidth, endurance |
A B+ tree engine achieves high write throughput through mechanisms like WAL (Write-Ahead Logging), buffering, group commits, and sequential I/O. Meanwhile, LSM engines can deliver excellent read performance via filtering, caching, and compaction strategies. Ultimately, performance decisions must be validated using representative workloads.
Operational metrics
General metrics:
logical write/read rate and latency percentiles
WAL bytes and sync latency
mutable/immutable memtable bytes/count
flush throughput and duration
files/bytes per level or run
pending compaction bytes/debt
compaction read/write bytes and CPU
write slowdown/stall duration
block cache hit by block type
Bloom useful/positive/false-positive counters
tombstone/obsolete versions and oldest snapshot age
disk free headroom and temporary compaction spaceMetric names may vary by engine or version; always refer to the current official property/statistics definitions. Viewing estimate-pending-compaction-bytes alone does not equate to write amplification.
Benchmark Design
- Data volume exceeds memory cache size, or the test explicitly reports cache-resident conditions;
- Pre-populate data and wait/record compaction state;
- Use real key/value sizes and actual update/insert/delete ratios;
- Run point and range reads concurrently with writes;
- Continue until the system reaches compaction steady state;
- Report p50, p95, and p99 latency, stall duration, physical bytes written, and storage space usage;
- Log engine version, configuration options, and hardware specifications;
- Validate durability and recovery time after crash and reopen.
Writing to an empty database for one minute primarily measures memtable and WAL burst behavior. It cannot establish the long-term capacity of the storage design.
Order of Operational Changes
- First confirm that the bottleneck is compaction, not application/OS/device workloads;
- Modify a single primary option;
- Maintain sufficient free space to accommodate the worst-case compaction scenario;
- Limit the scope and concurrency of manual compactions;
- Monitor read tail latency and write stall guardrails;
- Gradually roll out to a small number of shards or instances;
- Preserve the ability to roll back, but be aware that certain on-disk format options may not support lossless reversion.
Chapter Summary
The advantages of LSM trees come from batching, immutable sorted runs, and background merging. The costs include compaction debt, data amplification, and version retention. When selecting an LSM-based storage system or tuning its parameters, one should not rely solely on the claim that "sequential writes are faster." Instead, the design must be validated to ensure that steady-state ingestion, read latency, space efficiency, and recovery performance all meet their respective targets.