5.2 Compaction, Amplification Effects, and LSM Operations Maintenance
New sorted files continuously accumulate in the storage warehouse, and read operations begin spanning deeper and deeper layers. If background compaction runs unchecked, they can consume the bandwidth intended for foreground writes.
Compaction is not a "background tidy-up task." It is the core ongoing operation of LSM trees: selecting overlapping sorted runs, merging them into new files, discarding versions and tombstones that have safely expired, and atomically installing the new version. It directly determines steady-state write capacity, read latency, and space usage.
Why Compaction Is Necessary
If you flush data without compaction:
- The number of files and runs keeps growing;
- Point lookups and range merges must check more sources;
- Tombstones and overwritten values consume unnecessary space;
- Pressure on open files, metadata, and cache increases;
- Restart, recovery, and backup enumeration become significantly heavier.
Compaction transforms the cost of "updating in place" into a background merge operation, but the overall cost still falls on the device throughput and CPU capacity.
Leveled compaction
A typical leveled layout:
L0: overlapping flush files
L1: sorted files, key ranges non-overlapping within level
L2: larger total target, non-overlapping within level
...When a level exceeds its size/file target, the picker selects an input range and merges files that overlap with the next level. The advantages typically include fewer point lookup candidates at lower levels and controlled space amplification; the cost is that data may be rewritten multiple times across levels.
Level size multiplier, file size, and L0 triggers are configuration strategies, not fixed, version-agnostic defaults like "10x multiplier, 64 MB." These values are determined by the current engine options and live metadata.
Tiered / size-tiered / universal
This strategy allows multiple sorted runs to coexist within the same key space, merging them when size, age, or count thresholds are reached. It typically reduces redundant data movement and performs well under write-heavy workloads, but reads must traverse more runs. Older versions and tombstone entries may persist for longer periods.
"Tiered" is not a single, universally applied algorithm across all products. The compaction strategies in Cassandra's SizeTieredCompactionStrategy, RocksDB's Universal Compaction, and other engines use different selection and space allocation rules. Therefore, a threshold or configuration from one product cannot be directly applied to another.
FIFO and Time-Window
TTL/time-series workloads can use either FIFO or time-window strategies to delete entire batches of expired files, preventing the repeated merging of data that is destined to expire. However, this approach is only valid when query and time ordering, TTL settings, and out-of-order writes meet certain prerequisites.
Late-arriving data, cross-window range queries, updates to existing keys, and legal retention policies can all alter the design.
Three Types of Amplification
Write Amplification
physical bytes written by storage stack / logical user bytes writtenThe numerator must be clearly defined: only SST compaction is excluded, while WAL and filesystem/device write amplification are included. The resulting value will vary depending on what's counted. Compression may reduce physical bytes below the logical payload, but this does not imply the absence of compaction-related CPU usage or read overhead.
Read Amplification
Can be defined in terms of files, blocks, probes, bytes, or I/O operations. The amplification for point negative lookups, point positive lookups, and range scans differs significantly, so a single "level" metric cannot universally represent all types of requests.
Space Amplification
physical live storage / logical latest live dataIt's essential to clarify whether the denominator includes WAL, obsolete-but-not-deleted files, snapshots, tombstones, replicas, or backups. Reporting only a ratio without specifying the measurement scope renders the metric non-comparable.
These objectives are interdependent but do not follow a fixed triangular relationship. Workload characteristics, compression ratios, key distribution, retention policies, and the underlying storage device all influence the frontier between logical and physical storage.
Compaction Debt and Steady State
Let logical ingest be W bytes/s and the average compaction write amplification be A. The bandwidth demand solely for compaction scales as:
W × AThis does not include WAL, flush, read, or filesystem overhead. If the underlying storage device's sustained throughput for background operations falls below this demand, pending compaction bytes will accumulate and eventually lead to slowdown or stall.
Bursts can be temporarily buffered in memtables or L0 levels, but steady-state performance cannot be sustained by an ever-growing backlog. Capacity planning must focus on stable windows of several hours or days, rather than one-minute peak benchmarks alone.
Write stall is a protection mechanism
Common triggers:
- Too many immutable memtables;
- L0 file count or size exceeding thresholds;
- Pending compaction bytes surpassing limits;
- WAL or space limits reached;
- Background threads or I/O unable to keep up.
The slowdown or stall mechanism protects memory and read amplification, rather than an "LSM tree defect switch alone." The real issue lies in why compaction capacity falls short of ingestion rate: device saturation, CPU-bound compression, poor compaction configuration, oversized value sizes, snapshot retention policies, or contention over shared resources.
Applications must implement backpressure and deadlines. Unbounded retries will exacerbate stalls.
Tombstone and Snapshot Retention
Compaction can only remove tombstones when it is proven that older values are no longer visible to any relevant readers. Long-lived snapshots, transactions, iterators, replication lag, or backup pins can extend retention periods.
Symptoms:
- Space does not decrease after a logical delete;
- High read/write bytes during compaction;
- Point or range reads scan large numbers of old versions;
- The age of the oldest snapshot continues to increase.
Performing a manual full compaction immediately after deleting large amounts of data may trigger an I/O storm. First, verify the retention blockers and business window, then choose between range or manual compaction.
Bloom and cache cannot replace compaction
A Bloom filter can avoid some negative point reads, but it cannot:
- Remove overwritten values;
- Recycle tombstones;
- Reduce all costs associated with range merges;
- Decrease space occupied by obsolete files;
- Resolve compaction debt.
Block cache can hide some I/O overhead, but an excessive number of files or runs still consumes metadata, CPU, iterators, and cache space.
B+ Tree and LSM Are Not a "Read-Heavy vs. Write-Heavy" Binary Choice
When evaluating storage engines, at least the following dimensions should be compared:
| Dimension | Questions to Ask |
|---|---|
| Write | Sustained vs. burst write rate, update distribution, durability, value size |
| Point read | Positive-to-negative request ratio, tail latency, cache budget |
| Range read | Range width, sort order, snapshot support, merge sources |
| Space | Compression efficiency, TTL settings, tombstone handling, temporary compaction headroom |
| Consistency | Transaction model, secondary indexes, constraint enforcement |
| Operations | Backup capabilities, repair procedures, upgrade path, observability, stall behavior |
| Hardware | Local SSD, network block or object storage, IOPS, bandwidth, endurance |
A B+ tree engine achieves high write throughput through mechanisms like WAL (Write-Ahead Logging), buffering, group commits, and sequential I/O. Meanwhile, LSM engines can deliver excellent read performance via filtering, caching, and compaction strategies. Ultimately, performance decisions must be validated using representative workloads.
Operational metrics
General metrics:
logical write/read rate and latency percentiles
WAL bytes and sync latency
mutable/immutable memtable bytes/count
flush throughput and duration
files/bytes per level or run
pending compaction bytes/debt
compaction read/write bytes and CPU
write slowdown/stall duration
block cache hit by block type
Bloom useful/positive/false-positive counters
tombstone/obsolete versions and oldest snapshot age
disk free headroom and temporary compaction spaceMetric names may vary by engine or version; always refer to the current official property/statistics definitions. Viewing estimate-pending-compaction-bytes alone does not equate to write amplification.
Benchmark Design
- Data volume exceeds memory cache size, or the test explicitly reports cache-resident conditions;
- Pre-populate data and wait/record compaction state;
- Use real key/value sizes and actual update/insert/delete ratios;
- Run point and range reads concurrently with writes;
- Continue until the system reaches compaction steady state;
- Report p50, p95, and p99 latency, stall duration, physical bytes written, and storage space usage;
- Log engine version, configuration options, and hardware specifications;
- Validate durability and recovery time after crash and reopen.
Comparing performance under "empty database with continuous writes for one minute" primarily measures memtable and WAL burst behavior, this is insufficient to evaluate the long-term performance and resilience of the storage architecture.
Safety Change Order
- First confirm that the bottleneck is compaction, not application/OS/device workloads;
- Modify a single primary option;
- Maintain sufficient free space to accommodate the worst-case compaction scenario;
- Limit the scope and concurrency of manual compactions;
- Monitor read tail latency and write stall guardrails;
- Gradually roll out to a small number of shards or instances;
- Preserve the ability to roll back, but be aware that certain on-disk format options may not support lossless reversion.
Chapter Summary
The advantages of LSM trees come from batching, immutable sorted runs, and background merging. The costs include compaction debt, data amplification, and version retention. When selecting an LSM-based storage system or tuning its parameters, one should not rely solely on the claim that "sequential writes are faster." Instead, the design must be validated to ensure that steady-state ingestion, read latency, space efficiency, and recovery performance all meet their respective targets.