19.2 Cache Breaks, Penetration, Cascading Failures, and Cache Operations

Cache outages rarely stem from Redis slowing down on a single node, they typically expose the underlying capacity gaps in the backend services that were previously hidden by caching. Design goals go beyond high hit rates; they must also ensure that cache misses are bounded, aggregated, and gracefully degraded.

Breaking: A Hot Key Hits the Source Simultaneously

When a hot key expires or is evicted, a surge of concurrent requests simultaneously detects a cache miss and all attempt to query the origin server. This phenomenon is known as a cache stampede.

Single-Process Request Consolidation

Only one loader is allowed to execute for a given key at a time; all other requests wait for the result:

singleflight(key):
  first caller -> load source -> fill cache -> wake waiters
  followers    -> await same result

It's essential to limit the number of waiting requests, total timeout duration, and propagate cancellation signals. If the loader fails, no request should be left indefinitely queued.

Cross-Instance Mutual Exclusion

Distributed locks can reduce the likelihood of multiple instances simultaneously hitting the source. However, they must include expiration time and a unique owner token. The lock must be released via an atomic compare-and-delete operation to prevent older holders from accidentally deleting a newer lock. Lock leases may expire during slow queries, so source updates still require version checks or fencing tokens. Cache rebuild locks are not guarantees of business consistency.

Logical Expiration and Stale-While-Revalidate

When it's acceptable to return a temporarily stale value, a logical expiration timestamp can be stored within the cache entry: after expiration, one request refreshes the value, while others continue reading the outdated value. This trades staleness for availability. A maximum allowable staleness duration must be defined, this approach is unsuitable for data that must reflect immediate changes, such as revocation or account balances.

Penetration: The Requested Data Does Not Exist

An attack or bug might continuously generate invalid keys, causing every request to bypass the cache and directly access the database.

The defense layers, ordered from least to most costly, are:

• API authentication, format validation, and scope checking;
• Rate limiting and tenant-based quotas;
• Short-lived negative caching for responses confirming non-existence;
• Using a Bloom filter for pre-screening against a known set of valid IDs;
• Monitoring metrics such as distinct miss keys, 404 rates, and their origins.

Bloom filters have false positives but no false negatives, this only holds when the set is maintained correctly and no erroneous deletions occur. A "possibly exists" result still requires a lookup to the source; only a "definitely does not exist" result can safely block the request from reaching the database. Dynamic deletion requires a counting Bloom filter or a reconstitution strategy.

Negative caching must not treat transient database errors as "non-existent," otherwise failures would be cached as persistent 404s.

Snowball: Mass Key or Entire Cache Layer Failure

Causes include:

• A large number of keys sharing the same absolute expiration time;
• Redis cluster failures, network isolation, or credential expiration;
• A namespace change that causes all keys to cold-start simultaneously;
• Inadequate eviction policies and capacity leading to sustained churn;
• Hotspot migration with new nodes lacking pre-warming.

Mitigation requires a combination of strategies:

• Add small, business-compatible jitter to TTLs;
• Merge requests by key or tenant and enforce parallel back-end request limits;
• Implement circuit breakers, queues, and load reduction;
• Asynchronously refresh critical hotspots and perform controlled pre-warming;
• Return acceptable stale values or enable feature degradation during cache failures;
• Conduct capacity drills on the origin server based on the minimum traffic required to survive a full cache outage.

Random TTL only prevents batch expiration and does not address overall cluster unavailability.

Eviction Strategy Determines Who Leaves First; No Guarantee of Hit

Redis supports eviction policies such as allkeys or volatile, combined with LRU, LFU, random, TTL, or noeviction. LRU and LFU are approximate strategies; their real-world effectiveness depends on versioning, sampling, object size, and access patterns.

Before selecting a policy, consider the following:

• Whether all keys can safely be evicted;
• Which keys have TTLs set;
• Whether hot keys are stable or rapidly changing;
• Whether large keys are displacing many small, hot keys;
• Whether it's better to reject writes when capacity is reached or to evict older cache entries.

A drop in hit rate may result from insufficient capacity, overly short TTLs, a sudden explosion in key count, changes in request patterns, or large keys displacing smaller ones. Simply switching eviction algorithms does not resolve these underlying issues.

Large Keys, Hot Keys, and Serialization Costs

Large objects create spikes in network traffic, replication, persistence, deletion, and deserialization. Monitor the distribution of value sizes and split bounded objects into pages or fields to avoid unbounded growth of Hash, List, or Set structures.

Hot keys (even if small) can saturate a single shard’s CPU or network bandwidth. Mitigation strategies include:

• Performing local, short TTL read replicas for hot spots;
• Partitioning and aggregating exchangeable counters across shards;
• Merging requests to reduce redundant reads;
• Redesigning the data model to distribute keys, accepting read fan-out as a trade-off;
• Acknowledging the serial bottleneck for strongly consistent hotspots that cannot be partitioned.

Cache Observation Metrics

Metrics must be observed at multiple layers and by business domain:

• hit/miss rate and distinct miss keys;
• p50/p95/p99 latency for hits, misses, and source-originated requests;
• evictions, expired keys, and rejected writes;
• memory usage, fragmentation, and key/value size distribution;
• hot keys, per-shard QPS, and network bandwidth utilization;
• loader concurrency, wait queues, timeouts, and failure rates;
• sampled differences between cache values and authoritative versions;
• source server remaining capacity after a full cache invalidation.

A global 95% hit rate may mask a critical endpoint with only a 20% hit rate, or it may conceal "high hit" rates caused by incorrect or stale caching. These metrics must be evaluated in conjunction with business correctness.

Fault Simulation

Validate in a pre-production or controlled environment:

1. Clear a namespace and observe pre-warm and cache reload peaks;
2. Disconnect network paths between some applications and cache to verify timeout behavior and thread pool isolation;
3. Expire multiple hot keys simultaneously to test singleflight logic;
4. Inject a large number of non-existent IDs to validate rate limiting and negative caching;
5. Simulate a delayed old expiration event to confirm that the source version does not regress;
6. When the cache becomes completely unavailable, ensure degradation mechanisms activate instead of causing a cascading failure that drains the database.

References

• Redis: Key Eviction
• Redis: Memory Optimization

The next chapter introduces search indexing. Like caching, it is a derived view, but synchronization latency, correlation, and index reconstitution introduce a different set of correctness challenges.