14.3 AI System Caching: Keys, Permissions, Versions, and Expiry

The same query (“How much supply remains this month”) carries different meaning when evaluated in northern versus southern account contexts. The answer may also shift from morning to evening. If semantic caching relies solely on embedding similarity, even high similarity scores might return someone else’s result or outdated inventory data.

Caching optimizes repeated computation without altering the correctness boundary. Whether a result can be reused depends on all inputs that influence the output: identity, permissions, data version, model, prompt, tools, and time, rather than the query text itself alone.

Learning Objectives

• Distinguish between prefix/KV, retrieval, exact response, and semantic response caching;
• Establish a correctness contract for cache keys;
• Design event expiration, TTL, version namespaces, and deletion propagation;
• Handle stampede, negative caching, partial failures, and multi-region consistency;
• Evaluate caching performance using staleness rate, error reuse rate, and unit success cost.

1. Four Types of Cache Reuse Do Not Reuse the Same Thing

Type	Reused Object	Still Executes	Primary Risks
Prefix/KV cache	Intermediate model state for identical token prefixes	Subsequent decode steps	Version/tenant mixing, GPU memory consumption
Retrieval cache	Candidate documents matching a query or filter	Reranking or generation steps	Access control lists (ACLs), index updates, deletions
Exact response cache	Results of identical request specifications	Typically returned directly	Incomplete keys, randomness, expiration
Semantic response cache	Results of "sufficiently similar" requests	Typically returned directly	Misjudgment of intent, entities, permissions, or timeliness

"Cache-augmented generation" sometimes refers to preloading knowledge into the model context or reusing long prefixes, this does not encompass all forms of "caching in generation." First, we must clarify which layer of reuse is actually occurring, and then examine consistency across those layers.

2. Cache Key Is a Correctness Statement

An exact response key can be constructed from the canonical request:

tenant / principal or authorization scope
task and normalized user input
conversation/state version
model/tokenizer/template/prompt/adapter revisions
sampling and output schema
retriever/index/document snapshot
tool data/version or freshness token
safety/policy/locale versions

Not every item needs to be included verbatim in the key, some can be aggregated under a single deployment or content version. However, the system must prove that any omitted factor does not alter the visible output.

Do not include bearer tokens directly in the key or logs. Use a stable authorization-scope identifier instead, and ensure it updates or expires when permissions change.

3. Semantic Implications of Randomly Generated Caching

When temperature/sampling parameters are non-deterministic, the response cache locks in a specific sample. This may align with product requirements for stable responses to identical requests, but it can also undermine diversity or invalidate assumptions used in evaluations.

Clarify the following:

• Whether the cache stores the final text, structured business results, or a set of candidate responses;
• Whether a cache hit bypasses safety or authorization checks;
• Whether the user expects a re-generation of the response;
• Whether seed or sampling configuration is included in the cache key;
• Whether a cached response still requires re-validation against current policies and references.

High-risk facts and external state should be cached only if they are verifiable, prioritizing intermediate, checkable data over full natural language outputs.

4. A Semantic Hit Requires a Second Check

Embedding distance only indicates proximity in representation space, it does not prove that two requests can share the same answer. Dangerous distinctions include:

• "Can I delete my account" versus "Delete my account for me";
• "Balance of User A" versus "Balance of User B";
• "Policy from last year" versus "Current policy";
• "Who is eligible for this medication" versus "Can I take this drug?";
• Differences in negation, numbers, geographic regions, and versioning.

The semantic caching workflow begins with ANN to identify candidate queries, followed by a detailed validation of task type, entities involved, time scope, authorization levels, knowledge version, and risk level. For domains such as side effects, personalized recommendations, real-time data, and high-risk scenarios, final answers are not directly reused by default.

There is no universal threshold value. Instead, use query pairs to label cases as "safe to reuse," rather than relying on generic notions of "semantic similarity." Evaluate the false-hit cost and identify critical slices to ensure reliability.

5. TTL Is Just the Final Safety Net

TTL is suitable for limiting the maximum age of cached data, but it cannot effectively handle timely responses to permission revocations, document deletions, or urgent policy changes. Instead, prioritize implementing event-driven invalidation:

document v7 revoked
→ publish invalidation(document_id=v7)
→ retrieval entries evict
→ response entries tagged with v7 evict
→ regional replicas acknowledge
→ audit lag / failures

Cache entries store dependency tags: document IDs and versions, index metadata, policy rules, model configurations, and tool data snapshots. An inverted dependency index enables targeted invalidation. When precise tracking becomes difficult, switch to a version namespace so that old keys no longer match.

Deletion operations must cover the cache, replicas, snapshots, and logs. Backups can use short retention periods with replay of the deletion ledger upon restoration.

6. Permission Checks Cannot Be Bypassed by Cache Hit

Two secure approaches:

• Bind the key to an authorization scope so that changes in permissions trigger updates to the scope version;
• After a cache hit, re-authorize the cached result based on the current principal’s access to the resource.

A shared public cache stores only results that are explicitly public and contain no personalized data. In multi-tenant caching, namespaces, quotas, and access auditing are used to enforce isolation; data in transit and at rest are encrypted, but such measures cannot correct logical leaks caused by incorrect keys.

Negative caches can also leak information about whether a resource exists, or continue returning "not found" after permissions are restored. Care must be taken in designing cache keys, TTLs, and the semantic meaning of HTTP responses for 403 and 404 errors.

7. Stampede and Request Coalescing

When a popular key expires, a surge of requests simultaneously penetrates into expensive models, causing overload:

• single-flight/request coalescing: Only one filler is allowed per key to prevent redundant work;
• stale-while-revalidate: Return outdated content for requests while background refreshing continues;
• jittered TTL: Introduces random delays to expiration times to avoid batched key expirations;
• refresh-ahead: Proactively refreshes hot keys based on prediction models;
• bounded fill concurrency with admission control: Limits the number of concurrent fill operations and controls access;
• failure negative caching, but with short TTLs and error classification to avoid caching errors.

Failure to fill cannot treat error pages or partial streams as valid cache hits. An entry is only atomically published after completing generation, schema validation, citation checks, and safety validations.

8. Streaming Responses and Partial Results

Clients may cancel mid-stream, and models might emit invalid JSON at the end of output. Do not write incomplete text to a shared cache as it is being generated.

Instead, stream responses into a request-local buffer, validate the full output before committing it. For very large results, use a temporary object combined with an atomic manifest. Cache metadata should record the reason for completion, validation status, and content hash. Canceled, timed-out, policy-blocked, and truncated outputs are by default excluded from normal cache hits.

9. Multi-Region and Disaster Recovery

Cross-region cache replication introduces invalidation lag. Define an acceptable level of staleness and monitor the applied version across regions; permission and security revocations must follow high-priority paths, and in critical cases, it's acceptable to disable hits rather than continue serving stale copies.

During region failover, the new region may initially contain only the old cache or entries with different model or index versions. Recovery strategies must first validate namespace and deployment identity, never mix incompatible entries to reduce cold start latency.

10. Cache Evaluation Cannot Rely Solely on Hit Rate

Core metrics:

• exact/semantic/prefix/retrieval hit rate;
• saved prefill tokens, saved model/tool calls;
• false semantic hit / wrong-scope hit;
• stale answer or citation rate;
• invalidation propagation latency and failure;
• hit/miss quality, safety, and latency distribution;
• stampede/coalescing rate;
• cache memory/storage utilization versus unit successful-task cost;
• deletion and ACL test pass rate.

Rising hit rates accompanied by increased reuse of outdated or incorrect content are not an improvement. During A/B testing, requests should be paired or stably binned, with high-risk slices carefully monitored and evaluated.

Common Misconceptions

• Similar problems can share the same solution: Identity, entities, timing, and actions may differ significantly.
• TTL (time-to-live) solves cache consistency: Urgent deletions and permission revocations require active invalidation, not passive expiration.
• A cache hit means no further authentication is needed: That would turn the cache into a vector for privilege escalation.
• Storing the final rendered text is cheapest and therefore best: When authoritative data changes, reusing stale content introduces higher error costs.
• Higher hit rates are always better: You must also consider error hits, freshness, and the cost per successful operation.

Exercise

1. Write a complete exact cache key for "query current inventory".
2. Construct ten query pairs that are semantically similar but should not share the same response.
3. Design an event chain for rolling back a document to a multi-region cache upon cache miss.
4. Design an atomic cache fill mechanism for streaming JSON responses.
5. Establish a semantic cache rollout threshold that accounts for the cost of false hits.

Summary

The correctness of AI caching depends on the key, permissions, dependency versions, and expiration mechanisms. Prefix cache reuses computed results, retrieval cache reuses candidate outputs, and response cache reuses final responses; semantic response caching carries the highest risk and requires additional intent and scope validation.

The final lesson connects inference, routing, RAG, tools, caching, and evaluation into a publishable, degradable, and rollable production pipeline.