18.3 Evaluation, Caching, and Production Operations

The output of AI systems is inherently stochastic, with models, prompts, retrieval indexes, and tools all subject to change. Without reproducible evaluation sets and observable production results, teams cannot distinguish genuine improvements from sample noise or hidden regressions.

Start with Task and Failure Classification

Every evaluation task includes inputs, environment, success criteria, and allowed operations. The dataset should cover:

• High-frequency normal requests;
• Business-critical but low-frequency scenarios;
• Edge cases, ambiguities, missing information, and refusal scenarios;
• Actual production incidents and user-reported issues;
• Permission overreach, prompt injection, and dangerous tool invocations;
• Recovery, cancellation, and retry operations within long-running workflows.

Keep a holdout set out of routine tuning and report results by task type, so known examples do not become the whole target. Start each trial from an isolated, recoverable environment state. Refund records, files, or caches left by the previous trial can make the next agent look mysteriously clever.

Evaluate the Outcome, Then Evaluate the Trace

The agent's final statement of "refund completed" does not guarantee that the refund record actually exists. First, verify the outcome in the environment: database state, generated files, unit test results, sent records, or account balances.

Then evaluate the trace:

• Did it select the appropriate tools?
• Did it violate permission or sequence constraints?
• Did it make unnecessary or hazardous calls?
• Did it properly pause when results were unknown?
• Did it respect step count, delay, and cost budgets?

Matching only the final output misses correct-looking answers with incorrect execution. Trace checks should enforce required authorization, dependencies, and side-effect constraints, without forcing every valid solution through one exact tool sequence.

Combine Different Graders

Priority is typically:

1. Deterministic checks: schema validation, tests, state queries, precise rules;
2. Comparison against reference answers or structured facts;
3. Human scoring;
4. Model scoring that has been manually calibrated.

LLM judges are well-suited for evaluating style, coverage, and partial open-ended quality, but require clear rubrics, randomized, blinded candidate presentation, and regular consistency calibration with human reviewers. Avoid having the same vague prompt generate both the answer and the justification for its correctness.

For critical tasks, run multiple trials and report pass rates, sample sizes, and confidence intervals. Suppose independent trials of the same task each succeed with probability 90%. At least one success in five trials has probability 1 - 0.1^5 = 99.999%, but all five succeeding has probability 0.9^5 ≈ 59.0%. These are different questions, expressed by pass@5 and pass^5. The multiplication does not directly apply to correlated trials or tasks with different difficulty.

Regression Evaluation and Online Monitoring Complement Each Other

Offline evaluation is used to compare candidate versions, while online monitoring detects actual distribution patterns and environmental issues. Production systems must at least record:

• Model, prompt, tool, and index versions;
• Total latency, first token latency, token count, and cost;
• Tool calls, errors, approvals, iteration steps, and termination reasons;
• Retrieval sources, references, and permission filtering outcomes;
• Final business outcomes, user corrections, and the proportion of manual escalations.

Logs must be redacted and retained for a defined period. Full prompts, retrieval snippets, and tool outputs may contain personally identifiable information or sensitive keys.

Distinguish Four Types of Caching

Cache Type	What is Reused	Primary Risks
Prompt / prefix cache	Computation results for requests with the same prefix	Vendor semantics, unpredictable hit behavior
Exact response cache	Results of identical requests	Model/data versioning and randomness
Semantic response cache	Results of similar requests	Similarity does not imply equal permissions or intent
Retrieval cache	Candidate fragments matching a query	Index updates, deletions, and ACL changes

A semantic cache cannot rely solely on vector distance to determine reuse. Tenant identity, user permissions, language, time range, model/prompt version, knowledge base version, and tool availability may all need to be included in the cache key or validation condition.

A prefix cache reuses computed prefix state; the system still processes new input and generates an answer. It does not return the previous complete response. Hit behavior, retention, and billing depend on the provider. A response cache instead skips generation and returns an answer saved by the application. Their invalidation risks differ.

Context caching is not long-term memory. The Cache-Augmented Generation paper describes preloading a bounded knowledge corpus into a long context and reusing cached computation to reduce per-query retrieval. This is different from reusing old answers based on semantic similarity. Preloading and rebuilding still have costs when knowledge exceeds the context budget or changes frequently.

Caching Serves Only Acceptable Freshness

First, define data freshness and the cost of error reuse:

• Static product documentation can be cached for longer periods;
• Inventory and pricing need TTLs chosen for their freshness targets; permissions need checks that meet revocation requirements, which a short TTL alone does not guarantee;
• High-risk responses (such as those in healthcare, legal, or financial domains) should never be reused simply because they are semantically similar;
• Caches must be able to proactively invalidate entries when source documents are deleted or permissions change.

An administrator and a regular member may ask the identical question, "What is the refund status for this order?", yet be entitled to different fields. Even a user-specific cache key needs invalidation or reauthorization after permissions change. A cached "refund succeeded" answer cannot replace a current operation-status query.

Evaluate and monitor cache hits too. Lower cost can conceal more stale answers.

Release Gate

Any change to a model, prompt, tool, or index must go through the following sequence:

Offline regression → Security assessment → Shadow traffic → Limited rollout → Online metrics → Expand exposure or roll back

Shadow traffic must remain observational or write only to isolated environments. It must not resend real emails, issue real refunds, or update production records. Comparing two agents on the same request does not authorize both to execute it.

Rollback requires coordination of model/prompt versions, tool schemas, indexes, and caches, not just an application image switch. Do not restore deleted content or revoked access along with an old index. A retired model version may no longer be available either; verify a usable fallback in advance.

References

• OpenAI, Working with evals
• Chan et al., Cache-Augmented Generation
• Anthropic, Demystifying evals for AI agents
• OpenAI, Prompt caching