18.3 Evaluation, Caching, and Production Deployment

The output of AI systems is inherently stochastic, with models, prompts, retrieval indexes, and tools all subject to change. Without reproducible evaluation sets and observable production results, teams cannot distinguish genuine improvements from sample noise or hidden regressions.

Start with Task and Failure Classification

Every evaluation task includes inputs, environment, success criteria, and allowed operations. The dataset should cover:

• High-frequency normal requests;
• Business-critical but low-frequency scenarios;
• Edge cases, ambiguities, missing information, and refusal scenarios;
• Actual production incidents and user-reported issues;
• Permission overreach, prompt injection, and dangerous tool invocations;
• Recovery, cancellation, and retry operations within long-running workflows.

The evaluation set must include a separate, independently maintained subset to prevent teams from overfitting to publicly available examples.

Evaluate the Outcome, Then Evaluate the Trace

The agent's final statement of "refund completed" does not guarantee that the refund record actually exists. First, verify the outcome in the environment: database state, generated files, unit test results, sent records, or account balances.

Then evaluate the trace:

• Did it select the appropriate tools?
• Did it violate permission or sequence constraints?
• Did it make unnecessary or hazardous calls?
• Did it properly pause when results were unknown?
• Did it respect step count, delay, and cost budgets?

Matching only the final output misses issues where the answer appears correct but the execution was flawed.

Combine Different Graders

Priority is typically:

1. Deterministic checks: schema validation, tests, state queries, precise rules;
2. Comparison against reference answers or structured facts;
3. Human scoring;
4. Model scoring that has been manually calibrated.

LLM judges are well-suited for evaluating style, coverage, and partial open-ended quality, but require clear rubrics, blind candidate ordering, and regular consistency calibration with human reviewers. Avoid having the same vague prompt generate both the answer and the justification for its correctness.

Model outputs exhibit variance. For critical tasks, run multiple trials and report pass rates and confidence intervals, rather than selecting just the single best result.

Regression Evaluation and Online Monitoring Complement Each Other

Offline evaluation is used to compare candidate versions, while online monitoring detects actual distribution patterns and environmental issues. Production systems must at least record:

• Model, prompt, tool, and index versions;
• Total latency, first token latency, token count, and cost;
• Tool calls, errors, approvals, iteration steps, and termination reasons;
• Retrieval sources, references, and permission filtering outcomes;
• Final business outcomes, user corrections, and the proportion of manual escalations.

Logs must be anonymized and retained for a defined period. Full prompts, retrieval snippets, and tool outputs may contain personally identifiable information or sensitive keys.

Distinguish Four Types of Caching

Cache Type	What is Reused	Primary Risks
Prompt / prefix cache	Computation results for requests with the same prefix	Vendor semantics, unpredictable hit behavior
Exact response cache	Results of identical requests	Model/data versioning and randomness
Semantic response cache	Results of similar requests	Similarity does not imply equal permissions or intent
Retrieval cache	Candidate fragments matching a query	Index updates, deletions, and ACL changes

A semantic cache cannot rely solely on vector distance to determine reuse. Tenant identity, user permissions, language, time range, model/prompt version, knowledge base version, and tool availability may all need to be included in the cache key or validation condition.

"Context caching" is an optimization for computation, not long-term memory; a "semantic response cache" is not equivalent to Cache-Augmented Generation. Terminology must align with actual mechanisms in order to discuss consistency and cache invalidation accurately.

Caching Serves Only Acceptable Freshness

First, define data freshness and the cost of error reuse:

• Static product documentation can be cached for longer periods;
• Inventory, pricing, and permissions require short TTLs or should not be cached at all;
• High-risk responses (such as those in healthcare, legal, or financial domains) should never be reused simply because they are semantically similar;
• Caches must be able to proactively invalidate entries when source documents are deleted or permissions change.

Cache hits must also be evaluated and monitored. Otherwise, reduced latency may mask an increase in outdated or incorrect responses.

Release Gate

Any change to a model, prompt, tool, or index must go through the following sequence:

Offline regression → Security assessment → Shadow traffic → Limited rollout → Online metrics → Scale out or rollback

A rollback must include the model/prompt version, tool schema, index, and cache, never just the application image.

References

• OpenAI, Working with evals
• Anthropic, Demystifying evals for AI agents
• OpenAI, Agents SDK