14.4 Production Reference Architecture: SLOs, Change Units, Degradation, and Rollback

Three failures can occur at the same time: a remote model starts rate-limiting requests, a new index omits permission tags, and an old cache returns answers from the previous version. Each component may still pass its own health check while the end-to-end system is already unreliable.

Building AI applications isn't "LLM service + UI." It's a request chain spanning identities, data, models, tools, and observability. The value of architecture lies in containing failures, enabling comparability of changes, making results traceable, and allowing safe degradation when dependencies are unreliable.

This lesson's objectives

• Draw the end-to-end request/data/control planes;
• Break product objectives down into component SLOs and error budgets;
• Define change units covering models, prompts, indexing, tools, and caching;
• Design isolation, degradation, canary releases, and atomic rollback;
• Operate continuously using traceability, business outcomes, and event flowback.

1. A reference request chain

Client
  ↓ identity / rate / request limits
API Gateway
  ↓ canonical request + trace + deadline
AI Policy & Model Gateway
  ├─ route → Model Serving / Provider
  ├─ retrieve → ACL Filter → Hybrid Search → Rerank
  └─ tools → Validate → Authorize → Approve → Execute
           ↑             Orchestrator state             ↓
           └──── evidence / tool results / budgets ─────┘
  ↓ schema / citation / safety / business validation
Response / Artifact Store
  ↓
Telemetry + Evaluation + Feedback + Incident Pipeline

The control plane manages deployment bundles, policies, schemas, index versions, quotas, and feature flags; the data plane handles user requests. Separating these two components limits runtime components from directly modifying their own policies or release states.

2. Identity and Deadline Span the Entire Workflow

Each layer propagates principal, tenant, purpose, trace ID, deadline, and cancellation. Downstream must reauthorize using its own resources and cannot rely solely on upstream claiming "it has already been verified."

Deadline is the total budget, not 30 seconds allocated to each layer:

request deadline
  - gateway queue
  - retrieval/rerank
  - model TTFT/decode
  - tool calls
  - validation/serialization
  = remaining budget

Retries and fallbacks share the remaining deadline/cost budget. After client cancellation, cancellation is propagated to the model, retrieval, and tools; previously generated side effects still require status queries and logging.

3. SLO Breakdown from User Outcomes

For example, "95% of evidence-based questions are answered with supported responses within 5 seconds" includes:

• availability: The system accepts and completes requests;
• latency: TTFT, E2E, and tool delay;
• quality: correct/grounded/citation-complete;
• safety/security: no unauthorized retrieval or hazardous side effects;
• freshness: cited within the allowed version window;
• business outcome: Whether the user completes the task or upgrades to manual processing.

Component SLI helps with diagnosis but can't replace end-to-end SLO. Retriever Recall@K is normal, yet users may still fail due to context truncation or generation failure.

Six independent required components that each succeed 99% of the time yield only ${0.99}^6\approx 94.15\mathrm{\%}$ joint success. Shared failure causes make that independence assumption unsafe. Likewise, summing component P95 latencies does not in general produce the end-to-end P95; measure complete requests and their joint behavior. Google’s SLO workbook develops the user-outcome approach.

An error budget is used to determine how fast changes can be deployed and how much reliability investment is required. Safety, privacy, and significant rights violations are typically hard guardrails and should not be offset by the available budget for uptime.

4. The Deployment Bundle Is the Unit of Change

A reproducible version includes:

application/orchestrator image
model/provider revision + tokenizer/template + adapter
system prompts + output schemas
retriever/embedding/reranker + index snapshot
tool schemas + tool service/API versions
safety/authorization policies
cache namespace/key version
feature flags + routing rules
evaluation report + migration/rollback plan

Only versioning models misses more common prompt, index, and tool changes. Bundle manifest uses immutable IDs/hashes and records dependency compatibility matrices.

If DB or index migrations are irreversible, first design dual-read/write, backfill verification, and forward-fix/restore paths. Model rollbacks cannot automatically rollback already-sent emails or completed payments.

5. Gradually Expand the Impact of the Release Pipeline

unit/schema/property tests
→ component + end-to-end offline eval
→ security/privacy/red-team regression
→ shadow replay
→ internal / tenant canary
→ small traffic canary
→ staged rollout
→ post-release hold and review

Define promote/stop/rollback criteria for each phase. Route Canary traffic stably to user/tenant/session to prevent version mixing within the same workflow; high-risk tenants should not be default recipients of early access traffic.

Compare online in real time: success, quality guardrails, incident signals, TTFT/TPOT, cost, fallback, human escalation, and cache correctness. Even when short-term metrics appear normal, maintain observation windows to cover document updates, permission changes, and long-running tasks.

6. Rollbacks must atomically restore compatible combinations

If a new prompt depends on a new tool schema, rolling back only the model could cause protocol misalignment. Instead, roll back the deployment bundle or use compatible feature flags:

• route/model aliases;
• prompt/schema/policy;
• index read version;
• tool contract;
• cache namespace;
• application/orchestrator.

Confirm before rolling back that the previous version still complies with current security policies and data requirements. The old index might be missing new ACLs, and the old model provider might have discontinued service. Regularly practice rollbacks rather than executing them for the first time during an incident.

“Atomic” should describe the compatible version a request observes, not simultaneous replacement of every service. Publish an immutable bundle, check dependencies, then switch the routing pointer for new runs. Pin in-flight runs to a compatible bundle until completion or an explicit migration. Current revocations and authorization policy still apply; they must not roll back with an old index or prompt.

7. Design Levels of Degradation

When a dependency fails, the system can proceed based on risk assessment:

1. Feature Degradation: Disable write tools, retaining only read-only Q&A;
2. Quality Downgrade: Switch to a smaller model with capability compatibility and explicitly enforce limits;
3. Data Degradation: When retrieval is unavailable, respond only to requests that don't depend on private or real-time facts;
4. Interaction Degradation: Pause the long Agent workflow and switch to a fixed process or manual intervention;
5. Capacity Downgrade: Reduce maximum output, limit low-priority traffic;
6. Safe Stop: Refuse execution if authorization, verification, or isolation cannot be achieved.

Don't let the model continue responding with "current policy" based on parameter memory during RAG failures; nor should you default to approving when the approval service is unavailable. Degradation behaviors should be included in product documentation and testing sets.

8. Isolate Failure Domains

• Each tenant/priority has its own queue, quota, and budget to prevent noisy neighbors;
• High-risk tools use different executors/credentials than standard generation;
• Isolate retrieval and model provider with a circuit breaker;
• Batch and evaluation traffic doesn't compete with online critical requests for the same unlimited resource;
• Cache, index, and object store capacity/entry limits;
• An external provider failure does not trigger cross-layer infinite retry;
• A compromise of the control plane should not directly grant business data access.

A bulkhead sacrifices some resource utilization to prevent faults from spreading. Validate it with chaos/fault injection testing, rather than by looking at the architecture diagram alone.

Separate queues do not isolate a shared GPU, thread pool, or credential. Saturate one tenant in a fault test and verify that another tenant still meets its reserved capacity and deadline. Also test control-plane changes: a writable policy or route can indirectly grant access unless deployment privileges are constrained.

9. Observability Connects Technical and Business Outcomes

A single trace record:

• resolved deployment bundle;
• queue, retrieval, rerank, model, tools, validation spans;
• documents/citations, authorization decisions, and policy reasons;
• tokens, latency, cost, cache events, retries;
• state transitions, side effects, and stop reason;
• final artifact, business result, user correction/escalation.

Metrics are used for trends and alerts, traces for cross-component troubleshooting, logs for recording essential events, and evaluation samples for assessing behavior quality. All four require sampling, de-identification, access control, retention, and deletion policies.

Linking technical metrics to "whether the task is completed." Token usage dropped 20%, but user rework increased, this isn't effective optimization.

10. Cost is a controlled resource

Break down unit costs by input/output tokens, embedding/rerank, tool/API, GPU time, storage/cache, manual review, and event processing. In addition to cost per request, calculate cost per successful task, cost per grounded answer, and cost per approved action.

The optimization order is typically: remove useless calls, shorten or structure context, select the right model, reuse safe intermediate results, then optimize kernel or serving. Never sacrifice quality or bypass validation for superficial cost reductions.

Set per-run/tenant budget and alert on budget overruns for Agent; reconcile financial invoices with usage traces to prevent undetected changes in supplier token accounting or retry behaviors.

11. Operations Cycle

Production signals / Appeals / Events
→ De-identified reproduction and failure taxonomy
→ Locate the data/model/prompt/retrieval/tool/orchestration failure
→ Fix + Nearby variants
→ Offline Regression and Risk Review
→ staged rollout
→ Monitor residual risk

Ownership goes beyond ML engineers: Product, domain, security, privacy, SRE, legal/compliance, and dispute operations each hold distinct controls. Acceptance of significant residual risk must be signed off by someone with the appropriate authority.

12. Final Architecture Review Checklist

Requests and Permissions

• principal, tenant, deadline, cancellation are consistently referenced?
• Is each data and tool access reauthorized at the resource level?
• Are side effects idempotent, observable, and auditable?

Data and Version

• Can documents, indexes, cache, model, and prompt be traced to an immutable version?
• Can deletions, ACLs, and emergency policy updates be propagated?
• Can third-party changes be detected and regression-tested by probe and regression testing?

Reliability

• Is the queue bounded? Are retries sharing a budget?
• Are there safe degradation and failure isolation mechanisms in place for dependency failures?
• Have we practiced rollout/rollback/kill switch?

Quality and Risk

• Can component-level and end-to-end evaluations identify the fault layer?
• Do high-risk slices, red team, and production events get into the gate?
• Are manual reviews, corrections, and appeals accessible and effective?

Common Misconceptions

• Component health means system health: Protocol mismatches and cross-layer errors can still occur.
• Roll back the app image is enough: Models, indexes, prompts, schema, and cache are also versions.
• Having a fallback ensures high availability: A fallback might not meet capability, data, or security requirements.
• Downgrading is just switching to a smaller model: Sometimes high-risk facts and actions must be halted.
• Model metrics are business metrics: Track task outcomes, rework, appeals, and real-world side effects.

Practice

1. Draw the request/control/data planes for RAG + Tool Agent.
2. Break "Reliably answer in 5 seconds" down into at least six SLIs.
3. Write a deployment manifest containing model, prompt, index, tool, and cache.
4. Design the degradation order when provider rate limiting, ACL index errors, and cache staleness occur simultaneously.
5. Run through a rollback of an index and tool schema release incident.

Summary

An AI architecture puts uncertain models into defined identities, permissions, versions, budgets, and recovery boundaries. Inference services provide capacity, gateway and routing compatibility, cache reuse proves identical computation, workflow execution ensures controlled actions, and evaluation and observability provide evidence for deployments and events.

When extending the system, repeat the same decision sequence: define the task and potential harm, select data and models, verify behavior and permissions, then expand traffic with a tested recovery path.