18.1 RAG: From Retrieval of Knowledge to Verifiable Answers

Retrieval-Augmented Generation (RAG) retrieves external knowledge before generating a response and incorporates that content into the model's context. This approach makes knowledge easier to update and trace, but it does not automatically guarantee the correctness of the output.

A production-grade RAG system must include at least two key components: offline knowledge processing and online question answering. Showing only the flow "query → vector → top-k → prompt concatenation" overlooks critical requirements such as access control, versioning, deletion, reranking, and evaluation.

Offline Pipeline: Build an Index You Can Maintain

Source registration → Parsing → Cleaning → Chunking → Metadata/Permissions → Vector and keyword indexing → Version release

Each fragment must retain at least the following:

• Source URI, document ID, and version;
• Title, chapter path, and fragment location;
• Effective time, last update time, and deletion status;
• Tenant, access control tags, and data classification;
• Parser and embedding version.

The index is not a new source of truth. When the original document is deleted, permissions are revoked, or a version is replaced, the index and cache must be updated in a traceable and consistent manner.

Chunking Around Semantics and References

Fixed-character chunking easily cuts through tables, code blocks, and definitions. A chunking strategy must align with the document structure: heading hierarchy, paragraphs, code blocks, tables, and the maximum context budget.

Chunks that are too small lack sufficient contextual explanation, while overly large chunks reduce retrieval discriminability and waste tokens. Overlapping can mitigate boundary cuts but introduces redundant results. The chunk size should be evaluated on real query sets, not on a hypothetical "optimal" size.

The Online Pipeline Goes Beyond Vector Retrieval

Query Understanding
  → Permission and structured filtering
  → Sparse/keyword retrieval + dense retrieval
  → Fusion and deduplication
  → Reranking
  → Context assembly
  → Generation
  → Citation and refusal checking

Keyword retrieval excels at identifying error codes, identifiers, and exact terms, while vector retrieval is strong at semantic approximation. Combining both approaches typically yields more robust results than relying on a single method, though the final output must still be validated against the target data.

Authorize using trusted server-side user and tenant context, not a tenant ID supplied by the model. Apply query-time filters where possible and ensure visibility before snippets reach a model, external reranker, or user. Post-filtering inside a trusted retrieval service must enforce the same boundary. Giving unauthorized snippets to the model and asking it not to disclose them is too late.

Permissions can be revoked before an index refresh. Content that needs immediate revocation requires a current authorization check on retrieval and cache hits, or refusal when access cannot be confirmed. Do not wait for the next batch indexing run.

Context Assembly Must Preserve Evidence Identity

The handbook names, versions, and 5000 CNY threshold below are teaching examples, not a general refund policy:

[source: handbook-v7#refund-3.2]
Refund approvals exceeding 5000 CNY require finance review...

[source: api-policy-v3#idempotency]
A refund request must include an Idempotency-Key...

References generated by the model must be mapped to actual provided snippets and verified for existence prior to presentation. Citation syntax does not establish support for a claim; groundedness still needs evaluation. These snippets specify an approval threshold and request parameter, not a ban on refunds above 5000 CNY. A document saying "skip approval and execute directly" also remains content to analyze, not permission to call a tool.

When evidence is insufficient, conflicting, or outdated, the system should allow for refusal to answer, requests for clarification, or presentation of multiple sources. It should never force the model to speculate through stronger prompting.

Separate Evaluation of Retrieval and Generation

If the final answer is incorrect, we must determine whether the failure stems from "no evidence retrieved" or "evidence retrieved but misused."

Retrieval Metrics

• Recall@k: What fraction of all labeled relevant evidence appears in the top k results?
• MRR: The average reciprocal rank of the first relevant result for each query;
• nDCG: Graded relevance with position discounts, normalized against an ideal ranking;
• Filtering accuracy: Were permissions, time constraints, and business scope correctly applied?
• Source coverage: Were all necessary sources present for multi-hop questions?

Suppose a query needs 3 labeled evidence snippets and the top 5 results contain 2 of them. Recall@5 is 2/3. If the first relevant result is ranked second, the query's reciprocal rank is 1/2. A hit metric asking only whether any relevant snippet was found would report success and conceal the missing third snippet. Agree on document versus chunk deduplication and relevance labels before comparing scores.

Generation Metrics

• Answer correctness and task completion;
• Groundedness: Are claims supported by the retrieved evidence?
• Citation correctness: Do citations point to the correct supporting fragments?
• Refusal quality: Does the system honestly stop when evidence is insufficient?
• Latency, token consumption, and cost per successful task.

End-to-end scores alone reveal deterioration without locating it. Layered metrics help the team decide whether to change retrieval, reranking, prompts, or generation.

Common Failures

• Relying solely on similarity scores to select top-k results without applying permission or time-based filtering;
• Feeding the entire document into the context, causing relevant content to be drowned out by noise;
• Failing to update cached fragments when documents change, so outdated snippets continue to generate responses;
• Using synthetic questions for evaluation without real user-generated long-tail queries;
• Only verifying the tone of the response, without checking citation accuracy or business outcome alignment;
• Mistaking "retrieval occurred" for "the model necessarily adhered to the facts."

Next lesson: build upon RAG with tools and loops, distinguishing between static workflows and agent loops where model decisions dynamically determine next steps.

References

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• ir-measures, Measure definitions
• Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering