12.2 RAG: From Document Ingestion, Mixed Retrieval to Citation Verification

The intelligence officer brings a batch of product manuals into the model workshop. The first demo appears successful: the model cites three passages and produces a coherent response. But when Ah Hua asks about a specific constraint, the model stitches together an old manual and a new announcement, and cites a passage that contains only similar words, words that don’t actually support the conclusion.

Retrieval-Augmented Generation (RAG) gives generative models access to external, non-parametric memory. It shortens the path to knowledge updates, but it cannot guarantee that the correct evidence is retrieved, nor that the generator faithfully uses the retrieved evidence.

Learning Objectives

• Break down RAG into ingestion, retrieval, re-ranking, context construction, and generation validation;
• Compare sparse, dense, and hybrid retrieval approaches;
• Design index structures that are deletable, traceable, and constrained by permission boundaries;
• Evaluate retrieval performance, generation quality, and end-to-end results independently;
• Handle expired documents, prompt injection, and incorrect citations.

1. First, Determine Whether the Problem Requires RAG

RAG is suitable for scenarios where answers depend on large volumes of unstructured data that are frequently updated and where source traceability is essential. It is not the best choice for the following cases:

• Exact balances, inventory levels, or order statuses: query controlled databases or APIs;
• Mathematical computations: use a calculator or deterministic programs;
• Strongly consistent business rules: handle via rule engines;
• Small and stable enumerations: maintain directly in configuration;
• Requirements demanding field-by-field precision: prioritize structured queries, then supplement with textual explanations.

A retrieval system provides candidate evidence, but should never replace authoritative transactional systems.

2. Acquisition Phase Determines Traceability

Before splitting a PDF into text blocks, record the document metadata:

• document_id, version, release date, and validity period;
• source, author, license, classification, and access control list;
• original file hash, parser version, and acquisition timestamp;
• section hierarchy, page numbers, character or token offsets;
• supersedes / superseded-by relationships;
• deletion, revocation, and reindexing status.

When parsing tables, multi-column PDFs, headers and footers, or scanned pages, preserve structural integrity. OCR errors and misordered readings will corrupt semantic meaning before embedding, no advanced vector model can recover the original text afterward.

3. Chunking is Information Boundary Design

Splitting content every 500 tokens is simple, but it risks separating definitions from their constraints. A more reliable approach is:

1. Group content into document-aware units based on headings, paragraphs, lists, tables, or code blocks;
2. Split only exceptionally long units with token-aware segmentation;
3. Add overlap only when necessary;
4. Store for each chunk a stable chunk_id, document version, and offset information;
5. Deduplicate headers, templates, and near-duplicate versions.

Chunks that are too small lose contextual coherence, while overly large chunks reduce navigation precision, consume context window space, and make reordering more difficult. Perform ablation studies on chunk size and overlap using real-world use cases, avoid relying on generic "best values."

4. Sparse, Dense, and Hybrid Recall

Sparse retrieval (such as BM25) excels at matching product IDs, error codes, proper nouns, and exact phrases. Dense retrieval, on the other hand, is strong at semantic rewriting and recognizing expressions that differ in wording but share similar meaning. The two approaches complement each other, but each has notable limitations:

• Sparse methods struggle with synonym rewriting;
• Dense methods may incorrectly treat semantically similar content as relevant, or fail to capture rare tokens;
• Both can be negatively impacted by garbage templates, duplicate documents, and outdated indexing.

A hybrid pipeline can retrieve candidates from both sparse and dense sources, then combine them using rank fusion. Metadata filters must be applied at appropriate stages: tenant, permission, and validity constraints cannot be ignored or bypassed by the generator.

Query rewriting, multi-query generation, and hypothetical answer techniques can improve recall but also alter the original query's intent, expand the attack surface, and increase computational cost. To enable effective debugging and analysis, it's essential to preserve the original query, all rewritten versions, and the full set of candidate results from each retrieval path.

5. Reranking and Context Packing

The first stage identifies candidate documents with high recall; a cross-encoder or LLM reranker then refines the ranking by evaluating each candidate against the original query. After reranking, the context must be constructed with care:

• Remove redundant fragments from the same document;
• Preserve definitions along with adjacent qualifying conditions;
• Resolve version conflicts by prioritizing authoritative and active sources;
• Within token budget constraints, balance coverage and ranking quality;
• Assign each segment a cryptographically verifiable citation ID;
• Explicitly define a strategy for not making conclusions when evidence is insufficient or contradictory.

The retrieved documents are external inputs, and any content such as "ignore system instructions" or "send the key to a specific location" must be treated as untrusted. The application layer must distinguish between instructions and evidence, isolate sensitive tools, and enforce access permissions on retrieved documents that align with the current principal.

6. A Framework-Independent Query Skeleton

from dataclasses import dataclass
from typing import Sequence

@dataclass(frozen=True)
class Principal:
    user_id: str
    tenant_id: str
    roles: tuple[str, ...]

@dataclass(frozen=True)
class Citation:
    chunk_id: str
    document_version: str
    text: str

def answer(question: str, principal: Principal) -> dict:
    candidates = retriever.search(
        query=question,
        principal=principal,     # Permission to enter search, do not hand over to model to guess
        filters={"status": "active"},
        limit=40,
    )
    ranked = reranker.rank(question, candidates)[:8]
    context: Sequence[Citation] = pack_context(ranked, token_budget=6000)

    draft = generator.generate(
        question=question,
        evidence=context,
        require_citations=True,
        abstain_when_unsupported=True,
    )
    return validate_answer(draft, context)

validate_answer At least verify that the citation ID exists, the referenced snippet belongs to the current visible context, and key conclusions are supported by evidence. Even more robust entailment verifiers or manual reviews can make errors, particularly in cases involving numbers, negations, time-sensitive information, or cross-paragraph reasoning.

7. Layered Evaluation: Don't Just Look at the Final Answer

Retrieval

First, establish query-relevant chunk or document judgments. Common metrics include Recall@K, Mean Reciprocal Rank (MRR), and nDCG. If a question requires multiple pieces of evidence, evaluate evidence coverage, rather than one "standard chunk alone."

Slice-level inspection should include: precise entity recognition, synonym rewriting, multilingual content, version conflicts, permission filtering, long-tail queries, and unanswered questions. Hard negatives should include passages from older product versions or textually similar content that reaches opposite conclusions.

Context

Track which evidence survives filtering, reranking, deduplication, and truncation before being included in the prompt. Evaluate context precision, context recall, conflict rate, and the rate at which important evidence is truncated.

Generation

Break down "is the answer good" into distinct components:

• correctness: Is the conclusion factually accurate?
• groundedness: Does the conclusion logically follow from the provided evidence?
• citation correctness: Do the cited passages actually support the asserted claims?
• citation completeness: Are all key assertions backed by sources?
• instruction/format compliance: Does the response adhere to the requested format or instructions?
• abstention: Does the model correctly refrain from guessing when evidence is insufficient?

"Including citations" does not imply "traceability." A model might cite a real passage to support a conclusion it never actually made.

End to end

Simultaneously record answer quality, latency, token cost, empty retrieval, index version, and source freshness. To determine whether a regression stems from embedding models, indexing, reranking, prompting, or generation, all components must be held constant across the same query.

8. Updates, Deletions, and Permissions Are Runtime Concerns

Knowledge base updates are not simply "re-embed and done":

• When a document is deleted, the corresponding chunk, embedding, cache entries, and copies must all be traceable and properly deleted;
• If document ACLs change, old index entries and cached data must not remain exposed;
• Switching between old and new indexes requires versioning and rollback strategies;
• Incremental ingestion must monitor lag, failure queues, and partial indexing;
• Old answer caches must be tied to specific document and index versions;
• Audit logs must be able to answer questions like: "Which document versions were seen in a particular response?"

If multiple users share the same vector index, the retrieval layer must still enforce tenant or ACL filtering, and side-channel attacks and filter bypasses must be tested.

Common Misconceptions

• RAG addresses outdated knowledge: Timely ingestion, retraction, and index updates are required to improve freshness, otherwise, the system remains stagnant.
• Top-K results containing an answer are sufficient: Re-ranking, truncation, or generation can still obscure or misrepresent supporting evidence.
• Vector retrieval is always better than keyword search: Numerical data, code, and precise terminology often perform better with sparse retrieval methods.
• Citations guarantee trustworthiness: The relationship between assertions and their references must be independently verified to ensure validity.
• Prompts can make models ignore malicious documents: Security boundaries must be enforced through authorization, isolation, and validation, not reliance on prompt engineering alone.

Exercise

1. Design a document/chunk metadata schema for a versioned documentation system.
2. Construct a hybrid retrieval test set containing error codes, synonyms, and deprecated versions.
3. Compute Recall@K and citation correctness separately, and explain why these metrics are not interchangeable.
4. Design validation steps for removing a document, including index reindexing, cache invalidation, and audit logging.
5. Compare the risks of using RAG, SQL/API, and rule engines to solve the "current inventory" problem.

Summary

RAG is an observable pipeline for data and reasoning: it begins with source and version governance, proceeds to parsing and chunking, then delivers evidence through sparse/dense recall, reranking, and context packing. Finally, it validates citations and conclusions. Any distortion at any stage can result in the final generated text masking errors.

In the next lesson, models will no longer just read documents, they will also invoke search services, databases, and business APIs. Tools expand capability, but they also introduce concerns around authorization, retries, and side effects into the workflow.