11.4 Prompt, In-Context Learning, and Structured Output: Strings Can't Control Permission Boundaries

The Model Workshop hasn't retrained Ah Hua's base model (instead, it only modifies the input by specifying tasks, examples, and output formatting. New workflows can be tested in minutes, which is the strength of prompts. Yet, even small changes) such as swapping templates, altering the order of examples, or adjusting generation configuration, can lead to significantly different results. This is the cost of prompt-based systems.

A prompt is versioned program input, not natural language magic. Ultimately, the chat role is converted into a token sequence by the tokenizer template, and the model continues to perform conditional generation.

Learning Objectives

• Distinguish between zero-shot, few-shot, and weight-updating fine-tuning;
• Correctly use chat templates associated with specific checkpoints;
• Design tasks, context, examples, constraints, and output schema;
• Understand the impact of temperature, top-p, and greedy sampling on generation and evaluation;
• Use permission boundaries instead of "stronger prompts" to mitigate prompt injection.

1. In-Context Learning: No Weight Updates

Zero-shot: Provide only the task or input. Few-shot: Include demonstration examples within the same context. The model's parameters remain unchanged, and the output probability becomes:

$$ p(y\mid instruction, examples, input). $$

This approach enables rapid adaptation to format and local patterns, but the examples are not permanently stored in the model's weights once the context ends. It is incorrect to refer to few-shot learning as "temporary fine-tuning" and then dismiss the differences between the two in terms of state, cost, and safety.

2. Chat Messages Ultimately Become Token Sequences

Even instruct checkpoints that share a base model may use different control tokens:

<system>...</system><user>...</user><assistant>

or entirely different formats. Manually stitching together role tokens in error can significantly alter model behavior. Use the chat template provided with the tokenizer and lock it to a fixed version.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-pinned-instruct-checkpoint"
tokenizer = AutoTokenizer.from_pretrained(model_id, revision="pinned-revision")
model = AutoModelForCausalLM.from_pretrained(model_id, revision="pinned-revision")

messages = [
    {"role": "system", "content": "Return one JSON object matching the schema."},
    {"role": "user", "content": "Extract severity from: disk latency is critical"},
]

encoded = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)

generated = model.generate(
    **encoded,
    max_new_tokens=64,
    do_sample=False,
)

new_tokens = generated[:, encoded["input_ids"].shape[1]:]
answer = tokenizer.batch_decode(new_tokens, skip_special_tokens=True)[0]

If you first apply_chat_template(tokenize=False) and then separately tokenize, be cautious of duplicating special tokens. During training, generation prompts are typically omitted; instead, adhere strictly to the template's contract.

3. Components of a Maintainable Prompt

task / objective
input contract and delimiters
authoritative context
decision policy / abstention
few-shot examples
output schema
edge cases and constraints
untrusted content

Separate stable policies from dynamic user or data inputs. Use clear delimiters to mark data boundaries, but delimiters serve only to help the model recognize structure, they do not create a secure sandbox.

Phrases like "you must always..." in a prompt cannot prevent the influence of subsequent untrusted text, nor can they restrict the application's actual tool or network permissions.

4. Few-shot Example Selection

Examples should simultaneously teach the task, output format, label priors, and style. When selecting examples, ensure coverage of:

• Normal and edge case scenarios;
• All label and rejection pathways;
• Variations in length, language, and format;
• Negative examples that are easily confused;
• Correct schema versus unacceptable, invalid outputs.

The order, similarity, and label frequency of examples can significantly influence results. Dynamic retrieval of examples may introduce data leakage or injection; both the retrieval index and the examples themselves must be versioned.

It’s not true that “three carefully chosen examples are always better than ten.” Instead, evaluate the number, coverage, and ordering of examples using token budget constraints and held-out evaluation sets.

5. "Think step by step" is not a reliable validator

The rationale generated by a model may improve certain tasks, but it can also produce fluent yet incorrect explanations, leak sensitive intermediate information, or increase token usage and latency. To ensure reliability, prioritize verifiable artifacts such as:

• Final structured fields;
• Reference or evidence IDs;
• Executable code and accompanying tests;
• Computed results and independent checkers;
• Clear statements of uncertainty or explicit refusal to answer.

Do not treat the model's chain-of-thought as a reflection of its true internal causal process. For high-risk systems, rely on external rules, tool-based verification, multi-model or human review, never just ask the model to "think more carefully."

6. Generation Config is Part of the Behavior

• greedy: selects the token with the highest probability at each step;
• sampling: draws from a modified probability distribution;
• temperature: scales the logits, with lower values typically producing sharper, more deterministic outputs;
• top-k/top-p: truncates the candidate distribution to limit the search space;
• beam search: maintains multiple high-scoring sequences, though it may not be well-suited for open-ended conversations;
• max_new_tokens/stop: controls when generation terminates.

Setting temperature and do_sample=False may not yield the expected results and depends on the specific API. Changing decoding strategies while keeping the prompt fixed is not guaranteed to behave consistently across versions of the system.

Greedy decoding tends to be more stable under fixed input conditions, but floating-point precision, kernel differences, or version variations can still affect token ties or subsequent sequences. Evaluating stochastic generation requires multiple samples and seeds, along with clear definitions of metrics like variance and pass@k.

7. Structured Output Can't Rely on "Please Return JSON" Alone

A reliable workflow must include:

1. Define a JSON Schema or grammar;
2. Leverage constrained decoding capabilities (if available) from the provider or model;
3. Parse the output;
4. Validate against the schema;
5. Perform semantic validation (e.g., range constraints, field cross-references, permission boundaries);
6. Clearly define retry, repair, or abstain strategies;
7. Ensure both raw output and error classification are audit-ready.

Parsed JSON does not guarantee factual correctness. Never execute unverified model-generated SQL, shell commands, URLs, or tool arguments directly.

Prefilling the output with a structured template can improve format consistency, but it must align with the chat template’s assistant role and EOS (end-of-sequence) rules.

8. Prompt Injection Is a Trust Boundary Issue

RAG documents, web pages, emails, and tool outputs can contain content that instructs the model to "ignore previous instructions." Without reliable natural language-based permission isolation, adversarial text and trusted commands end up in the same context, creating a vulnerability.

Key defenses:

• Minimum tool permissions and per-action authorization;
• Allowlisting with schema, type, and range validation;
• Marking untrusted data and restricting its usage;
• Sensitive data should not be automatically included in context;
• Side-effect actions require user confirmation or secondary policy review;
• Egress, network, and domain controls;
• Detection and logging of injection and adversarial evaluation attempts;
• Model outputs must always be treated as untrusted proposals.

The system prompt holds semantic priority as a reflection of the model's training conventions, not as a firewall. Hiding the system prompt does not provide secure storage for sensitive information.

9. Context Budget and Information Placement

The token budget includes system messages, conversation history, examples, retrieved documents, tool schemas, user input, and reserved output tokens. Exceeding the budget results in truncation or errors; even if the budget is not exceeded, the model does not guarantee equal utilization of all available positions.

Strategies:

• Compute token quantiles for each component;
• Prioritize retaining context relevant to the current task and authorized information;
• Create traceable summaries of history, preserving key original texts;
• Deduplicate, sort, and cite retrieved documents;
• Reserve tokens for output and tool invocation rounds;
• Test the sensitivity of critical information when placed at the beginning, middle, or end of the context.

"Supporting 1M context" does not mean stuffing one million tokens into the model in the most optimal way. Costs, latency, noise, and utilization efficiency must all be evaluated.

10. Prompt Evaluation Goes Beyond Three Demos

Build a versioned evaluation set that includes:

• normal, edge, adversarial, and multilingual cases;
• schema and parse rate;
• task correctness and grounding;
• over- and under-reporting of refusals;
• injection and tool misuse;
• latency, token count, and cost;
• stability across paraphrasing, input order, and seed values;
• human rubric scoring and inter-rater agreement.

A system version is composed of prompt, model revision, tokenizer/template, retrieval corpus, tools, and decoding configuration. Recording only the prompt text is insufficient to reproduce results.

Repeatedly refining the prompt based on the same evaluation set leads to overfitting. Therefore, blind, private, and future evaluation sets must be preserved.

11. Prompt, RAG, or Fine-tune

Requirement	Common Starting Point	Reason
Modify task description or output format	Prompt/schema	Fast, reversible
Provide frequently updated or referenceable knowledge	RAG/tool	No need to embed facts into model weights
Stable changes to behavior or style	SFT/PEFT	Reduces need for context examples on every call
Precise business calculations or permissions	Code/tool/rules	Verifiable, auditable, and permission-controlled
Multiple requirements combined	Prompt + RAG + fine-tune + tools	Layered approach to address different problem types

Fine-tuning is not the only way to update a knowledge base, and RAG alone does not guarantee that answers use the provided documents. These components will be integrated into a full LLM system in Chapter 12.

Common Misconceptions

• Few-shot examples temporarily update model weights: They only modify the weights in response to the current input context.
• System prompts act as a safety boundary: Permissions must be enforced at the application and tool layer, rather than through prompt design alone.
• Requiring a chain-of-thought ensures correctness: Even the reasoning process can contain errors.
• Returning valid JSON is sufficient for execution: Semantic validity and permission checks are still required.
• A fixed prompt guarantees reproducibility: Model, template, retrieval, and decoding components all have versioning, so prompt stability alone is not enough.

Exercise

1. Compare token IDs with outputs using the correct/incorrect chat template.
2. Design zero-shot and few-shot prompts for an extraction task, and perform an order ablation study.
3. Add schema validation, semantic validation, and failure handling to JSON output.
4. Construct retrieval documents with injection attacks, and design a tool allowlist with clear permission boundaries.
5. Build a prompt regression dataset to track model, template, and decoding version changes.

Summary

Prompts and in-context examples modify conditional generation through input tokens without updating model weights. For reliable deployment, it's essential to match the chat template, freeze the generation configuration, validate structured outputs, and continuously evaluate performance. Security cannot be achieved through stronger wording alone, it must be enforced outside the model via well-defined permissions, input validation, and clear boundaries for side effects.

The next chapter integrates the model into real systems: SFT and preference alignment shift strategies, RAG brings external knowledge into the pipeline, tools and agents execute actions, and evaluation is embedded throughout the entire workflow.