12.1 Preference-Based Post-Training: SFT, Reward Models, RLHF, and DPO

A pretrained causal LM learns to predict continuations. A product also needs instruction following, formatting, and a policy for insufficient evidence. Preference post-training changes which responses are favored; it does not turn the model into a fact checker.

Learning Objectives

• Distinguish the responsibilities of Supervised Fine-Tuning (SFT), reward modeling, and policy optimization;
• Explain the three phases of InstructGPT-style RLHF;
• Write the core loss functions for the reward model and DPO;
• Identify issues such as preference data bias, reward hacking, and capability regression;
• Design auditable offline evaluation for preference-based post-training.

1. First, Define Who Owns the "Preferences"

"Human preferences" is not a naturally consistent label. It may originate from product specifications, domain experts, security policies, target users, or annotators. Different groups can diverge in terms of preference length, tone, risk tolerance, and value judgments.

Therefore, before training, it's essential to clearly specify:

• Where the prompt comes from in terms of task distribution;
• Who the annotators represent and what domain expertise they possess;
• What the ranking criteria are: correctness, helpfulness, safety, or formatting;
• How to resolve conflicts among multiple objectives;
• Whether ties, abstentions, or "none acceptable" are permitted.

Without this explicit definition, alignment becomes merely pushing the model toward an unspecified set of annotator preferences, rather than aligning with universal truths.

2. SFT First Teaches the Model Task Forms

Supervised Fine-Tuning (SFT) uses demonstration data $(x,y)$ to minimize the negative log-likelihood of the target response at the token level:

$${\mathcal{L}}_{SFT}=-\sum _t\log {\pi }_{\theta }(y_t\mid x,y_{<t}).$$

It is effective for learning dialogue templates, structured outputs, refusal patterns, and step-by-step task procedures. However, SFT is limited by the scope of demonstrated examples: showing only "good answers" does not directly instruct the model on which of two candidate responses is better, nor does it automatically calibrate factual accuracy.

In practice, the typical pipeline begins with SFT, then samples candidate responses from the SFT policy and collects preference data for comparison.

3. Training a Reward Model from Rankings

Given a prompt $x$, a winning response $y_w$, and a losing response $y_l$, the reward model outputs a scalar $r_{\varphi }(x,y)$. A common pairwise objective is:

$${\mathcal{L}}_{RM}=-\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)).$$

The score learns relative preferences under the annotation protocol; it is not an absolute quality scale across tasks. A scalar scorer sees one response at a time, so presentation order normally affects it indirectly through biased labels. Length or style can become direct shortcuts in the response itself.

Data auditing must at least include:

• Randomly swapping candidate orderings to detect position bias;
• Recording inter-annotator agreement rates and reasons for disagreements;
• Slicing data by task, language, risk level, and response length;
• Isolating prompts with identical or near-duplicate responses;
• Retaining gold-standard checks and allowing annotators to opt out;
• Versioning labeling guidelines, model candidate sources, and sampling parameters.

4. RLHF Optimizes a Policy with a Reward Signal

The InstructGPT workflow can be summarized as:

1. Train a supervised fine-tuning (SFT) policy using human demonstrations;
2. Train a reward model by ranking model responses;
3. Sample responses from the policy and update the policy using the reward signal, while constraining its deviation from the reference policy.

A high-level objective is often expressed as:

$$\underset{\theta }{max}{\mathbb{E}}_{x\sim D,y\sim {\pi }_{\theta }}[r_{\varphi }(x,y)-\beta D_{KL}({\pi }_{\theta }(\cdot \mid x)\parallel {\pi }_{ref}(\cdot \mid x))].$$

The KL penalty keeps the policy close to the reference model, reducing extreme distribution shifts that would occur if the policy were solely driven by maximizing rewards. PPO is one algorithm that implements such policy optimization, and it includes key components like advantage estimation, ratio clipping, and value loss. Referring to loss = -reward + beta * kl as a complete PPO would obscure these essential steps.

The KL term is a soft penalty, not a hard bound on every response. Its exact form averages over the policy distribution; implementations often estimate it from sampled token log-probability differences. This is the high-level setup described in InstructGPT.

Training logs should simultaneously monitor reward, KL divergence, response length, entropy, clip fraction, value error, and held-out task metrics. Relying solely on reward improvement is insufficient to determine whether the model has improved on the target task.

5. DPO Directly Encodes Preference Targets into the Policy

Direct Preference Optimization (DPO) uses the relationship between a reward model and the optimal policy, transforming preference pairs directly into a classification-style policy objective. A common formulation is:

$${\mathcal{L}}_{DPO}=-\log \sigma \left(\beta [\log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{ref}(y_w\mid x)}-\log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{ref}(y_l\mid x)}]\right).$$

This approach increases the probability that preferred responses are selected over less preferred ones, while anchoring the optimization to a reference policy. Standard DPO does not require training a separate explicit reward model or performing on-policy sampling and PPO within the training loop. However, it does not eliminate preference modeling, instead, the underlying assumptions are embedded directly into the objective function.

beta, reference model, pair quality, sequence log-prob calculations, and length distributions all influence the final results. While DPO is simpler to implement, it does not guarantee superior performance across all dataset sizes and tasks compared to RLHF.

The standard DPO objective uses sums of log probabilities over response tokens, with prompt and padding positions excluded. Replacing sums with length averages changes the objective. The following PyTorch example checks the loss and gradient direction; it does not train a language model.

import torch
from torch.nn import functional as F

# Response-token log-probability sums for one chosen/rejected pair.
policy_logp = torch.tensor([-2.0, -3.0], requires_grad=True)
reference_logp = torch.tensor([-2.0, -2.0])
beta = 0.1
relative = policy_logp - reference_logp.detach()
loss = -F.logsigmoid(beta * (relative[0] - relative[1]))
loss.backward()
print(round(loss.item(), 4))  # 0.6444
print(policy_logp.grad)      # approximately [-0.0475, 0.0475]

Gradient descent raises the chosen log probability relative to the rejected one in this two-variable example. With shared model parameters, both absolute response probabilities can still fall while their ratio improves. The reference weights stay frozen.

6. Reward Hacking and the Goodhart Effect

As long as an agent optimizes a reward signal, it may learn to exploit it:

• Generate longer, more confident responses to earn higher scores labeled "detailed and professional";
• Paraphrase or repeat user input to gain perceived validation;
• Stack disclaimers without actually reducing risk;
• Mimic the style of reference answers while avoiding the core tasks that cannot be completed;
• Craft tailored responses specifically to circumvent fixed judges or evaluation rubrics.

When the policy moves outside the coverage of the training data, extrapolation errors in the reward model grow, leading to situations where reward scores increase while human evaluations decline, this is over-optimization. To mitigate this, use KL divergence-based regularization, early stopping, independent holdout sets, diverse evaluators, adversarial examples, and manual review. Always inspect results after enforcing length constraints.

7. Preference Optimization Does Not Guarantee Factual Accuracy

An annotator's preference for a response only indicates that the chosen answer appears more aligned with the rubric among the visible candidates. If all candidates are incorrect, the annotator cannot verify the facts, or if the reference material is outdated, the training process will reinforce those errors.

Fact-intensive tasks also require:

• Authoritative data sources or executable verifiers;
• Citation and evidence-based validation checks;
• Clear guidelines for uncertainty and abstention;
• Independent fact sets and temporal holdout periods;
• Post-deployment error correction and event feedback loops.

The next lesson on RAG will enable the model to access external information, but it only expands the available evidence, it does not automatically ensure that answers are faithful to the evidence.

8. An Executable Evaluation Protocol

Freeze these before post-training:

• instruction-following, format, and task utility datasets;
• factuality/grounding, safety, and adversarial datasets;
• base capability regression suite;
• prompt/source/user/time group splits;
• decoding configuration and multi-random-seed schemes.

When reporting pairwise win rates, explicitly state the opponent, sampling method, and confidence interval. If using an LLM judge, randomize answer order, hide model identity, and use manually curated samples to calibrate position, verbosity, and self-preference bias. Do not report "a judge's win rate" as evidence of a model's general quality.

Common Misconceptions

• RLHF makes models tell the truth: It optimizes preferences captured by labeled and rewarded data, not truth itself.
• PPO is simply a reward minus KL divergence loss: That’s just a high-level objective; it’s not the full algorithm.
• DPO has no concept of reward: It doesn’t explicitly train a reward model, but its objective is derived from the relationship between preferences, rewards, and policies.
• More preference pairs always lead to better results: Systematic biases can grow with data scale.
• A rising reward signal means it’s safe to release: You must also validate human judgment, safety, factual accuracy, and the preservation of original capabilities.

Exercise

1. Create a pairwise rubric for the JSON extraction task that includes rules for correctness, formatting, and refusal behavior.
2. Explain why placing candidate A on the left consistently biases the reward model.
3. Use a formula to illustrate the role of the reference policy in DPO.
4. Design a length-paired test that exposes verbosity bias.
5. List five fundamental capabilities that must be preserved after preference training.

Summary

Preference-based post-training doesn't install "correct values" into a model, it transforms explicit data distributions and evaluation protocols into behavioral tendencies. SFT learns from demonstration data, the reward model learns relative ranking, RLHF optimizes the policy under constraints of rewards and reference strategies, and DPO directly uses preference pairs. Regardless of the chosen path, the origin of preferences, biases, over-optimization, and independent evaluation must never be omitted.

The next lesson connects external documents to a model and separates RAG into ingestion, retrieval, context construction, and answer validation instead of treating “vector database + prompt” as a complete system.