11.2 Causal, Masked, and Denoishing Pretraining: Self-supervised Labels Come from Data, Not from the Absence of Assumptions

The tokenizer on disk converts corpus text into token IDs. But Ah Hua’s training logs reveal three distinct production lines: one that predicts the next token, another that reconstructs masked positions, and a third that restores damaged spans back into complete text.

None of these require manual annotation of individual examples, yet they are not "the model learning arbitrarily." Corruption, masking, factorization, and data mixing explicitly define what the model sees and what it is expected to predict.

Learning Objectives

• Derive the conditional decomposition and label shift in causal language modeling;
• Understand the corruption mechanism and bidirectional context in masked language modeling;
• Distinguish between span-denoising and encoder–decoder training objectives;
• Accurately compare encoder-only, decoder-only, and encoder–decoder architectures;
• Incorporate data deduplication, corruption, licensing, and memorization into pretraining evaluation.

1. Self-supervised learning is not a synonym for unsupervised learning

Self-supervised learning constructs targets from within the raw sample: next token, masked token, rotation angle, missing span. It requires no manual labeling, but the objective function, data sampling strategy, and corruption mechanisms are all deliberately designed by humans.

Language models learn conditional statistics under the training distribution and loss function. They may acquire transferable representations and factual patterns, but they also inherit biases, errors, redundancies, and privacy risks. The claim that "reading the entire internet therefore understanding all language" remains unverified.

2. Causal Language Modeling

For a token sequence $x_{1:T}$:

$$ p(x_{1:T})=\prod_{t=1}^{T}p(x_t\mid x_{<t}). $$

Negative log-likelihood:

$$ L_{CLM}=-\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t}). $$

Training inputs and labels are typically shifted: the hidden state at position $t$ predicts the next token. A causal mask prevents access to future tokens.

import torch
from torch.nn import functional as F

# logits: [B, T, V], token_id: [B, T]
def causal_lm_loss(logits, token_id, padding_id):
    prediction = logits[:, :-1, :].contiguous()
    target = token_id[:, 1:].contiguous()
    return F.cross_entropy(
        prediction.view(-1, prediction.shape[-1]),
        target.view(-1),
        ignore_index=padding_id,
    )

Real model classes may internally handle the shift, so manual shifting should not be applied twice. Review the specific forward contract and verify position–target alignment using manually constructed short sequences.

3. Teacher Forcing and Generation Differences

During training, each position's context is drawn from the true prefix, enabling parallel computation of all token losses. In contrast, during generation, the context includes tokens sampled by the model itself; errors in early tokens can propagate and alter the distribution of subsequent outputs.

A low perplexity achieved through teacher forcing does not guarantee coherence, factual accuracy, or adherence to task requirements in long sequences. Generation is also influenced by decoding strategies, context construction methods, and post-training modifications.

4. Masked Language Modeling

Masked Language Modeling (MLM) selects certain positions to corrupt, allowing the encoder to recover the original token from the surrounding left and right contexts:

$$ L_{MLM} = -\sum_{t\in M} \log p_\theta(x_t \mid \widetilde{x}). $$

The original BERT model uses a fixed ratio and a specific corruption strategy involving MASK, random, or unchanged tokens; other masked language models may employ different ratios and strategies. Do not describe "randomly masking 15%" as part of the definition of MLM.

MLM enables bidirectional context, making it well-suited for representation learning, classification, and token-level labeling. Since pretraining only computes loss on selected positions, the corruption distribution does not align with the downstream text that remains unmasked, this mismatch is a known limitation. Subsequent methods address this by modifying the objective or the training data, but this does not invalidate the fundamental effectiveness of MLM.

5. NSP is a historical approach, not a binary conclusion

The original BERT model included Next Sentence Prediction (NSP) to determine whether two sentences were consecutive. Later models either removed NSP entirely, replaced it with sentence-order or contrastive objectives, or leveraged cross-sentence training for specific tasks.

Therefore, a more accurate statement is: the value of NSP depends on data construction, model architecture, and downstream task characteristics; it is incorrect to simply conclude "it has been found to be unimportant," nor should NSP be treated as a mandatory component of all encoder pretraining approaches.

6. Denoising / Span Corruption

Encoder-decoder denoising begins by corrupting the input and then uses the decoder to autoregressively reconstruct the target. For example, it can remove or replace consecutive spans and marks the gaps with sentinel tokens.

This approach simultaneously trains:

• the encoder to process a corrupted source;
• the decoder to recover content based on the source and the previously generated target;
• cross-attention alignment between source and target.

It is well-suited for text-to-text transfer tasks, but the training cost is determined by generation loss, source/target length, and corruption strategy. Denoising does not automatically provide factual grounding.

7. Three Common Architectures Are Not Absolutely Task-Bound

Typical Family	Attention Visibility	Common Objective	Common Use Cases
Encoder-only	Bidirectional / Full	MLM, contrastive learning	encoding, classification, retrieval
Decoder-only	Causal	next-token language modeling	generation, in-context tasks
Encoder–decoder	source full + target causal	denoising / sequence-to-sequence	conditional generation, transformation

This represents common architectural patterns, not mathematically exclusive categories. Architectures like Prefix LM, unified transformers, and non-causal decoders blend information flows across layers. When selecting a checkpoint, evaluate the specific architecture, configuration, and objective, rather than relying solely on the model's brand name.

8. Data Mixture Is an Invisible Objective

The overall loss is a weighted sum of losses from different data sources or token types. Sampling weights determine how much update each type of text receives:

$$ L=\sum_d\lambda_d\mathbb E_{x\sim D_d}[L(x)]. $$

The proportions of web content, books, code, papers, conversations, and content in different languages influence both capabilities and biases. The resulting mixture varies depending on whether sampling is done by raw bytes, documents, or tokens.

Record source provenance, timestamps, licenses, language or domain classification labels, filter thresholds, deduplication decisions, and mixture scheduling. Without a data card, a "dozens of TB" dataset cannot be reproduced or audited.

9. Deduplication and Benchmark Contamination

Duplicate content can lead to:

• Overweighting of high-frequency documents;
• Increased risk of memorization or privacy leakage;
• Overly inflated validation/test metrics that closely mirror training data;
• Repeated sampling of copyrighted material.

Exact hashing only removes content that is textually identical; near-deduplication requires techniques such as shingling, MinHash, or embeddings, along with careful selection of granularity and threshold. Code snippets, templates, references, and translations introduce complex boundaries.

Benchmark decontamination must be performed from multiple perspectives (tokenizer-level, normalized, and raw text) while documenting the methods used and any missed cases. Simply excluding benchmarks based on filename is insufficient.

10. Perplexity's Comparison Boundaries

The average token cross-entropy $H$ corresponds to:

$$ PPL = e^H. $$

It depends on tokenization, evaluation text, context length, stride or boundary settings, and whether special or padding tokens are included. Since token units vary across vocabularies, PPL cannot be directly compared across models.

In fixed-window models, evaluating long texts with disjoint chunks results in loss of contextual continuity. Sliding windows better approximate real conditional context, but at the cost of redundant computation. The evaluation protocol must be explicitly reported.

Perplexity measures next-token prediction likelihood and does not equate to factual accuracy, safety, calibration, or task utility.

11. Memorization and Privacy

Language modeling requires remembering patterns, and certain sequences may be verbatim memorized. The risk evolves with repetition, rarity, model capacity, and training dynamics.

Governance measures include:

• Secret or PII detection and source filtering prior to training;
• Deduplication and deletion lineage tracking;
• Canary, extraction, and membership audits;
• Privacy and safety controls on outputs;
• Data subject rights and licensing processes;
• Access controls to checkpoints and incident response protocols.

Filtering does not guarantee complete removal. Model weights are not queryable like a row-by-row database. Deletion requests may require retraining or unlearning strategies, along with well-defined evidence boundaries.

12. Pre-training Evaluation Is Multidimensional

• held-out loss/perplexity (strict decontamination);
• downstream transfer and few-shot tasks;
• factuality and temporal cutoff;
• robustness, calibration, and long-context capabilities;
• language and domain coverage;
• bias, toxicity, privacy, and security;
• compute requirements, throughput, energy efficiency, and failure rates.

The more test sets available, the more likely the evaluation will align with benchmark expectations. Preserve truly blind, private, or future-facing evaluations and document the full history of model and data selection.

Common Misconceptions

• Self-supervised learning has no labels or human assumptions: Targets are automatically generated, but the objective function is still defined by humans.
• BERT is a permanent template of MLM + NSP: The specific encoder objectives are highly diverse and context-dependent.
• GPT training and generation are identical: During training, the model looks at real prefixes; during generation, it uses its own output as the prefix.
• Perplexity can be directly compared across tokenizers: Differences in tokenization units and protocols make such comparisons invalid.
• More data always leads to better performance: Data quality, redundancy, licensing, bias, and mixture design are just as critical as quantity.

Exercise

1. Manually align CLM inputs, logits, and next-token labels to check for off-by-one errors.
2. Design three types of MLM corruption and explain how they illustrate a mismatch between pretraining and downstream tasks.
3. Provide an example of span corruption and draw encoder and decoder masks to illustrate the masking pattern.
4. Explain why it's not valid to directly compare PPL values of the same text across different tokenizers.
5. Create a checklist for a pretraining corpus that includes provenance tracking, deduplication, and contamination from benchmark datasets.

Summary

Causal, masked, and denoising objectives generate different supervision signals from the original text. The visibility of information, the type of corruption, and the data mixture determine the conditional task the model learns; a low loss is only a sign, further auditing of contamination, memory retention, transfer, and safety is required.

The next lesson begins with a foundational checkpoint: whether to continue pretraining, perform full-parameter fine-tuning, or train only low-rank adapters depends on the data, computational resources, risk of forgetting, and deployment strategy.