11.1 Tokenizer, Special Tokens and Long Text Chunking: What the Model Sees Are Token IDs, Not Raw Text

A checkpoint expects a particular tokenizer protocol: vocabulary IDs, normalization, boundary tokens, and chat format. Swapping only the tokenizer changes what the same model weights receive.

Learning Objectives

• Dissect the components of normalization, pre-tokenization, subword modeling, and post-processing;
• Distinguish between BPE, WordPiece, Unigram, and byte-level tokenization designs;
• Properly handle special tokens, padding, truncation, and loss masking;
• Design document-aware chunking for long texts;
• Evaluate the efficiency of multilingual or domain-specific tokens and character alignment.

1. Tokenization Is a Pipeline

Common stages:

raw text
  -> Unicode / case / whitespace normalization
  -> pre-tokenization (whitespace, punctuation, byte boundaries)
  -> subword segmentation
  -> vocabulary IDs
  -> BOS/EOS/SEP/role tokens
  -> padding / truncation / attention mask

Different tokenizers may preserve case or convert to lowercase; they may apply Unicode normalization or retain the original byte sequence. It is not safe to assume that decoding produces the exact original text: normalization, unknown tokens, and whitespace cleanup can all affect the round-trip fidelity.

For span labeling and citations, retain the original text and check the backend’s offset convention. With a tokenizer that lowercases and strips accents, Café may become cafe while its offset still points to the original span. Normalization is not necessarily reversible; special tokens may have an empty offset such as (0, 0).

2. Why Not Use "One Word, One ID"

A word-level vocabulary is fragile when dealing with morphological variations, spelling errors, code identifiers, or open-vocabulary terms. A character- or byte-level vocabulary is smaller but supports longer sequences. Subword tokenization strikes a balance between vocabulary size and sequence length.

BPE

BPE starts with small units and iteratively merges frequent adjacent pairs. In practice, its behavior is also influenced by the pre-tokenizer, byte-level mapping, and merge tie-breaking rules.

WordPiece

WordPiece also builds subwords, but its vocabulary learning criteria and encoding details differ from BPE. ## is merely a continuation representation used in certain BERT tokenizers and does not represent the general behavior of WordPiece.

Unigram Language Model

This approach begins with a candidate vocabulary and selects or prunes subword pieces based on probabilistic segmentation likelihood. The same input text can yield multiple valid segmentations.

Byte-level

Covering all 256 byte values, directly or through byte fallback, allows valid Unicode text to be represented through UTF-8 bytes without unknown characters. Merely calling a tokenizer “byte-level” does not prove that coverage. A character can still consume several bytes or tokens.

"The idea that 'GPT uses one BPE and BERT uses one WordPiece' is overly simplistic." You must inspect the actual tokenizer files of specific model checkpoints to verify this.

3. Vocabulary ID Has No Numerical Order

ID 50257 is not semantically greater than ID 100. The embedding table uses IDs as indices to access learned vectors; reordering the vocabulary without simultaneously reordering the embedding or output head weights completely alters the model's behavior.

Many causal language models tie the input embedding weights to the output projection weights. After adding new tokens, you must:

1. Update the tokenizer;
2. Resize the model's embeddings and output head;
3. Initialize and train the new rows in the embedding table;
4. Save the tokenizer and model in the same version;
5. Verify that the serving engine loads the updated vocabulary.

Simply calling add_tokens will not automatically assign semantic meaning to new tokens.

4. Special Tokens Are Protocol Control Symbols

Common examples include:

• BOS/EOS: beginning and end of a sequence;
• PAD: padding to fill batches;
• MASK: marks selected tokens to reconstruct;
• SEP/CLS: special tokens used to structure encoder input;
• role/control tokens: define boundaries between system, user, assistant, and tool roles;
• sentinel tokens: mark missing spans for corruption or infilling.

A visible string that is not registered as a special token may split into ordinary pieces. Duplicate BOS/EOS or a missing assistant-start token changes the sequence. Some causal models intentionally share PAD and EOS IDs; their masks must distinguish padding positions from real EOS targets.

Do not treat ordinary user-input strings as immutable or unforgeable boundaries of permission; role and control token parsing and message role assignment must be constructed at the application layer.

5. Padding Mask and Loss Mask Are Not the Same Thing

The tokenizer's attention_mask typically indicates which input positions contain valid tokens; the model or architecture then transforms this into an actual attention mask. Training labels must independently determine which positions contribute to the loss calculation.

In instruction tuning scenarios, the following patterns may occur:

system/user tokens: Can be attended by the assistant, labels = -100 (loss not counted)
assistant tokens:   Can be attended, labels = token IDs (loss is computed)
padding tokens:     Should not be attended, labels = -100

Training only the assistant response is an objective choice. Here -100 is the ignore index expected by the loss implementation, not a tokenizer ID or a universal masking protocol. Attention visibility and loss supervision remain separate.

Left/right padding affects position IDs, cache management, and generation batching, and must align with the checkpointing and serving implementation.

6. Truncation Can Remove Questions or Answers

truncation=True enforces a length limit; the configured side and strategy determine what is removed. Pair strategies longest_first, only_first, and only_second differ. Right-truncating a chat can remove the latest user question.

Record the following:

• Original token count versus post-truncation token count;
• Proportion of samples that are truncated;
• Retention strategy for each field;
• Whether BOS, EOS, or role tokens remain intact;
• Whether labels are truncated;
• Consistency between train and serve truncation side configurations.

The model’s context limit, tokenizer model_max_length, and serving configuration may not align. Furthermore, large placeholder values specified in configurations should not be treated as actual usable context windows.

7. Long Documents First by Document Split, Then by Chunks

The correct workflow:

documents -> split into train/valid/test by source, time, or entity -> tokenize and chunk within each split

If chunks are created first and then randomly split, overlapping windows or adjacent segments from the same document will span across train and test sets, leading to near-duplicate data leakage.

Chunk design considerations include:

• boundary: sentence, paragraph, code unit, or fixed token count;
• maximum token budget and allocation for special tokens;
• overlap or stride;
• document ID, offset, timestamp, and license metadata;
• whether short documents are packed across document boundaries;
• EOS or document boundary signals for causal language models.

Overlap improves boundary recall but also increases redundant computation and complicates evaluation correlation. Metrics and bootstrapping should be computed per document or source, not by treating overlapping chunks as independent samples.

EOS marks a boundary but does not block attention across packed documents. If documents must remain independent, use a compatible block-diagonal attention mask or separate sequences. Allowing cross-document context is a separate, explicit training choice.

8. An Auditable Encoding Example

This fragment requires tokenizer_dir to name a local, versioned tokenizer artifact. It was checked with Transformers 5.16.1 and a small local tokenizer; vocabulary-dependent output will differ.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    tokenizer_dir, local_files_only=True, use_fast=True,
)
if not tokenizer.is_fast:
    raise ValueError("this example requires a fast tokenizer with offsets")

encoded = tokenizer(
    "First segment of task log. Second segment contains alerts.",
    add_special_tokens=True,
    truncation=True,
    max_length=32,
    return_attention_mask=True,
    return_offsets_mapping=True,
    return_special_tokens_mask=True,
)

print({
    "tokens": tokenizer.convert_ids_to_tokens(encoded["input_ids"]),
    "ids": encoded["input_ids"],
    "attention_mask": encoded["attention_mask"],
    "special_mask": encoded["special_tokens_mask"],
    "offsets": encoded["offset_mapping"],
})

Fast tokenizer often supports offset mapping, but specific backend/checkpoint implementations must be verified. In production, model IDs should not be dynamically resolved at runtime to download the latest files; instead, fixed revisions, hashes, licenses, and local artifacts should be used.

9. Token Efficiency Must Be Measured

Differences in tokens per character or tokens per byte across different tokenizers can significantly impact:

• The amount of information that fits within a context window;
• Training and fine-tuning computational requirements;
• API or serving token costs;
• Response length and latency across languages;
• Fragmentation of code, numbers, URLs, and rare terms.

There is no stable rule that English words consistently map to 1–2 tokens per word while Chinese characters map to exactly 1 token per character. Real-world, domain-specific, and mixed-language text corpora must be used to measure length quantiles, unknown or byte fallback behavior, compression ratios, and truncation rates.

10. Tokenizer Training Is Also Data Governance

Vocabulary frequencies reflect the training mixture. Language proportions affect which pieces are economical to encode; PII, secrets, and control strings in the corpus also require artifact-level inspection.

When training or selecting a tokenizer, document the following:

• normalization/pre-tokenizer/configuration;
• source of the corpus, license, user consent, and timestamp;
• per-language or per-domain sampling weights;
• deduplication and filtering strategies;
• vocabulary size and reserved tokens;
• stability and compatibility policies.

The presence of strings within a tokenizer's vocabulary does not mean the model has learned or retained factual knowledge. Additionally, the tokenizer artifact itself may contain sensitive fragments that could be exposed, these must be audited.

Common Misconceptions

• Tokenizer is merely a preprocessing step before deployment: It is part of the checkpoint's input protocol.
• Subword tokens can perfectly reconstruct any original text: Normalization, unknown token handling, and cleanup operations can alter the input text.
• PAD mask automatically suppresses training loss: Training loss is masked by attention and label masks separately.
• Enabling truncation ensures safety: It may inadvertently remove the latest question or supervised answer.
• Each Chinese character is a single token: Efficiency depends on the specific vocabulary and text characteristics.

Exercise

1. Compare the length distributions of two pinned tokenizers across Chinese, English, code, and URL inputs.
2. Investigate how encode→decode and offset mappings change under whitespace and Unicode normalization.
3. Design a truncation strategy for pair/chat inputs that preserves the integrity of questions and answers.
4. First, chunk and then split, observing overlap leakage; then switch to a document-first split approach.
5. After adding domain tokens, verify compatibility of embedding resizing, persistence, and serving.

Summary

A tokenizer converts raw text into IDs that the model actually receives, and defines the input protocol through special tokens, masks, padding, and truncation. It determines sequence length, character alignment, and deployment compatibility, while also inheriting the governance challenges from the training corpus.

The next lesson explores how these token sequences generate supervision signals: causal, masked, and denoising objectives derive different information flows and model capabilities from the same original text.