11.3 Full-Parameter Fine-Tuning, LoRA, and Domain Adaptation: Fewer Trainable Parameters Don’t Mean Lower Risk or Reduced VRAM Usage

Ah Hua’s base model can generate general text but doesn’t understand the task formatting required by the model workshop. The team has three options: continue training on domain-specific data, use annotated instructions to reconfigure behavior, or freeze most of the weights and only allow small adapter layers to learn.

Fine-tuning isn’t simply “feed it hundreds of examples and it’ll learn the new task.” It modifies parameters under a specific data/objective context. The quality, coverage, forgetting, and validation protocols ultimately determine the outcome.

Learning Objectives

• Distinguish between continued pretraining, task fine-tuning, and instruction tuning;
• Compare head-only, full fine-tuning, and PEFT (Parameter-Efficient Fine-Tuning);
• Derive the low-rank update formulation and trainable parameter count for LoRA;
• Correctly construct chat labels, split datasets, and evaluate model performance;
• Version base models, adapter layers, tokenizers, and serving configurations.

1. Select the Target Approach

Continued pretraining / Domain-adaptive pretraining

Continue training on domain-specific text using self-supervised objectives like causal language modeling or masked language modeling (MLM) to improve terminology and distribution modeling. This approach does not directly teach the model how to follow instruction formats and may introduce domain bias or cause the loss of general-purpose capabilities.

Task supervised fine-tuning

Train the model on explicit input–output pairs for tasks such as classification, information extraction, or generation. This can be achieved by adding a task-specific head or using text-to-text or causal loss functions.

Instruction tuning / SFT

Train the model to respond in a structured dialogue or task format using system/user/assistant or instruction–response examples. This improves behavioral alignment, but does not guarantee factual accuracy or safety in generated responses.

Preference/alignment post-training

Use comparative data or reward signals to shape model preferences and behavior, this topic is covered in Chapter 12. Do not conflate SFT, RLHF, and LoRA into a single dimension: SFT defines the objective and training data, while LoRA specifies a parameter update method.

2. Three Parameter Update Ranges

Head-only / Feature extraction

Freeze the backbone and train only the classification or projection head. This approach is cost-effective and minimizes knowledge forgetting, but the learned representation remains unable to adapt to deep domain mismatches.

Full fine-tuning

Update all model weights, offering the largest capacity. However, this requires storing optimizer states, gradients, and activations, making checkpointing and serving more resource-intensive. With small datasets and high learning rates, overfitting or degradation of original capabilities can occur.

PEFT

Freeze most base model weights and train only adapters, prompt/prefix parameters, or low-rank updates. This reduces the number of trainable parameters and lowers memory usage for optimizers and gradients. Still, forward and backward passes go through the base model, so activations and base-weight memory do not diminish proportionally to the trainable parameter ratio.

3. LoRA limits updates to low-rank modifications

For frozen weights $W_0\in\mathbb R^{d_{out}\times d_{in}}$:

$$ W=W_0+\Delta W,\qquad \Delta W=sBA, $$

where:

$$ A\in\mathbb R^{r\times d_{in}},\qquad B\in\mathbb R^{d_{out}\times r}, $$

$r\ll\min(d_{in},d_{out})$, and $s$ is typically determined by lora_alpha/r or a variant thereof. The trainable parameter count reduces from $d_{out}d_{in}$ to:

$$ r(d_{in}+d_{out}). $$

Low rank represents a structural assumption about how task-specific updates should be applied. A rank that is too low may lead to underfitting, while a rank that is too high increases computational cost. Careful validation is required for which target modules to update, scaling factors, dropout settings, and whether bias terms should be trainable.

4. Where LoRA Is Targeted Matters More Than Just "Using LoRA"

LoRA can be applied to various components such as the attention layer's Q/K/V/O projections, FFN layers, embeddings, or heads. Module names vary depending on the model architecture, and incorrect targeting patterns may include:

• No matching layers found;
• Only a small number of unintended layers being trained;
• Inclusion of output heads that should remain unmodified;
• Name changes due to tensor parallelism or quantization wrappers.

Before training begins, print the names and counts of trainable parameters, and perform a backward pass check to verify that frozen parameters (e.g., .grad is None) have zero gradients and that adapter gradients are finite and non-zero.

5. A PEFT Construction Skeleton

from peft import LoraConfig, TaskType, get_peft_model

# base_model Fixed as specified revision Load and with tokenizer Match.
config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],  # Must be verified against the specific architecture
    bias="none",
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()

This builds an adapter, not a full training pipeline. Package and API changes rapidly; projects must lock in a specific transformers/peft/accelerate version and base revision, and store the resolved configuration.

6. Quantized Base + LoRA

QLoRA-style approaches store frozen base weights in low-precision/quantized form and train LoRA adapters to further reduce weight memory usage. However, several key issues remain to be addressed:

• Compute dtype versus accumulation precision;
• Quantization scheme and block-level statistics;
• Which modules should retain high precision;
• Optimizer and activation memory overhead;
• Hardware and kernel-level support;
• Quality and memory implications after merge and dequantization;
• Consistency between training and serving quantization.

"4-bit fine-tuning" does not mean that all tensors, computations, or checkpoints are stored in 4-bit format, nor does it guarantee equivalence to full fine-tuning.

7. SFT Formatting and Label Masking

A chat sample is a message structure, rather than a string concatenation alone. When using a checkpoint's chat template to generate control tokens, it's essential to explicitly define loss positions.

One example of assistant-only loss:

<system> ... </system>       labels = ignore
<user> ... </user>           labels = ignore
<assistant> response ...     labels = token IDs

Other models train on all non-padding tokens across the full conversation. The choice must align with the objective.

High-risk errors:

• Training template differs from serving template;
• Repeated BOS/EOS tokens;
• Response truncation leaving only the prompt;
• Prompt tokens are not masked but incorrectly labeled as assistant-only;
• Packed examples exhibit cross-boundary attention;
• Padding labels lack an ignore index.

Write unit tests to decode input and labels, and visualize the supervision mask token by token.

8. Data Quality Matters More Than Sample Size

Check the following:

• Whether the instruction is clear and actionable;
• Whether the response is accurate, sourced reliably, and consistently formatted;
• Whether multiple conflicting answers arise from the same input;
• Whether model-generated data has been validated against ground truth;
• Whether safety refusals are balanced with normal helpful responses;
• Whether the data covers language, length, domain, and difficulty variations;
• Whether personally identifiable information (PII), licenses, or consent requirements are properly handled, including deletion of data lineage;
• Whether near-duplicate responses or benchmark contamination is present.

"There are enough samples if you have hundreds to thousands" is not a universal guarantee. The required sample size depends on task complexity, base model capability, noise levels, and acceptance thresholds. Decision-making should be guided by learning curves and error taxonomy.

9. Split: Group by Source and Template to Deduplicate

After generating ten QA pairs from the same document, split them randomly by line, this approach causes train and test sets to share source facts and stylistic patterns. Even swapping only the entity names in the template leads to near-duplicate outputs.

Instead, group splits by document, source, user, time, or task family, and apply near-deduplication to instructions and responses. When the goal is to evaluate future knowledge or new customer segments, use temporal or domain holdout strategies.

Retain the following:

• in-domain validation: for hyperparameter tuning;
• held-out tasks or templates: to enable combination-based generalization;
• safety and adversarial test sets;
• base capability regression suite;
• final blind/private test.

10. Catastrophic Forgetting and Trade-offs

Fine-tuning may improve performance on the target task while harming:

• General language or multilingual capabilities;
• Calibration and uncertainty expression;
• Safety refusal behaviors;
• Formatting and tool calling;
• Long context retention;
• Previously established domain expertise.

Mitigation strategies include smaller learning rates or fewer training steps, regularization, replay or mixed data, adapter isolation, multi-task training, and selective layer updates. None of these can replace regression evaluation.

Adapters enable task isolation and switching, but the interactions between multiple adapters when combined or merged also need to be tested.

11. Training Diagnostics

• Train/validation token loss, stratified by response/source;
• Supervised token count, rather than example count alone;
• Truncation and empty-label rates;
• Gradient and adapter update norms;
• Base versus tuned outputs on fixed prompts;
• Exact, semantic, and task-level metrics compared to human evaluation rubrics;
• Memorization and copy rates;
• Throughput, peak memory usage, and checkpoint size.

A decrease in loss does not necessarily indicate improved instruction following or factual accuracy. Generation metrics must be evaluated using a frozen decoding configuration and should include variability across multiple seeds and samples.

12. Deployment and Version Contract

The adapter artifact must reference:

• the exact base model ID, revision, or hash;
• the revision of the tokenizer and chat template;
• the PEFT configuration, target modules, rank, and scaling factors;
• the merge status and data types (dtype) or quantization settings;
• the training data version, version range, and associated metrics;
• the license and usage restrictions.

Serving unmerged adapters may introduce additional module management overhead and minor latency; merged weights reduce flexibility in switching between adapters and require re-generation and full checkpoint validation. Logits before and after merging should be compared within a defined tolerance.

Common Misconceptions

• LoRA is a training objective: It is a parameterization or update method, not a training goal.
• 1% trainable parameters means only 1% VRAM usage: The base weights and activations still consume significant resources.
• LoRA quantization is always 4-bit throughout the entire process: Computation, adapter layers, and optimizers may require higher precision.
• Fine-tuning only improves capabilities: It can lead to forgetting, safety regressions, and calibration drift.
• Adapters can be loaded onto any base model with the same name: They must match a specific revision or architecture.

Exercise

1. Specify the data/objective/update scope for continued pretraining, supervised fine-tuning (SFT), and LoRA.
2. Calculate the number of LoRA parameters for a $4096\times4096$ projection at ranks 8, 16, and 64.
3. Print and verify the actual target modules and gradients.
4. Decode an SFT sample and annotate each token with the loss mask.
5. Compare the target objectives of base fine-tuning, full fine-tuning, and LoRA against the regression suite.

Summary

When adapting a foundation model, start by selecting the objective, then define the parameter update scope. Full fine-tuning offers the greatest flexibility, while LoRA reduces trainable parameters through a low-rank assumption, and quantization further compresses base memory. None of these approaches can skip data governance, data splitting, regression evaluation, or version contracts.

In the next lesson, we won’t modify weights at all: instead, prompts and in-context examples alter the conditional distribution via input sequences. This approach deploys quickly and relies heavily on templates, context, generation configurations, and continuous evaluation.