4.3 Generalization, Learning Curves, and Regularization: Low Training Error Only Proves Optimization Succeeded

A degree-15 polynomial passes through every training point, oscillating wildly between them. The training loss approaches zero, indicating the model has successfully optimized on this specific dataset and its target values. But it does not prove the model has captured any generalizable patterns, its ability to perform well on truly independent data remains to be seen.

Learning Objectives

• Distinguish between optimization, approximation, estimation, and irreducible error;
• Use training and validation learning curves to diagnose issues;
• Understand the scope of applicability of the bias–variance decomposition;
• Select appropriate remedies (regularization, data augmentation, model redesign, or distributional adjustments) based on the root cause of failure.

1. Underfitting and overfitting are phenomena, not causes

Typical observations:

• Both training and validation performance are poor: likely due to high bias, insufficient features, optimization failure, noisy labels, or an inherently unpredictable task;
• Training performance is good, but validation is poor: possibly caused by high variance, data leakage that was fixed, mismatched data distributions, or a small validation set;
• Both training and validation perform well, but production results are still subpar: could indicate incorrect evaluation distribution, feedback loops, implementation bias, or misalignment between metrics and actual actions.

Looking at just one pair of scores is insufficient for a definitive diagnosis. You must combine baseline performance, learning curves, grouped or time-sliced analysis, and the provenance of misclassified samples.

2. Four Sources of Error

• Optimization error: The model hasn't fully converged to a low training loss solution within the hypothesis class.
• Approximation error: The model class is unable to represent the true underlying relationship.
• Estimation error: A finite sample size causes the selected model to deviate from the best generalizing model within the class.
• Irreducible/noise: There remains inherent randomness or measurement error in the labels, even with perfect information.

Increasing model capacity reduces approximation error but may increase estimation difficulty. More effective, independent data primarily helps reduce estimation error. Longer training only addresses optimization error.

3. The Learning Curve Provides More Insight Than a Single Score

Repeat fitting on different sample sizes and compute train/validation loss:

from sklearn.model_selection import learning_curve

sizes, train_scores, valid_scores = learning_curve(
    estimator=pipeline,
    X=X,
    y=y,
    cv=group_or_time_aware_cv,
    scoring="neg_log_loss",
    train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0],
    n_jobs=-1,
)

When interpreting, focus on the mean, inter-fold distribution, and overall trend:

• If both curves remain close at a suboptimal level: the model, features, or target may be insufficient;
• A large gap with validation loss improving as data increases suggests additional similar data could be beneficial;
• Early platforming of validation loss indicates data volume is not the primary bottleneck;
• Deterioration in some time folds may signal data drift, not traditional overfitting.

The training subset must preserve group/time structure to avoid disrupting the split when plotting curves.

4. The Accurate Boundary of Bias–Variance Decomposition

Under standard settings such as squared-error regression and fixed input features:

$$ E[(Y-\hat f(X))^2] = Bias^2 + Variance + Noise. $$

This offers intuitive insight: the variance captures how sensitive a model is to different training samples, while the bias measures the average deviation of the model's predictions from the true regression function.

However, classification error, log loss, and modern overparameterized neural networks do not always adhere to this simple decomposition. The notion that "reducing bias necessarily increases variance" is not a universal rule in practice; engineering changes (such as better features or more data) can often improve both bias and variance simultaneously.

5. Explicit Regularization

Add a complexity penalty to the empirical loss:

$$ \hat R(f) + \lambda \Omega(f). $$

• L2 regularization shrinks parameters and often improves stability;
• L1 can produce sparse solutions, but feature selection remains unstable;
• constrain tree depth and minimum sample size during splits;
• encode inductive biases such as margin, smoothness, and monotonicity.

$\lambda$ must be selected via inner validation or cross-validation. The scale, parameterization, and inclusion of an intercept before applying the penalty all affect its interpretation.

6. Implicit Regularization and Training Process

• Early stopping constrains the optimization trajectory;
• The batch size, learning rate, and noise in SGD influence the solution;
• Network architecture limits the set of expressible functions;
• Dropout introduces random regularization to specific neural network architectures;
• Pruning and quantization primarily serve to compress models, though they may also affect generalization.

These are not universal solutions. Early stopping uses validation data to determine the stopping point, and validation data has already been involved in model selection; therefore, final evaluation must remain on an independent boundary.

7. Data augmentation: Preserve label semantics

Image translation, text rewriting, audio perturbation, these operations declare a certain invariance. If the label should not remain consistent after transformation, augmentation introduces erroneous data: flipping road signs left-right, altering the orientation of medical images, or failing to preserve meaning in negated sentence paraphrasing.

Validation requirements:

• The transformation must be reasonable in real-world deployment;
• Labels or structural integrity must be preserved;
• No data leakage into validation or test sets;
• Performance must not systematically degrade across demographic groups;
• The performance gain must stem from the target distribution, not from artificial features in the augmented samples.

8. Label Noise and Hard Samples

Samples with high loss may be:

• Truly rare but important;
• Mislabeled or corrupted input;
• Out-of-distribution;
• Samples the model has not yet learned to handle.

Automatically removing "hard samples" risks eliminating minority classes and true decision boundaries. Before discarding any sample, trace its origin, verify its correctness, and document the handling process. Robust loss functions, label smoothing, and noise models rely on assumptions about label noise and cannot substitute for label auditing.

9. Distribution Shift Is Not Ordinary Variance

Good generalization between training and validation data does not guarantee robustness under changes in covariate shift, label shift, concept drift, or policy changes.

P(X) changes: variation in input population or devices
P(Y) changes: shift in baseline rate
P(Y|X) changes: alteration in relationship or definition
selection changes: different individuals are observed or labeled

Slice by time, location, and key demographic groups; establish fresh holdout sets, shadow evaluations, and post-deployment monitoring. Regularization cannot correct for erroneous target distributions.

10. Action Plan by Symptom

Evidence	Priority Action
Training and validation performance both worse than a simple baseline	Investigate the task, features, optimization strategy, and labeling process
Performance gap shrinks noticeably as sample size increases	Collect more similar and independent data
Large performance gap with excessive model capacity	Apply regularization, simplify the model, or improve data splitting stability
System crashes at specific times or locations	Fix distribution coverage issues or restrict usage to safe contexts
Poor quality labels for minority groups	Improve data collection and labeling practices, do not rely solely on model tuning
Discrepancy between production and offline performance	Audit pipeline consistency and feedback loops

Common Misconceptions

• Low training error means the model has learned the pattern: It might have merely memorized the training samples or suffered from data leakage.
• Underfitting is just a matter of using a more complex model: Poor optimization, insufficient or inaccurate labels, and lack of informative features can also cause underfitting.
• All forms of regularization prevent overfitting: Regularization must be aligned with the specific cause of overfitting and the model architecture.
• More data is always beneficial: If the data contains biased error distributions or incorrect labels, adding more data can actually exacerbate the problem.

Exercise

1. Plot training and validation curves for polynomial regression, rather than reporting training MSE alone.
2. Construct two cases (one with underfitting and one with high bias) where both training and validation performance are poor.
3. Compare the stability of feature selection under L1 and L2 regularization when features are correlated.
4. Explain why adding older data might worsen performance in a time-drifting scenario.

Summary

Generalized diagnosis requires considering training performance, independent validation, sample size, and distribution shifts together. Regularization is just one way to control model behavior; task design, label consistency, data coverage, and deployment alignment typically set the ultimate limits.

The next chapter moves into linear models: starting from squared loss and probabilistic modeling, we'll understand linear regression and logistic regression and their optimization, rather than simply invoking an fit().