6.3 Regularization, Cross-Validation, and Prediction Intervals: Model Selection Can't Look at the Answers

Training error can fall while error on new data rises. Possible causes include overfitting, distribution shift and leakage. Regularization limits flexibility: fit preprocessing and coefficients on each training fold, then use its validation fold to compare regularization strengths. Reserve the final test set until the selection procedure is fixed.

This lesson's objectives

• Distinguish the objectives of ridge, lasso, and elastic net
• Use a pipeline to include preprocessing within cross-validation;
• Split based on independent, grouped, or time-based structures;
• Report baseline, prediction error, and prediction interval.

1. Ridge and Lasso Penalty Differences

Ridge:

$$\underset{\beta }{min}\sum _i(y_i-x_i^T\beta {)}^2+\lambda \sum _j{\beta }_j^2.$$

Lasso:

$$\underset{\beta }{min}\sum _i(y_i-x_i^T\beta {)}^2+\lambda \sum _j|{\beta }_j|.$$

• Ridge continuously shrinks feature coefficients, typically not reducing them to exactly zero;
• Lasso can produce sparse solutions, but feature selection among correlated features may be unstable;
• Elastic Net combines L1 and L2 regularization, often providing more stability when features are correlated.

Intercepts are typically not penalized; specific library behavior should be verified. Penalties alter the optimization objective; regularization coefficients cannot be directly interpreted using ordinary OLS p-values.

The formulas above use an unnormalized sum of squared errors. Library parameters need not use that convention: scikit-learn Ridge uses SSE plus an L2 penalty, while Lasso uses SSE divided by $2n$ plus alpha times the L1 norm. Equal alpha values across those estimators do not mean equal penalty strength.

2. Scaling Changes the Meaning of a Penalty

A feature scale measured in bytes versus one measured in proportions can differ greatly. The same regularization coefficient represents different functional changes on the original scale, so it's typical to standardize numerical features before regularization.

Standardized parameters can only be fitted within each training fold. scikit-learn pipeline example:

The example uses scikit-learn 1.4 or later and runs in order. It reserves four entire teams for final testing and uses the other 16 teams for model selection. Missing numeric values are imputed inside each fold. This fixture has no missing categories; unknown prediction categories map to an all-zero one-hot block, which is a fallback representation rather than a learned understanding of the new category.

import numpy as np
import pandas as pd

rng = np.random.default_rng(31)
frame = pd.DataFrame({
    "team_id": np.repeat(np.arange(20), 5),
    "team_size": rng.integers(2, 9, 100).astype(float),
    "resources_used": rng.uniform(10, 50, 100),
    "mission_type": rng.choice(["survey", "repair"], 100),
})
y = 30 - 2 * frame["team_size"] + .4 * frame["resources_used"] + rng.normal(0, 3, 100)
frame.loc[3, "team_size"] = np.nan
features = ["team_size", "resources_used", "mission_type"]
train_mask = frame["team_id"] < 16
X_train, y_train = frame.loc[train_mask, features], y.loc[train_mask]
X_test, y_test = frame.loc[~train_mask, features], y.loc[~train_mask]
team_ids_train = frame.loc[train_mask, "team_id"]

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["team_size", "resources_used"]
categorical = ["mission_type"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), categorical),
])

pipeline = Pipeline([
    ("preprocess", preprocess),
    ("model", Ridge()),
])

Fitting preprocessing on all data before cross-validation lets validation values influence scaling, imputation or learned category vocabularies. A pipeline moves those fits inside the split; preprocessing performed before the pipeline can still leak.

3. Three Layers of Data Responsibility

• training: fitting parameters;
• validation/CV: Select features, models, and hyperparameters;
• test: Estimate generalization performance only after the final solution is fixed.

Keep checking test results to tweak the model, turning test into a validation set. Either preserve a new independent dataset or honestly report that evaluations are overly optimistic.

Nested cross-validation puts hyperparameter selection in the inner loop and evaluates the full selection procedure in the outer loop. This reduces optimism from selecting and scoring on the same folds. It still needs splits that match the deployment population and dependence structure; nested does not mean automatically unbiased.

4. Split units must match the application

Independent Samples

For independent, exchangeable observations, shuffled K-fold is a reasonable starting point. Check that each fold has enough observations and relevant groups; ordinary K-fold does not guarantee their balance.

Repeated Entities

To estimate performance on new users or teams, keep each entity entirely in one fold. Predicting future records for already-known entities is a different target: retaining those entities may be appropriate, but time and label-availability boundaries must still hold.

Time Prediction

Training must come before validation; random shuffling would use future distributions to predict the past. Use rolling or expanding windows, and introduce a necessary gap between the feature and label windows.

Spatial or site generalization

For deployment to unseen sites, hold out complete sites or spatial blocks. Randomly splitting nearby rows may leave shared environmental information in both sets. The held-out unit should match the unit that will be new in deployment.

5. Hyperparameter Selection Includes Uncertainty

from sklearn.model_selection import GridSearchCV, GroupKFold

search = GridSearchCV(
    pipeline,
    param_grid={"model__alpha": [0.01, 0.1, 1.0, 10.0, 100.0]},
    scoring="neg_mean_absolute_error",
    cv=GroupKFold(n_splits=5),
)
search.fit(X_train, y_train, groups=team_ids_train)

Scoring metrics should align with business loss. Search scope, split randomness, and candidate quantity all influence the "optimal" alpha. A one-standard-error rule can be used to select a simpler model with performance only slightly below the best.

GroupKFold(n_splits=5) requires at least five distinct training groups and keeps each group out of its corresponding training fold. GridSearchCV refits the chosen pipeline on all training data by default. Fold scores share much of their training data, so a one-standard-error rule is a selection heuristic rather than an independent-sample confidence guarantee.

6. Metrics and Baselines

• MAE: target unit, less amplifies extreme errors;
• RMSE: Squared penalty, higher weight on large errors;
• R²: improvement in squared error relative to the mean baseline;
• MAPE: Unstable near zero and disproportionately sensitive to low values.

At least compare:

• Training set mean/median baseline;
• Simple business rules;
• Current production model;
• New candidate model.

The overall average must also be broken down by group, time period, and target range to prevent a large group from masking significant errors in smaller groups.

from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

prediction = search.predict(X_test)
baseline = DummyRegressor(strategy="median").fit(X_train, y_train).predict(X_test)
print({
    "test_mae": mean_absolute_error(y_test, prediction),
    "test_rmse": root_mean_squared_error(y_test, prediction),
    "baseline_mae": mean_absolute_error(y_test, baseline),
    "selected_alpha": search.best_params_["model__alpha"],
})

The final test records do not take part in selecting alpha or fitting the median baseline. Check that every held-out group is represented in the group report. Standard $R^2$ uses the evaluation-set mean as its scoring reference; compare a fitted training-mean baseline separately.

7. Prediction intervals are not mean intervals

Confidence interval for the mean response $E[Y|X=x]$; a prediction interval for a new observation also includes individual noise and is typically wider.

Heteroskedasticity, incorrect distribution assumptions, and distribution drift can undermine coverage in linear models with constant variance; consider:

• Conditional variance model;
• Quantile regression;
• conformal prediction (provides marginal coverage under exchangeability and similar conditions);
• Check actual coverage by group.

Reporting only point predictions can mislead users into thinking the model accuracy is the same; the interval width itself carries important decision-making information.

A separate split-conformal example shows how calibration works: hold the predictor fixed, collect absolute residuals on independent calibration examples, then take the finite-sample corrected quantile.

from math import ceil
import numpy as np

# Toy absolute errors from a separate calibration set, with the predictor already fixed.
calibration_errors = np.arange(1., 20.)
alpha = 0.10
n_calibration = len(calibration_errors)
rank = ceil((n_calibration + 1) * (1 - alpha))
q = np.inf if rank > n_calibration else np.sort(calibration_errors)[rank - 1]
new_prediction = 30.0
prediction_interval = (new_prediction - q, new_prediction + q)
print(prediction_interval)  # (12.0, 48.0)

Under exchangeability of calibration and future scores, this construction gives marginal coverage, not a 90% guarantee at each feature value or for every subgroup. These 19 toy residuals illustrate the rank calculation only. Do not tune the predictor on calibration labels; repeated entities or distribution shift need methods that account for their structure. See Angelopoulos and Bates.

8. Lasso selection doesn't guarantee stable interpretation

In highly correlated features, Lasso might randomly retain one and drive another to zero; switching folds or making slight data perturbations can change the selection. Zero coefficients don't prove a feature is irrelevant, and non-zero coefficients don't imply causal importance.

Evaluation:

• bootstrap or across-fold selection frequency;
• Coefficient path;
• Related feature groups;
• out-of-sample performance;
• Domain reasonableness.

To validly perform variable selection and inference, specialized methods and pre-specified analysis are required, you can't first use Lasso for screening and then treat ordinary OLS p-values as if no selection had occurred.

9. Model Delivery Record

Objectives and Scope of Use
Training/Validation/Test times and entity boundaries
Feature Definition and Availability Timeline
Preprocessing parameters and category strategies
Candidate models and hyperparameter search
Overall and group-wise errors
Prediction interval coverage
Known failure interval and fallback
Data/code/model versions
Monitoring and retraining trigger conditions

Common Misconceptions

• Lasso sets coefficients to zero indicates feature uselessness: Selection is influenced by scaling, correlation, and lambda.
• Scaling before cross-validation is fine: Leakage of validation fold information.
• Random K-fold fits all data: The splitting is determined by entity, temporal, and site structure.
• A confidence interval is the range for a single prediction: It differs from a prediction interval.

Practice

1. Compare the ridge coefficients and predictions for the same data before and after scaling.
2. Compare the random K-fold and GroupKFold scores for repeated group data.
3. Use time splitting to identify future leakage caused by random splitting.
4. Report MAE, RMSE, baseline errors, and the error intervals for the four groups.

Summary

Regularization limits model flexibility but can't replace independent evaluation. Preprocessing must be included in the pipeline, splits must reflect the true generalization goal, and the test set can be used for evaluation only once. Model outputs must also include intervals, group-wise errors, and failure regions.

The next chapter turns to feature engineering: more features aren't better, but features must be obtainable at prediction time, aligned with entities and time, and parameterized by transformations fitted on the training data.