6.3 Regularization, Cross-Validation, and Prediction Intervals: Model Selection Can't Look at the Answers

The candidate models in the prophecy room are getting better on the training set, but quickly lose accuracy when evaluated on new data, the model selection process might have peeked at the answers.

After adding more features, training error keeps decreasing while test error rises. Regularization can limit model flexibility, but the strength of regularization, scaling, and feature processing must be learned solely within the training fold; otherwise, cross-validation becomes merely formal isolation.

This lesson's objectives

• Distinguish the objectives of ridge, lasso, and elastic net
• Use a pipeline to include preprocessing within cross-validation;
• Split based on independent, grouped, or time-based structures;
• Report baseline, prediction error, and prediction interval.

1. Ridge and Lasso Penalty Differences

Ridge:

$$ \min_\beta\sum_i(y_i-x_i^T\beta)^2+\lambda\sum_j\beta_j^2. $$

Lasso:

$$ \min_\beta\sum_i(y_i-x_i^T\beta)^2+\lambda\sum_j|\beta_j|. $$

• Ridge continuously shrinks feature coefficients, typically not reducing them to exactly zero;
• Lasso can produce sparse solutions, but feature selection among correlated features may be unstable;
• Elastic Net combines L1 and L2 regularization, often providing more stability when features are correlated.

Intercepts are typically not penalized; specific library behavior should be verified. Penalties alter the estimation target; regularization coefficients cannot be directly interpreted using ordinary OLS p-values.

2. Scaling Determines Punishment Fairness

A feature scale measured in bytes versus one measured in proportions can differ greatly. The same regularization coefficient represents different functional changes on the original scale, so it's typical to standardize numerical features before regularization.

Standardized parameters can only be fitted within each training fold. scikit-learn pipeline example:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["team_size", "resources_used"]
categorical = ["mission_type"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

pipeline = Pipeline([
    ("preprocess", preprocess),
    ("model", Ridge()),
])

Standardizing the entire dataset before cross-validation will affect the mean, variance, and class distribution in the validation folds.

3. Three Layers of Data Responsibility

• training: fitting parameters;
• validation/CV: Select features, models, and hyperparameters;
• test: Estimate generalization performance only after the final solution is fixed.

Keep checking test results to tweak the model, turning test into a validation set. Either preserve a new independent dataset or honestly report that evaluations are overly optimistic.

Small datasets can use nested cross-validation: the inner loop selects hyperparameters, the outer loop estimates the performance of the selection process. It's more costly, but avoids using the same cross-validation score for both model selection and unbiased final evaluation.

4. Split units must match the application

Independent Samples

Random K-fold is available, but maintains the target and important group distributions.

Repeated Entities

The same user or team shouldn't appear in both training and validation, otherwise the model might memorize entities. Use a group split.

Time Prediction

Training must come before validation; random shuffling would use future distributions to predict the past. Use rolling or expanding windows, and introduce a necessary gap between the feature and label windows.

Spatial or site generalization

If the goal is to deploy to a new fortress, leave room for the fortress instead of randomly splitting lines within each fortress.

5. Hyperparameter Selection Includes Uncertainty

from sklearn.model_selection import GridSearchCV, GroupKFold

search = GridSearchCV(
    pipeline,
    param_grid={"model__alpha": [0.01, 0.1, 1.0, 10.0, 100.0]},
    scoring="neg_mean_absolute_error",
    cv=GroupKFold(n_splits=5),
)
search.fit(X_train, y_train, groups=team_ids_train)

Scoring metrics should align with business loss. Search scope, split randomness, and candidate quantity all influence the "optimal" alpha. A one-standard-error rule can be used to select a simpler model with performance only slightly below the best.

6. Metrics and Baselines

• MAE: target unit, less amplifies extreme errors;
• RMSE: Squared penalty, higher weight on large errors;
• R²: improvement in squared error relative to the mean baseline;
• MAPE: Unstable near zero and disproportionately sensitive to low values.

At least compare:

• Training set mean/median baseline;
• Simple business rules;
• Current production model;
• New candidate model.

The overall average must also be broken down by group, time period, and target range to prevent a large group from masking significant errors in smaller groups.

7. Prediction intervals are not mean intervals

Confidence interval for the mean response $E[Y|X=x]$; a prediction interval for a new observation also includes individual noise and is typically wider.

Heteroskedasticity, incorrect distribution assumptions, and distribution drift can undermine coverage in linear models with constant variance; consider:

• Conditional variance model;
• Quantile regression;
• conformal prediction (provides marginal coverage under exchangeability and similar conditions);
• Check actual coverage by group.

Reporting only point predictions can mislead users into thinking the model accuracy is the same; the interval width itself carries important decision-making information.

8. Lasso selection doesn't guarantee stable interpretation

In highly correlated features, Lasso might randomly retain one and drive another to zero; switching folds or making slight data perturbations can change the selection. Zero coefficients don't prove a feature is irrelevant, and non-zero coefficients don't imply causal importance.

Evaluation:

• bootstrap/interstitial selection frequency;
• Coefficient path;
• Related feature groups;
• out-of-sample performance;
• Domain reasonableness.

To validly perform variable selection and inference, specialized methods and pre-specified analysis are required, you can't first use Lasso for screening and then treat ordinary OLS p-values as if no selection had occurred.

9. Model Delivery Record

Objectives and Scope of Use
Training/Validation/Test times and entity boundaries
Feature Definition and Availability Timeline
Preprocessing parameters and category strategies
Candidate models and hyperparameter search
Overall and group-wise errors
Prediction interval coverage
Known failure interval and fallback
Data/code/model versions
Monitoring and retraining trigger conditions

Common Misconceptions

• Lasso sets coefficients to zero indicates feature uselessness: Selection is influenced by scaling, correlation, and lambda.
• Scaling before cross-validation is fine: Leakage of validation fold information.
• Random K-fold fits all data: The splitting is determined by entity, temporal, and site structure.
• A confidence interval is the range for a single prediction: It differs from a prediction interval.

Practice

1. Compare the ridge coefficients and predictions for the same data before and after scaling.
2. Compare the random K-fold and GroupKFold scores for repeated group data.
3. Use time splitting to identify future leakage caused by random splitting.
4. Report MAE, RMSE, baseline errors, and the error intervals for the four groups.

Summary

Regularization limits model flexibility but can't replace independent evaluation. Preprocessing must be included in the pipeline, splits must reflect the true generalization goal, and the test set can be used for evaluation only once. Model outputs must also include intervals, group-wise errors, and failure regions.

The next chapter dives into feature engineering: more features aren't better, but features must be obtainable at prediction time, aligned with entities and time, and parameterized by transformations fitted on the training data.