8.2 Splitting, Cross-Validation, and Uncertainty: Validation Structure Must Mirror Deployment Structure

The Model Workshop splits the logs from a single device into five folds, achieving nearly perfect results. But when the intelligence officer tests the model on a new device, performance plummets. The issue isn’t insufficient fold count; it’s that each training fold has already seen the “fingerprint” of every validation device.

Cross-validation only makes sense when the splitting unit aligns with the deployment boundary. Random K-Fold, stratified K-Fold, GroupKFold, and time-based replay estimate different generalization scenarios.

Learning Objectives

• Choose among random, group, or temporal split strategies for deployment setup;
• Enclose all learned preprocessing steps within each fold;
• Distinguish between model selection, performance estimation, and final testing;
• Accurately interpret CV mean, fold-to-fold standard deviation, and confidence intervals;
• Use nested cross-validation and paired comparisons to control for selection bias.

1. Splitting Is About Defining "New Data"

Common deployment challenges:

Deployment Issue	What Should Be Isolated During Validation
Independent new samples from the same distribution	Random rows (iid assumption approximately holds)
New users, new devices, new hospitals	Entire group
Future behavior of existing users	Time; historical users may still appear
Future behavior of new users	Both group and time must be isolated
New regions or new sites	Location or domain

Start by identifying the deployment problem, then choose the appropriate splitter. Never reverse this order simply because cv=5 is convenient, this does not mean the model can generalize to new devices.

2. Train, Validation, Test Have Distinct Responsibilities

• train: Fits model parameters and performs in-fold preprocessing;
• validation / inner CV: Selects features, models, hyperparameters, calibrates and sets thresholds;
• test / outer fold: Evaluates the entire selection pipeline once it's frozen.

As soon as you review test results and make any changes to the selection, the test set has become part of development. The value of a single test comes not from its filename, but from the process constraints it enforces.

For continuously iterating products, maintain a rolling backtest and conduct final shadow/champion–challenger evaluations over time. Avoid long-term, repeated "preservation" of a test result that has already been widely known and remembered across the team.

3. Stratification is an engineering measure, not a restoration of iid

StratifiedKFold Striving to maintain balanced class proportions across folds helps prevent rare classes from vanishing entirely in a single fold and reduces noise in certain evaluation metrics. However, it does not address:

• User-level data leakage across folds;
• Temporal leakage;
• Spatial or organizational correlation;
• Sample duplication;
• Overlapping label time windows.

When grouping is available and a roughly balanced class distribution is desired, consider StratifiedGroupKFold. However, group non-overlap should take precedence over perfect class balance; the number of groups, their sizes, and class distributions may make ideal stratification practically unattainable.

4. The statistical unit for Group Split is Group

Related observations (such as multiple visits by the same patient, multiple log entries from the same device, or multiple segments in the same document) typically belong to the same group. If the goal is to generalize a new group, all rows belonging to that group must be assigned to the same fold.

Evaluations and interval estimates should respect group structure: one thousand log entries from ten devices do not equate to one thousand independent samples. It is appropriate to report all of the following:

• micro/row-weighted metrics;
• macro/group-equal metrics;
• group-level distributions and worst-case or low-percentile performance;
• leave-one-site/group-out sensitivity.

The choice of weighting depends on the overall deployment strategy: will future traffic occur row-by-row, or will each site be treated equally?

5. Time Splitting Must Address Label Maturation and Gaps

A time model cannot train on future data to predict the past. You must also consider the feature window and label horizon:

feature window ---- cutoff ---- label horizon
train labels mature | gap | validation features/labels

If you're predicting a fault 30 days into the future, the labels of the final training samples must mature (i.e., become available) after 30 days. An embargo or gap period may be necessary between training and validation to prevent overlapping windows or label leakage from future data.

TimeSeriesSplit provides expanding-window and gap mechanisms, but by default assumes that data rows are time-sorted and evenly spaced, which affects the comparability of metrics across folds. In real-world systems, additional constraints such as rolling windows, fixed test durations, seasonal coverage, and group-time dual constraints may also be required.

6. All Learned Transforms Fit Within Fold

Leakage is not limited to target encoding. The following steps (any that estimate parameters from data) belong to the training process:

• Missing value imputation, scaling, winsorization;
• Vocabulary construction and one-hot category sets;
• PCA, feature selection;
• Target and frequency encoding;
• Resampling or SMOTE;
• Calibration and threshold tuning;
• Anomaly or cluster representation.

Use Pipeline/ColumnTransformer to include these within cross-validation. Even if a transform does not inspect y, fitting on the full dataset will cause the validation distribution to influence the learned representation.

7. A Group-Aware Multi-Metric Evaluation Skeleton

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedGroupKFold, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

candidate = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=2_000)),
])

cv = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

result = cross_validate(
    candidate,
    X,
    y,
    groups=device_id,
    cv=cv,
    scoring={
        "roc_auc": "roc_auc",
        "average_precision": "average_precision",
        "neg_log_loss": "neg_log_loss",
    },
    return_train_score=True,
    return_estimator=True,
    n_jobs=-1,
)

Whether sklearn's metadata routing is enabled currently affects the argument passing of groups. Projects should lock down versions and align with the official API. More importantly, each fold's group intersection must be verified to be empty before relying solely on the splitter's name.

8. The Inter-Section Standard Deviation Is Not a Confidence Interval

K CV scores do not represent K independent and identically distributed experiments: the training sets are highly overlapping, and the validation sets are drawn from the same finite dataset. mean ± std describes the dispersion among these splits, but it cannot be directly interpreted as a 95% confidence interval for the true model performance.

Sources of uncertainty include at least:

• A finite test sample size;
• Data splitting strategies;
• Model initialization and stochastic training;
• Selection of hyperparameters during search;
• Label noise and latency;
• Future distribution drift.

Reporting a mean with three decimal places does not eliminate these sources of uncertainty.

9. Bootstrap Must Use Correct Units

Independent bootstrap sampling of test units can estimate metric sampling uncertainty in a frozen model. When a user has multiple rows, users should be clustered during bootstrap rather than sampling rows independently; for time series data, block bootstrap is appropriate, though the block length itself is a modeling choice.

Metrics for rare events may result in bootstrap samples with no positive examples or highly skewed distributions. It is essential to report the number of effective replicates, the interval method used, and the sampling unit, and to evaluate the suitability of methods such as percentiles or BCa intervals.

Bootstrap intervals only capture finite-sample uncertainty under a specified sampling mechanism and do not account for future drift, hidden leakage, or selection bias introduced by continual model iteration.

10. Compare Models Using Paired Evidence

When comparing two models, the difference in their errors on the same test sample is meaningful. Compute the difference:

$$ \Delta = M_A - M_B, $$

and resample the paired predictions within the same row or group. This approach is typically more powerful than comparing two independent intervals. For classification accuracy, consider McNemar's test; for AUC, use specialized methods; for general metrics, paired bootstrap or permutation tests work well, though each must satisfy the appropriate independence assumption.

Statistical significance does not imply business importance. Predefine a minimum meaningful difference, a non-inferiority boundary, or a cost threshold. Additionally, evaluate runtime, calibration, and impact on user groups.

11. Nested CV Evaluates the "Selection Process"

A plain GridSearchCV performs cross-validation within the development set and then evaluates on a separate, untouched test set, this is a holdout test design, not nested CV.

Nested CV consists of two layers:

1. The outer split reserves data for the current evaluation round;
2. The inner search runs only within the outer-training set;
3. The selected workflow predicts the outer-test set;
4. All outer-test predictions are aggregated.

It estimates how well the selection process would perform if re-tuned on similar training data. It's especially valuable when data is scarce and hyperparameter tuning is extensive, though computationally expensive. Additionally, outer folds are not fully independent experiments.

12. Learning Curve Responding to Data or Model Limitations

Repeat a full fit across multiple training scales and compare train/validation loss using a fixed validation protocol:

• If both are poor and close: the model may lack sufficient representation capacity, or the target signal may be noisy;
• If training loss is low but validation loss is high: this suggests high variance, data leakage, or distribution shifts;
• If validation loss improves steadily as data grows: additional data from the same distribution may still be valuable;
• If the curve plateaus: the marginal benefit of collecting more similar data may be small.

Each scale must preserve the group/time structure, and the transform must be reapplied before re-fitting. It is not valid to draw smooth lines from a single random subsample and overinterpret the results.

Common Misconceptions

• K=5/10 is naturally robust: The value of K is determined by independent units, class counts, cost, and estimation goals, not by inherent reliability.
• StratifiedKFold fixes everything with class imbalance: It only controls the class distribution within each fold, not across folds.
• CV standard deviation is equivalent to error bars: It reflects fold-to-fold variation, not a confidence interval and cannot be directly interpreted as such.
• Unsupervised preprocessing can be fitted on the full dataset: Once included in the pipeline, it introduces data leakage.
• GridSearchCV comes with nested cross-validation: Only when an explicit outer loop is defined is it truly nested.

Exercise

1. Draw splitting boundaries for "new devices" and "future failures of existing devices."
2. Verify the intersection and temporal ordering of each train/validation group.
3. Compare estimation differences between full-data scaling and scaling within each fold.
4. Perform row bootstrap and group bootstrap on the same test predictions.
5. Implement outer group cross-validation with inner search, and compare the results to non-nested approaches.

Summary

Validation methods are not just template-based splits, but simulators of deployment conditions. The split is determined jointly by independent units, time, tag windows, and preprocessing; CV mean and standard deviation offer only limited evidence. More rigorous approaches like nested cross-validation, paired comparisons, and properly-specified bootstrapping can address deeper questions, but they cannot replace ongoing monitoring for future drift.

The next lesson integrates this validation structure into model selection: the search space, budget, early stopping, calibration, and thresholds must all be determined within the inner loop.