4.2 Data Splitting, Cross-Validation, and Leakage: The Evaluation Set Must Simulate the Unknown Data the Model Will Encounter

If one task produces dozens of log rows, a random split can place parts of it in both training and test. Recognizing task IDs and fixed patterns may earn a high score without answering how the model handles a new task. Moving data to another folder does not make it a stranger.

Splitting isn't a fixed 60/20/20 formula. It's a deployment simulation: the size of the isolation unit is determined by what the model will face next, new records, new time periods, new teams, or new locations.

Lesson Objectives

• Assign different responsibilities to training, validation/CV, and test sets;
• Split based on independent, grouped, temporal, and spatial structures;
• Put preprocessing, feature selection, and calibration within their proper boundaries;
• Identify adaptive overfitting caused by repeatedly viewing the test set.

1. Three Types of Data Responsibilities

• training: Fit the model and preprocessing parameters;
• validation/CV: Select features, algorithms, hyperparameters, thresholds, and stopping points;
• test: After the plan is frozen, estimate the generalization performance of the entire selection process.

Changing the model, features, or paper narrative after inspecting test results makes those results part of selection. Use new independent evaluation data or label the results exploratory. Retrofitting nested CV onto already inspected data does not erase human adaptation; design nested CV before the selection process begins.

A fixed percentage isn't the goal. For small samples, use cross-validation; for fast-changing data, try multiple rolling windows; and the test set must adequately cover key populations and rare failures.

2. Conditions for Appropriate Random Splitting

Random row splitting assumes that the sample is exchangeable and that the deployment target resembles the current distribution. Classification can be stratified by label to reduce fluctuations in class proportions, but:

• Does not guarantee balance across all important intersectional groups;
• Does not process duplicate entries for the same entity;
• Doesn't simulate future drifting;
• Can't fix selection bias.

A random seed only guarantees that a pseudo-random process can be reproduced in a single run; it does not prove the robustness of the results. Repeat the split multiple times and report the distribution.

3. Group split

For highly correlated rows involving the same user, device, team, patient, or document, the entire group should be kept together in the same fold to meet generalization goals.

This Python 3 + scikit-learn fragment (partial) expects NumPy arrays X, y, and team_id with matching sample counts and at least five distinct teams. It checks group isolation without training a model.

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5)
for train_idx, valid_idx in cv.split(X, y, groups=team_id):
    assert set(team_id[train_idx]).isdisjoint(team_id[valid_idx])

If a person belongs to multiple families/sites, group definitions may overlap, requiring connected components or higher-level isolation. Grouping by a single field may still result in leakage.

4. Time Splitting

When predicting the future, training should precede validation. A rolling/expanding window can evaluate multiple periods:

train Jan–Mar → validate Apr
train Jan–Apr → validate May
train Jan–May → validate Jun

You'll also need to set:

• label maturation: training sample labels have been fully observed;
• feature/label gap: prevent leakage from overlapping windows;
• embargo: isolate overlapping event scenarios in finance and related fields during nearby periods;
• late-arriving data: Historical replay uses only the version available at the time.

Suppose a model predicts on April 1 and labels require a 30-day observation window. A March 31 task has no complete label yet and cannot be included in training with that label. A further two-day reporting delay moves the maturity cutoff earlier. Holding out the last 20% by task time does not automatically enforce availability times.

5. Spatial and Site Generalization

Geographic proximity can create spatial correlation. Randomly holding out points near training locations may make evaluation too easy; regional blocks and buffers can better represent transfer to new areas.

6. Preprocessing Must Be Fitted Within the Fold

All steps involved in learning parameters from data are part of the model selection process:

• imputation, scaling, encoding;
• feature selection, PCA;
• target encoding;
• resampling/SMOTE;
• calibration and threshold selection.

This partial fragment constructs a preprocessing and model Pipeline. Pass the whole object to cross-validation so each fold learns its own imputation and scaling parameters.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000)),
])

Cross-validating pipeline refits preprocessing in every fold. Standardizing or selecting features on the full table before CV lets validation data influence the learned transformation, even without using its labels. Target encoding and supervised selection can leak more directly. See scikit-learn's leakage examples.

7. Resampling can only be done on the training fold

For minority class oversampling, undersampling, or synthetic sampling, you should apply it within each training fold. Performing SMOTE before splitting will cause an original sample and its synthetic neighbors to cross folds, leading to significant overoptimism.

The evaluation set should retain the natural distribution at deployment time. If business changes sampling or filtering, additional evaluation using weights or a separate target distribution is needed.

8. Nested cross-validation

When the same CV fold is used to select hyperparameters and report the best score, optimism in selection occurs. Nested CV:

• inner loop selects pipeline and hyperparameters;
• The outer loop evaluates the entire selection process.

More expensive, but important when dealing with small datasets, many candidates, and the need for reliable comparisons. The final model can be retrained on all development data using a selected process, but the independent test is still only run once.

9. Data deduplication cannot cross the test boundary to peek inside

Detecting near-duplicates across all data can use test content to change training. The appropriate workflow depends on the intended evaluation:

• Remove true duplicates at the global source layer, before splitting; rules and reasons must be fixed.
• Semantically similar or repetitive images should be grouped first, then sliced by group;
• Removing troublesome samples because of their test performance contaminates evaluation.

Version, hash, source ID, and deduplication cluster should be retained for auditing.

10. Test Set Operating Discipline

Restrict access and permitted uses
Freeze evaluation scripts and primary metrics before testing
Return only essential aggregations to avoid repetitive debugging per sample
Log every visit, model, data, and rationale.
Changes made after testing must reestablish the evaluation boundaries
Resample when testing distribution and deployment changes

A test set isn't a permanent benchmark. As teams repeatedly publish benchmark results, organizational knowledge can overfit to it.

Common Misconceptions

• Splitting it into three parts eliminated the leak: Dividing the unit and all fit steps is equally important.
• Stratify addresses distribution issues: It only constrains the proportion of specified labels.
• A fixed random seed makes the result reliable: It might only reproduce a single accidental split.
• Test only looks, doesn't train, so you can look at it more: Human choices can also overfit.

Practice

1. Compare row-by-row random and GroupKFold scores for repetitive task logs.
2. Design a time split and gap for a 30-day label window.
3. Demonstrate the difference between full-data feature selection and fold-level feature selection.
4. Set up access logs for the test set and establish rules for "when a new test is needed."

Summary

Credibility assessment comes from isolating the real unknown, not from the filename test.csv. Time, entities, space, and all data learning steps must align with the deployment target.

In the next lesson, we'll diagnose underfitting, overfitting, and distribution shifts using learning curves and error decomposition, and then select appropriate corrective actions.