4.2 Data Splitting, Cross-Validation, and Leakage: The Evaluation Set Must Simulate the Unknown Data the Model Will Encounter

The model performs nearly perfectly on random splits. After inspecting the samples, you find that the same task is split into dozens of lines by the log, with part of it in the training set and part in the test set; the model only recognizes the task ID and fixed patterns. The data was never truly "unknown."

Splitting isn't a fixed 60/20/20 formula. It's a deployment simulation: the size of the isolation unit is determined by what the model will face next, new records, new time periods, new teams, or new locations.

Lesson Objectives

• Assign different responsibilities to training, validation/CV, and test sets;
• Split based on independent, grouped, temporal, and spatial structures;
• Put preprocessing, feature selection, and calibration within their proper boundaries;
• Identify adaptive overfitting caused by repeatedly viewing the test set.

1. Three Types of Data Responsibilities

• training: Fit the model and preprocessing parameters;
• validation/CV: Select features, algorithms, hyperparameters, thresholds, and stopping points;
• test: After the plan is frozen, estimate the generalization performance of the entire selection process.

If the model, features, or paper narrative are changed based on test results, the test has been involved in selection. New holdout sets, nested CV, or honest labeling are needed to avoid overly optimistic performance estimates.

A fixed percentage isn't the goal. For small samples, use cross-validation; for fast-changing data, try multiple rolling windows; and the test set must adequately cover key populations and rare failures.

2. Conditions for Appropriate Random Splitting

Random line splitting assumes that the sample is exchangeable and that the deployment target resembles the current distribution. Classification can be stratified by label to reduce fluctuations in class proportions, but:

• Does not guarantee balance across all important intersectional groups;
• Does not process duplicate entries for the same entity;
• Doesn't simulate future drifting;
• Can't fix selection bias.

A random seed only guarantees that a pseudo-random process can be reproduced in a single run; it does not prove the robustness of the results. Repeat the split multiple times and report the distribution.

3. Group split

For highly correlated rows involving the same user, device, team, patient, or document, the entire group should be kept together in the same fold to meet generalization goals.

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5)
for train_idx, valid_idx in cv.split(X, y, groups=team_id):
    assert set(team_id[train_idx]).isdisjoint(team_id[valid_idx])

If a person belongs to multiple families/sites, group definitions may overlap, requiring connected components or higher-level isolation. Grouping by a single field may still result in leakage.

4. Time Splitting

When predicting the future, training should precede validation. A rolling/expanding window can evaluate multiple periods:

train Jan–Mar → validate Apr
train Jan–Apr → validate May
train Jan–May → validate Jun

You'll also need to set:

• label maturation: training sample labels have been fully observed;
• feature/label gap: prevent leakage from overlapping windows;
• embargo: isolate overlapping event scenarios in finance and related fields during nearby periods;
• late-arriving data: Historical replay uses only the version available at the time.

Even after simply sorting by time and taking the last 20%, future revisions of the same event might still leak into training features.

5. Spatial and Site Generalization

If the model is deployed to unseen fortresses, testing should reserve the entire fortress; if predictions are made only on existing fortresses to forecast the future, then time slices can be applied within each fortress. These are two distinct problems and should be evaluated separately.

Geographic proximity causes spatial correlation. Random points in adjacent grids may be too easily generated; use regional blocks and buffers instead.

6. Preprocessing Must Be Fitted Within the Fold

All steps involved in learning parameters from data are part of the model selection process:

• imputation, scaling, encoding;
• feature selection, PCA;
• target encoding;
• resampling/SMOTE;
• calibration and threshold selection.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000)),
])

Cross-validate pipeline, re-fitting preprocessing for each fold. Performing standardization or filtering on the entire dataset before CV means that the transformation affects the validation fold even if labels aren't directly used; target encoding and supervised feature selection suffer more from leakage.

7. Resampling can only be done on the training fold

For minority class oversampling, undersampling, or synthetic sampling, you should apply it within each training fold. Performing SMOTE before splitting will cause an original sample and its synthetic neighbors to cross folds, leading to significant overoptimism.

The evaluation set should retain the natural distribution at deployment time. If business changes sampling or filtering, additional evaluation using weights or a separate target distribution is needed.

8. Nested cross-validation

When the same CV fold is used to select hyperparameters and report the best score, optimism in selection occurs. Nested CV:

• inner loop selects pipeline and hyperparameters;
• The outer loop evaluates the entire selection process.

More expensive, but important when dealing with small datasets, many candidates, and the need for reliable comparisons. The final model can be retrained on all development data using a selected process, but the independent test is still only run once.

9. Data deduplication cannot cross the test boundary to peek inside

First, perform near-duplicate detection on the entire dataset, which might use test content to modify training. The correct workflow depends on the goal:

• Remove true duplicates at the global source layer, before splitting; rules and reasons must be fixed.
• Semantically similar or repetitive images should be grouped first, then sliced by group;
• Based on test performance, identify and remove "troublesome samples" that are found to be contaminated.

Version, hash, source ID, and deduplication cluster should be retained for auditing.

10. Test Set Operating Discipline

Visitors and limited use
Evaluating scripts and primary metrics are pre-frozen
Return only essential aggregations to avoid repetitive debugging per sample
Log every visit, model, data, and rationale.
Changes made after testing must reestablish the evaluation boundaries
Resample when testing distribution and deployment changes

A test set isn't a permanent benchmark. As teams repeatedly publish benchmark results, organizational knowledge can overfit to it.

Common Misconceptions

• Splitting it into three parts eliminated the leak: Dividing the unit and all fit steps is equally important.
• Stratify addresses distribution issues: It only constrains the proportion of specified labels. -Fixed random seed is reliable: It might only reproduce a single accidental split.
• Test only looks, doesn't train, so you can look at it more: Human choices can also overfit.

Practice

1. Compare row-by-row random and GroupKFold scores for repetitive task logs.
2. Design a time split and gap for a 30-day label window.
3. Demonstrate the difference between full-data feature selection and fold-level feature selection.
4. Set up access logs for the test set and establish rules for "when a new test is needed."

Summary

Credibility assessment comes from isolating the real unknown, not from the filename test.csv. Time, entities, space, and all data learning steps must align with the deployment target.

In the next lesson, we'll diagnose underfitting, overfitting, and distribution shifts using learning curves and error decomposition, and then select appropriate corrective actions.