8.3 Hyperparameter Search and Final Evaluation: The Larger the Search Budget, the Greater the Risk of Overfitting the Validation Process

Hyperparameter search is a model selection process. The more candidates we evaluate and the more frequent the feedback, the more the validation set begins to resemble training data. Therefore, the final test or outer cross-validation must assess the entire selection process, rather than the performance of the winning model on its training results alone.

Learning Objectives

• Distinguish between model parameters, hyperparameters, and decision parameters;
• Design search spaces with scale, conditions, and budget constraints;
• Correctly apply grid search, random search, and adaptive search strategies;
• Integrate pipeline components, early stopping, calibration, and thresholding within the validation boundary;
• Freeze the final selection and evaluate only using test data.

1. What Counts as Choosing from Data

• model parameters: coefficients learned from training loss, tree splitting rules, neural network weights;
• hyperparameters: regularization strength, tree depth, learning rate, representation selection;
• decision parameters: classification thresholds, top-$K$ values, rejection regions;
• procedure choices: feature windows, missing data handling strategies, evaluation metrics, ensemble rules.

The last three categories (once tuned based on validation performance) fall under the scope of model selection. Manually "adjusting by experience twice" is statistically indistinguishable from automated RandomizedSearchCV, as both consume validation data.

2. Define Search Objectives and Budget

Before starting the search, fix the following:

• Primary selection metric or utility;
• Secondary guardrails (latency, calibration, fairness, memory);
• Splitter and random seed strategy;
• Maximum number of candidates, total fit evaluations, and time limit;
• Handling of failures or NaN values;
• Minimum meaningful improvement;
• When testing is unblocked.

Otherwise, the team will change metrics after seeing results, expand search spaces, remove failing folds, and ultimately end up with a best score that cannot be explained.

3. Search Space Should Respect Parameter Scales

Regularization strength and learning rate often span several orders of magnitude, making log-uniform distributions appropriate. Depth and neighbor count are discrete integers, while some parameters are only valid under specific model or solver configurations.

An unreasonable grid:

C = [1, 2, 3, 4, 5]

If the plausible range could extend from ${10}^{-4}$ to ${10}^2$, a linear grid with only five points explores very little of the scale. Start by using a broad logarithmic space to identify the relevant region, then refine within that region under a fixed budget. Never expand the search range after observing test performance.

Conditional spaces must match the selected solver. For example, an elastic-net mixing ratio requires a solver that supports that mixture; scikit-learn 1.9 deprecates the old penalty parameter in favor of l1_ratio. Different boosters and grow policies also admit different parameters. Pin the API before enumerating combinations.

4. Grid and Random Search

Grid search enumerates the Cartesian product, making it suitable for search spaces with few candidates where each value has a clear and interpretable meaning. As the number of dimensions increases, the total number of combinations grows multiplicatively.

Random search samples from a defined distribution under a fixed n_iter budget:

• Separates search budget from dimensionality;
• Enables exploration of a broader range of values for continuous parameters;
• Results depend on the parameter distribution and random seed;
• Candidates can be added incrementally within the budget, but the addition rule must be predefined in advance.

Random search is not equivalent to blind exploration. The upper and lower bounds of the distribution still serve as strong priors. It is essential to record sampled values and failure cases.

5. Group-aware Random Search within a Pipeline

This Python 3 fragment was checked with SciPy 1.18.1 and scikit-learn 1.9.0. Supply numeric development/test matrices, binary 0/1 development labels, and aligned group IDs. At least five groups and both classes in every train/validation fold are required.

import numpy as np
from scipy.stats import loguniform
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV, StratifiedGroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=3_000)),
])

inner_cv = StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

y_development = np.asarray(y_development)
development_group = np.asarray(development_group)
splits = list(inner_cv.split(X_development, y_development, groups=development_group))
for train_idx, valid_idx in splits:
    assert not np.intersect1d(development_group[train_idx], development_group[valid_idx]).size
    assert len(np.unique(y_development[train_idx])) == len(np.unique(y_development[valid_idx])) == 2

search = RandomizedSearchCV(
    pipeline,
    param_distributions={
        "model__C": loguniform(1e-4, 1e2),
        "model__class_weight": [None, "balanced"],
    },
    n_iter=40,
    scoring={
        "neg_log_loss": "neg_log_loss",
        "average_precision": "average_precision",
    },
    refit="neg_log_loss",
    cv=splits,
    random_state=42,
    n_jobs=1,
    error_score="raise",
    return_train_score=True,
)

search.fit(X_development, y_development)

test_probability = search.predict_proba(X_test)[:, 1]

best_score_ is the negative loss with respect to neg_log_loss, where "larger is better", do not forget the negative sign. In multi-metric search, only the metric specified by refit is used to select the final estimator; other metrics serve as guardrails or diagnostic tools and are not automatically optimized for multi-objective performance.

The splitter receives groups explicitly and passes fixed index pairs to the search. Forty candidates across five folds require 200 fits, plus one refit on all development data. No budget is reserved here for further calibration or threshold fitting; add those costs if they are part of the selected procedure.

6. The Best Candidate May Be a Noise Winner

Check the following in cv_results_:

• The fold-wise mean and standard deviation;
• The train–validation performance gap;
• Fit and score times;
• Candidates that failed or produced NaN values;
• Whether parameter boundaries consistently dominate;
• Whether the differences between top candidates are smaller than sampling noise;
• Whether rankings flip across different time periods or groups.

When multiple candidates perform similarly, use a pre-defined tolerance or a one-standard-error heuristic (not a formal confidence guarantee from correlated CV folds) to select the simpler, faster, and more stable model. A difference in only the fourth decimal place does not necessarily justify increased latency.

7. Multi-Objective Selection Requires Clear Priorities

Real-world systems often demand:

Among candidate models where validation log loss is no worse than the baseline by more than 1%, select the one with the lowest p99 latency and where group recall meets the required threshold.

This is not simply averaging multiple metrics. Approaches such as constrained selection, Pareto frontier analysis, or refit=callable can be used, but the selection rules must be frozen before testing. The final outcome may be jointly determined by model architecture, thresholds, and capacity, so it's best to directly evaluate the final decision policy.

8. The Cost of Adaptive/Bayesian Search

Bayesian optimization determines the next set of parameters by building a surrogate model and an acquisition function based on historical trials; successive halving performs an initial low-cost screening, then allocates more resources to a small number of candidates.

While both approaches can reduce computational cost, they do not eliminate the risk of validation bias:

• Early rankings based on limited resources may not accurately predict final training performance;
• Learning curves can cross, leading to premature elimination of candidates that later perform well;
• Adaptive trials use validation data more intensively and closely, increasing variance;
• Parallel trials, failure recovery, and randomness affect reproducibility;
• sklearn's halving search remains an experimental API in the current version.

Record for each trial: the code, data, and version, resource allocation, random seed, metric value, and reason for termination.

9. Early Stopping Requires a Third Layer of Training Boundary

When evaluating boosting candidates within an inner fold:

• Outer validation must remain untouched. If inner validation selects the stopping round, its score is selection evidence rather than an independent estimate of that candidate;
• A subset can be split from inner-train for fitting and early stopping, or a custom workflow that properly cross-fits across rounds must be used;
• How the selected number of rounds is applied to refit the full training set must be predefined;
• It is absolutely forbidden to use the final test set to control stopping.

The general SearchCV does not automatically pass each fold’s validation data to a third-party library’s eval_set. If the code feeds the same global X_valid into every fit, cross-fold leakage may occur. Custom estimators or callbacks, or a staged approach, are required, along with tests that verify index integrity.

10. Calibration and Threshold Also Need Nesting

A complete binary classification system might proceed in sequence through:

1. Representation and model hyperparameters;
2. Calibration mapping;
3. Decision threshold.

If all three stages are optimized repeatedly on the same validation set, the results tend to be overly optimistic. A viable design alternative includes:

• Separate data segments for training/parameter tuning, calibration, and threshold setting;
• Cross-validation to generate out-of-fold scores, then fit calibration and threshold models on those scores;
• Embedding the full meta-estimator within an outer cross-validation loop;
• Final independent test evaluation of a frozen pipeline.

Optimizing the decision threshold does not alter the ROC or PR curve rankings of the original scores, but it does affect hard predictions and overall utility. Even if calibration is a strictly increasing mapping, it does not change the ranking order. In cases of limited data or non-strict transformations, empirical validation remains essential.

11. Engineering Pitfalls of Parallel Search

Combining n_jobs=-1 with internal model multithreading can oversubscribe CPU resources. Parallel search candidates may also trigger data duplication, leading to memory peaks and out-of-memory (OOM) errors.

Mitigation strategies:

• Limit parallelism to the search layer or estimator layer only;
• Set pre_dispatch or thread environment variables;
• Monitor peak RSS, fit time, and energy consumption;
• Assign timeouts and failure classifications to each trial;
• Do not silently discard OOM candidates as ordinary low-scoring entries.

Deployability must be enforced as a selection guardrail, not checked only after a run has succeeded.

12. Final Testing and Retraining Protocol

A clear holdout workflow:

1. Lock the test snapshot and evaluation script;
2. Complete all searches, feature engineering, calibration, threshold tuning, and code selection on development data;
3. Freeze the pipeline, dependencies, seed policy, and primary metric;
4. Unlock the test set and generate a single prediction batch;
5. Report the point estimate, uncertainty in appropriate units, sliced performance, and failure cases;
6. Make a release decision based on predefined release gates.

If the test fails and the model is revised, development may continue, but the original test must be acknowledged as evidence of prior performance. To restore confidence in the final evaluation, a new future window, external data, or a rigorously documented sequential testing protocol is required.

Whether to retrain on train+validation after a final decision depends on how calibration and thresholds align, and whether performance estimation remains necessary after the test. It is not permissible to include the test set in training and then use the original test score to represent the new model’s performance.

Common Misconceptions

• Manually tuning parameters doesn't count as search: Any adjustment based on validation feedback consumes the available search budget and undermines the exploration process.
• GridSearchCV's best score is a generalization-unbiased estimate: It represents the highest cross-validation score among the evaluated configurations, but is often overly optimistic in practice.
• Random search is inherently more scientific: While it's more efficient in exploration, the quality of results still depends on the parameter distribution, budget allocation, and validation structure.
• Early stopping automatically prevents overfitting: It functions as a validation-driven selection mechanism, inherently limiting training on noisy or overfitted data.
• SearchCV automatically handles third-party eval_set: In reality, integration with external components typically requires explicit design and thorough testing.

Exercise

1. Change the linear grid to a log-uniform search and plot the actual sampled values.
2. From cv_results_, identify candidate models with similar scores but differing latency or complexity.
3. Implement outer GroupKFold + inner RandomizedSearchCV, and compare the best inner score with the outer score.
4. Plot the data boundaries during fitting, early stopping, inner validation, outer validation, and testing in boosting.
5. Write a frozen protocol that includes calibration, thresholding, and final test evaluation.

Summary

Hyperparameter search expands the candidate space and increases sensitivity to validation results. A reliable workflow must predefine the target, search space, budget, and data split, ensuring all learned choices are confined within an inner loop. The full selection process should be evaluated independently using a separate test set or outer cross-validation. A model that ranks first in validation is merely a candidate, only the frozen decision system constitutes the final deliverable.

The next chapter moves into neural networks. Training scale and non-convex optimization will grow in complexity, but the evaluation principles remain unchanged: data boundaries, objectives, early stopping, and the final test must remain strictly isolated.