7.3 Feature Selection and Production Consistency: From Candidate Columns to Monitorable Interfaces

The same feature yields different values online and offline, and the Model Workshop rejected this batch of data that hasn't established a computational contract.

The model passed validation in offline replay, but after deployment, it was found that the "last seven days supply count" on production underreported a batch of late-arrival records, and unknown equipment types were silently encoded as zeros. The issue wasn't with the algorithm, it was that training and serving weren't following the same feature contract.

This lesson covers two "tightening" issues: how to select features without peeking at the validation set, and how to ensure that selected features are reproducible, monitorable, and rollable back in production environments.

Lesson Objectives

• Distinguish filter-based, wrapper-based, and embedded feature selection;
• Put feature selection inside cross-validation;
• Check for stability and genuine incremental value;
• Design offline/online consistency testing and monitoring.

1. Choosing a target isn't about having the fewest items on the list

Feature selection may serve different objectives:

• Reduce the costs associated with data collection, computation, and online latency;
• Reduce high-dimensional noise and overfitting;
• Improve interpretability or comply with regulatory constraints;
• Eliminate inputs that are unavailable, unstable, or redundant during prediction.

First, establish the constraints, then compare the approaches. There is no universal rule that sample size must be ten times the number of features: identifiability depends on the model, regularization, sparsity, noise, structural correlation, and validation design.

2. Three Types of Selection Methods

Filtered

Screened by missing rate, near-zero variance, correlation, or univariate statistics; computationally efficient but ignores feature interactions. Any score that uses labels must be computed within the training fold.

Parcel-style

Recursive elimination, sequential selection, and other iterative training methods are used to validate model performance and select subsets. These approaches can account for model interactions but are computationally expensive, and prone to overfitting in the validation process when the number of candidates is large.

Embedded

Lasso, tree splitting, and some sparse models perform feature selection during fitting. Results are model-dependent: Lasso can be unstable among correlated features, and tree importance does not equate to causal effect.

Domain review is the fourth gate: features that are legal but costly, hard to interpret, or possibly encode sensitive attributes cannot simply enter production based on a score.

3. You Must Choose Inside the Cross-Validation

First, identify the "20 most relevant columns to the label" from the full dataset, then perform cross-validation, where the validation labels are involved in the column selection. The correct approach is to place the selector inside a pipeline:

from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

candidate = Pipeline([
    ("preprocess", preprocess),
    ("select", SelectKBest(mutual_info_classif, k=20)),
    ("model", LogisticRegression(max_iter=1000)),
])

If you still need to search for k, encoding methods, and model hyperparameters, they all fall under inner selection. Final performance is estimated using outer cross-validation or a test set not involved in the selection process.

4. Check Stability and Incremental Value

A single selection does not imply reliability. Record results across different time windows, bootstrap samples, or cross-validation folds:

• Selection frequency;
• Coefficient direction and range;
• The distribution of permutation importance;
• Changes in validation metrics after addition/removal;
• Impact on delay, cost, and group error.

Highly correlated alternative features compete for importance. They should be evaluated by semantic groups, not by treating the occasional victory of a single column as evidence of a unique mechanism.

5. An executable feature definition

Production features don't just require SQL snippets:

name: team_supply_count_7d
entity: team_id
value_type: int64
event_source: confirmed_supply_events
event_time: confirmed_at
available_time: ingested_at
window: "(cutoff - 7d, cutoff]"
default: 0
freshness_slo: 15m
owner: logistics-data
version: 3

Definitions should also be tied to code versions, dependency data, permissions, deprecated plans, and allowed usage scenarios. Feature repositories can help register, materialize, and reuse definitions, but they don't automatically guarantee correctness.

6. Why Offline and Online Data Show a Bias

Common training-serving skew includes:

• Training uses batch SQL; production uses a separate application code;
• Different time zones, window boundaries, or rounding rules on both sides;
• Offline data has been replenished; the online view showed the state before the delay occurred;
• The category vocabulary list, fill values, and scaling parameters differ;
• Used an unrecorded default value when lookup failed online.

Prioritize reusing the same transformation output or the same declarative definition. When sharing an execution engine is not possible, conduct a parity test with fixed samples: for a given entity and cutoff, both paths must yield identical results or be within a clearly defined tolerance.

7. Pre-launch Verification

History Replay

Reconstruct the visible data at each past prediction time point using only data available at that time, without incorporating later backfilled truth snapshots.

Shadow Running

New features are computed online but not involved in decision-making; compare distribution, latency, missing data, and availability.

Contract Testing

Check types, ranges, uniqueness, window boundaries, default values, and behavior for unknown categories.

Rollback Deployment

Feature versions are bound to model versions; old versions are retained until the model is no longer in use and cannot be silently modified semantically.

8. Monitor Three Types of Changes

Data Quality

Missing rates, unknown category rates, duplicate entities, delays, refresh failures, and range violations.

Distribution and Consistency

Online/offline differences, quantiles, category proportions, drift statistics, and distribution by group. Drift is a signal, not automatically indicative of model failure.

Model Association

Feature coverage, predictive accuracy, calibration, group-level bias, and true performance with label delay. A feature's stable distribution may still lose its relationship with the label.

Alerts must be tied to owners, response procedures, and fallbacks, such as using the previous successful snapshot, a degradation model, or pausing automated decisions.

9. Feature Review Checklist

[ ] Legal and accessible during prediction
[ ] Entity, time, and window boundaries are clearly defined
[ ] Conversion and selection occur only during training fold fitting
[ ] Validate the split aligns with go-live scenarios
[ ] Incremental gains exceed calculation and governance costs
[ ] Offline/online parity test passed
[ ] Default values, unknown values, and late data strategies are clearly defined
[ ] Version, owner, monitoring, and rollback paths are all in place

Common Misconceptions

• The model can automatically ignore irrelevant features: Irrelevant or leaky features will still increase variance, cost, and risk.
• Validate after selection is objective: The selection itself has already used the labels.
• With a feature store, consistency is achieved: Tools can't fix erroneous windows or duplicate implementations.
• Distribution drift means retraining immediately: First determine if it's data faults, seasonal changes, or relationship breakdowns.

Practice

1. Move a label-related filter into the pipeline and compare the cross-validation scores before and after.
2. For each of 10 resampling iterations, record the feature selection frequency and explain the sources of instability.
3. Write a YAML definition and five boundary tests for a rolling feature.
4. Design training/service skew monitoring and define escalation actions for three types of alerts.

Summary

Feature selection belongs inside model selection and must not peek at validation results. Production features are versioned, operationally accountable interfaces with explicit time semantics. A usable feature improves offline scores and can be reconstructed consistently at inference time.

The next chapter shifts to sampling and causal inference: even if features and models are reliable, observed samples may still fail to represent the target population, and correlations do not necessarily justify action.