7.3 Feature Selection and Production Consistency: From Candidate Columns to Monitorable Interfaces

An offline seven-day supply count may include records backfilled after the historical prediction, while the online value correctly omits them. An unknown category’s all-zero encoding can also be intentional. The defect is a mismatch with the agreed availability and fallback rules; compare both paths against that contract before choosing which output to “fix.”

This lesson connects feature selection to its validation procedure, then checks whether selected features can be reproduced, monitored and rolled back in production.

Lesson Objectives

• Distinguish filter-based, wrapper-based, and embedded feature selection;
• Put feature selection inside cross-validation;
• Check for stability and genuine incremental value;
• Design offline/online consistency testing and monitoring.

1. Choose Features for a Defined Goal

Feature selection may serve different objectives:

• Reduce the costs associated with data collection, computation, and online latency;
• Reduce high-dimensional noise and overfitting;
• Improve interpretability or comply with regulatory constraints;
• Eliminate inputs that are unavailable, unstable, or redundant during prediction.

First, establish the constraints, then compare the approaches. There is no universal rule that sample size must be ten times the number of features: identifiability depends on the model, regularization, sparsity, noise, structural correlation, and validation design.

2. Three Types of Selection Methods

Filter Methods

Screened by missing rate, near-zero variance, correlation, or univariate statistics; computationally efficient but ignores feature interactions. Any score that uses labels must be computed within the training fold.

Wrapper Methods

Recursive elimination, sequential selection, and other iterative training methods are used to validate model performance and select subsets. These approaches can account for model interactions but are computationally expensive, and prone to overfitting in the validation process when the number of candidates is large.

Embedded

Lasso, tree splitting, and some sparse models perform feature selection during fitting. Results are model-dependent: Lasso can be unstable among correlated features, and tree importance does not equate to causal effect.

Domain review is the fourth gate: features that are legal but costly, hard to interpret, or possibly encode sensitive attributes cannot simply enter production based on a score.

3. Keep Selection Inside Cross-Validation

This complete scikit-learn example uses six continuous measurements from independent synthetic observations. It searches only feasible feature counts inside training CV and scores the reserved test split after selection.

First, identify the "20 most relevant columns to the label" from the full dataset, then perform cross-validation, where the validation labels are involved in the column selection. The correct approach is to place the selector inside a pipeline:

from functools import partial
import numpy as np
import pandas as pd
from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(9)
X = pd.DataFrame(rng.normal(size=(120, 6)), columns=[f"measurement_{i}" for i in range(6)])
y = (X["measurement_0"] - .8 * X["measurement_1"] + rng.normal(0, .5, 120) > 0).astype(int)
X.loc[4, "measurement_2"] = np.nan
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.25, stratify=y, random_state=0)
preprocess = Pipeline([
    ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()),
])
mi_score = partial(mutual_info_classif, discrete_features=False, random_state=0)
candidate = Pipeline([
    ("preprocess", preprocess),
    ("select", SelectKBest(mi_score, k=3)),
    ("model", LogisticRegression(max_iter=1000)),
])
search = GridSearchCV(
    candidate, {"select__k": [2, 3, "all"]}, scoring="balanced_accuracy",
    cv=StratifiedKFold(n_splits=3, shuffle=True, random_state=0),
)
search.fit(X_train, y_train)
selected = X_train.columns[search.best_estimator_.named_steps["select"].get_support()]
print(selected.tolist())
print(search.score(X_test, y_test))

If you still need to search for k, encoding methods, and model hyperparameters, they all fall under inner selection. Final performance is estimated using outer cross-validation or a test set not involved in the selection process.

mutual_info_classif needs the discrete/continuous status of every transformed feature. Its auto mode treats dense input as continuous and sparse input as discrete, which can be wrong for a mixed one-hot/numeric matrix. This example sets all six measurements continuous and fixes the random seed. Mixed representations need an aligned mask or a suitable alternative scoring method; k also cannot exceed the number of available columns. See the MI API.

4. Check Stability and Incremental Value

A single selection does not imply reliability. Record results across different time windows, bootstrap samples, or cross-validation folds:

• Selection frequency;
• Coefficient direction and range;
• The distribution of permutation importance;
• Changes in validation metrics after addition/removal;
• Impact on delay, cost, and group error.

Highly correlated alternative features compete for importance. They should be evaluated by semantic groups, not by treating the occasional victory of a single column as evidence of a unique mechanism.

5. A Feature Definition You Can Check

Production features don't just require SQL snippets:

name: team_supply_count_7d
entity: team_id
value_type: int64
event_source: confirmed_supply_events
event_time: confirmed_at
available_time: ingested_at
window: "(cutoff - 7d, cutoff]"
default_when_complete: 0
oncomplete_or_lookup_failure: missing
event_key: event_id
revision_order: increasing_revision
retraction_status: cancelled
freshness_slo: 15m
owner: logistics-data
version: 3

Definitions should also be tied to code versions, dependency data, permissions, deprecation plans, and allowed usage scenarios. Feature repositories can help register, materialize, and reuse definitions, but they don't automatically guarantee correctness.

The YAML is an illustrative contract, not a standard configuration accepted by every feature store. Zero applies only when the relevant source is known complete. A timeout, incomplete partition or failed lookup must not become “zero supplies.” The following reference calculation runs after the earlier Python block:

def supply_count_7d(events, team_id, cutoff, source_complete):
    if not source_complete:
        return {"value": None, "reason": "incomplete_source"}
    visible = events.loc[events["ingested_at"] <= cutoff]
    if visible.duplicated(["event_id", "revision"]).any():
        raise ValueError("Resolve duplicate event revisions before feature computation")
    latest = visible.sort_values(["event_id", "revision"]).drop_duplicates("event_id", keep="last")
    matches = (
        latest["team_id"].eq(team_id) & latest["status"].eq("confirmed")
        & latest["confirmed_at"].gt(cutoff - pd.Timedelta(days=7))
        & latest["confirmed_at"].le(cutoff)
    )
    return {"value": int(matches.sum()), "reason": "ok"}

cutoff = pd.Timestamp("2026-08-08T12:00:00Z")
events = pd.DataFrame([
    ("a", 1, "T1", cutoff - pd.Timedelta(days=7), cutoff, "confirmed"),
    ("b", 1, "T1", cutoff, cutoff, "confirmed"),
    ("c", 1, "T1", cutoff - pd.Timedelta(days=1), cutoff + pd.Timedelta(hours=1), "confirmed"),
    ("d", 1, "T1", cutoff - pd.Timedelta(hours=1), cutoff - pd.Timedelta(hours=1), "confirmed"),
    ("d", 2, "T1", cutoff - pd.Timedelta(hours=1), cutoff, "cancelled"),
], columns=["event_id", "revision", "team_id", "confirmed_at", "ingested_at", "status"])
print(supply_count_7d(events, "T1", cutoff, source_complete=True))  # value=1
print(supply_count_7d(events, "T1", cutoff, source_complete=False))  # value=None

Only b counts: a is exactly at the excluded left boundary, c is not available yet, and d has a visible cancellation. The source contract requires non-null UTC instants, immutable event IDs and strictly increasing revisions; later revisions may alter or retract an earlier fact. Here ingested_at is assumed to equal prediction-path availability. Production sources must verify that assumption and the completeness flag independently.

6. Why Offline and Online Data Show a Bias

Common training-serving skew includes:

• Training uses batch SQL; production uses a separate application code;
• Different time zones, window boundaries, or rounding rules on both sides;
• Offline data has been backfilled; the online prediction used only records that had arrived at its cutoff;
• The category vocabulary list, fill values, and scaling parameters differ;
• Used an unrecorded default value when lookup failed online.

Prioritize reusing the same transformation output or the same declarative definition. When sharing an execution engine is not possible, conduct a parity test with fixed samples: for a given entity and cutoff, both paths must yield identical results or be within a clearly defined tolerance.

7. Pre-launch Verification

History Replay

Reconstruct the visible data at each past prediction time point using only data available at that time, without incorporating later backfilled truth snapshots.

Shadow Running

New features are computed online but not involved in decision-making; compare distribution, latency, missing data, and availability.

Contract Testing

Check types, ranges, uniqueness, window boundaries, default values, and behavior for unknown categories.

Rollback Deployment

Feature versions are bound to model versions; old versions are retained until the model is no longer in use and cannot be silently modified semantically.

8. Monitor Three Types of Changes

Data Quality

Missing rates, unknown category rates, duplicate entities, delays, refresh failures, and range violations.

Distribution and Consistency

Online/offline differences, quantiles, category proportions, drift statistics, and distribution by group. Drift is a signal, not automatically indicative of model failure.

Model Association

Feature coverage, predictive accuracy, calibration, group-level bias, and true performance with label delay. A feature's stable distribution may still lose its relationship with the label.

Alerts must be tied to owners, response procedures, and fallbacks, such as using the previous successful snapshot, a fallback model, or pausing automated decisions.

9. Feature Review Checklist

[ ] Legal and accessible during prediction
[ ] Entity, time, and window boundaries are clearly defined
[ ] Conversion and selection occur only during training fold fitting
[ ] Validate the split aligns with go-live scenarios
[ ] Incremental gains exceed calculation and governance costs
[ ] Offline/online parity test passed
[ ] Default values, unknown values, and late data strategies are clearly defined
[ ] Version, owner, monitoring, and rollback paths are all in place

Common Misconceptions

• The model can automatically ignore irrelevant features: Irrelevant or leaky features will still increase variance, cost, and risk.
• Validate after selection is objective: The selection itself has already used the labels.
• With a feature store, consistency is achieved: Tools can't fix erroneous windows or duplicate implementations.
• Distribution drift means retraining immediately: First determine if it's data faults, seasonal changes, or relationship breakdowns.

Practice

1. Move a label-related filter into the pipeline and compare the cross-validation scores before and after.
2. For each of 10 resampling iterations, record the feature selection frequency and explain the sources of instability.
3. Write a YAML definition and five boundary tests for a rolling feature.
4. Design training/service skew monitoring and define escalation actions for three types of alerts.

Summary

Feature selection belongs inside model selection and must not peek at validation results. Production features are versioned, operationally accountable interfaces with explicit time semantics. A usable feature improves offline scores and can be reconstructed consistently at inference time.

The next chapter shifts to sampling and causal inference: even if features and models are reliable, observed samples may still fail to represent the target population, and correlations do not necessarily justify action.