7.2 Numerical, Categorical, and Temporal Encoding: Transformations Belong to the Model

After passing time-based validation, temperature, equipment level, task type, and departure time are now valid inputs. But the model interprets these values differently: categorical indices are not quantities, midnight and 11:59 PM are not close in time, and scaling alters regularization penalties.

Transformation is not a permanent "cleaning result" of a data table. It is part of the model itself. It must be fitted alongside training, versioned, and deployed with the model.

Learning Objectives

• Scale numerical values based on the model's mechanism;
• Properly handle unordered, ordered, and high-cardinality categorical features;
• Understand why target encoding must be cross-fitted;
• Use a pipeline to ensure consistent transformations across training folds and online deployment.

1. Scaling Solves What Problem

Models sensitive to distance, inner product, gradients, or coefficient penalties are affected by feature scaling:

• k-nearest neighbors and clustering algorithms are dominated by features with large scales;
• ridge and lasso regularization penalize coefficients, but the penalty's meaning changes when scales are inconsistent;
• neural networks and gradient-based optimization often train more reliably when feature scales are similar.

Tree-based models make splits based on thresholds and generally do not require standardization for prediction purposes. However, specific implementations, regularization techniques, or downstream explanation tools may introduce exceptions.

• StandardScaler uses the training set mean and standard deviation;
• RobustScaler uses median and interquartile range, making it less sensitive to outliers;
• MinMaxScaler maps features into a training range, but does not ensure future values remain within that interval.

"It's always best to use RobustScaler when there are outliers" is not a rule. The choice should be driven jointly by the model type, loss function, data distribution, and validation outcomes.

2. Unordered Categories: One-Hot Encoding and Rare Values

The task type has no natural ordering, so feeding scout=1, rescue=2, transport=3 directly into a linear model would fabricate distance and size relationships. One-hot encoding creates indicator columns for each category.

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(
    handle_unknown="infrequent_if_exist",
    min_frequency=20,
)

min_frequency Should be determined by training scale and business semantics, and there's no universal boundary like "one-hot encoding when fewer than 10 categories." Merging rare categories can reduce dimensionality but might also blend together groups with different risks.

The unknown category strategy must be clearly defined: ignore, assign to a rare class, reject the request, or fall back to the old model. Offline validation should simulate new categories that might appear online.

In linear models, arbitrarily dropping a one-hot category changes the reference coefficient; in penalized models, removing a category may introduce asymmetry. Encoding parameters should be designed in tandem with the model's interpretability.

3. Ordered Categories: Order Does Not Imply Equidistance

low < medium < high With order, explicit category tables can be used:

from sklearn.preprocessing import OrdinalEncoder

ordinal = OrdinalEncoder(
    categories=[["low", "medium", "high"]],
    handle_unknown="use_encoded_value",
    unknown_value=-1,
)

Encoding values as 0, 1, 2 implicitly assumes equal distance between adjacent levels. If this assumption is invalid, consider one-hot encoding, monotonic constraints, or domain-driven numerical definitions. Avoid substituting alphabetical ordering for business-defined sequence.

4. High-Cardinality Categories and Target Encoding

Categories like castles, devices, or merchants may have tens of thousands of distinct values. Possible approaches include:

• Merging hierarchically meaningful levels;
• Frequency or count encoding;
• Hash encoding;
• Learned embeddings;
• Smoothed target encoding.

Target encoding substitutes a category with the mean label of that category, making it prone to data leakage. If training rows participate in computing the mean of their own category, rare categories will effectively memorize their labels. The correct procedure is:

1. Split the training set into folds internally;
2. The encoding for each fold is computed solely from labels in the other folds;
3. Validation, test, and online data use only the mapping derived from the full training set;
4. Smooth the encoded values using a global prior and category sample size;
5. For unseen categories, fall back to a predefined prior.

This is cross-fitting. Simply placing a target encoder within an outer pipeline does not imply that its internal logic has been cross-fitted; you must verify that the implementation adheres to the fit_transform semantics.

5. Time Is Not an Integer

From a timestamp, you can derive:

• Hours, weeks, months, and holidays;
• Duration since a specific event;
• Rolling counts, rates, and trends;
• Business cycles and seasonal patterns.

Hours are periodic. For linear models, they can be represented as:

$$ x_{\sin} = \sin(2\pi h / 26),\qquad x_{\cos} = \cos(2\pi h / 24). $$

However, periodic encoding does not handle time zones or daylight saving time. First, clarify whether the business operates on local time or UTC, and account for missing or duplicated local times. elapsed_days The model may also retain trends from its training period. Before deployment into a new period, perform time-out validation to ensure robustness.

6. Nonlinear Transformations and Interactions

Log transformations can compress right-skewed positive values, but zero, negative, and unit values must be explicitly handled. Binning can represent thresholds, yet it loses internal information within bins; bin boundaries should be learned during training folds. Interaction terms express the idea that "the effect of one variable depends on the value of another." Ratio features require a stable denominator and should retain both numerator and denominator to enable diagnostic analysis.

Transformations should correspond to a plausible underlying mechanism or be validated independently to demonstrate benefit. Excessive automated feature interactions can inflate dimensionality and expand the space for overfitting.

7. Put the Transformation into the Pipeline

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["temperature", "days_since_supply"]
categorical = ["mission_type", "equipment_level"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
    ]), numeric),
    ("cat", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical),
])

model = Pipeline([
    ("features", preprocess),
    ("classifier", LogisticRegression(max_iter=1000)),
])

Cross-validation will refit the imputation, scaling, and vocabulary mapping for each training fold. Save and deploy the entire pipeline, rather than the final classifier alone.

Common Misconceptions

• All models need to be standardized: It depends on the model mechanism, not on data science rituals.
• Integer encoding for categories saves the most space: It may introduce false ordinal relationships and distances into the model.
• Target encoding is safe as long as you split the data first: Out-of-fold encoding is still required during training.
• The finer the time split, the better: High-dimensional calendar features may memorize historical noise.

Exercise

1. Explain which numerical columns require scaling for linear models versus tree models.
2. Test the behavior of unknown category entries using task types that did not appear in the training set.
3. Implement out-of-fold target encoding and compare its performance against incorrect full-training-set encoding to validate accuracy.
4. Generate hour features using both UTC and local time, and examine the behavior at the daylight saving time boundary.

Summary

Both coding and scaling introduce mathematical assumptions. Parameters must learn exclusively from training data, unknown values and time boundaries must be predefined, and the complete transformation chain must be delivered alongside the model.

The next lesson will constrain candidate features into maintainable interfaces: selections must occur within the validation process, and training and serving must remain aligned on the same semantic meaning.