7.2 Numerical, Categorical, and Temporal Encoding: Transformations Belong to the Model

Once availability checks pass, temperature, equipment level, task type and departure time can become inputs. Their representations still matter: category IDs are not quantities, 23:00 and midnight are one hour apart on the daily cycle, and scaling changes regularization penalties.

Transformation is not a permanent "cleaning result" of a data table. It is part of the model itself. It must be fitted alongside training, versioned, and deployed with the model.

Learning Objectives

• Scale numerical values based on the model's mechanism;
• Properly handle unordered, ordered, and high-cardinality categorical features;
• Understand why target encoding must be cross-fitted;
• Use a pipeline to ensure consistent transformations across training folds and online deployment.

1. Scaling Solves What Problem

Models sensitive to distance, inner product, gradients, or coefficient penalties are affected by feature scaling:

• k-nearest neighbors and clustering algorithms are dominated by features with large scales;
• ridge and lasso regularization penalize coefficients, but the penalty's meaning changes when scales are inconsistent;
• neural networks and gradient-based optimization often train more reliably when feature scales are similar.

Tree-based models make splits based on thresholds and generally do not require standardization for prediction purposes. However, specific implementations, regularization techniques, or downstream explanation tools may introduce exceptions.

• StandardScaler uses the training set mean and standard deviation;
• RobustScaler uses median and interquartile range, making it less sensitive to outliers;
• MinMaxScaler maps features into a training range, but does not ensure future values remain within that interval.

"It's always best to use RobustScaler when there are outliers" is not a rule. The choice should be driven jointly by the model type, loss function, data distribution, and validation outcomes.

2. Unordered Categories: One-Hot Encoding and Rare Values

Run the blocks in order with scikit-learn 1.9 or later, NumPy and pandas. TargetEncoder first appeared in 1.3; this example uses the CV-object interface introduced in 1.9, with shuffling configured on KFold.

The task type has no natural ordering, so feeding auditor=1, rescue=2, transport=3 directly into a linear model would fabricate distance and size relationships. One-hot encoding creates indicator columns for each category.

import numpy as np
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(
    handle_unknown="infrequent_if_exist", min_frequency=2, sparse_output=False,
)
encoder.fit(np.array([["survey"]] * 5 + [["rescue"], ["transport"]], dtype=object))
print(encoder.get_feature_names_out())
print(encoder.transform([["survey"], ["rescue"], ["new_type"]]))

min_frequency Should be determined by training scale and business semantics, and there's no universal boundary like "one-hot encoding when fewer than 10 categories." Merging rare categories can reduce dimensionality but might also blend together groups with different risks.

The unknown category strategy must be clearly defined: ignore, assign to a rare class, reject the request, or fall back to the old model. Offline validation should simulate new categories that might appear online.

Here survey has its own column; rescue, transport and new_type use the rare-category column. With infrequent_if_exist, that column exists only if fitting actually formed a rare group. Otherwise an unseen value becomes all zeros, as with ignore. Inspect the fitted categories and test both cases. See OneHotEncoder.

In linear models, arbitrarily dropping a one-hot category changes the reference coefficient; in penalized models, removing a category may introduce asymmetry. Encoding parameters should be designed in tandem with the model's interpretability.

3. Ordered Categories: Order Does Not Imply Equidistance

For ordered levels such as low < medium < high, define the order explicitly:

from sklearn.preprocessing import OrdinalEncoder

ordinal = OrdinalEncoder(
    categories=[["low", "medium", "high"]],
    handle_unknown="use_encoded_value", unknown_value=-1,
)
ordinal.fit([["low"], ["medium"], ["high"]])
print(ordinal.transform([["low"], ["high"], ["unknown"]]))  # 0, 2, -1

Encoding values as 0, 1, 2 implicitly assumes equal distance between adjacent levels. If this assumption is invalid, consider one-hot encoding, monotonic constraints, or domain-driven numerical definitions. Avoid substituting alphabetical ordering for business-defined sequence.

The sentinel -1 marks an unknown value; it does not mean “below low.” A linear model will nevertheless treat it numerically unless the representation handles unknownness separately. Ordinal coding also does not enforce a monotonic prediction by itself.

4. High-Cardinality Categories and Target Encoding

• Merging hierarchically meaningful levels;
• Frequency or count encoding;
• Hash encoding;
• Learned embeddings;
• Smoothed target encoding.

Target encoding substitutes a category with the mean label of that category, making it prone to data leakage. If training rows participate in computing the mean of their own category, rare categories will effectively memorize their labels. The correct procedure is:

1. Split the training set into folds internally;
2. The encoding for each fold is computed solely from labels in the other folds;
3. Validation, test, and online data use only the mapping derived from the full training set;
4. Smooth the encoded values using a global prior and category sample size;
5. For unseen categories, fall back to a predefined prior.

This is cross-fitting. Simply placing a target encoder within an outer pipeline does not imply that its internal logic has been cross-fitted; you must verify that the implementation adheres to the fit_transform semantics.

from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import KFold

rare_categories = np.array([[f"category_{i}"] for i in range(12)], dtype=object)
labels = np.array([0., 1.] * 6)
target_encoder = TargetEncoder(
    target_type="continuous", smooth=0.0,
    cv=KFold(n_splits=3, shuffle=True, random_state=0),
)
out_of_fold = target_encoder.fit_transform(rare_categories, labels)
full_mapping = target_encoder.transform(rare_categories)
print(out_of_fold.ravel())
print(full_mapping.ravel())  # exactly labels here: every category occurs once
print(target_encoder.transform([["unseen"]]))  # [[0.5]]

With zero smoothing and singleton categories, the full mapping reproduces the training labels exactly. Out-of-fold values use only other folds’ priors. Zero smoothing is chosen to expose the failure, not as a general recommendation. TargetEncoder distinguishes fit_transform(X,y) from fit(X,y).transform(X). Random internal folds do not establish time-safe or group-safe encoding; the inner split and each prior must respect the same information boundary as the outer evaluation.

5. Time Is Not an Integer

From a timestamp, you can derive:

• Hours, weekdays, months, and holidays;
• Duration since a specific event;
• Rolling counts, rates, and trends;
• Business cycles and seasonal patterns.

Hours are periodic. For linear models, they can be represented as:

$$x_{\sin }=\sin (2\pi h/24),x_{\cos }=\cos (2\pi h/24).$$

However, periodic encoding does not handle time zones or daylight saving time. First, clarify whether the business operates on local time or UTC, and account for missing or duplicated local times. elapsed_days may encode training-period trends; evaluate on a later time period before relying on that trend in deployment.

import pandas as pd

hours = np.array([23., 0., 6., 12., 18.])
cyclic = np.column_stack((np.sin(2 * np.pi * hours / 24), np.cos(2 * np.pi * hours / 24)))
print(np.linalg.norm(cyclic[0] - cyclic[1]))  # approximately 0.2611
utc_times = pd.to_datetime(["2026-11-01T05:30:00Z", "2026-11-01T06:30:00Z"], utc=True)
local_times = utc_times.tz_convert("America/New_York")
print(local_times)  # two different instants, both local 01:30, with different offsets

The two local 01:30 values are an hour apart in UTC. Hour features intentionally discard that distinction; event ordering and elapsed durations must retain the actual instants. One sine/cosine pair represents a smooth first daily harmonic, not arbitrary hour-specific effects.

6. Nonlinear Transformations and Interactions

• Log transformations can compress right-skewed positive values, but zero values, negative values, and units must be handled explicitly;
• Binning can represent thresholds but loses information within each bin, and boundaries should be learned inside training folds;
• Interaction terms express that one variable's effect depends on another variable;
• Ratio features need a stable denominator, while the numerator and denominator should remain available for diagnosis.

Transformations should correspond to a plausible underlying mechanism or be validated independently to demonstrate benefit. Excessive automated feature interactions can inflate dimensionality and expand the space for overfitting.

7. Put the Transformation into the Pipeline

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["temperature", "days_since_supply"]
categorical = ["mission_type", "equipment_level"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
    ]), numeric),
    ("cat", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical),
])

model = Pipeline([
    ("features", preprocess),
    ("classifier", LogisticRegression(max_iter=1000)),
])

Cross-validation will refit the imputation, scaling, and vocabulary mapping for each training fold. Save and deploy the entire pipeline, rather than the final classifier alone.

training = pd.DataFrame({
    "temperature": [10., 20., np.nan, 30., 15., 35., 18., 28.],
    "days_since_supply": [1., 2., 3., 9., np.nan, 8., 2., 7.],
    "mission_type": ["survey", "repair"] * 4,
    "equipment_level": ["low", "high", "low", "high", "low", "high", "low", "high"],
})
y_training = np.array([0, 0, 1, 1, 0, 1, 0, 1])
model.fit(training, y_training)
new_rows = pd.DataFrame({
    "temperature": [np.nan], "days_since_supply": [4.],
    "mission_type": ["unseen"], "equipment_level": ["high"],
})
print(model.predict_proba(new_rows))

This smoke example checks the fitted pipeline and one unknown category; it does not estimate predictive quality. Missing-indicator columns are learned from missingness present at fit time, so monitoring must also detect a newly missing feature that had no missing training values. Business-defined fixed thresholds or calendar periods need not be learned; data-estimated boundaries and parameters do.

Common Misconceptions

• All models need to be standardized: It depends on the model mechanism, not on data science rituals.
• Integer encoding for categories saves the most space: It may introduce false ordinal relationships and distances into the model.
• Target encoding is safe as long as you split the data first: Out-of-fold encoding is still required during training.
• The finer the time split, the better: High-dimensional calendar features may memorize historical noise.

Exercise

1. Explain which numerical columns require scaling for linear models versus tree models.
2. Test the behavior of unknown category entries using task types that did not appear in the training set.
3. Implement out-of-fold target encoding and compare its performance against incorrect full-training-set encoding to validate accuracy.
4. Generate hour features using both UTC and local time, and examine the behavior at the daylight saving time boundary.

Summary

Both coding and scaling introduce mathematical assumptions. Parameters must learn exclusively from training data, unknown values and time boundaries must be predefined, and the complete transformation chain must be delivered alongside the model.

The next lesson will constrain candidate features into maintainable interfaces: selections must occur within the validation process, and training and serving must remain aligned on the same semantic meaning.