7.3 Anomaly Detection, Novelty, and Thresholds: Anomaly Scores Only Matter After Entering the Response Workflow

A detector can rank unusual transactions without knowing which ones deserve investigation. Choosing who gets investigated is a separate decision, shaped by capacity, error cost, and the current event rate.

Learning Objectives

• Distinguish between outlier detection and novelty detection;
• Understand the assumptions behind Isolation Forest, LOF, and robust covariance;
• Separate anomaly scores, thresholds, and business response actions;
• Design evaluation strategies in the presence of sparse labels, latency, and data contamination;
• Address time drift, group differences, and feedback effects.

1. "Anomalies" Must Be Defined by Context

Rarity does not imply harm, and frequency does not mean normality. A massive order might signal fraud, or it could be a legitimate large customer. A low temperature reading on one device might be normal, but could indicate a failure on a different model.

First, clearly define:

• entity: Whether the scoring applies to users, transactions, devices, or time windows;
• cutoff: Which fields are available for prediction at the time of inference;
• anomaly event: What specific outcome warrants action;
• horizon: Over what time period should the result be validated;
• action/capacity: Whether the response should be an investigation, rate limiting, shutdown, or simply logging;
• label delay: How long after the event must we wait to know whether it was truly anomalous.

Without these definitions, "outlier" remains just a geometric observation.

2. Outlier and novelty are two distinct training conditions

Outlier detection

The training dataset contains a small amount of outliers. The model is trained to identify points that are low-density, easily separable, or deviate from the main structure. fit_predict(X) is commonly used for exploration within the same snapshot.

Novelty detection

The training data is assumed to represent a normal state approximately. The goal is to determine whether future, unseen samples deviate from this normal distribution. It should be trained on a baseline period or normal population fit, and then new data should be evaluated predict or scored.

Both approaches may be referred to as anomaly detection, but their APIs and evaluation methods differ. Treating a contaminated training set as a pure normal baseline causes the model to learn outlier patterns as normal. Conversely, labeling outliers in the training data as anomalies for future deployment performance leads to overly optimistic expectations.

3. Why Use Path Length in Isolation Forest?

Isolation Forest repeatedly chooses a feature and a random split within its current range. Points isolated after fewer splits have shorter paths and tend to receive more anomalous scores.

This intuition stems from the fact that anomalous points often reside in sparse or edge regions of the data space, where fewer random splits are sufficient to separate them from the rest. Averaging path lengths across multiple trees reduces random noise and improves stability.

It is not a supervised random forest: splits are not optimized for labels or Gini impurity. In high-dimensional data, irrelevant features, clustered anomalies, categorical encodings, and distribution shifts can still degrade performance.

4. From Scores to Validation Thresholds

Score directions can vary across libraries (some may be positive, others negative) so minimum testing is required to confirm behavior. In scikit-learn, IsolationForest.score_samples values generally increase with normality, so taking the negative of this score yields a unified metric where "higher values indicate anomalies":

This Python 3 / scikit-learn 1.9.0 fragment requires finite numeric train/validation/test matrices in the same representation. Validation and test rows must be unseen; each set is nonempty. Any fitted preprocessing uses training data only.

import numpy as np
from sklearn.ensemble import IsolationForest

detector = IsolationForest(
    n_estimators=400,
    max_samples="auto",
    contamination="auto",
    random_state=42,
    n_jobs=1,
)
detector.fit(X_train)

valid_anomaly_score = -detector.score_samples(X_valid)

# If reliable labels are unavailable, temporarily define candidate thresholds based on manual capacity review.
# This targets roughly 1% of validation rows, not a known anomaly prevalence.
threshold = np.quantile(valid_anomaly_score, 0.99)

test_anomaly_score = -detector.score_samples(X_test)
test_alert = test_anomaly_score >= threshold

The 99th percentile is only an approximate validation budget. Ties at the threshold may select far more than 1%, and drift changes the test alert rate. For a hard daily limit of $K$, rank each day’s eligible cases and use a deterministic tie-breaker to select at most $K$.

If validation data includes well-established labels, threshold selection should be based on cost, precision/recall, or daily top-$K$ capacity. Testing is only used for final evaluation after model freezing.

contamination is primarily used to convert continuous scores into decision thresholds within the training distribution. Setting it to 0.01 does not inform the model of the actual business-level anomaly rate, nor does it guarantee that all 1% alerts are true anomalies.

5. LOF Comparison of Local Density

Local Outlier Factor (LOF) compares the local reachability density of a sample to that of its neighbors. A point may still be considered normal even if it lies in a globally sparse region, as long as its neighbors are also sparse. Only when a point is significantly more isolated than its surrounding neighbors does the LOF score increase.

n_neighbors determines the scale of "locality." If too small, it becomes sensitive to noise; if too large, it approaches global density assessment. The choice of distance metric, scaling, and the quality of nearest neighbors in high-dimensional spaces are equally important.

In scikit-learn, it's essential to distinguish between two distinct modes:

• Default novelty=False: used for outlier detection during training of a snapshot, invoking fit_predict, with training scores evaluated at negative_outlier_factor_;
• novelty=True: applied after a baseline dataset fit has been established, to evaluate new samples using predict, decision_function, or score_samples. These methods should not be used on training samples, and results from this mode should not be mixed with those from the default mode for comparison.

This fragment requires a baseline X_train_normal with more than 30 rows and a separate X_new, both transformed using the same training-fitted distance representation:

from sklearn.neighbors import LocalOutlierFactor

novelty_detector = LocalOutlierFactor(
    n_neighbors=30,
    contamination="auto",
    novelty=True,
)
novelty_detector.fit(X_train_normal)
new_anomaly_score = -novelty_detector.score_samples(X_new)

6. Other Approaches Each Come With Assumptions

Robust covariance / Elliptic Envelope

This method uses robust location and covariance estimates to compute Mahalanobis distance, suitable when the main data distribution approximates a unimodal ellipsoid or Gaussian. Caution is needed in cases of multimodal distributions, nonlinear manifolds, or when $p$ is close to or exceeds $n$.

One-Class SVM

Learns a boundary that encloses normal data, with the shape determined by the kernel and bandwidth. It is sensitive to scale and hyperparameters, and training with large datasets and complex kernels can be computationally expensive.

Reconstruction error

Trains a PCA or autoencoder on normal data and evaluates anomalies based on reconstruction error. If the model capacity is too high, it may reconstruct even anomalies well; if normal states are highly diverse, small normal subgroups may be incorrectly flagged as outliers.

Rules and Supervised Models

Explicitly defined impossible conditions often work well for rule-based approaches. When sufficient mature labels are available, a supervised rare-event model typically provides a more direct path to optimizing business objectives. Unsupervised scores can serve as a feature or baseline, not as a substitute for domain knowledge or decision-making.

7. Preprocessing Defines Who Is Outlier

• Large units can dominate distance-based detectors. Ideal Isolation Forest is invariant to positive affine rescaling of individual features; do not apply the same scaling argument to every detector;
• One-hot encoded rare categories may appear isolated because of their sparse representation;
• Imputed missing values can create false high-density clusters;
• Statistics from future time windows introduce temporal leakage;
• Standardizing the full dataset before splitting lets future distributions influence the baseline.

Training, validation, and test sets are split by time or entity, and all transformations are fitted only within their corresponding training segments. For periodic or seasonal systems, conditional baselines (such as by hour-of-week or device model) can be used instead of a single global "normal."

8. Evaluating When Labels Are Scarce

When mature labels are available, prioritize reporting the following:

• precision-recall curves, rather than relying solely on ROC AUC;
• recall at daily alert budget or top-$K$ thresholds;
• the cost per investigation, the cost of false negatives, and net financial gain;
• time-to-detect;
• false positive rates stratified by group, device model, and time.

When complete labels are unavailable, consider:

• conducting blinded expert reviews on the top-ranked samples;
• tracking delayed outcomes such as subsequent outages, refunds, or manual interventions;
• performing time-backtests using historically known events;
• injecting controlled, mechanism-consistent faults to verify the model’s minimum detection capability;
• reporting on selection bias in review samples and unlabelled uncertainty.

Generating synthetic anomalies to create extreme cases only demonstrates that the model can identify such extremes, it does not prove it can detect real-world anomalies. Synthetic anomalies cannot substitute for production backtesting.

9. Alerts Can Alter Subsequent Data

After a model goes live, samples that trigger alerts receive increased scrutiny and labeling, while non-alert samples may remain unexamined for a long time. Retraining the model directly on these updated labels introduces verification or selection bias.

Mitigation strategies include:

• Retaining a random audit sample outside of alert triggers;
• Recording score values, thresholds, model versions, and disposition outcomes;
• Separating "uninvestigated" from "confirmed normal";
• Monitoring alert volume, score distribution, missing rate, and investigation hit rate;
• Establishing rules for threshold degradation, model failure, and manual intervention.

Attackers can also adapt to publicly known rules, so in fraud or security scenarios, adversarial adaptation must be incorporated into validation.

10. Re-calibrate Thresholds After Drift

Device updates, seasonal shifts, and business growth alter the underlying distribution. Fixed score thresholds can suddenly flood or silence alert queues.

Monitoring must distinguish between:

• covariate drift: changes in input features or embeddings;
• score drift: a systematic shift in the overall anomaly score;
• alert-rate drift: a change in the proportion of alerts above the threshold;
• outcome drift: a change in actual risk associated with the same score;
• capacity drift: the number of alerts human operators can realistically process per day.

Periodic retraining does not guarantee safety. Each update must replay a stable window, retain incident cases, compare group-level impacts, and version the threshold settings.

Common Misconceptions

• Anomalies are simply the ones farthest from the mean: Local density, conditional groups, and task semantics often matter more than distance from the mean.
• contamination represents the true anomaly rate learned by the model: It primarily governs the threshold for scoring.
• Unlabeled data makes evaluation impossible: Evaluation can still be performed using audit logs, delayed results, backtesting, and capacity metrics, though limitations must be disclosed.
• A good score on the training set guarantees detection of future novelty: The training conditions and API behavior may differ significantly from real-world usage.
• More alerts mean greater security: Under capacity constraints, this leads to reduced precision and alert fatigue, clogging incident response workflows.

Exercise

1. Define entity, cutoff, horizon, action, and label delay for transactional or device-based scenarios.
2. Verify the direction and threshold relationships of each scoring API in IsolationForest.
3. Compare LOF's outlier detection with novelty detection patterns to avoid invoking methods on erroneous samples.
4. Compare precision, recall, and time-to-detect when evaluating 200 alerts per day.
5. Design a set of random audit samples from non-alerted cases to estimate false negatives in undetected regions.

Summary

Anomaly detection learns a reference structure, then passes scores to a threshold and response process. Isolation Forest uses isolation paths, LOF compares local densities, and other methods rely on covariance, boundary, or reconstruction assumptions. Time splits, review capacity, feedback bias, and drift determine whether alerts remain useful.

The next chapter moves into model evaluation: organizing metrics, cross-validation, hyperparameter tuning, and final testing into a protocol that avoids contamination between components.