8.1 Metrics, Probability, and Decision Thresholds: Model Scores and Business Actions Are Not the Same Layer

Suppose one model has higher ROC-AUC, while another catches more costly events among the 200 cases that can be reviewed each day. Their rankings disagree because they summarize different parts of the score distribution. The deployment decision needs evidence at the actual review capacity.

Evaluation must be layered: can the model rank items reliably? Is the probability output trustworthy? And is the action taken after a threshold decision actually worthwhile? Reporting just one metric (like accuracy, F1, or AUC) blurs all three questions into one.

Learning Objectives

• Derive threshold-dependent metrics from a confusion matrix;
• Distinguish between discrimination, calibration, and decision utility;
• Accurately interpret ROC curves, precision-recall curves, AP (average precision), and probabilistic loss;
• Select optimal thresholds on validation data based on cost or capacity constraints;
• Choose appropriate evaluation metrics for regression and multiclass classification tasks.

1. Define the Prediction Problem Before Computing Metrics

Before computing any numerical value, write down:

• Unit of observation: user, device, task, or time window;
• Cutoff: what information is available at the time of prediction;
• Outcome/horizon: what constitutes a positive event within how much time in the future;
• Deployment population: new users, future records from the same device, or randomly sampled data from a shared distribution;
• Action: alerting, review, ranking, pricing, or resource allocation;
• Costs and capacity: false positives (FP), false negatives (FN), latency, and daily processing capacity.

If the labeling scheme or deployment population differs, metrics from two models cannot be directly compared. Model evaluation begins with estimand design, not with calling classification_report.

2. Confusion Matrix Threshold Dependence

For a binary classifier, the prediction of positive or negative class is first determined by comparing the score or probability against a threshold, which then leads to the following confusion matrix:

	Predicted Positive	Predicted Negative
Actual Positive	TP	FN
Actual Negative	FP	TN

Common metrics include:

$$\text{Precision}=\frac{TP}{TP+FP},\text{Recall/TPR}=\frac{TP}{TP+FN},$$$$\text{Specificity}=\frac{TN}{TN+FP},\text{FPR}=\frac{FP}{FP+TN},$$$$\text{Accuracy}=\frac{TP+TN}{N}.$$

Accuracy is not merely valid when class distributions are balanced. It can be appropriate when correct decisions have zero cost, both types of error have equal cost, and the evaluation population matches deployment. Class imbalance merely makes high accuracy from always predicting the majority class more likely to mask poor performance. The real issue is a mismatch between the chosen metric and the actual cost of decisions.

3. Precision and Recall Have Different Denominators That Answer Different Questions

• Precision: Among the actions that have been triggered, what proportion actually succeed? This is strongly influenced by prevalence and the chosen threshold.
• Recall: Out of all the actual positive cases, what fraction did the system successfully identify? This depends on whether the labeling is complete and whether the detection horizon has matured.
• Specificity measures correctly rejected negatives; FPR measures incorrectly flagged negatives. Even if the FPR is small, when the number of negative cases is large, the absolute number of false positives can still be substantial.

Operational reports should include both numerator and denominator. For example, FPR=0.1% over ten million negative cases translates to approximately ten thousand false positives.

F-score:

$$F_{\beta }=(1+{\beta }^2)\frac{PR}{{\beta }^{2}P+R}.$$

When $\beta >1$, recall is emphasized; when $\beta <1$, precision is prioritized. The F1 score does not use true negatives (TN) and lacks explicit modeling of real-world costs, such as financial or security implications. It is well-suited for certain retrieval-based objectives, but it should not be treated as a default compromise when choosing between precision and recall.

4. Multi-class Averaging Methods Can Alter Conclusions

For multi-class classification, precision, recall, and F1 are first computed per class, then aggregated across classes using one of several methods:

• macro: Each class is given equal weight, emphasizing smaller classes;
• weighted: Scores are weighted by the support (number of samples) of each class, potentially dominated by larger classes;
• micro: TP, FP, and FN are aggregated across all classes first, then used to compute a single global metric;
• per-class: Results are reported per class, offering the most insight into specific failures.

For single-label multiclass predictions over all classes, micro-precision, micro-recall, and micro-F1 equal accuracy: every wrong label contributes one FP and one FN. This equivalence does not generally hold for multilabel data or a selected subset of classes. State averaging, label selection, sample weights, and zero-division handling.

5. Measuring Ranking, Not Choosing Action Points

Changing the threshold produces different TPR/FPR pairs, which are connected to form a ROC curve. The ROC-AUC can be interpreted as: the probability that a randomly selected positive example is scored higher than a randomly selected negative example (ties are handled according to convention).

It has several key boundaries:

• It is invariant under strictly increasing score transformations, so it does not assess probability calibration;
• It aggregates performance across all possible thresholds, many of which may never be deployed in practice;
• A large number of true negatives enter the FPR denominator, so in cases of low prevalence, the curve may appear favorable, yet the actual number of false positives may still be unacceptable;
• If the case mix across groups or time periods changes, the AUC may also shift;
• If the primary concern is a very low FPR region, reporting partial AUC or operating points within that region is more appropriate.

"AUC is unaffected by threshold selection" is correct; "AUC is unaffected by class distribution and sampling" requires qualification. Uniformly replicating every sample within a class preserves its empirical conditional score distribution and ROC curve, but spectrum, sampling weights, and population composition can change the comparison objects.

6. Precision–Recall for a More Direct View of Alert Quality

The precision–recall (PR) curve plots precision against recall across different threshold values. For randomly ordered data, precision is expected to correlate with positive prevalence, so when comparing average precision (AP) across datasets with varying prevalence, the baseline prevalence must be explicitly stated.

Average Precision (AP) is a weighted discrete summation of precision values at recall increments, this definition should not be conflated with the trapezoidal PR-AUC commonly computed by arbitrary plotting libraries.

When prevalence is low and audit capacity is limited, it is common to report the following metrics together:

• AP/AUPRC (with clear definition);
• precision at $K$ / recall at $K$;
• recall at a fixed alert rate;
• absolute daily counts of true positives (TP) and false positives (FP);
• lift compared to random or rule-based baselines.

7. Probability Quality Is Not AUC

If predicted probabilities are being used in cost, inventory, or pricing calculations, the probabilities must have semantic meaning.

Binary log loss:

$$L_{\text{log}}=-\frac{1}{n}\sum _i[y_i\log p_i+(1-y_i)\log (1-p_i)].$$

It heavily penalizes confidently wrong predictions. The Brier score is the mean squared error of predicted probabilities:

$$L_{\text{Brier}}=\frac{1}{n}\sum _i(p_i-y_i{)}^2.$$

Both are proper scoring rules, but the overall score simultaneously reflects calibration, resolution/discrimination, and data uncertainty. You cannot simply conclude that calibration is better because the Brier score is lower. You should also examine the reliability diagram, sample size per bin, calibration-in-the-large, and the slope.

The calibration curve is sensitive to binning; with small sample sizes, the actual rate in each bin becomes unstable. Confidence intervals should be provided, or at minimum, the bin support should be reported.

8. Score, Probability, and Action

Break the system into this flow:

features -> model score -> calibrated probability -> decision rule -> action

For ranking tasks, the model score alone may suffice. However, to compute expected loss, calibrated probabilities are required. The final action is also influenced by cost, capacity, fairness, or safety constraints, as well as the option to reject a prediction.

The threshold of 0.5 only emerges naturally under conditions of probability calibration and specific symmetric cost assumptions. If the cost of a false negative is $C_{FN}$ and the cost of a false positive is $C_{FP}$, a simple binary decision can be derived from expected cost under ideal conditions. In real systems, however, capacity limits and individualized costs are common, making it more appropriate to optimize business utility directly on validation data.

With zero cost for correct decisions, predict positive when $C_{FP}(1-p)\le C_{FN}p$, or $p\ge C_{FP}/(C_{FP}+C_{FN})$. Costs 1 and 9 give threshold 0.1. This requires nonnegative costs with positive sum and a probability for the target population; capacity constraints change the rule.

9. Thresholds Can Only Be Selected from Validation

Select the threshold that maximizes recall while meeting the minimum precision requirement from the validation curve. The last precision/recall point in precision_recall_curve does not correspond to a defined threshold, so alignment of the length must be ensured:

This Python 3 / scikit-learn 1.9.0 fragment requires validation labels y_valid containing both 0 and 1, matching finite probabilities in [0, 1], and separate test_probability values. The 80% requirement is an empirical selection rule, not a confidence bound.

import numpy as np
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(
    y_valid,
    valid_probability,
)

eligible = np.flatnonzero(precision[:-1] >= 0.80)
if len(eligible) == 0:
    raise RuntimeError("No validation threshold reaches 80% precision")

best = eligible[np.argmax(recall[eligible])]
chosen_threshold = thresholds[best]

test_prediction = test_probability >= chosen_threshold

With validation labels [1, 0, 1, 0, 1] and probabilities [0.9, 0.8, 0.7, 0.6, 0.5], only threshold 0.9 meets 80% precision, giving recall $1/3$. A perfect one-case hit rate is weak evidence; report the selected support and uncertainty. Prevalence drift changes precision, while a fixed cost-optimal threshold can remain unchanged if probabilities are recalibrated to the new population.

Calibrators, thresholds, and model hyperparameters are all parameters selected from data. They should not be exhaustively tested on the same small validation subset. Cross-validation, nested validation, or reserved data segments with distinct purposes should be used instead.

10. Regression Metrics Also Include Cost Considerations

• MAE: Applies linear penalty, is related to median-based loss, and is less sensitive to large errors compared to MSE;
• MSE/RMSE: Imposes a quadratic penalty on large residuals; RMSE returns the error to the target unit, but cannot be directly compared across different scales;
• $R^2$: Measures relative improvement over a constant mean baseline, expressed as a squared error; it can be negative and does not imply "explanation of causal variance";
• MAPE: Can explode when true values are close to zero and is problematic for positive/negative or zero-valued data;
• Pinball/quantile loss: Predicts conditional quantiles, making it suitable for asymmetric cost structures;
• Interval coverage plus width: Evaluates prediction intervals, coverage alone is insufficient.

Residuals should be reported by target range, time period, demographic group, and key business region. A single average RMSE may mask significant failures in high-value tail cases.

Common Misconceptions

• Accuracy is completely invalid on imbalanced data: Whether it's appropriate depends on the loss relative to the overall distribution; imbalance is just a common warning signal.
• A high AUC implies reliable probabilities: AUC only evaluates ranking, not the absolute confidence of predicted probabilities.
• PR curves are unaffected by the negative class: Precision explicitly depends on false positives and the prevalence of the negative class.
• F1 balances all costs equally: It ignores true negatives and does not account for actual cost implications.
• Raising the threshold on test data still counts as final testing: The test set has already been involved in the threshold selection process.

Exercise

1. Derive precision, recall, specificity, false positive rate (FPR), and absolute alarm count from the same confusion matrix.
2. Apply a strictly increasing transformation to the score ordering and verify that AUC remains unchanged, but log loss can vary.
3. Alter the prevalence of the positive class and observe how precision and average precision (AP) baseline change relative to the ROC curve.
4. Select a threshold based on daily capacity and the relative costs of false positives and false negatives, then compare the resulting decision rules.
5. Explain why one might choose MAE, RMSE, or quantile loss for a regression task.

Summary

Metrics are not inherent scores of a model; they are measurements tailored to the prediction target, data distribution, and actionable questions. ROC and AP assess ranking performance, while log loss and Brier score, along with reliability diagrams, evaluate probabilistic calibration. Confusion matrices and utility functions examine decisions made after thresholding. Separating these three layers enables a clear explanation of why a model is worth deploying.

The next lesson addresses the origin of evaluation data: k-fold cross-validation is just one tool for the i.i.d. scenario. In practice, real-world splits are determined by users, devices, time windows, and labeling boundaries.