5.2 Logistic Regression, Log Loss, and Thresholds: Probability Models and Action Rules Are Two Layers

Now in the Developer Workshop, we're predicting whether a task requests an emergency supply. A simple linear regression would produce values less than 0 or greater than 1. Logistic regression maps the linear predictor into a conditional probability, and then the decision layer selects a threshold based on cost and capacity considerations.

Learning Objectives

• Derive binary cross-entropy from Bernoulli likelihood;
• Explain how log-odds relate coefficients to the decision boundary;
• Address separation, class imbalance, and numerical stability;
• Distinguish discrimination, calibration, and threshold utility.

1. Modeling from log-odds

Let $ p(x) = P(Y=1 \mid X=x) $:

$$ \log\frac{p(x)}{1-p(x)} = x^T\beta, $$

thus:

$$ p(x) = \sigma(x^T\beta) = \frac{1}{1 + e^{-x^T\beta}}. $$

Increasing a feature $ x_j $ by 1 (while holding all other design columns constant) multiplies the odds by $ e^{\beta_j} $. This is not an increase in probability by $ \beta_j $; the change in probability depends on the current value of $ p(x) $.

Interactions and nonlinear basis functions make the boundaries of the original input space more complex, but the log-odds remain linear in the design features.

2. Bernoulli Likelihood and Log Loss

For $y_i \in {0,1}$:

$$ P(y_i \mid x_i) = p_i^{y_i}(1 - p_i)^{1 - y_i}. $$

The negative average log-likelihood is:

$$ \mathcal{L} = -\frac{1}{n} \sum_i \left[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \right]. $$

This loss penalizes confident but incorrect predictions heavily and is a proper scoring rule. Training accuracy is not differentiable and ignores confidence levels, making it unsuitable as a standard logistic objective.

Numerical implementation directly computing log(sigmoid(z)) will suffer from overflow or underflow; instead, use stable logaddexp/softplus or a library's logits loss function.

3. Logistic Regression Does Not Require Linear Separability

When data cannot be linearly separated, the model can still find a maximum likelihood or regularized solution, though the decision boundary will not perfectly classify all instances. The real issue arises in cases of perfect or quasi-separation: when a particular feature direction can completely isolate one class from another, the maximum likelihood estimates without regularization may diverge to infinity, leading to unstable finite solutions.

Mitigation strategies include:

• L2 or L1 regularization;
• Collecting samples near the decision boundary;
• Merging sparse or underrepresented classes;
• Using specialized estimation methods such as Firth correction (particularly in inference scenarios);
• Investigating for data leakage that may artificially create "perfect features."

A training accuracy of 100% may indicate separation, but it could also be a sign of feature leakage.

4.0.5 Is Not a Universal Threshold

Given a model output $p$, and an action rule that selects a threshold $t$:

$$ \hat y = \mathbf{1}[p \ge t]. $$

If the probability distribution is well-calibrated, costs are stable, and the binary action is simple, the threshold can be derived from the false-positive/false-negative cost trade-off. However, in real systems, additional constraints such as review capacity, group-level limitations, and uncertainty must also be considered.

The threshold must be selected on validation data, test data only evaluates frozen rules. After deployment, shifts in baseline performance will cause the optimal threshold and precision to change.

5. How to Evaluate When Class Imbalance Exists

Having fewer instances in a class does not automatically require resampling. Start by examining:

• log loss / Brier score to assess predicted probabilities;
• precision-recall curves and recall at capacity;
• the numerator and denominator of the confusion matrix;
• how the performance compares to a prevalence-based or business-rule baseline.

class_weight="balanced" Adjusting the training objective to assign higher loss weights to the minority class can help. However, this does not magically increase information content, and the output probabilities may still need recalibration against the original target distribution. Oversampling also alters the training prior.

6. Discrimination and Calibration

• Discrimination: Whether positive examples generally appear before negative ones in the ranking;
• Calibration: Whether samples predicted to have a probability of 0.2 actually have approximately 20% positive outcomes over time;
• Decision utility: Whether a specific threshold or action leads to meaningful outcomes in practice.

A model with a high AUC may exhibit excessive confidence in its probability estimates; conversely, a weak model that is well-calibrated may still fail to provide sufficient discrimination. Report reliability curves, Brier score or log loss, group- or time-based calibration, and compare these metrics against baseline rates.

Calibration methods (such as Platt/logistic or isotonic regression) must be applied to data from an untrained base model or through cross-validation. Calibration should not be performed on the probability estimates derived directly from the trained model.

7. Multiclass Softmax

For mutually exclusive $K$ classes:

$$ P(Y=k\mid x)=\frac{e^{z_k}}{\sum_{j=1}^Ke^{z_j}}. $$

Logits are invariant to a common constant, so parameters must be referenced or constrained to avoid non-identifiability. Numerical stability is achieved using the log-sum-exp trick.

• Multinomial/softmax models train jointly across all classes;
• One-vs-rest trains $K$ binary classifiers, with probabilities that do not naturally sum to 1;
• Multilabel problems should not use mutually exclusive softmax.

8. Missing, Unknown Categories, and Intercept Drift

Category vocabularies, imputation values, and scaling parameters must be integrated into the pipeline. When deployment prevalence shifts (even if the likelihood ratio relationship remains stable) the intercept or calibration can drift. Monitor the following:

• Distribution of scores or logits;
• Prevalence and label delay;
• Calibration-in-the-large (average predicted probability versus actual rate);
• Calibration slope;
• Group- or time-based bias.

9. Interpretation of Coefficients and Boundaries

Odds ratios are often misinterpreted as risk ratios, especially when the results are common, differences become stark in such cases. Nonlinear features, interactions, and regularization make it harder to interpret a single coefficient in isolation.

A conditional association within a predictive model does not imply causation. Decisions to improve calibration or fairness using group-level variables also involve considerations of applicable laws, policies, and trade-offs related to harm.

Common Misconceptions

• Logistic regression requires linearly separable data: Even when data is not linearly separable, the model can still fit well. In fact, perfect separability can cause the unregularized maximum likelihood estimation (MLE) to diverge.
• Sigmoid output is inherently calibrated: Calibration can be lost due to incorrect model specification or improper regularization.
• Class imbalance calls for class weights: Before applying class weights, evaluate decision metrics and the target distribution to determine if imbalance is actually a problem.
• Fixed threshold at 0.5: The threshold should be chosen based on cost, capacity, and validation performance, not assumed a priori.

Exercise

1. Convert a coefficient to an odds ratio and compute probability changes under different baseline probabilities.
2. Create perfect separation and observe the solutions without regularization and with L2 regularization.
3. Compare AUCPR and calibration before and after applying class weighting.
4. Select a threshold under fixed review capacity, then evaluate performance on an independent test set.

Summary

Logistic regression interprets the linear predictor as log-odds and uses a Bernoulli likelihood to model probabilities. The reliability of these probabilities, the appropriateness of the threshold, and the value of the resulting actions must be validated in three distinct layers.

The next lesson returns to the optimization process: even convex objectives can be difficult to optimize due to scale and conditioning issues, and regularization can alter the semantic meaning of the objective and the coefficients.