13.2 Fairness, Explainability, and Appealability: Metrics Can't Decide What Harm Means

Before deploying the model to the Northern Outpost, low-scoring groups raised concerns: Why does the system make these judgments? How are erroneous records corrected? Who handles appeals?

The Northern Outpost's poor performance sparked controversy. After analysis, the team found that overall accuracy across regions was similar, yet the outpost had a significantly higher rate of missed detections. Another diagnostic tool pointed to "device uptime as the top contributor," but this didn't prove that devices caused poor performance, nor did it provide a clear path for the outpost to correct its missed records.

Fairness, explainability, and appealability address distinct challenges: fairness compares outcomes against the distribution of harm; explainability clarifies how a model or system arrives at a given output; and an appeal mechanism enables affected individuals to challenge the data, reasoning, and decisions behind a result.

Learning Objectives

• Accurately compute group-level confusion metrics and selection rates;
• Choose a fairness criterion based on harm, rather than aiming for a comprehensive set of metrics;
• Handle sample size, cross-group comparisons, thresholds, and uncertainty;
• Distinguish between feature attribution, causal explanation, and recourse;
• Integrate notifications, corrections, and appeals into system workflows.

1. "Don't Use Sensitive Attributes as Features" Doesn't Mean Fairness

Postal codes, schools, occupations, languages, and device types can serve as proxies for demographic groups. Historical labeling may also reflect past inequities. Removing explicit sensitive fields can leave teams unable to group and evaluate performance.

First, distinguish between:

• Representation harm: stereotypes, insults, stigma, or invisibility;
• Allocation harm: unfair distribution of opportunities or resources like jobs, loans, supplies, or healthcare;
• Quality-of-service harm: certain language groups, skin tones, device types, or disability groups experience more errors or poor service;
• Surveillance/privacy harm: some groups face greater monitoring and inference;
• Feedback harm: model outputs alter opportunities, and future data may revalidate old biases.

Each type of harm has distinct metrics, stakeholders, and mitigation strategies.

2. First, Get the Denominator Right

In binary classification:

$$ TPR=\frac{TP}{TP+FN},\qquad FPR=\frac{FP}{FP+TN},\qquad PPV=\frac{TP}{TP+FP}. $$

Dividing FP by the total population gives the proportion of false positives in the group, not the FPR. A minimal working example:

from dataclasses import dataclass
from math import nan

@dataclass(frozen=True)
class GroupMetrics:
    n: int
    selection_rate: float
    tpr: float
    fpr: float
    ppv: float

def safe_ratio(num: int, den: int) -> float:
    return num / den if den else nan

def binary_group_metrics(y_true, y_pred) -> GroupMetrics:
    pairs = list(zip(y_true, y_pred))
    tp = sum(y == 1 and p == 1 for y, p in pairs)
    fp = sum(y == 0 and p == 1 for y, p in pairs)
    tn = sum(y == 0 and p == 0 for y, p in pairs)
    fn = sum(y == 1 and p == 0 for y, p in pairs)
    n = len(pairs)
    return GroupMetrics(
        n=n,
        selection_rate=safe_ratio(tp + fp, n),
        tpr=safe_ratio(tp, tp + fn),
        fpr=safe_ratio(fp, fp + tn),
        ppv=safe_ratio(tp, tp + fp),
    )

Beyond the point estimate, confidence intervals or bootstrap distributions should also be reported. In small samples, a single error can lead to large fluctuations; nan should not be assumed to be zero by default.

3. Fairness Defined by Harm Models

• Demographic parity: Predictive rates across groups are similar. Suitable for scenarios focused on equitable distribution of outcomes, but may overlook actual needs or label quality.
• Equal opportunity: True positive rates (TPR) across groups are similar. Appropriate when missing qualified individuals is a primary source of harm.
• Equalized odds: Both true positive and false positive rates (TPR and FPR) are similar across groups. Balances concerns about both missed positives and false positives.
• Predictive parity: Positive predictive values (PPV) across groups are similar. Focuses on the proportion of individuals flagged as positive who actually are positive.
• Calibration within groups: For the same predicted score, the actual positive rates are similar across groups. Ensures consistency in how predictions map to real-world outcomes.

When group base rates differ and the model is imperfect, some calibration or predictive parity metrics often cannot be satisfied simultaneously with error-rate parity. This conclusion is conditional, not a universal claim that all fairness metrics are mutually incompatible. The choice should be driven by specific harms, rights at stake, business processes, and applicable laws, and must be clearly documented with identified trade-offs.

4. Thresholds Are Not Pure Technical Parameters

Using the same score threshold across different groups may result in varying error rates. Applying different thresholds can lead to disparate treatment and legal concerns. Engineering teams cannot simply rely on performance curves to determine thresholds.

A threshold review must clearly address:

• Who is harmed by false positives and false negatives;
• Whether there is a follow-up review or correction process in place;
• Whether the score is calibrated and the labels are reliable;
• Whether capacity constraints have turned the model into a resource allocation tool;
• Whether group-specific thresholds are both permissible and interpretable;
• How changes in thresholds affect the overall system and individual slices.

"A false positive rate disparity exceeding threefold" is not a universal hard standard. Acceptable variation and testing methodologies should stem from applicable rules and pre-defined risk thresholds.

5. Cross-group Effects and Measurement Error

Looking at averages across single dimensions (such as gender or region) can obscure intersectional harms, like "Northern region + outdated equipment + night shift." However, continuous segmentation introduces problems such as small sample sizes and multiple comparisons.

Use pre-registered key slices, hierarchical or confidence intervals, minimum sample disclosure, and qualitative review. For rare but severe harms, even if we cannot reliably estimate the overall incidence rate, we must not conclude that such harms do not exist.

Group attributes themselves may be missing, inferred from proxies, or not accepted by the individuals being studied. Document the rationale for data collection, its intended use, access restrictions, and the presence of unknown or multiple categories; avoid collecting sensitive data indefinitely in pursuit of perceived fairness.

6. Feature Attribution Is Not Causal Explanation

Linear coefficients, tree paths, permutation importance, SHAP, and LIME answer different questions. SHAP and LIME are typically used to describe how a model responds to changes in features around a given representation; their results depend on background data, feature dependencies, perturbation methods, and the scale of model outputs.

A SHAP value of "-0.3" for "device uptime" indicates that, under specific explanation conditions, the model output shifts in a particular direction, it does not prove that:

• Device uptime causes a decline in actual performance;
• Changing device uptime necessarily alters the final decision;
• The model did not use relevant proxy features;
• The entire system is fair or correct;
• This is a sufficient reason for users to act.

Relevant features distribute attribution, and local explanations can become unstable when inputs change slightly. Perform fidelity, stability, and sensitivity tests on explanation methods to validate their reliability.

7. Internal Explanability Does Not Equal Ease of Understanding

The computational structure of linear models is transparent, yet even with thousands of interactions, normalizations, missing-value encodings, and surrogate features, the model remains difficult to interpret. Deep models can also achieve partial transparency through carefully designed constrained tasks, verifiable intermediate outputs, and evidence traces.

The kind of explanation required depends on the stakeholder:

• Developers: feature dependencies, error slicing, gradients/attribution;
• Reviewers: evidence, thresholds, rules, and uncertainty;
• Affected individuals: which data was used, the reasoning behind decisions, how to correct or appeal;
• Auditors: version history, data lineage, control evidence, and accountability chains.

A visually appealing SHAP plot cannot simultaneously satisfy the needs of all four groups.

8. Recourse Must Be Actionable and Not Shift Responsibility

A counterfactual explanation might say: "If income increased by 20%, the loan would have been approved." This doesn't necessarily mean it's actionable, fair, or causally valid. A good recourse should:

• Avoid suggesting changes to immutable attributes like age or ethnicity;
• Not require unrealistic or harmful costs;
• Use fields that are permissible for modification within the business context;
• Account for field dependencies and future stability;
• Not expose risk mitigation rules that could be misused;
• Not turn systemic data errors into personal obligations for self-correction.

When the original record contains an error, the first step should be correcting the data and reevaluating the decision, not asking the user to change their behavior.

9. Accountability Requires Real Operational Permissions

Notifications must clearly state the extent of AI involvement, the purpose of decisions, the primary data categories used, contact points for users, and the timeframe for review. The appeals process must enable:

1. Secure identity verification;
2. Access to relevant data and the rationale behind decisions;
3. Correction of errors or supplementation of context;
4. Review by a human with appropriate authority and no blind reliance on model outputs;
5. Modification of downstream outcomes and notification to affected systems;
6. Feedback routing back to improvements in data, models, and policies.

Beyond tracking appeal rates, it’s essential to monitor accessibility, response times, reversal rates, group disparities, and recurring reasons for appeals. A low appeal rate may indicate a well-functioning system, or it may signal that appeal pathways are invisible or prohibitively costly.

Common Misconceptions

• Not inputting sensitive attributes doesn’t mean no discrimination: The proxy features and labeling mechanisms can still propagate bias.
• Same overall accuracy means fairness: The types of errors and their real-world consequences may differ significantly.
• All fairness metrics must be satisfied: Metrics should be selected and recorded based on their impact, recognizing that some are mutually exclusive.
• SHAP provides "why" explanations: It typically offers model attribution, not causal or justification-based proof.
• Human review can always correct errors: Without information, access, or time for proper validation, oversight remains superficial.

Exercise

1. Fix a code snippet that incorrectly divides false positives by total population.
2. Select a primary fairness metric for supply allocation and explain why the other two were not chosen.
3. Design an intersectional slice table that includes confidence intervals for small sample sizes.
4. List four key differences between SHAP attributions and causal explanations.
5. Write a complete appeal and downstream correction process for a low score resulting from a data omission.

Summary

Fairness assessments map specific harms to group metrics and slices, but they do not provide a single, context-independent answer. Explanatory methods reveal local behaviors of models or systems, yet they cannot prove causality, fairness, or legitimacy; only notifications, corrections, human review, and appeals provide affected individuals with actual pathways to redress.

The next lesson shifts focus to active adversaries. Data, models, RAG documents, and toolchains can all serve as entry points, stronger system prompts are insufficient to defend against such threats.