10.1 From Use Case to Harm Path: Data Projects Must First Prove They Are Worth Doing

Performance models consistently rate the northern site as underperforming. Upon investigation, it's revealed that the site's legacy communication devices frequently fail to report task completion events. The model didn't detect "poor performance", it simply translated infrastructure differences into resource allocation recommendations.

Data ethics isn't a post-hoc check after a model is built. From the very first stage of project planning, we must ask: Is this problem worth automating? Why was this data collected? Who benefits? Who bears the risk of error?

Learning Objectives

• Rewrite abstract “ethical risk” as a concrete path to harm;
• Review data sources, intended use cases, and how scenarios may evolve over time;
• Distinguish between representativeness, measurement quality, and historical injustice;
• Establish necessity, proportionality, and exit conditions for high-impact applications.

1. Define Decisions Before Finding Data

The risk associated with a prediction varies significantly depending on its intended use. For example, the statement "the team may miss upcoming shifts" could be used to proactively offer support or to disqualify an employee from promotion.

A project justification must include at least the following:

Who uses the output to make what decision
Whether the model provides recommendations, rankings, thresholds, or automated actions
The people and groups affected by the decision
The consequences of false positives, false negatives, and not using the system at all
How expected value is measured and who benefits from it
Existing non-model approaches and manual workflows
Prohibited uses and conditions under which the system should be shut down

If it's not clear how the output will change decisions, then appropriate metrics cannot be selected, and it's impossible to determine whether data collection is necessary.

For the same prediction of a high risk of missing next week’s shifts, a support use could offer voluntary scheduling help, while a punitive use could reduce promotion opportunities. The first asks whether people needing help receive it; the second must first establish valid labels and decision grounds. Equal model AUC does not make the two uses interchangeable for approval.

2. Replace Risk Lists with Harm Paths

Risks can be expressed as causal chains:

Outdated equipment fails to report data
→ Northern region records incomplete
→ Model underestimates performance
→ Reviewers trust a uniform score
→ Northern region receives fewer supplies and promotion opportunities
→ Underreporting and poor performance worsen

For each path, annotate: affected parties, severity, likelihood of occurrence, reversibility, duration, and existing controls. Then place control points along the chain, such as fixing data collection, displaying data completeness, blocking automatic resource deductions, and enabling review and appeal processes.

A risk matrix can only help with prioritization. It cannot convert intangible values like dignity, rights, or collective harm into a "tolerable score."

3. Data Sources Determine How They Can Be Interpreted

For each data source, record:

• Who collected it, why, and in what relational context;
• Reasonable expectations of use by the parties involved;
• Scope of collection, device variability, and methods of withdrawal;
• Field definitions, temporal ranges, and retention policies;
• Authorization requirements, contractual terms, licensing conditions, and geographic constraints;
• Whether the data has undergone filtering, review, annotation, or algorithmic processing.

"Publicly accessible" does not equate to suitability for arbitrary reuse. Ethical review and legal compliance are related but not identical: meeting minimum legal standards does not necessarily mean a use is appropriate, especially if it deviates from intended purposes, creates power imbalances, or poses significant harm.

4. Consent Is Not a Universal Pass

Effective consent requires that information be understandable, choices be genuinely free, the cost of refusal be low, and revocation be feasible. In high-power relationships (such as employment, education, or healthcare) a signed consent form may not reflect true autonomy.

You must also adhere to purpose limitation and data minimization:

• Is this field necessary for the defined purpose?
• Can the task be accomplished with broader, less granular data or a shorter retention period?
• Has use expanded without a new review?
• Can model outputs infer sensitive information that was never originally collected?

Deleting explicit names does not automatically eliminate ethical or privacy risks. Quasi-identifiers, rare combinations, behavioral trajectories, and external data sources may still allow individuals or small groups to be identified.

5. Representativeness Is Not "Equal Proportion"

Even when a group’s sample share matches its population share, measurement can still differ unfairly:

• Older equipment leads to higher missing data rates;
• The same label is generated by different institutions using inconsistent standards;
• Only individuals who are inspected have records of violations;
• Historical decisions determine who is eligible to receive a positive label;
• The demographic composition and behaviors of users change after system deployment.

Therefore, it is essential to simultaneously examine coverage, measurement bias, label generation processes, and historical decision-making. Simply increasing the number of rows for underrepresented groups does not correct a systemic bias in labeling.

Consider a checkable example: both sites complete 100 tasks, with an independent reconciliation source recording 90 successes each. New devices upload all 100 outcomes; old devices omit 30 successes, leaving 60 successes and ten failures. Success among uploaded records is 90% versus about 85.7%; treating missing outcomes as failures instead gives 60%. Repair collection and denominator semantics before adjusting a threshold to improve a regional score.

6. Stakeholders Go Beyond Interview Subjects

Engineering teams, domain experts, operators, impacted users, and complaint handlers perceive failures differently. Engagement must occur during problem definition, metric selection, pilot phases, and post-incident reviews, rather than at release time to gain approval for a pre-determined plan alone.

Disagreements must be documented: who considers certain harms unacceptable, which voices were excluded, and who ultimately bears responsibility. Participation does not mean shifting accountability onto affected parties.

7. Go / No-Go Review

Answer before development begins or when scope changes:

[ ] Purpose, decisions, and affected stakeholders are clearly defined
[ ] Alternatives to building the system have been evaluated and compared
[ ] Data sources align with reasonable expectations
[ ] Critical failure paths have been identified, controlled, and assigned owners
[ ] High-severity failures have manual mitigation and recovery procedures
[ ] The system can explain, correct, and provide remedies to affected parties
[ ] If key failure modes cannot be monitored, the use case is narrowed or eliminated

An unmeasured risk is not a zero risk; it may be a valid reason to pause automation.

A usable stop condition connects evidence to action. For example, if completion records cannot be reconciled to a reliable source, pause score-driven resource cuts while retaining supportive human review. Resume only after checking backfilled records, affected decisions, and revised evaluation. This is an illustrative project rule, not a universal legal threshold. NIST AI RMF 1.0 addresses context, affected parties, and risk throughout the lifecycle.

Common Misconceptions

• "Models only provide suggestions, so the risk is low": Automatic sorting and default recommendations can strongly influence human decisions, shaping outcomes in ways that are not always transparent or controllable.
• "Public data can be freely combined": Combining datasets can fundamentally alter their usage, identifiability, and downstream impact, sometimes in ways that were not anticipated.
• "Balanced sample proportions ensure fairness": Even with equal representation, measurement and labeling processes may still introduce systematic biases.
• "Ethics is only reviewed by compliance teams at the end": Path dependencies are already established during the stages of problem definition and data selection, meaning ethical risks are embedded from the outset.

Exercise

1. List two use cases for a performance model and compare their error costs.
2. Map a path from underreporting by older devices to reduced maintenance, and place three controls along it.
3. Complete a source attribution for a public data set, including origin, intended use, and reusability gap.
4. Design a clear no-go condition instead of merely assigning risk scores.

Summary

Responsible data projects begin with the question: "Should we do this, and for whom?" Only after clearly defining the purpose, data sources, and potential harm paths can technical evaluation be grounded. Without such clarity, high accuracy merely represents an optimized fit for a vague or even harmful objective.

The next lesson explores fairness metrics: group differences must be broken down into decision semantics, baseline rates, error types, and uncertainty. A single accuracy metric or arbitrary threshold is insufficient to assess fairness.