4.1 From Business Decisions to Supervised Learning Tasks: Labels Are Just Measurements of the Target

Supervised learning starts with a task: at what time, using which inputs X, do you predict which label Y and take which action? Choosing the wrong target makes a high score evidence of success on the wrong proxy. The table has been trained; the problem may still need treatment.

This lesson's objectives

• Define observation unit, prediction time, features, label, and decision;
• Distinguish the target construct, observed indicators, and proxy variables;
• Connect loss, offline metrics, and actual decision costs;
• Understand what a model learns from samples using the principle of minimizing empirical risk.

1. First, write the prediction contract

This hypothetical prediction contract is a structural example. It records available information alongside the resulting action:

Decision: Should an additional supply review be scheduled before departure?
Unit of prediction: one planned task
Prediction timestamp: departure_time - 30min
Inputs: task, team, and inventory information available at this time
Objective: Was an emergency resupply request made within 24 hours of departure?
Output: calibrated probability, with approval or rejection handled separately
Action: Enter manual review when exceeding the validated threshold
Error costs: missed risks versus unnecessary reviews
Target population: eligible regions, task types, and time periods

A prediction time lets you check leakage. An action determines useful thresholds and metrics. A target population tells you what generalization is meant to cover.

2. Labels Are Not Inherent Truths

The target construct might be "actual need for urgent resupply," while the observed label is "whether a request was submitted." These are different because:

• Some teams need supplies but cannot send a request because communication fails, producing false-negative labels;
• Some requests are for preventive purposes and do not indicate a shortage;
• Cultural and approval processes vary by region;
• After the model goes live, additional reviews will change whether a request is made.

Record label source, generation rules, observation window, delay, missing data, and version. Review a sample for agreement between labelers or compare it with a more authoritative source. When using historical decisions as labels, also ask whether those decisions involved selection bias or historical bias.

3. Features must be available at prediction time

$X$ is not "all columns of the training table except the label." Every feature must be available at prediction time and must be correct as of a specific point in time. Event time, availability time, and cross-fitting are discussed in Chapter 7 of Volume 12; here, the feature pipeline is considered part of the model.

IDs, post-processing states, handling outcomes, and full window aggregations are often sources of leakage. Even if they don't directly contain a label, highly correlated proxies can be generated by the result pipeline.

4. The output format determines the subsequent approach

• regression: predicting continuous conditional quantities/distributions;
• binary/multiclass classification: mutually exclusive categories;
• multilabel: multiple labels can all be true at the same time;
• ranking: focuses on relative order;
• survival/time-to-event: handles event times and censoring;
• structured prediction: sequence, graph, or combined output.

Forcing "how long until" into a 7-day binary classification loses both censoring and time information; forcing nonexclusive labels into a multiclass setup introduces an incorrect exclusivity constraint.

5. Loss, Metric, and Utility

Training loss is a proxy handled by the optimizer, such as squared error or log loss. Offline metrics are used to compare models, such as MAE, AUCPR, or calibration error. Business utility measures the benefits and harms of taking action.

They need not be the same:

Training: minimize log loss
Verification: calibration plus recall at review capacity
Decision: reduce costly missed risks under a fixed review capacity

For a small decision example, let each review cost 1 and each missed real risk cost 20. Assume review prevents the entire loss, probabilities are calibrated, and capacity is unlimited. Review when 20p>1, giving a threshold of 0.05. Imperfect reviews or a capacity limit require a different calculation; 0.05 should not become a team tradition.

Accuracy treats all samples and errors equally; when false positives or false negatives have different costs or classes are imbalanced, it may be irrelevant. You can't just pick one metric that looks good for the model, you must define a primary metric and constraints before training.

6. Minimizing Empirical Risk

Given a training sample $(x_i,y_i)$ and a hypothesis class $\mathcal{H}$:

$$\hat{f}=\arg \underset{f\in \mathcal{H}}{min}\frac{1}{n}\sum _{i=1}^n\ell (f(x_i),y_i).$$

This is empirical risk. What we care about is the expected risk under the target distribution:

$$R(f)=E_{(X,Y)\sim P_{target}}[\ell (f(X),Y)].$$

Training samples can only estimate target risk if sampling, timing, and deployment mechanisms are sufficiently aligned. The algorithm also has inductive biases, such as linearity, smoothness, sparsity, local similarity, or architectural invariance. Without bias, finite samples cannot uniquely determine predictions for unseen inputs.

7. Baseline Before Complex Models

At least compare:

• constant/majority/mean baseline;
• Simple business rules;
• Current production process;
• An interpretable model using a few strong features;
• Candidate complex models.

Baseline uses the same splits, available features, and metrics. If a new model is only better than random but worse than the current rules, it has no deployment value. If improvement comes only from post-hoc fields, it's not an algorithmic advancement.

8. Data Generation Process

Draw:

Population → Who is observed → Who receives a label → How they enter the training set
→ Model output → Action → Next round of data and label

Common selection patterns include violation labels only for inspected cases, success outcomes only for completed tasks, and payoff observations only for cases approved by the historical policy. The training distribution reflects these processes rather than the population automatically.

9. Task Definition Review

[ ] Decision, entity, prediction time, and horizon clearly defined
[ ] The gap between the label and the target construct has been assessed
[ ] Features are available at prediction site
[ ] The relationship between loss, metric, and utility is clearly defined
[ ] The target population and sampling mechanism are documented
[ ] The baseline and the no-model approach have been compared.
[ ] Feedback loops, appeals, and termination conditions have been considered

Common Misconceptions

• Supervised learning doesn't require hand-written rules: the task, labels, loss function, and actions are still defined by humans.
• Database fields are just objective labels: They might be proxies generated by a process.
• The model's output categories are the decisions: thresholds, capacity, and manual processes still need to be designed.
• High test accuracy has value: It must exceed the true baseline and improve decision utility.

Practice

1. Rewrite "predict task risk" as a complete prediction contract.
2. List three measurement/selection biases for historical approval labels.
3. Design two different actions for the same probability output under different error costs.
4. Draw how the model deployment changes the next round of labels.

Summary

Machine learning first learns the task you define, then learns patterns in the data. To determine whether an offline score corresponds to a real-world problem, you must clearly define the prediction context, the source of labels, and the cost of decisions.

The next lesson divides the sample into training, selection, and final evaluation responsibilities and blocks leakage by time, entity, and deployment goals.