8.2 Random Experiments, ITT, and Contamination: Assignment Does Not Equal Exposure

Simply comparing pre- and post-implementation reports doesn't prove whether a new supply reminder system improves delivery. To assess true impact, administrators must first design the assignment and observation process.

The new supply reminder system is ready for rollout. If historical data is randomly augmented with a new column treatment and then compared, the observed differences between groups have no experimental meaning: random numbers don’t alter anyone’s actual experience. A valid experiment requires random assignment before treatment occurs, and must record how that assignment influences subsequent exposure and outcomes.

Learning Objectives

• Define estimation targets using potential outcomes;
• Randomize units and perform reproducible assignment;
• Distinguish between intention-to-treat and actual treatment received effects;
• Address clustering, interference, missing data, and multiple outcome measurements.

1. First, Define the Treatment, Outcome, and Estimand

For each unit $i$, define:

• $Y_i(1)$: the potential outcome under treatment;
• $Y_i(0)$: the potential outcome under no treatment.

The individual treatment effect $Y_i(1) - Y_i(0)$ cannot be directly observed, as each unit occupies only one state at a time. Experiments typically estimate the average treatment effect:

$$ ATE = E[Y(1) - Y(0)]. $$

Before randomization begins, clearly specify:

• Treatment version, intensity, and start time;
• Primary outcome, units, and measurement window;
• Target population and analysis unit;
• Whether the primary estimand is the assigned treatment effect or the actual received treatment effect;
• Exclusion, dropout, and outlier treatment rules.

A vague question like "Does the reminder work?" provides no actionable guidance for study design or analysis.

2. What Randomization Provides

If the assignment mechanism is properly implemented, the assignment of treatment becomes independent of potential outcomes prior to treatment. In this way, the treatment and control groups are comparable across repeated randomizations, allowing the difference in group outcomes to estimate the causal effect of treatment assignment.

Even with a finite sample, covariate imbalance can still occur. Do not interpret the p-values from each balance test as definitive evidence of experiment success or failure; instead, verify the implementation of randomization, report standardized differences, and apply precision adjustments for pre-specified strong predictors.

Randomization does not automatically resolve issues such as measurement error, dropout, crossover, contamination, execution bias, or lack of representativeness of the target population.

3. Select Randomization Units

If team members share reminders, individual randomness can affect one another, so randomization should be done at the team or fortress level. The randomization unit must align with both the interference boundary and the delivery method.

The effective sample size in cluster-randomized experiments depends primarily on the number of clusters and intra-cluster correlation, rather than the total number of participants alone. When analyzing results, standard errors must be clustered by randomization unit.

Stratified or block randomization can balance key variables such as region or baseline risk across the two groups:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)

def blocked_assignment(frame, block_col):
    assignments = []
    for _, block in frame.groupby(block_col, sort=True):
        ids = block.index.to_numpy().copy()
        rng.shuffle(ids)
        z = np.zeros(len(ids), dtype=int)
        z[: len(ids) // 2] = 1
        rng.shuffle(z)
        assignments.append(pd.Series(z, index=ids))
    return pd.concat(assignments).sort_index()

units["assigned_treatment"] = blocked_assignment(units, "region")

Production experiments must also preserve the randomization seed, snapshots of candidate units, stratification variables, allocation timestamp, and execution logs. The random assignment code should run before any results are generated.

4. Difference in means and regression adjustment

The fundamental estimate in a completely randomized experiment is:

$$ \hat\tau = \bar Y_{Z=1} - \bar Y_{Z=0}. $$

Including pre-treatment covariates can improve precision, but variables and models should be specified in advance. Standard errors must align with the randomization design; for clustered randomization, row-wise independent standard errors are not appropriate.

Effect sizes, confidence intervals, and original group means should all be reported, rather than p-values alone. For binary outcomes, risk difference, risk ratio, or odds ratio should also be reported depending on the decision context, each has a distinct interpretation.

5. ITT: Analysis by Assignment

Some participants receive alerts but disable notifications, while others in the control group see a peer share. Rerouting participants based on whether they actually saw the alert breaks the randomization.

Intention-to-treat (ITT) compares outcomes based on the original random assignment, regardless of actual compliance. It estimates the effect of a policy that "offers and implements this intervention," and is typically the primary analysis.

Per-protocol or as-treated analyses are more susceptible to confounding by adherence motivation. To estimate the effect among those who actually received the intervention, additional assumptions and methods are required; random assignment can serve as an instrumental variable, but exclusion restrictions, independence, and weak instrument diagnostics must still be satisfied.

6. Spillover and Contamination

The standard potential outcomes notation implicitly assumes a key condition: an individual's outcome is unaffected by others' assignments, and the treatment has no hidden versions. In practice, however, spillover effects may propagate within a group, and shared resources may be depleted across teams.

Optional design strategies include:

• Randomization by cluster, grouping units that may influence one another;
• Two-stage randomization, separately estimating direct effects and coverage effects;
• Establishing geographic or network buffers;
• Clearly defining treatment versions and measuring actual exposure.

When spillover is present, the "treatment effect" must specify whether it refers to direct, indirect, total, or saturation effects.

7. Lost to Follow-Up and Missing Outcomes

If interventions make unsatisfied participants more likely to drop out, complete case comparisons can introduce bias. In experiments, it is essential to:

• Report the dropout rates and reasons for loss to follow-up in both groups;
• Obtain primary outcomes from independent sources whenever possible;
• Retain randomly assigned units in the intention-to-treat (ITT) analysis;
• Apply weighting or imputation under a plausible missing data assumption;
• Provide bounds or conduct sensitivity analyses for unobserved differences.

A similar dropout rate between groups does not imply that the dropout mechanisms are the same.

8. Efficacy, MDE, and Stopping Rules

Sample size planning requires: baseline rate, minimum detectable effect (MDE), variance, significance level, target power, allocation ratio, and adjustments for clustering and multiple comparisons.

First, determine the MDE based on decision value, then assess whether the study can detect it, rather than reverse-engineering a convenient target from an existing sample size. Repeatedly checking p-values during experiment execution and stopping upon significance increases the false positive rate. Instead, use a pre-specified sample size, or implement an effective sequential design with alpha spending.

9. Experiment Analysis Plan

Research questions and estimand
Experimental units, randomization units, and strata
Treatment/control versions and execution checks
Primary and secondary outcomes, observation window
Sample size, minimal detectable effect, power, and stopping rules
ITT main analysis and standard error methods
Non-adherence, dropout, interference, and outlier rules
Multiple comparison correction
Heterogeneity analysis and pre-registration status

Common Misconceptions

• Adding random grouping columns to historical tables makes it an experiment: Randomization must precede intervention and outcome measurement.
• Randomization guarantees complete balance in the current sample: It ensures the allocation mechanism is fair, but does not guarantee equal group sizes in any given run.
• Analyzing based on actual treatment assignment is more realistic: Re-randomization may introduce selection bias.
• Having many individuals with few clusters still yields high statistical power: Within-cluster correlation significantly reduces the effective sample size.

Exercise

1. Define the analysis of treatment, primary outcome, window, and ITT estimand for a supply reminder intervention.
2. Compare the contamination risks and required standard errors between individual randomization and group randomization.
3. Construct 20% non-compliance data and compare ITT results with as-treated outcomes.
4. Write a valid stopping rule for an experiment with daily monitoring metrics.

Summary

The power of randomized experiments comes from pre-assignment, not from a column label in a data table. The ability to interpret results as causal effects depends on the interplay of key elements: the target of estimation, the unit of randomization, compliance, contamination, and stopping rules.

The next lesson explores identification strategies when randomization is not possible: DAGs are not decorative diagrams, and matching or instrumental variables do not automatically eliminate confounding.