8.2 Randomized Experiments, ITT, and Interference: Assignment Does Not Equal Exposure

The new supply reminder system is ready for rollout. If historical data is randomly augmented with a new column treatment and then compared, the observed differences between groups have no experimental meaning: random numbers don’t alter anyone’s actual experience. A valid experiment requires random assignment before treatment occurs, and must record how that assignment influences subsequent exposure and outcomes.

Learning Objectives

• Define estimation targets using potential outcomes;
• Randomize units and perform reproducible assignment;
• Distinguish between intention-to-treat and actual treatment received effects;
• Address clustering, interference, missing data, and repeated looks at results.

1. First, Define the Treatment, Outcome, and Estimand

For each unit $i$, define:

• $Y_i(1)$: the potential outcome when assigned the intervention offer;
• $Y_i(0)$: the potential outcome when assigned the control policy.

The individual treatment effect $Y_i(1)-Y_i(0)$ cannot be directly observed, as each unit occupies only one state at a time. Experiments typically estimate the average treatment effect:

$$ATE=E[Y(1)-Y(0)].$$

Here $Z$ denotes assignment and $D$ denotes actual receipt. These potential outcomes are defined by assignment, so the ATE is the average effect of the assigned policy. Interpreting it directly as the effect of receipt requires conditions such as full compliance.

Before randomization begins, clearly specify:

• Treatment version, intensity, and start time;
• Primary outcome, units, and measurement window;
• Target population and analysis unit;
• Whether the primary estimand is the assigned treatment effect or the actual received treatment effect;
• Exclusion, dropout, and outlier treatment rules.

A vague question like "Does the reminder work?" provides no actionable guidance for study design or analysis.

2. What Randomization Provides

Correct randomization makes assignment independent of potential outcomes; blocked randomization gives this independence conditional on the block. Comparisons must follow that design to estimate assignment effects. Potential outcomes include the later outcomes the policy could produce, not just baseline measurements.

Even with a finite sample, covariate imbalance can still occur. Do not interpret the p-values from each balance test as definitive evidence of experiment success or failure; instead, verify the implementation of randomization, report standardized differences, and apply precision adjustments for pre-specified strong predictors.

Randomization does not automatically resolve issues such as measurement error, dropout, crossover, contamination, execution bias, or lack of representativeness of the target population.

3. Select Randomization Units

If colleagues forward reminders, an individual’s outcome can depend on others’ assignments. Consider randomizing teams or sites, while checking for spillover across clusters; a cluster label does not stop a forwarded message.

The effective sample size in cluster-randomized experiments depends primarily on the number of clusters and intra-cluster correlation, rather than the total number of participants alone. When analyzing results, standard errors must be clustered by randomization unit.

Stratified or block randomization can balance key variables such as region or baseline risk across the two groups:

import numpy as np
import pandas as pd

def blocked_assignment(frame, block_col, seed=42):
    if frame.empty or not frame.index.is_unique or frame.index.hasnans:
        raise ValueError("nonempty frame with unique, nonmissing unit IDs required")
    if frame[block_col].isna().any():
        raise ValueError("missing block")
    rng = np.random.default_rng(seed)
    assignments = []
    for _, group in frame.groupby(block_col, sort=True, observed=True):
        ids = group.sort_index().index.to_numpy()
        if len(ids) < 2 or len(ids) % 2:
            raise ValueError("this design requires even block sizes of at least two")
        treated_ids = rng.permutation(ids)[:len(ids) // 2]
        z = pd.Series(0, index=ids, dtype=int)
        z.loc[treated_ids] = 1
        assignments.append(z)
    return pd.concat(assignments).reindex(frame.index)

units = pd.DataFrame(
    {"region": ["east"] * 6 + ["west"] * 6},
    index=pd.Index([f"unit-{i:02d}" for i in range(12)], name="unit_id"),
)
units["assigned_treatment"] = blocked_assignment(units, "region")
print(units.groupby("region")["assigned_treatment"].agg(["sum", "count"]))
# east: sum=3, count=6; west: sum=3, count=6

Production experiments must also preserve the randomization seed, snapshots of candidate units, stratification variables, allocation timestamp, and execution logs. The random assignment code should run before any results are generated.

This example requires even block sizes and allocates half to each arm. Odd blocks need a prespecified probability rule; flooring their treated count would give a singleton no chance of treatment. Sorting unique unit IDs and creating the generator inside the function makes assignments reproducible even if input rows are reordered.

4. Difference in means and regression adjustment

The fundamental estimate in a completely randomized experiment is:

$$\hat{\tau }={\overline{Y}}_{Z=1}-{\overline{Y}}_{Z=0}.$$

Including pre-treatment covariates can improve precision, but variables and models should be specified in advance. Standard errors must align with the randomization design; for clustered randomization, row-wise independent standard errors are not appropriate.

When assignment proportions differ across blocks, estimate effects within blocks and aggregate using target-population block weights. Cluster experiments must likewise distinguish average individual effects from average cluster effects when cluster sizes differ.

Effect sizes, confidence intervals, and original group means should all be reported, rather than p-values alone. For binary outcomes, risk difference, risk ratio, or odds ratio should also be reported depending on the decision context, each has a distinct interpretation.

5. ITT: Analysis by Assignment

Some participants receive alerts but disable notifications, while others in the control group see a peer share. Rerouting participants based on whether they actually saw the alert breaks the randomization.

Intention-to-treat (ITT) compares outcomes based on the original random assignment, regardless of actual compliance. It estimates the effect of a policy that "offers and implements this intervention," and is typically the primary analysis.

Per-protocol or as-treated analyses are more susceptible to confounding by adherence motivation. To estimate the effect among those who actually received the intervention, additional assumptions and methods are required; random assignment can serve as an instrumental variable, but exclusion restrictions, independence, and weak instrument diagnostics must still be satisfied.

The following constructed data have zero treatment effect and illustrate the consequences of regrouping; assigning labels to historical data would not create an experiment. Both assigned arms initially have equal baseline composition. Ten of the 50 assigned-treated units, all from the low-baseline group, do not comply:

baseline = np.repeat([0, 1], 50)
assigned = np.tile([0, 1], 50)
received = assigned.copy()
noncompliers = np.flatnonzero((assigned == 1) & (baseline == 0))[:10]
received[noncompliers] = 0
outcome = baseline.copy()
itt = outcome[assigned == 1].mean() - outcome[assigned == 0].mean()
as_treated = outcome[received == 1].mean() - outcome[received == 0].mean()
print(round(itt, 6), round(as_treated, 6))
# 0.0 0.208333

The outcome equals baseline, giving ITT zero. The as-treated difference is 20.83 percentage points, entirely from the changed group composition.

6. Interference and Spillover

Writing only $Y_i(1)$ and $Y_i(0)$ assumes no interference and well-defined treatment versions. With interference, outcomes can depend on the group’s assignment vector, requiring definitions of direct, indirect, or coverage effects. Hudgens and Halloran develop these estimands under two-stage randomization.

Optional design strategies include:

• Randomization by cluster, grouping units that may influence one another;
• Two-stage randomization, separately estimating direct effects and coverage effects;
• Establishing geographic or network buffers;
• Clearly defining treatment versions and measuring actual exposure.

When spillover is present, the "treatment effect" must specify whether it refers to direct, indirect, total, or saturation effects.

7. Lost to Follow-Up and Missing Outcomes

If interventions make unsatisfied participants more likely to drop out, complete case comparisons can introduce bias. In experiments, it is essential to:

• Report the dropout rates and reasons for loss to follow-up in both groups;
• Obtain primary outcomes from independent sources whenever possible;
• Retain randomly assigned units in the intention-to-treat (ITT) analysis;
• Apply weighting or imputation under a plausible missing data assumption;
• Provide bounds or conduct sensitivity analyses for unobserved differences.

A similar dropout rate between groups does not imply that the dropout mechanisms are the same.

Retaining assignment records does not recover missing outcomes. Calling a complete-case calculation ITT does not make it a full ITT estimate; state the missingness assumptions, estimation method, and remaining identification limits.

8. Power, MDE, and Stopping Rules

Choose the smallest practically meaningful effect from the decision context, then plan sample size using the baseline rate, variance, significance level, power, allocation ratio, and clustering. MDE is the smallest effect detectable at the target power under that design; it is distinct from the smallest effect worth acting on.

With a fixed budget, calculate the MDE to assess whether the design is sensitive enough. Repeatedly checking ordinary p-values and stopping at significance inflates false positives; prespecify sample size or use a valid sequential design with alpha spending.

9. Experiment Analysis Plan

Research questions and estimand
Experimental units, randomization units, and strata
Treatment/control versions and execution checks
Primary and secondary outcomes, observation window
Sample size, minimal detectable effect, power, and stopping rules
ITT main analysis and standard error methods
Non-adherence, dropout, interference, and outlier rules
Multiple comparison correction
Heterogeneity analysis and pre-registration status

Common Misconceptions

• Adding random grouping columns to historical tables makes it an experiment: Randomization must precede intervention and outcome measurement.
• Randomization guarantees complete balance in the current sample: It controls assignment probabilities, but does not ensure exact covariate balance in each realization.
• Analyzing by actual treatment receipt is more realistic: Regrouping can introduce selection bias.
• Having many individuals with few clusters still yields high statistical power: Within-cluster correlation significantly reduces the effective sample size.

Exercise

1. Define the treatment, primary outcome, window, and ITT estimand for a notification intervention.
2. Compare the contamination risks and required standard errors between individual randomization and group randomization.
3. Construct 20% non-compliance data and compare ITT results with as-treated outcomes.
4. Write a valid stopping rule for an experiment with daily monitoring metrics.

Summary

The power of randomized experiments comes from pre-assignment, not from a column label in a data table. The ability to interpret results as causal effects depends on the interplay of key elements: the target of estimation, the unit of randomization, compliance, contamination, and stopping rules.

The next lesson explores identification strategies when randomization is not possible: DAGs are not decorative diagrams, and matching or instrumental variables do not automatically eliminate confounding.