8.3 DAGs and Observational Identification: Control Variables Can Also Introduce Bias

The core of observational causal inference isn't any single algorithm; it's: which assumptions make the data comparisons equivalent to the target counterfactual comparison?

Learning Objectives

• Use DAGs to distinguish confounders, mediators, and colliders;
• State the assumptions of consistency, exchangeability, and positivity;
• Understand when regression, matching, and weighting work and when they don’t;
• Review instrumental variables, difference-in-differences, and regression discontinuity identification conditions.

1. Turn an Association Question into a Causal Question

First, write the basic elements of the target trial:

• The population of interest;
• Treatment strategies and time zero;
• Control strategy;
• Outcomes and follow-up window;
• Allocation mechanism;
• estimand and analysis plan

For example, "teams with more training have higher success rates" is just an association; only "the average impact on future 30-day task success rate of increasing training from 10 to 20 hours for eligible teams" approaches an identifiable cause-and-effect question.

Align eligibility, treatment assignment, and the start of follow-up. Defining baseline treatment by future training completion, then crediting the treated group with time during which units had to remain alive or event-free to qualify, can create immortal time bias. Other date misalignments can cause different biases; they are not all immortal time bias. See Hernán and colleagues on target trials.

2. A DAG expresses assumptions

Consider:

Experience → Training Duration → Collaboration Proficiency → Success Rate
└────────────────────────────→ Success rate

Training duration ──→ Injury ──← Task difficulty ──→ Success rate

• Experience simultaneously affects both treatment and outcome, acting as a confounding variable;
• Collaboration proficiency lies on the path from treatment to outcome and is a mediator;
• Injury is jointly caused by training duration and task difficulty and is a collider.

When estimating the total effect of training, you should block backdoor paths created by experience, but typically do not control for collaborative proficiency; in this diagram, controlling for injury will open training duration → injury ← task difficulty → success rate path.

The arrows in a DAG come from domain knowledge, temporal order, and mechanistic evidence, not from automatically generating truth based on correlation matrices. Nodes for omitted variables should be included even in the absence of data, because they determine whether identification is possible.

Run the NumPy blocks in order. This example uses centered abstract variables to demonstrate collider bias. Let $A$ and $U$ be independent, $C=A+U$, and $Y=U$. The graph is $A\to C\leftarrow U\to Y$, with no causal effect of $A$ on $Y$:

import numpy as np

A = np.array([-1., -1., 1., 1.])
U = np.array([-1., 1., -1., 1.])
C = A + U
Y = U.copy()
plain = np.linalg.lstsq(np.column_stack([np.ones(4), A]), Y, rcond=None)[0]
adjusted = np.linalg.lstsq(np.column_stack([np.ones(4), A, C]), Y, rcond=None)[0]
print(round(plain[1], 6), round(adjusted[1], 6))
# 0.0 -1.0

Without $C$, the regression coefficient for $A$ is zero. After adding $C$, the identity $Y=C-A$ makes it −1. The extra control creates an association where the construction has no causal effect.

3. Three Basic Identification Criteria

Consistency

The observed outcome for units assigned to treatment $A=a$ equals their corresponding potential outcome $Y(a)$, which requires that "20 hours of training" not be multiple versions with different effects lumped together.

Conditional Exchangeability

Given sufficient pretreatment confounders $L$:

$$Y(a)\perp A\mid L.$$

It means there is no remaining unmeasured confounding, an assumption typically unverifiable from the observed data itself.

Positivity

For binary treatment $A\in \{0,1\}$, both treatment states must have positive probability at every relevant covariate combination in the target population:

$$0<P(A=a\mid L=l)<1.$$

If the most dangerous task never accepts short training, we can't compare the two strategies of this group from data. Out-of-distribution extrapolation can't create real overlap.

4. An adjustment set is not "all available variables"

For total-effect identification through the backdoor criterion, block every backdoor path and exclude descendants of treatment. Adjusting for a treatment-induced mediator changes the target effect; adjusting for a collider may open a path. The relevant distinction is causal role, not merely when a column appears in a table.

You don’t need to control for all pre-treatment variables: some instrumental variable predictors strongly predict treatment but don’t directly predict the outcome, and in small samples they can increase variance and amplify unmeasured confounding bias. Variable selection should be guided by causal diagrams and the estimand, then balanced with statistical efficiency.

“Stratified correlation coefficients decreasing” is not sufficient causal analysis. Direct estimation of the target effect is needed, along with propagation of model and sampling uncertainty.

5. Outcome Regression, Matching, and Weighting

Outcome Model

Fit $E[Y\mid A,L]$, predict each unit’s outcome difference under the two treatments, and average over the target population. This requires the identification assumptions above and a correctly estimated conditional mean; small predictive error alone does not establish accuracy in counterfactual regions.

Propensity Score

The propensity score $e(L)=P(A=1\mid L)$ describes the probability of treatment given covariates. It can be used for matching, stratification, or inverse probability weighting.

For a binary-treatment ATE, inverse probability weights can be written as $w=A/e(L)+(1-A)/(1-e(L))$. Compute the difference between separately normalized weighted arm means. After estimating weights, check:

• Propensity overlap between treatment and control groups;
• Weighted covariate balance, rather than the propensity model AUC alone;
• Extreme weights and effective sample size;
• Sensitivity to weight truncation and model selection.

Matching can only balance variables that have been measured and entered into the design. Using independent sample standard errors immediately after matching may also be incorrect.

Construct ten units at each of two covariate levels, with treatment fractions 0.2 and 0.8 and a true effect of two. The propensity scores are known in this example:

L = np.repeat([0., 1.], 10)
A = np.array([1] * 2 + [0] * 8 + [1] * 8 + [0] * 2)
Y = 3 * L + 2 * A
propensity = np.where(L == 0, 0.2, 0.8)
assert np.all((propensity > 0) & (propensity < 1))
w = np.where(A == 1, 1 / propensity, 1 / (1 - propensity))
raw = Y[A == 1].mean() - Y[A == 0].mean()
weighted = np.average(Y[A == 1], weights=w[A == 1]) - np.average(
    Y[A == 0], weights=w[A == 0]
)
ess = [(w[A == a].sum() ** 2 / np.square(w[A == a]).sum()) for a in (0, 1)]
print(round(raw, 6), round(weighted, 6), np.round(ess, 6).tolist())
# 3.8 2.0 [6.4, 6.4]

The raw difference is 3.8 and the weighted difference is two. Both weighted arm means of $L$ equal 0.5. Each arm has ten rows but a weight-based effective sample size, $(\sum w{)}^2/\sum w^2$, of only 6.4. This construction checks arithmetic; it supplies neither identification evidence nor a confidence interval for a real observational study.

Doubly Robust Methods

Combine the outcome model with the propensity score model; under appropriate conditions, correctness of either model suffices to maintain consistency. However, "doubly robust" does not protect against unmeasured confounding, positivity violations, or incorrect time ordering.

AIPW augments outcome-model predictions with inverse-probability-weighted residual corrections. In the preceding example, deliberately misspecify each model in turn, then both:

def aipw(m0, m1, e):
    return np.mean(m1 - m0 + A * (Y - m1) / e - (1 - A) * (Y - m0) / (1 - e))

correct_outcome = aipw(3 * L, 3 * L + 2, np.full(len(A), 0.5))
correct_propensity = aipw(np.zeros(len(A)), np.zeros(len(A)), propensity)
both_wrong = aipw(np.zeros(len(A)), np.zeros(len(A)), np.full(len(A), 0.5))
print(np.round([correct_outcome, correct_propensity, both_wrong], 6).tolist())
# [2.0, 2.0, 3.8]

Either correct model recovers two in this construction; two incorrect models give 3.8. The general property is consistency under suitable regularity conditions, not exact recovery in every finite sample. Flexible machine learning for these nuisance models requires attention to cross-fitting and convergence conditions for inference. See Bang and Robins on doubly robust estimation.

6. Four Assumptions to Examine for Instruments

For an instrument $Z$, treatment $A$, and outcome $Y$, justify at least the following:

1. Relevance: $Z$ actually changes $A$;
2. Independence: $Z$ is independent of unmeasured factors that affect the outcome;
3. Exclusion restriction: $Z$ influences $Y$ only through $A$;
4. Treatment-version and interference assumptions support the target interpretation.

In randomized trials with noncompliance, adding monotonicity and other conditions typically identifies the local average treatment effect (LATE) of compliers, but not necessarily the overall average treatment effect (ATE).

For a binary assignment that raises the outcome mean by 0.02 and receipt probability by 0.40, the assumptions above plus monotonicity give a Wald LATE of $0.02/0.40=0.05$. This concerns units whose receipt changes with assignment, not all recipients. Peng Ding’s textbook develops randomization, instrumental variables, and local effects.

Weak instruments lead to unstable estimates and amplify finite-sample bias. Statistical significance in the first stage is not sufficient proof; strength diagnostics, confidence intervals, and arguments about the exclusion restriction should be reported.

7. Quasi-experiments still rely on design

The Difference in Differences (DiD)

Compare the groups’ before-after changes. The key assumption concerns parallel trends in potential outcomes without treatment. Failure to reject a pretrend test does not prove future parallel trends. Check anticipation, concurrent policies, and composition changes. Under staggered adoption with heterogeneous effects, an ordinary two-way fixed-effects coefficient need not estimate the desired average effect.

Regression Discontinuity (RD)

Treatment assignment changes at a threshold of a running variable. RD identifies a local effect near that threshold under conditions including continuity of potential outcomes and no precise sorting around the cutoff. The observed outcome may jump; that jump is the signal being studied. Check bandwidth and functional-form sensitivity, and distinguish sharp assignment from fuzzy changes in treatment probability.

Natural Experiment

A naturally occurring change is not automatically random. Explain why it is plausibly exogenous for the target comparison and who is affected. If it serves as an instrument, separately justify that it affects the outcome only through the target treatment.

8. Sensitivity Analysis and Falsification Checks

Observational conclusions should demonstrate stability under changes in assumptions:

• Change the adjustment set, model form, bandwidth, or weight truncation;
• Use negative control outcomes/exposures to check for relationships that should not exist;
• Check whether future treatment predicts pretreatment outcomes;
• Quantify how strong unmeasured confounding would need to be to overturn the conclusion;
• Identify groups with inadequate overlap and narrow the target population;
• Present the results of different identification strategies side by side, rather than selecting only the favorable ones.

Sensitivity analysis can't prove that an assumption is correct, but it can show how much a conclusion depends on that assumption.

9. Minimum Contents of a Causal Report

Target trial and estimand
Data sources, inclusion criteria, and time zero
DAG and justification for the adjustment set
Identification assumptions and their untestable components
Overlap, balance, and weight diagnostics
Main effects, intervals, and absolute risk
Alternative models and sensitivity analysis
Applicable population and questions the analysis cannot answer

Common Misconceptions

• The more control variables you have, the closer you get to establishing causality: mediators and colliders can change the estimand or introduce bias.
• Propensity score matching equals a randomized experiment: It only addresses measured confounders and relies on overlap.
• An instrument’s association with treatment is sufficient: Independence and exclusion restrictions are usually harder to justify.
• DAGs learned from data are inherently objective: Pure observational distributions usually can't uniquely determine causal directions.

Practice

1. Draw a DAG for training duration, experience, collaboration proficiency, injury, and success rate, and identify the total effect adjustment set.
2. Use simulated data to demonstrate how controlling for a collider creates an association.
3. Conduct propensity score weighting, report standardized differences before and after weighting, overlap, and effective sample size.
4. Review a candidate instrumental variable for relevance, exogeneity, exclusion restriction, and target estimand.

Summary

Causal inference requires agreement among the target trial, causal graph, timing, and identification assumptions. Methods estimate effects under those assumptions; a successful model run does not establish them.

The next chapter tackles scalability: when a single machine's memory can't hold the data, first understand partitioning, shuffle, fault tolerance, and execution semantics, then decide whether to use a distributed system.