8.3 DAGs and Observational Identification: Control Variables Can Also Introduce Bias

Training duration can't be randomly assigned, so command staff still need to judge whether extending training improves mission success. Stuffing all fields (experience, injury status, mission difficulty) into a regression might look like "full control," but it could be controlling for training-induced intermediate outcomes or opening up a previously closed colliding pathway.

The core of observational causal inference isn't any single algorithm; it's: which assumptions make the data comparisons equivalent to the target counterfactual comparison?

This lesson's objectives

• Use DAGs to distinguish confounders, mediators, and colliders;
• State the assumptions of consistency, exchangeability, and positivity;
• Understanding when regression, matching, and weighting work and when they don’t;
• Review instrumental variables, difference-in-differences, and regression discontinuity identification conditions.

1. Rewrite relevant questions as cause-and-effect questions

First, write the basic elements of the target trial:

• The population of interest;
• Handling strategies and their time origins;
• Benchmark strategy;
• Outcomes and follow-up window;
• Allocation mechanism;
• estimand and analysis plan

For example, "teams with more training have higher success rates" is just an association; only "the average impact on future 30-day task success rate of increasing training from 10 to 20 hours for eligible teams" approaches an identifiable cause-and-effect question.

The time zero must be consistent. If the treatment group enters at the start of training, but the control group begins at an arbitrary date, it can introduce immortal time and other time biases.

2. A DAG expresses assumptions

Consider:

Experience → Training Duration → Collaboration Proficiency → Success Rate
└────────────────────────────→ Success rate

Training duration ──→ Injury ──← Task difficulty ──→ Success rate

• Experience simultaneously affects both treatment and outcome, acting as a confounding variable;
• Collaboration proficiency lies along the path from input to output and acts as a medium;
• Injury is caused jointly by training duration and task difficulty, making it a collision variable.

When estimating the total effect of training, you should block backdoor paths created by experience, but typically do not control for collaborative proficiency; in this diagram, controlling for injury will open training duration → injury ← task difficulty → success rate path.

The arrows in a DAG come from domain knowledge, temporal order, and mechanistic evidence, not from automatically generating truth based on correlation matrices. Nodes for omitted variables should be included even in the absence of data, because they determine whether identification is possible.

3. Three Basic Identification Criteria

Consistency

The observed outcome for units assigned to treatment $A=a$ equals their corresponding potential outcome $Y(a)$, which requires that "20 hours of training" not be multiple versions with different effects lumped together.

Conditionally Commutative

Given sufficient preprocessing of the confounding variable $L$:

$$ Y(a)\perp A\mid L. $$

It means there is no remaining unmeasured confounding, an assumption typically unverifiable from the observed data itself.

Positivity

In each $L$-combination that needs promotion, each treatment has positive probability:

$$ 0<P(A=a\mid L=l)<1. $$

If the most dangerous task never accepts short training, we can't compare the two strategies of this group from data. Out-of-distribution extrapolation can't create real overlap.

4. An adjustment set is not "all available variables"

An appropriate adjustment set should block all backdoor paths between treatment and outcome and should not include post-treatment variables or colliders that open paths.

You don’t need to control for all pre-treatment variables: some instrumental variable predictors strongly predict treatment but don’t directly predict the outcome, and in small samples they can increase variance and amplify unmeasured confounding bias. Variable selection should be guided by causal diagrams and the estimand, then balanced with statistical efficiency.

“Stratified correlation coefficients decreasing” is not sufficient causal analysis. Direct estimation of the target effect is needed, along with propagation of model and sampling uncertainty.

5. Outcome Regression, Matching, and Weighting

Result Model

Fit $E[Y\mid A,L]$, then predict outcomes for each unit under both $A=1$ and $A=0$ and average them. Accuracy depends on the result model being sufficiently reasonable.

Propensity Score

The propensity score $e(L)=P(A=1\mid L)$ describes the probability of treatment given covariates. It can be used for matching, stratification, or inverse probability weighting.

The stable weight form of ATE can be defined according to specific objectives; regardless of the form, always verify:

• Propensity overlap between treatment and control groups;
• Weighted covariate balance, rather than the propensity model AUC alone;
• Extreme weights and effective sample size; -Sensitivity of weighting truncation and model selection.

Matching can only balance variables that have been measured and entered into the design. Using independent sample standard errors immediately after matching may also be incorrect.

Double Robust Method

Combine the outcome model with the propensity score model; under appropriate conditions, correctness of either model suffices to maintain consistency. However, "doubly robust" does not protect against unmeasured confounding, positivity violations, or incorrect time ordering.

6. Four Tests for Instrumental Variables

When using tool $Z$ to process $A$ and produce result $Y$, at least the following must be justified:

1. Correlation: $Z$ actually changes $A$;
2. Independence: $Z$ is independent of unmeasured factors that affect the outcome;
3. Remove restrictions: $Z$ influences $Y$ only through $A$;
4. Handling version and interference hypotheses aligns with the target explanation.

In non-compliance randomization, adding monotonicity and other conditions typically identifies the local average treatment effect (LATE) of compliers, but not necessarily the overall average treatment effect (ATE).

Weak instruments lead to unstable estimates and amplify finite-sample bias. Statistical significance in the first stage is not sufficient proof; strength diagnostics, confidence intervals, and arguments about the exclusion restriction should be reported.

7. Quasi-experiments still rely on design

The Difference in Differences (DiD)

Compare the changes in the treatment group versus the control group before and after intervention. The key is the appropriate parallel trends assumption, rather than "having pre- and post-data makes it valid alone." You should check for pre-intervention trends, early reactions, concurrent policies, changes in composition, and estimation issues arising from staggered adoption.

Regression Discontinuity (RD)

The analysis depends on the running variable near the threshold; identifying local effects near the threshold relies on conditions such as the inability to precisely manipulate the threshold, continuity of the outcome at the threshold, and requires robustness checks for bandwidth and functional form.

Natural Experiment

"Spontaneous occurrence" doesn't mean random. We must explain why exogenous changes affect outcomes only through the treatment group and who is affected.

8. Sensitivity Analysis and Proof by Contradiction

Observational conclusions should demonstrate stability under changes in assumptions:

• Change the tuning set, model form, bandwidth, or weight truncation;
• Use negative control results/exposure to check for relationships that should not exist;
• Check whether the pre-processing result was "processed" predicted;
• How strong quantitatively unmeasured confounding would need to be to overturn the conclusion;
• Identify underrepresented groups and narrow the target population;
• Present the results of different recognition strategies side by side, rather than selecting only the favorable ones.

Sensitivity analysis can't prove that an assumption is correct, but it can show how much a conclusion depends on that assumption.

9. A causal report must include at least what?

Target trial and estimand
Data sources, inclusion criteria, and time zero
Reasons why DAG and adjustment sets are necessary
Identify assumptions and their untestable components
Overlapping, balance, and weight diagnostics
Main effects, intervals, and absolute risk
Alternative models and sensitivity analysis
Questions applicable to the overall topic and those that cannot be answered

Common Misconceptions

• The more control variables you have, the closer you get to establishing causality: mediators and confounding variables can introduce or create bias.
• Propensity score matching equals a randomized experiment: It only addresses measured confounders and relies on overlap.
• Tool correlation is sufficient: Independence and exclusion constraints are usually harder to justify.
• DAGs learned from data are inherently objective: Pure observational distributions usually can't uniquely determine causal directions.

Practice

1. Draw a DAG for training duration, experience, collaboration proficiency, injury, and success rate, and identify the total effect adjustment set.
2. Use simulated data to demonstrate how control variables create associations.
3. Conduct propensity score weighting, report standardized differences before and after weighting, overlap, and effective sample size.
4. Review a candidate instrumental variable for relevance, exogeneity, exclusion restriction, and target estimand.

Summary

Causal inference isn't about replacing correlation analysis with more complex models; it's about ensuring consistency among the target experiment, causal diagrams, time order, and identification assumptions. Methods can only test assumptions, they can't prove them for you.

Chapter two tackles scalability: when a single machine's memory can't hold the data, first understand partitioning, shuffle, fault tolerance, and execution semantics, then decide whether to use a distributed system.