2.3 Bayesian Networks, Conditional Independence, and Inference: Arrows Do Not Automatically Imply Causation

In the Developer Workshop, Ah Hua receives evidence that "the ground is wet" and must update her beliefs about whether it rained, whether the sprinkler system was activated, or if another factor is responsible.

The ground being wet could result from rain, from a sprinkler, or from other causes. While logic can enumerate possibilities, it struggles to quantify the strength of evidence. A Bayesian network uses a directed acyclic graph (DAG) combined with local conditional probability distributions to decompose the joint probability distribution, enabling the system to update beliefs in response to new evidence.

The arrows in the graph represent probabilistic dependencies and conditional independence. To interpret these as causal relationships, additional structural assumptions, temporal context, or intervention semantics are required.

Learning Objectives

• Derive the joint distribution decomposition from a DAG and a CPT;
• Use d-separation to determine conditional independence;
• Understand enumeration, variable elimination, and approximate sampling;
• Distinguish between observational and interventional probability distributions.

1. How Networks Define Joint Distributions

A Bayesian network consists of:

• A directed acyclic graph (DAG);
• A conditional distribution $P(X_i \mid \text{Pa}(X_i))$ for each random variable $X_i$.

The joint distribution factors as:

$$ P(x_1,\ldots,x_n) = \prod_i P(x_i \mid \text{pa}_i). $$

Example: Rain → Wet ← Sprinkler:

$$ P(R,S,W) = P(R)P(S)P(W \mid R,S). $$

If Rain and Sprinkler both have no parents, the model assumes marginal independence between them. In real systems, seasonal influences often affect both variables simultaneously; whether to omit such factors must be determined by modeling assumptions.

2. Three Basic Path Structures

Chain: $A\rightarrow B\rightarrow C$

A and C are typically associated when B is unspecified; the path is blocked once B is given.

Fork: $A\leftarrow B\rightarrow C$

B is a common cause; once B is known, A and C become conditionally independent.

Collider: $A\rightarrow B\leftarrow C$

The path is blocked when B and its descendants are not observed, allowing A and C to be conditionally independent. However, observing B opens the path, creating a conditional association between A and C, this is known as "explaining away."

In Rain → Wet ← Sprinkler, if we first learn that the ground is wet and then learn that the sprinkler is on, our belief in rain decreases, even though the two causes were initially independent.

3. d-Separation

d-separation extends the above rules to arbitrary paths. If, given a set of evidence $Z$, all paths between $X$ and $Y$ are blocked, then the graph implies:

$$ X\perp Y\mid Z. $$

This expresses independence implied by the graph structure, not necessarily that the data supports this structure. Structure learning may only identify Markov equivalence classes and cannot uniquely recover the direction of all arrows from purely observed distributions.

4. Enumeration Inference

To compute $P(Q\mid e)$, enumerate the unobserved variables $Y$:

$$ P(Q\mid e)=\alpha\sum_Y\prod_iP(x_i\mid pa_i). $$

For small networks, truth table combinations can be used to implement and serve as a test oracle. The original draft references enumeration_ask, which calls non-existent functions net.variables() and _enumerate_all(), it is not a runnable implementation; the course should not present interface sketches as complete algorithms.

Enumeration redundantly computes the same factors, and the computational burden grows exponentially as the number of variables increases.

5. Variable elimination

Convert a CPT into factors:

1. Apply evidence to constrain factors;
2. Choose an ordering of hidden variables;
3. Multiply all factors that contain the selected variable;
4. Sum out (sum out) the selected variable;
5. Multiply the resulting factors together and normalize.

Correctness does not depend on the elimination order, but runtime and memory usage are highly sensitive to the size of intermediate factors generated. Complexity depends on the induced width or treewidth of the graph, rather than the number of nodes alone.

Heuristics such as min-fill and min-degree are used to select elimination orders. Exact inference can be exponential in the worst case; however, efficient performance is still possible on sparse graphs with good variable orders.

6. Approximate Inference

Prior/rejection sampling

Draw samples from the joint distribution and reject those that do not satisfy the evidence. This approach is extremely inefficient when dealing with rare evidence conditions.

Likelihood weighting

Fix the evidence nodes and weight each sample by the likelihood of the evidence. In cases involving deep or extremely low-probability evidence, the weights can degrade significantly.

Gibbs sampling

Iteratively resample each non-evidence variable from its conditional distribution given the Markov blanket. Performance may be hindered by slow mixing, multimodality, or deterministic relationships.

Approximate algorithms must report effective sample size, chain diagnostics, differences across repeated runs, and error estimates, rather than providing only a single probability value.

7. Parameter Estimation Also Has Uncertainty

A Conditional Probability Table (CPT) can be derived from frequency counts, maximum likelihood estimation, or Bayesian methods. Rare parent configurations may result in zero counts and unstable estimates, which can be mitigated using Dirichlet priors or smoothing, though the prior must be explicitly recorded.

Missing data and latent variables can be handled using methods such as EM, but the resulting estimates depend heavily on the model structure and initial values. Treating estimated CPTs as if they were known true values will underestimate the uncertainty in predictions.

8. When Can a Probabilistic Arrow Be Interpreted as Causal?

An edge in a statistical Bayesian network represents a direct probabilistic dependency (relative to the chosen graph), but it does not automatically imply a causal relationship.

A causal interpretation requires:

• Nodes and arrows to represent stable mechanisms;
• Key common causes to be properly addressed;
• Absence of biases such as selection bias that would distort the inference;
• The intervention on variable X to be modeled by replacing its generative mechanism with do(X=x);
• The structure to be grounded in design decisions, domain knowledge, or well-justified assumptions.

In general:

$$ P(Y\mid X=x) \ne P(Y\mid do(X=x)). $$

Thus, the idea that "direction implies causality" is incorrect. A DAG inferred from correlational data still requires formal causal identification arguments.

9. Modeling and Verification Checklist

Variable state exclusivity is complete and consistently granular DAG is acyclic, with parent node ordering aligned to CPTs Probabilities sum to 1 for each parent configuration Key independence assumptions are grounded in domain knowledge Common causes and selection mechanisms have been explicitly discussed Exact small examples match hand-calculated results Approximate inference includes convergence and error diagnostics Observational predictions are separated from causal interventions

variable state exclusivity is complete and consistently granular
DAG is acyclic, with parent node ordering aligned to CPTs
probabilities sum to 1 for each parent configuration
key independence assumptions are grounded in domain knowledge
common causes and selection mechanisms have been explicitly discussed
exact small examples match hand-calculated results
approximate inference includes convergence and error diagnostics
observational predictions are separated from causal interventions

Common Misconceptions

• Arrows naturally imply causality: Statistical decomposition does not equate to an intervention mechanism.
• Fewer edges mean a more realistic model: Every missing edge represents a conditional independence assumption.
• Variable elimination is always fast: Intermediate factors are governed by treewidth and elimination order.
• More samples guarantee accuracy: Correlated samples and collapsing weights can degrade effective information.

Exercise

1. Write out the joint distribution of Rain → Wet ← Sprinkler and manually compute explaining away.
2. For each of chain, fork, and collider structures, determine independence under different evidence conditions.
3. Compare the maximum factor dimensions resulting from two different elimination orders for the same network.
4. Provide an example illustrating the difference between $P(Y\mid X)$ and $P(Y\mid do(X))$.

Summary

Bayesian networks compress joint probabilities using local conditional distributions, while the graph structure explicitly encodes conditional independencies. Inference algorithms compute the consequences of these assumptions; causal interpretation requires an additional layer of reasoning about mechanisms and interventions.

The next chapter moves into reinforcement learning: agents no longer simply query a fixed knowledge base, but instead make continuous decisions that actively shape future data.