2.3 Bayesian Networks, Conditional Independence, and Inference: Arrows Do Not Automatically Imply Causation

The ground being wet could result from rain, from a sprinkler, or from other causes. While logic can enumerate possibilities, it struggles to quantify the strength of evidence. A Bayesian network uses a directed acyclic graph (DAG) combined with local conditional probability distributions to decompose the joint probability distribution, enabling the system to update beliefs in response to new evidence.

The arrows in the graph represent probabilistic dependencies and conditional independence. To interpret these as causal relationships, additional structural assumptions, temporal context, or intervention semantics are required.

Learning Objectives

• Derive the joint distribution decomposition from a DAG and a CPT;
• Use d-separation to determine conditional independence;
• Understand enumeration, variable elimination, and approximate sampling;
• Distinguish between observational and interventional probability distributions.

1. How Networks Define Joint Distributions

A Bayesian network consists of:

• A directed acyclic graph (DAG);
• A conditional distribution $P(X_i\mid \text{Pa}(X_i))$ for each random variable $X_i$.

The joint distribution factors as:

$$P(x_1,\dots ,x_n)=\prod _{i}P(x_i\mid {\text{pa}}_i).$$

Example: Rain → Wet ← Sprinkler:

$$P(R,S,W)=P(R)P(S)P(W\mid R,S).$$

If Rain and Sprinkler both have no parents, the model assumes marginal independence between them. In real systems, seasonal influences often affect both variables simultaneously; whether to omit such factors must be determined by modeling assumptions.

2. Three Basic Path Structures

Chain: $A\to B\to C$

In a graph containing only this chain, A and C may be associated. Conditioning on B blocks this path; larger graphs require checking other paths too.

Fork: $A\leftarrow B\to C$

B is a common parent. In this three-node graph, conditioning on B makes A and C independent. Calling B a common cause requires a causal interpretation.

Collider: $A\to B\leftarrow C$

The path is blocked when neither B nor a descendant is observed. Conditioning on B or a descendant opens it, allowing dependence between A and C. The graph does not guarantee dependence for every parameter choice, and the direction of association depends on the conditional distributions.

For a concrete example, take independent rain and sprinkler priors of 0.2 and 0.1, and define Wet as their deterministic OR. Then P(Wet)=1−0.8×0.9=0.28 and P(Rain|Wet)=0.2/0.28=5/7. Learning that the sprinkler is on brings the rain probability back to 0.2. This is explaining away that can be checked by hand; a different CPT need not produce the same decrease.

3. d-Separation

d-separation extends the above rules to arbitrary paths. If, given a set of evidence $Z$, all paths between $X$ and $Y$ are blocked, then the graph implies:

$$X\perp Y\mid Z.$$

This expresses independence implied by the graph, without establishing that the data supports the graph. Conversely, failure of d-separation does not guarantee statistical dependence: particular parameter choices can cancel an effect. See Berkeley CS 188 on d-separation for the graphical criterion. Structure learning may only identify Markov equivalence classes and cannot uniquely recover the direction of all arrows from purely observed distributions.

4. Enumeration Inference

To compute $P(Q\mid e)$, enumerate the unobserved variables $Y$:

$$P(Q\mid e)=\alpha \sum _Y\prod _{i}P(x_i\mid p{a}_i).$$

In the OR example, the four assignments to (R,S) have joint probabilities 0.72, 0.08, 0.18, and 0.02. Remove (0,0), the only dry assignment, then normalize to check the posterior. If the evidence has probability zero, normalization is undefined; report that the evidence is incompatible with the model.

Enumeration redundantly computes the same factors, and the computational burden grows exponentially as the number of variables increases.

5. Variable elimination

Convert a CPT into factors:

1. Apply evidence to constrain factors;
2. Choose an ordering of hidden variables;
3. Multiply all factors that contain the selected variable;
4. Sum out the selected variable;
5. Multiply the resulting factors together and normalize.

Correctness does not depend on the elimination order, but runtime and memory usage are highly sensitive to the size of intermediate factors generated. Complexity depends on the induced width or treewidth of the graph, rather than the number of nodes alone.

Heuristics such as min-fill and min-degree are used to select elimination orders. Exact inference can be exponential in the worst case; however, efficient performance is still possible on sparse graphs with good variable orders.

6. Approximate Inference

Prior/rejection sampling

Draw samples from the joint distribution and reject those that do not satisfy the evidence. This approach is extremely inefficient when dealing with rare evidence conditions.

Likelihood weighting

Fix the evidence nodes and weight each sample by the likelihood of the evidence. In cases involving deep or extremely low-probability evidence, the weights can degrade significantly.

Gibbs sampling

Iteratively resample each non-evidence variable from its conditional distribution given the Markov blanket. Performance may be hindered by slow mixing, multimodality, or deterministic relationships.

Approximate algorithms must report effective sample size, chain diagnostics, differences across repeated runs, and error estimates, rather than providing only a single probability value.

7. Parameter Estimation Also Has Uncertainty

A Conditional Probability Table (CPT) can be derived from frequency counts, maximum likelihood estimation, or Bayesian methods. Rare parent configurations may result in zero counts and unstable estimates, which can be mitigated using Dirichlet priors or smoothing, though the prior must be explicitly recorded.

Missing data and latent variables can be handled using methods such as EM, but the resulting estimates depend heavily on the model structure and initial values. Treating estimated CPTs as if they were known true values will underestimate the uncertainty in predictions.

8. When Can a Probabilistic Arrow Be Interpreted as Causal?

An edge in a statistical Bayesian network allows the child distribution to depend on the parent. Some parameter choices make that edge redundant; the edge does not establish an intervention effect.

A causal interpretation requires:

• Nodes and arrows to represent stable mechanisms;
• Key common causes to be properly addressed;
• Absence of biases such as selection bias that would distort the inference;
• The intervention on variable X to be modeled by replacing its generative mechanism with do(X=x);
• The structure to be grounded in design decisions, domain knowledge, or well-justified assumptions.

In general:

$$P(Y\mid X=x)\ne P(Y\mid do(X=x)).$$

Thus, the idea that "direction implies causality" is incorrect. A DAG inferred from correlational data still requires formal causal identification arguments.

9. Modeling and Verification Checklist

variable state exclusivity is complete and consistently granular
DAG is acyclic, with parent node ordering aligned to CPTs
probabilities sum to 1 for each parent configuration
key independence assumptions are grounded in domain knowledge
common causes and selection mechanisms have been explicitly discussed
exact small examples match hand-calculated results
approximate inference includes convergence and error diagnostics
observational predictions are separated from causal interventions

Common Misconceptions

• Arrows naturally imply causality: Statistical decomposition does not equate to an intervention mechanism.
• Fewer edges mean a more realistic model: Every missing edge represents a conditional independence assumption.
• Variable elimination is always fast: Intermediate factors are governed by treewidth and elimination order.
• More samples guarantee accuracy: Correlated samples and collapsing weights can degrade effective information.

Exercise

1. Write out the joint distribution of Rain → Wet ← Sprinkler and manually compute explaining away.
2. For each of chain, fork, and collider structures, determine independence under different evidence conditions.
3. Compare the maximum factor dimensions resulting from two different elimination orders for the same network.
4. Provide an example illustrating the difference between $P(Y\mid X)$ and $P(Y\mid do(X))$.

Summary

Bayesian networks compress joint probabilities using local conditional distributions, while the graph structure explicitly encodes conditional independencies. Inference algorithms compute the consequences of these assumptions; causal interpretation requires an additional layer of reasoning about mechanisms and interventions.

The next chapter moves into reinforcement learning: agents no longer simply query a fixed knowledge base, but instead make continuous decisions that actively shape future data.