8.1 Probability Space and Conditional Probability: First Build the Model, Then Update Your Judgment

When a monitoring system reports “anomaly hit rate of 99%,” that number might refer to recall, precision, or simply the accuracy of historical samples. Before performing any probability calculations, you must first define the random experiment, the events of interest, and the conditional information available.

Probability Space

A probability space is written as:

(Ω, 𝔽, P)

• Ω: the sample space, comprising all possible outcomes;
• 𝔽: the collection of events that can be discussed;
• P: the function that assigns probabilities to events.

In finite sample spaces, it is common to treat every subset as an event. In general probability spaces, the event family must contain Ω and be closed under complements and countable unions to avoid assigning probabilities to pathological subsets arbitrarily.

Probability satisfies the Kolmogorov axioms:

P(A) ≥ 0
P(Ω) = 1
If countably many events A_i are pairwise disjoint, then P(⋃A_i) = ΣP(A_i)

From these, it follows that:

P(Aᶜ) = 1 - P(A)
P(A ∪ B) = P(A) + P(B) - P(A ∩ B)

"Favorable outcomes / All outcomes" must be equally likely

A fair six-sided die:

P(even number) = 3/6 = 1/2

This counting formula relies on the assumption that each face has equal probability. If the die is biased, the probability of an event must be the sum of the probabilities of its individual outcomes, rather than the count of those outcomes alone.

Software traffic is often uneven: 10 types of requests do not imply equal probability for each type 1/10. Replacing production distribution with test case counts leads to incorrect estimates of expected failure rates.

Conditional Probability

Given that event B occurs, and P(B)>0:

P(A|B) = P(A ∩ B) / P(B)

It restricts the sample space to B and then counts how much of that subset also belongs to A.

Multiplication rule:

P(A ∩ B) = P(A|B) × P(B)

Chain rule for multiple events:

P(A₁ ∩ ... ∩ A_n) = P(A₁) × P(A₂|A₁) × ... × P(A_n | A₁ ∩ ... ∩ A_{n-1})

The probability of subsequent events in a without-replacement sampling process changes based on prior outcomes, making chain rule computations particularly suitable.

For this conditional-probability form of the chain rule, each conditioning prefix must have positive probability. A zero-probability prefix makes the joint event null, but subsequent conditional quotients may be undefined.

The Law of Total Probability

If B₁,...,B_k are mutually exclusive and their union is Ω, and each has non-zero probability:

P(A) = Σ_i P(A|B_i)P(B_i)

For example, if requests originate from Web, Mobile, or Batch entry points, the overall timeout rate is a weighted average of each entry point's timeout rate based on traffic proportion, rather than a simple arithmetic mean of the three percentages.

Bayes' Theorem

P(B_i|A) = P(A|B_i)P(B_i) / Σ_j P(A|B_j)P(B_j)

In addition to the partition assumptions, require P(A)>0. Bayes’ theorem reverses the conditioning direction; it does not establish causality. Consider these hypothetical monitoring parameters:

Fault base rate P(D) = 0.001
Alert recall P(+|D) = 0.99
False positive rate P(+|¬D) = 0.02

The probability of a fault given an alert:

P(D|+) = 0.99 · 0.001 / (0.99 · 0.001 + 0.02 · 0.999) ≈ 0.0472

About 4.7% is the model’s alert precision, not its overall accuracy. Across 100000 identically distributed windows, expected counts would be 99 true alerts and 1998 false alerts, giving 99/(99+1998). This Python 3 example verifies the calculation with exact fractions:

from fractions import Fraction

base = Fraction(1, 1000)
recall = Fraction(99, 100)
false_positive = Fraction(2, 100)
true_alert = recall * base
false_alert = false_positive * (1 - base)
posterior = true_alert / (true_alert + false_alert)
print(f"{float(posterior):.4%}")  # 4.7210%

Independence and Mutual Exclusivity

Events A,B are independent when:

P(A∩B) = P(A)P(B)

If P(B)>0, this is equivalent to P(A|B)=P(A).

Mutual exclusivity, on the other hand, is defined as:

A∩B = ∅

Two mutually exclusive events with positive probabilities are not independent; knowing that A has occurred makes the probability of B drop to zero.

Pairwise independence does not imply mutual independence. It's possible for three events to be pairwise independent, yet their joint probability still satisfies a constraint.

For independent fair bits X,Y and Z=X XOR Y, every pair is jointly 1 with probability 1/4, and each bit has marginal probability 1/2 of being 1. All three cannot be 1. Enumerate the four equally likely outcomes:

from itertools import product
from fractions import Fraction

outcomes = [(x, y, x ^ y) for x, y in product((0, 1), repeat=2)]
def probability(predicate):
    return Fraction(sum(predicate(row) for row in outcomes), len(outcomes))

for i, j in [(0, 1), (0, 2), (1, 2)]:
    assert probability(lambda row: row[i] == row[j] == 1) == Fraction(1, 4)
print(probability(lambda row: row == (1, 1, 1)))  # 0, rather than 1/8.

Conditional Independence

Given C with P(C)>0:

P(A∩B|C) = P(A|C)P(B|C)

A shared cause can create marginal dependence that disappears within strata. Suppose a data-center outage has probability 0.01 and makes both services fail; without an outage, each independently fails with probability 0.01. Each marginal failure rate is 0.01+0.99×0.01=0.0199, while joint failure is 0.01+0.99×0.01²=0.010099, far above the marginal product 0.00039601. Conditional independence follows from explicit assumptions here, not merely from controlling a variable.

The naive Bayes classifier assumes feature independence given the class label; this is often a pragmatic modeling choice, not a natural property guaranteed by the data.

Probability Tree Check for Dependencies

A bag contains 3 red and 2 blue balls. Two balls are drawn without replacement. What is the probability that the second ball is red?

P(R₂) = P(R₂|R₁)P(R₁) + P(R₂|B₁)P(B₁)
     = (2/4)(3/5) + (3/4)(2/5)
     = 3/5

Symmetry directly shows that the marginal probability of a red ball at any position is 3/5. The conditional branches illustrate that the two draws are not independent.

Model Assumptions Must Be Reviewable

Before computing failure probabilities, it is essential to clearly specify:

• The time window under consideration;
• Whether requests are independently sampled or represent bursty, correlated events;
• The source population or traffic layer the data represents;
• How missing data and retries are handled;
• Whether the observation of an event alters the system state;
• Whether the probability is based on subjective prior beliefs, historical frequencies, or model predictions.

The same formula cannot correct for sample bias or erroneous causal assumptions.

Completion Checklist

1. Use a probability tree to calculate the probability of drawing two balls of the same color without replacement;
2. Construct two mutually exclusive but not independent events;
3. Construct three events that are pairwise independent but not mutually independent;
4. Define the baseline rate, recall, false positive rate, and positive predictive value for an alerting system;
5. Apply the total probability formula to aggregate timeout rates from three traffic entry points.

References

• MIT Mathematics for Computer Science