8.3 Sampling and Statistical Inference: From Observed Data to Limited Conclusions

In an A/B test, the B group shows a 0.4 percentage point higher conversion rate. This difference could stem from a genuine effect, random variation, sample bias, changes in metric definitions, or the concurrent rollout of other features. Statistical inference does not automatically establish causality; instead, it quantifies uncertainty under clearly defined experimental designs and assumptions.

Population, Sample, and Estimators

• Population: The entire set of objects or processes being studied;
• Sample: The actual data observed in practice;
• Parameter: An unknown quantity in the population, such as the mean μ;
• Estimator: A function of the sample, such as the sample mean X̄.

The sampling distribution describes how an estimator varies across repeated sampling. While this repeated process cannot be observed with a single sample, inference formulas are built upon this conceptual foundation.

Large sample sizes cannot correct systematic bias. For example, if only successful requests are collected (no matter how many millions of samples are gathered) the true latency that includes failures will never be accurately estimated.

The Law of Large Numbers Is Not a Finite-Sample Guarantee

Under appropriate conditions of independence, identical distribution, and finite E|X|, the Law of Large Numbers states that the sample mean converges to the population mean as the sample size increases.

It does not imply:

• That flipping a coin 10,000 times will result in a guaranteed proportion close to one half;
• That the sample mean moves monotonically closer to the true mean with each new observation;
• That any dependent or heavy-tailed process converges quickly;
• That a single observed sequence can validate a distributional model.

"Convergence" is an asymptotic result in the probabilistic or almost sure sense; finite-sample deviations must still be quantified and understood.

The Central Limit Theorem and Standard Error

For common conditions such as independence, identical distribution, and finite positive variance, the distribution of the standardized sample mean approaches a standard normal as n increases:

(X̄ - μ) / (σ / √n) ⇒ N(0, 1)

Under this IID model, the standard error of the mean is exactly:

SE(X̄) = σ / √n

Quadrupling the sample size halves the standard error; it does not reduce it to one quarter.

When population σ is unknown, substituting sample standard deviation s gives the estimated standard error s/√n. Standard error describes estimator variation across samples; standard deviation describes variation of individual observations.

Heavy-tailed distributions, strong correlations, clustered sampling, and extreme skewness can invalidate the simple approximation. Latency data are often skewed. A normal approximation for the sample mean does not make the underlying latency distribution normal.

Correct Interpretation of Confidence Intervals

In the frequentist framework, a 95% confidence interval means that if the same sampling and construction procedure were repeated many times, approximately 95% of the resulting intervals would contain the true, fixed parameter value.

Once a specific interval is computed, the parameter does not randomly fluctuate within that interval according to the frequentist model. While it's common in casual speech to say "95% credible," the formal interpretation must distinguish between frequentist confidence intervals and Bayesian credible intervals.

The width of the interval is influenced by:

• Sample size;
• Data variance;
• Confidence level;
• The estimation method and underlying model assumptions.

A narrow interval may still center on a biased estimate; precision alone does not guarantee validity, and the sampling design must be carefully evaluated.

Hypothesis Testing and p-Values

The null hypothesis H₀ states "no effect." The p-value is the probability of observing a statistic as extreme as, or more extreme than, the one observed, assuming H₀ and the model of the test are valid.

The p-value is not:

• The probability that H₀ is true;
• The probability that the result was caused by chance;
• Effect size;
• The probability of experimental reproducibility.

Under a valid test model and prespecified analysis, a level-α test controls Type I error probability at or below α. Report effect size and an interval alongside the p-value. For the opening 0.4 percentage-point difference, suppose each group has 2000 independent users with one conversion outcome each, and conversion counts are 400 and 408. Python 3.8+ standard-library code gives:

from math import sqrt, erfc
from statistics import NormalDist

x_a, n_a, x_b, n_b = 400, 2000, 408, 2000
p_a, p_b = x_a / n_a, x_b / n_b
difference = p_b - p_a
se = sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
critical = NormalDist().inv_cdf(.975)
interval = (difference - critical * se, difference + critical * se)
pooled = (x_a + x_b) / (n_a + n_b)
se_null = sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
p_value = erfc(abs(difference / se_null) / sqrt(2))
print(f"difference: {100 * difference:.3f} percentage points")
print(f"95% CI: [{100 * interval[0]:.3f}, {100 * interval[1]:.3f}] pp")
print(f"two-sided p: {p_value:.3f}")
# difference: 0.400 percentage points
# 95% CI: [-2.088, 2.888] pp
# two-sided p: 0.753

The interval is an unpooled Wald normal approximation; the test uses a pooled proportion under H₀:p_A=p_B, as in NIST’s two-proportion test. The interval crosses zero; failure to reject does not establish equality. Rare outcomes, small samples, or proportions near 0/1 need more suitable interval and testing methods.

A Type I Error, A Type II Error, and Test Power

• Type I error: rejecting a true null hypothesis (H₀);
• Type II error: failing to reject H₀ when a true effect exists;
• Power: the probability of correctly rejecting H₀ under a specified true effect size.

Sample size planning requires baseline rate, minimum detectable effect, significance level, target power, and allocation ratio. Deciding sample size after the experiment has concluded risks introducing selection bias.

Multiple Comparisons and Continuous Monitoring

If all 100 null hypotheses are true and their tests are independent with Type I error probability exactly 5% each, the chance of at least one false positive is 1−0.95^100≈99.4%. Dependence changes that calculation; 100 identical tests still have probability 5%. Bonferroni uses α/100 per test to control the familywise chance of any false positive without independence. FDR instead controls the expected proportion of false positives among discoveries. Specify the primary metric and error criterion first.

Checking results daily and stopping at the first significant signal undermines the error rate guarantees of fixed-sample testing. Instead, use pre-designed sequential testing procedures or alpha spending plans, rather than making ad hoc decisions to stop.

Causal Assumptions in A/B Testing

Random assignment ensures that the two groups are comparable in expectation. Additionally, it's essential to verify:

• Sample proportion mismatch;
• User contamination across groups;
• Network effects and interference;
• Novelty effects and time trends;
• Consistency of metric instrumentation across groups;
• Whether the experimental unit is user, session, or request;
• Whether observations from the same user are correlated across requests.

Treating positively correlated observations from the same user as independent can underestimate standard errors. Align the randomization unit, effect estimator, and variance calculation; cluster randomization generally needs cluster-level analysis or suitable dependence adjustments.

Model Probabilities Need Calibration

A classifier's output 0.8 does not automatically mean "80% of such samples are positive in the long run." To interpret it as a probability, calibration must be evaluated: does the actual proportion of positive instances among those predicted as 0.8 closely match 0.8?

Distinguish between:

• Discriminative ability: whether positive instances are ranked higher than negative ones, such as measured by ROC-AUC;
• Calibration: whether the predicted probability values align with observed frequencies;
• Decision quality: whether the outcome is useful under specific cost structures and threshold settings.

Distribution drift can render historical calibration obsolete, requiring continuous monitoring in production.

Latency and SLO Metrics

"95% of requests are below 200 ms" does not mean the remaining 5% can be ignored. With one hundred million requests per day, that 5% translates to five million slow requests.

When defining metrics, be explicit about:

• The time window and sampling rate;
• Whether successes, failures, or retries are included;
• Whether timing is measured on the client or server side;
• The method used to estimate percentiles;
• Grouping dimensions and traffic weights;
• How data loss is handled.

SLOs are goals for product reliability, not statistical theorems. Statistical estimation helps determine whether the target is met, but the target itself stems from user needs and risk trade-offs.

Completion Checklist

Design an A/B conversion experiment:

1. Define the experimental unit, population, and random assignment;
2. Select the primary metric and the minimum detectable effect;
3. Specify sample size and stopping rules;
4. Report effect size, confidence intervals, and p-values;
5. Address multiple metrics and repeat users;
6. List at least three implementation issues that could compromise causal interpretation.

References

• MIT OpenCourseWare: Introduction to Probability
• NIST/SEMATECH e-Handbook of Statistical Methods