8.3 Sampling and Statistical Inference: From Observed Data to Limited Conclusions
In an A/B test, the B group shows a 0.4 percentage point higher conversion rate. This difference could stem from a genuine effect, random variation, sample bias, changes in metric definitions, or the concurrent rollout of other features. Statistical inference does not automatically establish causality; instead, it quantifies uncertainty under clearly defined experimental designs and assumptions.
Population, Sample, and Estimators
- Population: The entire set of objects or processes being studied;
- Sample: The actual data observed in practice;
- Parameter: An unknown quantity in the population, such as the mean
μ; - Estimator: A function of the sample, such as the sample mean
X̄.
The sampling distribution describes how an estimator varies across repeated sampling. While this repeated process cannot be observed with a single sample, inference formulas are built upon this conceptual foundation.
Large sample sizes cannot correct systematic bias. For example, if only successful requests are collected (no matter how many millions of samples are gathered) the true latency that includes failures will never be accurately estimated.
The Law of Large Numbers Is Not a Finite-Sample Guarantee
Under appropriate conditions of independence, identical distribution, and finite expectation, the Law of Large Numbers states that the sample mean converges to the population mean as the sample size increases.
It does not imply:
- That flipping a coin 10,000 times will result in exactly half heads and half tails;
- That the sample mean moves monotonically closer to the true mean with each new observation;
- That any dependent or heavy-tailed process converges quickly;
- That a single observed sequence can validate a distributional model.
"Convergence" is an asymptotic result in the probabilistic or almost sure sense, finite-sample deviations must still be quantified and understood.
The Central Limit Theorem and Standard Error
For common conditions such as independence, identical distribution, and finite variance, the distribution of the standardized sample mean approaches a standard normal as n increases:
(X̄ - μ) / (σ / √n) ⇒ N(0, 1)The standard error of the mean is approximately:
SE(X̄) = σ / √nDoubling the sample size quadruples the sample size, reducing the standard error by half, not to one-fourth.
Heavy-tailed distributions, strong correlations, clustered sampling, and extreme skewness can invalidate the simple approximation. Time-series data often exhibit significant skewness, and the normal approximation of the mean under the Central Limit Theorem does not necessarily hold for the original delayed distribution.
Correct Interpretation of Confidence Intervals
In the frequentist framework, a 95% confidence interval means that if the same sampling and construction procedure were repeated many times, approximately 95% of the resulting intervals would contain the true, fixed parameter value.
Once a specific interval is computed, the parameter does not randomly fluctuate within that interval according to the frequentist model. While it's common in casual speech to say "95% credible," the formal interpretation must distinguish between frequentist confidence intervals and Bayesian credible intervals.
The width of the interval is influenced by:
- Sample size;
- Data variance;
- Confidence level;
- The estimation method and underlying model assumptions.
A narrow interval may still center on a biased estimate, precision alone does not guarantee validity, and the sampling design must be carefully evaluated.
Hypothesis Testing and p-Values
The null hypothesis H₀ states "no effect." The p-value is the probability of observing a statistic as extreme as, or more extreme than, the one observed, assuming H₀ and the model of the test are valid.
The p-value is not:
- The probability that
H₀is true; - The probability resulting from random causes;
- Effect size;
- The probability of experimental reproducibility.
The significance threshold controls part of the long-term Type I error rate. Effect size and confidence intervals should also be reported to assess business relevance.
A Type I Error, A Type II Error, and Test Power
- Type I error: rejecting a true null hypothesis (
H₀); - Type II error: failing to reject
H₀when a true effect exists; - Power: the probability of correctly rejecting
H₀under a specified true effect size.
Sample size planning requires baseline rate, minimum detectable effect, significance level, target power, and allocation ratio. Deciding sample size after the experiment has concluded risks introducing selection bias.
Multiple Comparisons and Continuous Monitoring
Testing 100 invalid metrics simultaneously, each with a 5% significance threshold, dramatically increases the probability of at least one false positive. It's essential to predefine a primary metric and apply appropriate corrections (such as Bonferroni or FDR) to control the error rate for the specific use case.
Checking results daily and stopping at the first significant signal undermines the error rate guarantees of fixed-sample testing. Instead, use pre-designed sequential testing procedures or alpha spending plans, rather than making ad hoc decisions to stop.
Causal Assumptions in A/B Testing
Random assignment ensures that the two groups are comparable in expectation. Additionally, it's essential to verify:
- Sample proportion mismatch;
- User contamination across groups;
- Network effects and interference;
- Novelty effects and time trends;
- Consistency of metric instrumentation across groups;
- Whether the experimental unit is user, session, or request;
- Whether observations from the same user are correlated across requests.
Randomization at the request level, while user behavior remains correlated across requests, leads to an underestimation of standard errors. Cluster randomization requires corresponding analytical adjustments.
Model Probabilities Need Calibration
A classifier's output 0.8 does not automatically mean "80% of such samples are positive in the long run." To interpret it as a probability, calibration must be evaluated: does the actual proportion of positive instances among those predicted as 0.8 closely match 0.8?
Distinguish between:
- Discriminative ability: whether positive instances are ranked higher than negative ones, such as measured by ROC-AUC;
- Calibration: whether the predicted probability values align with observed frequencies;
- Decision quality: whether the outcome is useful under specific cost structures and threshold settings.
Distribution drift can render historical calibration obsolete, requiring continuous monitoring in production.
Delay and SLO Metrics
"95% of requests are below 200 ms" does not mean the remaining 5% can be ignored. With one hundred million requests per day, that 5% translates to five million slow requests.
When defining metrics, be explicit about:
- The time window and sampling rate;
- Whether successes, failures, or retries are included;
- Whether timing is measured on the client or server side;
- The method used to estimate percentiles;
- Grouping dimensions and traffic weights;
- How data loss is handled.
SLOs are goals for product reliability, not statistical theorems. Statistical estimation helps determine whether the target is met, but the target itself stems from user needs and risk trade-offs.
Completion Checklist
Design an A/B conversion experiment:
- Define the experimental unit, population, and random assignment;
- Select the primary metric and the minimum detectable effect;
- Specify sample size and stopping rules;
- Report effect size, confidence intervals, and p-values;
- Address multiple metrics and repeat users;
- List at least three implementation issues that could compromise causal interpretation.