5.2 Test Selection, Power, and Multiple Comparisons: Plan the Analysis Before the Results

Statistical software makes many tests a short function call. Choosing the analysis unit, comparison structure, and error model still takes judgment, and the decision rules need to be written before the results arrive.

This lesson's objectives

• Choose a method based on result type, design, and dependency structure;
• Report the raw effect size, standardized effect size, and confidence interval;
• Link sample size to the smallest meaningful effect using power analysis;
• Control for multiple comparisons, selective reporting, and optional stopping.

1. The testing method is determined by the study design

Scenario	Common Starting Point	Key Assumption or Alternative
Two independent groups, continuous outcomes	Welch's t-test	Independent units; for heavy tails, consider justified robust or permutation methods
Pre- and post-test within the same unit	paired t-test	paired differences are the object of analysis
Binary Classification Results	Proportion Difference/Score Test, Logistic Model	Retain Numerator and Denominator; Adjust for Small Samples
Continuous outcomes across multiple groups	ANOVA/Regression	First define the overall comparison and then specify contrasts
Count or rate	Poisson/negative binomial model	exposure, overdispersion
Repeated Measures/Clustering	cluster-robust, GEE, mixed models	Number of clusters and hierarchical structure
Survival time	Kaplan–Meier/Cox models	Non-informative censoring; Cox also assumes proportional hazards

This table is not an automatic selector. Random assignment, missing data, covariate adjustment, and measurement error all affect the method.

2. Keep Paired Data Paired

For the same team, complete one task under the old plan and one under the new plan, analyze the difference in each pair:

$$d_i=Y_{i,new}-Y_{i,old}.$$

Pairwise utilization of within-subject correlations typically reduces noise. Using an independent t-test ignores pairing and may lose statistical power.

The paired example below uses NumPy and SciPy 1.11 or later; run this lesson’s Python blocks in order. It estimates the mean within-pair change, -1.6 minutes. A before/after contrast is not automatically a treatment effect: time trends, learning and order effects still need a design that separates them.

Reverse, forcing mismatched records to pair based on recency introduces selection bias. The pairing rule is part of the design and should be fixed before observing the results.

import numpy as np
from scipy import stats

old = np.array([12., 14., 13., 15., 16.])
new = np.array([11., 12., 13., 12., 14.])
if old.shape != new.shape or not np.isfinite(old).all() or not np.isfinite(new).all():
    raise ValueError("Pairs must align and have an explicit missing-value policy")
differences = new - old
paired_result = stats.ttest_rel(new, old)
print(differences.mean(), paired_result.confidence_interval())

3. Report the effect size on the original scale first

Average duration difference: -4.2 minutes
Completion rate difference: +3.1 percentage points
Relative risk: 1.08

The original scale can directly correspond to business costs and the smallest meaningful difference. Cohen's $d$ facilitates cross-scale comparisons, but it depends on standard deviation; when group heterogeneity changes, $d$ also changes.

"0.2/0.5/0.8 is small/mid/large" is just a rough convention in a specific context and should not replace domain-specific thresholds. For security events, even minor relative changes can be significant; for highly volatile experience metrics, larger changes are needed before a release is warranted.

4. Connect Meaningful Effects to Study Sensitivity

Separate two quantities before collecting data. The smallest meaningful effect is a decision threshold: smaller effects would not justify changing practice. The minimum detectable effect (MDE) is the effect a particular design can detect at a specified power and significance level. A study can be too small to detect a meaningful effect; naming both quantities the same does not fix its sensitivity.

Sample size depends on:

• Significance level;
• Target power $1-\beta$;
• baseline rate or variance;
• MDE;
• Allocation ratio;
• Cluster, repeated measures, and attrition;
• Multiple comparison plans.

Power is the probability of rejecting the null under a specified true effect and model. It is not an unconditional probability of a successful study. For equal-size independent groups, a two-sided normal approximation gives the following planning calculation:

from math import ceil
from scipy.stats import norm

alpha, target_power = 0.05, 0.80
sigma, target_mde = 10.0, 2.0  # minutes
n_per_group = ceil(
    2 * (norm.ppf(1 - alpha / 2) + norm.ppf(target_power)) ** 2
    * sigma ** 2 / target_mde ** 2
)
print(n_per_group)  # 393 independent observations per group

This assumes a common standard deviation of 10 minutes and targets a detectable 2-minute difference. The 393 is per group, not the total. It is an approximation for a continuous mean; a t-based design, proportions, attrition, clustering or repeated looks need their own calculation. Planning assumptions should come from relevant prior data rather than the hoped-for result.

5. Observed Power Usually Adds No Information

Plugging the observed effect back into the formula to calculate "observed power" is usually just a rearrangement of the p value and doesn't explain non-significant results. More useful is:

• Prospective power and sample-size planning;
• An interval showing which meaningful effects remain compatible with the data;
• Conduct a sensitivity analysis;
• If the goal is to exclude large effects, use an interval or equivalence design.

6. Multiple comparisons change the error rate

For 100 true null hypotheses, each tested at an exact 5% false-positive rate, the expected number of false positives is 5. Under independence, the probability of at least one is $1-{0.95}^{100}$, about 99.41%. Dependence changes that probability; the expected count alone does not determine it.

Common control objectives:

• FWER: the probability of observing any false positive, such as Bonferroni/Holm;
• FDR: The expected proportion of false discoveries among rejected hypotheses, as in Benjamini–Hochberg.

The choice depends on the consequences: safety-critical blocking may prioritize FWER; large-scale exploratory screening might use FDR. The correction of the family must be predefined and not simply exclude undesirable tests from the family.

FDR is $E[V/max(R,1)]$, where V is the number of false rejections and R is the total number of rejections. It is not the probability that an individual finding is false. Holm controls FWER with valid p-values under arbitrary dependence. The usual BH guarantee requires independence or suitable positive dependence; other dependence structures need a justified method, such as BY. SciPy’s FDR documentation states the BH and BY assumptions.

p_values = np.array([0.001, 0.01, 0.03, 0.20])
bonferroni = np.minimum(1.0, len(p_values) * p_values)
bh = stats.false_discovery_control(p_values, method="bh")
print(bonferroni)  # [0.004, 0.04, 0.12, 0.8]
print(bh)          # [0.004, 0.02, 0.04, 0.2]
print(1 - (1 - 0.05) ** 100)  # approximately 0.9941, assuming independence

At a 5% threshold, Bonferroni rejects the first two hypotheses and BH the first three. They target different error criteria, so the larger rejection count does not make BH universally preferable.

7. Optional stopping inflates error rate

Checking ordinary p-values daily and stopping at the first significant result does not preserve the fixed-sample test's 5% error rate. Options include:

• Reserve a fixed sample size and end time;
• Group sequential design and alpha spending;
• always-valid inference / sequential method;
• Clearly define futility and safety criteria for stopping a trial.

Business monitoring can involve daily review of descriptive metrics, but confirmatory decisions must adhere to predefined analysis rules.

8. Covariate adjustment should be pre-specified

In randomized experiments, adjusting for baseline covariates can improve precision; the adjusted model still needs to respect the randomization units and clustering structure. Trying many combinations of covariates after seeing the results and selecting only those with significant findings introduces researcher degrees of freedom.

Report both unadjusted and prespecified adjusted results, and explain how missing covariates were handled. Randomization specifies the assignment mechanism; it does not guarantee exact covariate balance in every finite sample.

9. An Auditable Report

Questions and estimands
Sample source, exclusion, and time window
Assignment/Sampling Units and Dependency Structure
Key Metrics and MDE
preset test, one/two-tailed, alpha
Point estimate, 95% CI, effect on original scale
p values and multiple comparison methods
Missing, lost, and deviation from agreements
Sensitivity and Group Analysis (Marked Exploratory)
Business decisions and still-undetermined scope

Don't just output "significant/insignificant." For example, if the interval [-1.0, 7.2] percentage points simultaneously includes minor harm and meaningful benefit, the correct conclusion is that current precision is insufficient, not "no effect."

10. Permutation tests must also follow the design

Permutation tests establish a reference distribution by swapping labels under the null hypothesis. They are valid only when exchangeability holds: in cluster randomized trials, labels should be permuted at the cluster level; in paired designs, labels should be swapped within pairs; in stratified randomization, labels should be permuted within strata.

A label permutation tests the null justified by the assignment mechanism or exchangeability assumption. Equality of means alone does not make two unequal-variance populations exchangeable. Randomization tests, studentized statistics and restricted permutations address different designs; choose the null and resampling scheme together.

"Nonparametric" doesn't mean no assumptions. If you misassign units, the p-value can still be overly optimistic.

Common Misconceptions

• Data isn't normal? Just switch to Mann–Whitney: The distribution relationship it tests doesn't always equate to a difference in means or medians.
• Standardized effects have uniform size standards: Business significance depends on the original scale and consequences.
• Observed power explains a nonsignificant result: the confidence interval and pre-planned comparisons are more useful.
• Just need each p < 0.05: Multiple comparisons and optional stopping inflate false positives.

Practice

1. Define the analytical units separately for independent, paired, and clustered designs.
2. Define the MDE for a completion rate experiment and list the inputs required for sample size calculation.
3. Compare Holm's and BH's control of the error target.
4. Rewrite a report containing only p-values into an auditable template.

Summary

Testing extends the study design, not an automatic function of data type. Effect sizes, minimum detectable effects, power, multiple comparisons, and stopping rules must be determined before results are obtained; final reports should show uncertainty ranges, rather than significance labels alone.

The next chapter uses linear regression to organize conditional relationships among multiple variables. The first step is to distinguish prediction from causal interpretation, avoiding the mistake of treating a high $R^2$ model as proof of a mechanism.