5.2 Testing Selection, Power, and Multiple Comparisons: Planning Analysis Before Results

Statistical software can return a p-value for any two columns. The real challenge lies in selecting the correct analysis unit, comparison structure, and error model, and writing decision rules before even looking at the results.

This lesson's objectives

• Choose a method based on result type, design, and dependency structure;
• Report the raw effect size, standardized effect size, and confidence interval;
• Link sample size to the smallest meaningful effect using efficacy analysis;
• Control for multiple comparisons, selective reporting, and optional stopping.

1. The testing method is determined by the study design

Scenario	Common Starting Point	Key Assumption or Alternative
Two independent groups, continuous outcomes	Welch's t-test	Independent units; robust in the presence of heavy tails; consider permutation methods
Pre- and post-test within the same unit	paired t-test	paired differences are the object of analysis
Binary Classification Results	Proportion Difference/Score Test, Logistic Model	Retain Numerator and Denominator; Adjust for Small Samples
Multiple consecutive results	ANOVA/Regression	First define the overall comparison and then specify contrasts
Count or rate	Poisson/negative binomial model	exposure, overdispersion
Repeated Measures/Clustering	cluster-robust, GEE, mixed models	Number of clusters and hierarchical structure
Survival time	Kaplan–Meier/Cox models	censoring and proportional hazards assumptions

This table is not an automatic selector. Random assignment, missing data, covariate adjustment, and measurement error all affect the method.

Don't split paired data into separate groups

For the same team, complete one task under the old plan and one under the new plan, analyze the difference in each pair:

$$ d_i=Y_{i,new}-Y_{i,old}. $$

Pairwise utilization of within-subject correlations typically reduces noise. Using an independent t-test ignores pairing and may lose statistical power.

Reverse, forcing mismatched records to pair based on recency introduces selection bias. The pairing rule is part of the design and should be fixed before observing the results.

3. Report the effect size on the original scale first

Average duration difference: -4.2 minutes
Completion rate lag: +3.1 percentage points
Relative risk: 1.08

The original scale can directly correspond to business costs and the smallest meaningful difference. Cohen's $d$ facilitates cross-scale comparisons, but it depends on standard deviation; when group heterogeneity changes, $d$ also changes.

"0.2/0.5/0.8 is small/mid/large" is just a rough convention in a specific context and should not replace domain-specific thresholds. For security events, even minor relative changes can be significant; for highly volatile experience metrics, larger changes are needed before a release is warranted.

4. Decision for Minimum Meaningful Effect Connection

Define the MDE or practical threshold before collecting data: any difference smaller than this will not change behavior, even if it exists.

Sample size depends on:

• Significance level;
• Target power $1-\beta$;
• baseline rate or variance;
• MDE;
• Allocation ratio;
• Cluster, repeated measures, and attrition;
• Multiple comparison plans.

Effect size is not an unconditional guarantee of study success. It is the probability of rejecting the null hypothesis under specified true effect and model assumptions. Even with overly optimistic assumptions, sample size may still be insufficient.

5. Post-hoc effects typically don't provide new information

Plugging the observed effect back into the formula to calculate "observed power" is usually just a rearrangement of the p value and doesn't explain non-significant results. More useful is:

• Pre-specification of efficacy and sample size design;
• The report estimates the interval covering meaningful effects;
• Conduct a sensitivity analysis;
• If the goal is to exclude large effects, use an interval or equivalence design.

6. Multiple comparisons change the error rate

If you perform a $\alpha=0.05$ test on each of 100 truly null metrics, you'd expect about 5 false positives; the probability of getting at least one is higher.

Common control objectives:

• FWER: the probability of observing any false positive, such as Bonferroni/Holm;
• FDR: The expected proportion of false discoveries among rejected hypotheses, as in Benjamini–Hochberg.

The choice depends on the consequences: safety-critical blocking may prioritize FWER; large-scale exploratory screening might use FDR. The correction of the family must be predefined and not simply exclude undesirable tests from the family.

7. Optional stopping inflates error rate

Checking p-values every day, stopping as soon as one is significant, and not maintaining the fixed sample t-test's 5% error rate, this can be avoided by using:

• Reserve a fixed sample size and end time;
• Group sequential design and alpha spending;
• always-valid inference / sequential method;
• Clearly define futility and safety criteria for stopping a trial.

Business monitoring can involve daily review of descriptive metrics, but confirmatory decisions must adhere to predefined analysis rules.

8. Covariate adjustment should be pre-specified

In randomized experiments, adjusting for baseline covariates can improve precision; the adjusted model still needs to respect the randomization units and clustering structure. Trying many combinations of covariates after seeing the results and selecting only those with significant findings introduces researcher degrees of freedom.

The report does not adjust for or address the pre-specified adjustments, and lacks explanation of how missing covariate handling was managed. Randomization ensures the assignment mechanism, but does not guarantee complete balance of all covariates across finite samples.

9. An Auditable Report

Questions and estimands
Sample source, exclusion, and time window
Assignment/Sampling Units and Dependency Structure
Key Metrics and MDE
preset test, one/two-tailed, alpha
Point estimate, 95% CI, effect on original scale
p values and multiple comparison methods
Missing, lost, and deviation from agreements
Sensitivity and Group Analysis (Marked Exploratory)
Business decisions and still-undetermined scope

Don't just output "significant/insignificant." For example, if the interval [-1.0, 7.2] points simultaneously includes minor harm and meaningful benefit, the correct conclusion is that current precision is insufficient, not "no effect."

10. Permutation tests must also follow the design

Permutation tests establish a reference distribution by swapping labels under the null hypothesis. They are valid only when exchangeability holds: in cluster randomized trials, labels should be permuted at the cluster level; in paired designs, labels should be swapped within pairs; in stratified randomization, labels should be permuted within strata.

"Nonparametric" doesn't mean no assumptions. If you misassign units, the p-value can still be overly optimistic.

Common Misconceptions

• Data isn't normal? Just switch to Mann–Whitney: The distribution relationship it tests doesn't always equate to a difference in means or medians.
• Standardized effects have uniform size standards: Business significance depends on the original scale and consequences.
• After seeing the results, the calculated power can't explain the non-significance: the confidence interval and pre-planned comparisons are more useful.
• Just need each p < 0.05: Multiple comparisons and optional stopping inflate false positives.

Practice

1. Define the analytical units separately for independent, paired, and clustered designs.
2. Define the MDE for a completion rate experiment and list the inputs required for sample size calculation.
3. Compare Holm's and BH's control of the error target.
4. Rewrite a report containing only p-values into an auditable template.

Summary

Power analysis is an extension of study design, not an automatic function of data type. Effect sizes, minimum detectable effects, power, multiple comparisons, and stopping rules must be determined before results are obtained; final reports should show uncertainty ranges, rather than significance labels alone.

The next chapter uses linear regression to organize conditional relationships among multiple variables. The first step is to distinguish prediction from causal interpretation, avoiding the mistake of treating a high $R^2$ model as proof of a mechanism.