5.1 Estimators, Confidence Intervals, and p-Values: Conclusions Depend on Sampling

A four-percentage-point difference sounds precise until you ask who entered the sample and how many independent units it contains. Inference connects the estimate to that sampling process; a small p-value cannot supply a missing design.

Learning Objectives

• Clarify the distinctions between population, sample, estimand, and estimator;
• Accurately explain standard error and frequentist confidence intervals;
• Construct a confidence interval directly for the difference between two groups;
• Correctly interpret p-values and the concept of "failing to reject the null hypothesis".

1. First, write the estimand

An estimand is the quantity we hope to learn from the target population. For example:

Target population: Tasks that meet eligibility criteria over the next quarter
Intervention: Using the new training protocol
Control: Using the old protocol
Outcome: Tasks completed within 24 hours
Estimand: The difference in completion probability (in percentage points) between the two protocols

Sample means, sample proportions, and regression coefficients are estimators. Whether these estimators accurately capture the target estimand depends on sampling, assignment, measurement, and missing data mechanisms. Statistical formulas can only quantify uncertainty within a model, they cannot correct for selection bias.

Strictly, an estimator is a rule applied to a random sample; the number it produces from the observed data is an estimate. Generalizing today’s sample to next quarter also needs an assumption about how the target population and operating conditions change.

2. Standard Deviation and Standard Error

• Standard deviation describes the dispersion among observed values;
• Standard error describes the variability of an estimator across repeated sampling.

The estimated standard error of the mean for independent and identically distributed observations is:

$$SE(\overline{X})=\frac{s}{\sqrt{n}}.$$

Tasks from one team may be correlated. With positive within-team correlation, treating every row as an independent unit tends to understate the standard error. Cluster by the assignment or dependence unit, use an appropriate hierarchical model, or aggregate with weights that preserve the estimand. Aggregation alone can change a task-average target into a team-average target.

3. Frequentist Explanation of a 95% Confidence Interval

A 95% confidence interval construction procedure, when repeatedly applied under identical conditions, will yield intervals that cover the true parameter value in approximately 95% of cases over the long run.

Once a specific interval has been computed, within the frequentist framework the parameter is considered fixed and deterministic. It is therefore incorrect to say that "the true parameter has a 95% probability of lying within this already calculated interval." Instead, we can state:

Under the current sampling conditions and model assumptions, the estimated difference is 4.0 percentage points,
with a 95% confidence interval of [1.2, 6.8] percentage points.

A Bayesian credible interval offers an alternative interpretation of posterior parameter probabilities, but it requires a prior distribution and a specified probabilistic model. It should not be conflated with or used interchangeably with frequentist confidence intervals.

4. Differences Between Two Groups Should Be Directly Estimated

For one continuous outcome measured in two independent groups, Welch’s method does not require equal variances. The Python blocks below run in order with NumPy and SciPy 1.11 or later; the reference result’s confidence-interval method was added in 1.11.

import numpy as np
from scipy import stats

old = np.asarray([30, 32, 35, 29, 34, 31], dtype=float)
new = np.asarray([24, 28, 25, 27, 26, 29], dtype=float)
if any(x.ndim != 1 or len(x) < 2 or not np.isfinite(x).all() for x in (old, new)):
    raise ValueError("Each independent group needs at least two finite observations")

estimate = new.mean() - old.mean()
new_component = new.var(ddof=1) / len(new)
old_component = old.var(ddof=1) / len(old)
se = np.sqrt(new_component + old_component)
if se == 0:
    raise ValueError("Both groups have zero sample variance; review the measurement design")
degrees_of_freedom = (new_component + old_component) ** 2 / (
    new_component ** 2 / (len(new) - 1)
    + old_component ** 2 / (len(old) - 1)
)
critical = stats.t.ppf(0.975, degrees_of_freedom)
interval = (estimate - critical * se, estimate + critical * se)
reference = stats.ttest_ind(new, old, equal_var=False)
print(estimate, interval, reference.pvalue)
print(reference.confidence_interval(confidence_level=0.95))

It is incorrect to treat the lower bound of the new group's confidence interval minus the lower bound of the old group's confidence interval as the confidence interval for the difference. The difference itself has its own sampling distribution and standard error.

The fixture estimates -5.333 minutes with a 95% interval of approximately [-8.058, -2.608] minutes and a two-sided p-value of 0.00151. The difference is defined as new minus old, so a negative value means shorter duration under the new protocol. Independence is still required, and a small heavy-tailed sample can make the t approximation unreliable. SciPy’s Welch test supplies the same difference direction when called with (new, old).

The overlap of 95% confidence intervals from two groups is not a standard 5% significance rule for difference. Even when intervals slightly overlap, the difference may still be statistically significant. Instead, the difference interval should be directly computed or the appropriate statistical test applied.

5. Methods Need to Match Their Scale

When each task outcome is success/failure, keep counts:

new_success / new_total
old_success / old_total

Differences, risk ratios, and odds ratios answer different questions. Point differences in completion rates are often easiest to explain in percentage points; relative risk requires specifying a baseline.

new_success, new_total = 3, 5
old_success, old_total = 3, 10
risk_difference = new_success / new_total - old_success / old_total
relative_risk = (new_success / new_total) / (old_success / old_total)
new_rate_ci = stats.binomtest(new_success, new_total).proportion_ci(method="wilson")
print(risk_difference, relative_risk, new_rate_ci)
zero_rate_ci = stats.binomtest(0, 5).proportion_ci(method="wilson")
print(zero_rate_ci)

The point difference is 0.30, or 30 percentage points, and relative risk is 2. The Wilson interval here is for the new group’s probability alone, not for the difference. Even 0 successes in 5 trials has a positive upper bound, about 43.45%; zero observed events does not establish zero risk. These intervals assume independent Bernoulli observations with a common probability. See SciPy proportion intervals.

With small samples or near 0/1 values, simple normal approximation intervals may perform poorly. Consider using score/Newcombe intervals, exact methods, or appropriate statistical models. Treating each aggregated group's success_rate as a continuous observation for a t-test will ignore denominator variability.

6. p-value is a conditional probability

Given:

• The null hypothesis $H_0$;
• A test statistic;
• Sampling and model assumptions;

The p-value is the probability of observing the current statistic or something more extreme, under the assumption that $H_0$ is true.

It is not:

• The probability that $H_0$ is true;
• The probability that the result is due to random chance;
• The probability that the effect is practically important;
• The probability that the study is reproducible.

"More extreme" is defined by the alternative hypothesis (one-sided or two-sided) and the test statistic. Choosing a one-sided test after seeing the direction of the result will alter the error rate.

The ASA statement on p-values also separates statistical compatibility from effect size and practical importance. The relevant model assumptions are part of the claim, not optional footnotes.

7. Significance Thresholds Should Be Set in Advance

The significance level $\alpha$ controls the long-run probability of incorrectly rejecting the null hypothesis when it is true and all assumptions hold. 0.05 is a convention, not a natural law.

Report continuous p-values and confidence intervals, rather than treating 0.049 and 0.051 as fundamentally different types of results. p=0.06 cannot be rebranded as "nearly significant"; instead, report estimates, confidence intervals, sample size, and decision rules.

Failure to reject $H_0$ does not prove equality. Equivalence asks whether the effect lies within predeclared lower and upper bounds. Non-inferiority asks whether it is no worse than one allowed margin, usually on one side; that does not establish equivalence. The bounds and testing procedure must be chosen before observing the results.

8. Bootstrap Still Relies on Resampling Units

Bootstrap can estimate the distribution of a statistic by repeatedly resampling units, making it suitable for complex estimators. However, it is not a method free of assumptions.

When tasks are correlated within a team, resampling should be done at the team level (cluster resampling), not row by row. Time series data requires methods like block bootstrap that preserve dependencies. Incorrect resampling units can lead to overly narrow confidence intervals.

Common Misconceptions

• "A 95% CI means the parameter has a 95% probability of lying within it": This is a common misinterpretation of confidence intervals from a frequentist perspective.
• "If two confidence intervals overlap, the difference is not significant": The correct approach is to directly evaluate the difference between the groups.
• "A small p-value proves a large effect": Sample size also influences p-values, larger samples can produce smaller p-values even with small actual effects.
• "Bootstrap can ignore correlation": When resampling, units must be drawn in a way that preserves the data generation process; ignoring correlation can lead to invalid inferences.

Exercise

1. State the population, sample, estimand, and estimator for the new and old approaches.
2. Explain how it's possible for a large standard deviation and a small standard error of the mean to occur simultaneously.
3. Compute a confidence interval for the difference in means using Welch's formula, and compare it to the individual confidence intervals of the two groups.
4. Design a cluster bootstrap for repeated user data.

Summary

Statistical inference begins with the estimand and the sampling mechanism. Confidence intervals describe the precision of an estimate, while p-values indicate how extreme a statistic is under the null hypothesis. Both must be interpreted in the context of the hypothesis, units, and the defined difference.

The next lesson will focus on specific tests and integrate effect size, power, multiple comparisons, and analysis planning into a unified decision-making framework.