8.1 Sampling Frame, Inclusion Probability, and Weights: Who Does the Sample Represent?

The dashboard shows higher activity at sites using the newest client. Generalizing that mean to every site misses an awkward fact: sites on older clients or unreliable networks barely enter the dataset. The sample mean may be precise, but it precisely describes only “the people the system sees.”

Determining who a sample can represent begins with how data enters the sample. This lesson focuses exclusively on representativeness, not causality.

Learning Objectives

• Distinguish between the population of interest, sampling frame, sampling unit, and observation unit;
• Calculate inclusion probabilities and weights for common probability sampling designs;
• Identify sources of coverage bias, nonresponse, and convenience sampling limitations;
• Retain information on stratification, clustering, and weighting when estimating variance.

1. From Target Population to Analytical Sample

A sampling plan should define the following in sequence:

• Target population: Who the conclusions are intended to apply to, for example, all services in production this quarter;
• Sampling frame: The actual list from which samples can be drawn, for example, services in an asset registry;
• Sampling unit: The entity randomly selected from the frame, such as a service;
• Observation unit: The unit ultimately measured, such as one request handled by that service;
• Analytical sample: The observations that are successfully collected and pass quality checks.

If services are sampled but every request becomes an independent row, high-traffic services contribute more observations. To estimate the average service’s performance, aggregate by service first. To estimate performance for an average request, define request inclusion probabilities and the corresponding denominator.

2. The Common Ground of Probability Sampling

Probability sampling requires that every unit in the population have a known and positive inclusion probability ${\pi }_i$. The basic weight is given by:

$$w_i=\frac{1}{{\pi }_i}.$$

A sample unit with weight 20 can be interpreted as representing approximately 20 units in the population. This is not an importance score.

The inclusion probability comes from the sampling design. It differs from the probability of obtaining a response after selection; nonresponse needs separate treatment.

If a class of units has ${\pi }_i=0$, it is impossible to recover the missing representation through post-stratification weighting. The report must either restrict the target population or revise the sampling frame and data collection mechanism.

3. Simple Random Sampling

Sampling $n$ units without replacement from $N$ units, where each unit has a selection probability of $n/N$.

import numpy as np
import pandas as pd

frame = pd.DataFrame({
    "service_id": [f"svc-{i:02d}" for i in range(40)],
    "traffic_tier": ["high"] * 8 + ["low"] * 32,
    "error_rate": [0.10] * 8 + [0.01] * 32,
})
assert frame["service_id"].is_unique
rng = np.random.default_rng(42)
n = 10
if not 0 < n <= len(frame):
    raise ValueError("n must be between 1 and frame size")
positions = rng.choice(len(frame), size=n, replace=False)
srs_sample = frame.iloc[positions].copy()
srs_sample["base_weight"] = len(frame) / n
assert len(srs_sample) == 10
assert srs_sample["base_weight"].eq(4).all()

Run this page’s NumPy and pandas blocks in order. The estimand is the equally weighted mean of 40 service error rates, not a request-weighted site error rate. Each service occupies one row. Sampling positions avoids extra rows from .loc when DataFrame index labels repeat. Preserve the seed, frame version, ordering, and deduplication rules.

4. Stratified Sampling

Divide the population into strata based on regions, task types, or risk levels, then sample independently within each stratum. This approach ensures that smaller or underrepresented subgroups are included in the sample and can improve estimation precision.

When the $h$th stratum contains $N_h$ units and $n_h$ are sampled:

$${\pi }_{hi}=\frac{n_h}{N_h},w_{hi}=\frac{N_h}{n_h}.$$

With proportional allocation, set $n_h\propto N_h$; when over-sampling is used to better represent small subgroups, the stratum weights differ, and the overall population estimate cannot be obtained by a simple unweighted average.

Stratum definitions must be known and mutually exclusive and exhaustive prior to sampling. Defining strata based on sample outcomes is not stratified sampling.

Continuing with the same population, select four services per traffic tier. This samples half of the high-traffic tier and one eighth of the low-traffic tier:

strata = []
for _, group in frame.groupby("traffic_tier", sort=True):
    positions = rng.choice(len(group), size=4, replace=False)
    selected = group.iloc[positions].copy()
    selected["base_weight"] = len(group) / 4
    strata.append(selected)
stratified_sample = pd.concat(strata, ignore_index=True)
print(stratified_sample.groupby("traffic_tier")["base_weight"].first().to_dict())
# {'high': 2.0, 'low': 8.0}

5. Cluster and Multistage Sampling

Cluster sampling selects clusters and observes every unit within selected clusters. Multistage sampling selects a further sample inside each chosen cluster. For example, select 20 of 100 sites, then 10 of the 50 services at each selected site. With $f$ denoting a site, a service’s inclusion probability is:

$${\pi }_i={\pi }_f{\pi }_{i|f}.$$

Here the probability is $(20/100)(10/50)=0.04$, giving weight 25. Recalculate the stage probabilities for unequal cluster sizes or complete observation within a cluster. Positive within-site correlation usually makes independent-row standard errors too small. Preserve strata, primary sampling units, and weights for variance estimation.

6. Systematic sampling cannot start fixed at the first row

Systematic sampling requires a random start. With a correctly implemented equal-probability random start, the sample mean can remain design-unbiased. A list period matching the sampling interval can nevertheless produce extreme individual samples and high variance. Always starting at the first row removes that randomization guarantee.

target_n = 8
if not 0 < target_n <= len(frame):
    raise ValueError("target_n must be between 1 and frame size")

step = len(frame) / target_n
start = rng.uniform(0, step)
positions = np.floor(start + step * np.arange(target_n)).astype(int)
systematic_sample = frame.iloc[positions]

For a list repeating [1, 0, 0, 0] ten times, interval four gives possible sample means 1, 0, 0, and 0. Their average, 0.25, equals the population mean, yet each individual sample misses it. Check ordering and periodicity before use. Also, k = len(df) // 1000; df.iloc[::k] produces invalid step zero below 1,000 rows.

7. Weighted Estimation and Weight Diagnostics

The overall mean can be estimated using a weighted mean:

$$\hat{\mu }=\frac{\sum _{i\in s}{w}_i{y}_i}{\sum _{i\in s}{w}_i}.$$

weighted_mean = np.average(
    stratified_sample["error_rate"], weights=stratified_sample["base_weight"]
)
print(round(stratified_sample["error_rate"].mean(), 3), round(weighted_mean, 3))
# 0.055 0.028
assert np.isclose(weighted_mean, frame["error_rate"].mean())

Actual weights may undergo several stages: design weights, nonresponse adjustments, calibration, or post-stratification. Each step must be documented. Extreme weights can inflate variance; truncation may stabilize the estimate but introduces bias, and such cases must be evaluated through sensitivity analyses rather than being silently ignored.

The normalized weighted mean is a ratio estimator and is not generally exactly design-unbiased because its denominator is random. With known population size $N$, the Horvitz–Thompson mean is $\sum _{i\in s}{w}_i{y}_i/N$. Here fixed stratum sample sizes make the weights sum to 40, so both forms coincide. Constant within-tier error rates also make the weighted estimate exactly recover the population mean, 0.028.

The design effect compares variances under the complex design and simple random sampling of the same size:

$$DEFF=\frac{{\mathrm{Var}}_{design}(\hat{\theta })}{{\mathrm{Var}}_{SRS}(\hat{\theta })}.$$

It varies by estimand and variable and can be below one, for example when stratification improves precision. It describes variance; it neither diagnoses nor removes coverage bias.

8. Nonresponse and Selection Bias

If nonresponse is related to the study variables, increasing sample size will not eliminate bias. Instead, consider:

• Recording contact attempts and reasons for nonresponse;
• Modeling response probability using auxiliary variables from the sampling frame;
• Calibrating to known marginal totals of the population;
• Comparing observable characteristics of early, late, and nonrespondents;
• Conducting sensitivity analyses for unobserved differences.

These adjustments rely on assumptions such as "nonresponse can be explained given auxiliary variables." While convenience samples, active participation, and platform logs can offer analytical insights, they cannot be treated as representative probability samples of the population.

9. Variance Estimation Must Respect Design

The standard error formulas commonly used assume independent and identically distributed observations. For complex sampling designs, methods that account for the survey structure (such as Taylor linearization, replicate weights, or bootstrap resampling at the primary sampling unit level) are required.

Performing individual-row bootstrap within clustered sampling designs is invalid, as it violates the original sampling structure. Additionally, when dealing with finite populations and large sample fractions, a finite population correction may be necessary.

For simple random sampling without replacement, estimate the mean’s standard error as $\sqrt{(1-n/N)s^2/n}$, where $s^2$ is the sample variance. Sampling 10 of 40 units multiplies the independent-sampling formula by $\sqrt{0.75}$. A census removes sampling error but can still contain undercoverage and measurement errors. See Statistics Canada’s probability sampling guide for the distinctions among sampling designs.

Common Misconceptions

• Large sample size implies representativeness: A large sample can still miss certain subgroups or populations due to systemic biases.
• Randomly selecting data rows means we have a random sample: Rows do not necessarily represent the fundamental sampling units.
• A stratified sample can be averaged directly: When sampling is unequal across strata, appropriate weighting is required.
• Weighting can fix any bias: Weighting fails when inclusion probabilities are zero or when critical auxiliary variables are missing.

Exercise

1. Define the target population, sampling frame, units, and analysis sample for “the error rate of all services this quarter.”
2. Design a sample stratified by traffic tier and calculate an analysis weight for each tier.
3. Compare row-wise standard errors to cluster-design standard errors for a dataset sampled by service.
4. Check a systematic sampling list for periodicity and switch to random starting points if necessary.

Summary

Whether a sample represents anyone depends not on the number of rows, but on the sampling frame, inclusion probabilities, and response mechanisms. Weights bring the design back into alignment with the estimates, while variance estimation methods account for the uncertainty introduced by stratification and clustering.

The next lesson moves into randomized experiments: representativeness answers the question "To whom does this apply?", while random assignment helps answer "Does the treatment cause a change in outcomes?"