6.2 Residuals, Heteroscedasticity, and Influential Points: Diagnosing Which Failure Mode to Address

The model has produced its coefficients, but Ah Hua notices arcs, funnel shapes, and a few unusually influential observations in the residual plot.

Model fit is just a software state. An arc in the residual plot indicates that the mean structure has missed a nonlinear relationship; a funnel shape signals changing error variance; and a small number of high-leverage points may entirely determine the direction of the regression line. Each of these diagnostic patterns corresponds to a distinct failure mode, never to be reduced to a single fix like "remove outliers."

Learning Objectives

• Connect linear, independent, homoscedasticity, and distributional assumptions to specific conclusions;
• Explain residual-vs-fitted plots, Q-Q plots, leverage, and Cook's distance;
• Apply appropriate boundaries when using robust or clustered standard errors;
• Understand how multicollinearity affects coefficient stability rather than necessarily harming predictive performance.

1. Distinguish Prediction Conditions from Inference Conditions

Common assumptions:

1. The conditional mean form is correct, $E[\varepsilon\mid X]=0$;
2. The observed or error dependence structure is properly addressed;
3. The conditional variance is constant;
4. The error distribution is compatible with finite-sample inference methods;
5. The design matrix lacks complete multicollinearity;
6. The distribution of the training data matches that of the application setting.

Heteroskedasticity does not necessarily bias OLS coefficient estimates, but it can lead to incorrect standard errors, and OLS is no longer the most efficient linear unbiased estimator. On the other hand, omitted confounding variables that cause $E[\varepsilon\mid X]\ne0$ can directly introduce bias into the coefficient estimates.

2. Residuals vs Fitted: Checking Mean and Variance Structure

import matplotlib.pyplot as plt
import seaborn as sns

fitted = model.fittedvalues
residuals = model.resid

sns.scatterplot(x=fitted, y=residuals, alpha=0.3)
plt.axhline(0, color="black", linestyle="--")
plt.xlabel("Fitted values")
plt.ylabel("Residuals")

Observations:

• Arc-shaped patterns: indicate missing nonlinearity or interaction effects;
• Funnel shape: suggests variance changes with the mean;
• Grouped bands: may point to categorical variables, measurement discretization, or omitted subgroups;
• Large structured regions: could reflect temporal or spatial dependence or model specification errors.

Smooth lines can help identify systemic patterns, but residuals and fitted values share estimated parameters. Therefore, graphical interpretation must be contextualized with knowledge of the data generation process.

3. Q-Q Plots Are Not the Final Switch for Model Validity

A Q-Q plot compares the quantiles of standardized residuals against those of a reference distribution. Deviations in the tails may arise from heavy-tailed distributions, outliers, mixture populations, or incorrect variance specification.

Normality of errors is primarily relevant for certain small-sample exact t/F inference; ordinary least squares (OLS) point estimation does not require either the response variable $Y$ or the residuals to be perfectly normal. Large-sample approximations also depend on independence, moment conditions, and design structure, sample size alone is not sufficient to justify ignoring clustering.

If the goal is prediction, the performance of the validation set and interval coverage are more directly relevant than whether residuals appear normally distributed.

4. How to Handle Heteroskedasticity Depends on the Objective

• For inference only: use heteroskedasticity-consistent covariance (HC series);
• When the variance structure is known: apply weighted least squares;
• When variance changes with the mean: transform the response or use an appropriate GLM;
• For prediction intervals: explicitly model the conditional variance or use quantile regression.

robust_model = model.get_robustcov_results(cov_type="HC3")
print(robust_model.summary())

Robust standard errors do not correct for nonlinearity, omitted variables, incorrect dependence structures, or bad data. They only adjust the covariance estimates and leave the original OLS fitted values unchanged.

5. Clustering and Time Dependence

Tasks within the same team may share unobserved factors. Standard statistical estimation treats them as independent, leading to overly narrow confidence intervals. When processing or sampling occurs at the team level, cluster-robust covariance or hierarchical modeling should be considered:

clustered = model.get_robustcov_results(
    cov_type="cluster",
    groups=training["team_id"],
)

When the number of clusters is small, conventional asymptotic approximations may be unreliable and require small-sample corrections, randomization inference, or other specialized methods.

Temporal error components may also exhibit autocorrelation; methods like Newey–West/HAC rely on bandwidth selection and stationarity assumptions and cannot be treated as a one-size-fits-all fix. cov_type

6. Leveraging, Residuals, and Influence Are Different

• Leverage: Whether a point is far from others in the $X$ space;
• Residual: How much the observed value differs from the fitted value;
• Influence: How the fit changes when that point is removed.

Cook's distance combines residual and leverage into a single diagnostic metric, it is a screening tool, not an automatic rule for deletion. 4/n serves as an empirical threshold, not a rigid rule that declares any point "wrong" upon exceeding it.

Handling high-influence points:

1. Verify data collection and parsing processes;
2. Determine whether the point represents a valid member of the target population;
3. Assess the sensitivity of results to inclusion or exclusion;
4. Consider using robust regression or a more appropriate model;
5. Document the rules and how conclusions change.

Removing genuine outliers can cause the model to only apply to typical, non-extreme cases. It is essential to clearly define the new scope of applicability.

7. Uncertainty in Variance Inflation Factors

When columns are nearly linearly dependent, the model struggles to isolate individual contributions. This manifests as coefficient sensitivity to small data changes, large standard errors, and potential sign flips.

VIF:

$$ VIF_j = \frac{1}{1 - R_j^2}, $$

where $R_j^2$ is derived from using all other features to predict the $j$th column. VIF > 5 or >10 are merely empirical thresholds and cannot replace judgment based on the specific context.

Collinear features may still jointly provide good predictive performance; the primary issue lies in the interpretability of individual coefficients and the stability of extrapolation. Removing features, creating composite indicators, redesigning sampling, or applying regularization each carries distinct semantic consequences.

8. Missing Patterns and Selection Also Require Diagnosis

Regression libraries often silently remove rows with missing data using the complete cases approach. First, compare the cases entering the model with those that are excluded:

required = ["duration_minutes", "team_size", "resources_used"]
included = training[required].notna().all(axis=1)
print(included.mean())
print(training.groupby(included)["mission_type"].value_counts(normalize=True))

If the excluded cases are related to the outcome or to the population characteristics, the target population of the complete-case model has been altered. You must return to examining the missing data mechanism and the estimand, rather than the final nobs alone.

9. Conduct Sensitivity Analysis After Diagnosis

Compare:

• Linear versus spline or nonlinear specifications;
• Ordinary, robust, and clustered standard errors;
• Models with and without influential outliers;
• Different approaches to handling missing data;
• Pre-specified combinations of covariates;
• Time or subgroup slicing.

If the direction of conclusions changes dramatically with reasonable specification choices, report this sensitivity, do not simply select the specification that yields the most attractive results.

Common Misconceptions

• Non-normal residuals don’t mean OLS is invalid: It’s the target variable and the inference method that matter, not the residuals alone.
• Robust standard errors fix model specification errors: They do not correct for misspecified mean structures or confounding.
• Cook’s D exceeding a threshold requires deletion: It only signals influential observations that warrant further investigation, not a justification for removal.
• Multicollinearity implies poor prediction: It primarily undermines the interpretability of coefficients, not the predictive accuracy of the model.

Exercise

1. Construct nonlinear, heteroskedastic, and high-leverage data sets, and compare residual plots.
2. Compare ordinary, HC3, and cluster-standard error estimates for the same model.
3. Remove observations with high Cook's D values, compare coefficient estimates before and after, and document your findings.
4. Assess whether complete-case deletion alters the population composition.

Summary

Diagnosis is not about assigning a simple "pass/fail" label to a model; it's about identifying which specific conclusions are at risk. Issues like mean shift, heteroscedasticity, correlated errors, influential points, and multicollinearity require distinct corrective actions and sensitivity reporting.

The next lesson integrates prediction workflows with regularization into cross-validation, ensuring that scaling, missing data handling, and hyperparameter selection do not inadvertently peek at the final test data.