3.2 Relationships, Grouping, and Visualization Integrity: Correlation in a Graph Does Not Equal Mechanism

A plot shows higher success rates for longer tasks. That alone does not justify extending every task: duration and success may both vary by task type, and the groups may even use different definitions of success. Inspect those definitions and group composition before interpreting the slope.

Learning Objectives

• Select appropriate relationship diagrams and summary statistics for variable types;
• Distinguish between Pearson, Spearman, and nonlinear correlations;
• Identify confounding, aggregation bias, and Simpson’s paradox;
• Honestly represent uncertainty through denominator, interval, and visual encoding.

1. Handle Overlapping Points in Scatter Plots

Run the Python blocks in order with pandas 3.0, seaborn 0.13 and Matplotlib. The small dataset is for checking plot and calculation behavior, not for drawing population conclusions.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

frame = pd.DataFrame({
    "duration_minutes": [5, 10, 20, 30, 40, 60],
    "resource_cost": [8, 13, 19, 40, 44, 70],
    "team_size": [1, 1, 2, 2, 3, None],
    "mission_type": ["survey"] * 3 + ["repair"] * 3,
})
fig, ax = plt.subplots()
sns.scatterplot(
    data=frame, x="duration_minutes", y="resource_cost",
    hue="mission_type", alpha=0.5, s=30, ax=ax,
)
ax.set_xlabel("Duration (minutes)")
ax.set_ylabel("Resource cost (credits)")
plt.show()

When many points overlap, point density is obscured. Consider using:

• Transparency and smaller point sizes;
• Hexagonal binning or 2D density plots;
• Faceting by group;
• Sampling for display, while maintaining statistical analysis on the full dataset;
• A legend or caption specifying the sampling method.

Trend lines can mask local structure. Before fitting a linear model, inspect residuals and consider nonlinearity. For smooth curves, clearly state the method and bandwidth used.

2. Correlation Coefficients Address Different Types of Relationships

Pearson correlation measures linear association and is sensitive to extreme values. Spearman correlation measures association between ranks: strictly increasing transformations preserve it when ties do not change, while reversing the order of one variable changes its sign. Neither measure detects every nonlinear relationship.

columns = ["duration_minutes", "resource_cost", "team_size"]
pearson = frame[columns].corr(method="pearson", min_periods=3)
spearman = frame[columns].corr(method="spearman", min_periods=3)
observed = frame[columns].notna().astype(int)
pair_counts = observed.T @ observed
print(pair_counts)

u_shape = pd.DataFrame({"x": [-2, -1, 0, 1, 2], "y": [4, 1, 0, 1, 4]})
print(u_shape.corr(method="pearson").loc["x", "y"])  # 0.0
print(u_shape.corr(method="spearman").loc["x", "y"])  # 0.0

Check the following:

• Whether the correlation uses complete cases and whether missing values cause each pair of observations to differ;
• Whether repeated measurements are treated as independent points;
• Whether range restriction artificially lowers the correlation;
• Whether extreme values dominate the result;
• Whether a time trend causes both variables to rise together.

A correlation near zero does not rule out strong U-shaped, threshold, or group-specific inverse relationships.

The U-shaped fixture has a deterministic relation, $y=x^2$, yet both correlations are zero. In the earlier table, duration–cost uses six pairs while pairs involving team size use five. pandas correlation drops missing values pair by pair; a constant column produces an undefined coefficient. min_periods=3 is a demonstration threshold, not evidence that three points suffice for inference.

3. Grouped Comparisons to Show Distributions and Sample Sizes

Box plots display the median, quartiles, and whiskers defined by rules, without directly indicating "outlier truths." They also cannot be relied upon alone to determine statistical significance based solely on non-overlapping boxes.

fig, ax = plt.subplots()
sns.boxplot(data=frame, x="mission_type", y="duration_minutes", ax=ax)
sns.stripplot(data=frame, x="mission_type", y="duration_minutes", color="black", ax=ax)
ax.set_xlabel("Mission type")
ax.set_ylabel("Duration (minutes)")
plt.show()

For small sample sizes, raw data points can be overlaid; for large samples, even minor differences may be statistically significant, so effect size and business-scale context should be reported together. When distributions are skewed, consider using violin plots or ECDFs, though kernel density shapes depend heavily on smoothing parameters.

With seaborn’s default whis=1.5, whiskers reach the most extreme observed values within the 1.5-IQR fences; they do not necessarily end at the fences themselves. A violin plot smooths a density estimate, while an ECDF steps through observed cumulative proportions and needs no bandwidth. See the seaborn boxplot API.

4. Simpson's Paradox Reminds You to Check the Composition

Within each subgroup, the new approach has a higher success rate than the old one. Yet, when aggregated across all groups, the overall success rate may be lower, because the new approach is assigned more difficult tasks.

Simple tasks: new 90/100, old 80/100
Difficult tasks: new 20/100, old 1/10

The weight given to subgroup comparisons versus overall comparisons differs. Grouping is not an automatic solution; you should determine which variables to control for, and which act as mediators or colliders, based on the causal question at hand. Exploratory data analysis (EDA) can reveal stratification effects or reversal patterns. Causal reasoning will be covered in depth in Chapter 8.

Here, new wins within both strata: 90% versus 80%, and 20% versus 10%. Overall, new is 110/200 = 55%, while old is 81/110 ≈ 73.64%. Difficult tasks account for 50% of new assignments but only about 9.09% of old assignments. Those different weights produce the reversal.

5. Be Cautious of Common Trends in Time Series

Two time-series cumulative metrics might show a correlation close to 1, even if they have no direct relationship. Start by plotting the original sequences before analyzing further:

• Detrend or use differencing;
• Apply seasonal decomposition;
• Examine lagged relationships;
• Investigate structural breaks;
• Account for autocorrelation, which can reduce effective sample size.

Simply shifting one series backward to find the highest correlation performs numerous implicit comparisons. If inferential conclusions are intended, a pre-defined window or correction for selection bias must be applied.

6. Graphical Encoding Can Amplify or Minimize Differences

Axes

Bar chart lengths should typically start at zero, otherwise small differences are exaggerated. Line charts showing trends may use non-zero ranges, but the axis must be clearly labeled to avoid misleading truncation.

Area and Three-Dimensional Representations

Bubble area should scale proportionally with the value, not the radius. Three-dimensional bar charts introduce perspective and occlusion, which generally hinder accurate comparisons.

Color

Use a continuous color gradient for ordered data. For positive and negative deviations, use diverging color scales centered on a meaningful baseline. Use discrete colors for categorical data. Consider color vision deficiencies and avoid relying solely on color to convey critical information.

Dual Axes

Dual Y-axes can create a false sense of synchronization through arbitrary scaling. Prefer faceting, standardization of indices, or direct labeling. If dual axes are necessary, clearly define the scales and provide a justification.

7. Uncertainty and the Denominator Must Appear in the Chart

Group success rates can be displayed together:

Point estimate + confidence interval + sample size n

The interpretation of a confidence interval depends on the estimation method and sampling assumptions. For clustered, repeated-user, or time-dependent data, a simple binomial independent interval may be too narrow.

Error bars must specify whether they represent standard deviation, standard error, or confidence interval. These are not interchangeable: standard deviation describes the dispersion of observed values, while standard error describes the sampling variability of the estimate.

8. Multiple Explorations Require Traceability

After reviewing dozens of plots, selecting the most striking one risks amplifying chance patterns. While exploratory data analysis (EDA) allows for free-ranging investigation, subsequent validation should:

• Be reproducible on independent data or in a new time window;
• Document the variables and slices examined;
• Clearly distinguish between exploratory and confirmatory conclusions;
• Apply corrections for multiple testing or use hierarchical modeling;
• Not treat the p-values from filtered analyses as results from pre-registered studies.

9. Automated Reporting Is an Index, Not a Conclusion

Automated profiling can quickly identify distributions, missing values, and correlations, but it also:

• Generates a high volume of false alarms in high-dimensional data;
• Lacks understanding of business-level granularity and denominator logic;
• Provides no context regarding time windows, clustering, or sampling design;
• May consume significant memory and processing time.

Use it to identify columns that warrant further investigation, then return to data contracts and generation processes. Tool versions and dependencies should be selected at implementation time, rather than treating a package name as a permanent standard.

Common Misconceptions

• Non-overlapping boxes imply significant differences: Graphs are not statistical tests.
• Spearman's correlation detects all nonlinear relationships: It primarily captures monotonic relationships.
• Grouping by more variables always brings you closer to the truth: Conditioning on colliders can introduce bias; adjusting for mediators may change the effect being estimated.
• Error bars always represent data variability: SD, SE, and CI answer fundamentally different questions.

Exercise

1. Construct data with a Pearson correlation near zero but exhibiting a clear U-shaped relationship, and plot it.
2. Demonstrate a dataset where aggregated trends and stratified trends move in opposite directions.
3. Modify a bar chart with a truncated Y-axis to present the data more honestly and transparently.
4. Enhance a success rate chart by including the denominator, interval-based methods, and a statement about the assumption of repeated user exposure.

Summary

A relationship plot shows associations under specific sampling, grouping, and visual encoding conditions. Correlation coefficients, trend lines, and box plots all require denominators, structural context, and uncertainty interpretation. Exploratory Data Analysis (EDA) identifies patterns worth validating, not causal mechanisms.

The next chapter shifts to SQL: before writing window functions, first establish the granularity of a row, the join cardinality, and the denominator for metrics, otherwise the query will precisely reproduce erroneous calculations.