3.2 Relationships, Grouping, and Visualization Integrity: Correlation in a Graph Does Not Equal Mechanism

A scatter plot shows a positive correlation between task duration and success rate. The intelligence officer wants to extend all tasks based on this trend, but the administrator asks you to first color-code by task type: high-difficulty tasks are longer and have a success rate governed by a separate rule. The apparent slope in the aggregate might simply reflect the composition of the task group, not a true underlying relationship.

Learning Objectives

• Select appropriate relationship diagrams and summary statistics for variable types;
• Distinguish between Pearson, Spearman, and nonlinear correlations;
• Identify confounding, aggregation bias, and Simpson’s paradox;
• Honestly represent uncertainty through denominator, interval, and visual encoding.

1. Handle Overlapping Points in Scatter Plots

import seaborn as sns

sns.scatterplot(
    data=frame,
    x="duration_minutes",
    y="resource_cost",
    hue="mission_type",
    alpha=0.25,
    s=20,
)

When many points overlap, point density is obscured. Consider using:

• Transparency and smaller point sizes;
• Hexagonal binning or 2D density plots;
• Faceting by group;
• Sampling for display, while maintaining statistical analysis on the full dataset;
• A legend or caption specifying the sampling method.

Trend lines can mask local structure. Before fitting a linear model, inspect residuals and consider nonlinearity. For smooth curves, clearly state the method and bandwidth used.

2. Correlation Coefficients Address Different Types of Relationships

Pearson correlation measures linear association and is sensitive to outliers. Spearman correlation, based on ranks, assesses monotonic relationships and is more robust to scale transformations, but it is not a universal solution for nonlinear dependencies.

columns = ["duration_minutes", "resource_cost", "team_size"]
pearson = frame[columns].corr(method="pearson")
spearman = frame[columns].corr(method="spearman")

Check the following:

• Whether the correlation uses complete cases and whether missing values cause each pair of observations to differ;
• Whether repeated measurements are treated as independent points;
• Whether range restriction artificially lowers the correlation;
• Whether extreme values dominate the result;
• Whether a time trend causes both variables to rise together.

A correlation near zero does not rule out strong U-shaped, threshold, or group-specific inverse relationships.

3. Grouped Comparisons to Show Distributions and Sample Sizes

Box plots display the median, quartiles, and whiskers defined by rules, without directly indicating "outlier truths." They also cannot be relied upon alone to determine statistical significance based solely on non-overlapping boxes.

ax = sns.boxplot(
    data=frame,
    x="mission_type",
    y="duration_minutes",
)
ax.set_xlabel("Mission type")
ax.set_ylabel("Duration (minutes)")

For small sample sizes, raw data points can be overlaid; for large samples, even minor differences may be statistically significant, so effect size and business-scale context should be reported together. When distributions are skewed, consider using violin plots or ECDFs, though kernel density shapes depend heavily on smoothing parameters.

4. Simpson's Paradox Reminds You to Check the Composition

Within each subgroup, the new approach has a higher success rate than the old one. Yet, when aggregated across all groups, the overall success rate may be lower, because the new approach is assigned more difficult tasks.

Simple tasks: new 90/100, old 80/100
Difficult tasks: new 20/100, old 1/10

The weight given to subgroup comparisons versus overall comparisons differs. The solution isn't always to split the data, instead, you should determine which variables to control for, and which act as mediators or confounders, based on the causal question at hand. Exploratory data analysis (EDA) can reveal stratification effects or reversal patterns. Causal reasoning will be covered in depth in Chapter 8.

5. Be Cautious of Common Trends in Time Series

Two time-series cumulative metrics might show a correlation close to 1, even if they have no direct relationship. Start by plotting the original sequences before analyzing further:

• Detrend or use differencing;
• Apply seasonal decomposition;
• Examine lagged relationships;
• Investigate structural breaks;
• Account for autocorrelation, which can reduce effective sample size.

Simply shifting one series backward to find the highest correlation performs numerous implicit comparisons. If inferential conclusions are intended, a pre-defined window or correction for selection bias must be applied.

6. Graphical Encoding Can Amplify or Minimize Differences

Axes

Bar chart lengths should typically start at zero, otherwise small differences are exaggerated. Line charts showing trends may use non-zero ranges, but the axis must be clearly labeled to avoid misleading truncation.

Area and Three-Dimensional Representations

Bubble area should scale proportionally with the value, not the radius. Three-dimensional bar charts introduce perspective and occlusion, which generally hinder accurate comparisons.

Color

Use a continuous color gradient for ordered data. For positive and negative deviations, use diverging color scales centered on a meaningful baseline. Use discrete colors for categorical data. Consider color vision deficiencies and avoid relying solely on color to convey critical information.

Dual Axes

Dual Y-axes can create a false sense of synchronization through arbitrary scaling. Prefer faceting, standardization of indices, or direct labeling. If dual axes are necessary, clearly define the scales and provide a justification.

7. Uncertainty and the Denominator Must Appear in the Chart

Group success rates can be displayed together:

Point estimate + confidence interval + sample size n

The interpretation of a confidence interval depends on the estimation method and sampling assumptions. For clustered, repeated-user, or time-dependent data, a simple binomial independent interval may be too narrow.

Error bars must specify whether they represent standard deviation, standard error, or confidence interval. These are not interchangeable: standard deviation describes the dispersion of observed values, while standard error describes the sampling variability of the estimate.

8. Multiple Explorations Require Traceability

After reviewing dozens of plots, selecting the most striking one risks amplifying chance patterns. While exploratory data analysis (EDA) allows for free-ranging investigation, subsequent validation should:

• Be reproducible on independent data or in a new time window;
• Document the variables and slices examined;
• Clearly distinguish between exploratory and confirmatory conclusions;
• Apply corrections for multiple testing or use hierarchical modeling;
• Not treat the p-values from filtered analyses as results from pre-registered studies.

9. Automated Reporting Is an Index, Not a Conclusion

Automated profiling can quickly identify distributions, missing values, and correlations, but it also:

• Generates a high volume of false alarms in high-dimensional data;
• Lacks understanding of business-level granularity and denominator logic;
• Provides no context regarding time windows, clustering, or sampling design;
• May consume significant memory and processing time.

Use it to identify columns that warrant further investigation, then return to data contracts and generation processes. Tool versions and dependencies should be selected at implementation time, not hardcoded into a permanent package name.

Common Misconceptions

• Non-overlapping boxes imply significant differences: Graphs are not statistical tests.
• Spearman's correlation detects all nonlinear relationships: It primarily captures monotonic relationships.
• Grouping by more variables always brings you closer to the truth: Controlling for mediators or confounding factors can introduce bias.
• Error bars always represent data variability: SD, SE, and CI answer fundamentally different questions.

Exercise

1. Construct data with a Pearson correlation near zero but exhibiting a clear U-shaped relationship, and plot it.
2. Demonstrate a dataset where aggregated trends and stratified trends move in opposite directions.
3. Modify a bar chart with a truncated Y-axis to present the data more honestly and transparently.
4. Enhance a success rate chart by including the denominator, interval-based methods, and a statement about the assumption of repeated user exposure.

Summary

A relationship plot shows associations under specific sampling, grouping, and visual encoding conditions. Correlation coefficients, trend lines, and box plots all require denominators, structural context, and uncertainty interpretation. Exploratory Data Analysis (EDA) identifies patterns worth validating, not causal mechanisms.

The next chapter shifts to SQL: before writing window functions, first establish the granularity of a row, the join cardinality, and the denominator for metrics, otherwise the query will precisely reproduce erroneous calculations.