3.1 EDA and Distribution Diagnostics: First Understand a Row of Data, Then Interpret a Chart

After cleaning, the dataset contains 100,000 rows and 50 columns. The dashboard in the analytics system can generate a histogram in under a second, but it won’t tell you whether each row represents a single task, a retry, or a daily summary. Without clear granularity, any mean value might be calculated with the wrong unit.

Learning Objectives

• Begin exploratory data analysis (EDA) by examining the data generation process, observational units, and coverage scope;
• Select appropriate summary statistics for numeric, categorical, rate, and time-based variables;
• Identify common data artifacts such as histogram binning, long tails, truncation, and missing values that can create misleading patterns;
• Translate exploratory findings into testable hypotheses rather than drawing definitive conclusions.

1. First, Write a Data Specification Card

Before drawing any diagrams, answer these questions:

What does each row represent?
What is the primary key or unit of repetition?
What time, geographic, and demographic scope does it cover?
How was the data sampled or filtered?
Which fields are computed after collection?
How are missing or late entries handled?
What cleaning steps have been applied in this version?

If retries produce several rows for one mission_id, event-level failure counts and task-level failure counts answer different questions. For daily summaries, equal weighting describes an average day; weighting by task count describes an average task. Name the unit before calculating the mean.

Exploratory Data Analysis (EDA) should examine both raw and curated data: raw data helps identify issues in collection, parsing, and missingness; curated data enables understanding of the final analysis dataset. Focusing solely on data after cleaning will obscure which rules removed specific records.

2. Establish Coverage and Granularity Checks

The blocks run in order with pandas 3.0 and Matplotlib. This eight-event fixture contains four tasks, so event summaries must not be labeled as task summaries.

import pandas as pd

frame = pd.DataFrame({
    "event_id": [f"e{i}" for i in range(8)],
    "mission_id": [f"M{i // 2}" for i in range(8)],
    "occurred_at": pd.to_datetime(
        ["2026-08-01T10:00:00Z"] * 4 + ["2026-08-03T10:00:00Z"] * 4
    ),
    "mission_type": ["survey", "repair"] * 4,
    "duration_minutes": [5, 10, 12, 20, 40, 60, 120, None],
})
overview = {
    "rows": len(frame),
    "missions": frame["mission_id"].nunique(),
    "min_time": frame["occurred_at"].min(),
    "max_time": frame["occurred_at"].max(),
    "missing_event_ids": frame["event_id"].isna().sum(),
    "duplicate_event_ids": frame["event_id"].dropna().duplicated().sum(),
}
print(overview)

The ratio of row count, entity count, and event count can quickly reveal granularity issues. Group by source, date, schema version, and status to identify any missing partitions or sudden spikes.

3. Numerical Variables: Center, Dispersion, and Shape

duration = frame["duration_minutes"]
summary = duration.describe(percentiles=[0.01, 0.1, 0.5, 0.9, 0.99])
summary["missing_rate"] = duration.isna().mean()
print(summary)

The mean is suitable for expressing total distribution across units, but it is sensitive to long tails. The median describes a typical value in the data, yet it does not retain information about the total sum. Standard deviation relies on both the mean and squared deviations, while the interquartile range (IQR) is more robust to outliers.

When reporting percentiles, it is essential to specify whether the analysis includes failed records, zero values, truncated values, or imputed missing values. Reproduction requires the dataset version and the exact inclusion rules.

4. Histograms Are Determined by Binning

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
duration.plot.hist(bins=20, ax=axes[0], title="20 bins")
duration.plot.hist(bins=80, ax=axes[1], title="80 bins")
for axis in axes:
    axis.set_xlabel("Duration (minutes)")
plt.tight_layout()
plt.show()

Wide bins can obscure multiple peaks, while overly narrow bins amplify random noise. Try at least several different bin widths, and consider business boundaries, measurement precision, and sample size when making your choice.

Only seven durations are observed here. The 80-bin plot mostly displays empty space; it demonstrates bin sensitivity, not evidence for 80 separate regimes. Matplotlib excludes missing values from this histogram, so report the missing rate alongside it.

For positive values spanning several orders of magnitude, logarithmic coordinates may be appropriate, though you must clearly address how zero and negative values are handled. Logarithmic scales highlight relative changes, not absolute differences, and should never be misrepresented in the legend as linear.

5. Long Tail and Truncation Reveal Generation Process

Duration caps cluster tightly around exactly 60 or 120 minutes, potentially due to timeout rules. Accumulations of integers at 999 or 9999 may indicate upper-bound encoding. A sudden shift in precision at a certain point in time could point to schema updates or device version upgrades.

These patterns should not be smoothed out immediately. Instead, split the data along the following dimensions:

• source / device model
• schema version
• time and release batch
• region, channel, task type
• whether cleaned or truncated

Bimodal distributions may result from the mixing of two distinct user groups, or from batch processing errors. Visualizations propose hypotheses, and source metadata help validate them.

6. Viewing Both Counts and Proportions for Categorical Variables

counts = frame["mission_type"].value_counts(dropna=False)
shares = frame["mission_type"].value_counts(
    dropna=False, normalize=True
)
category_summary = pd.DataFrame({"count": counts, "share": shares})
print(category_summary)

Looking only at percentages hides small sample sizes, while focusing solely on counts makes it difficult to compare periods of different scales. Charts should simultaneously display both the numerator and denominator.

For high-cardinality categories, avoid plotting thousands of individual bars. Instead, show only the top-N categories and "Others," though the "Others" group may have significant internal variation, complete category breakdowns must still be preserved in the analysis table.

Missing, unknown, and not applicable values should be separated. Excluding nulls and then re-normalizing the proportions can make known category shares appear stable, masking a decline in data quality.

7. Ratios Must Return to Numerator and Denominator

A single line success_rate=0.8 could originate from 4/5 or from 8000/10000, these have fundamentally different levels of uncertainty.

In EDA, we retain:

success_count
attempt_count
success_rate = success_count / attempt_count

When comparing group rates, first visualize the denominator distribution and examine how task difficulty, exposure, and observation window vary across groups. The average rate across rows typically does not equal the total successes divided by total attempts; the choice between macro-averaging and micro-averaging depends on the specific problem.

For example, rates 9/10 and 1/100 have a macro-average of 45.5%, while the pooled rate is 10/110, about 9.09%. Neither is an arithmetic error: the first weights groups equally, the second weights attempts equally. A group with zero attempts has an undefined rate, not a zero rate.

8. Distinguish Missing Data from True Zeros in Time Series

On some days, there may be no records, this could mean no events occurred, or it could simply indicate that the partition has not arrived. Before performing resampling and filling in zeros, it's essential to verify the business semantics.

observed = (
    frame.assign(day=frame["occurred_at"].dt.floor("D"))
    .groupby(["day", "mission_type"], dropna=False)
    .size().rename("observed_events").reset_index()
)
coverage = pd.MultiIndex.from_product(
    [pd.date_range("2026-08-01", "2026-08-03", tz="UTC"), ["survey", "repair"]],
    names=["day", "mission_type"],
).to_frame(index=False)
coverage["partition_complete"] = [True, True, True, False, True, True]
daily = coverage.merge(observed, on=["day", "mission_type"], how="left", validate="1:1")
daily["events"] = daily["observed_events"].fillna(0).astype("Int64").where(
    daily["partition_complete"]
)
print(daily)

The number of events should also be considered in conjunction with active users, device online counts, or exposure metrics. A rise in total volume might simply reflect expanded coverage, not necessarily an improvement in per-unit performance.

The coverage fixture marks August 2 survey data complete, so its absent rows become a verified zero. Repair data for that day is incomplete, so events remains missing. In production, completeness comes from the ingestion manifest, not from guessing at the event table. Resampling counts can produce zeros for empty bins automatically; counting a column also skips null IDs. This example uses row counts and an explicit coverage grid. A business day in a local timezone also needs conversion before daily bucketing.

9. EDA Output Is a List of Questions

Record format:

Observation: The temperature distribution of v3 devices is approximately 8°C higher than that of v2 devices.
Possible explanations: Calibration differences, deployment region variations, actual hardware changes.
Data needed: Device region, calibration records, version transition dates.
Verification method: Pairwise comparison under the same region and operating conditions.
Current conclusion: Not determined; versions cannot be merged and compared directly.

Separating observations, explanations, and conclusions helps avoid treating coincidental patterns as established facts during the exploration phase.

Common Misconceptions

• You can't do EDA before cleaning: Raw profiling is actually a key way to identify what data cleaning is needed.
• Mean vs. median: which is more representative?: They answer different questions.
• The shape of a histogram defines the data uniquely: Both binning and coordinate scaling affect how the data appears.
• Ratios can be compared without a denominator: Rates from small samples are more volatile, and their weights differ significantly.

Exercise

1. Write a data card for an event table, clearly specifying the row granularity and the unit of repetition.
2. Create three different binning schemes for the same distribution and document which structural patterns remain consistent.
3. Construct ratio data where macro-averaged and micro-averaged results yield opposite conclusions.
4. Identify "missing date" gaps in a time series and determine whether they represent zero events or missing partitions.

Summary

EDA first interrogates the process of data generation, then examines the numerical values. Granularity, coverage, missingness, denominator, and version determine whether the shapes in the plot are interpretable. The output of exploration should be a set of hypotheses with evidence requirements, not definitive statements following a visually appealing chart.

The next lesson extends single-variable analysis to relational diagrams and addresses issues such as correlation, confounding, grouping paradoxes, and the expression of uncertainty.