Vol 12: Data Processing and Data Science

This volume moves from data lifecycle, cleaning, and analysis into inference, features, distributed processing, governance, and privacy. Charts present evidence; they cannot take responsibility for the sampling frame, time boundary, or error model.

Prerequisites

• Be able to read and write Python and understand SQL queries, joins, and aggregates. Consult Volume 1: Programming Basics and Volume 5: Databases and Data Systems.
• Statistics and regression use basic probability, algebra, and matrices from Volume 10: Mathematics.

Coverage

Chapters 1–4 follow data from creation to analysis. Chapters 5–8 cover inference, regression, features, and causal questions; Chapter 9 covers distributed processing. Chapters 10–14 address ethics, governance, lineage, organizational coordination, and privacy.

Chapter 1

• 1.1 Data Products, Event Semantics, and Contracts: Data Must Be Interpretable from the Moment It Is Generated
• 1.2 Ingest, Store, and Retain: Building a Replayable Data Pipeline

Chapter 2

• 2.1 Data Profiling, Types, and Missingness: Explain Missing Values Before Imputing Them
• 2.2 Anomalies, Duplicates, and Entity Consistency: Investigate Suspicious Records Before Deleting Them
• 2.3 Reproducible Data Cleaning and Quality Gates: Explain and Roll Back Every Change

Chapter 3

• 3.1 EDA and Distribution Diagnostics: First Understand a Row of Data, Then Interpret a Chart
• 3.2 Relationships, Grouping, and Visualization Integrity: Correlation in a Graph Does Not Equal Mechanism

Chapter 4

• 4.1 Analytical Grain, Join Cardinality, and Metric Denominators: Prevent Double Counting in SQL
• 4.2 Window Functions and Time Series SQL: Sorting, Frames, and Missing Dates
• 4.3 CTE, Grouping Sets, and Query Validation: Treat Analytical SQL Like a Program

Chapter 5

• 5.1 Estimators, Confidence Intervals, and p-Values: Conclusions Depend on Sampling
• 5.2 Test Selection, Power, and Multiple Comparisons: Plan the Analysis Before the Results

Chapter 6

• 6.1 OLS, Coefficients, and the Design Matrix: Exactly What Is a Linear Model Estimating?
• 6.2 Residuals, Heteroscedasticity, and Influential Points: Diagnosing Which Failure Mode to Address
• 6.3 Regularization, Cross-Validation, and Prediction Intervals: Model Selection Can't Look at the Answers

Chapter 7

• 7.1 Point-in-Time Correctness and Feature Leakage: Use Information Available at Prediction Time
• 7.2 Numerical, Categorical, and Temporal Encoding: Transformations Belong to the Model
• 7.3 Feature Selection and Production Consistency: From Candidate Columns to Monitorable Interfaces

Chapter 8

• 8.1 Sampling Frame, Inclusion Probability, and Weights: Who Does the Sample Represent?
• 8.2 Randomized Experiments, ITT, and Interference: Assignment Does Not Equal Exposure
• 8.3 DAGs and Observational Identification: Control Variables Can Also Introduce Bias

Chapter 9

• 9.1 From Single-Machine Overflow to Partitioned Computing: First Shrink the Working Set, Then Add Machines
• 9.2 Partitioning, Shuffle, and Data Skew: Understanding Distributed Batch Processing Plans
• 9.3 Event Time, Watermarks, and End-to-End Semantics: How Stream Processing Handles Late Arrivals and Replay

Chapter 10

• 10.1 From Use Case to Harm Path: Data Projects Must First Prove They Are Worth Doing
• 10.2 Fairness Metrics and Group Evaluation: Which Errors Harm Whom?
• 10.3 Human Oversight, Appeals, and Incident Response: Responsibility Can't Stop at "Model Recommendation"

Chapter 11

• 11.1 Governance Responsibilities and Decision Authority: Every Rule Must Have an Explanation, an Executor, and an Exception Approval Process
• 11.2 Data Quality SLOs and Incident Response: What Should the System Do After a Check Fails

Chapter 12

• 12.1 Data Assets, Terminology, and Metadata Catalogs: Enabling Correct Use of Search Results
• 12.2 Runtime Lineage and Impact Analysis: Tracing an Incorrect Metric Back to a Specific Version

Chapter 13

• 13.1 Data Mesh's Applicability Conditions: Address Organizational Issues Before Distributing Responsibility
• 13.2 Data Products, Platforms, and Federated Governance: Autonomy Must Be Interoperable

Chapter 14

• 14.1 Privacy Engineering and Data Mapping: Start by Reducing Unnecessary Processing
• 14.2 Pseudonymisation, Anonymisation, and Linkage Attacks: Removing Names Is Not Enough
• 14.3 Differential Privacy, Sensitivity, and Budget: Noise Is Just Part of the Mechanism
• 14.4 Access, Retention, and Compliance Operations: Permissions Will Eventually Expire, and Copies Must Be Found

Outcomes

• Explain what a row represents and check missing values, duplicates, join cardinality, and metric denominators.
• Use sampling, time boundaries, and validation design to state the scope of a statistical or predictive conclusion.
• Establish replay, quality checks, and lineage for a pipeline, and assign responsibility for access, retention, and failures.

How to use this map

Bring a manageable dataset with a known source to Chapters 1–4. Continue to Chapters 5–8 for inference or prediction, and use the later chapters when the analysis needs to operate over time. Check the denominator and data scope before plotting. Even an excellent color palette cannot rescue an order counted twice.