2.3 Reproducible Data Cleaning and Quality Gates: Every Change Is Explainable and Rollable

In the Data Forecasting Lab, a manually edited report fails to reproduce, archival staff demand that every data cleaning rule record its version, evidence, and rollback path.

Fixing a CSV file by hand is fast, but it can't answer questions three days later: which rows were modified, which rule version was applied, what was the original value, and does rerunning yield the same result. The professional boundary of data cleaning isn't defined by API count; it's defined by reproducibility, auditability, and verifiability.

Learning Objectives

• Design the cleaning steps as versioned transformations;
• Isolate bad records using quarantine rather than silently discarding them;
• Establish quality gates using counts, conservation laws, and attribute checks;
• Decouple fit parameters from transform execution to prevent leakage.

1. Input immutability, output versioning

raw/source_version
  + code_version
  + rule_config_version
  + reference_data_version
  → curated/output_version

Do not modify the raw input data in place. Each run must record the input manifest, code commit or hash, configuration, start and end timestamps, and output checksum. Users may request that the repository not commit changes, this does not affect the general concept of versioning in the material. The version of the running code must remain identifiable.

Cleaning functions should ideally follow the pattern of "input DataFrame → new DataFrame + quality report," minimizing implicit global state.

2. Each Rule Returns a Result and a Reason

from dataclasses import dataclass
import pandas as pd

@dataclass(frozen=True)
class CleanResult:
    accepted: pd.DataFrame
    quarantined: pd.DataFrame
    metrics: dict[str, int | float]

def validate_temperature(frame: pd.DataFrame) -> CleanResult:
    invalid = ~frame["temperature_c"].between(-50, 150)
    quarantine = frame.loc[invalid].assign(
        rejection_code="TEMPERATURE_OUT_OF_RANGE"
    )
    accepted = frame.loc[~invalid].copy()
    return CleanResult(
        accepted=accepted,
        quarantined=quarantine,
        metrics={
            "input_rows": len(frame),
            "accepted_rows": len(accepted),
            "quarantined_rows": len(quarantine),
        },
    )

The example isolates an entire row for invalid temperature values; in a real system, the affected field might simply be left empty. The strategy is determined by downstream requirements, but silent deletion must be avoided.

The isolated area requires access controls at the same security level, as it often contains raw sensitive data. Retention periods, reprocessing flags, and ownership assignments should be established to prevent it from becoming permanent garbage.

3. Pipeline Steps Must Have a Clear Sequence

A transparent execution order:

1. Decoding and schema parsing;
2. Uniform explicit handling of missing values;
3. Logical type conversion;
4. Field-level and cross-field validity validation;
5. Deterministic deduplication;
6. Standardization and derivation of new fields;
7. Statistical imputation or model-based transformations;
8. Output contract definition and reconciliation.

The order directly affects outcomes. If invalid 999 temperature values are used in median calculations before being isolated, the imputation parameters become corrupted. Dependencies between rules must be explicitly declared in the pipeline definition.

4. Line Count Verification Is an Equation

If every input ultimately ends up in either accepted or quarantined:

$$ N_{input} = N_{accepted} + N_{quarantined}. $$

If deduplication is also performed:

$$ N_{input} = N_{accepted} + N_{quarantained} + N_{duplicates}, $$

provided that the three sets are mutually exclusive. If duplicates may also be invalid, a classification priority must be defined or multi-label counting must be used, equations cannot be blindly applied in such cases.

Numerical values such as monetary amounts, event counts, and entity counts can also be validated through conservation checks. When data cleaning alters numeric values, the original input values and a summary of the differences should be preserved and reported.

5. Quality Gate Comparison of Absolute Rules and Baselines

Hard Constraints

• Primary keys must not be null;
• Schema must remain compatible;
• Target partitions must not be missing;
• Referential keys must satisfy integrity constraints.

Statistical Monitoring

• Row counts compared against seasonal baselines;
• Missing rate trends;
• New or disappearing categories;
• Changes in percentiles and distribution drift;
• Proportion of records quarantined or retried.

Failure of hard constraints can block deployment entirely; deviations in statistical metrics may trigger alerts requiring manual verification. Blocking all changes leads to frequent false positives, while allowing all changes undermines the value of quality gates.

6. fit and transform Stored Separately

from dataclasses import dataclass

@dataclass(frozen=True)
class TemperatureImputer:
    median: float

    @classmethod
    def fit(cls, training: pd.Series) -> "TemperatureImputer":
        return cls(median=float(training.median()))

    def transform(self, values: pd.Series) -> pd.Series:
        return values.fillna(self.median)

Store parameter values, training data versions, feature definitions, and code versions. If median calculations are recomputed in production, it introduces bias between training and serving; historical replay cannot reproduce the original results.

Non-machine learning components also require versioning of parameters. For example, anomaly thresholds, exchange rate tables, and regional mappings evolve over time.

7. Idempotence and Determinism

An ideal cleaning transformation produces the same output given the same input and version. Verify the following:

• The tie-breaker for deduplication is unique;
• Explicit injection of current time or random numbers is avoided;
• External dimension tables use a fixed version;
• Floating-point order in parallel aggregation does not affect the final result;
• Output sorting is part of the contract.

Some transformations should also be idempotent:

$$ clean(clean(x)) = clean(x). $$

Not all transformations are naturally idempotent, examples include appending a suffix on each run or re-normalizing data. Property-based testing can quickly detect such issues.

8. Testing from Examples to Invariants

Unit Cases

Cover valid, boundary, missing, unknown enum, and parsing failure scenarios.

Property-Based Testing

• Output keys are unique;
• No additional rows are introduced unless explicitly specified;
• Span or temporal order is not reversed;
• Double-cleaning produces identical results;
• The accepted and quarantine sets do not overlap.

Golden Dataset

Store small, representative inputs and expected outputs for rule change diffing. Do not merely assert file hashes; when outputs change, show specific rows, columns, and the reasons behind the changes.

Shadow Run

New rules are run in parallel without being published first, comparing row counts, metrics, and downstream impacts before switching versions.

9. What Changed in the Release Report

run_id: clean-2026-08-03-01
input: 1,000,000
accepted: 982,140
quarantine: 12,300
duplicates: 5,560
changed values:
  normalized location: 320,441
  imputed temperature: 8,201
top rejection reasons:
  invalid timestamp: 7,104
  impossible temperature: 5,196

Sudden shifts in proportions should drill down to source, partition, schema version, and rule level. The report serves operations staff as well as analysts who need to understand differences between data versions.

Common Misconceptions

• A re-run of a cleaning script is enough for reproducibility: The external reference tables, parameters, and input versions must also be fixed to ensure true reproducibility.
• Deleting bad rows directly is the cleanest approach: This loses root cause analysis, audit trails, and opportunities for corrective actions.
• Testing a few example rows is sufficient: Data rules require invariants, boundary conditions, and replay of real-world data distributions.
• Statistical drift is always a data fault: It may instead reflect genuine business changes; responses must be tiered and context-aware.

Exercise

1. Modify a script that mutates a DataFrame in-place to use the CleanResult interface.
2. Write reconciliation equations for missing, anomalous, and duplicate sets and handle their intersections.
3. Write clean(clean(x)) == clean(x) attribute tests for a data cleaning function.
4. Design a shadow run report comparing new and old rule behavior.

Summary

Reliable data cleaning is versioned data transformation: the original input remains unchanged, rule execution has clear reasoning, bad records can be isolated, outputs satisfy invariants, and parameters are constrained to fit only within allowed data ranges. Only under these conditions does data cleaning transition from a one-off notebook operation to an auditable production process.

The next chapter uses the cleaned data for exploratory data analysis (EDA). The first step is still not about producing visually appealing charts, but rather about clearly identifying whether each visualization is intended to examine a distribution, reveal relationships, or explore the data generation process.