2.3 Reproducible Data Cleaning and Quality Gates: Explain and Roll Back Every Change

Fixing a CSV file by hand is fast, but it can't answer questions three days later: which rows were modified, which rule version was applied, what was the original value, and does rerunning yield the same result. The professional boundary of data cleaning isn't defined by API count; it's defined by reproducibility, auditability, and verifiability.

Learning Objectives

• Design the cleaning steps as versioned transformations;
• Isolate bad records using quarantine rather than silently discarding them;
• Establish quality gates using counts, conservation laws, and property checks;
• Decouple fit parameters from transform execution to prevent leakage.

1. Input immutability, output versioning

raw/source_version
  + code_version
  + rule_config_version
  + reference_data_version
  → curated/output_version

Keep raw inputs immutable. Record the input manifest, an identifiable code revision or content hash, configuration, start and end times, and output checksums for every run.

Cleaning functions should ideally follow the pattern of "input DataFrame → new DataFrame + quality report," minimizing implicit global state.

2. Each Rule Returns a Result and a Reason

from dataclasses import dataclass
import numpy as np
import pandas as pd

@dataclass(frozen=True)
class CleanResult:
    accepted: pd.DataFrame
    quarantined: pd.DataFrame
    metrics: dict[str, int | float]

def validate_temperature(frame: pd.DataFrame) -> CleanResult:
    values = frame["temperature_c"]
    valid = values.between(-50, 150).fillna(False)
    reasons = pd.Series("TEMPERATURE_OUT_OF_RANGE", index=frame.index)
    reasons = reasons.mask(values.isna(), "TEMPERATURE_MISSING")
    quarantine = frame.loc[~valid].assign(rejection_code=reasons.loc[~valid])
    accepted = frame.loc[valid].copy()
    return CleanResult(
        accepted=accepted,
        quarantined=quarantine,
        metrics={
            "input_rows": len(frame),
            "accepted_rows": len(accepted),
            "quarantined_rows": len(quarantine),
        },
    )

sample = pd.DataFrame({
    "record_id": ["a", "b", "c", "d"],
    "temperature_c": pd.Series([-50, 150, 999, pd.NA], dtype="Float64"),
})
result = validate_temperature(sample)
print(result.metrics)  # 4 input, 2 accepted, 2 quarantined

This pandas 3.0 example expects a numeric column after parsing and quarantines both missing and out-of-range temperatures, with different reasons. fillna(False) makes the decision explicit for nullable dtypes: an unknown boolean must not cause a row to disappear from both outputs. Other contracts may allow missing temperatures or reject only the field; make that policy explicit.

The isolated area requires access controls at the same security level, as it often contains raw sensitive data. Retention periods, reprocessing flags, and ownership assignments should be established to prevent it from becoming permanent garbage.

3. Pipeline Steps Must Have a Clear Sequence

A transparent execution order:

1. Decoding and schema parsing;
2. Uniform explicit handling of missing values;
3. Logical type conversion;
4. Field-level and cross-field validity validation;
5. Deterministic deduplication;
6. Standardization and derivation of new fields;
7. Statistical imputation or model-based transformations;
8. Output contract definition and reconciliation.

The order directly affects outcomes. If invalid 999 temperature values are used in median calculations before being isolated, the imputation parameters become corrupted. Dependencies between rules must be explicitly declared in the pipeline definition.

4. Row Count Verification Is an Equation

If every input ultimately ends up in either accepted or quarantined:

$$N_{input}=N_{accepted}+N_{quarantined}.$$

If deduplication is also performed:

$$N_{input}=N_{accepted}+N_{quarantined}+N_{duplicates},$$

provided that the three sets are mutually exclusive. If duplicates may also be invalid, a classification priority must be defined or multi-label counting must be used, equations cannot be blindly applied in such cases.

Count equality is necessary but insufficient. Verify that stable input record IDs appear exactly once across the output partitions; [a, b] and [a, a] have the same size. Track rule hits separately when one row violates several rules.

Numerical values such as monetary amounts, event counts, and entity counts can also be validated through conservation checks. When data cleaning alters numeric values, the original input values and a summary of the differences should be preserved and reported.

5. Quality Gate Comparison of Absolute Rules and Baselines

Hard Constraints

• Primary keys must not be null;
• Schema must remain compatible;
• Target partitions must not be missing;
• Referential keys must satisfy integrity constraints.

Statistical Monitoring

• Row counts compared against seasonal baselines;
• Missing rate trends;
• New or disappearing categories;
• Changes in percentiles and distribution drift;
• Proportion of records quarantined or retried.

Failure of hard constraints can block deployment entirely; deviations in statistical metrics may trigger alerts requiring manual verification. Blocking all changes leads to frequent false positives, while allowing all changes undermines the value of quality gates.

6. fit and transform Stored Separately

@dataclass(frozen=True)
class TemperatureImputer:
    median: float

    @classmethod
    def fit(cls, training: pd.Series) -> "TemperatureImputer":
        observed = training.dropna()
        if observed.empty or not np.isfinite(observed).all():
            raise ValueError("Training requires finite observed temperatures")
        return cls(median=float(observed.median()))

    def transform(self, values: pd.Series) -> pd.Series:
        if not np.isfinite(values.dropna()).all():
            raise ValueError("Validate nonfinite temperatures before imputation")
        return values.fillna(self.median)

imputer = TemperatureImputer.fit(pd.Series([10.0, 20.0, np.nan]))
print(imputer.transform(pd.Series([100.0, np.nan])).tolist())  # [100.0, 15.0]

Store parameter values, training data versions, feature definitions, and code versions. If median calculations are recomputed in production, it introduces bias between training and serving; historical replay cannot reproduce the original results.

The example keeps the fitted median at 15 even when inference includes 100. All-missing or nonfinite training values fail explicitly; domain-range validation still belongs upstream. These classes freeze their attributes, not the contents of any DataFrame they reference.

Non-machine learning components also require versioning of parameters. For example, anomaly thresholds, exchange rate tables, and regional mappings evolve over time.

7. Idempotence and Determinism

An ideal cleaning transformation produces the same output given the same input and version. Verify the following:

• The tie-breaker for deduplication is unique;
• Current time and random seeds are explicit inputs;
• External dimension tables use a fixed version;
• Floating-point order in parallel aggregation does not affect the final result;
• Output sorting is part of the contract.

Some transformations should also be idempotent:

$$clean(clean(x))=clean(x).$$

Not all transformations are naturally idempotent, examples include appending a suffix on each run or re-normalizing data. Property-based testing can quickly detect such issues.

Only test idempotence where it belongs to the contract. Applying a fitted scaler twice usually changes values again, and CleanResult is a result object, not a valid second input. For this validator, revalidate result.accepted and compare its accepted rows with the original accepted rows.

8. Testing from Examples to Invariants

Unit Cases

Cover valid, boundary, missing, unknown enum, and parsing failure scenarios.

Property-Based Testing

• Output keys are unique;
• No additional rows are introduced unless explicitly specified;
• Span or temporal order is not reversed;
• Double-cleaning produces identical results;
• The accepted and quarantine sets do not overlap.

Golden Dataset

Store small, representative inputs and expected outputs for rule change diffing. Do not merely assert file hashes; when outputs change, show specific rows, columns, and the reasons behind the changes.

Shadow Run

New rules are run in parallel without being published first, comparing row counts, metrics, and downstream impacts before switching versions.

9. What Changed in the Release Report

run_id: clean-2026-08-03-01
input: 1,000,000
accepted: 982,140
quarantine: 12,300
duplicates: 5,560
changed values:
  normalized location: 320,441
  imputed temperature: 8,201
top rejection reasons:
  invalid timestamp: 7,104
  impossible temperature: 5,196

Sudden shifts in proportions should drill down to source, partition, schema version, and rule level. The report serves operations staff as well as analysts who need to understand differences between data versions.

Common Misconceptions

• A re-run of a cleaning script is enough for reproducibility: The external reference tables, parameters, and input versions must also be fixed to ensure true reproducibility.
• Deleting bad rows directly is the cleanest approach: This loses root cause analysis, audit trails, and opportunities for corrective actions.
• Testing a few example rows is sufficient: Data rules require invariants, boundary conditions, and replay of real-world data distributions.
• Statistical drift is always a data fault: It may instead reflect genuine business changes; responses must be tiered and context-aware.

Exercise

1. Modify a script that mutates a DataFrame in-place to use the CleanResult interface.
2. Write reconciliation equations for missing, anomalous, and duplicate sets and handle their intersections.
3. Write clean(clean(x)) == clean(x) property tests for a data cleaning function.
4. Design a shadow run report comparing new and old rule behavior.

Summary

Reliable data cleaning is versioned data transformation: the original input remains unchanged, rule execution has clear reasoning, bad records can be isolated, outputs satisfy invariants, and parameters are fitted only on the permitted training data. Only under these conditions does data cleaning transition from a one-off notebook operation to an auditable production process.

The next chapter uses the cleaned data for exploratory data analysis (EDA). The first step is still not about producing visually appealing charts, but rather about clearly identifying whether each visualization is intended to examine a distribution, reveal relationships, or explore the data generation process.