11.2 Data Quality SLOs and Event Response: What Should the System Do After a Check Fails

The morning task list suddenly becomes empty, yet downstream reports still publish on schedule, until the commander notices that attendance across the entire system is zero. A pipeline's "success" only means that no exceptions were thrown in the code; whether a data service is actually available depends on freshness, completeness, semantic correctness, and recoverability.

Learning Objectives

• Select appropriate quality dimensions and measurable SLIs (Service Level Indicators) based on use case requirements;
• Distinguish between SLIs, SLOs (Service Level Objectives), SLAs (Service Level Agreements), and error budgets;
• Design fail-safe mechanisms including blocking, isolation, degradation, and alerting to handle failures;
• Establish a data-driven event flow from detection through response to prevention of recurrence.

1. Quality is relative to usability for a given purpose

Common dimensions include:

Dimension	Issue	Example
Completeness	Are required fields present	Completed tasks must have completed_at
Validity	Do values conform to declared rules	Status must be a versioned enum
Accuracy	Does it reflect reality	Inventory matches physically counted stock
Consistency	Are there contradictions across systems or fields	Total summary amount equals sum of line items
Uniqueness	Are business entities redundantly represented	Each task version key must be unique
Integrity constraints	Do relationships hold	Every log entry must have a valid task foreign key
Freshness	Is it available when needed	Coverage for the previous day is complete by 08:00

These are not the only "six dimensions" in industry, nor can they be uniformly scored to replace specific rules. age > 0 validates correctness but does not confirm accuracy of age; the presence of @ in a string does not prove the email is valid.

2. Deriving Rules from Business Failures

Start by asking: What kind of error would cause a decision to be flawed?

Business failure: Attendance reports treat unsynchronized sentinels as zero attendance
Data condition: Every sentinel that should report has a daily batch; the batch status is complete
SLI: Number of sentinels received on time / Total number of sentinels that should report
SLO: Attendance coverage at 08:00 on workdays reaches the agreed target
Response: When falling short, freeze the official report and display the previous successful version along with missing sentinels
Owner: Lead engineer for attendance data product

Rules must specify granularity, time window, denominator, time zone, exceptions, and versioning. A vague statement like "NULL rate below 5%" could permit all critical fields to be empty, or it could incorrectly reject fields that are legitimately missing in business contexts.

3. SLI, SLO, and SLA

• SLI: A measurable metric, such as the proportion of timely batch executions over a 30-day period;
• SLO: An internal target, for example, the on-time availability rate of critical batches;
• SLA: A commitment agreed upon with customers or stakeholders, including consequences when targets are not met;
• error budget: The amount of allowable unreliability within a target window, used to balance innovation with system stability.

Do not combine response time, resolution time, and data availability into a single metric. Instead, define them separately:

detect time: the time from when an anomaly occurs to when it is detected
acknowledge time: the time from alert to when someone takes ownership
mitigate time: the time to restore availability or activate fallbacks
resolve time: the time from root cause identification to full recovery and data reconciliation

Target values must be derived from consumer deadlines and error costs, never copied from a generic template.

4. Rule Layering and Handling Actions

Row-Level Constraints

Types, enumerations, ranges, required fields, primary and foreign keys, and other row-level validations. Errors in individual rows can be isolated, but care must be taken to avoid propagating errors that compromise the integrity of the entire dataset.

Batch-Level Constraints

Constraints related to row count, partition coverage, uniqueness, reconciliation, distribution, and freshness. A failure at this level may prevent the entire batch from being deployed.

Cross-System and Business Constraints

Consistency between accounts and physical records, conservation of funnel metrics, validity of state transitions, and relationships among key business indicators. These typically require judgment from business owners.

Each rule must predefine an action:

• fail closed: Block deployment; suitable when failure has high cost and a fallback exists;
• quarantine: Isolate the affected batch or record and preserve evidence;
• degrade: Fall back to the previous successful version or reduce functionality;
• warn: Proceed with deployment but clearly flag limitations;
• observe only: Establish a baseline for monitoring; no production action is triggered yet.

"Accept everything first and fix gradually" and "reject any anomaly" are not universally valid. The appropriate choice depends on reversibility, downstream impact, and recovery capability.

5. Contract Validation and Observability Complement Each Other

Contract validation checks known invariants, such as key uniqueness or schema compatibility. Observability, on the other hand, detects unforeseen changes not pre-defined in the system, such as distribution shifts, scale anomalies, latency spikes, or data lineage issues.

Simple three-standard-deviation alerts often fail due to: insufficient sample size, zero standard deviation, or noise introduced by trends and weekly patterns. A more robust baseline should include:

• Comparisons across weekly cycles, holidays, and seasonal periods;
• Use of robust percentiles or count-based models;
• Recording of change and activity calendars;
• Aggregation and suppression of alerts across multiple metrics;
• Including affected partitions, lineage paths, and runbooks in alert notifications.

Exception detection can only signal "difference", it cannot prove "error."

6. Quality Results Themselves Must Be Versioned

A quality check record must at least include:

dataset/version/partition
rule_id and rule_version
observed value and expected condition
sample denominator
status and severity
evaluation time
code/input version
owner, incident, and resolution status

After a rule is modified, historical pass rates may no longer be directly comparable. Versioning enables trend analysis and allows for replay during disputes.

7. Data Event Response

1. Detection: Confirm it's not a self-monitoring failure
2. Prioritization: Categorize by impact, scope, reversibility, and sensitivity
3. Containment: Halt faulty deployments, roll back versions, or activate fallbacks
4. Notification: Alert owners, consumers, and schedule for the next update
5. Remediation: Fix the code or source and re-populate from a trusted checkpoint
6. Validation: Verify rules, reconciliation, and key downstream metrics pass
7. Recovery: Restart consumers in sequence and monitor for stability
8. Postmortem: Add controls, assign owners, and define deadlines

A quality fix cannot be limited to just the current table. Use lineage to identify reports, models, and outbound files that have already consumed erroneous data, and determine whether reprocessing, version rollback, or notification is required.

8. Avoid Alert Fatigue

• Alerts must be actionable and have a clear owner.
• Downstream alerts caused by the same root cause should be aggregated.
• Maintenance windows and known changes should suppress expected noise.
• Alert severity should determine whether notifications go to a pager, create a ticket, or trigger a dashboard view.
• Track false positives, unclaimed alerts, and long-term silent alerts.
• High-priority alerts without runbooks should be downgraded or supplemented with proper response procedures.

Severity is determined by business impact, not automatically by a percentage deviation from baseline.

Common Misconceptions

• Pipeline success means data is available: Completion of code does not equate to semantic correctness.
• Applying the same thresholds across all tables is standard practice: Data quality depends on granularity, purpose, and the cost of errors.
• Accuracy can be proven by format validation alone: Authority sources, spot checks, or real-world reconciliation are required.
• More alerts mean greater safety: Unattended alerts are equivalent to no control at all.

Exercise

1. Define three SLIs, corresponding SLOs, and consumer deadlines for a daily attendance report.
2. Assign one of fail, quarantine, degrade, warn, or observe to each of five quality rules.
3. Construct a time series with weekly seasonality and explain why three-standard-deviation alerts produce false positives.
4. Walk through a scenario where a blank table is published, and list all downstream assets that must be rolled back or re-computed.

Summary

Data quality isn't a dimensional scoring sheet; it's a service operation mechanism: clearly defining what failures would harm decisions, measuring those with SLIs, prioritizing them with SLOs, and establishing verifiable containment, remediation, and recovery procedures after failure.

The next chapter dives into metadata and lineage: to identify affected assets and owners, the system must know what the data is, which versions generate it, and where it flows.