11.2 Data Quality SLOs and Incident Response: What Should the System Do After a Check Fails

After an upstream task table becomes empty, downstream reports still publish on schedule and calculate zero attendance for every site. A pipeline's “success” only means its code raised no exception; data-service availability also depends on freshness, completeness, semantic correctness, and recoverability.

Learning Objectives

• Select appropriate quality dimensions and measurable SLIs (Service Level Indicators) based on use case requirements;
• Distinguish between SLIs, SLOs (Service Level Objectives), SLAs (Service Level Agreements), and error budgets;
• Design fail-safe mechanisms including blocking, isolation, degradation, and alerting to handle failures;
• Establish a data incident process from detection through response to prevention of recurrence.

1. Quality is relative to usability for a given purpose

Common dimensions include:

Dimension	Issue	Example
Completeness	Are required fields present	Completed tasks must have completed_at
Validity	Do values conform to declared rules	Status must be a versioned enum
Accuracy	Does it reflect reality	Inventory matches physically counted stock
Consistency	Are there contradictions across systems or fields	Total summary amount equals sum of line items
Uniqueness	Are business entities redundantly represented	Each task version key must be unique
Integrity constraints	Do relationships hold	Every log entry must have a valid task foreign key
Freshness	Is it available when needed	Coverage for the previous day is complete by 08:00

These are not the only "six dimensions" in industry, nor can they be uniformly scored to replace specific rules. age > 0 checks validity but does not confirm accuracy of age; the presence of @ in a string does not prove the mailbox exists.

2. Deriving Rules from Business Failures

Start by asking: What kind of error would cause a decision to be flawed?

Business failure: Attendance reports treat unsynchronized sites as zero attendance
Data condition: Every reporting site has a daily batch; the batch status is complete
SLI: Number of sites received on time / Total number of reporting sites
SLO: Attendance coverage at 08:00 on workdays reaches the agreed target
Response: When falling short, freeze the official report and display the previous successful version along with missing sites
Owner: Accountable owner of the attendance data product

Rules must specify granularity, time window, denominator, time zone, exceptions, and versioning. A vague statement like "NULL rate below 5%" could permit all critical fields to be empty, or it could incorrectly reject fields that are legitimately missing in business contexts.

Take expected sites from an independent schedule or asset snapshot, not the data that arrived; otherwise a total reporting failure also erases the denominator. This standard-library example checks the previous day against a UTC deadline and accepts one already-selected batch version per site/date:

from datetime import datetime, timezone

expected_sites = {"north", "south", "west"}
cutoff = datetime(2026, 8, 2, 8, tzinfo=timezone.utc)
reports = [
    ("north", "2026-08-01", "complete", datetime(2026, 8, 2, 7, 50, tzinfo=timezone.utc)),
    ("south", "2026-08-01", "complete", datetime(2026, 8, 2, 8, 5, tzinfo=timezone.utc)),
]

def coverage_at_deadline(expected, records, data_date, deadline):
    if deadline.tzinfo is None:
        raise ValueError("timezone-aware deadline required")
    seen, ready = set(), set()
    for site, date, status, published_at in records:
        if date != data_date:
            continue
        if site not in expected or site in seen:
            raise ValueError("unexpected site or duplicate site/date snapshot")
        if published_at.tzinfo is None:
            raise ValueError("timezone-aware publication time required")
        seen.add(site)
        if status == "complete" and published_at <= deadline:
            ready.add(site)
    return {"coverage": len(ready) / len(expected) if expected else None,
            "not_ready": sorted(expected - ready)}

status = coverage_at_deadline(expected_sites, reports, "2026-08-01", cutoff)
print(status)
# {'coverage': 0.3333333333333333, 'not_ready': ['south', 'west']}

South eventually arrived but missed 08:00; west never arrived. Eventual completeness may recover while the deadline SLI remains a miss. No expected sites means not applicable, not 100%. Distinguish a legitimate empty batch from missing source data.

3. SLI, SLO, and SLA

• SLI: A measurable metric, such as the proportion of timely batch executions over a 30-day period;
• SLO: An internal target, for example, the on-time availability rate of critical batches;
• SLA: A commitment agreed upon with customers or stakeholders, including consequences when targets are not met;
• error budget: The amount of allowable unreliability within a target window, used to balance innovation with system stability.

Do not combine response time, resolution time, and data availability into a single metric. Instead, define them separately:

detect time: the time from when an anomaly occurs to when it is detected
acknowledge time: the time from alert to when someone takes ownership
mitigate time: the time to restore availability or activate fallbacks
resolve time: the time from incident onset to root-cause repair and complete backfill

Target values must be derived from consumer deadlines and error costs, never copied from a generic template.

With 100 expected sites per day over 30 days, there are 3,000 site-batches. A 99% deadline target allows 30 misses; 12 late batches consume 40% of that budget, even if backfilled later. If the unit becomes one whole daily report, a 99% target over 30 reports allows only 0.3 misses, so one late report exceeds the budget. Do not silently change denominators. See the good-event/total-event approach in the Google SRE Workbook.

4. Rule Layering and Handling Actions

Row-Level Constraints

Types, enumerations, ranges, required fields, primary and foreign keys, and other row-level validations. Errors in individual rows can be isolated, but care must be taken to avoid propagating errors that compromise the integrity of the entire dataset.

Batch-Level Constraints

Constraints related to row count, partition coverage, uniqueness, reconciliation, distribution, and freshness. A failure at this level may prevent the entire batch from being published.

Cross-System and Business Constraints

Consistency between accounts and physical records, conservation of funnel metrics, validity of state transitions, and relationships among key business indicators. These typically require judgment from business owners.

Each rule must predefine an action:

• fail closed: Block publication; suitable when failure has high cost and a fallback exists;
• quarantine: Isolate the affected batch or record and preserve evidence;
• degrade: Fall back to the previous successful version or reduce functionality;
• warn: Proceed with publication but clearly flag limitations;
• observe only: Establish a baseline for monitoring; no production action is triggered yet.

"Accept everything first and fix gradually" and "reject any anomaly" are not universally valid. The appropriate choice depends on reversibility, downstream impact, and recovery capability.

When serving a previous version, include its data timestamp, version, and maximum allowed age. Stop using it for the affected decision after that limit. Quarantine must also report excluded counts and group distribution; “all remaining rows passed” can conceal systematic missingness.

5. Contract Validation and Observability Complement Each Other

Contract validation checks known invariants, such as key uniqueness or schema compatibility. Observability, on the other hand, detects unforeseen changes not pre-defined in the system, such as distribution shifts, scale anomalies, latency spikes, or data lineage issues.

Simple three-standard-deviation alerts often fail due to: insufficient sample size, zero standard deviation, or noise introduced by trends and weekly patterns. A more robust baseline should include:

• Comparisons across weekly cycles, holidays, and seasonal periods;
• Use of robust percentiles or count-based models;
• Recording of change and activity calendars;
• Aggregation and suppression of alerts across multiple metrics;
• Including affected partitions, lineage paths, and runbooks in alert notifications.

Anomaly detection can only signal "difference", it cannot prove "error."

6. Quality Results Themselves Must Be Versioned

A quality check record must at least include:

dataset/version/partition
rule_id and rule_version
observed value and expected condition
sample denominator
status and severity
evaluation time
code/input version
owner, incident, and resolution status

After a rule is modified, historical pass rates may no longer be directly comparable. Versioning enables trend analysis and allows for replay during disputes.

7. Data Incident Response

1. Detection: Confirm it's not a self-monitoring failure
2. Prioritization: Categorize by impact, scope, reversibility, and sensitivity
3. Containment: Halt faulty publications, roll back versions, or activate fallbacks
4. Notification: Alert owners, consumers, and schedule for the next update
5. Remediation: Fix code or source and backfill from retained trustworthy raw data or snapshots
6. Validation: Verify rules, reconciliation, and key downstream metrics pass
7. Recovery: Restart consumers in sequence and monitor for stability
8. Postmortem: Add controls, assign owners, and define deadlines

A quality fix cannot be limited to just the current table. Use lineage to identify reports, models, and outbound files that have already consumed erroneous data, and determine whether reprocessing, version rollback, or notification is required.

8. Avoid Alert Fatigue

• Alerts must be actionable and have a clear owner.
• Downstream alerts caused by the same root cause should be aggregated.
• Maintenance windows and known changes should suppress expected noise.
• Alert severity should determine whether notifications go to a pager, create a ticket, or trigger a dashboard view.
• Track false positives, unclaimed alerts, and long-term silent alerts.
• Escalate high-impact alerts for human response even without a runbook, then add the missing procedures.

Severity is determined by business impact, not automatically by a percentage deviation from baseline.

Common Misconceptions

• Pipeline success means data is available: Completion of code does not equate to semantic correctness.
• Applying the same thresholds across all tables is standard practice: Data quality depends on granularity, purpose, and the cost of errors.
• Accuracy can be proven by format validation alone: Authority sources, spot checks, or real-world reconciliation are required.
• More alerts mean greater safety: Unattended alerts are equivalent to no control at all.

Exercise

1. Define three SLIs, corresponding SLOs, and consumer deadlines for a daily attendance report.
2. Assign one of fail, quarantine, degrade, warn, or observe to each of five quality rules.
3. Construct a time series with weekly seasonality and explain why three-standard-deviation alerts produce false positives.
4. Walk through a scenario where a blank table is published, and list all downstream assets that must be rolled back or re-computed.

Summary

Data quality isn't a dimensional scoring sheet; it's a service operation mechanism: clearly defining what failures would harm decisions, measuring those with SLIs, prioritizing them with SLOs, and establishing verifiable containment, remediation, and recovery procedures after failure.

The next chapter examines metadata and lineage: to identify affected assets and owners, the system must know what the data is, which versions generate it, and where it flows.