12.2 Runtime Lineage and Impact Analysis: Tracing a Single Error Metric Back to a Specific Version

The task completion rate in the report suddenly dropped. The directory tells you the metric definition and owner, but it can't tell you which snapshot was read, which version of SQL was used, or whether the northern region partition was skipped last night. Real troubleshooting requires connecting static design to actual execution facts.

This lesson's objectives

• Distinguish design lineage, runtime lineage, and field-level lineage;
• Model the relationship between dataset, job, run, process, and version;
• Select data collection methods such as parsing, engine events, and explicit event tracking;
• Use kinship to analyze changes, events, and privacy impacts.

1. Kinship is a graph with time

Basic nodes include:

• dataset: table, file collection, stream, report, or model input;
• job/process: The job or process that executes the transformation logic;
• run: a specific execution of a job;
• field: column, metric, or feature;
• code/config: SQL, images, parameters, and schema versions.

The edge must express run READ dataset_version, run WROTE dataset_version. Only record:

missions_clean → daily_report

Can't tell whether it describes the current design, yesterday's run, or logic retired three years ago; can't distinguish between a rerun and a refill.

2. Design Lineage and Runtime Lineage

Design Lineage

From code, DAGs, and declarative configurations, answer "what dependencies are designed to rely on." It can perform change checks before deployment, but dynamic branches might not be actually executed.

Runtime lineage

From query engines, schedulers, computation frameworks, or job telemetry, answers the question "What was actually read or written this time?" It can bind run, time, and state, yet may only cover already-executed paths.

Both should coexist: design for prevention, runtime for evidence. The difference itself warrants alerting, for example, if the code declares reading three tables, but the production run only reads two.

3. An auditable runtime event

{
  "run_id": "2026-08-03/daily-report/attempt-2",
  "job_id": "analytics.daily-report:v17",
  "started_at": "2026-08-03T06:01:00Z",
  "ended_at": "2026-08-03T06:08:20Z",
  "status": "complete",
  "inputs": [
    {"asset": "missions_clean", "version": "snapshot-2026-08-03T05:55Z"}
  ],
  "outputs": [
    {"asset": "daily_mission_summary", "version": "partition=2026-08-02/run=attempt-2"}
  ],
  "code_version": "git:9f2…",
  "schema_version": "3"
}

Production implementation must also handle start/complete/fail, retry attempts, idempotent event IDs, out-of-order arrivals, and access control. The lineage events themselves might leak table names, field classifications, or tenant information.

4. SQL Regular Expressions Cannot Reliably Parse Lineage

Regex extraction of FROM and JOIN will fail in the following situations:

• CTEs, nested queries, and correlated subqueries;
• quoted identifiers, catalog/schema, and temporary views;
• MERGE, UPDATE, Dynamic SQL and macros;
• Reads internally, external functions, and stored procedures within UDF;
• The actual physical origin of the engine rewrite.

Priority is typically:

1. Structured plans or events provided by the query engine/execution framework;
2. Use a full SQL parser and dialect AST;
3. The editor's declared inputs and outputs;
4. Explicitly track entry/exit points for assignments;
5. Manually enter the data and assign a confidence level.

Different collectors might produce conflicting edges. Preserve source, confidence, and observation time, don’t silently merge them into a single "truth."

5. Field-level provenance requires expression semantics

Table-level lineage can only indicate that a certain table is affected. Field-level lineage further records:

daily.completed_count
  ← count(distinct missions.mission_id)
     where missions.status in COMPLETED_STATUS_SET(v4)

daily.completion_rate
  ← completed_count / eligible_count

Direct mapping, aggregation, conditional logic, constants, UDFs, and multi-input expressions should be distinguished. If the system only knows "possibly from these columns," label it as unknown/indirect, rather than fabricating a precise expression.

Field-level lineage is costly; prioritize coverage of CDE fields, key metrics, sensitive fields, and high-risk model features.

6. Impact analysis Goes Beyond a simple BFS list

Graph traversal can identify candidate downstream nodes, but actual priority still depends on:

• Is the edge currently valid, and was it recently executed;
• The change involves name, type, unit, value range, or historical semantics;
• Is the downstream field read-only and unchanged;
• Assets are for experiments, internal reports, or external decisions;
• Are there cache, export, model, and manual copies?
• owner, SLO, sensitive categories, and change windows.

The impact report should output path and evidence:

changed field → expression → output field → report/model → owner
last successful run / active consumers / severity / confidence

The graph has cycles, aliases, and duplicate edges; traversal must maintain a visited set and enforce version or time ranges.

7. Three Typical Uses

Before the change

Check schema and semantic changes, identify active consumers, schedule compatibility windows and regression samples.

Incident Response

Trace the actual run and input version backward from error output, then forward to identify contaminated reports, features, models, and exports.

Privacy and Deletion

Track the derivation, caching, and external distribution of sensitive fields. Lineage can narrow the candidate scope but cannot prove all copies have been identified; additional analysis using access logs, storage scans, and asset inventories is required.

8. Integrity and Freshness

Surveillance:

• Production runs show coverage of edge cases;
• Does every complete run have inputs, outputs, and versions?
• Does the active asset in the directory have recent edges?
• Design and run edge differences;
• Field-level parse failure rate and unknown expressions;
• Event delays, duplicates, and isolated nodes.

Lineage gaps can create a false sense of security in impact analysis. Query results should show coverage and confidence levels, rather than a pretty diagram alone.

9. Change Access Controls

[ ] Currently and recently running downstream already identified
[ ] Key field-level paths have been verified
[ ] Active owner has confirmed or has been notified
[ ] Compatibility strategy and rollback version are ready
[ ] Coverage of regression samples on old/new semantics
[ ] Outsourced, models, and caching have been included
[ ] Unknown/Low-confidence lineage has been manually verified

Common Misconceptions

• Table-to-table arrows mean complete lineage: missing version, run, and field expression.
• Parsing SQL Can Cover All Transformations: Dynamic code and engine behavior require additional signals.
• Draw all downstream impacts to complete the impact analysis: Additionally filter by activity level, field, and business risk.
• Lineage can prove deletion is complete: Unconnected systems and manual copies still require other discovery methods.

Practice

1. Add three run events for failure, retry, and rollback of the same job.
2. Find a SQL containing CTE, subqueries, and UDFs, and compare regular expression vs. AST parsing results.
3. Create field-level expression lineage for the completion rate metric and label the business rule version.
4. Simulate a unit change and generate an impact report featuring owner, engagement level, and confidence.

Summary

Lineage isn't a static arrow; it's a dynamic graph with runtime, version, field semantics, and evidence sources. It takes change and event response from "might be related" to "which run, which output column, and which active decisions were impacted."

The next chapter discusses Data Mesh: how to distribute data ownership as domains grow, without losing interoperability, platform capabilities, or shared governance.