12.2 Runtime Lineage and Impact Analysis: Tracing an Incorrect Metric Back to a Specific Version

The task completion rate in the report suddenly dropped. The catalog tells you the metric definition and owner, but it can't tell you which snapshot was read, which version of SQL was used, or whether the northern region partition was skipped last night. Real troubleshooting requires connecting static design to actual execution facts.

Learning Objectives

• Distinguish design lineage, runtime lineage, and field-level lineage;
• Model the relationship between dataset, job, run, process, and version;
• Select data collection methods such as parsing, engine events, and explicit event tracking;
• Use lineage to analyze changes, events, and privacy impacts.

1. Lineage is a graph with time

Basic nodes include:

• dataset: table, file collection, stream, report, or model input;
• job/process: The job or process that executes the transformation logic;
• run: a specific execution of a job;
• field: column, metric, or feature;
• code/config: SQL, images, parameters, and schema versions.

The edge must express run READ dataset_version, run WROTE dataset_version. Only record:

missions_clean → daily_report

Can't tell whether it describes the current design, yesterday's run, or logic retired three years ago; can't distinguish between a rerun and a backfill.

2. Design Lineage and Runtime Lineage

Design Lineage

From code, DAGs, and declarative configurations, answer "what dependencies are designed to rely on." It can perform change checks before deployment, but dynamic branches might not be actually executed.

Runtime lineage

From query engines, schedulers, computation frameworks, or job telemetry, answers the question "What was actually read or written this time?" It can bind run, time, and state, yet may only cover already-executed paths.

Keep both: design edges support prevention, runtime edges support evidence. Interpret differences against branch conditions. A design declaring three tables with a run observing two may reflect a valid branch, a collection gap, or an execution error.

3. An auditable runtime event

This is a simplified custom teaching schema with example identities and versions, not an OpenLineage protocol payload. Keep the logical job ID stable and record code version and attempt separately:

{
  "run_id": "2026-08-03/daily-report/attempt-2",
  "job_id": "analytics.daily-report",
  "job_version": "v17",
  "event_id": "example-daily-report-attempt-2-complete",
  "started_at": "2026-08-03T06:01:00Z",
  "ended_at": "2026-08-03T06:08:20Z",
  "status": "complete",
  "inputs": [
    {"asset": "missions_clean", "version": "snapshot-2026-08-03T05:55Z"}
  ],
  "outputs": [
    {"asset": "daily_mission_summary", "version": "partition=2026-08-02/run=attempt-2"}
  ],
  "code_version": "example-release-v17",
  "schema_version": "3"
}

Production implementation must also handle start/complete/fail, retry attempts, idempotent event IDs, out-of-order arrivals, and access control. The lineage events themselves might leak table names, field classifications, or tenant information.

An OpenLineage integration must follow the chosen specification’s namespace/name, UUID runId, event types, and facet schemas. START and COMPLETE updates for one run share a runId. A completion event does not itself prove transactional publication; retain the storage commit ID and publication evidence. A delayed START must not overwrite an already known COMPLETE merely because it arrived later.

4. SQL Regular Expressions Cannot Reliably Parse Lineage

Regex extraction of FROM and JOIN will fail in the following situations:

• CTEs, nested queries, and correlated subqueries;
• quoted identifiers, catalog/schema, and temporary views;
• MERGE, UPDATE, Dynamic SQL and macros;
• Reads internally, external functions, and stored procedures within UDF;
• The actual physical origin of the engine rewrite.

Priority is typically:

1. Structured plans or events provided by the query engine/execution framework;
2. Use a full SQL parser and dialect AST;
3. The orchestrator’s declared inputs and outputs;
4. Instrument job input/output boundaries explicitly;
5. Manually enter the data and assign a confidence level.

Different collectors might produce conflicting edges. Preserve source, confidence, and observation time, don’t silently merge them into a single "truth."

5. Field-level provenance requires expression semantics

Table-level lineage can only indicate that a certain table is affected. Field-level lineage further records:

daily.completed_count
  ← count(distinct missions.mission_id)
     where missions.status in COMPLETED_STATUS_SET(v4)

daily.completion_rate
  ← completed_count / eligible_count

Distinguish direct value sources from indirect control dependencies: mission_id contributes values to the distinct count, while status controls which rows contribute. Changing the status rule can therefore change the count. Preserve constant, UDF, and multi-input semantics too. “Indirect” is a dependency type, not a synonym for “unknown”; record unresolved lineage separately. See the OpenLineage column lineage facet.

Field-level lineage is costly; prioritize coverage of CDE fields, key metrics, sensitive fields, and high-risk model features.

6. Impact analysis Goes Beyond a simple BFS list

Graph traversal can identify candidate downstream nodes, but actual priority still depends on:

• Is the edge currently valid, and was it recently executed;
• The change involves name, type, unit, value range, or historical semantics;
• Does the consumer read only unchanged fields;
• Assets are for experiments, internal reports, or external decisions;
• Are there cache, export, model, and manual copies?
• owner, SLO, sensitive categories, and change windows.

The impact report should output path and evidence:

changed field → expression → output field → report/model → owner
last successful run / active consumers / severity / confidence

The graph has cycles, aliases, and duplicate edges; traversal must maintain a visited set and enforce version or time ranges.

The following traverses candidate edges for explicitly selected versions. Observation times use uniform UTC ISO strings of identical precision; production code should parse timestamps. It returns one representative path per candidate and propagates uncertainty along that path:

from collections import defaultdict, deque

# source field/version, target field/version, run evidence, observation time, confidence
edges = [
    ("missions@v3.status", "summary@v2.completed", "run-17", "2026-08-03T06:00:00Z", "observed"),
    ("summary@v2.completed", "summary@v2.rate", "run-17", "2026-08-03T06:00:00Z", "observed"),
    ("summary@v2.rate", "report@v5.rate", "run-18", "2026-08-03T07:00:00Z", "observed"),
    ("summary@v2.rate", "export@v1.rate", "manual-1", "2026-08-03T07:00:00Z", "unknown"),
    ("missions@v2.status", "report@v4.rate", "run-10", "2026-08-02T07:00:00Z", "observed"),
]

def impacted(start, observations, as_of):
    adjacency = defaultdict(list)
    for src, dst, evidence, observed_at, confidence in observations:
        if observed_at <= as_of:
            adjacency[src].append((dst, evidence, confidence))
    paths, queue = {start: ([], False)}, deque([start])
    while queue:
        node = queue.popleft()
        path, uncertain = paths[node]
        for target, evidence, confidence in adjacency[node]:
            if target not in paths:
                paths[target] = (path + [(node, target, evidence)],
                                 uncertain or confidence != "observed")
                queue.append(target)
    return {node: value for node, value in paths.items() if node != start}

impact = impacted("missions@v3.status", edges, "2026-08-03T08:00:00Z")
print(sorted(impact))
# ['export@v1.rate', 'report@v5.rate', 'summary@v2.completed', 'summary@v2.rate']
assert impact["export@v1.rate"][1] is True
assert "report@v4.rate" not in impact

This is not a complete impact decision. The example neither collects lineage nor verifies publication and does not enumerate every path; retain the original graph for alternative evidence routes. Version selection avoids treating an old report as a descendant of this version. An unknown export remains a manual-investigation candidate rather than disappearing because confidence is low.

7. Three Typical Uses

Before the change

Check schema and semantic changes, identify active consumers, schedule compatibility windows and regression samples.

Incident Response

Trace the actual run and input version backward from error output, then forward to identify contaminated reports, features, models, and exports.

Privacy and Deletion

Track the derivation, caching, and external distribution of sensitive fields. Lineage can narrow the candidate scope but cannot prove all copies have been identified; additional analysis using access logs, storage scans, and asset inventories is required.

8. Integrity and Freshness

Monitor:

• Fraction of production runs that emit lineage events;
• Does every complete run have inputs, outputs, and versions?
• Does the active asset in the catalog have recent edges?
• Design and run edge differences;
• Field-level parse failure rate and unknown expressions;
• Event delays, duplicates, and isolated nodes.

Lineage gaps can create a false sense of security in impact analysis. Query results should show coverage and confidence levels, rather than a pretty diagram alone.

9. Change Gates

[ ] Currently and recently running downstream already identified
[ ] Key field-level paths have been verified
[ ] Active owner has confirmed or has been notified
[ ] Compatibility strategy and rollback version are ready
[ ] Coverage of regression samples on old/new semantics
[ ] External exports, models, and caches have been included
[ ] Unknown/Low-confidence lineage has been manually verified

Common Misconceptions

• Table-to-table arrows mean complete lineage: missing version, run, and field expression.
• Parsing SQL Can Cover All Transformations: Dynamic code and engine behavior require additional signals.
• Draw all downstream impacts to complete the impact analysis: Additionally filter by activity level, field, and business risk.
• Lineage can prove deletion is complete: Unconnected systems and manual copies still require other discovery methods.

Practice

1. Add three run events for failure, retry, and backfill of the same job.
2. Find a SQL containing CTE, subqueries, and UDFs, and compare regular expression vs. AST parsing results.
3. Create field-level expression lineage for the completion rate metric and label the business rule version.
4. Simulate a unit change and generate an impact report featuring owner, recent activity, and confidence.

Summary

Lineage isn't a static arrow; it's a dynamic graph with runtime, version, field semantics, and evidence sources. It takes change and event response from "might be related" to "which run, which output column, and which active decisions were impacted."

The next chapter discusses Data Mesh: how to distribute data ownership as domains grow, without losing interoperability, platform capabilities, or shared governance.