12.1 Data Assets, Terminology, and Metadata Catalogs: Enabling Correct Use of Search Results

A business owner asks, “What was the completion rate last quarter?” Seven identically named tables appear. One analyst picks the most recently updated table without noticing that it covers only one region; another finds the authoritative table but mistakes “settled” for “completed.” A catalog that shows only table names and column types makes the wrong data easier to find.

The goal of a metadata catalog is not to collect the most information, but to help users determine: what this is, whether it's relevant to their current problem, who's responsible, and whether it's currently healthy.

Lesson Objectives

• Distinguish technical, business, operational, quality, and usage metadata;
• Design stable identities and versions for data assets;
• Connect business terminology, physical fields, and metric definitions;
• Establish a searchable, verifiable, and sustainably maintainable catalog.

1. What are "Assets" in the Catalog?

Data assets can be:

• Tables, views, file collections, and streams;
• Fields, schema, and partitioning;
• Metrics, features, model inputs, and reports;
• Pipeline jobs, code versions, and run instances;
• Business terminology, rules, and data products.

Each asset needs a stable identifier, rather than a display name alone. For example:

platform://production/logistics/table/missions

Identifiers should include environment and namespace; renamings must preserve aliases or migration relationships. Development tables and production tables, logical metrics and physical columns cannot be conflated into a single object simply because they have the same name.

The example is a custom logical asset ID. Store asset_version=v3, schema version, and storage snapshot ID separately so successive snapshots remain linked to the same asset. Preserve logical identity through renames, and define identity rules for deletion and recreation to avoid joining unrelated histories.

2. Five Types of Metadata Answer Different Questions

Technical Metadata

Schema, types, keys, partitions, storage format, location, and compatible versions answer “How do I read it?”

Business/Semantic Metadata

How to explain: definition, unit, granularity, scope, permitted uses, owner, and term relationships.

Runtime Metadata

Job/run IDs, start and finish times, code version, input/output versions, status, and resources answer “How was this version generated?”

Quality and Reliability Metadata

SLIs, rule results, incidents, certification status, freshness, and known limits answer “Can I use it reliably now?”

Use Metadata

Queries, downstream dependencies, access frequency, active consumers, and last-use time answer “Who depends on it?”

Classification is merely a perspective for organization; the same fact can span multiple classes. The core is shared asset identity and temporal versioning.

3. Business terminology doesn't equal field notes

“Task Completion Rate” should first become a business term with an owner:

Name: Task Completion Rate
Definition: the number of tasks that meet completion criteria within the observation window divided by the number of tasks that meet inclusion criteria
Numerator: Deduplicated tasks whose status belongs to the versioned completed-state set
Denominator: tasks that have reached the observation cutoff and are not in the test/cancel reason set
Granularity: global, regional, task type
Event Time and Deadline Rules: ……
Unit: ratio
Owner: Operations Analysis Lead

Then connect terms to metric implementations, fields, and reports. A term can have multiple physical implementations, but one authoritative version must be designated; the same column name may carry different terms.

The glossary needs synonyms, deprecated terms, and conflict relationships. When searching for "success rate," it should indicate whether it differs from "completion rate," rather than silently merging the terms.

4. Automatic data collection and manual maintenance each have their boundaries

Can automatically collect:

• Columns, types, partitions, and constraints in the catalog/schema;
• Job, run, and dependencies in the scheduler and query engine;
• Inspection status in the quality platform;
• Audit statistics on access and usage within the system.

Need domain judgment:

• Why are the metrics defined this way;
• The business meaning of NULL, default values, and states;
• Which uses are permitted or prohibited;
• Who holds the final interpretation rights;
• Known bias and decision constraints.

Using pd.read_parquet() to load an entire dataset for "metadata collection" mistakes scan cost for catalog cost. Prefer the catalog, schema, or file footer, and state whether row counts are exact or estimated.

This PyArrow example reads a file’s footer and schema without invoking the table-data read API:

from pathlib import Path
from tempfile import TemporaryDirectory
import pyarrow as pa
import pyarrow.parquet as pq

with TemporaryDirectory(prefix="atlas-metadata-") as directory:
    file = Path(directory) / "part.parquet"
    fixture = pa.table({"mission_id": ["m1", "m2", "m3"], "resources": [10, 20, None]})
    pq.write_table(fixture, file, row_group_size=2)
    with pq.ParquetFile(file) as parquet:
        record = {"row_count": parquet.metadata.num_rows,
                  "row_groups": parquet.metadata.num_row_groups,
                  "fields": parquet.schema_arrow.names}
    print(record)
# {'row_count': 3, 'row_groups': 2, 'fields': ['mission_id', 'resources']}

This is an exact physical row count for one file. A dataset count requires a trustworthy file manifest plus deletion-vector, filtered-view, and snapshot rules; summing every old file in a directory can be wrong. Footer statistics can also expose sensitive values, so metadata access still needs authorization. See PyArrow ParquetFile.

5. Sample values might leak data

A catalog showing sample values is convenient, but it might expose sensitive information like names, device identifiers, rare diagnostics, or keys to individuals who have no right to access the original table.

May include:

• Does not default to sampling sensitive columns;
• Use approved synthetic examples or format templates;
• Have the profile results inherit the source data access policy;
• Provide small-group protection for aggregations such as count, min, max, and others;
• Record who triggered the sample, the sample version, and the deletion time.

Authorize catalog content by sensitivity: general terminology may be broadly visible, while locations, schemas, profiles, and samples follow appropriate source-data controls. Even “asset exists, request access” can expose an investigation or tenant identity, so visibility of existence needs its own policy.

6. Certification and Deprecation States

Search results should be distinguished:

• draft: still in design phase;
• verified/certified: Has passed review for definition, owner, quality, and purpose;
• deprecated: retained solely for migration purposes;
• quarantined: severe quality issues have been identified; product should not be consumed further;
• retired: No longer in production; retained or deleted according to policy.

Certification is not a lifetime badge. It must be reviewed again when there are significant changes to schema, logic, owner, SLO, or purpose. Deprecation must include a replacement asset, a migration timeline, and a list of active consumers.

These names form an illustrative lifecycle, not a universal product enum. Store semantic certification separately from operational health: a certified product can still have today’s partition quarantined. Search should show certification scope and recent quality status together so one green badge cannot hide a failed batch.

7. Search and sorting must prevent "most popular equals most authoritative"

Search relevance can combine:

• Name, synonyms, and business definitions match;
• Authority or certification status;
• Data freshness and quality;
• Usage visible to the current user;
• Organizational context and real-world usage;
• Deprecation and incident state.

High query volumes might just be historical debt and shouldn't override authoritative status. The search results page must at least display granularity, time coverage, owner, freshness, certification, and restrictions, so users don't have to click through ten layers of pages to assess them.

8. Metadata Quality Also Needs SLOs

Monitor:

• CDE's owner/definition/category coverage;
• Delay in production asset operation metadata;
• The number of active queries on deprecated assets;
• Missing owners, expired certification, and invalid links;
• Time to resolve terminology conflicts;
• Success rate or abandonment rate after catalog search.

“Having a catalog with 100,000 assets” doesn’t mean it’s useful. What matters more is whether key assets can be accurately located, understood, and managed.

Common Misconceptions

• Fetching the schema completes the catalog: Technical structure can't replace business definitions.
• Displaying real samples is the clearest option: A catalog can become a side channel for sensitive data.
• The most commonly used table is the authoritative table: Popularity will solidify historical debt.
• Certified once, always trusted: Certification expires with changes in definition or context.

Practice

1. Define the complete terminology for task completion rate and connect two physical implementations.
2. Design a stable asset ID for production tables, development tables, and logical metrics.
3. Review a catalog profile page to identify potential leaks of sensitive subgroup information.
4. Design migration status and consumer tracking for deprecated assets.

Summary

A catalog only helps users make the right choice by tying asset identity, business semantics, runtime status, and usage constraints together. Visibility is not the end goal; explainability, verifiability, and the ability to retire assets safely are equally important.

Runtime lineage then answers what a version actually read and wrote, and which columns, jobs, and decisions are affected by a change.