13.2 Data Products, Platforms, and Federated Governance: Autonomy Must Be Interoperable

Each initial domain released a “product table,” but they differed vastly in format, identity keys, temporal semantics, and permission request methods. Consumers now have to piece together three tables, making the process slower than before. The issue isn't domain autonomy; it's that autonomy lacks a common interface, and the platform hasn't made the right path the default.

Learning Objectives

• Define the interfaces, SLOs, and lifecycle of data products;
• Design a self-service paved road for the platform;
• Convert federated rules into templates, policies, and validation evidence;
• Measure migration by consumer outcomes, not by product volume.

1. Data products deliver consumable capabilities

A product specification must at least include:

The YAML is a custom teaching description, not executable configuration for a particular product. table://, topic://, and registry:// are illustrative identifiers requiring real platform resolution, publication, policy enforcement, and validation.

id: logistics.confirmed-supply-events
owner: logistics-domain
purpose: confirmed supply movement for operational and analytical use
interfaces:
  batch: table://prod/logistics/confirmed_supply_events/v2
  stream: topic://prod/logistics/confirmed-supply-events/v2
grain: one row/event per confirmed supply movement
keys: [supply_event_id]
event_time: confirmed_at
ingestion_time: ingested_at
available_time: published_at
schema_contract: registry://logistics/supply-event/v2
quality_slos:
  freshness: consumer-agreed target
  completeness: based on expected sites
classification: internal-sensitive
support: on-call route and issue tracker
deprecation: notice window and migration policy

Specific SLOs cannot be copied from templates; they must be negotiated between consumers based on deadlines, failure costs, and capacity. Whether an external SLA includes breach consequences depends on the organizational relationship.

Ingestion is not availability: a record may still await quality checks or commit after reaching the platform, so expose published_at separately. This product is defined as immutable confirmed events, with corrections represented by new events referring to originals. A mutable state-snapshot interface needs different key, version, and retraction rules.

2. The Eight Core Capabilities of a Product

• Discoverable: catalogs, names, terms, and examples;
• Understandable: granularity, time, units, NULL, and state semantics;
• Trustworthy: Quality results, incident status, and certification;
• Addressable: Stable asset ID and interface;
• Interoperability: shared identity, time, schema, and policy agreements;
• Security: Classification, purpose, least privilege, and auditing;
• Valuable: Clearly defines consumers, decision-makers, and feedback mechanisms;
• Maintainable: ownership, support, versioning, cost, and retirement.

You don’t need to promote every internal temporary table into a product. The more products you have, the higher the support, catalog, and compatibility costs become.

3. Contract Separation of Syntax and Semantics

Schema contract constraints include types, nullability, enumerations, and compatibility; semantic contract specifies granularity, units, time, and business rules; service contract defines freshness, availability, support, and changes.

Compatibility goes beyond just the schema:

• duration changed from minutes to seconds, type unchanged but semantics are broken;
• State done is now interpreted as "Settlement Completed," enumeration unchanged;
• Historical backfills alters past metrics, but queries can still run.

Therefore, change detection requires machine rules and owner/consumer review.

A concrete compatibility check compares a v2 batch table in integer grams with a v1 stream in decimal kilograms at the same source progress point. This standard-library example handles replay, excludes later offsets, and rejects conflicting content under one event ID:

from decimal import Decimal

stream_v1 = [
    {"offset": 1, "event_id": "logistics:1", "quantity_kg": "2.5"},
    {"offset": 2, "event_id": "logistics:2", "quantity_kg": "0.5"},
    {"offset": 2, "event_id": "logistics:2", "quantity_kg": "0.5"},
    {"offset": 3, "event_id": "logistics:3", "quantity_kg": "9"},
]
batch_v2 = {"logistics:1": 2500, "logistics:2": 500}

def snapshot_in_grams(events, high_water):
    result = {}
    for event in events:
        if event["offset"] > high_water:
            continue
        grams = Decimal(event["quantity_kg"]) * 1000
        if not grams.is_finite() or grams <= 0 or grams != grams.to_integral_value():
            raise ValueError("positive whole grams required")
        key, value = event["event_id"], int(grams)
        if key in result and result[key] != value:
            raise ValueError("same event ID with conflicting content")
        result[key] = value
    return result

normalized = snapshot_in_grams(stream_v1, high_water=2)
assert normalized == batch_v2
print(sum(normalized.values()))
# 3000

The example has one source partition. Real batch/stream reconciliation needs progress per partition, snapshot commit IDs, and correction/deletion rules. Querying “latest” at different times cannot establish parity. Unit adapters must not silently round values that cannot be expressed in whole grams; reject them or define a precision contract.

4. Platforms Should Also Be Operated as Products

Platform consumers are the domain product teams. The platform team should observe how many steps it takes to go from code to a secure release, and where failures occur, rather than just delivering a collection of infrastructure components.

A paved road can provide:

Product templates and repositories
Schema/Contract registration and compatibility checking
Deployment, scheduling, and environment isolation
Quality SLI, Logging, Lineage, and Catalog Registration
Identity, access control, and auditing
Cost, Quotas, and Resource Recommendations
Workflows for incidents, rollbacks, and retirement

The default path should be safe and low-friction, while still preserving exception extension points. Forcing all scenarios to use the same computation engine would make the platform restrictive; allowing unlimited freedom would undermine self-service and interoperability.

5. How Federated Computational Governance Is Put Into Practice

Common rules can be layered:

Globally Non-Negotiable

Identity authentication, sensitive data classification, minimum audit fields, significant use restrictions, and incident response.

Interoperability Standards

Asset ID, timestamp format, schema version, generic entity key, and contract agreement.

Domain Rules

Business enumerations, quality thresholds, release frequency, internal models, and support details.

Implement the rules as:

• Templates and default configurations;
• Schema/policy validation in CI;
• Access and category gatekeeping at time of release;
• Runtime monitoring and auditing;
• Exceptions with owner, compensating control, and expiration date.

"Computational governance" doesn't automatically decide all policies. Ambiguous or value-conflicting cases still require human judgment, with the system providing evidence and carrying out decisions.

For a rule forbidding publication of unclassified sensitive fields, CI can check classification coverage, the publisher can enforce destination policy, runtime controls can audit writes and access, and exception handling can enforce expiry. A classification field in a template cannot stop direct writes that bypass it; controls belong on actual publication and access paths.

6. Cross-Domain Identity and Composition

The product can function correctly on its own, but fails to integrate properly due to inconsistencies among entity keys, event timestamps, and slowly changing dimensions. Prioritize building a small set of shared capabilities:

• Global/mappable entity identifier;
• Master data and identity resolution;
• Event time, available time, and time zone conventions;
• Reference data version;
• Composition and attribution of cross-domain metrics.

Shared standards should address real consumer needs, not be created solely for the sake of unification, resulting in enterprise-wide data models that go unused.

7. Costs and Incentives

The domain bears production costs while other teams may receive the benefits. If owners are only evaluated on costs, they'll reduce sharing; if the platform is completely free and invisible, waste will result.

Needed:

• Display production, storage, and consumption costs;
• Provide a shared budget for cross-domain public goods;
• Incorporate reliability and consumer outcomes into domain goals;
• Retire low-value, unused products promptly;
• Don't use query volume as the sole metric of value.

8. Measuring Whether True Improvement Has Occurred

More useful metrics than product count include:

• Time from first discovery to successful use;
• Contract change lead time and breaking-change rate;
• SLO attainment, incident recovery, and recurring incidents;
• Owner/Consumer Issue Response Time;
• The number of manual mappings required for cross-domain integration;
• Platform paved road usage rate and escape reasons;
• Active products, products without consumers, and retirement dates;
• The total cost of each business outcome.

Metrics must be compared to the pre-migration baseline and safeguards must be put in place to prevent domains from optimizing numbers by splitting products or hiding incidents.

9. Progressive Migration

1. Pick a consumer-facing cross-domain value stream
2. Clearly define the domain owner, product interfaces, and the current baseline
3. Improve the two or three most painful steps in the platform release process
4. Establish a minimum global policy and exception mechanism
5. Run both old and new interfaces to verify semantics and SLO
6. Consumers migrate and provide feedback
7. Retire old paths, review organizational and platform gaps
8. Extend into the next domain only after the pattern is reusable

Don't preset generic phases of 6, 12, or 24 months. The pace depends on the existing platform, team capabilities, risk factors, and scope of migration.

Common Misconceptions

• A table with an owner is a data product: it's still missing interfaces, semantics, SLOs, support, and a lifecycle.
• Self-service means no platform support: The platform handles turning complex controls into usable capabilities.
• Federated means all rules are optional: Global baselines and interoperability agreements still need to be enforced.
• The more products, the more successful the transition: Over-splitting increases consumption and maintenance costs.

Practice

1. Complete the semantics, service-level, and security contracts for an event data product.
2. Draw the paved road from the domain team submitting code to product release and identify manual bottlenecks.
3. Divide the three governance policies into global, interoperability, and domain layers, then define execution points.
4. Select five consumer outcome metrics for the first pilot, rather than product quantities.

Summary

Data Mesh autonomy must be achieved through stable product APIs, shared platform capabilities, and enforceable governance to enable interoperability. True success isn't about changing the organizational chart; it's about domains delivering faster and more securely, and consumers relying less on oral knowledge and manual coordination.

The final chapter examines privacy engineering: regardless of whether data is centrally or locally managed, control over the risks to individuals caused by collection, linkage, access, retention, and dissemination must be ensured.