18.2 Data Architecture, Derived Views, and Safe Evolution

Keep a clear authoritative core, then add derived read models when a workload justifies them. Each derived system needs its own rebuild, monitoring, and retirement path; otherwise the architecture becomes harder to change with every addition.

Evolution Order Starting from a Single Database

transaction database
  -> add indexes and query fixes
  -> read replica / partitioning when measured
  -> cache for hot reconstructable reads
  -> outbox + event log for reliable propagation
  -> search index / analytical store / graph projection

This check order is meant to manage complexity, not enforce a migration path. Each layer added should clearly address a specific SLO and introduce new failure modes.

Derived views must be traceable to their source

Search documents, cache entries, and analytical rows should carry source metadata where practical:

• Source entity or event ID;
• source version / LSN / offset / event time;
• Projection schema version;
• Last successfully updated time and error status.

These fields help trace stale data, duplicates, and conversion errors, but are not interchangeable clocks. A Kafka offset orders one partition; an LSN belongs to a particular log stream; event time may arrive out of order. Freshness comparisons need a defined source version or ordering protocol.

Lambda and Kappa should not be reduced to an architecture diagram

The Lambda architecture maintains both batch and stream processing paths, then combines the results at the serving layer. It can use batch processing to correct stream computations, but it requires maintaining two separate logical systems, which easily leads to semantic drift.

Kappa architecture uses a replayable event log as its primary input, reprocessing it with the same stream processing logic. It reduces two codebases, but only if the log is retained long enough, the event schema can evolve, replay doesn't overwhelm the sink, and external reference data can be reconstructed.

When choosing, look at the conditions below; the name doesn't matter:

• Can an authoritative history be fully reconstructed;
• How late events and corrections are represented;
• How state and checkpoint recovery works;
• How rebuilt results replace the live view;
• How to measure the freshness and completeness of results.

Lakehouses Depend on Table Protocols and Governance

Object storage is cheap and scalable, but raw files lack transaction commits, snapshots, schema evolution, and small file management. A lakehouse maintains metadata and snapshots on top of objects, enabling multiple compute engines to see a consistent table version.

Table operations still need rules for:

• Partition and clustering design;
• Small file compaction;
• schema/partition evolution;
• snapshot expiration and orphan files
• Catalog permissions, lineage, and data quality;
• Write concurrency and engine compatibility.

Engines also need compatible catalogs and table protocols and must read an identified snapshot. Files from an in-progress writer may temporarily appear unreferenced; orphan cleanup with a retention window shorter than the longest write can delete files being committed. See Iceberg maintenance.

Putting data into object storage does not automatically make it a lakehouse.

Why Dual-Write Migrations Are Dangerous

When an application writes to both the old and new data stores, a timeout in either step leaves an uncertain state. Do not treat two ordinary API calls as a transaction.

A safer migration typically includes:

1. Obtain a consistent snapshot and matching change-stream position through the connector protocol, retaining the required logs;
2. Backfill the snapshot while continuously capturing changes after that position;
3. Apply changes idempotently and checkpoint, preventing old backfill from overwriting newer applied versions;
4. Compare the semantic results of shadow reads;
5. Shift a small percentage of reads while retaining a fast rollback path;
6. Fence the old write path, drain in-flight writes, and catch up through the final change before switching write authority;
7. Retire the old system only after preserving the audit window.

Suppose a snapshot contains order v7, but CDC has already written v8 into the new store. An unconditional late backfill upsert would restore v7. Sequence changes after the snapshot, or reject stale backfill using a per-entity source version. After write cutover, an old store that no longer receives updates cannot support rollback by changing a connection string alone; synchronize the reverse delta or explicitly limit rollback to safe read paths.

Comparisons cannot rely solely on row counts. Sorting, NULL values, floating-point arithmetic, time zones, duplicate keys, and aggregation rounding can all introduce semantic differences.

Reversibility is a design requirement

Prepare an exit path when introducing a system:

• Can the full data, schema, offset, and ACL be exported?
• Can the export speed meet the migration window?
• How are proprietary functions or data types mapped?
• Can encryption keys, audit logs, and backups be transferred?
• Do licenses and hosting services restrict data migration?

Unverified export and recovery paths are equivalent to having no exit strategy.

ADR Template Skeleton

Decision:
Add a standalone inverted index for product search; order facts will still be based on PostgreSQL.

Context:
Keyword, synonyms, and sorting requirements cannot be met by the current B+ tree index; allow 30 seconds of staleness.

Correctness boundary:
Search results are not a source for inventory and pricing decisions; inventory and price validation occur against the authoritative database at checkout.

Sync:
Outbox + CDC; event_id idempotency; monitor source LSN and index checkpoint.

Failure behavior:
If the index is unavailable, fall back to category browsing; keep the search system from writing back into the order database.

Exit:
Full-rebuild script, event retention period, index schema, and export workflow.

The professional theme throughout the entire volume

This volume organizes twenty topics along a continuous chain:

Relationships and Constraints
-> Physical pages, indexes, logging, and recovery
-> Query execution, optimization, and CPU efficiency
-> Transactions, MVCC, and Concurrency Control
-> Special data model
-> Sharding, Replication, and Cross-Shard Transactions
-> Event propagation
-> Derived read models and architectural evolution

The next chapter examines the most common derived read model, caching. The focus shifts to the correctness protocol between cache and authoritative data, rather than GET/SET API.