18.2 Data Architecture, Derived Views, and Security Evolution

A mature data architecture rarely relies on a single system to do everything, and it's not advisable to pile on all components from day one. A more prudent approach is to maintain a clear, authoritative core, add derived read models only as needed and demonstrably justified, and ensure each derived system can be rebuilt, monitored, and shut down independently.

Evolution Order Starting from a Single Database

transaction database
  -> add indexes and query fixes
  -> read replica / partitioning when measured
  -> cache for hot reconstructable reads
  -> outbox + event log for reliable propagation
  -> search index / analytical store / graph projection

This check order is meant to manage complexity, not enforce a migration path. Each layer added should clearly address a specific SLO and introduce new failure modes.

Derived views must be traceable to their source

It's best to index, cache, and analyze each data entry in tables for search.

• Source entity or event ID;
• source version / LSN / offset / event time;
• Project schema version;
• Last successfully updated time and error status.

Only then can we answer whether this is old data, redundant data, or a logic error in transformation, and support replaying from a known position.

Lambda and Kappa should not be reduced to an architecture diagram

The Lambda architecture maintains both batch and stream processing paths, then combines the results at the serving layer. It can use batch processing to correct stream computations, but it requires maintaining two separate logical systems, which easily leads to semantic drift.

Kappa architecture uses a replayable event log as its primary input, reprocessing it with the same stream processing logic. It reduces two codebases, but only if the log is retained long enough, the event schema can evolve, replay doesn't overwhelm the sink, and external reference data can be reconstructed.

When choosing, look at the conditions below; the name doesn't matter:

• Can an authoritative history be fully reconstructed;
• How to express being late and making corrections;
• How state and checkpoint recovery works;
• How to switch between recalculation period and online results;
• How to measure the freshness and completeness of results.

Lakehouse is a problem with table protocols and governance

Object storage is cheap and scalable, but raw files lack transaction commits, snapshots, schema evolution, and small file management. Lakehouse maintains metadata and snapshots on top of objects, enabling multiple compute engines to see a consistent table version.

Still requires governance:

• Partition and clustering design;
• Small file compaction;
• schema/partition evolution;
• snapshot expiration and orphan files
• Catalog permissions, lineage, and data quality;
• Write concurrency and engine compatibility.

Putting data into object storage doesn’t automatically make it a lakehouse.

Why Double-Booking Transfers Are Dangerous

When applying writes to both the old and new libraries simultaneously, a timeout in either step leaves an uncertain state. Don't treat two regular API calls as a transaction.

A safer migration typically includes:

1. Populate the new system with a full snapshot from the authoritative repository;
2. Continuously capture incremental changes via WAL/CDC/outbox;
3. Use a stable checkpoint to bridge full and incremental updates, avoiding gaps;
4. Compare shadow read with semantic results;
5. Small-scale scroll reading, retain fast backward navigation;
6. Freeze and verify the old write entry before switching to authoritative write;
7. Retire the old system only after preserving the audit window.

Comparisons can't be based solely on line count. Sorting, NULL values, floating-point arithmetic, time zones, duplicate keys, and aggregation rounding can all introduce semantic differences.

Reversibility is a design requirement

Exit upon system introduction:

• Can the full data, schema, offset, and ACL be exported?
• Can the export speed meet the migration window?
• How are proprietary functions or data types mapped?
• Can encryption keys, audit logs, and backups be transferred?
• Do licenses and hosting services restrict data migration?

Unverified export and recovery paths are equivalent to having no exit strategy.

ADR Template Skeleton

Decision:
Add a standalone inverted index for product search; order facts will still be based on PostgreSQL.

Context:
Keyword, synonyms, and sorting requirements cannot be met by the current B+ tree index; allow 30 seconds of staleness.

Correctness boundary:
Search results are not a source for inventory and pricing decisions; price validation occurs against the authoritative database at checkout.

Sync:
Outbox + CDC; event_id idempotency; monitor source LSN and index checkpoint.

Failure behavior:
Falls index unavailable, degrade to category browsing; do not allow search system to reverse-write order database.

Exit:
Full-rebuild script, event retention period, index schema, and export workflow.

The professional theme throughout the entire volume

This volume organizes twenty topics along a continuous chain:

Relationships and Constraints
-> Physical pages, indexes, logging, and recovery
-> Query execution, optimization, and CPU efficiency
-> Transactions, MVCC, and Concurrency Control
-> Special data model
-> Sharding, Replication, and Cross-Shard Transactions
-> Event propagation
-> Derived read models and architectural evolution

The next chapter delves into the most common derived read model, caching. The focus shifts to the correctness protocol between cache and authoritative data, rather than GET/SET API.