20.2 Near-real-time Indexing, Synchronization, Sharding, and Search Operations Maintenance

Search indexes are typically derived views of authoritative databases. They can be temporarily outdated, can be rebuilt, but they cannot quietly become a second arbiter of inventory, pricing, or permissions.

Near-real-time from Refresh, not write failure

Lucene organizes its index using immutable segments. New documents first enter in-memory index structures and a log path, then a refresh operation opens a new searchable segment; background merges subsequently consolidate smaller segments.

After Elasticsearch's index API returns success, documents aren't guaranteed to be immediately discoverable by the search API. The default refresh behavior varies by deployment and index activity level, so you can't hardcode "always 1 second." For a read-after-write search experience, evaluate refresh=wait_for, which makes write requests wait for the next natural refresh; frequently forcing refresh=true creates numerous small segments and increases merge costs.

Also need to distinguish:

• refresh: Make the most recent actions searchable;
• flush: advance the persistence boundary and handle translog/commit-related states;
• merge: merge immutable segments and reclaim physical space occupied by deleted documents;
• replica acknowledgment: the conditions under which replica acknowledgments determine index write commitment.

They're not the same "disk spinning" action.

Primary Shard Decides Data Sharding

Document IDs or routing values are mapped to the primary shard, and replica shards store a copy of the primary's data. Search requests are forwarded by the coordinating node to the relevant shard copies, then the top results from each shard are merged.

query
  -> shard 0 top-k
  -> shard 1 top-k
  -> shard 2 top-k
  -> coordinator merge/rerank

Too few primary shards limit parallelism and capacity, while too many increase heap usage, file handles, cluster state overhead, scatter/gather operations, and recovery costs. Replicas improve fault tolerance and potential read throughput without increasing the number of shards for write data.

Custom routing can allow a single tenant/entity query to access only a subset of shards, but it may also create hotspots and excessively large shards. Planning should use real document sizes, query fan-out, recovery times, and growth curves, rather than fixed "X GB per shard" universal numbers.

Synchronizing Database to Search Index

Reliable links are typically:

database transaction
  -> outbox/WAL/CDC
  -> durable event log
  -> idempotent indexer
  -> search index
  -> checkpoint + lag metrics

Events must carry a stable entity ID, source version, and deletion semantics. The indexer must reject out-of-order events where an older version attempts to overwrite a newer one; deletion cannot be inferred solely from the absence of a source table entry, it must be explicitly propagated via tombstone or explicit delete events.

Full-cycle synchronization easily mixes in increments during scanning, causing gaps or rollbacks. Correct reconstruction requires a consistent snapshot position: first record or obtain the WAL/offset corresponding to the snapshot, then fill in the snapshot, and finally consume the increments from the handoff position.

Reindex Switching to a New Index and Alias

When modifying the analyzer, field types, or primary shard count, a new version of the index is typically created:

products-v3 (current alias target)
products-v4 (backfill + catch up + verify)

Process:

1. Create an explicit mapping/settings for v4;
2. Fill in full data from a consistent snapshot;
3. Catch up with the incremental data and record the checkpoint;
4. Compare quantities, sample documents, filtering, and relevance;
5. Atomically switch read-write alias;
6. Keep the rollback window intact, then delete v3.

Dual-writing v3/v4 still requires handling scenarios where one side succeeds and the other fails; projecting from the same CDC log typically makes replay and auditing easier.

Search results cannot directly settle transactions

When users see "in stock" on the search page and click to place an order, they must re-verify in the authoritative transaction database and perform atomic deduction. If permission revocation needs to take effect immediately, it can't just wait for the index to refresh.

Search services should declare a freshness SLO, for example:

• 99% of products are searchable within 30 seconds;
• Remove/remove from public results within 10 seconds;
• Strong permission validation is performed by the authoritative service when returning details or executing actions.

Different fields can have different risks and degradation paths.

Deep Paging and Result Consistency

The deeper from + size, the more candidates each shard needs to retain and return, causing the coordination node cost to rise rapidly. For rolling lists, prefer search_after with a stable tie-breaker; when maintaining consistent views across multiple pages, combine it with point in time (PIT).

The sort field must be stable and deterministic. When sorting by score alone, documents with the same score may change order after a refresh, replica, or shard state change.

Lexical, Vector, and Hybrid Search

Vector retrieval excels at semantic similarity but doesn't naturally guarantee exact matching, negation, up-to-date facts, or permission filtering. Lexical retrieval is strong at exact term matching and interpretable filtering. A hybrid approach typically:

lexical candidates + vector candidates
  -> normalize / RRF / learned reranker
  -> hard business filters

The embedding model version should be written to the index metadata; upgrading a model often requires recomputing vectors and evaluations. Vector retrieval also requires offline judgement, latency, and memory/disk cost testing, and cannot be assessed based on just a few demonstration queries.

Operations Metrics

At least monitor:

• indexing rate, reject, bulk error, and CDC lag;
• refresh, segment count, merge time/throttle;
• shard size, allocation, recovery, and cluster health;
• search p50/p95/p99, timeout, cancel, and shard failures;
• query fan-out, slow log, cache, and heap/GC;
• Zero-result rate, Recall/nDCG sampling and permission leakage inspection;
• Gap between source version and index checkpoint;
• Time required for full rebuild and alias rollback.

Green cluster indicates only shard allocation status, not that data has synchronized with the authoritative source, nor that search quality is accurate.

Reference

• Elasticsearch: Refresh API
• Elasticsearch: Search Shard Routing
• Elasticsearch: Index Statistics
• Lucene: BM25Similarity

Thus, the professional narrative arc of Volume Five has evolved from relational models all the way through caching and search-derived views. The next phase of full-volume verification will examine code, links, directories, and terminology, before moving on to narrative consistency and writing style.