20.1 Inverted Indexes, Analyzer, and Relevance

A B+ tree can narrow a lookup by product ID. A query such as “lightweight waterproof jacket” also needs tokenization, retrieval, and ranking. Search systems prepare terms and statistics so that queries can start with candidate documents.

What LIKE '%word%' Cannot Provide

Leading wildcard queries typically cannot use standard B+ tree prefix ordering and require scanning large volumes of text. While database extensions like trigrams or inverted full-text indexing can address certain scenarios, it's incorrect to claim that "database indexes can't search for keywords."

Dedicated search systems combine several capabilities:

• analyzers, synonym expansion, stemming, and multilingual support;
• inverted indexes with phrase position tracking;
• boolean filtering, relevance scoring, field weighting, and function scoring;
• aggregation, highlighting, suggestions, and distributed queries;
• large-scale index lifecycle management and near-real-time updates.

If the requirement involves only a few exact substrings or small-scale in-app table searches, native full-text capabilities built into databases might be simpler and more adequate.

An Inverted Index Maps Terms to Postings

Start with these supplied token sequences, using positions numbered from 1. These illustrate postings without relying on a particular language analyzer:

d1: waterproof jacket lightweight
d2: waterproof backpack
d3: jacket commute

The inverted index can be conceptualized as:

waterproof -> (d1, tf=1, positions=[1]), (d2, tf=1, positions=[1])
jacket -> (d1, tf=1, positions=[2]), (d3, tf=1, positions=[1])

In practice, the index compresses encodings of document ID differences, term frequencies, and positions, and maintains additional structures such as field lengths and doc values. Whether positions, offsets, or term vectors are stored directly affects performance in phrase queries, highlighting, and disk and write overhead.

Analyzer is the Index Contract

A typical analysis pipeline:

character filters -> tokenizer -> token filters

• Character filters normalize the original text before tokenization;
• The tokenizer defines boundaries for tokens;
• Token filters apply operations such as lowercasing, stopword removal, synonym expansion, and stemming.

Chinese text lacks inherent whitespace boundaries, so tokenization based on single characters, dictionary entries, statistical models, or subword splitting can yield different recall rates. Choose analyzers for product names, model numbers, codes, and natural language according to query semantics; a shared configuration needs validation.

The standard analyzer combines standard tokenization with lowercasing; the exercise below checks its output.

The index analyzer and query analyzer can differ, but they must be semantically compatible. Modifying an analyzer typically does not automatically rewrite existing postings and requires creating a new index and reindexing.

text and keyword Are Two Search Paths

These request bodies target the Elasticsearch 8.x/9.x REST API and require an accessible practice cluster with index creation, write, and search permissions. Confirm that products-lab does not exist, then send this JSON as the body of PUT /products-lab. Use a fresh practice index. The explicit standard analyzer makes the English tokens easy to inspect:

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "raw": { "type": "keyword" }
        }
      },
      "status": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 }
    }
  }
}

name is used for full-text matching, while name.raw is used for precise filtering, sorting, or aggregation. Automatically mapping dynamic fields into the index can lead to mapping explosion; in production environments, template complexity, field count, and type evolution must be carefully controlled.

Query and Filter Division

Send POST /products-lab/_analyze with this body. The expected tokens are waterproof and jacket:

{ "field": "name", "text": "Waterproof Jacket" }

Next, send the following bodies to PUT /products-lab/_doc/1?refresh=wait_for, /2?refresh=wait_for, and /3?refresh=wait_for, respectively. The latter two paths also start with /products-lab/_doc:

{ "name": "Waterproof Jacket", "status": "available", "price": 80 }

{ "name": "Waterproof Backpack", "status": "available", "price": 65 }

{ "name": "Jacket Commute", "status": "available", "price": 120 }

This exercise assumes automatic refresh is enabled; the next lesson explains the waiting boundary of wait_for. Prices here are in yuan. scaled_float stores values after multiplying by 100 and rounding to an integer; this does not replace monetary validation in the transaction database.

Send the following body to POST /products-lab/_search:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": { "query": "waterproof jacket", "operator": "and" } } }
      ],
      "filter": [
        { "term": { "status": "available" } },
        { "range": { "price": { "lte": 10000 } } }
      ]
    }
  }
}

Only document 1 should match because both terms are required. The default match operator is OR; removing operator: "and" allows all three documents to match at least one term, and all pass the price filter. AND does not require adjacent terms; express phrase order with a query such as match_phrase.

Full-text queries participate in relevance scoring, while structured filters only determine whether a document matches. Hard conditions such as permissions, tenant identity, and inventory status must be enforced by filters. A low score does not remove an unauthorized result.

BM25 Is Balancing Three Key Factors

Lucene BM25 balances three factors:

• The more times a term appears in a document, the greater its contribution, though this effect eventually plateaus;
• Terms that are rarer across the entire corpus tend to offer stronger discriminative power;
• Longer documents naturally contain more relevant matches, so they are normalized for length to prevent bias.

The score produced by BM25 is not a direct probability of relevance across queries or across indexes. Factors such as shard-level statistics, field boosting, synonym expansion, and index schema changes all influence _score. Never interpret a score greater than 1 as a probability of relevance.

Lexical relevance should be augmented with business signals, but only within bounded, controlled limits:

final_score = lexical relevance
            + bounded freshness boost
            + bounded popularity boost

If popularity is simply added without bounds, high-volume items will permanently dominate and suppress more relevant newer content.

Relevance Must Be Evaluated with a Judgement Set

Build relevance judgments for representative queries that cover popular queries, long-tail queries, typos, polysemous terms, zero-results, and permission-based filtering. Common offline metrics include:

• Precision@k: the proportion of relevant results among the top k results;
• Recall@k: the fraction of relevant results that appear in the top k;
• MRR: the mean, across queries, of the reciprocal rank of the first relevant result; a query with no relevant result contributes zero;
• nDCG@k: a metric that accounts for graded relevance and ranking decay.

Suppose d1 and d3 are judged relevant and the returned order is d1, d2, d3. Precision@2 = 1/2, Recall@2 = 1/2, and reciprocal rank = 1. If the first relevant result for a second query ranks third, MRR = (1 + 1/3) / 2 = 2/3. Recall depends on the scope of the judgments; missing relevance labels affect its interpretation.

After evaluating with offline metrics, conduct A/B testing to observe user behavior such as click-through rates, conversion rates, zero-result rates, and query reformulation. Click logs are subject to positional bias and cannot be treated as unbiased ground truth.

Section Checkpoint

• Can explain the purpose of inverted lists, term frequency, position, and field length.
• Can design different fields for natural language queries, precise filtering, and sorting.
• Can distinguish between retrieval, filtering, and ranking, without treating _score as a probability.
• Can evaluate changes in analyzers and ranking models using a judgment set.

The next section moves into production workflows: when new documents become searchable, how indexing is split across shards, and how authoritative databases stay synchronized with search projections.