BM25 in Elasticsearch
The relevance formula quietly running behind every search you've ever typed into Elasticsearch — and the three ideas that make it work.
Imagine grading two student essays that both use the word “photosynthesis.” One essay is a single paragraph that mentions it once; the other is a ten-page report that mentions it fifty times. Naively, you might think the ten-page report is “fifty times more about photosynthesis” — but a fair grader knows that’s not quite right, because a longer document naturally contains more words of everything. A fair grader also knows that a rare, specific word like “photosynthesis” tells you more than a common word like “the.” Balancing exactly these two instincts — reward genuine repetition, but don’t get fooled by document length or common words — is the entire job of the scoring formula every modern Elasticsearch search relies on by default: BM25.
1What Is BM25?
Let’s build the plain-English picture before looking at the individual ingredients that make up the formula.
BM25 stands for “Best Matching 25,” a name that traces back to a series of information-retrieval research experiments where it was the twenty-fifth formula variant tested. It is a mathematical formula used to calculate how relevant a document is to a search query, and it has been Elasticsearch’s default relevance-scoring algorithm since version 5. Every time you run a search and see documents returned in a particular order with an associated score, that score — unless you’ve deliberately configured something else — was almost certainly computed by BM25.
Think of BM25 as a strict but fair judge scoring how well a book answers your question. The judge cares how many times your search words appear in the book (but doesn’t keep getting more impressed after a certain point), rewards words that are rare and specific across the whole library rather than common everyday words, and adjusts expectations based on how long the book actually is, since a thick book naturally contains more words of everything.
Term Frequency (TF)
How many times a specific search word appears within a single document.
Inverse Document Frequency (IDF)
A measure of how rare or common a word is across the entire collection of documents in an index.
Field-Length Normalization
An adjustment that accounts for a document’s length, so shorter and longer documents are compared fairly.
Relevance Score
The final number BM25 produces for a document, used to rank search results from most to least relevant.
A practical example: searching a collection of recipe articles for “cardamom.” Because “cardamom” is a relatively rare, distinctive ingredient compared to a common word like “salt,” BM25’s rarity component gives it more scoring weight. Meanwhile, an article that mentions cardamom five times will score higher than one mentioning it once — but not proportionally five times higher, and not higher at all once the count gets unusually large, which brings us to the specific mathematical shaping BM25 applies, covered in detail in the next two chapters.
BM25 is not unique to Elasticsearch — it is a widely studied, decades-old algorithm from the broader field of information retrieval, used across many different search systems. Elasticsearch simply adopted it as its default.
2Architecture and Components
BM25 doesn’t operate in isolation — it depends on statistics collected across an entire index. Let’s see how it fits into Elasticsearch’s broader search architecture.
graph TD
A["Index Statistics"] --> B["Term Frequency per document"]
A --> C["Document Frequency across the index"]
A --> D["Average field length across the index"]
B --> E["BM25 Formula"]
C --> E
D --> E
E --> F["Per-document relevance score"]
F --> G["Ranked search results"]
Component: The Inverted Index
Elasticsearch’s core search structure already tracks, for every unique term, which documents contain it and how often — this is precisely the raw data BM25 draws on for term frequency and document frequency.
Component: Per-Field Similarity Settings
BM25 is configured as the “similarity” algorithm for a field in the index mapping. While BM25 is the default, Elasticsearch allows swapping in alternative similarity algorithms per field, and BM25 itself exposes tunable parameters covered later in this tutorial.
Component: The Query Execution Layer
When a match or multi-match query runs, the query execution layer is what actually invokes the configured similarity algorithm (BM25, by default) once per matching document to compute its score.
This architecture explains an important practical fact: BM25 scores are always relative to the specific index they were computed against, since document frequency and average field length are index-wide statistics. The same document, indexed into two different indices with different overall content, could receive two different BM25 scores for an identical search term.
3Internal Working — How the Formula Actually Behaves
This is where the three ingredients from Chapter 1 combine, and where BM25’s real cleverness becomes visible.
The first ingredient, term frequency, does not increase a document’s score in a straight, ever-climbing line. Instead, BM25 applies a saturation effect: going from zero to one mention of a term boosts the score noticeably, going from one to two mentions boosts it a bit less, and by the time a term has appeared a dozen times, each additional mention adds almost nothing further. This deliberately mirrors real reading intuition — a document mentioning your search term fifty times isn’t meaningfully “more about” that topic than one mentioning it fifteen times.
flowchart LR
A["1 mention"] --> B["Score contribution: fast rise"]
C["5 mentions"] --> D["Score contribution: rising, slower"]
E["20 mentions"] --> F["Score contribution: nearly flat"]
The second ingredient, inverse document frequency, works by comparing how many documents in the entire index contain a given term against the total number of documents. A term appearing in only a handful of documents out of millions is treated as highly distinctive and given a large scoring weight; a term appearing in nearly every document (common words like “the” or “and,” if they weren’t already filtered out during analysis) is treated as carrying very little discriminating value and given a much smaller weight.
If a witness says “the suspect was wearing clothes,” that tells you almost nothing, because it’s true of everyone. If a witness says “the suspect was wearing a bright orange scarf,” that tells you a great deal, precisely because it’s rare. IDF is BM25’s way of automatically recognizing which search words behave like “clothes” and which behave like “bright orange scarf.”
The third ingredient, field-length normalization, adjusts term frequency relative to how long the document’s field actually is compared to the average field length across the index. Without this adjustment, a naturally long document would tend to accumulate higher raw term counts purely by virtue of containing more words overall, unfairly outscoring a shorter, tightly-focused document that may actually be a better match. BM25 counterbalances this by slightly discounting term frequency in longer-than-average fields and slightly boosting it in shorter-than-average ones.
It’s tempting to assume “the document with the most matching words wins.” BM25 explicitly resists this by combining diminishing returns on repetition with length normalization, meaning a short, focused document can legitimately outscore a much longer one containing the same term more times in raw count.
These three effects — frequency saturation, rarity weighting, and length normalization — are controlled by two tunable parameters built into the formula, conventionally named k1 and b. The k1 parameter controls how quickly term frequency saturates (a higher value lets repeated terms keep contributing longer before flattening out), and the b parameter controls how strongly field-length normalization is applied, ranging from no length adjustment at all to full adjustment.
4Data Flow and Lifecycle
Let’s trace exactly when BM25’s ingredients are gathered and when the actual score gets calculated.
Indexing Time — Term Statistics Collected
As documents are indexed, Elasticsearch continuously updates term frequency and document frequency statistics for every field configured with BM25 as its similarity.
Indexing Time — Field Length Recorded
The length of each document’s analyzed field is also recorded, contributing to the index-wide average length figure BM25 will later use for normalization.
Query Submission
A search request (such as a match query) is submitted, specifying the field and search terms to evaluate.
Candidate Retrieval
Elasticsearch uses the inverted index to quickly identify the set of documents that contain at least one of the query’s search terms.
Per-Document BM25 Calculation
For each candidate document, BM25 combines that document’s term frequency, the index-wide document frequency, and the field-length normalization into a single relevance score.
Shard-Level and Cluster-Level Ranking
Each shard ranks its own candidates by BM25 score, and these rankings are merged across shards to produce the overall top results for the whole index.
Response
Results are returned to the client sorted by BM25 score by default, unless the query explicitly specifies a different sort order.
An important detail from this lifecycle is that document frequency and average field length are collected continuously as documents are added, updated, or removed — meaning BM25 scores for the exact same query can shift slightly over time purely because the overall content of the index has changed, even if the specific document being scored hasn’t been touched at all.
5BM25 vs. Classic TF-IDF
BM25 didn’t appear out of nowhere — it improved on an earlier, simpler scoring approach called TF-IDF, which Elasticsearch itself used as its default in older versions.
| Aspect | Classic TF-IDF | BM25 |
|---|---|---|
| Term frequency growth | Grows roughly without limit as a term repeats | Saturates — extra repetitions add diminishing value |
| Document length handling | Limited or no normalization | Built-in, tunable length normalization |
| Tunability | Largely fixed behavior | Adjustable via k1 and b parameters |
| Elasticsearch default | Used in versions before 5 | Default since version 5 |
| Resistance to keyword stuffing | Weaker — repeated terms keep helping | Stronger — repetition quickly stops mattering much |
The practical upshot of this evolution is that BM25 behaves more sensibly across a wider range of real-world content — long articles, short snippets, keyword-heavy spam, and naturally-written prose all get scored more fairly than they would have under the older, simpler formula. This is precisely why Elasticsearch made the switch by default, and why deliberately reverting to classic TF-IDF today is rarely recommended except in specific research or compatibility scenarios.
6Advantages, Disadvantages, and Trade-offs
BM25’s design reflects decades of refinement, but like any general-purpose formula, it has boundaries worth understanding.
Advantages
- Produces intuitive, well-behaved relevance rankings across a huge variety of document types and lengths without manual tuning.
- Naturally resistant to simple keyword-stuffing tactics, since repeated terms quickly stop adding meaningful score.
- Backed by decades of information-retrieval research and widely proven across countless production search systems.
- Exposes just two tunable parameters (k1 and b), making customization approachable without needing a deep mathematical background.
- Works automatically as soon as documents are indexed, requiring no explicit configuration to get reasonable relevance out of the box.
Disadvantages / Trade-offs
- Relies purely on term statistics — it has no understanding of meaning, synonyms, or context beyond what analyzers provide beforehand.
- Scores are index-relative, meaning they cannot be meaningfully compared across two different indices with different content.
- Doesn’t account for business signals like popularity or recency on its own — those require layering additional query behavior, such as a function score query, on top.
- Default k1 and b values, while sensible for many use cases, are not automatically optimal for every unique dataset or domain.
These trade-offs are less a flaw and more a reflection of BM25’s intentionally narrow, well-defined job: score text relevance based on term statistics, and do that one job extremely well, leaving everything else — semantics, business logic, personalization — to other tools working alongside it.
7Security Considerations
BM25 itself is a read-only scoring calculation, so its security surface is narrower than many other Elasticsearch topics, but a few considerations still apply.
Because BM25 rewards rare, distinctive terms with higher scores, and index-wide statistics like document frequency are derived from the full dataset, exposing raw relevance scores or detailed scoring explanations in an untrusted, public-facing interface can occasionally leak subtle information about the overall composition of an index’s content.
Mitigation: Avoid Exposing Raw Scores and Explanations Publicly
Reserve detailed score breakdowns (such as those from the explain API) for internal debugging and trusted administrative tooling, rather than surfacing them directly to end users of a public search interface.
Mitigation: Apply Standard Access Controls Before Scoring
Since BM25 only scores documents a query is allowed to see, ensure document-level and field-level security are correctly configured upstream, so scoring never even considers content a given user shouldn’t have access to in the first place.
Mitigation: Be Cautious When Allowing Custom k1 or b Overrides
If an application ever allows adjusting similarity parameters per request or per index programmatically, restrict who can make such changes, since poorly chosen values could be used to degrade search quality or performance for other users of a shared index.
Overall, BM25’s security footprint is small precisely because it does one narrowly-scoped, read-only job — the broader access-control and data-governance practices that apply to Elasticsearch generally are what matter most here.
8Monitoring, Logging, and Metrics
Because BM25 quietly underlies almost every search result, monitoring its effective behavior is really about monitoring search relevance quality over time.
Average Field Length Drift
Tracking how an index’s average field length changes over time helps explain why relevance rankings might shift even without any query changes.
Top-Result Click-Through Rate
A gradual decline in users clicking top-ranked results can indicate that default k1 and b values no longer fit how the dataset or user queries have evolved.
Zero and Near-Zero Score Rate
A rising share of matches with very low scores may indicate overly common terms dominating queries, worth investigating alongside your analyzer and stop-word configuration.
Explain API Output During Investigations
When a specific ranking looks wrong, the explain API’s detailed BM25 breakdown — term frequency, document frequency, and length normalization contributions — is the most direct diagnostic tool available.
A practical monitoring habit worth adopting is periodically re-running a fixed set of representative benchmark queries against production data and comparing their top results over time, since BM25’s index-relative nature means relevance can genuinely drift as the underlying dataset grows or changes shape, even without a single line of query code being modified.
sequenceDiagram
participant App
participant ES as Elasticsearch
participant Mon as Monitoring
App->>ES: Benchmark query set
ES-->>App: BM25-ranked results
App-->>Mon: Log top results and scores
Mon->>Mon: Compare against historical baseline
Mon-->>App: Flag drift in ranking quality
9Design Patterns and Anti-patterns
A handful of proven approaches for working with BM25 well, along with mistakes that consistently undermine relevance quality.
Pattern: Tune b Down for Fields With Naturally Variable Length
For fields where length differences are expected and not meaningful — like article bodies of varying depth — lowering the b parameter reduces how harshly longer documents are penalized purely for their length.
Pattern: Layer Business Logic on Top, Not Instead Of, BM25
Use BM25 for what it does best — text relevance — and combine it with a function score query for recency, popularity, or geographic signals, rather than trying to force BM25 itself to account for factors it was never designed to consider.
Problem
Repeating important keywords artificially many times within a document’s content, expecting a proportional relevance boost.
Why It’s Harmful
BM25’s term frequency saturation means this tactic yields rapidly diminishing returns, while also degrading the actual readability and quality of the content for real users.
Correct Approach
Focus on writing clear, naturally-worded content and, where appropriate, use field boosting or well-chosen field selection in queries to emphasize genuinely important fields instead.
Problem
Directly comparing raw BM25 scores returned from two entirely different indices, or from the same index at two very different points in time, to make relevance judgments.
Why It’s Harmful
Because BM25 depends on index-wide statistics like average field length and document frequency, scores are only meaningful relative to the specific index and dataset state they were computed against.
Correct Approach
Compare relative rankings within a single query’s results, or use consistent benchmark queries against the same index over time, rather than treating raw score values as universally comparable numbers.
The recurring lesson here is that BM25 is a precise, well-understood tool for one specific job — respecting that scope, rather than fighting against it or over-relying on it for unrelated concerns, is what separates smooth relevance tuning from frustrating trial and error.
10Best Practices and Common Mistakes
A concentrated, practical checklist for getting the most out of BM25 in real applications.
Start with the default k1 and b values
They are well-tested defaults suited to a broad range of content; only adjust them after observing a specific, measured relevance problem.
Combine BM25 with function score for non-textual signals
Let BM25 handle text relevance and layer recency, popularity, or location logic on top rather than expecting one formula to do everything.
Use the explain API when debugging odd rankings
It reveals exactly how term frequency, document frequency, and length normalization each contributed to a confusing result.
Assume more keyword repetition always means better ranking
BM25’s saturation effect makes this assumption false past a fairly low number of repetitions.
Compare raw scores across different indices
BM25 scores are only meaningful in the context of the specific index’s own term statistics.
Forget that analyzer choices affect BM25 indirectly
Stop-word removal, stemming, and tokenization all shape what BM25 even sees as a “term,” making analyzer configuration just as important as the BM25 parameters themselves.
If search results feel biased toward unusually long or unusually short documents, adjusting the b parameter is usually a more targeted fix than reworking your entire query structure.
11Real-World and Industry Examples
BM25’s influence is everywhere in modern search, even when users never see its name.
Enterprise Document and Knowledge-Base Search
Internal company search tools rely on BM25 to surface the most relevant policy documents, wikis, or support articles purely from the words a user types, without any manual ranking rules required.
E-Commerce Product Search
Baseline product relevance in most Elasticsearch-powered storefronts starts with BM25 scoring product names and descriptions, later enhanced with function score boosting for popularity or promotions.
Log and Security Event Search
When engineers search through massive volumes of log data for a specific error message or identifier, BM25 helps surface the log entries most specifically matching the rare, distinctive terms in that search rather than the most common, generic log lines.
Academic and Legal Document Retrieval
Systems searching large bodies of case law, research papers, or regulatory text lean heavily on BM25’s rarity-weighting behavior, since precise, uncommon legal or technical terminology is exactly what such searches are usually trying to match.
Across all these examples, BM25’s quiet, unglamorous consistency is exactly the point — it does the fundamental, unopinionated work of judging text relevance well enough that most applications never need to think about it directly, freeing developers to focus their attention on the business-specific ranking logic layered on top.
12Frequently Asked Questions
No, it is Elasticsearch’s default similarity algorithm and is used automatically for text scoring unless a different similarity is explicitly configured.
It refers to its origin as roughly the twenty-fifth variant explored during the original research into this family of scoring formulas, rather than describing any parameter or setting within the formula itself.
Only cautiously — scores are best used to rank results within a single query’s response, since absolute score values can vary based on the specific terms searched and the index’s overall statistics.
A b value of zero disables field-length normalization entirely, meaning documents are no longer adjusted for length differences at all when their term frequency is scored.
No, BM25 only works with the exact terms produced after your field’s analyzer has processed the text; synonym handling must be configured separately at the analysis stage.
It’s relatively uncommon for general-purpose search, but it can be worthwhile for specialized datasets with unusual length distributions or repetition patterns, ideally guided by measured relevance testing.
Yes, many modern Elasticsearch deployments combine traditional BM25 text scoring with vector-based semantic search in a hybrid approach, since each captures different aspects of relevance.
13Summary and Key Takeaways
BM25 is the quiet mathematical engine behind Elasticsearch’s default relevance scoring, combining three carefully balanced ideas — diminishing returns on repeated terms, rarity-based weighting for distinctive words, and fair normalization across documents of different lengths — into a single, well-tested formula. Its improvement over older approaches like classic TF-IDF lies specifically in this restraint: rewarding genuine relevance signals without being fooled by document length or gamed by simple keyword repetition. Because BM25 scores are index-relative and purely statistical, understanding its behavior is essential for correctly interpreting search results, debugging unexpected rankings, and knowing exactly where to layer additional tools, like function score queries, when text relevance alone isn’t the full ranking story.
Key Takeaways
- BM25 is Elasticsearch’s default relevance formula — it powers scoring automatically without any explicit configuration required.
- Term frequency saturates, it doesn’t scale linearly — repeated terms help less and less, resisting keyword-stuffing tactics.
- Rare terms carry more scoring weight — inverse document frequency rewards distinctive, specific words over common ones.
- Field length is normalized fairly — longer documents aren’t automatically favored just for containing more words.
- Two parameters, k1 and b, allow careful tuning — but defaults are well-suited to most use cases and should only be changed with measured justification.
- Scores are index-relative — never compare raw BM25 scores across different indices or assume they carry absolute meaning.
- BM25 handles text relevance, not everything else — combine it with function score queries or hybrid semantic search for recency, popularity, or meaning-based ranking needs.