Three plain ideas, put together: a list of numbers, a fixed length, and closeness. Once each is clear, the rest follows naturally.

A vector is simply an ordered list of numbers, such as 0.12, negative 0.83, 0.05, and so on. Each number says how far along one direction a thing lies. With two numbers you can place a point on a flat sheet of paper. With three, you can place it in a room. Vectors used in search have many more numbers, commonly a few hundred or a thousand, so they place things in a space with that many directions. We cannot picture such a space, but the arithmetic works exactly as it does on paper.

A dense vector is one in which almost every position holds a meaningful number. This is different from a sparse vector, where most positions are zero and only a few carry values. Dense vectors are typically produced by machine-learning models called embedding models, which read a piece of content and return a list of numbers that captures its meaning. The list is called an embedding. Content with similar meaning receives embeddings that lie close together.

The dense vector field is the Elasticsearch field type that stores one such vector per document. You declare it in the mapping, telling Elasticsearch how many numbers each vector has, and what kind of closeness to use. Then you fill it with the embeddings of your content, and search it with a technique called k-nearest-neighbor search, or kNN: given a query vector, find the k stored vectors that lie closest to it.

What it is not

A dense vector field is not a text field. It does not split words, it has no postings lists of terms, and it cannot be searched with ordinary word queries. It is also not the place where the model lives. The embedding model is a separate program, inside or outside Elasticsearch, that produces the numbers. The field just stores them, indexes them for fast nearest-neighbor lookup, and compares them at search time. Finally, it is not a replacement for keyword search. It answers a different question, “what is similar in meaning?”, and works best together with the keyword tools, as Chapter 9 shows.

The one-sentence version

If you remember one line from this chapter, make it this: a dense vector field stores, for each document, a fixed-length list of numbers produced by an embedding model, and lets Elasticsearch find the documents whose lists lie closest to a query list, which means finding the documents most similar in meaning.

Every feature is born from a frustration. The frustration here is that people mean one thing and write many.

Keyword search matches the words you type against the words stored in documents. It is fast, precise, and predictable. But human language is slippery. The same idea is expressed in many ways: “cheap flights,” “budget airfare,” “low-cost tickets.” A search for the first may miss documents that use only the second. The same word can mean different things: “bank” of a river and “bank” for money. And people often do not know the exact words: a shopper types “shoes for running in the rain” while the catalog says “waterproof trail sneakers.”

Search engineers have fought this for decades with synonym lists, stemming, spelling correction, and hand-tuned rules. These help, but they are labor-intensive, brittle, and always incomplete. Embedding models offer a different approach. Instead of listing which words are equivalent, they learn from vast amounts of text what tends to appear in similar contexts, and turn that knowledge into positions on a map. Content about the same thing ends up in the same neighborhood, whichever words it uses, and often whichever language it is written in.

A short history

The exact release for each step is worth double-checking in the release notes of your version. The direction of travel is what matters: from “compare everything” to “search a graph,” and from “raw numbers” to “compressed, managed, and combined with keywords.”

Why not just use synonyms?

Synonyms are still useful, and often the best answer for a known, small vocabulary, such as product codes and brand names. But a synonym list cannot cover every way that people phrase a question, it needs constant upkeep, and it cannot handle whole sentences. Vectors handle the long tail of phrasing automatically. The price is that they are approximate, they cost memory, and they depend on the quality of the model that produced them.

A dozen words explain nearly everything about vector search. Learn them here, and later chapters will read like common sense.

Where the numbers come from

An embedding model is trained on a huge collection of examples until it learns to place similar things near each other. After training, you give it a sentence, an image, or another kind of content, and it returns a vector of fixed length. The same model always returns vectors of the same length, and only vectors from the same model can be compared. A vector produced by one model means nothing on the map of another, just as coordinates from one atlas are meaningless on a different planet’s map.

Distance and similarity are two sides of one coin

When two vectors are close, their distance is small and their similarity is large. Elasticsearch turns each measure into a score, so that a bigger score always means a better match. You choose the measure in the mapping. The three classic ones are the straight-line distance, the angle between the vectors, and the dot product. Chapter 6 explains them.

Why “approximate” is acceptable

Finding the exact nearest neighbors among millions of vectors means comparing the query with every one of them. With 768 numbers each, that is billions of multiplications per query. Approximate methods build a network of links between close vectors and follow it from a starting point toward the query, checking only a tiny fraction of all vectors. They may occasionally miss the true closest one, but they answer in milliseconds. For search applications, where the “best” result is a matter of degree anyway, this trade is almost always worth it.

Quick glossary

Where do vectors come from, where do they live, and which parts work together when a user searches?

Vector search has two halves that mirror each other. On the indexing side, content is turned into vectors and stored. On the search side, the user’s query is turned into a vector with the same model, and Elasticsearch looks for stored vectors near it. The embedding model sits outside the field, and the field itself is the storage and the index.

flowchart TB
    T["Text, image, or other content"] --> EM["Embedding model: turns content into a list of numbers"]
    EM --> V["Vector with a fixed number of dimensions"]
    V --> F["dense vector field in the document"]
    F --> S["Segment: raw vectors plus an HNSW graph"]
    Q["User query"] --> QE["Same embedding model makes a query vector"]
    QE --> K["kNN search: find the nearest vectors"]
    S --> K
    K --> R["Top results with scores"]
    

Read the picture from top to bottom. Content passes through an embedding model, producing a vector of fixed length. The vector is stored in the document’s dense vector field. When the document is written into a segment, the vectors are saved in a file and, if the field is indexed, linked into a graph. At search time, the query text goes through the same model, giving a query vector. The kNN search walks the graph of each segment, collects the nearest candidates, and returns the best results with their scores.

The main components

Vectors live per segment

Like other indexed data, dense vectors are stored inside segments. Each segment has its own vector file and its own graph, covering only the documents of that segment. A shard is made of several segments, so a kNN search on a shard runs once per segment and combines the findings. When segments merge, the vectors are copied into a new segment and the graph is rebuilt or extended. This makes the number of segments, and the cost of merges, important themes of the chapters that follow.

Walking one search through the machine

Now we open the mapping and look at each option. A small number of choices control almost everything.

When you declare a dense vector field, you tell Elasticsearch a handful of things. The most important are the number of dimensions, the type of each number, whether the field should be indexed for fast search, which similarity to use, and how the index should be built. The table below lists them in plain words.

Dimensions must match the model

The number of dimensions is fixed for the field, and every vector stored in it must have exactly that many numbers. If your model produces vectors of 768 numbers, the field must say 768. A vector with a different length is rejected at indexing time. The maximum allowed number has grown across versions, and now reaches several thousand, but higher numbers mean more memory and slower comparison, so more is not better by itself. Choose a model whose size fits your budget.

Element types

One vector per field

In the standard arrangement, a dense vector field holds one vector per document. It cannot hold a list of vectors for the same document. If a long document needs several vectors, for example one per paragraph, the usual answers are to cut the document into chunks and index each chunk as its own document, or to place the chunks in nested objects, each with its own vector. Both designs are discussed in Chapter 11.

The vector and the original source

By default, the document’s original source contains the vector as an array of numbers, and the same numbers are also stored in the special vector files. This means the numbers are kept twice, once in the source and once in the vector storage, and every time the source is fetched or a document is updated or reindexed, hundreds of numbers travel with it. Newer versions let you keep vectors out of the returned source, or handle them more economically, to save space and bandwidth. Check the behavior of your version, because the defaults have changed over time.

Changing the mapping later

Most of these choices are permanent for an index. You cannot change the number of dimensions, or the similarity, of an existing field. Some newer versions allow certain upgrades of the compression setting, but in general, changing the design of a dense vector field means creating a new index with a new mapping and reindexing the data. If you also change the embedding model, all vectors must be recomputed, because the old ones are meaningless to the new model. Plan for this from the start, for example by giving indexes names that include the model version, and by using aliases to switch between them.

Nearest neighbor search needs a ruler. The choice of ruler changes which results come first, so it deserves care.

Two vectors are “close” when a chosen measure says so. Elasticsearch offers several measures. Three of them are classic, and one is a specialized variant. Each turns a pair of vectors into a score in which a higher number always means a better match.

flowchart TD
    A["Choose a similarity"] --> B{"Does the embedding model recommend one?"}
    B -- Yes --> C["Use the recommended metric"]
    B -- No --> D{"Are the vectors normalized to length one?"}
    D -- Yes --> E["dot product: fastest, same ranking as cosine"]
    D -- No --> F{"Does vector length carry meaning?"}
    F -- Yes --> G["maximum inner product or straight-line distance"]
    F -- No --> H["cosine: compares direction only"]
    

Straight-line distance

This is the everyday idea of distance: the length of the straight line between two points. It takes into account both the direction and the length of the vectors. Points that lie physically close have a small distance. Elasticsearch converts it to a score so that small distances give large scores. It suits models that were trained with distance in mind, and data where the magnitude of a vector is meaningful.

Cosine similarity

Cosine similarity looks only at the angle between two vectors, ignoring their lengths. Two vectors pointing in the same direction are considered identical in meaning, even if one is much longer than the other. Perfectly aligned vectors score at the top, unrelated ones in the middle, and opposite ones at the bottom. Most text embedding models are designed for cosine similarity, which is why it is the most common choice. A vector with all zeros has no direction, so it cannot be used with cosine similarity, and is rejected.

Dot product

The dot product multiplies the numbers of two vectors position by position and adds them up. When both vectors have been scaled to length one, the dot product equals the cosine, and gives the same ranking, but is cheaper to compute. Elasticsearch’s dot product option therefore expects unit-length vectors. Normalizing your vectors once, before indexing, and using the dot product option is a common way to save work on every search.

Maximum inner product

This variant uses the dot product without requiring unit length, so that vector length counts. Some recommendation models use it deliberately: a longer vector might mean a more popular item, and the ranking should reflect that. Because the raw values are unbounded, Elasticsearch scales them into a positive score. Use it only when your model documentation says to.

A tiny example

Let us use a two-number picture. Suppose vector A points along the horizontal axis, vector B points straight up, and vector C points diagonally between them. The angle between A and C is 45 degrees, and the angle between A and B is 90 degrees. By cosine similarity, C is closer to A than B is, because the angle is smaller. If you stretch C to twice its length, its direction does not change, so cosine similarity does not change, but the straight-line distance between C and A grows. This shows why the choice of ruler matters: the same three points give different rankings under different measures.

How scores are shaped

The similarity produces a raw value, which Elasticsearch converts into a positive score. For the angle-based measures, the range is mapped so that identical direction gives a score near one, and opposite direction gives a score near zero. For distance, small distances give scores near one, and large distances give scores approaching zero. These scores are for ranking and for combining with other scores. They are not probabilities, and a score of 0.9 does not mean 90 percent chance of being relevant.

Similarity thresholds

A kNN search can also set a minimum similarity, so that results below the threshold are dropped, even if fewer than k results remain. This is useful when you would rather return nothing than return something irrelevant. The threshold is expressed in the units of the raw measure, not the converted score, so read the documentation for your version before choosing a value.

Normalizing vectors

Normalizing means dividing each number of a vector by its length, so that the new length is exactly one. Many embedding services already return normalized vectors. If yours does not, and you want the fast dot product option, you can normalize in your loading program or in an ingest step. Never normalize some vectors and not others in the same field, and never normalize the stored vectors but forget to normalize the query vector, since mixed lengths would distort the ranking.

A vector is only a list of numbers, but storing millions of them so that they can be searched quickly takes careful machinery.

flowchart LR
    A["Document with a vector arrives"] --> B["Vector is added to the in-memory segment and its graph"]
    B --> C["Refresh writes a new segment with vectors and graph"]
    C --> D["Merge combines segments and builds larger graphs"]
    D --> E["Search asks every segment for its nearest candidates"]
    E --> F["Shard combines candidates, coordinator picks the global best"]
    

Two structures per segment

Inside each segment, a dense vector field creates two things. The first is a file of raw vectors, stored one after the other, in the order of the documents. This lets the search read the numbers of any document quickly. The second, if the field is indexed, is the graph: a network in which each vector is linked to a small number of its close neighbors. The graph is what makes approximate search fast. Both are stored in files that the operating system can map into memory.

Building the graph as documents arrive

When a document with a vector is indexed, the vector is inserted into the graph of the segment that is currently being built in memory. The insertion searches the existing graph for the new vector’s closest neighbors, and links the new vector to them. This is real work, involving many distance calculations, so indexing vectors is much slower than indexing plain text or numbers. The cost grows with the number of dimensions and with the quality settings of the graph.

Refresh and new segments

When a refresh happens, the in-memory graph is written out as a new segment with its own vector file and graph files. Each refresh under heavy indexing creates a new small segment. Small segments have small graphs, which are quick to search but add to the number of graphs that every search must visit.

Merging: the expensive part

In the background, segments are merged into larger ones. For vectors, a merge must produce a graph for the combined segment. Depending on the version, Lucene may reuse the graph of the largest input and insert the vectors of the others into it, or build a new one. Either way, merging vectors involves many distance calculations and uses a lot of processor time, far more than merging text segments of the same size. This is one reason that indexing many vectors continuously can keep a cluster’s processors busy long after the documents have stopped arriving.

How much memory do vectors need?

The single most important sizing fact is that vector search is fast only when vectors and graphs stay in memory, in the operating system’s file cache, not on the Java heap. Reading a vector from disk for every step of a graph walk would be slow. A rough estimate of the memory needed for unquantized decimal vectors is the number of vectors, times four bytes, times the number of dimensions, plus a smaller amount for the graph links, roughly four bytes times the number of links per vector.

A numbers walkthrough

Let us use invented numbers. Suppose you store 10 million vectors of 768 dimensions each as four-byte decimals. The raw vectors take 10 million times 768 times 4 bytes, which is about 30.7 gigabytes. With sixteen links per vector, the graph adds about 10 million times 4 times 16 bytes, which is about 640 megabytes. So around 31 to 32 gigabytes should ideally fit in the file cache across the nodes that hold this data, for good latency. If the same data were stored with each number in one byte, the vectors would take about 7.7 gigabytes. If in a single bit, about 960 megabytes. Compression, which Chapter 10 discusses, is therefore the main way to fit large collections in modest memory.

Disk is bigger than memory needs

Even when vectors are compressed for searching, the original decimal vectors are usually kept on disk as well, so that they can be used for rescoring or for rebuilding the compressed copy. This means the disk footprint may be larger than the memory footprint of the searched form. Plan disk and memory separately: memory for what the search touches, disk for everything that must be kept.

Deletes and updates

Deleting a document does not remove its vector from the graph of its segment. The vector stays, marked as deleted, and searches skip it, until a merge rewrites the segment. An update is a delete plus a new document in a new segment. Indexes with constant updates therefore carry dead vectors in their graphs, consuming memory and search effort until merged away.

The name sounds frightening, but the idea is friendly: use highways to get near, then local streets to arrive.

HNSW stands for Hierarchical Navigable Small World. It is a graph in which each vector is a point, connected by links to a few of its closest neighbors. The graph has several layers. The bottom layer contains every vector, with links to close neighbors. Each higher layer contains only a random sample of the vectors below it, so it is sparser, and its links span longer distances. The top layer contains very few points.

flowchart TD
    E["Entry point in the top layer"] --> L2["Top layer: few points, long jumps"]
    L2 --> L1["Middle layer: more points, shorter jumps"]
    L1 --> L0["Bottom layer: all vectors, fine search"]
    L0 --> C["Keep the num_candidates closest vectors found"]
    C --> T["Return the best k"]
    

How a search works

A search starts at an entry point in the top layer. From there, it repeatedly moves to whichever linked neighbor is closer to the query vector, until no neighbor is closer. That point is the best found in that layer, and it becomes the starting point in the layer below, where the same greedy walk continues with finer steps. At the bottom layer, the search does not stop at the first best point. It keeps exploring neighbors and maintains a list of the closest candidates seen so far, of a size that you control. When no better candidates can be found, it stops, and the best of the list are returned.

Why it is fast

The walk touches only a tiny part of the graph. Even with tens of millions of vectors, a typical search visits a few thousand of them. The number of steps grows slowly with the size of the collection, roughly with its logarithm, rather than in direct proportion. That is why doubling the data adds only a little to the search time, as long as the graph stays in memory.

The knobs

The accuracy and speed trade

Every knob trades accuracy for cost. A graph built with more links and a larger construction list is better at finding true neighbors, but takes more time and memory to build. A search with more candidates finds more of the true neighbors, but takes longer. The measure of accuracy is recall: of the true k nearest neighbors, how many did the search return? A recall of ninety percent means that nine of the ten true nearest neighbors were found. Most applications aim for a recall in the high nineties and accept a small loss for large savings in time.

One graph per segment

Because each segment has its own graph, a search runs the walk once for every segment and combines the candidates. A shard with twenty small segments does twenty walks. A shard with two large segments does two. Each walk in a small segment is quick, but the overhead adds up. This is why heavy indexing, which creates many small segments, can slow vector searches until merges catch up, and why merging read-only data into few large segments is such a helpful step.

Filters during the walk

Real searches often combine similarity with conditions such as “only in stock” or “only this customer’s documents.” Elasticsearch applies such filters during the graph walk itself: it only counts vectors of documents that pass the filter as candidates, and keeps walking until it has enough. This means the search still returns k results that all satisfy the filter, if enough exist. When a filter is very restrictive, so that very few documents pass, walking the graph becomes wasteful, and the engine switches to an exact comparison against just the passing documents, which is cheap when there are few of them.

There are two ways to search a dense vector field, exact and approximate, and a powerful way to combine them with keywords.

Approximate kNN search

The main way to search an indexed dense vector field is the kNN search. You give it the field to search, a query vector, the number of results you want, and the number of candidates to consider on each shard. Optionally, you give a filter, a minimum similarity, and a boost that scales its scores. The query vector can be supplied directly, or Elasticsearch can produce it from text using a deployed embedding model, so that the client never handles vectors at all.

How shards and candidates interact

Each shard searches its own segments and returns its best candidates. The coordinating node then merges the lists from all shards and keeps the overall best k. Since every shard contributes, an index with many shards does more total work per search, and the candidate count applies per shard. If you raise the number of candidates while the number of shards is large, the total effort rises in proportion. Keep this in mind when tuning.

Exact search with a scoring script

The older method, still valuable, is to compute similarity exactly for a chosen set of documents inside a scoring script. It compares the query vector with the vector of every document that passes a query, and ranks them. This is exact, since no shortcut structure is used, and it works on fields that are not indexed. But its cost is proportional to the number of documents compared, so it is practical only when a filter narrows the set to thousands, not millions.

Hybrid search: words and meaning together

Vector search is strong on meaning and weak on exact terms, such as product codes, names, and rare words. Keyword search is the opposite. Combining them gets the best of both. A hybrid request runs a keyword query and a vector query, and merges their results into one ranking.

flowchart TD
    Q["User question"] --> L["Keyword query: words and postings lists"]
    Q --> V["Vector query: meaning and nearest neighbors"]
    L --> M["Combine rankings: weighted scores or reciprocal rank fusion"]
    V --> M
    M --> R["Final ranked results"]
    

Paging and sizes

A kNN search returns at most k results. It is not designed for deep paging through thousands of results. If you need more, raise k, keeping in mind that the cost rises. Ordinary paging with offset and size works only within the k results. For very deep listing, use a different approach, such as a filtered keyword query or exact scoring over a narrowed set.

Practical query tips

Vectors are big. Compression is the main way to make large collections affordable. The trick is to shrink them without shrinking the quality of the results.

flowchart LR
    A["Decimal: 4 bytes per dimension"] --> B["8-bit: about 1 byte per dimension"]
    B --> C["4-bit: about half a byte per dimension"]
    C --> D["Binary: about one bit per dimension"]
    D --> E["Smaller memory, faster search, some loss of accuracy"]
    

The idea

Quantization means storing each number of a vector with fewer bits. A number such as 0.3721845 is stored as a small whole number that stands for a range of values, such as “somewhere in this band.” The comparison of two vectors is then done on these small numbers, which is faster and needs much less memory. The original number cannot be recovered exactly, but the comparison of two vectors stays close to the true one, and the ranking is usually nearly the same.

The common levels

Rescoring: getting the accuracy back

The most aggressive levels lose enough detail that the top of the ranking may be slightly wrong. The fix is rescoring. The search first finds a generous number of candidates using the compressed vectors, which is fast and cheap. Then it recomputes the exact similarity, using the original full-precision vectors read from disk, for only those candidates, and reorders them. The final list is much closer to what full precision would give, and the expensive comparison ran on a small fraction of the data. This “coarse first, fine second” pattern is why binary quantization can keep good accuracy with a tiny memory footprint.

Quantization and the original vectors

Element types are different from quantization

Do not confuse choosing a small element type in the mapping with quantization performed by Elasticsearch. With a small element type, your model or your pipeline produces small numbers to begin with, and Elasticsearch stores exactly what you provide. With quantization, you provide normal vectors, and Elasticsearch compresses a searching copy for you and keeps the originals. The second is easier to adopt and reversible in spirit, and in recent versions it is often the default.

A numbers walkthrough

Suppose you have 100 million vectors of 384 dimensions. Stored as decimals, they need about 154 gigabytes of memory to be searched well, spread over the cluster. With eight-bit compression, this falls to about 38 gigabytes. With binary compression, to about 5 gigabytes. In the last case, a modest cluster can hold a search index that would otherwise need a large one. The price is that the search must rescore a candidate list from disk, and that a small share of results may differ from the full-precision ranking. The figures are invented, but the ratios are real.

How to choose

The dense vector field is only as good as the vectors placed into it. Here is how vectors are made, chosen, and kept consistent.

Where the model can run

Embedding at indexing time

If the model runs inside or through Elasticsearch, an ingest pipeline can embed each document as it arrives. The pipeline sends the chosen text field to the model and stores the returned vector in the dense vector field. If your application embeds outside, it sends the vector as part of the document. In both cases, the same model, with the same settings, must be used for every document in the field.

Embedding at query time

At search time, the query text must be embedded by the same model. Elasticsearch can do this itself, given the model’s identifier and the text, so that clients never handle vectors. This adds the model’s latency to every search, typically tens of milliseconds for a small model on a good processor, and much more for large models or slow networks. For repeated queries, caching the query vectors saves time and money.

Choosing a model

Dense versus sparse models

Not all semantic models produce dense vectors. Some, such as Elastic’s own learned sparse model, produce sparse vectors: a small set of terms with weights, including related terms that were not in the text. These are stored in a different field type and searched with different queries. They tend to be more interpretable, since you can see which terms carry weight, and use less memory, while dense vectors often capture deeper meaning and work across languages. Many systems use both, together with keywords.

Chunking long content

Embedding models have a maximum input length, and even below it, one vector for a long article averages many ideas into a blur. The remedy is to cut long content into chunks, such as paragraphs or windows of a few hundred words, embed each chunk, and store each with its vector. There are two common designs. In the first, each chunk is its own document, carrying a reference to its parent. In the second, chunks are nested objects inside the parent document, each with its own vector, so that the parent is returned with the best matching chunk highlighted.

Model-specific details that surprise people

• Prefixes. Some models expect a short marker before the text, one for documents and another for queries. Forgetting it lowers quality quietly.
• Normalization. Some models return unit-length vectors and some do not. Check before choosing dot product.
• Truncation. Text beyond the input limit is usually cut without warning.
• Versions. A new version of a model with the same name may produce different vectors. Record the exact version used.
• Batching. Sending many texts to the model in one call is much faster and cheaper than one at a time.

A vector index is a small factory with a start and an end. Following the whole line shows where things can go wrong.

The pipeline from source to result

When content changes

Vectors describe the content at the time they were made. If a document’s text changes, its vector must be recomputed, or it will describe the old text. A common bug is to update the text field and forget the vector, so that searches return the document for words that are no longer in it. Build the update path to re-embed whenever the embedded text changes, and consider storing a hash of the text alongside the vector, to detect when a vector is stale.

Updates and deletes at the segment level

As Chapter 7 explained, an update creates a new document in a new segment and leaves the old vector in its old graph, marked as deleted. Merges eventually clean up. In an index with frequent updates, dead vectors accumulate and merges work harder. If you can, design the data as append-mostly, with updates being rare.

Changing the model: the big move

Replacing the embedding model is the most disruptive change to a vector index. All vectors must be recomputed with the new model, and the mapping may need new dimensions or a new similarity. The safe way is a parallel build.

Merging and force merging

Merges keep the number of segments in check, and therefore the number of graph walks per search. For an index that is complete and will not receive more writes, a force merge down to a few segments makes searches faster, since fewer graphs are walked, and removes dead vectors. It also costs a lot of processing power, because graphs are rebuilt, so run it in a quiet period, and only on finished data.

Refresh interval and small segments

While loading large volumes, a short refresh interval creates many small segments, each with a small graph, and the merges that follow rebuild graphs again and again. A longer refresh interval during loading reduces this churn. Restore a sensible value when the load is over.

Vector search is expensive in three currencies: memory, processing time, and indexing effort. Here is where each is spent and how to save it.

Where the cost comes from

A numbers walkthrough for search cost

Let us use invented numbers. Suppose an exact search compares the query with 10 million vectors of 768 numbers. Each comparison needs about 768 multiplications and additions, so one query needs about 7.7 billion operations. Even on a fast processor, that takes seconds. An approximate search through a graph might compare the query with only a few thousand vectors, perhaps five thousand, which is about 3.8 million operations, roughly two thousand times fewer. That is the difference between a few seconds and a few milliseconds, and it is why the graph exists.

Tuning levers

Indexing speed

Vector indexing is slower than text indexing, because of graph construction and, if embedding happens on the way in, model calls. To speed up a large load, embed in batches and outside the critical path if you can, use bulk requests of moderate size, lengthen the refresh interval, use zero replicas for a first load, and let the graph quality settings stay at moderate levels. Because each replica builds its own graph, the cost of replicas is real: one replica doubles the graph-building work.

Search latency

Search latency depends on whether the vectors and graphs are in the file cache. When they are, latency is set by the number of comparisons. When they are not, every step of the walk may need a disk read, and latency rises by orders of magnitude. The most common cause of slow vector search is not the query, but too little memory for the cache. After that come too many segments, too many candidates, and overly large shards.

Shard layout

Every shard runs its own walks, so more shards mean more total work per search. Choose a modest number of shards, of moderate size, and avoid spreading a small vector index over many shards. On the other hand, a single huge shard has a huge graph and long merges, and does not use several nodes’ memory. The right balance depends on the size of the index and the number of nodes.

Hardware

• Memory first. Size the cluster so that the searched form of the vectors fits in the cache of the data nodes.
• Modern processors. Distance calculations benefit from vector instructions of modern processors, and recent Java versions used by Elasticsearch make use of them. Old hardware wastes this.
• Fast local disks. Rescoring and cold reads need fast random access, so solid-state drives are strongly preferred.
• Separate roles. Keep model inference on machine-learning nodes, so that it does not compete with search for memory and processors.

Every design carries a price. Here is the honest balance sheet of searching by meaning.

How ambitious should you be?

Vector indexes are ordinary indexes in the way they are replicated and recovered, with a few special sensitivities.

Replicas and graphs

In the standard model, each replica repeats the indexing work, so each replica builds its own graph, in its own segments. The graphs on two copies of a shard therefore differ in shape, although they hold the same vectors. Two searches served by different copies may return slightly different approximate results, because the walks follow different graphs. The differences are small, and the ranking of the best results is usually the same, but they mean that identical queries can occasionally give slightly different lists.

Recovery and memory

When a replica is rebuilt by copying files, vector files and graph files are copied like any other segment file. After a node restart, the file cache is cold: vectors and graphs must be read from disk again, and searches are slow until the cache warms. For large vector indexes, this warm-up can take minutes or longer, and search latency in that period may be many times higher. Plan restarts accordingly: restart one node at a time, and let each node warm up before moving on.

Designing for resilience

A vector looks like harmless numbers. It is not. It carries the meaning of your content, and the path to it has its own risks.

Vectors leak meaning

An embedding is a compressed description of the content. Research has shown that, with enough effort, parts of the original text can be reconstructed from its embedding, or at least that sensitive attributes can be inferred. You should therefore treat vectors with the same sensitivity as the text they came from. If the text is confidential, so are its vectors, in the index, in snapshots, and in backups.

Access control still applies

The embedding path is a data path

When embedding happens in an outside service, your content leaves your environment. Check the provider’s terms: whether data is stored, used for training, or logged, where it is processed, and how it is protected. For regulated or confidential data, prefer models that run inside your own cluster or network. Protect the credentials used to call the service, keep them in secure storage, rotate them, and give each application its own.

Risks specific to retrieval for language models

When vector search feeds a large language model, a further set of risks appears. The retrieved text becomes part of the model’s input. If a retrieved document contains hidden instructions, such as text that tells the model to ignore its rules or reveal secrets, the model may obey them. This is called prompt injection through retrieved content. Reduce the risk by controlling what enters the index, marking retrieved text clearly as data and not as instructions, limiting what the model is allowed to do, and reviewing outputs where the stakes are high.

Poisoning and quality attacks

Someone who can add documents to the index can also try to influence what is retrieved, by writing content designed to be close to popular queries. If your index accepts content from the public, treat it as untrusted: moderate it, limit who can write, and separate it from trusted content.

Deletion and retention

When a document is deleted, its vector remains in the segment’s files until a merge removes it, and older history may be kept for recovery. Snapshots also keep copies. For privacy requests to erase personal data, know how long these copies last, and plan deletion and merging accordingly. Remember that vectors of derived chunks must be deleted as well.

Resource abuse

Vector search is costly. A user, or an attacker, who can send unlimited queries with huge candidate counts or large k can burn processors and memory. Set limits on k and candidates, add rate limits and timeouts in front of search endpoints, and separate heavy workloads from interactive ones.

Vector search can fail in a way that keyword search rarely does: it can keep answering quickly while quietly getting worse. Watch both speed and quality.

Windows onto the index

Metrics to chart

Measuring quality: recall and relevance

There are two different questions. Recall asks whether the approximate search finds what the exact search would find. You can measure it by taking a sample of real queries, running both the approximate search and an exact scoring search, and counting how many of the exact top results appear in the approximate top results. Relevance asks whether the results are actually useful to people. It needs human judgments or behavior data, and depends on the model and the chunks more than on the search settings. Both matter, and they are affected by different changes.

Reading the patterns

Good habits make vector search accurate and affordable. Bad habits are behind most disappointing results and most surprising bills.

Patterns that work

Anti-patterns to avoid

A short list you could hand to a new teammate on their first day.

Best practices

Common mistakes

• Treating vector search as magic. It is only as good as the model, the chunks, and the data.
• Forgetting that scores are relative. A high score is not proof of relevance, and scores differ between models.
• Setting the wrong number of dimensions. The field must match the model exactly, or indexing fails.
• Skipping normalization when using dot product. The ranking is then distorted.
• Ignoring model input limits. Text beyond the limit is not embedded.
• Forgetting the prefixes some models need. Quality drops quietly.
• Using a post-filter and expecting k results. Filters given outside the kNN part can leave fewer results.
• Neglecting the cold cache after restarts. Latency is high until vectors are read back into memory.
• Assuming the mapping can be changed in place. Dimensions and similarity generally cannot.

Myths versus facts

Where does the dense vector field quietly change what users can do?

Many well-known organizations have said publicly that Elasticsearch is part of their systems, including Wikipedia’s search, Netflix, Uber, and Stack Overflow. They rarely publish how they use vectors, so the scenarios below are typical patterns for such workloads, not descriptions of anyone’s private configuration.

Retrieval for question answering with a language model

One of the most common uses today is to give a large language model the right information to answer a question. The pattern is often called retrieval-augmented generation. The documents of a knowledge base are chunked and embedded into a dense vector field. When a user asks a question, the question is embedded, the nearest chunks are retrieved, and both the question and the chunks are handed to the language model, which writes an answer based on them.

flowchart LR
    U["User question"] --> E["Embed the question"]
    E --> K["kNN search in Elasticsearch"]
    K --> C["Top chunks of text"]
    C --> P["Prompt with the question and the chunks"]
    P --> L["Language model writes the answer"]
    L --> A["Answer with sources"]
    

Short answers to the questions that come up most often.

If you skim only one part of this guide, make it this one.

The dense vector field lets Elasticsearch search by meaning. It stores, for each document, a fixed-length list of numbers made by an embedding model, indexes those lists in a layered graph, and finds the nearest ones to a query in milliseconds. It costs memory, indexing effort, and care, and it pays back with search that understands paraphrases, works across languages, and feeds language-model applications.