Search is easy when you want ten hits. It gets hard when you want ten million.

Elasticsearch is a search and analytics engine. Most searches ask for a small answer, such as the top ten products or the latest fifty log lines. But real projects also need to read everything that matches. A team may need to export all customer records to a spreadsheet, copy an entire index into a new index, feed millions of documents into a machine learning job, or run a nightly report over a whole month of events.

Asking for everything in one response is impossible in practice. The answer would be enormous, it would exhaust memory on the servers and on the client, and it would take so long that the connection might time out. So the work must be split into many small steps. The real question is how to split it so that each step continues exactly where the last one stopped, without missing documents and without repeating any.

A second difficulty: the data keeps changing

Elasticsearch is usually a live system. While your export runs for ten minutes, other programs may add new documents, update old ones and delete some. If each trip looked at the current state of the index, you could see a document twice, because it moved position, or miss one entirely, because it shifted past your bookmark. A trustworthy export needs a stable picture of the data for the whole journey.

The Scroll API solves both difficulties together. It gives you batches, and it gives you a consistent view. As you will see, it pays for this with some resources held inside the cluster, and it has a newer relative that is now preferred for many jobs. Understanding both the power and the price is the goal of this tutorial.

A plain definition, then the words you will keep meeting.

The Scroll API is an Elasticsearch feature that lets you run one search and then fetch its results in many consecutive batches, all taken from a consistent snapshot of the data as it was when the search began. It works like a cursor in a traditional database: you open it, read from it repeatedly, and close it when you are done.

The process starts with an ordinary search request that carries one extra piece of information, a time value telling Elasticsearch how long to keep the search alive between batches. The response contains the first batch of documents and a special identifier called the scroll ID. You then send that ID back in follow-up requests, and each one returns the next batch, until a response comes back with no documents left.

Words you need to know

To appreciate the scroll, first see why ordinary paging breaks down.

The simplest way to page through results is to use the from and size parameters. The first page has from zero and size ten. The second page has from ten and size ten. And so on. This is fine for a user browsing a few pages of results. It becomes a problem for deep pages, and the reason lies in how Elasticsearch distributes the work.

The hidden cost of deep pages

An index is split into shards, and each shard searches its own data. Suppose you want the page starting at position one hundred thousand with ten results. Each shard cannot just return its own ten. It must find its top one hundred thousand and ten hits and send them to the coordinating node, which merges all of them from every shard, sorts them and throws away all but the last ten. The deeper you go, the more each shard and the coordinator must work, and the memory grows with the depth.

flowchart TD
    A["Client asks for page at from 100000, size 10"] --> B["Coordinating node"]
    B --> C1["Shard 1 finds its top 100010"]
    B --> C2["Shard 2 finds its top 100010"]
    B --> C3["Shard 3 finds its top 100010"]
    C1 --> D["Coordinating node merges 300030 hits"]
    C2 --> D
    C3 --> D
    D --> E["Sorts, drops 100000 hits, keeps 10"]
    E --> F["Client receives only 10 hits"]
        

The safety limit

To protect the cluster, Elasticsearch sets a limit called index.max_result_window, ten thousand by default. A request whose from plus size goes beyond that limit is rejected. This is a wise guard. It means ordinary paging can reach only the first ten thousand hits of any search.

The data-changes problem again

Paging with from and size also suffers from the shifting-data problem. Each page request is a fresh search of the current index. If a document is added between page one and page two, every later position shifts by one, and you may see one result twice or miss one. The scroll fixes both issues, since it holds one view and moves a cursor through it.

Two simple ideas combine to make scrolling work.

The first idea is the frozen view. When you start a scroll, each shard involved in the search takes note of the exact state of its data at that moment and promises to keep serving from that state until the scroll ends. New documents that arrive later, updates to existing documents and deletions are all ignored by that scroll. This is often called a snapshot, though it is unrelated to the backup feature that has the same name.

The second idea is the bookmark. Each shard remembers how far it has already delivered. When you ask for the next batch, the shard continues from its bookmark instead of starting the search again from the top. Because it continues, the work of each batch stays about the same however far you have travelled. Deep positions cost no more than early positions.

Stateful means the cluster remembers you

A normal search is stateless. Once the response is sent, the cluster forgets the request. A scroll is stateful. The cluster must remember your frozen view and your position between requests. That memory lives inside the search contexts on the shards, and it is the source of both the strengths and the costs of scrolling. Anything kept alive on your behalf uses memory, file handles and disk space, so it must eventually be released.

A scroll always follows the same three phases: open, read repeatedly, close.

sequenceDiagram
    participant C as Client
    participant N as Coordinating Node
    participant S as Shards
    C->>N: Initial search with scroll time and batch size
    N->>S: Create search contexts and run query
    S-->>N: First hits and context references
    N-->>C: First batch and scroll ID
    loop Until a batch is empty
        C->>N: Scroll request with scroll ID and new keep-alive
        N->>S: Continue from each bookmark
        S-->>N: Next hits
        N-->>C: Next batch and scroll ID
    end
    C->>N: Clear scroll
    N->>S: Release search contexts
        

The scroll ID may change

Each response includes a scroll ID. In many cases it stays the same, but the documentation says it can change between requests, so always use the most recent one you received. Treat it as an opaque token. Never try to read or build it yourself.

Where the ID travels

The scroll ID can be long, especially for indices with many shards, since it encodes references to each shard’s context. For that reason, send it in the body of the request instead of the web address, to avoid hitting limits on address length.

The most misunderstood part of scrolling is how long it should stay alive.

The time you give, such as one minute, is not the total time of the whole scroll. It is the time the shards will hold your context after each request. Every request that reaches the shards resets that timer. So a scroll of two hours may work perfectly with a keep-alive of only one minute, as long as you always come back for the next batch within that minute.

How to choose a value

The keep-alive should be long enough to process one batch and come back for the next, with a healthy safety margin. It should not be long enough to cover the entire job. A shorter value means that if your program crashes, the cluster releases the resources sooner. A longer value gives more room for slow processing but holds resources longer after a failure.

The cluster ceiling

Elasticsearch protects itself with a maximum keep-alive, controlled by the setting search.max_keep_alive, which is twenty-four hours by default. Asking for a longer time is rejected. There is also a default keep-alive for cases where none is given, controlled by search.default_keep_alive. Both are cluster settings that an administrator can adjust, though raising the maximum should be done with care.

Two small choices strongly affect how fast a scroll runs.

The batch size

The size parameter of the first search decides how many hits each batch contains. A larger size means fewer round trips, but each response is bigger and needs more memory on the servers and on your client. A very small size makes the job chatty and slow. A common starting range is between several hundred and a few thousand documents per batch, adjusted to document size and network speed.

There is a limit here as well. The size of a scroll batch must not exceed the index result window setting, ten thousand by default, because a scroll batch costs as much memory as a result window of the same size. If you ask for more, Elasticsearch replies with an error that mentions this setting.

The sort order

If you do not care about the order of the documents, ask for them sorted by the special value _doc. This means “in the natural index order”. Elastic’s documentation notes that scroll requests are faster when sorted this way, because Elasticsearch does not have to compare and order documents by score or by a field, and each shard can simply walk through its data.

Finishing politely is part of the contract.

A scroll ends in one of three ways. The best way is that your program finishes reading and sends a clear scroll request naming the scroll ID. Elasticsearch then releases the search contexts on every shard at once. The second way is that the keep-alive expires because no request touched the scroll in time, and the contexts are removed automatically. The third way is a failure, such as a node restart, which destroys the contexts along with the node’s memory.

Why you should always clear

Automatic expiry is a safety net, not a plan. If you rely on it, then for the entire keep-alive after your last request the cluster keeps holding a frozen view that nobody will read. Multiplied by many jobs, this quickly adds up. Clearing a finished scroll is a small extra call that returns resources at once.

flowchart TD
    A["Scroll is open"] --> B{"How does it end?"}
    B -->|Client sends clear request| C["All shard contexts released immediately"]
    B -->|Keep-alive runs out| D["Contexts expire and are removed automatically"]
    B -->|Node restarts or shard is lost| E["Context disappears, next request fails"]
    C --> F["Resources returned right away"]
    D --> G["Resources held until the timeout"]
    E --> H["Client must start a new scroll"]
        

Now let us look inside a shard to see how the frozen view really works.

Each shard is a Lucene index made of immutable segments. When the initial scroll search reaches a shard, the shard creates a search context. Inside it is a reader that points at the exact list of segments that existed at that moment, along with the record of which documents in those segments were alive. This reader is the frozen view.

Because segments never change once written, holding on to a set of segments is a cheap and elegant way to freeze data. Nothing needs to be copied. The shard simply refuses to delete those segment files while a reader still uses them. New documents go into new segments that the frozen reader does not know about, so the scroll never sees them.

graph TD
    A["Scroll starts"] --> B["Shard creates search context"]
    B --> C["Reader pins segments A, B and C"]
    C --> D["Live index continues"]
    D --> E["New segment D appears, invisible to the scroll"]
    D --> F["Merge combines A and B into segment E"]
    F --> G["Old segments A and B stay on disk while the scroll needs them"]
    C --> H["Each batch continues from the bookmark inside the pinned view"]
        

How the bookmark works

Along with the reader, the search context stores the position reached so far for the chosen sort order. When the next batch is requested, the shard resumes after that position. It does not re-run the whole search from the beginning and discard the first hits. This is why a scroll’s cost per batch stays steady, unlike deep paging.

How the batches come together

For each batch, the coordinating node asks every shard for its next portion, merges the portions, keeps the best according to the sort order and returns exactly the requested batch size. Each shard advances only as far as the hits that were actually delivered, so nothing is lost or repeated between batches.

Holding a frozen view is not free, and the cost appears in three places.

Elasticsearch continually merges small segments into larger ones in the background, and afterwards deletes the old small ones. A scroll disturbs this tidy routine. Any segment still needed by an open scroll cannot be deleted, even if a merge has already replaced it. The old files stay on disk until the scroll finishes.

Why this matters most on busy indices

If an index is nearly static, such as yesterday’s log index, a scroll pins segments that will not change anyway, and the extra cost is small. If the index is busy, with constant indexing and heavy merging, a long scroll can hold a large amount of obsolete data. The longer the scroll runs and the more churn the index sees, the greater the temporary footprint.

A few guard rails protect the cluster from runaway scrolls.

What happens when limits are hit

If too many scrolls are open on a node, new scroll requests are rejected with an error stating that too many scroll contexts are open. The message names the limit. This usually means that scrolls are being started without being cleared, or that many parallel jobs are running at once. Raising the limit is possible, but the better fix is usually to find out why so many contexts are needed and to clear them properly.

If the requested keep-alive is greater than the maximum, the request is rejected as invalid, with a message that points to the maximum setting. If the batch size is above the result window, the request fails with a message about batch size. All three errors are clear and quick to diagnose once you know the settings behind them.

One reader is sometimes too slow. Slicing lets many readers share the work.

A single scroll walks through results one batch after another, with only one client busy at a time. For very large jobs, you can split the work into slices. Each slice is an independent scroll that covers a separate portion of the documents. You give every slice a number, from zero up to the total number of slices minus one, and the total count. Different workers can then read different slices at the same time.

graph LR
    Q["One query and one frozen view idea"] --> S0["Slice 0 with its own scroll ID"]
    Q --> S1["Slice 1 with its own scroll ID"]
    Q --> S2["Slice 2 with its own scroll ID"]
    S0 --> W0["Worker 0"]
    S1 --> W1["Worker 1"]
    S2 --> W2["Worker 2"]
    W0 --> R["Together they read every matching document once"]
    W1 --> R
    W2 --> R
        

How documents are divided

By default, Elasticsearch decides which slice a document belongs to using a hash of its identifier, spread evenly. Each document falls into exactly one slice, so together the slices cover all matches without overlap. You may also name a numeric field with doc values, and the slice is chosen from that field’s value instead. That can be more efficient in some cases, but the field must exist in all documents and should hold a single value each.

Elastic now recommends a newer pair of features for many jobs that once used scroll.

A point in time, often shortened to PIT, is a lightweight handle that freezes a view of one or more indices for a chosen period, similar to the frozen view of a scroll. Search after is a way of continuing a search by giving the sort values of the last hit you saw. When you combine them, you get stable paging that does not need a heavy per-request cursor.

Why the newer way is often better

With search after, your program stores the sort values of the last hit. If the program crashes, it can restart from those values instead of losing its place. The cluster holds less per-request state, and the cost of keeping the frozen view alive is lower. Search after is also friendlier to user-facing paging, where the user moves forward and backward.

When scroll still makes sense

Scroll remains valid and is very common in existing tools. It is simple to reason about, it has a natural fit for reading everything once, and many client libraries offer helper functions that wrap it. If your Elasticsearch version or your client library does not yet support the newer approach, scroll is a safe choice.

Different reading jobs need different tools.

Ask whether you need the documents at all

Many scroll jobs read millions of documents only to compute a total, an average or a top-ten list. Aggregations do that work inside the cluster and return a tiny answer. Before writing an export loop, check whether a single aggregation can give you what you need.

You may be using scrolls without knowing it.

Several built-in features of Elasticsearch are, at heart, a loop that scrolls through the matching documents and does something with each batch. Knowing this helps you understand their behaviour, their resource use and the settings that affect them.

What this means for you

When a long reindex or update by query runs, it holds frozen views on the source index, with all the disk and file handle costs discussed earlier. This is one reason why large reindex jobs are best run during quieter periods, and why their duration matters. It also explains version conflicts in update and delete by query: the frozen view may contain an older version of a document than the live index now holds.

The frozen view gives strong guarantees, and a few surprises.

What the scroll guarantees

Every document that was searchable at the moment the scroll began, and matches your query, will be delivered exactly once. No document appears twice, and none is skipped because of changes made after the start. This is the property that makes scrolls trustworthy for exports and migrations.

What the scroll will not show

sequenceDiagram
    participant W as Writer
    participant I as Live Index
    participant S as Scroll View
    participant R as Reader
    Note over S: Scroll starts, view frozen with documents 1 to 5
    W->>I: Add document 6
    W->>I: Update document 2
    W->>I: Delete document 4
    R->>S: Request next batch
    S-->>R: Documents from the frozen view only
    Note over R: Reader sees old document 2 and document 4, and never sees document 6
        

The difference between consistent and current

The scroll is consistent, but not current. Its answers are perfectly repeatable, and they describe the index at one moment. They may be minutes or hours out of date by the time you finish. If your task needs the freshest data, such as syncing a live system, you must plan a follow-up pass to catch changes made during the scroll, for example by reading the documents whose timestamps are newer than the scroll’s start.

Because scrolls are stateful, some failures cannot be repaired by simply retrying.

The search context lives in the memory of the node that holds the shard copy. It is not saved to disk and it is not copied to other nodes. If that node restarts, crashes, or if the shard is moved away, the context disappears. The next scroll request that refers to it fails with an error saying that no search context was found for the given identifier.

Designing for restartability

Long export jobs should expect to fail sometimes. A robust design records progress outside the cluster, for example the last value of a sort field that was safely written, and knows how to begin a new scroll from that point using a range query. That way, a lost context costs a few minutes of work, not the whole job.

The same technique serves very different teams.

A scroll ID is a key to a set of results, so treat it with care.

When security features are enabled in Elasticsearch, a scroll is tied to the user who created it. The cluster checks that scroll requests come from the same authenticated user as the initial search. A different user who somehow obtains the scroll ID should be refused. This protects results from being read by another account, but it is still wise to avoid leaking scroll IDs in logs or shared tools.

Frozen views and deletion promises

Because a scroll pins a snapshot, a document deleted for privacy reasons can still be delivered by a scroll that started before the deletion. If you have a policy that data must vanish immediately, be aware that an in-flight scroll may still hold it, and limit scroll duration accordingly. The segments holding it are also kept on disk until the scroll ends.

You should know how many scrolls are open before they cause trouble.

Elasticsearch reports scroll activity in its node and index statistics. A handful of numbers gives you a clear picture of whether scrolls are healthy, and they are easy to chart and alert on.

Alerts worth having

Habits that keep scrolls safe, and traps that catch careful teams.

Patterns worth copying

Anti-patterns to avoid

An honest balance sheet for the scroll API.

A checklist to keep next to any export script.

Best practices

Common mistakes

Match the symptom to the likely cause and check the simplest explanation first.

A calm routine

First, read the exact error message, because most scroll errors name their cause and the setting behind it. Second, look at the number of open contexts and the free disk space on the nodes involved. Third, compare the keep-alive with the real time between requests. Fourth, check whether nodes restarted or shards moved during the job. These four checks explain the vast majority of scroll problems.

Short answers to doubts that come up again and again.

It lets you run one search and read its results in many consecutive batches from a consistent snapshot of the data, using a scroll ID to mark your position.

No. It is designed for batch processing such as exports and reindexing. User-facing pages should use ordinary paging or search after with a point in time.

No. It only needs to cover the time between one request and the next. Each request refreshes it.

No. The scroll shows the index as it was when the scroll started. Later additions, updates and deletions are not visible to it.

It will expire on its own after the keep-alive, but clearing it as soon as you finish frees memory, file handles and pinned disk space at once, so you should always clear it.

Each shard can simply walk through its documents without comparing scores or field values, so batches are cheaper to produce.

A way of dividing one scroll into several independent slices, each with its own scroll ID, so that different workers can read different portions in parallel.

The contexts held by that node are lost, and your next request fails with a missing search context error. You must begin again, ideally from a checkpoint you saved.

Both provide a stable view. With scroll the cluster remembers your position in a search context. With search after your client remembers the last sort values, which makes restarting easier and the cluster’s load lighter. Current documentation recommends the latter for deep paging.

Segments still needed by the scroll cannot be deleted after merges, so old and new segments coexist. This is worst on busy indices with long scrolls.

With security enabled, the scroll is bound to the authenticated user who created it, so other users should be refused. Still, avoid exposing scroll IDs in logs or shared tools.

Yes, they read documents in batches from a consistent view in a similar way, which is why their behaviour, resource use and conflicts resemble those of a scroll.

The Scroll API lets you read a very large result set in steady batches from a frozen view of the data. You open it with a search that carries a keep-alive time, receive a scroll ID, come back for batch after batch, and clear it when finished. Inside the cluster, each shard holds a search context that pins the segments of the frozen view and remembers your position, which keeps every batch cheap but costs memory, file handles and disk space until the scroll ends. It is ideal for exports, reindexing and batch clean-ups, poorly suited to real-time paging, and today the documentation prefers a point in time with search after for deep paging. Use it with short keep-alives, natural ordering, proper clearing, monitoring and a plan for failure.