The shortest possible answer: search_after is a search request option that continues a sorted search right after a position that you name, using the sort values of the last result you already received.

Let us build the definition slowly. Elasticsearch is a search and analytics engine. Companies use it to search products, explore application logs, count events, and draw charts. Data lives in indices, which are like big labeled cabinets of documents, and each index is cut into pieces called shards so that many servers can share the work.

When you search, Elasticsearch returns a list of matching documents, called hits. A search usually returns only a small batch of hits at a time, called a page, whose length is set by the size option. To read the next page, you need a way to say where the previous page ended.

The search_after option is that way. You must sort the search by one or more fields. Every hit that comes back carries its sort values, the values of the sort fields for that document. To ask for the next page, you send the same search again, with the sort values of the last hit of the previous page placed into search_after. Elasticsearch then returns the next size documents that come after that position in the sort order.

Why it is sometimes called a live cursor

A cursor is a marker that remembers a position in a list of results. Many databases keep a cursor open on the server. With search_after, the server keeps nothing between requests, and the marker lives entirely in the values that you send back. For this reason, the documentation describes it as a live cursor: it always looks at the current state of the index, and it holds no memory on the server unless you add a point in time to it.

What it is not

The option does not save results on the server, it does not guarantee that the pages come from one frozen view of the data, and it does not let you jump to page fifty. It also does not work without a sort. It is a simple, efficient way to walk forward through a sorted result list. When you need a frozen view, you combine it with the point in time feature, which Chapters 12 and 13 explain.

Where you will meet it

You will meet search_after when you build an infinite scroll page that keeps loading more items, when you export millions of documents to another system, when a batch job must process every record exactly once, when a log viewer offers a “load older events” button, and when a search request fails because it asked for results beyond the ten thousandth. In each case, the bookmark idea gives a clean and cheap solution.

Asking for a page far from the start is expensive in a distributed search engine. The search_after option exists to make deep reading cheap and safe.

To see the problem, we must understand how a search works across shards. A search request lands on a coordinating node, which forwards it to one copy of every relevant shard. Each shard finds its best matching documents, and the coordinating node combines the partial lists into one final list.

The cost of skipping

Suppose you ask for the results from position 10,000 to 10,010, which means skipping ten thousand hits and returning ten. No shard knows where the global position 10,000 is, because the global order only exists after combining all shards. So, to be correct, each shard must return its top 10,010 hits. If the index has five shards, the coordinating node receives about fifty thousand candidate hits and sorts them all, just to throw away ten thousand and keep ten.

The cost grows with depth. Page one costs a little. Page one thousand costs a thousand times more. Memory use on the shards and on the coordinating node grows in the same way, and many users asking for deep pages at once can bring a cluster to its knees.

The safety valve: the result window

To protect clusters from this cost, Elasticsearch has a limit called the result window. The index setting index.max_result_window defaults to ten thousand. A request whose from plus size is larger than that limit is rejected with an error. This protects the cluster, but it also means that ordinary pagination cannot reach beyond the ten thousandth hit. If you need to read further, you need a different tool.

The earlier answer: scroll

The first tool built for this need was the scroll API. A scroll opens a search context on the server that remembers a frozen view of the index, and you fetch the results in batches by sending a scroll identifier back and forth. Scroll works well for exports, but it ties up server resources, it is designed for one fixed query, and it is not meant for live user-facing pagination. Today, the documentation no longer recommends it for deep pagination and points to search_after with a point in time instead.

The bookmark answer

With search_after, each shard is told the exact sort position after which to look, so each shard returns only its next size hits after that position. The coordinating node merges a few hundred entries, not tens of thousands. The cost of every page is about the same, whether it is the second page or the second millionth.

An illustrative story

A short history

The search_after option arrived in Elasticsearch 5.0 as a lighter alternative to scroll for live pagination. Its main weakness was the lack of a consistent view, since the index could change between requests. Version 7.10 added the point in time API, which freezes the view without the drawbacks of scroll. Later versions added an automatic tiebreaker for such searches. Together, these pieces are today’s recommended way to read deeply and reliably.

To follow the rest of this guide you need eight simple ideas: hit, page, sort, sort values, cursor, tiebreaker, segment, and point in time.

Sort values are not always what you expect

The sort values that come back with a hit are the internal keys used for ordering, and they can look different from the fields in the document. A date field, for example, shows its sort value as a number of milliseconds since a fixed starting date in 1970. A text label stored as a keyword shows up as the string itself. A number shows as a number. When you build the next request, you send these values back exactly as you received them, not the original document values.

Why sorting must be predictable

The bookmark works only if the sort order is stable and complete. Stable means that the same documents always sort in the same relative order. Complete means that no two documents share exactly the same sort values, so that the position of a bookmark is unique. Much of this guide is about making sorting stable and complete, because everything else builds on that.

The path of a search request

The diagram shows how a paged search travels. The client usually reaches the cluster through a load balancer or an API gateway, which passes the request to a coordinating node. That node asks the shards for their partial results, merges them, and returns one page.

flowchart LR
    A["Client application"] --> B["Load balancer or API gateway"]
    B --> C["Coordinating node"]
    C --> D1["Shard 1: top hits after the bookmark"]
    C --> D2["Shard 2: top hits after the bookmark"]
    C --> D3["Shard 3: top hits after the bookmark"]
    D1 --> E["Coordinating node merges partial lists by sort values"]
    D2 --> E
    D3 --> E
    E --> F["One page of hits, each with its sort values"]
    F --> A

Look closely at Figure 1. The client receives the sort values of the last hit and keeps them. On the next request, the client sends them back, and every shard starts from that same position. The whole state of the pagination lives in the client, in the form of a few values.

To appreciate the bookmark, look closely at the method it replaces: skipping a count of results with from and size.

The classic pagination options are from, the number of hits to skip, and size, the number of hits to return. To read the third page of ten results per page, you set from to twenty and size to ten. This is simple to understand, and it allows jumping to any page number directly. It is perfect for the first few pages of a result list.

What each shard must do

Every shard does not know how many of its hits fall before the global position of the requested page. So it must return at least from plus size hits. The shard keeps a sorted queue of that many candidates while it scans its matching documents, and every candidate takes memory and time.

Where from and size still shine

Ordinary pagination is the right tool when users look at the first few pages and want to jump between them. Product search pages often work this way, where hardly anyone goes beyond page five. Its cost at that depth is small, and its simplicity is valuable. You do not need a bookmark for a result list that people read shallowly.

Where it breaks down

• Deep reading. Cost and memory rise with every page, and the result window limit stops the reading at ten thousand hits.
• Full exports. Reading all documents of a large index page by page costs far more than the data itself.
• Endless scrolling. Feeds and timelines invite users to read far beyond the first pages.
• Concurrent users. Many deep requests at once multiply the pressure on memory and CPU.

Another quiet problem: shifting pages

Ordinary pagination also suffers from shifting. If a new document arrives at the top of the sorted list between two requests, every later document moves down by one position. The user who requests the next page then sees the last document of the previous page again, because it shifted into the next page’s range. A bookmark does not have this problem, since it names a place relative to a document, not a count from the start.

The whole technique fits in one loop: ask for a page, keep the sort values of the last hit, and send them back to ask for the next page.

Let us walk through the loop from the point of view of an application. The steps are the same whether the application is a web feed or a nightly export job.

flowchart TD
    A["Send first search with sort and size"] --> B["Receive a page of hits"]
    B --> C{"Any hits in the page?"}
    C -->|No| D["End of results: stop"]
    C -->|Yes| E["Process the hits"]
    E --> F["Copy the sort values of the last hit"]
    F --> G["Send the same search with search_after set to those values"]
    G --> B

What the engine does with the bookmark

When a shard receives a request with search_after, it treats the bookmark as a position in the sort order. It looks only for documents that sort strictly after that position and returns its best size among them. It does not need a long queue of skipped documents. The work per page depends on the page size, not on how many pages came before.

Rules that the request must follow

• A sort is required. Without a sort, there is no order in which to find a position.
• The skip value must be zero. The from option must be zero or unset when search_after is used, because the bookmark replaces skipping.
• The bookmark must match the sort. It must contain one value for every sort field, in the same order, with values of the right types.
• The query and sort must stay the same. If you change the query or sort between pages, the bookmark loses its meaning.

A first taste of the problems ahead

The loop looks simple, and for many uses it works well. Two questions remain. What if several documents share the same sort values, so that the bookmark does not point to a single place? And what if documents are added, changed, or removed between pages? The next chapters answer these questions, and the answers determine whether your pagination is merely convenient or truly dependable.

A paged search has only a few moving parts. Knowing each one, and what the engine expects, prevents almost every beginner error.

A search request with search_after has the same shape as an ordinary search. It carries a query that decides which documents match, a sort that decides their order, a size that decides how many hits come back, and, from the second page on, the search_after bookmark.

What a hit looks like in the response

The response contains a list of hits. Each hit has its usual details, such as the index it lives in, its id, and its source. When a sort was used, each hit also contains a sort array. The array holds one value for every sort field, in the order in which you listed the fields. The array of the last hit on the page is your next bookmark.

The first page and the last page

The first page is an ordinary sorted search with no bookmark. The last page is the one that returns no hits. Some applications also treat a page with fewer hits than the requested size as the last page, since the engine had nothing more to return. Be careful with this shortcut when partial shard failures can occur, because a page may be short for the wrong reason. Chapter 24 returns to this point.

Counting matches costs extra

By default, Elasticsearch counts the total number of matching documents, but it stops counting at ten thousand. For a deep walk, this counting is wasted effort, since you do not need the total on every page. You can switch it off with the track_total_hits option to save time. If your user interface needs a total, ask for it once, in a separate request, instead of on every page.

What errors look like

• A missing sort. The request is rejected, because a bookmark needs an order to refer to.
• A skip value above zero. The request is rejected, because skipping and bookmarking cannot be combined.
• A bookmark with the wrong number of values. The engine complains that the count of bookmark values differs from the count of sort fields.
• A bookmark value of the wrong type. For example, a text value where the sort field is a number.
• A page size larger than the result window. The size itself is limited, not only the sum with the skip value.

Follow one paged request from the coordinating node to the shards and back, and you will see why the bookmark makes deep pages cheap.

Elasticsearch runs a search in two phases, called query then fetch. In the query phase, every shard finds its best matching documents in the requested order and returns only small records: the sort values and the document position. In the fetch phase, the coordinating node asks the owners of the winning documents for their full contents.

sequenceDiagram
    participant C as Client
    participant N as Coordinating node
    participant S1 as Shard 1
    participant S2 as Shard 2
    C->>N: Search with sort, size, and search_after
    N->>S1: Query phase with the bookmark
    N->>S2: Query phase with the bookmark
    S1-->>N: Next size hits after the bookmark, sort values only
    S2-->>N: Next size hits after the bookmark, sort values only
    N->>N: Merge by sort values and keep the best size
    N->>S1: Fetch phase for winning documents
    N->>S2: Fetch phase for winning documents
    S1-->>N: Document sources
    S2-->>N: Document sources
    N-->>C: One page of hits, each with its sort values

Is the cost really flat?

The cost per page is bounded in memory, since each shard keeps only a small queue. It is not always constant in time. A shard still has to scan matching documents to find those after the bookmark, comparing sort values as it goes. For a query that matches a large share of the index, and a sort that is unrelated to how the index is stored, the shard may examine many documents that come before the bookmark before it finds those that come after. Still, this work is cheap compared with maintaining huge queues and merging huge lists, and it does not grow with the memory of deeper pages.

Index sorting can help

Elasticsearch offers a feature called index sorting, which stores the documents of each segment in a chosen order. When a search sorts by the same fields, the engine can stop scanning early once it has enough hits, and with a bookmark, it can jump close to the right place. This helps for stable, high-volume walks, but it has a cost at indexing time and must be decided when the index is created.

Memory on the coordinating node

With ordinary pagination, the coordinating node receives about the number of shards times the sum of the skip and the size. With search_after, it receives about the number of shards times the size. If you have two hundred shards and a page size of one thousand, that is two hundred thousand small records, which is manageable but not trivial. A smaller page size, or fewer shards, keeps the coordinating node calm.

Why the fetch phase stays small too

The heavy fetch work is done only for the final page, since the query phase already reduced the candidates. This is the same for every page. Trimming the returned fields with source filtering makes the fetch phase lighter still, which matters for exports of large documents.

The bookmark is only as good as the sort behind it. Not every field is suited for sorting, and some choices cause errors or surprises.

Sorting relies on a compact, column-like structure called doc values, which Elasticsearch builds at indexing time for most field types. Doc values let a shard read the sort value of any document quickly. A field without doc values cannot be sorted on efficiently.

Direction and missing values

Each sort field has a direction, ascending or descending. The bookmark refers to a position in the direction that you chose, so after means later in that direction. If some documents lack the sort field, the missing option decides where they go: first, last, or with a chosen value. The sort values of those documents are special extreme numbers, and they must be sent back like any other value. Keep the setting the same on every page.

Fields with several values

A field can hold several values in one document, for example a list of tags or prices. For sorting, Elasticsearch needs one value per document, so the mode option chooses which one: the smallest, the largest, or a computed value. The bookmark then contains that chosen value. Because the choice affects the order, it must be the same on every page.

Searching several indices

A search can target several indices at once. If the sort field is mapped differently in some of them, for example as a date in one and a keyword in another, sorting fails or gives odd results. If an index does not have the field at all, the unmapped_type option tells the engine how to treat it. When you page through data spread over many indices, such as daily logs, make sure that the sort fields have the same type everywhere.

Sort on stable values

• Prefer immutable fields. A creation time does not change, while a “last updated” time does. Sorting on a value that changes moves documents around during the walk.
• Prefer indexed values over computed ones. Scripts are slower and can hide instability.
• Keep the mapping stable. Changing a field’s type or format during a walk invalidates the bookmark.
• Use the same sort in every request. The list of fields, their order, their directions, and their options must all match.

The most common mistake with search_after is sorting by a field that is not unique. The result is silent: pages simply skip documents, and nobody notices.

The bookmark means “give me documents that sort strictly after these values.” If several documents share the same sort values, the bookmark cannot tell them apart. A page that ends in the middle of a group of tied documents leaves some of them behind, and the next page starts after all of them.

A small example with numbers

Imagine eight documents, sorted from oldest to newest by a timestamp field. Five of them share the same timestamp, because they were created in the same second. The page size is three.

flowchart TD
    A["Documents A B C D E share timestamp 10"] --> B["Page 1 returns A B C, bookmark is timestamp 10"]
    B --> C["Page 2 asks for timestamps strictly after 10"]
    C --> D["D and E have timestamp 10, so they are not after 10"]
    D --> E["D and E are silently skipped forever"]
    A --> F["With a unique id in the sort, bookmark is timestamp 10 and id 3"]
    F --> G["Page 2 asks for positions strictly after that pair"]
    G --> H["D and E come next and nothing is lost"]

Ties are more common than you think

• Timestamps at low resolution. Events created in the same second or millisecond share the same time, especially with bulk loading.
• Categories and statuses. Fields with few distinct values are tied for almost all documents.
• Prices and scores. Many products have the same price, and many hits share the same relevance score.
• Imported data. Batch imports often assign one timestamp to a whole batch of documents.

The silent nature of the error

What makes this problem dangerous is that nothing fails. The requests succeed, the pages look fine, and the loop ends normally. Only a comparison of the number of documents read with the number that exist reveals the loss. Good exports therefore verify counts at the end, and good designs prevent the problem with a tiebreaker.

Ties also make the order unstable

Even the order within a group of ties is not guaranteed. Different shard copies, or the same shard at different times, may list tied documents in different orders. Without a tiebreaker, even a single page can change between two identical requests. A unique tiebreaker gives the whole list one fixed order.

A good tiebreaker is unique, stable, cheap to sort on, and known when the index is designed. Here are your options and their trade-offs.

Add the tiebreaker to the mapping early

The best time to think about tiebreakers is when you design the mapping. If every document has a unique key, map it as a keyword or a number so it has doc values. If not, copy the document id into a dedicated field at indexing time, either in the application or with an ingest step. Adding such a field later means reindexing, which is far more work.

The automatic tiebreaker of a point in time

When you search with a point in time and do not include a tiebreaker of your own, Elasticsearch adds one for you. It is a hidden position value called _shard_doc, which combines the shard and the internal document number, and it is unique within that point in time. You can also list it explicitly in your sort, for example to choose its direction. It appears as the last value in the sort array of every hit.

Why the automatic tiebreaker is not always enough

The automatic value is only meaningful for one point in time. If your export job crashes and later restarts with a new point in time, the old value refers to a position that no longer exists in the new view. To resume across points in time, you need a tiebreaker that is stored in the data, such as a business key or a copied id. This is a key point for long-running exports and is discussed again in Chapters 20 and 24.

Order of the sort fields

The tiebreaker goes last. The main sort field, such as a date, comes first, and the tiebreaker resolves ties among documents with the same main value. Putting the unique field first would sort by id alone and lose the meaning of the main field.

Each request sees the index as it is at that moment. If documents change between pages, the walk can show surprising results. Knowing exactly how helps you decide whether it matters.

Without a point in time, search_after is stateless, and every request sees the current data. This is often what you want for a live feed. But for an export that must be exact, changes between requests can cause trouble.

Different shard copies, different views

Each shard has a primary copy and possibly replica copies. Requests may be served by any of them, and copies can differ slightly in what they have refreshed or merged. Without a frozen view, two consecutive pages may be served by two copies that see the data a little differently. For sorting on immutable, unique fields, this rarely matters. For sorting by score, it can. The preference option lets you route related requests to the same copies, which reduces this effect.

When live behavior is what you want

A live feed or a “load more” button often benefits from seeing current data. New items appear as users scroll. The occasional repeated or skipped item in a very active feed is acceptable. In such cases, plain search_after with a good unique tiebreaker is exactly right, and adding a point in time would only hold extra resources.

When you need a frozen view

• Exports and migrations. Every document must be read exactly once.
• Reports and audits. Numbers must match a defined moment.
• Batch processing. A job that changes documents while reading them must not read its own changes.
• Relevance paging. Pages by score must not shift because of background changes.

A point in time is a photograph of an index. Every search that uses it sees exactly the same data, no matter how much the index changes afterward.

The point in time feature, usually shortened to PIT, was added in Elasticsearch 7.10 to solve the consistency problem of Chapter 11. You open a PIT on one or more indices, and Elasticsearch returns an identifier, a long, opaque text value. From then on, you can run searches against that identifier, and every search sees the index exactly as it was when you opened the PIT.

How it works in simple terms

Remember that shards store data in read-only segments, and a search works with a consistent set of segments called a reader. When you open a PIT, each shard keeps its current reader alive, even if new segments appear or old ones are merged away. Searches with the PIT identifier use those preserved readers. New documents, updates, and deletions made afterward are invisible to them. The photograph stays sharp until you close it or it expires.

sequenceDiagram
    participant A as Application
    participant E as Elasticsearch
    A->>E: Open a point in time on the index with a keep alive
    E-->>A: Point in time identifier
    A->>E: Search page 1 using the identifier, sort, and size
    E-->>A: Hits with sort values and a possibly updated identifier
    A->>E: Search page 2 using the latest identifier and search_after
    E-->>A: Hits with sort values and a possibly updated identifier
    A->>E: Close the point in time using the latest identifier
    E-->>A: Confirmation

The keep alive time

When you open a PIT, you set a keep_alive value, such as one minute. This is not the total lifetime of the PIT. It is the time that the PIT stays alive after its last use. Each search that uses the PIT can include its own keep alive value, and that extends the life again. As long as you keep asking within the interval, the PIT lives on. If you go quiet for longer than the interval, Elasticsearch releases it.

The identifier can change

Each search response using a PIT contains an identifier. It may be different from the one you sent. Always use the identifier from the most recent response for the next request. Treat it as a value that can be renewed.

Special rules for searches with a PIT

• No index name. The PIT already defines which indices are searched, so the search request does not name them.
• No routing or preference. The PIT has fixed which shard copies hold the frozen view, so these options do not apply.
• Any query. Unlike scroll, a PIT is not bound to one query. You may run different queries against the same frozen view.
• Automatic tiebreaker. As Chapter 10 explained, a unique position value is added to the sort unless you add your own.

Closing the PIT

When the walk is finished, close the PIT explicitly with the delete operation, passing the latest identifier. This releases the resources on the shards at once instead of waiting for expiry. A good program closes the PIT even when errors occur, in the same way that it closes files.

What a PIT does not do

It does not copy data. It does not protect you from deletion of the whole index, because deleting an index removes its files. It does not make the results permanent; it is a temporary view. And it does not survive a restart of the nodes that hold it or the movement of a shard to another node, because the preserved readers live in the memory and files of those nodes.

Together, the two features give an exact, resource-friendly way to read every matching document once and only once.

flowchart TD
    A["Open PIT with a keep alive"] --> B["Search page with pit, sort, size"]
    B --> C{"Any hits?"}
    C -->|No| G["Verify shard failures, then close the PIT"]
    C -->|Yes| D["Process hits"]
    D --> E["Save last sort values and the newest PIT id"]
    E --> F["Search next page with search_after and the newest PIT id"]
    F --> C

Choosing the keep alive

Pick a value that comfortably covers the time between two requests, including the time to process a page and to retry after a hiccup. One to five minutes is common. A very long value wastes resources when a client crashes, since the PIT lingers. A too-short value risks expiry during a slow step. Because each request extends the life, the value need not cover the whole walk.

Combining an automatic and an explicit tiebreaker

For simple, single-run walks, the automatic tiebreaker is enough. For jobs that must be able to resume after a crash, add your own unique field to the sort. You may even keep both: the business key first, so that a resumed walk under a new PIT can continue from a stored bookmark, and the automatic value last, which does no harm.

Processing while reading

If your job changes the documents it reads, for example by updating them, the PIT keeps its frozen view, so the job does not see its own updates and does not loop. Be aware that the changes still create new segments and may lead to extra disk use while the PIT lives, which Chapter 18 explains.

Checking for completeness

At the end of an exact walk, compare the number of documents processed with an independent count taken at the same PIT, if the count matters. Since the PIT is frozen, a count query against it returns a number that must match your processed total. This simple check catches configuration errors such as a non-unique tiebreaker.

Elasticsearch offers several ways to read many results. Each fits a different need, and choosing the right one saves both effort and cluster resources.

flowchart TD
    A["I need to read many results"] --> B{"Only the first few pages, with jumping between them?"}
    B -->|Yes| C["Use from and size"]
    B -->|No| D{"Must every document be read exactly once?"}
    D -->|No| E["Use search_after alone with a unique tiebreaker"]
    D -->|Yes| F["Use search_after with a point in time"]
    F --> G{"Is the export huge and time critical?"}
    G -->|Yes| H["Add slices and read them in parallel"]
    G -->|No| I["Use one sequential walk"]

Why scroll is no longer the first choice

Scroll works, but it is heavier, it binds the context to a single query, and it is not flexible about changing the query mid-way. Its state must be kept on the server for every reader, and it competes for a limited number of open contexts. Elasticsearch still uses scroll-like behavior internally for tasks such as reindexing, and older applications rely on it, but new code is advised to use a PIT with search after.

Mixing methods in one product

A common design uses from and size for the first few pages of a search result screen, where jumping is useful, and switches to search_after when the user goes deeper or when a background job needs everything. This combination gives users the comfort of page numbers where they matter and the efficiency of bookmarks where depth matters.

Cost summary in one sentence each

• From and size: cost grows with the page number, and it stops at the result window.
• Scroll: cost is steady, but it holds a heavy context.
• Search after alone: cost is steady and stateless, but the view is live.
• Search after with a point in time: cost is steady, the view is frozen, and the held resource is light.

A bookmark points forward. Yet users want a Previous button and sometimes a page number box. There are honest ways to offer these, with limits.

The search_after option is a forward-moving tool. It says “what comes after this position.” It has no built-in way to say “what came before,” and it cannot jump to the fiftieth page without walking there. Still, applications can build the missing features.

The cursor stack for Previous

The simplest way to go back is to remember the bookmarks you used. Every time the user moves forward, the application pushes the bookmark used for the page just shown onto a stack. To go back, it pops one bookmark and requests that page again with it. Since each stored bookmark points to the position just before the page, the same page comes back, subject to whatever changes happened in the live index in the meantime.

The reverse sort trick

A second way to go backward is to flip every direction in the sort, use the sort values of the first hit of the current page as the bookmark, and ask for a page. The engine returns the documents that sort just before the current page, but in reverse order. The application reverses the list before showing it. This trick also gives a “Last page” button: ask for the first page with all directions reversed and reverse the result.

Why random jumping is not offered

To jump to page fifty, the engine would need to know the sort values at the start of page fifty, and those exist only after computing the first forty-nine pages. There is no shortcut, because the bookmark is a value taken from the data, not a number counted from the start. Some applications precompute bookmarks for every hundredth page in the background, and offer coarse jumps. Others redesign the interface: filters, search boxes, and date pickers help users reach the region they need in a few steps instead of paging.

User interface designs that fit

• Infinite scroll or “load more.” Perfect for forward-only reading, and the bookmark is all you need.
• Next and previous buttons. Served by the cursor stack.
• Filters and date jumps. Instead of page numbers, let users narrow the results.
• Approximate page counts. Show “more than ten thousand results” instead of an exact page count.

A single walk reads one page at a time. For very large exports, slices let several workers read different parts of the same frozen view at once.

A slice is a disjoint part of the result set. When you ask for slice number two out of eight, you receive a well-defined eighth of the matching documents, and the other seven slices contain the rest, with no overlap. The slice option takes two numbers: the identifier of the slice and the total number of slices. Each worker reads its own slice with its own bookmark, all against the same point in time.

flowchart TD
    A["Open one point in time"] --> B["Slice 0 of 4"]
    A --> C["Slice 1 of 4"]
    A --> D["Slice 2 of 4"]
    A --> E["Slice 3 of 4"]
    B --> B1["Worker 0: own search_after loop"]
    C --> C1["Worker 1: own search_after loop"]
    D --> D1["Worker 2: own search_after loop"]
    E --> E1["Worker 3: own search_after loop"]
    B1 --> F["Downstream system: all documents exactly once"]
    C1 --> F
    D1 --> F
    E1 --> F

Rules for slicing

• One point in time for all slices. The slices are consistent only if they share the same frozen view.
• Same query and sort in all slices. Only the slice identifier differs between workers.
• No global order. The order of documents is guaranteed only inside one slice. Do not expect worker zero’s last page to come before worker one’s first page.
• Bookmarks per slice. Each worker keeps and, if needed, checkpoints its own bookmark.
• Keep alive for everyone. Each search extends the life of the shared point in time, so make sure that at least one worker’s requests keep it alive, or that each uses a keep alive that covers gaps.

How many slices?

More slices mean more parallelism, but also more load on the cluster at once. A sensible start is a number of slices that is not more than the number of shards, since slicing along shard boundaries is the cheapest way to divide the work. Using many more slices than shards can make the division itself more expensive. Check the documentation of your version for details, and measure the effect on cluster load while the export runs.

Do not overload the cluster

Four workers reading pages of a thousand hits produce four times the load of one worker. On a cluster that also serves live searches, a large parallel export can hurt user latency. Run exports at quiet times, limit the number of workers, use moderate page sizes, and watch search latency while the job runs.

Handling failures with slices

If one worker fails, the others can continue. The failed worker’s slice can be resumed from its last saved bookmark, as long as the point in time is still alive. If the point in time has expired, a resumed worker needs a new point in time and a business-unique tiebreaker so that it continues from the stored bookmark under the new view. Because the new view may differ from the old one, the job must tolerate small differences or must verify counts afterward.

The bookmark already removes the biggest cost of deep paging. A handful of other choices decide how quickly each page arrives.

Page size in practice

For user interfaces, a page of ten to fifty is typical. For exports, hundreds to a few thousand work well. There is a sweet spot: very small pages cause too many round trips, and very large pages create heavy responses that take long to serialize and to process. Try several sizes with your data, and choose the one where throughput stops improving.

Sorting by fields, not by score

When a search sorts by a field and not by score, Elasticsearch does not need to compute scores, which saves time. If your query is just a filter with no relevance needs, express it in a filter context and sort by a stable field. This combination is the fastest way to walk large sets of documents.

Keep the request shape stable

Because every page repeats the same query and sort, the engine can reuse work such as parsed queries and warmed file caches. Do not vary the query between pages, and avoid adding unneeded features such as highlighting or aggregations to every request of a walk. Aggregations should be computed once, in a separate request.

Measure with the reported time

Each response contains a took value, the time in milliseconds spent inside Elasticsearch. Compare it across pages in a walk. If it stays flat, your setup works as intended. If it rises steadily, something unexpected is happening: a sort that is not using doc values, a filter that scans too much, or a cluster under load.

A frozen view is not free. It holds old files on disk and small structures in memory for as long as it lives. Small habits keep that cost tiny.

To keep the photograph sharp, each shard must keep the segments that were current when the PIT was opened, even if merges would normally replace and delete them. So, as long as the PIT lives, disk space that would have been reclaimed remains in use.

stateDiagram-v2
    [*] --> Open
    Open --> InUse: A search uses the PIT and extends the keep alive
    InUse --> InUse: More searches keep it alive
    InUse --> Closed: The application closes it
    InUse --> Expired: No search within the keep alive time
    Expired --> [*]
    Closed --> [*]

What is held

• Old segments on disk. If the index has heavy updates or merges during the life of the PIT, more old files stay behind. The extra space depends on how much the index changes.
• Small in-memory structures. Each shard keeps a reader and some bookkeeping. This is small for one PIT, but it adds up with many.
• Open contexts. Each PIT creates a context on every shard that it covers. Clusters have limits on the number of open search contexts.

A small sizing example

Imagine fifty exports running at the same time, each with a PIT over an index that has twenty shards. That creates one thousand open contexts. If your cluster’s limit or practical budget is lower, some exports will fail with errors about too many contexts. This example is invented to show the arithmetic, but it explains why large numbers of long PITs need planning.

Habits that keep the cost low

• Short keep alive. Use just enough to cover the gap between requests.
• Close when done. Release resources immediately instead of waiting for expiry.
• Limit concurrency. Cap the number of simultaneous exports.
• Prefer smaller scopes. Open PITs on the specific indices you need, not on wildcard patterns that cover everything.
• Do not open one per user request. Interactive interfaces that need only “load more” usually do not need a PIT at all.
• Watch disk margins. On heavily updated indices, leave free space for the segments that PITs keep alive.

How to see what is open

The node statistics for search report the number of open contexts on each node. A number that rises and does not come back down points to PITs or scrolls that are never closed. Chart this number over time, and alert on sudden growth.

The bookmark approach trades some convenience for a large gain in efficiency. Here is an honest scorecard so that you can decide when to use it.

The central trade-off: simplicity against efficiency

Skip-based paging is the simplest to understand and the easiest to build. The bookmark is only slightly more complicated, but it requires the application to store values and to design its interface around forward movement. In return, it removes the depth limit and the growing cost. For shallow reading, simplicity wins. For deep reading, efficiency wins.

The second trade-off: freshness against consistency

A live walk shows the newest data but may repeat or miss documents that change during the walk. A walk with a point in time shows a consistent view but hides changes made after it started and holds resources. The right choice depends on whether exactness or freshness matters more for the task.

The third trade-off: keep alive against cost

A long keep alive time protects a slow job from expiry, but it also leaves resources held longer if the job crashes. A short time saves resources but demands quick processing between pages and good retry logic. Balancing the two is part of designing a robust job.

The fourth trade-off: parallelism against load

Slices finish large jobs sooner but load the cluster harder. On a shared cluster, speed must be balanced against the needs of live users.

When search_after is the wrong tool

If users need to jump to arbitrary pages and the result lists are short, use skip-based paging. If you need aggregated numbers, use aggregations instead of walking documents. If you need a one-time bulk copy between clusters or indices, purpose-built features such as reindexing or snapshots may be simpler and faster than a custom walk.

Reliable pagination is mostly a matter of a few good habits, chosen at design time. These eight habits cover almost every case.

Pattern 3 in action

A nightly job copies three hundred million documents into a warehouse. After each page of two thousand documents, it writes the last sort values into a small table together with a job identifier. On the tenth night, the network fails at the two hundredth million. The next morning, the job reads the stored bookmark, opens a new point in time, and continues from the same position, using the business key in the sort. It finishes the remaining hundred million documents without repeating the first two hundred million. The job’s design took an afternoon and saved a whole night of work.

Pattern 4 in action

The same warehouse job writes each document to the target using the document id as the key, so that a repeated write replaces the same record. When the resumed job re-reads the last page before the crash, the duplicates overwrite themselves, and the target ends up with exactly one record per document.

Pattern 6 in action

A product search page shows page numbers up to page ten using skip-based paging, which costs little at that depth. A “show more results” button beyond that uses bookmark paging with a signed token. Ninety-nine percent of visitors never go beyond page three, and the rest get a fast, safe experience.

A pattern for documentation

Write a short design note for each paged feature: the query, the sort with its unique key, the page size, whether a point in time is used, the keep alive time, the maximum depth allowed, and how failures are handled. When someone changes the mapping or the sort later, this note shows what depends on them.

These are the traps behind most pagination bugs. Each is common and easy to avoid once you know it.

Other frequent mistakes

• Sending a non-zero skip value together with the bookmark.
• Reformatting sort values instead of sending them back untouched.
• Using an outdated point in time identifier after a newer one was issued.
• Choosing a page size so large that responses time out.
• Using scroll in new code where a point in time would be lighter.
• Forgetting to close points in time when errors occur.
• Assuming that pages of a live walk are stable snapshots.

When your own API hands bookmarks to callers, you become responsible for keeping them safe, honest, and useful. A small amount of design prevents leaks and abuse.

Most applications do not let browsers talk to Elasticsearch directly. A web or mobile client calls your application service through a load balancer or an API gateway. The service queries Elasticsearch and returns a page of results with something the client can use to ask for the next page. What should that something be?

flowchart LR
    A["Browser or mobile app"] -->|Page request with cursor token| B["API gateway with rate limits"]
    B --> C["Application service"]
    C -->|Decode and verify token| D["Build search with sort and search_after"]
    D --> E["Elasticsearch cluster"]
    E --> F["Hits with sort values"]
    F --> G["Application builds a new signed cursor token"]
    G -->|Page of results plus next cursor token| A

Why not return raw sort values?

• They reveal data. Sort values are copies of field values. They might contain identifiers, times, or names that the caller should not learn.
• They can be tampered with. A caller could craft arbitrary bookmarks to probe positions or to request strange combinations that your code did not expect.
• They tie your API to your internal mapping. Changing the sort fields later would break every client that stored bookmarks.

Opaque, signed tokens

The safer design is an opaque cursor token. The service packs the bookmark, the identifier of the query or filter settings, the user identity, an expiry time, and possibly the point in time identifier into one text value, then encodes and signs it with a secret key. The client only stores and returns the token. On the way back, the service verifies the signature, checks the expiry and the user, and rebuilds the search from trusted parts.

Access control still applies

Elasticsearch checks the privileges of the user or service account on every search, including searches that use search_after. Roles that limit which indices, documents, or fields a user may read are applied to each request. Creating a point in time also needs read access to the target indices. Use dedicated service accounts with the least privileges that the feature needs, and never give an export job wider rights than its indices require.

Do not share point in time identifiers between users

Treat the identifier as a secret that grants access to a frozen view of the data. Do not put it into URLs, logs, or browser storage in the clear. In most designs, the identifier never needs to leave the server side.

Limit depth and rate for public APIs

For APIs exposed to the public, agree on a maximum number of pages or documents that one caller may read, and enforce it in the gateway or the service. Bookmark paging makes deep reading cheap for the cluster, which also makes it cheap for scrapers. Rate limits and quotas protect both your data and your budget.

A healthy paging feature is quiet. A few numbers show early when pages slow down, contexts leak, or an export loses documents.

Metrics worth charting

• Page latency by depth. A line that rises with depth reveals a configuration problem.
• Open contexts per node. Shows leaks and overload.
• Search rejections. A sign of cluster saturation.
• Documents per second for each export. Shows throughput and the effect of tuning.
• Checkpoint age for running jobs. Shows whether jobs are progressing.
• Count mismatches. The most important quality metric for exact jobs.
• Coordinating node heap and garbage collection. Large pages and many slices show up here first.

Logging in the application

Log, for every job, the job identifier, the query fingerprint, the page number, the number of hits, the time taken, the shard failure count, and the current checkpoint. Do not log raw bookmarks or documents if they contain private data. With these logs, you can reconstruct exactly what happened in any run.

Alerts that help

Create a short list of alerts. First, an alert when open contexts on any node stay high for more than a set period. Second, an alert when an export reports a count mismatch or shard failures. Third, an alert on search rejections. Fourth, an alert when page latency at depth rises above a limit. Fifth, an alert when a job has not advanced its checkpoint for a long time. Each alert should link to a short runbook.

Long walks meet failures: expired contexts, restarts, overloaded nodes, and half-finished pages. A robust job expects them and knows how to continue.

flowchart TD
    A["A page request fails"] --> B{"What kind of error?"}
    B -->|Timeout or rejection| C["Wait with growing delay, then retry the same page"]
    B -->|Missing search context| D["Open a new point in time"]
    D --> E["Resume from the last saved bookmark with the same query and sort"]
    B -->|Shard failures reported| F["Retry the page, and fail the job if failures persist"]
    B -->|Sort or bookmark error| G["Stop: query, sort, or mapping changed. Start a new walk"]
    C --> H["Continue the loop"]
    E --> H
    F --> H

Scenario: the point in time expired

If the job pauses longer than the keep alive time, the shards release the frozen view. The next request fails with an error saying that no search context was found. The recovery is to open a new point in time and continue from the last saved bookmark, using the same query and sort. This works only if the sort contains a stored, business-unique tiebreaker, because the automatic position value belongs to the old point in time. The new view may differ from the old one, so the job should tolerate small changes or check counts at the end.

Scenario: a node restarts or a shard moves

The frozen view lives on specific nodes. If one of them restarts, or a shard relocates, the context is lost, and the point in time fails in the same way as an expired one. The same recovery applies. If this happens often, review why nodes restart, and schedule big jobs away from maintenance windows and rebalancing.

Scenario: some shards fail during a page

A search can succeed partially. The response lists how many shards failed, and the hits come only from the others. An exact job must not accept such a page. The best practice is to check the failure count on every response, and either retry, or request that a partial result be treated as an error. Otherwise the job may silently miss a whole shard’s worth of documents.

Scenario: timeouts and rejections

Under heavy load, the cluster may reject requests when its queues are full, or a request may time out. Searches are read-only, so retrying the same page is safe. Use growing delays between attempts, and reduce the page size or the number of parallel workers if the problem persists.

Scenario: duplicates or misses in a live walk

Without a point in time, updates and deletions during the walk can cause repeats or misses, as Chapter 11 explained. Design the consumer to be idempotent to handle repeats, and use a later reconciliation pass, for example by comparing counts or by running the walk again for documents changed since the start, to catch misses.

Scenario: the mapping changed mid-walk

If a sort field changes type or format while a walk is in progress, bookmarks no longer match, and requests fail with errors about the type of a value. Stop the walk, fix the cause, and start a new one. Do not attempt to continue with converted values unless you are sure of the meaning.

Scenario: the client crashes

A crash leaves the point in time to expire and the job to be restarted. The stored checkpoint tells the new run where to continue. Without a checkpoint, the job starts from the beginning, which is safe but slow. With a short keep alive, the abandoned point in time disappears quickly and costs little.

Habits that make recovery easier

• Store a checkpoint after every page, or every few pages.
• Include a business-unique tiebreaker in the sort for resumable jobs.
• Make consumers idempotent.
• Check shard failures and final counts automatically.
• Close points in time in a cleanup step that always runs.

When something looks wrong, match the message or symptom to its likely meaning. This table covers what people meet most often.

A calm way to debug

Here are four common situations, with rough numbers, showing search_after at work. The companies are imaginary, but the patterns are the ones teams meet every day.

A worked example with numbers

The search_after option is a simple idea with a big effect: continue from where you stopped, instead of counting from the start. Here is everything to remember.

Keep in mind that option names, defaults, limits, and the availability of features such as the point in time API can change between Elasticsearch versions, so confirm exact details against the documentation of the release you run. With that habit, and the ideas in this guide, you can read as deep into your results as you need, quickly and safely, without ever counting from the front cover again.