Before we can appreciate the background version, we need to understand what happens when you run an ordinary search.

Clusters, shards and the search request

Elasticsearch stores documents in indices, and each index is divided into shards, which are smaller pieces that can live on different machines called nodes. When you send a search request, one node receives it and becomes the coordinating node for that request. It works out which shards must be asked, sends the query to each of them, and waits for their answers. Each shard searches its own data and returns its best results. The coordinating node then merges all these partial answers into one combined result and sends it back to you.

This two-step dance, ask every shard and then merge, is often called scatter and gather. The important thing for us is the waiting. The coordinating node cannot answer you until every shard has replied, or until the request runs out of time. If one shard is slow, everybody waits for that shard. The client, meanwhile, keeps its connection open the whole time, holding a line until the answer arrives.

sequenceDiagram
  participant C as Client
  participant N as Coordinating node
  participant S1 as Shard 1
  participant S2 as Shard 2
  participant S3 as Shard 3
  C->>N: Search request (connection held open)
  N->>S1: Run query
  N->>S2: Run query
  N->>S3: Run query
  S1-->>N: Results
  S2-->>N: Results
  S3-->>N: Results (slowest)
  N-->>C: Merged final result
        

Where the time goes

The time a search takes depends on several things: how many shards are involved, how much data each holds, how complicated the query is, whether aggregations must be computed, and how fast the underlying storage is. A simple lookup on a small index is instantaneous. A wide aggregation over a year of data spread across hundreds of shards is a different story, and the work adds up quickly.

A slow search is not just an inconvenience. It creates a chain of small problems that all trace back to one long wait.

Long-held connections are fragile

When a client waits for a long search, a connection stays open between the client and Elasticsearch, and often through several other machines in between, such as load balancers, reverse proxies and API gateways. Each of these middle machines usually has its own idle timeout, commonly somewhere between thirty and sixty seconds. If your search takes longer, one of them may cut the connection, and you receive a gateway error even though Elasticsearch is still happily working on your request.

Mobile networks, office Wi-Fi and laptops that go to sleep add more ways for a connection to break. When it breaks, the client has no answer and no way to get one, even if the cluster finishes the work moments later. All that effort is wasted, and the user must start again, making the cluster do the same heavy work a second time.

The user experience suffers

People are patient with a progress bar and impatient with a frozen screen. A dashboard that shows nothing for forty seconds and then either loads or errors out feels broken. A dashboard that shows a bar creeping forward, along with early numbers that keep improving, feels alive. The difference is not the total time; it is whether the user is kept informed.

Slow storage and huge data make it worse

Modern deployments often keep recent data on fast machines and older data on slower, cheaper storage, sometimes even on object storage in the cloud. Searching such data can take much longer, because bytes must be fetched from far away before they can be examined. Users still want to search it, since investigations often need to look back months or years, but they cannot reasonably be asked to wait on an open connection for minutes.

Async search lets you start a search, stop waiting for it, and come back later for the answer.

A precise definition

Async search is a feature of Elasticsearch that lets a client submit a search request and get control back immediately, without holding the connection open until the search finishes. The search keeps running inside the cluster. Elasticsearch gives the client an identifier for the running search, and the client can use that identifier to check progress, read partial results while the search is still running, fetch the final result once it completes, and finally delete the search when it is no longer needed.

The word “async” is short for asynchronous, which means “not happening at the same moment.” In a normal search, asking and answering happen in one continuous exchange. In async search, asking and answering are separated in time. You ask now, and you collect the answer whenever it suits you, as long as you do it before the stored result expires.

The three big ideas

What async search is not

It is very important to understand that async search does not make a search faster. The same shards do the same work, using the same amount of processing time and memory. What changes is how the client and the cluster communicate about that work. It is a change in the conversation, not a shortcut through the data. If a search takes two minutes normally, it will take about two minutes as an async search too, but you will not have to stand still for those two minutes.

graph LR
  A["Client submits search"] --> B["Cluster starts work and returns an id"]
  B --> C["Client does other things"]
  B --> D["Search keeps running in the cluster"]
  C --> E["Client checks progress with the id"]
  D --> F["Result stored for a limited time"]
  E --> G["Client fetches result when ready"]
  F --> G
        

The whole feature is built from four simple operations. Learn these and you understand the entire API.

Submit, get, status and delete

The async search API is a small family of operations. Submitting starts a search and looks very much like the ordinary search request, sent to an async search endpoint under the target index, and it accepts the same kind of query body. Getting fetches the current state and results of a previously submitted search by its identifier. Status returns only the lightweight state of the search, without the results. Deleting removes the search, cancelling it if it is still running, and discarding any stored result.

Why a separate status operation?

The get operation returns the whole response, which may contain large lists of hits and complicated aggregation results. If you just want to know whether the search has finished, fetching all that is wasteful. The status operation returns only a small summary: whether the search is still running, whether the result is partial, when it started, when it will expire, how many shards have finished, and, once complete, a completion code. It is meant for polling, the polite repeated checking that a client does while waiting.

The identifier

The submit operation returns an identifier, a long, opaque string. You should treat it as a handle that you pass back unchanged, without trying to read anything into its contents. The identifier is only returned if the search is still running when the submit call returns, or if you asked Elasticsearch to keep the result even when the search completes quickly. If the search finishes fast and you did not ask for storage, there is nothing left to point to, and no identifier is returned at all.

Let us follow a single async search from birth to cleanup so that every moving part has a place in your mind.

sequenceDiagram
  participant C as Client
  participant N as Coordinating node
  participant S as Shards
  C->>N: Submit search (wait up to a short time)
  N->>S: Run query on all shards
  Note over C,N: Wait window ends, search still running
  N-->>C: Id, running flag, partial result
  S-->>N: Shard results arrive over time
  C->>N: Status check with id
  N-->>C: Still running, 40 of 100 shards done
  C->>N: Status check with id
  N-->>C: Completed
  C->>N: Get result with id
  N-->>C: Final result
  C->>N: Delete search with id
  N-->>C: Acknowledged
        

Three settings decide almost everything about how an async search behaves. Get comfortable with them.

Wait for completion timeout

This setting says how long the submit call should wait for the search to finish before returning. Its default is a very short time, about one second. If the search finishes within that time, you get the full result immediately. If not, the call returns anyway, with an identifier and whatever partial result is available. Raising it makes fast-ish searches behave like ordinary ones, while lowering it makes the call hand back a ticket almost instantly.

Keep on completion

By default, Elasticsearch stores a search result only if the search is still running when the submit call returns. If the search finishes inside the wait window, the result is returned directly and nothing is stored. Setting keep on completion to true changes this: the result is stored even when the search finishes within the window, so that it can be fetched again later using its identifier. This is useful when a user might reload a page and expect to see the same result.

Keep alive

This is the lifetime of the stored search. After the keep alive period passes, Elasticsearch will delete the search and its result. The default is several days, which is generous. You can shorten it to save space, or extend it when you fetch a result, for example if a user is still looking at a report. Think of it as the number of days the dry cleaner promises to keep your shirt on the rack.

An async search response is a normal search response wrapped in a small envelope. Understanding the envelope is the key.

The envelope fields

The shard progress counters

Inside the response body, a section about shards reports the total number of shards involved, how many have successfully answered, how many were skipped because they could not possibly match, and how many failed. While the search runs, the successful count climbs toward the total. This gives a natural progress measure: forty successful shards out of a hundred means roughly forty percent of the shards have answered, though it is not a precise percentage of time remaining, because shards differ in size and speed.

What the partial flag really means

The partial flag is easy to misread. While the search is still running, the flag is always true, because you are looking at an incomplete picture. Once the search has finished, the flag has a second meaning: it is false if the search succeeded on all shards, and true if it failed or some shards could not be searched. This means a finished search with the partial flag set to true is a warning sign. It tells you that the results may not represent all your data, and you should look at the shard failure information before trusting them.

One of the nicest features of async search is that early answers arrive while the rest is still being computed.

How partial results are built

As shard answers arrive at the coordinating node, they are not merged all at once at the very end. Instead, the coordinating node merges them in batches. The number of shard results merged at a time is controlled by a setting called batched reduce size, with a default of five. Each time a batch is merged, the running combined result is updated, and that updated result is what a client sees when it asks for the current state.

This has a second benefit that has nothing to do with progress bars: merging in batches keeps memory use under control on the coordinating node. Holding hundreds of full shard results at once before merging can consume a lot of memory, whereas merging as you go lets earlier pieces be released.

graph TD
  S1["Shard results 1-5"] --> R1["Partial merge A"]
  S2["Shard results 6-10"] --> R2["Partial merge B"]
  R1 --> P["Running combined result"]
  R2 --> P
  S3["More shard results"] --> R3["Partial merge C"]
  R3 --> P
  P --> V["Visible to the client as a partial response"]
  P --> F["Final result once every shard has answered"]
        

What a user can do with partial results

A dashboard can draw charts from partial aggregations, and let them refine as more shards report in. A search page can display the first hits found while the rest of the shards are still working. An analyst can decide early that a query is on the wrong track and cancel it, saving cluster resources. All of these turn a long blank wait into an active experience.

The honesty problem

Partial results are exactly that: partial. A count computed from thirty percent of the shards is not thirty percent of the final answer in any reliable sense, because data is not evenly spread. Top-ten lists can change entirely as new shards contribute. Averages can shift. Minimums and maximums can jump. A well-designed interface labels early numbers clearly as provisional, for example with a “still loading” indicator, and avoids drawing firm conclusions or triggering actions such as alerts from them.

Checking on a search should be cheap, polite and predictable. The status operation exists for exactly this.

What the status operation returns

The status response is a compact summary. It includes whether the search is running, whether it is partial, the start and expiry times, the shard progress counters, and, once the search is complete, a completion status code that mirrors the code an ordinary search would have returned, such as success or a failure code. It does not include hits or aggregations, so it is small and quick to produce.

Polling strategies

Polling means asking repeatedly. Done carelessly, it can flood the cluster with pointless requests. Done thoughtfully, it is almost free. The most common good practice is to start with short intervals, since many searches finish soon, and then lengthen the interval gradually the longer the search runs, a technique known as backoff. A pause of one second, then two, then four, up to a modest ceiling such as ten or fifteen seconds, works well for most interfaces.

Long polling as an alternative

The get operation itself supports a wait time. If you ask for the current result and provide a short wait, Elasticsearch will hold the request for up to that long, returning early if the search completes. This gives you a form of long polling, where each check waits a little inside the cluster instead of returning immediately. It reduces the number of round trips, at the cost of holding a connection briefly, so keep waits short enough to stay within any proxy timeouts.

flowchart TD
  A["Submit search"] --> B{"Finished within wait window?"}
  B -->|"Yes"| C["Show final result"]
  B -->|"No"| D["Receive id and partial result"]
  D --> E["Wait, then check status"]
  E --> F{"Still running?"}
  F -->|"Yes"| G["Increase delay, update progress bar"]
  G --> E
  F -->|"No"| H{"Partial flag false?"}
  H -->|"Yes"| I["Fetch and show final result"]
  H -->|"No"| J["Inspect failures and warn the user"]
        

Stored results are real data on real disks. They deserve the same care you give to any other data.

A dedicated system index

Elasticsearch keeps the state and results of async searches in a special hidden system index reserved for this purpose. You do not create or manage it directly, and you should not write to it yourself. It is an ordinary index in terms of storage, which means the stored results occupy disk space on the cluster, take part in replication, and are affected by disk watermarks like any other data.

Expiry and clean-up

Every stored search has an expiry time, based on its keep alive period. After that moment, the search is no longer accessible, and a background maintenance task periodically deletes expired entries from the system index. Because clean-up runs periodically, an expired result may linger briefly on disk, but it can no longer be retrieved. You can extend the lifetime by fetching the search and asking for a longer keep alive, and you can shorten its life by deleting it explicitly.

A size limit for each stored response

To protect the cluster, there is a limit on the size of a stored async search response, controlled by a cluster setting with a default of ten megabytes. If a completed search would produce a response larger than the limit, the search is marked as failed instead of storing an enormous result. This is a good reminder that async search is meant for delivering answers, such as summaries and reasonably sized lists of hits, not for exporting millions of documents. For bulk export, other tools are more appropriate.

Good housekeeping is part of using async search properly. Every search you start should have a planned ending.

Extending the lifetime

When you fetch a search, you may supply a new keep alive value. Elasticsearch then moves the expiry time forward to match. This is handy for interactive tools: as long as the user keeps a report open, the tool can quietly extend the lifetime each time it refreshes, and when the user leaves, the lifetime simply runs out and the search disappears on its own. It is much better than setting an enormous lifetime at the start “just in case.”

Cancelling and deleting

The delete operation does two jobs. If the search is still running, it cancels the underlying work, so shards stop spending effort on it as soon as they can. Then it removes the stored state and any result. This makes delete the right tool both for cleaning up finished searches and for stopping ones that are no longer wanted, such as when a user changes the filters and starts over.

Cancellation is cooperative. Shards check for cancellation at regular points while they work, so a heavy search does not stop in the same instant, but it stops quickly, and, importantly, the cluster no longer keeps merging results that nobody wants.

Elasticsearch offers several search styles, and each solves a different problem. Choosing the right one avoids confusion.

The family of search approaches

A regular search is the default and the best choice for anything that finishes quickly. The scroll approach, which is now discouraged for most uses, walked through large result sets by keeping a snapshot open. The point in time feature combined with a search-after parameter is the modern way to page through large results consistently. Multi-search sends several searches in one request but still waits for them all. Async search is the one designed for long-running searches where you want to detach from the connection and observe progress.

Async search and paging are different things

A common confusion is to think of async search as a way to fetch many pages of results. It is not. It returns a single response, the same shape as a normal search response, containing the hits for the requested size and the aggregations you asked for. If you need to walk through millions of documents, use a point in time with search-after, or a purpose-built export process, not a giant async search response.

flowchart TD
  Q["I need to run a search"] --> A{"Will it finish in a second or two?"}
  A -->|"Yes"| R["Use a regular search"]
  A -->|"No"| B{"Do I need progress, partial results or to detach?"}
  B -->|"Yes"| AS["Use async search"]
  B -->|"No"| C{"Do I need to read a very large number of documents?"}
  C -->|"Yes"| PIT["Use point in time with search-after"]
  C -->|"No"| R2["Regular search with a sensible timeout"]
        

Knowing the boundaries of a feature is as valuable as knowing its strengths.

It does not make queries cheaper

Each shard still executes the query in full. The processing, memory and disk reads are the same as for a regular search. If a query is expensive, an async version is equally expensive. Async search is a communication improvement. If you want a query to cost less, you must improve the query, the mapping, the number of shards, the time range or the hardware.

It is not a job scheduler

Async search runs a search now, in the background. It does not schedule a search for later, repeat it every hour, or guarantee that it will run to completion after a restart. If a node involved in the search leaves the cluster or restarts, the search may fail or be incomplete. Regular reporting jobs belong in a scheduler, which can then use async search underneath if it wishes.

Some search options are restricted

Not every option of the regular search is supported in the async version. Options that are about walking through results in a stateful way, such as scroll, are not available, and the search type is limited to the standard query-then-fetch approach. Some defaults also differ from a regular search, so it is worth reading the current documentation for your version before assuming that every parameter behaves identically.

A background search is still a real search. It uses real threads, real memory and real disk.

Threads and queues

Shard-level search work runs on a pool of search threads on each node. When many searches arrive at once, work waits in a queue, and if the queue is full, new work is rejected. Async searches use the same pool as regular searches, so a burst of long-running background searches can crowd out the quick interactive ones that users expect to be instant. Because async searches are longer, they hold the resources longer.

Memory on the coordinating node

The coordinating node keeps the running merged result in memory. For aggregations with many buckets, this can be large. Batched merging helps by releasing memory as it goes, but large partial results still occupy space until the search is finished and the result is handed off to storage. Circuit breakers guard against runaway memory use and will fail a search rather than risking the node.

Disk and indexing of results

When a search completes and its result is stored, Elasticsearch writes the result into the system index. That is an indexing operation with a real cost, though small compared with the search itself, and it also has to be replicated. For heavy use, hundreds of stored results of several megabytes each represent noticeable storage.

Ways to protect the interactive experience

The situations that made async search necessary are becoming more common, not less.

Data tiers and slower storage

To control costs, many organisations keep fresh data on fast, expensive nodes, and move older data to cheaper storage through several tiers. On the coldest tiers, data may live in object storage and be fetched on demand into a local cache when searched. A search that touches such data can take many seconds or longer, especially the first time, because the needed pieces must be downloaded before they can be examined.

This is a natural fit for async search. The user asks a question about last year, gets a ticket immediately, sees early results from the faster tiers, and receives the complete answer when the slower tiers finish. Without async search, the user would either face a long blank wait or be told that old data is off limits.

Cross-cluster search

Cross-cluster search lets one cluster query others, often in different regions. The slowest remote cluster, plus network delays, sets the pace for a normal search. With async search, results from faster clusters can appear early while slower ones are still working. Partial results are especially valuable here, since some clusters may be far away or briefly unreachable, and a user would rather see most of the picture than nothing at all.

graph LR
  U["User dashboard"] --> C["Local cluster: coordinating node"]
  C --> H["Hot tier: fast local shards"]
  C --> W["Cold tier: slower shards"]
  C --> R1["Remote cluster in region 1"]
  C --> R2["Remote cluster in region 2"]
  H -->|"answers first"| C
  W -->|"answers later"| C
  R1 -->|"answers later"| C
  R2 -->|"may be slowest"| C
        

You may already be using async search without knowing it, because friendly tools rely on it behind the scenes.

Kibana and progress feedback

Kibana, the visual interface for Elasticsearch, uses async search to run many of its queries, which is why you often see a progress indicator while a large dashboard loads. Rather than hold an unbroken connection open, Kibana submits searches, polls their progress, updates the interface as results arrive, and cancels searches when you navigate away or change a filter. Understanding the pattern helps you reason about what Kibana is doing when a dashboard is slow, and about the load it places on your cluster.

Background searches and sessions

Some versions of Kibana have offered a feature that lets a user send a long search to the background and come back later to see its result, often called a search session. This relies on stored async search results with a suitable lifetime. If you use such a feature, remember that each stored session keeps data in the system index until it expires, and that the lifetimes you configure directly affect storage use.

The same pattern elsewhere

A stored result is a copy of data taken from your documents, so it must be protected like the documents themselves.

Results belong to the submitting user

When security features are enabled, each async search is tied to the user who submitted it. Other users cannot simply fetch, extend or delete that search by knowing its identifier. Administrators with sufficiently powerful privileges can typically manage other users’ searches, which is important for clean-up and for stopping runaway queries. The exact privileges required are documented for your version, and it is worth reviewing them when you design roles.

Data access rules still apply

A search submitted by a user runs with that user’s permissions. Index-level, field-level and document-level restrictions apply to the search just as they would in a normal search, so the stored result contains only what that user was allowed to see at the time. This is good news, but it also means a stored result reflects a moment in time. If permissions are later withdrawn, the stored copy may still exist until it expires or is deleted.

Treat identifiers as sensitive

Even though access is tied to the user, it is sensible to treat identifiers as sensitive and avoid pasting them into public chat channels, tickets or logs. They are handles to stored data. If an application logs identifiers for debugging, protect those logs with the same care as other operational data.

A healthy async search setup shows up in a few simple numbers. Watch them and problems announce themselves early.

What to watch

Where to look

The status operation gives per-search detail. The task management view lists running search tasks, which is useful for finding long-running or stuck searches and cancelling them. Node statistics show thread pool queue lengths and rejections, and the cluster’s health and disk statistics reveal storage pressure. Slow logs help identify which queries take the longest, so that they can be improved. Your own application should also log the durations and outcomes of the async searches it submits, since it sees the entire experience from the user’s side.

flowchart LR
  A["Alert: search rejections rising"] --> B["List running search tasks"]
  B --> C{"Many long async searches?"}
  C -->|"Yes"| D["Find owners, cancel or narrow their queries"]
  C -->|"No"| E["Check regular search load and hardware"]
  D --> F["Add limits on concurrent async searches per user"]
        

Things go wrong in distributed systems. A good client is ready for each situation in advance.

The reassuring part

One of the great advantages of async search is that a dropped client connection does not destroy the search. If the network hiccups while you are polling, you can simply reconnect and poll again with the same identifier, as long as it has not expired. This is a large improvement over the regular search, where a dropped connection often means starting from scratch.

Let us walk through a realistic story to see all the pieces working together.

The scenario

A retail company collects web server logs into daily indices, keeping ninety days of data. A support engineer uses an internal dashboard to see the number of errors per hour for each service over the last ninety days. The dashboard sends one search with a date histogram aggregation nested inside a terms aggregation across all ninety daily indices, which together contain around nine hundred shards. On a busy day, this search takes about forty seconds.

The old behaviour

With a regular search, the dashboard shows a blank chart for forty seconds. The office reverse proxy cuts connections after thirty seconds, so the engineer often sees a gateway error, reloads the page, and starts another heavy search, doubling the load on the cluster. Sometimes three or four of these pile up before anyone gets an answer.

The new behaviour

sequenceDiagram
  participant E as Engineer
  participant D as Dashboard
  participant ES as Elasticsearch
  E->>D: Open 90-day error chart
  D->>ES: Submit async search (wait 2s, keep alive 10m)
  ES-->>D: Id, running, partial buckets
  D-->>E: Early chart and progress bar
  D->>ES: Status check with backoff
  ES-->>D: 30 percent of shards done
  E->>D: Filter to one service
  D->>ES: Delete first search
  D->>ES: Submit narrower async search
  ES-->>D: Finished, no failures
  D-->>E: Final chart
  D->>ES: Delete search
        

Most of the quality of an async search experience is decided in the client code, not in the cluster.

The building blocks of a good client

A well-behaved client has five habits. It submits with a short wait so that quick searches finish in one round trip. It records the identifier as soon as it receives one. It polls the status with backoff and jitter. It checks the completion state and shard failures carefully before showing results as final. And it always deletes searches it no longer needs. If you build these five habits into a small shared library, every screen in your application gets them for free.

Designing the interface

Good interfaces label early results honestly. A small “still loading” tag, a progress bar based on shards completed, and a gentle change of colour when the search is complete let the user understand what they are looking at. Offer a visible cancel button, since the ability to stop a wrong search is one of the most valuable features of the pattern. If the search fails partway, show a clear message with a one-click retry rather than an unexplained error.

Working with proxies and gateways

Because each individual request in the pattern is short, proxy timeouts stop being a problem. The submit call returns after the short wait, each status check is quick, and the final fetch is normally quick as well. If you use long polling with a wait on the get operation, keep the wait comfortably below the shortest timeout in your chain, so that no middle machine ever cuts a request.

These patterns have proved themselves in many production systems.

graph TD
  UI["User interface"] --> API["Application service with async search helper"]
  API --> LIM["Concurrency limiter per user"]
  LIM --> ES["Elasticsearch cluster"]
  API --> STATE["Session state: current search id"]
  ES --> IDX["Hidden system index for stored results"]
  API -->|"delete on change or exit"| ES
        

Recognise these habits early and you will save yourself and your cluster a lot of trouble.

A little arithmetic tells you how much space and effort a busy async search setup will need.

The three multiplying factors

The storage consumed by async search results depends on three things multiplied together: how many searches are stored at any moment, how big the average stored result is, and how many copies of the system index exist. The first factor is driven by your user numbers and by how long lifetimes are. If two hundred analysts each store five results at any one time, that is a thousand stored results. If each averages two megabytes, the store holds about two gigabytes, which is modest. If lifetimes are days long and results are never deleted, the number of stored results could grow into the tens of thousands and beyond.

Estimating concurrency pressure

On the compute side, estimate how many async searches will be running simultaneously at peak, multiply by the average number of shards each touches, and compare with the number of search threads available across your data nodes. If the requested shard work greatly exceeds what your threads can handle, queues will grow and rejections will appear. Cutting the number of shards a search touches, through better time filters and sensible shard sizes, has an enormous effect on this calculation.

Every feature is a bargain. Here is what you gain and what you pay.

Regular search compared with async search

Here is how organisations with different needs put the pattern to work.

The lesson across all these examples

In every case, the value came from respecting the difference between the computer’s timescale and the human’s. Clusters can work for a minute without complaint, but people cannot stare at nothing for a minute without frustration. Async search bridges the gap by keeping the person informed while the machine does its work, and by letting the person change their mind while work is still in progress.

A few misunderstandings appear again and again. Let us set them straight.

A short history helps you read older articles and understand why the feature looks the way it does.

Short answers to the questions beginners ask most often about async search.

It is a way to submit a search, receive an identifier and control back immediately, watch progress and partial results while it runs, and fetch the final result later, without holding a connection open.

No. The shards do the same work. What improves is the experience of waiting: you can detach, see early results, cancel, and recover from dropped connections.

Use it when a search may take long enough to hit proxy timeouts or to frustrate users, for example wide aggregations, cold tier data or cross-cluster searches. Use regular search for quick lookups.

Because the search finished within the wait window and you did not ask to keep the result, so nothing was stored. Set keep on completion to true if you want an identifier even for fast searches.

While the search runs it is always true, because you see an incomplete picture. After completion it is false if the search succeeded on all shards and true if it failed or missed some shards.

Until the expiry time shown in the response, which is based on the keep alive setting. You can extend it when you fetch, or shorten it by deleting the search.

It returns a small summary of the search, including running state, shard progress and completion code, without the potentially large results, so it is ideal for polling.

Yes. Deleting a running search cancels the underlying work and removes its stored state. Deleting a finished search removes the stored result.

With security enabled, searches are tied to the user who submitted them, so ordinary users cannot retrieve someone else’s search. Privileged administrators can typically manage them. Still, treat identifiers as sensitive.

Yes. There is a cluster setting that limits the size of a stored response, with a default of around ten megabytes. Larger responses cause the search to fail rather than being stored.

The search may finish with shard failures, marked as partial, or fail. Async search does not resume interrupted work, so rerun the search if the results are incomplete.

Start with short intervals, such as a second, and lengthen them gradually up to a modest ceiling, adding a little random jitter. Stop polling when the user leaves and set a maximum total wait.

Not sensibly. It returns a single size-limited response. Use a point in time with search-after, or a dedicated export approach, for bulk reads.

Kibana commonly uses it to run searches, which is why you see progress indicators on heavy dashboards. Related features such as event query language and SQL have their own asynchronous variants.

Treating partial or finished-but-incomplete results as final, and forgetting to delete searches. Always check completion flags and shard failures, and always clean up.

Async search is a way of running Elasticsearch searches in the background. You submit a query, receive an identifier if the search is still running, poll a lightweight status, read partial results as shards report in, fetch the final result when it is ready, and delete the search when you are done. It solves the practical problems of long searches: broken connections, blank screens, wasted work and users who cannot change their mind. It does not make the underlying work any smaller, so it must be combined with good data modelling, sensible shard counts, narrow queries and controlled concurrency. Used with care, checking completion flags, limiting lifetimes, securing identifiers and cleaning up after itself, it turns a long, fragile wait into a calm, informative conversation between people and their data.