Three plain ideas, put together: indexing, bulk, and tuning. Once each is clear, the whole subject makes sense.

Indexing means adding a document to Elasticsearch so that it can be searched. A document might be a product, a customer record, or one line from a log file. When you index it, Elasticsearch does not just save it. It also breaks the text into words, builds lookup structures, and stores the values in several shapes so that searching later is fast. That extra work is why indexing costs more than simply writing a line to a file.

The Bulk API lets you send many indexing operations in one single request. Each operation in the request says what to do (add a new document, replace one, update one, or delete one) and, for adds and replaces, carries the document itself. The cluster processes the whole batch and answers with one response that lists what happened to every item.

Tuning means adjusting the knobs that control speed. Some knobs live in your own loading program, such as how many documents go in each request and how many requests run at once. Some live in the index settings, such as how often new data becomes searchable and how many copies are kept. Some live in the cluster itself, such as the number of shards and the kind of disks. Bulk indexing tuning is the work of setting all of these knobs sensibly, using measurements instead of guesses.

The three places where tuning happens

The one-sentence version

If you remember one line from this chapter, make it this: bulk indexing tuning is the practice of sending well-sized batches, from the right number of parallel workers, into an index whose settings favor writing, while watching the cluster’s health signals so that you go as fast as it can safely go and no faster.

Every tuning practice exists because something was painfully slow or dangerously fragile. Here are the pains.

Every request sent over a network carries a fixed overhead. The client opens or reuses a connection, sends a header, waits for the cluster to receive it, waits for the work, and waits for the answer. This waiting is the same whether the request carries one document or five thousand. When you index documents one at a time, you pay that overhead again and again, and most of your time goes to waiting instead of working.

A numbers walkthrough

Let us use some invented numbers to feel the difference. Suppose one round trip to the cluster costs 3 milliseconds of pure waiting, and indexing one document takes about 1 millisecond of real work. You need to load 1 million documents.

Sending them one by one, in a single line, costs about 4 milliseconds each. That is 4,000 seconds, or a bit over an hour. Now send them in bulk requests of 5,000 documents. That is 200 requests. Each one pays the 3 milliseconds of waiting once, plus the real work of indexing 5,000 documents, which is about 5 seconds if you keep the same 1 millisecond estimate. The total is roughly 1,000 seconds. Real clusters can index many documents in parallel, so with several workers you might finish in a couple of minutes. The exact figures will differ on your system, but the shape never changes: batching removes most of the waiting.

A short history

The exact release for each step is worth double-checking in the release notes of your version. The direction of travel is what matters: from “send anything and hope” toward “the cluster pushes back, and clients cooperate.”

Why not just send as fast as possible?

A natural idea is to turn every knob to the maximum: giant batches, hundreds of workers, no pauses. The result is usually worse, not better. Giant batches consume memory on the node that receives them. Too many workers fill the queues and trigger rejections. Rejections lead to retries, and retries add more load. The cluster ends up busy handling failures instead of indexing. The goal is a steady, well-fed pipeline, like a conveyor belt that is always full but never jammed.

A dozen words explain nearly everything in this guide. Learn them here, and later chapters will read like common sense.

The write path in one breath

When a document arrives, Elasticsearch analyzes it, meaning it turns text into searchable words. It places the result in an in-memory buffer and also records the operation in the translog on disk. At the next refresh, the buffer becomes a new small segment that searches can see. Later, a flush stores segments durably and trims the translog. Meanwhile, background merges tidy up the growing collection of small segments. Every tuning trick in this guide is really a way to make one of these steps cheaper, rarer, or more parallel.

Backpressure is a feature, not a bug

Beginners often see the 429 answer as an error to be suppressed. It is actually a healthy signal, similar to a traffic light turning red. The cluster is telling the client, honestly, that it needs a short pause. A well-tuned loader listens, waits a little, and tries again, ideally with a growing delay each time. A badly tuned loader either ignores the signal and loses data, or hammers harder and makes the jam worse.

Two ways to measure speed

Documents per second is easy to explain, but it depends on how large the documents are. Megabytes per second is fairer when document sizes vary. When you compare two experiments, use the same documents in both, or use bytes per second, so that you are comparing the tuning and not the data.

Where does a bulk request go after it leaves your program, and which parts of the cluster can slow it down?

A production cluster commonly has a load balancer in front, some nodes that act as coordinators, and many data nodes that hold the shards. The bulk request enters at the front, is taken apart by a coordinating node, and its pieces are sent to the primary shards that own each document. The primaries then pass the work on to their replicas.

flowchart TB
    Client["Indexing workers: your application or ingest tool"] --> LB["Load Balancer"]
    LB --> Coord["Coordinating Node: splits the bulk by shard"]
    Coord --> P1["Primary Shard 1"]
    Coord --> P2["Primary Shard 2"]
    subgraph NodeA["Data Node A"]
        P1
        R2["Replica of Shard 2"]
        WQA["Write thread pool and queue"]
    end
    subgraph NodeB["Data Node B"]
        P2
        R1["Replica of Shard 1"]
        WQB["Write thread pool and queue"]
    end
    P1 --> R1
    P2 --> R2
    P1 -.-> WQA
    P2 -.-> WQB
    

Read the picture from top to bottom. Your workers send requests through the load balancer, which spreads them across coordinating nodes. The coordinating node looks at each document’s target shard, using its id or routing value, and groups the documents by shard. Each group is sent to the node holding that primary shard. On that node the work waits in the write queue for a free worker thread. The primary indexes the documents and then forwards the same operations to each replica, which indexes them again on its own node. Notice that the replica of Shard 1 sits on Node B, next to the primary of Shard 2, so every node is doing both primary work and replica work at once.

The main components

Walking one bulk request through the machine

Now we open the shard itself. Most of the settings you will tune act on the small machinery shown below.

flowchart LR
    B["Incoming documents"] --> IB["In-memory indexing buffer"]
    B --> TL["Translog on disk"]
    IB -->|refresh| SEG["New small segment: now searchable"]
    SEG -->|merge| BIG["Larger merged segment"]
    TL -->|flush| COM["Lucene commit: segments saved, translog trimmed"]
    SEG --> COM
    

Why frequent refreshes hurt bulk loads

Refresh is what gives Elasticsearch its “near real-time” feel. But every refresh writes a new segment, and every new segment adds work for the future: it must be searched, and it will eventually need to be merged. During a large load, nobody is waiting for the newest document to become searchable, yet the cluster keeps refreshing every second. Loosening the refresh, as Chapter 8 explains, saves a lot of work.

Why replicas double the work

In the standard design of Elasticsearch, a replica does not simply copy files from the primary. It receives the same operations and indexes them itself, doing the analysis and buffering again. With one replica, your cluster does the indexing work twice. That is the price of safety and search capacity, and it is why turning replicas off during a first-time load can nearly halve the total indexing effort. Later, when replicas are added back, they are created by copying finished segment files, which is cheaper than indexing every document a second time.

Why the merge scheduler matters

The number of segments a shard accumulates depends on how fast it is fed and how fast it can merge. Merges are driven by a policy that groups segments of similar size. Fast disks, such as solid-state drives, allow merges to keep pace. Slow or shared disks are the classic cause of indexing that starts fast and then falls off a cliff as merges pile up.

A big load is a small project with a beginning, a middle, and an end. Most disasters happen because someone skipped the end.

flowchart LR
    S1["Plan: mapping, shards, sizing"] --> S2["Prepare: refresh off, replicas 0, translog relaxed"]
    S2 --> S3["Load: parallel bulk workers with retries"]
    S3 --> S4["Verify: counts and failed items"]
    S4 --> S5["Restore: refresh on, replicas on, translog durable"]
    S5 --> S6["Optional: force merge on read-only data"]
    S6 --> S7["Serve searches"]
    

The phases in order

Three kinds of loads

Not all bulk indexing is the same, and the right tuning depends on the kind.

• Initial load: a one-time fill of a new index from another system. Speed is the top goal. Aggressive settings are safe because you can always start again.
• Continuous ingestion: a steady stream such as logs or events. You cannot switch off refresh or replicas, because searches and safety are needed all the time. Tuning here is about steady, efficient batches and headroom for spikes.
• Reindexing: copying data from one index to another, usually after a mapping change. It behaves like an initial load, with the extra detail that the source index is being read at the same time.

What happens to searches during the load

With refresh loosened or switched off, new documents will not appear in search results until the next refresh. That is fine for a fresh index nobody queries yet. For an index that live users search, do not switch refresh off. Choose a longer interval, such as ten or thirty seconds, so that data still becomes visible in a reasonable time while the refresh cost drops sharply.

The cheapest tuning happens in the program that sends the data. No cluster change is needed, and the gains are often large.

Choosing the bulk size

The right size is measured in bytes, not in document count, because documents vary in size. Elasticsearch’s own tuning guidance suggests starting with requests of roughly 5 to 15 megabytes and increasing slowly. For documents around one kilobyte, that means several thousand documents per request. Beyond a certain point, larger requests stop helping and start hurting: they consume memory on the coordinating node, they take longer to retry when they fail, and they can trigger rejections from the memory guard. There is also a hard ceiling on request size set by the HTTP settings of the cluster, which by default is 100 megabytes.

Finding the sweet spot, step by step

Parallel workers

A single sender can rarely fill a cluster, because it spends part of its time preparing the next batch or waiting for the answer. Several workers overlap those pauses. How many workers is best depends on how many data nodes, cores, and shards you have. The rule is simple: add workers until the cluster starts pushing back or the CPU on the data nodes is fully busy, then use a comfortable amount less than that.

Compression and response trimming

Bulk bodies are text, and text compresses very well. If the network is a bottleneck, and clients typically support compressing the request, turning it on can reduce transfer time at the price of a little CPU on both ends. The response can also be trimmed: by default it lists every item in detail, which is wasteful when nearly all items succeed. Requests can ask the cluster to return only the parts of the answer that you need, for instance the error flag and the details of failed items, which cuts down both network traffic and client parsing.

Retries done right

When a batch comes back with a “too many requests” signal, do not resend it immediately. Wait, and let the wait grow if the problem persists. Adding a little randomness to the wait prevents all workers from retrying at exactly the same moment. When only some items in a batch failed, retry only those items, not the whole batch, so that you do not repeat work that already succeeded.

Spreading the load

If every request goes to one node, that node’s coordinating work and memory become the bottleneck while the others sit idle. Point clients at several nodes, or at a load balancer that spreads requests evenly. For large clusters, dedicated coordinating or ingest nodes can take the splitting work off the data nodes.

A handful of index settings decide how much background work every document creates. Loosening them for a load can change the speed dramatically.

Most of these are dynamic index settings, so they can be changed on a running index without a restart. The memory setting is a node-level setting and needs a restart to change, so it is usually planned in advance if changed at all.

Refresh interval

Refresh makes new data searchable and creates a new small segment each time it runs. Switching it off, or stretching it to a long interval, removes a large amount of repeated work. There is one detail worth knowing: modern Elasticsearch skips scheduled refreshes on shards that nobody has searched for a while, by default around thirty seconds. That means an index that no one is querying may already refresh less often than you expect. Setting the value explicitly is still wise because it makes the behavior predictable.

Replicas

Because each replica indexes documents itself, zero replicas means your cluster does the analysis and indexing work only once. For a first-time load this can nearly double the speed. The trade-off is safety: while there are no replicas, a lost node means lost data, and the load must be redone. That is acceptable only when the source data still exists and the load can be repeated. When replicas are restored, the new copies are built from finished segment files, which is faster than replaying every document. Restoring them still uses disk and network, so plan for that time before opening the index to heavy traffic.

The translog

By default, Elasticsearch syncs the translog to disk before answering each request, so a confirmed write survives a crash. Switching to asynchronous durability syncs every few seconds instead, which saves time on each request but means that a crash can lose the last few seconds of acknowledged writes. For a load that can be rerun, that is a reasonable gamble. For continuous ingestion of data you cannot recreate, keep the safe default.

The indexing buffer

The in-memory buffer holds recently indexed documents before they are written out as a segment. Its total size is shared among all shards that are actively indexing on the node, and it defaults to ten percent of the heap. A larger buffer means bigger, fewer segments, which is good, but the benefit shrinks as the buffer grows, and the memory comes out of the same heap that everything else needs. Most teams leave it alone, and only consider changes when a node has many busy shards.

Auto-generated ids

When you give a document your own id, Elasticsearch must first check whether a document with that id already exists in the shard, because the new operation might be a replacement. That check is real work. When Elasticsearch generates the id itself, it can skip the check, because it knows that a fresh id has never been used. For append-only data such as logs and events, generated ids make indexing noticeably faster.

The trade-off is repeatability. With your own ids, sending the same document twice simply overwrites it, which makes retries safe. With generated ids, a retry after an uncertain timeout can create a duplicate. Choose according to your data: append-only streams favor generated ids, and data that must be updated or deduplicated needs your own ids.

Things that slow indexing on purpose

• Index sorting orders documents inside segments to help certain searches, but adds cost at indexing time. Use it only if you truly need it.
• Stronger compression of stored data saves disk space but costs CPU while writing. The default is faster to write.
• Waiting for extra shard copies before acknowledging adds delay. Keep the defaults unless you have a reason.
• Refreshing on every request by asking for it in the bulk call defeats the whole point of batching.

The cost of indexing one document is decided long before the bulk request is sent. It is decided by the shape of the document and the mapping that describes it.

A mapping is the blueprint that tells Elasticsearch what type each field has and how it should be stored and searched. Every field you index has a price: analysis, index structures, storage for sorting and grouping, and space for the original text. The more fields and the more structures per field, the more work per document. Cutting unnecessary work at this level speeds up every single request.

Dynamic mapping: convenient but costly

When a document contains a field that the index has never seen, Elasticsearch normally creates a mapping for it automatically. This is called dynamic mapping. It is convenient for exploring but has two costs during bulk loads. First, each new field requires an update to the cluster’s shared state, which is slow when it happens often. Second, the automatic guess may be wrong or wasteful, such as indexing every string as both text and keyword. For production loads, define the mapping first, using an index template, and set dynamic behavior to strict or to off for unknown fields if your data is well understood.

Document size and shape

Very large documents are slow to analyze and take large amounts of memory in each stage of the write path. If a document contains a huge blob of text or many repeated sections, consider whether all of it must be in Elasticsearch, or whether a reference to the full content stored elsewhere would do. Documents under a few kilobytes index most smoothly.

Work before the bulk request

Cleaning, joining, and enriching data can happen in two places: in your loading program before sending, or in an ingest pipeline inside Elasticsearch. Ingest pipelines are handy, and can be attached to a request, but they spend the cluster’s CPU on work that your own program could do on cheaper machines. If the cluster is your bottleneck, moving heavy transformations upstream is one of the most effective changes you can make.

Settings can only take you as far as the machinery allows. This chapter covers the ceiling.

Shards: the unit of parallel work

Documents are spread across the primary shards of an index. Each shard can accept several concurrent writers, but the number of shards also decides how the work spreads across nodes. An index with a single primary shard sends all its indexing to one node, no matter how big the cluster is. An index with primaries spread over all data nodes lets every node do its share.

Too many shards cause different trouble. Each shard has fixed overhead in memory, file handles, and cluster state, and a large number of tiny shards makes both indexing and searching less efficient. A common guideline is to keep shards in the range of tens of gigabytes for search-oriented data, and to avoid a huge number of shards per gigabyte of heap. For time-series data, rolling over to a new index when the current one reaches a target size keeps shards sensible without manual planning.

Hardware that helps indexing

Hot data on fast nodes

In time-based systems, the newest data is written to and searched the most, and the oldest data is mostly just stored. A layered design places the newest indices on the fastest nodes, the hot tier, and moves older indices to cheaper machines automatically. Indexing then always lands on hardware chosen for it, while cheaper hardware holds the archive.

Do the shard arithmetic before the load

Estimate the total size of the data, decide the target shard size, and divide. Then check that the result spreads evenly across your data nodes. If you have six data nodes and choose five primary shards, one node will do no primary work. Choosing six, or a multiple of six, gives every node an equal share. The number of primary shards cannot be changed on an existing index without a rebuild, so it pays to decide carefully.

A pipeline is only as fast as its slowest stage. Tuning the wrong stage changes nothing. The skill is knowing where to look.

Bulk indexing passes through many stages: your program, the network, the coordinating node, the write queue, the CPU that analyzes documents, the memory buffer, the translog, the disk, the merges, and the replicas. At any moment, one of these is the limit. Speeding up any other stage will not raise the total. The flowchart below is a simple way to find the limiting stage.

flowchart TD
    A["Throughput is too low"] --> B{"Are there 429 rejections?"}
    B -- Yes --> C["Reduce workers or bulk size and check queues and indexing pressure"]
    B -- No --> D{"Is data node CPU near full?"}
    D -- Yes --> E["Cut analysis cost: fewer fields, move pipelines upstream, add cores or nodes"]
    D -- No --> F{"Is the disk busy or indexing throttled?"}
    F -- Yes --> G["Improve disks, loosen refresh, check merges, add shards"]
    F -- No --> H{"Is the client the limit?"}
    H -- Yes --> I["Add workers, adjust batch size, compress, spread across nodes"]
    H -- No --> J["Check network, replicas, and uneven shard layout"]
    

Where the biggest wins usually are

Read the table from top to bottom. The easiest changes are also the most powerful, so try them first. Only when they are exhausted does it make sense to change the mapping, and only after that to pay for hardware.

A worked example

Imagine a company must load 100 million product records, each about one kilobyte, into a new index within two hours. That is about 100 gigabytes of source data. Two hours is 7,200 seconds, so the target rate is roughly 14,000 documents per second, or about 14 megabytes per second.

A quick test on one node with one shard shows that a single shard absorbs about 5,000 documents per second with the default settings, and about 9,000 with refresh switched off and no replicas. To reach 14,000 per second with a safety margin, the team wants at least three shards working at once. They choose six primary shards on six data nodes, so that every node carries one primary and load is even. They start with 8-megabyte batches, and add workers until rejections just begin, then back off one step. The invented numbers are only an illustration, but the method is real: measure one shard, multiply, add margin, and verify.

Scaling out

Indexing scales nearly linearly with the number of shards and nodes, as long as the load is spread evenly and clients send to all of them. Uneven spread is the usual reason scale-out disappoints: one hot shard, or one node receiving all client traffic, becomes the limit again. Watch the per-node numbers, not only the cluster totals.

The cost of the first minutes

Many loads look fast at the start and slow later. At the start the buffers are empty, segments are few, and the file cache is cold but idle. As data grows, merges become bigger, disks become busier, and throttling may begin. Judge a tuning change by sustained throughput over a long enough run, not by the first thirty seconds.

Every speed-up is a trade. Here is the honest balance sheet.

Which tuning fits which situation?

A fast load that silently drops data is worse than a slow one. Reliability is the other half of tuning.

Partial failure: the most important idea

A bulk request is not all-or-nothing. It is a bundle of separate operations, and each one can succeed or fail on its own. The response includes a flag saying whether any item failed and then a result for each item. If your program looks only at the overall HTTP status, it will miss failures, and documents will be quietly lost. Every serious loader reads the flag, walks through the items, and handles each failure by its type.

Sorting failures into kinds

A place for the failures: the dead-letter store

Documents that can never be indexed, such as those with wrong types, should not block the pipeline or vanish. A common design writes them to a separate store, often called a dead-letter queue, along with the error message. Someone then repairs the data or the mapping and resends them. Without this, a single bad record can stall a whole load or, worse, disappear without a trace.

Timeouts and the unknown

When a request times out, you do not know whether it succeeded. The cluster may still be working on it, or it may have finished after your program stopped listening. This is why retry safety matters. With your own document ids, resending is harmless because the same document overwrites itself. With generated ids, a resent batch that had in fact succeeded creates duplicates. If duplicates are unacceptable, either use your own ids, or deduplicate later, or use generous timeouts and careful checks.

When a node fails during a load

If a node holding a primary shard goes down, one of its replicas is promoted to be the new primary, and indexing continues. With no replicas, that shard’s data is gone, and the shard is unavailable until the node returns. Because a first-time load with zero replicas has no safety net, it should be designed to be restartable: keep track of how far the load got, so that it can resume or replay a range without starting from zero.

Cluster health colors

During loads, the cluster health may show yellow when replicas are missing or still being built, and red if a primary shard is unavailable. Yellow during a zero-replica load is expected. Red is never acceptable and should stop the load until it is understood.

A loading job is a powerful client: it writes a lot of data, quickly, with credentials. It deserves careful handling.

Least privilege for loaders

Give the loading program only the permissions it needs, and only on the indices it writes. In Elasticsearch’s security model, permission to add documents is granted per index or index pattern. A loader that only creates new documents does not need rights to delete indices or read other data. Using API keys created for a single job, with a limited lifetime, is safer than sharing a general-purpose administrator account. If the key leaks, the damage is limited to what that job could do.

Protect the connection

Bulk requests carry your data in text form. Always use encrypted connections between your loader and the cluster, and between nodes. Keep the cluster on a private network and expose only what the loaders and users need. Rate limiting and authentication at the load balancer add another layer.

Guard against accidental or hostile overload

• Request size limits: the cluster caps the size of an HTTP request. Keep this cap in place, so that one huge request cannot exhaust memory.
• Indexing pressure: the memory guard refuses work beyond its limit, which protects nodes from both mistakes and abuse.
• Field limits: a limit on the number of fields per index stops hostile or careless documents from creating thousands of new fields.
• Separate workloads: keep untrusted or experimental loads on their own indices, or even their own cluster, so that they cannot starve production.

Sensitive data in failures and logs

When a document fails, the error message and the failed document often end up in logs or a dead-letter store. If the data is personal or confidential, those copies become a leak risk. Restrict access to those stores, encrypt them, set retention limits, and avoid printing full document bodies in ordinary logs.

Ingest pipelines and scripts

Pipelines can run scripts and call other steps on every document. Restrict who can create or change them, because a bad pipeline changes every document that passes through, and a slow one becomes a denial-of-service against your own cluster.

You cannot tune what you cannot see. These are the signals to watch, and how to read them.

Elasticsearch exposes statistics for nodes and indices through its statistics endpoints, and the compact table views show several of them at a glance. Monitoring tools, including Elastic’s own, chart these over time. Keep charts of these numbers next to your throughput chart, so that a change in speed can be matched with a change in cause.

Reading the patterns

Logging that helps

Elasticsearch can record slow indexing operations in a special log, with thresholds you choose, which helps you find the documents or fields that cost the most. On the client side, log the size of each batch, the time it took, the number of failed items, and the reasons for failure. When a load misbehaves, those logs are the first place to look.

How to benchmark honestly

Good habits make loads smooth and repeatable. Bad habits are the source of most late-night incidents.

Patterns that work

Anti-patterns to avoid

A short list you could hand to a new teammate on their first day.

Best practices

Common mistakes

• Counting documents instead of bytes. A batch of 10,000 tiny documents and a batch of 10,000 huge ones are very different loads.
• Benchmarking with fake data. Synthetic documents that are too simple make every setting look better than it will be.
• Changing many things at once. When speed changes, you cannot tell which change did it.
• Sending everything to one node. The coordinating work and memory use of that node become the limit.
• Giving the heap too much memory. A huge heap starves the file cache and lengthens garbage-collection pauses.
• Letting dynamic mapping run wild. A stray field with thousands of names can slow the whole cluster.
• Using updates when inserts would do. A partial update still reads the old document and writes a whole new one, so it is more expensive than adding a fresh document.
• Forgetting that force merging is for finished data. Merging an index that still takes writes creates huge segments that are costly to keep tidy.

Myths versus facts

Where does bulk indexing tuning matter most?

Many well-known organizations have said publicly that Elasticsearch is part of their systems, including Wikipedia’s search, Netflix, Uber, and Stack Overflow. They rarely publish their exact loading settings, so the scenarios below are typical patterns for such workloads, not descriptions of anyone’s private configuration.

Short answers to the questions that come up most often.

If you skim only one part of this guide, make it this one.

Bulk indexing tuning is the discipline of getting large amounts of data into Elasticsearch quickly, safely, and repeatably. It starts with batching, continues with careful settings and a lean mapping, rests on well-laid-out shards and good hardware, and is checked at every step by measurement. The cluster will tell you when you are going too fast. Your job is to listen.