Start here. By the end of this chapter you will know what a reindex is, what the script adds, and why the two together are so useful.

An Elasticsearch index is a named collection of documents, much like a table in a database. The Reindex API is a built-in operation that reads documents from one index, called the source, and writes them into another index, called the destination. The source is not changed or deleted. It is simply read. The destination receives the copies.

A plain reindex makes faithful copies. A reindex with a script adds one more step in the middle. For every document that passes through, Elasticsearch runs your script, a short program written in a safe, purpose-built language called Painless. The script receives the document and may change it in almost any way before it is written to the destination. It can also decide that a document should not be copied at all, or should be copied under a different identifier, or into a different index.

Read, change, write

The easiest way to remember the idea is three verbs. Elasticsearch reads a batch of documents from the source. Your script changes each document in the batch. Elasticsearch writes the batch to the destination. Then it repeats, batch after batch, until the source has been fully read. The important part is the middle verb. Without the script, you get a copy. With it, you get a transformation.

Where you meet it

You use a scripted reindex whenever the data you have is right in substance but wrong in shape. You may need to fix a field type, split one field into two, convert units, remove personal details, merge several indices into one, or divide one large index into many. It is one of the most common maintenance jobs in real Elasticsearch systems, and it is a normal part of every major migration and schema change.

A short history

Early Elasticsearch users had to reindex by writing their own programs: read every document out with the scroll interface, transform it in application code, and send it back with bulk requests. Everybody wrote the same glue code, and everybody made the same mistakes. A built-in Reindex API arrived in the 2.x series, in 2016, and it did the reading, the writing, the retrying and the progress tracking for you, using the task system to run long jobs in the background. The ability to run a script on each document came with it. Over the years the feature gained sliced parallel execution, throttling, reindexing from remote clusters, and a safe scripting language: the older scripting engines were replaced by Painless, which became the default in the 5.x series and is designed to be fast and sandboxed. Today a scripted reindex is a mature, well-understood tool.

Data is rarely perfect on the day it is stored. To see why a scripted reindex is needed, list the reasons the shape of data must change, and see why simpler tools fall short.

Reason one: some things cannot be changed in place

Elasticsearch stores documents in a structure called a Lucene index, which is built once and is not rewritten field by field. Once a field has been created with a type, such as text or number, that type is fixed for the life of the index. The analyzers that break text into words are fixed too. So is the number of primary shards, which are the pieces an index is split into. If any of these choices turns out to be wrong, the only way forward is to build a new index with the right choices and move the documents into it.

Reason two: requirements change

The developer who stored the price as text was not careless. The requirement at the time was “show the price.” Two years later the requirement is “add up the prices.” Products evolve, teams reorganize, and data that was only displayed becomes data that is analyzed. Each new requirement may need a new shape.

Reason three: the data itself is untidy

Real data is messy. The same country appears as “US,” “USA,” “U.S.” and “United States.” Dates arrive in three formats. Numbers arrive as text. A boolean is spelled “yes,” “Y” and “1.” Names are stored in one field when you now need first and last names apart. Cleaning this at the moment of copying is far easier than living with it forever.

Reason four: privacy and safety

Sometimes a field must not survive into the next index: a national identifier collected by mistake, a free-text comment that turned out to contain phone numbers, a field that a new law says you may no longer keep. A scripted reindex is a practical way to produce a clean copy without the sensitive parts.

Reason five: restructuring

Indices need to be merged, split and reorganized. Ten small monthly indices become one yearly index. One huge index that mixes many customers is divided into one index per customer. Old documents from an outdated design are folded into a new design. A script can decide, document by document, where each one belongs.

Reason six: migrations and upgrades

Moving data from an old cluster to a new one, or from an index created by an old version to a modern one, often involves reindexing. Major upgrades in particular can require that indices created long ago are rebuilt so that they follow current rules.

Why simpler tools are not enough

Note that a scripted reindex is not always the right answer. Some jobs are better done in place, at write time, or at search time. Chapter 13 compares the alternatives so you can choose the lightest tool that solves your problem.

A dozen words unlock the whole feature. Each one is explained with what it is, why it exists, where you meet it, and a picture from everyday life.

Source and destination

The source is what you read from: one index, several indices, a data stream or an alias, optionally narrowed by a query so that only some documents are chosen. The destination is where the copies are written. Source and destination must be different, because reading from and writing to the same place at the same time would be a recipe for confusion. The source is the old house; the destination is the new one.

The script

The script is the small program that runs once for each document. It is written in Painless, a language made for Elasticsearch that is fast, safe and easy to read. Painless is deliberately limited: it cannot open files, use the network or run arbitrary Java code, so a script cannot damage the machine it runs on. The script is the sorter at the truck door.

The context object

The script does not see the whole cluster. It receives one object, called the context and written as ctx, that holds the document currently passing through and a few controls: the document’s fields, its identifier, its destination index, its routing value, its version, and an operation setting that says what to do with it. Reading and changing this object is all a reindex script does.

Parameters

A script can be given a set of named values, called params, alongside its code. Parameters let you keep the code fixed while the values vary, for example a conversion rate or a list of fields to remove. They also help Elasticsearch to reuse a compiled script instead of building a new one for each value. Params are the settings dial on the sorter’s instruction sheet.

The operation

By default, every document is indexed into the destination. The script can change that by setting the operation to “noop,” which means “skip this document,” or “delete,” which means “remove this document from the destination if it exists.” This turns the script into a filter as well as a transformer.

The batch

Documents are not copied one by one. They travel in batches, by default a thousand at a time. A batch is read, run through the script, and written with a single bulk request. Batching is what makes the copy fast.

Conflicts, operation type and version type

If the destination already holds a document with the same identifier, there is a conflict to resolve. The op type says whether to overwrite (“index”) or to create only if missing (“create”). The version type says how document versions are compared. Together with a setting called conflicts, they decide whether a clash stops the whole job or is counted and skipped. They make a reindex safe to repeat.

Slices

A reindex can be cut into slices, independent parts that run in parallel, like several movers carrying boxes at once. More slices can mean a faster copy, up to what the cluster can handle.

Throttling

A big reindex can compete with normal traffic. Throttling limits how fast it goes, so that customers’ searches are not slowed. It is a speed limit for the movers.

The task

A large reindex may run for hours, so it runs as a background task. You start it, receive a task identifier, and can check its progress, slow it down, speed it up or cancel it, all without keeping a connection open.

The alias

An alias is a nickname that points to one or more indices. Applications search the alias, not the real index name. When the reindex is finished and verified, you move the alias from the old index to the new one in a single atomic step, and the switch is invisible to users. The alias is the street address that stays the same when you move.

Now let us zoom out and look at the machinery: which parts take part in a scripted reindex, where each one lives, and how they connect.

The Reindex API is not a separate program or plugin. It is a built-in feature of every Elasticsearch node. When you start one, a node takes charge of the job, reads from the source, runs your script, and writes to the destination. The diagram below shows the whole path, including the optional pieces.

flowchart LR
    CLIENT["Client or Kibana Dev Tools"] --> TASK["Reindex task on a coordinating node"]
    TASK --> SRC["Source index shards"]
    SRC -->|"batch of documents"| SCR["Script engine - Painless runs once per document"]
    SCR --> BULK["Bulk request builder"]
    BULK --> PIPE["Optional ingest pipeline"]
    PIPE --> DST["Destination index shards"]
    TASK --> REC["Task record - progress and final results"]
    DST --> ALIAS["Alias moved after verification"]
    subgraph REMOTE["Optional remote cluster"]
        RSRC["Remote source index"]
    end
    RSRC -.->|"read over the network"| TASK

Walking through the diagram

You start the job from a client, such as an application, a command-line tool or the Dev Tools console in Kibana. The request lands on one node, which becomes the owner of a reindex task. That task asks the shards of the source index for a batch of documents. Each document in the batch is handed to the script engine, which runs your Painless script and returns the changed document together with an operation. The task then packs the results into one bulk request and sends it to the destination index. If the destination has an ingest pipeline attached, the pipeline runs on each document as it is indexed, after the script. Meanwhile, the task keeps a running record of its progress, which you can read at any time. When the job is finished and checked, you move an alias from the old index to the new one.

The dotted box on the right shows the optional case where the source lives in a different cluster. The task then reads the documents across the network, runs the script locally and writes into its own cluster. This is the usual way to move data between clusters, or between very different versions.

The components in detail

What is not copied

A reindex copies documents. It does not copy the settings or mappings of the source index. If you do not create the destination first, Elasticsearch will create it on the fly and guess the mappings from the data, which is rarely what you want. It also does not copy aliases, lifecycle policies, or other index-level configuration. This is why the standard advice is: create the destination index yourself, with explicit mappings and settings, before you reindex.

This is the engine room. You do not need to be an expert, because every idea comes with a picture.

sequenceDiagram
    participant C as Client
    participant T as Reindex task
    participant S as Source shards
    participant P as Painless script
    participant D as Destination shards
    C->>T: Start reindex with a script
    T->>S: Open a snapshot and read the first batch
    S-->>T: Batch of documents
    T->>P: Run the script on each document
    P-->>T: Changed documents and operations
    T->>D: Send one bulk request
    D-->>T: Bulk results with any failures
    T->>S: Ask for the next batch
    T-->>C: Final report with counts

Step one: take a snapshot of the source

When the job starts, Elasticsearch opens a consistent view of the source index as it is at that moment. This is like taking a photograph of a shelf before you start moving the items. Documents that are added to the source after the photograph are not part of the job, and changes to documents already photographed are not seen either. This gives the copy a clear meaning: it is the state of the source at the instant the job began. It also means that if applications keep writing to the source during a long reindex, you will need a plan to catch up afterward, which later chapters explain.

Step two: read a batch

The task asks for a batch of documents, by default one thousand, using the same scrolling mechanism that Elasticsearch offers for reading large result sets. Each result carries the document’s identifier, its stored original content, its routing value and its version. If you gave a query, only matching documents are returned. If you asked for only some fields, only those fields are returned, which saves network and memory.

Step three: run the script on each document

For every document in the batch, the script engine executes your Painless script. The script is compiled once into fast code and then run over and over, once per document. It receives the context object holding that document. Whatever the script does to the context, such as changing a field, removing one, setting a new identifier or setting the operation to “noop,” becomes the instruction for what to write. Because the script runs in a sandbox, it can only work with the data in front of it. Its cost is small for simple logic, but it does add processor work for every document, which matters at large scale.

Step four: build and send a bulk request

The task turns the batch into a single bulk request: one network call that carries a thousand indexing instructions, plus any deletions the script asked for. Bulk requests are far more efficient than sending documents one at a time. Documents whose operation was “noop” are simply left out.

Step five: handle the answer

The destination reports the result for each document. Most succeed. A few things can go wrong, and the task handles them in different ways. If the destination is temporarily too busy and rejects some requests, the task retries them with a growing pause, up to a limit. If a document clashes with one that already exists and you told the job to continue in case of conflicts, it counts the clash and moves on. If a document is rejected for a real reason, for example because its content does not fit the destination’s mapping, the failure is recorded and, by default, the job stops so that you can investigate.

Step six: repeat and finish

The task asks for the next batch, and the cycle repeats until the source is exhausted. At the end it clears the snapshot, optionally refreshes the destination so the new documents become searchable, and prepares a final report with all the counts.

How identifiers and versions travel

Each document normally keeps its identifier from the source, which is what makes a repeated reindex safe: the same document lands on the same identifier again. Each document also carries a version number. By default, the destination assigns its own versions and ignores the source’s, so the copy behaves like a fresh document. If you choose an external version type, the source versions are used and compared, so that an older copy can never overwrite a newer one. The script can change the identifier, and it can even clear the version, and those powers are useful but carry risk, as Chapters 8 and 11 explain.

A reindex is a short-lived job with a clear life: it is submitted, runs, sometimes waits, and ends in one of three ways. Knowing each stage helps you read its reports.

stateDiagram-v2
    [*] --> Submitted
    Submitted --> Running : task starts
    Running --> Throttled : rate limit wait
    Throttled --> Running : wait is over
    Running --> Completed : all documents processed
    Running --> Cancelled : cancel request
    Running --> Failed : unrecoverable error
    Completed --> [*]
    Cancelled --> [*]
    Failed --> [*]

Waiting or not waiting

By default, a reindex request waits until the whole job finishes and then returns the report. That is fine for a few thousand documents, but for millions the connection would have to stay open for hours. For big jobs, you ask Elasticsearch to run the reindex in the background. It replies at once with a task identifier, and you use that identifier to check progress, change the speed, cancel, or read the final result later. Think of the difference between standing at the counter until your food is ready, and taking a pager that buzzes when it is done.

Throttling and pauses

If you set a speed limit, the job deliberately pauses between batches so that it never exceeds the rate you chose. These pauses are the “throttled” state in the diagram. The limit can be changed while the job is running, which is very handy: slow it down during the busy afternoon, and speed it up overnight.

The three endings

A job that reads the last batch and writes it successfully is completed. A job that you stop is cancelled. A job that meets an error it cannot recover from is failed. In the last two cases, the documents already written to the destination stay there. Reindexing is not a transaction that rolls back. This is important: a cancelled or failed job leaves a partial destination, which is why the safe pattern is to reindex into a new index that nobody uses yet, and only switch to it after the job has completed and been checked.

Reading the final report

When a job ends, it reports a set of counters. They tell you exactly what happened and allow you to check that nothing was lost.

In a clean run, the total is roughly the sum of created, updated, deleted, noops and conflicts. Comparing these numbers with what you expect is the first and best check. If you told the script to skip documents with no email address and you know that about two percent of them lack one, then a noop count near two percent of the total tells you the script is doing what you meant.

Before going deeper, here is the route you follow for a first scripted reindex. Knowing the road makes the later chapters easier to follow.

The technical requests are short. Most of the work is thinking and checking. Here is the journey for a beginner who needs to turn a text price like “19.99 USD” into a numeric amount and a separate currency, across a large index of products.

Before you begin

Check four things. First, you can read the source and write the destination, and you have permission to create the destination index if it does not yet exist. Second, the source keeps its stored original documents, so that the fields you need are really there. Third, the cluster has spare disk space: the destination will hold a full new copy, so plan for roughly the size of the source, while both exist. Fourth, you have a plan for what happens if applications keep writing to the source during the copy.

A reindex script sees exactly one thing: the context object. Knowing what is inside it, and which parts are safe to change, is the key to writing good scripts.

The context, written ctx, is like a parcel that arrives at the sorter’s desk. The parcel has contents, which are the document’s fields, and a label on the outside, which holds its address and handling instructions. The script may read both, change some of them, and then send the parcel on. Here is what is on the parcel.

The content: your everyday workspace

Most scripts spend their whole life inside the source content. This is a collection of field names and values, and the script treats it like a small notebook. It can look up a field, write a new one, cross out an old one, or replace a value with a better one. Nested objects and lists inside the document are part of the notebook too. The rule of thumb is simple: whatever the notebook looks like when the script finishes is what is written to the destination.

The identifier: careful, this is the name tag

By default the document arrives with its old identifier and leaves with the same one. A script may replace it, for instance by combining a customer number and an order number into one new identifier. This is powerful, but there is a trap. If two different source documents end up with the same new identifier, the second overwrites the first in the destination, and no error is raised. The document count in the destination will quietly be lower than expected. The counters in the report can reveal this: many “updated” documents in a job that should only create new ones is a red flag.

The index: sending documents to different places

A script may change the destination index name for a single document. This turns one reindex into a sorting machine: documents from one large index can be fanned out to many smaller ones, for instance one per country or one per month. Each target index must already exist with the right mapping, because otherwise Elasticsearch may create it on the fly with guessed mappings, or refuse if automatic creation is switched off.

The routing: for advanced layouts

Routing lets you keep related documents on the same shard, which can speed up searches that always filter by that value. A script can set the routing, for example to a customer identifier. Change it only if your destination design calls for it, and remember that searches must then use the same routing to benefit.

The version: a delicate switch

The version arrives from the source. Whether the destination cares depends on the version type you chose for the job. A script can clear it, which is equivalent to saying “ignore the version entirely and just write this.” That is helpful when you deliberately want to overwrite. It is dangerous when you forgot that it disables the protection against overwriting newer data with older data.

The operation: your decision about the document

Every document has an operation, which is “index” unless the script changes it. Setting it to “noop” tells Elasticsearch not to write this document at all. Setting it to “delete” tells Elasticsearch to remove the document with that identifier from the destination. Any other value is rejected. The operation is the sorter’s yes, no or discard.

Here are the most common things people do with a reindex script, each described in words so that you can see the idea before you meet any code.

Convert a type or a format

This is the classic case, and the one from the start of this article. The script looks at the old field, for example a text price, turns it into a proper number, and writes it to the new field. It usually also removes the old field or leaves it behind, according to your plan. The same recipe applies to dates written in odd formats, to numbers stored as text, and to booleans spelled as “yes” and “no.” The main care point is what to do when the value cannot be converted. A wise script decides in advance: skip the document, use a default, or store the problem in a separate field for later review.

Rename a field

The script reads the value under the old name, writes it under the new name, and removes the old one. There is no built-in rename, so this is simply a move. The script should first check that the old field exists, otherwise it might create an empty new field for every document that lacked it.

Split one field into several

A full name becomes a first name and a last name. A price with a currency becomes an amount and a code. An address line becomes street, city and postal code. The script finds the separator, cuts the text at that place, and stores the parts. Good scripts prepare for surprises: names with several words, missing separators, or extra spaces.

Merge several fields into one

The opposite of splitting. Separate latitude and longitude fields become a single location object. A day, a month and a year become one date. A first and last name become a display name. The script joins the values in the right format and then removes the old parts if they are no longer needed.

Compute a new field

The script may compute something new from existing values: total price from quantity and unit price, an age from a birth date, a category from a list of rules, or a flag such as “is large order.” Precomputing at copy time makes later searches faster and simpler.

Fill in defaults and normalize values

Missing values may be replaced by defaults. Variants of the same thing may be brought to one standard form: “US,” “U.S.” and “United States” all become “US.” The script can read a lookup table given as a parameter, so that the mapping of variants to standard forms is kept outside the code and is easy to change.

Skip documents that should not be copied

If a document is empty, corrupt, a duplicate, or simply no longer needed, the script sets the operation to “noop.” The report counts these, so you can check that the number matches your expectation.

Delete from the destination

Sometimes the destination already holds documents that should no longer exist. Setting the operation to “delete” for a matching document removes it. This is a specialized use, but useful when reindexing into an existing index that is being cleaned.

Route documents to different indices

By setting the destination index per document, one source can feed many targets: one index per tenant, per country, per month, or per document type. The script reads a field, builds the target name from it, and sets it. Every target must exist and have the right mapping.

Create a new identifier

To make documents unique by a business key, the script builds an identifier from several fields, such as customer number plus order number. Done well, this makes repeated reindexing safe and prevents duplicates. Done badly, it makes documents overwrite each other.

Remove or mask sensitive parts

The script can remove fields, or replace them with a masked form, for instance keeping only the last four digits of a phone number. Combined with a field filter at the source, it produces a clean copy for a test system or a partner.

The recipes at a glance

You do not need to be a programmer to write a good reindex script, but you do need to be careful. This chapter gives the ideas that prevent most script failures.

What Painless is

Painless is the scripting language built into Elasticsearch. Its rules look familiar to anyone who has seen Java or similar languages, but it is smaller and safer. It runs inside a sandbox, so it cannot read files, reach the network or call arbitrary code. It is fast, because scripts are compiled to efficient instructions the first time they are used. It has a few basic kinds of values: text, whole numbers, decimal numbers, true-or-false values, lists and maps, which are collections of names and values. That is nearly everything a reindex script needs.

Rule one: never assume a field exists

Real data is uneven. Some documents lack a field entirely. Others have it, but empty. A script that reads a missing field and tries to use it will meet a “null” value and stop with an error. The single most valuable habit is to check first: does the field exist, and does it hold something? Only then use it. Decide what to do otherwise: skip the document, or use a default.

Rule two: never assume a type

The same field may hold a number in one document and text in another, or a single value in one and a list in another. Numbers in the original JSON might come through as a whole number in one document and a decimal in the next. Good scripts check the kind of value, or convert carefully, before doing arithmetic or text handling on it. It is like checking whether a parcel holds a book or a lamp before deciding which shelf it goes on.

Rule three: keep scripts small, pure and repeatable

A reindex script should decide everything from the document in front of it and its parameters. It should not depend on the time of day, on random numbers, or on what earlier documents looked like. Such scripts are deterministic: running the job twice produces the same result. That matters for safe re-runs, and for finding mistakes. A script that stamps “the current time” into each document gives a different result on every run, which is sometimes what you want and sometimes a nasty surprise.

Rule four: use parameters, not hard-coded values

Every unique script text has to be compiled before it runs, and Elasticsearch limits how many new scripts may be compiled in a short period, to protect the cluster. If you build a different script text for every value, for example by pasting a different rate into the code each time, you may hit that limit and be refused. Passing changing values as parameters keeps the script text the same, so it is compiled once and reused. Parameters also make scripts easier to read and to test.

Rule five: keep heavy work out

The script runs for every document. A tiny amount of extra work per document becomes a huge amount of total work across ninety million documents. Avoid long loops, complicated pattern matching and repeated searching inside big lists. Regular expressions in particular are restricted by default in many setups, because they can be slow, so use simple text operations, such as finding a separator and cutting, whenever they are enough. If your logic is really heavy, consider doing it in an external program instead.

Stored scripts

Instead of pasting the script into every request, you can save it in the cluster under a name and refer to it by that name. This keeps a single trusted version, avoids copy-and-paste mistakes, and lets teams review scripts like any other piece of code.

Testing your logic

Never run a new script on the whole data set first. Test in layers. First, think through the logic on paper with three or four typical documents and three or four ugly ones. Second, use small test facilities that let you run a snippet of Painless against sample values. Third, run the real reindex on a small sample into a scratch index and read the results by eye. Fourth, run a slightly larger sample, and compare the counters with your expectations. The cost of these steps is small, and the cost of finding an error after ninety million documents is large.

Big jobs fail halfway, get interrupted, or need to be repeated. This chapter shows how to make a scripted reindex safe to run again, and what to do when the source keeps changing.

What happens when the document already exists?

If the destination is empty, every document is simply created. But you may run the job again after an interruption, or reindex into an index that already has some documents. Then each incoming document may meet an existing one with the same identifier. Elasticsearch has clear rules for what to do, and you choose them with three settings: the operation type, the version type, and the conflict policy.

flowchart TD
    A["Document leaves the script"] --> B{"Same identifier already in destination?"}
    B -->|"No"| C["Created in destination"]
    B -->|"Yes"| D{"Operation type is create?"}
    D -->|"Yes"| E["Version conflict recorded and document skipped"]
    D -->|"No"| F{"Version type is external?"}
    F -->|"No"| G["Existing document overwritten"]
    F -->|"Yes"| H{"Incoming version higher than existing?"}
    H -->|"Yes"| G
    H -->|"No"| E
    E --> I{"Conflict policy is proceed?"}
    I -->|"Yes"| J["Conflict counted and job continues"]
    I -->|"No"| K["Job stops with a failure"]

Operation type: overwrite or create only

With the default operation type, “index,” an incoming document replaces any existing one with the same identifier. With “create,” the document is written only if no document with that identifier exists, otherwise a conflict is raised. “Create” is the safe choice for a re-run that should only fill gaps and never overwrite. It is also required when the destination is a data stream.

Version type: who wins when both exist?

By default, versions are handled internally by the destination, and an incoming document simply overwrites. With an external version type, the version from the source is compared with the version in the destination, and the write succeeds only if the incoming version is higher. This protects a newer document in the destination from being replaced by an older one from the source. Variants exist that also allow equal versions. External versioning is useful for “catch-up” runs, described below.

Conflict policy: stop or carry on?

By default a conflict stops the job, on the theory that something unexpected has happened. If you know that conflicts are normal, because you are re-running a job or using “create,” you can tell the job to proceed. It then counts the conflicts in the report and continues. Use this deliberately, and read the count at the end.

The moving source problem

Remember that the job works from a photograph taken at the start. If applications keep writing to the source while a long reindex runs, the destination will lack anything that changed after the photograph. There are four common ways to deal with this.

Deletions deserve special attention. A document deleted from the source after the photograph will still be copied, and a catch-up pass based on timestamps will not notice that it is gone. If deletions matter, use a soft-delete flag in your data, or run a comparison of identifiers at the end.

Making scripts and jobs repeatable

A job is safe to repeat when three things hold: the identifiers are stable, the script is deterministic, and the settings say what should happen on a clash. If all three are true, you can run the same request twice and end up in the same state, which is called being idempotent. It is like a light switch marked “on”: pressing it twice leaves the light on, unlike a toggle, which changes on every press.

The success of a reindex is often decided before it starts. A carefully prepared destination turns a risky job into a routine one.

Design the mapping on purpose

The mapping is the blueprint of an index: which fields exist and what type each one has. When you reindex, the mapping of the destination decides how your transformed documents are understood. If you skip this step and let Elasticsearch guess, it will look at the first documents and choose types by intuition: text for anything in quotes, numbers for digits. It may guess wrong, and once it has guessed, it cannot change its mind. So create the destination index yourself, with the exact mapping that you want. For your converted price field, that means a number type, and for the currency, a keyword type.

Use strict mapping as a safety net

Mappings can be set to reject documents that contain fields you did not declare. This is the strict setting. In a reindex, it acts as a quality check: if your script accidentally leaves an old field behind, or creates a misspelled new one, the destination refuses the document and the report lists the failure. That is annoying for a moment and very valuable overall, because it catches mistakes early on a small sample instead of late in a huge job.

Settings for a fast load

A destination that is being filled in bulk should be configured for speed and restored afterward. Two settings matter most. Setting the number of replicas to zero during the load avoids writing every document several times. Setting the refresh interval to a very long value, or switching it off, avoids the constant work of making new documents searchable during the load. When the job is done, you restore replicas to their normal number, which makes Elasticsearch build the copies in one efficient pass, and restore the refresh interval. Remember to do this: an index left with no replicas is not protected against the loss of a node.

Choose the shard count with care

A reindex is the chance to fix the number of primary shards. Too many small shards waste resources, and too few huge shards are slow to move and recover. A commonly used guide is to aim for shards in the range of tens of gigabytes, but the best value depends on your workload. Decide the number now, because changing it later will need another reindex or a special operation.

Templates, lifecycle and data streams

If your destination names follow a pattern, an index template can apply the right mappings and settings automatically to every matching new index. This is especially handy when the script fans documents out to many indices, such as one per month, since each new target then arrives with the correct blueprint. If the destination is a data stream, which is the modern structure for time-based data, the reindex must use the “create” operation type, because data streams accept only new documents, and each document needs a timestamp field. Lifecycle policies attached to the destination can move data through hot, warm and cold stages, but avoid letting a policy act on an index while it is still being filled.

The alias swap for zero downtime

Applications should never use real index names. They should use an alias, and real indices should carry a version in their names, such as “products-v1” and “products-v2.” Then a migration looks like this: applications search the alias, which points to the old index. You build and fill the new index. When it is verified, one atomic operation removes the alias from the old index and adds it to the new one, and every application switches at the same instant, without noticing. If something is wrong, the same operation in reverse switches back.

flowchart LR
    APP["Applications search the alias products"] --> AL["Alias products"]
    AL --> V1["Old index products-v1"]
    V1 -->|"reindex with script"| V2["New index products-v2"]
    AL -.->|"atomic switch after verification"| V2
    V1 -.-> KEEP["Kept for a while for rollback, then deleted"]

Writers need the alias too

Do not forget the applications that write. If writers use the real name of the old index, they will keep adding documents to it after you have switched, and those documents will be lost from the new view. Give writers a write alias, and include the switch of the write alias in your cutover plan, together with the catch-up pass described earlier.

A scripted reindex is a strong tool but not always the lightest one. Here is how to choose the simplest method that solves your problem.

flowchart TD
    Q["What do you need to change?"] --> A{"Only new incoming data?"}
    A -->|"Yes"| P["Ingest pipeline at write time"]
    A -->|"No"| B{"Mapping, analyzer or shard layout must change?"}
    B -->|"Yes"| S{"Need per-document logic?"}
    S -->|"Yes"| RS["Reindex with a script"]
    S -->|"No"| RP["Plain reindex, maybe with a pipeline"]
    B -->|"No"| C{"Only values inside the same index?"}
    C -->|"Yes"| U["Update by query in place"]
    C -->|"No"| D{"Only a different view at search time?"}
    D -->|"Yes"| RT["Runtime fields or field aliases"]
    D -->|"No"| E["External processing tool for very heavy logic"]

Update by query: fix documents in place

The update-by-query operation runs a script on every document that matches a query, and rewrites it in the same index. It uses the same context object and a similar script. It is perfect when only values need fixing and the mapping is already right, for example correcting a wrong code in a million documents. It cannot change the mapping, the shard count or the analyzer, and it cannot move documents to other indices or change their identifiers. It also rewrites documents in place, so there is no clean copy to fall back on unless you have a snapshot.

Ingest pipelines: fix data as it arrives

An ingest pipeline is a chain of processing steps applied to each document at the moment it is indexed. It has ready-made steps for renaming, converting, splitting, removing and enriching, and a script step for custom logic. Pipelines are ideal for new data. They can also be attached to a reindex, so that your script and a pipeline work together: the script runs first, then the pipeline runs as each document is written. Use a pipeline when the same fix should apply forever, and a script when the fix is a one-time migration.

Runtime fields and field aliases: change the view, not the data

A runtime field computes a value at search time from other fields, so you can offer a “price as a number” without touching stored data. A field alias gives an existing field a second name. Both need no reindex at all. The price is slower searches, since the calculation repeats on every query, and some limits on what can be done. They are wonderful for experiments and small needs, and a scripted reindex is the way to make the change permanent and fast.

Split, shrink and clone: change shard counts only

If the only thing you need is a different number of primary shards, dedicated operations can split an index into more shards or shrink it into fewer. They work at the file level and are much faster than reindexing, but they have rules about the allowed numbers and they do not change any content.

External processing tools

If your transformation needs data from other systems, complicated business rules, or heavy computing, a script inside Elasticsearch may not be the right place. A separate tool that reads, transforms and writes can do more, at the cost of more moving parts.

What happens when the index has a hundred million documents, or a billion? This chapter explains the levers for speed and the safety nets for long jobs.

What decides the speed

Four things set the pace of a scripted reindex. The first is reading from the source: how fast the source shards can serve batches. The second is the script: how much work it does per document. The third is writing: how fast the destination can index, which depends on the number of shards, the storage and the settings. The fourth is the network, especially for remote reads. The slowest of these decides the overall speed, so measure before you tune.

Batch size

The number of documents per batch, by default a thousand, sets how big each bulk request is. Very small batches waste effort on overhead. Very large batches use a lot of memory and may be rejected. A common guide is to aim for bulk requests of a few megabytes each, so if your documents are large, reduce the batch size, and if they are tiny, you may increase it. Try a few values on a sample and measure.

Slices: parallel movers

A single stream of batches is limited by one worker. Slicing divides the job into several independent parts that run at the same time, each reading a distinct subset of the source. You may name a number or ask Elasticsearch to choose automatically, in which case it picks a sensible value based on the number of source shards. Using about as many slices as the source has shards usually gives the best speed. More than that gives no benefit and adds overhead. Each slice appears as its own child task with its own counters, and the parent task shows the total.

flowchart TD
    JOB["Reindex job with four slices"] --> S1["Slice 1 - child task"]
    JOB --> S2["Slice 2 - child task"]
    JOB --> S3["Slice 3 - child task"]
    JOB --> S4["Slice 4 - child task"]
    S1 --> D["Destination index"]
    S2 --> D
    S3 --> D
    S4 --> D

A simple calculation shows the value. Suppose the source holds 120 million documents and one stream copies 15,000 documents per second. That takes 8,000 seconds, a little over two hours. If four slices each manage 12,000 documents per second, the total is 48,000 per second, and the job finishes in about 2,500 seconds, or roughly forty minutes. The gain is real, but only if the cluster has spare capacity: slicing on a busy cluster simply moves the queue.

Throttling: a speed limit for good manners

To protect users, you can limit how fast the job goes. The setting behaves roughly like documents per second, and the job waits between batches to stay under it. With slices, the limit is shared among them. You can change the limit while the job runs, using the rethrottle operation, which means you can go gently in the afternoon and quickly at night.

Do less work per document

Reduce what travels. If you need only some fields, ask the source for only those. Keep scripts simple. Avoid extra pipelines during the load unless you need them. If a heavy step can be done later, for example building a derived field, leave it until the bulk copy is done.

Reliability: what if it fails halfway?

A reindex is not a transaction and it has no built-in checkpoint. If a node fails, if you cancel, or if an error stops the job, the documents already written stay in the destination, and the job cannot simply resume where it stopped. Three habits make this manageable. First, design the job to be repeatable: stable identifiers, deterministic scripts, and the “create” operation type with conflicts set to proceed, so that a second run fills only the gaps. Second, cut the job into pieces, for example one run per month of data using a query, so that a failure costs one piece and not the whole. Third, always write into a fresh index, so a failed job never damages anything that users rely on.

Temporary trouble is handled for you. If the destination rejects requests because it is too busy, the job retries with growing pauses. Watch the retry counters: many retries mean you are pushing too hard, and a lower speed limit will finish sooner than a job that fights the cluster.

Resources that run quietly in the background

While the job runs, the source index must keep its photograph alive, which stops old files from being cleaned up. Disk use on the source may therefore grow, particularly if the source is busy with updates. And the destination grows as it fills. Check disk space on both. If a node crosses its disk safety threshold, Elasticsearch may make indices read-only, and the job will fail with a clear message.

Reindexing from another cluster

The source can also be a remote cluster. You provide its address and credentials, and the destination cluster reads across the network. An administrator must first allow the remote address in the destination cluster’s settings, which is a safety rule that prevents casual use. Remote reindexing is very useful for moving data between clusters and even between quite different versions. It has limits: it does not support automatic slicing, very large batches may be rejected because the reading buffer is limited, and the network becomes another possible bottleneck. Many teams cut remote jobs into pieces by query for that reason. For copying whole indices unchanged between clusters, snapshots or replication may be simpler; a scripted remote reindex earns its place when the data must be reshaped on the way.

No tool is perfect. Knowing where a scripted reindex shines and where it struggles helps you use it wisely.

Trade-off one: safety versus space

The safest method builds a whole new index and keeps the old one until you are sure. That doubles the storage for a while. Reworking data in place uses less space but leaves no clean copy to return to. When the data matters, pay for the space, and make a snapshot too.

Trade-off two: speed versus disturbance

Running the job at full speed finishes sooner but may slow users. Throttling protects users but lengthens the job, and a longer job means a longer catch-up. A good compromise is to run fast in quiet hours, slow in busy ones, and use rethrottling to move between the two.

Trade-off three: flexibility versus risk

A script can do almost anything to a document, which is exactly why it can do harm. Every extra power, such as changing identifiers or routing, brings a new way to go wrong. Give the script only the powers that the task needs, and test on samples.

Trade-off four: one big job versus many small ones

One large job is simple to start and monitor but costly to repeat. Many small jobs are easier to recover but take more organization. For very large data, pieces cut by time range or by another natural key are usually worth the extra bookkeeping.

These tools are teammates, not rivals. Many real migrations use all three: a scripted reindex to rebuild the history, an ingest pipeline to keep new data clean, and a short update by query to correct the few stragglers.

A reindex reads your data, changes it, and writes a new copy. Every one of those steps has security consequences, and a well-run migration takes them seriously.

Who may run a reindex

A reindex uses the permissions of the person or service that starts it. That identity needs permission to read the source index and to write to the destination. If the destination does not exist and you expect it to be created, the identity also needs permission to create indices. If the script deletes documents from the destination, it needs the delete permission as well. Keep these powers narrow: a dedicated migration role, assigned for the duration of the job and removed afterwards, is safer than running everything as an all-powerful administrator.

Field and document restrictions can silently shrink the copy

Elasticsearch can restrict what a user sees by hiding certain documents or fields. A reindex run by such a user copies only what that user can see. The destination will then lack the hidden documents or fields, without any error. This is helpful when you deliberately want a restricted copy, for instance for a partner, and dangerous when you expected a complete one. Run full migrations with an identity that has full read access to the data, and run restricted copies with the restricted identity on purpose.

The script sandbox

Painless was designed to be safe. It cannot touch files or the network, and administrators can go further and limit which kinds of scripts are allowed, for example allowing only stored scripts that have been reviewed. Treat scripts like code: review them, keep them in version control, and avoid letting untrusted users submit their own scripts to a production cluster.

Remote sources and credentials

When reading from another cluster, the request contains the address and credentials of that cluster. Use encrypted connections, use an account that can only read the needed indices, and make sure the request is not written to logs or shared in tickets. The destination cluster only reads from remote addresses that an administrator has explicitly allowed, and that rule is a useful check: it prevents a stolen credential from turning your cluster into a data pump to anywhere.

Privacy: making the new index safer than the old one

A scripted reindex is a practical way to remove or mask sensitive data while producing a copy: dropping fields, replacing values with masked forms, or generalizing precise values such as exact locations or birth dates. Keep three points in mind. First, the source index still holds the original data until it is deleted, and so do any snapshots taken earlier, so a privacy promise is only kept when those are handled too. Second, masking done carelessly can be reversible, for example when the mask keeps too much of the value. Third, test copies for development should be built from a masked reindex and never from raw production data.

Compliance and where copies live

A reindex creates a full second copy of data. If the data is subject to rules about location or retention, make sure the destination lives in an allowed place, has the same protection as the source, and is included in retention and deletion schedules. When you reindex from one cluster to another, remember that data crosses a network and possibly a border.

Audit trail

Record who ran which reindex, when, with which script, and with what results. Elasticsearch can log security events, and the task record keeps the final counters. Storing the script and the report together with the change ticket gives you a trail that an auditor, or a colleague a year later, can follow.

Long jobs need watching, and finished jobs need checking. This chapter shows how to follow a reindex while it runs and how to prove afterward that the copy is right.

Watching a running job

Every background reindex is a task, and the task interface tells you what it is doing. You can list all running reindex tasks, or ask about one by its identifier. The status shows how many documents are expected in total, how many have been created, updated, deleted, skipped or blocked by conflicts, how many batches have been processed, how many retries have happened and how long the job has been throttled. With slices, the parent task shows totals and each child task shows its own share. Comparing the number processed with the total gives a progress estimate: when the numbers match, the job is done.

Controlling a running job

You can cancel a task, which stops it at the next opportunity, and rethrottle it, which changes its speed limit without restarting. Remember that cancelling leaves the documents already written. When a background job finishes, its final result is kept in a system index for later reading, so tidy up old task results from time to time.

Watching the cluster

The reindex itself is only half the picture. The other half is how the cluster feels. Watch the queues and rejections of the write thread pool, processor use, memory pressure, disk space, indexing latency on the destination and search latency on the source. If users start to notice slow searches, or if rejections and retries climb, slow the job down with the rethrottle operation. The slow logs of the source and destination can also help you see which requests are heavy.

Verifying a finished job

A finished job is not a proven job. Verification is the step that turns “it ran” into “it is right,” and it should be done with numbers and with your own eyes.

Patterns are recipes that experienced teams keep repeating because they work. Anti-patterns are the traps those same teams learned to avoid.

Patterns worth copying

Traps to avoid

A short, practical checklist drawn from careful teams, followed by the mistakes that trip up nearly everyone once.

Common mistakes and their cures

Here is how the same tool adapts to very different industries. Named companies below are used as familiar illustrations of the kind of scale and problem involved, not as claims about their internal systems.

A complete walk-through: the product catalog

To make everything concrete, here is how a team might carry out the price migration from the start of this article, from a blank page to a proven result.

After the migration, the problem from the opening of this article is gone. Prices are numbers, the finance team can add them up, and the old text field is only a memory.

Short answers to the questions that beginners ask most often.

The whole article in one place. If you remember only this box, you still know the essentials.