The Transform API converts raw, event-level data in a source index into a summarized, entity-centric destination index, and can keep that destination continuously, automatically up to date.

The one-sentence answer

Elasticsearch commonly stores raw, granular data — one document per event, per transaction, per log line — in a source index. A transform is a configuration that describes how to group and aggregate that raw data, using the same aggregation concepts already familiar from ordinary Elasticsearch searches, and write the resulting, summarized documents into a separate destination index. The Transform API is the set of operations Elasticsearch exposes to create, preview, start, stop, monitor, and manage these transforms.

Two distinct styles of transform exist, covered in depth later in this guide: a pivot transform, which groups raw documents by one or more fields and computes aggregated metrics per group, and a latest transform, which instead keeps only the single most recent document for each unique combination of chosen fields. Both share the same underlying machinery for reading source data, tracking progress, and writing to a destination index; they differ only in what kind of destination document they ultimately produce.

What the Transform API is not

• It is not a general-purpose ETL or data-integration tool for moving data between entirely different systems; it operates specifically from an Elasticsearch source index to an Elasticsearch destination index.
• It is not a replacement for runtime aggregations run directly on a search request; those remain useful for ad-hoc, exploratory questions that were not anticipated in advance.
• It is not a real-time, sub-second streaming mechanism; even in its continuous mode, it operates on a periodic checking interval, not an instantaneous, event-by-event basis.

Elasticsearch already supports powerful aggregations at search time, so it is worth being precise about exactly which problem the Transform API solves that runtime aggregations do not.

The cost of repeated, on-demand aggregation

Elasticsearch can absolutely compute an aggregation, such as total spend per customer, directly at search time, scanning the raw source index fresh for every single request. This works well for occasional, exploratory questions. But for a dashboard loaded repeatedly by many users throughout the day, or an application that needs this same summary looked up instantly and constantly, recomputing that same expensive aggregation from scratch on every single request is wasteful, and the cost grows directly with how much raw historical data has accumulated over time.

The cost of unbounded aggregation cardinality

A related, deeper problem is that some aggregations become genuinely impractical to compute live, at query time, once the number of distinct groups involved grows large enough. Aggregating total spend for a handful of product categories at search time is cheap. Aggregating a separate total for each of several million individual customers, computed fresh on every request, is a fundamentally heavier operation, and one that competes for the same cluster resources as the ordinary search traffic the cluster is also trying to serve.

flowchart TD
  A["Dashboard needs per-customer
lifetime spend summary"] --> B{"Compute live at
every page load?"}
  B -->|"Yes"| C["Expensive, repeated aggregation
over the entire raw history"]
  B -->|"No: use a transform"| D["Pre-computed summary already
sitting in a small destination index"]
  D --> E["Dashboard reads instantly,
no aggregation needed at request time"]

Why hand-rolling your own pipeline is its own burden

Faced with this, teams have historically built their own custom pipelines: a scheduled job that periodically queries the raw index, computes the needed aggregations, and writes the results into a summary index by hand. This approach works, but it means owning and operating an entirely separate piece of infrastructure — scheduling it reliably, handling partial failures, figuring out how to process only new or changed data rather than reprocessing everything every single time, and keeping it in sync with the underlying Elasticsearch cluster’s own behavior and version changes. The Transform API exists specifically to remove this burden, by making incremental, aggregated summarization a first-class, built-in capability of Elasticsearch itself, rather than something every team has to reinvent independently.

The hidden difficulty of “process only what changed” logic

Of all the pieces a hand-rolled pipeline has to get right, correctly identifying exactly which records are new or changed since the last run is often the hardest and most error-prone part, more so than the aggregation logic itself. A naive implementation might simply track “the last time this job ran” and query for anything created after that timestamp, but this simple approach quietly breaks the moment data can arrive slightly out of order, or the moment an existing record is updated rather than newly created, since an update does not necessarily fall neatly after the previous run’s cutoff in the way a brand-new record would. Getting this incremental-detection logic genuinely correct, across every edge case a real production data source can produce, is exactly the kind of unglamorous, easy-to-get-subtly-wrong engineering work that the Transform API takes off a team’s plate entirely, having already solved it once, generally, inside the platform itself.

Why this becomes a bigger problem as more summaries are needed

A single hand-rolled summarization job might be a manageable, one-time engineering investment. Real applications, though, rarely need just one summary; a growing product typically accumulates a whole family of different entity-centric views over time — one for customers, one for devices, one for sessions, one for products — each with its own subtly different aggregation logic and its own incremental-processing requirements. Without a shared, built-in mechanism, each of these becomes its own bespoke pipeline, multiplying the same categories of operational risk, scheduling complexity, and failure-handling logic across every single one of them independently, rather than reusing one well-tested, shared mechanism across all of them.

A handful of precise terms make the rest of this guide click into place.

Batch versus continuous: a fundamental fork in behavior

A transform can be run as a one-time batch operation, processing all currently existing source data exactly once and then stopping, which is useful for a one-off analysis or a single historical summary. Alternatively, it can be configured to run in continuous mode, where it keeps running indefinitely, periodically waking up, checking the source index for anything new or changed since its last checkpoint, processing just that incremental slice, and updating the destination index accordingly, entirely without manual intervention.

Why the sync field has to be a date field

Continuous mode’s entire ability to process only new or changed data, rather than reprocessing everything from scratch on every run, depends on being able to reliably identify what counts as “new since last time.” A date-type field, updated whenever a relevant source document is created or modified, gives the transform a reliable, monotonically progressing marker to compare against its last recorded checkpoint, which is precisely why continuous transforms require designating such a field as part of their configuration.

A transform is implemented as a persistent, managed background task, built on top of Elasticsearch’s own existing aggregation and indexing machinery.

The pieces that make up a transform

A transform’s configuration bundles together several pieces: the source index or indices to read from, an optional query to filter which source documents are even considered, the grouping and aggregation logic for a pivot transform (or the unique-key and sort logic for a latest transform), the destination index to write results into, and, for continuous transforms, the sync configuration identifying the date field and how frequently to check for new data.

flowchart TD
  A["Transform configuration"] --> B["Source index
+ optional filter query"]
  A --> C["Grouping and
aggregation logic"]
  A --> D["Destination index"]
  A --> E["Sync configuration
(continuous mode only)"]
  B --> F["Transform task"]
  C --> F
  E --> F
  F --> D

Where transform state and checkpoints actually live

A transform’s own internal bookkeeping — its current checkpoint, its state, and related metadata — is itself stored in dedicated, internal Elasticsearch indices, rather than existing only in the memory of whatever node happens to be running the transform task at a given moment. This matters because it means a transform’s progress is durable and recoverable: if the node running a transform task fails or restarts, another node can pick the transform back up from its last recorded checkpoint, rather than losing track of what had already been processed.

The transform task itself

Once created and started, a transform runs as a persistent task assigned to a specific node in the cluster, one capable of handling this kind of workload. This task is what actually issues the aggregation queries against the source index, processes their results, and writes the resulting documents into the destination index, on an ongoing basis for continuous transforms or until completion for batch transforms.

Following one checkpoint cycle of a continuous pivot transform step by step reveals exactly how incremental processing actually happens.

Why using a deterministic destination document identifier matters

A pivot transform deliberately derives each destination document’s identifier from the values of its grouping fields, rather than generating a fresh, random identifier every time a group is recalculated. This is what allows an updated summary for an existing group to correctly overwrite its own previous version in the destination index, rather than accumulating duplicate, stale documents for the same underlying group every single checkpoint cycle.

Why re-aggregation, not simple addition, is necessary

It might seem like updating a running total for a customer could be as simple as adding the value of each new transaction to whatever total already exists. In practice, transforms are built around re-running the full aggregation logic for affected groups, which correctly handles more complex aggregations than simple sums, correctly handles updates or corrections to existing source documents, not merely brand-new ones, and avoids the subtle bugs that a purely incremental, add-only approach could introduce over a long-running series of updates.

Tracing one concrete scenario end to end makes the abstract checkpoint cycle in the previous chapter feel real.

Setting the scene

Imagine an online retailer, “BrightCart,” with a raw orders index recording every individual order as it is placed. BrightCart configures a continuous pivot transform that groups orders by customer identifier, computing each customer’s total lifetime spend, order count, and most recent order date, writing the results into a customer-summary destination index that powers the support team’s dashboard.

sequenceDiagram
  participant Orders as Raw orders index
  participant Transform as Continuous transform
  participant Summary as Customer-summary index
  participant Dash as Support dashboard

  Orders->>Transform: New order placed by Customer A
  Transform->>Orders: Query for changes since last checkpoint
  Orders-->>Transform: Customer A's new order
  Transform->>Transform: Recompute Customer A's aggregates
  Transform->>Summary: Overwrite Customer A's summary document
  Transform->>Transform: Advance checkpoint
  Dash->>Summary: Read Customer A's updated summary
  Summary-->>Dash: Up-to-date totals, instantly

What the dashboard sees, and when

The support dashboard querying the customer-summary index never touches the raw orders index at all, and never waits for any aggregation to be computed at request time; it simply reads an already up-to-date document. The dashboard’s view of a given customer’s totals becomes current within roughly one checkpoint cycle’s worth of delay after a new order is placed, rather than instantly at the exact millisecond the order was recorded, which is an important, deliberate characteristic of how continuous transforms operate, discussed further in a later chapter.

The two transform types solve related but distinctly different problems, and knowing which one fits a given need is an essential first decision.

Pivot transforms: aggregation across many documents

A pivot transform’s entire purpose is to combine many source documents belonging to the same group into one aggregated summary document, using aggregation building blocks such as sums, averages, counts, minimums, and maximums, exactly analogous to the aggregations already available for ordinary search-time queries, just computed once and persisted rather than recomputed on demand.

Latest transforms: the single most current document per key

A latest transform solves a different, narrower problem: given many documents that share a unique key, but where you only care about the single most recent one, it identifies and copies forward only that latest document for each unique key combination, with no aggregation across multiple documents involved at all. A natural use case is maintaining an always-current “latest known status” index for a set of entities, such as the most recent recorded state of each device in a large fleet, where only the newest reading actually matters for most purposes.

Why the distinction matters for choosing the right tool

Reaching for a pivot transform when what you actually need is simply “the latest record per entity” would mean configuring an aggregation, such as a maximum over a timestamp field, purely to simulate what a latest transform already does natively and more directly. Conversely, a latest transform cannot produce genuinely aggregated metrics, such as a running total across many historical documents, since it is fundamentally built around selecting one document rather than combining several. Recognizing which underlying question you are actually asking — “summarize many documents together” versus “show me only the newest one” — is the first, most important step in configuring the right kind of transform.

Two specific mechanisms, composite aggregations and configurable sync delay, are what let transforms scale to enormous cardinality and handle real-world data arrival patterns gracefully.

Why an ordinary aggregation would not scale to millions of groups

An ordinary aggregation computing a metric per unique value of some field is, by default, oriented toward returning a manageable, bounded number of resulting buckets in a single response. Asking for a separate result for each of several million unique customers in one ordinary aggregation request would be an enormous, unwieldy, memory-intensive operation to compute and return all at once. Transforms instead rely internally on a composite aggregation, a mechanism specifically designed to page through an arbitrarily large number of distinct groups in smaller, manageable, memory-bounded batches, processing one page of groups at a time until the entire set has been covered.

flowchart LR
  A["Millions of unique
customer groups"] --> B["Composite aggregation:
page 1 of groups"]
  B --> C["Process and write
page 1 results"]
  C --> D["Composite aggregation:
page 2 of groups"]
  D --> E["Process and write
page 2 results"]
  E --> F["... continues until
every group is covered"]

Sync delay: giving out-of-order data a chance to arrive

Real-world data does not always arrive in perfectly punctual, strictly time-ordered fashion; network delays, buffering, and distributed producers can mean a document with an earlier logical timestamp occasionally arrives at Elasticsearch slightly after documents that logically came after it. A continuous transform’s sync configuration includes an optional delay setting, which deliberately waits a short additional window past the current sync-field boundary before considering a range of time fully settled and ready to process, reducing the chance that a checkpoint advances past data that has not fully arrived yet.

The trade-off sync delay introduces

Configuring a longer sync delay reduces the risk of a transform advancing its checkpoint before some slightly-late data has arrived, at the direct cost of making the destination index’s freshness lag further behind real time by that same additional margin. Configuring too short a delay risks the opposite problem: a checkpoint advancing past a window that later turns out to have received additional, late-arriving data, which that transform will never automatically go back and reprocess into an already-passed checkpoint.

Understanding exactly how a transform behaves when things go wrong is essential for trusting it with production-critical summary data.

What happens when a checkpoint fails partway through

If a transform encounters an error while processing a checkpoint, such as a transient failure reaching the source or destination index, it does not advance its recorded checkpoint position to reflect a partially completed cycle. Instead, it retries the failed checkpoint, and only advances its durable checkpoint marker once that checkpoint’s work has genuinely, fully completed successfully. This behavior is what prevents a transient failure from silently leaving the destination index in an incomplete, inconsistent state while still believing that data range had been fully handled.

Transform health states and what they mean

A transform reports its own health and operational state, distinguishing between healthy, actively indexing, stopped, and failed conditions, among others. A transform that has moved into a failed state requires explicit operator attention to diagnose and resolve the underlying issue before it can resume normal operation; it does not simply retry forever silently in the background without surfacing that something is wrong.

Reprocessing historical data after a configuration change

If a transform’s aggregation logic needs to change after it has already been running for a while, the existing destination index generally needs to be rebuilt from scratch against the new logic, typically by creating a new transform pointed at a new destination index, rather than attempting to retroactively patch an already-running transform’s historical output in place. This is an important operational consideration to plan for, since correcting a mistake in a long-running transform’s logic is not simply a matter of tweaking a setting and continuing forward.

Pivot transforms support considerably richer aggregation logic than simple single-level grouping, unlocking more sophisticated entity-centric summaries.

Multi-level grouping

A pivot transform can group by more than one field simultaneously, producing a destination index organized by a compound key, such as one summary document per unique combination of customer and product category, rather than being limited to a single, flat grouping dimension. This allows entity-centric indices to reflect genuinely multi-dimensional summaries where a business question naturally spans more than one axis at once.

Nested and sub-aggregations

Within each group, a pivot transform can compute not just simple metrics but nested, richer aggregation structures, such as a breakdown of spend by product category within each customer’s own overall summary document. This lets a single destination document carry a genuinely rich, multi-faceted profile of an entity, rather than being limited to a flat handful of top-level numbers.

Filtering the source before aggregating

A transform’s configuration can include a query that filters which source documents are even considered before grouping and aggregation happen at all, which is useful both for excluding irrelevant or low-quality data from a summary, and for building multiple, differently-scoped transforms against the same underlying source index, each focused on a different relevant subset of the data.

Why richer logic still respects the same checkpoint discipline

No matter how elaborate a pivot transform’s grouping and aggregation logic becomes, it still operates under the exact same incremental checkpoint discipline described earlier: only groups touched by data newer than the last checkpoint are recomputed, regardless of how many nested sub-aggregations or derived fields are involved in producing each group’s final summary document.

Combining multiple metric types within one group

A single group’s summary document can freely combine several different kinds of metrics side by side: a count of matching events, a sum of some numeric value, an average, a minimum, and a maximum, all computed simultaneously for the same group in the same checkpoint pass. This means one well-designed pivot transform can often replace what might otherwise have felt like several separate, narrower summarization efforts, simply by including every metric a downstream consumer needs within one single, coherent transform configuration, rather than fragmenting closely related metrics across multiple, separately maintained destination indices.

Why this richness still has a practical ceiling

It is worth being realistic that this flexibility is not unlimited in practice. Every additional metric, nested sub-aggregation, or grouping dimension adds to the computational and memory cost of processing each affected group during a checkpoint, and an overly ambitious, everything-in-one-transform design can become its own performance liability at sufficiently high cardinality. A useful discipline is to include in a given transform only the metrics that genuinely belong together conceptually, from the perspective of how the resulting destination index will actually be consumed, rather than accumulating every metric anyone might someday want into one increasingly heavy, all-purpose transform.

The Transform API is a genuine trade, not a strictly-better alternative to every other approach, and understanding both sides leads to better decisions.

When the trade is clearly worthwhile

The Transform API earns its keep clearly and quickly whenever a summarized, entity-centric view of data is queried frequently, whenever the underlying aggregation would otherwise be expensive to compute repeatedly at search time, and whenever a modest, checkpoint-driven latency between an event happening and its summary reflecting that event is genuinely acceptable for the use case.

When it may not be the right tool

Use cases genuinely requiring sub-second, event-by-event freshness, or highly ad-hoc, unpredictable aggregation questions that were never anticipated in a fixed configuration, are poorly served by transforms, since the former conflicts with the checkpoint-based nature of continuous mode, and the latter is exactly what flexible, runtime search-time aggregations remain better suited for instead.

A transform’s performance profile depends heavily on cardinality, checkpoint frequency, and the complexity of its aggregation logic.

Checkpoint frequency versus per-checkpoint overhead

Configuring a continuous transform to check for new data very frequently reduces the latency between source changes and destination freshness, but each checkpoint cycle carries its own fixed overhead in querying and processing, even when relatively little new data has actually arrived. Configuring checkpoints too frequently for a workload with only modest data arrival rates can mean a disproportionate amount of resource consumption relative to the actual amount of new data being processed each cycle.

Cardinality as the dominant scaling factor

The number of distinct groups a pivot transform must track is generally the single largest factor influencing its resource consumption and runtime, since every affected group requires its own aggregation computation and its own destination-index write during a given checkpoint. A transform grouping by a field with tens of millions of distinct values is a fundamentally heavier operation than one grouping by a field with only a few thousand.

Resource isolation from ordinary search and indexing traffic

Because a transform’s own processing work competes for cluster resources alongside ordinary search and indexing traffic, teams running resource-intensive transforms in production should account for this shared resource consumption explicitly in capacity planning, and should consider dedicating appropriately resourced or specifically designated nodes for transform workloads in particularly demanding deployments, rather than assuming transform processing is a negligible, background-only cost with no bearing on the rest of the cluster’s performance.

Batch size and paging tuning

The size of each page a transform’s underlying composite aggregation requests can itself be tuned, trading off between fewer, larger requests against the source index and more, smaller ones. A larger page size generally means fewer round trips to cover the full set of affected groups in a given checkpoint, at the cost of a heavier individual request; a smaller page size spreads the same total work across more, lighter requests. As with many such settings, the right balance is workload-specific and benefits from empirical tuning against a representative data set and cardinality, rather than being assumed from general principles alone.

The relationship between source index shard count and transform throughput

Because a transform’s queries against its source index are, underneath, ordinary Elasticsearch search requests, the same general principles that affect search performance elsewhere also affect a transform’s own throughput: a source index spread appropriately across enough shards to allow real parallelism during the transform’s own aggregation queries will generally support a higher-throughput transform than one confined to a single, heavily loaded shard. Teams designing a source index specifically to feed a demanding, high-cardinality transform should keep this relationship in mind during the source index’s own shard-count planning, rather than treating shard count purely as a search-serving decision unrelated to transform performance.

Because a transform’s progress is durably recorded rather than held only in memory, it is designed to survive node failures gracefully.

Recovering from a node failure

Because a transform’s checkpoint state lives in durable, internal Elasticsearch indices rather than solely in the memory of the node currently executing it, if that node fails or is restarted, the cluster can reassign the transform’s task to a different available node, which resumes processing from the last durably recorded checkpoint rather than restarting from the very beginning or losing track of its progress entirely.

flowchart TD
  A["Node running transform
task fails"] --> B["Cluster detects
task is unassigned"]
  B --> C["Transform task
reassigned to a
healthy node"]
  C --> D["Resumes from last
durably recorded checkpoint"]
  D --> E["No progress lost,
no duplicate reprocessing
of already-completed work"]

Dependence on source and destination index availability

A transform’s own resilience does not exempt it from depending on the ordinary availability of its source and destination indices; if either becomes unavailable, whether due to a broader cluster issue or an unrelated index-level problem, the transform will be unable to make progress until that underlying availability issue is resolved, at which point it resumes from its last successful checkpoint rather than needing any special recovery procedure of its own.

Replication and the destination index

Because a transform’s destination index is an ordinary Elasticsearch index, it benefits from the exact same replication and shard-allocation reliability guarantees as any other index, and should be configured with an appropriate number of replicas for its importance, exactly as any other production-critical index would be, rather than being treated as somehow exempt from ordinary index-level reliability planning simply because it happens to be transform-generated.

Because a transform reads from one index and writes to another, its security model necessarily spans both, and deserves deliberate attention.

Distinct privileges for source and destination

Running a transform requires appropriate read privileges on its source index or indices, appropriate index-management and write privileges on its destination index, and a dedicated privilege for managing transforms themselves, reflecting the fact that a transform’s overall capability genuinely spans both ends of the pipeline it operates, not merely one side of it.

The destination index inherits its own, independent access control

A destination index populated by a transform is, from an access-control perspective, an entirely ordinary index, and should have field-level and document-level security configured deliberately and independently, based on who should actually be permitted to see the summarized data it contains, rather than assuming its permissions automatically or implicitly mirror whatever access controls exist on the original source index.

A subtle risk: aggregation can leak information across a security boundary

It is worth being deliberate about a subtle risk: an aggregated summary computed across documents a given user is not individually permitted to see could, in some circumstances, indirectly reveal aggregate information about that restricted underlying data through the destination index, if that destination index’s own access controls are not configured carefully and independently to prevent this. Teams building transforms across sensitive, access-restricted source data should think through this possibility explicitly, rather than assuming aggregation automatically anonymizes or safely obscures the underlying restricted data.

A small set of specific signals reveal whether a transform is healthy, keeping up, and producing trustworthy results.

Checkpoint recency and processing lag

Tracking how recently a continuous transform’s checkpoint has advanced, and how far behind the true current time that checkpoint sits, is the most direct way to understand whether a transform’s destination index is meaningfully fresh or has quietly begun falling behind, whether due to resource contention, an underlying failure, or simply a sync delay and checking interval that no longer match the actual pace of incoming data.

Documents processed and indexing failures

Monitoring the count of documents a transform has processed per checkpoint cycle, alongside any indexing failures encountered while writing to the destination index, surfaces both unexpected drops in throughput and any silent write failures that could otherwise leave the destination index subtly incomplete without an obvious, visible symptom.

Building alerting around transform health specifically

Because a transform silently falling behind or failing does not necessarily produce an obviously broken-looking destination index, at least not immediately, proactive alerting on transform state and checkpoint lag, rather than relying purely on downstream symptoms like an outdated-looking dashboard being noticed by a human, is a meaningfully better operational posture for any team depending on transform-generated data for anything genuinely important.

Practical guidance distilled from how the Transform API is used well, and used poorly, in real production systems.

A best practice around naming and documenting transform configurations

Because changing a transform’s logic generally requires standing up a new destination index rather than editing in place, teams that consistently document each transform’s purpose, its exact aggregation logic, and its destination index’s intended consumers avoid the confusion that otherwise accumulates over time as multiple generations of similarly-named transforms and destination indices proliferate across a growing deployment.

Concrete scenarios illustrate how the Transform API’s capabilities actually matter to real applications built on Elasticsearch.

Straightforward answers to the questions people most often ask once they first encounter the Transform API.

Bringing everything in this guide together into a single, quick reference.

The Transform API exists to remove a burden every data-driven application eventually faces: turning raw, event-level data into a fast, purpose-built, entity-centric summary, without forcing every query to recompute that summary from scratch, and without forcing every team to build and operate its own custom aggregation pipeline. By relying on durable, incremental checkpoints and Elasticsearch’s own composite aggregation machinery, a continuous transform can keep a destination index automatically, incrementally current at real-world scale, trading a modest, configurable amount of freshness latency for a dramatic reduction in both query-time cost and operational complexity. Understanding its checkpoint-driven nature, its two distinct transform types, and its genuine resource and latency trade-offs is what lets a team use it as a reliable production building block rather than a poorly understood black box.