What Is the Translog (Transaction Log) in Elasticsearch?
How Elasticsearch promises that a document you just indexed will still be there after a sudden power failure, seconds before that document was ever written into its real, searchable storage.
Imagine writing an important letter by hand, only to have the building’s lights flicker off the instant before you can seal it into an envelope and drop it in a mailbox. If the letter existed only as ink on a single loose sheet of paper on your desk, a power failure would not matter, because ink does not depend on electricity. But now imagine the letter exists only inside a half-finished word processor document that autosaves once every ten minutes. If the power goes out at minute nine, the letter is simply gone, and no amount of wishing brings back words that were never actually saved anywhere durable.
Every database and search engine that lets you add new information immediately, without forcing you to wait for a slow, heavyweight save operation first, faces exactly this problem internally. Elasticsearch is no exception: when a document is indexed, it does not instantly get written into its final, highly-optimized, searchable data structures, because doing that on every single document would be far too slow for any real workload. Instead, Elasticsearch keeps that final structure building up in memory for a while, and relies on a separate, much simpler, much faster append-only record of everything that happened, so that if the server crashes before the final structure is safely written to disk, nothing indexed is actually lost. That separate record is called the translog, short for transaction log, and it is one of the quietest, least glamorous, and most important pieces of engineering inside all of Elasticsearch.
This guide explains the translog from absolute first principles, assuming no prior background in Elasticsearch, Lucene, databases, or distributed systems. Every new term is defined the moment it appears, every abstract idea is tied to something familiar, and by the end you will understand not just what the translog is, but why it has to exist, how it behaves under real failures, and what a production engineer needs to know to configure and monitor it correctly.
1What Is the Translog in Elasticsearch?
The translog is a durable, append-only record of every write operation, kept specifically so that no acknowledged write is ever silently lost before it reaches its final, searchable home.
The one-sentence answer
Elasticsearch is a distributed search and analytics engine that organizes data into indices, which are internally split into pieces called shards. Each shard’s actual searchable content lives inside a Lucene index made of files called segments. The translog, short for transaction log, is a per-shard, append-only file that records every indexing, update, and delete operation the moment it happens, written to disk independently of Lucene’s own much slower segment-writing process, so that operations can be recovered and replayed even if the node crashes before Lucene has permanently written that data itself.
The word “transaction log” is borrowed from the world of relational databases, where the same basic idea has existed for decades under the name write-ahead log, or WAL. The core principle in both cases is identical: write a simple, fast, sequential record of an operation before or immediately alongside applying that operation to the real, complex, slow-to-update data structure, so that the sequential record can always be replayed to reconstruct any operation that the complex structure has not yet durably absorbed.
Picture a busy restaurant kitchen during dinner rush. Waiters do not walk straight into the kitchen and rearrange the pantry shelves every time a customer orders a dish; that would be catastrophically slow. Instead, every order gets scribbled onto a small paper ticket and clipped onto a rail in the exact order it arrived. The kitchen cooks from that rail of tickets, and even if a cook accidentally forgets what they were doing partway through a dish, the ticket is still clipped there as proof of exactly what was ordered and in what sequence. The ticket rail is fast to write to, cheap, and reliable. The fully cooked, plated meal — the equivalent of Lucene’s finished, searchable segment — takes much longer to produce. The translog is Elasticsearch’s ticket rail: a fast, reliable, sequential record that exists specifically to survive the moment between an order being placed and the kitchen finishing it.
What the translog is not
- It is not the primary storage format for search. Nobody ever searches the translog directly; all search queries run exclusively against Lucene’s segments. The translog exists purely for durability and crash recovery.
- It is not a full history or audit trail of every change ever made to a document. Old translog files are eventually removed once they are no longer needed for recovery, so it is not designed to be a permanent record of everything that ever happened.
- It is not a replacement for backups or snapshots. It protects against the narrow window between an operation happening and that operation reaching Lucene’s durable segments; it does not protect against deleting an entire index by mistake or against a catastrophic loss of the underlying disk itself.
The translog is the specific mechanism that lets Elasticsearch safely acknowledge a write to a client almost immediately, without waiting for that document to be folded into Lucene’s slower, more expensive segment-building machinery first. Without it, Elasticsearch would either have to make every single write dramatically slower, or accept that a crash could silently erase very recent writes that a client had already been told succeeded.
2Problem and Motivation: Why a Translog Has to Exist
To understand the translog, you first need to understand exactly what problem it is solving, and why the obvious alternatives do not work.
The expensive nature of Lucene’s segments
Underneath every Elasticsearch shard sits a Lucene index, and Lucene’s fundamental unit of storage is the segment: an immutable, self-contained mini-index of documents, along with inverted indexes, stored fields, and other structures that make searching fast. The word “immutable” is doing a lot of work here — once a segment is written, it is never edited again. Updating a document does not modify bytes inside an old segment; it marks the old version as deleted and writes a brand-new version into a newer segment. This design is what makes Lucene extremely fast at reading and searching, but it also means creating and finalizing a new segment on every single incoming document would be a wildly expensive, disk-thrashing operation if done constantly.
Two competing goals in tension
Every write-heavy system engineered for durability has to balance two goals that pull in opposite directions: writes need to feel fast to the client making them, and writes need to be safe even if the server crashes one microsecond later. Making writes safe by immediately, fully committing them to their final, optimized storage format on every single request would make the system slow. Making writes fast by only holding them in memory for a while, with no separate durable record, would make the system unsafe: a crash, a power outage, or an operating system kill signal could erase data a client was already told had been saved.
flowchart LR A["Fast writes,
no durability"] -->|"Naive in-memory only design"| C["Crash loses
recent data"] B["Fully durable writes,
segment written every time"] -->|"Extremely slow indexing"| D["Unusable at
real-world scale"] E["Translog: fast append-only
durable record"] -->|"Elasticsearch's actual design"| F["Fast AND
durable writes"]
Why an in-memory buffer alone is not enough
Elasticsearch does keep newly indexed documents in an in-memory buffer for a short time before they become part of a searchable, in-memory Lucene segment through an operation called a refresh, which by default happens roughly once per second. This buffer is what makes Elasticsearch’s near-real-time search possible. But an in-memory buffer, by definition, evaporates completely the instant a process stops running unexpectedly, whether from a crash, an out-of-memory kill, a hardware fault, or someone unplugging the machine. Without a separate durable record sitting underneath that buffer, every document indexed since the buffer was last durably saved would simply cease to exist after such an event, even though Elasticsearch had already told the client the write succeeded.
Writing directly to an in-memory buffer with no translog underneath is like keeping your grocery list only in your head while walking around the store. It works fine as long as nothing interrupts you. But if someone bumps into you hard enough to genuinely startle you, you might completely forget three items you were absolutely certain you would remember a moment earlier. Writing the list down on a scrap of paper first, even a messy, unsorted scrap, means that startling interruption no longer costs you anything: the paper remembers what your head might not.
Why not just make every refresh instantly durable instead
A natural question is: why not simply make the refresh itself write everything durably to disk in Lucene’s real format, and skip the translog entirely? The answer is that writing a genuinely durable, correctly ordered, crash-safe Lucene segment on every refresh, which happens roughly once per second by default, would still be considerably more expensive than appending a simple, sequential record to a translog file. The translog’s format is deliberately much simpler than a Lucene segment: it is closer to a plain sequential log of operations than to a fully built, queryable index, and that simplicity is exactly what makes it fast enough to write on every single operation, rather than merely once per second.
The cost of building a queryable structure versus the cost of appending a line
To appreciate why this distinction is so significant, it helps to picture what actually has to happen inside Lucene to make a document searchable at all. Lucene has to tokenize the document’s text fields, build or extend an inverted index mapping individual terms back to the documents that contain them, update any doc-value structures used for sorting and aggregations, and arrange all of this into a compact, efficient, on-disk layout that a search request can later scan through quickly. None of that work is optional, and none of it is something you would want to redo, from scratch, for a single incoming document, every single time. Appending an entry to the translog, by contrast, involves none of that: it is simply writing the operation, in the order it happened, to the end of an already-open file, which is one of the cheapest possible things a computer can do to persist information.
This gap in cost is precisely why Elasticsearch’s designers chose to build two separate systems rather than one. Lucene’s segment-building machinery is optimized entirely for the eventual reading side of the equation: making search queries fast, once the work of building the structure is done. The translog is optimized entirely for the opposite side: making the initial act of recording an operation as cheap and immediate as possible, with essentially none of the reading-side optimization work done at write time at all. Trying to force one single system to be excellent at both jobs simultaneously would have meant compromising on one goal to serve the other; keeping them separate lets each one be uncompromisingly good at exactly the job it was built for.
3Core Concepts You Need First
A small, precise vocabulary unlocks everything else in this guide. Learn these terms once and the advanced chapters later on will read easily.
Shard
A single, self-contained slice of an index. Every shard, whether primary or replica, has its own independent translog and its own independent Lucene index underneath it.
Segment
An immutable, self-contained mini-index of documents inside Lucene. Segments are the actual files that get searched; they are never edited once written, only created or eventually merged away.
Refresh
The operation that makes recently indexed documents visible to search, by building a new, small, in-memory Lucene segment from the buffer. Happens roughly every one second by default.
Flush
The operation that permanently, durably commits Lucene’s segments to disk and then clears out translog entries that are no longer needed, because they are now safely represented in Lucene’s own durable storage.
Translog
The append-only, per-shard durable record of every write operation, written independently of the refresh and flush cycle, specifically to survive a crash that happens before a flush occurs.
fsync
An operating-system-level instruction that forces data sitting in a buffer to be physically written to the underlying storage device, rather than merely handed to the operating system’s own cache.
Translog generation
One numbered file within a shard’s translog. New generations are created periodically, and old, no-longer-needed generations are eventually removed after a successful flush.
Checkpoint
A small companion file recording exactly how much of the current translog generation has been durably written, used to know precisely where a safe replay should begin.
Refresh, flush, and the translog are three different clocks
One of the most common points of confusion for people new to Elasticsearch is treating “refresh,” “flush,” and “translog write” as though they were the same event, or as though they always happen together. They do not. A refresh happens frequently, roughly once per second by default, and makes new documents searchable, but a refresh does not necessarily mean data has been made durable on disk in its final Lucene form; it can still be sitting only in the operating system’s file cache. A flush happens less frequently and is the operation that actually forces Lucene’s segments to be durably committed to disk and clears translog entries no longer needed for recovery. Writing to the translog itself, by contrast, happens on essentially every single write operation, independent of both the refresh and flush schedules, precisely because it is the fast, cheap, durable safety net sitting underneath both of the slower processes.
| Operation | What it actually does | Roughly how often |
|---|---|---|
| Translog write | Appends the operation to the shard’s translog file, optionally forcing it durably to disk immediately | On (or very near) every single write operation |
| Refresh | Builds a new, small, searchable in-memory Lucene segment from the buffer, making recent writes visible to search | Roughly every 1 second by default |
| Flush | Durably commits Lucene segments to disk and then trims translog entries no longer needed for recovery | Based on size or time thresholds, considerably less often than a refresh |
Why the distinction matters in practice
Understanding that these are separate mechanisms explains a lot of otherwise confusing Elasticsearch behavior. It explains why a document can become searchable, through a refresh, well before it would survive certain kinds of failure without the translog’s help. It explains why disabling or slowing down refreshes, a common tactic during heavy bulk-loading, has no bearing whatsoever on how safely your writes are being durably recorded, since the translog keeps operating on its own separate schedule regardless. And it explains why a flush, which trims the translog, is the operation that actually determines how large a shard’s translog is allowed to grow before Elasticsearch intervenes.
4Architecture and Components
The translog is implemented as a small set of files living alongside each shard, with a simple, deliberately unglamorous internal structure.
Where the translog physically lives
Every shard copy — whether it is a primary shard or a replica shard — maintains its own translog, stored in a dedicated directory alongside that shard’s Lucene data on disk. A ten-shard index with one replica per shard therefore has twenty separate translogs in total, one per shard copy, each recording only the operations relevant to that specific shard copy. This per-shard isolation matters: a translog never tries to record anything about any shard other than the one it belongs to, which keeps its internal logic simple and keeps a failure in one shard’s translog from having any bearing on any other shard.
flowchart TD
subgraph Index["An index with 2 primary shards, 1 replica each"]
subgraph P0["Primary shard 0"]
P0L["Lucene segments"]
P0T["Translog"]
end
subgraph R0["Replica shard 0"]
R0L["Lucene segments"]
R0T["Translog"]
end
subgraph P1["Primary shard 1"]
P1L["Lucene segments"]
P1T["Translog"]
end
subgraph R1["Replica shard 1"]
R1L["Lucene segments"]
R1T["Translog"]
end
end
Generations: the translog is not one single, ever-growing file
Rather than writing forever into a single, endlessly growing file, a shard’s translog is broken into numbered pieces called generations. New operations are appended to the current generation until a new generation is started, which typically happens around the time a flush occurs. Once a flush has confirmed that all operations in an older generation are now safely represented in Lucene’s own durable segments, that older generation is no longer needed for crash recovery and becomes eligible for deletion. This generational design keeps any single translog file from growing without bound, and keeps old, unneeded data from lingering on disk forever.
The checkpoint file: knowing exactly how far you can trust the log
Alongside the translog’s data files sits a small companion file usually referred to as the checkpoint. Its job is to record precisely how much of the current translog generation has actually been safely, durably written to the physical storage device, as opposed to merely handed to the operating system’s cache. This distinction matters enormously during recovery: without a reliable checkpoint, Elasticsearch would have no trustworthy way to know exactly where a safe replay of operations should begin and end, and might either replay operations twice or miss replaying some altogether.
Translog data files
The actual append-only sequence of recorded operations, grouped into numbered generations.
Checkpoint file
Tracks precisely how much of the current generation is confirmed durable, guiding safe recovery.
In-memory buffer
Where newly indexed documents sit before a refresh turns them into a small, searchable Lucene segment.
Lucene segments
The real, durable, optimized, searchable storage that the translog exists to protect data on its way toward.
Why the translog’s own format is deliberately simple
The translog does not attempt to be searchable, sortable, or queryable in any rich way. Each entry is essentially just “here is the operation that happened, in the order it happened.” This deliberate simplicity is a feature, not an oversight: a data structure that only ever needs to support appending new entries and replaying them sequentially, from a known starting point, can be made extremely fast to write to, in a way that a rich, queryable index fundamentally cannot be. Every ounce of complexity that was deliberately left out of the translog’s design is complexity that would otherwise have slowed down every single write Elasticsearch ever performs.
5Internal Working: What Happens on Every Single Write
Following one indexing request step by step through the translog and buffer reveals exactly how durability and speed are both achieved together.
Client sends an indexing request
An application asks Elasticsearch to index a new document into a particular index.
Request is routed to the primary shard
The coordinating node determines which primary shard is responsible for this document and forwards the request there.
The operation is appended to the primary’s translog
Before anything else happens to the document’s searchable representation, the raw operation is recorded in the primary shard’s translog.
The document is added to the in-memory indexing buffer
The document also enters an in-memory structure that will eventually become a searchable Lucene segment at the next refresh.
The operation is forwarded to replica shards
Each replica shard performs the same translog-then-buffer sequence independently, for its own copy of the data.
Acknowledgment is returned to the client
Once the required number of shard copies have durably recorded the operation in their own translogs, Elasticsearch reports success back to the client.
A later refresh makes the document searchable
Independently, on its own roughly-one-second schedule, a refresh turns the buffered document into a small, searchable segment.
A later flush makes Lucene’s storage durable and trims the translog
Eventually, a flush durably commits Lucene’s segments to disk, after which the corresponding translog entries are no longer needed and can be safely discarded.
Why the translog write comes before acknowledgment, not after
The ordering in step 3 and step 6 above is the entire point of the design. If Elasticsearch acknowledged a write to the client before the operation was durably recorded anywhere, a crash occurring in that gap would mean the client walks away believing a write succeeded that, in fact, left no durable trace anywhere on any machine. By requiring the translog write to complete first, Elasticsearch guarantees that any operation it has ever told a client succeeded is recoverable, even if that same operation has not yet made it anywhere near Lucene’s actual segments.
What “durably recorded” actually means at the operating-system level
Simply calling a write function in a programming language does not, by itself, guarantee data has reached a physical storage device; the operating system is often free to hold recently written data in its own memory cache for a while before actually pushing it to disk, purely for performance reasons. An operation called fsync is the mechanism that forces the operating system to flush that cached data out to the physical device immediately, rather than at its own leisure. Whether or when Elasticsearch performs this fsync on translog writes is governed by a configurable durability setting, covered in depth in the advanced chapters ahead, and this single setting is one of the most consequential durability knobs in the entire system.
Many newcomers assume that once Elasticsearch returns a success response, the document is already fully, permanently part of Lucene’s searchable index. In reality, the document may still only exist inside the in-memory buffer and the translog at that exact moment; it becomes searchable only at the next refresh, and it becomes part of Lucene’s own durable, on-disk representation only at the next flush. The translog is precisely what closes the durability gap in between.
6Data Flow and Lifecycle: A Worked Example
Tracing one concrete indexing request, from an application’s perspective all the way down to disk, makes the abstract timeline in the previous chapter feel real.
Setting the scene
Imagine an online ticketing platform, “VenueNow,” indexing a newly purchased concert ticket into Elasticsearch so support staff can immediately search for it by confirmation number. The index has three primary shards and one replica per shard. The new ticket document happens to be routed to primary shard 1, which currently has one replica living on a different node in the cluster.
sequenceDiagram participant App as VenueNow application participant Coord as Coordinating node participant Prim as Primary shard 1 participant PT as Primary translog participant Repl as Replica shard 1 participant RT as Replica translog App->>Coord: Index new ticket document Coord->>Prim: Route to primary shard 1 Prim->>PT: Append operation to translog PT-->>Prim: Durably written Prim->>Prim: Add document to in-memory buffer Prim->>Repl: Forward operation to replica Repl->>RT: Append operation to translog RT-->>Repl: Durably written Repl->>Repl: Add document to in-memory buffer Repl-->>Prim: Acknowledge Prim-->>Coord: Acknowledge (quorum reached) Coord-->>App: Success response
What happens if a crash occurs one second later
Now suppose, one second after that acknowledgment, the node hosting the primary shard suddenly loses power, before any refresh or flush has happened. The ticket document was never turned into a searchable Lucene segment on that node, and it was never part of any durably flushed Lucene storage either. And yet, nothing about VenueNow’s support-staff search experience is at risk of silently losing that ticket, for two independent reasons: first, the replica shard’s own translog and buffer already independently recorded the exact same operation on a different node entirely; second, even the primary shard’s own translog entry, sitting on the now-crashed node’s disk, is itself durable and can be replayed the moment that shard is recovered, whether on the same node after it restarts or by promoting the surviving replica.
Replaying the translog during recovery
When a shard needs to be recovered, whether because a node restarted, a replica is being rebuilt, or a primary is being reassigned elsewhere, Elasticsearch does not simply trust whatever Lucene segments happen to already exist on disk as the complete truth. It also consults the translog for that shard, replaying any operations recorded there that occurred after the most recent successful flush. Because the translog’s entries are ordered and durable, replaying them reconstructs precisely the same sequence of changes that would have happened if no crash had ever occurred, bringing the shard’s Lucene segments fully up to date before the shard is allowed to serve traffic again.
Worked recovery scenario
Suppose a shard’s last successful flush happened at 10:00:00, and the node then crashed at 10:00:07 after seven more documents had been indexed and acknowledged in the meantime. On restart, Elasticsearch opens the shard’s Lucene segments as they existed at the 10:00:00 flush, then reads the shard’s translog starting from that same point, and replays those seven recorded operations one by one back into the shard, restoring it to exactly the state it was in at 10:00:07 — with nothing missing, and nothing applied twice.
7High-Level Concepts: How the Translog Relates to Lucene Commits
Zooming out, the translog and Lucene’s own commit mechanism form a deliberate two-layer durability system, each covering the other’s weaknesses.
Lucene has its own notion of a durable commit
Independently of Elasticsearch’s translog, Lucene itself has a concept of a commit: a point at which a specific set of segments is recorded as the durable, official state of the index, safe to reopen even after a restart. A Lucene commit is a relatively heavyweight operation, involving durably writing metadata about exactly which segment files make up the index at that moment. This is deliberately not done on every single document, both because it would be slow and because Lucene’s own commit points are meant to represent stable, meaningful snapshots rather than a constant stream of tiny, incremental changes.
The flush is where these two systems meet
An Elasticsearch flush is, underneath, largely a request to Lucene to perform one of its own commits, followed by trimming translog entries that are now covered by that commit. Before a flush, Lucene’s on-disk commit point may be considerably out of date relative to what has actually been indexed, sometimes by many operations; the translog is precisely what covers that gap. After a flush, Lucene’s on-disk commit point is fully up to date as of that moment, and the translog entries covering everything up through that point become unnecessary for recovery purposes and are eligible to be discarded.
flowchart LR A["Last Lucene commit
(fully durable)"] -->|"Operations since
the last commit"| B["Covered only by
the translog"] B -->|"Flush occurs"| C["New Lucene commit
(fully durable)"] C -->|"Old translog entries
no longer needed"| D["Old translog
generation removed"]
Two layers of durability, each with a different job
It helps to think of these as two deliberately different layers working together rather than as one redundant system. Lucene’s commit mechanism is the slow, heavyweight, highly-optimized, occasionally-updated layer, responsible for the actual final searchable structure. The translog is the fast, lightweight, constantly-updated layer, responsible for making sure nothing acknowledged is ever lost in the time before it reaches that slower layer. Neither layer alone would be sufficient: Lucene’s commits alone would make every write agonizingly slow, and the translog alone, with nothing ever eventually folded into real Lucene segments, would never actually become efficiently searchable.
A photographer shooting a wedding takes rapid individual photos throughout the day, each one saved instantly to a memory card. Periodically, though, they also back up the entire card’s contents to a laptop and a cloud service, a slower, more deliberate process done only a few times during the event rather than after every single shot. The instant, per-photo save to the memory card is like the translog: fast, granular, and covering every single moment. The periodic full backup is like a Lucene commit: slower, but producing a complete, trustworthy, consolidated snapshot that does not need to be pieced back together from hundreds of tiny individual saves.
8Advanced Topics: Durability Modes and Tunable Behavior
Elasticsearch exposes several settings that let an operator directly trade off durability guarantees against raw write throughput, and understanding them is essential for running Elasticsearch responsibly in production.
Request durability versus asynchronous durability
Elasticsearch offers two fundamentally different durability modes for translog writes at the index level. In the mode generally referred to as request durability, an fsync of the translog is forced before Elasticsearch acknowledges a write back to the client, which means every acknowledged write is guaranteed to already be physically durable at the moment the client is told it succeeded. In the alternative mode, generally referred to as asynchronous durability, the fsync instead happens on a periodic timer, independent of any specific acknowledgment, which allows writes to be acknowledged faster but introduces a small window in which a crash could theoretically lose the very most recent operations that had been acknowledged but not yet physically synced.
Request durability
- Every acknowledged write is immediately physically durable
- Strongest possible guarantee against data loss on crash
- Simpler mental model: “acknowledged” always means “safe”
Async durability trade-off
- A small window of already-acknowledged writes could theoretically be lost on a hard crash
- Higher raw write throughput, since fsync happens on a timer rather than per write
- Requires deliberately accepting a bounded, small risk in exchange for speed
The sync interval setting
When asynchronous durability is chosen, an interval setting controls exactly how often the periodic fsync happens, commonly defaulting to every five seconds. This single number is the maximum possible amount of already-acknowledged data that could be lost in the absolute worst case of a hard crash landing at the least convenient possible moment: a five-second interval theoretically risks up to five seconds’ worth of acknowledged writes on an exceptionally unlucky crash, whereas request durability risks essentially none.
Flush thresholds: controlling how large a translog is allowed to grow
A size-based threshold setting controls how large a shard’s translog is allowed to accumulate before Elasticsearch triggers a flush on its own initiative, independent of any other scheduling. This exists for two practical reasons: an unbounded translog would eventually consume an unreasonable amount of disk space, and a very large translog also means a very long, slow replay would be required during recovery, since every single recorded operation since the last flush would need to be reapplied in sequence. Keeping this threshold at a sensible size keeps both disk usage and worst-case recovery time bounded and predictable.
Workloads that disable or heavily delay refreshes to speed up bulk indexing, a legitimate and common tuning technique, do not thereby disable or delay the translog. The translog keeps recording every operation regardless, and if flush thresholds are also pushed unusually high at the same time, a shard’s translog can grow far larger than typical, meaningfully lengthening how long recovery would take if that node crashed at an inopportune moment.
Retention beyond a single flush: keeping history for replica recovery
Beyond simply covering the gap since the last flush, Elasticsearch also has mechanisms to retain some additional translog history specifically to make certain kinds of replica recovery faster, allowing a replica that briefly fell behind to catch up by replaying a range of recent operations rather than requiring a full, expensive resynchronization of the entire shard’s data from scratch. This retained history is bounded and eventually reclaimed once it is no longer useful for that purpose, balancing faster typical-case recovery against unbounded disk growth.
A concrete, worked numeric picture of the trade-off
Numbers make this trade-off easier to reason about than abstract descriptions alone. Suppose a shard receiving a steady one thousand indexing operations per second is configured with a flush threshold sized to trigger roughly every thirty seconds under normal load. In the ordinary case, a crash occurring at a random moment would, on average, need to replay somewhere in the neighborhood of fifteen thousand operations during recovery, and in the worst case, right before the next scheduled flush, as many as thirty thousand. Doubling that flush threshold to trigger roughly every sixty seconds instead would roughly double both of those numbers, trading fewer, larger flush operations for a meaningfully longer worst-case replay. Neither choice is objectively correct; the right answer depends entirely on how many seconds of replay time a given team is comfortable accepting during a genuine incident, weighed against how much they value the reduced overhead of less frequent flushing.
This same kind of back-of-envelope reasoning is worth applying whenever a flush-related setting is being tuned away from its sensible default. A setting that looks like a harmless number in a configuration file translates directly into a very concrete quantity: how many operations, and therefore how many seconds, an on-call engineer will be staring at a replay progress indicator during the next unplanned crash, wondering exactly how much longer it will take before the affected shard is searchable again.
9Very Advanced Topics: Crash Recovery, Corruption, and Comparisons
The deepest, most operationally consequential material: what actually happens when the translog itself is damaged, and how Elasticsearch’s approach compares to write-ahead logging elsewhere.
What happens when the translog itself is corrupted
Disks fail in messier ways than a clean, total loss; sometimes only part of a file becomes unreadable or contains garbled bytes, perhaps from a hardware fault, an abrupt power loss mid-write, or a filesystem bug. Elasticsearch’s translog format includes checksums specifically so that corruption can be detected rather than silently trusted. When Elasticsearch opens a shard and finds that its translog appears corrupted, it deliberately refuses to simply guess at what the missing or garbled data might have been, since guessing wrong could silently reintroduce incorrect data into a search system that people rely on for accuracy.
The translog truncation tool: a deliberate last resort
For the rare situation where a translog is corrupted badly enough that the shard cannot even start normally, Elasticsearch ships a dedicated command-line remediation tool that allows an administrator to deliberately truncate the damaged translog, accepting the loss of whatever operations that translog alone was protecting, in exchange for being able to bring the shard back online at all. This is presented to operators as an explicit, manual, break-glass action rather than something Elasticsearch ever does automatically on its own, precisely because it trades a small, bounded amount of very recent data for shard availability, and that trade should always be a deliberate human decision made with full awareness of the consequences, not a silent background choice.
Anti-pattern
Treating translog truncation as a routine fix for “stuck” shards, applying it reflexively without first understanding why the translog became corrupted or unreadable in the first place.
Why it hurts
Truncation is a deliberately destructive operation. Reaching for it automatically, without investigating root cause first, both discards data unnecessarily in cases that might otherwise have been recoverable and risks masking a deeper, recurring hardware or filesystem problem that will simply corrupt the next translog too.
Better approach
Investigate the underlying disk, filesystem, and hardware health first, restore from a snapshot or a healthy replica wherever one is available, and reserve the truncation tool strictly for situations where no other recovery path exists and the resulting data loss has been consciously accepted.
How Elasticsearch’s translog compares to write-ahead logs elsewhere
The write-ahead log concept long predates Elasticsearch and appears, in various forms, across the entire database world: relational databases have used essentially the same idea for decades to guarantee that a transaction, once committed, survives a crash even though the underlying data pages it touched have not yet been fully written back to their final storage location. Elasticsearch’s translog follows the same essential logic, adapted specifically to Elasticsearch’s own architecture of shards, segments, refreshes, and flushes rather than pages and transactions, but the underlying insight — record the operation fast and sequentially first, apply it to the expensive final structure later — is a shared piece of distributed-systems and database wisdom, independently reinvented across many different systems because the underlying trade-off it solves is universal.
flowchart TD A["Universal write-ahead
logging principle"] --> B["Relational databases:
transaction logs"] A --> C["Elasticsearch:
the translog"] A --> D["Other distributed
systems: append-only
replication logs"] B --> E["Same core idea:
record fast, sequentially,
before the slow final write"] C --> E D --> E
Sequence numbers: ordering guarantees beneath the translog
Each operation recorded in a shard’s translog is also tagged with a strictly increasing sequence number, unique to that shard, which gives Elasticsearch an unambiguous, total ordering of every operation that has ever happened to that shard. This ordering is what makes replaying a translog safe and deterministic: replaying operations out of order, or replaying the same operation twice by mistake, could otherwise silently produce a different, incorrect final result than the one the client originally intended.
10Complex Concepts and Features: Global Checkpoints and Multi-Copy Consistency
The translog does more than protect a single shard copy from a crash — it is also a foundational building block for keeping a primary and its replicas provably consistent with one another.
Local checkpoints versus the global checkpoint
Each shard copy tracks its own local checkpoint: the highest sequence number for which that particular copy has durably recorded every single operation up to and including that number, with nothing missing in between. Separately, the primary shard tracks a global checkpoint: the highest sequence number that every in-sync copy of the shard, primary and all its current replicas together, has already durably recorded. The global checkpoint can never be higher than the slowest currently in-sync copy’s own local checkpoint, since it represents a guarantee about every copy, not merely the fastest one.
Local checkpoint
Per-shard-copy: the point up to which that one specific copy has durably recorded every operation with nothing missing.
Global checkpoint
Cluster-wide, per-shard: the point up to which every currently in-sync copy has durably recorded every operation.
In-sync copy
A shard copy currently considered fully caught up and eligible to be counted toward the global checkpoint and toward future promotion.
Why this matters when a primary shard fails
When a primary shard becomes unavailable, one of its replicas must be promoted to take over as the new primary. The global checkpoint is what guarantees this promotion is safe: because it represents an operation count that every in-sync replica already durably holds, promoting any in-sync replica to primary is guaranteed not to silently lose any operation that had already been safely, durably acknowledged across the required number of copies. Without the translog durably backing every shard copy’s own local checkpoint, this entire safety guarantee would have nothing solid underneath it.
sequenceDiagram participant Primary as Old primary participant R1 as Replica 1 (in-sync) participant R2 as Replica 2 (in-sync) Primary->>R1: Operation seq #100 Primary->>R2: Operation seq #100 R1-->>Primary: Ack (local checkpoint = 100) R2-->>Primary: Ack (local checkpoint = 100) Primary->>Primary: Global checkpoint advances to 100 Note over Primary: Primary crashes R1->>R1: Promoted to new primary Note over R1: Safe: R1's translog already
holds everything up to seq #100
Retaining translog history to speed up replica recovery
When a replica briefly falls behind, perhaps due to a short network hiccup or a brief restart, Elasticsearch prefers, wherever possible, to bring it back up to date by replaying the specific range of operations it missed directly from retained translog history, rather than resorting to a full, expensive resynchronization of the entire shard’s underlying data from scratch. This is considerably cheaper and faster for the common case of a short-lived disruption, and it is precisely the retained translog history discussed in the previous chapter that makes this optimization possible at all.
Why a full resynchronization is so much more expensive than a replay
It is worth being explicit about just how large the gap is between these two recovery paths. Replaying a bounded range of recorded operations from translog history means only the specific changes that were actually missed need to be transmitted and reapplied, which for a brief disruption might be a comparatively small number of operations even on a very large shard. A full resynchronization, by contrast, effectively means copying the shard’s entire underlying data set over the network from a healthy copy, regardless of how small the actual gap in knowledge was. For a shard holding many gigabytes of data, that difference can mean the gap between a recovery finishing in seconds and one that takes minutes or longer, entirely dependent on network bandwidth between nodes rather than on how much data was actually missed. This is precisely why Elasticsearch goes out of its way to retain a bounded window of translog history specifically to make the cheap, targeted replay path available whenever the length of the outage allows for it.
What happens when the gap is too large for a replay
Retained translog history is deliberately bounded, both in how much data it holds and in how long it is kept, since keeping it indefinitely would defeat the purpose of ever trimming the translog at all. If a replica has been disconnected long enough, or missed enough operations, that the required range of history is no longer available, Elasticsearch falls back automatically to the more expensive full resynchronization path instead. This automatic fallback means operators do not need to manually decide which recovery path applies in a given situation; Elasticsearch always attempts the cheaper, faster path first, and only resorts to the expensive path when the cheaper one genuinely is not possible anymore.
11Advantages, Disadvantages, and Trade-offs
Like every durability mechanism, the translog is a deliberate set of trade-offs rather than a free win, and understanding both sides makes for better operational decisions.
Advantages
- Lets Elasticsearch acknowledge writes quickly without waiting for a full, expensive Lucene commit on every single request
- Guarantees any acknowledged write survives a crash, restart, or unexpected process termination
- Enables fast, targeted recovery by replaying only recent operations rather than rebuilding an entire shard
- Underpins the sequence-number and checkpoint machinery that keeps replicas provably consistent with the primary
Costs and trade-offs
- Every write incurs the extra cost of an append to the translog, on top of the in-memory buffer update
- Choosing asynchronous durability for higher throughput introduces a small, bounded window of possible data loss on a hard crash
- A very large, un-flushed translog lengthens worst-case shard recovery time
- Translog files consume additional disk space beyond Lucene’s own segments until they are safely trimmed by a flush
There is no universally correct durability setting
Different workloads reasonably make different choices here. A system recording financial transactions or medical records will typically choose request durability without hesitation, accepting somewhat lower peak indexing throughput in exchange for the strongest possible guarantee that nothing acknowledged is ever lost. A system ingesting a high-volume stream of easily-replayable telemetry or log data, where losing a handful of seconds’ worth of recent entries during an exceptionally rare hard crash is a tolerable, recoverable inconvenience rather than a catastrophe, might reasonably choose asynchronous durability to maximize throughput instead.
The right question to ask is not “which durability mode is best,” but “what does it actually cost this specific workload if the absolute worst case, a handful of seconds of very recent, already-acknowledged writes, is lost during an exceptionally rare hard crash.” For some data, that cost is unacceptable. For other data, it is a rounding error compared to the throughput gained.
12Performance and Scalability Considerations
Because the translog sits directly in the path of every write, its performance characteristics have an outsized effect on overall indexing throughput.
Storage hardware matters enormously here
Because translog writes, especially under request durability, involve frequent, small, sequential fsync operations, the underlying storage device’s raw write latency has a direct, immediate impact on indexing performance. Fast solid-state storage with low write latency handles this workload far more gracefully than older spinning-disk storage, where every fsync can involve a comparatively slow mechanical operation. This is one of the more concrete, practical reasons that production Elasticsearch deployments handling meaningful write volume are generally steered firmly toward fast, low-latency local storage rather than slower networked or mechanical alternatives.
Bulk indexing and the translog
Applications that index data in large batches, using Elasticsearch’s bulk indexing capability, still write every individual operation within that batch to the translog; the translog does not somehow become optional or skip entries just because operations arrived grouped together in one request. What bulk indexing does help with is reducing the overhead of network round-trips and per-request coordination, not the fundamental need for every operation to still be durably recorded.
Larger flush thresholds versus smaller ones
Raising the size threshold that triggers an automatic flush reduces how often the relatively expensive flush operation runs, which can meaningfully help sustained indexing throughput. The cost of doing so is a translog that is allowed to grow larger before being trimmed, which in turn means a longer worst-case recovery replay if a crash happens to occur right before that larger flush would have run. As with durability mode, this is a genuine trade-off to be tuned deliberately for a given workload’s actual tolerance for longer recovery times, not a setting to raise blindly in pursuit of throughput alone.
13High Availability and Reliability
The translog’s contribution to overall cluster reliability goes well beyond protecting a single node against a single crash.
Translogs and replica shards work together
Because every replica shard maintains its own independent translog, a cluster’s overall resilience to a single node failure comes from the combination of replication and the translog acting together, not from the translog alone. If a node hosting a primary shard fails entirely and never comes back, the translog sitting on that now-dead node’s disk provides no help whatsoever; what saves the data in that scenario is that an in-sync replica, on a different node, already independently durably recorded the same operations in its own separate translog, and can be promoted to primary without any data loss.
Why replica count still matters even with a durable translog
It is a common misunderstanding to assume that having a well-configured, durable translog somehow reduces the need for replica shards. It does not: the translog protects against a crash of the process or the operating system on a given node, assuming that node’s own disk survives and remains reachable. It does nothing at all to protect against the total loss of that entire node, including its disk, whether from hardware failure, an accidental deletion, or a destroyed virtual machine. Only having the data durably present on a separate node, through replication, protects against that broader class of failure.
flowchart TD A["Failure type"] --> B["Process crash,
disk survives"] A --> C["Entire node lost,
including its disk"] B --> D["Translog on that
node's disk is enough"] C --> E["Translog on the
dead node is useless"] C --> F["Only a healthy
replica on another
node saves the data"]
Running an index with zero replicas, purely to save on storage or compute cost, removes the ability of that data to survive a total node loss, no matter how well the translog is configured on the single existing copy. A perfectly durable translog on the only surviving copy of a shard is no comfort at all if that copy’s entire disk has just been destroyed along with the node it lived on.
14Security Considerations
The translog is not primarily a security feature, but it does raise a few considerations worth understanding for anyone responsible for a production cluster’s data protection posture.
The translog contains real document data
Because the translog records the actual operations performed, including the content of newly indexed or updated documents, it necessarily contains the same sensitive information that lives in the index itself, at least for a window of time until that data has been folded into Lucene’s segments and the corresponding translog entries have been trimmed. Anyone with filesystem-level access to a node’s data directory who could read translog files would potentially be able to see recently written document content, exactly as they would if they could read the underlying Lucene segment files directly.
Encryption at rest covers the translog too
When an operator encrypts the underlying storage at the operating-system or disk level, that encryption applies to everything stored on that disk, including both Lucene’s segment files and the translog’s own files, since neither is treated specially by disk-level encryption. This means the standard best practice of enabling encryption at rest, already a recommended baseline for protecting Elasticsearch’s data files generally, automatically extends the same protection to the translog with no separate, translog-specific configuration required.
Filesystem permissions
The translog directory should carry the same restrictive file permissions as the rest of a node’s data directory, since it holds equally sensitive content.
Backup and snapshot scope
Snapshots generally capture a consistent point-in-time view of Lucene’s durable data rather than raw translog files, so operators should not assume a snapshot alone captures every last unflushed operation.
Node decommissioning
Disks removed from decommissioned nodes should be securely wiped or destroyed, since translog files on them may still contain readable, sensitive document content.
The truncation tool as a sensitive administrative capability
Because the translog truncation tool discussed earlier can deliberately discard data, access to run it should be restricted to trusted administrators, and its use should be logged and reviewed like any other destructive administrative operation, rather than being something available to a broad set of operators without oversight.
15Monitoring, Logging, and Metrics
A healthy translog is invisible; a struggling one shows up clearly in a handful of specific metrics worth watching deliberately.
Translog size per shard
Elasticsearch exposes, per shard, the current size on disk of its translog. A translog that is consistently large, or that keeps growing without ever shrinking back down after an expected flush, is a signal worth investigating: it can indicate flushes are not happening as expected, that a flush threshold has been configured unusually high, or that some other condition is preventing translog generations from being cleared.
Number of operations since the last flush
Alongside raw size, Elasticsearch also reports how many operations have been recorded in the translog since the shard’s last successful flush. This number is a direct proxy for how much work a recovery would need to replay if that shard crashed at that exact moment, making it a genuinely useful early-warning signal for unusually long potential recovery times, well before an actual crash ever puts that number to the test.
| Signal to watch | What a worrying trend suggests |
|---|---|
| Translog size steadily climbing | Flushes may be happening less often than expected, or a size threshold may be set unusually high |
| Operations since last flush growing large | Recovery time after an unexpected crash would grow correspondingly longer |
| Frequent, unusually slow flush operations | Underlying storage may be struggling to keep up with Lucene’s commit workload |
| Elevated indexing latency correlated with fsync activity | Storage write latency may be the limiting factor, especially under request durability |
Correlating translog metrics with hardware health
Because translog performance is so closely tied to underlying storage latency, a sustained, unexplained rise in indexing latency is worth cross-referencing against translog-related metrics and the health of the underlying disk itself. A struggling or degrading storage device often shows up first as slower fsync operations, which in turn shows up as slower acknowledgment of writes, well before that same degrading disk causes any more dramatic, obvious failure.
Treat a sudden, unexplained spike in translog size or in operations-since-last-flush the same way you would treat a sudden spike in any other core reliability metric: as a prompt to investigate immediately, not as background noise to note and move past, since it is one of the clearest early signals available that recovery time in a real incident is quietly getting longer.
16Design Patterns, Best Practices, and Anti-patterns
Years of production experience with the translog have produced a small set of well-understood patterns worth following, and traps worth deliberately avoiding.
Match durability mode to data value
Choose request durability for genuinely irreplaceable data, and consider asynchronous durability only for easily-replayable or lower-stakes data streams.
Watch translog size as a leading indicator
Treat translog size and operation count as early-warning reliability signals, not merely obscure internal details to ignore.
Never rely on the translog as a backup strategy
Continue taking regular snapshots regardless of translog durability settings, since the translog protects against a narrow class of failure, not against catastrophic data loss generally.
Provision fast, low-latency storage for write-heavy indices
Recognize that translog performance is closely tied to underlying storage latency, and provision hardware accordingly for demanding indexing workloads.
Anti-pattern
Aggressively raising flush-related thresholds purely to chase higher indexing throughput numbers in a benchmark, without ever considering the corresponding effect on worst-case recovery time in a real production incident.
Why it hurts
A benchmark rewards raw throughput; a real incident rewards fast, predictable recovery. Tuning exclusively for the first metric while ignoring the second can leave a team facing a much longer, more painful recovery than expected during an actual outage, precisely because nobody had considered how large the translog was quietly allowed to grow.
Better approach
Tune flush and durability settings with both throughput and worst-case recovery time deliberately in view, and periodically validate, through genuine failure testing rather than assumption alone, that recovery from a realistic crash scenario still completes within an acceptable window.
A best practice around bulk-loading windows
Teams that temporarily relax refresh intervals during large bulk-loading operations, a legitimate and common technique to speed up initial data loads, should remember that the translog is entirely unaffected by that particular tuning knob, and should separately confirm that flush-related settings and available disk space remain appropriate for the larger volume of data being written during that loading window specifically.
17Real-World and Industry Examples
Concrete scenarios illustrate how the translog’s guarantees, and its tuning knobs, actually matter to real applications built on Elasticsearch.
Financial transaction search platform
A payments company indexing every completed transaction into Elasticsearch for fraud-analyst search tooling chooses request durability without hesitation, since a lost transaction record, even for a handful of seconds around a rare hard crash, would represent an unacceptable gap in a system that regulators and internal auditors expect to be fully, provably complete.
High-volume application log ingestion
An engineering organization ingesting many millions of application log lines per hour into Elasticsearch for observability purposes deliberately chooses asynchronous durability, judging that losing at most a few seconds of the very newest log lines during an exceptionally rare hard crash is an acceptable cost in exchange for meaningfully higher sustained ingestion throughput across their fleet.
E-commerce order-tracking index
An online retailer indexing customer order updates chooses request durability for that specific index, even though other, less critical indices in the same cluster run with asynchronous durability, illustrating that the durability setting is configured per index rather than as one single, cluster-wide, one-size-fits-all choice.
Incident response after an unexpected data-center power event
Following an unplanned power interruption across an entire data center, an operations team observes that every shard recovers cleanly by replaying its translog, with monitoring dashboards confirming zero data loss for every index running under request durability, and a bounded, well-understood, acceptable amount of loss for the small number of indices deliberately running under asynchronous durability, exactly matching what each index’s chosen configuration had promised in advance.
Corrupted disk on an aging node
A team notices a shard failing to start after a node’s disk begins developing hardware errors, and rather than immediately reaching for the translog truncation tool, they first confirm a healthy, fully in-sync replica already exists elsewhere in the cluster, recover the shard cleanly from that replica instead, and retire the failing disk entirely, avoiding any unnecessary discarding of data.
Migrating a write-heavy index onto faster storage
A media streaming company notices unusually high indexing latency on an index tracking viewer session events, and after correlating the slowdown with translog fsync timings, traces the root cause to an aging class of network-attached storage volumes struggling under request durability’s frequent fsync demands. Migrating that specific index onto locally-attached, low-latency storage brings indexing latency back down immediately, illustrating how closely translog performance is tied to raw storage characteristics rather than to any application-level inefficiency.
Bulk-loading a historical archive under relaxed refresh settings
A research organization loading several years of archived sensor readings into a new index temporarily disables automatic refreshing to speed up the bulk load, correctly reasoning that search visibility does not matter until the load finishes. They separately verify, before starting, that disk space and flush-related settings are sized appropriately for the much larger volume of data being written during the load, recognizing that the translog itself continues recording every single operation throughout the entire process regardless of the refresh setting.
18Frequently Asked Questions
Straightforward answers to the questions people most often ask once they first encounter the translog.
19Summary and Key Takeaways
Bringing everything in this guide together into a single, quick reference.
The translog is Elasticsearch’s fast, append-only, per-shard durability record, sitting quietly underneath the far more visible processes of refreshing and flushing. It exists to solve one specific, unavoidable problem: making writes feel fast to a client without ever risking that an acknowledged write could silently vanish because the far slower, more complex Lucene segment it was headed toward had not yet been durably finished. Every core idea in this guide — refresh, flush, durability modes, checkpoints, sequence numbers, and recovery — ultimately traces back to that one central tension between speed and safety, and the translog is Elasticsearch’s carefully engineered answer to it.
Key Takeaways
- The translog is a fast, append-only, per-shard durability record, separate from Lucene’s own segments, recording every write operation the moment it happens.
- It exists to close the gap between a write being acknowledged to a client and that same write being durably represented in Lucene’s own, much slower, segment-based storage.
- Refresh, flush, and translog writes are three genuinely different processes on three different schedules, and confusing them is one of the most common sources of misunderstanding.
- Durability mode is a deliberate, configurable trade-off: request durability guarantees every acknowledged write is immediately physically safe, while asynchronous durability trades a small, bounded risk window for higher throughput.
- The translog is not a backup strategy, and does not protect against the total loss of a node’s disk; that protection comes from replication, working alongside the translog rather than replacing it.
- Sequence numbers and checkpoints, built on top of the translog, are what let Elasticsearch safely and provably promote a replica to primary without silently losing acknowledged data.
- Monitoring translog size and operation count gives an early, honest signal of how long recovery would actually take if a crash happened right now, long before any real incident puts that number to the test.