Two jobs, one goal: keep the database small, fast and healthy.

etcd (pronounced “et-see-dee”) is a small, reliable database that stores key-value pairs. A key is a name, such as the address of a Pod, and a value is the data saved under that name. Kubernetes uses etcd as its single source of truth. If etcd were lost, the cluster would forget every Deployment, Service, Secret and Node it ever knew about.

Because etcd never overwrites data in place, it needs two clean-up operations.

The one-sentence difference

Compaction decides what history to forget. Defragmentation decides how much disk space to give back. Forgetting never gives anything back by itself.

etcd keeps history on purpose

Most databases overwrite a value when it changes. etcd does not. It uses a design called multi-version concurrency control, usually shortened to MVCC. Each change creates a new revision, a number that always goes up. The old value stays readable under its old revision. This is not a flaw. It is what makes two of etcd’s most important abilities possible.

• Watching. A client can say “tell me everything that changed since revision 5000”. etcd can answer only because it still remembers what happened after that point. Kubernetes controllers rely on this all day long.
• Consistent reads. A client can read the state of many keys as they were at one exact revision, so it never sees a half-updated picture.

But history is not free

If nobody ever removes old revisions, the database grows forever. A busy Kubernetes cluster writes constantly: nodes report their health every few seconds, controllers update the status of thousands of objects, and Pods start and stop all day. Each of those writes adds another revision. Even if the number of live objects stays the same, the stored history keeps piling up.

The size limit makes this urgent

etcd has a built-in ceiling on the size of its database file, called the backend quota. The default is two gibibytes (GiB), and the project suggests staying well below eight. When the file reaches the quota, etcd raises an alarm and refuses every write except deletes. For a Kubernetes cluster, refusing writes is a disaster. No new Pods can be created, no status can be updated, and controllers cannot do their jobs. The cluster is effectively frozen in place, although workloads already running keep running.

Why two jobs and not one

Separating the jobs is a deliberate engineering choice. Compaction is cheap and safe to run often, because it only marks data as removable. Defragmentation is heavy: it rewrites the whole file and briefly pauses the member that is doing it. So the first is run constantly and automatically, and the second is run occasionally and with care. Understanding that difference is the heart of this topic.

A short vocabulary makes everything after this much easier.

Key, value and revision

A key is a name and a value is the data stored under it. In Kubernetes, the keys live under a prefix called /registry, for example a key for one Pod in one namespace. A revision is a cluster-wide counter that increases by one with every successful change anywhere in the database. Every version of every key is tagged with the revision at which it was written.

Tombstone

When a key is deleted, etcd does not erase it. It writes a small marker called a tombstone at a new revision. The tombstone says “this key stopped existing here”. Compaction later removes both the tombstone and the old versions behind it.

Backend and bbolt

etcd keeps the actual data in a single file on disk, called the backend. The storage engine that manages the file is a library named bbolt. It organises data in fixed-size blocks called pages and uses a copy-on-write approach: when something changes, it writes new pages and abandons the old ones rather than editing them. This is where fragmentation originates.

Free pages and the freelist

When compaction deletes old revisions, the pages that held them are no longer needed. bbolt records these pages in a list called the freelist and reuses them for future writes. The pages stay inside the file, so the file does not shrink. This is the central fact of the whole article.

Raft, members and leader

A production etcd runs as a group of members, usually three or five. They agree on every change using the Raft consensus protocol, which means a change counts only when a majority of members have recorded it. One member is the leader and handles the ordering of writes. Each member keeps its own copy of the database file, and so each has its own fragmentation. That is why defragmentation is done member by member.

Quota and alarm

The quota is the maximum size of the backend file. The alarm is a flag, named NOSPACE, that etcd raises when the quota is reached. While it is on, the cluster accepts reads and deletes only.

Only one component talks to etcd directly. Seeing this makes it clear who triggers what.

In Kubernetes the kube-apiserver is the only component that reads and writes etcd. The scheduler, the controllers and the kubelets all go through the API server. The API server sends requests to etcd over a secure connection, and etcd replicates each write across its members before confirming.

flowchart TB
  KUBECTL["kubectl, controllers, kubelets"] --> API["kube-apiserver replicas"]
  API -->|"gRPC with TLS"| E1["etcd member 1: leader"]
  API --> E2["etcd member 2"]
  API --> E3["etcd member 3"]
  E1 |"Raft replication"| E2
  E1 |"Raft replication"| E3
  E2 |"Raft replication"| E3
  E1 --> W1["WAL log and bbolt database file"]
  E2 --> W2["WAL log and bbolt database file"]
  E3 --> W3["WAL log and bbolt database file"]
  API -.->|"compact request about every 5 minutes"| E1
  OPS["Operator or automation"] -.->|"defragment one member at a time"| E2
    

Reading the picture

Two different kinds of “log”

Newcomers often confuse compaction with the clean-up of etcd’s write-ahead log. The write-ahead log, or WAL, is a sequential diary of changes used for crash recovery, and etcd trims it automatically by taking periodic snapshots. That is a separate mechanism. The compaction in this article concerns the versions of keys inside the database, not the recovery log.

Follow one key through five changes and watch the revisions pile up.

Suppose the cluster starts empty and the revision counter is at 100. A Deployment’s replica count is stored in a key. Here is what happens over a few minutes.

At revision 106 the Deployment is gone from the point of view of Kubernetes. Yet etcd still holds four versions plus a tombstone for that key. Nothing was lost, and nothing was freed.

What compaction does to this history

Say someone asks etcd to compact at revision 105. etcd then removes every version that is older than revision 105 and is not the newest version at or before it. The rule keeps the latest value as of revision 105, because that is what a reader at revision 105 would see. Older versions are dropped. The result: versions 1, 2 and 3 vanish, version 4 is kept for now, and the tombstone at 106 is newer than the compaction point so it stays. After the next compaction at a later revision, version 4 and the tombstone disappear too, and the key has fully left the database.

Why Kubernetes makes this pile up fast

Every Kubernetes update rewrites the whole object, not just the changed field. A Pod object might be a few kilobytes. Each status change stores another full copy. A cluster with ten thousand Pods, each updated a handful of times during its life, quickly generates millions of stored versions. The next chapters show how compaction removes the ones nobody needs.

Compaction is the everyday, low-risk job. Let us see who starts it, what it removes and what it leaves.

What a compaction request contains

A compaction request carries a single number: a revision. It means “forget all history older than this revision, except the newest version of each key as of that revision”. The request goes through the Raft process like any write, so every member applies it identically. Because it is replicated, you only need to send it once to any member.

Who triggers compaction in a Kubernetes cluster

How it runs without freezing the cluster

Compaction is designed to be gentle. etcd does the physical deletion in small batches with short pauses between them, so that reads and writes keep flowing while the clean-up proceeds in the background. For a database of several gibibytes, a compaction pass often completes in seconds, though a very large backlog can take longer and slightly raise latency while it runs. This is why compaction can safely happen every few minutes, while defragmentation cannot.

What compaction removes and what it keeps

Running a compaction by hand

Sometimes you want to compact yourself, for example during recovery from a full database. The procedure is simple in idea. First, ask any member for its current revision, a number shown in the status output of the command line client. Second, send a compaction request using that number. Because the request is replicated, one request to one healthy member is enough for all of them. Third, check the status again: the in-use size should drop sharply while the file size stays the same. That last observation is a useful lesson in itself, because it proves with your own eyes that compaction alone does not shrink the file.

How much history should be kept?

Keeping very little history keeps the database small, but if a controller restarts and needs to catch up on changes it missed a few minutes ago, there must still be enough history for it to do so. Kubernetes picks a short window because its components keep their own in-memory copies of state, as chapter 12 explains. A window of a few minutes is the default, and very long windows are rarely needed. Longer windows cost disk space in direct proportion to how many writes you make.

This is the idea people struggle with most, so we will take it slowly with pictures.

The page picture

The database file is divided into equal pages, like the pages of a notebook. As revisions are written they fill pages. Compaction removes old data, which empties some pages, but those pages are scattered all over the file between pages that are still in use. bbolt cannot hand back a page from the middle of a file to the operating system. A file is a single continuous object. You can only cut the end off a file, and the end is full of live data.

flowchart LR
  subgraph B["Before compaction: 10 pages, file is 10 pages long"]
    P1["live"] --- P2["old"] --- P3["old"] --- P4["live"] --- P5["old"] --- P6["live"] --- P7["old"] --- P8["old"] --- P9["old"] --- P10["live"]
  end
  subgraph C["After compaction: still 10 pages long"]
    Q1["live"] --- Q2["free"] --- Q3["free"] --- Q4["live"] --- Q5["free"] --- Q6["live"] --- Q7["free"] --- Q8["free"] --- Q9["free"] --- Q10["live"]
  end
  subgraph D["After defragmentation: 4 pages long"]
    R1["live"] --- R2["live"] --- R3["live"] --- R4["live"]
  end
  B --> C --> D
    

What the free pages are good for

The free pages are not wasted. bbolt remembers them in its freelist and uses them for new writes before it grows the file. In a steady cluster that compacts regularly, the file therefore settles at a size that is a little above the live data plus the recent history, and then stops growing. The file size reflects the high-water mark, the largest it ever needed to be, not the amount of live data today.

The life of a page

stateDiagram-v2
  [*] --> LiveData : write creates a new revision
  LiveData --> OldRevision : a newer revision of the key is written
  OldRevision --> FreeInFile : compaction removes it
  FreeInFile --> LiveData : reused by a later write
  FreeInFile --> ReleasedToDisk : defragmentation rewrites the file
  ReleasedToDisk --> [*]
    

So why ever defragment?

If free pages get reused, why not leave the file alone? There are three reasons.

• The quota counts the file, not the live data. A file that grew to its high-water mark during a bad hour still counts that size against the quota forever, even when most pages are free. The cluster can hit the quota with mostly empty pages.
• The file is memory-mapped. etcd maps the database file into memory for fast reads. A bloated file wastes address space and, when it exceeds the available memory, forces the operating system to read from disk more often, which slows reads.
• Backups and restores get bigger. A snapshot copies the file as it is, so a bloated file makes every backup, copy and recovery slower and larger.

Defragmentation is the heavy job. It is safe when done with care, and painful when done carelessly.

What happens during a defragmentation

sequenceDiagram
  autonumber
  participant OP as Operator or tool
  participant M as etcd member
  participant OLD as Old database file
  participant NEW as New temporary file
  participant OS as Operating system
  OP->>M: Request defragmentation of this member only
  M->>M: Pause serving reads and writes
  M->>NEW: Create a new empty database file
  M->>OLD: Read every live key and value
  M->>NEW: Write them out in compact order
  M->>NEW: Flush to disk
  M->>OS: Replace the old file with the new one
  OS-->>M: Old space returned to the filesystem
  M->>M: Resume serving requests
  M-->>OP: Done
    

Reading the steps

How long does it take?

The time depends on the amount of live data and the speed of the disk. On fast solid-state storage it is typically a few seconds for databases of around a gibibyte, and it can grow to a minute or more for large ones. Slow or shared disks stretch the pause much longer. Measure it in a test cluster before running it in production.

Practical tips that save trouble

The leader needs special care

The leader sends regular heartbeats to the followers to prove it is alive. If the leader is paused for defragmentation for longer than the election timeout, the followers may think it has died and start an election. A new leader will be chosen quickly and the cluster will recover, but writes will stall briefly in the meantime. The safe habit is to defragment followers first and the leader last. Some teams move leadership to another member before defragmenting the original leader.

Place the two jobs next to each other and the differences are easy to remember.

What actually happens on the bad day, and how to get out of it.

What the quota does

The quota is a safety brake. etcd is not designed for very large databases, and a runaway database could exhaust the disk and corrupt everything. So when the file grows to the quota, etcd raises the NOSPACE alarm and rejects all writes with an error message that says the database space is exceeded. Deletes still work, because deleting is how you recover.

What the cluster looks like when the alarm is on

The recovery path

flowchart TD
  A["Writes fail: database space exceeded"] --> B["Confirm the NOSPACE alarm is active"]
  B --> C["Find the current revision"]
  C --> D["Compact up to that revision"]
  D --> E["Defragment each member, one at a time"]
  E --> F["Disarm the alarm"]
  F --> G["Verify that writes succeed again"]
  G --> H["Find the root cause of the growth"]
  H --> I["Raise the quota only if the need is real"]
    

Why Kubernetes feels so broken when this happens

The NOSPACE alarm is dramatic because so much of Kubernetes’ normal operation is writing. Node heartbeats stop being recorded, so after a while the control plane may start marking nodes as unreachable. Controllers that use leader election cannot renew their leases, so they may give up leadership and stop working. The scheduler cannot record where it placed Pods. The kubelets cannot report status. The API server itself keeps answering reads, which is a trap for the unwary, because a quick look at the cluster can seem fine. Only attempts to change anything reveal the problem.

This is also why the alarm is better treated as a rehearsed emergency than as a surprise. Teams that have written the recovery steps in their runbook, and have tried them once in a test cluster, usually restore writes in minutes. Teams that discover the steps during the outage often spend hours.

Why every step matters

• Compact first so that old history becomes free pages. Without this, defragmentation has nothing to reclaim.
• Defragment second so that the file really shrinks below the quota. If you skip this, the file is still at the quota and the alarm will return immediately.
• Disarm third. The alarm does not clear itself, even when space is available. Until you disarm it, etcd keeps refusing writes.
• Investigate last. Something wrote far more than usual. Finding it is what stops a repeat.

Follow one update from the moment you press Enter until the disk space it used is finally given back.

Where the time goes

Compaction is invisible most of the time. Here is where it can touch the rest of the cluster.

Resource versions are revisions in disguise

Every Kubernetes object carries a field called resourceVersion. It is derived from the etcd revision at which the object last changed. When a controller says “watch Pods starting from this resource version”, the API server turns that into “watch from this etcd revision”.

The watch cache shields most clients

To avoid hitting etcd for every request, the API server keeps a watch cache, an in-memory copy of recent changes for each resource type. Almost all controller and kubelet watches are served from this cache and never reach etcd directly. The cache keeps a window of recent events, so a client that disconnects briefly can resume without any help from etcd history.

When a client falls too far behind

If a client asks to resume from a resource version that is older than both the watch cache window and the compaction point, the API server cannot supply the missing changes. It answers with an HTTP 410 response meaning the resource version is too old. Well-behaved clients, such as the informers used by controllers, react by doing a fresh full list of the resource and then starting a new watch from there. This is called a relist. It is a normal, designed-for event and not a bug.

sequenceDiagram
  autonumber
  participant C as Controller informer
  participant A as kube-apiserver
  participant E as etcd
  C->>A: Watch from resource version 5000
  A->>E: Watch from revision 5000
  E-->>A: Error: revision has been compacted
  A-->>C: 410 Gone: resource version too old
  C->>A: List all objects of this type
  A->>E: Read current state
  E-->>A: Current objects and latest revision
  A-->>C: Full list with new resource version
  C->>A: Start a new watch from the new version
    

When this becomes a problem

• Many clients relist at once. After a long etcd pause, such as a slow defragmentation or a leader election, hundreds of controllers and thousands of kubelets may fall behind together and relist simultaneously. The resulting burst of large list calls can load the API server heavily, a pattern sometimes called a thundering herd.
• Very aggressive compaction. Keeping almost no history leaves little room for slow clients to catch up.
• Overloaded or slow components. A controller that is starved of CPU may fall behind the window repeatedly and relist over and over.

Finding the biggest source of writes is usually more valuable than any amount of tuning.

How to find the biggest writers

Count objects by type under the Kubernetes prefix, and look at the API server’s request metrics broken down by resource and verb. The resource types at the top of the write counts are your suspects. Another useful approach is to list the largest keys. A handful of runaway objects is a very common cause of surprising growth.

A quick habit: ask “what changed?”

When growth suddenly speeds up, the most useful question is not “how do I clean this up?” but “what changed recently?”. Think through the likely suspects in order. Was a new controller or operator installed? Did a deployment tool start recording more history? Did a workload begin crash-looping and flooding Events? Was a very large ConfigMap or custom resource added and updated often? Did the cluster grow in node count, multiplying heartbeats? Teams that make a habit of asking this question, and of keeping a short log of significant changes to the cluster, find the answer in minutes instead of hours.

Separating the noisy tenants

Kubernetes allows certain resource types to be stored in a different etcd cluster from the rest. The official documentation for large clusters recommends storing Events in a separate, dedicated etcd instance, using an API server setting that overrides the etcd servers for a given resource. The very high write rate of Events then cannot slow down or bloat the main store that holds the objects your cluster truly depends on. Several large operators have described using this approach to keep their primary etcd healthy at scale.

Choose the quota on purpose

The default quota of two gibibytes is comfortable for small clusters and tight for large ones. Many production clusters raise it, with eight gibibytes being the figure the etcd project suggests as a practical upper limit. Beyond that, performance and recovery times degrade, and it is a sign that the cluster should be split or its noisy resource types separated. Set the quota high enough that normal growth plus a bad day does not hit it, and low enough that a runaway process is stopped before it harms the disk.

The memory-mapped file

etcd maps its database file into memory. For best read performance, the machine should have enough free memory to hold the whole file, in addition to what etcd itself needs. If the file is larger than the available memory, the operating system must fetch pages from disk during reads, and latency rises. This is a practical reason to keep the file compact and an argument for regular defragmentation even when the quota is far away.

Disk matters more than CPU

etcd waits for the disk on every write, because it must flush the write-ahead log before confirming. Slow disks cause slow writes, missed heartbeats and leader changes. Use dedicated solid-state storage with low and steady latency, and avoid sharing the disk with other busy workloads. Leave generous free space: the data file, the logs and the temporary file created during defragmentation all live on the volume.

A repeatable routine turns a frightening operation into a boring one.

When to defragment

Do not defragment on a fixed calendar alone. Base the decision on two numbers: the size of the file and the amount of it that is actually in use. A common rule of thumb is to act when the file is large in absolute terms, for example above a few hundred megabytes, and a large share of it, such as half or more, is free. The standard monitoring rules published with etcd raise a warning on very similar conditions, which makes them a good starting point.

The routine, step by step

Tools and automation

Quorum is the number to protect

A three-member cluster needs two members to agree on any write. While you defragment one member, two remain, which is exactly enough. If another member fails or is slow at the same time, the cluster loses its majority and stops accepting writes. For this reason a three-member cluster has no room for a second problem during maintenance. Five members tolerate one planned pause plus one unexpected failure, which is one reason larger or more critical clusters use five.

What can go wrong, and what to do

Backups and restores

A snapshot backup is a copy of the database at one moment. It is also a handy way to understand fragmentation, because restoring a snapshot creates a fresh file that contains only live data, in effect a defragmented database. Test restoring from your backups regularly, in a separate environment, so that you know it works and how long it takes. A backup that has never been restored is only a hope.

How a healthy or neglected database changes the speed of everything built on it.

How fragmentation hurts

A fragmented file is slower in several quiet ways. It spreads live data over more pages, so reads touch more of the disk and more of memory. It enlarges snapshots, which slows the process of bringing a new or recovering member up to date, because the leader has to send a bigger file. And it pushes the file towards the memory limit of the machine, after which the operating system starts to swap pages in and out.

How the maintenance jobs themselves cost performance

Reading the effect in your own graphs

After a defragmentation, three graphs should tell a clear story. The file size line drops sharply to close to the in-use line. The request latency line shows a small bump at the moment the member paused and then returns to its usual level. The leader change counter stays flat unless you defragmented the leader. If instead the latency stays high, or leader changes keep happening, something else is wrong, most likely a slow disk or a network problem between members, and the maintenance simply made it visible.

Scaling guidance

• Keep compaction frequent and automatic so that each pass is small and quick.
• Defragment only when the numbers say it will help, and run it in a quiet window.
• Reduce the cause of growth first, since every avoided write is cheaper than any clean-up.
• Split noisy resource types, especially Events, into their own etcd when the main one becomes crowded.
• At very large scale, consider several smaller clusters rather than one enormous one with a huge database.

etcd holds the keys to the kingdom, so every maintenance action must respect that.

Why access to etcd is access to everything

Everything in the cluster, including every Secret, lives in etcd. Anyone who can read etcd directly can read all of it and bypass the API server’s login and permission checks entirely. Anyone who can write to it can change the cluster without leaving a trace in the API audit logs. For this reason etcd must be reachable only from the control plane, with mutual certificate authentication between the API server and etcd, and between etcd members.

Security points specific to maintenance

Least privilege for automation

A scheduled defragmentation job needs a client certificate that can reach etcd. Give that job its own certificate, narrow in purpose and short in lifetime where possible. Run it from a control plane node or a locked-down pod, never from a developer laptop. Log every run, so that any maintenance action on the cluster’s most sensitive data store is traceable.

The signals that matter

etcd exposes metrics in a format that monitoring systems such as Prometheus can collect. The exact metric names can differ a little between versions, but the ideas are stable.

A calm path from symptom to cause

flowchart TD
  A["Symptom: slow API or failing writes"] --> B{"Do writes fail with space exceeded?"}
  B -->|"Yes"| C["Follow the NOSPACE recovery path"]
  B -->|"No"| D{"Is the file far larger than the in-use size?"}
  D -->|"Yes"| E["Plan defragmentation, one member at a time"]
  D -->|"No"| F{"Is the in-use size itself growing fast?"}
  F -->|"Yes"| G["Find the heaviest writers: events, leases, status churn, large objects"]
  F -->|"No"| H{"Are disk sync or commit times high?"}
  H -->|"Yes"| I["Fix the slow or shared disk"]
  H -->|"No"| J["Check leader changes, network and API server load"]
    

Reading common messages

Logging advice

etcd logs compaction and defragmentation events with timings. Keep these logs, because they are the best record of how long maintenance really took and of whether a slow period matches a maintenance window. Send them to a central place that survives the loss of the machine itself.

Patterns that work

Anti-patterns to avoid

Best practices

Common mistakes

• Treating compaction and defragmentation as the same thing. They are not. One forgets, the other reclaims.
• Judging health by file size only. Compare it with in-use size.
• Running defragmentation through a load balancer. It may hit the wrong member or several at once.
• Starting without enough free disk. The temporary file needs room.
• Ignoring slow disks. Many “etcd problems” are disk problems.
• Letting Events share the main etcd in a very busy cluster. They are the loudest writer in most clusters.
• Skipping the backup because the operation is “routine”. Routine operations are where rare failures hurt most.

Let us walk through a realistic story from calm, to trouble, to repair. The cluster here is an illustration, not a specific company.

The cluster

A three-member etcd cluster serves a mid-sized Kubernetes cluster. The quota is set to 8 GiB. The API server compacts every five minutes. Nobody has defragmented for a year.

What the numbers mean

The same two jobs appear across the industry, from managed services to the largest self-run clusters.

To truly understand why files do not shrink, it helps to look one level below etcd at the storage library it relies on.

A tree of pages in one file

bbolt stores data as a B+tree, a tree-shaped structure that keeps keys in sorted order so that lookups and range reads are fast. The tree is built out of fixed-size pages, usually the same size as a page of the operating system, commonly four kibibytes. Small pages near the top of the tree point to pages further down, and the actual data sits in the leaf pages at the bottom. Everything lives inside one single file.

Copy-on-write: never edit, always replace

When a value changes, bbolt does not edit the page in place. It writes a new copy of the changed page, and of every page above it in the tree that needs to point to the new copy, and then switches the root to the new version. The old pages are not erased. They are marked as free once no reader can still be using them. This is called copy-on-write, and it has a great benefit: a crash in the middle of a write cannot damage the existing tree, because the old version is still intact until the switch is complete. That safety is the reason etcd uses it.

One writer, many readers

bbolt allows a single writer at a time and any number of readers. Readers see a consistent picture of the tree as it was when they began, and are never blocked by the writer. The pages that a long-running reader might still be looking at cannot be reused yet. In practice this means a very long read can delay the reuse of free pages, which is one more reason to avoid holding enormous reads open in etcd.

How etcd lays its data on top

etcd does not use the user’s key directly as the tree key. Inside the main bucket, the tree key is the revision, and the value holds the user key together with the data. Separately, etcd keeps an in-memory index that maps each user key to the list of revisions at which it has existed. This design explains several behaviours.

• Writing is fast, because new revisions are always larger and are appended at the right-hand end of the tree.
• Reading the latest value of a key means looking up the key in the in-memory index, finding its newest revision, and fetching that revision from the tree.
• Compaction has two parts: pruning the in-memory index, and deleting the old revision entries from the tree. Only the second part creates free pages.
• A key’s life from creation to deletion is called a generation, and compaction can drop whole generations of keys that have been deleted.

Batching writes to disk

To avoid flushing to the disk after every single change, etcd groups writes and commits them to the file in batches, roughly ten times a second or whenever a batch limit is reached. The durability of an individual write is already guaranteed by the write-ahead log. The batch commit is what turns many small changes into one disk operation. The time this commit takes appears in the backend commit duration metric and is a good indicator of storage health.

Why the file cannot give pages back

The file is also memory-mapped, which means a region of the process’s address space mirrors the file. Free pages sit scattered through that region. Returning a middle page to the operating system would punch a hole in a structure that the engine assumes is contiguous. The designers chose simplicity and safety: the file only grows, free pages are tracked in a list, and the only way to compact the physical layout is to build a fresh file. That is precisely what defragmentation does.

Compaction looks simple, but it must give the same answer on every member. Raft is how that is guaranteed.

One log, applied in the same order everywhere

Raft keeps an ordered log of operations. A write is proposed to the leader, copied to a majority of members, and then applied to each member’s database in exactly the same order. Because every member applies the same operations in the same order, they all arrive at the same revision numbers. Revision 5000 means the same change on every member. That determinism is what makes revisions a safe thing for clients to rely on, even when they switch between members.

Compaction travels through the same log

A compaction request is also an operation in the log. When it reaches a member’s turn to be applied, the member removes the old revisions up to the chosen point. Since every member does it at the same logical position in the log, every member forgets exactly the same history. If compaction were applied independently on each member at different times, a client could be told “your revision is fine” by one member and “your revision is gone” by another. Replication avoids that confusion.

Defragmentation stays outside the log

Defragmentation changes only the physical layout of one member’s file. The logical contents are identical before and after, so there is nothing for the other members to agree about. It is therefore performed locally and is not part of the log. This is why you must repeat it on every member, and why a successful defragmentation of one member says nothing about the others.

flowchart LR
  REQ["Compact request for revision N"] --> L["Leader appends to Raft log"]
  L --> F1["Follower 1 receives entry"]
  L --> F2["Follower 2 receives entry"]
  F1 -->|"acknowledge"| L
  F2 -->|"acknowledge"| L
  L -->|"majority reached"| A0["Leader applies compaction"]
  F1 --> A1["Follower 1 applies compaction"]
  F2 --> A2["Follower 2 applies compaction"]
  DF["Defragmentation request"] -.->|"local only, not in the log"| A1
    

What happens to a member that falls behind

A member that is paused, for example during defragmentation, falls behind on applying new log entries. When it resumes, the leader either sends the missing entries or, if the member is too far behind for the leader to still hold them, sends a full snapshot. A snapshot transfer is slower and larger when the database file is bloated. A neat side effect is that a member rebuilt from a snapshot ends up with a compact file, because it starts fresh from live data.

Why keeping members similar helps

If one member has a much bigger file than the others, it also takes longer to defragment, longer to restore and longer to ship in a snapshot. It is wise to treat all members the same way in your maintenance plan, even though only one at a time is paused.

Several settings influence this topic. Knowing which component owns each one prevents contradictory choices.

Two auto-compaction modes of etcd

Choosing between them

In a Kubernetes cluster the API server already provides periodic-style compaction by default, so the simplest choice is to leave etcd’s own auto-compaction off and let the API server own the job. In an etcd cluster used by other applications as well, or where the API server’s compaction has been turned off, etcd’s own setting is the right place. Whichever you pick, write it down in your runbook so that the next engineer does not enable a second compactor and wonder about the extra activity.

Offline defragmentation

Besides the online command, newer etcd releases include a separate utility meant for operations on files while the member is stopped. It can defragment a data directory offline. This is useful when a member is so full that it cannot start or cannot serve the online command, or when you prefer to perform a long rewrite while the member is out of the cluster. Stop the member cleanly, run the offline step against its data directory, and then start it again so that it catches up from its peers.

Restoring from a snapshot

A snapshot restore builds a new data directory from the saved file. The result is a fresh, compact database. It also starts a new cluster identity, so restoring a whole control plane means restoring every member from the same snapshot, then restarting the API servers so that they reconnect. Two consequences are worth knowing.

• Revisions can go backwards. The restored database continues from the revision at the time of the snapshot, which may be lower than revisions that clients saw before the failure. Some clients then hold resource versions from the “future”. Newer etcd tools can raise the revision after a restore so that this does not confuse clients, and following current guidance for your version is wise.
• Caches must be refreshed. The API server and its clients should relist after a restore. Restarting the control plane components achieves this.

Upgrades

Upgrading etcd is done one member at a time, like defragmentation, and for the same reason: quorum must stay. Do not combine an upgrade with a defragmentation of the same member in the same window. Separate changes make a problem easy to attribute. After a successful upgrade, there is normally no need to defragment unless the numbers call for it.

Disaster recovery planning

Many clusters, one routine

Organisations with dozens of clusters cannot maintain each one by hand. They package the runbook as automation that reads health and fragmentation, acts only when thresholds are met, staggers work so that not all clusters pause on the same minute, and reports results to a central place. The principles in this guide are the same at any scale. Only the amount of repetition changes.

Keeping etcd lean is not only about safety. It also saves money and time.

What a bloated file really costs

A simple calculation

Suppose a cluster keeps its snapshots for thirty days, taking one every hour. That is about seven hundred and twenty snapshots. With a bloated file of six gibibytes, the archive holds more than four terabytes. With a lean file of one and a half gibibytes, the same archive is about a terabyte. Even before counting transfer costs between sites, the saving is substantial. Compression helps both cases, but it cannot hide a four-to-one difference in the raw size.

The price of the maintenance itself

Defragmentation costs a few minutes of an engineer’s attention, or nothing at all once automated, and a few seconds of one member’s availability. Compared with the costs above, and with the cost of even a short outage caused by the quota, it is among the cheapest forms of reliability work available to a platform team.

What not to optimise

• Do not shorten the compaction interval to extremes to save a few megabytes. Clients need a little history to catch up.
• Do not run defragmentation on a small, healthy file just to see the numbers fall. It adds risk for no benefit.
• Do not spend weeks tuning flags when a single misbehaving controller is the real source of the growth.

Because compaction frees space only inside the file. The file keeps its high-water mark size. Defragmentation is the only normal way to return that space to the disk and bring the file back under the quota margin.

No. It removes only old versions and deleted history. The latest value of every live key is always kept.

No. It copies all live data into a new file and swaps it in. Take a backup first anyway, as with any maintenance.

Running workloads are not affected. The control plane may see brief slowness while one member is paused. Done one member at a time on a healthy cluster, applications normally notice nothing.

As often as the measurements say. When the file is large and a big share of it is free, defragment. In a well-behaved cluster that may be rarely.

Each member keeps its own file, shaped by its own history of writes, restarts and defragmentations. Differences are normal.

The write-ahead log is the recovery diary, trimmed by snapshots. Compaction removes old key versions inside the database. They are separate mechanisms with separate purposes.

You can, within the supported range, but it hides the symptom rather than fixing it. Larger files are slower to back up, restore and map into memory.

The NOSPACE alarm does not clear itself. You must disarm it explicitly after compaction and defragmentation, then confirm that a write succeeds.

Not by itself. A delete adds a new revision. The space is only freed inside the file after compaction and returned to the disk after defragmentation.

Usually no, since the provider operates etcd. Your best contribution is to avoid runaway writes and very large objects, which reduces load on the shared design limits.

Compaction is. Defragmentation is usually safe on a healthy cluster with proper sequencing, but a quiet window is wiser, particularly for the leader.