etcd is a small, strongly consistent database that remembers the truth about your cluster.

etcd is a distributed key-value store. A key-value store is the simplest kind of database: you save a value under a name, called a key, and later you ask for it by that name. Think of a coat check where you hand over a coat and receive a ticket number. The word “distributed” means that etcd runs as a group of several programs on several machines, and all of them keep identical copies of the data. If one machine dies, the others still have everything.

The most important word to remember is consistent. When etcd says a write succeeded, every healthy member will agree on that fact, and no later read will show older data. Kubernetes depends on this promise because many controllers make decisions from the data they read. If two controllers saw different versions of the truth, they could create duplicates or delete the wrong things.

Why tuning is not optional at scale

A small test cluster with a handful of nodes will run happily on almost any etcd setup. The trouble appears as the cluster grows. More nodes mean more heartbeats. More Pods mean more status updates. More controllers mean more watches. Each of these becomes a write or a read in etcd. When the number of requests rises faster than etcd can answer them, delays pile up and the control plane starts to wobble.

One database, one gatekeeper, and a lot of people who depend on both.

In a Kubernetes cluster, only one component is allowed to talk to etcd: the kube-apiserver. Every other part of the system, including the scheduler, the controller manager, the kubelets on each node, and your own kubectl commands, talks to the API server instead. The API server checks who you are, validates the request, and then reads from or writes to etcd on your behalf.

This design has an important effect. The speed of every operation in Kubernetes is at most the speed of the API server, and the speed of the API server is at most the speed of etcd for anything that has to be stored. When you create a Deployment, the API server writes one object to etcd. The Deployment controller notices and writes a ReplicaSet. The ReplicaSet controller writes Pods. The scheduler writes a binding for each Pod. The kubelet writes Pod status. A single “scale up” command can therefore cause dozens or hundreds of etcd writes within seconds.

What happens when etcd is slow

Slow etcd does not usually look like an etcd problem. It looks like many other problems. Kubectl commands hang. Deployments roll out slowly. Pods stay in Pending even though nodes are free. Nodes flip between Ready and NotReady because their heartbeats cannot be saved in time. Controllers log errors about timeouts. People then waste hours investigating nodes and networks, when the real cause was a slow disk under one etcd member.

Twelve short words that you will meet in every etcd conversation.

Latency versus throughput

Two measurements appear again and again. Latency is how long one request takes from start to finish. Throughput is how many requests are handled per second. They are related but different. A road can have a high speed limit (low latency) and still be narrow (low throughput). etcd cares mostly about latency, because each write must wait for disks on several machines. Raising latency even a little can drop throughput a lot.

A small group of members behind a group of API servers.

flowchart TB
  KC["kubectl and controllers"] --> LB["Load balancer"]
  LB --> A1["kube-apiserver 1"]
  LB --> A2["kube-apiserver 2"]
  LB --> A3["kube-apiserver 3"]
  A1 --> E1[("etcd member 1 - leader")]
  A2 --> E2[("etcd member 2")]
  A3 --> E3[("etcd member 3")]
  E1 |"Raft peer traffic"| E2
  E2 |"Raft peer traffic"| E3
  E1 |"Raft peer traffic"| E3
  A1 -.->|"events only"| EV[("separate etcd for events")]
    

Look at Figure 1 from the top. People and controllers send requests to a load balancer, which hands them to one of several API servers. Each API server speaks to etcd. The etcd members continuously talk to each other to agree on every change. The dotted line at the bottom shows an optional trick that you will study later: sending the very noisy Event objects to their own etcd cluster.

Two ways to deploy etcd

Follow one write from the API server to success, and you will see exactly where time is spent.

sequenceDiagram
  participant C as kube-apiserver
  participant L as Leader member
  participant F1 as Follower 1
  participant F2 as Follower 2
  C->>L: Write request
  L->>L: Append to WAL and fsync
  L->>F1: Replicate log entry
  L->>F2: Replicate log entry
  F1->>F1: Append to WAL and fsync
  F1-->>L: Acknowledge
  Note over L: Majority reached so the entry is committed
  L->>L: Apply to database
  L-->>C: Success
    

Why quorum shapes your cluster size

A cluster of three members needs two for quorum, so it survives one failure. A cluster of five needs three, so it survives two. A cluster of four also needs three, so it survives only one failure, the same as three members but with more cost. This is why even numbers are wasteful. It is also why adding many members does not make writes faster. More members mean more copies to update and more messages to send, so very large groups are slower, not faster.

Understanding how data is stored on disk explains almost every size and space problem you will ever see.

flowchart LR
  W["Write arrives"] --> WAL["WAL file with fsync"]
  WAL --> RAFT["Raft commit by majority"]
  RAFT --> MV["MVCC adds a new revision to the in-memory index"]
  MV --> BB["bbolt database file with batched commits"]
  BB --> CP["Compaction drops old revisions"]
  CP --> DF["Defrag returns free space to the disk"]
    

Two places on disk

Each member keeps two main things on disk. The first is the write-ahead log, which records changes in order and is read again only if the member restarts. The second is the database file, which holds the current and past versions of every key. The database is built on a small embedded storage engine called bbolt, which stores data in fixed-size pages and is organized as a tree. Periodic snapshots let etcd trim old log files, so the log does not grow forever.

Every change creates a new version

etcd uses a design called MVCC, short for multi-version concurrency control. In plain words: when you change a value, etcd does not overwrite the old one. It adds a new version with a higher revision number and keeps the old version for a while. This is how watches work. A client can say “tell me everything that changed since revision 5000,” and etcd can replay the history. It is also why reads do not block writes.

Compaction and defragmentation are different jobs

Compaction removes old revisions logically. After compaction, the old data is gone from etcd’s view, but the database file does not shrink. The pages that held the old data are simply marked as free inside the file. Defragmentation is what actually rewrites the file tightly and hands the unused space back to the operating system. Without compaction, the data keeps growing. Without defragmentation, the file stays large even though much of it is empty.

The space quota: a safety fuse

etcd has a limit on the database file size, called the backend quota. By default it is two gibibytes, and the documentation advises not to go far beyond eight. When the file reaches the limit, etcd raises an alarm called NOSPACE and refuses further writes. Reads still work, so the cluster looks alive but frozen: nothing new can be created or changed. This is a classic outage, and it is entirely preventable by watching size and keeping compaction and defragmentation healthy.

If you only fix one thing, fix the disk.

As you saw in Chapter 5, every write needs an fsync on the leader and on at least one follower. If the fsync takes fifty milliseconds, no amount of CPU or clever settings can make a write complete faster than that. The etcd documentation gives a clear target: the 99th percentile of the WAL fsync time should stay below ten milliseconds, and the 99th percentile of the backend commit time should stay below twenty-five milliseconds. A 99th percentile means “ninety-nine out of a hundred operations were faster than this,” so it shows the slow tail, which is what hurts users.

Test the disk before you trust it

Do not guess. A common practice is to run a short test with a disk benchmarking tool such as fio, configured to mimic etcd by writing small blocks and calling fdatasync after each one. The result you want to see is a 99th percentile sync time under ten milliseconds. etcd also ships with a built-in performance check command that sends a standard load and reports pass or fail. Run the test on every new machine type before you put a cluster on it, and again after any storage change.

What a bad disk looks like in the logs

When a disk is too slow, etcd tells you. The logs fill with warnings such as “apply request took too long,” which appears when applying a request takes more than about a hundred milliseconds, and warnings that the leader failed to send out a heartbeat on time. Metrics show the WAL fsync histogram shifting to the right. If you see these, check the disk first before touching any setting.

The members must hear each other quickly and reliably, or they start electing new leaders for no reason.

The leader sends a short message to every follower at regular intervals to say, “I am still here.” This is the heartbeat interval. If a follower hears nothing for a period called the election timeout, it assumes the leader is dead and starts an election to choose a new one. The defaults are a heartbeat every hundred milliseconds and an election timeout of one thousand milliseconds, which suit members that are close together in one region.

Rules of thumb from the etcd documentation

Why leader changes hurt

While an election is running, no writes can be accepted. A single election lasts about the length of the election timeout. If members keep losing contact because of a congested network or a stalled disk, they may elect again and again, and the cluster spends more time choosing leaders than doing work. The metric that counts leader changes should be almost flat. A leader change every few minutes is a serious warning sign, and it is almost never fixed by simply raising the timeouts, because the true cause is usually a slow disk or a busy CPU.

Where to place the members

Members should be on a fast, low-latency network. Putting the three members in three availability zones of one cloud region is the usual balance: the zones are separate buildings, so one fire does not take everything, yet the round trip time is only a few milliseconds. Spreading members across regions, often hundreds of milliseconds apart, makes every write wait for that long, and is rarely a good idea for Kubernetes. If you need multi-region safety, use backups and a recovery plan rather than a stretched cluster.

etcd is not hungry for CPU, but it hates being starved.

For small and medium clusters, etcd needs only a few CPU cores. The important point is not raw power but steadiness. If another program on the machine grabs all the CPU for a moment, etcd cannot send its heartbeat on time, and followers may decide the leader has died. The etcd hardware guidance ranges from around two cores and eight gigabytes of memory for small clusters up to sixteen or more cores and sixty-four gigabytes for very large ones, with disk speed rising along with it.

Memory and the watch load

A watch is a long-lived connection where a client asks to be told about changes. In Kubernetes, the API server keeps its own cache and opens a small number of watches to etcd, then shares those updates with many clients. This design protects etcd from thousands of separate watchers. Still, a very large number of resource types or poorly written custom controllers can increase the watch load and the memory used per watcher.

How many members, where they live, and when to split the data across clusters.

Separate etcd for Events

Kubernetes creates an Event object whenever something notable happens, such as a Pod being scheduled or an image being pulled. Events are numerous, short-lived, and not critical to the cluster’s survival. By default they live in the same etcd as everything else, with a lifetime of one hour. In a large cluster, they can account for a big share of writes. A well-known tuning step is to configure the API server to send Events to a different etcd cluster. This keeps the heavy traffic away from the important data, so a flood of Events cannot slow down Deployment updates.

Learners: adding members safely

Adding a new member to a live cluster is risky if it is far behind, because it can slow down the quorum. etcd supports a learner role: a new member first joins as a non-voting learner, catches up with all the data, and is promoted to a full voting member only once it is ready. Using learners makes replacement and growth much safer. Always change membership one member at a time.

When one etcd is not enough

A single etcd cluster has natural limits on how much it can write per second and how large its database should be. The Kubernetes project documents clusters of up to about five thousand nodes as the supported scale. Beyond that, teams either split a workload across many smaller clusters, which is the most common and safest answer, or they move specific hot resources such as Events to their own etcd. A few very large operators have gone further and replaced etcd with other storage systems underneath the API server.

A small database is a fast database. Here is the routine that keeps it that way.

A big database slows etcd in several ways. Reads that scan many keys take longer. Snapshots and restores take longer. Followers that fall behind take longer to catch up. Starting a member takes longer. And as the file approaches the quota, the risk of an outage grows. The goal is therefore to keep the file comfortably below the quota, with plenty of safety room.

Two numbers to watch

etcd reports two sizes. The first is the total size of the database file on disk. The second is the size in use, which is how much of the file actually holds live data. When the total is much bigger than the size in use, the file is full of empty pages, and defragmentation will recover that space. When both are large, the problem is real data, and no cleanup will help. You must reduce what is stored.

flowchart TD
  S["Check total size versus size in use"] --> Q{"More than half of the file is wasted?"}
  Q -->|"No"| N["Do nothing"]
  Q -->|"Yes"| C["Confirm compaction has run"]
  C --> F["Defrag one follower"]
  F --> H["Wait until it is healthy"]
  H --> M{"More followers?"}
  M -->|"Yes"| F
  M -->|"No"| L["Defrag the leader last"]
  L --> D["Clear the NOSPACE alarm if it was set"]
    

Raising the quota carefully

If the database legitimately needs more than two gibibytes, you can raise the quota setting. The etcd documentation suggests staying at or below about eight gibibytes. A bigger limit is not a free gift. It also raises recovery time, memory use, and snapshot size. Raising the quota should always come with an investigation into why the database is large.

The cheapest request is the one you never make. Most real tuning happens on the Kubernetes side.

It is tempting to treat etcd as a fixed machine and only tune the machine. But in practice, the biggest improvements come from sending fewer and smaller requests to it. The API server and the programs that talk to it decide the load, so changing their behavior often beats any etcd setting.

Finding the heavy hitters

The API server publishes a metric showing how many objects of each resource type are stored. Sort it, and the biggest types stand out at once: often Events, Secrets, ConfigMaps, Pods in a busy batch cluster, or a custom resource created by some operator. Another set of metrics shows request rates per resource and per verb, such as list, watch, and update. Together they answer the question “who is using etcd, and for what?”

Read types and why they matter

etcd offers two kinds of reads. A linearizable read, which is the default, guarantees the newest data by checking with the leader and the quorum. A serializable read is answered by one member from what it already knows, which is faster but might be slightly behind. The Kubernetes API server uses the first kind when a client asks for the latest data, and relies on its own cache for most other requests. Programs that bypass the cache with frequent full list calls can therefore create heavy quorum read load.

A few guard rails keep one bad request from hurting everyone.

etcd has built-in limits on the size of a single request. The default maximum request size is about one and a half mebibytes. A request larger than that is rejected. This protects the cluster, because a single huge write would have to be copied to every member and could block the leader for a noticeable time. Raising this limit makes tail latency worse, so it is best left alone and the oversized objects reduced instead.

API Priority and Fairness in plain words

The API server groups requests into priority levels and gives each level a share of the available capacity. Requests from system components such as the kubelet and the scheduler can be protected from requests from a user script that lists everything in a loop. If one flow of requests is too heavy, the API server queues or rejects it, instead of letting it flood etcd. Tuning these rules is a way of protecting etcd indirectly.

You cannot tune what you cannot see. These are the numbers worth putting on a dashboard.

etcd publishes its metrics in a format that Prometheus can collect. The list is long, but a small set tells you almost everything. Group them into four questions: is the disk fast, is the network stable, is the database healthy, and are the clients behaving?

Alerts that earn their keep

Logs that matter

A few log messages deserve special attention. A line saying a request took too long means the disk or CPU is slow. A message about failing to send a heartbeat on time means the leader is overloaded or the network is delayed. A message about the database space being exceeded means the quota has been hit. Keep etcd logs at normal verbosity and send them to central storage, because the clues often disappear after a restart.

A calm order of questions that finds most causes in minutes.

flowchart TD
  A["API server slow or timing out"] --> B{"WAL fsync p99 above 10 ms?"}
  B -->|"Yes"| B1["Fix the disk: dedicated NVMe, remove noisy neighbors"]
  B -->|"No"| C{"Leader changes frequent?"}
  C -->|"Yes"| C1["Check network delay, CPU starvation and timing settings"]
  C -->|"No"| D{"Database size near quota?"}
  D -->|"Yes"| D1["Compact, defrag, and find the big objects"]
  D -->|"No"| E{"Slow large list calls?"}
  E -->|"Yes"| E1["Use paging and the watch cache, reduce object counts"]
  E -->|"No"| F["Check CPU, memory and noisy clients"]
    

Plan for the worst day while the cluster is healthy.

etcd was designed to survive failures. A three-member cluster keeps working if any one member dies, and a five-member cluster keeps working if any two die. The system elects a new leader if the old one disappears, and writes continue as soon as a quorum is available. What it cannot survive is the loss of a majority. At that point writes stop, because no one can guarantee agreement, and only a recovery from backup or the return of the missing members brings the cluster back.

Backups are not optional

Replication protects against machine failure but not against mistakes. If someone deletes a namespace by accident, all three members faithfully delete it too. Only a backup can bring it back. The standard approach is to take regular snapshots with etcd’s own tool, store them off the machine, and test restoring them. A backup you have never restored is only a hope. A restore builds a fresh cluster from the snapshot, so it has a new identity and every control plane component must be pointed at it.

etcd holds every secret in the cluster, so access to it is access to everything.

Anyone who can read etcd directly can read all Secrets, and anyone who can write to it can rewrite the cluster. That is why the access model is simple and strict: only the API servers should be able to connect to the client port, and only the other etcd members to the peer port. Everything else should be blocked by network rules.

The performance cost of security is small

TLS adds a little CPU work and a small delay per connection, but the API server keeps a small number of long-lived connections to etcd, so the cost is spread thin. Encryption at rest happens in the API server, not in etcd, and increases object size slightly. The cost of skipping these protections, on the other hand, is the exposure of every password and key in the cluster. There is no good performance argument for leaving etcd open.

A boring, repeatable routine is the best performance feature.

etcd supports rolling upgrades: you upgrade one member at a time, waiting for it to rejoin and catch up before touching the next. Upgrades should move one minor version at a time and follow the release notes, because defaults and behaviors do change between versions. Always take a fresh snapshot first, and always upgrade followers before the leader.

Changing one thing at a time

When tuning, change a single setting, measure for long enough to see a pattern, and write down the result. If you change five things together and the cluster improves, you will not know why. If it gets worse, you will not know what to undo. Keep a simple log of every change and its effect.

Every tuning choice trades one good thing for another. Knowing the trade helps you choose with open eyes.

A handful of habits separate calm control planes from noisy ones.

Patterns worth copying

Anti-patterns to avoid

A short checklist to read before you build or change an etcd setup.

Mistakes beginners make

• Blaming the wrong layer. Spending days on node and network problems when a slow etcd disk was the cause.
• Tuning timeouts before fixing the disk. Longer timeouts hide instability instead of removing it.
• Ignoring Events. They are the largest source of writes in many clusters and are rarely checked.
• Forgetting the alarm. After freeing space, the NOSPACE alarm must be disarmed or writes stay refused.
• Letting Helm and Jobs pile up. Old releases and finished Jobs quietly grow the database for months.
• Skipping restore drills. Finding out the backup is unusable during a real outage is the worst possible timing.
• Copying the data directory of a live member and calling it a backup.

The same lessons appear again and again in large production clusters.

Short answers to the questions beginners ask most often.

You started with a single master notebook in a busy library and finished with a clear picture of how Kubernetes keeps its truth. etcd is small, careful, and consistent. Its price is that every write waits for a majority of disks and networks to agree, so its performance depends on steady disks, quiet networks, small data, and polite clients. Tuning is mostly a matter of protecting those four things, measuring honestly, and practicing recovery before it is needed.