Before meeting etcd, understand what would go wrong without it.

A cluster is full of facts

A Kubernetes cluster is a group of machines that work together to run your applications. To do that job, the cluster must know many facts at every moment. Which applications should be running? How many copies of each? Which machine is each copy on? What secret passwords do the applications need? Which people are allowed to change what?

If these facts lived only in the memory of one program, a single restart would make the cluster forget everything. It would be like a shop owner who wakes up each morning with no idea what stock exists or what customers ordered.

Many components need the same truth

Kubernetes is made of several cooperating parts: a scheduler that picks machines, controllers that fix differences, and agents on every machine. They all need to read and update the same facts. If each kept its own copy, the copies would drift apart and the parts would work against each other.

flowchart LR
  A["Scheduler"] --> S["One shared source of truth"]
  B["Controllers"] --> S
  C["Node agents"] --> S
  D["Users and tools"] --> S
        

etcd is a distributed, consistent key-value database that Kubernetes uses to store all of its cluster data.

Let us unpack that sentence one word at a time.

Where the name comes from

The name is a playful mix of the Linux folder called “etc,” where configuration files live, and the letter “d” for distributed. The idea is a distributed place for configuration. It was created by the team at a company called CoreOS and is now a project hosted by the Cloud Native Computing Foundation, the same neutral home that hosts Kubernetes. It is written in the Go language, like Kubernetes itself.

A short glossary makes everything after this chapter easier.

Almost everything that describes the cluster itself.

Whenever you create something in Kubernetes, its description ends up in etcd. Here is what lives there.

What is not stored there

etcd does not hold your application’s own data such as customer orders or photos, and it does not store container images or application logs. Those belong in databases, image registries and logging systems. etcd holds the map of the cluster, not the cargo it carries.

One door, one doorman.

Here is a surprising design choice: in a normal Kubernetes cluster, only one component talks to etcd directly. That component is the API server. The scheduler, the controllers, the node agents and even human administrators all speak to the API server, and the API server speaks to etcd.

Why a single gateway?

graph TD
  U["Users and kubectl"] --> API["API Server"]
  SCH["Scheduler"] --> API
  CM["Controller Manager"] --> API
  KL["Kubelets on nodes"] --> API
  API --> E["etcd"]
        

How several machines can hold the same data without arguing.

A single database machine is a single point of failure. If it burns out, all the cluster’s memory disappears. So etcd runs on several machines that keep copies of the same data. The hard part is keeping those copies identical, even when messages are delayed or a machine suddenly stops. etcd solves this with a method called Raft, designed to be understandable and reliable.

The three roles

At any time each member plays one of three roles: follower, candidate or leader. Almost everyone is a follower most of the time. One member is the leader, who takes charge of all changes. A candidate is a member who is trying to become the new leader.

How a leader is chosen

The leader sends a tiny “I am still here” message to the followers again and again. If a follower stops hearing it for a random short time, it assumes the leader is gone, becomes a candidate and asks the others for votes. If a majority votes for it, it becomes the new leader. The random waiting time makes it unlikely that two members start an election at the same moment.

stateDiagram-v2
  [*] --> Follower
  Follower --> Candidate: No heartbeat heard
  Candidate --> Leader: Wins majority vote
  Candidate --> Follower: Another leader found
  Leader --> Follower: Sees a newer leader
        

How a change is recorded

Every change goes through the leader. The leader adds it to its log, sends it to the followers, and waits until a majority confirm they have stored it. Only then is the change considered committed and the client told it succeeded. Because a majority must always agree, two conflicting histories can never both be committed.

Majority rules, and that shapes how many members you should run.

A quorum is a majority: more than half of all members. If you have three members, a quorum is two. With five, it is three. As long as a quorum is alive and can talk to each other, etcd keeps working. If the number of healthy members falls below a quorum, etcd stops accepting changes to protect correctness. It prefers to pause rather than risk giving wrong answers.

Why odd numbers?

Look at the table: four members tolerate the same single failure as three, yet cost more and need more messages for every write. Odd numbers give the best protection for the price. Also, if a network split divides members into groups, an odd total guarantees that at most one group can hold a majority.

What actually happens when you create something in Kubernetes?

sequenceDiagram
  participant C as Client
  participant A as API Server
  participant L as etcd Leader
  participant F1 as Follower 1
  participant F2 as Follower 2
  C->>A: Create object
  A->>L: Write request
  L->>F1: Replicate entry
  L->>F2: Replicate entry
  F1-->>L: Saved
  F2-->>L: Saved
  L-->>A: Committed
  A-->>C: Success
        

Instead of asking “has anything changed?” a thousand times, components ask to be told.

Kubernetes works like a set of reactive helpers. The scheduler wants to know when a new pod appears with no home. A controller wants to know when a pod dies. The node agent wants to know when a pod is assigned to its machine. If each of these components repeatedly polled the database, the load would be enormous and the reactions slow.

etcd offers a watch feature. A client says “watch this range of keys and tell me about any change,” and etcd pushes notifications as changes happen. In Kubernetes, the API server holds these watches and shares the news with all the other components, so the cluster feels alive and responsive.

Missing nothing

Because every change has a revision number, a watcher that briefly disconnects can say “resume from revision so-and-so” and catch up on what it missed, as long as that history has not been cleaned away. This is one reason etcd is so well suited to controlling a busy system.

etcd remembers old versions, which is powerful and also needs housekeeping.

etcd does not overwrite a value in place. When something changes, it records a new version with a higher revision number and keeps the old versions for a while. This design makes watches and consistent reads possible, but the history grows every time anything is updated. In a busy cluster, updates happen constantly, for example every time a node reports its health.

Either alongside the control plane or on its own machines.

graph TD
  subgraph S["Stacked layout"]
    CP1["Control plane node 1 with etcd"]
    CP2["Control plane node 2 with etcd"]
    CP3["Control plane node 3 with etcd"]
  end
  subgraph X["External layout"]
    A1["API server node 1"]
    A2["API server node 2"]
    A3["API server node 3"]
    E1["etcd member 1"]
    E2["etcd member 2"]
    E3["etcd member 3"]
    A1 --> E1
    A2 --> E2
    A3 --> E3
  end
        

With managed Kubernetes services from cloud providers, you usually never see etcd at all. The provider runs, backs up and repairs it for you. Understanding etcd still helps you understand what that service is doing on your behalf and why some limits exist.

etcd is small, but it is demanding about a few specific things.

If etcd is the cluster’s memory, a backup is the cluster’s photograph album.

Having several members protects you from machine failures, but it does not protect you from human mistakes. If someone deletes a namespace by accident, that deletion is faithfully copied to every member. Replication is not a backup. That is why regular snapshots of etcd are vital.

Whoever controls etcd controls the entire cluster.

Because etcd holds every object and every secret, access to it is equivalent to being the administrator of the whole cluster. Protecting it is one of the most important security tasks.

Watch the memory of your cluster before it starts forgetting.

etcd publishes many measurements that monitoring tools can collect. A handful of them tell you most of what you need.

Set alerts on these signals so a human is told about slow disks or a filling database long before users notice symptoms such as pods that will not start.

Understanding failure modes shows why etcd is so central.

An important point for beginners: when etcd is unavailable, applications already running on the nodes usually keep running for a while, because the node agents continue to look after the containers they already have. What stops is change. You cannot deploy, scale or heal anything until etcd is back. The lights stay on, but nobody can flip a switch.

A specialist tool: excellent at its job, unsuitable for others.

Patterns teams follow, and traps they fall into.

Helpful patterns

Anti-patterns

You may already depend on etcd without knowing it.

If you remember only a page of this post, remember this one.

It is a separate open-source project that Kubernetes depends on. Kubernetes ships with instructions and tooling that assume etcd, but etcd is developed on its own.

The standard Kubernetes control plane expects etcd. Some lightweight distributions replace it with another storage backend, but the role of storing cluster state remains.

No. Applications talk to the Kubernetes API if they need cluster information, and only the API server talks to etcd. Applications should never be given direct access to it.

The cluster’s needs are simple: store objects under names, read them back, and be notified of changes. A key-value store with strong consistency and watches does exactly that without the weight of a full relational system.

The leader accepts and orders all changes, while followers copy those changes and stand ready to replace the leader if it disappears.

Split brain means two groups each believing they are in charge. etcd avoids it because only a group holding a majority can accept changes, and two groups can never both hold a majority.

An extra even member adds cost and slows writes without allowing more failures. Odd counts give the best fault tolerance per member.

Only if you protect them. By default they are stored in a form that can be decoded easily, so you should enable encryption at rest, restrict access and protect backups.

Usually not. The provider operates it. You still benefit from knowing how it works, especially the limits on the number and size of objects.

Slow storage. Because every change must be flushed to disk before it is confirmed, poor disk performance leads to slow clusters, timeouts and unstable leadership.

etcd is the quiet foundation of Kubernetes. It is a small, consistent, distributed database that stores the description and status of everything in the cluster, and it does so in a way that several machines always agree on. Every other component learns the truth from it, through the API server, and reacts to changes through watches. Because so much depends on it, etcd deserves careful sizing, fast disks, strong security, regular backups and steady monitoring.