A cluster that forgets what it was told is not a cluster for very long.

Kubernetes is built around a simple promise. You describe what you want, such as “run three copies of this web server,” and the system works day and night to make reality match your description. For that promise to mean anything, the description must be stored somewhere safe. The place where Kubernetes keeps every description, along with the current state of every Pod, Service, Secret, ConfigMap and Node, is a database called etcd.

Now think about what happens if that database is a single machine. If the machine crashes, the cluster loses its memory. Existing containers might keep running for a while, but nobody can create, change or repair anything, and if the disk is damaged, the record of what should be running may be gone forever. A single database is a single point of failure, and for the brain of a production platform that is unacceptable.

The obvious answer is to keep several copies on several machines. But copies create a new and subtler problem. If two copies receive different instructions at nearly the same moment, which one is right? If the network between them breaks, can each side keep accepting changes? If both do, the copies disagree, and when the network heals there is no clean way to merge two different histories of a cluster. Engineers call this the split-brain problem, and it can be far worse than an outage, because the system keeps running on conflicting beliefs.

etcd solves this with a consensus algorithm called Raft. Consensus means getting several computers to agree on one shared history, even though some of them may crash and the network between them may lose or delay messages. With Raft, etcd gives Kubernetes a memory that is copied across several servers, never contradicts itself, and keeps working as long as a majority of those servers can talk to each other.

Where etcd sits, who is allowed to talk to it, and what it actually stores.

etcd is a distributed key-value store. That means it holds pairs of a key, like a file path, and a value, like the contents of a file. Kubernetes stores each object under a key that includes its kind, namespace and name, and the value is the serialized object itself. When you create a Deployment, a record for it appears in etcd. When a controller updates the status of a Pod, the Pod’s record changes.

One rule of the architecture is easy to miss and extremely important: only the API server talks to etcd. The scheduler, the controller manager, the kubelets and your own tools never connect to it directly. They all talk to the API server, and the API server reads and writes etcd on their behalf. This keeps etcd well protected, keeps the number of clients small, and means all the validation, authentication and policy checks happen in one place before data is stored.

flowchart TD
    U["Users and kubectl"] --> A["API Server"]
    S["Scheduler"] --> A
    C["Controller Manager"] --> A
    K["Kubelets on every node"] --> A
    A --> E1["etcd member 1"]
    A --> E2["etcd member 2"]
    A --> E3["etcd member 3"]
    E1  E2
    E2  E3
    E1  E3
        

What lives in etcd

Almost every object you can list with a Kubernetes command is stored there: Pods, Deployments, Services, ConfigMaps, Secrets, Nodes, Namespaces, roles and bindings, custom resources, and Leases. It also holds cluster-wide records such as the registered API extensions. What it does not hold is application data. Your database rows, uploaded files and container images live elsewhere. etcd is built for small, precious, frequently read metadata, not for bulk storage.

A built-in size limit, and why

By default, etcd refuses a single request larger than about one and a half megabytes. This is the real reason why a Kubernetes object cannot grow without limit. Raft sends every write to every member, so very large values slow the whole group down. The limit nudges people to keep objects small and to put big data in proper storage systems.

Getting a group of unreliable computers to behave like one reliable one.

Computers fail in ordinary ways. A power supply dies, a disk slows down, a network cable is unplugged, or a program pauses for several seconds while the operating system does housekeeping. Worse, from the outside it is often impossible to tell whether a silent server has crashed or is just slow. Consensus algorithms are carefully designed recipes that let a group make decisions despite these uncertainties.

The standard way to use consensus is called a replicated state machine. Imagine every member starts with the same empty notebook. A shared list of instructions, such as “set key A to 5” and “delete key B,” is agreed upon in a fixed order. Every member then carries out the same instructions in the same order. Because each member performs identical steps from an identical start, they all end up with identical notebooks. The hard part is only one thing: agreeing on the order of the list. That agreed list is called the log, and Raft is a method for keeping it consistent.

What kinds of failure Raft tolerates

Raft is designed for crash faults and network faults. Servers may stop and later restart. Messages may be delayed, lost, duplicated or reordered. Raft is not designed for Byzantine faults, where a server lies or is controlled by an attacker and sends deliberately misleading messages. That is why the network between etcd members must be trusted and protected, a point we will return to in the security chapter.

Where Raft comes from

Raft was published in 2014 by Diego Ongaro and John Ousterhout, in a paper whose title describes its main goal: an understandable consensus algorithm. An older algorithm called Paxos solved the same problem but was famously hard to learn and to implement correctly. Raft breaks the problem into three independent pieces, leader election, log replication and safety, which made it much easier to build. etcd uses its own Raft library, and the same library has been reused by other well-known distributed systems.

Three roles, one rising counter, and a handful of simple rules.

Every member of a Raft group is always in exactly one of three roles. Understanding the roles is the key that unlocks the rest of the algorithm.

flowchart LR
    F["Follower"] -->|"No heartbeat for too long"| C["Candidate"]
    C -->|"Wins a majority of votes"| L["Leader"]
    C -->|"Another leader appears"| F
    C -->|"No winner, try again"| C
    L -->|"Sees a higher term"| F
        

Why terms matter

Terms act as a logical clock. Every message carries the sender’s term. If a member receives a message with a higher term than its own, it immediately updates its term and, if it was a leader or candidate, steps down to follower. If it receives a message with a lower term, it ignores it as outdated. This single rule is what prevents an old, disconnected leader from causing damage after it returns: the moment it hears a higher term, it knows its time is over.

How the group picks a boss within a second or two, and never picks two at once.

When an etcd group starts, every member begins as a follower and nobody is in charge. Each follower runs a countdown timer called the election timeout. The leader, once it exists, resets that timer on every follower by sending regular small messages called heartbeats. As long as the heartbeats keep arriving, followers stay calm.

sequenceDiagram
    participant A as Member A
    participant B as Member B
    participant C as Member C
    Note over A,C: Leader has failed, timers are running
    Note over B: B's timer fires first
    B->>A: Vote for me in term 5
    B->>C: Vote for me in term 5
    A-->>B: Vote granted
    C-->>B: Vote granted
    Note over B: B holds a majority and becomes leader
    B->>A: Heartbeat, I lead term 5
    B->>C: Heartbeat, I lead term 5
        

Randomness prevents deadlock

What if two followers time out at exactly the same moment? Each votes for itself, the other members split their votes, and nobody reaches a majority. This is called a split vote. Raft avoids endless repetition by making each member’s election timeout slightly random, within a range. After a split vote, one member will almost always time out first in the next round, win quickly and end the tie. In practice, a healthy etcd group completes an election in about the length of one timeout.

The majority rule guarantees a single leader

Because each member votes once per term, and a leader needs a majority, two leaders can never be elected in the same term. Any two majorities of the same group overlap in at least one member, and that member would have had to vote twice to elect both, which the rules forbid. This tiny piece of arithmetic is the foundation of everything Raft promises.

How one write travels from the leader to a majority and becomes permanent.

Once a leader exists, all changes go through it. If a client contacts a follower with a write, the follower passes the request along to the leader. The leader’s job is to turn each request into an entry in its log and make sure a majority of members store the same entry in the same position.

Each log entry has an index, which is its position in the list, a term, which is the term in which the leader created it, and the instruction itself. The leader appends the new entry to its own log, then sends it to all followers in a message called an append request. Each follower checks that its log matches the leader’s just before the new entry. If it does, the follower adds the entry and replies with a success. If not, the leader backs up and sends the earlier missing entries first until the follower’s log is repaired.

sequenceDiagram
    participant Cl as Client
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    Cl->>L: Write request
    L->>L: Append to own log and save to disk
    L->>F1: Append entry
    L->>F2: Append entry
    F1->>F1: Save to disk
    F1-->>L: Done
    Note over L: Leader plus one follower is a majority
    L->>L: Mark entry committed and apply it
    L-->>Cl: Success
    L->>F1: Next message tells follower the commit point
    L->>F2: Next message tells follower the commit point
        

Committed means safe

An entry is called committed once the leader knows it is stored on a majority of members. From that moment, Raft promises it will never be lost, even if the leader crashes immediately afterwards. The leader then applies the instruction to the actual key-value data and tells the client the write succeeded. The followers learn the new commit point through later messages and apply the same instruction in the same order.

Notice that the leader does not wait for every member, only for a majority. If one follower is slow or down, writes continue without delay. The slow follower catches up later, because the leader keeps resending entries until the follower’s log matches.

Why the leader writes to disk first

Before acknowledging anything, every member saves the entry to a file on disk and forces the operating system to flush it to permanent storage. If a member only kept the entry in memory, a power failure after it said “done” would erase a promise it already made, and the guarantees would break. This is why disk speed is the most important performance factor for etcd, a point we will explore in the performance chapter.

Five promises that Raft keeps no matter which servers crash or which messages go missing.

Speed and convenience mean nothing if the system can forget committed data. Raft therefore comes with a short list of safety properties, each of which is proven to hold. Knowing them helps you reason about almost any failure story.

The election restriction

The most clever rule sits inside the voting step. A member only votes for a candidate whose log is at least as up to date as its own, judged first by the term of the last entry and then by the length of the log. Combine this with the majority rule. A committed entry lives on a majority of members. A candidate needs votes from a majority to win. Any two majorities overlap, so at least one voter in the winning majority holds the committed entry, and that voter will refuse any candidate who lacks it. The result is that a candidate missing committed data can never become leader.

What happens to uncommitted entries

Not every entry in a log is committed. If a leader crashes after saving a new entry locally but before copying it to a majority, that entry was never promised to the client. A new leader, which lacks the entry, will simply carry on. When the old leader returns as a follower, the new leader overwrites its extra entries with the correct history. Because the client never received a success reply, nothing that was promised is lost. Clients that time out must simply retry or check the result, which is why idempotent operations and resource versions are so valuable on the Kubernetes side.

Why three or five members is the sweet spot, and why four is a trap.

A quorum is the smallest number of members that counts as a majority. For a group of N members, the quorum is half of N, rounded down, plus one. Raft needs a quorum both to elect a leader and to commit any entry. If a quorum is available, the group works. If not, the group stops accepting changes rather than risk disagreement.

Why even numbers do not help

Look at the table. A group of four needs three members to agree, and so tolerates only one failure, exactly like a group of three. The fourth member adds cost and slightly slows every write, because the leader must reach more members, without improving survival. An even group can also split evenly in a network partition, leaving neither side with a majority. For these reasons, odd sizes are strongly preferred.

Why more members is not always better

It is tempting to think that more members means more safety and more speed. Safety does improve a little, but speed does not. Every write goes through the single leader and must reach a quorum, so adding members never increases write throughput. It actually makes writes a bit slower, because the leader sends each entry to more places. Beyond five or seven members, the cost outweighs the benefit. The usual guidance is to use three members for most clusters, five where the extra resilience is worth it, and to avoid going past seven.

Following a single kubectl change from your keyboard all the way to durable storage.

Let us connect the dots. Suppose an engineer changes the replica count of a Deployment. Here is what happens, from the request to the safe storage of the new value.

flowchart TD
    A["Engineer sends the change"] --> B["API server: authenticate, authorize, admit, validate"]
    B --> C["API server sends a conditional write to etcd"]
    C --> D["Any etcd member forwards it to the leader"]
    D --> E["Leader appends to log and saves to disk"]
    E --> F["Leader sends entry to followers"]
    F --> G{"Majority saved?"}
    G -->|Yes| H["Commit, apply to the data, new revision"]
    G -->|No| I["Wait, retry, or fail on timeout"]
    H --> J["Reply to API server, then to the engineer"]
    H --> K["Watchers are notified of the change"]
        

Why the revision is trustworthy

The cluster-wide revision counter that Kubernetes uses to produce resource versions is itself a product of consensus. Because all members apply the same entries in the same order, they all assign the same revision to each change. That is why a version stamp read from one member means the same thing everywhere, and why compare-and-swap writes are safe even when different etcd members serve different requests.

Reading seems simple, yet it hides a subtle choice between accuracy and speed.

You might think reads are easy because they change nothing. But in a group of members, a follower may be a little behind the leader. If a read were served from a lagging follower’s local data, it could return an old answer even though a newer one has already been committed. etcd therefore offers two read modes.

sequenceDiagram
    participant Cl as Client
    participant F as Follower
    participant L as Leader
    Cl->>F: Read, latest data please
    F->>L: What is the current commit point?
    L->>L: Confirm still leader with a majority
    L-->>F: Commit point is 900
    F->>F: Wait until applied up to 900
    F-->>Cl: Answer from fresh data
        

How this shows up in Kubernetes

Remember the resourceVersion choices on reads. When a request leaves the version unset, the API server asks etcd for a quorum read, which is the linearizable mode, and gets the most recent data. When the version is set to zero, the API server answers from its own in-memory cache, which is fed by a watch on etcd and may lag slightly. The Kubernetes freshness dial is therefore directly tied to the etcd read modes underneath, and to protect etcd from heavy traffic, the API server leans on its cache for most list requests.

How etcd keeps its promises across restarts without growing forever.

Raft is about agreement, but each member must also remember what was agreed after a restart. etcd uses several layers of storage that work together, and knowing them clears up many operational mysteries.

Compaction and defragmentation

Because etcd keeps history, the database grows with every change, even when keys are overwritten. Compaction tells etcd to forget history older than a chosen revision. The Kubernetes API server asks etcd to compact on a regular schedule, by default about every five minutes. This is exactly why watchers that fall too far behind receive the 410 Gone response: the history they wanted was compacted away.

Compaction frees space inside the database file but does not shrink the file on disk. Returning that space to the operating system requires a separate step called defragmentation. Defragmentation briefly blocks the member it runs on, so it is performed one member at a time, starting with followers, and usually during quiet periods.

The space quota and the alarm

etcd protects itself with a storage quota. By default the database may grow to about two gigabytes, and operators are advised to stay well below a much higher ceiling, often cited as eight gigabytes. If the quota is exceeded, etcd raises an alarm and switches to a mode that rejects writes, while still allowing reads and deletions. The cluster then cannot change until space is freed, which is a well-known and painful incident.

How etcd groups are shaped, grown and shrunk without breaking the majority rule.

An etcd group is not frozen forever. Servers fail and need to be replaced, and clusters grow. But changing the membership of a consensus group is dangerous. If the old group and the new group could each form a majority at the same time, you could briefly have two leaders. etcd avoids this by changing the membership one member at a time, so that any majority of the old group and any majority of the new group always overlap.

Learners: joining safely

A new member that starts with an empty database has a lot of catching up to do. If it were counted in the quorum immediately, it could make the group more fragile while it is still behind, because writes might wait on a member that cannot yet keep up. etcd therefore supports learners, which are members that receive log entries but do not vote and do not count toward the quorum. An operator adds the new member as a learner, waits until it has caught up, and only then promotes it to a full voting member. Typically only one learner is allowed at a time.

Two ways to place etcd

Cluster builders choose between two layouts. In a stacked topology, each control plane machine runs an etcd member next to the API server, scheduler and controller manager. It is simple, needs fewer machines, and is the default for common cluster setup tools. In an external topology, the etcd group runs on its own dedicated machines, separate from the control plane.

flowchart TB
    subgraph ST["Stacked topology"]
        direction TB
        S1["Control plane node 1: API server plus etcd"]
        S2["Control plane node 2: API server plus etcd"]
        S3["Control plane node 3: API server plus etcd"]
    end
    subgraph EX["External topology"]
        direction TB
        X1["Control plane node 1: API server"]
        X2["Control plane node 2: API server"]
        X3["Control plane node 3: API server"]
        E1["Dedicated etcd 1"]
        E2["Dedicated etcd 2"]
        E3["Dedicated etcd 3"]
        X1 --> E1
        X2 --> E2
        X3 --> E3
    end
        

Walking through the bad days to see exactly how Raft and the cluster respond.

The best way to appreciate a consensus algorithm is to watch it handle trouble. Here are the most important failure situations for a three-member group and what each means for your cluster.

flowchart LR
    subgraph MAJ["Majority side: 2 members"]
        A["Member A"]
        B["Member B leader"]
    end
    subgraph MIN["Minority side: 1 member"]
        C["Member C"]
    end
    A  B
    B -.->|"Network cut"| C
        

What the data plane does meanwhile

An important point for operators is that the control plane and data plane fail differently. When etcd is unavailable, your application Pods generally keep serving traffic, because the kubelet and the container runtime do not need to ask etcd for permission to keep running. What stops working is everything that needs to change state: deployments, scaling, repairs, new Service endpoints and new Pods. That is a degraded but not dead cluster.

Disk first, network second, and only then the timers.

etcd is an unusual database. It does not need huge amounts of processor power or memory, but it is extremely sensitive to the speed and steadiness of its disk and network. Understanding why helps you size it correctly.

The disk is the heartbeat

Every Raft entry must be forced to permanent storage before it is acknowledged. The time this takes, called the fsync latency, sits in the middle of every write. If a disk is slow or has unpredictable delays, writes slow down, followers miss heartbeats and unnecessary elections occur. Guidance from the project says that the ninety-ninth percentile of the write-ahead log sync time should stay under about ten milliseconds. Fast solid-state drives on dedicated volumes are strongly recommended, and sharing a disk with noisy neighbours is a common cause of trouble.

The network sets the floor

Each write needs at least one round trip to a follower. If members are far apart, such as in different regions, every write inherits that delay, and cross-region stretching of a single etcd group is generally a bad idea. Members spread across nearby availability zones, with low and stable delay, are the usual compromise between resilience and speed.

Heartbeat and election timers

The heartbeat interval should be about as long as the typical round trip between members, and the election timeout should be at least ten times that round trip. If the election timeout is too short, ordinary hiccups cause pointless elections. If it is too long, real failures take longer to detect and recover from. The defaults of one hundred milliseconds and one second suit a single data centre. Clusters spread across zones with higher delay may raise both, trading slower failure detection for fewer false alarms.

Separating noisy data

Some Kubernetes data is high volume and short lived, with Events being the classic example. Clusters at large scale often point the API server at a separate etcd group for events, so that this chatter cannot slow the main group that holds the important configuration. It is a simple example of dividing workloads so that one cannot hurt the other.

Whoever can talk to etcd can rewrite your whole cluster, so the doors must be locked.

Because etcd holds the entire state of the cluster, including every Secret, access to it is effectively access to everything. Anyone who can write to etcd directly can bypass every check that the API server performs, including permissions and policies. Protecting it carefully is therefore a top priority.

Why Raft needs a trusted network

Remember that Raft tolerates crashes and message loss, but not liars. A malicious party who could join the peer network or forge peer messages could disrupt elections or inject changes. Mutual authentication of peers and tight network rules close that door. The safety proofs of Raft hold only when every member follows the rules, so membership must be controlled.

The handful of signals that tell you an etcd group is healthy or heading for trouble.

An etcd group usually runs quietly for months and then causes a major incident if ignored. A small set of well-chosen signals gives early warning. They fall into four families: leadership, replication, disk and size.

Logs and warnings

etcd writes clear warnings when things degrade. Typical messages mention requests taking too long to apply, heartbeats not being sent on time because of a slow disk or an overloaded processor, and elections caused by missed heartbeats. Treat these messages as the earliest evidence that disks or networks are struggling, since they appear before graphs turn red.

Raft survives a few failures, but it cannot help if a majority is destroyed or data is deleted by mistake.

It is a common misunderstanding that a three-member group is its own backup. It is not. Replication protects against machines failing, not against human mistakes. If someone deletes a namespace by accident, Raft faithfully replicates that deletion to every member. If all members are lost in one disaster, the data is gone. A separate backup is the only protection against these events.

Replacing a single failed member

Most incidents are not total disasters but a single dead member. As long as a quorum remains, the right approach is to remove the dead member from the group, add a fresh one as a learner, let it catch up and promote it. Doing this while the remaining members are healthy is far safer than waiting until a second failure leaves the group without quorum.

A short checklist that covers most of what experienced operators do.

Upgrade with care

Upgrading an etcd group is a membership operation in disguise. The safe approach is a rolling upgrade, one member at a time, checking after each step that the member has rejoined, caught up and that the group has a leader before moving on. Taking a fresh backup first gives a way back if something unexpected happens. Skipping several minor versions at once is risky, so follow the supported upgrade path.

Five habits that quietly weaken an etcd group.

An honest look at what the Raft choice buys Kubernetes and what it costs.

How it compares with other choices

The key trade-off is consistency over availability. When the network breaks, etcd prefers to stop accepting changes on the side that lacks a majority, instead of accepting changes that might later conflict. For a control plane that decides what runs in your cluster, that is the right trade, because a briefly frozen cluster is safer than one acting on contradictory instructions.

Where these ideas show up in everyday platform engineering.

Short answers to the questions beginners ask most often.

An even-sized group tolerates no more failures than the odd size just below it, costs more and can split evenly in a partition. Odd sizes give the best resilience for the cost.

Running containers generally keep running because kubelets continue with their last known instructions. What stops is anything that needs to change state, such as creating, scaling, scheduling or repairing, until etcd regains a quorum.

A client can contact any member, but a write sent to a follower is forwarded to the leader. All writes are ordered and committed by the leader.

No. Writes still pass through the single leader and need a majority, so more members add resilience at best and a little extra delay at worst. Speed comes from faster disks, lighter objects and lower network delay.

Typically about the length of one election timeout, which is one second by default, plus a short extra delay if a split vote occurs. Writes pause briefly during this time and then resume.

No. A term counts elections, while the revision that feeds resource versions counts changes to the data. They are separate counters with separate purposes.

No. Replication protects against machine failures but copies mistakes and deletions to every member. Backups stored elsewhere protect against human error and total loss.

A common cause is that the database passed its space quota, which makes etcd reject writes until space is freed through compaction, defragmentation and removal of unneeded data.

No. Raft assumes members follow the rules. Protection against intruders comes from mutual authentication, encryption and careful control of who can join or connect.

Every entry must be saved to permanent storage before it is acknowledged, so disk delay sits inside every write. Slow disks can cause missed heartbeats and unnecessary elections.

Kubernetes keeps its entire memory in etcd, and etcd keeps that memory safe by copying it across a small group of servers that agree on every change through Raft. One leader orders all writes, a majority must save each entry before it counts, and a strict voting rule guarantees that any new leader already holds everything that was ever committed. This makes the cluster’s history trustworthy, gives resource versions their meaning, and lets the control plane survive crashes and network splits without ever disagreeing with itself. The price is that the group needs a quorum to move forward, depends on fast disks and calm networks, and still needs real backups. Understand those few ideas and most etcd behaviour becomes easy to predict.