etcd is the one place where a Kubernetes cluster keeps its memory. Lose it, and the cluster forgets who it is.

etcd is a small, very reliable database. Its name comes from the Linux folder /etc, where configuration files live, plus the letter “d” for “distributed”. It stores information as key-value pairs, which means each piece of data has a name (the key) and a content (the value), much like a labelled folder in a filing cabinet. It was created at CoreOS in 2013 and is now a graduated project of the Cloud Native Computing Foundation.

Kubernetes uses etcd as the only permanent store of cluster state. When you create a Deployment, the record of that Deployment is written to etcd. When a node joins, its record is written to etcd. When you change a setting, the new value goes to etcd. Nothing else in the control plane keeps durable data of its own.

Who is allowed to talk to etcd?

Only the Kubernetes API server talks to etcd directly. The scheduler, the controller manager, the kubelets and all your tools talk to the API server instead. This keeps one gatekeeper in front of the data and means every change passes through authentication, authorization and validation first.

flowchart LR
    U["Users and kubectl"] --> A["API Server"]
    S["Scheduler"] --> A
    M["Controller Manager"] --> A
    K["Kubelets on Nodes"] --> A
    A --> E[("etcd Cluster")]
    E --> B["Backup Snapshot Files"]
    

What happens if etcd is lost?

Containers that are already running on worker nodes usually keep running for a while, because the kubelet on each node does not need etcd to keep an existing container alive. But the cluster can no longer be managed. You cannot deploy, scale, heal, or even list what exists. If the data is truly gone, the whole description of your cluster is gone with it, and you would have to rebuild every object by hand.

Knowing exactly what a snapshot contains tells you what a restore can bring back, and what it cannot.

etcd holds the desired state and the recorded state of every Kubernetes object. That includes Pods, Deployments, StatefulSets, Services, ConfigMaps, Secrets, Namespaces, Roles and Role Bindings, Persistent Volume records, Custom Resource Definitions and every custom resource, plus the records of nodes and short-lived leases. Each object is saved under a key that starts with a common prefix, usually /registry, followed by the type and the namespace and name.

A word about Secrets

Kubernetes Secrets are stored in etcd, and by default they are only encoded, not encrypted. A snapshot file therefore contains your passwords, tokens and keys in a readable form. If you turn on encryption at rest in the API server, the Secrets inside etcd are scrambled with a key that lives in a separate configuration file. That helps, but it adds one more thing to protect and back up, which later chapters explain.

You do not need to be a database expert, but four ideas explain almost every backup and restore rule you will ever meet.

Idea 1: A group of members that agree

A production etcd runs as a group of three or five members, each on its own machine. They use an agreement method called Raft. In Raft, one member is the leader and the others are followers. All writes go through the leader. The leader sends each change to the followers, and the change counts as saved only when a majority of members confirm that they wrote it to disk.

That majority is called a quorum. For three members the quorum is two, and for five members it is three. A group can lose members and keep working only while a quorum survives.

sequenceDiagram
    participant A as API Server
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    A->>L: Write request
    L->>L: Append to write-ahead log
    L->>F1: Send log entry
    L->>F2: Send log entry
    F1-->>L: Entry saved
    F2-->>L: Entry saved
    L->>L: Quorum reached, commit
    L-->>A: Write confirmed
    

Idea 2: The write-ahead log

Before a change is applied to the main database file, each member writes it to a write-ahead log, usually called the WAL, and forces it onto the disk. If the machine loses power in the next second, the log still holds the change and the member can replay it on restart. This is why etcd is very sensitive to slow disks: every write waits for the disk to confirm.

Idea 3: Revisions and history

etcd never overwrites a value in place. Every change creates a new revision, a number that only goes up. Old versions of a key stay on disk until a cleanup step called compaction removes history older than a chosen point. After compaction, the database file still has empty holes in it, and a second step called defragmentation squeezes them out. Kubernetes triggers compaction automatically on a regular schedule, but defragmentation is usually something operators schedule themselves.

Idea 4: Internal snapshots versus backup snapshots

etcd itself takes internal snapshots of its state from time to time so it can shorten the log. These are part of normal housekeeping and are not a backup plan. A backup snapshot is one that you ask for, copy out of the etcd data folder and store somewhere safe. The rest of this guide is about the second kind.

Many teams think three etcd members means they are safe. They are safe from some failures, and completely exposed to others.

Running several etcd members protects you when one machine dies. It does not protect you when the data itself is wrong, because every member holds a copy of the same wrong data a few milliseconds later. If someone deletes a namespace by mistake, all members agree that it is deleted. Replication spreads mistakes as faithfully as it spreads good data.

The failures you should plan for

The two kinds of recovery

It helps to split recovery into two very different jobs. Member repair is the gentle job: you remove a dead member and add a fresh one, and the new member copies the data from its healthy peers. No backup is used. Cluster restore is the heavy job: the group has lost its quorum or its data, so you build a brand-new group from a snapshot file. Always try to use the gentle job when it is enough, and keep the heavy job for when you truly need it.

Before you type a single command, decide how much you can afford to lose and how long you can afford to wait.

Two short terms guide every backup plan. RPO, the Recovery Point Objective, is how much recent data you can afford to lose. If your RPO is thirty minutes, you must have a snapshot at least every thirty minutes. RTO, the Recovery Time Objective, is how long you can afford to be down. If your RTO is one hour, your restore steps must be rehearsed and fast enough to finish within an hour.

Choosing how often to snapshot

Clusters change at different speeds. A development cluster that changes once a day may need a snapshot only every few hours. A busy production cluster where deployments run all day and operators update status constantly may need a snapshot every fifteen to thirty minutes. Remember that anything created after the last snapshot is lost in a restore, so frequency is the main lever for reducing loss.

Choosing how long to keep them

Keep several generations, not only the newest. Mistakes are often noticed days later, and the newest snapshot may already contain the mistake. A common pattern is to keep frequent snapshots for a day or two, daily ones for a few weeks and weekly ones for a few months. Apply the pattern with automatic rules so old files do not pile up forever.

Taking a snapshot is a short, safe operation, and understanding its parts helps you do it correctly every time.

The standard tool is etcdctl, the command-line client that ships with etcd. Its snapshot save action connects to one etcd member and asks it to stream a consistent copy of its database into a single file. The cluster keeps running while this happens. The copy represents the exact state of the cluster at one moment, even though the stream takes a few seconds or minutes to write.

What the tool needs from you

• An address. The location of one etcd member, usually on port 2379 for client traffic. A snapshot talks to a single member, so give it one address.
• Proof of identity. etcd in a Kubernetes cluster almost always requires TLS, so you must provide the certificate authority file, a client certificate and its private key. In clusters created by kubeadm, these usually live under the Kubernetes PKI folder in an etcd sub-folder on the control plane node.
• A matching tool version. Use an etcdctl that matches the major and minor version of the running etcd. Mismatched versions are a common cause of strange errors.
• A place to write. A folder with enough free space for a file about as large as the etcd database.

Where to run it

You can run the tool directly on a control plane node, where the certificates already exist. You can also run it inside the etcd Pod in clusters where etcd runs as a static Pod, or from a small helper Job scheduled onto a control plane node. The goal is the same: reach a healthy member with valid credentials.

Checking the result

A snapshot file that cannot be read is worse than no snapshot, because it gives false comfort. After saving, inspect the file with the status action of etcdutl, the companion tool for working on files. It shows the file’s integrity hash, the revision it represents, the total number of keys and the size. If the key count is zero or the size is tiny, something went wrong. Newer etcd releases moved offline file operations from etcdctl to etcdutl, so older tutorials may show different tool names.

Why not just copy the data folder?

You may see advice to copy the database file straight from the member’s data folder. This can work, but the file may be in the middle of a write, the copy lacks the safety checks, and restoring from it needs a special flag to skip the integrity hash. The snapshot action is the supported way because it produces a verified, consistent file.

A backup that sits on the same disk as the thing it protects is not a backup. Where you store the file matters as much as taking it.

The control plane node that holds etcd is exactly the machine most likely to be lost in a disaster. Copy every snapshot off that machine right away. Good homes for snapshots are object storage such as Amazon S3, a separate backup system, or at least a different server in a different zone. Name the files with the cluster name and the date and time, so you can find the right one quickly when you are tired and stressed.

A safe layout on Amazon S3

If you run Kubernetes on AWS, S3 is a natural home. These features make it much safer.

Verify, do not assume

For each new snapshot, run at least the cheap checks automatically: the file exists, its size is within an expected range of the last one, and its status can be read with the hash and revision. Record the result in a log or metric. Once in a while, run the expensive check as well: restore the snapshot in a scratch environment and make sure a test API server starts and shows the objects you expect. Chapter 18 explains these drills.

An etcd snapshot is the heart of the plan, but a few other pieces decide whether a restore really works.

A backup that depends on a person remembering will eventually be forgotten. Automation turns good intentions into a habit.

A reliable backup job has four small parts. It takes a snapshot, checks it, copies it to safe storage, and reports success or failure. The job runs on a schedule that matches your RPO. If any part fails, a person is told within minutes, not months.

flowchart LR
    T["Scheduler: CronJob or timer"] --> S["Take snapshot from one etcd member"]
    S --> V["Check hash, revision and size"]
    V --> U["Upload to remote object storage"]
    U --> R["Apply retention rules"]
    V -- "Failed" --> X["Raise alert"]
    U -- "Failed" --> X
    U --> M["Record success metric"]
    

Common ways to schedule it

Keep the backup path independent

Where possible, make the backup job work even when the Kubernetes API is down. A node-level timer that reads straight from etcd keeps working when the API server is broken. If you use a CronJob, remember that it needs a working scheduler and API server to run at all, so also watch for the case where backups silently stop because the cluster itself is unhealthy.

A restore is an emergency operation that rewinds time. Slow down for five minutes at the start, and you may save five hours at the end.

Restoring is a big step. It replaces the cluster’s memory with an older one. Everything changed since the snapshot is gone from Kubernetes’ point of view. So the first question is not “how do I restore?” but “do I need to?”

The pre-restore checklist

The restore itself is a handful of clear steps. Understanding why each step exists makes it hard to get wrong.

The key idea is that a restore does not put data back into a running etcd. Instead, the tool reads the snapshot file and builds a brand-new data folder for a brand-new cluster. Then you start etcd from that new folder. The restored cluster has a new identity, which protects it from accidentally mixing with any old members that might still be alive somewhere on the network.

flowchart TD
    A["Stop API servers and other writers"] --> B["Stop every etcd member"]
    B --> C["Keep old data folders safe"]
    C --> D["Run snapshot restore on each member using the same snapshot file"]
    D --> E["Give each member its name, peer address and the full member list"]
    E --> F["Point etcd to the new data folder"]
    F --> G["Start all etcd members"]
    G --> H{"Members healthy and has leader?"}
    H -- "No" --> D
    H -- "Yes" --> I["Start API server, controller manager, scheduler"]
    I --> J["Verify nodes, objects and workloads"]
    

A restored cluster has travelled back in time. The world outside the cluster did not.

When etcd comes back from a snapshot, Kubernetes believes it is the moment the snapshot was taken. Meanwhile, the real world has moved on: containers kept running, cloud resources were created, passwords were changed. Most of the effort after a restore is about reconciling that gap.

What you can expect

The lifecycle in order

In a managed service, the provider runs etcd for you, and the rules of backup change completely.

On services like Amazon EKS, Google GKE and Azure AKS, the control plane, including etcd, belongs to the provider. You cannot log in to the etcd machines, read their certificates or run a snapshot command against them. Amazon, for example, runs the EKS control plane across several Availability Zones, with multiple API server instances and an etcd group spread across three zones, so a single machine or zone failure does not stop the cluster.

What is still your job

The provider protects against infrastructure failure. It does not protect against your own mistakes. If someone deletes a namespace in your EKS cluster, the cloud company will not bring it back. Provider-side backups exist for their own disaster recovery and are not offered as a self-service restore point for individual customers.

Whoever holds an etcd snapshot holds the keys to the entire cluster. Treat the file like a master password.

A snapshot contains every object, including Secrets, service account tokens and configuration. Someone who obtains a snapshot can read it offline, at their leisure, with no logs and no alarms. This makes backups a favourite target. The following rules reduce the risk.

Keep the key apart from the lock

A good rule is that stealing one thing must never be enough. Store encryption keys in a different place, with different access, than the snapshots they protect. Log every read of the backup bucket and alert on unusual access. Review the list of people and systems that can restore at least once a year.

etcd is small by database standards, but it is picky. Disk speed and database size decide how healthy your backups and restores will be.

etcd waits for the disk on every write, so fast, low-latency disks matter more than anything else. Solid-state drives, and on cloud platforms provisioned or local SSD volumes, are the norm. Keep the etcd data on its own disk where you can, so a noisy neighbour such as logging cannot slow down the cluster’s memory.

Size and the quota

etcd has a storage size limit, 2 GiB by default and adjustable. If the database grows past the limit, etcd raises an alarm and switches to a read-only mode where it only accepts deletions. The cluster is effectively frozen. The cure is to compact old revisions, defragment and then clear the alarm. This is one of the most common etcd emergencies and is entirely avoidable with monitoring.

Keep objects and namespaces tidy

Large ConfigMaps, thousands of leftover Jobs and huge custom resources inflate etcd. Each Kubernetes request is also limited in size, around a megabyte and a half by default. Set time-to-live values on finished Jobs so Kubernetes cleans them up, and avoid storing big payloads in objects.

Good high availability means you rarely need the heavy restore, because most failures are handled by the group itself.

How many members?

Always use an odd number. Three members tolerate one failure, and five tolerate two. Four members tolerate only one failure, the same as three, so the extra machine only adds cost and slightly slower writes. More than five is rarely useful, because every write must reach a majority, and more members mean more messages. Spread members across at least three failure zones, so losing one zone still leaves a majority.

Stacked or external?

In a stacked topology, which is the kubeadm default, each control plane node runs an API server and an etcd member together. It is simple and needs fewer machines, but losing a node loses both a control plane instance and an etcd member. In an external topology, etcd runs on its own machines. It costs more but separates the failure of etcd from the failure of the control plane, and lets you tune and protect etcd on its own.

flowchart TB
    subgraph Stacked["Stacked topology"]
        direction LR
        N1["Control plane node 1: API server + etcd"]
        N2["Control plane node 2: API server + etcd"]
        N3["Control plane node 3: API server + etcd"]
        N1 --- N2
        N2 --- N3
        N1 --- N3
    end
    subgraph External["External topology"]
        direction LR
        C1["Control plane node 1: API server"]
        C2["Control plane node 2: API server"]
        C3["Control plane node 3: API server"]
        E1[("etcd 1")]
        E2[("etcd 2")]
        E3[("etcd 3")]
        C1 --> E1
        C2 --> E2
        C3 --> E3
        E1 --- E2
        E2 --- E3
        E1 --- E3
    end
    

Replacing a dead member without a restore

If one member of a healthy group is lost, you do not need a snapshot. You tell the group to forget the dead member, prepare a fresh machine, and ask the group to add it as a new member. The new member then copies the data from its peers on its own. Do this carefully and one member at a time, never replacing two at once, so the group never loses its quorum while you work.

You want to hear about a sick etcd or a missing backup from an alert, not from an angry user.

etcd publishes detailed metrics in the Prometheus format on its own metrics address. Collect them and graph them. Good monitoring watches two things: the health of the etcd group, and the health of the backup process. Many teams do the first and forget the second.

Signals about etcd itself

Signals about backups

• Freshness. Time since the last successful snapshot. This is the most important backup alert.
• Size trend. A sudden drop to a tiny size, or a jump to many times the usual, deserves a look.
• Upload success. Whether the copy to remote storage finished, and whether the remote copy can be read.
• Drill results. The date and outcome of the last restore test, and how long it took against your RTO.

Logs worth reading

etcd logs warnings such as requests that took too long and slow disk syncs. They often appear hours before a visible outage. Send etcd logs to your central logging system, keep them for a fair period, and add a search for those slow-disk and leader-change messages to your regular health review.

The only backup you can trust is one you have already restored.

Many teams discover on the worst day that their backups were empty, that the keys were missing, or that the steps took four hours instead of one. A restore drill is a rehearsal done on a quiet day in a safe place. It finds those surprises while they are cheap.

What to rehearse

• Replacing a single failed member in a healthy group.
• Rebuilding a whole group from a snapshot after total loss.
• Recovering when the encryption key is needed, and when it is missing, so you know the real consequences.
• Restoring into brand-new machines in a different zone or region.
• Handling the aftermath: stale Secrets, orphaned cloud resources and volume checks.

No single backup method does everything. Knowing what each tool is good at lets you combine them wisely.

Trade-offs to decide on purpose

• Frequency versus load. More frequent snapshots reduce data loss but add disk reads and storage cost. Tune to your RPO instead of guessing.
• Retention versus risk. Keeping snapshots longer helps with late-noticed mistakes, but it also keeps old Secrets around for longer. Balance both.
• Convenience versus isolation. A backup job inside the cluster is easy to build, but one outside the cluster still works when the cluster is down.
• Speed versus safety. Immutable, off-region copies are slower to reach and costlier, but they are the ones that survive the worst days.

Teams that recover calmly tend to follow the same few patterns. Teams that suffer tend to repeat the same few mistakes.

Patterns worth copying

Anti-patterns to avoid

A short list of habits, and a short list of traps, collected from many real incidents.

Best practices

Common mistakes

• Pointing the snapshot tool at a load balancer. A snapshot must come from one specific member. Use the address of a single healthy member.
• Forgetting the TLS credentials. The tool fails or hangs without the right certificate authority, client certificate and key.
• Using mismatched tool versions. An old client with a new server leads to confusing errors. Match them.
• Restoring only one member of a group. Every member must be restored from the same snapshot with the same member list.
• Reusing the old data folder. The restore needs a clean folder, so set the old one aside first.
• Losing the encryption key. A snapshot with encrypted Secrets is half-useless without the key.
• Ignoring expiring certificates. Clusters built with kubeadm issue certificates that last one year by default. A forgotten renewal can bring down the control plane and make a restore harder.
• Letting the backup folder fill the node. Old local snapshots pile up until the disk is full, which damages etcd itself. Clean up after upload.
• Skipping the aftermath. The restore is not finished when the API server starts. Reconcile Secrets, cloud resources and volumes.

Major platforms bake etcd backup into their products because they have learned how much it matters.

Short answers to the questions people ask most about etcd backups.

etcd is the memory of your cluster, and a good backup plan is the difference between a short outage and a rebuild from scratch.

You began with a simple picture, a hotel ledger, and followed it through how etcd stores data, how a snapshot is taken, how it is protected and automated, and how a restore rewinds the cluster and what must be fixed afterwards. The pattern behind everything is simple: decide how much loss and downtime you can accept, build a layered and automated plan to meet it, protect what you store, and practise until the steps feel boring.