Kubernetes was built around the idea that pods are disposable. Data is the opposite of disposable.

Kubernetes treats pods like paper cups. If one gets dirty, spills, or is on a table that is being cleared, Kubernetes throws it away and pours a fresh one somewhere else. This makes the platform resilient. But the same disposability creates a problem for any application that keeps data. A container’s own filesystem lives and dies with the container. When a pod is replaced, everything written inside it disappears.

Storing data safely therefore needs two things at once. First, the data must live outside the pod, on a disk that survives the pod’s death. Second, when a replacement pod starts, it must find the same disk again, and know who it is, so it can rejoin its group properly. Solving both together is the job of the StatefulSet and its storage system.

What can go wrong without the right design

A handful of storage words are enough to understand everything that follows.

The most important relationship to remember is the chain: an application asks for storage with a claim, the storage system creates a volume to satisfy it, and the claim and volume are bound together. The pod mounts the claim, and the data lives on the volume. A StatefulSet automates creating a separate claim for every pod.

A StatefulSet is a Kubernetes manager for pods that need stable names, stable storage, and orderly starting and stopping.

A StatefulSet looks after a group of pods built from one template, much like a Deployment does. But it makes three extra promises that a Deployment does not. First, every pod gets a permanent, predictable name with a number: the first is number zero, the next is number one, and so on. Second, every pod gets its own storage, which stays attached to that name even if the pod is replaced. Third, pods are created, updated, and removed in a careful order rather than all at once.

What it is, why it exists, where it is used

What it is: a workload manager for stateful applications, giving each pod a sticky identity and its own storage.

Why it exists: because clustered data systems need members that can be told apart, found by name, and reunited with their own data after any restart.

Where it is used: for databases, message brokers, search clusters, coordination services, and any application where each copy keeps its own state.

The best way to understand a StatefulSet is to see how it differs from the Deployment you already know.

flowchart LR
    subgraph DEP["Deployment: interchangeable pods"]
        D1["Pod with random name a1x9"] --> DV["One shared volume or none"]
        D2["Pod with random name b7k2"] --> DV
        D3["Pod with random name c4m8"] --> DV
    end
    subgraph STS["StatefulSet: numbered pods with their own disks"]
        S0["Pod db-0"] --> V0["Volume data-db-0"]
        S1["Pod db-1"] --> V1["Volume data-db-1"]
        S2["Pod db-2"] --> V2["Volume data-db-2"]
    end
    

Several small parts cooperate to give each pod a name, a disk, and a place to run.

flowchart TD
    USER["Engineer or pipeline"] -->|"Creates StatefulSet"| API["API Server"]
    API --> SSC["StatefulSet Controller"]
    SSC -->|"Creates numbered pod"| POD["Pod db-0"]
    SSC -->|"Creates claim from template"| PVC["PersistentVolumeClaim data-db-0"]
    PVC --> PVCC["Volume binding controller"]
    PVCC --> SC["StorageClass"]
    SC --> CSI["CSI provisioner"]
    CSI --> DISK["Real disk in cloud or data center"]
    PVCC --> PV["PersistentVolume"]
    PV --> DISK
    POD --> SCH["Scheduler picks a node"]
    SCH --> NODE["Node and kubelet"]
    NODE -->|"Attaches and mounts"| DISK
    HS["Headless Service"] -->|"Publishes db-0 DNS name"| POD
    

A StatefulSet has a small number of parts. Learning them once makes every StatefulSet easy to read.

Every StatefulSet pod carries a name that never changes, and a matching address that other pods can look up.

A StatefulSet named “db” with three replicas produces pods named “db-0”, “db-1”, and “db-2”. If db-1 is deleted or crashes, the replacement is again called “db-1”. It is not “db-4” or a random word. The number is called the ordinal. Ordinals let applications answer the question “which member am I?” simply by reading their own name. A database can decide that member zero is the first primary, or that members one and two join it.

The headless Service

A normal Service gives one shared address for a group and sends each request to any member. A headless Service does not have a shared address. Instead, it publishes separate DNS records for each pod, so “db-0” and “db-1” each have their own name that stays the same even when the pod’s underlying address changes. A StatefulSet must name a headless Service to provide these per-pod names.

The signature storage feature of a StatefulSet is a template that automatically creates a separate claim for every pod.

A volume claim template is a recipe, written once, describing the storage each pod needs: how large, which storage class, and which access mode. For every pod the StatefulSet creates, it uses the recipe to produce a separate claim. The claim’s name combines the template’s name, the StatefulSet’s name, and the pod’s number. So a template called “data” in a StatefulSet called “db” yields claims named “data-db-0”, “data-db-1”, and “data-db-2”.

The link between name and disk

When a pod named “db-1” starts, the StatefulSet looks for a claim named “data-db-1”. If it exists, the pod is reunited with it, with all the data. If it does not, a new one is created. This name-based matching is how a replacement pod finds the same disk as the pod it replaces.

Follow a new StatefulSet from creation to a running, storage-backed pod.

sequenceDiagram
    participant U as Engineer
    participant API as API Server
    participant SSC as StatefulSet Controller
    participant PVCC as Binding and Provisioning
    participant SCH as Scheduler
    participant K as Kubelet
    U->>API: Create StatefulSet with 3 replicas
    API->>SSC: Notify of new StatefulSet
    SSC->>API: Create claim data-db-0
    SSC->>API: Create pod db-0
    PVCC->>PVCC: Wait for first pod placement if class says so
    SCH->>API: Place db-0 on node in zone A
    PVCC->>API: Provision disk in zone A and bind claim
    K->>K: Attach disk to node, mount into container
    K-->>API: db-0 is running and ready
    SSC->>API: Create claim data-db-1 and pod db-1
    Note over SSC,K: Repeat in order for db-2
    

A storage claim passes through clear stages, and knowing them makes stuck volumes easy to diagnose.

stateDiagram-v2
    [*] --> Pending
    Pending --> Bound : "Disk created and matched"
    Pending --> Pending : "Waiting for a node or capacity"
    Bound --> InUse : "A pod mounts the claim"
    InUse --> Bound : "Pod stops or is replaced"
    Bound --> Resizing : "Larger size requested"
    Resizing --> Bound : "Disk and filesystem expanded"
    Bound --> Terminating : "Claim deleted"
    InUse --> Terminating : "Claim deleted while in use"
    Terminating --> Released : "Pod stops using it"
    Released --> [*] : "Disk deleted or kept, per policy"
    

A storage class is the menu that decides what kind of disk each claim receives.

A storage class names a type of storage, such as “fast solid-state disk” or “standard magnetic disk”, together with the driver that can create it and settings that control its behavior. When a claim names a class, the driver creates a disk of that type on demand. This is called dynamic provisioning, and it replaces the old way, where an administrator pre-created every disk by hand.

Drivers follow a standard called the Container Storage Interface, or CSI. Because of it, a single set of Kubernetes objects works with cloud disks, network file systems, and storage appliances alike. Each vendor supplies a CSI driver that translates Kubernetes requests into its own language.

A cloud disk exists in one place. If the pod cannot reach that place, it cannot start.

Most cloud block disks live in a single zone. A pod placed on a machine in a different zone cannot attach it. This is a source of the most common stuck-pod problem in stateful systems. Kubernetes gives two binding modes to handle it.

flowchart TD
    A["Pod db-1 needs to start"] --> B{"Does its claim already have a disk?"}
    B -->|No| C["Scheduler picks a node"]
    C --> D["Disk is created in that node's zone"]
    D --> E["Pod starts"]
    B -->|Yes| F{"Is there a node with room in the disk's zone?"}
    F -->|Yes| G["Pod is placed on that node"]
    G --> E
    F -->|No| H["Pod stays pending"]
    H --> I["Fix: add capacity in that zone, or restore from a snapshot elsewhere"]
    

An access mode says how many machines may use a volume at once, and whether they may write.

For StatefulSets, the usual pattern is a private, single-writer disk for each pod. That matches how most databases work: each member owns its own files. Sharing one disk among several database members that each expect to own it invites corruption.

Deleting the wrong thing can erase data forever. Understanding the defaults protects you.

Two settings decide what happens to data when things are removed. The reclaim policy belongs to the storage class or volume, and decides what happens to the real disk when a claim is deleted: either the disk is deleted along with its data, or it is retained so it can be recovered by hand. The claim retention policy belongs to the StatefulSet in newer versions, and decides whether claims themselves are removed when the set is scaled down or deleted.

flowchart TD
    A["StatefulSet is scaled down or deleted"] --> B{"Claim retention policy?"}
    B -->|"Retain (default)"| C["Claims stay, with their data"]
    B -->|"Delete"| D["Claims are deleted"]
    D --> E{"Reclaim policy of the storage class?"}
    E -->|"Delete"| F["Disk and data are destroyed"]
    E -->|"Retain"| G["Disk is kept, claim is gone; recovery by hand"]
    C --> H["Scaling back up reattaches the same data"]
    

By default, a StatefulSet does things one step at a time, in a fixed order. This is a feature, and sometimes a speed limit.

The default policy, called ordered ready, creates pods from the lowest number to the highest, waiting for each to be running and ready before starting the next. When scaling down or deleting, it removes them in reverse order, from the highest number to the lowest, and waits for each to finish. This suits systems where member zero must start first, such as a primary that others copy from, or where the group must never lose several members at once.

The alternative policy, parallel, launches or removes all pods at the same time without waiting. This speeds up large groups whose members do not depend on start order, such as search shards or cache nodes that discover each other on their own.

A careful update replaces one member at a time and gives you a way to test on a few before trusting all.

When you change the pod template, for example to a new version of the database software, the StatefulSet replaces pods according to its update strategy. The default is a rolling update that moves from the highest number to the lowest: pod two first, then pod one, then pod zero. Each replacement keeps the same name and reattaches to the same disk, and the controller waits for it to be ready before touching the next.

Two tools for control

• Partition. You can set a number below which pods are not updated. With five members and a partition of three, only pods three and four are updated; pods zero to two stay on the old version. Lowering the partition step by step lets you canary the new version on a few members first.
• On delete. A manual strategy in which nothing is replaced automatically. Each pod is updated only when you delete it yourself. This gives total control, at the cost of effort.

flowchart TD
    A["New version defined"] --> B["Partition set high: only top pods eligible"]
    B --> C["Highest numbered pod is replaced"]
    C --> D{"Ready and healthy for the wait time?"}
    D -->|No| E["Pause, investigate, roll back the change"]
    D -->|Yes| F["Lower the partition to include the next pod"]
    F --> G{"All pods updated?"}
    G -->|No| C
    G -->|Yes| H["Update complete"]
    

Databases only ever grow. Plan for the day the disk is nearly full, before it arrives.

Enlarging a volume is supported by many storage classes, if their expansion setting is enabled. You request a larger size on the claim, the storage system grows the disk, and the filesystem is expanded to use the new space. Depending on the driver, this may happen while the pod runs, or may need a restart of the pod.

The tricky part with StatefulSets is that the volume claim template cannot be edited once the set exists. Changing the template will not resize the existing claims, and the platform refuses to modify that section. The usual approach is to enlarge each claim directly, one by one, and then recreate the StatefulSet object itself while leaving its pods and claims running, so that its template records the new size for future pods.

A StatefulSet is cautious. It would rather wait than risk two pods writing the same disk.

When a node stops responding, Kubernetes cannot tell whether the machine has really died or is only cut off from the network. If the machine is still running, the old pod might still be writing to its disk. Starting a second pod with the same name and the same disk would then risk two writers and corrupted data. To avoid that, a StatefulSet guarantees at most one pod with a given identity at any time. It will not create the replacement until it is sure the old one is gone.

sequenceDiagram
    participant N as Failed node
    participant CP as Control plane
    participant SSC as StatefulSet Controller
    participant D as Cloud disk
    participant N2 as Healthy node
    N--xCP: Stops reporting
    CP->>CP: Marks node not ready after a grace time
    CP->>CP: Marks pod db-1 as terminating but cannot confirm
    Note over SSC: Will not create a second db-1 while the first might be alive
    CP->>CP: Node confirmed gone, or force deleted by an operator
    SSC->>CP: Create replacement db-1
    D->>D: Detach from failed node, possibly after a wait
    D->>N2: Attach to healthy node
    N2-->>CP: db-1 running with same data
    

Why recovery can take a while

• Detection delay. The cluster waits some time before deciding that a node is not ready.
• Terminating limbo. A pod on an unreachable node remains “terminating” until the node returns or is confirmed removed.
• Disk release. The cloud may hold the disk attached to the dead machine for a few minutes, causing errors saying the volume is already in use.
• Placement. The replacement must land in the disk’s zone, where capacity must exist.

The biggest storage decision is whether the disk is separate from the machine or part of it.

Real systems reuse a few well-tested designs. Knowing the catalogue lets you choose instead of invent.

flowchart TD
    A["What does the application need?"] --> B{"Does each member own its own data?"}
    B -->|Yes| C{"Can the app replicate data between members?"}
    C -->|Yes| D["Pattern 1: Private disk per pod with app replication"]
    C -->|No| E["Pattern 2: Private disk per pod plus strong backups"]
    B -->|No| F{"Must many pods share the same files?"}
    F -->|Yes| G["Pattern 3: Shared file storage"]
    F -->|No| H{"Can the data be rebuilt?"}
    H -->|Yes| I["Pattern 4: Temporary or local scratch storage"]
    H -->|No| J["Reconsider design: use a managed service or object storage"]
    D --> K["Add: snapshots, sidecar backups, zone spreading"]
    E --> K
    

This is the workhorse pattern. Understanding its moving parts explains most of what StatefulSets are for.

In this design, each member of a database has its own disk, and the software keeps the members in step. One member, the primary, accepts writes. The others, the replicas, copy those writes and can serve reads. If the primary fails, a replica is promoted. Kubernetes supplies stable names and disks. The database supplies the copying and the promotion.

flowchart LR
    C["Client application"] -->|"Writes"| P["db-0 Primary"]
    C -->|"Reads"| R1["db-1 Replica"]
    C -->|"Reads"| R2["db-2 Replica"]
    P -->|"Sends changes"| R1
    P -->|"Sends changes"| R2
    P --- VP["Volume data-db-0 in zone A"]
    R1 --- V1["Volume data-db-1 in zone B"]
    R2 --- V2["Volume data-db-2 in zone C"]
    HS["Headless Service"] -.->|"Names for each member"| P
    HS -.-> R1
    HS -.-> R2
    

Sometimes several pods need the same files. This is possible, but it changes the rules.

A shared file system can be mounted by many pods on many machines at once. This suits content such as uploaded images, shared configuration files, or build outputs. Because the volume is shared, one claim can serve a whole Deployment, and no per-pod claim is needed. It also means the StatefulSet’s main storage feature, one disk per pod, is not the point.

Small helper containers prepare the disk before the main application starts and look after it while it runs.

Init containers: getting ready

An init container is a short-lived helper that runs to completion before the main containers start. In stateful systems it is commonly used to prepare a new disk: fix ownership and permissions, generate a configuration file from the pod’s number, or copy initial data from a snapshot or a peer. Because the pod’s name contains its ordinal, the init container can decide whether this member should start as a primary or a replica.

Sidecars: looking after the disk

A sidecar is a second container that runs alongside the main one for the pod’s whole life and shares its volumes. Typical sidecar jobs for storage include streaming backups to object storage, shipping log files, exporting metrics about disk use, and running small checks.

A snapshot is a fast, frozen picture of a disk, and a clone is a new disk made from one.

The CSI standard supports volume snapshots where the driver allows it. A snapshot request produces a point-in-time copy, often within seconds, because many storage systems record only changes after that moment. From a snapshot you can create a new claim with the same contents, which is called restoring or cloning. Some drivers also clone one volume directly into another.

A persistent disk protects you from a pod dying. Only a backup protects you from everything else.

flowchart LR
    APP["Database pods"] --> SNAP["Volume snapshots"]
    APP --> DUMP["Logical export by sidecar or job"]
    APP --> WAL["Continuous change log shipping"]
    SNAP --> OBJ["Object storage in another account or region"]
    DUMP --> OBJ
    WAL --> OBJ
    OBJ --> TEST["Regular restore test in a scratch namespace"]
    TEST --> OK["Verified recovery time and data loss window"]
    

Two numbers to decide first

• How much data can we lose? Measured in minutes or hours. This sets how often backups must run.
• How long can we be down? This sets how fast your restore process must be, and therefore how often you must practice it.

Many storage problems turn out to be simple file ownership problems.

A new disk is often owned by the machine’s administrator. If your application runs as an ordinary, unprivileged user, as it should, it may be unable to write to the new disk. Kubernetes helps through a pod-level setting that assigns a group to the volume’s files, so the application’s user can write. On large disks with many files, changing ownership at every start can be slow, so newer versions let you control when it happens.

Storage speed is often the true limit of a database. It deserves early measurement.

A reliable stateful system layers several protections, because no single one is enough.

Failure scenarios and responses

• A pod crashes. It restarts with the same name and disk, and rejoins the group.
• A machine dies. The pod waits for confirmation, then restarts elsewhere in the disk’s zone. Replicas serve users meanwhile.
• A zone fails. Members in other zones continue. Members in the failed zone remain pending until it returns, because their disks are there.
• A disk is corrupted. Replication may copy the corruption. Restore from a backup or a clean replica.
• Someone deletes the claim. With protections, the disk survives. Without them, only a backup can help.

Data at rest is the most valuable thing in the cluster. Guard it in layers.

Disks fail quietly, and they fill slowly. Good monitoring turns both into calm, planned work.

Alerts worth setting up

• A volume is more than seventy or eighty percent full, and forecast to fill within days.
• A claim has been pending for longer than a normal provisioning time.
• A member has been not ready for longer than an update should take.
• Replication lag rises above the limit you set based on your data-loss tolerance.
• The latest successful backup is older than the agreed limit.
• A disk’s latency or throttling indicator stays high.

StatefulSets solve real problems, and they bring real responsibilities.

The main trade-offs

Control versus convenience. Running your own database on a StatefulSet gives control over versions, settings, and cost. A managed database service gives convenience and expert operation. Many teams use managed services for critical data and StatefulSets for supporting systems.

Speed versus resilience. Local disks are fastest and least resilient. Network disks are slower and survive machine loss. Replication can recover resilience on top of local disks, at the cost of complexity.

Safety versus automation. Deleting claims automatically is tidy for tests and dangerous for production. Keeping them costs money and needs manual cleanup, but protects data.

These habits lead to lost data, stuck pods, and long nights.

A short checklist prevents most stateful surprises.

Best practices

Common mistakes

The same storage ideas appear wherever data must be remembered on Kubernetes.

Match the symptom to the likely cause, then follow the steps.

Let us design storage for a small three-member database, describing each step in words rather than code.

You need a database with one primary and two replicas, spread across three zones, with backups and room to grow.

What this walkthrough teaches

Notice that the walkthrough begins with two business numbers, not with technology. The tolerable data loss and downtime decide everything that follows. Storage design is the art of turning those two numbers into disks, replicas, and backups.

Practice makes the choice quick. Consider four systems and choose a pattern for each.

Questions to ask about any system

• Does each copy own its own data, or is the data shared?
• Can the software replicate data between copies, or must the storage protect it?
• Could the data be rebuilt from somewhere else, and how long would that take?
• How much data loss and downtime can the business accept?
• Would a managed service make this problem someone else’s specialty?

A plain StatefulSet leaves many chores to you. Operators and managed services take some of them away.

A bare StatefulSet knows nothing about your particular database. It cannot promote a replica, take a consistent backup, or decide whether a member has caught up. An operator is a specialized controller, written by people who know one database well, that adds this knowledge. It watches a custom object that describes the whole database cluster, and creates the StatefulSets, claims, services, and backup jobs needed to match it. A managed service goes one step further: a cloud provider runs the database for you, and Kubernetes only hosts the applications that call it.

How to choose

• If the data is critical and your team has little database experience, prefer a managed service.
• If you need specific versions, extensions, or cost control, consider a well-maintained operator.
• If the workload is small, temporary, or a learning exercise, a plain StatefulSet is a good way to understand the ideas.
• Whatever you choose, test recovery. The tool that takes work away should be proven, not trusted.

Storage costs money every hour, and running out costs far more. A little arithmetic finds the balance.

Capacity planning answers three questions: how much space is needed now, how fast it grows, and how much headroom to keep. Because disks can be enlarged but not shrunk, and because each member of a replicated group holds its own full copy, sizing errors are expensive in both directions.

Quick answers to the questions that come up most often.