Before we meet the solution, let us feel the pain it removes. Storage only makes sense once you see what goes wrong without it.

A quick look at containers and Kubernetes

A container is a small, sealed box holding a program and everything it needs to run. Kubernetes is the free, open-source system that starts, stops, moves and repairs thousands of these boxes across many computers. A group of computers managed together is a cluster, and each computer in it is a node. Containers run inside small wrappers called Pods, the smallest thing Kubernetes manages.

Why data disappears

Every container gets a private, temporary writing area when it starts. It lives and dies with the container. Kubernetes is designed to replace containers freely. A Pod may be restarted after a crash, moved to a healthier node, or rebuilt during an update. Each time, the fresh container starts with a clean, empty area, exactly like the whiteboard after cleaning night.

• A container crashes: its writing area is wiped when it restarts.
• A Pod is deleted or updated: the new Pod knows nothing about the old one.
• A node fails: everything stored on that node’s temporary space may be lost.

For a static website this is fine. For a database, it would be a disaster.

Eight small ideas carry this whole topic. Learn them once and the rest will feel natural.

Kubernetes storage uses three objects that work together. Each has one clear job, like the roles in a restaurant.

Why split it into supply and request?

The split keeps two groups of people from stepping on each other. The platform team knows about disks, speeds and prices, so they define the supply and the menu. The application developers only know how much space they need and how it should behave. Developers ask, the platform delivers, and neither needs to learn the other’s whole job.

Let us define it in plain words, and then compare it with the other places a Pod could keep data.

A Persistent Volume is a Kubernetes object that represents a chunk of storage with its own life, separate from any Pod. It has a size, a set of allowed ways to use it, a note about what happens when it is no longer needed, and a pointer to the real storage behind it. Even if every Pod using it vanishes, the PV and the bytes inside continue to exist.

The word “persistent” simply means “keeps existing”. A regular volume is tied to a Pod’s life. A persistent one is tied to its own life, managed separately.

Comparing the ways to store data

Here is the whole picture in one drawing. Read it from the Pod on the left to the real storage on the right.

flowchart LR
    subgraph NS["Namespace: shop"]
        POD["Pod: Database"] --> PVC["PersistentVolumeClaim"]
    end
    PVC -->|bound to| PV["PersistentVolume"]
    PV --> CSI["CSI Driver"]
    CSI --> BACK["Storage Backend: Cloud Disk or File Server"]
    SC["StorageClass"] -. defines how to create .-> PV
    CTRL["PV Controller"] -. matches claims and volumes .-> PVC
    KUB["Kubelet on the Node"] -. mounts the volume into .-> POD
    

Reading the diagram

The solid arrows show the path that data takes: the Pod writes to a folder, the folder belongs to a claim, the claim is bound to a volume, the volume is served by a driver, and the driver reaches the real storage. The dotted arrows show the helpers that make this possible behind the scenes.

Meet each component

There are two ways a PV can come into being. One is like buying furniture in advance. The other is like ordering it made to fit.

Static provisioning: prepared in advance

An administrator creates the real disk by hand, then writes a PV object describing it. The PV sits in the cluster with the state “Available”, waiting. When a matching claim arrives, the two are paired. This gives tight control but takes manual effort for every disk.

Dynamic provisioning: made when asked

Here nobody prepares disks in advance. A developer writes a claim that names a StorageClass. The CSI driver automatically creates a new disk in the storage system, then creates a PV to represent it, and Kubernetes binds the two. This is how almost all modern clusters work, because it is fast, repeatable and hands-off.

sequenceDiagram
    participant Dev as Developer
    participant API as API Server
    participant CSI as CSI Provisioner
    participant Store as Storage Backend
    participant Kub as Kubelet
    Dev->>API: Create a claim that names a StorageClass
    API-->>CSI: New claim is waiting
    CSI->>Store: Please create a disk of this size
    Store-->>CSI: Disk created
    CSI->>API: Create a PersistentVolume for that disk
    API-->>API: Bind the claim and the volume together
    Dev->>API: Create a Pod that uses the claim
    API-->>Kub: Start the Pod on a node
    Kub->>Store: Attach and mount the disk
    Kub-->>Dev: Pod is running with its data
    

Every PV moves through a small number of states. Reading them is the fastest way to understand what is going on.

stateDiagram-v2
    [*] --> Available
    Available --> Bound : a matching claim arrives
    Bound --> Released : the claim is deleted
    Released --> Available : admin cleans and reuses it
    Released --> Failed : automatic cleanup did not work
    Released --> [*] : deleted under the Delete rule
    

How a claim finds its volume

When a claim looks for a volume, the matchmaker checks a few things. The volume must be big enough, and it must offer the access style the claim asked for. If the claim names a StorageClass, the volume must belong to the same one. Labels can narrow the choice further. Among matches, the smallest one that fits is usually chosen so bigger volumes are not wasted. A claim can also name a specific volume directly.

Protection from accidents

Kubernetes has a safety net called storage object in-use protection. If a Pod is still using a claim, and someone deletes the claim, Kubernetes holds off on the removal until the Pod is gone. A volume that is still bound to a claim is also protected. This prevents the nightmare of pulling a disk out from under a running database.

Two settings decide who may use the storage at the same time and in what form.

Access modes: who may share it

Notice that the word “once” in ReadWriteOnce refers to a node, not a Pod. Several Pods on the same node can share such a volume. If you need a strict guarantee of a single Pod, use the ReadWriteOncePod mode.

Volume modes: what shape it takes

In Filesystem mode, which is the default, the storage is formatted and shown to the container as a normal folder. In Block mode, the container receives the raw, unformatted disk. Some specialized databases prefer this because they manage the disk’s layout themselves. Most applications should use the ordinary Filesystem mode.

When a claim is deleted, what should happen to the storage and the data inside? The reclaim policy answers that question.

There was once a third option called Recycle, which wiped the data and made the volume available again. It is deprecated, and dynamic provisioning is recommended instead.

Timing matters too: binding mode

A StorageClass also has a binding mode. With Immediate, the disk is created the moment the claim is written. With WaitForFirstConsumer, Kubernetes waits until a Pod that uses the claim is placed on a node, and only then creates the disk in a location that node can reach. This second choice is very important on clouds where disks belong to a single zone, which the next chapter explains.

The objects look the same everywhere, but what sits behind them depends on where your cluster runs.

The zone trap

Many cloud disks live in one availability zone, which is a separate building within a cloud region. A disk in Zone A cannot be attached to a machine in Zone B. If a Pod using that disk gets rescheduled to a node in another zone, it will be stuck waiting, unable to reach its data.

flowchart LR
    subgraph ZA["Zone A"]
        DISK["Cloud Disk in Zone A"]
        NA["Node in Zone A"]
    end
    subgraph ZB["Zone B"]
        NB["Node in Zone B"]
    end
    DISK ==>|can attach| NA
    DISK -. cannot attach .-> NB
    

Kubernetes handles this by writing a location rule, called node affinity, onto the PV. The scheduler reads it and only places the Pod on nodes that can reach the disk. Using the WaitForFirstConsumer binding mode makes the disk appear in the same zone as the chosen node from the start.

What actually happens on the machine when your Pod starts and needs its data? Let us follow one write, from the moment the Pod is scheduled to the moment bytes land on disk.

Why moving a Pod can take a while

Detaching a disk from an old node and attaching it to a new one takes time. The storage system must finish the detach before it allows the attach. If the old node has crashed, the system may wait to be sure that node is truly gone before letting anyone else use the disk. This safety wait protects your data from being written by two machines at once, which could corrupt it. So during node failures, a stateful Pod may take a minute or more to recover, and you may see a warning about the disk still being in use elsewhere.

What if you run three copies of a database and each needs its own private storage? Kubernetes has a tool made exactly for this.

A normal Deployment treats all its Pods as identical and interchangeable, like paper cups at a party. A StatefulSet treats each Pod as an individual with a name, a number and an identity, like named guests at a dinner table. Pod number 0 is always called by its own name, and it always gets its own storage back after a restart.

The StatefulSet contains a template for claims. When it creates Pod 0, Pod 1 and Pod 2, it also creates three separate claims, one for each. If Pod 1 is killed and replaced, the new Pod 1 reconnects to exactly the same volume as before, with all its data intact.

Data changes and grows. These features help you resize and protect it without downtime nightmares.

Expanding a volume

If a StorageClass allows expansion, you can raise the size in your claim and the storage grows to match. Depending on the driver, the file system is stretched while the Pod keeps running, or after a restart. You can grow a volume but you cannot shrink it, so it is wise to start modestly and grow when needed rather than start huge.

Snapshots

A VolumeSnapshot is a frozen picture of a volume at one moment. It is like taking a photo of a whiteboard before you erase it. The storage system records the state efficiently, often without copying every byte at once. You can use a snapshot to restore data after a mistake, or to create a new volume that starts with the same contents. Snapshots need a CSI driver that supports them.

Cloning

A clone creates a new volume that begins as a copy of an existing claim. It is handy for testing new database versions on real-looking data without touching the original.

Every design has strengths and costs. Knowing both helps you decide well.

The trade-offs behind the choices

Speed versus sharing. Block disks are fast and simple but usually serve one node. Shared file systems serve many nodes but often have higher delay and cost more per gigabyte.

Safety versus tidiness. The Retain policy protects data but leaves orphaned disks to clean up. The Delete policy is tidy but unforgiving.

Own it versus rent it. Running a database yourself on Persistent Volumes gives control and portability. A managed database service removes chores such as backups and upgrades but ties you to a provider and often costs more.

Local speed versus resilience. Disks that sit inside the node are very fast, but they die with the node. Network disks are a little slower but survive machine failures.

Stored data is often the most valuable thing you own. Guard it in layers.

Cleaning up after yourself

When a volume is retired, the old data must not leak to the next user. Dynamic deletion normally removes the disk entirely. With reused volumes, make sure old contents are wiped. Dropping in a new claim on a dirty volume is like handing over a hotel room with the previous guest’s belongings still in the wardrobe.

A full disk at three in the morning is a classic emergency. A few alarms make it a non-event.

Think of the fuel gauge in a car. You do not wait for the engine to stop before you look at it. Storage deserves the same care.

What to watch

Alarms worth setting

• A volume passing about 80 percent full, with a stronger alarm near 90 percent.
• A claim that stays Pending longer than a few minutes.
• A Pod stuck in container creation because of a volume problem.
• Disk delay climbing above what your application tolerates.
• Any snapshot or backup job that fails or misses its schedule.

Patterns are solutions that many teams found reliable. Anti-patterns are traps many teams fell into.

Patterns that work

Traps to avoid

Small habits make a big difference when your data is on the line.

Habits that pay off

• Choose the right class for the job. Fast SSD for databases, cheaper storage for logs and archives.
• Turn on volume expansion. It lets you grow calmly instead of migrating in a panic.
• Prefer WaitForFirstConsumer on zonal clouds. The disk is then created near the node that actually needs it.
• Set the reclaim policy on purpose. Know whether deleting a claim deletes the data.
• Size with room to grow. Leave at least a fifth of the space free, and alarm before it runs out.
• Test your restores. A backup that has never been restored is only a hope.
• Use readiness checks and graceful shutdown. Give databases time to finish writing and close cleanly before a Pod is removed.
• Label everything. Add owner, team and purpose to claims so old volumes can be traced and cleaned up.
• Keep sensitive data encrypted. Use encrypted classes by default so nobody has to remember.

Mistakes beginners make

Here is how well-known organizations and typical setups use Persistent Volumes in practice.

Quick answers to the questions beginners ask most often.

You began with a whiteboard that gets wiped every night. Now you know how to build the filing cabinet next to it.