Before CSI existed, every storage system had to be built into Kubernetes. That created a painful bottleneck.

Containers forget, storage remembers

A container’s own filesystem disappears when the container is replaced. That is fine for a stateless web server, but a database needs its data to survive restarts, rescheduling and node failures. The answer is a volume: a piece of storage that lives outside the container and can be attached to whichever node runs the Pod. Volumes can be network disks from a cloud, shared file systems, local drives, or storage appliances in a data centre.

The in-tree era

In the early years, the code that knew how to talk to each storage system was part of the main Kubernetes source code. These were called in-tree volume plugins, because they lived inside the Kubernetes code tree. Support for a cloud disk, a file protocol or an appliance meant a new chunk of code inside Kubernetes itself, built and released together with everything else.

A short tour of the Kubernetes objects that drivers serve, so the rest reads easily.

The three everyday objects

Dynamic provisioning

In the old manual style, an administrator pre-created volumes and users claimed them. In dynamic provisioning, nobody pre-creates anything. When a claim names a StorageClass, the system asks the driver named in that class to create a fresh volume on demand, and then registers it as a PersistentVolume that binds to the claim. This is what makes self-service storage possible, and it is the main workflow CSI drivers support.

Reclaim policy

Every dynamically created volume carries a rule for what happens when its claim is deleted. Delete means the real storage is destroyed too. Retain means the PersistentVolume and its data are kept for an administrator to deal with. Choosing between them is a data-safety decision, not a technical detail, and we revisit it in the anti-patterns chapter.

Binding mode

A StorageClass also chooses when a volume is created. With immediate binding, the volume is created as soon as the claim appears. With wait-for-first-consumer binding, creation is postponed until a Pod that uses the claim has been scheduled, so the volume can be created in the right place. Chapter 11 explains why this matters.

A written contract between an orchestrator and a storage driver, spoken over a simple local channel.

A specification shared across systems

The Container Storage Interface is an open specification that describes a small set of operations a storage driver must offer: create a volume, delete a volume, attach it to a machine, mount it at a path, take a snapshot, and so on. It was designed together by several container orchestrators and storage vendors, so a driver written once can in principle serve more than one orchestrator. Kubernetes is the most prominent user.

The language of the contract: gRPC

The operations are defined as gRPC calls, a widely used way for programs to call functions in other programs over a connection. On a single node, the program that asks and the driver that answers talk through a local file-like endpoint called a Unix domain socket. There is no network involved for that conversation, only a socket file on disk, which keeps it fast and limits who can talk to the driver.

Three services in one contract

The specification groups its operations into three named services. A driver offers some or all of them, depending on what it can do.

Optional capabilities

Not every driver can do everything. A simple network file system may never need an attach step. A local disk driver may not offer snapshots. Each driver therefore reports a list of capabilities through the Identity and Controller and Node services, and Kubernetes only calls the operations that the driver says it supports. This keeps the contract small for simple drivers and rich for advanced ones.

Two kinds of running programs, placed in two different spots of the cluster.

Two halves of one driver

A CSI driver is usually delivered as two deployments of the same driver software. The controller plugin runs a small number of copies somewhere in the cluster, serves the Identity and Controller services, and talks to the storage system’s management interface, often a cloud API. The node plugin runs one copy on every node that can use the storage, serves the Identity and Node services, and performs the machine-level work of preparing and mounting volumes.

flowchart LR
  subgraph CP["Control plane side of the driver"]
    PR["external-provisioner"]
    AT["external-attacher"]
    SN["external-snapshotter"]
    RS["external-resizer"]
    CD["Driver: Identity and Controller services"]
    PR --> CD
    AT --> CD
    SN --> CD
    RS --> CD
  end
  CD --> ST[("Storage system API")]
  subgraph ND["Every node"]
    KL["Kubelet"]
    RG["node-driver-registrar"]
    NDD["Driver: Identity and Node services"]
    KL --> NDD
    RG --> KL
  end
  NDD --> ST
        

Why not one program?

It would seem simpler to ship one program that does everything. But the two halves have very different needs. Controller work needs credentials for the storage system and does not need access to any node’s filesystem. Node work needs deep access to the node’s filesystem and devices and should not carry broad cloud credentials. Separating them follows the principle of least privilege, scales the right part in the right way, and lets each half be upgraded or restarted independently.

What each service can be asked to do, in plain language.

Identity service

The Identity service answers three questions. Plugin info returns the driver’s name and version. The name is important because it must match the provisioner name written in StorageClasses. Plugin capabilities tells callers whether the driver offers controller features and whether volumes can be reached only from some parts of the cluster. Probe reports whether the driver is ready to work, and is used by health checks.

Controller service

Node service

The driver does not watch Kubernetes. A set of helper programs do that on its behalf.

The sidecar idea

The CSI driver itself knows nothing about Kubernetes objects. It only answers CSI calls. To connect Kubernetes to the driver, the Kubernetes storage community provides a set of generic helper programs called sidecars. Each one is a small controller that watches certain Kubernetes objects and translates changes into CSI calls to the driver, over a local socket shared in the same Pod. Driver authors reuse these helpers instead of writing Kubernetes logic themselves.

Informers and queues again

If you have read about how Kubernetes controllers work, these sidecars will feel familiar. Each one uses informers to watch objects, a work queue to process changes, and a reconcile step that calls the driver and writes results back. The same list-and-watch pattern, the same retries with back-off and the same level-based thinking apply. A sidecar that restarts simply re-lists and carries on.

High availability and leader election

The controller plugin is usually run with several replicas for availability. Only one replica of each sidecar should act at a time, otherwise two copies might create the same volume twice. The sidecars therefore use leader election, so that one copy is active and the others stand by. Because leader election is per sidecar, the active provisioner and the active attacher may in principle be in different replicas.

flowchart TD
  K[("Kubernetes API objects")] -- "watch" --> S1["Sidecar: provisioner"]
  K -- "watch" --> S2["Sidecar: attacher"]
  S1 -- "CSI call over local socket" --> D["Driver container: Controller service"]
  S2 -- "CSI call over local socket" --> D
  D -- "management API call" --> X[("Storage system")]
  D -- "result" --> S1
  S1 -- "write PV and status" --> K
        

The part that touches the machine: devices, filesystems and mount points.

Deployed on every node

The node plugin runs as a DaemonSet, which means Kubernetes keeps exactly one copy on each eligible node. Because it must create mounts that are visible to the kubelet and to containers, it needs special privileges and special access to the node: it normally runs as a privileged container and shares parts of the node’s directory tree with the host.

The kubelet is the caller

On the node side the caller is not a sidecar but the kubelet, the agent that starts Pods on that node. When a Pod needs a CSI volume, the kubelet calls the node plugin’s stage and publish operations over a Unix socket that the plugin has created in a known folder under the kubelet’s directory.

Registration

How does the kubelet find the driver’s socket? A helper sidecar called the node-driver-registrar runs beside the node plugin. It announces the driver to the kubelet through a registration socket that the kubelet watches. The kubelet then calls the driver’s node information operation, learns the node’s identity and limits in the storage system’s language, and records the driver in a per-node Kubernetes object. From that point the kubelet knows the driver is available on this node.

Mount propagation

A subtle but vital detail is how mounts made by the node plugin become visible to the application container. A container normally has its own view of the filesystem. For this to work, the node plugin’s container shares the kubelet’s Pod directory with the host using a setting called bidirectional mount propagation, so that a mount made inside the plugin appears on the host and therefore in the application’s Pod directory. Without it, the plugin would mount the volume in its own private view and the application would see an empty folder.

Besides claims and volumes, three more objects tell Kubernetes how a driver behaves.

CSIDriver: the driver’s settings

A CSIDriver object is a cluster-wide record that declares how Kubernetes should treat a driver. It is optional but very useful. It can say whether volumes need an attach step at all, whether the kubelet should pass Pod details to the driver when mounting, how file ownership should be applied to volumes, whether the driver supports tracking available capacity, and which lifecycle modes it supports, such as persistent volumes or short-lived ephemeral ones.

CSINode: what each node can do

A CSINode object exists for each node. It lists the drivers registered on that node, the node’s identifier in each driver’s world, the topology keys that describe where the node sits, and sometimes the maximum number of volumes the node can hold for each driver. The scheduler uses this to avoid placing a Pod on a node that has reached its volume limit.

VolumeAttachment: the attach request

A VolumeAttachment object records the intent to attach a particular volume to a particular node. It is created by Kubernetes when a Pod using a volume is scheduled to a node, and is watched by the attacher sidecar. Its status shows whether attachment succeeded, and any error. It is the single most useful object to inspect when a Pod is stuck waiting for its disk.

Why the attach step may not exist

Not all storage needs attach and detach. A cloud block disk must be attached to a specific virtual machine before it can be used. A network file system, on the other hand, is simply mounted over the network by any node that knows the address. A driver declares which style it is. When no attach is needed, Kubernetes skips the VolumeAttachment step completely and goes straight to node-level stage and publish, which makes startup of such Pods faster.

Follow one volume through every component, from the first request to a mounted folder.

The story in order

An application owner creates a claim naming a StorageClass. The provisioner sidecar sees it, asks the driver to create a volume, and records a PersistentVolume. Kubernetes binds the claim to that volume. When a Pod using the claim is scheduled to a node, Kubernetes creates a VolumeAttachment. The attacher sidecar asks the driver to attach the volume to that node. The kubelet then waits for attachment to finish, asks the node plugin to stage the volume, then to publish it into the Pod’s directory, and finally starts the containers with the folder in place.

sequenceDiagram
  participant U as User
  participant API as Kubernetes API
  participant PR as Provisioner sidecar
  participant CD as Controller driver
  participant AT as Attacher sidecar
  participant KL as Kubelet
  participant ND as Node driver
  U->>API: Create claim with a StorageClass
  PR->>API: Watch sees new claim
  PR->>CD: CreateVolume
  CD-->>PR: Volume ID
  PR->>API: Create PersistentVolume, claim binds
  U->>API: Create Pod that uses the claim
  API->>API: Scheduler picks a node
  API->>API: Attach logic creates VolumeAttachment
  AT->>CD: ControllerPublishVolume to that node
  CD-->>AT: Attached
  AT->>API: Mark VolumeAttachment attached
  KL->>ND: NodeStageVolume
  ND-->>KL: Staged at global path
  KL->>ND: NodePublishVolume
  ND-->>KL: Mounted into Pod directory
  KL->>KL: Start containers
        

Reading the diagram

The top half belongs to the cluster-wide side: creating the volume and binding it. The middle is the scheduler and the attach request. The bottom is the node’s own work. Notice that the controller driver is called twice, once to create and once to attach, while the node driver is called twice as well, to stage and to publish. Each call is a small, separate, retryable step, and failure at any point leaves the system in a state from which it can continue.

Why the node side has two steps, and what happens when a Pod goes away.

Why two node steps

A single volume can be used by several Pods on the same node at the same time, for instance when a shared volume is used by two Pods. The expensive, once-per-volume work, such as finding the device and creating a filesystem, should be done once. The cheap, per-Pod work, which is placing the volume into one Pod’s directory, should be done for each Pod. So stage does the expensive part once per node and volume and mounts the volume at a shared staging path. Publish then makes a lightweight bind mount from the staging path into each Pod’s own directory.

The reverse journey

When a Pod is deleted, the order reverses. Unpublish removes the bind mount for that Pod. When the last Pod on the node using the volume is gone, unstage unmounts the staging path. If the volume is no longer needed on that node, the attach logic removes the VolumeAttachment and the attacher asks the driver to detach. Finally, if the claim itself is deleted and the reclaim policy is Delete, the provisioner asks the driver to delete the real volume.

A volume’s location constrains where the Pod can run, and the Pod’s location constrains where the volume should be created.

Storage has a location

Many storage systems are tied to a place. A cloud block disk lives in one availability zone and can only be attached to machines in that same zone. A local disk lives on one specific node. If a volume is created in zone A and the Pod is later scheduled into zone B, the Pod can never mount it and will be stuck. CSI handles this with topology: drivers describe where nodes are and where volumes can be reached from, using labels such as zone.

The two binding modes revisited

With immediate binding, the volume is created as soon as the claim appears, before the scheduler has chosen a node for the Pod. The driver has no idea where the Pod will go, so it picks a location on its own. The scheduler is then forced to place the Pod where the volume is. This can leave a Pod unschedulable if that zone is full.

With wait-for-first-consumer binding, creation is postponed. The scheduler first picks a node considering everything else, such as CPU, memory and affinity, and also considering the topology that the driver’s storage supports. Only then is the volume created in the chosen node’s location, passed to the driver as a preferred topology. This is the recommended mode for location-bound storage.

flowchart TD
  C["Claim with wait-for-first-consumer class"] --> P["Claim stays pending, no volume yet"]
  P --> S["Pod created, scheduler evaluates nodes"]
  S --> N["Scheduler picks a node in zone B"]
  N --> T["Chosen node's topology sent to provisioner"]
  T --> V["Driver creates the volume in zone B"]
  V --> B["Claim binds, Pod proceeds to attach and mount"]
        

Allowed topologies

A StorageClass can also restrict volumes to certain locations, for example only two of three zones. This is useful when storage exists only in some places or when data should stay in specific regions. The scheduler and provisioner both honour the restriction.

Volume limits per node

Machines can only hold so many attached volumes. A driver reports the limit through the node information call, and Kubernetes stores it in the node’s record. The scheduler then counts volumes per node and avoids exceeding the limit, so Pods are not sent to a node that has no room for another disk.

What a driver can offer beyond create and mount, and how each feature maps onto the two plugins.

Snapshots

A snapshot is a point-in-time copy of a volume. Kubernetes models it with separate objects: a request for a snapshot, a record of the real snapshot, and a class that names the driver. Snapshots are handled by a snapshot controller in the cluster together with the external-snapshotter sidecar in the controller plugin. A snapshot is a storage-side operation, so it needs only the controller service, and the node plugin takes no part.

Cloning and restoring

A new claim can name an existing snapshot or even an existing claim as its data source. The provisioner passes that source to the driver’s create operation, and the driver builds the new volume from it. This makes it quick to create test copies of databases or to restore after a mistake, depending on how efficiently the storage system implements copying.

Expansion

Growing a volume has two halves. The controller side makes the storage bigger. The node side then grows the filesystem inside it so the extra room is usable. Some drivers can do both while the volume stays mounted, called online expansion, while others need the Pod to be stopped first. Shrinking is not supported.

Access modes and volume modes

Separately, a volume can be presented as a filesystem, which is a mounted folder, or as a raw block device, which the application manages itself, common for databases that want direct control. The node plugin handles the difference at publish time.

Ephemeral volumes

CSI also supports volumes that live and die with a single Pod. In the generic style, the Pod’s own definition carries a claim template, and a claim is created and removed along with the Pod, going through the normal provisioning path. Some drivers also support a lighter inline style, where the Pod names the driver directly and the node plugin does the work without any claim. These are useful for scratch space and for injecting data such as secrets from an external store.

Distributed systems fail halfway through jobs. CSI is designed so that doing a step twice is always safe.

Idempotent by contract

An operation is idempotent when repeating it has the same effect as doing it once. The CSI specification requires this for most calls. If a create-volume request is repeated with the same name, the driver must return the same volume, not create a second one. If a delete is repeated for a volume that is already gone, it must succeed rather than fail. This rule makes retries safe, and retries are the main recovery tool in the whole system.

Names make idempotency possible

The provisioner gives each create request a stable name derived from the claim’s unique identity. If the provisioner crashes after the driver created the volume but before it saved the result, the next attempt uses the same name, and the driver can recognise that the volume already exists and return it. Without stable names, such a crash would leak an orphaned volume.

Timeouts and in-progress states

Storage operations can be slow. A sidecar sets a timeout on each call. If a call times out, the sidecar does not know whether the operation completed, so it retries, and the driver should answer with the current state, or with an “operation in progress” signal if the previous attempt is still running. Drivers protect themselves with locks so that two concurrent calls about the same volume do not collide.

The hard case: a node that disappears

If a node vanishes suddenly, its attached disks cannot be cleanly unmounted. Kubernetes waits for a safety period before force-detaching, because attaching a read-write disk to a second node while the first might still be writing risks serious corruption. This is a deliberate trade of availability for safety. Newer features allow an administrator to mark a node as truly shut down, which allows faster, safe recovery. Always follow the documented procedure for your version instead of removing objects by hand.

How the pieces are packaged in a real cluster, and how to change them without trouble.

The typical layout

Installation approaches

Most drivers are installed with a packaged bundle, such as a Helm chart, an operator or a set of manifests published by the driver’s maintainers. These bundle the workloads, the access rules, the CSIDriver object and the default StorageClasses. Managed Kubernetes services often offer the main cloud driver as an add-on that the provider installs and updates for you. Using the provider’s own add-on is generally the safest choice where available.

Upgrades

How Kubernetes moved existing users from built-in volume code to drivers without breaking their clusters.

The goal

Once CSI matured, the project set out to remove the old in-tree volume code. But thousands of clusters already had claims and volumes created through the old plugins, with old-style names in their specifications. Forcing everyone to rewrite them would have been disruptive. The solution was CSI migration: a translation layer inside Kubernetes that quietly redirects calls meant for an old in-tree plugin to the matching CSI driver.

How it works in practice

When migration is active for a volume type, the existing in-tree volume specification is translated on the fly into its CSI form, and the CSI driver does the actual work. Users do not edit their existing objects. For this to work, the matching CSI driver must be installed and healthy in the cluster. Over several releases, migration was enabled by default for the major cloud disk and file plugins, and most of the old in-tree code has since been removed from the project.

What this means for you

Where CSI drivers show up, and how the controller and node roles differ between them.

Storage is rarely fast to create, so most tuning is about concurrency and avoiding pile-ups.

Where time goes

Creating a cloud disk may take several seconds. Attaching can take longer. Formatting a large volume adds more. Mounting is quick, but a volume with a huge number of files may be slow to apply ownership changes at mount time. Most delays are inside the storage system or the machine, not in Kubernetes itself, so it is important to know which stage is slow before you tune anything.

Tuning options

Scale effects

In a cluster with thousands of nodes, a node plugin on every node means thousands of registrations and thousands of per-node records. The records are small, but a rolling restart of the whole node plugin DaemonSet can briefly cause a storm of re-registrations. Pace rollouts, and make sure the API server and the driver’s own back-end can absorb the burst.

Storage drivers hold keys to data and to hosts. They deserve careful hardening.

Two privilege profiles

The two plugins are powerful in different ways. The controller plugin holds credentials for the storage system, so a compromise could let an attacker create, read through snapshots, or delete volumes across the whole environment. The node plugin holds host-level filesystem access, so a compromise could expose or change other Pods’ data on that node. Treat both as high-value targets.

Secrets in classes

Some drivers need extra credentials per class or per volume, for example for a storage array. A StorageClass can point to a Kubernetes Secret and the sidecars pass its contents to the driver. Keep these Secrets in a locked-down namespace, give the sidecars permission to read only those specific Secrets, and rotate them on a schedule.

Data at rest and in transit

File ownership policy

When a Pod mounts a volume, the group ownership of its files may need adjusting so the Pod’s user can read and write. The driver’s declared policy controls whether Kubernetes does this and how. Choosing the wrong policy can either block access or recursively change ownership on large volumes, which is slow and may surprise applications that expect existing ownership to remain.

How to see what the driver is doing, and how to find the stuck step when a Pod will not start.

Signals worth watching

A troubleshooting path for a Pod stuck waiting for its volume

Common symptoms and likely causes

Habits that make storage boring and reliable.

Patterns that work

Failures that appear again and again in real clusters.

An honest balance sheet for the CSI approach.

Choosing a storage style for a workload

Questions that beginners and experienced engineers keep asking.

The specification is separate and shared with other orchestrators. Kubernetes includes the machinery that talks CSI, such as the kubelet’s volume handling and the API objects, but the drivers and most sidecars live outside the core code and are released on their own schedules.

Because the work happens in two different places. Some actions affect the storage system as a whole and can run anywhere, others must run on the machine that will use the volume. Splitting them gives each the right privileges and scale, and keeps broad storage credentials away from every node.

Sidecars are generic helper controllers that watch Kubernetes objects and call the driver’s CSI operations. This keeps drivers free of Kubernetes-specific logic, so one driver can work with several orchestrators, and driver authors write only storage code.

Stage prepares a volume once per node, such as finding the device, formatting it and mounting it at a shared path. Publish makes it appear inside one Pod’s directory, usually by a bind mount from the staging path. This lets many Pods on one node share one prepared volume.

No. Block disks usually do, because they must be connected to a machine first. Network file systems typically do not, because nodes simply mount a network path. The driver declares which style it uses in its CSIDriver object.

Mounted volumes usually keep working because the kernel handles the data path. What stops is new work: creating, attaching, mounting, resizing and snapshotting. Once the driver returns and re-registers, those operations resume.

Binding only means a volume exists for the claim. The Pod may still be waiting for scheduling constraints, for the attach to complete, or for the node plugin to stage and publish. Look at the Pod’s events and at the VolumeAttachment to see which step is active.

Because location-bound volumes must be created where the Pod will run. Delayed binding waits for the scheduler’s decision and then creates the volume in the chosen node’s zone, avoiding Pods that can never mount a volume created somewhere else.

Not directly. A zonal volume cannot be attached across zones. The usual approach is to take a snapshot and create a new volume from it in the target zone, if the driver supports snapshots and cross-zone restore.

Not without understanding why it is stuck. Removing it hides the symptom while the real disk may still be attached or mounted elsewhere, which risks corruption. Investigate the node, the storage side and the documented recovery steps first.

It is a translation layer that redirects the old in-tree volume plugins to matching CSI drivers so existing objects keep working. You need to ensure the matching CSI driver is installed before your cluster version crosses the point where the old plugin is disabled or removed. Check the release notes for your version.

Yes, and this is common. Each driver has its own name, its own controller and node plugins and its own classes, and claims pick one through the class. Shared pieces such as the snapshot controller and snapshot definitions must be installed once for the whole cluster.

The Container Storage Interface replaced storage code built into Kubernetes with a standard contract spoken over local gRPC sockets. A driver offers up to three services: Identity for self-description, Controller for whole-system actions such as create, delete, attach, snapshot and expand, and Node for machine-level actions such as stage, publish, unpublish and unstage. In practice that becomes two deployments. The controller plugin runs a few replicas with generic sidecars that watch Kubernetes objects and call the driver, and holds the storage credentials. The node plugin runs on every node, is called by the kubelet, registers itself through a registrar sidecar, and holds host-level mount access. Supporting objects such as CSIDriver, CSINode and VolumeAttachment describe driver behaviour, node capabilities and attach progress.

Return to the apartment building from the start. The back office allocates your locker and records its number. The porter on your floor brings it to your door. Neither does the other’s job, each is repeatable and safe to ask twice, and a clear register lets anyone see where things stand. If you remember that picture, and choose delayed binding for location-bound storage, protect both privilege profiles, monitor each step and test your restores, CSI storage becomes predictable and quietly reliable.