To understand raw block volumes, we first need to see what normally sits between your program and a disk.

A disk is just a long row of numbered boxes

Deep down, a storage disk, whether it is a spinning drive, a flash drive, or a cloud volume, knows nothing about files or folders. It is a very long row of equally sized slots called blocks, usually 512 bytes or 4,096 bytes each. You can say “read block 7,000,001” or “write these bytes into block 42”, and that is all the disk understands. Anything you can do with such a raw disk is done by reading and writing numbered blocks. A disk used this way is called a block device.

The filesystem: a librarian for the blocks

Nobody wants to remember that “my photo lives in blocks 800 through 1,200”. So operating systems put a layer on top, called a filesystem. Common ones include ext4 and XFS on Linux. A filesystem is like a librarian with a catalogue. It keeps track of which blocks belong to which file, organizes files into folders, remembers who owns what, and protects against half-finished writes using a journal. Your programs then simply ask for “/data/report.csv”, and the filesystem translates that into block numbers.

By default, when a Kubernetes Pod asks for storage, the cluster takes a raw disk, creates a filesystem on it if one does not exist yet, and mounts it at a folder path inside your container. You never see the blocks. This is the Filesystem volume mode, and it is right for the vast majority of applications.

Why anyone would give up the librarian

Some software already keeps its own catalogue. A database, for example, often has a carefully designed on-disk layout, its own caching, and its own method of making sure writes survive a crash. When such software sits on top of a filesystem, the work is partly duplicated: the filesystem keeps a journal, the database keeps a log, both cache the same data in memory, and each adds its own small delays. Handing the database the raw disk lets it skip the middle layer.

Other uses are different. A virtual machine running inside Kubernetes expects a whole disk to install its own operating system on. A storage system that is itself built on Kubernetes, such as a software-defined cluster of disks, wants to own the disks directly. A backup or data-copy tool may want to read every block exactly as it is. For all of these, a raw device is the natural form.

A dozen words appear again and again in Kubernetes storage. Learn them once and the rest of the guide reads easily.

The two volume modes side by side

Every PersistentVolume and every PersistentVolumeClaim has a field called volumeMode. It can be Filesystem, which is the default if you say nothing, or Block. With Filesystem, Kubernetes prepares a directory for your container. With Block, Kubernetes places a device file, a special entry such as /dev/xvda, inside the container. Reading from that file reads raw blocks. Writing to it writes raw blocks. There is nothing else on it unless you put something there.

A raw block volume uses the same storage machinery as any other volume. Only the last step differs.

flowchart LR
  U["User creates PVC with volumeMode Block"] --> SC["StorageClass picks the CSI driver"]
  SC --> PR["CSI controller creates the disk"]
  PR --> PV["PersistentVolume with volumeMode Block"]
  PV --> BIND["PVC and PV bound"]
  BIND --> ATT["Attach disk to the chosen node"]
  ATT --> MAP["Kubelet maps the device, no filesystem mount"]
  MAP --> CT["Container sees a device file at devicePath"]
    

Read the diagram from left to right. A user files a request, the PVC, and states that they want the Block mode. A StorageClass tells Kubernetes which driver to use and with what settings. The driver’s controller part creates a real disk in the storage system and registers it as a PersistentVolume. Kubernetes binds the PVC to the PV. When a Pod using the claim is scheduled, the disk is attached to the node, and finally the kubelet exposes the device inside the container. Up to the last step the process is the same as for any volume.

The CSI contract: two kinds of access

The Container Storage Interface, or CSI, is the standard that lets storage vendors write one driver that works with every container platform. Part of the standard describes the type of access a volume is wanted for: either mount, meaning a filesystem, or block, meaning a raw device. When Kubernetes asks the driver to publish a volume on a node, it states which one. A driver that supports block mode is able to hand over a device file in place of a mounted folder. A driver that does not support it will refuse, and the Pod will not start.

Looking inside shows exactly what is removed in block mode, and what stays the same.

flowchart TD
  subgraph FS["Filesystem mode"]
    A1["Application"] --> A2["Filesystem: files, folders, journal"]
    A2 --> A3["Page cache in memory"]
    A3 --> A4["Block layer and I/O scheduler"]
    A4 --> A5["Disk"]
  end
  subgraph BL["Block mode"]
    B1["Application with its own layout"] --> B4["Block layer and I/O scheduler"]
    B4 --> B5["Disk"]
  end
    

Compare the two columns. In Filesystem mode, a write passes through the filesystem, which decides which blocks to use and records the change in its journal. It often lands in the page cache, a region of memory where the system keeps recent data for speed, before the block layer finally sends it to the disk. In Block mode, the application talks straight to the block layer. There is no filesystem to consult, and the application itself decides which blocks to write and when to flush them. The block layer and the disk are identical in both cases, which is why the physical limits of the disk do not change.

What the kubelet does differently

No formatting, ever, by Kubernetes

This is the most important rule to remember. In Block mode, Kubernetes never creates a filesystem on the device and never checks one. If the disk is new, it is blank. If it carried data from before, that data is still there. Your application must recognise whether the device is already initialised, and must never assume that a new volume has any particular content.

Only a few fields change when you move from a normal volume to a raw one. Knowing exactly which ones prevents most early errors.

Step one: ask for Block in the claim

In the PersistentVolumeClaim, set volumeMode to Block. Everything else, the size, the access mode, the StorageClass, works as usual. If you leave the field out, the claim asks for a Filesystem. The claim does not mention formats, because there is no format.

Step two: use volumeDevices, not volumeMounts

In the Pod, a Filesystem volume appears under volumeMounts with a mountPath, a folder inside the container. A raw block volume appears under a different list, volumeDevices, with a devicePath, which is the location of a device file, for example /dev/appdata. The two lists are separate. A single volume name cannot appear in both. Pick the path carefully: it must not clash with a device or folder that already exists in the container, and it should sit under the device folder by convention.

What the application does with the device

Inside the container, the application opens the device file the same way it opens any file, but reads and writes at block offsets. Tools commonly used for testing and inspection include lsblk to list devices, blockdev to find the size, dd to copy bytes, and fio to measure performance. A program can also create its own filesystem on the device and mount it, but that needs extra privileges and is usually a sign that Filesystem mode would have been simpler.

A claim only binds to a volume that matches its needs. Volume mode is one of the matching rules, and mistakes here leave claims waiting forever.

Mode must match

When Kubernetes looks for a PersistentVolume for a claim, it checks size, access mode, StorageClass and labels, and also the volume mode. A claim that asks for Block will not bind to a Filesystem volume, and the reverse is also true. If the only available volumes have the wrong mode, the claim stays Pending, and the events explain that no matching volume was found. For dynamic provisioning, the driver must be able to create a block volume, or the creation fails.

Dynamic provisioning: the usual route

In most clusters you do not create PersistentVolumes by hand. You create a claim that names a StorageClass, and the driver creates the disk on demand. For block mode, this works exactly the same way, as long as the driver supports it. The disk type, speed and encryption come from the StorageClass settings. A good habit is to create a dedicated StorageClass for block workloads, with a clear name, so nobody uses the wrong one by accident.

Static provisioning: bringing your own disk

With static provisioning, an administrator creates a PersistentVolume that points to an existing disk or network target and marks its mode as Block. This route is common for local NVMe drives on a node, for pre-created SAN volumes, and for special hardware. It gives total control but also requires care, because the disk may hold old data.

Timing: wait for the Pod

Many cloud disks exist in a single availability zone and can only attach to nodes in that zone. If the disk is created before the scheduler has chosen a node, the disk may land in a zone where the Pod cannot run. The setting volumeBindingMode: WaitForFirstConsumer on the StorageClass solves this: the disk is created only after the Pod is scheduled, in the same zone as the chosen node. This is strongly recommended for zone-bound cloud disks, in both volume modes.

The honest answer to “should I use block mode?” is “usually not”. Here is how to tell the exceptions.

flowchart TD
  S["Do I need raw block?"] --> Q1{"Does the app manage its own on disk layout?"}
  Q1 -->|yes| B["Use Block mode"]
  Q1 -->|no| Q2{"Is it a VM disk, Ceph OSD or similar?"}
  Q2 -->|yes| B
  Q2 -->|no| Q3{"Need files, folders, fsGroup or file level backup?"}
  Q3 -->|yes| F["Use Filesystem mode"]
  Q3 -->|no| Q4{"Measured a real filesystem overhead problem?"}
  Q4 -->|yes| B
  Q4 -->|no| F
    

Read the diagram from the top. The first two questions identify software that is built to own a raw device. The third catches everything that depends on files. The last question protects you from guessing: unless you have actually measured a filesystem problem on your own workload, the safe default is Filesystem mode.

Good reasons to choose Block

Poor reasons to choose Block

• “It must be faster.” Not automatically. The disk’s own speed is the same, and the kernel’s filesystem code is highly optimised. Gains appear only for particular patterns.
• “I do not want a filesystem lock-in.” Filesystem mode is portable and well supported. Block mode makes you responsible for your own formats.
• “My application just writes some files.” If it uses folders and files, it needs a filesystem. Giving it a raw device means it cannot work.
• “Everyone else uses it.” A rare mode with fewer tools should be adopted deliberately, not by imitation.

A raw device is easy to damage if two writers use it without coordination. Access modes are the guardrails.

Why sharing a raw device is dangerous

A filesystem is not just a tidy catalogue. It is also a referee. Within a single machine, it makes sure two programs do not overwrite each other’s files. Across machines, most filesystems cannot be mounted for writing in two places at once, because each machine would keep its own memory picture of the layout and corrupt it. With a raw device there is no referee at all. If two programs on two nodes both write to the same blocks, the data is quietly damaged, with no error message.

This is why RWX on a block volume is a specialised feature. It is correct only if the software on top coordinates writers itself, as clustered databases and clustered filesystems do, or if only one writer is ever active at a moment, as in the live migration of a virtual machine, where the source hands the disk to the destination in a controlled step.

Making sure only one writer is active

• Prefer RWOP for single-writer software. It prevents the scheduler from starting a second Pod that uses the same claim, even on another node.
• Mind the node failure case. If a node stops responding, the disk stays attached to it for a while. A new Pod on another node may be blocked by a multi-attach error until the system is sure the old node has really let go.
• Use application-level fencing for shared devices. Clustered software often uses reservations on the device or a lease in a separate service so that a faulty node cannot keep writing after it was declared dead.
• Test failover with real failures. Pull the network from a node and watch what happens to the disk.

A device file is protected by the same permission rules as any file on Linux, and Kubernetes does not tidy them for you.

No fsGroup for raw devices

In Filesystem mode, a Pod can set fsGroup in its security context, and Kubernetes changes the group ownership of the volume’s files so that the application can write them. This works because there are files to change. With a raw device there is no such tree of files. Kubernetes cannot apply the fix-up, and the permissions of the device file itself decide who may read and write.

The practical effect is that a container running as a non-root user may find the device unreadable or unwritable. The cure depends on the platform and driver: you may need to add the user to the group that owns the device through the Pod’s supplemental groups, run the process as a specific user, or have a small privileged setup step. Check the behaviour of your driver and runtime, and test as the same user your application will use.

Do you need a privileged container?

Simply reading and writing a device handed over through volumeDevices does not require a privileged container. The container runtime adds the device to the container with permission to read and write. Privileges are needed only when you do more, such as creating a filesystem and mounting it from inside the container, or changing the device’s settings. Keep privileges as low as the task allows.

Stale data and clean slates

A new cloud disk starts blank. But a disk that was used by someone else and then reused, for example a statically provisioned device or a retained volume, may still hold old data. With a filesystem, a fresh format would remove it. With a raw device, nothing happens automatically. In a shared cluster this can leak one team’s data to another. Drivers and administrators should wipe or replace devices between uses, and applications should never treat unexpected content as safe.

Most day-two work on storage happens here. Block volumes support the main features, with a few important differences.

Snapshots

A snapshot is a frozen picture of a volume at one moment. Kubernetes exposes snapshots through objects such as VolumeSnapshot, handled by the CSI driver. For raw block volumes, a snapshot captures the blocks exactly. It does not know about files, so it cannot say anything about what is inside, and it cannot be “browsed”. It is, however, an excellent building block for backup, cloning and recovery.

It is important to understand that a snapshot of a busy device is normally crash-consistent. That means it is as if the power had been cut at that instant. A well-designed application can recover from that, like a database replaying its log. If your software needs a cleaner state, it should flush or pause before the snapshot is taken.

Restoring and cloning, and the mode rule

A snapshot or an existing claim can be used as the source for a new claim. The new claim should have the same volume mode as the source. Kubernetes and the snapshot controller take care to prevent a silent conversion, for example restoring the blocks of a raw volume into a Filesystem claim, because the result would make no sense and could be misread as a damaged filesystem. An administrator can explicitly allow a mode change for special cases, but it is an unusual step.

Backups: file tools do not work

Many popular backup tools for Kubernetes work by walking through the files of a volume and copying them. With a raw block volume there are no files to walk, so these tools cannot back it up. The right approach is snapshot-based backup, where the backup system asks the driver for a snapshot and then stores that snapshot or copies it elsewhere. If you rely on a specific backup product, check its documentation for block support before you adopt block mode.

Expanding a volume

If the StorageClass allows volume expansion, you can edit the claim to request more space. The driver enlarges the underlying disk. In Filesystem mode, the kubelet then also grows the filesystem. In Block mode there is no filesystem, so Kubernetes does nothing more: the device simply becomes larger. Your application must notice the larger size and begin to use the new space, for example by re-reading the device size, then extending its own layout. Shrinking is not supported.

From request to retirement, a volume passes through clear stages. Knowing them helps you read events and plan clean-ups.

flowchart LR
  P["PVC created"] --> PR["Provisioned"]
  PR --> BD["Bound"]
  BD --> AT["Attached to node"]
  AT --> MP["Device mapped for Pod"]
  MP --> US["In use by container"]
  US --> UM["Pod stops: device unmapped"]
  UM --> DT["Detached"]
  DT --> RC{"Reclaim policy"}
  RC -->|Delete| DEL["Disk deleted"]
  RC -->|Retain| RET["Disk kept, data intact"]
    

Reclaim policy: Delete or Retain

When you delete a claim, the reclaim policy of the volume decides its fate. With Delete, the disk is destroyed along with its data. With Retain, the disk is kept, the data stays, and an administrator must clean it up or reuse it. For databases and other valuable data, Retain is a safety net against accidental deletion. For temporary or easily rebuilt data, Delete avoids forgotten disks that quietly cost money every month.

What happens when a Pod moves

If a Pod is rescheduled to another node, the system must detach the disk from the old node and attach it to the new one. With cloud disks this usually takes from several seconds to a minute or two. If the old node has failed, the system waits for a safety period, about six minutes in typical settings, before force-detaching the disk, so as to avoid two nodes writing to it. During that time the new Pod may report a multi-attach error. This is the platform protecting your data, not a fault.

Raw block can shorten the path, but the disk’s own limits stay the same. Performance work begins with knowing where the limits are.

What block mode removes

Where the real limits live

Cloud disks are sold with limits. On Amazon EBS, for example, a general-purpose gp3 volume comes with a baseline of 3,000 input/output operations per second and 125 megabytes per second regardless of size, which you can raise by paying more, while provisioned types such as io2 offer higher and more predictable performance. The instance itself also has an overall cap for all the disks attached to it. Choosing block mode will never push you past these ceilings. If you are hitting one, a change of volume type or instance size helps far more than a change of mode.

Block mode helps when the filesystem layer was the bottleneck, which happens with certain database patterns and very high rates of small writes. It may not help, or may even hurt, when the filesystem was doing useful work for you, such as merging small writes or caching reads. The only reliable way to know is to measure.

Scaling limits to remember

A single block volume is a single copy of your data. Reliability comes from how you design around it.

What one volume does and does not protect

Cloud block storage is usually replicated inside one zone by the provider, so a single hardware fault does not lose your data. But the volume still lives in one zone. If the zone has a problem, the volume is unreachable. If the application writes bad data, every copy holds the bad data. If someone deletes the claim with a Delete policy, the disk goes too. High availability therefore needs more than one volume or more than one copy.

Common failure situations

The missing safety net

When a computer loses power, a normal filesystem can usually repair itself on the next start, by replaying a journal or running a check. On a raw volume, that safety net is gone. Your application becomes responsible for making sure that a half-finished write can be detected and fixed. Storage engines built for raw devices do this carefully, with logs and checksums. Software that was never designed for raw access should not be pointed at one.

Raw devices remove some layers of protection, so the remaining layers need extra care.

Who can ask for storage

Anyone who can create a claim in a namespace can request disks from any StorageClass they are allowed to use. Control this with RBAC for who can create claims, with resource quotas to limit how many disks and how much space a team can request, and with admission policies that limit which classes are allowed in which namespaces. A claim is also a cost, so these controls help with the bill as well as with security.

Encryption

• Encryption at rest. Cloud providers can encrypt disks with keys from a key management service. Enable it in the StorageClass or through account defaults. It works the same way for block and filesystem volumes because it happens below the node.
• Encryption in transit. For network storage protocols, check whether data between the node and storage is encrypted. For cloud disks this is handled inside the provider network, but you should confirm for your own platform.
• Application-level encryption. Because the application owns the device, it can encrypt data itself, giving you independent control of the keys.
• Key lifecycle. Plan for key rotation, key loss and key revocation. A disk whose key is deleted is gone for good.

The privileged parts of the system

The CSI node component runs on every node, usually with high privileges, because it must attach disks and map devices. A fault or compromise in it is serious. Install drivers only from trusted sources, keep them updated, limit who may change their configuration, and watch their logs.

Isolation between tenants

Compliance and audit

Regulations often require proof that data is encrypted, that access is controlled, and that deleted data is really gone. For block volumes, collect evidence at three places: the cluster, where audit logs show who created, changed and deleted claims; the cloud account, where disk settings and key use are recorded; and the application, where access is logged. Define what “deleted” means for your policy, such as destroying the disk, destroying the key, or both, and test that the process works.

Block volumes are quieter than filesystems: some of the usual dashboards are blank, so you must know where else to look.

What you lose

For Filesystem volumes, the kubelet reports used and free space and inode counts, which feed the familiar alerts about full volumes. For raw block volumes there is no filesystem to inspect, so these numbers are not available. The application is the only one who knows how much space is truly in use, so it must report this itself.

What to watch instead

Symptoms and likely causes

A calm troubleshooting routine

Lessons that teams usually learn the hard way, collected in one place.

Best practices checklist

• Confirm that your driver supports block mode and the access mode you need, before designing around it.
• Match the volume mode in claim and volume, especially for static volumes.
• Use RWOP for single-writer software.
• Set a Retain policy for valuable data, and a Delete policy for temporary data.
• Take snapshots on a schedule and copy them to a separate place.
• Export capacity and usage metrics from the application, since the kubelet cannot.
• Benchmark with your real workload before and after changing mode.
• Test node failure, zone failure and restore procedures regularly.
• Document who owns the device format and what the application does on start-up.

Common mistakes

• Forgetting that the device has no filesystem and trying to read files from it.
• Mixing up the Pod fields. Raw volumes use device entries, not mount entries, and the container sees a path to a device, not a folder.
• Expecting expansion to be automatic for the application as well as the disk.
• Ignoring per-node attach limits until Pods become unschedulable.
• Treating a snapshot in the same system as a complete backup.
• Skipping a wipe on reused volumes.

Where raw block volumes show up in practice, and what each setting teaches us.

A journey through adoption

A typical team begins with Filesystem volumes for everything, because they are simple and well supported. After a while one workload, perhaps a database with its own storage engine or a platform for virtual machines, shows a real need for a raw device. The team adds a block StorageClass, tests a single workload, measures the effect, writes its recovery and backup procedures, and only then expands. The stable outcome is a small number of block users, each with a clear reason, rather than a cluster full of raw devices.

Raw block volumes give a Pod an empty, unformatted device and let the application decide how to use it. They remove a layer of work, and in return they remove a layer of help: no files, no permission fix-up, no usage numbers, no file-level backups, and no automatic repair after a crash.

Used for the right software, such as virtual machines, storage platforms and engines that manage their own disks, they are simple and powerful. Used without need, they add risk. The best habit is to start with the filesystem, move to block deliberately, and prove the benefit with measurements.