Begin with the plain meaning, then add detail one layer at a time.

When an application saves or loads data, it asks the operating system to read from or write to a storage device. Storage performance describes how fast and how smoothly that request is answered. It is not one number. It is a small family of numbers that describe different kinds of speed, and a system can be excellent at one and poor at another. A road is a good comparison: you can ask how many cars pass per minute, how many tonnes of cargo move per hour, and how long one car takes to travel from the start to the end. All three questions are legitimate, and the answers can differ.

In Kubernetes, storage performance matters more than in older setups for a simple reason: Pods move. A traditional server had its own disk, bolted to the same machine as the program. A Pod is scheduled onto whichever node has room, and its data often lives somewhere else entirely. That separation brings great flexibility, since a Pod can restart on another node and find its data waiting. The price is a longer path between the application and its bytes, and every extra step on the path can add delay or limit the flow.

Why tuning is different from buying

The first instinct when storage is slow is to buy a faster disk. Sometimes that is correct. Often it is wasteful, because the real limit sits somewhere else: a per-machine cap on bandwidth, an application that asks for one block at a time, a filesystem option that forces extra work, or a startup step that recursively changes permissions on millions of files. Tuning means finding the actual limit first and then spending effort or money exactly there.

A handful of words explain almost every storage conversation. Learn them once and the rest becomes easy.

IOPS: input and output operations per second

One I/O operation is one request to read or write a chunk of data. IOPS counts how many of these the storage completes in a second. A small database that updates many rows needs a high IOPS number. The size of each chunk is called the block size, and it changes the meaning of IOPS completely. Four thousand requests of 4 kilobytes each is a very different job from four thousand requests of 1 megabyte each, even though the IOPS count is identical.

Throughput: bytes per second

Throughput, also called bandwidth, is how much data moves each second, usually in megabytes per second. It is linked to IOPS by simple arithmetic: throughput is IOPS multiplied by block size. If a disk delivers 10,000 IOPS at 4 kilobytes, that is only about 40 megabytes per second. The same disk reading 1-megabyte blocks might reach its bandwidth ceiling at a few hundred IOPS. This is why a vendor can quote a huge IOPS figure and a database can still starve: the two numbers apply to different block sizes.

Latency: how long one request takes

Latency is the time from issuing a request to receiving the answer. It is measured in milliseconds or microseconds. Local flash storage often answers in well under a millisecond, while network-attached disks commonly take one to several milliseconds, depending on the provider and the load. Latency is the number a waiting user feels.

Percentiles, not averages

An average hides pain. If ninety-nine requests take 1 millisecond and one takes 500, the average looks like 6 milliseconds, yet one in a hundred users had a terrible time. Engineers therefore look at percentiles. The 99th percentile, written p99, is the time that 99 percent of requests beat. When people talk about “tail latency” they mean these slow outliers, and tail latency is where storage problems usually show first.

Queue depth: how many requests are in flight

Queue depth is the number of requests the application has sent but not yet seen finished. Modern storage is built to work on many requests at once. A program that sends one request, waits, and then sends the next, has a queue depth of one and cannot use the full power of the device. Higher queue depth usually raises IOPS up to a point, and then only raises latency.

Random and sequential access

Sequential access reads or writes data in order, like reading a book from page one. Random access jumps around, like looking up words in a dictionary. Spinning disks hate random access because the head must move. Flash storage handles it far better, but sequential access is still usually cheaper. Databases and key-value stores mix both, and knowing which dominates your workload shapes every later decision.

Reads, writes and the mix between them

Reads can often be served from memory caches. Writes must eventually reach durable storage, and many applications also demand that a write is truly safe before they continue. The command that forces this is called sync or flush, and it is expensive on distributed storage because every copy must confirm. A workload that is ninety percent reads behaves very differently from one that is ninety percent writes.

To tune a journey, you must first see every stop on it.

When an application inside a container writes a file, the request does not leap straight to a disk. It travels through a stack of layers. Each layer does useful work, and each layer can be a source of delay or a limit on flow. The map below shows the typical stops for a volume that lives on a network-attached disk.

flowchart TD
  A["Application in the container"] --> B["Container filesystem view and volume mount"]
  B --> C["Node operating system: page cache, filesystem, block layer"]
  C --> D["Node device driver or network storage client"]
  D --> E{"Where does the data live?"}
  E -->|"Local disk"| F["Physical flash or disk in the same machine"]
  E -->|"Network block"| G["Network, then storage service, then replicated disks"]
  E -->|"Network file"| H["Network, then file server or distributed filesystem"]
    

Walking down the stack

A short, friendly refresher on the objects that describe storage, because tuning means changing them.

Kubernetes separates the request for storage from the supply of storage, much like a customer who orders a table and a restaurant that decides which table to give. That split is expressed through a few objects.

Access modes

A claim also states how the volume may be shared. ReadWriteOnce means one node at a time can mount it for reading and writing, which suits network block disks. ReadWriteMany means many nodes may use it together, which needs a shared file system. ReadOnlyMany allows many readers. Newer versions also include a single-Pod variant of the first mode. The mode is not just a rule. It decides what kind of storage is even possible, and therefore what performance is available.

Dynamic provisioning

In dynamic provisioning, a developer creates a claim naming a class. The driver then creates the real disk in the background, and Kubernetes binds the claim to it. Because the class carries the parameters, changing the class changes the performance of every new volume. Existing volumes do not change by themselves, a point that surprises many teams.

The life of a volume, briefly

Notice how many steps happen before the first byte is read. Chapter 17 returns to this, because slow Pod starts are a performance problem too.

Not all volumes are equal. Each kind has a character, and choosing well is the cheapest tuning you will ever do.

Matching the volume to the job

• Scratch space for caches, sorting and temporary files: an emptyDir, perhaps on a fast local disk.
• A primary database: a network block volume with provisioned performance, or a local volume with replication handled by the database itself.
• Shared media or content used by many web Pods: a file share, with attention to small-file behaviour.
• Backups, logs and archives: object storage, accessed by a client library instead of a mount.

This is the biggest fork in the road, and it is a trade between speed and freedom.

A disk inside the same machine as the Pod answers in microseconds and has no network to share. A disk reached over the network answers in milliseconds and may be limited by the bandwidth of the machine. On the other hand, a network disk is independent of any one node. When the node dies, Kubernetes can start the Pod elsewhere and re-attach the disk. A local disk disappears with its node, so the data must be protected in another way, such as the application’s own replication.

How local volumes work in Kubernetes

A local persistent volume declares which node it belongs to. The scheduler reads that and only places the Pod there. Because the placement must be known before the claim is bound, local volumes depend on a binding mode that waits for the Pod, which Chapter 8 explains. A helper program called a local static provisioner can discover disks on nodes and publish them as volumes automatically.

Making the decision

The class is where platform teams turn performance up or down for everyone.

A StorageClass carries a driver name and a set of parameters that the driver understands. The parameters differ by vendor, and the documentation of your driver is the final authority, but a few themes repeat everywhere.

Offering a menu rather than one choice

A single default class forces every team into the same compromise. Mature platforms publish several classes with clear names, such as a standard tier for general use, a high-performance tier for databases, and a throughput-optimised tier for analytics. Each class documents what it costs and what it promises. Application teams then pick by need, and finance can see who is paying for premium performance.

Naming and ownership

Treat classes as a product. Give them clear names, publish expected numbers from your own benchmarks, state which workloads they suit, and assign an owner. A well-run class catalogue prevents the common pattern where everyone copies the biggest class “just to be safe” and quietly multiplies the storage bill.

A fast disk in the wrong place is a slow disk.

Clouds are divided into zones, which are separate groups of data centres within a region. A network disk usually lives in exactly one zone. A Pod in a different zone cannot attach it at all. Even where cross-zone access is possible, the extra distance adds latency and sometimes cost. Kubernetes handles this through topology information on volumes and nodes, and through the volume binding mode of the class.

flowchart LR
  subgraph IM["Immediate binding"]
    A1["Claim created"] --> A2["Disk created in a zone chosen at once"]
    A2 --> A3["Pod scheduled later"]
    A3 --> A4{"Same zone?"}
    A4 -->|"No"| A5["Pod stuck pending"]
    A4 -->|"Yes"| A6["Works"]
  end
  subgraph WF["Wait for first consumer"]
    B1["Claim created"] --> B2["Wait"]
    B2 --> B3["Pod scheduled to a node"]
    B3 --> B4["Disk created in that node's zone"]
    B4 --> B5["Works"]
  end
    

Why waiting is usually wiser

With immediate binding, the disk is created the moment the claim appears, before anyone knows where the Pod will run. The zone is chosen blindly. Later, the scheduler may find that the Pod must run in another zone because of CPU, memory or other rules, and the Pod then cannot start. With wait for first consumer binding, the system holds the claim until a Pod that uses it is placed. The disk is then created in the right zone. This one setting prevents a large share of “Pod stuck pending” incidents for zonal storage.

Spreading replicas across zones

For resilience you often want database members in different zones. Each member then gets its own disk in its own zone. Use topology spread rules so the replicas do not pile into one zone, and remember that a single-writer database still sends its replication traffic across zones, which has latency and cost consequences of its own.

Node-level limits that look like topology problems

Each node can attach only a limited number of disks, and the limit depends on the machine type and the driver. A Pod that needs a volume may be unschedulable on an otherwise empty node because the attach limit was reached. Kubernetes learns these limits from the driver and takes them into account when scheduling, but it is worth knowing the numbers for your machine types when you plan a dense layout.

A filesystem is the filing system on top of the raw disk. Its design shapes how the disk behaves under your workload.

A raw disk is just a long row of numbered blocks. A filesystem adds names, folders, permissions and the bookkeeping that keeps everything consistent. When Kubernetes first uses a new volume in filesystem mode, the node formats it with the type named in the class, commonly ext4 or XFS. Both are mature and widely used, and for most workloads the difference is smaller than the difference between a good and a bad disk tier. Still, the choice has consequences worth knowing.

Mount options that matter

A class can pass mount options, which are flags used when the volume is attached to the node. The ones that most often matter are small, and each deserves a moment of thought rather than copying from a blog.

• Access time updates. By default, every read may cause a small write to record when the file was last read. Options such as noatime or relatime reduce this. Modern kernels already default to a gentle version, but explicit settings on busy shared volumes can still help.
• Discard or trim. Telling flash storage which blocks are free helps it manage itself. Doing it on every delete can add latency. Periodic trimming on a schedule is often kinder to latency-sensitive workloads.
• Barriers and journalling modes. These protect data after a crash. Turning them off is a classic trap: speed rises, and so does the chance of corruption when power or a node fails. Leave safety features on unless you fully understand the consequences.
• Network filesystem options. For shared file systems, read and write sizes, number of connections and caching rules can change results dramatically. Chapter 23 covers them.

Inodes: the quiet limit

A filesystem tracks each file with a small record called an inode. The number of inodes is often fixed when the disk is formatted. A volume holding millions of tiny files can run out of inodes while still having free space, and writes then fail strangely. Watch inode usage as well as bytes used, particularly for caches, mail queues and build systems.

Sometimes the best way to speed up the filing system is to remove it.

Most volumes are presented in filesystem mode: the application sees a directory and works with files. Kubernetes also supports raw block mode, in which the Pod receives the disk itself as a device, with no filesystem in between. The application must then manage the layout of data on its own.

Direct I/O and caching

Even inside a filesystem, an application can ask to bypass the operating system’s page cache with a mode usually called direct I/O. Engines that keep their own cache use it to avoid caching the same data twice. This changes how memory is used by the node and affects how you read monitoring graphs, because the node will show little page cache for such a Pod while the process itself holds a large buffer pool.

Performance in the cloud is a contract with several clauses. A bottleneck often hides in the one you did not read.

When you rent a network disk, the provider promises certain numbers. Those numbers come with rules, and the rules differ among providers and change over time, so always confirm them in the current documentation before you plan. The general shapes, however, repeat.

The three ceilings

Burst credits

Some disk types give a modest steady baseline and let you go faster for short periods by spending credits that build up when the disk is quiet. This is excellent for spiky workloads and treacherous for steady ones. A database that runs well for the first hour of a benchmark and then slows dramatically has probably used up its credits. Always run tests long enough to drain any burst allowance, and prefer disk types with a flat, documented baseline for production databases.

Size and performance linkage

On some platforms a bigger disk is a faster disk, because performance is tied to capacity. A team may therefore buy more space than it needs, just to gain speed. Other platforms decouple the two, so you pay for the exact IOPS and throughput you require. Understanding which model your provider uses saves money and prevents the confusing situation where doubling the disk doubles the speed.

Resizing and modifying volumes

Kubernetes supports growing a claim when the class allows expansion and the driver supports it. Many drivers can expand a volume while it is in use, and some can also change the performance settings in place, though capabilities vary. Check your driver’s feature list, and test an expansion in a non-production cluster. Shrinking is generally not supported.

Not all storage is persistent, and the temporary kind is where many surprises hide.

Every container needs somewhere to write that is not a persistent volume: a writable layer on top of its image, temporary directories, logs. This space is called ephemeral storage, and it lives on the node, usually on the same disk as the operating system and the container runtime. Everything on that disk competes for the same speed.

What shares the node disk

• Unpacked container images and their layers.
• The writable layer of every running container.
• emptyDir volumes of every Pod, unless memory-backed.
• Container logs written by the runtime.
• Kubelet working data and system logs.

A single Pod that writes heavily to its temporary space can slow down image pulls, log writing and the other Pods on that node. Because this is all one pool of I/O, it is also a hidden source of noisy-neighbour problems.

Memory-backed emptyDir

An emptyDir can be told to use memory instead of disk. Speed is superb, but the space counts against the container’s memory limit, and a full memory volume can cause the Pod to be killed. Use it for small, hot scratch data, such as sockets or tiny caches, and set a size limit.

Giving the container runtime its own disk

On busy nodes, a good pattern is to put the container runtime’s data directory on a separate disk from the operating system, and sometimes to give heavy temporary space a third. Then an image pull storm does not delay the kubelet, and a runaway log does not fill the system disk. Many managed services and node images allow you to choose the size and type of the node’s root volume, and tuning that choice is worthwhile.

Requests, limits and eviction

Pods may declare how much ephemeral storage they request and the most they may use. The kubelet watches disk space and inodes on the node, and when free space falls below its thresholds, it starts evicting Pods to protect the machine. Notice that these controls count space, not speed. They keep the disk from filling up, but they do not stop one Pod from using all the I/O bandwidth.

You cannot tune what you have not described. Start by asking how the application actually touches its data.

Two workloads can both be “a database” and ask for opposite things. Before touching a setting, characterise the traffic using a few questions. How large are the requests? Are they random or in order? What share is reads versus writes? Do writes wait for confirmation? How much parallelism does the application create? How much data is the hot set that gets used repeatedly, compared with the cold bulk?

The write-ahead log

Many databases first write each change to a sequential journal called a write-ahead log and only later update the main data files. The log write must be confirmed as safe before the transaction is reported complete. That makes log latency the main limit on how many transactions per second a database can commit. The data files, by contrast, are written lazily in the background. This is why a favourite tuning move is to place the log on its own fast, low-latency volume.

Background work is part of the workload

Engines that organise data in sorted layers must periodically rewrite it, a process often called compaction or merging. It can use more I/O than the foreground requests. When a cluster looks fine and then stalls for a few minutes, the cause is frequently a background rewrite that saturated the disk. Include it when you size capacity.

A measurement you can trust is worth more than any opinion. A misleading one is worse than none.

The most widely used tool for storage testing is fio, a flexible program that generates controlled read and write patterns and reports IOPS, throughput and latency percentiles. Other tools exist, including wrappers that run these tests inside a cluster against real volumes. Whichever you choose, the method matters more than the tool.

flowchart TD
  A["Describe the real workload: block size, mix, parallelism"] --> B["Create a test volume from the exact StorageClass"]
  B --> C["Run a Pod on a representative node with matching resources"]
  C --> D["Warm up and run long enough to drain any burst allowance"]
  D --> E["Record IOPS, throughput and latency percentiles"]
  E --> F{"Do results meet the goal?"}
  F -->|"Yes"| G["Write down the numbers as the baseline"]
  F -->|"No"| H["Change one thing and repeat"]
  H --> C
    

Rules for honest results

1. Test the same path your application uses. A benchmark on the node’s local disk says nothing about a network volume mounted in a Pod.
2. Match the access pattern. Use the block size, read and write mix, and parallelism that you measured in Chapter 13. Defaults are rarely realistic.
3. Defeat the cache on purpose, or measure it on purpose. A file that fits in the page cache will report absurdly fast reads. Use a test file much larger than memory, or choose direct I/O when testing the device itself.
4. Run long enough. Burst credits, thin provisioning and background garbage collection show up after minutes, not seconds.
5. Test on a quiet and on a busy node. Shared resources change results when neighbours are active.
6. Change one thing at a time. Otherwise you cannot tell which change helped.
7. Repeat the test. Run it several times and at different hours to see the spread, not just a lucky number.

A standard set of tests

Numbers do not speak for themselves. They need a careful reader.

Compare to the limits you know

Take the observed IOPS and throughput and compare them with the three ceilings from Chapter 11. If the result sits exactly at the volume’s published limit, you have found a ceiling and the cure is to raise it. If it sits far below every published limit, something else is in the way: queue depth, application behaviour, or a mount setting.

Watch the shape of latency

Healthy storage under rising load shows latency that stays flat, then climbs sharply once the device saturates. Plot latency against queue depth. The knee of that curve is the practical sweet spot. Running beyond it adds waiting and no extra work.

Common ways to be fooled

Writing down a baseline

When you have a trusted result, record it with the date, the class, the size, the node type, the kernel and driver versions and the exact test settings. This baseline becomes your reference. After any upgrade or change, rerun the same test. Storage regressions are common after driver or kernel updates, and without a baseline they are almost impossible to prove.

The driver sits between Kubernetes and the storage system. It does not touch your data path, but it controls everything around it.

A CSI driver is usually deployed as two cooperating parts. A controller part, running as ordinary Pods, handles cluster-wide actions such as creating disks, attaching them to machines, resizing and taking snapshots. A node part, running on every node, handles local actions such as formatting, mounting and reporting usage. Small helper containers called sidecars translate Kubernetes events into driver calls.

flowchart LR
  subgraph CP["Controller side"]
    P["External provisioner"] --> CD["Driver controller service"]
    AT["External attacher"] --> CD
    RS["External resizer"] --> CD
    SN["External snapshotter"] --> CD
  end
  CD --> ST["Storage system API"]
  subgraph ND["Each node"]
    KL["Kubelet"] --> NP["Driver node service"]
    NP --> MT["Format and mount on the node"]
  end
  ST -.-> MT
    

What this means for performance

• Data does not flow through the driver. Reads and writes go from the node straight to the device or storage service. The driver’s speed does not set your IOPS.
• The driver does set how quickly volumes appear. Slow provisioning, attaching or mounting delays Pod starts, rescheduling after failure and scaling events. Sidecars that are throttled by API rate limits make this worse in large clusters.
• The node part sets what happens at mount time. Formatting options, permission changes and checks happen there, as the next chapter explains.
• Capacity and attach limits are reported by the driver. The scheduler relies on them.
• Metrics come from the driver and the kubelet. They are your window into attach time, mount time and volume usage.

Keeping drivers current

Storage drivers evolve quickly. Newer releases bring faster attach, better limits handling, volume modification support and bug fixes. Treat them like any critical component: read the release notes, upgrade in a staging cluster first, and rerun your baseline benchmarks afterwards.

Storage performance is not only about steady speed. It is also about how long a Pod waits before it can begin.

A Pod that uses a persistent volume cannot start its containers until the volume is ready. In a healthy setup this takes a few seconds to a minute. In an unhealthy one it can take many minutes, and when a database restarts during an incident, those minutes are painful. Several distinct causes are worth knowing.

The permission change trap

The group ownership step is the most common hidden start-up delay. Kubernetes offers a policy setting for it, fsGroupChangePolicy. With the value that checks the root directory only, the walk is skipped when the top-level ownership already matches. After the first run, restarts become fast. Many large stateful workloads gain minutes of start time from this single setting.

Reducing the failover window

• Use health checks and node monitoring that detect failure quickly.
• Know your cluster’s timing settings for unreachable nodes and volume detach.
• For databases with their own replicas, fail over at the application level rather than waiting for the old disk to move.
• Keep volumes lean. Millions of files make every volume-wide operation slower.
• Practice a node failure and record the time taken. The real figure often surprises teams.

Backups are essential, but they have a speed footprint. Know it before the backup runs at noon.

A snapshot is a point-in-time picture of a volume. Many storage systems create snapshots almost instantly, using a technique that records only the changes after the picture was taken. A clone creates a new volume from an existing one or from a snapshot, which is a fast way to build test copies. Kubernetes exposes these through snapshot objects and through a data source field on new claims.

How snapshots can affect speed

• Copy-on-write cost. In some systems, the first write to a block after a snapshot requires copying the old content aside. A heavily written volume with frequent snapshots may see added write latency.
• Many snapshots in a chain. Long chains can slow reads and make deletion expensive on certain platforms.
• Snapshot creation is not the same as a backup. On many clouds, a snapshot is copied to cheaper storage in the background, and that background copy can use bandwidth.
• Lazy loading after restore. A volume built from a snapshot may fetch blocks from the remote store on first access. The first read of each block is slow, and the workload may look broken until the volume is fully warmed. Some platforms offer ways to initialise the volume fully or in advance.

Application-consistent backups

A snapshot of a running database is only as good as the moment it was taken. Many databases need a short pause or a special flush for a snapshot to be consistent. Tools that coordinate this add a brief latency blip. Plan snapshots for quiet times, and test restores regularly, since a backup that cannot be restored is only a feeling of safety.

A shared node is a shared kitchen. One messy cook can slow everybody.

Since Kubernetes does not limit disk I/O per Pod by default, a single heavy Pod can consume most of a node’s disk bandwidth or most of the machine’s network storage bandwidth. Other Pods on the same node then see higher latency, even though their own volumes are separate disks. This is the classic noisy neighbour problem.

Finding the culprit

The symptom is a latency rise across unrelated Pods on one node. On that node, look at disk utilisation, the queue length and the per-process I/O reports to see who is generating the load. Correlating the time of the slowdown with scheduled jobs often points straight at the cause.

Some of the biggest levers sit just below the container, in the settings of the node’s operating system.

Most teams should change node-level settings carefully and only when measurements justify it, because these settings apply to everything on the machine. Managed Kubernetes services also restrict or reset some of them. Still, knowing what exists helps you read documentation and recommendations from database vendors.

The page cache is your friend, and also your trap

The kernel keeps recently used file data in memory. A node with plenty of free memory therefore serves many reads without touching the disk at all. But the cache also counts toward a container’s memory accounting in many setups, and memory limits that are too tight force the cache to shrink, which turns into more disk reads. When a Pod’s I/O rises after you lower its memory limit, you have probably squeezed its cache.

How to apply changes safely

1. Change one setting on one test node.
2. Repeat the benchmark from Chapter 14 and compare it with the baseline.
3. Roll it out through your node image or configuration management so it survives replacement.
4. Record why the setting exists, because six months later nobody will remember.

Databases are the most demanding storage customers in a cluster. A few layout decisions give most of the gain.

A StatefulSet is the Kubernetes object designed for applications that need stable identity and their own storage. Each member gets its own claim, created from a template, and keeps it across restarts. This is the usual way to run databases, brokers and search nodes.

Separate volumes for separate jobs

Databases do different kinds of I/O, and giving each its own volume lets you tune each separately and prevents them from interfering.

Placement rules

• Spread members across nodes and zones with anti-affinity or topology spread rules.
• Reserve nodes for heavy databases with taints, so general workloads do not crowd them.
• Give the Pod guaranteed CPU and memory by setting equal requests and limits, so the database is not throttled or evicted during busy moments.
• Mind the memory settings of the engine itself. A buffer pool that is too small forces extra disk reads. One that is too large starves the operating system and the container limit.

Operators

An operator is a controller that encodes the operational knowledge for a particular database: creating members, handling failover, taking backups, resizing volumes. Mature operators expose storage options directly, such as separate volume sizes for data and log, and know how to expand claims safely. Using one reduces the chance of a mistake during a stressful night.

Sharing one volume across many Pods is convenient. It also changes the performance rules.

When many Pods on many nodes must read and write the same files, you need a file system that can be mounted in many places at once. Common examples include the Network File System protocol, managed file services from cloud providers, and distributed file systems. They offer the many-node access mode, but their strengths and weaknesses are different from a block disk.

Why metadata dominates

Every file operation involves two kinds of work: moving the data, and handling the metadata, which means names, sizes, permissions and timestamps. On a local disk, metadata is cheap. On a file share, each metadata action may be a round trip to the server. A job that lists a folder with ten thousand files, then opens each one, performs tens of thousands of network conversations before reading a single byte. This is why small-file workloads feel slow on shares even when throughput figures look high.

Common tuning levers

• Read and write sizes. Larger transfers reduce overhead for big files.
• Number of parallel connections. Modern clients can spread one mount across several connections, which helps throughput on fast links where the kernel and server support it.
• Attribute and directory caching. Longer caching reduces round trips but lets clients see stale information for a while. Choose according to how much consistency the application needs.
• Hard versus soft mounts. Hard mounts retry forever and protect data integrity. Soft mounts give up and return errors, which can corrupt applications that do not expect them.
• Server-side tier or performance mode. Managed services often offer modes trading cost for throughput or latency. Choose by workload.
• Server location. The file server in another zone adds latency and cost to every operation.

Running your own storage system on the cluster’s nodes gives flexibility and freedom. It also gives you its performance consequences.

Projects such as Rook with Ceph, Longhorn, OpenEBS variants and others turn the local disks of your nodes into a pooled, replicated storage service. Applications request volumes in the usual way, and the system places copies on several nodes. The benefit is that you keep durability and mobility on any infrastructure, including your own data centre.

Why writes cost more

To keep data safe, these systems write each block to several machines and wait for confirmation before saying the write is done. With three copies, a single application write becomes three disk writes and a set of network messages. This is called write amplification. Latency is set by the slowest of the required copies, plus network time.

sequenceDiagram
  autonumber
  participant App as Application Pod
  participant Pri as Primary storage node
  participant R1 as Replica node 1
  participant R2 as Replica node 2
  App->>Pri: Write one block
  Pri->>R1: Copy the block
  Pri->>R2: Copy the block
  R1-->>Pri: Stored
  R2-->>Pri: Stored
  Pri-->>App: Write complete
    

What determines performance

Tuning without measurement is guessing with extra steps. Put the gauges in place before the incident.

Storage health can be seen from four viewpoints, and a good dashboard has something from each. Together they let you say which stop on the path from Chapter 3 is struggling.

The metrics that tell the most

Good alerts, not loud alerts

• Alert on user-visible symptoms first, such as commit latency, and on causes second.
• Alert on volume space with enough lead time to expand safely.
• Alert when a disk approaches its provisioned limits, before throttling begins.
• Alert on unusual attach and mount durations.

A calm, ordered path beats guessing. Work from the outside in, and from the cheap checks to the expensive ones.

flowchart TD
  A["Application is slow"] --> B{"Is the slowness at start-up only?"}
  B -->|"Yes"| C["Check attach, mount, ownership change and relabelling times in Pod events"]
  B -->|"No"| D{"Is the node disk busy with a long queue?"}
  D -->|"No"| E["Look at CPU, memory pressure, locks and the application itself"]
  D -->|"Yes"| F{"Is throughput or IOPS flat at a published limit?"}
  F -->|"Yes"| G["Raise the volume limit or machine storage cap, or spread the load"]
  F -->|"No"| H{"Do other Pods on the node cause the load?"}
  H -->|"Yes"| I["Isolate with dedicated nodes, separate disks or rate limits"]
  H -->|"No"| J["Check queue depth, block size, filesystem options and the access pattern"]
    

Symptom table

Learn from the mistakes other teams have already paid for.

Speed costs money, and the most expensive disk is the one nobody needs.

Every performance tier has a price, and prices differ widely. The skill is to buy performance where it is felt and nowhere else. A few principles keep spending sensible.

Orphaned volumes

When a claim is deleted but the reclaim policy keeps the disk, the volume remains and the bill continues. Likewise, volumes from retired test environments linger. A monthly report of volumes that are not attached to any Pod pays for itself surprisingly quickly.

Walk through one problem from complaint to cure, using plain arithmetic.

The situation

A company runs a relational database for an order system in a Kubernetes cluster. The database uses one network block volume for everything, with published limits of 3,000 IOPS and 125 megabytes per second. At peak, orders feel sluggish, and every evening a reporting job makes things worse. The machine type running the database has a storage bandwidth cap well above the disk limit, so the volume itself is the first ceiling.

Measure the demand

What the numbers say

• At peak, the IOPS demand of 3,700 is above the 3,000 limit by about 23 percent. Requests queue up and latency grows. The slowness is real, and it matches a ceiling, not an application bug.
• In the evening, throughput demand exceeds the bandwidth limit by 25 megabytes per second, so the report starves the orders, and the orders slow down the report.
• The log traffic and the data traffic share the same queue, so a burst of data reads delays log writes, which directly delays commits.

Applying queue depth arithmetic

The team measures a single-depth sync write latency of 2 milliseconds on the volume. Using the rule from Chapter 2, one request at a time gives at most about 500 operations per second (1 divided by 0.002). The database uses group commit, batching several transactions per log write, so it handles the 2,500 order writes with fewer log writes. Even so, the log’s latency sets the commit time, and 2 milliseconds is acceptable only if it stays steady. Sharing a queue with the report breaks that steadiness.

The plan

Typical situations where these ideas apply, described in general terms.

A short, ordered list you can print and follow.

It depends on the workload. Transactional systems feel latency and IOPS. Scans, backups and streaming feel throughput. Describe your workload first, then pick the number that it is bound by.

Usually it has lower latency, because there is no network. But a poorly chosen local disk can be slower than a strong provisioned network disk, and local storage has durability and mobility costs. Benchmark both against your workload.

Not directly. The class of a bound claim is fixed. You normally create a new volume from the new class and migrate the data, or use a driver feature that modifies the existing volume where supported.

The test probably did not match the application’s pattern. Block size, queue depth, sync behaviour and the read and write mix all change results. Or the real limit is elsewhere, in locks, CPU or the network.

Not by default. Resource requests and limits cover CPU, memory and ephemeral storage space, not I/O speed. Isolation comes from separate disks, dedicated nodes or features in the storage system.

Check whether a recursive ownership change is happening, and set the ownership change policy so it runs only when the root directory does not already match. Then look at attach and detach delays.

No, not for data you care about. The gain is usually small compared with the risk of corruption after a crash or power loss.

Some platforms load blocks lazily from a remote store on first access, so early reads are slow until the volume warms. Initialise the volume or read through it before relying on it.

On some platforms performance is tied to size, and on others it is independent. Check your provider’s rules. Do not buy extra space just to buy speed if you can purchase performance directly.

Usually not. Databases need low, steady latency for small synchronous writes and reliable locking behaviour. Block volumes or local disks are normally a better fit, unless your vendor documents support for a specific file service.

Compare observed IOPS and throughput with the documented limits and look at the provider’s metrics for throttling or burst balance. Flat lines at an exact round number are the classic sign.

Not for applications that need a filesystem or low-latency random writes. It is excellent for backups, archives, logs and large datasets that are accessed through an API or loaded into local scratch.

After any change to the driver, kernel, node image, machine type or storage class, and on a regular calendar, such as quarterly, to catch drift.

They manage the control plane and often supply default storage classes, but the workload design, volume layout, class choice and monitoring remain your responsibility.