Before we touch any setting, we need a shared language. Isolation is not one single thing. It is a family of protections, and each one solves a different kind of problem.

The simple definition

Resource isolation means that one application cannot take away the resources another application needs to do its job. A resource is anything that is limited and shared: processor time (CPU), memory (RAM), disk space, the number of running processes, and network bandwidth. A guarantee is a promise that a certain amount of that resource will be available when needed.

Notice the careful wording. We said “will be available when needed”, not “will always be reserved”. This difference is the secret behind almost every Kubernetes surprise, and we will come back to it many times in this post.

The noisy neighbor problem

Engineers call the cold-shower scenario the noisy neighbor problem. It happens when one workload consumes so much of a shared resource that other workloads on the same machine slow down or crash. It is the most common reason people ask for isolation in the first place. Large platforms have written about it for many years. Google, whose internal system called Borg inspired Kubernetes, built much of its container technology specifically to stop one job from starving another on shared machines.

Six kinds of isolation

Throughout this post we will use six labels. Keep them in mind, because every setting you will meet belongs to one of them.

Kubernetes itself does not build walls. It asks the Linux kernel to build them. Understanding two kernel features, namespaces and control groups, makes every later chapter easy.

A container is not a tiny computer

Many beginners picture a container as a small virtual machine. It is not. A container is just a normal Linux process (or a small group of them) that the kernel has dressed up with two kinds of costume. One costume controls what the process can see. The other controls how much the process can use. Every container on a machine shares the same kernel, which is why containers start in milliseconds, and also why isolation needs so much care.

Namespaces: controlling what a process can see

A Linux namespace gives a process its own private view of one part of the system. When a container starts, the container runtime wraps it in several namespaces at once. Inside, the process believes it is alone on the machine.

Control groups: controlling how much a process can use

A control group, usually shortened to cgroup, is a kernel feature that measures and limits resource use for a group of processes. If namespaces are walls, cgroups are meters and valves. Google engineers began the cgroup work in the mid-2000s, and it is now the foundation of every container platform.

Cgroups are arranged as a tree. The top of the tree represents the whole machine. Beneath it sit branches for different kinds of workloads. Kubernetes builds a tidy hierarchy: one branch for all pods, sub-branches for each quality-of-service class, and below those a branch for each pod, and finally one for each container. The kernel enforces the numbers written at each branch.

flowchart TD
  ROOT["Node: root cgroup"] --> SYS["system.slice: OS daemons"]
  ROOT --> KUBE["kubepods: all pods"]
  KUBE --> GUA["Guaranteed pods"]
  KUBE --> BUR["burstable branch"]
  KUBE --> BE["besteffort branch"]
  BUR --> PODA["Pod A cgroup"]
  BUR --> PODB["Pod B cgroup"]
  PODA --> C1["Container 1 cgroup"]
  PODA --> C2["Container 2 cgroup"]
  PODB --> C3["Container 3 cgroup"]
    

Look at the diagram from top to bottom. The node’s root cgroup splits into system daemons and a branch called kubepods for everything Kubernetes runs. Inside kubepods, the Burstable and BestEffort pods get their own branches, while Guaranteed pods sit directly under the top. Each pod gets its own cgroup, and each container inside the pod gets a child cgroup. When you set a limit on a container, the kubelet writes a number into that container’s cgroup, and the kernel obeys it.

Two generations: cgroup v1 and cgroup v2

Linux has two versions of the cgroup system. The older v1 keeps one separate tree per resource type. The newer v2 uses a single unified tree and offers better accounting, including pressure information that tells you how long processes have been waiting for a resource. Kubernetes supports v2 as generally available, and most modern Linux distributions now enable it by default. Several newer isolation features, such as improved memory controls, depend on v2.

Every guarantee in Kubernetes begins with two numbers you write in a pod’s specification: a request and a limit. They sound similar. They do completely different jobs.

The request: “I need at least this much”

A request is the minimum amount of a resource that a container asks for. The scheduler reads requests when it decides where to place a pod. If a node does not have enough unrequested capacity left, the pod is not placed there. After placement, the request also becomes the container’s fair share when the machine is busy. In short: requests are used for placement and for fair sharing under pressure.

The limit: “I may never use more than this”

A limit is the maximum amount a container may use. The kernel enforces it through cgroups. What happens when a container reaches its limit depends on the resource, and this is the single most important idea in the whole post:

How the numbers are written

CPU is measured in cores, and Kubernetes lets you use fractions called millicores. One core equals 1000 millicores, so 500m means half a core. Memory is measured in bytes, usually written with suffixes such as Mi for mebibytes and Gi for gibibytes. A common beginner trap is mixing up M and Mi, which differ by about five percent. Always prefer Mi and Gi so the arithmetic stays predictable.

Where each number goes

Understanding the journey of these numbers removes the mystery. When you create a pod, four different components touch it, in this order.

The scheduler does not look at real usage

Here is a fact that surprises almost everyone. The scheduler places pods using the requested numbers, not the actual usage. If a pod requests 4 cores but really uses 0.2 cores, the scheduler still treats 4 cores as taken. This protects guarantees, but it also means over-asking wastes money, because the unused reservation cannot be given to anyone else for placement.

CPU is the resource people argue about the most. The reason is that Kubernetes enforces CPU in two completely different ways, and they behave differently.

Mechanism one: shares (the request)

When the machine is busy, the kernel needs a rule for dividing processor time. Kubernetes translates each container’s CPU request into a weight, called CPU shares in cgroup v1 and CPU weight in cgroup v2. A container requesting 1000 millicores receives roughly twice the weight of a container requesting 500 millicores. Weights matter only during competition. When the machine has idle processors, a container may use far more than its request, even with no limit at all.

This is a key point about guarantees. A CPU request is a proportional guarantee, not a reserved core. If two containers both want everything, they split the processor in proportion to their requests. If only one wants it, that one gets it all.

Mechanism two: quota (the limit)

The CPU limit works differently. The kernel uses a feature called the CFS bandwidth controller. Time is cut into short periods, 100 milliseconds by default. A container with a limit of 1 core receives a budget of 100 milliseconds of CPU time in every 100-millisecond period. Once the budget is spent, every thread of that container is paused until the next period begins.

flowchart LR
  A["New 100ms period starts"] --> B["Container runs and spends quota"]
  B --> C{"Quota used up?"}
  C -- "No" --> B
  C -- "Yes" --> D["All threads paused: throttled"]
  D --> E["Wait until the period ends"]
  E --> A
    

Why throttling surprises people

The budget is counted across all threads together. Suppose your service has a limit of 1 core but runs four busy threads at the same moment. Together they burn 100 milliseconds of budget in just 25 milliseconds of real time. Then all four are frozen for the remaining 75 milliseconds. A request that needed a quick answer now waits for no reason you can see on a usage graph, because the average CPU usage over a minute still looks low.

The great debate: should you set CPU limits?

There is no universal winner. A shared batch cluster with many teams often benefits from limits. A dedicated, latency-critical service often benefits from none. What matters is that you choose deliberately and measure the outcome.

CPU pinning for the strictest guarantee

For workloads that cannot tolerate any sharing, such as trading engines or media encoders, the kubelet offers the static CPU manager policy. Pods in the Guaranteed class that request whole numbers of cores can be given exclusive cores, which no other pod may use. We explore this fully in the advanced hardware chapter.

Memory is the unforgiving resource. CPU can be shared by taking turns. Memory cannot. Once a byte is handed out, it stays handed out until the owner gives it back.

What the memory limit does

The memory limit becomes a hard ceiling in the container’s cgroup. If the processes inside try to hold more than that ceiling, the kernel first tries to free memory it can safely recover, such as cached file data. If that is not enough, the kernel’s OOM killer chooses a process inside that cgroup and terminates it. Kubernetes then reports the container as OOMKilled, usually with exit code 137, and restarts it according to the pod’s restart policy.

The most important point is the scope of the damage. The OOM kill happens inside the container’s own cgroup. A container that exceeds its own limit hurts only itself. That is the memory-isolation guarantee in its purest form: your memory mistakes stay inside your own walls.

What the memory request does

The memory request is mainly used by the scheduler for placement and by the kubelet when deciding who to remove first during node-wide pressure. By default, in the classic setup, the request itself is not written as a hard kernel guarantee. This means memory requests are a promise made at scheduling time. Newer Kubernetes work on memory quality of service, which depends on cgroup v2, aims to make the kernel protect the requested amount more directly, but you should check the maturity of that feature in your Kubernetes version before depending on it.

What counts as “memory used”?

Beginners are often confused because several numbers describe memory. The container’s cgroup counts program memory, file cache, and some kernel memory. File cache can usually be reclaimed, so it does not cause kills right away. The number that the kubelet watches when making eviction decisions is called the working set: memory in active use that the kernel cannot easily reclaim. When you watch dashboards, prefer the working-set metric, not the raw usage total, because the raw total includes cache that may look scary but is harmless.

Swap: the quiet exception

Historically, Kubernetes required nodes to run with swap turned off, because swapping memory to disk makes behavior unpredictable and weakens the limit guarantee. Newer versions have introduced carefully limited swap support, mainly for pods in the Burstable class. The purpose is to give bursty workloads a small safety cushion. For strict isolation, most production teams still keep swap disabled or tightly controlled.

When a machine runs short of memory, Kubernetes needs a clear pecking order. That order is called Quality of Service, or QoS. You never set it directly. Kubernetes derives it from the numbers you wrote.

The three classes

How the class turns into kernel behavior

The kubelet uses the QoS class to set a number called oom_score_adj on every container process. When the kernel itself must kill something because the whole node is out of memory, it prefers processes with higher scores. Guaranteed pods receive a very low score, so they are protected. BestEffort pods receive the maximum score, so they are chosen first. Burstable pods receive a score that depends on how much memory they requested compared to the size of the node, which means a Burstable pod that asked for more is better protected than one that asked for less.

Decision flow

flowchart TD
  S["Pod is created"] --> Q1{"Any request or limit set?"}
  Q1 -- "No" --> BE["BestEffort"]
  Q1 -- "Yes" --> Q2{"CPU and memory set on every container, request equals limit?"}
  Q2 -- "Yes" --> G["Guaranteed"]
  Q2 -- "No" --> BU["Burstable"]
    

Choosing a class on purpose

Pods are not the only residents of a machine. The operating system, the container runtime, and the kubelet also need resources. If pods eat everything, the node itself can freeze, and every pod on it falls with it.

Capacity versus allocatable

A node reports two numbers. Capacity is the total hardware: for example, 16 cores and 64 gigabytes of memory. Allocatable is the portion that pods are allowed to request. The scheduler uses allocatable, never capacity. The difference is the safety margin that protects the machine.

The three reservations

Managed services help here. Amazon EKS and Google GKE both calculate sensible reserved amounts automatically, based on the size of the machine, so beginners get a safer default. Self-managed clusters must set these values explicitly, and forgetting to do so is one of the most common causes of mysterious node freezes.

Node-pressure eviction

The kubelet continuously watches signals such as available memory, available disk space, free inodes, and available process IDs. When a signal crosses a configured threshold, the kubelet starts evicting pods: it gracefully stops them to reclaim resources before the situation turns into a crash. There are two kinds of thresholds.

Who is evicted first?

The kubelet ranks candidates using a sensible, fair order. First come pods using more of the starved resource than they requested. Among those, lower priority goes before higher priority. After that, the pod with the largest overage compared to its request goes first. This is why a pod that sticks near its request is much safer than a pod that regularly bursts far above it.

CPU and memory get all the attention, yet many real outages come from two quieter resources: disk space and process counts.

Ephemeral storage

Every container writes some files that disappear when the pod goes away: logs, temporary files, downloaded caches, and the container’s writable layer. These live on the node’s disk and are called ephemeral storage. You can set requests and limits for it exactly like memory. The scheduler uses the request to place the pod. If a pod exceeds its limit, the kubelet evicts it. Unlike memory, the kernel does not instantly kill the process; the kubelet notices during its regular checks.

Without ephemeral limits, one chatty application writing debug logs can fill the node disk. When disk space runs out, the kubelet marks the node as under disk pressure, new pods are refused, and running pods begin to be evicted. A single noisy logger can therefore disrupt unrelated teams.

Persistent volumes are different

Data that must survive pod restarts lives in persistent volumes, which usually sit on separate network storage rather than the node’s own disk. A persistent volume claim has a requested size, and storage quotas can cap how much each namespace may claim. Performance isolation for disks, such as how many operations per second a volume can perform, usually comes from the storage service itself. For example, a cloud block-storage volume has its own provisioned throughput, independent of its neighbors.

Process ID limits

Every process and thread uses a process ID from a finite pool. A buggy program that keeps creating new processes, sometimes called a fork bomb, can exhaust the pool and make the entire machine unable to start anything. Kubernetes can cap the number of processes per pod through a kubelet setting, and the kubelet can reserve process IDs for system use. Cloud providers often set safe defaults, but self-managed clusters should confirm them.

So far we protected pods from each other on a single machine. Now we zoom out to protect whole teams from each other across the entire cluster.

The cluster-wide budget problem

Picture a company with ten teams sharing one cluster. Team A deploys an enormous job that requests half of all the cluster’s CPU. The scheduler is happy to place it, because every request fits. Now Teams B through J find that their new pods stay in the Pending state with no room left. No machine was overloaded. No limit was crossed. The cluster was simply handed out unfairly.

Kubernetes solves this at the namespace level with two objects that work as a pair: ResourceQuota sets a total budget, and LimitRange sets per-container rules and defaults.

ResourceQuota: the namespace budget

A ResourceQuota caps the total amount a namespace may use. It can limit the sum of CPU requests, memory requests, CPU limits, and memory limits across all pods in the namespace. It can also limit the count of objects, such as the number of pods, services, secrets, or persistent volume claims, and the total storage that may be claimed. When a new pod would push the namespace over its budget, the API server rejects it immediately with a clear error.

LimitRange: the per-container rulebook

A LimitRange works inside a namespace and controls individual containers or pods. It can define a minimum and maximum size for any container, a default request and default limit applied automatically when someone forgets to specify them, and a maximum ratio between limit and request so that nobody sets a tiny request with a huge limit.

The two objects depend on each other in an important way. If a namespace has a quota on CPU or memory, every new pod must declare those values, otherwise the API server rejects it. A LimitRange with sensible defaults fills in the missing numbers, so developers do not have to remember them. Together they form a safety net: nobody can forget to set resources, and nobody can set absurd ones.

flowchart LR
  DEV["Developer submits pod"] --> API["API server"]
  API --> LR["LimitRange: apply defaults and min/max checks"]
  LR --> RQ{"ResourceQuota: budget left?"}
  RQ -- "Yes" --> SCH["Scheduler places pod"]
  RQ -- "No" --> REJ["Request rejected with error"]
    

Quota scopes and priorities

Quotas can be scoped. For example, you can give a namespace a small budget for high-priority pods and a larger budget for low-priority batch pods. This stops a team from labeling everything as top priority to avoid eviction. Scopes can also separate pods that have a deadline or that are terminating, helping platform teams shape behavior with precision.

The strongest form of isolation is never sharing a machine at all. Scheduling controls let you decide exactly who may live next to whom.

Taints and tolerations: reserved neighborhoods

A taint is a mark placed on a node that says “keep out unless you have permission”. A toleration is a permission slip carried by a pod. A pod without the matching toleration cannot be placed on a tainted node. Platform teams taint nodes for special purposes: machines with expensive graphics cards, nodes reserved for a sensitive team, or nodes dedicated to a noisy batch workload.

Taints work like a gated community. The gate stops outsiders. But remember that a toleration only allows a pod to enter; it does not force the pod to go there. To attract the pod to the tainted nodes as well, you combine the toleration with a node selector or node affinity.

Node affinity and pod anti-affinity

Node affinity tells the scheduler which kinds of nodes a pod prefers or requires. Pod anti-affinity tells the scheduler “do not put me on the same machine, or the same zone, as these other pods”. It is a powerful isolation tool in two directions. You can keep your own replicas apart so one machine failure cannot take down all of them. And you can keep a sensitive service away from a known noisy neighbor.

Priority and preemption

Not every pod is equally important. A PriorityClass attaches a number to pods. When the cluster is full, the scheduler can preempt: it removes lower-priority pods to make room for a higher-priority pod that is waiting. Priority also influences eviction order during node pressure, so important services are removed last.

Choosing the right level of separation

Some workloads need more than a fair share. They need a specific processor core, or memory physically close to that core, or exclusive use of a special device. Kubernetes has managers for each need.

Why plain cgroups are not enough

Even with CPU shares and quotas, the kernel may move a busy process from core to core, sharing each core’s tiny internal caches with strangers. For most web services this is fine. For tasks such as video encoding, high-frequency trading, or packet processing, the constant moving causes small but measurable delays. These workloads want a core to themselves.

The CPU Manager

The kubelet’s CPU Manager has two policies. The default policy, none, uses the shares and quota behavior described earlier. The static policy gives pods in the Guaranteed class that request whole numbers of cores an exclusive set of cores. Other pods are kept off those cores. The result is the closest thing to a private processor that Kubernetes offers.

NUMA: when memory has a geography

Large servers have more than one processor socket, and each socket has memory physically attached to it. This design is called NUMA, short for non-uniform memory access. A core reads from its own socket’s memory quickly and from the other socket’s memory more slowly. A pod that receives cores from one socket but memory from the other pays a constant speed penalty.

flowchart LR
  subgraph N0["NUMA node 0"]
    C0["Cores 0-7"] --- M0["Local memory"]
  end
  subgraph N1["NUMA node 1"]
    C1["Cores 8-15"] --- M1["Local memory"]
  end
  C0 -. "slower remote access" .- M1
  C1 -. "slower remote access" .- M0
    

Memory Manager and Topology Manager

The Memory Manager can assign memory and huge pages from specific NUMA nodes to Guaranteed pods. The Topology Manager acts as a coordinator. It asks the CPU Manager, the Memory Manager, and the device plugins which resources each can offer, then insists they all come from the same NUMA node. It supports policies from relaxed best-effort alignment to a strict mode that refuses to start a pod unless everything fits on a single node.

Huge pages

Memory is normally handed out in small 4-kilobyte pages. Databases and similar programs that manage enormous amounts of memory can be faster with much larger huge pages. Kubernetes treats huge pages as their own schedulable resource, separate from ordinary memory, with their own requests and limits. They are pre-reserved on the node, so the usual overcommit surprises do not apply.

Devices and accelerators

Graphics cards and other accelerators are exposed through device plugins. A pod asks for a whole number of devices, and the device is then dedicated to that pod alone. Newer Kubernetes work on dynamic resource allocation aims to make requesting and sharing such devices richer and more flexible, so check the status of that feature in your version.

Changing resources without a restart

Traditionally, changing a pod’s CPU or memory required recreating the pod. Recent Kubernetes versions are adding in-place pod resize, which lets some resource changes happen on a running pod, and work on pod-level resource settings that cover all containers in a pod together. These features are maturing quickly, so confirm their stage in your cluster’s version before relying on them.

By default, a Kubernetes cluster is wide open: any pod can talk to any other pod. Isolation here means two different things: controlling who can connect, and controlling how much bandwidth they may use.

Who may talk: NetworkPolicy

A NetworkPolicy is a rule that selects a group of pods and says which incoming and outgoing connections are allowed. The standard pattern is to apply a “deny all” policy to a namespace first, and then add small rules that open exactly the paths your application needs. A payment service might accept connections only from the checkout service and may call only the database and an outside bank gateway.

How much bandwidth: the missing guarantee

Kubernetes has no first-class request and limit for network bandwidth. Some network plugins support bandwidth-shaping annotations on a pod to cap ingress and egress rates. Cloud networks also impose per-machine limits that all pods on a node share. This means the network is the resource where noisy neighbors are hardest to prevent, and many teams rely on dedicated node pools and monitoring instead.

Shared pathways you might forget

Resource isolation keeps honest neighbors from accidentally hurting each other. Security isolation keeps dishonest or compromised neighbors from deliberately doing so. The two goals overlap, but they need different tools.

Containers share a kernel

Because every container on a node uses the same Linux kernel, a serious flaw in that kernel can let a process escape its container. This is the fundamental difference between a container and a virtual machine, which carries its own kernel behind a hardware boundary. For friendly teams inside one company, container isolation is usually considered strong enough. For running code from strangers, many organizations demand more.

Hardening layers inside a normal container

Enforcing the rules: Pod Security Standards

Kubernetes defines three standard profiles. Privileged allows everything and is meant for trusted system components. Baseline blocks the known dangerous options. Restricted follows current hardening best practice. The built-in Pod Security Admission controller can enforce these per namespace, so a team namespace can be made to reject any pod that asks for dangerous powers such as host networking or privileged mode.

Stronger walls: sandboxed runtimes

For untrusted workloads, Kubernetes supports alternative container runtimes through the RuntimeClass object. Two common families exist. A user-space kernel such as gVisor intercepts system calls so the real kernel is exposed to far fewer requests. A lightweight virtual machine runtime such as Kata Containers gives each pod its own small kernel behind a hardware boundary. Both cost some performance and a little memory overhead, which Kubernetes can account for in scheduling.

Do not forget identity and secrets

Resource walls mean little if every pod holds the keys to the whole cluster. Give each workload its own service account, grant it only the permissions it needs, and avoid mounting credentials that the pod does not use. On Amazon EKS, mapping individual service accounts to individual cloud roles keeps one compromised pod from reaching another team’s data.

Individual settings are useful, but real protection comes from stacking them. This chapter shows how the layers combine into one coherent picture.

flowchart TD
  U["Team submits workload"] --> ADM["Admission layer: LimitRange defaults, ResourceQuota budget, Pod Security checks"]
  ADM --> SCHED["Scheduling layer: requests, taints, affinity, priority, topology spread"]
  SCHED --> NODE["Node layer: allocatable, reservations, eviction thresholds"]
  NODE --> KERNEL["Kernel layer: cgroups for CPU, memory, PIDs; namespaces for visibility"]
  KERNEL --> RUN["Runtime layer: seccomp, capabilities, optional sandboxed runtime"]
  RUN --> NET["Network layer: NetworkPolicy, bandwidth shaping, service mesh"]
    

Reading the diagram

Follow one workload down the layers. First, the admission layer decides whether the pod may enter the cluster at all, fills in missing numbers, and checks the team’s budget. Next, the scheduling layer picks a suitable machine using requests and placement rules. The node layer ensures the machine keeps enough free resources for itself. The kernel layer then enforces the actual caps and hides what the pod should not see. The runtime layer reduces what the process may ask the kernel to do. Finally, the network layer controls the connections the pod may make.

If one layer fails, the layers below still catch part of the damage. A pod that slips past a missing quota still faces node eviction. A process that escapes a seccomp filter still sits inside a memory cgroup. This is called defense in depth, and it is the right mental model for the entire topic.

A sample tiered platform

To make the layers concrete, imagine a company that runs a shared cluster with four tiers of workloads.

A guarantee you cannot observe is only a hope. These are the signals that tell you whether isolation is working or quietly failing.

Signals for CPU

The most useful CPU signal is the throttling ratio: the share of scheduling periods in which a container hit its quota and was paused. Standard container metrics expose both the number of periods and the number of throttled periods, so dividing the second by the first gives a clear percentage. Sustained throttling above a small percentage on a latency-sensitive service is a sign the limit is too tight or the thread count too high.

Signals for memory

Watch the working-set size compared to the limit. A container that spends its days at 90 to 95 percent of the limit is one traffic spike away from an OOM kill. Also track the count of container restarts with the reason OOMKilled, because a single surprise kill may be harmless but a pattern is a warning.

Signals for the node

Signals for fairness

Track quota usage by namespace, the number of pods stuck in Pending and the reason, and the gap between requested and used resources. A big gap means money is being wasted. A tiny gap with frequent throttling or kills means requests and limits are too tight.

Right-sizing with real data

The Vertical Pod Autoscaler can watch real usage and recommend request values. Many teams run it in recommendation mode only, then review and apply the suggestions through their normal change process. Combine it with the Cluster Autoscaler, which adds machines when pending pods cannot fit, so extra requests turn into extra capacity rather than stuck workloads.

Concepts feel easier once you can see the numbers. In this chapter we fill one imaginary machine step by step and watch where the guarantees hold and where they quietly stretch.

The imaginary node

Suppose we have one worker machine with 16 CPU cores and 64 gigabytes of memory. After the platform team reserves space for the operating system, the Kubernetes agents, and the eviction buffer, the node reports an allocatable amount of roughly 15 cores and 58 gigabytes. These figures are illustrative, but the shape is realistic: a small slice is always held back. From now on, 15 cores and 58 gigabytes are the only numbers the scheduler cares about.

Three kinds of workloads move in

We place three groups of pods on this node. Each group has a different personality, and each uses requests and limits differently.

Adding up the requests

The scheduler adds only the requests. For CPU, the checkout pods contribute 3 cores, the search pods contribute 4 cores, and the report jobs contribute 2 cores. The total is 9 cores out of 15 allocatable. For memory, the checkout pods contribute 6 Gi, the search pods 16 Gi, and the report jobs 4 Gi, for a total of 26 Gi out of 58 Gi. The node has plenty of room on paper, so the scheduler is happy to keep adding pods.

Adding up the limits

Now add the limits instead. For CPU, the limits total 6 plus 8 plus 16, which is 30 cores. That is double the 15 cores the node allocates. For memory, the limits total 6 plus 16 plus 16, which is 38 Gi, comfortably below 58 Gi. This difference between the sum of requests and the sum of limits is called overcommitment. The CPU limits add up to 200 percent of allocatable, so CPU is overcommitted. The memory limits stay under 100 percent, so memory is not overcommitted yet.

Why CPU overcommitment is usually survivable

All the pods could in theory try to burn their CPU limits at once, wanting 30 cores on a 15-core machine. What happens then? The kernel falls back to the weights it created from the requests. Every pod receives processor time in proportion to what it requested. The checkout pods, requesting 1 core each, still get their fair share, and the report jobs slow down. Nobody crashes. CPU overcommitment trades speed for density, and it is the reason clusters can run so many workloads on so little hardware.

Pushing the node into danger

Now imagine a well-meaning team deploys ten more report jobs with the same settings. The extra requests add 5 cores and 10 Gi, so the totals become 14 cores of 15 and 36 Gi of 58 Gi. The scheduler accepts every pod, because requests still fit. But the memory limits have grown by 40 Gi, so the total allowed memory is now 78 Gi on a node that only has 58 Gi to give. The memory limits now add up to about 134 percent of allocatable, so memory is overcommitted.

If the report jobs all grow toward their 4 Gi limits at the same moment, the node runs out of memory long before any single job reaches its own ceiling. Each pod is behaving “legally”. Yet the machine is in trouble. The kubelet begins evicting pods, and the ranking rules from earlier chapters decide who goes. The report jobs, having used far more than they requested, are removed first, which is exactly the outcome a platform team would want. But the example shows why memory limits far above memory requests are risky: they make promises the machine cannot keep.

flowchart TD
  A["Scheduler checks only requests"] --> B{"Requests fit in allocatable?"}
  B -- "Yes" --> C["Pod is placed"]
  C --> D{"Do actual usage and limits grow together?"}
  D -- "CPU grows" --> E["Weights share CPU by request: slower, no crash"]
  D -- "Memory grows" --> F["Node memory pressure: kubelet evicts pods above request"]
  B -- "No" --> G["Pod stays Pending"]
    

Rules of thumb from the example

Theory is one thing. Real incidents unfold slowly and confusingly. This chapter follows a realistic outage from the first complaint to the final fix, so you can recognize the same pattern in your own cluster.

The setup

A company runs an online store on a shared Kubernetes cluster. The checkout service is the most important application, and it shares worker machines with a reporting platform used by the finance team. The checkout pods declare small CPU requests and no CPU limits. The reporting pods declare small requests and generous limits. Nobody has ever measured throttling, and no alerts watch it.

The incident timeline

What the investigation found

The post-incident review identified four separate weaknesses, and notice that none of them alone would have caused an outage.

The fixes, in the order they were applied

A short diagnosis checklist

When a service is slow and you suspect a neighbor, work through these questions in order. Start with the pod: is it throttled, and was it recently restarted or OOM-killed? Then look at the node: is it showing memory, disk, or process pressure, and who else is running there? Next look at the neighbors: which pods use far more than they requested? Finally check the policy: do requests, priority, and placement rules reflect how important each workload really is? This simple sequence solves most mysteries in a few minutes.

Autoscalers add and remove pods and machines automatically. They depend heavily on requests, so getting isolation settings wrong makes autoscaling behave strangely.

Three autoscalers, three jobs

Why requests control everything

The Horizontal Pod Autoscaler usually measures utilization as a percentage of the CPU request. If you set the request too low, usage looks like 300 percent, and the autoscaler launches far too many pods. If you set the request too high, usage looks like 10 percent, and the autoscaler never scales out even when the service is struggling. Accurate requests therefore protect both isolation and autoscaling at the same time.

Node autoscaling and Pending pods

Node autoscalers do not look at real usage either. They react to pods that cannot be placed because requests do not fit anywhere. This is why quotas and requests shape cost directly. A team that doubles its requests causes the autoscaler to add real machines, even if the application uses the same amount of CPU as before. On Amazon EKS, tools such as Karpenter choose machine sizes that fit the pending requests, which makes request accuracy even more valuable.

Keeping spare room with low-priority placeholders

Adding a machine can take a few minutes, which is too slow for sudden traffic spikes. A popular trick uses priority to keep a cushion. The platform team runs a few placeholder pods with very low priority that do nothing but reserve space. When a real workload needs room, the scheduler preempts the placeholders instantly, and the real pod starts right away. The displaced placeholders then become Pending, which triggers the node autoscaler to add capacity in the background.

flowchart LR
  P["Placeholder pods: lowest priority, reserve space"] --> N["Node has spare room on paper"]
  R["Real pod arrives with higher priority"] --> X{"Room available?"}
  X -- "No" --> PRE["Scheduler preempts placeholders"]
  PRE --> RUN["Real pod starts immediately"]
  PRE --> PEND["Placeholders become Pending"]
  PEND --> CA["Node autoscaler adds a machine"]
    

Watch out for these interactions

A tenant is any group that shares a cluster with others: a team, a product, a customer. How much you trust your tenants decides how much isolation you need and how much you will pay for it.

Soft multi-tenancy

Soft multi-tenancy assumes the tenants are friendly. They work for the same company, follow the same rules, and want the cluster to stay healthy. The main danger is accidents, not attacks. The standard toolkit is namespaces, quotas, limit ranges, network policies, pod security profiles, and role-based access control. It is cheap, easy to run, and good enough for most internal platforms.

Hard multi-tenancy

Hard multi-tenancy assumes some tenants may be hostile, or that a mistake in one tenant’s code could be exploited. This is the situation when you host customers’ own applications or run code supplied by strangers. Shared-kernel containers alone are not enough. You add sandboxed runtimes, dedicated nodes, strict network separation, and often a separate control plane per tenant.

The ladder of separation

Shared control plane concerns

Even when worker machines are separated, tenants often share the control plane: the API server, the scheduler, and the data store behind them. A tenant that floods the API server with requests can slow down everyone. Kubernetes includes an API priority and fairness feature that separates request flows and prevents one noisy client from starving the rest. Cluster-wide objects such as custom resource definitions and admission webhooks are also shared, and a broken webhook can block pod creation for every tenant.

Choosing a model

If you operate your own nodes, a handful of kernel-level details will save you hours of confusion. This chapter keeps them simple.

Four memory dials

Cgroup v2 gives each group of processes several memory controls, not just one ceiling. Think of them as four dials on the same cabinet.

The hard ceiling is what a normal Kubernetes memory limit sets. The other dials are what the newer memory quality-of-service work tries to use, so that a requested amount is protected like a floor and the limit has a gentler soft ceiling in front of it. Because this area is evolving, always read the notes for your Kubernetes version before depending on it.

Pressure stall information

One of the best features of cgroup v2 is pressure stall information, often shortened to PSI. Instead of only telling you how much of a resource is used, it tells you how much time tasks spent waiting for it. That is the real symptom of a noisy neighbor. A group might use a modest amount of CPU but spend a large percentage of its time waiting in the queue. PSI turns that hidden delay into a number you can graph and alert on. Support for exposing it through Kubernetes metrics depends on your version, and the data is also available directly from the node.

Disk and network controls

The kernel can limit disk operations through its input and output controller, but Kubernetes does not expose a pod-level disk-speed setting in its standard resource model. Platform teams who need it usually rely on separate volumes with provisioned throughput, local disks reserved for one workload, or storage classes that differ in performance. The same is true for network throughput, as covered earlier.

The cgroup driver choice

The kubelet and the container runtime must agree on who manages the cgroup tree. On modern Linux distributions that use systemd, both should use the systemd driver. A mismatch between the kubelet and runtime is a classic source of strange behavior, such as pods that start but never receive the limits you set, or nodes that become unstable under load. Managed services configure this correctly. Self-managed clusters should verify it during setup.

Kernel version matters

Many of the best isolation features arrive through the Linux kernel, not through Kubernetes itself. Older kernels had known problems with CPU throttling that caused containers to be paused even when they had not used their full quota. Those problems were fixed in newer kernels, and many distributions backported the fixes. If you see unexplained throttling with low usage on an old node image, upgrading the kernel is one of the first things to try.

Many readers run Kubernetes on AWS, so this chapter connects the earlier ideas to the choices you meet on Amazon EKS. Treat it as a map of where each layer lives in the AWS world.

Who manages which layer

On EKS, AWS operates the control plane: the API server and the data store. You operate, or choose to hand off, the worker side. That split matters, because it decides who is responsible for each isolation layer.

Choosing between nodes and Fargate

Pod-level cloud permissions

On EKS you can link a Kubernetes service account to an AWS identity role, so each workload receives only the cloud permissions it needs. This is an isolation feature in its own right. If one pod is compromised, it can reach only the storage buckets and queues that its own role allows, not the whole account. Combine it with a separate service account for every workload and with network policies for a strong baseline.

Network considerations on EKS

By default, pods on EKS receive addresses from your virtual network, and each machine type supports a limited number of network interfaces and addresses. This means the number of pods per node can be capped by networking, not by CPU or memory. Teams sometimes see Pending pods even when requests would fit, simply because a node has run out of pod addresses. Security groups can also be applied to pods in supported setups, adding a cloud-level network boundary beyond Kubernetes network policies.

A baseline checklist for EKS

Most isolation failures are not exotic. They come from the same handful of mistakes, repeated across companies. Learn them once and avoid years of pain.

These are the questions that come up most often when teams first take isolation seriously.