Capacity is the balance between reliability and cost. Planning it badly makes you pay either way.

Two ways to get it wrong

The first failure is too little capacity. Pods cannot be placed, so they stay in a waiting state. Nodes run hot, so applications slow down or get killed. A traffic spike or the loss of a machine pushes the whole cluster over the edge, and a small problem becomes an outage. These failures are loud and urgent, and they happen at the worst possible moments.

The second failure is too much capacity. Nothing breaks, so nobody complains. But machines sit mostly idle, and the company pays for them every hour. Because the waste is silent, it can last for years. In large organizations, idle capacity often adds up to a very large sum of money.

Why Kubernetes makes this trickier

Kubernetes adds layers between your application and the machine. A single sentence like “this app needs a lot of memory” becomes a request on a pod, which is placed by a scheduler on a node, which itself has reservations for system programs, which sits in a zone that may fail. Each layer has its own rules, its own counting and its own limits. Capacity planning means understanding all those layers and making them agree.

What a good plan answers

• How much CPU, memory, storage and network will our workloads need at their busiest?
• What shape should the machines have, small or large, balanced or memory-heavy?
• How much spare room do we keep for failures, upgrades and surprises?
• When will we run out if demand keeps growing, and how long does it take to add more?
• What limits other than machines could stop us, such as network addresses or cloud quotas?
• How do we know the plan is right before real users find out?

Many people think autoscaling makes planning unnecessary. It does not. The two work at different time scales and answer different questions.

What autoscaling does

Autoscalers react to the present. When load rises, the pod autoscaler adds pods, and the node autoscaler adds machines. When load falls, they remove them. This handles ups and downs within a day or a week very well. But autoscalers can only act inside a box. They need a place to put new machines, permission to create them, money to pay for them and limits that allow it.

What capacity planning does

Capacity planning draws the box. It decides the minimum and maximum size of node groups, which machine types are allowed, how many network addresses are reserved, how big the control plane must be and how much the organization will spend. It also looks far ahead, at the next quarter or the next year, and asks whether the box itself will be big enough.

flowchart TB
    PLAN["Capacity planning: sets the box"] --> BOX["Node types, min and max sizes, quotas, IP ranges, budget"]
    BOX --> AUTO["Autoscalers: move inside the box"]
    LOAD["Real traffic changes"] --> AUTO
    AUTO --> CLUSTER["Cluster size right now"]
    CLUSTER -->|"actual usage and limits hit"| FEEDBACK["Measurements"]
    FEEDBACK --> PLAN

Why the lead time matters

Autoscaling can add a cloud machine in minutes, but only if the cloud agrees. If you are close to an account limit, a request for more can take days. In your own data centre, buying and installing servers can take months. Capacity planning is the art of asking early enough that the answer arrives before you need it.

When people say “capacity”, they usually mean CPU and memory. In practice, many other limits can run out first.

The big four

The quiet limits

Beyond these four, there is a second group of limits that rarely appear on dashboards until they bite.

The weakest link decides

A cluster is only as large as its tightest limit. You might have plenty of CPU but no free pod addresses, or plenty of memory but no spare disk attachments. The goal of planning is to find the tightest limit for each workload type, and to make sure none of them is a surprise.

This is the single most important idea in the whole guide. Kubernetes plans with promises, not with measurements.

Three different numbers

For every container, three numbers matter, and beginners often mix them up.

The scheduler plans with requests

When a new pod arrives, the scheduler adds up the requests of all pods already on each node and subtracts them from what the node can offer. It places the new pod on a node that still has room for the new pod’s requests. Real usage is never consulted. So a node can be fully booked while its CPU is almost idle, or it can be heavily used while the scheduler thinks it has room, if the pods asked for much less than they use.

flowchart LR
    REQ["Pod requests: what was promised"] --> SCHED["Scheduler adds up requests per node"]
    SCHED --> FIT{"Does the new pod fit in the remaining requested room?"}
    FIT -->|"yes"| PLACE["Pod is placed on the node"]
    FIT -->|"no"| PEND["Pod stays Pending"]
    PLACE --> RUN["Pod runs and uses real resources"]
    RUN --> USE["Usage: measured by monitoring"]
    USE -.->|"not used for placement"| SCHED

Two kinds of capacity numbers

Because of this, every cluster has two different pictures. Allocation is what has been promised through requests. Utilization is what is really used. Capacity planning must look at both. Allocation tells you when the scheduler will run out of room. Utilization tells you whether the room is actually busy. The gap between them is the clearest sign of waste or of risk.

Quality of service classes

Kubernetes sorts pods into three classes depending on how their requests and limits are set. These classes decide who is protected and who is sacrificed when a node runs short.

The two main resources are not the same kind of thing. Understanding the difference explains most surprises in real clusters.

CPU can be squeezed

CPU is called a compressible resource. If a container wants more processing time than it is allowed, the node does not kill it. It simply gives the container less time, so the program runs slower. This slowing down is called throttling. The node divides processor time into tiny slices, and a container that has used its share for the current slice must wait for the next one.

Throttling is sneaky because nothing crashes. Instead, response times get longer, sometimes in short bursts that average out on a dashboard but hurt real users. A container can be throttled even when the node as a whole is quite idle, if its own limit is low.

Memory cannot be squeezed

Memory is an incompressible resource. Once a program has filled memory with data, the node cannot take it back without destroying the program. If a container goes above its memory limit, the node stops it abruptly. This is called an out-of-memory kill, and the container is then restarted if its pod rules allow. If the whole node runs short of memory, the node starts evicting pods to protect itself.

What this means for planning

Because CPU bends and memory breaks, teams often treat them differently. Many set CPU requests near typical busy usage and are relaxed about CPU limits, while setting memory requests and limits close to the real peak. Programs whose memory is managed by a runtime, such as those with garbage collectors, need extra care because their memory can jump in steps rather than grow smoothly.

A machine with 16 cores does not give your applications 16 cores. Part of it is spoken for before any pod arrives.

Where the missing room goes

Every node runs its own helper programs: the agent that talks to the cluster, the container runtime, the operating system and its logging. These need CPU and memory too. Kubernetes lets you reserve space for them and also sets safety thresholds so that the node does not collapse under pressure. What remains for your pods is called allocatable.

flowchart TB
    CAP["Node capacity: everything the machine has"] --> KR["Reserved for Kubernetes helpers"]
    CAP --> SR["Reserved for the operating system"]
    CAP --> EV["Eviction safety margin"]
    CAP --> ALLOC["Allocatable: what pods can request"]
    ALLOC --> DS["Per-node helper pods such as logging and monitoring agents"]
    ALLOC --> APPS["Your application pods"]

The pieces explained

• Kubernetes reserved: room kept for the node agent and the container runtime.
• System reserved: room kept for the operating system itself.
• Eviction thresholds: when free memory or disk falls below a set level, the node starts removing pods to protect itself. Typical defaults include keeping at least about 100 mebibytes of memory free and some percentage of disk free. This slice is also subtracted from what you can allocate.

The exact defaults vary by Kubernetes version and by cloud provider, and managed services often set their own reservations that grow with machine size. You can read the real numbers from each node’s status, which lists capacity and allocatable side by side.

The small-machine tax

Reservations do not shrink in proportion to the machine. A tiny node loses a larger percentage to overhead than a big one. On a very small machine, the fixed costs of the operating system and node agent might eat a quarter or more of the memory. On a large machine the same costs are a small share. This is one reason very small nodes can be wasteful.

Per-node helper pods

Some pods run once on every node, such as log collectors, monitoring agents, network components and security tools. They are called DaemonSet pods. Their requests are subtracted from the room available for your applications on each node. On a hundred nodes, a helper that asks for a quarter of a core costs twenty-five cores of your cluster. Count these costs; they grow with the number of nodes, not with the amount of traffic.

You cannot plan from a feeling. A plan starts with a trustworthy picture of today.

The measurements you need

Use percentiles, not only averages

An average hides the peaks that actually cause outages. If a service uses 1 core most of the day and 4 cores for ten minutes during a rush, its average may look like 1.1 cores, but the cluster must be able to supply 4. Planning therefore looks at high percentiles, for example the ninety-fifth or ninety-ninth, and also at the absolute maximum over a long window that includes busy days.

Look at a long enough window

One normal week may not contain the monthly report run, the end-of-quarter rush or the seasonal sale. A good rule is to look at at least one full business cycle, and preferably more, so that you see weekly rhythms and the rare peaks. Short retention in your monitoring system is a hidden capacity risk because it forces you to plan blind.

Allocation versus utilization: reading the gap

flowchart LR
    APPS["Workloads"] --> KUBELET["Node agents report usage"]
    KUBELET --> COLLECT["Metrics collector"]
    COLLECT --> STORE["Long-term metrics storage"]
    STORE --> DASH["Dashboards and reports"]
    STORE --> ALERT["Alerts"]
    STORE --> MODEL["Capacity model and forecasts"]
    BIZ["Business traffic data"] --> MODEL

All cluster capacity maths rests on pod requests. If requests are wrong, every later number is wrong too.

Why requests are usually wrong

People set requests once, often by copying from another service or by guessing high to be safe, and then never touch them again. Studies of real clusters repeatedly show that a large share of requested CPU and memory is never used. The opposite error also exists: teams that set requests very low to fit more pods, and then suffer noisy-neighbour trouble.

A simple method

Tools that help

The Vertical Pod Autoscaler can watch usage and suggest better requests. It can run in a recommendation-only mode, which is a safe way to learn what it would change without it restarting any pods. Dashboards from tools such as Goldilocks present these recommendations per workload. Cost tools such as OpenCost show how much each team’s over-requesting costs in money, which is often the most persuasive evidence for people who own the workloads.

Guard rails that prevent new waste

• LimitRange: sets default and maximum requests and limits in a namespace so that pods with no settings still get sensible ones.
• ResourceQuota: caps the total a namespace can request, so one team cannot take everything.
• Policy checks: admission rules that reject pods with no requests or with absurdly large ones.

Should you buy a few huge machines or many small ones? The answer depends on your pods, your risk and your overhead.

Big nodes

Large machines waste less on fixed overhead, fit big pods easily and reduce the number of nodes to manage. But losing one removes a large slice of the cluster at once, and rolling upgrades drain a lot of work at a time. They also hit the per-node pod limit and the per-node network limits more easily when pods are small.

Small nodes

Small machines lose a bigger percentage to overhead and may be unable to host a large pod at all. Each one is cheap to lose, and capacity can be added in fine steps. But there are more machines to patch, more per-node helper pods and a heavier load on the control plane.

Matching the CPU-to-memory ratio

Cloud machines come in families with different ratios of memory to CPU. General-purpose machines often offer around four gibibytes per core, compute-optimized ones offer around two, and memory-optimized ones offer eight or more. If your workloads request memory and CPU in a ratio that does not match your machines, one resource runs out while the other is stranded. Compute the ratio of total requested memory to total requested CPU, and pick a family close to it.

Bin packing and stranded capacity

The scheduler is essentially packing pods into nodes. Because pods come in different sizes, some leftover room on each node is too small for any pod you actually run. This unusable leftover is called stranded capacity. Planners handle it by applying a packing efficiency factor, such as assuming only eighty-five percent of allocatable room can truly be used. Standardizing pod sizes and choosing node sizes that are clean multiples of typical pod sizes improves packing.

flowchart TB
    START["Workload profile: pod sizes and memory-to-CPU ratio"] --> RATIO["Pick a machine family with a close ratio"]
    RATIO --> SIZE["Pick a size where the largest pod fits comfortably"]
    SIZE --> CHECK{"Pod count, IP and volume limits still fine?"}
    CHECK -->|"yes"| MIX["Consider a second node group for special workloads"]
    CHECK -->|"no"| SIZE2["Adjust size or relax the limit"]
    SIZE2 --> SIZE
    MIX --> DONE["Node shapes chosen"]

Separate groups for special workloads

Not everything belongs on the same machine type. Memory-hungry databases, graphics-using jobs, short-lived batch work and sensitive workloads may each deserve their own group of nodes, marked with labels and taints so that only the right pods land there. Too many groups, however, split your spare room into separate puddles and increase total headroom needs. Keep the number of groups small.

A cluster running at one hundred percent is not efficient; it is fragile. Headroom is the price of resilience.

Why headroom exists

Several things need spare room at the same moment you least expect them.

The N+1 idea

N+1 means having enough capacity for the normal load plus one extra unit that can fail. For nodes, you size the cluster so that the work still fits after losing the largest single node. For larger risks, you extend it to N+2, or to the loss of a whole zone.

Surviving a lost zone

Imagine a cluster spread over three zones with equal capacity. If one zone fails, only two thirds of the capacity remains. To still carry the full load, the total must be one and a half times the load. In general, with N equal zones and the need to survive the loss of one, you multiply the needed capacity by N divided by N minus one. With three zones that is 1.5; with four zones it is about 1.33; with two zones it is 2.

Many organizations decide that not every workload deserves full zone-loss protection. Critical customer-facing services get it; internal batch jobs do not. This is a business decision about how much insurance to buy, and capacity planners should present it as a choice with a price, not hide it as a technical detail.

Autoscaler lag

New nodes are not instant. The autoscaler notices pending pods, asks the cloud for a machine, waits for it to boot, joins it to the cluster, pulls images and starts the pods. This can take several minutes. Your headroom must cover the traffic growth during that window. If traffic can double in two minutes and node startup takes five, the spare room in the cluster, not the autoscaler, handles the first few minutes.

flowchart LR
    SPIKE["Traffic spike begins"] --> HPA["Pod autoscaler adds pods"]
    HPA --> ROOM{"Free room on existing nodes?"}
    ROOM -->|"yes"| FAST["Pods start in seconds"]
    ROOM -->|"no"| PEND["Pods wait as Pending"]
    PEND --> CA["Node autoscaler requests machines"]
    CA --> BOOT["Machine boots and joins: minutes"]
    BOOT --> SLOW["Pods start after minutes"]

Ways to hold headroom smartly

• Overprovisioning pods: run low-priority placeholder pods that reserve room. When real pods arrive, the placeholders are pushed out instantly and the real pods take the space, while the autoscaler quietly replaces the room.
• Percentage buffers: keep a fixed fraction, such as ten to twenty percent of allocatable room, free in each critical node group.
• Surge settings: tune rolling update speed so that deployments do not demand more room than you hold.

Let us size a cluster step by step with simple numbers. The figures are invented for teaching, but the method is the real one.

The starting facts

Imagine an online shop whose production cluster, in the busiest week of the last quarter, showed total pod requests of 280 cores of CPU and 840 gibibytes of memory. After right-sizing, the team trusts these numbers. The business expects twenty percent growth over the next six months. The team wants to use nodes with 16 cores and 64 gibibytes, spread over three zones, and wants to survive the loss of one zone.

Step 1: usable room per node

A 16-core node does not give 16 cores to applications. After reservations and eviction margins, suppose allocatable is about 15.2 cores and 59 gibibytes. Per-node helper pods use about 1.2 cores and 3 gibibytes. That leaves roughly 14 cores and 56 gibibytes usable per node.

Step 2: apply growth

CPU: 280 × 1.2 = 336 cores. Memory: 840 × 1.2 = 1,008 gibibytes.

Step 3: divide by usable room

CPU: 336 ÷ 14 = 24 nodes. Memory: 1,008 ÷ 56 = 18 nodes. CPU needs more nodes, so CPU is the binding constraint. Memory is not the limit here.

Step 4: account for packing efficiency

Real pods do not fill nodes perfectly. Assuming eighty-five percent efficiency: 24 ÷ 0.85 = 28.2, rounded up to 29 nodes.

Step 5: protect against losing a zone

With three zones, multiply by 1.5: 29 × 1.5 = 43.5, rounded up to 44 nodes, which is roughly 15 per zone after rounding to equal groups.

flowchart TB
    A["Peak requests: 280 cores, 840 GiB"] --> B["Add growth 20%: 336 cores, 1008 GiB"]
    B --> C["Divide by usable per node: 24 by CPU, 18 by memory"]
    C --> D["Take the larger: 24 nodes"]
    D --> E["Packing efficiency 85%: 29 nodes"]
    E --> F["Zone-loss factor 1.5: 44 nodes"]
    F --> G["Check pod, IP, volume and API limits"]

Reading the answer wisely

Notice what happened: 280 cores of today’s peak turned into 44 nodes, or 704 cores of raw capacity. That is a large jump, and every multiplier is a business choice. If zone-loss protection is only needed for part of the workload, the answer drops sharply. If requests are cut by right-sizing from 280 to 220 cores, the answer drops again. The example shows why measurement quality and policy choices matter more than the arithmetic.

Final checks

• With 44 nodes, can the pod count per node and the address range still hold the expected number of pods?
• Does memory still fit after rebalancing, considering the memory-to-CPU ratio of the machines?
• Does the cloud account permit 44 machines of that type in the region, and are they available in all three zones?
• Can the control plane cope with the extra nodes and pods?

A plan is only as good as its guess about the future. Forecasting turns past patterns and business plans into numbers.

Four ingredients of a forecast

Connecting business numbers to resources

Resource use rarely grows by itself; it follows a business driver such as orders per minute, active users or messages processed. The best forecasts first find the ratio, for example “each hundred orders per second needs about forty cores”, and then forecast the driver, which the business understands far better than engineers do. This also makes the forecast explainable to non-technical colleagues.

A helpful law from queues

A simple principle from queueing theory, often called Little’s Law, says that the number of things being worked on at once equals the arrival rate multiplied by the time each takes. For a web service: concurrent requests equal requests per second multiplied by average seconds per request. If each pod can safely handle a fixed number at once, you can estimate pods from traffic. For example, 2,000 requests per second at 0.1 seconds each means about 200 requests in flight, and if a pod handles 10 comfortably, you need roughly 20 pods at peak, before adding margin.

Ranges, not single numbers

A single forecast number gives false confidence. Good plans offer a low, expected and high case. You then decide what to build for each: typically, you commit to buying for the expected case and ensure that the high case can be reached quickly through flexible capacity.

Lead times decide when to act

Worker nodes are only half the story. The cluster brain has its own limits, and they surprise teams who grow quickly.

What the control plane is

The control plane is the set of programs that run the cluster: the API server that answers every question, the database called etcd that stores all objects, the scheduler that places pods, and controllers that keep reality matching the desired state. Every pod, service, secret and event is a record in that database, and every controller keeps watching it.

Documented scale guidance

Kubernetes documentation describes tested limits for a single cluster, commonly quoted as up to about 5,000 nodes, 150,000 pods, 300,000 containers and 110 pods per node. These are guidance numbers, not hard walls, and they assume reasonable object sizes and request patterns. Managed services often publish their own limits and may differ. Always check your provider’s current documentation.

What actually stresses the brain

• Object count and size: millions of small objects, or a few giant ones, slow down listing and storage.
• Database size: etcd has a storage quota, often two gibibytes by default unless raised, and when it fills the cluster stops accepting writes. Cleaning up old objects and compacting the database are routine duties.
• Request rate: many controllers and tools hitting the API server at once can overload it. Kubernetes includes a feature called API Priority and Fairness that sorts requests into queues so that important traffic is not starved by noisy clients.
• Watch pressure: every watcher holds a connection and receives every change. Tools that list everything every few seconds create heavy load.
• Churn: constant creating and deleting of pods, such as large batch jobs, loads the scheduler and database far more than a stable cluster of the same size.

flowchart LR
    CLIENTS["kubectl, controllers, operators, monitoring"] --> API["API server"]
    NODES["Node agents"] --> API
    API --> ETCD["etcd database"]
    SCHED["Scheduler"] --> API
    CTRL["Controller manager"] --> API
    API -->|"queues by priority"| APF["Priority and fairness"]

Planning for it

Include control plane checks in every sizing exercise. Track API request latency, request rates by client, etcd size and slow-write warnings. For very large or fast-changing clusters, consider splitting into several clusters instead of growing one forever. Separate clusters per environment, per region or per large team reduce the blast radius of a mistake and keep each control plane in a comfortable range. Managed services handle much of the sizing for you, but their limits still apply.

Network limits are quiet, hard to change and capable of stopping growth completely.

Pod addresses

Each pod gets its own IP address from a range reserved for pods. Each node is assigned a slice of that range. If the overall range is small, you cannot add more nodes or pods once it is used up, and changing it later often means rebuilding the cluster. Plan the range for the maximum expected size, not for today. A range that holds around sixty-five thousand addresses sounds large, but with a slice per node and many short-lived pods it can run out faster than expected.

Cloud limits on addresses per machine

In some cloud setups, pods get real addresses from the cloud network, and each machine type can hold only a limited number of network interfaces and addresses. That caps the pods per node regardless of CPU and memory. Small machine types can have very low ceilings. Check the numbers for each machine type you plan to use, and see whether the cluster’s network plug-in offers ways to extend them.

Other shared network limits

Storage has two sides: the temporary space on the node and the durable volumes that outlive pods. Both can limit your cluster.

Temporary node storage

Each node uses its own disk for container images, container writable layers, logs and temporary directories. Pods can request this ephemeral storage. When disk usage crosses thresholds, the node cleans up old images and then evicts pods. Large images, verbose logs and applications that write scratch files all consume this room. Plan disk size as part of node shape, and watch for log growth.

Persistent volumes

Databases and other stateful programs keep data on volumes provided by the cloud or by a storage system. These need their own planning.

• Space: how much data now and how fast it grows. Volumes can often be expanded, but shrinking is hard.
• Speed: measured in operations per second and throughput. A volume that is large enough but too slow still causes slow applications.
• Attachment limits: each node can attach only a limited number of volumes, and the number varies by machine type. Many stateful pods can fill a node’s attachments while CPU sits idle.
• Zone ties: many cloud volumes live in one zone and can only attach to nodes in that zone. Zone-loss planning must account for this.
• Snapshots and backups: these need space and bandwidth as well, often forgotten in budgets.

flowchart TB
    NODE["Node disk"] --> IMG["Container images"]
    NODE --> LOGS["Logs"]
    NODE --> SCRATCH["Pod scratch space"]
    POD["Stateful pod"] -->|"claims"| PVC["Volume claim"]
    PVC --> PV["Persistent volume in storage system"]
    PV -.->|"attach limit per node, zone tie"| NODE

A planning checklist for storage

• Total data today, monthly growth and the date the volume fills.
• Peak operations per second and throughput versus what the volume type provides.
• Maximum attachments per node versus planned stateful pods per node.
• Free disk percentage on nodes, image sizes and log rotation.
• Backup space, restore time and the capacity needed to restore in a hurry.

When many teams share a cluster, capacity becomes a question of fairness as well as arithmetic.

Quotas

A ResourceQuota sets the total amount a namespace may request or use, such as total CPU, total memory, number of pods and number of volume claims. It prevents one team’s mistake from taking the whole cluster. Quotas are also a planning tool: the sum of all quotas shows the maximum demand the cluster might face, and comparing it to real capacity shows how much you have overcommitted.

Priority and preemption

Pods can be assigned a PriorityClass, a number that says how important they are. When the cluster is full, the scheduler can preempt, meaning remove lower-priority pods to make room for higher-priority ones. This turns a shortage into a controlled decision: batch jobs and placeholder pods give way, customer-facing services stay. Without priorities, whichever pod arrives first wins, which is rarely what the business wants.

Queues for batch work

Large batch and machine-learning jobs often need many pods at once. Starting half of them and waiting for the rest wastes resources and can deadlock other jobs. Job-queueing add-ons such as Kueue admit a job only when its whole requirement fits, and share capacity between teams by agreed fair-share rules. This makes batch demand visible and plannable.

Overcommit as a choice

The total of all quotas can exceed real capacity, because teams rarely peak together. This is called overcommitment, and it is the same trick airlines use when they sell more tickets than seats. Done carefully, it raises efficiency. Done blindly, it creates the occasional day when everybody shows up and someone is turned away. Decide on an overcommit ratio deliberately and monitor how often the cluster nears full.

Showback and chargeback

Showing each team what its workloads cost, called showback, or billing them for it, called chargeback, changes behaviour more than any reminder. When teams see that an oversized request costs real money, they start right-sizing without being asked. Cost tools can group spend by namespace, label or team to make this simple.

Capacity is not just a technical amount; it is something you pay for. How you buy it matters as much as how much you buy.

Three ways to pay for machines

Layering the purchases

A common approach is to stack the layers. Buy commitments for the lowest level of demand that you are confident will remain all year. Use on-demand machines for the predictable daily rise above that. Use spot machines for flexible work that can handle interruption. If your baseline is 20 nodes and your daily peak is 44, you commit to something near 20, not 44.

flowchart TB
    PEAK["Peak demand layer: spot and on-demand burst"] --> MID["Daily rise: on-demand and autoscaled nodes"]
    MID --> BASE["Always-on baseline: committed discounts"]

Where waste hides

• Oversized requests: promised but unused room, usually the biggest source.
• Idle environments: development and test clusters running all night and weekend.
• Zombie resources: unused volumes, load balancers and snapshots that nobody owns.
• Poor packing: nodes that are half-empty because of constraints, spread rules or mismatched shapes.
• Cross-zone traffic: data transfer fees between zones that nobody noticed.

The efficiency trade-off

Squeezing out waste always works against safety margin. A cluster at ninety-five percent allocation costs less but has little room for failure. The right balance depends on what an outage costs your business. A shop that loses money every minute of downtime will happily pay for spare room. An internal reporting tool can run lean.

Autoscalers are powerful, and a plan sets their limits. Mistakes here can cost either a lot of money or an outage.

Settings that belong in your plan

• Minimum nodes: the floor, which should cover baseline load and zone spread. Too low and a sudden spike finds an empty house.
• Maximum nodes: the ceiling, which protects budget and quotas. Too low and growth stops silently.
• Allowed machine types: a short, tested list. Mixed types help find stock but complicate packing.
• Scale-down delay: how long a node must be idle before removal. Too short causes thrashing; too long wastes money.
• Pod autoscaler bounds: minimum and maximum replicas per service, tested against the cluster’s real room.

Match the two autoscalers

The pod autoscaler can ask for many more pods than the cluster can hold. The node autoscaler then adds machines, but it takes minutes. If the pod autoscaler’s maximum is higher than the node autoscaler’s maximum can support, the extra pods just sit pending. The totals must agree: the sum of every service’s maximum replicas, times its requests, must fit within the node maximum plus headroom.

Cloud account limits

Cloud accounts carry default quotas on machines per region, particular machine families, IP addresses, disks and load balancers. They differ by provider and can be raised on request, although stock of particular machine types can also run short in a zone during busy periods. Plan to request increases before they are needed, and test in all zones you rely on.

flowchart LR
    HPA["Pod autoscaler max replicas"] --> NEED["Total pods x requests"]
    NEED --> CAPCHK{"Fits within node autoscaler maximum?"}
    CAPCHK -->|"no"| STUCK["Pods stay Pending"]
    CAPCHK -->|"yes"| QUOTA{"Within cloud account quota?"}
    QUOTA -->|"no"| DENIED["Scale-up request refused"]
    QUOTA -->|"yes"| OK["Capacity added"]

A plan on paper is a hypothesis. Tests turn it into knowledge.

Kinds of tests

Test where it is safe, but be realistic

Testing in production gives the most realistic answer but carries risk. A staging cluster that is scaled down proportionally can teach you per-pod behaviour, but it cannot show cluster-wide limits such as control plane load or address ranges. Often teams combine both: per-service load tests in staging, and carefully controlled failure drills or traffic ramps in production during quiet periods.

What to record

• Requests per second a single pod handles within the target latency.
• Resource use per unit of business traffic.
• Time to add nodes and start pods.
• The point at which each limit is first hit, and what failed first.

These numbers feed straight back into the sizing model, closing the loop.

Plans decay. Monitoring tells you when reality has drifted away from the assumptions.

Metrics to watch

A troubleshooting path for pending pods

flowchart TB
    START["Pod stuck in Pending"] --> EV["Read the pod events"]
    EV --> Q1{"Insufficient CPU or memory?"}
    Q1 -->|"yes"| A1["Check requests and node group size; check autoscaler and its limits"]
    Q1 -->|"no"| Q2{"Node selector, taints or affinity mismatch?"}
    Q2 -->|"yes"| A2["Fix rules or add matching nodes"]
    Q2 -->|"no"| Q3{"Volume cannot attach or zone mismatch?"}
    Q3 -->|"yes"| A3["Check attachment limits and zones"]
    Q3 -->|"no"| Q4{"Quota exceeded?"}
    Q4 -->|"yes"| A4["Raise the quota or reduce demand"]
    Q4 -->|"no"| A5["Check IP availability and scheduler health"]

Regular reviews

Hold a short capacity review every month or quarter. Compare forecast and actual, review top consumers, check headroom and limits, list upcoming business events and decide on any purchases or limit increases. A review that ends with named owners and dates is what turns monitoring into planning.

Running out of capacity is a reliability problem, and it can also be a security problem. Attackers and accidents use the same weaknesses.

What happens when each limit is hit

The cascade

Capacity failures love to chain. A node is lost, pods reschedule onto the rest, those nodes run hot, more pods are evicted, retries from clients grow, and the surviving services are overwhelmed. This is called a cascading failure. Headroom, priorities and limits on retries are what stop the chain at the first or second link.

flowchart TB
    LOST["A node or zone is lost"] --> MOVE["Pods reschedule onto remaining nodes"]
    MOVE --> HOT{"Enough spare room?"}
    HOT -->|"yes"| STABLE["Service stays healthy"]
    HOT -->|"no"| PRESS["Nodes run hot, evictions begin"]
    PRESS --> RETRY["Clients retry and add more load"]
    RETRY --> OUT["Wider outage"]
    OUT -.-> LOST

Resource exhaustion as an attack

Anyone who can create workloads, or can send traffic that triggers workloads, can try to use up your capacity. This is a denial-of-service attack from the inside or the outside. A compromised service might create thousands of pods, a malicious tenant might request all the memory, a crypto-mining program might consume spare CPU, and a flood of requests might force expensive autoscaling.

• Quotas and limit ranges cap what each team or namespace can take.
• Admission policies reject oversized or unlabelled workloads.
• Access control limits who can create pods, volumes and load balancers.
• Autoscaler maximums put a ceiling on both capacity and bill, so an attack cannot create unlimited machines.
• Rate limiting and caching at the front door keep floods from becoming internal load.
• Isolation of untrusted work on dedicated node groups keeps noisy or hostile neighbours away from critical services.
• Anomaly alerts on sudden jumps in pod count, CPU and spend reveal abuse quickly.

Capacity planning brings real benefits and real costs. Knowing both helps you decide how much of it your team actually needs.

Key trade-offs

Teams that plan well share some habits, and teams that struggle share others.

Patterns worth copying

Anti-patterns to avoid

A short, practical list you can put in front of a team.

Best practices

Common mistakes

• Treating node CPU and memory as the only limits.
• Believing that autoscaling removes the need for planning.
• Planning from averages and short windows.
• Using raw machine size rather than allocatable room.
• Forgetting the cost of per-node helper pods as the cluster grows.
• Counting headroom that was never tested.
• Locking in commitments before right-sizing.
• Ignoring business events until they arrive.
• Leaving quotas, priorities and defaults unset in shared clusters.
• Forgetting that address ranges and cluster-wide limits are the hardest to change.

The ideas here come from long experience in large-scale operations. The scenarios below are typical patterns, described in general terms.