The most dangerous moments for a healthy application are often the ones its owners chose on purpose.

When people think about outages, they picture disasters: a machine catches fire, a network cable is cut, a data center loses power. Those events do happen, but they are rare. Far more often, outages come from ordinary, planned work. A team upgrades the operating system on every machine. A cloud provider replaces old servers with new ones. An autoscaler decides the cluster is too big and removes a few machines to save money. In each case, somebody pressed a button and Kubernetes obediently emptied a machine.

Emptying a machine means stopping every pod on it. A pod is the smallest unit Kubernetes runs, usually holding one copy of your application. If you run six copies of a web service, and three of them happen to sit on the machine being emptied, then half of your capacity vanishes in one moment. If the other three are already busy, the remaining copies may be overwhelmed, and users start to see slow pages and errors.

The core problem, in one sentence

Kubernetes is very good at moving pods, but by default it does not know how many of them your application can afford to lose at once. You know that. The cluster does not, unless you tell it. The Pod Disruption Budget is how you tell it.

A handful of everyday words are all you need to follow the rest of this guide.

Two more words will appear often. To cordon a node means to mark it as “do not place new pods here”. To drain a node means to cordon it and then politely ask every pod on it to leave, so the node can be worked on. Draining is the everyday event that Pod Disruption Budgets were designed to make safe.

A Pod Disruption Budget is a small Kubernetes object that limits how many pods of one application may be taken down voluntarily at the same time.

The word budget is a helpful clue. A budget is a limit on spending. Here, what you spend is availability. Suppose you run six copies of a service and decide you can tolerate losing one at a time. Then your budget is one disruption. Whenever maintenance wants to stop a pod, Kubernetes checks the budget. If the budget is not used up, the pod may go. If it is used up, the request is refused, and the maintenance tool must wait and try again.

What it is, why it exists, where it is used

What it is: a rule, stored as an object in the cluster, that names a group of pods and states either the minimum number that must stay available, or the maximum number that may be unavailable.

Why it exists: so that the people who own an application can protect it from well-meaning maintenance done by other people or by automated tools.

Where it is used: in nearly every production cluster, especially for web services, queues, databases, and any application where losing many copies at once would hurt. It is also checked by node upgrade tools, cluster autoscalers, and many cloud management systems.

The single most important idea about budgets is knowing which events they control and which they cannot.

Kubernetes divides the ways a pod can disappear into two groups. An involuntary disruption is something nobody chose: a machine crashes, the operating system freezes, a cloud provider takes back a machine without warning, or a pod is stopped because it used more memory than allowed. A voluntary disruption is something a person or program chose to do: draining a node for maintenance, upgrading a cluster, scaling down machines to save cost, or asking Kubernetes to move a pod.

A Pod Disruption Budget can only influence voluntary disruptions. This makes sense. If a machine loses power, no one asks permission first, so there is nothing to refuse. But when a person asks Kubernetes to stop a pod politely, Kubernetes has a moment to check the budget and say “not now”.

flowchart TD
    A["A pod stops running"] --> B{"Did someone choose this?"}
    B -->|No| C["Involuntary disruption"]
    B -->|Yes| D["Voluntary disruption"]
    C --> C1["Node hardware failure"]
    C --> C2["Kernel crash or power loss"]
    C --> C3["Container exceeds memory limit"]
    C --> C4["Cloud provider reclaims machine"]
    D --> D1["Node drain for maintenance"]
    D --> D2["Cluster upgrade"]
    D --> D3["Autoscaler removes a node"]
    D --> D4["Operator evicts a pod"]
    D1 --> E["Budget is checked"]
    D2 --> E
    D3 --> E
    D4 --> E
    C1 --> F["Budget is not consulted"]
    C2 --> F
    C3 --> F
    C4 --> F
    

Several small parts cooperate to enforce a budget. Knowing who does what makes troubleshooting far easier.

A Pod Disruption Budget is only a piece of written intent. It does nothing by itself. Enforcement happens through a chain of components, each with a narrow job.

flowchart LR
    ADM["Administrator or automation"] --> DR["Drain tool or autoscaler"]
    DR -->|"Eviction request"| API["API Server"]
    API --> ADMIT["Eviction admission check"]
    ADMIT --> PDB["PodDisruptionBudget object"]
    DC["Disruption Controller"] -->|"Computes status"| PDB
    DC -->|"Watches"| WL["Deployment, StatefulSet, ReplicaSet"]
    DC -->|"Counts ready pods"| PODS["Pods matched by selector"]
    ADMIT -->|"Allowed"| DEL["Pod is deleted gracefully"]
    ADMIT -->|"Refused with too many requests"| DR
    DEL --> KL["Kubelet on the node"]
    KL --> PODS
    

The cast of characters

A budget has only a few settings. Each one deserves a careful look.

Two ways to say the same thing

You may set minimum available or maximum unavailable, but never both in the same budget. They are two views of one rule. If you run six pods, then “minimum five available” and “maximum one unavailable” describe exactly the same limit. The difference appears when the number of pods changes. A minimum of five with ten pods allows five to be disrupted. A maximum of one with ten pods still allows only one.

The selector: the part beginners get wrong

The selector must match the labels on the pods, not the name of the Deployment. If it matches nothing, the budget protects nothing, and worse, it gives a false feeling of safety. If it matches too much, it may lump unrelated pods together and block maintenance for no reason. Always confirm, after creating a budget, that the number of pods it reports matches the number you expect.

Both forms work. The right choice depends on what you care about most and how your application scales.

flowchart TD
    A["Start: choose a budget style"] --> B{"Does the app need a fixed number of pods to work, such as a quorum?"}
    B -->|Yes| C["Use minimum available with a whole number"]
    B -->|No| D{"Does the number of pods change often, for example with autoscaling?"}
    D -->|Yes| E["Use maximum unavailable, often as a percentage"]
    D -->|No| F{"Do you prefer to think in terms of capacity or of loss?"}
    F -->|Capacity| G["Use minimum available"]
    F -->|Loss| H["Use maximum unavailable"]
    C --> Z["Check the result against replica counts"]
    E --> Z
    G --> Z
    H --> Z
    

Whole numbers and percentages

Either setting can be a whole number or a percentage. A percentage is calculated against the expected number of pods and rounded up. Rounding up is important. With three pods and a maximum unavailable of thirty percent, the exact result is 0.9, and that is rounded up to one. With a minimum available of ninety percent on three pods, the exact result is 2.7, rounded up to three, which means no pod may be disrupted at all. This surprises many people. Small numbers and percentages do not mix well.

The budget is enforced at one specific moment: when a polite request to remove a pod reaches the API server.

Kubernetes has two ways to remove a pod. The blunt way is a direct delete. It simply removes the pod, and no budget is consulted. The polite way is an eviction. An eviction is a special request that says, “Please remove this pod, if it is safe.” When the API server receives an eviction, it looks up every budget that matches the pod and checks whether the removal would break any of them.

If all budgets allow it, the API server deletes the pod using the normal graceful shutdown steps, and reduces the count of allowed disruptions. If any budget refuses, the API server answers with an error meaning “too many requests” and leaves the pod untouched. The caller is expected to wait and try again later.

sequenceDiagram
    participant Admin as Administrator
    participant Drain as Drain Tool
    participant API as API Server
    participant PDB as Budget Status
    participant Node as Node and Kubelet
    participant Dep as Deployment Controller
    Admin->>Drain: Drain node A
    Drain->>Node: Mark node A as unschedulable
    Drain->>API: Evict pod 1
    API->>PDB: Are disruptions allowed
    PDB-->>API: Yes, one allowed
    API->>Node: Delete pod 1 gracefully
    API->>PDB: Reduce allowed disruptions to zero
    Drain->>API: Evict pod 2
    API->>PDB: Are disruptions allowed
    PDB-->>API: No, zero allowed
    API-->>Drain: Refused, try again later
    Dep->>Node: Create replacement pod on node B
    Node-->>PDB: Replacement becomes ready
    PDB-->>PDB: Allowed disruptions rises to one
    Drain->>API: Retry evict pod 2
    API-->>Drain: Allowed
    

Step by step

A budget’s status section tells you everything about its current state in four simple numbers.

Worked examples

Notice the second row. When one pod is already unhealthy, the healthy count equals the required minimum, so no further disruption is permitted. This is the right behavior: if you are already down to your safe limit, planned maintenance must wait. But it also explains a common frustration. A single broken pod can freeze an entire drain until it is repaired.

A budget moves between states as pods leave and return, and the pattern is worth memorizing.

stateDiagram-v2
    [*] --> Created
    Created --> Counting : "Selector matches pods"
    Counting --> RoomAvailable : "Healthy pods exceed desired"
    RoomAvailable --> BudgetSpent : "An eviction is approved"
    BudgetSpent --> Recovering : "Replacement pod is starting"
    Recovering --> RoomAvailable : "Replacement becomes ready"
    Counting --> Blocked : "Healthy equals desired"
    Blocked --> RoomAvailable : "Unhealthy pod recovers"
    RoomAvailable --> [*] : "Budget deleted"
    

Knowing the limits of a tool prevents false confidence, which is more dangerous than no tool at all.

flowchart LR
    subgraph Governed["Protected by the budget"]
        A["Node drain"]
        B["Autoscaler scale-down"]
        C["Managed upgrade of nodes"]
    end
    subgraph NotGoverned["Not protected by the budget"]
        D["Direct pod deletion"]
        E["Rolling update of a Deployment"]
        F["Node crash"]
        G["Out-of-memory stop"]
    end
    A --> H["Eviction request"]
    B --> H
    C --> H
    H --> I["Budget check"]
    D --> J["Pod removed immediately"]
    E --> K["Deployment's own limits apply"]
    F --> J
    G --> J
    

A budget counts ready pods, so the quality of your readiness checks decides how useful your budget is.

The disruption controller does not count pods that merely exist. It counts pods that are ready. A pod is ready when its readiness probe passes, meaning the application says it can serve visitors. This link is powerful, because it means a new replacement pod does not restore the budget until it can truly take traffic. It gives the whole system a safe rhythm: evict one, wait for its replacement to be genuinely ready, then evict the next.

What goes wrong when readiness is poor

Three everyday operations lean on your budget. Each treats it a little differently.

Manual node drains

When an engineer drains a node, the drain tool asks for evictions and keeps retrying refused ones. It waits patiently, and by default it may wait a very long time. If a budget can never be satisfied, the drain simply hangs. Engineers can set a time limit on the drain, and can choose to skip certain kinds of pods, but they should not casually override budgets. Overriding is the path to the outage the budget was created to prevent.

Cluster upgrades

Upgrading a cluster usually means replacing or restarting every node in turn. Each node is drained before it is changed. Managed services from cloud providers automate this. They generally respect budgets for a limited time, and then either stop and report a problem or proceed anyway, depending on the provider and settings. This means an overly tight budget can turn a routine upgrade into a stuck, half-finished one, and a missing budget can turn it into an outage. Read your provider’s documentation on how long it waits and what it does when it gives up.

Autoscalers removing machines

A cluster autoscaler adds machines when pods cannot be placed and removes machines that are underused. Before removing a machine, it checks whether every pod on it can be moved without violating a budget. If a budget would be broken, the machine is left in place. Modern node provisioning tools follow the same principle. So an over-tight budget can keep your cluster larger, and more expensive, than it needs to be.

flowchart TD
    A["Maintenance trigger"] --> B{"Which kind?"}
    B -->|"Manual drain"| C["Drain tool retries refused evictions"]
    B -->|"Cluster upgrade"| D["Provider drains nodes one by one"]
    B -->|"Autoscaler scale-down"| E["Autoscaler simulates the move"]
    C --> F{"Budget allows?"}
    D --> F
    E --> F
    F -->|Yes| G["Pod evicted and replaced elsewhere"]
    F -->|No| H["Wait, retry, or give up after a limit"]
    H --> I["Drain stuck, upgrade delayed, or node kept"]
    G --> J["Node becomes empty and can be changed or removed"]
    

For databases and clustered systems, a budget is not a courtesy. It is a survival rule.

Some applications only work while a majority of their members are alive. This is called a quorum. A quorum is like a vote: a decision counts only if more than half of the members agree. If a three-member group loses two members, the remaining one cannot form a majority and stops accepting changes to avoid making conflicting decisions. Coordination services, distributed databases, and message brokers commonly work this way.

Extra care for stateful workloads

• Prefer a maximum of one unavailable. For data stores, losing one member at a time is usually the safest pace, even if the majority would tolerate more.
• Wait for full recovery, not only readiness. A replacement member may be “ready” while still copying data from its peers. Design the readiness check to report ready only after it has caught up.
• Remember the storage. A replacement pod must reattach to the right disk, which may be in one zone only. If the disk’s zone has no free node, the replacement stays pending and the budget stays spent.
• Do not rely on the budget alone. Also spread members across machines and zones, keep backups, and test recovery.

A budget limits how many pods leave at once. Spreading limits how many are on one machine in the first place. You need both.

Suppose you run six copies with a budget that allows one to be disrupted at a time. If the scheduler places all six on one machine, then draining that machine tries to evict six pods, and the budget allows only one at a time. Each replacement has to be placed on another machine, and the drain becomes slow and awkward. Worse, if that one machine crashes, all six are lost at once, and the budget does nothing.

flowchart LR
    subgraph Z1["Zone A"]
        N1["Node 1: pod 1, pod 2"]
    end
    subgraph Z2["Zone B"]
        N2["Node 2: pod 3, pod 4"]
    end
    subgraph Z3["Zone C"]
        N3["Node 3: pod 5, pod 6"]
    end
    LB["Service"] --> N1
    LB --> N2
    LB --> N3
    B["Budget: at most 1 unavailable"] -.-> N1
    B -.-> N2
    B -.-> N3
    

When the number of pods changes with load, the budget must still make sense at the smallest size.

A horizontal pod autoscaler adds and removes pod copies as demand rises and falls. Suppose it runs between two and twenty copies. At night you may have only two. A budget of “minimum available 80 percent” on two pods rounds up to two, so no disruption is allowed. Any drain touching those pods hangs until morning. During the day, with twenty pods, the same rule allows four disruptions. The rule that suits daytime blocks night-time maintenance.

The table shows that maximum unavailable, especially as a percentage, adapts more gracefully to changing sizes. A fixed count of one is the most cautious and the most predictable. Choose according to how much loss your service can bear at its smallest size, not its average size.

The budget decides whether a pod may leave. Graceful termination decides how well it leaves.

Once an eviction is approved, the pod does not vanish instantly. It goes through the same graceful shutdown as any deleted pod. First, it is marked as terminating and removed from the list of pods receiving traffic. Then the application is sent a polite stop signal and given a grace period, thirty seconds by default, to finish what it is doing. Only after that is the container stopped by force.

If your application ignores the stop signal or takes longer than the grace period to finish, users lose in-flight requests every time a pod is evicted. A budget cannot fix that. It only limits how many such losses happen at once.

A broken pod can block the very maintenance that would fix it. Newer versions of Kubernetes offer a way out.

Imagine a pod stuck in a crash loop on a node that needs an urgent security update. The pod is not ready, so it is not counted as healthy. Under the default behavior, Kubernetes allows evicting an unhealthy pod only if the budget would still be satisfied without it, which usually means only when enough other pods are healthy. If several pods are unhealthy at the same time, the budget may refuse to evict any of them, and a broken pod can hold the node hostage.

To handle this, newer versions of Kubernetes let you choose an unhealthy pod eviction policy. With the default choice, an unhealthy pod is evictable only when the budget is otherwise satisfied. With the “always allow” choice, pods that are not ready may be evicted regardless of the budget, since they were not contributing to availability anyway. This frees drains from being blocked by pods that are already broken.

A budget covers whichever pods its selector matches, and mistakes here cause both silent gaps and hard failures.

Three selector situations

The overlap case deserves emphasis. Each application should have exactly one budget, and each pod should match exactly one. When several teams share a namespace, it is easy for one to create a broad budget that overlaps another team’s. The result is puzzling drain failures that seem to have no cause. A quick review of all budgets in a namespace, comparing selectors, catches this.

Which workloads can be covered

Budgets work best with pods managed by the well-known controllers: Deployments, ReplicaSets, StatefulSets, and their relatives. These controllers tell the disruption controller how many pods to expect, so percentages and maximum-unavailable rules can be computed. For pods managed by custom controllers or created directly, options are more limited, and often only a whole-number minimum available works. Check your version’s documentation when using unusual controllers.

Budgets are light, but their effect on the speed of maintenance is significant.

The disruption controller does very little work. It watches pods and workloads and updates a few numbers. Even in large clusters with thousands of budgets, this cost is small. The real effect of a budget is on the time that maintenance takes, because a budget serializes evictions. If your budget allows one disruption at a time and each replacement takes ninety seconds to become ready, then draining a node holding ten pods of that application takes at least fifteen minutes.

Factors that determine drain speed

• Disruptions allowed. A larger allowance lets more pods leave in parallel.
• Startup time. Slow-starting applications make every recovery slower. Startup probes and faster images help.
• Image pulling. If a replacement lands on a node that lacks the image, it must first download it. Keeping images small and pre-loaded reduces the delay.
• Spare capacity. Replacements can start only where there is room. A tight cluster makes pods wait for space.
• Number of nodes drained together. Upgrading many nodes at once multiplies the load on budgets. Upgrade a few nodes at a time.
• Grace period. Each pod’s shutdown time adds to the total.

A budget is one layer of a reliable system. It works best beside several others.

Design for the sum of losses

A useful way to plan is to add up everything that can be missing at the same time. Suppose you need four pods to handle your peak traffic. During a drain, one pod may be missing because of the budget. At the same time, a machine might fail and take another. A wise design therefore runs at least six pods: four for the load, one for the drain, and one for bad luck. This is often called N plus two planning.

A tool that can block maintenance can also be misused, on purpose or by accident.

You want to know a budget is blocking maintenance before an engineer is staring at a frozen drain at midnight.

Alerts worth setting up

• A budget has had zero allowed disruptions for longer than a chosen period, such as thirty minutes, while all its pods are supposed to be healthy.
• A budget’s expected pods is zero while a matching application exists, which suggests a selector mistake.
• An application with more than one replica has no budget at all. This can be checked by a periodic report.
• A budget’s minimum available equals the total number of replicas, which makes maintenance impossible.
• A node has stayed cordoned for a long time, meaning a drain probably stalled.

Budgets are simple and powerful, and like every safety rule, they have a price.

The main trade-offs

Safety versus speed. A strict budget makes maintenance safer and slower. A loose budget makes it faster and riskier. The right point depends on how costly a brief loss of capacity is for your service.

Predictability versus flexibility. A fixed whole number behaves the same at every size. A percentage adapts to growth but can surprise you at small sizes.

Ownership versus coordination. Letting each team write its own budget gives them control, but requires a shared standard so that budgets do not clash with the platform team’s need to upgrade.

A few tried habits produce budgets that protect without blocking.

These are the habits that turn a helpful budget into a source of pain.

A short checklist prevents almost all budget trouble.

Best practices

Common mistakes

The same idea appears in every kind of organization that runs Kubernetes at scale.

Match the symptom to the likely cause, then work through the steps in order.

If your provider runs the control plane and upgrades your nodes, you should know how it treats your budgets.

Cloud providers that offer managed Kubernetes each have their own way of upgrading nodes. Broadly, they cordon a node, drain it while respecting budgets for some time, and then either move on or stop and report. The details, such as how long they wait and whether they can be told to proceed anyway, differ from one provider to another and change over time. Always read the current documentation of the service you use.

Let us protect a small web service step by step, describing each action in words rather than code.

You run a product catalog service with six copies across three nodes. It needs at least four copies to handle normal evening traffic. You want to keep it safe during a node upgrade.

What this walkthrough teaches

The order matters. First decide what loss you can tolerate, then write the rule, then check that the numbers agree with your intention, and finally rehearse. A budget you have never watched working is only a hope.

Practice makes the arithmetic automatic. Here are three services, each needing a different approach.

Reasoning about drain time

Consider the coordination cluster. If each replacement takes four minutes to become ready, upgrading five nodes, each holding one member, takes about twenty minutes at the very least, and only if the members are spread one per node. That is a good trade for a service where losing quorum would stop all writes.

Understanding the problem that led to the feature makes its design feel natural.

In the early days of Kubernetes, clusters were smaller and maintenance was mostly manual. An engineer would drain a node, watch the pods move, and go on to the next node. If something looked risky, the engineer simply waited. The knowledge of what was safe lived in people’s heads.

As clusters grew to hundreds of nodes, and as upgrades became automated, that approach stopped working. Machines were now drained by scripts, by cloud services, and by autoscalers, none of which knew anything about the applications running on them. The application owners, on the other hand, had no way of telling those tools what was acceptable. The gap between “the people who know the application” and “the tools that move the pods” needed a formal language. The Pod Disruption Budget became that language.

The feature began as an early, experimental addition and matured over several releases into a stable, widely used part of Kubernetes. Along the way it gained better handling of percentages, support for more kinds of workloads, and, in newer versions, the option to decide what to do with pods that are already unhealthy. Its basic idea has hardly changed: a simple, written promise about how much can be missing at once.

Three other ways pods can be removed behave differently from evictions requested by maintenance tools. Knowing them prevents confusion.

A safety rule proves its value only when it is exercised. Rehearsal turns a theory into a fact.

Many teams write a budget once and never see it work. Months later, during an upgrade, the first real test is also the first real failure. A little deliberate practice prevents that. The goal is not to break things but to learn, in a safe place, exactly how your application behaves when machines are emptied.

A simple rehearsal plan

In a large organization, hundreds of budgets are written by dozens of teams. A little governance keeps them consistent.

When only one team uses a cluster, budgets are a private matter. In a shared platform, they become a shared agreement. The platform team needs to patch machines on a schedule, while each application team needs its service to stay up. Both goals are legitimate, and they collide at the budget.

A simple set of shared rules

• Every multi-replica service has one budget. Enforce this through templates and periodic reports.
• Budgets must allow at least one disruption at normal size. A policy rule can reject budgets that make maintenance impossible.
• Minimum replica counts must fit the budget. If a budget needs three healthy pods, the service should run at least four.
• Stateful systems are reviewed by hand. Numbers for databases and brokers come from their own recovery rules, not from a general template.
• Exceptions are recorded. If a team needs a stricter rule, they document why and set a review date.
• The platform publishes its maintenance calendar. Teams know when nodes will be drained and can prepare.

Quick answers to the questions that come up most often.