Most teams fear upgrades because they are rare, large and poorly rehearsed. The cure is to make them small, frequent and boring.

The fear is understandable

A cluster runs everything: the website, the payments, the background jobs. If an upgrade goes wrong, many things break at once. People remember the one terrible upgrade night and decide to postpone the next one. The postponement makes the next upgrade bigger, because more versions have piled up, and a bigger upgrade is scarier. This loop is how teams end up running a cluster that is years old and nobody dares touch.

Why you cannot simply stand still

What “zero downtime” really means

Zero downtime does not mean that nothing changes. Machines are stopped, Pods are moved, and components restart. It means that users never see an error or a long pause. To reach that, three things must all be true:

• The cluster itself keeps working, so existing applications continue to run and be reachable while the control plane is upgraded.
• Each application can be moved without dropping requests, because it has several copies, shuts down politely and starts quickly.
• The upgrade is observed and reversible, so that a problem is spotted early and the damage stays small.

A helpful distinction

The control plane being briefly unreachable is not the same as the applications being down. Running Pods keep serving traffic even if the API server restarts, because the nodes and the network keep working by themselves. What you lose temporarily is the ability to change things. That is why a well-built upgrade can be invisible to users even when the control plane is not perfectly available the whole time.

Kubernetes ships on a steady calendar. Knowing the rhythm tells you how often you must upgrade and how much time you have.

Version numbers

A Kubernetes version has three parts: major, minor and patch. For example, in “1.31.4” the major version is 1, the minor is 31 and the patch is 4. The major number has stayed at 1 for years. The minor number changes with each release that brings new features and sometimes removes old ones. The patch number changes with bug and security fixes only, and patch upgrades are low risk.

The cadence

The project publishes about three minor releases a year, roughly one every four months. Each new minor version receives patch releases for around a year, and then it is declared end of life. In practice, the community maintains the three most recent minor versions at any time. Exact dates move a little from release to release, so check the official release page for the current schedule.

What this means for planning

• If you upgrade every minor release, you do about three small upgrades a year.
• If you upgrade every second release, you do about two per year, but you hop across more changes at once. You still pass through each minor version in between.
• If you fall more than a year behind, you leave the community support window and begin a catch-up marathon of several sequential upgrades.

Cloud provider calendars

Managed services such as Amazon EKS, Google GKE and Azure AKS publish their own calendars. They usually offer a version for a while after the community release, and then support it for a fixed period. Some offer extended support for a fee. When a version leaves support, the provider may upgrade your control plane automatically. A cluster that you do not plan for will eventually be upgraded on the provider’s schedule, which is the worst time to find problems.

During an upgrade, different parts of the cluster run different versions for a while. Kubernetes defines exactly how far apart they may be.

Why skew exists

You cannot upgrade a whole cluster in one instant. The control plane is replaced first, then the nodes, one by one. For hours, or even days, parts of the cluster speak different versions. Kubernetes makes this safe by publishing a version skew policy: a set of rules about which components may lag behind which.

The rules in plain terms

The golden rule

The API server must always be the newest component. Nothing talks to a server that is older than itself, because it might ask for features that the server does not know. That single rule explains the whole upgrade order: control plane first, then workers.

Why the kubelet allowance matters

Because kubelets may lag, you can upgrade the control plane quickly and then take your time with the nodes. For a large fleet, this gap is a relief. It also lets you take a break in the middle of an upgrade if something looks wrong. Do not abuse it, though. Running nodes far behind for months makes the next upgrade harder, and some features only work when nodes are current. Check the exact numbers in the official policy for your version, since the allowed gap has changed over time.

Skew in add-ons

The skew policy covers the core components. Add-ons such as the network plugin, the storage drivers, ingress controllers and operators have their own compatibility tables. These are often more restrictive than the core rules and are a frequent source of surprises. Always read the compatibility table of each add-on before upgrading Kubernetes.

“Upgrading the cluster” hides many separate parts. Listing them makes the plan concrete.

flowchart TB
  subgraph CP["Control plane"]
    API["API server"]
    ETCD[("etcd - the cluster's memory")]
    SCH["Scheduler"]
    CM["Controller manager"]
    CCM["Cloud controller manager"]
  end
  subgraph NODES["Worker nodes"]
    KL["Kubelet"]
    KP["Kube-proxy"]
    RT["Container runtime"]
    OS["Operating system and kernel"]
  end
  subgraph ADD["Add-ons"]
    CNI["Network plugin"]
    DNS["CoreDNS"]
    CSI["Storage drivers"]
    ING["Ingress controller"]
    MET["Metrics and monitoring agents"]
  end
  subgraph APPS["Your workloads"]
    P1["Applications"]
    P2["Operators and custom resources"]
  end
  API --> ETCD
  SCH --> API
  CM --> API
  KL --> API
  KP --> API
  ADD --> API
  APPS --> NODES

Reading the diagram

The four boxes are four layers. The control plane holds the decision-makers and the memory. The worker nodes run containers. The add-ons are helpers installed on top, often as ordinary Pods. Your own workloads sit on the nodes. Arrows show who talks to whom: nearly everything talks to the API server, which is the front door of the cluster, and only the API server talks to etcd.

What each layer means for an upgrade

Three kinds of “upgrade” people mix up

• Control plane upgrade: new versions of the decision-making components.
• Node upgrade: new kubelet and often a new machine image, which usually replaces or restarts the machines.
• Add-on upgrade: new versions of helpers, which are managed separately and sometimes by a different team.

A complete plan covers all three, and a good plan assigns an owner to each.

The order of steps is not a matter of taste. Each step prepares for the next, and each has a way back.

flowchart LR
  A["1. Read release notes and deprecations"] --> B["2. Fix removed APIs and check add-on compatibility"]
  B --> C["3. Back up etcd and key resources"]
  C --> D["4. Upgrade control plane"]
  D --> E["5. Verify control plane health"]
  E --> F["6. Upgrade add-ons that require it"]
  F --> G["7. Upgrade nodes in small batches"]
  G --> H["8. Verify workloads and metrics"]
  H --> I["9. Upgrade remaining add-ons and clean up"]

Walking through the order

Why control plane before nodes

Because the API server must be at least as new as everything that talks to it, workers can never go first. And because nodes may lag, there is no rush to finish them right after the control plane. You have the freedom to pause between steps.

Why small batches

If you upgrade ten percent of the nodes and something looks wrong, ninety percent of the fleet is still healthy and you can stop. If you upgrade everything at once, a hidden problem hits everywhere together. Small batches turn a possible disaster into a minor, containable incident.

The control plane is the cluster’s brain. Upgrading it safely depends mostly on having more than one of everything.

High availability is the foundation

A highly available control plane runs at least three copies of the API server and the memory store, spread over different machines or zones. A load balancer sits in front of the API servers and sends requests only to healthy ones. With this setup, you can upgrade one instance at a time: the others keep answering while one is replaced. Without it, a single control plane machine means a short but real period when nobody can change the cluster.

The one-at-a-time dance

1. Remove one API server from the load balancer, or let health checks do it when it restarts.
2. Upgrade it and wait until it reports healthy.
3. Return it to service and check logs and metrics.
4. Move to the next instance.

During this time, two versions of the API server run together. That is allowed by the skew policy, as long as they differ by at most one minor version.

The other control plane components

The scheduler and the controller manager use a leader election system. Several copies run, but only one is active at a time. When the active one restarts for the upgrade, another takes over within seconds. During these seconds, no new Pods are placed and no controllers react, but running Pods are unaffected. Their short pause is generally invisible to users.

What users may notice, and what they will not

Clients that talk to the API server

Your own operators, autoscalers, CI systems and tools call the API server constantly. They should retry on errors, and reconnect when a watch connection drops. An upgrade is a good test of whether they do. Controllers written with standard libraries handle this well. Hand-written scripts often do not.

Managed control planes

On managed services, the provider runs and upgrades the control plane for you, usually with high availability built in. You press a button or update a setting, and wait. You still own the planning: choosing the time, preparing the workloads and checking compatibility. And you cannot easily roll back a managed control plane, so the preparation matters even more.

Everything Kubernetes knows is stored in etcd. If it is lost and you have no backup, the cluster forgets what it was running.

What etcd holds

etcd is a small, reliable database. It stores every object in the cluster: Deployments, Services, Secrets, configuration, custom resources and more. It does not hold your application data, such as customer records in a database, but it holds the instructions that tell Kubernetes how to run everything. Lose it, and the running Pods continue for a while, but the cluster can no longer manage them.

Why it matters during upgrades

Some upgrades change the version of etcd, or the format in which objects are stored. A new Kubernetes version may also migrate stored objects to newer API versions. If something fails halfway, the stored data is what you must restore. That is why a verified backup is the first safety step.

The backup rules

• Take a snapshot right before the upgrade, not last week. A recent one loses the least.
• Store it outside the cluster, in a separate account or location, so that it survives a cluster failure.
• Protect it, because it contains secrets in a readable form unless encryption at rest is turned on.
• Test a restore at least once, in a scratch environment. A backup that has never been restored is a hope, not a plan.
• Record the version of etcd and Kubernetes the snapshot belongs to.

Cluster health and quorum

etcd uses a voting system called quorum. With three members, two must agree for changes to be accepted, so one member can fail safely. With five members, two can fail. During an upgrade of a three-member etcd, upgrading one member at a time keeps quorum, but taking two down together stops the cluster from writing. Always upgrade etcd members one by one, and wait for each to rejoin and sync before touching the next.

Managed services and backups

On managed services you cannot reach etcd directly, and the provider handles its safety. You should still keep your own copies of the important resources, usually by storing manifests in version control or using a backup tool that exports cluster objects, so that you can rebuild if needed. Note that stored cloud volumes and databases need their own backups as well.

Nodes are the machines that carry your applications. How you replace them decides how smooth the upgrade feels.

Strategy one: in-place

You take an existing node out of service, upgrade the software on it, and bring it back. This keeps the same machine, the same address and local disks. It is the traditional method for self-managed clusters. It is simple, but the machine’s history accumulates over time, and a failed upgrade leaves a half-changed machine that is hard to reason about.

Strategy two: rolling replacement

You create a new node from a fresh image with the new version, join it to the cluster, move workloads over, and delete the old node. Each machine is new and clean, and the old one is thrown away. This is the usual method on managed services and with auto scaling groups. It needs a little extra capacity during the process, often called “surge.”

Strategy three: blue/green node pools

You create a whole new group of nodes at the new version, called the green pool, while the old blue pool continues to run. Then you move workloads across in controlled steps, watching the results. If something is wrong, you can send workloads back to the blue pool, which still exists. When satisfied, you delete the blue pool. This costs the most in temporary capacity, but it gives the best way back.

Surge and unavailable settings

Managed node groups and similar tools let you set two numbers. The surge says how many extra nodes may be created above the normal count while upgrading. The max unavailable says how many nodes may be out of service at once. A larger surge makes the upgrade faster and safer for workloads, because capacity is never reduced. A larger unavailable count makes it faster but cheaper, at more risk. For production, a common safe default is a surge of at least one node, with no more than one node unavailable per zone.

Immutable nodes

The more your nodes are treated as disposable, the easier upgrades become. If nodes are built from an image and nobody logs in to change them by hand, then replacing a node is a routine event, and every node of a given pool is identical. Teams that keep hand-tuned “pet” machines find upgrades far harder.

Before a node is touched, its Pods must be moved away. Kubernetes has a gentle, well-defined way to do that.

Cordon: no new guests

Cordoning marks a node as unschedulable. Pods already there keep running, but the scheduler will not place any new Pods on it. It is like putting a “no vacancy” sign on a hotel: current guests stay, no new ones are admitted.

Drain: please move out

Draining cordons the node and then asks every Pod on it to leave. It does so using the eviction process, not by simply deleting Pods. The difference is crucial: eviction checks the rules you have set for each application, especially the PodDisruptionBudget, and refuses to remove a Pod if doing so would break the budget. The drain tool then waits and retries until the Pod can leave safely.

What drain does and does not move

• Moves: ordinary Pods managed by Deployments, StatefulSets, Jobs and similar controllers. The controller recreates them elsewhere.
• Skips: Pods managed by DaemonSets, since they are meant to run on every node. Drain is usually told to ignore them.
• Refuses by default: Pods that are not managed by any controller, since they would be lost forever, and Pods using local temporary storage, unless you allow it explicitly.
• Waits for: a Pod’s termination grace period, so that it can finish its work.

The eviction journey of a single Pod

sequenceDiagram
  autonumber
  participant OP as Operator or upgrade tool
  participant API as API server
  participant PDB as Disruption budget check
  participant KL as Kubelet on the node
  participant POD as Pod
  participant CTL as Deployment controller
  participant SCH as Scheduler
  OP->>API: Cordon node
  OP->>API: Request eviction of Pod
  API->>PDB: Would this break the budget?
  PDB-->>API: Allowed
  API->>POD: Mark for deletion
  KL->>POD: Run preStop step, send termination signal
  POD-->>KL: Finishes in-flight work and exits
  CTL->>API: Create replacement Pod
  SCH->>API: Place it on another node
  Note over OP,SCH: If the budget forbids, the request is refused and retried later

Reading the diagram

The tool first asks the API server to evict a Pod. The server consults the disruption budget; if the answer is no, the request is refused and the tool tries again later. If yes, the Pod is marked for deletion and the kubelet on the node tells it to stop politely. Meanwhile, the controller notices a missing replica and creates a new one, and the scheduler puts it on another node. The ideal is that the replacement is ready before the original has finished leaving.

Timeouts and stuck drains

A drain can hang when a budget blocks eviction forever, for example when an application has a single replica and a budget that allows zero disruptions, or when a Pod refuses to terminate. Tools let you set a timeout. When a drain gets stuck, investigate why, instead of forcing it. Forcing a delete bypasses the safety checks and is the usual cause of user-visible errors during upgrades.

A PodDisruptionBudget is the way an application tells the cluster how many of its copies may be taken away at once during planned maintenance.

What it says

A PodDisruptionBudget, usually shortened to PDB, selects a group of Pods by label and states one of two rules: “at least this many must stay available,” or “at most this many may be unavailable.” The number can be a count or a percentage. When a drain tries to evict a Pod, the cluster counts how many Pods of the group are healthy and refuses the eviction if it would break the rule.

Choosing the right number

Two ways to get it wrong

The budget covers voluntary disruptions only

PDBs protect against planned actions: drains, upgrades and similar requests that go through the eviction process. They cannot protect against a node crashing or a machine being removed without warning. For those, you need more replicas and good spreading, which later chapters cover.

Budgets and health

A Pod counts as available only when it is ready. If an application is already struggling, with several Pods not ready, a budget may refuse any further eviction. This is good behavior: it stops the upgrade from making a bad situation worse. It also means that a broken application can block an upgrade until it is fixed.

An audit worth doing before every upgrade

• Find all workloads with only one replica, and decide whether to scale them up for the upgrade.
• Find all budgets that allow zero disruptions.
• Find all workloads that are important but have no budget.
• Check that each budget selects the right Pods.

Moving a Pod is easy. Moving it without losing a single customer request needs a precise choreography.

The two things that happen at once

When a Pod is told to terminate, two processes begin at the same time, and they do not wait for each other. First, the Pod is marked as terminating and removed from the lists of endpoints that Services use to send traffic. Second, the kubelet tells the container to shut down. The trouble is that removal from the endpoint lists takes a moment to reach every proxy and load balancer. If the container exits too quickly, some requests are still sent to it and fail.

sequenceDiagram
  autonumber
  participant API as API server
  participant EP as Endpoint update
  participant LB as Proxies and load balancers
  participant KL as Kubelet
  participant APP as Application container
  API->>KL: Pod marked as terminating
  API->>EP: Remove Pod from service endpoints
  par Traffic side
    EP->>LB: New endpoint list spreads
    Note over LB: Takes a few seconds to reach everyone
  and Container side
    KL->>APP: preStop pause begins
    Note over APP: Keeps serving while traffic drains away
  end
  KL->>APP: Send termination signal
  APP->>APP: Finish in-flight requests, close connections
  APP-->>KL: Exits cleanly
  Note over KL,APP: If the grace period ends first, a forced kill follows

The ingredients of a clean exit

Reading the diagram

Step through the picture from the top. The two branches in the middle are the heart of the matter. On the left, the news that the Pod is leaving spreads to proxies and load balancers. On the right, the container is told to wait a little. The termination signal is sent only after that pause, by which time most of the traffic has already gone elsewhere. The final note shows the safety limit: if the application takes longer than the grace period, it is killed anyway, and any request still running is lost.

Slow-starting replacements

Graceful exit is only half of the story. The replacement Pod must start and become ready quickly, or capacity will sag. Long start-up times, large images, heavy cache warming or slow health checks all lengthen the period when you are short of capacity. Startup probes can protect slow starters from being killed too early, and pre-pulled images shorten start time.

Test it before the upgrade

You can test graceful shutdown in staging by deleting a Pod during a steady load test and watching for errors. If a single deletion causes failed requests, a full upgrade will cause many. This one test finds a large share of upgrade problems.

An application with ten replicas is not safe if all ten live on one machine. Spreading makes sure that losing any one place costs only a small slice.

The tools for spreading

• Topology spread constraints tell the scheduler to distribute Pods evenly across zones, nodes or other groupings, with a limit on how uneven the spread may be.
• Pod anti-affinity says that Pods of a certain kind should not share a node or zone with each other.
• Node selectors and affinity place Pods on certain kinds of nodes, which also affects what happens when those nodes are upgraded.

Why spreading matters for upgrades

During a node upgrade, the machines go out of service one at a time, or a few at a time. If an application has three replicas, and two of them sit on the same node, draining that node removes two thirds of the capacity at once. A PDB would block the drain, which is safe but slow. Good spreading means every drain removes at most one replica of any application, and the upgrade flows steadily.

A small worked picture

Zones

Spread across at least three zones for important services, so that a zone-wide problem or a zone-by-zone upgrade never removes more than a third. Remember that spread rules apply at scheduling time. After nodes are replaced, Pods may drift into an uneven layout. A descheduler tool, or simply the natural churn from regular upgrades, helps re-balance them.

Do not over-constrain

Strict rules, such as “never two Pods on one node,” can leave Pods unschedulable when nodes are few, especially during a node upgrade when capacity is already reduced. Use “preferred” rules for secondary goals, and strict rules only for the real requirements. Check that the cluster has room for the spread you demand.

An upgrade temporarily removes machines. If the rest cannot carry the load, users feel it, no matter how clever your drain is.

The simple arithmetic

If you drain one node out of ten, you lose ten percent of capacity. If you drain three at once, you lose thirty percent. If the cluster normally runs at 80 percent of capacity, losing 30 percent takes it past 100 percent, and Pods will stay pending or compete for resources. Your cluster’s headroom, which is the spare capacity above normal use, decides how fast you can safely go.

Ways to create room

Autoscalers during upgrades

Cluster autoscalers add and remove nodes in response to pending Pods and idle capacity. During an upgrade they can help, by adding capacity when drained Pods have nowhere to go. They can also cause confusion, by removing a node you just added because it looks idle, or by creating nodes with the old version. Decide before starting whether to pause the autoscaler, or to make sure it uses the new node image. Horizontal Pod autoscalers, which scale replica counts by load, may react to the brief rise in load as Pods move; ensure their limits allow it.

Resource requests matter

The scheduler places Pods by their declared requests, not by actual usage. If requests are wildly too large, nodes look full when they are mostly empty, and drained Pods cannot find space. If requests are too small, nodes are overpacked and Pods squeeze each other when they are all moved together. Realistic requests are a quiet prerequisite for smooth upgrades.

Quota, limits and cloud limits

Surge nodes need room in your cloud account: enough instance quota, IP addresses in the subnets and capacity in the chosen instance type. Many upgrades stall because the account could not create the extra machines. Check the quotas, and have a fallback instance type, before the maintenance window.

Applications with data attached are harder to move than those without. The upgrade must respect where the data lives.

How stateful sets move

A StatefulSet gives each Pod a stable identity and its own storage volume. When a Pod is drained, it is recreated with the same name and reconnects to the same volume. The default update behavior replaces Pods one at a time, in reverse order, waiting for each to become ready. This is the natural rhythm of a safe upgrade, but it also means a slow one, particularly if each Pod takes minutes to start.

Volumes and where they live

• Zone-bound volumes: Many cloud disks can only attach to machines in the same zone. A Pod with such a volume can only move to another node in the same zone. If that zone has no spare node, the Pod stays pending.
• Attach and detach time: Moving a volume from one node to another takes time, sometimes a minute or more, and occasionally gets stuck.
• Local disks: Volumes on the node’s own disk disappear when the node is replaced. Replicated databases can rebuild from peers, but that takes time and load.
• Shared file systems: Network file storage moves easily, but performance and locking behavior vary.

Databases and replication

For replicated databases, the golden rule is to move one replica at a time and wait for it to catch up before touching the next. Moving a leader requires a failover, which causes a short pause for writes. Plan for that by upgrading followers first, and then switching leadership deliberately at a calm moment. Use budgets that prevent more than one member from leaving at once, and probes that report a replica as ready only when it is in sync.

Storage drivers and snapshots

Storage is connected to Kubernetes through drivers that run as Pods and DaemonSets. They must be compatible with the new version, and they should be upgraded in the order their documentation says. Before upgrading, take volume snapshots or database backups of important data. A Kubernetes upgrade should not harm data, but the combination of an upgrade and an unfortunate detach can.

Operators

Many stateful systems are run by operators, which are controllers with specialized knowledge, such as how to fail over a database cluster. Their behavior during node drains varies. Some handle them intelligently, while others need special annotations or a maintenance mode. Read the operator’s upgrade notes, and test a drain in staging.

On Amazon EKS, Google GKE and Azure AKS, much of the heavy lifting is done for you, but the responsibility for a smooth experience still has a split.

Shared responsibility

Common patterns across providers

• Control plane first, node pools next. The provider upgrades the control plane on request, then you upgrade each node group separately.
• Surge-based node upgrades. Managed node groups create new nodes, drain old ones and respect disruption budgets, up to a time limit. If a drain exceeds the limit, some services may force the upgrade to continue or mark it failed, so know your provider’s behavior.
• Auto-upgrade channels. Some providers offer release channels or automatic upgrades. These are convenient but can surprise you; use maintenance windows and exclusions to control the timing.
• Extended support. Some providers keep older versions alive for an additional fee, which can buy time but not safety forever.
• Limited rollback. You generally cannot downgrade a managed control plane. Plan on rolling forward or switching to a second cluster.

Provider-specific details to check

Each provider has its own particulars, such as how the add-ons for networking and DNS are versioned, how long a node drain is allowed to take, what the default surge setting is, and whether nodes may be upgraded straight to the control plane version. These details change over time. Read the current documentation of your provider before every minor upgrade, instead of relying on habit.

Serverless and autopilot modes

Some offerings run nodes entirely for you, so there are no node pools to upgrade. The provider replaces nodes on its own schedule, which makes your workloads’ tolerance for moves even more important. You control the timing mostly through budgets and maintenance windows.

If you run Kubernetes on your own machines, you perform what the managed services do for you. The sequence is well established.

The shape of the process with common tooling

Installers such as kubeadm provide a guided upgrade path. In outline, the process runs like this:

Things you must handle yourself

• Load balancing of the API server: your own load balancer must remove a control plane node while it restarts.
• Certificates: control plane certificates are renewed during upgrades, and the upgrade is a good time to check their expiry dates.
• Package repositories: Kubernetes packages are published in repositories per minor version, so the repository setting must be changed before each minor step.
• Container runtime: the runtime may need its own upgrade, and it should be compatible with the kubelet version.
• Operating system: kernel and system updates often arrive with node maintenance; decide whether to combine them with the Kubernetes upgrade or keep them separate.
• Configuration drift: check that kubelet settings are the same across nodes, and that custom flags are still valid in the new version.

Automation tools for self-managed fleets

Cluster API treats clusters as declared objects, so changing the version in a definition triggers a rolling replacement of control plane and worker machines. Other tools, such as configuration management systems and image-based approaches, let you rebuild nodes from a fresh image. The more of the process is declared in files and automated, the less one-off human work remains, and the lower the risk of a typing mistake at midnight.

Rehearsal is worth more here

With no provider safety net, rehearse the entire procedure on a copy of the cluster, including the control plane load balancer and certificate behavior. Time each step, and write down surprises. The runbook that comes out of a rehearsal is the best insurance you can buy.

The network plugin, DNS, storage drivers and ingress controllers are Pods like any other, but a mistake with them affects everything at once.

The big add-ons and their risk

Rules for add-on upgrades

• Read compatibility tables first. Each add-on lists which Kubernetes versions it supports. Check both the current and target versions.
• Know the order. Some add-ons must be upgraded before the control plane, others after. The vendor documentation says which.
• Run more than one copy. DNS, ingress and similar critical add-ons should have several replicas, spread across nodes, with budgets and graceful shutdown.
• Roll out slowly. DaemonSet add-ons, which run on every node, should update with a limit on how many nodes are changed at once.
• Watch the basics. After each add-on change, test name resolution, Pod-to-Pod connectivity, an external request and a volume mount.
• Keep a way back. Know the previous version and how to return to it quickly.

A smoke test for the platform

Create a small set of canary checks that run continuously: one that resolves a name, one that calls a service in another namespace, one that reaches an outside address, one that writes to a volume, and one that sends a request through the ingress. After each upgrade step, these tell you in seconds whether the platform basics are intact, long before any customer reports a problem.

The most common reason an upgrade breaks applications is not the machines. It is a Kubernetes feature that your manifests still use but the new version no longer offers.

How Kubernetes retires an API

Kubernetes objects have versions in their names, such as a beta version and a stable version. When a better version arrives, the older one is marked deprecated, which means “still working, but please move.” After a promised period, spanning several releases, the old version is removed. After removal, any request that uses it fails. Objects already stored in the cluster are converted, but your deployment files, charts and tools that still name the old version will break.

Where old APIs hide

Finding them before they find you

• Read the deprecation guide published for each release. It lists exactly which versions are removed.
• Ask the API server about use of deprecated versions. The server records metrics and warnings every time a deprecated API is called, which reveal who is still using them.
• Scan your repositories and charts with a static checking tool that flags removed versions for a chosen target release.
• Scan live objects in the cluster, including those created by tools and operators, using a similar checker.
• Watch the warnings printed by your tools. Deprecation warnings appear on the command line and in logs for months before removal. Teams that read them are rarely surprised.

Fixing them

For each finding, update the manifest or template to the newer version, check whether the shape of the object changed between versions, apply it to a test cluster, and confirm that it works. Do this at least one release before the upgrade, so that the change itself is not mixed with the upgrade. Third-party charts and operators may require their own upgrade, which in turn has its own compatibility needs.

Changes that are not API versions

Besides removed versions, releases sometimes remove whole features, change default behavior or tighten security settings. Examples across past releases include removal of old volume plugin types, changes to default security profiles and removal of built-in cloud integrations in favor of external ones. These are in the release notes, in sections about urgent upgrade notes and deprecations. Read those sections line by line.

A good checklist turns worry into a list of small questions with yes or no answers.

The checklist, by theme

Health first

Never start an upgrade on a cluster that is already unhealthy. If nodes are flapping, if Pods are crashing or a controller is stuck, the upgrade will make a confusing situation worse and hide the cause. Fix the baseline problems first, even if it means a delay. An upgrade should begin from green.

Go and no-go criteria

Write down, before the window opens, what would make you stop. For example: “if error rate doubles for more than three minutes after any batch, pause,” or “if two nodes fail to rejoin within ten minutes, stop and investigate.” Decisions made calmly in advance are better than those made at 2 a.m. under stress.

The first time you do something should never be on production. A rehearsal environment turns surprises into lessons.

What makes a good rehearsal cluster

It should resemble production in the ways that matter: the same Kubernetes version, the same add-ons and versions, similar node types and counts per zone, similar policies and admission rules, and representative workloads. It need not be as large, but it should have more than one node per zone and several replicas of key services, because the interesting behavior of upgrades is how things move, which a single-node playground cannot show.

The rehearsal plan

Upgrade tests as automation

Teams with mature practices create a fresh cluster on demand at the old version, deploy a standard set of workloads, upgrade, and run a battery of checks, all automatically, every time a new Kubernetes version is released. This gives early warning about incompatibilities, weeks before the production upgrade is due.

Canary clusters

If you have many clusters, upgrade the least critical first and let it run for a week or two while you observe. Then proceed to more important ones in waves. The first cluster plays the role of the coal-mine canary: if something is wrong, it shows there, before it can hurt the important ones.

Limits of rehearsal

No test cluster has exactly the same traffic, data size or odd corners as production. Rehearsal finds most problems but not all. Therefore even a well-rehearsed production upgrade is done in small steps with active watching.

The extra machinery you have added to your cluster is often the part that breaks during an upgrade. Three things deserve special attention.

Admission webhooks

An admission webhook is a service that the API server calls whenever someone creates or changes an object, to approve, reject or modify it. Policy engines, certificate managers, and service meshes use them. The risks during upgrades are:

• A webhook that is down can block everything. If its failure policy is “fail,” then when the webhook Pods are being moved, no new Pods can be created anywhere, including the replacements for the Pods being drained. This can freeze an upgrade.
• Webhooks that call the API they guard. A circular dependency can make recovery impossible.
• Webhooks that use removed API versions in their registration, and so stop working after the upgrade.
• Short timeouts that fail when the system is busy during node moves.

Good practice includes running several replicas of every webhook, spreading them, giving them disruption budgets, excluding system namespaces from their scope, and choosing the failure policy thoughtfully: “ignore” for convenience checks, “fail” only for essential security rules and then only with high availability.

Custom resource definitions

CRDs extend Kubernetes with new object types. They bring their own versions and conversion rules. When Kubernetes is upgraded, CRDs themselves keep working, but their conversion webhooks must be reachable, and any CRD that used a now-removed definition API version needs updating. Before upgrading, list all CRDs, note which controller owns each and check that controller’s compatibility.

Operators and controllers

Operators are programs that watch objects and act on them. During a control plane upgrade they reconnect and resync, so they must handle dropped connections. During node drains, they are moved like any other Pods, but their own work, such as failing over a database, may be interrupted. Questions to ask for each operator:

The deadlock to avoid

One notorious situation: a webhook Pod that must be running for new Pods to be admitted is itself evicted and cannot restart, because its own creation requires the webhook. The upgrade freezes with the webhook missing. Prevent it by exempting the webhook’s own namespace from its checks and keeping at least one replica on a node that is upgraded last.

An upgrade without monitoring is a journey with the lights off. What you watch decides how quickly you notice trouble.

Three layers of signals

The upgrade dashboard

Build a single page that shows, side by side: the number of nodes at each version, the count of ready and not-ready nodes, Pods pending and recently restarted, API server error rate and latency, and the main user-facing success rate and latency. Mark the start and end of each step on the graphs. During the window, the whole team looks at this one page, not at twenty separate ones.

Alerts that suit upgrades

• Nodes not ready for more than a few minutes.
• Pods pending for longer than expected.
• Crash loops and image pull failures.
• Rise in 5xx errors or latency of any critical service.
• Webhook call failures and API server errors.
• DNS lookup failures.

Stop rules

Agree on explicit thresholds that cause a pause. A pause means you stop starting new steps and investigate; it does not mean rollback. Examples: user error rate above the normal level for three minutes after a batch, any node stuck unready beyond ten minutes, more than a set number of pending Pods, or anything unexplained in the control plane logs. When a stop rule triggers, the person leading the upgrade pauses, and the group decides on next steps together.

Logs and events

Keep a running view of Kubernetes events and of the logs of the API server, scheduler, controller manager, kubelets and add-ons. Many problems announce themselves in events, such as failed mounts, failed scheduling or failed probes, minutes before they become visible to users. Centralized logging that survives node replacement is vital, since the logs on an old node vanish with it.

A timeline kept by a scribe

Ask one person to write down what happens and when: step started, step finished, anything odd. This record becomes the basis of the review afterwards and is invaluable if you need to reconstruct what caused a problem.

People ask, “What if we need to go back?” The answer depends on what part of the cluster you ask about, and it is often less comforting than hoped.

What can be undone and what cannot

Roll forward is the usual answer

In practice, most problems are solved by fixing the issue and moving on, not by going back. A removed API is fixed by updating a manifest. A broken add-on is fixed by upgrading it further or reverting only that add-on. A misbehaving node is replaced. This is why small, observable steps are so important: a small step has a small, quick fix.

The point of no return

Identify in your plan the moment after which going back to the old version is no longer feasible. For many clusters, it is the upgrade of the control plane. Before this point, the decision to continue should be careful and well informed, with all checks complete. After it, the plan changes from “go back” to “make the new version work.”

The real safety nets

• A tested backup of etcd and your resource definitions.
• Infrastructure as code, so a cluster can be recreated from files.
• Blue/green node pools, so the old workers still exist during early observation.
• A second cluster, in the most critical environments, to which you can shift traffic. The next chapter describes this.
• Rehearsal, since a problem found in the test cluster never needs a rollback.

Do not over-promise

Do not tell stakeholders that the upgrade is “reversible” unless you have actually tested the reversal. Honest wording is better: “We can pause at any step, we can replace the worker nodes with old-version machines for a limited time, and we have a plan to rebuild the cluster from backup in a few hours if something catastrophic occurs.”

Instead of changing a running cluster, build a new one at the new version and move the traffic across. It costs more, but it gives the best safety.

flowchart TB
  U["Users"] --> DNS["Traffic entry: DNS or global load balancer"]
  DNS -->|"100 percent at start"| BLUE["Blue cluster - current version"]
  DNS -.->|"0 percent at start"| GREEN["Green cluster - new version"]
  GIT["Git repository - declared state"] --> BLUE
  GIT --> GREEN
  DATA[("Shared or replicated data stores")] --> BLUE
  DATA --> GREEN
  GREEN --> CHK["Smoke tests and canary checks"]
  CHK -->|"healthy"| SHIFT["Shift traffic in steps: 1, 10, 50, 100 percent"]
  SHIFT --> DONE["Retire blue cluster after observation"]
  SHIFT -.->|"problem found"| BACK["Shift traffic back to blue"]

How it works

1. Create a brand new cluster at the target version, from the same declared configuration as the old one.
2. Deploy all workloads and add-ons to it using the same pipeline that deploys to the old cluster.
3. Run smoke tests and canary checks on the new cluster, with no user traffic.
4. Move a small share of traffic, for example one percent, to the new cluster using DNS weights or a global load balancer.
5. Increase the share in steps while watching error rates and latency.
6. When all traffic is on the new cluster and has been stable for an agreed period, delete the old one.

Why teams like it

The data problem

The hardest part is state. If your databases live outside the cluster, such as managed cloud databases, both clusters can use the same ones, and the migration is simple. If state lives in the cluster, you must replicate it to the new cluster, or move it with snapshots and a cut-over, which is complex. Teams often choose blue/green clusters for stateless tiers and in-place upgrades for clusters that hold heavy state.

Other dependencies to duplicate

• Certificates and domain names for ingress.
• Network rules and firewall allow-lists that mention cluster addresses.
• Identity and permission bindings for workloads that call cloud services.
• Monitoring, logging and alert routes.
• Scheduled jobs, which should not run in both clusters at once.

Humans typing commands at night make mistakes. Automation makes upgrades repeatable, reviewable and boring.

Cluster definitions as code

Describe your clusters in files: versions, node pools, add-ons and settings. Tools for infrastructure as code create and change clusters from these files. An upgrade becomes a change to a version number in a file, reviewed by colleagues, tested in a pipeline and applied by automation. The history of changes is in version control, so you can always see what was done and by whom.

GitOps for what runs inside

GitOps tools keep the cluster’s workloads and add-ons in sync with a repository. For upgrades, this has two benefits. First, rebuilding after a problem, or filling a new green cluster, is just pointing the tool at the repository. Second, changes to manifests that fix removed APIs go through the same reviewed path as any other change.

Cluster API and similar controllers

Cluster API lets you manage clusters as Kubernetes objects. A cluster’s control plane and machine pools have a version field. Changing it makes controllers create new machines at the new version, wait until they are ready, and delete the old ones, honoring settings for surge and maximum unavailable. It works with many infrastructure providers, and it brings the same declare-and-reconcile model to cluster upgrades that Kubernetes brings to application deployment.

Guardrails in the automation

Where to keep humans in the loop

Automate the routine, and keep people for the judgment calls: approving the start of the production window, approving the move from control plane to nodes, and deciding what to do when a stop rule triggers. A fully automatic upgrade is attractive for non-critical clusters, but critical ones deserve a human hand on the pause button.

Upgrade as a regular pipeline

Treat the upgrade as a pipeline that runs through environments in order: ephemeral test cluster, development, staging, a small production cluster, and finally the rest. Each stage must pass health checks and wait for a soak period before the next begins. Over time, the pipeline absorbs lessons: each failure adds a new check.

Technology is half of a smooth upgrade. The other half is making sure that people know what is happening and what to do.

Tell the right people at the right time

Roles during the window

• Upgrade lead: runs the steps and makes the call to pause or continue.
• Observer: watches the dashboards and calls out anything unusual.
• Scribe: records the timeline.
• Communicator: posts updates and fields questions, so the lead can focus.
• Application contacts: reachable engineers from important services, in case a service misbehaves.

Choosing the window

Pick a time when traffic is moderate and the team is rested and available, and not at the end of the week or just before a holiday or a business peak. For global businesses, there may be no truly quiet hour, so zero-downtime techniques matter even more. Avoid starting a long upgrade that you cannot pause safely at the end of the working day.

Pace and fatigue

Long upgrades tire people. Plan breaks, hand-offs and clear points where it is safe to stop for the night. A cluster with its control plane upgraded and nodes half done is a perfectly valid stopping place, since the skew policy allows it, as long as you have documented the state.

Learn afterwards

Within a few days, hold a short, blameless review. What surprised us? Which checks would have caught it sooner? What did the runbook not say? Update the checklist and the automation. An upgrade that taught nothing was a wasted chance.

Upgrades go wrong in a small number of repeating ways. Knowing the patterns shortens the time between “something is off” and “I know why.”

A disciplined way to investigate

1. Pause further steps, so that the situation does not change while you look.
2. Ask what changed last and where. Most problems relate to the most recent step.
3. Look at events for the affected objects, since Kubernetes often states the reason in plain words.
4. Compare a healthy example with a broken one: a node that works with one that does not.
5. Fix the smallest thing that solves the problem, and write it down.
6. Resume only after the signals are green again.

The “one change at a time” habit

Many long incidents come from changing several things in the middle of a problem: tuning a budget, restarting an add-on and replacing a node at once. When things improve, nobody knows why. Keep to one change at a time, and note it.

Between your users and your Pods sit load balancers and proxies. Their behavior decides whether a Pod’s departure is smooth or abrupt.

The traffic path

A typical request travels from the user to a cloud load balancer, then to a node, then through the service routing rules to a Pod, perhaps through an ingress controller on the way. Each hop keeps its own list of healthy destinations, and each list updates at its own speed. When a Pod or node is removed, the lists must catch up before the destination disappears.

Key settings and behaviors

Why node drains drop traffic if you are not careful

When a node is replaced, the cloud load balancer may still send requests to it for a few seconds or more, until its health check fails or the node is deregistered. If the node disappears first, those requests are lost. The remedy is to deregister the node, wait for connections to drain, and only then stop it. Managed node group tools usually do this, but check that your load balancer settings, such as the deregistration delay and the health check interval, are in agreement with the drain timing.

Clients that help

Good clients also help to produce zero downtime. Clients that retry idempotent requests with backoff, that use timeouts and that reconnect when a connection breaks, hide small disruptions. Your internal services should do this too, since service-to-service calls face the same moves. A request that fails once and succeeds on a quick retry is often invisible to the user.

Idempotency

Retries are only safe when repeating a request does no harm. A “create order” request that is retried may create two orders unless it carries a unique key that lets the server notice the repeat. Making important operations idempotent is a valuable design habit that pays back during every upgrade and incident.

One cluster is an event. Fifty clusters are a program. The same principles apply, but order, pace and standardization matter even more.

Waves and rings

Group clusters into rings by importance and risk, and upgrade ring by ring:

Standardize to make it tractable

• Few versions at once. Allow at most two or three Kubernetes versions across the fleet at any time.
• Same add-on sets. A standard platform bundle per version, tested together, is easier than a different mix per cluster.
• Same automation. One pipeline, one runbook, parameterized for each cluster.
• Clear ownership. A platform team owns the cluster and add-ons; product teams own their workloads and their readiness.
• Visible inventory. A dashboard listing every cluster, its version, its end-of-life date and its upgrade status.

Soak time

Between rings, wait long enough to see slow problems, such as memory leaks and rare errors, which do not show up in the first hour. A week in the lower rings is common. This soak period is the cheapest protection a fleet has.

Exceptions and the long tail

Some clusters will be hard to upgrade: they host special software, belong to teams that are slow to respond, or hold unusual configurations. Make exceptions visible and time-limited, with an executive sponsor and a date. Without that, the “just one old cluster” turns into a permanent risk.

Measuring the program

These are goals to aim at, not promises. Track them over time and ask why when they slip.

This is an illustrative story, not a report about a real company. It shows how the ideas in this guide fit together over a year.

The starting point

“Gatepass,” a fictional company that sells event tickets online, ran one big production cluster on a cloud provider. The cluster was two and a half years old, four minor versions behind, and the provider had announced that support for its version would end in three months. The team of six platform engineers had avoided upgrading because an earlier attempt had caused an hour of failed purchases.

Month one: understand the risks

The team did not touch the cluster. They read the release notes for each of the four versions in between, and made a list of every removed API and changed feature. They ran scans of all repositories and live objects, finding 37 objects using removed versions, several of them in third-party charts. They checked every add-on’s compatibility table and found that the ingress controller and the storage driver both needed upgrades first.

Month two: fix the foundations

They discovered that eleven services had only one replica, nine services had no disruption budget, and five had budgets that allowed no disruption at all. Six services ignored the termination signal. The application teams fixed these over four weeks, supported by a short guide and a policy that warned about missing budgets. The team also built an upgrade dashboard and a set of canary checks.

Month three: rehearse

The team created a copy of the cluster at the old version, loaded it with a traffic replay and rehearsed the first upgrade. They found that the first drain failed because a policy webhook with a single replica and a “fail closed” setting blocked its own replacement. They added replicas and namespace exclusions. They rehearsed three times, until the process produced fewer than a handful of failed requests, and the runbook had grown from one page to six.

Months four to six: one step at a time

Rather than build a giant jump, they upgraded one minor version about every five weeks, starting with the less important development cluster and then production on a Tuesday morning with a surge of three extra nodes. The first production upgrade took five hours, mostly waiting for careful drains. A single increase in checkout errors, lasting about a minute, was caused by a load balancer deregistration delay that was shorter than the Pod shutdown pause. They aligned the numbers, and the next step produced no visible blip.

Months seven to twelve: routine

With the cluster current, the team adopted a rhythm: test every new version in an automated ephemeral cluster in the first weeks after release, upgrade development a month later, and production a month after that. Upgrades dropped to two hours. They moved node upgrades to a blue/green pool method for the busiest season, which let them hold the old nodes for two days as a safety net. By the end of the year, they upgraded three times with no customer-visible incident.

What made the difference

Many painful upgrades follow the same few mistakes. Naming them helps you notice them in your own plans.

Teams improve in stages. Knowing where you stand lets you pick the next step instead of trying to do everything at once.

flowchart LR
  S0["Stage 0: Fear - upgrades are rare and manual"] --> S1["Stage 1: Planned - checklist, backups, release notes read"]
  S1 --> S2["Stage 2: Prepared apps - budgets, graceful shutdown, spread"]
  S2 --> S3["Stage 3: Rehearsed - test clusters with traffic"]
  S3 --> S4["Stage 4: Automated - pipeline with health gates"]
  S4 --> S5["Stage 5: Continuous - small, frequent, boring upgrades"]

Reading the diagram

The boxes read from left to right, each building on the last. At the far left is a team afraid of upgrades, doing them rarely and by hand. At the far right is a team whose upgrades are small and frequent, so that nobody pays them much attention. The path between them is not about learning new tools. It is about adding preparation, practice and automation, one stage at a time.

What each stage looks like

Questions to place yourself

• How many minor versions behind are we, and when does our version leave support?
• When did we last restore a backup in a test?
• If we drain a node right now, will users notice?
• Do we know every deprecated API we still use?
• How long would it take to rebuild this cluster from files?
• Who has done an upgrade, and who could do it next time?

Short answers to questions that come up again and again.

You have followed the whole path from the first worry to a routine. Here is the guide in a nutshell.

Your first month

Back to top