To appreciate runbooks, let us first feel the problem they solve. It is not a lack of skill. It is a lack of calm, shared, written knowledge at the exact moment it is needed.

A typical bad night

A phone buzzes at three in the morning. An alert says that the checkout service has a high error rate. The engineer on call is new to this service. She opens a dashboard that shows many graphs, but she is not sure which ones matter. She asks in the team chat, but everyone who knows the service is asleep. She starts running commands she half remembers from a training course. Twenty minutes pass. A senior colleague wakes up and joins, and in two minutes says, “Oh, that is the certificate again. It expires every year.” Total time to fix: forty minutes. Real fixing time: two minutes. The rest was searching.

Why Kubernetes makes this harder

What runbooks give you

Before writing a runbook, we need a shared meaning for the word “incident” and a picture of how one unfolds from start to finish.

A plain definition

An incident is an unplanned event that harms, or threatens to harm, the service your users depend on. A single failed pod that Kubernetes replaces within seconds is usually an event, not an incident. A checkout page that returns errors for ten minutes is an incident. The difference is impact on users or on the business, or a credible risk of it soon.

The life of an incident

flowchart LR
  D["Detect: an alert or a user report"] --> T["Triage: judge impact and severity"]
  T --> M["Mitigate: stop the harm quickly"]
  M --> R["Resolve: fix the real cause"]
  R --> V["Verify: confirm users are healthy"]
  V --> P["Review: write the postmortem and improve runbooks"]
  P -.-> D
    

Mitigate before you resolve

The most important idea in incident response is the difference between mitigation and resolution. Mitigation means stopping the pain for users, even if you do not yet know the cause. Resolution means fixing the underlying problem. During an outage, users care about the first. A rollback to the previous version, moving traffic away from a bad zone, or adding capacity are mitigations. They buy time. The deeper investigation can often wait until the service is healthy again.

Time matters in two ways

The word runbook is used loosely. Let us separate it from its close cousins so that your team builds the right thing.

Four kinds of document

What makes something a good runbook

A runbook is tied to a symptom or alert, not to a component. Responders do not wake up thinking “something is wrong with the scheduler”. They wake up with “pods are pending” or “latency is high”. A runbook should begin at the symptom, because that is what the responder holds in their hand. It should then lead them, step by step, through the checks that separate one cause from another.

Qualities to aim for

Every runbook in this post follows the same shape. A fixed shape matters, because a tired responder can find what they need without thinking about where it is.

The standard sections

flowchart TD
  A["Alert fires with a link"] --> B["Runbook opens at the symptom"]
  B --> C["Check impact: how bad, who is affected"]
  C --> D["Quick mitigation if available"]
  D --> E["Diagnosis steps in order"]
  E --> F{"Cause found?"}
  F -->|"yes"| G["Apply the matching fix"]
  F -->|"no"| H["Escalate to the named owner"]
  G --> I["Verify recovery"]
  H --> I
    

Decision points make runbooks useful

A runbook is not a list; it is a small map with branches. The best ones use plain questions: “Is the pod restarting? If yes, go to step four. If no, go to step six.” These questions turn a long document into a path of only a few steps for any given case. If a runbook has more than about ten steps on any path, it probably needs splitting into two.

A runbook guides what to do. Roles decide who does it. Without clear roles, many well-meaning people try to fix the same thing at once, or everyone assumes someone else is communicating.

The core roles

flowchart TD
  IC["Incident commander: decides and coordinates"] --> OPS["Operations lead: investigates and acts"]
  IC --> COM["Communications lead: updates stakeholders"]
  IC --> SCR["Scribe: records the timeline"]
  OPS --> SME["Subject experts: called in as needed"]
    

Rules that keep it calm

Not every problem deserves the same response. Severity levels are a shared language that tells everyone how fast to move, who to wake, and how often to update.

A simple severity scale

How to judge severity fast

In the first minutes, three questions are enough. How many users are affected? How badly: slow, partly broken, or fully down? Is it getting worse? If you are unsure, choose the higher level, and lower it later. Raising severity late costs more than lowering it early.

Escalation: asking for help on time

Escalation is not failure. It is the normal part of the process. A good runbook gives a time limit: “If you have not found the cause within fifteen minutes, escalate.” This removes the pressure on the responder to prove themselves, and prevents the common mistake of struggling alone while the outage grows.

flowchart TD
  A["Alert fires"] --> B["Primary on-call acknowledges"]
  B --> C{"Resolved or mitigated in the runbook time limit?"}
  C -->|"yes"| D["Verify and close"]
  C -->|"no"| E["Escalate to secondary on-call and service owner"]
  E --> F{"Severity 1 or widening impact?"}
  F -->|"yes"| G["Open formal incident, assign roles, notify leadership"]
  F -->|"no"| H["Continue with owner and update on a schedule"]
    

Before the individual runbooks, you need a map of where things can break. Kubernetes is a stack of layers, and a good responder works through them in a sensible order.

The layers, from top to bottom

flowchart TD
  U["Users and clients"] --> L1["Layer 1: Application code and configuration"]
  L1 --> L2["Layer 2: Pods and containers"]
  L2 --> L3["Layer 3: Workload controllers and scheduling"]
  L3 --> L4["Layer 4: Nodes and the container runtime"]
  L4 --> L5["Layer 5: Cluster networking and DNS"]
  L5 --> L6["Layer 6: Control plane and its datastore"]
  L6 --> L7["Layer 7: Cloud infrastructure: machines, disks, networks, identity"]
    

What each layer contains

Two helpful questions at the start

The first question is “What changed?” Most incidents follow a change: a deployment, a configuration edit, a certificate expiry, a traffic surge, or a cloud event. The second is “What is the blast radius?” If only one service is broken, look at layers one to three. If many unrelated services break together, look lower: nodes, networking, control plane, or cloud. The blast radius is a strong hint to the layer.

If you write only one runbook, write this one. It is the opening sequence that works for almost any Kubernetes alert, and it points you to the more specific runbooks that follow.

Step by step

flowchart TD
  S["Symptom reported"] --> A["Confirm impact and severity"]
  A --> B["Ask: what changed recently?"]
  B --> C{"Recent change likely the cause?"}
  C -->|"yes"| D["Mitigate: roll back the change"]
  C -->|"no"| E["Check blast radius"]
  E --> F{"One service or many?"}
  F -->|"one"| G["Pods, events, logs, resources, dependencies"]
  F -->|"many"| H["Nodes, network, DNS, control plane, cloud"]
  D --> V["Verify recovery"]
  G --> V
  H --> V
    

Every runbook step asks you to look at something. This chapter explains the three main sources of evidence in Kubernetes, so that you know what each one can and cannot tell you.

Pod status: the headline

Each pod has a status that is a one-word summary of its state. It is a headline, and headlines can mislead, but they point you in the right direction. The common values mean the following.

Events: the cluster’s diary

Kubernetes records events as short messages about what it is doing: pod scheduled, image pulled, probe failed, volume could not attach. When you describe a pod, the bottom of the output lists its events. They often state the cause in plain words, such as “insufficient memory” or “failed to pull image”. Two cautions apply: events are kept only for a short time, often about an hour, and they are summarized, so rapid repeats appear as one line with a count.

Logs: the application’s own words

Container logs are what the application printed. They tell you about problems inside the program, such as failed database connections, bad settings, or unhandled errors. If a container has restarted, you must ask for the previous container’s logs, because the current one may be brand new and empty. If your cluster ships logs to a central system, search there, since the pod may no longer exist.

This is the most famous Kubernetes symptom. A container starts, dies, and is restarted again and again, with longer and longer pauses between attempts. The pause is the “back off”.

Symptom and impact

Symptom: pods show restart counts rising, and a status of CrashLoopBackOff. Impact: fewer healthy replicas, possibly none, so traffic fails or capacity falls. Usual severity: two if a main service has few healthy replicas, three if only some replicas are affected.

Quick mitigation

Diagnosis

flowchart TD
  A["Container restarting"] --> B["Describe the pod: read last state, reason, exit code"]
  B --> C{"Reason is OOMKilled?"}
  C -->|"yes"| D["Go to memory runbook"]
  C -->|"no"| E["Read previous container logs"]
  E --> F{"Logs show a clear error?"}
  F -->|"missing config or secret"| G["Fix the missing setting"]
  F -->|"cannot reach dependency"| H["Check the dependency and network"]
  F -->|"application bug"| I["Roll back or hotfix"]
  F -->|"no logs at all"| J["Check the command, entry point, and image"]
  E --> K{"Liveness probe failing?"}
  K -->|"yes"| L["Probe too strict or app too slow to start"]
    

The common causes, in order of how often they occur

Verification and escalation

Verify that pods reach ready, that restart counts stop rising for at least ten minutes, and that error rates return to normal. Escalate to the owning team if you cannot find a clear error after fifteen minutes, or if the logs point to a defect you cannot fix by configuration.

A pending pod has been accepted by the cluster but has not been placed on a node. It is waiting. The question is: waiting for what?

Symptom and impact

Symptom: new pods remain in the Pending state for minutes. A rollout stalls, or an autoscaler adds replicas that never start. Impact: the service cannot grow, and during a traffic spike this becomes an outage. Usual severity: two if it blocks scaling during load, three otherwise.

The key to the diagnosis

The scheduler writes the reason onto the pod’s events. It usually says something like “zero of N nodes are available”, followed by the reasons it rejected each node. Reading that message is almost always the first and best step.

flowchart TD
  A["Pod pending"] --> B["Describe the pod and read the scheduling message"]
  B --> C{"What does the message say?"}
  C -->|"insufficient CPU or memory"| D["Capacity or requests too large"]
  C -->|"node taints not tolerated"| E["Pod lacks a needed toleration"]
  C -->|"node selector or affinity not matched"| F["Labels do not match any node"]
  C -->|"volume problem"| G["Storage not bound or in another zone"]
  C -->|"too many pods"| H["Node pod limit reached"]
  C -->|"no message at all"| I["Check scheduler health and quotas"]
    

Causes and fixes

When the autoscaler should have helped

If the cause is simple lack of capacity, the node autoscaler should add machines. If it does not, check its own logs and events. Common reasons are that the cloud account has reached its machine quota, the chosen machine type is unavailable in the zone, the pod’s rules cannot be satisfied by any available machine type, or the autoscaler has hit its configured maximum size.

Before a container can run, the node must download its image from a registry. When this step fails, pods sit in a state called ImagePullBackOff or ErrImagePull.

Symptom and impact

Symptom: pods never start, and events say that the image could not be pulled. Impact: a new release cannot roll out, and any replacement pods cannot start, which becomes serious if nodes are also being replaced. Usual severity: three, rising to one or two if it blocks recovery from another incident.

Checks in order

Make this failure rare

Memory problems are among the most frequent causes of Kubernetes incidents. They come in two forms: a container exceeding its own limit, and a whole node running short.

Symptom and impact

Symptom: containers restart with the reason OOMKilled, or pods are evicted, or latency rises as the system struggles. Impact: dropped requests, lost in-memory work, and sometimes cascading failures. Usual severity: two to three.

Two different problems

Diagnosis

flowchart TD
  A["Memory trouble suspected"] --> B{"Which scope?"}
  B -->|"one container restarts with OOMKilled"| C["Compare usage over time with the limit"]
  B -->|"many pods evicted on one node"| D["Check node memory pressure and total requests"]
  C --> E{"Does usage grow steadily without ever falling?"}
  E -->|"yes"| F["Likely a leak: roll back, restart on schedule as a stopgap, file a defect"]
  E -->|"no, it spikes"| G["Limit too low for the real peak: raise limit and request"]
  D --> H["Move pods off, add nodes, and lower overcommitment"]
    

Fix steps

Node-level pressure

Nodes report pressure conditions for memory, disk, and process counts. When pressure appears, the node’s agent starts evicting pods, beginning with those using more than they requested and of lower priority. Pods with no requests set are the most exposed. Setting honest requests for every container, and priorities for important services, protects them during pressure.

A node is a machine. When a node stops reporting that it is healthy, Kubernetes marks it NotReady and, after a delay, begins moving its pods elsewhere. This is both a protection and a source of trouble.

Symptom and impact

Symptom: one or more nodes show NotReady. Pods on them become unreachable, and later are rescheduled. Impact: lost capacity, and a burst of restarts as pods move. If several nodes fail together, replacements may not fit. Usual severity: three for one node, one or two for many at once.

The sequence of events

sequenceDiagram
  participant N as Node agent (kubelet)
  participant C as Control plane
  participant P as Pods on the node
  N->>C: Regular heartbeat
  Note over N: Node problem begins
  N--xC: Heartbeats stop
  C->>C: Waits for a grace period
  C->>C: Marks node NotReady and adds taints
  C->>P: Evicts pods after the toleration period
  C->>C: Creates replacement pods on other nodes
    

Diagnosis steps

Mitigation and fix

Almost every request inside Kubernetes begins with a name lookup. When DNS misbehaves, many unrelated services seem to break at once, which is why DNS is a famous suspect.

Symptom and impact

Symptom: errors such as “could not resolve host”, intermittent timeouts to other services, slowness at the start of requests. Impact: widespread, because most service calls depend on names. Usual severity: one or two when it is cluster-wide.

How lookups work, in brief

Each pod is configured to send name lookups to the cluster’s DNS service, which is run by a set of DNS pods. Those pods answer names for cluster services themselves, and forward other names to outside resolvers. A lookup can fail at the pod’s own settings, at the DNS pods, at the network between them, or at the upstream resolver.

flowchart LR
  APP["Application pod"] --> RES["Pod resolver settings"]
  RES --> SVC["Cluster DNS service"]
  SVC --> DNSP["DNS pods"]
  DNSP -->|"cluster names"| API["Cluster service records"]
  DNSP -->|"outside names"| UP["Upstream resolver"]
    

Diagnosis steps

Common causes and fixes

The pods look fine, yet clients cannot reach the service. This runbook walks the path a request takes, from the outside to the pod, and finds where it stops.

Symptom and impact

Symptom: timeouts, connection refused, or errors from the gateway such as “bad gateway” or “no healthy upstream”. Impact: users cannot use the feature, even though the application may be healthy. Usual severity: one or two for public services.

Follow the path of a request

flowchart LR
  C["Client"] --> LB["External load balancer"]
  LB --> ING["Ingress or gateway"]
  ING --> SVC["Service"]
  SVC --> EP["Endpoints: ready pods"]
  EP --> POD["Pod and container"]
    

Check each hop, from the inside out

Typical findings

The control plane is the brain of the cluster, and the API server is its front door. When it struggles, commands hang, and nothing seems to change. Fortunately, running applications usually keep running.

Symptom and impact

Symptom: commands time out, dashboards show errors, deployments do not progress, autoscalers do not react. Impact: you cannot change anything, and failed pods are not replaced. Existing pods keep serving traffic, which is a comfort, but any new failure will not heal. Usual severity: one or two.

Who owns the control plane?

On managed services, the cloud provider operates the control plane, and your levers are limited. You can still check the provider’s status page, your own request patterns, and the components you installed. On self-managed clusters, you own all of it, including the machines and the datastore. The runbook should state clearly which case applies, because the available actions are very different.

Diagnosis steps

flowchart TD
  A["API slow or failing"] --> B{"Managed control plane?"}
  B -->|"yes"| C["Check provider status, open a support case, reduce your own request load"]
  B -->|"no"| D["Check API server, scheduler, controller manager, and datastore health"]
  C --> E["Find heavy callers and stop them"]
  D --> E
  E --> F{"Webhook failing or slow?"}
  F -->|"yes"| G["Fix or temporarily relax the webhook failure policy"]
  F -->|"no"| H["Scale or repair control plane components"]
    

What still works

Remind the team that running pods continue to serve, existing connections continue, and node agents keep their pods alive using the last instructions they received. This knowledge prevents panicked restarts, which would be fatal since restarted pods cannot be recreated if the control plane is down. The rule during a control plane problem is: do not make it worse by disturbing healthy workloads.

Behind the API server sits a small, careful database called etcd, which stores the entire state of the cluster. If you use a managed service, the provider runs it for you. If you run your own cluster, this runbook can save your cluster.

Why it matters so much

Everything Kubernetes knows is stored in this datastore: every object, every secret, every setting. It uses a voting system among several members, and needs a majority to accept changes. If a majority is lost, the cluster can no longer change anything. If the data is lost or corrupted without a backup, the cluster’s memory is gone.

Symptoms

Runbook steps for self-managed clusters

Prevention

Most incidents begin with a change. This runbook covers the single most useful mitigation in all of operations: undoing the change.

Symptom and impact

Symptom: errors, latency, or crashes that began shortly after a release or configuration change. Impact: depends on how many replicas have the new version. Usual severity: depends on traffic share affected.

How a rolling update behaves

A Deployment replaces old pods with new ones gradually. It starts a few new pods, waits for them to become ready, then removes old ones. If the new pods never become ready, the rollout stalls, and the old pods keep serving. That is a safety feature. The danger is a bad version that does pass the readiness check but behaves badly, because then the rollout completes and the bad version takes over.

sequenceDiagram
  participant R as Release pipeline
  participant D as Deployment
  participant O as Old pods
  participant N as New pods
  R->>D: New version declared
  D->>N: Start a few new pods
  N-->>D: Become ready
  D->>O: Remove a few old pods
  Note over N: Errors begin to rise
  R->>D: Roll back to the previous version
  D->>O: Start old pods again
  D->>N: Remove new pods
    

The rollback steps

Ways to make rollbacks fast

Certificates are the quiet time bombs of the platform. They work perfectly for a year and then, at an exact second, everything that relied on them stops.

Symptom and impact

Symptom: sudden errors mentioning “certificate has expired” or “x509”, across many components at once, or a public site showing a browser warning. Impact: can be total. Expired control plane certificates can stop the cluster from functioning. Expired ingress certificates make sites unreachable for users. Usual severity: one.

Where certificates live in a cluster

Steps

Stateless pods are easy to replace. Pods with data are not. When a disk cannot attach, fills up, or sits in the wrong place, the pod waits, and so do your users.

Symptom and impact

Symptom: pods stuck in ContainerCreating or Pending with messages about volumes, claims stuck in Pending, applications failing to write, or “no space left on device”. Impact: stateful services such as databases, queues, and caches are down or read-only. Usual severity: one or two, because data is involved.

The pieces involved

Diagnosis

flowchart TD
  A["Pod waiting on storage"] --> B{"Is the claim bound?"}
  B -->|"no"| C["Check storage class, provisioner health, cloud quota, and zone"]
  B -->|"yes"| D{"Is the volume attached to a node?"}
  D -->|"no"| E["Check attach errors, zone match, and whether the old node still holds it"]
  D -->|"yes"| F{"Is the disk full or read-only?"}
  F -->|"full"| G["Free space or expand the volume"]
  F -->|"read-only"| H["Check filesystem errors and node health"]
  F -->|"fine"| I["Check application and permissions"]
    

Common causes and fixes

Backups are part of the runbook

Every stateful runbook should say where the latest backup is, how recent it is, and how to restore it. A storage incident that turns into data loss is much worse than an outage. Regular snapshots and restore tests are the real mitigation for the worst case.

Sometimes there is simply too much demand for the capacity you have, or the machinery that adds capacity has stopped working. This runbook handles both.

Symptom and impact

Symptom: rising latency under load, pods pending, autoscalers at their maximum, CPU throttling, or queues growing. Impact: a gradual slide into an outage as demand exceeds supply. Usual severity: two, rising to one if load keeps growing.

The chain of scaling, and where it breaks

flowchart TD
  L["Load rises"] --> M["Metrics collected"]
  M --> H["Horizontal autoscaler decides more pods"]
  H --> P["New pods created"]
  P --> S["Scheduler finds room"]
  S -->|"no room"| N["Node autoscaler adds machines"]
  N --> C["Cloud provides machines"]
  C --> R["Pods start and become ready"]
    

Check each link

Mitigation while you diagnose

Not every incident is an accident. Sometimes a container is compromised, a credential leaks, or a strange workload appears. Security runbooks have a different priority: contain first, preserve evidence, then clean up.

Symptom and impact

Symptom: unknown pods or workloads, unexpected outbound traffic, sudden high CPU from mining software, alerts from security tools, unfamiliar changes by a service account, or leaked credentials found in public places. Impact: potential data theft, further spread, and legal obligations. Usual severity: one or two, until proven harmless.

Containment without destruction

flowchart LR
  A["Suspicious signal"] --> B["Involve security"]
  B --> C["Isolate: cut network, keep the workload"]
  C --> D["Preserve logs and state"]
  D --> E["Revoke credentials and permissions"]
  E --> F["Find entry point and scope"]
  F --> G["Rebuild from trusted sources"]
  G --> H["Review and harden"]
    

Questions to answer

Good technical work can be ruined by poor communication. Silence makes people anxious, and rumor fills the gap. A runbook for communication is as important as one for pods.

Who needs to hear what

A good status update has five parts

A runbook nobody finds is useless. The link between an alert and its runbook is what turns documents into response. This chapter covers how to design alerts so that they help rather than overwhelm.

Alert on symptoms, not causes

The best alerts describe what users feel: high error rate, high latency, failed checkouts. Alerts on causes, such as “CPU is above eighty percent”, fire often without any harm and train people to ignore them. Cause-level signals are better kept as dashboards and as diagnostic steps inside runbooks. A few high-quality symptom alerts, each with a runbook, beat hundreds of noisy ones.

What every alert should carry

Page only when action is needed

Useful symptom alerts for a cluster

Once a runbook is stable and well understood, many of its steps can be done by machines. Automation is the natural next step, but it carries its own risks.

The automation ladder

Start with diagnosis automation

The safest and most valuable first automation is gathering evidence. A responder who receives the pod states, recent events, recent deployments, and key graphs in one message saves several minutes and nothing can go wrong, because the automation only reads. Many teams build this as a chat command or as an enrichment attached to each alert.

Rules for automatic remediation

flowchart TD
  A["Alert fires"] --> B["Automation gathers evidence"]
  B --> C{"Known failure with safe fix?"}
  C -->|"yes"| D{"Within rate limits and scope?"}
  C -->|"no"| H["Page a human with the evidence attached"]
  D -->|"yes"| E["Apply the fix and notify the channel"]
  D -->|"no"| H
  E --> F{"Did the signal recover?"}
  F -->|"yes"| G["Close and record"]
  F -->|"no"| H
    

A runbook that has never been used is a theory. Practice turns it into a tool, and reveals the steps that no longer work.

Why untested runbooks fail

Ways to practice

How to run a good game day

An incident that teaches nothing will repeat. The postmortem is the step that turns pain into improvement, and the runbook is where much of that improvement should land.

What a blameless postmortem is

A postmortem is a written review of what happened, why it happened, how it was handled, and what will change. Blameless means that it looks for weaknesses in the system and process, not for a person to punish. People make mistakes in every system. A good system makes mistakes hard to make, easy to spot, and cheap to undo.

Parts of a useful postmortem

Feeding the runbooks

flowchart LR
  I["Incident"] --> P["Postmortem"]
  P --> RB["Update or create runbooks"]
  P --> AL["Improve alerts and detection"]
  P --> AU["Automate repeated steps"]
  P --> DS["Fix the design weakness"]
  RB --> G["Next incident is shorter"]
  AL --> G
  AU --> G
  DS --> G
    

Questions that improve runbooks

An anti-pattern is a habit that looks sensible but leads to trouble. Each one below is written as a short decision record: why people choose it, what goes wrong, and what to do instead.

To finish the practical part, here is a one-page map from symptom to runbook, and a plan for building your own library from nothing.

Symptom to runbook

Build your library in this order

flowchart TD
  A["Few or no runbooks today"] --> B["Write universal triage"]
  B --> C["Write top five from real incidents"]
  C --> D["Link every paging alert to a runbook"]
  D --> E["Practice with drills and game days"]
  E --> F["Improve after each incident"]
  F --> G["Automate the safe parts"]
  G --> F
    

These are the questions people ask most often when they begin to build incident response runbooks for Kubernetes.

We began with a pilot reaching for a checklist when an alarm sounds. An incident runbook is that same checklist, written calmly in advance so that a tired person can do the right thing in the right order.