Before we design anything, let us agree on what we are building and why it matters more for Kafka than for many other systems.

Runbook, playbook and checklist

A runbook is a document that tells an operator how to handle a specific situation, step by step. Some teams call a similar document a playbook, and a very short version a checklist. The names matter less than the purpose: to move knowledge out of a few experts’ heads and into a form that anyone on the team can use under pressure.

Why Kafka in particular

Apache Kafka is a system that stores and moves streams of events between software services. It usually sits in the middle of many applications, so when it has a problem, many teams feel it at once. Several features make Kafka failures tricky:

• Many moving parts. Brokers, controllers, partitions, replicas, producers, consumers and connectors all interact.
• Symptoms far from causes. A slow disk on one broker can show up as consumer lag in an application that has never heard of that broker.
• Actions with lasting effects. Some fixes, such as allowing an unsafe leader election or resetting consumer positions, can lose or repeat data. A wrong move is hard to undo.
• Self-healing that can be interrupted. Kafka repairs a lot on its own. Well-meant restarts can interrupt that repair and make things worse.

What runbooks give you

Every runbook entry should have the same shape. A consistent shape means responders know where to look, and authors know what to write.

flowchart LR
  A["Detect"] --> B["Triage"]
  B --> C["Mitigate"]
  C --> D["Verify"]
  D --> E["Follow up"]
    

The parts of an entry

An example of the flavor

Suppose a runbook entry for a full disk contains this step: “Check how full the data disk is on the affected broker. Expected result: a percentage. If it is above ninety percent, go to the section on freeing space. If it is below eighty, the disk is probably not the cause; go to the section on slow requests.” Notice what this step does. It names the action, states the expected result, and says where to go next for each possible answer. Nobody has to guess.

Keep it scannable

• Put the most important information first: what this is, how bad it is, and the first safe action.
• Use short numbered steps, not paragraphs.
• Give the expected result for every check.
• Use bold text sparingly for warnings.
• Link to deeper explanations, so the main page stays short.

A handful of principles separate runbooks that save the night from runbooks that are ignored.

Principle one: written for the worst moment

Assume the reader is tired, stressed, new to the team and reading on a phone. Use plain words, short sentences and clear headings. Avoid jokes, jargon that is not defined and long explanations. Save the teaching for a separate background page.

Principle two: symptom first

Organize by what responders see, such as “producers are timing out”, not by what the cause might be, such as “network partition”. In an incident, the cause is unknown. That is the whole problem.

Principle three: safe by default

Put reading and observing steps before changing steps. Put reversible changes before irreversible ones. Mark dangerous steps clearly, and require a second person to approve them. A runbook should make the safest choice the easiest one.

Principle four: every step is checkable

For each action, state the result that proves it worked. “Restart the broker” is weak. “Restart the broker, then wait until it appears in the cluster’s broker list and the count of under-replicated partitions returns to zero” is strong.

Principle five: decisions, not just steps

Real incidents branch. Include decision points with clear conditions, and give a default path when the situation is unclear. Where a judgment call is needed, say who makes it and what information they need.

Principle six: honest about risk

State what can go wrong. If a step may lose data, say so in plain words at the top of that step, not in a footnote. If the step needs the approval of a data owner, say so.

Principle seven: kept alive

An untested runbook is a rumor. Test entries regularly, update them after every incident, and record the last test date on the page. Retire pages that describe systems you no longer run.

Write for the reader you will be at 3 a.m.

The person using a runbook is rarely at their best. They may have been asleep a minute ago, they may be new to the team, and they may be worried about the customers affected. Good design accepts this. It avoids clever shortcuts, gives exact names and locations, and never assumes the reader remembers a detail from a training session six months ago. Every time you write a step, imagine handing it to a capable colleague from another team who has never seen this cluster. If they could follow it without calling you, the step is ready.

A runbook cannot teach everything, but responders must share a small common picture of how Kafka works. Here is the shortest version that still lets you follow the scenarios later in this guide.

Brokers, topics and partitions

A broker is one Kafka server. A group of brokers is a cluster. Data is organized into topics, and each topic is split into partitions, which are separate ordered logs. Every event in a partition has a running number called an offset.

Replicas, leaders and the ISR

Each partition is stored as several copies, called replicas, on different brokers. One replica is the leader, which handles reads and writes. The others are followers, which copy from the leader. The followers that are fully caught up form the in-sync replica set, or ISR. Producers that ask for full confirmation wait until the whole ISR has their data, and a setting called minimum in-sync replicas decides how small the ISR may become before writes are refused.

The controller

One component, the controller, manages metadata for the cluster: which broker leads which partition, which topics exist and what the settings are. When a broker fails, the controller picks new leaders from the ISR. In modern Kafka, a small group of controller nodes agree among themselves using a protocol called KRaft. Older clusters used a separate system called ZooKeeper for this job.

Producers, consumers and groups

A producer sends events. A consumer reads them. Consumers usually work in a consumer group, where the partitions of a topic are shared among the members so that each partition is read by only one member at a time. Each group remembers how far it has read by saving its offset. The gap between the newest event and the group’s position is called lag.

A runbook says “check the disk” or “look at the lag”. That only works if the dashboards, logs and tools it names actually exist and are easy to reach.

The core dashboard

Logs and command-line tools

Runbooks often name tools that ship with Kafka, such as the topics tool (to list and describe topics and to find problem partitions), the consumer groups tool (to see lag and group state), the configuration tool (to view and change settings), the leader election tool, the reassignment tool and the metadata quorum tool. Name the tool and the purpose in the runbook, and keep a tested copy of the exact commands in a shared place with variables for cluster names. Make sure the tools are installed where responders will work, and that they can authenticate to the cluster.

Broker logs, garbage collection logs and operating system logs must be searchable from one place. During an incident, nobody wants to log in to a dozen machines one by one.

Access before the incident

• On-call engineers can log in to dashboards, log search and the cluster tools before they are ever paged.
• Emergency access to dangerous tools is arranged in advance, with approvals and logging, not invented at midnight.
• Links in the runbook go directly to the right dashboards for the right cluster and time range.

Not every alert deserves a midnight phone call. Clear severity levels and clear roles keep the team calm and focused on the right things.

A simple severity scale

Write the severity guidance into every runbook entry, along with the conditions that raise or lower it. For example, “one broker down” is high, but becomes critical if the minimum in-sync replicas setting is violated on important topics.

Roles during an incident

Escalation paths

• List who to call when the on-call engineer is stuck, with phone numbers or paging groups, not just names.
• State the time limit: “if you have not found the cause in twenty minutes, escalate.”
• Include vendor and cloud provider support contacts, account numbers and how to raise the priority of a case.
• Name the data owners who must approve risky steps, such as allowing data loss to restore availability.

Communicating during an incident

Good technical work can be undone by poor communication. Colleagues who hear nothing will assume the worst, start their own investigations, and flood the responders with questions. A steady rhythm of short updates keeps everyone calm and lets the operators concentrate.

• Announce early. Within the first ten minutes of a high or critical incident, post what is known: what is affected, since when, and who is leading.
• Update on a schedule. Every twenty or thirty minutes, even if the message is “no change, still investigating, next update at three fifteen”.
• Separate facts from guesses. Say “we see” for facts and “we suspect” for theories.
• Name the impact in business terms. “Orders are delayed by about ten minutes” means more to a manager than “consumer lag on the orders group is high”.
• Announce the end. Say when service is restored, what is still being watched, and when the review will happen.

Put message templates into the runbook, with blanks for the affected service, symptoms, start time and next update. Filling in a template at 3 a.m. takes ten seconds; composing a clear message from scratch takes ten minutes and often goes wrong.

Ten excellent pages are useless if nobody can find the right one in a hurry. Structure the whole collection for fast navigation.

A three-layer structure

The triage page

The triage page turns a vague alert into a specific entry. It should fit on one screen and work like a decision tree.

flowchart TD
  A["Alert or report"] --> B{"Are reads or writes failing for users?"}
  B -- Yes --> C["Treat as high severity and open an incident"]
  B -- No --> D["Investigate calmly and check for a worsening trend"]
  C --> E{"Any offline partitions?"}
  E -- Yes --> F["Go to the offline partitions entry"]
  E -- No --> G{"Any under-replicated or under-minimum partitions?"}
  G -- Yes --> H["Go to the broker and ISR entries"]
  G -- No --> I{"Are consumers behind or erroring?"}
  I -- Yes --> J["Go to the consumer entry"]
  I -- No --> K["Check broker health: disk, CPU, network, certificates"]
    

Naming and searching

• Name entries by symptom, for example “Producers get not-enough-replicas errors”.
• Add the exact alert names and error messages to each page so search finds them.
• Link every alert directly to its runbook entry.
• Keep a single index, sorted by symptom, and another sorted by component.

Where the runbooks live

Store them in a place that stays available when your main systems fail, and where changes are tracked. Many teams keep them as text files in version control, published to an internal website, and also exported as a printable bundle. Anyone can suggest changes, and every change has an author and a reviewer.

Most Kafka incidents begin the same way. This general entry covers the opening minutes, before you know which specific scenario you are in.

Habits for the whole incident

• Observe before you act. Reading costs nothing. Changing things may.
• One change at a time. Wait to see the effect before the next change. Otherwise you will never know what helped.
• Say it out loud in writing. Post each action and result in the incident channel.
• Prefer reversible actions. Choose steps you can undo.
• Do not restart things without a reason. A restart is a change with side effects, not a universal fix.

Handing over between shifts

Long incidents outlast a single person’s energy. Tired responders make mistakes, so plan to hand over after a few hours. A good handover note covers the current state, what has been tried and what happened, the current theory, the next planned step, the pending approvals, and the contacts already involved. Keep the incident log up to date so the note writes itself.

Also decide who may say “stop”. Any responder who feels a step is too risky should be able to pause the action and ask for a second opinion, without being seen as slow. Many bad outcomes come from someone noticing a doubt and not saying it.

This is the most common Kafka failure, and a well-configured cluster survives it with little noise. Your runbook should confirm that recovery is on track and guide the repair of the broker itself.

sequenceDiagram
  participant B as Failed Broker
  participant C as Controller
  participant F as Follower Brokers
  participant P as Producers and Consumers
  B-xC: Stops responding
  C->>F: Promote in-sync followers to leaders
  C->>P: Publish new leader information
  P->>F: Retry requests against the new leaders
    

Symptoms

• An alert that a broker is unreachable, or the broker count is one lower than normal.
• Under-replicated partitions rise, because the missing broker’s replicas are out of sync.
• Brief bumps in produce and fetch errors while leaders move, then a return to normal.

Quick checks

1. Is the machine alive? Check the host, virtual machine or container status from the cloud or orchestration tool.
2. Is the Kafka process running? Check its service status and recent logs.
3. How many partitions are under-replicated, under the minimum or offline? Under-minimum or offline partitions raise the severity.
4. Did anyone stop it on purpose, for maintenance or an upgrade?
5. Why did it stop? Look for out-of-memory errors, disk failures, “no space left” messages or a crash trace.

Actions

If the machine or its disk is lost

Provision a replacement with the same broker identity. It starts empty and copies all its data from the other replicas. This can take hours, depending on data size and any speed limit you set. Watch network and disk load on the source brokers, and keep an eye on capacity, since the cluster runs with fewer safe copies until the copy is done.

Verification

• Broker count is back to normal.
• Under-replicated, under-minimum and offline partition counts are zero.
• Client error rates and latency are normal.
• Leadership is spread evenly across brokers.

Do not

• Do not restart other brokers to “help”. Each restart shrinks more ISRs.
• Do not delete the broker’s data directory unless you are sure other in-sync replicas exist for every partition.
• Do not allow unsafe leader election just because a broker is down.

Planned restarts versus surprise failures

A planned restart and a crash look similar on a dashboard, but they are not the same. During a planned restart, the broker announces that it is leaving, and leadership moves away cleanly before it stops. A crash gives no warning, so clients must notice and retry, and the restarted broker must check its files. Your runbook should say how to tell them apart: check the change calendar, the deployment log and the last lines of the broker log for a clean shutdown message.

It should also set expectations about time. A clean restart of a healthy broker typically takes a minute or a few minutes, while recovery after a crash on a broker with many partitions and large files may take much longer. Write down the normal recovery time for your own cluster, measured in a drill, so responders know when to worry and when simply to wait.

Here the brokers are running, but many replicas have fallen out of sync. The runbook’s job is to find the shared cause quickly, since scattered symptoms usually have one origin.

Symptoms

• Under-replicated partition count is above zero for several minutes.
• ISR shrink rate spikes without a matching rise in expands.
• Producers using full confirmation report not-enough-replicas errors when the ISR falls below the minimum.
• Produce latency climbs before the shrink, then drops.

Quick checks

1. List the affected partitions. Group them by missing replica, by leader and by zone.
2. One broker missing everywhere points to that broker (disk, pauses, network, recent restart). One leader everywhere points to an overloaded leader. One zone points to zone networking.
3. Everything at once at a sharp time points to a certificate, permission, setting or deployment.
4. Check for a reassignment in progress and for leftover replication throttles.
5. Check request idle percentages, disk latency, network errors and garbage collection pauses on the affected brokers.

Actions

• Freeze changes and pause reassignments and heavy jobs.
• Fix the shared cause you found: renew the certificate, restore a permission, remove a stale throttle, repair a link, or add capacity.
• Reduce load with client quotas if brokers are saturated.
• Watch expansions. Do not restart brokers while they catch up.

Decision point

If writes are failing on critical topics and the cause is not found within your agreed time limit, the incident commander may decide on an emergency relaxation, such as lowering the minimum ISR on named topics. This reduces protection, so it requires the approval of the data owner, a written note and a reminder to restore the setting afterwards.

Verification

• Under-replicated and under-minimum counts return to zero and stay there.
• Shrink and expand rates return to baseline.
• Any temporary setting is restored.

Do not

• Do not restart many brokers.
• Do not raise the replica lag limit as a first reaction.
• Do not enable unsafe leader election cluster-wide.

An offline partition has no leader at all, so nobody can read or write it. This is the most serious data-path failure, and the runbook must be careful, because the fastest fix can also lose data.

Why a partition goes offline

A partition needs a leader chosen from its in-sync replicas. If every in-sync replica is gone, Kafka has no safe choice. It waits, because promoting an out-of-date copy would silently throw away messages. So the partition stays offline until one of the in-sync replicas returns, or a human accepts data loss.

Quick checks

1. List offline partitions and the topics they belong to. How important are those topics?
2. For each, list its replicas and its last known ISR. Which brokers were in it?
3. Are those brokers down, and can any be recovered? A broker that is merely slow to restart is a very different case from a destroyed disk.
4. Was there a recent unclean shutdown, a zone loss or a mass restart that explains it?

Actions, in order

Verification

• Offline partition count is zero.
• Producers and consumers work on the affected topics.
• A data-loss assessment is recorded, even if the answer is “none found”.

Do not

• Do not enable unclean election cluster-wide as a shortcut.
• Do not wipe the data of a broker that might hold the only up-to-date copy.
• Do not skip the approval and the written record.

Disks are where Kafka keeps everything, so disk trouble is never small. There are two different cases: a disk that is filling up, and a disk that is broken.

Case A: the disk is filling

Kafka stops a broker that can no longer write, and a full disk often causes messy recovery. The goal is to act while there is still space.

Symptoms

• Disk usage alert at eighty or ninety percent, or a steep upward slope.
• Log directory going offline messages, or “no space left” errors.
• Growth in one topic that does not match usual traffic.

Quick checks

1. Which broker and which directories are affected? Is it all brokers or one?
2. Which topics use the most space? Did one grow suddenly?
3. Is retention working? Compare oldest data age with the configured retention. Check for stuck cleaners, huge segments or slow deletion.
4. Is a consumer or replica far behind so that data cannot be removed, or a compacted topic that is growing without bound?
5. Is a reassignment copying large amounts of data to this broker?

Actions, from safest to riskiest

Case B: a disk has failed

If the broker uses several disks and one fails, Kafka can mark that directory offline and keep serving from the others, depending on version and configuration. Replicas on the dead disk become unavailable and must be rebuilt elsewhere.

• Confirm which directory is offline and which partitions were on it.
• Check that other replicas of those partitions are in sync before doing anything.
• Replace or repair the disk, then let the broker replicate the missing data back, or reassign those partitions to other brokers.
• If the whole machine’s storage is lost, follow the “machine lost” steps in Chapter 9.

Verification

• Free space is comfortably above your threshold, and the trend is flat or falling.
• All log directories are online. No offline or under-replicated partitions remain.
• Any temporary retention change is reverted or turned into a reviewed permanent decision.

Setting early-warning thresholds

The best disk runbook is the one you rarely need, because alerts arrive early. Use two kinds of alerts. A level alert fires at a fixed usage, such as seventy and eighty-five percent. A prediction alert fires when the current growth rate will fill the disk within a set time, such as twelve hours. The prediction alert is the more valuable, because a slow climb from sixty to seventy percent in a week is calm, while the same change in an hour is an emergency.

Tie each alert to the action expected. Seventy percent creates a ticket to plan capacity. Eighty-five percent pages the on-call engineer in working hours. Ninety-five percent pages at any time and follows the steps above. Clear rules stop arguments about whether an alert can wait.

Many Kafka pages are about consumers, not brokers. The broker is usually fine; the applications reading from it are slow, stuck or constantly reshuffling. The runbook should separate these three cases quickly.

flowchart TD
  A["Consumer lag is rising"] --> B{"Is the group active with members?"}
  B -- No --> C["Group is empty or dead: check the application and deployment"]
  B -- Yes --> D{"Are rebalances happening again and again?"}
  D -- Yes --> E["Rebalance storm: check timeouts, restarts, slow processing"]
  D -- No --> F{"Is lag rising on all partitions or only a few?"}
  F -- All --> G["Consumers too slow or downstream slow: scale or fix the dependency"]
  F -- Few --> H["Hot partition or poison message: inspect the stuck partitions"]
    

Symptoms

• Lag growing beyond the agreed limit for the group, or time-behind rising.
• Downstream data is late: dashboards stale, emails delayed, orders not processed.
• Group state flipping between stable and rebalancing.

Case A: the group is dead or empty

• Check whether the application is running, crash-looping, or blocked by a deployment.
• Check for authentication or authorization errors, such as expired credentials or a changed access rule.
• Restart or roll back the application. Lag will drain once it returns, so watch the drain rate and make sure downstream systems can handle the burst.

Case B: a rebalance storm

A rebalance pauses reading while partitions are reassigned among members. If members keep being kicked out and rejoining, the group spends its time reshuffling instead of working.

• Look for members leaving because processing a batch takes longer than the allowed poll interval.
• Look for missed heartbeats caused by long pauses, overloaded machines or network trouble.
• Check for rolling restarts or autoscaling that constantly changes membership.
• Mitigation: pause a rollout, reduce batch size or raise the poll interval for slow processing, use static membership and cooperative assignment where supported, and fix the slow dependency.

Case C: consumers are slow

• If all partitions lag equally, processing capacity is too low, or a downstream database or service is slow. Check that dependency first, since adding consumers to a struggling database makes it worse.
• Adding consumers helps only up to the number of partitions. Beyond that, extra members sit idle.
• If a few partitions lag, look for a hot key, a skewed partition or one very slow record.

Case D: a poison message

A single message that always fails processing can block a partition, because the consumer retries the same record forever.

• Identify the partition and offset where progress stopped, using the application’s error log.
• Route the record to a dead-letter topic with its error details, then continue.
• Only if the owner approves, skip past it by moving the group’s offset, after saving a copy of the record.

Verification

• Lag is falling at a rate that clears the backlog in an acceptable time.
• Group state stays stable and rebalances are rare.
• Downstream freshness is back within its target.

Choosing lag limits that mean something

Raw lag numbers are hard to judge. A lag of one million messages may be a few seconds on a busy topic and a whole day on a quiet one. Wherever possible, express lag as time: how old is the oldest unprocessed message? Then agree a limit with the owner of each application, such as “orders must be processed within two minutes”. Alerts based on time-behind produce fewer false alarms and speak directly to business impact.

Record in the runbook the normal processing rate of each important group, so that responders can estimate how long a backlog will take to clear. If lag is one hour and the group can process at twice real-time speed, the backlog clears in an hour, and this helps you decide whether scaling is worth the effort.

When applications cannot write, they see specific error messages. A runbook that maps each message to its meaning and first action saves valuable minutes.

General actions

1. Find out which applications and topics are affected, and since when.
2. Compare client errors with broker health. If brokers are healthy, the problem is likely in the client, credentials, network path or configuration.
3. Confirm the applications retry safely. With idempotent producers, retries do not create duplicates. Without them, be careful about “just retry”.
4. If producers have buffers or local queues, estimate how long they can hold data before dropping it. This is your deadline.

Client settings that turn outages into blips

Applications can be made much more forgiving. Sensible retry counts and delivery timeouts allow a producer to ride out a leader change without losing messages. Idempotent sending prevents duplicates when retries happen. Enough buffer memory absorbs short slowdowns. The runbook for producing teams should include a checklist of these settings, with recommended values, so that a brief broker event does not become an application incident.

The controller manages leaders and cluster metadata. When it struggles, many things go odd at once, even if each broker looks fine.

Symptoms

• No active controller, or more than one reported.
• Topic creation, deletion or leader elections hang or fail.
• Brokers reporting stale metadata, or leader changes that do not propagate.
• Repeated controller changes in a short time.

In the newer architecture with its own quorum (KRaft)

1. Check that a majority of controller nodes are alive and able to talk to each other. With three controllers, you need two; with five, three.
2. Use the metadata quorum tooling to see the leader, the followers and how far each lags.
3. If a controller is down, restore it. If a majority is down, the cluster cannot change metadata. Existing data traffic may continue for a while, but leader elections cannot happen.
4. Check disk space and latency on the controller nodes, since they store a log of all cluster changes.

In older ZooKeeper-based clusters

1. Check the health of the ZooKeeper ensemble: majority alive, low latency, no session expirations.
2. Look for brokers repeatedly losing and regaining sessions, often due to long garbage collection pauses or network hiccups.
3. Fix the coordination layer first. Restarting brokers will not help while it is unhealthy.

Verification

• Exactly one active controller, stable for a reasonable time.
• Metadata lag near zero. Topic operations work.
• Brokers all report the same cluster view.

Why controller problems spread widely

Because the controller decides who leads each partition, a slow or unstable controller delays every leader change in the cluster. During a broker failure, for example, thousands of partitions may need new leaders, and a controller that is short of CPU, or that is struggling to write its log, takes far longer to finish. That is why cluster-wide symptoms with no obvious broker fault deserve a look at the controller early. Record in the runbook how long a normal election burst takes on your cluster, so that a slow one stands out.

Sometimes brokers are healthy but cannot reach each other or their clients. This is one of the trickiest situations, because each side believes the other has failed.

Symptoms

• Several brokers in one zone or rack disappear from the view of others at the same moment.
• Sudden ISR shrinkage focused on partitions with replicas across the affected zone.
• Clients in one location see timeouts while clients elsewhere are fine.
• Cloud provider status pages or network dashboards report problems.

Quick checks

1. Which brokers are affected, and do they share a zone, rack, switch or subnet?
2. Are the machines themselves running? If yes, this is a network issue, not a crash.
3. Can you reach the affected brokers from different places, such as a bastion in each zone?
4. Is rack-aware replica placement configured, so each partition keeps copies in more than one zone? If yes, most partitions have surviving replicas.
5. Do the remaining brokers hold a majority of the controller quorum?

Actions

Do not

• Do not restart the isolated brokers hoping to fix the link.
• Do not move or delete data during the split.
• Do not enable unclean election unless a partition is truly stranded and approved.

Preparing for zone loss before it happens

The best time to handle a zone outage is months earlier, when you design the cluster. Spread each partition’s replicas across at least three zones where possible, and set the minimum in-sync replicas so that losing one zone leaves writes still accepted. Keep enough spare capacity that the remaining brokers can carry the extra load, which often means running each broker well below its limits during normal times. The runbook should state these assumptions explicitly, and a game day that simulates the loss of a zone proves whether they are really true.

This failure is embarrassing because it is completely predictable. A certificate has a date on it, and when that date arrives, connections fail all at once.

Symptoms

• Sudden simultaneous failures across many clients or between brokers at a specific time, often at midnight or on a round date.
• Handshake, trust or expired certificate messages in client and broker logs.
• ISR shrink across many partitions, if inter-broker links use expired certificates.
• Authentication errors after a password or credential rotation.

Quick checks

1. Read the exact error. Does it mention expiry, an unknown authority, a name mismatch or a bad password?
2. Check the dates on the broker certificate, the client certificate and the signing authority certificate.
3. Ask what changed: rotation job, new authority, new host names, a secret store update.
4. Is only one class of connection failing: clients, inter-broker traffic, or tools?

Actions

• Issue and install the new certificate through your standard process.
• Where your Kafka version supports it, update broker certificates dynamically without restarting. Otherwise, do a careful rolling restart, one broker at a time, waiting for full ISR between each.
• For a broken rotation, roll back to the previous secret if it is still valid.
• Tell client teams what to reload or restart, since many clients read certificates only at startup.

Preventing a repeat

• Alert on certificate expiry at sixty, thirty, fourteen and seven days.
• Keep an inventory of every certificate, owner and expiry date.
• Automate renewal and test it in a staging cluster.
• Make the runbook for renewal a routine task page, not just an emergency page.

Credentials other than certificates

Certificates are not the only secrets with a lifetime. Passwords, tokens and keys used by applications also expire or get rotated. A rotation that updates the server side but not every client leads to a wave of authentication failures. The runbook should list where each credential is stored, who rotates it, and how to roll back to the previous credential quickly. A safe rotation keeps both the old and the new credential valid for an overlap period, so clients can change over without a hard cut-off.

Nothing is down, yet everything is slow. Slow failures are hard to spot and hard to pin on a cause, so the runbook follows the path of a request from the client to the disk and back.

Symptoms

• Produce and fetch latency rising at the high percentiles.
• Request queues growing and request handler idle percentage falling low.
• Clients timing out occasionally; consumer lag creeping up.

Follow the request

Actions

1. Find out whether the slowness is on one broker or many. One broker means a local problem, such as a failing disk. Many means load or a shared dependency.
2. Look for a new heavy client: a big backfill, a consumer restarted from the beginning, or a burst of new connections.
3. Apply quotas to the noisy client while you talk to its owner.
4. Move leaders away from the struggling broker if it is a local problem.
5. If it is broad load, add capacity or reduce work, and schedule the longer-term fix.

Baselines make slowness visible

You cannot tell what is slow without knowing what is normal. Keep baseline values for latency, request rates, network use and disk latency at different hours and days of the week. A Monday morning peak might double the request rate every week without being a problem. Show the baseline beside the live graph on the dashboard, and mention typical values in the runbook, so responders can judge quickly. A number without context is just a number.

Some incidents are about the data itself rather than availability: deleted topics, corrupt files, missing offsets, or bad content. These need the most care, because mistakes can be permanent.

Accidental topic deletion

• Stop and assess. Deletion of a topic removes its data on all replicas, with no recycle bin.
• Check whether any replica or mirror cluster still holds the data. If a backup or mirror exists, restore from it.
• Recreate the topic with the same name and settings, so producers can continue, and let consumers know what was lost.
• Afterwards, restrict delete permissions and require review for destructive operations.

Corrupted log segments

• Symptoms include broker startup failures, checksum errors or repeated fetch failures on one partition.
• If other in-sync replicas exist, remove the bad replica and let it rebuild from a healthy copy.
• If it is the only copy, stop and escalate. Recovery tools exist but may lose data, so the owner must approve.
• Investigate the cause: failing hardware, filesystem issues or abrupt power loss.

Consumer offsets expired or lost

• If a consumer group was inactive longer than the offset retention period, its saved position is deleted, and it may restart from the beginning or the end depending on its setting.
• Decide with the application owner whether to reprocess or skip, and set the starting position deliberately.
• Prevent recurrence by aligning offset retention with the longest expected outage of any consumer.

Bad or malformed data in a topic

• Stop the producer or fix the source. Do not try to edit the log.
• Have consumers skip or quarantine bad records using a dead-letter topic.
• If the schema was changed incompatibly, roll it back and use the schema registry’s compatibility checks in future.

Recovering with backups and mirrors

Kafka replicates data between brokers, but replication is not a backup. If a topic is deleted, or bad data is written, every replica faithfully copies the mistake. Teams that care about recovery keep a second copy in a different place: a mirrored cluster, or an export of important topics to cheap object storage, or the original data in the systems that produce it.

• Write down, per topic, where the data can be rebuilt from and how long that takes.
• Rehearse restoring one small topic so the steps are proven, not theoretical.
• Record which topics are “source of truth” and which can be recreated from elsewhere. This shapes how much care each one deserves.

When the recovery point is different from the moment of failure, consumers need to be told. Some will see duplicates, and some will see gaps. The application runbooks should describe how each consumer reconciles its state, for example by checking totals against the source system.

Upgrades and configuration changes are a leading cause of incidents. The runbook must cover how to stop a bad rollout and how to go back.

Symptoms

• Errors started right after a broker was restarted on a new version or setting.
• The upgraded broker does not rejoin, or ISRs fail to recover between restarts.
• Clients report protocol or compatibility errors.

Actions

Before any change: the pre-flight list

• Cluster healthy: no offline or under-replicated partitions, one active controller.
• A rollback plan written down, with the exact previous version and configuration.
• A quiet time chosen, with no other big jobs running.
• Someone on standby who can approve a stop and who is not the person doing the work.
• Clients notified if brief errors or reconnections might occur.

Changes to configuration deserve the same care as software upgrades. A setting that is harmless on one broker can be disastrous when applied to every broker at once. Apply changes to one broker first, watch the health signals for a full cycle of your traffic pattern, and only then continue.

Real Kafka platforms include more than brokers. Connectors, schema services and cross-cluster mirrors break too, and users often see those failures first.

Kafka Connect

• Symptoms: a connector or task shows a failed state, data stops flowing to or from an external system.
• Checks: read the task error, check the external system and credentials, check worker health and rebalances, check whether a message format changed.
• Actions: fix the cause, then restart the failed task. Use error handling settings and dead-letter topics to keep one bad record from stopping a pipeline.

Schema Registry

• Symptoms: producers or consumers fail to serialize or deserialize; errors about unknown schema identifiers or compatibility.
• Checks: is the registry service up, reachable and using a healthy storage topic? Did a schema change break compatibility?
• Actions: restore the service, then fix or revert the schema. Clients usually cache schemas, so already-running applications may keep working for a while, which gives you time.

Mirroring between clusters

• Symptoms: replication lag between clusters rises, or topics are missing in the target.
• Checks: health of the mirroring workers, network between sites, permissions on both clusters, and quota limits.
• Actions: restore the workers or link, then watch the lag drain. If failing over to the second cluster, follow a separate, rehearsed disaster recovery plan and remember that consumer positions need translating between clusters.

A runbook that a person reads and follows is good. A runbook that a tool can partly run for them, with safety checks, is better. The goal is to automate the boring and repetitive steps while keeping humans in charge of decisions.

The automation ladder

flowchart LR
  A["Alert fires"] --> B["Automatic evidence gathering"]
  B --> C{"Is the fix safe and known?"}
  C -- Yes --> D["Dry run shows the planned change"]
  D --> E{"Approval needed?"}
  E -- Yes --> F["Human confirms"]
  E -- No --> G["Apply the change"]
  F --> G
  G --> H["Verify health checks"]
  C -- No --> I["Page a human with the evidence"]
  H --> J["Record everything in the incident log"]
    

Guardrails

• Dry run first. Show what would change before changing it.
• Scope limits. A tool that acts on one named topic or one broker cannot accidentally act on all of them.
• Rate limits and cool-downs. No more than one broker restart per period, and never while another is recovering.
• Health gates. The tool refuses to proceed unless the cluster is healthy enough.
• Approvals for destructive actions. Deletion, unclean election and offset resets always need a human.
• Full logging. Who or what did what, when, and with what result.
• An off switch. Any automation can be paused quickly.

What to automate first

Begin with tasks that are frequent, boring and low-risk, because these give the quickest benefit and the least chance of harm. Gathering evidence is the perfect starting point: a single command that prints the offline partitions, the under-replicated partitions grouped by broker, the controller identity and the recent changes saves five minutes on every incident. Next come read-only reports and dashboards linked from alerts. Only later add actions that change the cluster, and always start those with a dry-run mode.

Measure whether automation helps. If responders ignore the tools because they are slow, confusing or often broken, the effort is wasted. Treat the tools as products with users, collect their feedback, and keep them as well maintained as the runbooks they support.

An untested runbook is a guess. Testing finds missing steps, outdated commands and unclear wording while the stakes are low.

How to run a good game day

1. Pick one scenario from the library and define what success looks like.
2. Tell stakeholders, choose a safe time, and prepare a stop signal that ends the test immediately.
3. Have someone play the responder who follows the runbook, and someone else observe and take notes.
4. Time the key steps: detect, diagnose, mitigate, verify.
5. Afterwards, list every place where the runbook was wrong, missing or confusing, and fix each one within a week.

Systems change constantly, and runbooks that do not change with them turn from helpful to harmful. Treat them like software, with owners, reviews and a release process.

Ownership and review

• Every entry has a named owning team and a last reviewed date shown at the top.
• Review each entry at least twice a year, and immediately after any major change to the platform.
• Mark entries as verified when they have been tested, and mark them draft otherwise, so readers know how much to trust them.
• Keep the runbooks in version control with pull requests, so changes are discussed and traceable.

The post-incident loop

Every incident should produce runbook changes. In the review, ask three questions: did the runbook exist, was it found, and was it correct? Each “no” becomes a task with an owner and a date. Over time, this loop is the most powerful source of quality.

Measures worth tracking

• Track how often responders leave the runbook to improvise. High rates show gaps.
• Track repeat incidents. The same problem twice means the fix did not stick.
• Collect quick feedback from on-call staff after each use: helpful, partly helpful, or wrong.

The lifecycle of an entry

Keeping on-call staff healthy is part of keeping runbooks alive. If people are woken often by noisy alerts, they stop reading carefully. Review the pages each week: which alerts fired, which were useful, which were ignored. Every alert that cannot be acted upon is either a candidate for deletion, or a sign that a runbook entry is missing.

Large technology companies that run Kafka at scale, including LinkedIn, where it was created, Netflix, Uber and many banks and retailers, have talked publicly about the importance of clear operational procedures and rehearsal. Exact details of their internal runbooks are not public, so the stories below are illustrative composites showing how the ideas play out.

What these stories have in common

• The entry could be found quickly from the symptom.
• The decision rules were agreed before the crisis.
• Someone followed a check-then-act approach.
• The incident review improved the documents.

Some approaches work again and again, and others fail predictably. Knowing both saves you from learning the hard way.

Patterns that work

Anti-patterns

Here is a condensed checklist you can use to review your own library.

A short review checklist

1. Can a new on-call engineer find this entry from the alert in under a minute?
2. Does it say how bad this is and when to escalate?
3. Are the first checks safe and read-only?
4. Does each action have a check to confirm it worked, and an undo?
5. Are dangerous steps flagged and approvals named?
6. Was it tested in the last six months?
7. Does it link to background material rather than repeating it?

Writing style that helps under stress

• Use short imperative sentences. “Check the number of offline partitions” is better than “You may wish to consider examining offline partitions”.
• One action per step. If a step contains “and then”, split it.
• Say what a good result looks like. “The count should be zero. If not, go to step seven.”
• Avoid ambiguous words. Replace “soon”, “large” and “some” with numbers and times.
• Give exact names. Use the real dashboard title, the real topic name and the real channel, so nobody has to guess.
• Use the same words everywhere. If one page says “broker” and another says “node”, readers wonder if they are different things.

Finally, remember accessibility: many alerts arrive on a phone. Entries should be readable on a small screen, with short lines and simple tables, and should not depend on colours alone to convey meaning.

A runbook is a way of turning experience into a form that helps anyone, at any hour. For Kafka, a good runbook library covers the common failures, makes the safe path obvious, and keeps improving with every incident.