Some people believe a well-run cluster should never need restarting. In reality, restarts are a regular part of life, and the only choice is whether they are painful or boring.

Apache Kafka is a system that stores and moves streams of events between programs. It runs as a cluster of servers called brokers. Like any software running on real machines, brokers need attention over time. Here are the everyday reasons a broker has to stop and start again.

What goes wrong when restarts are done badly

A careless restart can cause real harm. If all brokers are stopped at once, the cluster is simply down. If brokers are stopped too quickly one after another, some data may exist on only one server for a while, which is risky. If clients are not prepared, they may throw errors, lose messages, or trigger long delays while they rediscover where their data lives. Teams that fear restarts postpone patching, and postponed patching turns into a security problem.

Before we chase zero downtime, we must agree on what it means. The phrase hides several different promises.

When people say “no downtime”, they usually mean that users and applications do not notice. But there are levels of not noticing, and each needs a different amount of care.

Most teams aim for the first three: the cluster stays available, clients do not fail, and no acknowledged data is lost. Tiny latency bumps are accepted as the price. You should decide, before you start, which of these levels your business needs, and write it down. It affects how carefully you pace the work and how much spare capacity you keep.

How to measure it

Decide on a few numbers before the maintenance window. For example: producer error rate stays at zero, the slowest one percent of produce requests stay under two seconds, no consumer group falls more than a minute behind, and no partition is ever offline. If any number is crossed, you pause the restart and investigate. Without such numbers, “no downtime” is a feeling, not a fact.

To restart brokers safely, you must understand who holds what. This short refresher covers exactly the parts that matter.

Topics and partitions

Kafka stores events in named channels called topics. Each topic is split into partitions, which are ordered logs. Partitions let Kafka spread a topic across many brokers, so no single machine has to carry all the traffic.

Replication

To survive machine failures, every partition is copied to several brokers. The number of copies is the replication factor. A common choice is three. Each copy is called a replica. One replica of each partition is the leader. All reads and writes for that partition go through the leader. The other replicas are followers, and they continuously copy data from the leader.

The in-sync replicas

A follower that is keeping up with the leader is called in-sync. The set of in-sync replicas, including the leader, is the ISR. Only members of the ISR can be promoted to leader without risking data loss. A follower that falls too far behind, by default more than about thirty seconds, is dropped from the ISR until it catches up.

Leaders are spread out

Kafka balances leadership so each broker leads roughly the same number of partitions. That means every broker is the leader for many partitions at once, and a follower for many others. When one broker stops, it takes the leadership of many partitions with it, and those partitions need new leaders. Everything about zero-downtime restarts comes down to moving that leadership smoothly.

Three settings decide how many brokers can be down at once without harm. Get them right before you ever restart anything.

Replication factor

This is the number of copies of each partition. With a replication factor of one, restarting the broker that holds the partition makes it unavailable for the whole restart. That cannot be rolled without downtime. With three copies, two other brokers can take over.

Minimum in-sync replicas

The setting min.insync.replicas is the smallest number of in-sync copies that must exist before Kafka accepts a write from a producer that asks for full safety. If the ISR shrinks below this number, writes are refused instead of risking a single fragile copy.

Acks

A producer chooses how many acknowledgments it wants. With acks=all, the leader waits until every in-sync replica has the message before confirming. That, combined with the minimum ISR rule, gives the strongest promise: an acknowledged message lives on at least the minimum number of brokers.

The golden rule

With a replication factor of three and a minimum ISR of two, you can restart exactly one broker at a time and keep accepting safe writes. The key words are one at a time. The next broker may only go down after the previous one has fully returned and every partition is back to three in-sync copies.

Check the internal topics too

Kafka keeps its own bookkeeping in internal topics, such as the topic that stores consumer offsets. Their replication factor must also be three. A small test cluster that was created with a replication factor of one for these topics will lose its consumer positions or coordinator service during a restart.

A restart is not just one broker stopping. Several pieces react. Here is the full picture.

flowchart LR
  subgraph Clients["Client Applications"]
    P["Producers"]
    C["Consumers in groups"]
    S["Streams and Connect apps"]
  end
  subgraph Cluster["Kafka Cluster"]
    B1["Broker 1"]
    B2["Broker 2"]
    B3["Broker 3"]
  end
  subgraph Control["Controller Quorum"]
    K1["Controller 1 active"]
    K2["Controller 2"]
    K3["Controller 3"]
  end
  subgraph Ops["Operations"]
    ORC["Restart orchestrator"]
    MON["Monitoring and alerts"]
  end
  P -->|"produce to leaders"| B1
  P --> B2
  C -->|"fetch from leaders"| B2
  C --> B3
  S --> B1
  S --> B3
  B1 |"replicate"| B2
  B2 |"replicate"| B3
  B1 |"replicate"| B3
  B1 -.->|"heartbeat and metadata"| K1
  B2 -.-> K1
  B3 -.-> K1
  K1 --- K2
  K1 --- K3
  ORC -->|"stop, start, wait"| B2
  MON -->|"health checks"| ORC
    

Who does what during a restart

The difference between a clean stop and a crash is the heart of this guide. Kafka has a built-in way to leave gracefully.

When you ask a broker to stop politely, it does not just switch off. It performs a controlled shutdown. That is a short routine in which the broker first tells the cluster “I am leaving, please move my leaders elsewhere”, waits until that is done, and only then closes. This setting is turned on by default in Kafka.

sequenceDiagram
  autonumber
  participant Op as Operator
  participant B as Broker being restarted
  participant Ctl as Active Controller
  participant F as Other Brokers
  participant Cl as Clients
  Op->>B: Send polite stop signal
  B->>Ctl: Request controlled shutdown
  Ctl->>Ctl: Pick new leaders from in-sync replicas
  Ctl->>F: New leader assignments
  F-->>Ctl: Leadership taken over
  Ctl-->>B: All leaders moved, safe to stop
  B->>B: Flush data, write clean shutdown marker
  B->>Ctl: Deregister and close connections
  Cl->>F: Refresh metadata, send requests to new leaders
  F-->>Cl: Normal responses
    

The steps in words

Why the clean marker matters

When a broker starts again, it checks whether it stopped cleanly last time. If the clean marker is present, it can trust its data files and start almost at once. If not, it must scan its log segments to check for damaged data and rebuild indexes. On a broker with terabytes of data, that recovery can take many minutes, and the broker cannot serve during that time. This is the biggest practical reason to avoid hard kills.

The brokers do the handover, but the clients are the ones who could get hurt. Here is what they experience, and how they recover.

sequenceDiagram
  autonumber
  participant P as Producer
  participant Old as Old Leader Broker
  participant New as New Leader Broker
  P->>Old: Send batch for partition 7
  Old-->>P: Not the leader anymore
  P->>P: Mark metadata stale, wait a short backoff
  P->>New: Ask for fresh cluster metadata
  New-->>P: Partition 7 leader is now this broker
  P->>New: Resend the same batch
  New->>New: Write and replicate to followers
  New-->>P: Acknowledged
    

The producer’s experience

A producer keeps a map of which broker leads each partition. When leadership moves, that map goes stale. The next request to the old leader gets a “not the leader” reply. The producer’s client library quietly refreshes its map from any broker and resends the batch to the new leader. If the client is set up well, the application code sees nothing but, at worst, a slightly slower send.

Producers also retry when a connection drops, which happens when a broker closes at the end of its shutdown. Retries are automatic and controlled by a total time limit, called the delivery timeout. As long as the leadership move finishes inside that limit, the send succeeds.

The consumer’s experience

A consumer fetches from partition leaders in the same way, and it refreshes its map in the same way. There is one extra thing. Each consumer group has a group coordinator, a broker that tracks members and their positions. The coordinator for a group is simply the leader of a partition of an internal offsets topic. If the coordinator’s broker restarts, the coordinator role moves too, and consumers must find the new one.

How long does the blip last?

In a healthy cluster, the leadership handover for a partition takes tens to hundreds of milliseconds. A broker with a huge number of partitions takes longer overall, because each must be moved, but any individual client only cares about the partitions it uses. Most applications see a delay of well under a second, if they see anything at all.

A restart is not finished when the process is running again. The broker is still empty-handed and behind. The next steps are just as important.

flowchart TD
  S["Broker process starts"] --> M{"Clean shutdown marker present?"}
  M -- "Yes" --> L["Load logs quickly"]
  M -- "No" --> R["Log recovery: scan segments and rebuild indexes"]
  R --> L
  L --> REG["Register with controller"]
  REG --> F["Become follower for its partitions"]
  F --> CU["Fetch missed data from leaders"]
  CU --> ISR{"Fully caught up?"}
  ISR -- "No" --> CU
  ISR -- "Yes" --> J["Rejoin the ISR"]
  J --> PL["Preferred leader election returns leadership"]
  PL --> OK["Broker fully back in service"]
    

Step 1: load or recover the logs

The broker opens its data directories. After a clean stop, it loads the log files quickly. After a crash, it checks the tail of each partition for damaged data and rebuilds indexes. A setting controls how many threads do this work in parallel per disk. More threads shorten recovery on machines with many partitions.

Step 2: register and become a follower

The broker tells the controller it is back. Because other brokers took over leadership, this broker starts out as a follower for all its partitions, not a leader.

Step 3: catch up

While the broker was away, leaders kept receiving new messages. Now it must fetch all those missing messages. The amount depends on how long it was down and how fast data arrives. During this period, the broker is not in the ISR for those partitions, so those partitions are under-replicated. That is normal and expected during a restart, but it is exactly the state that must be fully resolved before touching the next broker.

Step 4: rejoin the ISR

Once a follower has caught up to the leader, it is added back to the ISR. Now each partition again has its full set of in-sync copies.

Step 5: get leadership back

Each partition has a preferred leader, the first replica in its list, chosen when the topic was created to spread leadership evenly. Kafka can check periodically and hand leadership back to preferred leaders when they are in sync. By default this happens automatically every few minutes if the imbalance passes a threshold. You can also trigger it yourself. Restoring balance matters, because otherwise the restarted broker sits nearly idle while others carry extra load.

Now we combine the pieces into a repeatable loop. The loop is simple, and its discipline is what makes it safe.

flowchart TD
  A["Start maintenance window"] --> PRE{"Pre-flight checks pass?"}
  PRE -- "No" --> ABORT["Stop and fix first"]
  PRE -- "Yes" --> PICK["Pick the next broker"]
  PICK --> STOP["Controlled shutdown"]
  STOP --> WORK["Apply change: patch, config, upgrade"]
  WORK --> START["Start broker"]
  START --> W1{"Broker registered with controller?"}
  W1 -- "No, timeout" --> HALT["Halt and investigate"]
  W1 -- "Yes" --> W2{"Under-replicated partitions equal zero?"}
  W2 -- "No, still catching up" --> W2
  W2 -- "Stuck too long" --> HALT
  W2 -- "Yes" --> W3["Optionally rebalance preferred leaders"]
  W3 --> CH{"Client errors and lag within limits?"}
  CH -- "No" --> HALT
  CH -- "Yes" --> MORE{"More brokers left?"}
  MORE -- "Yes" --> PICK
  MORE -- "No" --> DONE["Finish and record results"]
    

The three gates

Why not restart faster?

Suppose you restart broker two the moment broker one’s process is running, but before broker one has caught up. Now some partitions have one copy that is out of sync, one that is down, and only one healthy. With a minimum ISR of two, those partitions stop accepting writes. Worse, if the lone healthy copy fails, data could be lost. Patience between brokers is the cheapest insurance you will ever buy.

Why not slower?

Waiting for perfection has costs too. The longer the maintenance window, the longer the cluster runs with reduced fault tolerance. If another broker fails unexpectedly during the window, you are exposed. A good loop is thorough between brokers and efficient in everything else.

Pilots run through a checklist before every flight, however many times they have flown. Do the same before every restart window.

A rolling restart assumes the cluster starts in good health. If something is already wrong, a restart can turn a small problem into a big one. Spend ten minutes checking before you touch anything.

Find the fragile topics

Before the window, list every topic with a replication factor below three, or a minimum ISR that equals the replication factor. These are the topics that will suffer during a rolling restart. Some may be forgotten test topics you can delete. Others may be real, and need a plan: either raise their replication before maintenance, or warn their owners.

Tell people

Announce the window to the teams that use the cluster. Not because you expect trouble, but so that if a team sees a small blip, they know why. Share the time, the expected effect, and a contact. This also gives teams a chance to say “please do not touch us today, we have a launch”.

Have a way back

For a plain restart, going back means simply starting the broker again. For an upgrade or a configuration change, going back means restoring the old version or settings. Keep the old package, the old configuration files, and a written procedure ready.

A well-configured producer barely notices a restart. A poorly configured one loses data or fails loudly. These are the settings that decide which you get.

Why acks matter so much

With acks=1, the leader confirms as soon as it has written the message itself, before any follower has it. If that leader stops before the followers copy the message, and a follower becomes leader, the message is gone, even though the producer was told it succeeded. Controlled shutdown makes this less likely but never impossible. With acks=all and a minimum ISR of two, an acknowledged message is on at least two brokers, so losing any one broker cannot lose it.

Handling errors in code

Even with perfect settings, a send can fail if the outage outlasts the delivery timeout. Application code should handle failure callbacks: log them, retry at a higher level, or park the message in a fallback place. Never ignore the result of a send.

Consumers are more delicate than producers, because they work in teams. A restart that shakes the team can cause a pause called a rebalance.

What is a rebalance?

Consumers in a group split up the partitions of a topic. Whenever the membership changes, or the group thinks someone has left, the partitions must be reassigned. This reassignment is called a rebalance. In older styles of rebalance, every consumer stops reading while the new plan is worked out. On a big group, that pause can last many seconds or even minutes.

How a broker restart can trigger a rebalance

Consumers send regular heartbeats to their group coordinator. If the coordinator’s broker restarts, heartbeats fail for a moment while the role moves. If the delay is longer than the session timeout, the group thinks its members are dead and starts a rebalance. So two things matter: making the timeout comfortably longer than a normal coordinator move, and making rebalances cheap when they do happen.

Commit offsets carefully

A consumer’s position in each partition is saved as an offset. If a commit fails during a restart, the consumer may re-read a few messages after recovery. This is called at-least-once processing. To be safe, make your processing idempotent, which means applying the same message twice gives the same result. Then a repeated message is harmless.

Higher-level tools have their own behavior when brokers restart. Know it before you begin.

Transactions

Kafka supports transactions, which let a producer write to several partitions and have them appear all together or not at all. Transactions are managed by a transaction coordinator, which, like the group coordinator, lives on a broker and moves when that broker restarts. Producers see temporary errors, such as “coordinator not available”, and retry. A transaction in progress during the move might be aborted if it runs longer than the transaction timeout, so keep transactions short. Consumers that read only committed messages will simply skip aborted ones.

Kafka Streams

Streams applications are built on the ordinary producer and consumer, and they add local state stores. They are designed to ride through broker restarts, retrying for a configured time before giving up. Set that retry window longer than your typical broker restart. If a Streams application’s own instance restarts, it must rebuild its local state from a change log topic, which can take a while for big state. Standby replicas keep a warm copy and shorten this wait.

Kafka Connect

Connect runs connectors as tasks spread over workers. When a broker restarts, Connect’s internal topics briefly move leaders, and connectors retry. When a Connect worker itself restarts, its tasks move to other workers. A delay setting gives a restarted worker time to come back before its tasks are permanently reassigned, so you avoid a reshuffle for a short restart. That delay is a few minutes by default.

Brokers hold data. Controllers hold the cluster’s brain. Restarting the brain needs its own care.

The controller decides who leads which partition, and remembers the list of topics, settings, and brokers. In modern Kafka, this job is done by a small group of controller nodes running a built-in consensus system called KRaft. The nodes agree on every change by majority vote, so the cluster’s brain survives the loss of a minority of them. Older Kafka versions used a separate service called ZooKeeper for the same purpose. Recent major releases have removed it entirely.

The majority rule

A quorum of three controllers can lose one and keep working. A quorum of five can lose two. Lose the majority and the cluster can no longer change leaders, create topics, or react to broker failures. Existing traffic often keeps flowing for a while, but the cluster is frozen and fragile.

Rolling controllers

Combined and separate nodes

Some small clusters run controller and broker roles on the same machines. Then a single restart affects both the data plane and the control plane at once. That is workable with three or more nodes, but it makes each step heavier. Larger clusters usually keep controllers on dedicated machines, which makes restarts of each kind independent and easier to reason about.

If your replicas are spread across failure zones, you can restart a whole zone at once, which turns a long window into a short one.

Every broker can be labeled with the rack or availability zone it lives in, using the setting broker.rack. When this is set, Kafka places the replicas of each partition in different racks whenever it can. With three racks and a replication factor of three, every partition has exactly one copy in each rack.

flowchart LR
  subgraph RA["Zone A"]
    A1["Broker 1"]
    A2["Broker 4"]
  end
  subgraph RB["Zone B"]
    B1["Broker 2"]
    B2["Broker 5"]
  end
  subgraph RC["Zone C"]
    C1["Broker 3"]
    C2["Broker 6"]
  end
  P1["Partition 0 replicas"] --> A1
  P1 --> B1
  P1 --> C1
  P2["Partition 1 replicas"] --> A2
  P2 --> B2
  P2 --> C2
    

The zone-at-a-time approach

With this layout, restarting all brokers in Zone A at the same time removes exactly one replica of every partition. The minimum ISR of two is still met by Zones B and C, so writes continue. You can restart a whole zone in one step, wait for it to catch up, and then do the next zone. For a large cluster, three zone-sized steps replace dozens of single-broker steps.

The catch is that the surviving zones must have the capacity to absorb the load of the missing zone: about one and a half times their normal traffic for a three-zone layout. Confirm you have that headroom before using this approach.

Many settings can be changed on a running broker. Learning which ones saves you a surprising number of restarts.

Kafka has three kinds of broker settings. Read-only settings are read once at start-up, so changing them needs a restart. Per-broker settings can be updated on a running broker, one broker at a time. Cluster-wide settings can be updated for all brokers in a single command. Kafka stores dynamic values in its own metadata and applies them without stopping the process.

Certificates without restarts

Security certificates expire on a schedule, and forgetting one is a classic cause of outages. Kafka can reload a listener’s key store and trust store while running, when they are updated through the dynamic configuration mechanism. That lets you rotate certificates broker by broker, without a restart, and with no interruption to clients. It is worth setting this up long before the first certificate is close to expiring.

Change one broker first

Even with dynamic settings, apply a per-broker change to one broker first, watch it for a few minutes, and only then apply it cluster-wide. A bad value that takes effect everywhere at once can do more harm than any restart.

An upgrade is a rolling restart where each broker comes back running newer software. It adds one careful idea: keep the old and new versions compatible until everyone has moved.

During an upgrade, the cluster runs a mix of old and new brokers for some time. To make this safe, new brokers are told to keep speaking the old dialect of Kafka’s internal protocol until every broker is upgraded. Only then do you switch the whole cluster to the new dialect. In older versions, this was controlled by a setting for the inter-broker protocol version. In current versions, it is controlled by a cluster-wide metadata version that you raise with a special command.

Why the version step is separate

The step that raises the cluster-wide version is the point of no return. Before it, you can still stop and put the old software back. After it, some data formats or metadata may have changed. Separating the software swap from the version switch gives you a safe soak period in between. Use it.

Containers make restarts easy to trigger and easy to get wrong. A few Kubernetes settings decide which.

On Kubernetes, Kafka brokers usually run as a StatefulSet, a kind of workload that gives each pod a stable name and its own persistent disk. When you change the pod definition, Kubernetes replaces pods one at a time, in reverse order. That is a rolling restart already. But by itself, it only waits for each pod to be ready. It does not know whether Kafka’s partitions are fully replicated.

The settings that matter

Operators that understand Kafka

Kafka operators, such as those from the Strimzi project and commercial vendors, take over the restart order. A good one restarts brokers one at a time, checks that a restart will not push any partition below its minimum ISR before it acts, waits for replicas to catch up after each restart, and handles controllers before brokers. It can also decide which pods truly need a restart and skip the rest. For most teams running Kafka on Kubernetes, using an operator is the safest path.

Common Kubernetes traps

Outside Kubernetes, the loop is yours to build. The good news is that the logic is short and clear.

Most teams use a configuration or orchestration tool to run the loop across hosts. Whatever tool you use, it should do these things, in this order, for each broker.

Design principles for the automation

Instead of letting the shutdown move leaders in a rush, you can move them out gently beforehand. It makes the actual stop almost invisible.

Controlled shutdown moves leadership at the moment of stopping. For brokers with a very large number of partitions, that can take a while and produces a burst of leader changes. A gentler approach is to drain the broker first: move leadership away from it in the background, while it is still healthy, in small batches. When you finally stop it, it leads nothing, so nothing has to move.

How teams do it

Kafka’s tools can reassign partition leadership by changing the order of replicas or by running a preferred leader election after the order has changed. Cluster management tools such as Cruise Control offer a “demote broker” action that moves all leadership off a chosen broker in a controlled, throttled way, making it a follower only. After maintenance, the same tool can restore the balance.

Draining is most useful on large brokers, on brokers that serve latency-sensitive topics, and on clusters where you want extra caution. For small clusters, controlled shutdown alone is usually enough.

How long should each step take? A little arithmetic turns guesswork into a schedule.

How long will catch-up take?

When a broker returns, it must fetch everything that was written while it was away. The amount depends on the rate of new data and the length of the outage. Consider a broker that holds copies of partitions receiving 30 MB per second in total. If it is down for 10 minutes, that is 600 seconds times 30 MB, or about 18 GB of data to catch up. If the broker can replicate at 200 MB per second while still handling normal traffic, catching up takes about 90 seconds. If its share is bigger, or the network is slower, the time grows in proportion.

The lesson: the faster the restart itself, the less there is to catch up. Patching that takes 30 minutes means 54 GB to catch up in the same example. Do slow work such as long file operations before stopping the broker, and keep the stopped time short.

The cold cache problem

Kafka leans on the operating system’s page cache, memory that keeps recently used file data. Consumers reading recent messages are served straight from memory, which is very fast. After a restart, or a reboot of the machine, that cache is empty. Reads must go to the disk until the cache warms up. If a restarted broker immediately becomes leader for many partitions, consumers reading through it may see slower responses for a while.

This is one more reason to avoid rebooting whole machines when a process restart would do, and to give the cache a little time to warm up before you restart the next broker.

Throttling the catch-up

Catch-up traffic competes with client traffic. A broker fetching 18 GB at full speed can saturate the network and slow everyone. Kafka lets you limit replication bandwidth so that catch-up runs at a gentle speed. The trade-off is that the cluster stays in the reduced-safety state for longer. Choose the rate based on how much headroom you have.

A rolling restart is a live operation. Watch the right dashboards, and you will see trouble before your users do.

Cluster health signals

Client-facing signals

Sooner or later a restart will not go to plan. Knowing the common failures in advance turns a crisis into a checklist.

The recovery mindset

A quick lookup table for the situations you will most often meet during a window.

Maintenance windows are also when security slips. Keep the walls up while the brokers come and go.

Security upgrades as restarts

Rolling restarts are often driven by security work, for example moving to a stronger encryption setting or a new login method. Add new options first and let clients adopt them while the old ones still work. Once every client has moved, remove the old option in a later window. This two-step pattern avoids locking out applications that are slower to change.

Some changes cannot be rolled through a single cluster. For those, you need another cluster to stand in.

Certain big changes, such as moving to a different storage layout, changing the number of controller nodes, or swapping very old infrastructure, are safer or only possible with a second cluster. The idea is simple: build the new cluster next to the old one, copy data across, move applications over gradually, and retire the old one.

Disaster recovery as a safety net

If your business cannot tolerate even a small risk during a major upgrade, a standby cluster in another place gives you a way out. If the upgrade goes badly, you switch applications to the standby, which is on the old version. It costs more, and it is the strongest insurance available.

Rolling restarts are the standard, and for good reason. Still, an honest look at both sides helps you plan.

The core trade-offs

Shapes that work, and shapes that look fine until the day you need to restart.

Patterns worth copying

Anti-patterns to avoid

A short list of habits that separate calm maintenance windows from stressful ones.

Best practices

Common mistakes

You do not know whether your cluster can survive a restart until you try it, in a place where a mistake costs nothing.

A game day is a planned practice session. The team gathers, chooses a failure or a maintenance step, runs it on a test cluster or, with care, in production, and watches what happens. The goal is not to pass. The goal is to learn what breaks while everyone is calm and ready.

What to test

How to run a good game day

A runbook is a checklist that a tired person can follow at night. Here is a shape you can copy and shorten to fit your cluster.

Before the window

For each broker

After the window

Let us follow a made-up company through one full window, so the pieces click together.

Imagine an online bookshop called PageTurn. It runs a nine-broker Kafka cluster spread over three zones, with three brokers in each zone. Every important topic has a replication factor of three and a minimum ISR of two. Producers use acknowledgement from all in-sync replicas with idempotence on. Consumers use static membership and cooperative rebalancing. A tiny canary program writes and reads a message every second and reports the delay.

The team needs to apply a security patch to the operating system, which requires a reboot of every machine. They choose a Tuesday morning, when orders are at their daily low.

Nothing here was heroic. The safety came from settings chosen months earlier, gates that refused to hurry, and a team that treated a slow broker as a reason to wait rather than a reason to push.

How different kinds of teams tend to organize this work in practice.

Different as they look, all of these share the same three ideas: keep enough copies, move one failure domain at a time, and let the cluster’s health, not a timer, decide when to continue.

Plain-language reminders of the words used throughout this guide.

Short, direct answers to the questions people ask most often.

Everything above fits on one page in your head.

A rolling restart works because Kafka keeps several copies of every partition. When one broker leaves politely, its leadership moves to a copy on another broker within moments. When it returns, it catches up from those copies and rejoins the in-sync list. Do this for one failure domain at a time, and let the cluster’s own health decide when to take the next step, and the service never stops.