Before we talk about shrinking, we need to know what is being shrunk. The idea is simple once you see it as a team of copy-keepers.

Copies of your data

Apache Kafka stores streams of events in named categories called topics. Each topic is divided into partitions, and each partition is a separate ordered log. To protect against machine failures, every partition is stored as several copies, called replicas, on different servers called brokers. The number of copies is the replication factor, and three is the most common choice.

One replica is chosen as the leader. Producers write to it and consumers read from it. The other replicas are followers. A follower has one job: keep asking the leader for new data and add it to its own copy, so that it stays identical.

The in-sync group

Not every follower is always up to date. A follower might be slow, disconnected or restarting. Kafka keeps a list of the replicas that are fully caught up, and calls it the in-sync replica set, or ISR. The leader is always in it. A follower is in it only while it keeps pace.

Why the list is so important

• Safety of acknowledged writes. When a producer asks for full confirmation, Kafka waits until every replica in the ISR has the data. Being in the ISR means “I hold everything that was confirmed”.
• Safe leadership. If the leader dies, the new leader is normally chosen only from the ISR, since only those replicas are guaranteed to have all confirmed data.
• Honest reporting. The size of the ISR tells you, at any moment, how many safe copies of a partition actually exist.

What “shrinkage” means

ISR shrinkage means the list gets shorter: the leader decides a follower is no longer keeping up and removes it. With three copies, the ISR drops from three members to two, or to just the leader alone. When the follower catches up again, the list expands back. A little shrinking and expanding is a normal part of life. Widespread, simultaneous shrinking is a warning that something serious is happening.

The rule is smaller and simpler than most people expect. Understanding it precisely removes a lot of confusion during an incident.

The follower’s loop

A follower runs a constant loop. It sends a fetch request to the leader saying, “Here is the position I have reached. Please send anything newer.” The leader replies with new data, and the follower adds it to the end of its log and asks again. The follower’s position is called its log end offset, and the leader remembers the latest one it heard from each follower.

sequenceDiagram
  participant F as Follower
  participant L as Leader
  participant C as Controller
  F->>L: Fetch new data from my offset
  L-->>F: Send data and current high watermark
  F->>F: Append data to local log
  Note over L: Leader tracks how recently each follower was caught up
  L->>L: Follower silent or behind for too long
  L->>C: Ask to shrink the in-sync set
  C-->>L: Approve and record the new set
    

The time-based rule

The leader does not count how many messages a follower is behind. It watches the clock. A setting called replica.lag.time.max.ms gives the maximum time allowed. If a follower has not fetched, or has not caught up to the leader’s latest data, for longer than this limit, the leader removes it from the ISR. In recent Kafka versions the default is thirty seconds, though older versions used ten, so check your own setup.

This detail matters a great deal. A follower that is slightly behind for a moment is fine. A follower that is continuously behind, or silent, for that whole time window is out. Anything that stops many followers from fetching and catching up for thirty seconds will shrink many ISRs at once.

The high watermark

The high watermark is the offset up to which all in-sync replicas have the data. Consumers can only read up to this point, because data beyond it could still be lost in a failover. When a producer asks for full confirmation, its request waits until the high watermark passes its message. So a single slow follower who is still in the ISR holds back the high watermark for the whole partition, and delays every writer waiting for full confirmation.

Who changes the list?

The leader decides, but the decision must be recorded in a central place so that everyone agrees. In modern Kafka, the leader sends a request to the controller, the part of the cluster that manages metadata, asking it to change the ISR. The controller records the new list and tells the other brokers. In older setups, the leader wrote the change to ZooKeeper directly. Either way, if this recording step is slow or fails, ISR changes can be delayed or rejected, which is one of the causes covered later.

The expand side

When a removed follower fetches all the way up to the leader’s high watermark, the leader asks to add it back. This rejoining is the expansion. If a follower keeps falling behind and catching up in cycles, you see repeated shrink and expand events, which people call ISR flapping.

Some shrinkage is healthy. The skill is telling everyday ripples from a wave that needs action.

Expected shrinkage

Certain events legitimately make followers drop out of the ISR for a while:

• A broker is restarted for an upgrade or patch. Every partition it hosts loses a replica from the ISR until it returns and catches up.
• A new replica is added during a partition move. It starts far behind and is not in the ISR until it catches up.
• A brief network blip makes one follower miss its window.

These are usually limited, short and explainable. You know why they happened, and they end by themselves.

The signature of a storm

The numbers to watch

One partition losing a follower is a local repair job. Thousands of partitions losing followers together is a different kind of event, and it needs a different mindset.

Common cause, not many bad luck events

If one disk on one broker fails, a few hundred partitions may shrink, all with the same broker as the missing follower. That is a local fault. But when partitions shrink across many brokers at once, the chance that a separate fault hit each one is tiny. Almost always there is one shared cause that touches everybody: the network they all use, the certificate they all present, the setting they all obey, the controller they all report to, or the load they all share.

Why the blast radius is larger

• Many applications feel it at once. Every producer that uses full confirmation on any affected topic sees errors or delays.
• The safety margin vanishes everywhere. With ISRs reduced across the cluster, one more failure anywhere could take partitions offline or lose data.
• Recovery competes with itself. When followers rejoin, they all fetch at once, adding load exactly when the cluster is weakest.
• Fixes can hurt. Restarting brokers in a panic makes still more replicas leave the ISR.

A calm mindset

The single most valuable habit in a storm is to resist random action. Kafka is doing its job by protecting confirmed data. Your first goals are to find the pattern and to avoid adding load or churn. A ten-minute investigation that finds the shared cause usually beats an hour of restarts.

A smaller ISR is not just a number on a dashboard. It changes what your system can promise to users, and different settings decide which promise breaks first.

Three settings that decide the impact

• Replication factor: how many copies exist in total. Three is standard.
• Minimum in-sync replicas (min.insync.replicas): the smallest ISR size for which the leader still accepts writes that need full confirmation. Two is a common choice with three copies.
• Producer acknowledgement setting (acks): how much confirmation the producer waits for. The safest value, all, waits for the whole ISR.

What happens as the ISR shrinks

Availability: refused writes

When the ISR falls below the minimum, producers using full confirmation receive errors named not enough replicas. Reads continue, but writers cannot proceed. If your applications do not retry, they lose events. If they retry blindly, they add load to a cluster that is already struggling.

Durability: fewer safe copies

Even when writes are accepted, an ISR of one means the only up-to-date copy lives on a single machine. If it dies, the partition goes offline. The system may be forced into a painful choice between waiting and allowing an out-of-date replica to lead, which can lose data that was already confirmed.

Latency: the slow follower problem

Before a slow follower is removed, it holds back the high watermark, so writers wait. The thirty-second window means a follower that stalls can add many seconds of delay to every full-confirmation write on its partitions before Kafka acts. After removal, latency drops, because the writers no longer wait for the slow follower. Thus a latency spike followed by a shrink is a classic pattern: the delay was the early warning.

Clients that do not wait for the ISR

Producers that use weaker confirmation settings do not experience the errors, because the leader does not wait for followers. That may look like good news, but it is the opposite. Those writes are being accepted without protection, and the loss of the leader could silently destroy them. A quiet shrinkage for these clients is the most dangerous kind.

Storms rarely stay the size they started. Understanding the feedback loops that grow them explains why speed and calm both matter.

flowchart TD
  A["Trigger: slowdown or fault"] --> B["Followers fall behind"]
  B --> C["ISR shrinks on many partitions"]
  C --> D["Producers see errors or delays and retry"]
  D --> E["More load on leaders"]
  E --> B
  C --> F["Followers later catch up and rejoin"]
  F --> G["Catch-up fetch traffic surges"]
  G --> B
    

Loop one: retries pile on

When producers get errors or timeouts, most clients retry automatically. If thousands of clients retry at once, leaders receive more requests, not fewer, right when they are already too slow. More load makes followers fall further behind, which shrinks more ISRs, which causes more errors. This is called a retry storm.

Loop two: catching up costs more

A follower that has been away is far behind. When it returns, it fetches large amounts of data as quickly as it can. Many followers doing so together create a burst of traffic between brokers, using network bandwidth, disk reads on the leaders and processor time. That can push other followers past the time limit, and the shrinking continues even though the original trigger has gone.

A typical timeline

Followers and leaders talk to each other constantly. Anything that disturbs that conversation for half a minute can shrink many ISRs at once.

Typical network causes

How to recognize a network cause

• Followers are missing from ISRs across many leaders, but the machines themselves look healthy in disk and processor graphs.
• Affected brokers cluster in one zone, one rack or behind one switch.
• Network error and retransmission counters rise on the machines, and round-trip times between brokers grow.
• Cloud provider status pages or network teams report an incident.

What to do

• Ask the network or cloud team about events in the relevant zones and links.
• Measure round-trip time and loss between a few affected pairs of brokers with ordinary network tools.
• Check bandwidth graphs against the machine’s documented sustained limit, not the burst figure.
• Remove sudden extra traffic where possible: pause backups, mirroring or bulk consumers that share the same links.

Both leaders and followers depend on the disk. A follower that cannot write fast enough falls behind, and a leader that cannot read fast enough cannot feed its followers.

How a slow disk shrinks an ISR

Followers must write every fetched batch to their log. Leaders must read data for followers that are behind the cached recent data. If a disk stalls for even a few seconds, requests wait, queues grow, and the thirty-second window becomes easy to miss. Because Kafka spreads partitions over many disks, a shared storage problem hits many at once.

Common storage causes

• Exhausted burst credits. Many cloud volumes allow fast speeds for a while and then fall back to a low baseline. A busy cluster can run out of credits together, and every broker slows at nearly the same time.
• Shared-storage limits. Network-attached volumes may share a bandwidth cap with the machine’s other network traffic.
• Failing hardware. A dying disk shows growing latency and error counts before it fails outright.
• Background pressure. Log cleaning for compacted topics, deletion of very large old files, or another program sharing the disk can steal speed.
• Nearly full disks. A disk close to capacity slows down and can stop accepting writes.
• File system trouble. Errors, forced checks or a remount can pause everything.
• Page cache pressure. When memory for caching is squeezed, reads that used to come from memory hit the disk instead, overwhelming it.

Clues in the data

What to do

• Check the volume’s credit or throughput graphs from the provider, and its configured maximum.
• Reduce write load where you can, for instance by slowing bulk producers with quotas.
• Free space on nearly full disks by adjusting retention carefully, or by adding capacity.
• Increase provisioned throughput if the platform allows it, since some volume types can be tuned while running.

A broker that cannot get its work done in time looks, to the rest of the cluster, exactly like a broker that is broken.

Busy request threads

Brokers process requests with pools of threads. Network threads read and write on connections, and I/O threads carry out the requests. Follower fetch requests are ordinary requests that wait in the same queue as client requests. If threads are saturated, the queue grows, and follower fetches are delayed. A watch on the request handler idle percentage, discussed later, shows this clearly.

Garbage collection pauses

Kafka runs inside the Java virtual machine, which periodically pauses to clean up unused objects. A healthy broker pauses for a fraction of a second. A broker with a mis-sized heap, or under memory pressure, can pause for many seconds. During that pause it sends nothing, receives nothing, and looks dead. A pause near or beyond the ISR time limit removes the broker from many ISRs at once. If the pauses happen on several brokers, for instance because of a shared change that increased memory demand, the storm is cluster-wide.

Machine-level pauses

• Swapping. If the operating system pushes Kafka’s memory to disk, response times collapse.
• Noisy neighbors and steal time. On shared virtual machines, another tenant can take processor time. The machine appears busy even though your processes cannot run.
• Live migration or maintenance. Cloud providers occasionally freeze or move virtual machines, causing short stalls.
• Throttled processors. Some machine types earn processor credits and slow down when they run out.

How to recognize it

• Garbage collection logs show long pauses that line up with the shrink events.
• Processor use is pegged, or steal time is high, on the affected brokers.
• Request queue size and request wait times are elevated while disks and networks look calm.
• Memory graphs show swap use or a heap near its limit.

What to do

• Confirm the heap size is sensible, not oversized, and that a modern collector is in use.
• Reduce the request load temporarily with client quotas.
• Turn off or fix any recent change that increased memory or processor demand, such as a new feature, a new client wave or a larger batch setting.
• Move the workload to machines with more headroom if processor credits or a noisy neighbor are to blame.

Sometimes nothing is broken at all. The cluster is simply asked to do more than it can, and replication is the first thing to suffer.

Surges from clients

A marketing campaign, a flash sale, a replay job, or a new application that starts reading from the beginning can multiply traffic in minutes. Extra producer traffic means more data to replicate. Extra consumer traffic means more outgoing bytes competing with replication for network and cache. A cluster sized for average load will show ISR shrinkage during a peak.

Surges from operators

Some of the worst storms are self-inflicted:

• Partition reassignments copy large amounts of data between brokers. Without a speed limit, they can saturate network and disks and starve normal replication.
• Adding brokers and rebalancing triggers large data moves.
• Mass consumer restarts cause everyone to re-read from an old position.
• Big batch jobs that read whole topics at full speed.

The forgotten speed limit

Kafka lets you set replication throttles so that data moves during a reassignment do not overwhelm the cluster. The danger is the opposite mistake. If a throttle is set too low, or left in place after the move is finished, it also limits ordinary follower replication for the throttled partitions or brokers. Followers then fall behind whenever traffic rises, and ISRs shrink. This is a classic cause of mysterious, recurring shrinkage that begins a few hours after a reassignment and ends only when someone removes the limit.

Client quotas as a shield

Kafka also supports client quotas that cap how much bandwidth or request time a client may use. They protect replication from being crowded out by a runaway application. They are best set in advance for known heavy clients, and they can be applied quickly during an incident to slow the worst offenders.

How to recognize overload

What to do

• Stop or pause any reassignment, rebalance or bulk job that is running.
• Check for and remove stale replication throttles once the move they were made for is done.
• Apply quotas to the heaviest clients.
• Ask application teams to slow down or delay non-urgent work.

These causes are sneaky because the machines are healthy. Everything is running. The brokers simply are no longer allowed, or able, to talk to each other properly.

Expired or changed certificates

Brokers often prove their identity to each other with digital certificates. Certificates expire on a fixed date. If the certificates that brokers use for replication expire, or are replaced with ones that other brokers do not trust, the follower connections fail. Because certificates are usually issued together, they often expire together, so every follower loses the connection at the same moment. The result is an instant, cluster-wide shrinkage with no hardware trouble at all.

Permission changes

Kafka uses access control rules to decide who may do what. Followers connect using a special internal identity, and they need permission to act on the cluster to fetch replication data. If a rule change, a clean-up script or an automation accidentally removes or overrides those permissions, followers are refused. Look at recent changes to access rules and to the identity brokers use to talk to each other.

Listener, address and name changes

• Changing the advertised addresses of brokers so that other brokers can no longer reach them.
• Switching the security protocol for inter-broker traffic in only part of the cluster.
• DNS changes that point names at the wrong machines.
• Load balancers placed between brokers with short idle timeouts that silently cut long-lived connections.

Setting changes that alter behavior

How to recognize it

• Broker logs on followers show authentication, authorization or connection errors to other brokers.
• The storm began at a precise moment that matches a certificate expiry, a deployment or an automation run.
• Hardware graphs are unremarkable.
• The affected brokers form no pattern by zone, but all show the same error text.

The ISR is not only a local idea on the leader. Every change must be recorded by the controller. If the recording system is unhealthy, the whole cluster’s view of ISRs can go wrong.

What the controller does here

When a leader wants to shrink or expand its ISR, it sends a request to the controller. The controller checks it, saves the new list and shares it with the brokers. In older clusters the coordination service ZooKeeper played this role, and in newer ones Kafka’s own KRaft controllers do. If the controller is overloaded, slow, mid-election or cut off, requests time out or are rejected, and brokers may have a stale picture of which replicas are in sync.

How controller trouble appears

Likely reasons for controller trouble

• A controller node under heavy load, for instance because it shares a machine with a busy broker.
• Too many partitions, making metadata changes large and slow to apply.
• Long pauses or network issues between controller nodes, breaking their agreement protocol.
• Storms of leadership changes caused by another problem, which flood the controller with work.
• Problems in the older coordination service, such as slow disks or overloaded nodes.

What to do

• Confirm there is exactly one active controller and that it is stable.
• Check the health, disk speed and processor use of the controller nodes.
• Avoid actions that generate large metadata changes during the incident, such as mass topic creation or partition moves.
• If controllers share machines with brokers in production, plan to separate them.

Many storms are simply maintenance done too quickly. The safe pace for restarting brokers is set by the ISR itself.

Why a restart shrinks ISRs

When a broker stops, every partition it hosts loses a replica. If it was a leader, leadership moves. When it comes back, its replicas must catch up on everything they missed before they can rejoin the ISR. A clean shutdown lets Kafka move leadership in an orderly way first, which reduces disruption. A crash or forced kill skips that, and on restart the broker may need to check and rebuild parts of its logs, which takes time proportional to how much data it has.

The dangerous pattern

The dangerous habit is restarting the next broker before the previous one has fully caught up. Now two brokers are behind at once. Partitions that had replicas on both are down to a single in-sync copy, perhaps below the minimum, and writes fail. Do this across a whole cluster and you have manufactured a cluster-wide storm on purpose.

Other maintenance triggers

• Rolling upgrades that skip the required order of steps for protocol versions.
• Configuration pushes that restart all brokers together through an automation tool.
• Operating system patching that reboots machines on a shared schedule.
• Autoscaling or instance replacement that removes several machines at once.
• Cloud provider maintenance events that touch many machines in the same window.

How to recognize it

The timing lines up with a change record, a deployment tool’s log or a scheduled patch window. The missing followers are the brokers that restarted, and the restarts happened close together. Logs show recovery work such as checking log segments after an unclean stop.

What to do

• Halt any rollout in progress.
• Let restarted brokers catch up fully before touching another.
• Fix the automation so it waits for a healthy ISR state between steps. Chapter 21 shows how.

The best storm is the one you catch in the first two minutes. Good alerts turn a surprise into a routine.

The core alert set

Supporting signals that explain the cause

• Request handler idle percentage and network thread idle percentage show whether brokers are out of processing capacity.
• Produce request time, and especially the part spent waiting for followers, shows replication lag before the ISR shrinks. When the waiting part of produce time rises, followers are struggling.
• Follower fetch request time and queue time show whether followers’ requests are slow to be served.
• Maximum replica lag shows how far behind the worst follower is.
• Disk latency, network errors, processor steal and garbage collection pauses confirm the physical cause.
• Producer error rates and retry rates from the client side show impact and warn of retry storms.

Designing alerts that people trust

• Alert on sustained conditions, so routine restarts do not page anyone at night.
• Suppress or annotate alerts during announced maintenance, and make sure the suppression ends.
• Group alerts by broker, zone and cause, so ten thousand messages become one clear incident.
• Link every alert to a short runbook.
• Graph shrinks and expands together, so the gap is obvious at a glance.

You do not need to investigate everything at once. A few good questions, asked in order, point to the right family of causes.

flowchart TD
  S["ISR shrinkage alert"] --> Q1{"One follower broker missing from many partitions?"}
  Q1 -- Yes --> A1["Follower host problem: disk, pauses, network, restart"]
  Q1 -- No --> Q2{"Many followers missing from one leader broker?"}
  Q2 -- Yes --> A2["Leader overloaded, or its network or disk is slow"]
  Q2 -- No --> Q3{"Do the affected brokers share a zone or rack?"}
  Q3 -- Yes --> A3["Zone or rack network or infrastructure problem"]
  Q3 -- No --> Q4{"Did anything change recently?"}
  Q4 -- Yes --> A4["Certificates, permissions, settings, throttles, deployments"]
  Q4 -- No --> A5["Systemic: controller, cloud event, traffic surge"]
    

How to get the answers

Kafka’s command-line tools can list partitions that are under-replicated (--under-replicated-partitions) or below the minimum ISR (--under-min-isr-partitions). Export that list and group it in a spreadsheet or small script in two ways:

• By missing replica. For each partition, note which replica is not in the ISR. If one broker appears almost everywhere, you have a follower-side problem with that broker.
• By leader. Count affected partitions per leader. If a single leader dominates, the leader cannot serve its followers.
• By zone or rack. Add each broker’s zone and see whether the affected set clusters.

A good runbook is short enough to follow under stress. Here is a practical order of actions for the first thirty minutes.

Under pressure, teams reach for settings that promise quick relief. Some help, some hide the problem, and some make it much worse. Here is how to tell them apart.

Safe actions to take early

• Pause changes and heavy jobs. This cannot hurt and often helps immediately.
• Apply or tighten client quotas on the heaviest producers and consumers.
• Remove stale replication throttles once you are sure they are no longer needed.
• Ask application teams to slow retries or add delays between attempts.
• Add capacity where it is obviously short, such as raising a volume’s provisioned throughput.

Knobs that need careful thought

Things not to do

• Do not restart many brokers. It adds churn and recovery work.
• Do not delete data or topics to “clear the problem”. It rarely helps and cannot be undone.
• Do not switch on unclean leader election cluster-wide. The damage can be silent and permanent.
• Do not start large reassignments in the middle of an incident.
• Do not raise the lag limit as your first step. Find the cause first.

Fixing the cause is only half of the job. The cluster then has to bring all those lagging followers back, and how you manage that decides whether recovery is smooth or a second storm.

What recovery looks like

Once the cause is gone, followers fetch the data they missed and, when caught up, rejoin their ISRs. Expand events climb, under-replicated partitions fall, and the under-minimum count drops to zero. The amount of data to copy equals the incoming rate multiplied by the length of the outage, so longer incidents mean heavier recovery traffic.

Watch for the second wave

All followers fetching at once can saturate the same resources that just recovered. Signs of trouble include the shrink count rising again while expands are still low, request queue times increasing, and network use hitting its limits. If this happens, ease the load rather than restarting anything.

Techniques that help

• Keep client load moderate until the under-replicated count is near zero, by leaving quotas in place.
• Use replication throttles deliberately only if catch-up traffic is threatening clients, and then remove them as soon as it is safe.
• Recover the most important topics first where you can, by prioritizing their partitions, or by giving non-critical clients lower priority.
• Avoid restarts while brokers are catching up, since each restart resets progress for the partitions it hosts.
• Let automatic leader rebalancing wait until the ISRs are healthy, so leadership does not move again during catch-up.

How to know it is really over

• Under-replicated, under-minimum and offline partition counts are all zero and stay there.
• Shrink and expand rates have returned to baseline.
• Request latencies and queue times are back to normal.
• Client error and retry rates have returned to normal.
• Leadership is spread evenly, or a rebalance has been planned.
• Any temporary emergency settings, such as a lowered minimum ISR, quotas and throttles, have been reviewed and restored.

Your applications decide whether a shrinkage storm becomes lost data, or a short pause that nobody notices. Their settings matter as much as the brokers’.

Settings for critical data

Behave well when errors occur

When a producer receives a not-enough-replicas error, it should wait and retry with exponential backoff, meaning longer and longer delays between tries, plus a little random variation so thousands of clients do not all retry at the same instant. It should also have a bounded local buffer, and a clear plan for what to do when the buffer fills, such as slowing the source, spilling to local storage, or failing the request visibly.

Tiers of importance

Not every topic deserves the same settings. Classify topics by how much loss or delay the business can accept, and choose settings to match:

Consumers during a storm

Consumers only read data up to the high watermark, so they see slightly older data while followers lag, and they are otherwise unaffected. Delays in end-to-end freshness are the main effect. If consumers are configured to read from followers in their own zone, they follow the same watermark rule and see the same delay. Consumers rarely need changes, but downstream teams should know that data may be delayed, not lost.

The cheapest storm is the one that never happens. These design choices make shrinkage rarer, smaller and shorter.

Spread replicas across failure domains

Tell each broker which rack or availability zone it lives in, and Kafka places the replicas of each partition in different ones. Then a problem in one zone can remove at most one replica from a partition, and with three copies and a minimum of two, writes continue. Make sure the number of brokers in each zone is equal.

Leave headroom

Many storms are just clusters running too hot. Size brokers so that normal peak load uses about half of tested capacity, leaving room for a lost broker or zone, for recovery traffic and for surges. Watch the busiest broker, not the average.

Give replication its own road

Kafka lets you define separate network listeners for inter-broker traffic and client traffic. Using a dedicated inter-broker listener, and where possible a dedicated network path or quality-of-service class, keeps a client surge from crowding out replication. It also makes it easier to secure and monitor the replication path.

Right-size the fetching machinery

• Set enough replica fetcher threads for the number of partitions each broker follows.
• Keep replica fetch sizes large enough to move data efficiently.
• Set socket buffers appropriate for links between zones.
• Keep the wait time for fetches well below the replica lag limit.

Keep partitions manageable

The more partitions a broker leads and follows, the more work its fetchers and the controller must do. Stay within tested partition counts per broker, and avoid creating thousands of tiny partitions “just in case”.

Choose the lag limit consciously

The default replica lag limit works for most clusters. If your links between zones are long and your pauses are occasional, you might raise it modestly after testing, accepting that real stalls take longer to detect. Change it as a deliberate design decision, with measured evidence, not as a panic fix.

Isolate the coordination layer

Run controllers on dedicated machines, in separate zones, with reliable disks and modest but steady resources.

Good habits matter as much as good design. Most storms trace back to a small number of preventable operational mistakes.

The safe rolling restart

Restart one broker at a time, and never start the next until the ISR state is healthy again. This single rule prevents the most common self-inflicted storm.

flowchart LR
  R1["Restart one broker"] --> R2["Wait for it to rejoin"]
  R2 --> R3{"Under-replicated partitions at zero?"}
  R3 -- No --> R2
  R3 -- Yes --> R4["Move to the next broker"]
  R4 --> R1
    

• Use controlled shutdown, so leadership moves away before the broker stops.
• Do not pass the wait step just because a timer expired. Check the actual under-replicated count.
• Follow upgrade steps in the documented order, including protocol version changes.
• Restart the controller nodes carefully, one at a time, keeping the majority alive.

Certificates and secrets

• Keep an inventory of every certificate and its expiry date.
• Alert at least thirty days ahead, then again at fourteen and seven.
• Stagger expiry dates across brokers where possible, so a single missed renewal cannot hit everyone at once.
• Practice renewal in a test cluster.

Change hygiene

• Roll out changes gradually, to one broker or one zone first, and watch the ISR metrics before continuing.
• Require review for changes to access rules and to inter-broker settings.
• Record every replication throttle you set, with an owner and an end date, and verify removal afterwards.
• Keep configuration in version control, and compare running values with the stored ones regularly.

Capacity and growth

• Review capacity quarterly with real numbers.
• Start growth work when disk, processor or network use passes agreed thresholds, before they become emergencies.
• Apply quotas to clients that are known to be spiky.

Practice

Run game days in a test environment or in a carefully chosen quiet window. Simulate a lost zone, a slow disk, an expired certificate and a forgotten throttle. Time how long it takes your team to find the cause using the decision tree. Every drill exposes gaps in dashboards, alerts and runbooks while the stakes are low.

A storm that teaches nothing is a storm you will meet again. A good review turns pain into a stronger system without blaming people.

A blameless approach

The aim is to understand how the system allowed a mistake or a failure to cause harm, not to find someone to blame. If a person made a change that caused the storm, ask why the change was possible, why nothing warned them, and why the effects were not caught sooner.

Questions to answer

Typical follow-up actions

• Add or tune the alerts that would have fired earlier.
• Update the runbook with the steps that worked.
• Fix the automation or process that allowed the trigger.
• Adjust capacity, settings or architecture.
• Share the story with other teams that run Kafka.

Numbers make the ideas concrete. Let us walk through an invented but realistic storm and do the arithmetic that shapes how long recovery takes.

The setup

A cluster has nine brokers spread over three zones. Producers send 60 MB/s in total. Every partition has three copies, so each byte is followed by two follower copies. There are about 3,000 partitions in all. Critical topics use a minimum of two in-sync replicas. The replica lag limit is 30 seconds.

The incident

At 14:00, the certificates used by brokers to talk to each other expire, and all follower connections fail. The team finds the cause and installs new certificates by 14:10. Let us trace what happens.

How much data must be caught up?

Followers were away for ten minutes, which is 600 seconds. Producers kept sending, or tried to, at up to 60 MB/s. Assume the writes that were accepted continued, so the missed data per follower copy is about 60 MB/s times 600 seconds, which is 36,000 MB. There are two follower copies of each byte, so the total to copy is about 72,000 MB, or 72 gigabytes. Spread over nine brokers, each has about 8 GB to catch up.

How long does catch-up take?

What the numbers teach

• Detection speed matters. Every extra minute of outage adds more data to catch up.
• Throttles and headroom decide whether recovery takes minutes or half an hour.
• Writes with full confirmation were refused for about ten minutes, so client retry behavior decided how much the applications suffered.
• A cluster running near capacity would have struggled to absorb the catch-up traffic and might have shrunk again.

Each family of causes leaves a different fingerprint on your graphs. Learning the shapes lets you make a good guess in seconds.

Flat lines are clues

Natural traffic wobbles. When a graph is suspiciously flat at its top, a limit is at work, because real load would rise and fall. A flat ceiling on replication traffic across brokers is a very strong sign of a throttle, a quota or a bandwidth cap.

Compare, do not stare

• Put the same metric for all brokers on one graph and look for outliers.
• Overlay the shrink count with the candidate cause, such as disk latency, and see which moved first.
• Compare with the same hour yesterday and a week ago to separate normal peaks from new problems.
• Mark deployments, certificate dates, configuration changes and cloud events on the timeline.

The advice so far assumes a typical self-managed cluster with three copies. A few common situations need extra care.

Replication factor and minimum ISR together

Very small clusters

In a cluster of exactly three brokers, taking one down for maintenance puts every partition at the minimum. If anything else goes wrong in that window, writes stop. Plan maintenance for quiet times, keep brokers healthy, and consider more brokers for critical workloads.

Managed Kafka services

If your provider runs the brokers, you cannot log in and restart machines, but the same logic applies. Check the provider’s service health page and the metrics they expose. Look at your own clients: are they overloading the service, using more partitions than the plan supports, or reaching quotas? Contact support with evidence, such as the time the shrinkage began, the affected topics and the metrics. Meanwhile, use client-side protections such as backoff, quotas and buffering. Many providers also offer scaling options; use them early in an incident if load is the cause.

Consumers reading from followers

Some consumers are set to read from the nearest replica, including followers, to save cross-zone traffic. They see data only up to the high watermark, which may lag while followers struggle. Warn those teams that freshness can drop during a storm.

Tiered storage

With tiered storage, older data lives in object storage while recent data stays local. The catch-up of a lagging follower usually concerns only the recent local data, so recovery traffic is bounded. But heavy reads of old data from remote storage can add network and cache pressure. Keep an eye on them when you are recovering.

Topics with unusual settings

• Compacted topics run background cleaning that competes for disk. Watch them if disks are the suspect.
• Very high-throughput topics may need dedicated brokers or quotas so they cannot dominate replication.
• Topics with a per-topic lag or minimum setting that differs from the cluster default can behave differently from their neighbors. Check for exceptions when a subset of topics is affected.

Many settings touch replication. Knowing what each one does, and how it can help or hurt during a storm, lets you review your cluster calmly before trouble arrives. Default values change between Kafka versions, so always check the documentation for the version you run.

Broker and topic settings

Producer settings

A sensible review routine

1. List the running values of these settings on every broker and compare them with each other and with your stored configuration.
2. Look for topics that override the cluster defaults, especially for minimum ISR and unclean election.
3. Check producers of the most important topics for the settings above, including defaults that come from libraries.
4. Note any dynamic settings, such as throttles and quotas, that were applied by hand and never removed.
5. Write down the reason for every value that differs from the default.

The stories below show the causes from earlier chapters in action. The first group reflects publicly known practices, and the rest are illustrative scenarios written to show how a typical team would meet each situation.

Patterns are habits that keep clusters healthy, and anti-patterns are traps that turn small issues into big storms.

Patterns worth adopting

Anti-patterns to avoid

Here is a compact summary of what to do before, during and after a shrinkage storm, followed by the mistakes that trip up most teams.

The most common mistakes

• Treating shrinkage as the cause. It is a symptom. The cause is what made followers slow or disconnected.
• Looking at brokers one by one. A cluster-wide pattern needs a cluster-wide view: group and compare.
• Forgetting the boring causes. Expired certificates, changed permissions and leftover throttles cause a surprising share of storms.
• Running too hot. With no headroom, any surge becomes a shrinkage event.
• Ignoring producer settings. Weak confirmation hides risk, and aggressive retries multiply load.
• Mixing controllers with busy brokers. Metadata slows exactly when it is needed.
• Never rehearsing. Untested runbooks fail at the worst moment.

A short pre-flight checklist

• Do critical topics use three copies, a minimum of two, and full confirmation?
• Are replicas spread evenly over zones?
• Do we alert on the metrics in Chapter 14, with links to a runbook?
• Do we know every certificate’s expiry date and every temporary throttle?
• Does our rolling restart process check the ISR state, not just a timer?
• When did we last practice this incident?

Quick answers to questions that come up during and after shrinkage incidents.

A cluster-wide ISR shrinkage looks alarming, but it follows patterns you can learn to read. Understand the mechanism, find the shared cause, act calmly, and design so the next one is small.