Kafka is a system built on promises of speed and steady behavior, and a pause quietly breaks both.

A quick picture of Kafka

Apache Kafka stores streams of events in named containers called topics, and each topic is cut into partitions. The servers that store partitions are brokers. Programs that write events are producers, programs that read them are consumers, and every partition has several copies called replicas, one of which is the leader that clients talk to. All of these parts constantly send small messages to one another: heartbeats that say “I am alive,” fetch requests that say “give me new data” and acknowledgments that say “I stored it.”

What a pause does to that conversation

A garbage collection pause is a moment when the Java program stops running its own code so the collector can clean up unused memory. During this moment the broker cannot answer requests, cannot send heartbeats and cannot fetch data from leaders. To the rest of the cluster, a broker that is paused looks exactly like a broker that has crashed or lost its network. Only when the pause ends does the difference become clear.

Short pauses of a few milliseconds are harmless. Pauses of a few hundred milliseconds create latency spikes that clients feel as a slow request. Pauses of several seconds can make the cluster believe the broker is dead, and pauses of half a minute can remove it from the in-sync group of a great many partitions at once.

Why averages hide the problem

Most dashboards show average latency, and averages hide pauses. If a broker serves 10,000 requests per second and freezes for one second, the other 9,999 seconds look perfect. The average barely moves, yet in that one second thousands of requests waited. Engineers who study large systems call this the tail latency problem: the rare slow requests matter more than the typical fast ones, because a user or a service that sends many requests will hit a slow one sooner or later. Diagnosing GC pauses is largely about looking at the tail instead of the average.

To understand what the garbage collector is cleaning, first see where a broker keeps its data. Most of it is not where beginners expect.

Three different kinds of memory

A running broker uses memory in three main places, and only one of them is managed by the garbage collector.

Why the heap is small on purpose

Kafka does not keep your messages in the Java heap. It appends them to files on disk and relies on the operating system’s page cache to serve recent data quickly. That is why a broker with 64 gigabytes of memory usually runs with a heap of only a few gigabytes. The rest of the memory is left to the page cache, which is much faster and cheaper for Kafka’s needs than a giant heap that the garbage collector must constantly examine.

flowchart TB
    subgraph HOST["Broker machine memory"]
        subgraph JVM["Broker Java process"]
            HEAP["Java heap: young and old areas"]
            META["Metaspace: class definitions"]
            DIRECT["Direct buffers for network and files"]
            STACK["Thread stacks"]
            MMAP["Memory-mapped index files"]
        end
        PC["Operating system page cache holding log data"]
    end
    CLIENTS["Producers and consumers"] --> DIRECT
    DIRECT --> HEAP
    HEAP --> PC
    PC --> DISK[("Disks with log segments")]

A helpful rule

Give the heap enough room for the broker’s working objects, and give everything else to the operating system. A heap that is too small causes constant cleaning. A heap that is too large steals memory from the page cache, forcing more reads from disk, and makes each cleaning longer. Good sizing sits in between, and Chapter 13 shows how to find that spot.

Budgeting the off-heap part

Because off-heap memory is invisible to the garbage collector, it is easy to forget when planning. Yet the total memory of the Java process is the heap plus several other pieces. Direct buffers grow with the number of connections and the sizes of network reads and writes. Metaspace holds the definitions of classes and grows with the code the broker loads. Every thread has a stack, and a busy broker runs hundreds of threads. Memory-mapped index files add address space that grows with the number of log segments. If you run in a container with a memory limit, all of these plus the page cache count toward the limit, and the operating system may kill the process when it goes over, with no warning from the JVM at all.

A practical planning rule is to add up the heap, a generous allowance for direct buffers and metaspace, and a fair amount of headroom, and then leave the majority of the machine’s remaining memory to the page cache. The exact numbers vary, so confirm them by watching the real process over a busy week, not by guessing.

Before diagnosing anything, you need a clear mental model of what a garbage collector does and why it sometimes has to stop the world.

What is garbage?

Whenever a Java program needs a piece of data, such as the contents of a request, it asks the JVM for a chunk of heap memory. This creates an object. When the program has finished with the object and nothing refers to it any more, the object becomes garbage: it still occupies memory but is of no use. The garbage collector is the part of the JVM that finds such objects and reclaims their space so it can be reused. Programmers in other languages have to free memory by hand and often forget. Java does it automatically, and the price is occasional pauses.

Why the world must sometimes stop

To find garbage safely, the collector must know which objects are still in use. While it studies the objects, the program must not change who refers to whom, or the collector could throw away something still needed. The simplest way to keep the picture still is to freeze all application threads. This freeze is called a stop-the-world pause, often shortened to STW. Modern collectors do as much work as possible while the program keeps running, but they still need short freezes at certain moments.

Young objects die young

Studies of real programs found that most objects live for a very short time. A request object is created, used within milliseconds and is then garbage. So the heap is divided into two main areas. The young generation is where new objects are born. It is cleaned often and cheaply, because most objects there are already dead and only a few survivors must be copied. The old generation holds objects that have survived several cleanings and are likely to live a long time. It is cleaned less often, but cleaning it costs more.

Throughput collectors versus low-pause collectors

Collectors are designed with different goals. Some aim for the highest total work done and accept longer pauses now and then. Others aim for short, predictable pauses and accept spending a little more processor time to achieve them. A message broker is in the second group, because its clients and its cluster peers are always waiting on short deadlines. This is why Kafka guidance has recommended a low-pause collector called G1 for many years.

Collector threads and the price of speed

Modern collectors use several threads at the same time to speed up their work. During a pause, many threads clean different parts of the heap in parallel, so a machine with more processor cores can finish a pause faster. Some work also runs in the background alongside the application, for example the marking of old objects. This background work uses processor time that the broker would otherwise spend on requests. On a machine that is already busy, the background threads compete with request handling, and both can slow down. The takeaway is that garbage collection is never free. It uses processor time, and a broker with too few cores can suffer pauses that would be short on a bigger machine.

The number of collector threads is normally chosen by the JVM based on the number of cores it sees. In containers and on shared virtual machines, the JVM may see more or fewer cores than are really available. If it starts too many threads for the real processor share, the threads fight each other and pauses stretch. Checking what the JVM believes about the machine is a quick and useful step in any investigation.

G1 is the collector used by nearly every production Kafka broker. Knowing its rhythm makes GC logs easy to read.

Regions instead of two big blocks

The G1 collector, whose name comes from “garbage first,” splits the heap into many equal-sized pieces called regions. At any moment each region plays a role: some hold young objects, some hold old objects and some are free. Because the regions are small, G1 can clean a few of them at a time and choose the ones that contain the most garbage first, which is where the name comes from. You can ask G1 to aim for a pause target, and it picks how many regions to clean in each step to try to stay within the target.

The rhythm of G1

flowchart LR
    Y["Young collection: copy survivors, short pause"] --> Y
    Y --> M["Concurrent marking: runs alongside the program"]
    M --> MX["Mixed collections: clean young plus some old regions"]
    MX --> Y
    M -.->|"if old area fills too fast"| F["Full collection: long stop-the-world pause"]
    MX -.->|"if space runs out"| F
    F --> Y

In a healthy broker, the loop of young collections runs many times per minute. Each one stops the program for a few milliseconds up to a few tens of milliseconds. When the old area of the heap becomes about a certain percentage full, G1 starts a concurrent marking cycle, which studies the old objects while the program keeps running. Afterwards, G1 performs several mixed collections that clean old regions full of garbage together with the young area. The setting that controls when concurrent marking begins is called the initiating heap occupancy percentage, and it plays a central role in tuning.

The dangerous path: the full collection

If G1 falls behind and the heap runs out of free regions before marking and mixed collections have freed enough space, the collector gives up on being clever. It performs a full collection, which stops the entire program and cleans the whole heap. In older Java versions this work was done by a single thread, which made it extremely slow. Even in newer versions it is long compared with normal pauses. Many terrible incidents in Kafka clusters are full collections, so in the GC log, hunting for them is the first job.

Humongous objects

G1 has a special rule for very large objects. Any object that is at least half the size of a region is called humongous, and it is placed in its own group of regions directly in the old area, skipping the young area. Humongous objects can cause trouble: they are not cleaned by cheap young collections, they can fragment the heap because they need several neighboring free regions, and a burst of them can trigger extra marking cycles. Kafka allocates large buffers when clients send big batches or ask for big fetches, so humongous allocations are a real risk. Choosing a larger region size, for example 16 megabytes, makes more of these buffers count as ordinary objects, and this is one reason the Kafka documentation’s sample tuning includes a region size setting.

The pause target is a hint, not a promise

The pause target setting tells G1 how long you would like each pause to be. The default is 200 milliseconds, which is too relaxed for many Kafka clusters. The Kafka operations documentation describes a busy cluster at LinkedIn tuned with a target of 20 milliseconds, where the 90th percentile of pauses was about 21 milliseconds and there was less than one young collection per second. But a very low target has a cost. G1 must clean fewer regions per step, so it needs more steps and more processor time, and if the target is unrealistic for the amount of live data, G1 may fall behind and end up in a full collection. Targets should be set from measurement, not from hope.

Evacuation failure and “to-space exhausted” in plain words

During a young or mixed collection, G1 copies the surviving objects from the regions it is cleaning into empty regions. This copying is called evacuation, and the empty regions it copies into are the to-space. If G1 runs out of empty regions in the middle of a collection, it has nowhere left to copy survivors. The log then reports that the to-space is exhausted, and G1 must finish the collection in a slower, more careful way, which produces a much longer pause. In the worst case it follows with a full collection.

The usual reasons are a heap that is too small, a sudden burst of surviving or humongous objects, or marking that began too late so that old regions are still full of garbage. G1 keeps a reserve of free space for this exact case, and tuning the reserve upward is one option, but it is usually better to address the reason for the shortage. In the GC log, phrases about to-space exhaustion or evacuation failure are strong signals that the heap sizing or thresholds need attention.

Every GC problem comes from one of two sources: too much garbage made too quickly, or too much long-lived data. Here is where each comes from in Kafka.

Sources of fast garbage: the allocation rate

Sources of long-lived data: the live set

The second group grows slowly and stays. It decides how much the collector must examine in each mixed or full collection.

Why the two problems feed each other

A vicious circle is common. A pause makes requests wait. Waiting requests pile up in queues, and clients time out and retry, which creates even more requests and more garbage. When the pause ends, the broker faces a bigger burst than before, so the next collection is longer, and the cycle repeats. This is why a small tuning problem can suddenly become a full outage under load.

A pause by itself only stops one machine. Timeouts across the cluster are what turn it into a bigger event.

The chain reaction

flowchart LR
    GC["Long GC pause on one broker"] --> HB["Heartbeats and fetches stop"]
    HB --> TO["Session timeout passes"]
    TO --> FENCE["Controller marks broker as dead or fenced"]
    FENCE --> LE["Leader elections for its partitions"]
    LE --> CE["Client errors and metadata refresh storm"]
    GC --> ISR["Followers of its leaders drop out of sync"]
    ISR --> WE["Write errors if too few copies remain"]
    GC --> RETURN["Pause ends, broker returns"]
    RETURN --> CATCH["Catch-up replication and leader rebalance"]

The timeouts that decide how bad it gets

Each connection between two Kafka components has a timeout. If the pause is shorter than the timeout, the effect is only extra latency. If the pause is longer, something is declared dead. Knowing the timeouts lets you predict how much pause your cluster can tolerate. The defaults below are typical for recent versions, but they change between releases and are often customized, so always confirm them in your own configuration.

Why a follower’s pause matters as much as a leader’s

If a leader pauses, clients cannot write to its partitions until it wakes or a new leader is chosen. If a follower pauses, the leader may stay perfectly healthy, yet the follower stops fetching. When producers require several in-sync copies before accepting a write, a paused follower can bring the count below the required minimum, and writes to many partitions start failing even though the leader never paused at all.

sequenceDiagram
    participant P as Producer
    participant L as Leader Broker
    participant F as Follower Broker paused
    participant C as Controller
    P->>L: Produce with all replicas acknowledging
    L->>F: Waiting for follower to fetch
    Note over F: GC pause of several seconds
    F--xL: No fetch requests arrive
    L->>L: Follower lag grows past limit
    L->>C: Shrink in-sync set
    L-->>P: Error if minimum copies not met
    Note over F: Pause ends
    F->>L: Fetch resumes and catches up
    L->>C: Expand in-sync set

The clean-up cost after a pause

Recovery is not instant. After a broker returns, it must catch up on missing data for every partition, rejoin in-sync groups and take back its leader duties through preferred leader election. During this time it uses extra network and disk capacity, which can create another round of pressure if the cluster is already busy. Engineers who only look at the pause itself often miss this second wave of problems.

Working out your pause budget

A helpful exercise is to compute how long a pause your cluster can survive without visible damage. List the timeouts that matter and pick the shortest one that you cannot change quickly. For many teams it is not one of Kafka’s own timeouts, but a deadline in a client application, for example a payment service that gives up after two seconds. Call that the hard limit. Your target for the longest normal pause should be well below it, say a quarter or less, because some pauses will be longer than average and because the pause is only part of the delay a request experiences.

Then compare with what your logs show. If the longest pause each day is under a quarter of the hard limit, you have healthy headroom. If it is within half of the limit, you are on thin ice. If it exceeds the limit, clients are already failing and the priority is clear. Recording this budget in the team’s documentation gives everyone a shared, numeric definition of “too long.”

You usually do not start with a GC log. You start with a strange graph, and the pattern tells you where to look.

Symptoms in Kafka metrics

Symptoms in logs

Broker logs often hold the clearest hints. Look for messages about the broker’s session with ZooKeeper timing out, or, in newer clusters, about the controller fencing the broker or about failing to send heartbeats in time. Look for statements that a follower was removed from the in-sync set, and for warnings from the JVM that the application threads were stopped for an unusually long time. Line up the timestamps of these messages, because the order tells the story: first the freeze, then the missed heartbeats, then the removal.

The signature of a GC pause versus other problems

One broker or many?

Ask first whether the pattern appears on a single broker or on all of them at the same moment. A single broker suggests something local: a hot partition, too many leaders, a bad disk or a machine problem. All brokers at once suggests something shared: a change in client behavior, a scheduled job, a common dependency, a cloud event or a cluster-wide configuration change. This single question halves the search space.

You cannot diagnose what you cannot see. Set up these three sources of evidence before the next incident, not during it.

GC logs: the black box recorder

The JVM can write a detailed record of every collection to a file. In Kafka’s standard start scripts, this logging is switched on when the broker runs as a background service, and the file is stored beside the other broker logs, rotated by size. In older Java versions, the settings were named after “print GC details” and a log file path. In Java 9 and later, a single unified logging option controls it and lets you choose which tags to record and how to decorate each line with a date and uptime. Whatever version you run, make sure of four things: the log is on, it includes timestamps, it keeps enough history to cover several days and it is written to a fast local disk.

JMX metrics: the live dashboard

The JVM exposes counters through JMX. For each collector, it reports how many collections have happened and the total time spent in them. Sampling these values every 15 to 60 seconds and taking the difference gives you collections per second and the share of time spent collecting. Also collect the heap size after collection, the sizes of the young and old areas and the number of threads. If you use a metrics exporter, add these to your standard broker dashboard. On Amazon MSK, some of these are published for you in CloudWatch, for example a metric for the heap in use after collection, but you should check the current list of metrics for your broker type.

Java Flight Recorder: the microscope

Java Flight Recorder, or JFR, is a low-overhead profiling tool built into modern OpenJDK builds. It can record collections, allocations, lock waits, thread stalls and safepoints for a period of minutes and save them to a file that you open in a viewer. It is ideal for questions such as “which parts of the code allocate the most memory?” and “what were threads doing during the stall?” Because the overhead is small, many teams enable a rolling recording so that the last few minutes are always available when an incident strikes.

flowchart TB
    JVM["Broker JVM"] --> GCL["GC log files"]
    JVM --> JMX["JMX metrics"]
    JVM --> JFR["Flight Recorder recordings"]
    OS["Operating system counters"] --> COL["Metrics collector"]
    JMX --> COL
    GCL --> SHIP["Log shipper"]
    SHIP --> STORE["Central log store"]
    COL --> DASH["Dashboards and alerts"]
    JFR --> ANALYSIS["Engineer analysis on demand"]
    STORE --> DASH
    DASH --> RUNBOOK["Runbook and on-call response"]

Other useful tools

Setting up a rolling recording plan

A GC log looks scary at first. In practice you read only a handful of fields.

What each line tells you

Each collection produces a record with a time, a type, a cause, the heap size before and after, and the duration. Your eyes should look for these items in order.

The three time numbers

Most log formats print three times for each pause. User time is processor time spent in the collector’s own code, added across all its threads. System time is processor time spent inside the operating system on the collector’s behalf, for example handling memory. Real time is the actual clock time that passed. Compare them, because their relationship exposes hidden problems.

Typical causes and what they suggest

Patterns over time

One line rarely tells the story. Zoom out and look for shapes. A steady sawtooth of the heap rising and dropping is normal. A sawtooth whose lowest points climb steadily is a growing live set, possibly a leak or ever more partitions. A sudden vertical jump followed by a long full collection is often a burst of humongous allocations or a coordinator loading a large state. Regular spikes at exact intervals, such as every hour, point at scheduled jobs on the broker or on its clients.

A log line read out loud

Since logs are hard to read at first, here is how an engineer would narrate one entry, without showing the raw text. “At 14:02:31 on Tuesday, a young collection began because the young area was full. Before the collection the heap held 4.6 gigabytes out of a maximum of 6. After it, the heap held 2.3 gigabytes. The collector’s threads worked for a combined 60 milliseconds of processor time, the operating system spent 4 milliseconds on its behalf, and the whole pause lasted 22 milliseconds on the clock.” That is a healthy line: the young area was cleaned, about half the heap was freed, the live set is 2.3 gigabytes, and the pause is close to a 20 millisecond target.

Now imagine the next entry says: “At 14:07:10, a full collection began after an allocation failure. Before it, the heap held 5.9 out of 6 gigabytes. After it, the heap held 3.8 gigabytes. The pause lasted 6.4 seconds.” This is a warning. The heap was nearly full, a full collection was needed, and the live set of 3.8 gigabytes is larger than it was earlier in the day. Such a pair of entries tells you the live set is growing and that marking started too late or the heap is too small, so the next step is to look at what changed between the two.

Finally, picture an entry where the clock time is 4.1 seconds, the collector’s own processor time is 80 milliseconds and the operating system time is 20 milliseconds. Nearly all of the time was spent neither computing nor in the kernel on the collector’s behalf. The threads were waiting, most likely for a disk, for a starved processor or for something that swapped. Reading the entry this way, you know at once that the fix is outside the JVM settings.

Many “GC pauses” turn out to be something else that freezes the machine. A good diagnosis rules these out.

Safepoints: the JVM’s parking spots

A safepoint is a moment when the JVM brings all application threads to a stop so it can do certain jobs safely. Garbage collection is the most common job, but not the only one. Others include revoking special locks, recompiling code, dumping threads and applying certain tools’ requests. Each time, the JVM must wait until every thread has reached a safe spot, and a thread running a long loop without safe spots can hold everyone else up. The JVM can log the total time that application threads were stopped and the time spent waiting for them to reach the safepoint. If your clients see a stall and the GC log shows nothing, safepoint logging is the next place to look.

Memory pressure and swapping

If the operating system runs short of memory, it can move parts of a process to the swap area on disk. When the garbage collector then touches those parts, they must be read back, and a pause that should take 20 milliseconds can take seconds. Kafka guidance is to keep swapping to an absolute minimum, either by turning swap off or by setting the swappiness value very low. A high system time in the GC log is the classic clue.

Transparent huge pages

Linux can group small memory pages into big ones automatically, a feature called transparent huge pages. In theory it saves work. In practice, defragmenting memory to build these big pages can cause unpredictable stalls, again visible as high system time. Many database and messaging systems recommend disabling it or setting it to a mode that avoids on-demand compaction. Check what your operating system image uses by default.

Processor starvation: steal and throttling

Two situations can stop the collector from getting the processor time it needs. On virtual machines, a busy neighbor can take processor time that was promised to you, which shows up as a steal counter. On burstable cloud instance types that give a baseline of processor use and let you spend saved credits, running out of credits reduces you to the baseline and everything slows down together. In containers, a processor quota can cause the kernel to pause your threads for the rest of each accounting period once they have used their share, which looks like random freezes of tens or hundreds of milliseconds. The collector uses many threads at once, so it hits the quota especially fast.

Disk and file system stalls

Kafka writes to disk in the background, and the operating system flushes dirty pages when it chooses. If the disk becomes slow, for example because of a storage volume that has used up its burst allowance, threads that touch the file system can block. If a thread that is needed to reach a safepoint is blocked on the disk, the whole JVM waits for it. Such stalls appear as long real times with little user or system time. On AWS, check the volume’s throughput and operations limits and its credit or burst state.

Random guessing wastes hours. This method moves from the simplest question to the most detailed, and each step narrows the search.

flowchart TD
    A["Symptom: latency spikes, ISR shrinks, timeouts"] --> B["Step 1: fix the time window and the broker or brokers affected"]
    B --> C{"Does the GC log show a long pause at that time?"}
    C -->|Yes| D{"Is it a full collection?"}
    C -->|No| E{"Do safepoint records show a long stop?"}
    D -->|Yes| F["Heap too small, live set too big, or marking too late"]
    D -->|No| G{"Is real time far above user plus system?"}
    G -->|Yes| H["Machine problem: disk, steal, throttling"]
    G -->|No| I{"Is system time high?"}
    I -->|Yes| J["Swap or huge pages"]
    I -->|No| K["Too much garbage or oversized pause target"]
    E -->|Yes| L["Non-GC safepoint work or blocked thread"]
    E -->|No| M["Not a JVM pause: check disk, network, load, client behavior"]

The steps in detail

Questions that speed up the search

A complete worked diagnosis

A team sees complaints: every day around 9:15 in the morning, some producers report timeouts for about ten seconds, and dashboards show a burst of in-sync set shrinks on broker 4. Follow the method step by step.

Step one fixes the window: 9:14 to 9:16, broker 4 only, on weekdays. Only one broker is involved, so the cause is probably local. Step two opens the GC log for broker 4. At 9:15 there is one long collection, a full collection, lasting eleven seconds. Step three reads the times: user time is large, system time is small and real time is close to user time divided by the number of threads. The collector was doing honest work, so this is a heap problem and not a machine problem. Step four looks at the heap shape: after collection the floor sits at 5 of 6 gigabytes, much higher than on the other brokers, and the log shows the humongous allocation cause a few seconds before the full collection.

Step five looks at the surroundings and finds that broker 4 leads the partitions of a new reporting topic whose producers send batches of many megabytes when the business day starts. Those batches become humongous objects that fill the old area. Step six forms the hypothesis and tests it in one controlled change: increase the G1 region size on broker 4 only, cap the producers’ batch size and move half the reporting partitions’ leadership to other brokers. The full collection disappears the next morning. Step seven records the finding: batch-size limits are now part of the onboarding checklist for new topics. The whole diagnosis followed evidence at each step and needed no guessing.

Most real incidents fit one of a handful of patterns. Learn them, and the diagnosis often takes minutes.

Once you know the cause, these are the JVM-level levers you can pull, in order of how often they matter.

Choose a supported Java version

Newer Java versions bring a faster and better G1, and they fix bugs that older ones still have. Kafka 3.x supports several long-term releases, and Kafka 4.0 requires Java 17 for brokers, controllers and tools. Running an old release just because “it works” often means running with slower collectors and unfixed memory bugs. Always test an upgrade in staging under load, since the pause profile can change in either direction.

Size the heap

Start with the starting and maximum sizes set equal, so the heap never needs to resize during operation. A common range for busy brokers is roughly 4 to 8 gigabytes, and Kafka’s own operations guidance quotes 6 gigabytes for a large production cluster. Then measure the heap after collection under peak load. A good sign is that the live set uses well under half of the heap, leaving room for bursts. If it uses more than that, increase the heap in steps of a gigabyte or two, and confirm that page cache hit rates and disk reads do not suffer.

Set G1 with measured intent

The recommended starting point in the Kafka documentation is to use G1 with a modest pause target, an earlier marking threshold than the default and a larger region size. Use those as a baseline, then adjust based on your logs. If mixed collections come too late and full collections occur, lower the marking threshold. If marking runs almost constantly and wastes processor time, raise it a little. If humongous allocations are frequent, increase the region size. If pauses cluster around the target but clients still complain, the target may be simply too high for their deadlines, and the answer might be to reduce the amount of work per collection by reducing the live set.

Pre-touching memory

By default, the operating system gives the JVM memory pages as they are first used, and that first touch can cause small stalls as the heap grows into new memory. A setting called always pre-touch makes the JVM touch all heap pages at startup, which slows the start a little and removes those stalls later. It is worth testing on brokers with large heaps, especially in virtualized environments.

Direct buffers and explicit collections

Java networking code uses direct buffers that live outside the heap. When the total of direct memory approaches its limit, the JVM may request a collection to free unused buffers. If that request causes a full stop-the-world collection, you get a mysterious full pause with a modest heap. Setting the JVM to treat explicit requests as concurrent cycles avoids this and is part of the Kafka documentation’s sample settings.

What about other collectors?

Newer low-pause collectors, such as ZGC and Shenandoah, aim for pauses in the range of a millisecond or so, regardless of heap size. Some teams have experimented with them for brokers. The trade-offs are a higher processor and memory overhead and far less shared operating experience with Kafka than G1. If you are curious, test in a staging cluster with production-like traffic, and compare the whole picture including throughput and page cache use, not just pause length. For most clusters, a well-tuned G1 on a current Java release is the right default.

How to run a safe tuning experiment

Often the best “GC fix” is not a JVM flag. It is making the broker create less garbage and hold less data in the first place.

Control the live set

Control the allocation rate

Protect the broker with quotas

Client quotas cap how many bytes per second, and what share of request processing time, a client may use. A single runaway client, such as a job that reads a year of history at maximum speed, can raise the allocation rate for everyone on that broker. Quotas turn a flood into a manageable stream and are one of the cheapest tools for smoothing garbage collection behavior. Set them per service account and review them regularly.

Spread the load evenly

If one broker leads far more busy partitions than the others, it produces far more garbage. Keep leaders balanced using preferred leader election, keep partitions balanced across brokers, and consider tooling that watches traffic and rebalances automatically. A single hot broker is a common reason why GC pauses seem to follow one machine around.

The advice so far applies to ordinary brokers. Other components and environments change what you can see and what you can fix.

KRaft controllers

In modern Kafka clusters, a small group of controller nodes holds the cluster’s metadata. Controller processes are also Java programs with a heap and a garbage collector. A long pause on the active controller delays every cluster-wide decision: elections, topic creation and broker registrations. Since the controller stores the entire metadata in memory, its heap should be sized for the total partition count. Controllers should run on dedicated, well-provisioned machines in large clusters, and their GC logs need the same monitoring as brokers’.

ZooKeeper in older clusters

If you still use ZooKeeper, remember that it is a separate Java program that also suffers from pauses. A ZooKeeper pause can cause broker sessions to expire even when brokers themselves are healthy. Give ZooKeeper a dedicated machine, a modest heap that is sized so it never swaps, and a fast dedicated disk for its transaction log. When brokers report lost ZooKeeper sessions, check both sides of the connection.

Amazon MSK and other managed services

On a managed service such as Amazon MSK, you do not control the JVM flags, the operating system or the instance internals. What you can do is watch the metrics the service publishes, such as heap memory in use after garbage collection, processor use by user and system, memory and disk metrics, and lag or under-replicated partition counts. Your levers are the workload and the size of the brokers. Reduce partitions and garbage from clients, add brokers to spread load, move to a larger broker size and, when the evidence points at the platform rather than your usage, open a support case with exact times and broker identifiers. Check the current documentation to see which metrics exist for your broker type and monitoring level.

Containers and Kubernetes

When brokers run in containers, memory and processor limits shape the JVM’s behavior. Modern Java versions read the container limits and size their thread pools and default heap accordingly, but you should still set the heap explicitly. Leave enough memory outside the heap for direct buffers, metaspace, thread stacks and the page cache, because the container’s memory limit covers all of them. Avoid tight processor limits for the reasons in Chapter 10, and set requests equal to limits so that the platform treats the broker as a guaranteed workload. Use local or high-performance storage, and do not put GC logs on a shared network volume.

During an incident the goal is to protect users first and to preserve evidence second. Fixing the root cause comes later.

Temporary safety nets

While you work on the root cause, a few settings buy safety. A somewhat longer session timeout for brokers can prevent needless fencing during rare pauses, though it delays the detection of real failures. Allowing producers to retry with sensible back-off protects against short errors. A less strict minimum for in-sync replicas can restore write availability at the cost of some durability, and that trade should be a deliberate business decision, never a silent default. Document each safety net and schedule its removal, so that temporary changes do not become permanent hidden risks.

Communication during a pause incident

Technical work is only half of an incident. Assign one person to keep a running timeline in a shared document, one to talk to affected teams and one to do the hands-on investigation, so that the person analyzing logs is not interrupted every few minutes. Tell client teams plainly what they should do: keep retries with back-off, avoid restarting their applications all at once and expect a short period of catch-up. Clear, calm messages prevent the well-meaning but harmful reactions that make garbage collection worse, such as a hundred applications restarting and reconnecting at the same time.

The best pause is the one you find in a test, before customers do.

What to alert on

Test before you ship

Before changing the JVM version, the flags, the heap size, the client mix or turning on encryption, run a load test on a staging cluster that has the same machine type and a realistic number of partitions. Record GC logs during the test and compare them with your baseline. Include a failure test: stop a broker, then bring it back, and watch the collection behavior during catch-up, because recovery often has the highest allocation rate of all.

Game days

Schedule regular exercises in which the team practices the response in Chapter 16 on a test cluster. Inject a stall using a controlled tool, for example pausing the process for several seconds, and see how the cluster behaves: which alarms fire, how long elections take and what the clients see. The practice reveals gaps in dashboards and runbooks while the stakes are low.

Feed findings back into design

Each incident should improve something structural. If pauses were caused by partition growth, add partition governance. If they were caused by an old client, add a client version policy. If they were caused by a noisy job, add quotas. If they were caused by the environment, add a platform check. Over time, the number of surprise pauses falls, and the ones that remain are easy to explain.

flowchart LR
    CHG["Proposed change: JVM, clients, encryption, size"] --> STG["Staging load test with GC logs"]
    STG --> CMP{"Compare with baseline"}
    CMP -->|Worse| FIX["Adjust plan"]
    FIX --> STG
    CMP -->|Same or better| CAN["Canary on one broker"]
    CAN --> OBS["Watch pauses, ISR, latency for a full cycle"]
    OBS -->|Healthy| ROLL["Roll out gradually"]
    OBS -->|Unhealthy| BACK["Roll back and record findings"]

Every choice about memory and collectors trades one good thing for another. Make the trade on purpose.

Patterns that work

Common mistakes checklist

• Leaving the default one-gigabyte heap on a production broker.
• Judging memory health by total machine memory instead of heap after collection.
• Ignoring GC logs until an incident, then discovering they were never enabled or were overwritten.
• Assuming every stall is a collection without checking safepoints and the machine.
• Running on burstable instances or with tight container processor limits.
• Leaving swap and transparent huge page defaults unexamined.
• Taking a heap dump from a struggling broker still in service.
• Upgrading Java or enabling encryption without a load test.
• Letting partitions, groups and topics grow without limits.

Best practices in one place

Use a current supported Java version with G1. Set equal starting and maximum heap sizes that fit the measured live set. Keep GC logs, JMX metrics and a rolling flight recording. Learn your baseline. Read the three GC times to separate collector problems from machine problems. Keep the live set small through governance, and the allocation rate small through modern clients and quotas. Change one thing at a time, and write down what you learn.

The same story plays out in companies of every size, with the same handful of endings.

Lessons that repeat

Teams that handle pauses well share four habits. They always have GC logs and metrics before a problem starts. They check the machine before blaming the collector. They reduce garbage and live data rather than only tuning flags. And they treat every incident as a chance to improve monitoring and governance. The technology is the same for everyone. The habits are what differ.