Before looking at threads, understand how much a single broker is asked to do at the same moment.

A broker is a busy server

A Kafka broker is one server process in a Kafka cluster. It stores a share of the data and answers requests from many programs. Those programs include producers that write data, consumers that read data, other brokers that copy data between themselves, and administrative tools that ask questions about the cluster. In a large deployment, a single broker may hold thousands of open network connections and receive tens of thousands of requests every second.

Every one of those requests needs a series of small jobs. Bytes must be read from a network connection and assembled into a complete request. The request must be understood, checked for permission, and carried out against files on disk. A response must then be built and sent back. If these jobs were done one after another by a single worker, the broker would crawl. So the broker uses many workers, called threads, that run at the same time.

Why two kinds of workers

The jobs of a broker fall into two very different families. The first family is about moving bytes across the network: waiting for data to arrive, reading it, and later writing the answer. This work is light on thinking but must be extremely responsive, because thousands of connections are waiting. The second family is about doing the task: writing to the log, finding data on disk, checking rules. This work is heavier and can occasionally wait on a slow disk.

Mixing the two would be harmful. If a worker stuck waiting on a slow disk was also the one responsible for greeting new data on the network, then every connection it served would freeze along with it. Kafka therefore separates the families into two pools of threads with a queue between them. That separation is the heart of this tutorial.

Everything that follows depends on a clear picture of what a thread is, and what limits how many are useful.

Programs, processes and threads

When you start a program, the operating system creates a process: a running copy of the program with its own memory. Inside a process, work can be divided among several threads. Each thread is an independent line of execution that follows its own instructions, yet all the threads of a process share the same memory. The Kafka broker is one process containing dozens or hundreds of threads, each with a specific job.

Cores and time slices

A computer has a limited number of cores, the physical units that actually execute instructions. A machine with sixteen cores can truly run sixteen threads at the same instant. If there are more threads than cores, the operating system rotates them, giving each a short time slice. Switching from one thread to another is called a context switch, and it has a cost: the processor must save one thread’s position and load another’s, and its caches become less useful.

This is why threads have a sweet spot. Too few, and cores sit idle while work waits. Too many, and the machine spends more time switching than working.

Busy threads and waiting threads

A thread is either busy, using a core to compute, or waiting, sleeping until something arrives such as data from a disk or a network. Waiting threads use almost no processor time. Threads that mostly wait, for example on a slow disk, can outnumber cores without harm, while threads that compute constantly cannot. Network threads and I/O threads differ in exactly this respect, and that difference guides how many of each to run.

flowchart TD
    P["Broker process"] --> N["Network thread pool"]
    P --> H["I/O thread pool"]
    P --> O["Other pools: replication, cleaning, background"]
    N --> C["Machine cores"]
    H --> C
    O --> C
        

The diagram shows the key fact: every pool competes for the same cores. Adding threads to one pool takes processor time away from the others, so tuning is always about balance rather than about raising one number in isolation.

A small glossary makes every later chapter easier to read.

Three more terms worth knowing

Producer is a client program that sends records to Kafka. Consumer is a client program that reads records. Follower is a broker that copies data from the partition leader so the data survives failures. From the leader’s point of view, a follower is just another client sending fetch requests, and it uses the same threads as everyone else. That detail matters later, because replication traffic competes with client traffic for the same pools.

Finally, latency is the time one request takes from start to finish, and throughput is the number of requests, or bytes, handled per second. Thread tuning is largely a bargain between the two: keeping latency low when the broker is busy, while keeping throughput high.

Follow one request from the moment a client sends it until the client receives an answer.

The six steps

sequenceDiagram
    participant Client
    participant Net as Network thread
    participant RQ as Request queue
    participant IO as I/O thread
    participant Log as Log on disk
    Client->>Net: Send request bytes
    Net->>RQ: Enqueue complete request
    RQ->>IO: Dequeue request
    IO->>Log: Append or read
    Log-->>IO: Result
    IO->>Net: Response via response queue
    Net-->>Client: Send response bytes
        

Why the queue in the middle matters

The request queue is a buffer that lets each group run at its own speed. If a burst of requests arrives, the network threads can keep reading them and queue them up, even when the I/O threads are momentarily busy. If the queue fills up completely, the network threads must pause and stop reading new requests, which is the broker’s way of saying “slow down”. This behaviour is a useful signal, because a full queue is the first visible sign that the I/O threads are the bottleneck.

Zoom out to see every piece of the broker’s request machinery in one picture.

flowchart LR
    CL["Clients and follower brokers"] --> ACC["Acceptor per listener"]
    ACC --> P1["Network thread 1"]
    ACC --> P2["Network thread 2"]
    ACC --> P3["Network thread 3"]
    P1 --> RQ["Shared request queue"]
    P2 --> RQ
    P3 --> RQ
    RQ --> H1["I/O thread 1"]
    RQ --> H2["I/O thread 2"]
    RQ --> H8["I/O thread N"]
    H1 --> LOG["Log and page cache"]
    H2 --> LOG
    H8 --> LOG
    H1 --> PG["Purgatory for delayed requests"]
    H1 --> P1
    H2 --> P2
    H8 --> P3
        

Reading the picture

On the left, clients connect. Each network endpoint, called a listener, has its own acceptor. A broker often has more than one listener: for example one for applications, one for other brokers, and one for administrators. Each listener owns its own group of network threads, so the total number of network threads is the configured count multiplied by the number of listeners.

In the middle sits the single shared request queue. Every network thread of every listener puts requests into it, and every I/O thread takes from it. Because the queue is shared, any idle I/O thread can help with any request, which spreads the work evenly.

On the right are the log files and the operating system’s page cache, a region of memory where the operating system keeps recently used file data. Most reads that a healthy Kafka broker serves come from the page cache and never touch the physical disk at all, which is why I/O threads can be so fast.

Before any request can be read, a connection must exist, and one thread is dedicated to creating it.

What the acceptor does

The acceptor is a single thread per listener whose only job is to notice new connection attempts and accept them. When a client opens a connection, the acceptor completes the handshake at the operating system level and then hands the new connection to one of the network threads. It distributes new connections among the network threads in rotation, so that each thread gets a fair share of connections.

Why distribution by connection matters

Assignment happens once per connection, not once per request. After a connection is given to a network thread, that thread serves it for its whole life. This makes the design simple and fast, but it has an important consequence: the load on network threads depends on how busy each connection is, not merely how many connections each has. If one network thread happens to receive several very chatty connections, for example a heavy producer sending large batches, it may be overloaded while others are nearly idle.

Over time, with many connections and random arrival, this evens out. But in clusters with a small number of very heavy clients, the imbalance can be real, and it can be seen in the individual idle values of the network threads.

Connection limits

The broker can cap how many connections it accepts, both in total and per client address. When a limit is reached, the acceptor stops accepting new connections until existing ones close. These limits protect the broker from a misbehaving application that opens connections in a loop, which could otherwise exhaust memory or file handles. Idle connections are also closed after a configurable time, so connections abandoned by clients do not pile up.

Network threads are the broker’s ears and mouth. They handle a huge number of connections with very little effort each.

One thread, many connections

A network thread does not dedicate itself to a single connection. Instead, it watches all the connections assigned to it at once, using an operating system facility that reports which connections have data ready. When bytes arrive on any of them, the thread reads them. When a connection is ready to accept outgoing bytes, the thread writes to it. This approach is called non-blocking input and output, and it allows a handful of threads to serve thousands of connections.

What a network thread actually does

What limits a network thread

A network thread is limited by the processor time it needs to move bytes. Reading and writing large messages costs more than small ones. Decrypting and encrypting traffic costs much more still. A network thread that is fully busy cannot read the next request in time, and every connection it owns waits. Because these threads are usually computing rather than waiting, the sensible number is close to the number of cores available for network work, and adding more than that rarely helps.

The queue is small, simple, and the single most informative place to look when something is slow.

A bounded waiting line

The request queue has a limit on how many requests it can hold, set by a broker setting whose default is five hundred. It is bounded on purpose. A queue that could grow without limit would happily accept requests until the broker ran out of memory. A bounded queue instead pushes back: when it is full, network threads block while trying to add another request, and therefore stop reading from the network. Clients then see slower responses and their own buffers fill, which slows them down naturally. This is called backpressure.

Reading the queue size

The broker reports the number of requests waiting in the queue at any moment. Three kinds of pictures are common:

A byte-based limit too

Besides the count limit, an optional setting can bound the total bytes of queued requests. It is off by default. When enabled, it protects the broker from a pile of very large requests that would fit within the count limit but consume too much memory, such as large produce batches sent by many clients at once.

flowchart TD
    A["Request queue size"] --> B{"Persistently near the limit?"}
    B -->|"Yes"| C["I/O threads or disk are the bottleneck"]
    B -->|"No"| D{"Clients still slow?"}
    D -->|"Yes"| E["Inspect network threads, TLS cost and the network"]
    D -->|"No"| F["Broker is healthy for this load"]
        

The I/O threads are the kitchen staff. They do the actual work that clients care about.

The request handler pool

The I/O threads form a pool named the request handler pool. Each thread runs the same simple loop: take a request from the shared queue, work out what kind of request it is, run the matching logic, and place the response into the correct response queue. Then it goes back to the queue for the next one. Since all threads take from one queue, work is shared automatically, and no thread sits idle while others have a backlog.

What kinds of work happen here

What limits an I/O thread

Unlike a network thread, an I/O thread often waits. It may wait for the operating system to load data from a physical disk, for a lock on a partition, or for a remote response. While waiting it uses no processor, yet it is unavailable for other work. So the right count for I/O threads depends on how much of their time is spent waiting. The more disk waiting, the more threads are useful, up to the point where the disks themselves become the limit and extra threads only queue up at the disk.

Writing data is the most common heavy request, and it touches every part of the pipeline.

Step by step

A producer sends a produce request carrying batches of records for one or more partitions. The network thread receives it, decrypts it if needed, and queues it. An I/O thread picks it up and checks that this broker is the leader for each partition, that the client is permitted to write, and that the batch is valid. It then appends the batch to the end of the partition’s log, which is a sequential write into the operating system’s page cache.

What happens next depends on the producer’s acknowledgement setting. If the producer asked for no acknowledgement, the broker may answer right away. If it asked for the leader only, the broker answers as soon as its own append is done. If it asked for all in-sync replicas, the broker must wait until the follower brokers have copied the batch. That wait is not spent by an I/O thread. Instead, the request is parked in purgatory, described in the next chapter, so the thread is free to serve others.

sequenceDiagram
    participant P as Producer
    participant N as Network thread
    participant H as I/O thread
    participant L as Leader log
    participant F as Follower brokers
    P->>N: Produce request
    N->>H: Queued request
    H->>L: Append batch
    H->>H: Park request until followers confirm
    F->>L: Fetch new data
    L-->>H: Followers caught up
    H->>N: Response queued
    N-->>P: Acknowledgement
        

What makes produce requests expensive

Several factors push the cost of a produce request up. Large batches take longer to read from the network and to write. Compression that does not match the topic’s setting forces the broker to decompress and recompress the batch, using processor time in the I/O thread. Many partitions in a single request multiply the number of log appends. And a busy disk lengthens each append. Understanding which of these dominates in your cluster tells you whether to add threads, change client settings, or improve disks.

Reading data has a different shape from writing, and it is where network threads work hardest.

Two kinds of readers

Fetch requests come from two kinds of clients. Consumers read data to process it. Followers read data to keep their own copy. Both send fetch requests to the leader, and the leader treats them almost identically. This means that in a cluster with replication factor three, every byte written by a producer is fetched at least twice by followers, before any consumer reads it. Replication traffic is therefore a large, constant share of fetch work.

How an I/O thread answers a fetch

The I/O thread finds the requested position in the log and, when possible, avoids copying the data through the application at all. Instead, it prepares a reference to a region of the file and lets the operating system send it straight to the network connection. This technique is called zero copy, and it is why Kafka can serve enormous read volume with little processor use. Zero copy works only for unencrypted connections. With encryption, the data must be brought into the broker’s memory and encrypted by the network thread, which is much more expensive.

When there is nothing to read

Consumers often ask for data that has not arrived yet. Rather than answering “nothing” repeatedly, which would flood the broker with empty requests, the fetch request includes a maximum wait and a minimum amount of data. If the minimum is not there, the request is parked in purgatory until enough data appears or the wait expires. Again, no thread is held during the wait.

Some requests cannot finish immediately. Kafka parks them so that threads never sit idle waiting.

What purgatory is

The purgatory is an in-memory holding area for requests whose completion depends on something that has not happened yet. A produce request waiting for followers to copy its data sits there. A fetch request waiting for enough new data sits there. Each parked request has a condition to finish and a timeout. When the condition is met, or the timeout arrives, the request is completed and its response is sent. Meanwhile, the I/O thread that started it has moved on.

Why it matters for thread tuning

Without purgatory, every request needing a wait would tie up an I/O thread for the whole duration of the wait. With thousands of consumers polling for new data, all threads would be permanently blocked and the broker would appear frozen. Because of purgatory, a modest pool of I/O threads can serve a very large number of waiting clients. It also means that request duration and thread busy time are different things. A request may take half a second from the client’s view while consuming only a fraction of a millisecond of thread time.

flowchart LR
    A["I/O thread handles request"] --> B{"Can finish now?"}
    B -->|"Yes"| C["Respond immediately"]
    B -->|"No"| D["Park in purgatory"]
    D --> E{"Condition met or timeout?"}
    E -->|"Condition met"| F["Complete and respond"]
    E -->|"Timeout"| G["Respond with what is available"]
    D -.-> H["I/O thread is free for other requests"]
        

The journey is not over when the work is done. The answer still has to reach the client.

Response queues

When an I/O thread finishes a request, it does not write to the network itself. It places the response into the response queue belonging to the network thread that owns the client’s connection. That network thread notices the new response, and writes it to the connection when the connection is ready to accept bytes. This division keeps the rule simple: only network threads touch sockets, and only I/O threads touch logs.

Slow clients and partial writes

Not every client reads its answers quickly. A consumer on a congested link may accept only a few kilobytes at a time. The network thread cannot wait for it, or every other connection would suffer. Instead, it writes as much as the connection accepts, remembers how much is left, and returns to the rest of its work. When the connection is ready again, it continues. In this way, a single slow client does not stop the thread from serving the others, but it does keep part of that thread’s attention.

Time in the response queue

If the network thread is busy, finished responses wait in its response queue before being sent. That wait is one of the components of total request time, and it is a direct sign of network thread overload. A response that took a millisecond to prepare but waited twenty milliseconds to be sent shows exactly where the shortage lies.

flowchart LR
    A["I/O thread finishes work"] --> B["Response queue of owning network thread"]
    B --> C{"Network thread busy?"}
    C -->|"No"| D["Written to socket quickly"]
    C -->|"Yes"| E["Response waits in queue"]
    E --> D
    D --> F["Client receives answer"]
        

Not all requests are equal, and some must never wait behind bulk traffic.

Lightweight but critical requests

Besides data requests, the broker handles small, important ones. Metadata requests let clients discover which broker leads which partition. Group requests let consumers join a group, send heartbeats and record their progress. Heartbeats are especially sensitive: a consumer whose heartbeat is answered too late may be declared dead and removed from its group, causing a disruptive reshuffle of partitions among the remaining consumers. Each of these requests is tiny, yet if they queue behind a pile of large produce requests, they may be delayed enough to cause real harm.

Separating traffic with listeners

Because each listener has its own network threads and its own acceptor, operators can dedicate one listener to broker-to-broker traffic and another to clients. In this arrangement, a flood of client connections cannot starve the replication connections of network threads. Both listeners still share the same request queue and the same I/O threads, so the separation protects the network stage but not the I/O stage.

Control plane isolation

Requests that come from the cluster controller, which manages cluster-wide changes such as leadership moves, can be given their own listener and their own dedicated queue. When this is configured, control requests are processed by a separate thread and are never stuck behind data requests. This matters in very busy clusters, where a delayed leadership change can prolong an incident. It is an advanced option, best introduced with care and after measuring.

Network and I/O threads are the stars, but they are not the only pools competing for cores and disks.

Why they matter here

These pools share cores and disks with network and I/O threads. A large compaction can saturate a disk and slow every I/O thread. A follower that has fallen far behind and fetches aggressively can flood the leader with heavy fetch requests. A rolling restart with many partitions can cause a wave of catch-up work. When tuning, always ask whether the observed slowness is really about network or I/O threads, or whether one of these neighbours is consuming the shared resource.

Replica fetchers and network threads

Since followers ask leaders for data through ordinary fetch requests, adding replica fetcher threads on followers increases the number of concurrent fetch requests arriving at leaders. This can help a lagging follower catch up, but it also raises the load on the leader’s network and I/O threads. Every increase on one side should be considered against the capacity on the other.

A short list of settings governs the whole pipeline. Know each one, its default, and what it changes.

Reading the table sensibly

The defaults of three network threads and eight I/O threads suit a modest broker. They are a starting point chosen to work on small machines, not a recommendation for a large, busy one. Notice also that the ratio between the two, roughly one to three, is deliberate: I/O threads often wait on disks and are cheap to have in larger numbers, while network threads compute and need to stay close to the available cores.

Defaults can differ slightly between versions, and new settings appear over time. Always confirm the exact defaults for the version you run in its official documentation before using any number in this article as a plan.

Modern Kafka lets you resize both pools while the broker keeps running, which changes how you should tune.

Dynamic broker configuration

Historically, changing thread counts meant editing a file and restarting the broker, which is slow, risky, and disruptive for a cluster under stress. Newer versions allow certain settings, including both thread counts, to be changed dynamically, that is, while the broker runs. The change can be applied to one broker or to the whole cluster through an administrative tool. The broker then adds or removes threads on its own.

Practical limits

Dynamic changes should be modest. A common guideline is to move the count by no more than about half up or down in one step, then observe. Reducing threads sharply can leave requests stranded, and raising them sharply can produce an abrupt burst of activity on the disks. After each change, wait long enough to see the idle percentages settle before deciding on another.

Why this changes the approach

Because changes are cheap and reversible, tuning becomes an experiment loop rather than a one-shot guess. Measure, change a little, measure again, and keep or revert. Store the final values in the permanent configuration as well, so that a later restart does not silently return to the old numbers. Dynamic values stored with the cluster survive restarts, but mixing file values and dynamic values can confuse the next person, so document which one is authoritative.

flowchart LR
    A["Measure idle percent and queue"] --> B["Change one setting a little"]
    B --> C["Wait and observe"]
    C --> D{"Improved?"}
    D -->|"Yes"| E["Keep and record"]
    D -->|"No"| F["Revert and rethink"]
    E --> A
    F --> A
        

Two numbers tell you more about thread health than any others.

Idle percent explained

The broker reports how much of the time each pool was idle, meaning waiting with nothing to do. It is expressed as a fraction between zero and one. A value of one means the pool did nothing at all. A value of zero means every thread was constantly busy. The metric for network threads is called the network processor average idle percent, and the one for I/O threads is called the request handler average idle percent.

How to read the values

A widely used rule is to keep both idle values above roughly 0.3 during normal peaks, and to treat sustained values below 0.2 as a problem needing action. These are practical guidelines rather than laws, and each cluster’s own history is the best reference.

Why not just use processor percent

Operating system processor use mixes everything on the machine together. It cannot tell you whether the network stage or the request stage is short of capacity, and threads waiting on a disk show up as low processor use yet are not available. The idle metrics measure exactly what you need: the share of time each pool could not take more work.

The broker records where each request spent its life. Reading these pieces points straight at the guilty stage.

Five slices of time

Every request’s total time on the broker is split into named slices, and each slice belongs to a particular stage of the pipeline.

Using the slices

Compare each slice against the total. If most of the time is request queue time, the queue is the problem and the I/O threads or the disks are the cause. If most is response queue time, the network threads are overloaded. If most is remote time, then threads are not the issue at all, and one should examine followers, network links between brokers, or how the clients set their waits. This simple decomposition prevents a whole class of wasted effort.

flowchart LR
    A["Request arrives"] --> B["Request queue time"]
    B --> C["Local time"]
    C --> D["Remote time"]
    D --> E["Response queue time"]
    E --> F["Response send time"]
    F --> G["Client receives"]
    B -.->|"large means"| H["I/O threads short"]
    E -.->|"large means"| I["Network threads short"]
    D -.->|"large means"| J["Look at followers, not threads"]
        

Combine the queue, the idle values and the time slices into one clear decision.

The decision path

flowchart TD
    S["Clients report slow requests"] --> Q{"Request queue persistently high?"}
    Q -->|"Yes"| I{"I/O idle below 0.3?"}
    I -->|"Yes"| D{"Disk latency high?"}
    D -->|"Yes"| DISK["Fix disks or spread partitions"]
    D -->|"No"| IO["Raise I/O threads, check cores"]
    I -->|"No"| OTH["Look at purgatory and followers"]
    Q -->|"No"| N{"Network idle below 0.3 or response queue time high?"}
    N -->|"Yes"| NET["Raise network threads, check TLS cost"]
    N -->|"No"| CL["Look at clients, network links, and quotas"]
        

Three typical signatures

When the front of the pipeline is the problem, these are the levers, in the order to pull them.

Step 1: Confirm it is really the network stage

Verify that response queue time is large, or that network idle is low with a healthy request queue. Without confirmation, adding network threads is only a guess.

Step 2: Raise the count modestly

Increase the network thread count by a small step, for example from three to five, and watch the idle value. Each thread can use one core at most, so the total across listeners should stay well below the number of cores, leaving room for I/O threads and the operating system. A common ceiling in practice is around a third to a half of the cores, though measurement should decide.

Step 3: Reduce the work per byte

Many network overloads are better solved by making each byte cheaper. Larger, better compressed batches from producers mean fewer requests and fewer bytes to move. Fewer chatty clients or lower polling frequency reduces the count of tiny requests. Reviewing whether every connection really needs encryption between brokers inside a trusted network can free significant processor time, though security requirements must decide that, not performance.

Step 4: Separate traffic

Move replication traffic to its own listener so that client bursts cannot starve followers. Each listener gets its own network threads, which isolates the two kinds of traffic at the network stage.

When the kitchen is overloaded, more cooks help only if the ovens can keep up.

Step 1: Check the disks first

Look at disk latency and utilization. If the disks are saturated, more I/O threads will simply wait in longer lines. Fix the disks by spreading partitions across more drives, using faster media, or reducing write amplification from other activities such as heavy compaction.

Step 2: Raise the count in steps

A guideline is to have at least as many I/O threads as there are data disks, since each disk can serve requests in parallel. Beyond that, increase gradually while the request handler idle value is low and disks still have spare capacity. Since I/O threads often wait, values above the number of cores can be reasonable, but each addition should show a clear improvement in queue time.

Step 3: Reduce the work per request

Align compression between producers and topics to avoid recompression. Avoid extremely large batches that hold a thread for a long time. Limit the number of partitions per request where clients allow it, and remove unused topics that add metadata and housekeeping overhead.

Step 4: Protect important traffic

Use quotas so that one aggressive client cannot occupy all handler time, and consider lowering the intensity of background jobs that compete for disk during peak hours.

The two pools are partners. A change to one shifts pressure onto the other.

Moving the bottleneck

Fixing one stage usually reveals the next. Raise network threads and the request queue may suddenly fill, because requests now arrive faster than the I/O threads can consume them. Raise I/O threads and the response queues may grow, because answers are produced faster than the network threads can send them. This is not failure, it is the ordinary process of finding the next limit. What matters is to stop when the overall latency and idle values are acceptable, not to chase perfect numbers.

A budget for cores

Treat the machine’s cores as a budget. Reserve a share for the operating system and for other pools: replication fetchers, compaction, and background tasks. Divide most of the rest between network and I/O threads. A helpful mental split for a busy broker is that network threads are the expensive, steady consumers of cores and I/O threads are the bursty ones that spend part of their time waiting. Exact splits depend on encryption, compression and disks, so they must be measured.

flowchart TD
    C["Total cores"] --> OS["Operating system and page cache work"]
    C --> BG["Replication, compaction, background"]
    C --> NT["Network threads, steady computing"]
    C --> IT["I/O threads, computing plus waiting"]
    NT --> Q["Request queue"]
    Q --> IT
    IT --> R["Response queues"]
    R --> NT
        

Keep a reserve

Aim to leave headroom, with idle values comfortably above the alarm line during peaks. A broker that is exactly sufficient in normal times will fail during a failover, when the survivors take over the load of a lost broker. Sizing must therefore consider the situation with one broker down, not only with all brokers healthy.

The numbers below are invented to show the reasoning. Your own measurements will differ.

The situation

A broker runs on a machine with sixteen cores and six data disks. It has two listeners, one for clients and one for replication, and uses the defaults of three network threads and eight I/O threads. Producers report timeouts during the evening peak. The team looks at its dashboard at peak time.

Reading the evidence

Request handler idle at 0.55 means the kitchen is comfortable. The request queue is small, so requests are picked up quickly. But network idle is only 0.12 and response queue time is eight times the request queue time. Finished answers are waiting for a network thread. The bottleneck is clearly the network stage of the client listener, and the I/O pool should be left alone.

The change

Two features change the balance between the pools more than almost anything else.

Encryption shifts work to network threads

When connections are encrypted using a secure transport, every incoming byte must be decrypted and every outgoing byte encrypted, and this happens in the network threads. It also removes zero copy for reads, because data must pass through memory to be encrypted. Clusters that adopt encryption commonly see network idle drop dramatically and processor use rise, even though the traffic is the same. Plan to raise network thread counts, and to verify modern processor features for encryption are in use, when enabling it.

Compression shifts work to I/O threads or clients

If producers compress batches with the same type the topic expects, the broker stores them as they are, saving processor time and disk space. If the types differ, the broker must decompress and recompress in the I/O threads. Consumers fetching compressed data receive it compressed as well, so the broker does no work per record on the read path. Consistent compression is thus one of the cheapest performance improvements available.

What to check

Threads are a shared resource, and quotas are how a broker stops one client from consuming them all.

Three kinds of limits

A quota is a limit on how much of a resource a client, or a group of clients, may use. Kafka supports limits on produce bandwidth, on fetch bandwidth, and on request time, which is the share of I/O and network thread time a client may consume. The last kind is the one most directly related to thread tuning. It is expressed as a percentage of a thread’s time, so that a client allowed twenty percent may use up to one fifth of a single thread on average.

How throttling works

When a client exceeds its quota, the broker does not reject its requests. Instead, it computes a delay, completes the request, and holds the response for that time, telling the client how long it was delayed. Well-behaved clients slow down accordingly. The delay is applied outside the busy path, so throttled clients do not tie up threads while waiting.

Why request-time quotas help tuning

Thread shortages are often caused by one or two noisy clients, such as a runaway batch job or a consumer group in a tight polling loop. A request-time quota caps the damage such a client can do to the handler pool, protecting every other client. It is often a better first response than raising thread counts, because it treats the cause, not the symptom.

I/O threads are only as fast as the storage beneath them, so any tuning discussion must include the disks.

The page cache

Kafka writes to files, but the operating system does not write each byte to the physical disk immediately. It keeps recent data in memory, in the page cache, and flushes it to disk in the background. Reads of recent data are served from that memory. This is why a healthy broker with enough free memory has I/O threads that rarely wait for the physical disk, and why memory left for the operating system is as important as the memory given to the Kafka process itself.

When the cache is not enough

A consumer that reads far behind the newest data, for example when restarting after a long outage, asks for data no longer in memory. The operating system must load it from disk, and the I/O thread waits. A handful of such catch-up consumers can turn many handler threads into waiting threads. Request handler idle drops, the queue grows, and healthy real-time clients suffer. The fix is not more threads but limiting catch-up readers with quotas, or scheduling them at quiet times.

Multiple data directories

A broker can spread partitions across several directories, ideally each on its own drive. Different partitions then land on different disks and I/O threads can work on several drives at once. If all data sits on a single volume, the volume becomes the shared limit regardless of thread counts. This is the reason for the guideline that I/O threads should be at least as numerous as data disks.

flowchart LR
    H["I/O threads"] --> C{"Data in page cache?"}
    C -->|"Yes"| F["Fast: memory read, thread returns quickly"]
    C -->|"No"| D["Read from physical disk"]
    D --> W["Thread waits, unavailable for other work"]
    W --> S["Slower response and lower handler idle"]
        

Real incidents rarely announce their cause. Match what you see to a likely explanation.

The cascade to watch for

The most dangerous pattern is a self-reinforcing loop. A spike fills the queue. Heartbeats are delayed. Consumers leave and rejoin, triggering rebalances that create more requests. Producers time out and retry, adding even more requests. Each step worsens the load that started it. Breaking such a loop usually needs relief at its source: throttle the heaviest clients, pause non-essential jobs, and only then adjust threads.

flowchart TD
    A["Traffic spike"] --> B["Request queue fills"]
    B --> C["Heartbeats and responses delayed"]
    C --> D["Consumers rebalance, producers retry"]
    D --> E["More requests arrive"]
    E --> B
    D -.-> F["Break the loop: throttle heavy clients, pause batch jobs"]
        

Some approaches to thread tuning repeatedly work. Others repeatedly cause surprises.

Patterns worth copying

Anti-patterns to avoid

A short checklist, followed by the mistakes it is designed to prevent.

Common mistakes

Different industries stress the two pools in different ways.

Direct answers to the questions that come up most often.

Network threads read requests from and write responses to network connections. I/O threads carry out the requests against the log and metadata. They are separate pools joined by queues.

The network thread count is set by num.network.threads, per listener, and the I/O thread count by num.io.threads. Both can be changed while the broker runs in recent versions.

I/O threads spend part of their time waiting for disks, so more of them can share a core budget. Network threads compute steadily and would compete more heavily for cores.

When network idle stays low, response queue time is large, and the request queue is not full. The finished work is waiting for a network thread to send it.

When request handler idle is low, the request queue stays high, and request queue time is large, while the disks still have spare capacity.

A common guideline is to stay above roughly 0.3 at peak and to act when values stay below 0.2. Your own history is the best reference.

For I/O threads, sometimes, since many are waiting. For network threads, rarely, because they compute steadily. Measure the effect rather than relying on a rule.

Extra threads may have increased context switching or disk contention, or the real bottleneck lies elsewhere, such as a slow disk or followers. Revert and re-diagnose.

Usually yes. Encryption and decryption run in network threads and disable zero copy on reads, raising processor use per byte.

A brief fullness during a burst is normal. A persistently full queue means the I/O stage cannot keep up, and network threads are being forced to wait.

It is time spent in purgatory waiting for followers or for more data. Large remote time points to replication or client settings, not to thread counts.

Yes. Followers send fetch requests like any other client, so replication competes for the same network and I/O threads unless it uses a separate listener for the network stage.

The configured number is applied per listener, so the total is the count multiplied by the number of listeners.

Dynamic changes are convenient for experiments and incidents. Record the final decision consistently so that a restart does not silently revert it.

Modest, often within about half of the current value. Observe through a full peak before another change.

Request-time quotas limit the share of thread time a client can use, preventing one client from starving others. They often solve the problem without adding threads.

Low request handler idle, large local time, and modest processor use. The threads are waiting, not computing.

Their heartbeats wait in a saturated request queue and arrive late, so the group coordinator considers them dead. Relieve the queue and review timeouts.

Often the defaults suffice. Watch the idle values, and tune only when they show pressure.

Compare idle percent, queue size and time slices before and after under similar load, and confirm that client-side latency and timeouts improved too.

A Kafka broker divides its work between two pools joined by queues. Network threads read requests and send responses, and I/O threads carry out the work against the log. A bounded request queue passes work forward, response queues carry answers back, and purgatory parks requests that must wait so that no thread is held idle. Two idle metrics, the queue size and the time slices of each request show which stage is short of capacity. Tuning is then a careful loop: diagnose the saturated stage, change only that pool by a modest step, check that the bottleneck has not simply moved, and record the result. Encryption, compression, disks, followers, quotas and the shared supply of cores all shape the balance, so thread counts are never tuned in isolation.