Broker Configuration Tuning in Apache Kafka
How a Kafka broker really handles requests, what settings such as num.network.threads and num.io.threads control, how to read the right metrics, and how to tune step by step without breaking a production cluster.
Imagine a busy restaurant. Customers arrive at the door, a host greets them and passes their orders to the kitchen, and cooks prepare the food and send it out through waiters. If there are too few hosts, guests queue outside even though the kitchen is quiet. If there are too few cooks, orders pile up on the counter while the hosts stand idle. If there are too many cooks crammed into a tiny kitchen, they get in each other’s way. Tuning a Kafka broker is the same kind of balancing job: making sure each stage of the request journey has the right amount of capacity, so that no single stage becomes the bottleneck.
This tutorial explains broker tuning from the ground up. We start with how a broker handles a request, then examine the main thread pools and the settings that control them, then replication, storage, message sizes and the world outside the configuration file, such as memory and disks. We finish with metrics, a repeatable tuning method, rollout habits and common traps. Every new term is explained the first time it appears, with an everyday comparison.
1Why Tune a Broker at All?
Kafka works out of the box, so it is fair to ask why anyone should touch the settings.
A quick Kafka refresher
Apache Kafka stores streams of events in topics. Producers write events, consumers read them, and brokers, the servers of the cluster, store and serve them. Each topic is split into partitions, and each partition is copied to several brokers for safety. One broker is the leader of a partition and handles the reads and writes, while the others follow it. A broker therefore spends its life accepting connections, reading requests, moving data to and from disk, copying data to followers and sending responses.
Defaults are made for the average case
Kafka’s default settings are chosen to work reasonably on modest hardware for many kinds of workloads. They are safe starting points, not perfect answers. A small development cluster may never need a change. A production cluster on powerful machines with many clients and many partitions often benefits from carefully chosen changes, for example more threads to use the available processor cores, or larger buffers for long network links.
What tuning can and cannot do
Remove internal bottlenecks
Ensure request handling, replication and storage work use the machine’s capacity well.
Balance speed, safety and cost
Choose the right trade-off between latency, throughput and durability for your business.
Beat hardware limits
No setting makes a saturated disk or network card faster. Tuning finds limits, it does not remove them.
Fix poor design
Too many partitions, tiny producer batches or badly chosen keys need design fixes, not thread counts.
Tuning an engine can squeeze more performance from a car, but it cannot turn a small city car into a lorry. You first find out what the car is and what job it must do, and only then adjust it.
2The Tuning Mindset and the Layers of the Stack
Broker settings are only one of several layers that decide performance. Knowing which layer owns a problem saves weeks of wasted effort.
flowchart TB
L6["Layer 6: Client settings, batching, compression, fetch sizes"] --> L5["Layer 5: Topic settings, partitions, replication, retention"]
L5 --> L4["Layer 4: Broker settings, threads, queues, buffers, replication"]
L4 --> L3["Layer 3: JVM, heap size and garbage collection"]
L3 --> L2["Layer 2: Operating system, page cache, file limits, network stack"]
L2 --> L1["Layer 1: Hardware, CPU, memory, disks, network cards"]
A delivery company’s speed depends on the roads, the vehicles, the drivers, the depots, the parcel sizes and the customers’ opening hours. Hiring more drivers will not help if the roads are jammed or if customers send one parcel at a time.
Principles that guide good tuning
Measure before changing
Guessing which stage is slow leads to changes that do nothing or make things worse. Look at the metrics first and let them point at the bottleneck.
Change one thing at a time
If several settings change together and performance improves, you will not know which one helped. If it worsens, you will not know which one to undo.
Know what you are optimising for
Throughput, low latency and maximum durability pull in different directions. Decide your priority before touching any knob.
Prefer fixing the cause over raising limits
A queue that keeps filling is a symptom. Adding capacity may hide a slow disk or a badly behaved client that should be fixed instead.
Keep changes reversible and documented
Record what changed, why, and what the numbers were before and after, so the next engineer understands the reasoning.
Most tuning wins come from a small number of settings and from checking the layers around Kafka. Resist the urge to change dozens of settings because a guide lists them.
3How a Broker Handles a Request
To tune the threads, you must first see the path a request takes through the broker. It looks much like a restaurant’s flow from door to table.
flowchart LR
C["Clients"] --> A["Acceptor: accepts new connections"]
A --> N1["Network threads: read requests from sockets"]
N1 --> RQ["Request queue"]
RQ --> IO["I/O threads: process requests, read and write logs"]
IO --> LOG[("Log files and page cache")]
IO --> RESQ["Response queues, one per network thread"]
RESQ --> N2["Network threads: send responses"]
N2 --> C
The acceptor takes the connection
Each listener has an acceptor that accepts new connections and hands each one to a network thread. It does no other work.
A network thread reads the request
A network thread owns a set of connections. It reads bytes from those sockets, builds complete requests and places them in the shared request queue.
The request waits in the queue
The queue holds requests until an I/O thread is free. Its size is bounded, so it cannot grow without limit.
An I/O thread does the work
An I/O thread, also called a request handler thread, takes the request and does the real job: appending events to a log, reading events for a consumer, answering metadata questions and more.
The response goes back through a queue
The I/O thread places the response in the queue belonging to the network thread that owns the client’s connection.
The network thread sends it
The same network thread that read the request writes the response to the socket and the client receives it.
In the restaurant, the host at the door seats guests, the waiters take orders and place them on a rail in the kitchen, the cooks pick orders from the rail and prepare the food, and the waiters bring the plates back to the same tables. The rail is the request queue.
Each stage has its own capacity. If the waiters cannot keep up, orders reach the kitchen slowly. If the cooks cannot keep up, the rail fills. The settings we tune in the following chapters control the number of waiters, the length of the rail and the number of cooks.
4Network Threads: num.network.threads
These are the waiters. They move bytes between the network and the broker, and they do no heavy thinking.
What they do
The setting num.network.threads controls how many network threads the broker runs for its listeners. Each thread is responsible for a share of the connections. It reads incoming requests, forwards them to the request queue, and sends back the responses that the I/O threads prepared. Network threads also handle encryption work for TLS connections, which makes them busier on secured listeners than on plain ones.
The default and when it is not enough
The default is a small number, three in current versions, which suits light to moderate traffic. When a broker has many connections, large requests, TLS encryption, or high message rates, the network threads can become saturated. Saturation means they are busy nearly all the time, so new requests wait to be read and responses wait to be sent, and latency grows even though the disks and the I/O threads may be idle.
How to tell whether they are the bottleneck
Idle percentage of network threads
The metric for the average idle time of network processors shows what fraction of time they are free. A common rule of thumb is to keep it comfortably above roughly a third. A value close to zero means they are saturated.
Response queue size and response send time
Growing response queues, or long times to send responses, suggest network threads or the network itself cannot keep up.
Latency without disk pressure
If requests are slow but disks are quiet and I/O threads are idle, the network stage is a prime suspect.
If guests wait outside a restaurant while the kitchen is quiet, the problem is not the cooks. There are too few waiters to take orders and carry plates.
Choosing a value
A useful starting point relates the number of network threads to the number of processor cores and the number of listeners, while leaving cores for I/O threads and other work. Increase gradually, watch the idle metric, and stop when it recovers. More threads than cores usually adds contention rather than speed.
Advantages of Raising It
- Faster reading of requests and sending of responses
- Better handling of many connections and TLS work
- Lower request queue waiting caused by network saturation
Disadvantages / Trade-offs
- Uses processor time that other threads may need
- More threads than cores adds context switching
- Does nothing if the real bottleneck is disk or I/O threads
Raising the network thread count because latency is high, without checking the idle metric first. If the network threads are mostly idle, adding more will not help and wastes processor capacity.
5The Request Queue and Backpressure: queued.max.requests
The queue between the network threads and the I/O threads is small on purpose. It protects the broker from overload.
A bounded queue
The setting queued.max.requests limits how many requests may wait in the request queue. The default is a few hundred. When the queue is full, network threads that want to add another request must wait until space becomes free.
Backpressure
Because network threads wait when the queue is full, they stop reading more data from sockets. The unread data fills the operating system’s socket buffers, and eventually the network layer tells the clients to slow down. This chain reaction is called backpressure, and it prevents the broker from accepting more work than it can handle. It is a protective feature, not a fault.
flowchart LR
IO["I/O threads too slow"] --> Q["Request queue fills up"]
Q --> N["Network threads must wait to add requests"]
N --> S["Socket buffers fill"]
S --> CL["Clients slow down or time out"]
The order rail in the kitchen holds only so many tickets. When it is full, waiters cannot hang new tickets, so they stop taking orders, and guests wait longer at their tables. The kitchen is protected from an avalanche of orders.
Should you enlarge the queue?
A bigger queue absorbs short bursts of traffic without slowing clients, but it also lets more work pile up, which increases memory use and the time each request waits. If the queue is constantly full, the real issue is that I/O threads or storage cannot keep up, and a bigger queue only hides the problem while adding delay. Use the queue size metric to see whether it is regularly full or only occasionally.
| Observation | Likely Meaning | Typical Action |
|---|---|---|
| Queue usually near empty | I/O threads keep up | No change needed |
| Queue fills briefly during bursts | Short spikes exceed capacity | Possibly a modestly larger queue, or accept the backpressure |
| Queue constantly full | I/O threads or disks cannot keep up | Find the true bottleneck, do not simply enlarge the queue |
A full queue is a message from the broker telling you that it is overloaded. Read it as a symptom, and look one stage downstream for the cause.
6I/O Threads: num.io.threads
These are the cooks. They do the real work of every request, and they are where disk speed shows up.
What they do
The setting num.io.threads controls how many request handler threads process requests from the queue. They append incoming events to log files, read events for consumers and followers, handle metadata requests, manage group coordination and answer administrative calls. Since much of their work touches storage, their speed depends on disks and on the operating system’s page cache.
The default and when it is not enough
The default is a small number, eight in current versions. A broker with many disks, many partitions, many clients or heavy replication traffic can keep more I/O threads usefully busy. The documentation suggests the count should at least reflect the number of disks, because each disk can handle work in parallel. On the other hand, adding threads when the disk is already saturated only creates a longer line of threads waiting for the same disk.
How to tell whether they are the bottleneck
Idle percentage of request handlers
The metric for average idle time of request handler threads shows how much of the time they are free. A common rule of thumb is to keep it well above a fifth. When it approaches zero, requests are waiting for a handler.
Request queue size and queue time
A growing request queue and rising time spent waiting in it show that handlers are not keeping up.
Disk utilisation and wait times
If the disks are nearly fully busy while handlers are also busy, more threads will not help. The disk is the limit.
Processor usage
If cores are already busy, more handler threads may compete for them rather than add capacity.
Hiring more cooks helps only if there are free stoves. If every stove is in use, extra cooks just stand in line. The number of cooks should match the number of stoves and the space in the kitchen.
flowchart TD
S["Requests waiting in the queue"] --> Q1{"I/O threads mostly busy?"}
Q1 -- "No" --> N["Look at network threads or clients"]
Q1 -- "Yes" --> Q2{"Disks saturated?"}
Q2 -- "Yes" --> D["Fix storage: faster disks, more disks, fewer partitions per disk"]
Q2 -- "No" --> Q3{"Processor cores saturated?"}
Q3 -- "Yes" --> C["Add cores or reduce work: compression, TLS, request rate"]
Q3 -- "No" --> T["Raise num.io.threads gradually"]
Advantages of Raising It
- More requests processed in parallel across disks
- Shorter queue waits under heavy load
- Better use of multi-disk and many-core machines
Disadvantages / Trade-offs
- More contention on shared locks and disks
- No benefit if storage is already saturated
- Extra memory and scheduling overhead
Setting the I/O thread count to a very large number in the hope of general speed-up. Beyond what the disks and cores can support, extra threads increase contention and can make latency worse.
7Where the Time Goes: Reading Request Timing Metrics
Kafka breaks the life of every request into stages and measures each one. This breakdown tells you exactly which stage to tune.
sequenceDiagram
participant C as Client
participant N as Network thread
participant Q as Request queue
participant H as I/O thread
participant R as Replicas and purgatory
C->>N: Request arrives
N->>Q: Request queued
Note over Q: Request queue time
Q->>H: Handler picks it up
Note over H: Local time, work done on this broker
H->>R: Wait for followers or data if needed
Note over R: Remote time
R-->>H: Condition met
H->>N: Response queued
Note over N: Response queue time
N-->>C: Response sent
Note over N,C: Response send time
| Stage | What It Measures | If It Is High | Where to Look |
|---|---|---|---|
| Request queue time | Waiting for a free I/O thread | I/O threads are overloaded | I/O thread count, disks, processor |
| Local time | Work done by the leader broker, such as writing to the log | Slow storage or heavy processing | Disks, page cache, flush settings |
| Remote time | Waiting for followers to confirm, or for enough data to fetch | Slow replication or intentional fetch waiting | Replica fetchers, network between brokers, acknowledgement settings |
| Response queue time | Waiting for a network thread to send | Network threads overloaded | Network thread count |
| Response send time | Time to push bytes to the client | Slow client or network path | Socket buffers, network bandwidth, client speed |
A restaurant manager who times each stage of an order, from waiting at the door, waiting on the rail, cooking, waiting for the waiter and eating, knows exactly whether to hire more cooks, more waiters or to buy a bigger oven.
Reading it in practice
Look at the timing breakdown separately for the main request types: producing, consumer fetching and follower fetching. A produce request with a high remote time usually reflects waiting for replicas. A consumer fetch with a high remote time may simply reflect the fetch waiting for data as configured by the client, which is normal and not a fault. Always interpret the numbers in light of what each request type is supposed to do.
Build a dashboard with the five stages stacked for produce requests. When latency rises, the stage that grew tells you which part of the broker to investigate, and saves guesswork.
8Replication Settings: Replica Fetchers and Lag
Every broker also acts as a client of other brokers, copying data for the partitions it follows. Replication has its own threads and rules.
Replica fetcher threads
The setting num.replica.fetchers controls how many threads a broker uses to fetch data from each other broker for the partitions it follows. With the default of one, all follower traffic between a pair of brokers goes through a single thread, which can limit replication speed when there are many partitions or a lot of data. Raising it adds parallelism, and the effect multiplies with the number of source brokers.
flowchart LR
L1["Broker 1: leader for some partitions"] --> F1["Replica fetcher threads on Broker 2"]
L3["Broker 3: leader for other partitions"] --> F2["Replica fetcher threads on Broker 2"]
F1 --> B2["Broker 2 stores follower copies"]
F2 --> B2
Signs that replication is struggling
Under-replicated partitions
Partitions whose followers have fallen behind and dropped out of the in-sync set. Persistent under-replication points to slow replication, an overloaded follower or network limits.
Frequent shrinking and expanding of the in-sync set
Followers dropping out and rejoining repeatedly suggests they only just keep up, and small delays push them over the lag limit.
High remote time on produce requests
When producers require acknowledgement from all in-sync replicas, slow followers directly slow the producers.
Related settings to know
replica.lag.time.max.ms
How long a follower may go without catching up before being removed from the in-sync set. Too low causes flapping, too high delays detection of a truly slow follower.
replica.fetch.max.bytes
The amount of data a follower asks for per partition in one fetch. It should be at least as large as the biggest message allowed.
replica.fetch.wait.max.ms
How long a leader may wait to gather data before answering a follower fetch, balancing latency and request count.
replica.socket.receive.buffer.bytes
The network buffer used for replica fetching, relevant on long or fast links between brokers or data centres.
Copying a large library’s books to a second building needs enough movers and enough trucks. One mover on one route is fine for a small library, but a big one needs several movers working in parallel, or the second building always lags behind.
Raising the allowed lag time to hide flapping in the in-sync set. That makes the metric look better while allowing badly lagging followers to remain in the set, which weakens the safety that replication is supposed to give.
9Background Work: Recovery, Log Cleaning and Housekeeping Threads
Besides serving requests, brokers do a lot of housekeeping. It has its own thread pools, and it matters most at start-up and during heavy retention work.
flowchart TB
BR["Kafka broker"] --> P1["Network threads: sockets"]
BR --> P2["I/O threads: request processing"]
BR --> P3["Replica fetchers: follower replication"]
BR --> P4["Recovery threads: log loading at start-up and flushing at shutdown"]
BR --> P5["Log cleaner threads: compaction of compacted topics"]
BR --> P6["Background threads: deletion and other housekeeping"]
num.recovery.threads.per.data.dir
Threads per data directory used to recover logs at start-up after an unclean stop and to flush logs at shutdown. The default is one.
log.cleaner.threads
Threads that compact log-compacted topics, keeping only the latest value per key. Compaction uses memory and disk bandwidth.
background.threads
General housekeeping threads, for example deleting old log segment files.
log.cleaner.dedupe.buffer.size
Memory the cleaner uses to track which keys have been seen. Too small slows compaction of large topics.
Recovery threads and restart time
After a broker stops uncleanly, it must check its log files before rejoining the cluster. With many partitions on many disks and only one recovery thread per directory, this can take a long time. More recovery threads per directory speed up start-up and shutdown, which shortens rolling restarts and reduces the time a broker is unavailable. It is one of the most useful settings for clusters with thousands of partitions.
Log cleaner threads and compaction
Topics that use compaction depend on the log cleaner to reclaim space. If the cleaner cannot keep up, compacted topics grow larger than expected and consumers that start from the beginning read more data. Signs of trouble include a cleaner that is constantly busy, a large fraction of uncompacted data, or a dead cleaner thread. Raising the number of cleaner threads or the buffer size can help, as long as disk bandwidth is available.
A restaurant has staff who serve guests, but also staff who open the kitchen in the morning, clean at night and take out rubbish. A big kitchen needs more people at opening time, or it opens late every day.
Measure broker start-up time after an unclean shutdown in a test environment. If it is measured in tens of minutes, more recovery threads are usually worth trying.
10Socket Buffers and Connection Settings
Buffers are the waiting areas between the broker and the network. Their size matters most on fast or long links.
Send and receive buffers
The settings socket.send.buffer.bytes and socket.receive.buffer.bytes set the size of the operating system buffers for each connection. If they are too small on a link with high bandwidth and long delay, the sender cannot keep enough data in flight, and throughput stays far below what the link could carry. Setting a value of minus one lets the operating system choose and adjust sizes automatically, which is often the best approach on modern systems.
Request size limit
The setting socket.request.max.bytes caps the size of a single request. It protects the broker from being forced to allocate huge memory for one request, and it must be large enough to fit the biggest legitimate requests your clients send.
Connection limits and idle timeouts
Maximum connections
Overall and per-address limits protect the broker from being swamped by a misbehaving application that opens thousands of connections.
Idle connection timeout
Connections idle for too long are closed to free resources. A very short value causes clients to reconnect often, while a very long one wastes resources on dead connections.
Connection creation rate limits
Limits on how fast new connections may be created protect against reconnect storms, for example after a network blip when thousands of clients reconnect at once.
File descriptor limits
Each connection and each open log file uses a file descriptor from the operating system’s limited supply. Brokers with many partitions and clients need this limit raised well above the default.
Think of a pipe carrying water over a long distance. If you fill it with only a trickle at a time, the far end stays nearly empty even though the pipe is wide. Larger buffers let more water be in the pipe at once, which matters most on a long pipe.
Setting very large fixed buffer sizes on every connection of a broker that has thousands of connections. Memory use multiplies, and the broker may run short. Prefer letting the operating system tune buffers, or raise them only where links need it.
11Message Size Settings Must Line Up
Allowing larger messages requires changes on several settings at once, and mismatches cause confusing failures.
The chain of size limits
A message travels from a producer through the broker, to followers and then to consumers. Each stage has its own limit. If any stage has a smaller limit than the message, that stage fails, and the failure often looks unrelated to the real cause.
flowchart LR
P["Producer: max.request.size"] --> B["Broker: message.max.bytes"]
B --> R["Followers: replica.fetch.max.bytes"]
B --> C["Consumers: max.partition.fetch.bytes and fetch.max.bytes"]
| Setting | Where | What It Limits | If Too Small |
|---|---|---|---|
| max.request.size | Producer | Largest request the producer will send | Producer rejects the message before sending |
| message.max.bytes | Broker, or topic level override | Largest batch the broker accepts | Broker rejects the write |
| replica.fetch.max.bytes | Broker as a follower | Data followers request per partition | Followers cannot copy large batches and fall out of sync |
| max.partition.fetch.bytes | Consumer | Data fetched per partition per request | Consumer may be unable to read large messages or slow down |
| socket.request.max.bytes | Broker | Largest request accepted on the network | Connection is closed on very large requests |
A big sofa must fit through the shop door, the delivery van, the customer’s front door and the living room door. A single narrow doorway anywhere along the route stops the delivery, no matter how wide the others are.
Think before allowing large messages
Very large messages put strain on memory, slow down replication and lengthen garbage collection pauses. Consider whether the data would be better stored in an object store, with only a small reference sent through Kafka. If large messages truly are needed, raise the limits deliberately at every stage and test them together.
Prefer raising the limit for one topic through a topic-level override instead of raising it for the whole cluster. Only the topics that need large messages then bear the cost.
12Log Storage Settings: Directories, Segments, Retention and Flushing
How Kafka lays out and manages its files affects disk usage, recovery time and read speed.
Data directories
The setting log.dirs lists one or more directories where partition data is stored. Listing several directories on separate physical disks spreads the load, and the broker places new partitions across them. Combined with I/O threads and recovery threads, this is how brokers use many disks in parallel.
Segments
Each partition is stored as a series of segment files. The segment size setting controls when a new one starts. Smaller segments mean more files and more open file handles, but finer-grained deletion. Larger segments mean fewer files but coarser deletion and longer index work. The defaults suit most workloads, and changes should be made with a clear reason.
Retention
Retention settings decide how long or how much data is kept before old segments are deleted, by time or by size. Retention affects disk usage, and, through it, recovery and rebalancing time, since more data means more to copy when a broker is replaced.
Flushing to disk
Kafka does not force every write to disk immediately. It writes to the operating system’s cache and relies on replication for durability, with the operating system flushing to disk in the background. Settings exist to force flushes at fixed intervals, but leaving them at their defaults is the recommended approach for nearly all deployments. Forcing frequent flushes reduces throughput and rarely improves real safety, since replication already protects against single-machine loss.
Multiple data directories
Use separate disks to parallelise work, and plan for how a failed disk is handled.
Segment size
Balances the number of files against the granularity of deletion and index work.
Retention by time and size
Controls disk use and the amount of data to move during recovery or reassignment.
Flush settings
Best left at defaults. Forced flushing costs throughput and is rarely needed with replication.
A newspaper archive can bind papers into big volumes or small ones, and can keep them for a month or a decade. Small volumes are easier to discard but produce a crowded shelf. Large volumes save shelf space but throw away in big chunks.
Forcing a flush to disk after every message in the belief that it makes data safer. It cuts throughput sharply, and the real protection of data comes from replication and safe acknowledgement settings.
13Trade-offs: Latency, Throughput and Durability
No setting is free. Every choice shifts the balance between speed, volume and safety.
flowchart TB
G["Your priorities"] --> L["Low latency: fast individual responses"]
G --> T["High throughput: most data per second"]
G --> D["Strong durability: no acknowledged data lost"]
L --- T
T --- D
D --- L
| Goal | Typical Direction | Cost |
|---|---|---|
| Lower latency | Smaller batches, shorter waiting times, more threads | Lower throughput per request, more overhead |
| Higher throughput | Larger batches, compression, longer waiting to fill fetches | Higher latency for individual events |
| Stronger durability | Acknowledgement from all in-sync replicas, higher replication factor, minimum in-sync replicas of at least two, no unclean leader election | Higher latency, lower availability during failures |
Settings that shape the balance
Replication factor and minimum in-sync replicas
A replication factor of three with a minimum of two in-sync replicas is a common durable setup. It survives one broker failure without losing acknowledged data, but refuses writes if too many replicas are unavailable.
Unclean leader election
Allowing an out-of-date replica to become leader improves availability after failures but may lose acknowledged data. Most durable setups keep it disabled.
Client acknowledgement level
Producers choose how many replicas must confirm a write. This is a client setting, but the broker settings above decide what that confirmation is worth.
Fetch waiting behaviour
How long and how much data a fetch waits for trades a few milliseconds of latency for far fewer requests, which can lift broker capacity considerably.
A courier can leave immediately with one parcel, which is fast for that parcel but costly overall, or wait to fill the van, which is cheap per parcel but slower. Insuring every parcel with signatures at every step is safest but slowest.
Write down your priority order for each topic, for example durability first for payments, throughput first for click streams. Then tune each accordingly, and avoid one-size-fits-all settings.
14Beyond the Config File: Operating System, JVM, Disks and Network
Some of the most important tuning happens outside Kafka’s own configuration.
The Java virtual machine
Kafka brokers run on the Java virtual machine. Its heap, the memory area for Java objects, should be sized moderately, not as large as possible. Brokers rely heavily on the operating system’s page cache, and a huge heap takes memory away from it. Garbage collection, the process of cleaning unused objects, can pause the broker, so a modern collector configured for short pauses is standard practice. Long pauses cause followers to drop out of the in-sync set and clients to time out.
The operating system
Page cache and free memory
Leave plenty of memory free for the operating system to cache log files. Consumers that read recent data are usually served from this cache without touching the disk.
File descriptor and memory map limits
Each log segment uses open files and memory-mapped index files. Brokers with many partitions need much higher limits than the operating system defaults.
Swapping
Swapping memory to disk causes severe stalls. Configure the system to avoid swapping on broker machines.
File system choice and options
Widely used, well-tested file systems perform well. Options such as not recording file access times reduce needless writes.
Network stack
Operating system limits on buffers and connection queues affect how many clients and how much bandwidth a broker can use.
Disks
Kafka’s access pattern is mostly sequential writes and reads, which suits many disk types, but latency and throughput of the disks decide the ceiling of a broker. Separate disks for separate data directories let I/O threads work in parallel. Sharing disks with other busy programs causes unpredictable delays.
Network
Bandwidth on the broker’s network cards is shared between producers, consumers and replication. With a replication factor of three, every byte written by producers is sent over the network twice more, to the followers, before consumers even read it. Networks must be sized with that multiplication in mind.
A restaurant’s speed depends not only on its staff but on the size of its stoves, the strength of the electricity supply, the width of its doorways and the deliveries of ingredients. Improving the staff schedule does not help if the power keeps tripping.
Giving the Java heap most of the machine’s memory. The page cache then has little room, reads hit the disk more often, and performance drops even though the heap looks generous.
15The Page Cache and Memory Planning
Kafka’s speed relies on a clever trick: it uses the operating system’s cache as its memory, not a large cache of its own.
How it works
When a broker writes events, they land in the operating system’s page cache and are written to disk later. When a consumer reads recent data, the data is usually still in the page cache, so no disk read is needed. The operating system can also pass data from the cache straight to the network without copying it through the application, a technique known as zero-copy, which saves processor time. Zero-copy does not apply when the connection is encrypted with TLS, since the data must be processed by the broker first.
flowchart LR
P["Producer write"] --> PC["Operating system page cache"]
PC -->|"Background flush"| D[("Disk")]
PC -->|"Recent reads served from memory"| C["Consumers"]
D -->|"Old data read from disk"| PC
Planning memory
A rough way to plan is to think about how much recent data you want to keep in memory. If consumers normally read data seconds or minutes after it was written, you need cache room for that window of data across all active partitions. Machines with generous memory beyond the Java heap are standard for busy brokers. Consumers that read old data, such as a job that re-reads days of history, bypass the cache and hit the disks hard, which can slow other clients.
| Memory Use | Who Uses It | Guidance |
|---|---|---|
| Java heap | Broker objects, buffers, metadata | Moderate size, tuned for short garbage collection pauses |
| Page cache | Operating system, holding log data | As much as possible of the remaining memory |
| Off-heap and native memory | Network buffers, memory-mapped files | Leave headroom, and watch limits |
| Operating system and other agents | Monitoring agents, tools | Reserve a small fixed share |
A chef keeps the most-used ingredients on the counter instead of walking to the store room for each order. The bigger the counter, the more is within reach. Giving the chef a giant personal notebook instead of counter space would not help cooking.
Watch disk read rates for consumers that should be reading recent data. If they read heavily from disk, the page cache is too small for the workload, or some client is reading old data and evicting recent data.
16Changing Settings Safely: Static and Dynamic Configuration
Some settings need a restart, and others can be changed on a running broker. Knowing which is which shapes how you tune.
Three kinds of broker settings
Read-only
Changed only by editing the configuration file and restarting the broker. Many structural settings fall here.
Per-broker dynamic
Can be changed on one specific broker while it runs, using the administration tools.
Cluster-wide dynamic
Can be changed for all brokers at once while they run, stored in the cluster’s metadata.
Settings often changed dynamically
Many of the settings in this tutorial can be changed on a running broker, including the sizes of the network, I/O, replica fetcher, recovery and log cleaner thread pools, and several log and connection settings. Thread pool changes take effect gradually, and there are limits on how far a single change may move a pool size, so large adjustments are made in steps. Always check the documentation for your version to see which settings are dynamic and what limits apply.
Precedence
When the same setting is defined in several places, a per-broker dynamic value overrides a cluster-wide dynamic value, which overrides the static file value, which overrides the built-in default. Knowing the order avoids confusion about why a change does not seem to apply.
flowchart TB
A["Per-broker dynamic value"] --> B["Cluster-wide dynamic value"]
B --> C["Static value in the configuration file"]
C --> D["Built-in default"]
Some restaurant rules are fixed in the building plans and need a builder to change. Others are on a whiteboard that the manager can rub out at any time. And a note on one particular table overrides the general whiteboard rule for that table.
Changing a dynamic value in the running cluster and forgetting to update the static file. After a restart, the broker may return to the old file value, and the change appears to vanish. Keep the file in step with what you rely on.
17The Dashboard: Metrics That Guide Tuning
Good tuning is driven by a small, well-chosen set of metrics. These are the ones experienced operators watch first.
| Metric | What It Shows | Rule of Thumb | Related Setting |
|---|---|---|---|
| Network processor idle percent | Free time of network threads | Keep comfortably above about a third | num.network.threads |
| Request handler idle percent | Free time of I/O threads | Keep well above about a fifth | num.io.threads |
| Request queue size | Requests waiting for handlers | Should be small most of the time | queued.max.requests |
| Response queue size | Responses waiting to be sent | Should be small most of the time | num.network.threads |
| Request time by stage | Where each request spends its time | Compare against your own baseline | Depends on the stage |
| Under-replicated partitions | Followers behind the leader | Should be zero in steady state | num.replica.fetchers, replica.lag.time.max.ms |
| In-sync set shrinks and expands per second | Flapping followers | Should be near zero in steady state | Replication and network health |
| Bytes in and out per second | Traffic volume | Compare against network capacity | Capacity planning |
| Disk utilisation and wait | Storage saturation | Watch for sustained high values | Disks, data directories |
| Garbage collection pause time | JVM pauses | Pauses should stay short | Heap and collector settings |
| Purgatory sizes | Requests waiting for conditions | Watch for steady growth | Acknowledgement and fetch waiting |
The rules of thumb above are common starting guidelines, not laws. Establish your own baselines during healthy periods, and judge changes against those.
Baselines and dashboards
Record what the metrics look like during normal and peak periods, so you can spot deviations. A dashboard that groups thread idle times, queues, request time stages, replication health and resource use on one screen makes diagnosis fast. Add annotations for changes and deployments, so you can connect a shift in the graphs with its cause.
A pilot flies with a small set of key instruments in front of them: speed, height, fuel and direction. They do not read every dial in the aircraft each moment, but they know exactly which ones to check first.
18From Symptom to Setting: A Diagnostic Guide
A practical way to tune is to start from what you observe and follow the path to the likely cause.
flowchart TD
S["Clients report high latency"] --> Q1{"Network thread idle low?"}
Q1 -- "Yes" --> N["Raise network threads, check TLS load and bandwidth"]
Q1 -- "No" --> Q2{"Request queue time high?"}
Q2 -- "Yes" --> H["Check I/O threads, disks and processors"]
Q2 -- "No" --> Q3{"Remote time high?"}
Q3 -- "Yes" --> R["Check replication: fetchers, followers, network between brokers"]
Q3 -- "No" --> Q4{"Local time high?"}
Q4 -- "Yes" --> D["Check disks, page cache, garbage collection"]
Q4 -- "No" --> C["Check clients, batching and network path"]
| Symptom | Likely Cause | What to Check First | Possible Action |
|---|---|---|---|
| Slow produce with idle disks | Network stage saturated | Network processor idle percent, response queue | Raise network threads, check TLS |
| Request queue constantly full | I/O stage overloaded | Handler idle, disk wait, processor use | Fix the true bottleneck, then consider more handlers |
| Produce with acknowledgement from all replicas is slow | Followers lagging | Under-replicated partitions, replica fetch rates | More replica fetchers, review network and disks of followers |
| Followers keep dropping from the in-sync set | Slow followers or long pauses | Garbage collection pauses, disk and network of followers | Fix pauses and capacity, avoid just raising the lag limit |
| Very slow broker start-up | Log recovery work | Number of partitions and directories, recovery threads | Raise recovery threads per directory |
| Compacted topics larger than expected | Log cleaner behind or stopped | Cleaner metrics and logs | More cleaner threads or buffer, check for failures |
| Consumers read from disk constantly | Page cache too small or old data being read | Memory use, disk read rate, who reads old data | More memory, isolate replay jobs |
| Large messages rejected | Mismatched size limits | Producer, broker, replica and consumer limits | Align limits, prefer a topic-level override |
A doctor does not give the same medicine for every complaint. They ask about the symptoms, run a few targeted tests, and only then choose a treatment. Tuning brokers follows the same discipline.
Most latency problems are found by two questions: which stage of the request grew, and is any resource, disk, processor or network, saturated? The answers usually point straight at the fix.
19A Repeatable Tuning Method
A steady loop of measuring, changing and checking works far better than one-off guesses.
flowchart LR
A["Measure baseline"] --> B["Find the bottleneck"]
B --> C["Form one hypothesis"]
C --> D["Change one setting in a test"]
D --> E["Measure again under similar load"]
E --> F{"Improved without side effects?"}
F -- "Yes" --> G["Roll out gradually and document"]
F -- "No" --> H["Revert and try another idea"]
G --> A
H --> B
State the goal in numbers
For example, keep the ninety-ninth percentile of produce latency under a set limit at the peak rate. A clear target tells you when to stop.
Capture a baseline
Record the key metrics at normal and peak load, plus the current configuration and hardware. Without this, you cannot judge any change.
Find the bottleneck
Use the timing stages, thread idle percentages and resource metrics to see which stage or resource is saturated.
Reproduce load in a test environment
Use realistic message sizes, partition counts, client counts and replication settings. A test with unrealistic traffic gives misleading answers.
Change one setting and compare
Apply a single change, run the same test, and compare the metrics with the baseline, including side effects such as processor use and garbage collection.
Roll out gradually
Apply the change to one production broker, watch for a while, then extend to others, keeping an easy way back.
Document and revisit
Record the reason, evidence and result. Review after hardware, workload or version changes, since good settings age.
A recipe developer changes one ingredient at a time and tastes after each change. If they changed salt, spice and cooking time together, they would never know which one made the dish better or worse.
Keep a tuning log in your team’s shared documents. When performance changes months later, the log shows what was tried and why, and saves the team from repeating old experiments.
20Working With Clients, Quotas and Partition Counts
Many broker problems are caused or cured outside the broker. Tuning should look at the whole conversation between clients and brokers.
Client batching
Producers that send tiny batches create many small requests. Each request costs network and handler time regardless of size. Raising producer batching and allowing a short wait to fill batches reduces broker load enormously, often more than any broker setting can. Compression also reduces network and disk use, at the cost of some processor time at the producer and at the consumer.
Consumer fetch behaviour
Consumers that fetch with no minimum data and no waiting time create a stream of near-empty requests. Allowing consumers to wait briefly for a reasonable amount of data cuts request rates and broker work.
Quotas
Quotas limit the bandwidth or request share a client or user may consume. They protect the whole cluster from one noisy application, and are a governance tool rather than a performance tweak. Combine sensible thread and queue settings with quotas so that one client cannot use up all the handler time.
Partition counts
Each partition adds files, memory, replication work and metadata. A broker with far more partitions than it needs spends resources on housekeeping, slows recovery and has more leadership elections to do after a failure. Reducing needless partitions is often a bigger win than adjusting any thread count.
flowchart TB
CL["Client behaviour: batching, compression, fetch waiting"] --> RATE["Request rate reaching brokers"]
PC["Partition counts and replication factor"] --> WORK["Housekeeping and replication work"]
Q["Quotas"] --> FAIR["Fair share between clients"]
RATE --> BR["Broker load"]
WORK --> BR
FAIR --> BR
A kitchen struggles when customers place a hundred tiny orders instead of one big one, or when the menu has thousands of items. Improving how customers order and how the menu is designed often helps more than hiring more cooks.
Adding more broker threads to cope with a flood of tiny requests from badly configured clients. The clients should batch better. Broker changes only treat the symptom.
21Security and Governance of Configuration Changes
Broker settings can affect safety and availability, so changing them deserves the same discipline as changing code.
Restrict who can change configuration
The ability to alter broker settings is powerful. Give it to a small group, protect it with access rules, and log every change with the person who made it.
Keep configuration in version control
Store the static files in a repository with review, so changes are visible, discussed and reversible. Keep secrets out of it and use secret providers instead.
Review changes that affect durability
Settings such as replication, minimum in-sync replicas and unclean leader election affect data safety. Require sign-off from the data owners before changing them.
Guard against configuration drift
Brokers in a cluster should normally have consistent settings unless there is a deliberate reason. Regular automated comparisons find accidental differences.
Protect the tuning environment
Test clusters used for load experiments should not hold real customer data, or should hold it under the same protections as production.
Audit dynamic changes
Changes made on running brokers leave less visible traces than file edits. Capture them in an audit log and reconcile them regularly with the files.
In an aircraft, only certified engineers may adjust the engines, every change is recorded in a logbook, and safety-critical changes need a second signature. Broker settings deserve the same seriousness.
22Rolling Out Changes Without Downtime
Even a good change can cause trouble if it is applied carelessly. A gradual rollout limits the damage.
flowchart TD
A["Change approved and documented"] --> B["Confirm cluster is healthy: no under-replicated partitions"]
B --> C["Apply to one broker"]
C --> D["Watch metrics for a set period"]
D --> E{"Healthy?"}
E -- "No" --> R["Roll back that broker"]
E -- "Yes" --> F["Apply to next broker"]
F --> D
Check the cluster is healthy first
Do not start a change while partitions are under-replicated or a broker is recovering. Fix existing problems before adding new ones.
Pick a quiet, representative broker
Start with one broker, ideally at a time of moderate load, and keep an eye on the metrics that the change should affect.
Apply and observe
For dynamic settings, apply without restart. For static settings, restart the broker gracefully and wait for it to rejoin and catch up before continuing.
Compare with untouched brokers
The unchanged brokers act as a control group, showing whether the change or the general load caused any movement in the graphs.
Continue or roll back
If metrics improve or stay healthy, continue broker by broker. If not, revert the change, and record what happened.
A restaurant testing a new kitchen layout tries it in one station first, keeps the others as before, and compares how orders flow. Only after it clearly helps does the whole kitchen change.
Restarting several brokers at the same time to finish sooner. Partitions may lose enough replicas to become unavailable, and recovery work overlaps, multiplying the disruption.
23Design Patterns and Anti-Patterns
Habits that keep tuning effective, and traps that cost teams time and stability.
Patterns worth adopting
Metrics before knobs
Never change a setting without a metric that says it is needed and a metric that will show whether it helped.
Standard configuration profiles
Define a small number of tested profiles for different hardware and workload types, and apply them consistently across clusters.
Load test before big changes
Maintain a test cluster and a realistic load generator, so changes and version upgrades can be tried safely.
Fix clients and design first
Check batching, fetch behaviour, partition counts and key design before adding broker capacity or threads.
Anti-patterns to avoid
Problem
Copying a configuration from a blog post or another company’s cluster into production without testing.
Why It’s Harmful
Their hardware, workload and goals differ from yours. Values that helped them may hurt you, and nobody on your team understands why the values were chosen.
Correct Approach
Use outside advice as a source of ideas, and test each idea against your own metrics and load.
Problem
Setting thread pool sizes far above the number of processor cores, on the theory that more is faster.
Why It’s Harmful
Threads compete for cores and locks, context switching rises, and latency can get worse, not better.
Correct Approach
Increase in small steps guided by idle-percentage metrics, and stop when they recover.
Problem
Giving the broker a huge Java heap and leaving little memory for the page cache.
Why It’s Harmful
Reads go to disk instead of memory, and long garbage collection pauses cause followers to drop out and clients to time out.
Correct Approach
Use a moderate heap tuned for short pauses and leave the remaining memory to the operating system cache.
Problem
Trading away durability for speed, for example minimum in-sync replicas of one and unclean leader election enabled, on important data.
Why It’s Harmful
A single broker failure at the wrong moment can lose acknowledged data, and nobody notices until it is too late.
Correct Approach
Choose durability settings by the value of the data, and get explicit sign-off for any reduction.
Problem
Changing several settings at once, in production, with no baseline or rollback plan.
Why It’s Harmful
If something breaks, nobody knows which change caused it, and if something improves, nobody knows why. The team learns nothing and risks an outage.
Correct Approach
Follow the measured loop: one change, tested, compared with a baseline, rolled out gradually and documented.
24Best Practices and Common Mistakes
A checklist to keep beside you when tuning Kafka brokers.
Best Practices
- Start from metrics: thread idle percentages, queues and timing stages
- Change one setting at a time and compare with a baseline
- Match thread counts to disks and cores, not to guesswork
- Leave generous memory for the operating system page cache
- Keep the Java heap moderate and pauses short
- Align message size limits on producers, brokers, replicas and consumers
- Raise recovery threads for brokers with many partitions
- Fix client batching and partition counts before adding capacity
- Roll changes out one broker at a time and document them
Common Mistakes
- Raising threads without checking idle metrics
- Enlarging queues to hide an overloaded I/O stage
- Forcing frequent disk flushes for imagined safety
- Setting huge fixed socket buffers on every connection
- Mismatched message size limits between stages
- Raising the replica lag limit to mask flapping followers
- Giving the heap most of the memory
- Forgetting to sync dynamic changes back to configuration files
- Restarting many brokers at once
Defaults, dynamic options and recommended values change between Kafka versions. Always check the documentation for the version you run, and treat every number in any guide, including this one, as a starting point for your own measurement.
A good mechanic listens to the engine, checks the gauges, changes one part, and drives again before touching anything else. A careless one swaps five parts at once and hopes.
25Real-World Scenarios
Five common situations show how the reasoning of this tutorial plays out.
Scenario 1: A newly secured cluster gets slower
After turning on TLS on the client listener, latency rises although disks are quiet. The network thread idle metric has dropped close to zero, because encryption work now runs on those threads. The team raises the network thread count in small steps, watches the idle metric recover and confirms the latency returns to its old level.
Scenario 2: A broker with many disks underused
A machine has twelve disks, each in its own data directory, but the request handler pool is still at the default. Disks sit half idle while handlers are always busy and the request queue is full. The team raises the I/O thread count in steps and sees the queue drain and disk use rise evenly.
Scenario 3: Rolling restarts take hours
A cluster with thousands of partitions per broker takes a long time for each broker to come back after a restart. The team raises the number of recovery threads per data directory, tests unclean shutdown recovery on a test broker, and cuts each restart substantially, shortening upgrade windows.
Scenario 4: Followers keep falling out of sync
Under-replicated partitions appear every afternoon. Investigation shows long garbage collection pauses on some brokers caused by an oversized heap, and replication running through a single fetcher thread. The team shrinks the heap, adds fetcher threads and finds the flapping stops without touching the lag limit.
Scenario 5: Broker overload from tiny producer batches
A new application sends millions of tiny requests, and handler idle time falls. Rather than adding threads, the team asks the application owners to raise batching and enable compression. Request rates fall sharply, and brokers recover without any configuration change.
| Scenario | Key Evidence | Action |
|---|---|---|
| TLS slowdown | Low network thread idle percent | More network threads, in small steps |
| Many disks underused | Busy handlers, full queue, idle disks | More I/O threads |
| Slow restarts | Long log recovery | More recovery threads per directory |
| Followers flapping | Long pauses, single fetcher | Smaller heap, more replica fetchers |
| Tiny batches | Huge request rate, low handler idle | Fix client batching, no broker change |
26Frequently Asked Questions
Network threads read requests from sockets and send responses back, and they handle encryption for secured connections. I/O threads, also called request handlers, take requests from the shared queue and do the real work, such as reading and writing logs.
Check the request handler idle percentage and the request queue. If handlers are nearly always busy and the queue keeps growing, while disks and processors still have spare capacity, more I/O threads may help.
No. Beyond what disks, cores and networks can support, extra threads add contention. Increase in small steps guided by metrics, and stop when the relevant idle percentage recovers.
It means requests arrive faster than the I/O stage can process them. The broker protects itself by making network threads wait, which slows clients. Look for the real bottleneck downstream, such as disks or processors.
Many of them, including several thread pool sizes, can be updated dynamically on a running broker, with limits on how far one change may move a pool size. Check the documentation for your version and remember to update the configuration file too.
Kafka relies on the operating system’s page cache for fast reads. A huge heap takes memory from the cache and lengthens garbage collection pauses. A moderate heap plus generous free memory performs better.
A message passes through the producer, the broker, follower replication and the consumer. Each stage has its own limit, and the smallest one blocks large messages, so all must be aligned.
Almost never. Kafka relies on replication for durability, and forced flushing reduces throughput without adding much real safety. Leave flush settings at defaults unless you have a specific, tested reason.
More recovery threads per data directory, fewer partitions per broker, and clean shutdowns. Test unclean recovery time on a test broker to see whether more threads help your setup.
Common causes are long garbage collection pauses, saturated disks or networks on the follower, and too little replication parallelism. Fix those instead of only raising the allowed lag time.
Treat them as starting points only. Hardware, workloads and Kafka versions differ, so always test against your own baseline before adopting a value.
Yes, often dramatically. Better producer batching, compression and consumer fetch settings reduce request rates and broker load more than most broker settings can.
27Summary and Key Takeaways
A Kafka broker handles each request through a pipeline: network threads read it, a bounded queue holds it, I/O threads do the work, and network threads send the response. Separate thread pools handle replication, recovery, log cleaning and housekeeping. Tuning means giving each stage the right capacity for your workload and hardware, guided by metrics such as thread idle percentages, queue sizes and the timing stages of each request, and balancing latency, throughput and durability deliberately.
The most valuable habits are not particular numbers but a method: measure a baseline, find the bottleneck, change one thing, verify under realistic load, roll out one broker at a time and document the result. Just as important, look beyond the file: memory and page cache, the Java heap, disks, networks, client batching and partition counts often decide performance more than any thread setting. If you remember one sentence, let it be this: tune the stage the metrics point to, not the setting a guide happens to list.
Key Takeaways
- Know the request path — network threads read and send, a bounded queue holds requests, and I/O threads do the real work, so each stage has its own capacity.
- num.network.threads and num.io.threads have different jobs — raise network threads when their idle percentage is low, raise I/O threads only when handlers are busy and disks and processors are not the limit.
- A full queue is a symptom — it means backpressure from an overloaded downstream stage, so find the bottleneck rather than simply enlarging the queue.
- Use the timing stages — request queue, local, remote, response queue and response send times show exactly which part of the broker is slow.
- Tune the housekeeping pools too — replica fetchers, recovery threads and log cleaner threads decide replication speed, restart time and compaction health.
- Look beyond the config file — a moderate heap, ample page cache, aligned message size limits, fast disks, network capacity, client batching and sensible partition counts often matter more than thread counts.
- Follow a disciplined method — baseline, one change at a time, realistic testing, gradual rollout, documentation and version-aware checks keep tuning safe and repeatable.