Before tuning anything, we need clear meanings for the two ideas that everything else depends on.

Throughput: how much

Throughput is the amount of work completed in a period of time. In Kafka it is usually measured as messages per second or megabytes per second. A cluster that accepts one million messages every second has high throughput. Throughput answers the question: “How much data can this system handle?”

Latency: how long

Latency is the time one piece of work takes from start to finish. In Kafka it is usually measured in milliseconds. It might be the time from a producer sending a message to the broker confirming it, or the time from the message being created to a consumer receiving it. Latency answers the question: “How quickly does one message get through?”

Why they are not the same thing

It is tempting to think a fast system must be fast in both ways. In practice they are separate. A system can move a huge volume of data while each message waits a long time in queues and batches. Another system can respond to each request instantly, yet collapse when the load doubles. A mail truck that leaves once a day carrying a million letters has enormous throughput and terrible latency. A courier on a bicycle has low throughput and excellent latency.

A trade-off can only be managed if you measure both sides properly. Averages alone can mislead badly.

Measuring throughput

Throughput is measured over a period, and it helps to record it in two ways: records per second and bytes per second. A million tiny records and a thousand huge records may have the same byte rate but stress the system differently. Small records make the cluster spend more effort per byte on bookkeeping. Large records make network and disk the limits.

Measuring latency with percentiles

Latency varies from message to message, so a single number hides the story. Engineers use percentiles. The 50th percentile, called the median, is the time that half of requests beat. The 99th percentile is the time that ninety-nine percent of requests beat, so it describes the slowest one in a hundred.

The tail matters more than it seems

Modern applications often call many services in a chain. If each step has a one percent chance of being slow, a request that passes through fifty steps has a large chance of meeting at least one slow step. This is why large companies pay attention to the 99th and 99.9th percentiles. Rare slowness at each layer becomes common slowness for the whole request.

Different kinds of latency in Kafka

To tune Kafka you need to know where a message spends its time. Follow one message from creation to consumption.

flowchart LR
    A["Application creates a message"] --> B["Producer buffer: waits to join a batch"]
    B --> C["Network to the partition leader"]
    C --> D["Leader appends to its log"]
    D --> E["Followers copy the batch"]
    E --> F["Leader acknowledges the producer"]
    D --> G["Consumer fetch reads the message"]
    G --> H["Application processes the message"]
    

Reading the diagram

The message is born in your application, which hands it to the Kafka producer library. The producer does not usually send it immediately. It places the message in a buffer, grouped by destination partition, and waits until a batch is ready. The batch then travels over the network to the leader, the broker that owns that partition. The leader appends the batch to the end of its log, which is a file on disk. Other brokers holding copies, called followers, fetch the batch and store it too. Once enough copies exist, as required by your settings, the leader sends an acknowledgement to the producer.

Consumers work separately. A consumer asks the leader for data after its last position. The leader answers with whatever is available, possibly after waiting for a minimum amount to accumulate. The consumer then processes the messages and saves its position.

Where the time goes

If you remember one idea from this guide, remember batching. Almost every Kafka throughput gain comes from it, and almost every latency cost comes with it.

What batching means

Instead of sending each message alone, the producer collects several messages destined for the same partition and sends them as one batch. The batch has one header, one network request and one disk write, no matter how many messages it holds. The fixed costs are shared among all of them.

sequenceDiagram
    participant App as Application
    participant Buf as Producer buffer
    participant Br as Broker
    App->>Buf: message 1
    App->>Buf: message 2
    App->>Buf: message 3
    Note over Buf: Waits until the batch is full or the linger time ends
    Buf->>Br: one request with three messages
    Br-->>Buf: one acknowledgement
    

Why batching improves throughput

• Fewer requests: the broker handles one request instead of many, so its threads and queues are less crowded.
• Less overhead per message: headers, checksums and network framing are paid once per batch.
• Better compression: compressing many similar messages together shrinks them far more than compressing each one alone.
• Sequential disk writes: larger appends make the disk and operating system work smoothly.

Why batching adds latency

The first message in a batch must wait for the batch to fill or for a timer to expire. When traffic is heavy, batches fill in a moment and the cost is tiny. When traffic is light, a batch may never fill, so the timer decides how long each message waits. This is why the same setting can be harmless on a busy system and painful on a quiet one.

A handful of producer settings control most of the balance. Think of them as dials on a mixing desk.

Linger time: the timer

The linger setting says: “if the batch is not full yet, wait this many milliseconds for more messages before sending.” A value of zero means send as soon as possible, though messages that arrive while a request is in flight still end up batched together naturally. A value of five to twenty milliseconds is a common throughput-friendly choice, adding a small, bounded delay in return for much bigger batches. The exact behaviour of the default has changed between client versions, so check the documentation for the version you run.

Batch size: the container

The batch size is the size of the container, in bytes, per partition. If messages arrive quickly, the container fills and is sent without waiting for the timer. A larger container means more messages per request and better compression, but more memory used per partition. If the container is much larger than what fills within your linger time, the extra size is never used.

Buffer memory and back-pressure

The producer holds unsent messages in a memory buffer. If the application produces faster than the brokers accept, the buffer fills. When it is full, further send calls wait, up to a limit, and then fail. This back-pressure protects the application from running out of memory, but it shows up as sudden latency for the application thread. A larger buffer absorbs bursts but hides slowness longer.

In-flight requests

A producer can send several batches to a broker without waiting for each reply. Allowing more outstanding requests keeps the pipe full and raises throughput, especially over long distances. With idempotence enabled, Kafka can allow up to five outstanding requests while still preserving order within a partition, which is why the modern defaults are friendly to both speed and safety.

How the dials interact

The dials never act alone. A long linger time only helps if the batch size is large enough to hold what accumulates. A large batch size only helps if traffic is heavy enough to fill it. Compression works better with bigger batches, but it also takes processor time that a busy application server may not have to spare. Acknowledgement of all lengthens each round trip, so allowing more requests in flight keeps the pipe full while waiting. Understanding these links is what separates thoughtful tuning from random guessing.

A useful way to think about it is to ask two questions for every application. First: how long may a message wait before it must leave the producer? That answer sets linger time. Second: how much data typically arrives during that wait? That answer sets a sensible batch size. Once those two are matched, compression and buffer memory can be sized around them.

Compression shrinks batches so they use less network, less disk and less time on the wire. It costs some processor time, so it is a trade-off of its own.

Why compression often raises throughput

Networks and disks have limits measured in bytes per second. If a batch shrinks to a third of its size, three times as many messages fit through the same pipe. On many clusters the network or disk is the bottleneck, not the processor, so spending a little processor time to save a lot of bandwidth is a good exchange. Compressed batches also reduce replication traffic between brokers and the disk space needed.

The common codecs

How compression interacts with batching

Compression works on whole batches. A larger batch of similar messages compresses better than a small one, so linger time and batch size influence compression ratios. Text-like data such as JSON or logs compresses well. Data that is already compressed, such as images or encrypted payloads, barely shrinks and wastes processor time.

Producers compress, brokers usually store the batch as received, and consumers decompress. This means the producer machines and consumer machines pay the processor cost, while brokers benefit from smaller data. Brokers can be forced to recompress if the topic’s setting differs from the producer’s, which costs broker processor time, so keep them consistent.

The most important latency decision in Kafka is not about speed at all. It is about how many copies must confirm a message before the producer is told “done”.

Copies and acknowledgements

Kafka keeps several copies of each partition on different brokers. The number of copies is the replication factor, commonly three. The producer’s acks setting says how many of them must confirm a write before the leader replies.

sequenceDiagram
    participant P as Producer
    participant L as Leader broker
    participant F1 as Follower 1
    participant F2 as Follower 2
    P->>L: batch of messages
    L->>L: append to log
    F1->>L: fetch new data
    F2->>L: fetch new data
    L-->>P: acknowledgement once required copies exist
    

The minimum in-sync replicas rule

The in-sync replicas are the copies that are keeping up with the leader. The min.insync.replicas setting says how many of them must exist for a write with acknowledgement of all to be accepted. A common safe setup is replication factor three with a minimum of two, so that one broker can fail without stopping writes or losing acknowledged data. If too few copies are in step, writes are refused rather than accepted unsafely.

What replication actually costs in time

With acknowledgement of all, send latency includes the time for followers to fetch the batch and the leader to see that they have it. If followers are in the same data center, this may be a millisecond or two. If they are in different availability zones, a few milliseconds more. If one follower is slow, the whole acknowledgement waits for it or until it is removed from the in-sync set. This is why slow disks or busy machines on a single broker can raise latency for many partitions at once.

Choosing by the value of the data

A practical way to decide the acknowledgement level is to ask what losing one message would cost. A lost page-view event in a clickstream may be statistically invisible, while a lost payment instruction is a serious incident. Many organizations therefore classify topics by importance and apply strict durability to the critical ones while relaxing it only where loss is genuinely acceptable. The classification, not the speed graph, should drive the decision.

Kafka achieves remarkable throughput on ordinary hardware by leaning on how operating systems and disks like to work.

Sequential writes

Kafka appends every batch to the end of a log file. Writing sequentially is far faster than jumping around a disk, even on old spinning disks, and it is comfortable for solid-state drives too. There is no complicated rearranging of data on write. Reading is also sequential for consumers that follow the log.

The operating system’s page cache

Kafka does not keep its own large cache of messages. It relies on the operating system’s page cache, spare memory that holds recently used file data. A message just written is usually still in memory when consumers ask for it, so reads are served from memory rather than disk. This is why leaving plenty of free memory for the operating system, instead of giving everything to the Java heap, is a well-known Kafka habit.

Zero-copy transfer

When a consumer fetches data that is already in the page cache, Kafka can ask the operating system to send it straight from the cache to the network card without copying it through the application’s memory. This “zero-copy” approach saves processor time and memory bandwidth. Note that it works best when data is not encrypted by the broker itself. When TLS is enabled, data must be encrypted in the broker process, so this shortcut is lost for those connections.

Flushing to disk

Kafka writes data into the page cache and lets the operating system flush it to disk in the background. It relies on replication, not immediate disk flushes, for durability, since other brokers hold copies. Forcing a flush after every message is possible through settings, but it sharply reduces throughput and raises latency, and is rarely needed.

Broker threads and queues

A broker has network threads that read requests from connections, a queue of waiting requests, and request handler threads that do the actual work. When traffic is light, a request passes through immediately. When threads are busy, requests wait in the queue, and this waiting is a major source of latency spikes. Monitoring the idle percentage of these thread pools tells you how close the broker is to the point where latency starts to climb.

Partitions are how Kafka spreads work across machines. More partitions can lift throughput, but they bring costs that show up in latency and stability.

How partitions raise throughput

A topic is split into partitions, each an independent log. Different partitions can live on different brokers and be read by different consumers in a group at the same time. More partitions therefore mean more parallel writing and reading. A single partition is limited by what one leader and one consumer can handle, so a very busy topic needs many.

flowchart TB
    T["Topic with 6 partitions"] --> P0["Partition 0 on Broker 1"]
    T --> P1["Partition 1 on Broker 2"]
    T --> P2["Partition 2 on Broker 3"]
    T --> P3["Partition 3 on Broker 1"]
    T --> P4["Partition 4 on Broker 2"]
    T --> P5["Partition 5 on Broker 3"]
    P0 --> C1["Consumer A"]
    P3 --> C1
    P1 --> C2["Consumer B"]
    P4 --> C2
    P2 --> C3["Consumer C"]
    P5 --> C3
    

The costs of too many partitions

How many partitions?

A widely used method is to estimate the throughput you want for the topic, measure what one partition can handle for producing and for consuming, and divide. Add headroom for growth, since increasing partitions later changes which partition a key maps to and can disturb ordering for keyed data. Avoid absurd numbers chosen “just in case”. Many production topics work well with a dozen to a few dozen partitions, with only the busiest having more.

Keys and ordering

Messages with the same key go to the same partition, which preserves their order. If one key is far busier than the others, its partition becomes a “hot” partition, limiting throughput and raising latency for that key regardless of how many partitions exist. Choosing keys that spread evenly is as important as choosing the partition count.

Adding partitions later

Increasing the partition count of an existing topic is possible, but it changes the mapping from keys to partitions. Messages for a key that used to land in one partition may now land in another, so consumers that rely on all events for a customer arriving in one place can see order disturbed during the change. Teams that use keyed data therefore plan a comfortable partition count at the start, or create a new topic and migrate carefully. This is one more reason to choose the number from measurement and growth forecasts instead of guessing.

Reducing the partition count is not supported directly at all. A topic with too many partitions usually needs to be replaced by a new one, which is a heavier operation. The lesson is to start moderate and grow, rather than start huge and be unable to shrink.

Consumers make their own trade-off between reading in big, efficient gulps and reacting to each message as soon as it appears.

Fetch settings

A consumer reads by sending fetch requests. Two settings shape how the broker answers.

The default is latency-friendly

By default, the minimum fetch size is one byte, so the broker replies as soon as any data exists. This favours low latency, at the cost of many small responses when traffic is light. Raising the minimum lets the broker collect more before replying, reducing request counts on busy clusters, but every message may wait up to the maximum wait time if traffic is slow.

Processing time is part of the story

The time a consumer takes to process each batch often exceeds all Kafka delays combined. If processing a batch takes longer than the allowed gap between polls, the consumer is considered dead and removed from its group, triggering a rebalance that pauses reading for everyone. Big poll sizes with slow processing are a classic cause of this trouble, which again turns “throughput per poll” into a latency and stability problem.

Parallel consumption and lag

Adding consumers to a group, up to the number of partitions, increases throughput. Consumer lag, the number of messages waiting to be read, is the key sign of whether consumers keep up. Stable low lag means throughput is adequate. Growing lag means the consumers are slower than the producers, and end-to-end latency will grow along with it.

Rebalances and the price of stopping

Whenever consumers join or leave a group, partitions are reassigned in what is called a rebalance. During older-style rebalances, every consumer briefly stops reading, which adds a pause to end-to-end latency for the whole group. Frequent restarts, long processing gaps or unstable networks cause repeated pauses. Newer cooperative rebalancing and static membership reduce the disruption by moving only the partitions that must move and by letting a restarted consumer keep its old place. For latency-sensitive applications, stability of the group is as important as raw speed.

When a business gives you a target such as “under 200 milliseconds”, you must split it among all the stages. That split is called a latency budget.

Spending the budget

Imagine a goal of 200 milliseconds from event creation to a consumer finishing its work. You cannot spend it all on Kafka. Some goes to the application creating the event, some to the network, some to Kafka, some to processing. Writing the budget down makes each team responsible for a slice and shows which slice is too big.

flowchart LR
    A["Create event: 10 ms"] --> B["Producer wait: 10 ms"]
    B --> C["Send and replicate: 20 ms"]
    C --> D["Wait for consumer fetch: 10 ms"]
    D --> E["Consumer processing: 100 ms"]
    E --> F["Spare margin: 50 ms"]
    

Where Kafka teams find savings

Putting the dials together gives three typical starting points. Treat them as starting points to test, never as final answers.

Different settings for different topics and clients

Producer and consumer settings belong to each client application, not to the whole cluster. This means you can run a latency-focused payment service and a throughput-focused log shipper on the same cluster, each with its own client settings. Use quotas on the brokers to stop the bulk applications from crowding out the latency-sensitive ones, and consider separate clusters when the workloads are extreme and clash.

Revisiting the profile over time

Workloads change. A service that launched with a few hundred messages a second may grow to tens of thousands, and a profile that was ideal at launch can become wasteful. Review each application’s settings whenever traffic doubles, message sizes change, hardware is replaced or the business goal shifts. A quarterly look at the dashboards, with a short note on whether the balance still matches the goal, prevents slow drift into a configuration nobody chose on purpose.

Correctness features are not free. Each one asks the system to do a little more work, and that work has a shape you should understand.

Ordering and retries

Kafka guarantees order within a partition. Suppose a producer sends batch one and batch two without waiting, and batch one fails and is retried. Without care, batch two might be stored first, and the order would be wrong. In the past, the safe answer was to allow only one outstanding request, which hurt throughput badly. With idempotent producers, the broker tracks sequence numbers and can keep order even with up to five outstanding requests, so you keep both order and speed.

The cost of idempotence

Idempotence adds a small header to each batch and a small amount of bookkeeping on the broker. In practice the impact on throughput and latency is very small, and it is enabled by default in recent client versions. Its main requirement is acknowledgement of all, which is where any noticeable latency effect comes from.

The cost of transactions

Transactions let a producer write to several partitions atomically, so that consumers see all or none of the writes. They add coordination: the producer registers partitions with a transaction coordinator, and commit or abort markers are written into each partition. This adds latency, particularly if transactions are very short, because the fixed cost of coordinating is paid every time.

The commit interval trade-off

Stream processing applications that use exactly-once processing commit a transaction at regular intervals. A short interval gives fresh results to read-committed consumers but pays the coordination cost more often. A long interval reduces overhead and raises throughput but delays visibility. Kafka Streams, for example, uses a much shorter default commit interval when exactly-once is turned on than when it is off, showing that the designers considered this trade-off carefully.

Settings can only do so much. Eventually the machines, disks and network draw the real boundaries.

Disks

Because Kafka writes sequentially, ordinary disks can deliver good throughput, and many clusters use several disks per broker. Solid-state drives help most with latency variation, since they handle simultaneous reads and writes and recover quickly from bursts. Slow or failing disks are a common hidden cause of latency spikes, because a single slow broker can hold back the acknowledgements of every partition it leads.

Memory

Memory is important mostly as page cache. A common approach is a moderate Java heap and the rest of the memory left to the operating system. If the working set of recent data fits in the cache, consumers reading near the end of the log get answers without touching the disk, which keeps latency low.

Network

Network bandwidth often becomes the first limit. Each message is received once, replicated to followers, and sent to each consumer group, so traffic multiplies. A topic with three copies and three separate consuming applications generates far more network traffic than its incoming rate suggests. Compression, sensible replication factors and avoiding needless consumers all help.

Distance and placement

Light in fibre still takes time. A round trip inside one data center takes a fraction of a millisecond, between availability zones a millisecond or a few, and across continents tens to hundreds of milliseconds. Acknowledgement of all across distant regions therefore adds noticeable latency. Many organizations run a Kafka cluster in each region and copy data between them with tools such as MirrorMaker or Confluent’s Cluster Linking, accepting a delay in the copy in exchange for fast local writes.

Cloud considerations

In the cloud, data moving between availability zones is often charged. Spreading replicas across zones improves resilience but adds cost and a little latency. Consumers can be allowed to read from a nearby replica instead of the leader in newer Kafka versions, which saves cross-zone traffic and can lower latency. Instance types with fast local or well-provisioned network disks matter, and burstable disks can suddenly slow down when their credits run out.

A tuning change is only an improvement if a fair test says so. Careless benchmarks produce confident nonsense.

Start from a goal and a baseline

Write down the target throughput and latency percentiles, measure the current system under a realistic load, and record the numbers. Without a baseline, you cannot say whether a change helped. Change one setting at a time, run the same test, and compare.

Use realistic data and load

Message size, key distribution, compression ratio and traffic pattern all affect results. Random test data compresses very differently from real data. A steady synthetic load misses the bursts of a real day. Where possible, replay a sample of production-like data at production-like rates.

Tools that come with Kafka

Kafka ships with performance test tools for producers and consumers. They send a chosen number of records of a chosen size at a chosen rate, and report throughput and latency percentiles. They are useful for quick comparisons of settings, though they do not reproduce your application’s real behaviour. Other community tools generate workloads and collect end-to-end latency, and the open-source OpenMessaging Benchmark has been used to compare messaging systems under common scenarios.

Common benchmarking traps

Kafka publishes metrics that show exactly where you sit on the throughput and latency scale. Watching them lets you tune with evidence.

flowchart TB
    S["Latency is too high"] --> Q1{"Is producer queue time high?"}
    Q1 -- "yes" --> A1["Reduce linger time or batch size"]
    Q1 -- "no" --> Q2{"Is broker request time high?"}
    Q2 -- "yes" --> A2["Check thread idle percentages, disks, replication"]
    Q2 -- "no" --> Q3{"Is consumer lag growing?"}
    Q3 -- "yes" --> A3["Speed up processing or add consumers"]
    Q3 -- "no" --> A4["Check network path and clocks"]
    

Logging and correlation

Metrics show that latency rose. Logs show why. Broker logs may reveal garbage collection pauses, disk errors or leader changes at the exact time. Keep machine clocks synchronized so that timestamps line up, and consider putting a creation timestamp inside each message so consumers can compute true end-to-end age.

Building a useful dashboard

A single page that shows the essentials helps everyone speak the same language. Put message rate and byte rate at the top to show throughput. Beside them, show the 50th, 95th and 99th percentile of produce request time, and consumer lag expressed in seconds. Below, show thread idle percentages, under-replicated partitions and disk usage to explain causes. Mark deployments and configuration changes on the graphs, so a sudden shift can be linked to what changed. When throughput and latency graphs move together, you can see the trade-off with your own eyes, and you can tell whether a tuning change moved the balance in the direction you wanted.

Encryption and access control are essential, and they have a measurable price in throughput and latency. Knowing the price lets you budget for it rather than switch protection off.

Encryption in transit

TLS encrypts traffic between clients and brokers and among brokers. It adds a one-time handshake cost when a connection opens and a continuous processor cost to encrypt and decrypt data. On modern processors with hardware support the cost is moderate, but it can reduce peak throughput noticeably and increase broker processor use. As described earlier, TLS also prevents the zero-copy shortcut for reads, since the broker must handle the data to encrypt it.

Authentication and authorization

Authentication proves who the client is, using methods such as mutual TLS, SASL with passwords or tokens, or Kerberos. It costs time at connection setup, so applications that open many short-lived connections pay repeatedly. Long-lived connections and shared producer instances avoid this. Authorization checks, through access control lists, happen on requests and are cached in memory, so their per-request cost is small unless the number of rules is enormous.

Quotas: fairness as a performance tool

Kafka quotas limit how much data a client or user can produce or consume, and how much broker processing time they can use. Quotas protect latency-sensitive applications from noisy neighbours, and they are a security and stability feature too. A client that exceeds its quota is slowed by the broker, which appears in metrics as throttle time. If you see throttle time, the “slowness” is a deliberate limit, not a fault.

Recurring designs help teams manage the trade-off well, and recurring mistakes damage it.

Patterns worth copying

Anti-patterns to avoid

A short list of habits that keep the throughput and latency balance healthy, and traps that catch almost every team once.

Mistakes beginners meet first

A short tuning routine

Culture matters as much as settings

Teams that handle this trade-off well share a habit: they treat performance goals as requirements with owners, not as afterthoughts. Someone owns the latency target, someone owns the throughput target, and both look at the same dashboards. When a new feature threatens either goal, the conversation happens early, with numbers, rather than after users complain. This shared discipline keeps tuning from becoming an argument between people who each optimize only their own metric.

Public talks, papers and documentation from well-known organizations show the trade-off in action.

Short answers to questions that newcomers and experienced engineers ask often.

If you keep only one page of this guide, keep this one.

Throughput and latency are two different measures of speed, and Kafka lets you slide between them. Grouping messages into batches, compressing them and fetching them in larger pieces moves more data for less effort, at the cost of making individual messages wait. Sending and reading eagerly does the opposite. Neither end is right for every job, so the skill lies in choosing where each application should stand, measuring the result and keeping enough headroom for bursts and failures.