The plain-English idea first. The details come one layer at a time.

Kafka brokers store data in logs, and clients read from those logs. The way a client reads is not by having the broker push messages out. Instead, the client sends a request that says, “Here is the partition I care about and here is the offset where I stopped. Please give me what comes next.” The broker answers with the next chunk of data. This request and its answer are called Fetch. Because the reader asks and the broker responds, we call Kafka a pull-based system.

The Fetch protocol is the exact set of rules for how that conversation happens. It says which fields a request must carry, which fields a response contains, how long the broker may wait before replying, how much data may be sent, and what errors can come back. A fetch session is a shortcut layered on top of those rules. It lets a client and a broker remember what they talked about last time, so that the next request only needs to mention what changed.

Why this topic deserves your time

Fetch sits under almost every performance and reliability story in Kafka. Slow consumers, lagging followers, shrinking ISRs, high network bills, sudden CPU spikes on brokers after a restart, and strange delays of exactly half a second are all connected to how fetch behaves. Engineers who understand fetch can explain these symptoms in minutes. Engineers who do not often chase them for days.

A short tour of the parts that fetch talks about. Skim it if you already know Kafka well.

Kafka is a system for moving streams of events. Programs that write events are producers, and programs that read them are consumers. The servers in the middle are brokers, and a group of brokers is a cluster. Events are organized into topics, and each topic is cut into partitions. A partition is the unit that fetch works on: every fetch request asks about one or more specific partitions.

Records travel in batches

Kafka does not store or send records one at a time. Producers group records into batches, often compressed, and brokers store each batch exactly as it arrived. When a consumer fetches, the broker sends whole batches. This one design choice makes Kafka fast, because the broker can send bytes straight from disk without unpacking them. It also explains a few surprising details you will meet later, such as why a fetch response can start slightly before the offset you asked for.

flowchart LR
  subgraph CL["Kafka Cluster"]
    L["Broker 1
Leader of Partition 0"]
    F1["Broker 2
Follower of Partition 0"]
    F2["Broker 3
Follower of Partition 0"]
  end
  P["Producer App"] -->|"Produce"| L
  F1 -->|"Fetch"| L
  F2 -->|"Fetch"| L
  C1["Consumer A"] -->|"Fetch"| L
  C2["Consumer B"] -->|"Fetch"| L
  LB["Cluster Metadata
tells clients who leads"] -.-> C1
  LB -.-> C2
  LB -.-> P

Notice in Figure 1 that followers and consumers look almost the same from the leader’s point of view. Both send fetch requests, and both receive batches of records. The only difference is who they are and what the leader does with the information. That shared design keeps Kafka simple: one mechanism serves both readers and replicas.

Two very different kinds of clients use the same request, and the broker treats them differently.

The first kind of client is the ordinary consumer. It fetches records so that an application can process them. The second kind is the follower replica, which is a broker copying a partition from its leader. In the request there is a field that says who is asking. Consumers put the value -1 in that field. Brokers put their own broker ID there. The leader looks at this value and knows whether it is serving a reader or a replica.

Consumers, followers and the cluster’s own logs

There is a third user of fetch that many people miss. In modern clusters that run without ZooKeeper, the controllers keep the cluster’s metadata in their own replicated log, and they copy it between themselves using the same pull-based fetch idea. The mechanics are slightly extended for that purpose, but the shape is the same: a follower keeps asking the leader, “What is new after this offset?” This shows how central the pattern is. Kafka uses one simple idea for user data and for its own bookkeeping.

Let us open the envelope and look at every important field. Field names vary a little between versions, so read this as the shape, not as a word-for-word specification.

Kafka clients and brokers speak a binary protocol over TCP. Every message on the wire starts with a small header that says which kind of request it is (for us, Fetch), which version of that request it uses, a number that lets the client match answers to questions, and the client’s name. Clients and brokers first agree on the highest version they both understand, so that new brokers can still talk to old clients and the other way around. After the header comes the body. For Fetch, the body has two layers: fields that apply to the whole request, and fields that apply to each partition.

Fields that apply to the whole request

Fields that apply to each partition

The key idea is that the request is per broker, not per partition. A consumer that owns twelve partitions on the same broker sends one request that lists all twelve. This batching saves network round trips, but it also creates the problem that fetch sessions later solve: the list of partitions can be very long, and most entries repeat unchanged.

What the broker sends back, and the small details in it that keep the whole cluster honest.

The response mirrors the request. It carries whole-response fields, and then a list with one entry for each partition that has something to say. The most obvious content is the records themselves, delivered as batches. But the response also carries several other facts that clients and followers depend on.

A response that starts too early

Because the broker sends whole batches, the first batch in a response can begin a little before the offset the client asked for. Imagine a batch that holds offsets 100 to 119, and the client asks for offset 105. The broker cannot cut the batch in half without unpacking it, so it sends the whole batch, and the client quietly ignores records 100 to 104. This is normal and costs very little, and it is a good example of Kafka trading a tiny bit of wasted bandwidth for a large saving in broker work.

A response can be empty and still useful

Sometimes a response carries no records at all. This is not a mistake. A follower that is fully caught up still needs to hear the leader’s latest high watermark, and the leader still needs to hear that the follower is alive and keeping up. An empty response is how a quiet partition keeps the relationship going. It is also how a broker tells a client, “Nothing new, but here is an error, a leader change or a new suggested replica.”

The broker deliberately sits on your request. That pause is a feature, not a delay.

Imagine a consumer that reads a quiet topic. If the broker answered instantly with “nothing yet”, the consumer would immediately ask again, and again, hundreds of times a second, wasting network, CPU and battery. To avoid this, Kafka uses long polling. When a fetch request arrives and there is not enough data to fill it, the broker does not reply. It parks the request in a waiting area and answers only when one of two things happens: enough data has arrived, or the maximum wait time has run out.

Two request fields control this. Min bytes says how much data must be available before an early answer is worth sending. The default is one byte, which means “answer as soon as there is anything at all”. Max wait says the longest time the broker may hold the request. On consumers the default is 500 milliseconds. The waiting area is called the purgatory in Kafka’s code, and the parked request is a delayed fetch.

sequenceDiagram
  autonumber
  participant C as Consumer
  participant B as Broker Leader
  participant PG as Purgatory
  C->>B: Fetch offset 500, min 1 byte, max wait 500 ms
  B->>B: Look for data at offset 500 and find none
  B->>PG: Park the request and start the timer
  Note over PG: Waiting for new data or for the timer
  B->>B: Producer writes offset 500
  B->>PG: New data is now available
  PG-->>C: Fetch response with the new batch
  C->>B: Fetch offset 501, min 1 byte, max wait 500 ms

What ends the wait

The trade-off behind the two numbers

A higher min bytes and a longer max wait make the broker collect more data per answer. Fewer, bigger responses mean higher throughput and less CPU per byte, but a message may sit and wait until the batch fills up. A tiny min bytes and short wait give faster delivery of each record but more requests. For most applications the defaults are a sensible middle. If you run a heavy analytics job, raising the min bytes can pay off. If you run a live trading feed, you keep it small.

Three different limits shape every response, and one clever rule stops them from blocking progress.

If a consumer could ask for unlimited data, one busy partition could fill its memory and starve the others. So the fetch protocol has several size limits that work together. Each one protects a different thing.

Followers have their own matching settings on the broker side, such as replica.fetch.max.bytes for each partition, replica.fetch.response.max.bytes for the whole response, replica.fetch.min.bytes and replica.fetch.wait.max.ms. They do the same jobs for replication traffic.

The “at least one batch” rule

Here is a subtle problem. Suppose a partition holds a single batch of 3 megabytes, and the per-partition limit is 1 megabyte. If the broker obeyed the limit strictly, it could never send that batch, and the consumer would be stuck forever. Older Kafka versions really did suffer from this. The fix is a rule that is now part of the protocol: if the first batch in the first non-empty partition is larger than the limits, the broker still sends that one batch. The limits are therefore soft for the first batch and strict for everything after it.

Fairness between partitions

With a shared budget for the whole response, the order in which partitions are served matters. If the broker always started with the same partition, that partition could use up the budget every time, and later partitions would starve. Kafka avoids this by rotating the order of partitions from one request to the next, so that every partition gets its turn at the front of the line. Fetch sessions, which we meet soon, made this rotation cleaner.

Follow one request from the network cable to the disk and back, and see why Kafka reads are so cheap.

When a fetch request reaches a broker, it does not go straight to the log. It passes through a small assembly line made of separate groups of threads, each doing one job. This design keeps slow work from blocking fast work, and it is the reason a broker can serve thousands of clients at the same time.

flowchart LR
  CL["Client or Follower
sends Fetch request"] --> NT["Network Thread
reads bytes from the socket"]
  NT --> RQ["Request Queue"]
  RQ --> IO["Request Handler Thread
checks leader and epoch"]
  IO --> LG["Log Read
find segment and offset"]
  LG --> DEC{"Enough data for
min bytes?"}
  DEC -->|"Yes"| RS["Response Queue"]
  DEC -->|"No"| PU["Purgatory
wait for data or timer"]
  PU --> RS
  RS --> NT2["Network Thread
sends bytes to the socket"]
  NT2 --> CL

Why reads are so cheap: page cache and zero-copy

Kafka leans heavily on the operating system’s page cache, a part of memory that holds recently used file data. A consumer that keeps up with producers reads records that were written moments ago, so they are almost always still in the page cache, and the disk is barely touched. On top of that, when the data can be sent unchanged, Kafka asks the operating system to copy it directly from the cache to the network. This trick is called zero-copy, and it removes two copies of the data and a lot of CPU work.

Warm readers and cold readers

A consumer reading fresh data is a warm reader. A consumer that restarts after a day of downtime and reads old data is a cold reader. Cold reads must fetch old segments from disk, and they push recent data out of the page cache. When that happens on a leader, the fetches from its followers can become slower, followers can slip out of the ISR, and everything gets worse together. This is one reason why a large replay job should be throttled or scheduled carefully.

Replication is just fetch, and the leader learns everything it needs from the follower’s offsets.

On every follower broker, a small group of replica fetcher threads runs all the time. Each thread is responsible for the partitions that follow the same leader broker. It sends fetch requests, appends whatever comes back to the local logs, and goes again. The number of such threads per source broker is set by num.replica.fetchers, one by default. More threads spread the work over more connections, which helps when a broker follows a very large number of partitions.

The clever part is what the follower says. In each fetch request, it states the offset it wants next, and that offset is exactly its own log end offset. So the request itself tells the leader how far the follower has copied. No special progress report is needed. The leader records that number, decides whether the follower is still in sync, and checks whether every in-sync follower now holds a message that was not yet committed. If so, it moves the high watermark forward.

sequenceDiagram
  autonumber
  participant P as Producer App
  participant L as Leader
  participant F as Follower
  P->>L: Produce with acks=all
  Note over L: Append record at offset 10, log end is 11
  F->>L: Fetch offset 10
  L-->>F: Batch with offset 10, high watermark is 10
  Note over F: Append record, log end is 11
  F->>L: Fetch offset 11
  Note over L: Follower holds offset 10, so high watermark moves to 11
  L-->>P: Acknowledgement
  L-->>F: Empty response after waiting, high watermark is 11
  Note over F: Follower now knows offset 10 is committed

Why the follower learns the news late

Look at Figure 4 closely. The leader moves the high watermark only when it receives the follower’s second fetch, and the follower hears about it in the response to that same fetch or a later one. So the follower’s view of the committed line always trails the leader’s by about one round trip. This small lag is normal and is the reason Kafka added leader epochs to fetch. If a leader fails at the wrong moment, a follower’s idea of the high watermark may be a little behind or ahead of the truth, and the epoch is used to sort out any differences safely.

How the ISR uses fetch

The leader also keeps a note of the last time each follower fetched all the way up to the end of the log. This time is what decides ISR membership: if it is older than replica.lag.time.max.ms, the follower is removed. Here is a rule that many people miss. A caught-up follower’s fetch request is parked in purgatory when there is no new data, and it is released when the wait time replica.fetch.wait.max.ms ends. That wait must be much shorter than the lag limit. Otherwise, a healthy follower on a quiet partition could be judged as late simply because its own fetch was waiting politely.

The same fetch request can show two different views of the log, depending on one field.

Not every record below the high watermark is safe to show to every application. Kafka supports transactions, where a producer writes several records and then either commits them all or aborts them all. While a transaction is still open, its records are already in the log, but the outcome is not yet decided. The fetch request therefore carries an isolation level.

The last stable offset

The last stable offset, or LSO, is the smaller of two things: the high watermark, and the offset of the first record that belongs to a transaction that is still open. Everything below the LSO has a known fate. A consumer in read-committed mode gets records only up to the LSO, along with a list of aborted transaction ranges, and it discards the aborted records before handing data to the application.

Why the plain protocol starts to hurt on large clusters.

Remember that a fetch request lists every partition the client wants to read from that broker. That is fine when the list is short. But large clusters have brokers that follow thousands of partitions from the same leader, and consumers that own hundreds of partitions on one broker. In the original protocol, the client repeated the entire list in every single request, and the broker answered with an entry for every partition again, even if nothing had happened there.

Let us do a rough calculation with round numbers, only to get a feeling. Imagine a broker that follows 4,000 partitions from one leader. Each partition entry in a request needs somewhere around 40 to 60 bytes, once you count the topic reference, partition number, offsets and limits. That is about 200 kilobytes for one request. Now imagine that request being sent and answered many times every second. Most of the partitions in it are quiet, so nearly all of those bytes describe things that did not change. The network, the CPU that reads and writes the entries, and the memory that holds them are all spent on repeating known facts.

The numbers above are illustrations, not measurements, but the pattern is real. Before fetch sessions, the cost of a fetch grew with the number of partitions, not with the amount of work actually done. A cluster with many partitions but modest traffic could spend a surprising share of its resources just on describing itself. Worse, this overhead hit the replication path, so followers could fall behind and leave the ISR because the leader was busy processing giant request lists.

The idea behind the fix

The fix came from a simple observation. Between two consecutive requests, almost nothing changes. The client wants the same partitions, with the same limits, and only a few of them have moved to a new offset. If both sides remember the last full list, the client can send only the differences. This is exactly what happens when you edit a document and save only the changes rather than the whole file. Kafka proposal KIP-227 turned this observation into fetch sessions, and they arrived in Kafka 1.1.

A session is a memory shared by one client and one broker. Two numbers, a session ID and an epoch, keep them in step.

A fetch session lives between one client and one broker. The broker keeps a small record for the session that holds the list of partitions being fetched, along with the last state it told the client about each of them, such as the last high watermark and log start offset. The client keeps a matching record. Because both sides remember the same list, they can talk in differences.

The three kinds of fetch request

sequenceDiagram
  autonumber
  participant C as Client
  participant B as Broker
  C->>B: Full fetch, session 0, epoch 0, 1000 partitions
  B-->>C: Data for 40 partitions, new session id 77
  C->>B: Incremental, session 77, epoch 1, 40 changed partitions
  B-->>C: Data for 12 partitions only
  C->>B: Incremental, session 77, epoch 2, 12 changed partitions
  B-->>C: Empty response after the wait
  Note over B: Broker evicts session 77 to make room
  C->>B: Incremental, session 77, epoch 3
  B-->>C: Error, session id not found
  C->>B: Full fetch again, session 0, epoch 0
  B-->>C: Response with a new session id

What the epoch protects against

The session epoch is a step counter that must go up by exactly one on each request. It protects the shared memory from getting out of step. If a request is lost or repeated, or if two threads mistakenly use the same session, the broker sees an epoch that does not match and refuses the request with an invalid-epoch error. The client then discards its session and begins a fresh one. It is a small price for making sure that “same as last time” always means the same thing to both sides.

What counts as a change

A partition needs to appear in an incremental request when its fetch offset has moved, which happens whenever the client has consumed data from it. It also appears when the client changes its size limit, its leader epoch view, or other per-partition settings. An idle partition that the client keeps asking about at the same offset does not appear at all. On the broker side, a partition is included in the response when it returns records, when its high watermark, last stable offset or log start offset changed since the last response, or when there is an error or a new preferred replica to report.

A session costs the broker memory, so the number of sessions is capped. Understanding the cap explains many odd production behaviours.

Each fetch session stores a list of partitions on the broker, so the broker cannot keep an unlimited number of them. It holds sessions in a session cache with a fixed number of slots, controlled by the broker setting max.incremental.fetch.session.cache.slots. The default is 1,000. One slot holds one session, and one session belongs to one client and one broker. A consumer that reads from five brokers therefore uses five slots, one on each broker.

Who gets a slot when the cache is full

Kafka treats two groups of sessions differently. Sessions opened by follower brokers are privileged. Replication is essential to the health of the cluster, so these sessions may push out ordinary sessions if the cache is full. Sessions opened by consumers are unprivileged. A new consumer session may take a slot from an older one, but the cache aims to protect sessions that were used recently and are large, and it prefers to remove ones that are small and idle. In simple words, the cache tries to spend its limited room where it saves the most work.

How to size the cache

A practical way to size it is to count the clients that will talk to a broker at the same time and add some headroom. The count should include every consumer instance that reads from it, every follower fetcher thread from other brokers that follow its partitions, and any tools or connectors. If you run thousands of small consumer applications, the default of 1,000 can be too small, and constant eviction may cost more CPU than it saves. Raising the number costs some memory, so pick a value from measurements, not from guesses.

The restart stampede

Sessions live in a broker’s memory only. When a broker restarts, every session on it disappears. All the clients that were connected to it now receive session errors at about the same moment and reconnect with full fetch requests together. In a large cluster this stampede of full requests can briefly load a freshly started broker much more than usual. Rolling restarts, spaced out and monitored, keep the effect small, and healthy clients back off slightly between retries.

Fetch is built for a world where brokers move, leaders change and sessions vanish. Each error has a planned reaction.

A fetch response can carry an error for the whole request or for individual partitions. Most of these errors are not disasters. They are signals that the client’s picture of the cluster is out of date, and the client is expected to refresh that picture and try again. Good clients do this automatically, with small waits between attempts so that they do not hammer a broker that is busy recovering.

Backoff: being a good citizen

When errors appear, retries must be spaced out. Consumers use settings such as retry.backoff.ms and the reconnect backoff settings to wait a little between tries. Followers use replica.fetch.backoff.ms. Without spacing, a broker that is already stressed would receive a flood of repeated requests from every client at once, which makes recovery slower. Backoff turns a crowd rushing a door into a queue.

Metadata refresh and the cost of being wrong

Almost every fetch error leads to the same next step: ask the cluster who leads what, right now. Clients keep a cached map of partition leaders and refresh it when errors occur and also on a regular timer. If many clients refresh at once after a leader change, the metadata requests themselves add load. This is another reason why stable clusters with few leader changes are so much easier to run.

Two small fields let the fetch protocol detect that a log has split from the leader’s log, even when nothing looks wrong.

The leader epoch is a number that goes up every time a partition gets a new leader. Fetch requests carry two epoch values. The first is the current leader epoch, meaning the era the client thinks is in force. The second is the last fetched epoch, meaning the era of the newest record the client has actually received. The broker compares both against its own records.

flowchart TD
  R["Fetch request arrives
with current epoch and last fetched epoch"] --> Q1{"Client epoch older
than broker epoch?"}
  Q1 -->|"Yes"| E1["Reply: fenced leader epoch
client must refresh metadata"]
  Q1 -->|"No"| Q2{"Client epoch newer
than broker epoch?"}
  Q2 -->|"Yes"| E2["Reply: unknown leader epoch
client waits and retries"]
  Q2 -->|"No"| Q3{"Does the client log match
this leader at last fetched epoch?"}
  Q3 -->|"No"| E3["Reply with diverging epoch
client truncates to the given point"]
  Q3 -->|"Yes"| OK["Serve records from the fetch offset"]

Stale views: the first pair of checks

If the client’s epoch is older than the broker’s, the client is looking at an outdated cluster map, perhaps because a leader change happened while it was busy. The broker refuses the request and the client refreshes its metadata. If the client’s epoch is newer, then the broker is the one that is behind. It has not yet heard about the latest leader change, so the client waits and tries again. These two checks stop a request from being served by a broker that no longer has authority over the partition.

Diverging logs: the second check

The last fetched epoch solves a harder problem. After certain failures, especially an unclean leader election, a follower or consumer may hold records that the new leader never had. Their logs have diverged at some offset. Without help, the reader would keep asking for the next offset and might either get an error far too late or, worse, quietly continue on top of different data. With the last fetched epoch, the leader can look at its own epoch history, see that the client’s last record belongs to an epoch that ended earlier than the client thinks, and reply with a diverging epoch hint: “Your log and mine split here.”

A follower reacts to this hint by cutting its log back to the split point and continuing from there. A modern consumer reacts by either resetting to that point or raising a truncation error, depending on its reset policy. Either way, the problem is caught on the first fetch after the split, not weeks later in a data-quality report. This feature arrived through KIP-320 for consumers in Kafka 2.1, and it works hand in hand with the leader-epoch truncation described in our guide on unclean leader election.

Sometimes the nearest copy of the data is not the leader. Kafka can let consumers read from it.

In the original design, consumers always read from the leader. If your brokers are spread over three cloud zones, two out of three consumers will typically sit in a different zone from the leader. Every byte they read then crosses a zone boundary, which adds delay and, on most cloud providers, adds a data transfer charge. For a topic read by many consumer groups, this cost can become one of the largest lines on the bill.

Kafka 2.4 introduced fetching from the closest replica, described in KIP-392. Brokers are labelled with their rack or zone using broker.rack, and consumers announce their own location with client.rack. A broker-side plug-in, selected by the replica.selector.class setting, decides which replica is closest. Kafka ships with a rack-aware selector for exactly this purpose. When it is enabled, the leader answers a consumer’s fetch with a hint called the preferred read replica, and the consumer then sends future fetches to that follower instead.

flowchart LR
  subgraph ZA["Zone A"]
    LD["Broker 1
Leader"]
  end
  subgraph ZB["Zone B"]
    FL["Broker 2
Follower"]
    CS["Consumer
client.rack = Zone B"]
  end
  CS -->|"1. First fetch"| LD
  LD -->|"2. Response names preferred read replica: Broker 2"| CS
  CS -->|"3. Later fetches stay inside Zone B"| FL
  LD -->|"Replication fetch"| FL

What you give up

A handful of settings decide whether your consumers are quick or efficient. Usually you cannot have both at the maximum.

Every tuning choice in fetch is a version of one question: should the broker answer sooner with less or later with more? Answering sooner gives lower delay for each record. Answering later gives fewer, larger responses, which use less CPU and network per byte. The table below shows the main dials for consumers.

Three starting profiles

Do not fight the batch size

Producers control how large the stored batches are. If producers send many tiny batches, no fetch setting can make reads efficient, because the broker must still handle many small pieces. Tuning the producer side with a short linger time and a reasonable batch size often improves fetch efficiency more than any consumer setting does.

How fetch behaves as your cluster grows in partitions, clients and traffic.

Fetch scales well because it is simple and because Kafka lets the operating system do most of the heavy lifting. But several resources can still run short, and each one leaves its own fingerprint. Knowing them helps you fix the right thing.

Fan-out multiplies outgoing traffic

A topic that receives 100 megabytes per second and is read by five independent consumer groups sends out about 500 megabytes per second to consumers, plus two more copies to the followers if the replication factor is 3. The outgoing side of a Kafka cluster often carries several times more traffic than the incoming side. When you plan network capacity, always plan for the readers as well as the writers.

Partitions per broker

Each partition a broker leads or follows adds work for fetch: more entries in requests, more files, more state in sessions. Thousands of partitions on one broker are common and fine, but tens of thousands need special care. Fetch sessions were created exactly to keep the cost of many quiet partitions low, yet they do not make partitions free. Keep partition counts reasonable and give every partition a reason to exist.

Quotas keep neighbours polite

Kafka can limit how many bytes per second a client, a user or a group of them may fetch, and how much broker processing time they may use. When a client goes over its quota, the broker delays its responses and reports the delay in the throttle time field. Quotas stop one runaway replay job from stealing bandwidth from everyone else. For replication traffic during partition moves, separate throttle settings exist for the same reason.

Fetch is the door through which data leaves the cluster, so it is also the door that must be guarded.

Reading data is a powerful action. A leaked read permission can expose customer records, payments or private messages without any warning. Kafka lets you protect fetch in several layers, and a healthy cluster uses all of them.

Mistakes that expose data

The most common security problem with fetch is not a clever attack but a broad permission. A wildcard read permission for a whole team, given “just to make it work”, quietly allows access to every topic, including new ones. Another frequent mistake is sharing one identity between many applications, which makes it impossible to tell who read what. A third is turning off encryption inside a “trusted” network, when a single compromised machine could then listen to all traffic between brokers.

Fetch sessions and abuse

Because sessions use limited memory, a badly behaved or malicious set of clients could try to fill the session cache and force other clients into heavier full fetches. Kafka’s cache rules favour replication and recently active sessions, and connection limits and quotas add further protection. Still, the safest position is to allow only authenticated clients, watch the eviction metrics, and set connection limits per address where it makes sense.

Which numbers to watch, what they mean, and the one metric that surprises almost everyone.

Brokers and clients both publish metrics about fetch. The names below are the ones commonly used in Kafka’s monitoring interface. They can differ slightly between versions, so confirm them in the documentation of the version you run.

Broker-side metrics

Client-side metrics

flowchart LR
  BM["Broker metrics
queue time, idle percentage, sessions, follower lag"] --> COL["Metrics Collector"]
  CM["Client metrics
fetch rate, size, latency, lag"] --> COL
  COL --> DASH["Dashboards"]
  COL --> AL{"Thresholds crossed?"}
  AL -->|"Yes"| ON["Alert the on-call engineer"]
  AL -->|"No"| REV["Weekly capacity review"]
  ON --> RB["Runbook: find the layer that is slow"]

Logs

Kafka can log every request, including fetch requests, at a detailed level. This is very useful for a short investigation, but it produces a huge volume of lines on a busy cluster, so turn it on for a short time and then off again. Broker logs also record follower fetch problems, such as errors talking to a leader, and messages about truncation. Client logs show metadata refreshes, session resets and offset resets. Learn what a calm cluster’s logs look like, so that unusual lines stand out.

Now that you know the pieces, let us watch a real consumer use all of them in a loop.

A consumer application does not send a single fetch and stop. It lives in a loop that repeats for hours or months. Each turn of that loop connects several ideas from this guide: metadata, positions, fetch requests, sessions, buffering and offset commits. Seeing the loop as one story makes it much easier to find where something goes wrong.

flowchart TD
  A["Start: get cluster metadata
learn which broker leads each partition"] --> B["Join the consumer group
receive a list of partitions"]
  B --> C{"Is there a committed offset
for each partition?"}
  C -->|"Yes"| D["Start at the committed offset"]
  C -->|"No"| E["Apply auto.offset.reset
earliest or latest"]
  D --> F["Send one fetch request per broker
full first, then incremental"]
  E --> F
  F --> G["Broker waits, then replies with batches"]
  G --> H["Client buffers the batches"]
  H --> I["poll returns up to max.poll.records"]
  I --> J["Application processes the records"]
  J --> K["Commit offsets"]
  K --> F

Step by step through the loop

Backpressure comes for free

The consumer does not ask a broker for more data while it still holds unprocessed data for that broker’s partitions. If your application is slow, the buffer stays full, the client sends fewer requests, and the network naturally calms down. No special flow-control message is needed. This is one of the quiet benefits of pull-based fetching, and it protects both the consumer’s memory and the broker.

What a rebalance does to a session

When group membership changes, the partitions a consumer owns change too. The consumer then stops reading some partitions and starts reading others. In session terms, the partitions it lost are placed in the forgotten list of the next incremental request, and the partitions it gained appear as new entries. If the change is large, the client may simply drop the session and begin with a fresh full fetch. Either way, the session adapts without any special ceremony.

Real problems with fetch tend to repeat. Here are the common ones and how to tell them apart.

flowchart TD
  S["Consumers are slow or lagging"] --> A{"Is fetch latency high
on the client?"}
  A -->|"No"| B["Look at the application
processing time and poll loop"]
  A -->|"Yes"| C{"Is the broker request
queue time high?"}
  C -->|"Yes"| D["Broker is busy
check handler idle percentage and evictions"]
  C -->|"No"| E{"Is the client throttled
by a quota?"}
  E -->|"Yes"| F["Adjust the quota or the workload"]
  E -->|"No"| G{"Are disks or page cache
under pressure?"}
  G -->|"Yes"| H["Cold reads or slow disks
throttle replays and fix storage"]
  G -->|"No"| I["Check the network path
and leadership changes"]

A simple order of investigation

Habits that keep reads cheap and predictable, and habits that cause the incidents above.

Patterns that work

Anti-patterns to avoid

How well-known organizations meet the pressures that fetch and fetch sessions were built for.

A checklist to carry into your next design review or capacity meeting.

Common mistakes in more detail

Mistake one: believing fetch sessions change the data. A session only changes how requests are described. The records, the ordering and the guarantees are identical with or without sessions. If a problem involves missing or repeated data, look at offsets, epochs and commits, not at sessions.

Mistake two: forgetting the follower side. Many teams tune consumers carefully and never look at replica fetch settings. Yet replication is often the busiest fetch traffic on the cluster, and its health decides how safe your data is.

Mistake three: reading more than you need. Consumers that subscribe to a topic and throw most of the records away still cost the cluster the full bandwidth. Filtering at the source, splitting topics sensibly or using compacted views can save far more than any tuning.

Mistake four: treating the client and the broker as separate worlds. The best fixes often need a change on both sides, for instance a producer batching change and a consumer min bytes change together.

The core idea stays the same, while the edges keep improving.

The fetch protocol has changed steadily over the years by adding fields, not by breaking old ones. Topic IDs now let requests refer to a topic by a unique number, which avoids confusion when a topic is deleted and re-created with the same name. Leader epochs made truncation and divergence detection precise. Rack-aware reads moved data closer to its readers. In clusters without ZooKeeper, the controllers copy their own metadata log using the same pull-based pattern, with extensions for their special needs.

Other developments touch fetch indirectly. Tiered storage moves old data to remote storage, so cold reads no longer disturb local disks in the same way. Newer consumer group protocols move more of the coordination work to the broker, which reduces the disruption caused by rebalances. Proposals for queue-like consumption have been discussed and are arriving in stages. Since these features are at different levels of maturity in different releases, read the release notes for your version before depending on any of them.

Let us put numbers on a made-up company, so that the ideas turn into decisions.

Imagine a food-delivery company called QuickBite. Its Kafka cluster has six brokers, two in each of three zones. There is one busy topic called order-events that receives about 40 megabytes per second. Six independent consumer groups read it: billing, kitchen tablets, delivery routing, fraud checks, analytics and a customer-notification service. In total, about 1,200 consumer instances run across the company, and most of them read from every broker. Each broker also follows partitions from the other five brokers using two replica fetcher threads per source.

Step one: size the session cache

The team sets the slot count to 2,000, which leaves comfortable room. Before the change, the eviction metric showed a steady stream of evictions and many clients running full fetches. After the change, evictions drop close to zero, and broker CPU during peak hours falls.

Step two: cut cross-zone traffic

The topic’s traffic is 40 megabytes per second in, and six groups read it, so about 240 megabytes per second flow out to consumers. Because leaders are spread over three zones, roughly two thirds of reads cross a zone boundary. That is about 160 megabytes per second, which adds up to nearly 14 terabytes per day. These are illustrative numbers, and real prices depend on the provider, but the size is clear. The team labels brokers with broker.rack, sets client.rack on the consumers, and enables the rack-aware replica selector. The delivery routing group, which needs the freshest data, stays on the leader by using a client without a rack label. The others read locally and accept a tiny extra delay.

Step three: give each group a profile

Everything in this guide can be held in a few sentences.