Before we talk about two special numbers, we need a shared vocabulary. Here is the smallest useful picture of Kafka.

Apache Kafka is a system for moving streams of events between programs. An event (Kafka usually calls it a message or a record) is a small fact: “order 4471 was paid”, “driver 88 moved to a new street”, “user clicked Buy”. Programs that create events are called producers. Programs that read them are called consumers. The machines in the middle that store the events are called brokers, and a group of brokers working together is a Kafka cluster.

Events are grouped into topics. A topic is like a named channel or folder: one for payments, one for clicks, one for driver locations. A single machine cannot store or serve a giant stream alone, so each topic is cut into partitions. A partition is an ordered, append-only list of messages. “Append-only” means new messages are added at the end and old ones are never edited in place.

Offsets: the line numbers of a partition

Every message in a partition receives a number called its offset. The first message gets offset 0, the next gets 1, then 2, and so on. An offset is not a timestamp and not a global ID. It is a position inside one partition. Partition 0 and partition 1 of the same topic each have their own offset 0, and they never share a counter.

The log on disk

On disk, a partition is stored as a log: a folder holding a series of files called segments. Kafka writes only to the end of the newest segment. Sequential writes are very fast because disks and operating system caches are built for them. Old segments are deleted or compacted later according to retention rules, such as “keep seven days of data”.

One copy of your data on one machine is a single point of failure. Kafka keeps several copies of every partition on different brokers.

Each copy of a partition is called a replica. The number of copies is the replication factor. With a replication factor of 3, every partition lives on three different brokers, and the cluster can lose a machine without losing the data. Many production clusters use a factor of 3 because it balances safety against storage cost.

Among the replicas of a partition, exactly one is elected the leader. All writes from producers go to the leader. By default, reads from consumers also go to the leader. The other replicas are followers. A follower has one job: copy the leader’s log, in order, as quickly as it can. Followers do this by sending the leader ordinary fetch requests, the same kind of request a consumer uses.

The In-Sync Replicas (ISR)

Not every follower keeps up. A broker can be slow, overloaded, or cut off by a network problem. Kafka therefore tracks a list called the In-Sync Replicas, or ISR. It contains the leader plus every follower that is currently caught up closely enough. A follower that lags for too long is removed from the ISR, and it is added back once it catches up again.

The ISR is the group of “students in good standing”. It matters enormously for our two numbers, because the High Watermark is computed from the ISR only, not from every replica that exists on paper.

Every replica, leader and followers alike, keeps its own private Log End Offset. It answers one question: how much have I written so far?

The Log End Offset (LEO) of a replica is the offset that the next new message will receive in that replica’s log. Another way to say it: LEO equals the offset of the last message plus one. If a replica holds messages at offsets 0 through 9, its LEO is 10. If its log is empty, its LEO is 0.

Every replica has its own LEO

This is the first big idea. There is no single “LEO of the partition”. There is one LEO per replica. The leader’s LEO moves up when a producer’s batch is appended to the leader’s log. A follower’s LEO moves up when the follower appends data it fetched from the leader. Because followers copy after the leader writes, follower LEOs are usually equal to or smaller than the leader’s.

The gap between the leader’s LEO and a follower’s LEO is called replica lag. In the example above, Broker 3 has a lag of 3 messages. A small lag that comes and goes is normal, since copying always trails writing by a moment. A lag that keeps growing is a warning sign that a broker is struggling.

When can a LEO change?

A LEO grows whenever new data is appended. Under normal conditions it never shrinks. There is one exception that we will study in detail later: after a crash and a leader change, a follower may discover that its log contains messages the new leader never had. It must then cut those messages off, and its LEO moves backwards. This deliberate rewinding is called truncation.

If LEO says how much has been written, the High Watermark says how much is safe. It is the line that separates “written” from “committed”.

The High Watermark (HW) of a partition is the offset up to which every in-sync replica has a copy of the data. Precisely, the HW equals the smallest LEO among all replicas in the ISR. Messages with an offset below the HW are called committed. Messages at or above the HW exist on the leader but are still uncommitted.

Take the earlier numbers: the leader has LEO 10, one follower has LEO 10, and another follower has LEO 7. The smallest of the three is 7, so the HW is 7. Offsets 0 to 6 are committed. Offsets 7, 8 and 9 sit on the leader (and one follower) but are not yet committed, because the slowest in-sync replica has not copied them.

The two promises of the High Watermark

Why not let consumers read everything?

Imagine a consumer could read the uncommitted tail of the leader’s log. Then the leader crashes before any follower copies those messages. A follower becomes the new leader, and it has never seen them. The consumer has now processed events that, from Kafka’s point of view, never happened. It could have charged a card, sent an email, or updated a balance based on a ghost. Worse, its saved position could point past the end of the new leader’s log.

By hiding everything above the HW, Kafka guarantees that what consumers see is a stable history. Any message a consumer has been shown will still be there after a leader change. This guarantee is the entire reason the High Watermark exists.

Now that each number has a definition, let us put them in one picture and one table, so the difference sticks.

flowchart LR
  P["Producer"] --> L["Leader replica on Broker 1
LEO = 10"]
  L -->|"followers fetch"| F1["Follower on Broker 2
LEO = 10"]
  L -->|"followers fetch"| F2["Follower on Broker 3
LEO = 7"]
  L --> HW["High Watermark = smallest ISR LEO = 7"]
  HW --> C["Consumer can read offsets 0 to 6"]

Read the figure from left to right. The producer writes to the leader, whose LEO becomes 10. Two followers copy from the leader, and their LEOs are 10 and 7. The leader looks at all three LEOs, picks the smallest, and calls that number the High Watermark. Consumers are then allowed to read offsets 0 through 6.

The three zones of a leader’s log

At any moment the leader’s log can be divided into three zones, using the two numbers as borders. Understanding these zones makes almost every Kafka replication behaviour easy to predict.

Let us follow one message through its whole life and watch both numbers move at each step.

We will use a partition with three replicas, all in the ISR. At the start, every replica has LEO 10 and the HW is 10, which means offsets 0 to 9 are committed and nothing is in flight. A producer now sends one new payment message.

The leader never guesses how far a follower has got. It learns it from the follower’s own requests, and this small trick is the engine behind the High Watermark.

Replication in Kafka is pull-based. The leader does not push data to followers. Instead, each follower runs a background worker, called a replica fetcher, that repeatedly sends fetch requests to the leader. A fetch request says, in effect: “I am replica number 2, and I want data for partition 0 starting at offset 10. Please send me at most this many bytes, and if you have nothing new, wait a little before answering.”

The number inside that request, the starting offset, is the important part. A follower always asks for the next offset it does not have, which is exactly its own Log End Offset. So the fetch offset is a free progress report. When the leader receives a request for offset 11 from a follower, it knows that the follower already holds everything below 11, and therefore that the follower’s LEO is 11.

sequenceDiagram
  participant P as Producer
  participant L as Leader on Broker 1
  participant F as Follower on Broker 2
  P->>L: Produce one message
  Note over L: Message gets offset 10, leader LEO goes 10 to 11, HW stays 10
  F->>L: Fetch from offset 10
  L-->>F: Message 10 plus leader HW = 10
  Note over F: Follower LEO goes 10 to 11, follower HW stays 10
  F->>L: Fetch from offset 11
  Note over L: Leader learns follower LEO = 11, so HW moves to 11
  L-->>F: No new data, but leader HW = 11
  Note over F: Follower HW moves to 11
  L-->>P: Success reply when acks = all

What a fetch request and reply carry

Long polling keeps the system calm

What if the leader has no new data when a fetch arrives? It does not answer with an empty reply and force the follower to ask again instantly, which would flood the network. Instead it parks the request in a waiting area (Kafka calls it a purgatory) and answers as soon as enough new data arrives or the wait time runs out. The default maximum wait for replica fetches is 500 milliseconds. This technique is called long polling, and it gives low delay without wasted traffic.

The rule is simple: the HW is the smallest LEO in the ISR. The surprise is in the timing, because followers learn about it one step later than the leader.

Whenever the leader appends data, or receives a fetch from a follower, it may recompute the HW. It looks at its own LEO and at the last known LEO of every follower in the ISR, takes the minimum, and, if that minimum is higher than the current HW, moves the HW up. The HW only moves forward on a healthy leader.

Followers cannot compute the HW themselves, because they do not know each other’s LEOs. They simply copy the value the leader includes in each fetch response. To stay safe, a follower sets its own HW to the smaller of two numbers: the leader’s HW from the reply and its own LEO. It would be wrong for a follower to claim a HW beyond what it has actually written.

A step-by-step example

Start with three in-sync replicas that are fully caught up at LEO 10 and HW 10. Then a producer sends a batch of two messages. Follow the numbers in the table below, row by row.

Notice three things. First, at T2 and T3 the followers already hold the new messages, yet nothing is committed, because the leader does not yet know that. Knowledge travels only with the next fetch. Second, the leader’s HW moves at T5, the moment the slowest in-sync replica reports in. Third, the followers’ own HW values trail the leader’s by one more fetch round.

Special cases

So far we watched the numbers from the outside. Now let us step inside one broker and see which parts touch them, and in what order.

A Kafka broker is a busy program with several groups of workers. Network threads accept connections and read incoming requests from the wire. Request handler threads do the real work: appending to logs, reading from logs, and building replies. A component often called the replica manager keeps track of every partition that lives on this broker, and for each one it knows whether the local copy is a leader or a follower. Everything we have discussed happens inside these pieces.

The write path on the leader

The read path: who may read how far

Here is a detail that surprises many people. The leader treats followers and consumers differently when serving fetch requests. A follower is allowed to read all the way up to the leader’s LEO, including the uncommitted tail, because copying that tail is exactly its job. A consumer is only allowed to read up to the High Watermark, or up to the Last Stable Offset if it uses read_committed. The same code path serves both, and the difference is simply which limit is applied.

When a follower’s fetch arrives, the handler thread does three extra things beyond reading the data. It records the follower’s fetch offset as that follower’s newest known LEO. It notes the time, so it can later judge whether the follower is still keeping up. And it recomputes the HW, because the follower’s new position might raise the minimum. If the HW did move, the handler completes any produce requests waiting on the rail. Only then does it send the reply, which carries the data plus the leader’s current HW.

Waiting rails, not busy loops

Two kinds of waiting operations exist on a broker. Delayed produce operations wait for the HW to reach a target. Delayed fetch operations wait for enough new data to arrive, which is the long polling from Chapter 7. Both are stored in purgatories that use an efficient timer structure, called a hierarchical timing wheel, so that even hundreds of thousands of waiting requests cost very little to track. Whenever something changes, such as new data appended or the HW moved, the broker checks only the waiting operations tied to that partition, instead of scanning everything.

The producer setting called acks decides when the producer is told “your message is safe”. Two of its values are directly tied to our two numbers.

A producer can choose how much confirmation it wants from the cluster before it treats a send as successful. This setting is named acks. Each choice trades speed against safety.

Where the waiting happens

With acks=all, the leader appends the batch to its log and then parks the producer’s request in a purgatory. It stays there until the High Watermark moves past the last offset of the batch. When that happens, the leader releases the request and sends success. If this does not happen within the producer’s timeout, the producer sees an error and can retry. So the delay of an acks=all write is roughly the time for followers to fetch the data plus the time for the leader to learn about it.

min.insync.replicas: a floor for the ISR

There is a catch. “All in-sync replicas” can shrink to just the leader, and then acks=all would behave like acks=1. To prevent this, Kafka offers min.insync.replicas. It sets the minimum ISR size required for a write with acks=all to be accepted. If the ISR is smaller than this floor, the leader rejects the write with an error rather than pretending it is safe. This protects committed data at the cost of availability for writes.

This trio is a widely used starting point for important topics. With it, one broker can fail and writes still succeed, because two in-sync replicas remain. If a second broker fails, the ISR drops below two, and writes are refused, but existing committed data is still readable. An acknowledged message always lives on at least two brokers, so losing any single broker never loses it.

The High Watermark is only as trustworthy as the ISR behind it. Let us look at how replicas earn and lose their place.

A follower stays in the ISR while it keeps up with the leader. The exact test is time-based. If a follower has not fully caught up with the leader’s log for longer than replica.lag.time.max.ms, the leader declares it out of sync and removes it. In current Kafka versions this setting defaults to 30 seconds; older versions used 10 seconds. Kafka used to also have a rule based on how many messages a follower could lag, but it was removed because bursts of traffic made healthy followers flap in and out.

stateDiagram-v2
  [*] --> InSync
  InSync --> OutOfSync : falls behind for too long
  OutOfSync --> InSync : catches up to the leader HW
  InSync --> [*] : broker stops or is removed

Who updates the ISR list?

The leader notices a lagging or recovered follower, but the official ISR list is stored in the cluster’s metadata, which the controller manages. In modern Kafka, the leader sends a request asking the controller to approve the change, and then updates its own state once approved. Older clusters recorded this in ZooKeeper instead. Either way, the effect is the same: the shared record of “who is in sync” is kept consistent, so that after a crash the cluster knows which replicas are safe leaders.

What an ISR change does to the numbers

Common reasons for ISR flapping include long garbage collection pauses on a broker, saturated disks, network problems between racks, overloaded brokers hosting too many partitions, and replication throttles set too low during maintenance. Frequent shrinks and expands are therefore one of the best early warning signs in a cluster, and we will return to them in the monitoring chapter.

A replica is not just “in the ISR” or “out”. It is born, catches up, works, sometimes leads, and eventually retires. Following that path shows how LEO and HW behave in each stage.

Consider what happens when a cluster grows. An operator adds a new broker and asks Kafka to move some partitions onto it. For each moved partition, a brand-new replica is created on the new machine. It starts with an empty log, so its LEO is at the beginning, while the leader’s LEO may be in the millions. The stages that follow are the same ones a recovering replica goes through after a long outage.

Why newcomers stay out of the ISR

Keeping a catching-up replica out of the ISR is a deliberate safety rule. If a replica with LEO 0 were counted immediately, the minimum would drop to 0 and the High Watermark would collapse, hiding the whole log from consumers and stalling every acks=all producer. Waiting until the replica reaches the HW means that joining the ISR can never pull the watermark backwards.

A consumer’s view of a partition ends at the High Watermark. Understanding this explains lag numbers, delays and a few surprising tool outputs.

A consumer keeps a position: the offset of the next message it wants. It sends fetch requests to the partition leader with that offset. The leader returns messages starting there, but only those below the HW. If the consumer is fully caught up and asks for an offset equal to the HW, the leader holds the request briefly and answers when new committed data appears.

Periodically, the consumer saves its progress by committing an offset. Kafka stores these commits in an internal topic, so a restarted consumer resumes where it stopped. The committed offset is the consumer’s bookmark. It is a different thing from the HW, even though both are offsets in the same partition.

Consumer lag

Consumer lag is the distance between what is available to read and what the consumer has finished. In simple terms it is the latest readable offset minus the consumer’s committed offset. A lag of 5,000 means there are 5,000 messages waiting. Lag is the single most watched number in most streaming systems, since growing lag means consumers cannot keep up.

Transactions and the Last Stable Offset

Kafka also supports transactions, where a producer writes several messages that should become visible together or not at all. Consumers configured with the read_committed isolation level must not read messages from a transaction that is still open. For them, Kafka uses another boundary called the Last Stable Offset (LSO): the offset before the first still-open transaction. The LSO is always less than or equal to the HW. So a read_committed consumer sees a slightly shorter log than a default one, and both see a shorter log than the leader’s LEO.

Reading from the nearest replica

By default all reads go to the leader, which can be in a different data centre zone from the consumer and cause cross-zone network cost. Since Kafka 2.4, a feature often called follower fetching lets a rack-aware consumer read from a nearby follower instead. The follower serves only offsets below its own HW. Since a follower’s HW trails the leader’s by a fetch round, data reaches a follower-reading consumer a little later, a small price for lower network cost and shorter distance.

This is the moment the High Watermark earns its keep. It draws the exact line between data that must survive and data that may be lost.

When the broker holding a partition’s leader crashes or becomes unreachable, the cluster’s controller notices and picks a new leader from the ISR. Only in-sync replicas are eligible in the normal case, because they are guaranteed to hold every committed message. The controller records the decision, increases a counter called the leader epoch (we will explain it soon), and informs the brokers involved.

flowchart TD
  A["Leader on Broker 1 fails"] --> B["Controller checks the ISR list"]
  B --> C{"Is another in-sync replica alive?"}
  C -->|"Yes"| D["Pick it as new leader and raise the leader epoch"]
  C -->|"No"| E{"Is unclean election allowed?"}
  E -->|"No"| F["Partition stays offline, no committed data lost"]
  E -->|"Yes"| G["Out-of-sync replica leads, committed data may be lost"]
  D --> H["Committed messages are safe on the new leader"]

What happens to each zone of the old log

The new leader’s log end becomes the new truth. Any follower that has messages beyond it must cut them off. The new leader also starts with whatever HW it last learned, which may be a little lower than the real one. It recomputes the HW as followers reconnect and fetch. This can cause a short pause in which consumers see no progress, a normal and brief effect of a leader change.

Unclean leader election: choosing availability over safety

The setting unclean.leader.election.enable answers this question: if no in-sync replica is alive, should the cluster wait, or promote an out-of-sync replica? The default is off, meaning Kafka waits and protects the data. Turning it on makes the partition available again sooner, but the promoted replica may be missing messages that consumers already read, and those messages can vanish or be overwritten by new ones. This is a conscious trade between the letters C and A of the CAP theorem: consistency of committed data versus availability of the partition.

For years, Kafka used the High Watermark to decide how much of a follower’s log to trust after a restart. That shortcut had a flaw, and the fix is one of the most elegant ideas in Kafka replication.

When a follower restarts or becomes a follower again, it must make sure its log is a true prefix of the new leader’s log. Any extra messages it holds that the leader lacks must be removed. That removal is truncation. In early versions, the follower truncated its log to its own stored High Watermark, on the theory that everything below the HW was certainly committed.

The theory ignored the one-round delay from Chapter 8. A follower’s stored HW can lag behind the truth. So a follower could truncate messages that were in fact committed and acknowledged.

The data-loss scenario in plain steps

A second scenario was possible too, called replica divergence. After certain crash sequences, two replicas could end up with different messages at the same offset, and each believed it was in sync. That is arguably worse than losing data, because the copies silently disagree.

The fix: leader epochs

Kafka 0.11 introduced the leader epoch. It is a number attached to each partition that increases by one every time a new leader is elected. Every batch of messages is stamped with the epoch of the leader that wrote it, and every replica keeps a small table recording, for each epoch, the offset at which that epoch began.

sequenceDiagram
  participant F as Restarted follower
  participant L as Current leader
  F->>L: Where did my latest epoch end?
  L-->>F: That epoch ended at offset 5
  Note over F: Truncate anything after offset 5
  F->>L: Fetch from offset 5
  L-->>F: Send messages from offset 5 onward

On restart, the follower asks the leader for the end offset of the latest epoch the follower knows about. The leader answers from its own history. The follower truncates to that answer, or to its own LEO if that is smaller, and then resumes fetching. Since the answer comes from the leader’s real log rather than from a possibly stale HW, acknowledged messages are no longer thrown away, and diverging logs are detected and repaired.

A broker’s memory disappears when it restarts, but the High Watermark and its friends must not. Kafka saves them in small checkpoint files.

Each broker keeps a few tiny files that act as bookmarks. They are rewritten regularly so that after a restart the broker does not have to start from zero. Think of them as sticky notes stuck to each partition’s folder. They are hints for recovery, not the source of truth, since the truth is always the log itself and the leader’s answers.

What happens when a broker starts

Planned restarts are gentler

Operators restart brokers constantly for upgrades. Kafka supports controlled shutdown: before a broker stops, it asks the controller to move leadership of its partitions to other in-sync replicas. Producers and consumers see a brief blip instead of an outage, and no ISR has to shrink because of a surprise. This is why rolling upgrades restart one broker at a time and wait for the cluster to be fully in sync before moving to the next.

Definitions are useful, but understanding sticks when you follow numbers through a crisis. Here are four short stories, each using the safe trio: replication factor 3, min.insync.replicas 2, acks=all.

Story 1: the sleepy follower

A partition has its leader on Broker 1 and followers on Brokers 2 and 3. All three LEOs are near 5,000. Then Broker 3 hits a long garbage collection pause and stops fetching. Its LEO freezes at 5,000 while the leader keeps accepting writes, reaching 5,300. The HW is the minimum, so it also freezes at 5,000. Producers using acks=all start seeing their sends wait, because nothing new is committed.

After the lag time limit, the leader removes Broker 3 from the ISR. The ISR now has two members, Broker 1 (LEO 5,300) and Broker 2 (LEO 5,300). The minimum is 5,300, so the HW jumps to 5,300 and every waiting producer is released at once. The ISR size of 2 still meets the floor, so writes continue. A few minutes later, Broker 3 wakes up, catches up, and rejoins the ISR. Users saw a short latency spike and nothing else.

Story 2: the leader dies with an uncommitted tail

Now the leader on Broker 1 has LEO 1,000. Broker 2 has LEO 995. Broker 3 has LEO 990. The HW is 990. The leader crashes. The controller picks Broker 2, which is in the ISR, as the new leader, and raises the leader epoch. Broker 2’s log ends at 995, so that is the new end of history.

Look at the messages at offsets 990 to 994. They were uncommitted under the old HW, so their producers never received success. But Broker 2 holds them, so they survive the failover, and they will become committed once Broker 3 copies them. The messages at offsets 995 to 999 exist only on the dead Broker 1. When it eventually returns, it asks the new leader where its epoch ended, learns that the answer is 995, and truncates offsets 995 to 999 away.

Story 3: the isolated old leader

A network fault cuts Broker 1, the leader, off from the rest of the cluster while it keeps running. From its own point of view nothing has changed, and it still believes it leads. Meanwhile the controller cannot reach it, elects Broker 2 as the new leader, and raises the leader epoch. Some producers, whose information is stale, still send data to Broker 1.

With min.insync.replicas set to 2, Broker 1 cannot safely commit anything. Its followers no longer fetch from it, so its HW cannot move, and acks=all requests just wait and then fail. This is the safety floor doing its job. When the network heals, Broker 1 learns about the higher epoch. Requests that still carry the old epoch are rejected as coming from a stale leader, and Broker 1 becomes a follower, truncates its uncommitted tail using the epoch answer, and resumes copying from the true leader. Two leaders never both commit data.

Story 4: the broker with a dead disk

Broker 3 loses its disk entirely and is replaced with a fresh machine with the same identity. The ISR for its partitions shrinks to two brokers. The partitions keep working with two copies, and the safe trio still allows writes. The new machine starts fetching every partition from the earliest available offset, and each of its replicas climbs from LEO 0 toward the leader’s LEO.

During this catch-up, the cluster is running at reduced protection. If another broker in one of those ISRs failed now, only one in-sync replica would remain, the ISR would fall below the floor, and writes to that partition would be refused. Committed data would still be safe on the survivor. When the new replicas reach the HW, they rejoin the ISR and full protection returns.

Kafka’s replication is a set of deliberate choices. Knowing what it gives, and what it costs, helps you decide when it is the right tool.

Many well-known databases use a majority quorum. With five replicas they consider a write committed when three of them have it, and they tolerate two failures. Kafka takes a different route with the ISR. A write is committed when all in-sync replicas have it, and the ISR itself can shrink. The two approaches make different trades.

The ISR model needs fewer copies for the same failure tolerance, which matters when each partition may hold terabytes. The cost is that one struggling follower can slow writes until the lag limit removes it. Kafka accepts that cost because copies are expensive and because the removal rule bounds the damage.

What Kafka guarantees, and what it does not

The CAP view in one paragraph

The CAP theorem says that when a network splits a system, you must choose between staying consistent and staying available. Kafka’s defaults lean toward consistency for committed data: it refuses writes when the ISR is below the floor, and it keeps a partition offline rather than electing an out-of-date leader. Each of these choices can be relaxed per topic. Writes with acks=1 favour availability and speed, and unclean election favours availability at the cost of possible loss. The High Watermark is the mechanism that keeps the consistent side of that choice honest.

Replication is not free. Every byte written to a leader must be copied to its followers, and that copying decides how quickly the High Watermark can move.

Suppose a topic receives 100 megabytes per second of new data and has a replication factor of 3. The leaders receive 100 megabytes per second from producers, and must send out another 200 megabytes per second to two followers each. On top of this come the consumers’ own reads. In short, replication multiplies network use by roughly the replication factor minus one, so capacity planning must include it.

Why acks=all latency is a chain

The time for a message to be committed is roughly: leader append time, plus the wait for each follower’s fetch to arrive, plus the follower’s append time, plus one more fetch to report back. The slowest in-sync follower sets the pace, because the HW is the minimum. This is why a single overloaded broker can raise tail latency for all the partitions it follows, even if their leaders are healthy.

Scaling out

Kafka scales by adding partitions and brokers. More partitions spread leaders across more machines, which lifts total throughput. But every partition means more replication state, more fetch work and more files. Very high partition counts per broker slow down leader elections and recovery, because every partition needs its own epoch handling and checkpoint. Good designs keep partition counts sensible and grow the cluster horizontally before a single broker becomes crowded.

The High Watermark protects the integrity of your history. Protecting who can read or change it is a separate job.

Replication traffic is just brokers talking to each other, so it needs the same care as any internal service traffic. Brokers should authenticate to one another, and the link between them should be encrypted. Kafka lets you configure a dedicated listener for communication between brokers, separate from the one clients use, so you can apply stricter rules to it and keep it away from the public network.

Note the link between safety settings and security. A fake follower that fetches data is really just a reader with access to everything. An unclean leader election can also be seen as an integrity risk, since it may let unreplicated, out-of-date history become the official one. Keeping unclean election off on sensitive topics is therefore a data protection choice as well as an availability choice.

Because LEO and HW describe the health of replication, watching them, and the metrics built on them, reveals trouble long before users notice.

Kafka exposes many metrics through JMX and monitoring agents. You do not need all of them. A small set tells the story of replication health, and most on-call playbooks start there.

A simple diagnosis path

Every company that runs Kafka at scale lives with these two numbers, even if their dashboards only show the friendly summaries built on top.

Good replication design is mostly a handful of proven combinations, plus a list of tempting shortcuts to avoid.

Patterns that work

Anti-patterns to avoid

A short checklist for design reviews and a table of mistakes that show up again and again.

Some beliefs about these two numbers feel obvious and are quietly wrong. Here are the ones that cause the most confusion in design meetings and incident calls.

Gotchas worth writing on the wall

Let us apply everything to a practical exercise. A company runs one cluster with four kinds of topics. Each needs different answers to the same questions.

Start by asking four questions about any topic. How costly is it to lose one message? How much delay can the writers tolerate? How many broker failures must the topic survive? And do consumers act on the data in ways that cannot be undone? The answers give you the three settings from Chapter 10 and the election policy from Chapter 14.

The reasoning behind each row

Payment events use the safe trio. Every acknowledged message is on at least two brokers, one broker can fail without stopping writes, and a partition without an in-sync replica waits rather than gambling. Producers are idempotent, and consumers that trigger irreversible actions use read_committed if the pipeline is transactional.

The audit trail is treated the same way, with attention to retention time and to replicas spread across zones, since these records must be there when someone asks for them years later.

Website clickstream can tolerate losing a handful of events, but not losing whole partitions during an outage, so it keeps three replicas and leaves unclean election off. Whether to use acks=all depends on the delay budget. Many teams choose acks=1 here to keep the front-end snappy and accept a tiny loss window.

Device debug metrics are high in volume, low in value, and useful mainly in aggregate. Two replicas and light acknowledgements keep costs down. Allowing unclean election is a conscious choice that keeps the topic writable through outages, at the risk of a gap in a chart.

Short answers to the questions people ask most often when these two numbers first click.

Two numbers, one simple idea: write first, commit when everyone who matters has a copy.

Every replica of a Kafka partition keeps a Log End Offset that says how much it has written. The leader combines the LEOs of all in-sync replicas into a High Watermark that says how much is safe. Producers, consumers, leader elections and restarts all depend on the gap between those two lines. Understanding that gap turns many mysterious Kafka behaviours, such as stalled acknowledgements, sudden lag, or surprising data after a failover, into predictable outcomes.