A short refresher on the moving parts, so that the rest of the guide has firm ground to stand on.

Apache Kafka moves streams of events between programs. A program that writes events is a producer, and a program that reads them is a consumer. The servers that store the events are brokers, and a group of brokers is a cluster. Events are organised into named streams called topics, and each topic is cut into partitions, which are ordered lists of messages that can live on different brokers so the work can be shared.

For safety, every partition is copied to several brokers. Each copy is a replica. One replica is the leader and handles all reads and writes for that partition. The others are followers that copy the leader. The set of replicas that are keeping up is the in-sync replica set, or ISR.

Metadata means “data about data”. In Kafka it is the description of the cluster itself, not the messages inside it.

Your messages are the cargo. The metadata is the map of the airport: which gates exist, which flight uses which gate, and which staff are on duty. It is small compared with the cargo, but without it nothing can be delivered. Kafka keeps several kinds of metadata, and each kind answers a different question.

Why every broker needs the whole picture

A broker stores only some partitions, yet it must know about all of them. A client may connect to any broker and ask about any topic, so each broker must be able to answer for the whole cluster. A broker also has to check, on every produce or fetch request, whether it is the correct leader for the partition being asked about, and it must reject requests that arrive at the wrong place. These checks happen thousands of times a second, and they all read the metadata.

To see why brokers keep a local copy, imagine the design without one.

Suppose the truth about the cluster lived in one place, and every broker had to ask that place a question each time a request arrived. A single busy broker may handle tens of thousands of requests each second. Multiplied across a cluster with dozens of brokers, that is a flood of lookups aimed at one small service. The lookups would add delay to every request, and the service would become both a bottleneck and a single point of failure.

What a cache buys

What a cache costs

A copy can become out of date. If the control room changes a gate and one screen has not updated yet, a passenger may walk to the wrong place. Kafka accepts this, because a wrong answer is cheap to correct: the request is rejected with a clear error, the client asks again, and the next answer is right. The rest of this guide is about making those corrections quick, safe and rare.

Here is the whole system in one picture, followed by a walk through each part.

flowchart LR
  CQ["Controller quorum
holds the metadata log"] -->|"metadata records"| B1["Broker 1
local metadata cache"]
  CQ -->|"metadata records"| B2["Broker 2
local metadata cache"]
  CQ -->|"metadata records"| B3["Broker 3
local metadata cache"]
  P["Producer"] -->|"Metadata request"| B1
  C["Consumer"] -->|"Metadata request"| B2
  A["Admin tool"] -->|"Metadata request"| B3

Read the figure from left to right. On the left is the controller quorum, a small group of servers that makes decisions about the cluster and records each decision in an ordered log. In the middle are the brokers. Each one reads that log and builds its own in-memory picture of the cluster, which is the metadata cache. On the right are the clients. They send Metadata requests to whichever broker they can reach and receive answers straight from that broker’s cache.

Two layers of caching

There are really two caches involved, and it helps to keep them apart. The first lives inside every broker, and it is a copy of the cluster’s state. The second lives inside every client, and it is a copy of the parts of that state the client cares about. A client asks a broker, keeps the answer for a while, and reuses it. The airport version: the broker cache is the departure board, and the client cache is the gate number printed on the passenger’s boarding pass.

Caches are copies, and copies need an original. In Kafka, the original is managed by the controller.

The controller is the part of the cluster that makes decisions about the cluster itself. It reacts when a broker joins or fails, chooses a new leader when one is lost, creates and deletes topics, changes partition counts, and updates configuration. Brokers do not make these decisions on their own, because two brokers deciding differently would leave the cluster in conflict. One authority avoids that.

The word controller has meant two slightly different things over the years. In older clusters, one ordinary broker was elected controller and used an external coordination service called ZooKeeper to store cluster state. In modern clusters, a dedicated group of controller nodes runs a built-in consensus protocol called KRaft and keeps the state in Kafka’s own log. Only one controller is the active one at a time. The others are ready to take over.

What the controller must guarantee

The same software can run in different roles. How you arrange those roles decides how well the metadata system copes with load and failure.

In modern Kafka, a server process can be given one or more roles. A broker role stores partitions and serves clients. A controller role takes part in the quorum that makes and records cluster decisions. A single process can hold both roles, called combined mode, or each can run on its own machines, called dedicated mode.

How many controllers?

The controller quorum needs a majority of its members to agree before any decision is committed. That is why quorums use an odd number of nodes. With three controllers, a majority is two, so the quorum tolerates the loss of one node. With five controllers, a majority is three, so it tolerates the loss of two. More than five brings little benefit, since each additional member adds more communication to every decision.

What brokers need from the controllers

Brokers are not voters. They do not count toward the majority. They only need to reach the active controller to fetch the metadata log, to register, and to send heartbeats. Each broker is told where the controllers are through configuration, and it keeps its own copy of the log on local disk. Controllers should sit on their own listener, reachable only by brokers and administrators.

Understanding the old design makes the new one easier to appreciate. It also explains many older articles you may still read.

For most of Kafka’s history, cluster state was stored in an external system called ZooKeeper. The elected controller broker read that state, made decisions, wrote results back to ZooKeeper, and then pushed updates to every other broker over ordinary requests. Two request types were central. A LeaderAndIsr request told specific brokers that they had become leader or follower of certain partitions. An UpdateMetadata request told every broker the new facts about leaders, replicas and live brokers, and each broker stored them in its metadata cache.

Why this design hit limits

When a controller failed, the replacement had to read the state of every topic and partition before it could work. With very large clusters this could take a long time, during which no leader changes were possible. This was a major reason the Kafka community designed a new approach, described in the next chapter. Recent Kafka releases, starting with version 4.0, no longer support ZooKeeper at all.

In modern Kafka, the cluster’s state is itself a Kafka-style log, and brokers keep their cache up to date by reading it, much as a consumer reads a topic.

In KRaft mode, all cluster decisions are stored as records in a special single-partition log, often called the metadata log (its internal topic name is __cluster_metadata). The active controller appends each decision to this log, and the controller quorum replicates it using a Raft-style consensus protocol so that a majority of controllers hold every record before it counts as committed. Brokers are not part of the voting group, but they read the log as followers, using the same fetch mechanism that replicas use.

What the records look like

Each record in the log describes one small change. Examples include a broker registering itself, a topic being created, a partition being added, a leader or ISR changing, a configuration value being set, an access rule being added, and a feature level being raised. Since each record is a tiny, self-contained fact, a broker can apply them one by one and always know exactly which version of the cluster picture it has reached, because every record has a position number, its offset.

Images and deltas: how the cache is built

A broker turns the stream of records into its cache in two steps. First, it keeps an immutable snapshot of the full cluster picture, known as the metadata image. Immutable means it is never edited in place. Second, when a batch of new records arrives, the broker builds a delta that describes what changed, applies it to produce a brand-new image, and then swaps the new image in for the old one. Threads that were reading the old image keep working with it safely until they finish. Because unchanged parts are shared between old and new images rather than copied, this is cheap even for very large clusters.

Many organisations ran ZooKeeper-based clusters for years. Moving them to the log-based design changes how the broker caches are filled, though not what clients see.

Because modern releases no longer support ZooKeeper, a cluster that began life in the older mode must be migrated before it can move to the newest versions. Kafka provides a migration path that lets operators do this in stages, without taking the cluster offline. Understanding what each stage does to the metadata caches makes the process less mysterious.

What clients notice

Very little. Clients use the same Metadata request before, during and after the move. Rolling restarts of brokers cause the usual short waves of leader changes and client refreshes, so the migration should be done at a quiet time with the same care as any rolling upgrade. The internal path by which brokers learn about changes is what differs: pushed requests before, a fetched log after.

A broker that starts up has an empty head. Before it can serve anyone, it must rebuild the picture of the whole cluster, and it must prove to the controller that it has done so.

When a KRaft broker starts, it does not ask the controller for a giant dump of state. It relies on two things it keeps on its own disk: a recent snapshot of the cluster picture and the tail of the metadata log after that snapshot. It loads the snapshot, replays the log records that came after it, and ends up with an image that is close to current. Then it connects to the active controller, registers itself, and continues fetching new records to close any remaining gap.

Snapshots keep the log short

The metadata log would grow forever if every record were kept. To prevent this, Kafka periodically writes a snapshot: a compact file holding the whole cluster picture as of a certain log position. Once a snapshot exists, older log records can be discarded. By default a new snapshot is taken after a sizeable amount of new records has been added, or after a long time interval, whichever comes first. A broker that has been offline so long that the records it needs are gone can simply download the latest snapshot from the controller and continue from there.

Fenced until ready

A broker that has not caught up must not receive client traffic, because its cache would give wrong answers. Kafka handles this with a state called fenced. A new or lagging broker registers as fenced. The controller keeps it fenced until the broker reports that it has read the log at least as far as its own registration record. Only then is the broker unfenced and eligible to lead partitions and to appear in metadata answers.

stateDiagram-v2
  [*] --> Starting
  Starting --> Fenced : loads snapshot and registers
  Fenced --> Active : caught up with the metadata log
  Active --> Fenced : misses heartbeats
  Active --> ShuttingDown : controlled shutdown
  ShuttingDown --> [*]

Heartbeats tie the cache to liveness

An active broker sends a regular heartbeat to the controller. By default this happens every couple of seconds, and if the controller hears nothing for several heartbeats, roughly nine seconds by default, it fences the broker and moves its partition leadership elsewhere. Each heartbeat reply also tells the broker whether it is still allowed to serve. In this way, having a fresh cache and being alive are joined together: a broker that cannot keep its cache current, or cannot talk to the controller, drops out of service rather than giving stale answers.

A new image in memory is only useful if the parts of the broker that need it actually notice. Here is how a change spreads through the inside of a single broker.

Inside each KRaft broker, a component reads batches of records from the metadata log as they arrive. For every batch, it collects the changes into a delta and builds a new image from the previous one plus the delta. That new image is then handed to several listeners, sometimes called publishers, each responsible for one part of the broker’s behaviour. Every listener looks at the delta and acts only on what concerns it.

Ordering inside the broker

Changes are applied in log order, and a new image only becomes visible after its delta has been fully applied. A request handler thread therefore never sees half of a change. Consider a leader change: the partition’s entry in the image gains its new leader and epoch in one step, not first the leader and then the epoch. This all-at-once switch is exactly what the immutable image design provides.

When something goes wrong while applying

Occasionally, applying a record fails, for instance because of a full disk when creating a partition folder. A failure in one listener does not stop the image from being updated for the others, and the failure is counted in error metrics so operators can see it. Serious failures make the broker shut itself down rather than continue with a cache that does not match reality, which is safer than serving wrong answers indefinitely.

Let us follow one small change all the way from a request to every broker’s memory.

An engineer asks for a new topic named orders with 6 partitions and 3 replicas each. This is a typical metadata change, and it touches every stage we have discussed.

sequenceDiagram
  participant A as Admin client
  participant B as Any broker
  participant C as Active controller
  participant O as Other brokers
  A->>B: Create topic orders
  B->>C: Forward the request
  C->>C: Validate and choose replicas and leaders
  Note over C: Append topic and partition records to the metadata log
  C-->>B: Topic created
  B-->>A: Success
  C-->>O: Brokers fetch the new records
  Note over O: Each broker updates its local cache

Topic creation is calm. A leader change during a failure is the moment when the metadata cache matters most, and when stale copies do the most damage.

flowchart TD
  A["Leader broker stops"] --> B["Controller notices missed heartbeats"]
  B --> C["Controller writes a partition change record"]
  C --> D["Brokers fetch and apply the record"]
  D --> E["New leader starts serving, others follow it"]
  E --> F["Clients get errors then refresh metadata"]
  F --> G["Clients send traffic to the new leader"]

Suppose Broker 2 leads partition 3 of the orders topic, and it crashes. The controller stops hearing its heartbeats. After the session limit passes, it fences Broker 2 and looks at every partition that Broker 2 led. For partition 3 it picks a new leader from the in-sync replicas, say Broker 1, increases the partition’s leader epoch (a counter of leader changes), and writes a partition change record to the metadata log.

Every broker fetches the record and updates its cache. Broker 1 sees that it is now the leader and begins accepting writes. The other replicas see who the new leader is and start following it. Meanwhile, the producers and consumers that were using Broker 2 discover the problem the hard way: their connections fail or their requests to the wrong broker are refused. They then ask any live broker for fresh metadata and learn that Broker 1 leads partition 3.

The cache is not just something brokers hold. It is consulted at nearly every step of handling a request.

What a metadata answer contains

A response to a Metadata request lists the live brokers with their addresses, then for each requested topic it lists every partition together with the id of its leader, the leader epoch, the replica list, the in-sync replica list, and any replicas that are offline. The cluster identifier is included as well. Since the response is built directly from the broker’s own image, it always reflects exactly what that broker knew at that moment, no more and no less.

To tie everything together, let us watch a single message from the moment a brand-new application starts, and see which cache is used at each step.

An order service starts up for the first time. It has never heard of the cluster, apart from three bootstrap addresses in its configuration. It is about to send its first message to the orders topic.

Clients do not ask a broker about metadata for every message. They keep a private copy and refresh it on a schedule and whenever something goes wrong.

flowchart TD
  S["Client needs the leader of a partition"] --> Q{"Is it in the local cache and still fresh?"}
  Q -->|"Yes"| U["Use it right away"]
  Q -->|"No, or an error occurred"| R["Send a Metadata request to any known broker"]
  R --> V["Update the local cache"]
  V --> T["Retry the original action"]

When a producer sends its first message to a topic, it does not yet know the partitions or leaders. It sends a Metadata request to a bootstrap broker, stores the answer, and sends the message to the correct leader. All later messages to that topic use the stored answer, at no extra network cost. Consumers do the same, and also use the answer to find the coordinator for their consumer group.

The knobs that control freshness

What triggers an early refresh

Consumers use metadata for more than finding partition leaders. Their teamwork, called a consumer group, depends on it too.

Several consumers can share the work of reading a topic by joining a consumer group. Kafka spreads the topic’s partitions among the members so that each partition is read by exactly one member of the group at a time. A special broker, the group coordinator, keeps track of who is in the group, receives their heartbeats, and starts a rebalance whenever the membership or the list of partitions changes.

Finding the coordinator

Which broker is the coordinator for a given group? Kafka answers with a simple rule. The group name is hashed to pick one partition of an internal topic that stores group progress. The broker that leads that partition is the coordinator for that group. A consumer sends a find-coordinator request to any broker, and the broker works out the answer using its metadata image: the partition count of the internal topic and the current leader of the chosen partition.

When metadata changes cause rebalances

Newer group protocols

Recent Kafka versions include a newer consumer group protocol in which the coordinator broker computes assignments on the server side, instead of one client doing it. With that design, the broker relies on its own cached view of topics and partitions when deciding who gets what, and changes in metadata reach the group through the broker’s normal updates rather than through each client’s refresh timer. The core idea stays the same: assignment is only as good as the metadata behind it.

Two copies of the metadata can disagree. Leader epochs give Kafka a simple way to decide which one is newer.

A leader epoch is a number attached to each partition that increases by one every time leadership changes. Every metadata answer includes it, and every produce or fetch request from a modern client includes the epoch the client believes is current. This small number solves two problems at once.

sequenceDiagram
  participant P as Producer with an old cache
  participant Old as Former leader
  participant New as New leader
  P->>Old: Produce to partition 3, leader epoch 7
  Old-->>P: Not the leader, newer leader information attached
  P->>New: Produce again with leader epoch 8
  New-->>P: Success
  Note over P: The client cache now says epoch 8

Ignoring answers that are older than what we have

Clients use epochs in one more way. When a Metadata response arrives, the client compares each partition’s epoch with the one it already stored. If the new answer carries a lower epoch, it came from a broker that is lagging behind, and the client keeps its own newer information instead of going backwards. Without this rule, a client that happened to ask a slow broker could replace a correct cache with an outdated one.

Newer Kafka versions go a step further. When a broker refuses a produce or fetch request because it is not the leader, it can attach the identity and epoch of the leader it knows about within the error response. The client can then use that immediately without a separate Metadata round trip, which shortens recovery after a leader change.

Cached metadata is eventually consistent. Copies may briefly differ, but they always converge. Kafka expresses each kind of disagreement as a specific error code.

Eventual consistency sounds abstract until you see the errors it produces. Every error below is a normal part of life on a busy cluster. A handful per minute during a change is expected. A constant stream is a sign that something needs attention.

The client-side timeout

The most common metadata problem in application logs is a producer failing with a message that the topic was not present in the metadata after a set time, typically 60 seconds. It means the producer asked for metadata, waited up to its blocking limit, and never received a usable answer for that topic. Typical causes are a wrong topic name with auto-creation disabled, a lack of permission to describe the topic, a bootstrap address that leads nowhere, or an entire cluster being unreachable.

Metadata does not only say who leads a partition. It also says how to reach that leader. When the addresses are wrong, everything looks fine at first and then fails.

A broker may be reachable on several networks at once: a private network for other brokers, an internal network for applications in the same data centre, and a public address for outside clients. Kafka models each as a named listener. The setting called listeners tells the broker which addresses and ports to open. The setting called advertised.listeners tells the broker which addresses to announce to clients through the metadata, and these can be different.

How the connection flows

The classic mistake

A team moves Kafka from a laptop to a cloud server. The bootstrap address is the server’s public name, so the client connects and receives metadata. But the broker still advertises its internal machine name, which only makes sense inside the cloud network. The client sees the leader’s address, cannot resolve or reach it, and every send times out. The broker is healthy and the topic exists, yet nothing flows, because the map the client was given points to an address it cannot use.

A Metadata request can do more than ask. Under some settings, asking about a topic that does not exist makes Kafka create it.

Brokers have a setting called auto.create.topics.enable. When it is on, a client that asks for metadata about an unknown topic, or sends a message to one, can cause that topic to be created with default settings. The Metadata request carries a flag saying whether the client allows this. Clients can turn it off with their own setting, so that an accidental typo does not create a stray topic.

What happens behind the scenes

When a broker receives a Metadata request for an unknown topic with auto-creation allowed, it does not create the topic itself. It forwards a creation request to the active controller, just like an administrator’s request, following the steps in Chapter 12. The first answer to the client is usually “leader not available”, because the topic’s partitions do not have leaders yet. The client retries after its backoff, and soon receives real metadata. This explains a familiar pattern in logs: one warning about leader not available on the very first send to a new topic, followed by normal operation.

The best way to feel how the metadata cache behaves is to watch it under pressure. These four stories use plausible numbers to show the mechanics.

Story 1: the reconnection storm

A cluster of six brokers serves 20,000 client connections across many applications. Broker 4 crashes, and it led about 500 partitions. The controller notices after the heartbeat limit of roughly nine seconds, moves the leaders and records the changes. Within a second or two, all brokers have applied the change. But in the meantime, thousands of clients with connections to Broker 4 saw errors and each sent a Metadata request to a surviving broker.

Here the design pays off. Every broker answers from memory, so the flood is spread over five machines and handled quickly. Clients also wait a short backoff before retrying, so their requests do not all arrive in the same instant. Most applications recover in a few seconds beyond the detection time. If clients had no backoff, or if every metadata answer required a call to the controller, the same event could turn into a second, self-made outage.

Story 2: the broker that fell behind

Broker 2 shares a disk with a noisy neighbour, and its disk stalls for twenty seconds. During that time, the controller commits a leader change for a partition that Broker 2 used to lead. Broker 2 cannot apply the record yet, so for a while its cache still says it is the leader. A producer with an old cache sends it a message with the old epoch. Broker 2 has not yet learned the new epoch, so it cannot yet reject the message on epoch grounds alone.

What saves the day is layered protection. First, Broker 2 has stopped sending heartbeats normally, so the controller fences it and moves leadership away. Second, when Broker 2 does catch up and applies the record, it stops accepting writes for that partition. Third, replicas follow the new leader, so an old leader’s late writes cannot be committed by the in-sync replicas. The metadata cache is one line of defence, and the epoch checks in the replication protocol are another.

Story 3: doubling the partitions

An orders topic has 6 partitions and the team needs more throughput, so they raise it to 12. The controller commits the change instantly and all brokers apply it within a second. But producers still hold cached metadata that says 6 partitions, and they keep hashing keys modulo 6 until their next refresh, which can take up to the maximum metadata age of five minutes by default.

Once each producer refreshes, it starts hashing modulo 12, and a customer whose orders used to go to partition 2 may now go to partition 8. Consumers reading partition 2 have older messages for that customer, and consumers of partition 8 have newer ones. Order across the boundary is lost. Consumers subscribed by pattern also notice new partitions only at their own refresh interval. The lesson is to plan partition counts up front, and to treat an increase as an event that can break per-key ordering.

Story 4: the controller failover

The active controller crashes. The remaining controllers hold an election, and one of them becomes active, typically within a few seconds. During this window, no new decisions can be made. Nobody can create a topic, and if a broker fails during the window, its leaders cannot be moved until the new controller takes over.

Yet ordinary produce and consume traffic continues without any interruption, since brokers serve requests from their cached metadata and do not need the controller for the data path. When the new active controller comes up, brokers reconnect to it and continue fetching the log from where they stopped. Because the log is replicated to a majority before any decision counts, the new controller holds every committed decision, so nothing already announced is lost.

The cache is fast because it lives in memory, but it is not free. Its cost grows with the size of the cluster and the behaviour of its clients.

Every partition of every topic has an entry in the cache: its leader, epoch, replica list and ISR. A cluster with a few thousand partitions has a trivial cache. A cluster with hundreds of thousands has one measured in tens or hundreds of megabytes, and every broker holds a full copy. That is still small next to the message data, but it is not nothing, and other costs grow with it.

The all-topics request

A Metadata request can name specific topics or ask for all of them. The second form is convenient and expensive. On a big cluster, the answer lists every partition of every topic. As a rough illustration, if each partition entry takes around a hundred bytes in the response, a cluster with 200,000 partitions would send about twenty megabytes for a single request. If hundreds of monitoring tools, dashboards or pattern-subscribed consumers make that request every few minutes, brokers spend much of their time building and sending enormous answers.

Why the modern design scales better

The older push-based design had each metadata change sent from the controller to every broker, and a new controller had to reload everything from ZooKeeper. The modern log-based design changes both. Changes are small records that brokers pull, and a new active controller already holds the whole log in memory, since standby controllers apply it continuously. Also, applying a delta to an immutable image shares all unchanged parts, so a single change to one partition does not copy the entire cluster picture. One goal of the design was to allow clusters with a very large number of partitions, in the order of millions, with fast controller failover.

Kafka could have chosen a different way of sharing the cluster picture. Comparing the alternatives shows why this one was chosen, and what it costs.

Kafka’s choice gives strong ordering of changes but not instant delivery to every reader. All brokers apply the same records in the same order, so they never disagree about the sequence of events, only about how far along the sequence they have reached. Since a leader epoch accompanies every change, any request can be judged as fresh or stale, and mistakes are corrected without harm.

Where CAP shows up

When the network splits, a system must choose between staying consistent and staying available. For metadata, Kafka chooses consistency for decisions: changes need a majority of the controller quorum, so a minority partition cannot create topics or elect leaders. For reads, it favours availability: brokers keep answering from their last known cache during the split. The result is a cluster that will not make conflicting decisions but will continue serving the existing traffic.

Metadata may look harmless, but a list of topics, partitions and addresses is exactly what an attacker wants to see first.

Topic names often reveal business activity: a topic called merger-negotiations or customer-refunds tells a story. Broker addresses reveal network layout. Access rules define who can do what. Because all of this sits in the metadata cache, it deserves protection just like the messages.

Permission checks come from the cache

In KRaft clusters, access rules are stored in the metadata log and every broker keeps them in memory. This makes checks very fast, but it also means a rule change reaches each broker when that broker applies the record. In the short window before that, a broker may still use the old rule. For most systems this is invisible, but for urgent revocations, such as removing a compromised account, it helps to verify that all brokers have applied the change, and to disable the credential at the identity source too.

A healthy cache is one that is current, quiet and fast. A few metrics tell you whether it is all three.

A simple diagnosis path

Companies running Kafka at scale meet the metadata cache in daily life, mostly through its side effects.

A few habits keep the metadata path healthy. A few shortcuts quietly damage it.

Patterns that work

Anti-patterns to avoid

A short checklist for reviews and a table of mistakes that keep coming back.

Some beliefs about metadata caching sound sensible and are wrong. Here are the most common.

Gotchas worth remembering

Let us apply what we have learned to three different workloads. The method matters more than the specific numbers.

Ask four questions for any system. How quickly must clients recover from a leader change? How many clients and partitions exist? Do clients connect from several networks? And how costly is a mistake in topic setup? The answers point to different choices for the metadata path.

Short answers to questions that come up again and again.

A cache of the cluster’s map, kept in every broker and every client, is what lets Kafka answer “who leads this partition?” in microseconds.

The controller decides, the metadata log records, and each broker turns those records into an in-memory picture that it can read without locks or network calls. Clients copy the parts they need and refresh them on a timer, after errors, or when they meet something new. Leader epochs let every party tell fresh from stale, so the occasional disagreement is corrected quickly and safely. Understanding this chain explains both the speed of Kafka and the handful of surprises that come with it: unknown topics right after creation, unreachable advertised addresses, and keys that move when partitions grow.