Before looking at any bytes, we must understand the real-world problem that versioning solves.

A distributed system never upgrades all at once

A Kafka cluster may hold dozens of brokers. Around it may sit hundreds of producers, consumers, stream processors, connectors, monitoring agents and administrative tools. These pieces are owned by different teams, written in different programming languages, and deployed on different days. It is almost never possible to stop everything and upgrade everything at the same moment.

That means, at any given time, a cluster is likely to be talking to software of several ages. A broker from this year may be serving a client library from three years ago and a command-line tool from last month. Everyone still expects messages to flow.

What could go wrong without versioning

Suppose the Kafka team wants to add a new field to the request that fetches messages, for example a field telling the broker which transaction isolation level the consumer wants. If the client and broker simply assumed the newest format, an old client would send a request without the field, and a new broker would misread the following bytes as if they belonged to that field. The result would be garbage, crashes or silent corruption.

Going the other way is just as dangerous. A new client sending an extra field to an old broker would confuse it, because the old broker has no idea what the field means or how long it is.

The solution in one sentence

Every request type in Kafka carries an explicit version number, every broker advertises the range of versions it understands for each request type, and every client picks a version that both sides support before doing real work. That simple agreement, repeated for every request type, is the heart of Kafka’s compatibility story.

Kafka does not use a web-style protocol. It has its own compact, binary request and response conversation over ordinary network connections.

A custom binary protocol over TCP

Kafka clients open a normal network connection to a broker, typically using the TCP transport, optionally wrapped in TLS encryption. Over this connection they exchange messages in a purpose-built binary format. It is not text, it is not JSON, and it is not HTTP. Binary formats are much smaller and much faster to parse than text formats, which matters when a broker handles hundreds of thousands of requests each second.

The request and response rhythm

The conversation follows a strict pattern. The client sends a request. The broker sends back exactly one response for that request. There is no unsolicited chatter from the broker. Requests on one connection are answered in the order they were received, which keeps the protocol simple and predictable.

A client may send several requests without waiting for earlier answers. This is called pipelining. It improves throughput because the network does not sit idle while waiting for each individual answer.

Size-prefixed framing

Every request and every response begins with a number that says how many bytes follow. This is like putting a note on the front of each parcel that says “this parcel weighs 300 grams”. The receiver reads the number, then reads exactly that many bytes, and knows the parcel is complete. Without this, a reader would not know where one message ends and the next begins on a continuous stream of bytes.

Why the version sits in the header

The receiver must know how to read the body before reading it. Since the body’s layout depends on the version, the version is placed in the header, which itself has a small, stable shape. The receiver reads the header first, looks up the request type and version, chooses the matching layout, and then parses the body accordingly.

Every kind of request in Kafka has a small numeric identifier called an API key.

Think of a restaurant where each dish on the menu has a number. Instead of saying “one grilled chicken with rice and vegetables”, a customer can say “number twelve”. The kitchen knows exactly what that means. Kafka does the same: the first meaningful item in a request header is the API key, a small number that tells the broker what the client wants.

Internal APIs

Not all requests come from clients. Brokers and controllers also talk to each other with their own requests. Replica followers use fetch requests to copy data. Controllers exchange requests to elect leaders, register brokers and share metadata. Some of these APIs are marked for internal use only, and their availability to ordinary clients may be restricted. They still use the same version mechanism.

API keys never change meaning

Once a number is assigned to a request type, it belongs to that type forever. If Kafka removes a request type, its number is retired and not reused for something else. This stability is important, because an old client using API key one must always mean the same request as it did years ago, even if the broker has a completely different set of newer request types.

The header is the small label on every message. It, too, has versions.

What the request header carries

Header versions

The header format itself has evolved. The earliest header held only the four fields above. A later header version, introduced together with the flexible encoding described in a later chapter, added a section at the end for optional tagged fields. So there are several header versions, and the header version used for a request is decided by the request’s own version.

The special case of ApiVersions

There is one deliberate exception. The response to the ApiVersions request always uses the oldest, simplest response header, even when the request itself uses a newer version. The reason is practical: the ApiVersions exchange is the very first conversation, so the client cannot yet know what the broker supports. Using the simplest header for this one response means even a very old client can read at least the essential part of the answer.

Versions in Kafka are not global. Each request type keeps its own counter.

Independent counters

Imagine a large company where every department publishes its own policy handbook. The finance handbook may be on edition nine, the travel handbook on edition three, and the security handbook on edition twelve. There is no single edition number for the whole company, because each department changes its rules at its own pace.

Kafka works the same way. The fetch request might be at a high version because it has been refined many times, while a rarely used administrative request might still be on its first or second version. Each version number is meaningful only together with its API key.

Why not one number for everything

What a version range means

For each API key, a broker supports a contiguous range of versions, from a minimum to a maximum. For example, a broker might support versions three through nine of one request. A client that only understands versions zero through two cannot use that request with this broker. A client that understands versions five through twelve can use versions five through nine.

The very first request on a fresh connection is usually a polite question: what can you do?

The conversation step by step

sequenceDiagram
    participant C as Client
    participant B as Broker
    C->>B: Open connection
    C->>B: ApiVersions request
    B-->>C: List of API keys with min and max versions
    C->>C: Choose highest shared version per API key
    C->>B: Metadata request at chosen version
    B-->>C: Metadata response
    C->>B: Produce request at chosen version
    B-->>C: Produce response
        

What the response contains

The ApiVersions response is essentially a table. Each row says, for one API key, the lowest and highest version this broker supports. In newer versions of the response, the table also includes information about broker features, such as the cluster-wide metadata version and other finalized feature levels. We cover those later.

Software name and version information

A later version of the ApiVersions request lets the client tell the broker the name and version of its client software, for example “the standard Java client, release such-and-such”. Brokers can expose this in metrics and logs. This is very valuable in real operations, because it lets administrators ask “which applications are still using an ancient client library?” and plan upgrades with evidence.

The negotiation has a clever trick for the case where even the ApiVersions request itself is too new for the broker.

The awkward first message

A client always begins with ApiVersions. But suppose the client is very modern and sends the newest version of ApiVersions to a very old broker that has never heard of that version. The old broker cannot parse the request. If it simply dropped the connection, the client would have no clue why.

The graceful fallback

Kafka handles this with a special rule. When a broker receives an ApiVersions request whose version it does not support, it replies using the oldest response format, with an error code meaning “unsupported version”, and it still includes its table of supported versions in that reply. The client reads the error, looks at the table, sees the highest ApiVersions version the broker does support, and simply asks again using that older version. This is why the ApiVersions response always uses the oldest header: even the most ancient broker can produce it.

sequenceDiagram
    participant C as New client
    participant B as Old broker
    C->>B: ApiVersions at newest version
    B-->>C: Error unsupported version, plus supported table in oldest format
    C->>C: Note broker maximum for ApiVersions
    C->>B: ApiVersions again at older version
    B-->>C: Full list of supported API versions
    C->>B: Normal requests at negotiated versions
        

Older brokers with no ApiVersions at all

Very early Kafka brokers, from before the negotiation mechanism was invented, do not understand the ApiVersions request at all. Modern clients that want to support such ancient brokers traditionally need a fallback: either a manual setting telling the client which broker release to assume, or a guess based on trial and error. Some client libraries offered exactly such a “broker version fallback” setting for this reason. Today this situation is rare, and newer Kafka releases have deliberately dropped support for the oldest protocol versions, as we discuss later.

What happens to unsupported requests in general

If a client sends some other request at a version the broker does not support, the broker will typically return an error indicating the version is unsupported. Depending on the type of request and the broker’s design, it may also close the connection. Well-written clients avoid this entirely because they only choose versions from the negotiated overlap.

A major redesign of the encoding made future changes cheaper and safer.

The old problem: every change needs a new version

In the classic encoding, adding even a small optional field meant creating a whole new version of the request, and every client and broker had to know about it. The version numbers rose quickly, and old brokers that did not know a new field could not tolerate it at all.

The idea: optional extras that can be skipped

A Kafka improvement proposal introduced flexible versions. In a flexible version, every message ends with a section of tagged fields. A tagged field has a numeric tag, a size and its data. A reader that recognizes the tag reads it. A reader that does not recognize the tag knows exactly how many bytes to skip, so it can safely ignore the field and continue with everything else. This turns many changes into harmless, ignorable additions.

Compact encodings

Flexible versions also switched to more compact ways of writing lengths. Instead of writing every string and array length in a fixed number of bytes, they use variable-length integers, which take a single byte for small values. This saves space when there are many small strings and arrays, which is very common in Kafka requests that list many topics and partitions.

Where tagged fields help most

The records inside a produce or fetch request have their own format version, separate from the request version.

Two layers of versioning

Picture a shipping container filled with boxes. The container has a standard shape so that cranes and trucks can handle it. The boxes inside have their own packing layout. Kafka has this same double layer. The request is the container, and the records are the boxes. The request version says how the container is laid out. The message format version says how the boxes inside are packed.

The historical generations

The number that identifies this generation is stored in a small field within each batch, historically called the magic byte. A reader looks at it first and then knows exactly how to interpret the rest.

Down-conversion for older consumers

If an old consumer asks for messages using an old version of the fetch request that only understands an older message format, the broker may need to convert stored messages to that older layout on the fly. This is called down-conversion. It costs CPU and memory, and it prevents the broker from using its most efficient path for sending data straight from disk to network, called zero-copy. A cluster full of old consumers can therefore be noticeably slower than one full of modern ones.

Changing the protocol in Kafka is a deliberate, public and carefully reviewed process.

The improvement proposal process

Kafka changes that affect the protocol, public interfaces or compatibility are proposed in a written document called a Kafka Improvement Proposal, usually shortened to KIP. The document describes the problem, the proposed new fields or requests, how old and new versions will coexist, and how upgrade and downgrade will behave. The community reviews it, discusses it openly and votes before the change is accepted.

Schemas as the single source of truth

Each request and response type is defined in a small structured schema. For every field, the schema states the versions in which the field exists, whether it may be empty in certain versions, and whether it is an optional tagged field. The client and broker code that reads and writes the bytes is generated from these schemas. This keeps every version consistent and lets developers see, at a glance, the entire history of an API.

Freedom to evolve depends on a few strict rules that Kafka developers follow.

Backward and forward compatibility

Backward compatibility means new software can still work with old partners. Forward compatibility means old software can tolerate the presence of newer partners. Kafka aims for both at the level of clients and brokers within supported ranges: a new client should be able to talk to a reasonably old broker, and an old client should be able to talk to a newer broker, as long as the versions they share still exist.

The rules of the road

Examples of changes and their consequences

The negotiation mechanism opened the door to a very friendly policy: new clients and old brokers can work together, and so can old clients and new brokers.

The historical turning point

In the early days of Kafka, a client library could only talk to brokers of the same age or newer. Upgrading a client required upgrading the brokers first. Once the ApiVersions request was introduced and client libraries learned to use it, a new client could look at what an older broker supported and speak accordingly. Newer clients could now talk to a broad range of older brokers, and older clients continued to talk to newer brokers, as newer brokers kept the older versions available.

flowchart LR
    OC["Older client library"] -->|"uses older shared versions"| NB["Newer broker"]
    NC["Newer client library"] -->|"uses older shared versions"| OB["Older broker"]
    NC -->|"uses newest shared versions"| NB
    OC -->|"uses its own versions"| OB
        

What this policy gives you

Limits of compatibility

Compatibility is broad but not infinite. New features require new request versions, so an old broker cannot support a feature that the client needs a newer version to express. For example, if a client wants a feature that only exists in a later version of some request, and the broker’s range stops before that version, the client must either do without the feature or fail with a clear error. The most modern behaviors are only available when both sides are modern enough.

Supporting every version forever would become a heavy burden. Kafka retires old versions carefully.

The cost of keeping old versions alive

Every supported version is code that must be maintained, tested and secured. Over many years, the number of supported combinations grows very large. Developers must reason about behaviors from many eras, and every new feature must also remain compatible with all the historic paths. This slows development and can hide bugs.

Announced, staged removal

A real example of a floor being raised

The Kafka 4.0 release, which also removed ZooKeeper, took a big step here. It dropped support for many of the oldest protocol versions of the main request types, raising the minimum client version that can talk to the broker. The exact minimum differs per request type, but the practical message was clear: clients from many years ago need to be upgraded before, or together with, moving to the newest brokers. Anyone planning such an upgrade should check the official upgrade guide for their exact target release.

Protocol versions tell you what a single broker can understand. Feature flags tell you what the whole cluster has agreed to use.

The gap that versions alone leave

During a rolling upgrade, some brokers run new software while others still run old software. A new broker may be capable of a new behavior, but if it started using it immediately, the still-old brokers could not cooperate. The cluster needs a way to say “everyone has been upgraded, now we may all start using the new behavior together.”

Supported versus finalized

Kafka distinguishes two ideas. A supported feature range describes what one broker’s software is capable of. A finalized feature level describes what the cluster has collectively decided to use right now. Brokers advertise the supported range when they register. Operators then raise the finalized level, and the cluster refuses to raise it beyond what every member supports. The finalized information is stored in the cluster metadata and is also returned by the ApiVersions response, so clients can see it.

The cluster-wide metadata version

The most important finalized feature is the cluster metadata version. In modern KRaft-based clusters, it controls the format of internal metadata records and which cluster behaviors are enabled. It has taken over much of the role that older broker-to-broker protocol settings used to play in ZooKeeper-based clusters, where an operator would edit a configuration setting on each broker in stages.

Clients are not the only speakers. The servers of the cluster talk to each other using the same versioning idea.

Replication uses ordinary requests

When a follower replica copies data from a partition leader, it does so by sending fetch requests, just like a consumer, with a few extra details identifying it as a replica. The follower and the leader must therefore agree on a fetch version, and the same negotiation ideas apply. A cluster with brokers of different ages during an upgrade relies on this to keep replication healthy.

Controller communication

Brokers register with the active controller, send heartbeats and report leadership changes using dedicated request types. Controllers talk to each other to elect a leader and replicate the metadata log. All of these are ordinary versioned APIs with their own key numbers and version ranges, and the same rules of compatibility and deprecation apply to them.

Why internal upgrades need extra care

If a client fails to understand a broker, one application suffers. If a broker fails to understand another broker, replication or coordination may fail for many partitions at once. That is why internal protocol changes are guarded by the feature-finalization mechanism from the previous chapter. New internal behaviors are only switched on after every member of the cluster has been upgraded and the operator has explicitly finalized the new level.

Let us follow one realistic connection from start to finish to see every idea working together.

An application team runs a consumer built with a client library from a few years ago. The platform team has just upgraded the cluster to a recent release. Nothing has been changed in the application. Will it still work? Let us watch.

sequenceDiagram
    participant C as Older consumer
    participant B as Recent broker
    C->>B: ApiVersions
    B-->>C: Supported ranges for every API
    C->>C: Select own highest version per API
    C->>B: Metadata at chosen version
    B-->>C: Partition leaders
    C->>B: JoinGroup and SyncGroup at chosen versions
    B-->>C: Assignment
    C->>B: Fetch at older version
    B->>B: Down-convert if needed
    B-->>C: Messages in older layout
        

What the team should conclude

The application works, but it is leaving benefits on the table and may be causing extra work for the brokers through down-conversion. That is a good reason to schedule an upgrade of the client library during normal maintenance, without any rush or outage. Compatibility gives you the time to do so calmly.

Most version problems show up as a small family of recognizable symptoms.

Protocol versioning touches security in several quiet but important ways.

Authentication has its own versioned requests

Security mechanisms such as SASL use dedicated requests: one to choose a mechanism and one to carry the authentication exchange. These requests have versions, and the way the exchange is wrapped depends on the version. Older versions sent the authentication tokens in a raw form outside of normal request framing, while newer versions wrap them as ordinary requests and responses. Clients and brokers must agree on this, which is one more reason the ApiVersions exchange happens before authentication.

Practical protective measures

You can only manage version diversity if you can see it.

Questions worth answering with metrics

Logs that help

Broker logs record connection problems, unsupported requests and authentication failures. Client logs commonly print the versions negotiated when connecting, or provide a debug setting that does. When investigating, collect the client’s negotiated versions and the broker’s supported ranges side by side, and the mismatch usually becomes obvious.

Kafka ships tools that let you look at versions directly, with no guesswork.

Practical uses

The official protocol documentation is full of tables. Learning to read them turns them from a wall of text into a quick answer machine.

Anatomy of an API entry

For every request type, the documentation shows the API key, the name, a description of what it does and the list of versions. Under each version you find the exact fields of the request and of the response, in order. Some fields appear only in later versions. Some are marked as optional, and some are described as nullable, meaning they may hold an explicit “nothing” value in certain versions.

A method for answering “can I use this feature”

Many familiar Kafka features exist only because a particular request grew a new version.

It is instructive to see how well-known capabilities map onto protocol evolution. The exact version numbers differ between requests and are best confirmed in the documentation, but the pattern is consistent: a capability arrives with a new request version or a new request type, and older versions simply cannot express it.

What to notice in this list

Consumer groups add a second layer of agreement: the members of a group must also agree with each other.

Two kinds of compatibility in one system

So far we have looked at client-to-broker compatibility. Consumer groups add another dimension. The members of a group must agree on how partitions are divided among them. In the classic design, the members themselves run the assignment logic, and they announce which assignment strategies they support when they join. Because the group may contain members running different library versions, a common strategy must be chosen that every member understands.

The classic rebalance conversation

The newer group protocol

Recent Kafka releases introduced a redesigned consumer group protocol in which the broker side takes over computing assignments, and members communicate through a dedicated heartbeat request. This reduces the disruption of rebalances and simplifies client logic. It is enabled through a combination of broker capability, cluster feature level and client configuration, so adopting it is a good example of how several kinds of versions must line up before a new behavior is used.

Kafka has no single official client for every language. Each library implements the protocol on its own schedule.

A diverse ecosystem

The Java client is maintained together with the Kafka broker and usually receives new protocol versions first. Libraries for other languages are built by other communities and companies. Some are thin wrappers around a shared native library, others are entirely separate implementations. Each decides how quickly to adopt new request versions and how much of the protocol to implement at all.

Version fallback settings

Because some old brokers could not answer the ApiVersions request, certain libraries historically offered settings such as a switch to disable the request and a fallback broker version to assume in its place. In today’s environment, where nearly all brokers support the request, these settings should normally be left alone, and depending on them is a sign of an outdated deployment.

Compatibility that is never tested is only a hope. A little structure turns hope into evidence.

Build a compatibility matrix

List the client libraries and versions that matter in your organization on one axis and the broker releases you run or plan to run on the other. Each cell answers a question: does a representative application using this library work correctly against this broker release, including the features it relies on? Even a small matrix, filled in honestly, reveals risks that no amount of reading can.

What to test in each cell

Everything in this tutorial exists so that upgrades can be done calmly, one piece at a time.

flowchart TB
    A["Read upgrade notes"] --> B["Inventory client libraries"]
    B --> C{"Any client below new floor?"}
    C -->|"yes"| D["Upgrade those clients first"]
    C -->|"no"| E["Rehearse in staging"]
    D --> E
    E --> F["Roll controllers then brokers"]
    F --> G["Observe through a full cycle"]
    G --> H["Finalize feature level"]
    H --> I["Clients upgrade at their own pace"]
        

Kafka’s versioning approach has made it easy to evolve, but it is not free of cost.

Comparison with other approaches

These habits keep upgrades boring, which is exactly what you want.

Each of these looks harmless until an upgrade day arrives.

Protocol versioning shows up in daily practice across the Kafka ecosystem.

A few persistent misunderstandings cause a surprising number of upgrade mistakes.

No. Every request type has its own version numbering. A single connection can carry requests of many types at many different versions.

The client chooses. The broker advertises the range it supports for each request type, and the client picks the highest version that both sides understand.

Because it is the first exchange on a connection, before anything is known about the peer. Using the simplest header lets even very old clients read at least the essential part, including an error and the supported range.

The broker replies in the oldest response format with an unsupported-version error and includes its supported table, so the client can retry with a version the broker understands.

The request version describes the layout of the request and response bodies. The message format version describes the layout of the records inside a batch. They evolve separately.

It is the broker converting stored messages into an older layout for an old consumer. It costs CPU and memory and stops the broker from using its most efficient data path, so many old consumers can noticeably reduce broker performance.

They are optional fields at the end of a flexible-version message, each with a tag number and a size. A receiver that does not recognize a tag can skip it safely, which makes many changes possible without a new version.

Usually yes, as long as the old versions it uses are still supported. Major releases can remove very old versions, in which case very old clients must be upgraded first.

Generally yes within a broad range, using the highest versions the old broker supports. Features that need newer request versions will not be available.

A request version range describes what a piece of software can read. A finalized feature level describes what the whole cluster has agreed to use. Operators raise the latter deliberately, after all members are upgraded.

Recent brokers record the client software name and version reported during the handshake and expose it in metrics and logs. Use that to build an inventory.

Because the highest shared version of the relevant request may differ. One cluster may offer the version that carries the feature and the other may not.

Rarely. The protocol has many special cases, such as the header rule for ApiVersions and fallback behavior, and official libraries have already solved them. If you must, follow the public protocol documentation closely and test against many broker versions.

Not directly. It is a cluster-level decision about internal behavior. Clients continue to negotiate request versions with brokers as before, though new cluster capabilities may later show up as newer version ranges or features.

Measure client library versions continuously and upgrade them steadily, so no application is ever far behind the versions your brokers support.

No. The broker only publishes what it accepts. Every decision about which version to send is taken inside the client library, which is why a buggy or outdated library is a common source of version errors.

Kafka reuses one well-tested request for both jobs, adding a few identifying details for replicas. This means replication between brokers of different ages relies on the same negotiation rules that clients use, which keeps rolling upgrades of the cluster safe.

Yes. Every request header carries the version, so packet captures show it directly, and broker request metrics can be grouped by request type and version. Client debug logging often prints the negotiated versions at connection time.

A transparent network proxy that only forwards bytes is fine. A proxy that understands Kafka requests must itself support the versions in use, otherwise it may reject or damage newer requests. Always check the proxy’s supported range.

Generally newer versions carry more capability and better efficiency, but they are not automatically better for every application. The right approach is to let the client choose the highest shared version, and to test that the resulting behavior matches your expectations.

Because each version is frozen once released, even a tiny change becomes a new version. The many small steps are a deliberate price paid for never altering the meaning of something that peers already depend on.

The release notes and the upgrade guide for every release between your current and target version, with special attention to removed protocol versions, changed defaults and any required intermediate release.

Kafka’s protocol versioning is a quiet piece of engineering that allows an enormous, mixed-age fleet of clients and brokers to work together. Every request type carries its own version, every broker publishes the ranges it accepts, and every client chooses the highest version both sides understand, starting with a special handshake that even the oldest software can begin. Flexible encodings, careful compatibility rules, generated code, feature flags and a disciplined deprecation process complete the picture. Understanding these pieces lets you plan upgrades with confidence, diagnose odd compatibility errors quickly and keep your platform healthy for years.