The trouble starts because producers and consumers never change at the exact same moment.

Two teams, two calendars

Imagine the Orders team writes a message every time a customer buys something. The Billing team, the Shipping team, and the Analytics team all read those messages. Each team has its own schedule. The Orders team wants to add a “coupon code” field this week. Billing can upgrade next week. Analytics will only upgrade next month.

For a while, some programs will speak the new message shape and others will still understand only the old one. If nothing protects them, someone will crash. Compatibility rules are that protection.

Kafka never forgets quickly

A normal message queue hands over a message and forgets it. Kafka is different. It keeps messages on disk for days, weeks, or even forever, depending on the topic. That means a consumer can start reading today and find messages written a year ago in a shape that no longer exists in your code base. Some consumers deliberately replay a whole topic from the beginning to rebuild a report or a table.

So the question is not only “can today’s program read tomorrow’s message?” It is also “can tomorrow’s program read yesterday’s message?” Both directions matter, and that is exactly why we need two different words, backward and forward.

What breaks when compatibility is ignored

A handful of words unlock everything else, so let us define each one plainly.

Message, key, and value

A Kafka message, also called a record, has a key, a value, and some extra details. The value is the main content, such as an order with a customer, an amount, and a time. Kafka itself does not look inside the value. To Kafka it is just a bundle of bytes.

Schema

A schema is a written description of what a message value looks like: the names of its fields, the type of each field (number, text, true or false), and which fields are optional. It is like a form template that says “Name: text, Age: number, Nickname: optional text”.

Serializer and deserializer

Computers store bytes, not forms. A serializer turns a structured object into bytes before it is sent. A deserializer turns bytes back into a structured object after it is read. Popular formats for this job are Avro, Protobuf, and JSON Schema. All three are compact, and all three come with a schema.

Writer schema and reader schema

These two names are the heart of compatibility. The writer schema is the schema that was used when the message was written. The reader schema is the schema the consumer’s program expects when it reads. They may or may not be the same.

Schema Registry

Sending the whole schema with every message would waste space. Instead, teams use a small service called a schema registry. It stores every schema and gives each one a short number, called a schema ID. Each message carries only that small ID. The most widely used registry is Confluent Schema Registry, and other options include Apicurio Registry and AWS Glue Schema Registry.

Subject and version

The registry groups schemas by subject, which is normally the topic name plus “value”. For a topic called orders, the subject is orders-value. Every time a schema for that subject changes, the registry stores it as a new version: version 1, version 2, version 3, and so on. Compatibility rules are checked between these versions.

flowchart LR
    P["Producer with writer schema"] -->|"1. register schema"| R["Schema Registry"]
    R -->|"2. returns schema ID 7"| P
    P -->|"3. message = ID 7 + data bytes"| K["Kafka topic"]
    K -->|"4. message = ID 7 + data bytes"| C["Consumer with reader schema"]
    C -->|"5. fetch schema for ID 7"| R
    

The picture shows the journey. The producer registers its schema once and receives an ID. Every message then holds that ID followed by the data bytes. The consumer sees the ID, asks the registry which schema it stands for, and now knows exactly how the writer laid out the data. Both sides also cache this answer, so the registry is not asked again for every single message.

Here is the one idea that clears up most of the confusion.

The words backward and forward describe the direction in time of the data that a program can read. Always ask two questions: “Who is reading?” and “Whose data is it?”

flowchart TD
    subgraph BW["Backward compatible"]
        A1["Old data written with version 1"] --> B1["Reader using version 2 can read it"]
    end
    subgraph FW["Forward compatible"]
        A2["New data written with version 2"] --> B2["Reader using version 1 can read it"]
    end
    subgraph FU["Full compatible"]
        A3["Data from any neighbouring version"] --> B3["Reader using any neighbouring version can read it"]
    end
    

Read the diagram from the top. In the backward case, the data is old and the reader is new. In the forward case, the data is new and the reader is old. In the full case, it works no matter which is which. Notice that the word describes the reader’s ability, looking back to old data or looking forward to new data.

The default in Kafka

In Confluent Schema Registry, the default rule is backward. Whenever someone tries to register a new version of a schema, the registry checks: “Can a program using this new schema read data written with the previous schema?” If not, the registration is refused, and the developer sees an error before any message is sent.

The most common rule, and the safest starting point for most teams.

The promise

Backward compatibility means that a consumer using the new schema can correctly read data written with the previous schema. In other words, upgrading the reader never leaves it unable to handle older messages.

Changes that keep it safe

Let us use a small order message with two fields: orderId and amount. This is version 1.

• Add a field that has a default value. Version 2 adds couponCode, with a default of “none”. When the new reader meets an old message that has no coupon, it fills in the default. Everything works.
• Delete a field. Version 2 drops amount. When the new reader meets an old message that still contains an amount, it simply ignores the extra field. Everything works.

Changes that break it

• Add a field with no default. The new reader expects couponCode but old messages do not have it, and there is nothing to fall back on. The read fails.
• Change a field’s type in an unsupported way, such as text to number. The new reader cannot understand old text values.

Who must upgrade first?

With backward compatibility, consumers upgrade first, then producers. Why? A consumer with the new schema can read both old and new data. A consumer still on the old schema might not be able to read the new data. So you make every reader ready before anyone starts writing in the new shape.

flowchart LR
    S1["Step 1: consumers upgrade to version 2"] --> S2["Step 2: producers switch to version 2"]
    S2 --> S3["Topic now holds version 1 and version 2 messages"]
    S3 --> S4["All version 2 readers can read both"]
    

Why it is the default

Most organizations have many consumers and only a few producers per topic, and consumers often need to replay history. Guaranteeing that new readers can read old data fits that reality. It also lets you rewind a consumer to the beginning of a topic after an upgrade and still get correct results.

The mirror image of backward: this time the old readers must survive new data.

The promise

Forward compatibility means that a consumer still using the previous schema can correctly read data written with the new schema. Producers can move ahead, and old readers keep working without any change.

Changes that keep it safe

• Add a new field, with or without a default. Old readers do not know about it, so they simply ignore it. Nothing breaks.
• Delete a field that had a default. An old reader that still expects the field will find it missing in new data, and will use its default. Nothing breaks.

Changes that break it

• Delete a field that had no default. The old reader expects the field, finds nothing, and has no fallback.
• Change a type in a way the old reader cannot understand.

Who must upgrade first?

With forward compatibility, producers upgrade first, then consumers. Producers can start writing new messages immediately, because old consumers can still read them. Consumers can catch up later, whenever it suits each team.

flowchart LR
    S1["Step 1: producers switch to version 2"] --> S2["Old consumers on version 1 keep reading"]
    S2 --> S3["Step 2: consumers upgrade to version 2 later"]
    

When teams choose forward

Forward is popular when you cannot control the consumers. Suppose your topic is read by twenty teams across the company, or by outside partners. You cannot force them to upgrade before you ship. With forward compatibility, you ship, and they follow at their own pace.

The strongest promise: both directions work at the same time.

The promise

Full compatibility means a schema is both backward and forward compatible with the previous version. New readers can read old data, and old readers can read new data. Neither side is forced to move first.

What you are allowed to do

Because full compatibility must satisfy both rules, only the changes that are safe in both directions are allowed:

• Add a field that has a default value.
• Delete a field that has a default value.

Everything else, such as adding a mandatory field or deleting a mandatory field, would break at least one direction and is refused.

Upgrade order

With full compatibility, the upgrade order does not matter. Producers and consumers can upgrade in any sequence, on any day. This is very attractive for large organizations with many independent teams.

There is also a mode with no rules at all, and a handy table to compare everything.

The NONE mode

With NONE, the registry accepts any new version without checking. This gives total freedom, but zero safety. Teams use it for early experiments, private topics with a single consumer, or during a planned migration where everything will be replaced at once. In shared production topics, it is risky, because a careless change can break every consumer.

The seven modes at a glance

Three of these modes end with the word transitive. The next chapters explain what that word adds, and it matters more than most beginners expect.

A rule that only looks at the last step can still lead you off a cliff over several steps.

What “transitive” means

The plain modes (backward, forward, full) compare a new schema only with the latest registered version. The transitive modes compare it with every earlier version. The word comes from maths: if A is compatible with B, and B is compatible with C, does that always mean A is compatible with C? Not necessarily, and the difference can hurt.

A small example that shows the trap

Our order message evolves over three versions, and we use the plain BACKWARD mode.

Now imagine a consumer on V3 that rewinds a topic to the beginning. It meets a V1 message that has no note field. V3 requires a note and has no default. The read fails. The plain backward rule looked only at V2, so it never noticed this problem with V1.

flowchart LR
    V1["Version 1: orderId"] --> V2["Version 2: orderId + note with default"]
    V2 --> V3["Version 3: orderId + note without default"]
    V3 -. "non-transitive check passes against version 2" .-> OK["Accepted"]
    V3 -. "transitive check fails against version 1" .-> BAD["Rejected"]
    

The diagram shows one change, V3, being judged in two ways. A non-transitive check looks only at V2 and accepts it. A transitive check also looks at V1, and rejects it. The transitive rule protects consumers that read very old data.

When you must use transitive modes

• Long retention. Topics that keep data for months or forever hold messages from many versions.
• Replays and rebuilds. Consumers that reprocess from the beginning will meet the oldest versions.
• Compacted topics. A compacted topic can keep an old message for a key that never changed, for years.
• Data lake sinks. Jobs that read historical files written across many schema versions.

Upgrade order for transitive modes

The order is the same as the non-transitive twin. BACKWARD_TRANSITIVE still means consumers first. FORWARD_TRANSITIVE still means producers first. FULL_TRANSITIVE still means any order. Only the strength of the guarantee grows.

This table answers most day-to-day questions when you edit a schema.

The table below uses Avro-style rules, which Confluent Schema Registry applies for Avro schemas. A tick means the change is allowed under that mode, and a cross means the registry will refuse it.

Understanding each row

The role of defaults

A default value is the single most powerful tool for safe change. It gives a reader a fallback when the field is missing. Almost every “Yes” in the table above depends on a default existing somewhere, either in the new schema or the old one. If you remember one rule, remember this: give new fields defaults, and never remove a field that has no default unless every reader has already upgraded.

How to rename safely

The safe path is a two-step move rather than a single rename. First, add the new field with a default, and keep the old field. Fill both while consumers migrate. After every consumer has switched to the new name, remove the old field, which is safe once nobody reads it. Avro also has an alias feature that can help a reader recognize an old name, but treat it as an advanced tool and test it carefully.

A compatibility mode is really a promise about who can safely move first.

A rollout in practice

Rolling deployments blur the order

In real life, applications rarely switch in an instant. A rolling deployment replaces instances one at a time, so for several minutes a group of programs is mixed, some on the old version and some on the new. During that window, a producer on the new schema and a consumer on the old schema can exist together, and so can the reverse. This is where full compatibility shines: it stays safe even in the messy middle.

flowchart LR
    subgraph Mixed["During a rolling deployment"]
        P1["Producer instance on version 1"]
        P2["Producer instance on version 2"]
        C1["Consumer instance on version 1"]
        C2["Consumer instance on version 2"]
    end
    P1 --> T["Kafka topic"]
    P2 --> T
    T --> C1
    T --> C2
    

The picture shows four kinds of programs sharing one topic. Old and new producers both write. Old and new consumers both read. Every combination of writer and reader must work, at least briefly. Only a schema that is compatible in both directions handles all four cases without special ordering.

Let us look at every moving part involved in keeping messages readable.

flowchart TD
    DEV["Developer and CI pipeline"] -->|"test compatibility then register"| API["Schema Registry REST API"]
    subgraph SR["Schema Registry Cluster"]
        LB["Load balancer"]
        PRI["Primary node accepts writes"]
        SEC["Secondary nodes serve reads"]
        LB --> PRI
        LB --> SEC
    end
    API --> LB
    PRI --> ST[("Kafka topic _schemas compacted")]
    SEC --> ST
    PROD["Producer with serializer"] -->|"register or look up schema"| LB
    PROD -->|"messages with schema ID"| BRK["Kafka brokers"]
    BRK -->|"messages with schema ID"| CONS["Consumer with deserializer"]
    CONS -->|"fetch schema by ID"| LB
    

The developer’s pipeline talks to the registry’s REST interface. The registry runs as a small cluster behind a load balancer: one primary node accepts new schemas, and secondary nodes answer reads. All schemas are stored in a special Kafka topic named _schemas, so the registry inherits Kafka’s durability. Producers and consumers talk to the registry only for schema lookups. The heavy traffic, the messages themselves, flows through the brokers.

Subject naming strategies

The registry needs a rule for choosing the subject name. There are three common ones.

The choice matters for compatibility, because the rule is applied per subject. With the default strategy, all messages in one topic must evolve in a compatible chain. With record-name strategies, each record type evolves on its own chain, which allows mixed-event topics, such as an event-sourcing log.

Follow one schema change and one message through the whole system.

Part one: registering a new version

Part two: one message’s journey

Why the registry check matters so much

The registry check is a prediction: it says “if these two schemas ever meet, reading will work”. The read-time resolution is the reality. If you bypass the registry, for example by disabling checks or sneaking in messages with an unregistered format, the prediction disappears and reality can surprise you. This is why teams treat the registry as a gate that every schema must pass through.

Schema caching and speed

Producers and consumers hold schemas in a local cache, keyed by schema ID. After the first message of each type, no network call to the registry is needed. This makes the registry a low-traffic service, even when the topic carries hundreds of thousands of messages each second.

A schema is born, grows, gets old, and eventually retires, just like software.

flowchart LR
    D["Draft in source control"] --> R["Review by consuming teams"]
    R --> T["Automated compatibility test"]
    T --> G["Registered as new version"]
    G --> U["In use by producers and consumers"]
    U --> DP["Older versions deprecated"]
    DP --> RT["Retired once no data or readers remain"]
    

The path starts with a draft in source control, so that changes are visible and reviewed. An automatic test catches incompatible changes. The schema is then registered and used. When the data written with an old version has been read by everyone, or expired, that version can be marked deprecated and finally retired.

Soft delete and hard delete

The registry lets you delete a schema version in two stages. A soft delete hides the version but keeps it recoverable. A hard delete removes it permanently. Deleting versions that still have live data in a topic is dangerous, because consumers will be unable to find the writer schema for those messages and cannot decode them.

Breaking changes: the escape hatches

Sometimes the business truly needs a change the rules refuse, such as turning a text field into a number. There are three honest approaches:

The three big rules stay the same, but each format has its own personality.

Avro

Avro matches fields by name. Data on the wire holds no field labels, so the reader must have the writer schema to decode it. This is why the registry is essential with Avro. Avro’s rules around defaults, aliases, and type promotion are well defined, and they form the basis of the tables in this article.

Protobuf

Protobuf matches fields by number, not by name. Each field has a permanent number, and unknown numbers are skipped by readers. This makes many additions naturally safe. The golden rules are: never reuse a field number, never change the type of an existing number in an incompatible way, and reserve numbers of deleted fields so nobody reuses them later. Protobuf treats missing fields as having a zero-like default value, so the idea of “no default” works differently from Avro.

JSON Schema

JSON Schema is the most flexible and the trickiest for compatibility. Whether a change is safe depends on whether the schema is “open” (extra properties are allowed) or “closed” (extra properties are forbidden). A closed schema makes adding properties a breaking change for old readers. Teams using JSON Schema often need extra care and thorough testing.

Some Kafka tools read data in ways that make the choice of rule especially important.

Replaying a topic

Many teams rewind a consumer to the start of a topic to rebuild a report, to fix a bug, or to fill a new database. In that situation the consumer, using its current reader schema, meets messages written by every old version that is still on disk. That is exactly the backward promise, and it is best backed by the transitive version, which checks against all earlier versions.

Kafka Streams

A Kafka Streams application reads input topics, and it also stores data in its own state stores and internal topics, such as changelog topics. When you upgrade the application, the new code reads data that the old code wrote into those internal topics. The new schema must therefore read old data, which points again to backward-style evolution. Applications with long-lived state, such as KTables built from compacted topics, benefit strongly from the transitive rules.

Kafka Connect

Connect moves data between Kafka and other systems through converters, which are configured to use Avro, Protobuf, or JSON Schema with the registry. A source connector, for example one that captures database changes, creates a new schema version when a database column is added. If the registry rule refuses that version, the connector can fail. Planning compatibility for these auto-generated schemas is essential, since developers do not hand-write them.

• Database add-column changes map neatly to “add optional field with default”, which is safe under full compatibility.
• Database drop-column or type changes may be rejected. Plan the change in two steps or route through a new topic.
• Sink connectors writing to warehouses must also cope with new columns appearing, so check what the destination tolerates.

ksqlDB and other query layers

Query layers read a topic through a declared schema. When the schema evolves, the query may need to be updated or recreated, especially if a field it uses is removed. Backward compatibility protects the query from old data, while forward compatibility protects older queries from new data.

Walk through these questions in order the next time you create a topic.

Matching modes to real needs

Stricter rules buy safety at the price of freedom.

Backward

Forward

Full

Trade-offs at a glance

The deeper trade-off is between team independence and schema freedom. Strict rules give teams independence, because nobody has to wait for anybody else. Loose rules give the schema author freedom, but they make everyone else pay through coordination. Organizations with many teams almost always end up choosing independence.

The registry controls the shape of company data, so it deserves real protection.

Automatic registration: convenient but risky

Serializers can be told to register schemas automatically the first time they see one. This is handy during development, but in production it lets any application that gets permission push a new schema into the registry. A safer habit is to turn automatic registration off, register schemas through a reviewed pipeline, and let applications only look up the versions that were approved.

Sensitive data in schemas

A schema itself is not secret, but its field names can reveal a great deal, such as “passportNumber” or “medicalCondition”. Also, fields that carry personal information stay in Kafka for as long as retention allows. When you evolve a schema, think about whether you are adding personal data. Consider marking sensitive fields in the schema with tags, and use field-level encryption where the platform supports it.

Protecting the _schemas topic

The registry keeps everything in the _schemas topic. If someone deletes or corrupts that topic, every consumer loses the ability to decode messages. Give the topic a high replication factor, restrict who can write to it to only the registry itself, and back it up as you would any critical data store.

Compatibility problems usually announce themselves as errors, but only if you are listening.

What to watch

Handling poison messages

A poison message is a record that a consumer can never decode. If the consumer simply retries, it will be stuck forever. Good consumers catch the decoding error, send the raw bytes plus an error note to a dead-letter topic, and move on. Later, an engineer can study the dead-letter topic, work out which schema was missing or wrong, and replay the good messages.

Logging with care

When decoding fails, log the topic, partition, offset, and schema ID, which is enough to find the record later. Avoid logging the full message content, because it may carry personal data. Also log every schema registration with who did it, which subject, which version, and which compatibility rule was in effect.

A registry is small but sits on the critical path, so it must be dependable.

Why it stays fast

The registry is asked for schemas only occasionally. Producers ask when they meet a schema for the first time. Consumers ask when they meet a schema ID for the first time. After that, both use caches. Even a topic with a huge message rate produces just a handful of registry calls, so the service can be small and still serve very large deployments.

What happens if the registry goes down?

Programs that already have their schemas cached keep working. Trouble appears in three situations: a producer that starts up and needs to look up or register a schema, a consumer that meets a brand-new schema ID, and a pipeline trying to register a new version. So a registry outage rarely stops steady traffic immediately, but it blocks deployments and new applications.

flowchart LR
    APP["Application starts"] --> CH{"Schema in local cache?"}
    CH -->|"yes"| USE["Encode or decode straight away"]
    CH -->|"no"| REG["Ask Schema Registry"]
    REG -->|"available"| STORE["Save in cache then continue"]
    REG -->|"down"| FAIL["Wait, retry, or fail on that message"]
    

Making the registry highly available

• Run several nodes behind a load balancer, with one primary that accepts writes and the others serving reads.
• Replicate the _schemas topic across brokers, typically three copies.
• Spread nodes across zones so one data centre failure does not remove the whole registry.
• Keep clients patient. Configure sensible timeouts and retries in serializers.
• Consider multiple regions. Some setups replicate schemas to a second cluster so a disaster recovery site can read data with the same schema IDs.

A few simple pipeline habits turn compatibility from a hope into a guarantee.

The pipeline gate

The single most valuable practice is to test every schema change automatically before it merges. The build asks the registry whether the proposed schema would be accepted, and fails the build if not. Developers learn about a problem within minutes, on their own laptop or in the review, and not in production at three in the morning.

Same rule in every environment

It is a common mistake to use a relaxed rule in development and a strict rule in production. The developer’s change passes, then fails when it is finally deployed. Keep the compatibility setting identical across environments, so surprises appear early.

Managed services in the cloud

Cloud teams have several choices. Confluent Cloud includes a managed Schema Registry. On AWS, teams using Amazon MSK often choose the AWS Glue Schema Registry, which supports compatibility modes such as backward, forward, full, and none, with variants that check all earlier versions. Open-source options such as Apicurio Registry can be run on your own container platform. Whatever the choice, the same ideas apply: a subject, versions, a rule, and a gate in your pipeline.

Handling several environments and regions

If you run separate clusters for development, staging, and production, each has its own registry. Schema IDs may differ between them, which is fine, as long as messages are never copied raw between environments. When you must copy data between clusters, use a tool that also carries the schemas across, so IDs and meanings stay aligned.

The same good habits and the same bad habits appear again and again.

Patterns that work

Anti-patterns to avoid

Small habits that prevent most schema incidents.

Habits that pay off

Common mistakes

Let us watch one schema evolve under FULL_TRANSITIVE and see each decision.

Brightcart’s orders topic begins with a tiny schema. Over a year, the business asks for several changes. We will judge each one under the full transitive rule, which is the strictest ordinary choice.

flowchart LR
    V1["V1: orderId, amount"] --> V2["V2: adds couponCode with default"]
    V2 --> V3["V3: adds currency with default USD"]
    V3 --> V4["V4: removes couponCode which had a default"]
    V3 -. "rename amount to totalAmount" .-> X["Rejected, needs two-step change"]
    

The two-step rename

What if the team had used plain BACKWARD?

Under the non-transitive backward rule, one different path would have been allowed: making currency mandatory in V4 by removing its default. V4 versus V3 looks fine, because V3 always writes a currency. But V1 and V2 messages lack the field, so any V4 consumer that replays the topic would fail. The transitive rule catches this on day one.

Some wrong beliefs travel from team to team. Here is the correction for each.

The word “compatibility” appears in Kafka with a second, different meaning.

Everything so far has been about the shape of message data. Kafka also talks about compatibility between software versions: can a client library of one version talk to a broker of another version? This is a separate topic, and mixing the two causes confusion in meetings.

Kafka clients and brokers agree on which request versions to use when they connect, so many combinations of client and broker versions work together within the supported ranges. Over time the project retires very old versions, so platform teams should read the upgrade notes for their release before planning a cluster upgrade. That discussion belongs to cluster operations, not to schema design, and the two should be planned separately.

The same ideas appear wherever many teams share streams of data.

Structure checks catch many problems, but not all. Newer registry features try to close the gap.

So far, compatibility has meant one thing: does the structure line up? But real breakages often come from meaning. A field that used to hold a positive number now sometimes holds a negative one. A text field that used to contain an email address now contains a phone number. The shape is identical, so the registry sees nothing wrong.

What a data contract adds

A data contract is a formal agreement between the team that produces data and the teams that consume it. It includes the schema, but goes further. Modern registries, including recent versions of Confluent Schema Registry, let you attach extra information to a schema so the agreement becomes something a machine can enforce.

Compatibility groups

Some registries let you divide versions of one subject into groups, for example “major version 1” and “major version 2”. Compatibility is then checked only among schemas inside the same group. This lets a team introduce a truly breaking redesign as a new major version, while still guarding every change made within each major version. Consumers upgrade from one group to the next deliberately, often with a migration rule to help.

The mindset shift

The old view was, “the registry keeps my messages from crashing consumers.” The newer view is, “the registry helps carry an agreement between teams.” The shape check is the foundation, and the extra rules build on it. Even without any special features, you can take the same idea and apply it with plain habits: write clear field descriptions, name units in field names, review changes together, and keep example messages in source control.

Schema compatibility lets producers and consumers of a Kafka topic change their data shape at different times without breaking one another. The words backward, forward, and full describe which direction of reading is guaranteed. Transitive versions extend the guarantee across every earlier version. The rule you choose decides who must upgrade first and how much freedom schema authors have.