Schema evolution is the controlled change of a data format over time, while old and new versions of the data keep working together.

Start with the basics. A message (also called an event or record) is a small package of data that a program writes to Kafka, for example “order 1042 was placed by customer 77 for 3 items”. A topic is a named log where such messages are stored in order. A schema is the written description of the shape of a message: which fields it has, what type each field is (text, number, date) and which fields are required.

Schema evolution means changing that description over time. You might add a field, remove a field, or change a type. The word “evolution” hints at the important point: the change happens gradually, in versions, while the system stays alive. Version one of the order schema might have three fields. Version two adds a fourth. Version three retires the second. At any moment, messages written under every earlier version may still exist in the topic, and programs at different release levels may be reading and writing at the same time.

The core question

Every schema change raises one question: who might be surprised? A new producer might write data that an old consumer cannot understand. A new consumer might read old data that lacks a field it now expects. Schema evolution gives precise names and automatic checks for these surprises, so a human does not have to guess.

Kafka stores bytes and knows nothing about their meaning. That freedom is powerful, and it is also the reason problems appear.

Kafka brokers treat every message as a bag of bytes. They do not check that a message contains a customer ID, and they do not know whether a price is a number or a piece of text. This makes Kafka fast and flexible, but it means the meaning of the bytes must be agreed between the writer and the reader by some other mechanism. That agreement is the schema.

Now add the reality of large companies. A topic named “orders” may be written by one team and read by ten: billing, shipping, analytics, fraud, search, recommendations, a data lake loader, and more. These teams do not release on the same day. If the producing team changes the format on Tuesday, the ten consuming teams cannot all update on Tuesday. Someone has to define what kinds of change are allowed so that the slowest team is not broken.

What goes wrong without it

A schema is written in a format. The three most common formats in Kafka are Avro, Protobuf and JSON Schema, and each handles change a little differently.

Before a message can be stored in Kafka, the program converts it from an in-memory object into bytes. This step is called serialization. The reverse step, turning bytes back into an object, is deserialization. The serializer follows a schema so that the bytes have a known layout.

Avro in plain words

Avro is a data format that stores the values without repeating the field names in every message, which makes it small. The trade-off is that the reader must know the schema to decode the bytes. Avro was designed with evolution in mind: it allows a reader to hold a different version of the schema than the writer and reconciles them using a fixed set of rules. This is why Avro has been the traditional favorite in the Kafka world.

Protobuf in plain words

Protocol Buffers, or Protobuf, is a format from Google. Every field has a number called a tag, and the bytes carry these numbers instead of names. A reader that sees a number it does not recognize simply skips that field. This makes adding new fields naturally safe. The golden rule is that a field number, once used, must never be reused for something else.

JSON Schema in plain words

JSON is the text format everyone can read, and JSON Schema is a way to describe it. It is convenient for debugging, since you can open a message and read it. It is bulkier than the binary formats. Evolution rules can be subtle, because JSON Schema can either allow or forbid extra fields, and that choice changes what counts as compatible.

Schema Registry is a small service that stores every version of every schema and refuses changes that would break compatibility.

A schema registry is a central server that keeps a catalog of schemas. Confluent Schema Registry is the best known, and there are others, such as Apicurio Registry and the AWS Glue Schema Registry, that play the same role. The registry does three jobs. It stores schemas and gives each a unique numeric ID. It tracks versions of schemas over time. And it checks each proposed new version against a compatibility rule before accepting it.

flowchart LR
  P["Producer Application"] --> SER["Serializer in Producer"]
  SER --> LB["Load Balancer and API Gateway"]
  LB --> SR1["Schema Registry Instance 1"]
  LB --> SR2["Schema Registry Instance 2"]
  SR1 --> ST["Internal Schemas Topic in Kafka"]
  SR2 --> ST
  SER --> T["Kafka Topic with Partitions"]
  T --> DES["Deserializer in Consumer"]
  DES --> LB
  DES --> C["Consumer Application"]

The components in plain words

Why the ID trick matters

Putting a full schema in every message would waste space, since the schema can be larger than the data. Instead, the registry assigns a small number to each unique schema, and every message carries just that number. A consumer that sees the number asks the registry once, remembers the answer, and decodes all later messages with that ID from memory. The registry is therefore consulted only when a new schema ID appears, not for every message, which keeps it out of the hot path.

Follow a single order event from the producer to the consumer to see exactly where the schema and its ID are used.

flowchart TD
  A["Producer creates event"] --> B["Serializer asks registry for schema ID"]
  B --> C{"Is the schema compatible and registered"}
  C -->|"No"| D["Registration rejected and producer fails fast"]
  C -->|"Yes"| E["Bytes written with magic byte and schema ID"]
  E --> F["Kafka stores bytes in a partition"]
  F --> G["Consumer reads schema ID from header"]
  G --> H["Fetch writer schema once and cache it"]
  H --> I["Decode using writer schema and reader schema"]

Compatibility is a rule that answers one question: can readers and writers of different versions still understand each other?

Two words are used again and again: writer schema and reader schema. The writer schema is the version used when the message was produced. The reader schema is the version the consumer’s code uses to interpret it. Compatibility describes which combinations of the two are guaranteed to work.

Transitive versus non-transitive

The word transitive decides how many earlier versions a new schema is checked against. Without it, a new schema is compared only with the latest registered version. With it, the new schema is compared with every earlier version in the subject. This matters whenever old data stays in the topic. If a topic keeps messages for a year, a consumer that starts from the beginning must read every version written during that year, so a check against only the latest version is not enough.

Backward compatibility protects consumers that upgrade first. It is the most widely used mode in practice.

With BACKWARD compatibility, a consumer running the new schema can read messages written under the previous one. The natural question is what happens when the new schema has a field the old data never had. The answer is a default value. The consumer fills the missing field with the default that the new schema declares. That is why a new field must have a default in this mode: without one, the reader has no way to fill the gap.

The upgrade order

Backward compatibility tells you that consumers should be upgraded before producers. Think about why. If a producer moved to version two first, it would start writing new data, and any consumer still on version one might not understand it. If consumers move first, they hold version two and can read both old version-one data and new version-two data as it appears. Only after every consumer has moved is it safe to upgrade the producer.

flowchart LR
  S1["Step 1: Register new schema version"] --> S2["Step 2: Upgrade all consumers"]
  S2 --> S3["Step 3: Upgrade producers"]
  S3 --> S4["Step 4: New data flows to ready consumers"]

Forward compatibility protects consumers that cannot upgrade quickly. Full compatibility protects everyone, in any order.

With FORWARD compatibility, an old consumer can read messages written under the new schema. When a new producer adds a field, the old consumer does not know it exists, so it simply ignores it. When a new producer deletes a field, the old consumer would look for a value that is missing, so that field must have had a default. Forward compatibility means producers can be upgraded first, and consumers catch up later.

When forward compatibility fits

Full compatibility

FULL compatibility requires both backward and forward safety. In practice, this limits you to adding optional fields with defaults and removing optional fields with defaults. It is the most restrictive of the useful modes, and the most comfortable to live with, because producers and consumers can be upgraded in any order. Many organizations choose FULL_TRANSITIVE for their most important, widely shared topics for exactly this reason.

Some edits are almost always safe. Others break something, whatever the setting. Knowing the difference saves outages.

The dangerous one nobody can detect

A registry checks the structure of a schema, not its meaning. If a field called “amount” switches from dollars to cents, the type stays the same, the check passes, and every downstream calculation is quietly wrong. Meaning changes need human review and documentation. The safe habit is to never repurpose an existing field. Add a new field with a new name, and retire the old one on a schedule.

Avro compares the writer schema with the reader schema field by field. Knowing the rules makes every compatibility error easy to read.

When an Avro consumer decodes a message, it holds two schemas at once. The writer schema comes from the registry using the ID in the message header. The reader schema is the one compiled into the consumer’s code. Avro walks through the fields and applies a fixed set of rules to line them up.

Making a field optional

In Avro, “optional” is expressed by a union of null and another type, with a default of null. A union is simply a list of allowed types for one field. A field declared as a union of null and text can hold either nothing or a piece of text, and by default holds nothing. This is the standard pattern for adding a field safely: make it a union with null and give it a null default.

Renaming with aliases

Avro supports aliases, which let a reader schema say “this field used to be called something else”. A reader that knows the alias can match old data written under the earlier name. Aliases help in careful renames, but they only work for readers that have been upgraded and they behave differently across tools, so many teams prefer the safer approach of adding a new field and retiring the old one over time.

Enums need care

An enum is a field limited to a fixed list of names, such as order status values. Adding a new value is a common change, and it can break readers that meet a value they have never heard of. Avro allows a schema-level default for an enum that a reader can fall back to when it sees an unknown symbol. Without such a default, adding a value can be treated as unsafe, so plan the fallback value from the beginning.

Protobuf evolves through numbered fields. Respect the numbers and change becomes remarkably forgiving.

In Protobuf, every field is identified by a number called a tag, and the encoded bytes carry the tag, not the name. Names can change freely in the source file without touching old data, because the number is what matters on the wire. A reader that encounters a tag it does not know skips the value. New readers that find a tag missing simply use the type’s default, such as zero for numbers or empty text.

JSON Schema is readable but subtle. Its compatibility depends on whether unknown fields are welcome.

JSON Schema can describe an object in two styles. In an open content model, extra fields that the schema does not mention are allowed. In a closed content model, extra fields are forbidden. This single choice changes what counts as a compatible change.

Because of this, teams often favor the open model for topics expected to evolve, so old readers tolerate new fields. The tradeoff is that typos in field names slip through unnoticed, since anything extra is allowed. Text-based JSON is also larger on the wire than the binary formats, so high-volume topics often choose Avro or Protobuf instead.

The subject is the bucket in which versions are tracked and compatibility is checked. How you name it shapes your whole design.

A subject is a named list of versions of a schema. Compatibility checks compare a new schema with earlier versions in the same subject. The naming strategy decides how the serializer picks the subject name for each message.

A checklist that turns compatibility theory into a change you can deploy on an ordinary Tuesday.

Sometimes the business truly needs a change that no compatibility mode allows. The answer is migration, not force.

Imagine a payment event whose amount must change from a whole number of cents to a decimal with currency. No safe evolution path exists, because the old and new types cannot be read by each other. Turning compatibility off and hoping for the best is the wrong response. The usual approach is to create a second topic for the new shape and move consumers over deliberately.

flowchart LR
  P1["Producer on old schema"] --> T1["Topic payments v1"]
  T1 --> M["Migration Job or Stream Processor"]
  M --> T2["Topic payments v2"]
  P2["Producer on new schema"] --> T2
  T1 --> C1["Old Consumers"]
  T2 --> C2["Upgraded Consumers"]

A few repeatable moves handle nearly every real change without breaking anyone.

Expand and contract

This pattern, also called parallel change, splits a risky change into two safe ones. In the expand phase you add the new thing next to the old one. In the contract phase, much later, you remove the old thing. Between the two phases, every consumer has time to move.

Optional by default

Make new fields optional with sensible defaults. A required field with no default forces everything to change in lockstep, which is exactly what schema evolution is trying to avoid.

Deprecate before deleting

Mark a field as deprecated in its description, announce it to the consuming teams, and give a date. Watch usage. Delete it only when nobody reads it any more. Removing a field that a forgotten weekly report depends on is a classic way to ruin a Monday.

Old data keeps coming back to life. Replays, compaction and stateful applications all put pressure on old schema versions.

Replay and long retention

A topic with long retention holds messages of many schema versions side by side. A brand new consumer that starts from the earliest position will meet all of them in order. That is why the transitive compatibility modes matter: they guarantee the current reader can decode every version still sitting in the topic. If your retention is infinite, treat every version ever registered as still alive.

Compacted topics

Log compaction keeps the latest message for each key and discards older ones. It is often used for reference data such as customer profiles. Compaction does not rewrite old messages in a new schema. A compacted topic may hold, for one key, a message written years ago under version one, next to a message for another key written yesterday under version five. Readers must handle both.

Kafka Streams and state

Kafka Streams applications store running results, such as counts and totals, in state stores that are backed by internal changelog topics. Those topics contain serialized data. If you change the shape of the stored value, the application must be able to read what it wrote before. For long-lived state, evolve the stored schema with the same compatibility rules, or reset and rebuild the state deliberately during a planned migration.

Kafka Connect

Connectors move data between Kafka and other systems, using converters that talk to the registry. A source connector reading a database table may produce a new schema whenever a column is added, and a sink connector writing to a warehouse may need to add a matching column. Set the compatibility mode so that database changes such as adding a nullable column pass smoothly, and decide in advance how sinks react when a new field appears.

Schema management brings safety and costs a little speed. Most organizations decide the trade is well worth it.

The main trade-offs

The registry is on the edge of the hot path, not in the middle of it, so it scales easily when used properly.

Clients cache schemas by ID, so the registry receives a request only the first time a client meets a schema, or when a producer starts. In steady state, millions of messages per second may flow through Kafka while the registry sees almost no traffic. This is why a small registry cluster can support very large deployments.

A registry outage should never take down producers and consumers that are already running.

flowchart TD
  M["Consumer reads a message"] --> Q{"Is schema ID known in local cache"}
  Q -->|"Yes"| DEC["Decode immediately"]
  Q -->|"No"| REG["Ask registry for schema"]
  REG --> OK{"Registry reachable"}
  OK -->|"Yes"| CACHE["Cache schema and decode"]
  OK -->|"No"| RETRY["Retry with backoff and raise an alert"]
  DEC --> NEXT["Process the record"]
  CACHE --> NEXT

What to do with messages that cannot be decoded

Sometimes a message simply cannot be decoded, because of corruption, a wrong format, or an unknown schema ID. A consumer that crashes on the same message forever blocks its partition. Common practice is to catch the error, write the raw message and the error details to a separate dead letter topic, alert a person, and move on. The bad messages can then be investigated and replayed later without stopping healthy traffic.

Schemas describe your data, and controlling who can change them protects everything that depends on them.

A short list of signals catches almost every schema problem before customers do.

Log every registration with the subject, version, ID and the identity that made the request. When a mystery appears at three in the morning, that log answers the first question: what changed recently? Alert on deserialization errors and dead letter volume, because these are the earliest signs that a producer and a consumer no longer agree.

Walk through a realistic sequence of changes and see which are accepted, which are rejected, and why.

An online shop publishes an order event. Version one has three fields: an order number, a customer ID and a total in cents. The subject uses BACKWARD compatibility.

Change one: add a coupon code

The marketing team wants to record a coupon code. The developer adds a field called “coupon” as a union of null and text, with a default of null. The registry checks: can a reader on the new schema read old data? Yes, because old messages lack the coupon and the reader fills in null. The change is accepted as version two. Consumers upgrade first, then the producer starts writing coupons.

Change two: make the customer ID a text value

Another developer wants to change the customer ID from a number to a text value, so that IDs can hold letters. The registry checks the change and finds that a reader expecting text cannot decode old bytes that hold a number, because a number cannot be promoted to text. The registration is rejected. The developer follows the expand and contract pattern instead: adds a new optional field “customer_ref” as text, has the producer fill both fields, and plans to retire the numeric field later.

Change three: add “customer_ref”

The optional text field with a null default is accepted as version three. Over the next weeks, each consumer switches from the old numeric field to the new text field. When monitoring shows nobody reads the numeric field, a later version removes it. Deleting a field is allowed in BACKWARD mode, so this final step is accepted too.

A few shapes work well again and again. A few shortcuts cause most of the outages.

Patterns that work

Anti-patterns to avoid

A short checklist gathered from many production systems. Follow it and most schema pain never appears.

Mistakes seen again and again

Changing a database table and changing a Kafka schema look alike, yet the second is far less forgiving.

Anyone who has changed a database table knows the routine: write a migration script, run it during a quiet window, update the application, and finish. The old shape of the table disappears, and every program that uses it sees only the new shape. A Kafka topic behaves differently in three important ways.

This is why schema evolution is not just a nicer name for migration. It is a design approach where change is expected to be gradual, additive and reversible, and where readers must always be ready for several versions of the same data.

A change that passes the registry can still fail in real code. A few cheap tests close that gap.

The registry answers one question: is the new schema structurally compatible? It cannot answer whether your application handles the change correctly. A layered testing plan covers both.

Most schema incidents are not technical failures. They are communication failures between teams.

A registry stops incompatible changes, but it cannot tell a consuming team that a field is about to disappear. The human side of schema evolution needs its own habits. The most successful organizations treat a topic like a public product with an owner, documentation and a change policy.

A simple deprecation timeline

Suppose a team plans to retire the old “cust_id” field. In week one, it adds the new field and marks the old one as deprecated in its description. In weeks two to eight, it announces the plan and tracks which consumer groups still read the old field. In week ten, it contacts any team still using it. In week twelve, it removes the field only if usage has dropped to zero. If a critical consumer still depends on it, the date moves, and the plan is reviewed. Nobody is surprised, and nobody needs an emergency rollback.

Many teams start with plain JSON and no registry. Adopting schemas later is possible, and it can be done without a big-bang rewrite.

Early projects often send loosely defined JSON. It is quick and it works, until three teams depend on the same topic and one of them adds a field that another team misreads. Introducing schemas at that point feels daunting, since the topic already contains months of unstructured data. A gradual path exists.

The registry is small compared with Kafka itself, but a little planning avoids surprises.

Schemas are tiny text documents, and even a large company may hold only a few thousand of them. The registry therefore needs little storage. What deserves attention is the number of requests during busy moments and the health of the internal topic that stores the data.

Schema evolution appears wherever many teams share streams of data.

Kafka began at LinkedIn to carry activity data, and schema-based formats have been central to how large companies keep such data manageable. Public talks and blog posts from many companies describe the ideas below. Details differ from company to company and change over time, so treat these as illustrations of the technique.

A quick reference for the vocabulary used throughout this guide.

Quick answers to the questions that come up most when teams first manage schemas in Kafka.

Schema evolution lets a shared stream of data change safely over time.