Avro, Protobuf and JSON Schema in Kafka
A beginner-friendly guide to the three data formats that give Kafka messages a shared, checkable structure: what each one is, how each works with a Schema Registry, and how to choose between them.
Imagine two friends who write letters to each other in different countries. If one writes in English, the other must read English. If one suddenly switches to a new alphabet without telling the other, the letters become useless. Now imagine a thousand friends, all writing to each other through one giant post office, and each of them may change their handwriting or language at any time. Chaos would follow. Kafka faces this very problem every day, because many applications write messages and many others read them, often built by different teams at different times. Avro, Protobuf and JSON Schema are the three popular ways to give those messages a shared language and a rulebook. This guide explains all three from zero, without assuming you know anything about them.
1Kafka Only Moves Bytes
The most important fact to remember before anything else: Kafka does not understand your data.
What Kafka really stores
Apache Kafka is a system that stores and delivers streams of messages, which it calls records. A record has a key and a value, and both are simply sequences of bytes. Kafka does not know whether those bytes represent a customer name, a photograph, a payment or a sentence. It never opens the message, never checks it, and never complains if the content looks strange. This is a deliberate design choice, because it keeps Kafka fast and lets it carry any kind of data.
Kafka is like a shipping company that moves sealed boxes. The company weighs the box and delivers it to the right address, but it never looks inside. If the sender packs a fragile vase without labels, the receiver may open the box and find something unexpected. The shipping company is not to blame, because looking inside was never its job.
The problem this creates
Because Kafka does not look inside, the two ends of the pipeline must agree on what is inside. The application that writes the message (the producer) must turn its data into bytes in a certain way. The application that reads the message (the consumer) must turn those bytes back into meaningful data using the same rules. If the agreement breaks, for example when a producer renames a field or changes a number into text, consumers may crash, misread the data, or silently produce wrong results.
Many beginners believe that Kafka “supports JSON” or “supports Avro” the way a database supports tables. In truth, Kafka itself supports none of them. The formats are handled by libraries in your producers and consumers, and Kafka only carries the resulting bytes.
Record
One message in Kafka, made of a key, a value, a timestamp and optional headers.
Producer
An application that writes records into Kafka topics.
Consumer
An application that reads records from Kafka topics.
Topic
A named stream of records, like a mailbox for one kind of message.
2Why We Need Schemas
A schema is a written description of what a message must look like. It turns a loose agreement into a checkable contract.
What is a schema?
A schema is a formal description of the structure of data. It lists which fields exist, what type each field has (text, whole number, decimal number, true or false, list, and so on), which fields are required and which are optional. You can think of it as the blueprint of a house. Anyone who has the blueprint knows how many rooms to expect and where the doors are.
A schema is like a printed form at a government office. The form has labelled boxes for name, date of birth and address, and each box expects a certain kind of answer. Because everyone uses the same form, the clerks can process thousands of applications quickly without guessing what any box means.
What goes wrong without schemas
Silent changes break consumers
A producer team renames a field from “customer_id” to “customerId” during a routine cleanup. Nobody tells the consumers. The next morning, dashboards show empty values and downstream jobs fail, and it takes hours to find the cause.
Bad data poisons the topic
A buggy producer starts writing records with missing fields or wrong types. Kafka happily stores them. Every consumer that reads them now fails at the same offset again and again, which people call a poison message.
No shared documentation
New team members ask, “What does this topic contain?” With no schema, the answer lives in someone’s memory or in an old wiki page that may be wrong. With a schema, the answer is one lookup away and always current.
The idea of a data contract
When producers and consumers agree on a schema, the schema becomes a data contract. The producer promises to send data that follows the contract, and consumers promise to read data according to it. When someone wants to change the contract, there are clear rules for doing it safely, which we will explore in the chapters on schema evolution.
3Serialization: Turning Data Into Bytes and Back
Every format in this guide is, at heart, a way to do two jobs: pack data into bytes, and unpack bytes into data.
Serialization and deserialization
Your application works with objects in memory, such as an order with a customer, a list of items and a total. Kafka needs bytes. The process of converting an object into bytes is called serialization. The reverse process, turning bytes back into an object, is called deserialization. In Kafka, the components that do this work are often called serializers and deserializers, or together, a SerDe.
Serialization is like flat-packing a piece of furniture into a box so that it can travel. Deserialization is assembling it again at the destination. The instruction sheet that shows how to pack and unpack it plays the role of the schema.
Two big families of formats
Data formats fall into two broad groups, and understanding the difference makes the three formats in this guide much easier to place.
Text formats
- Readable by humans with any text editor
- Easy to debug and share
- Field names travel with every record
- Larger messages and slower parsing
Binary formats
- Compact, so they use less network and disk
- Fast to encode and decode
- Not readable without the right tool
- Usually need the schema to be understood
Plain JSON is a text format. Avro and Protobuf are binary formats. JSON Schema is a bit different: it is not a new way of writing data but a way of describing and checking ordinary JSON, so the messages themselves stay as readable text.
Where the schema fits in
A binary format saves space by leaving out labels. If a record was written without field names, the reader cannot know that the first value is a customer name and the second is an amount, unless it also has the schema. That is why binary formats and schemas travel together, and why a place to keep those schemas becomes necessary. That place is the Schema Registry.
4The Schema Registry: A Library for Schemas
Instead of sending the full schema with every message, we store schemas in one shared place and send only a small reference number.
What is a Schema Registry?
A Schema Registry is a separate service that stores schemas and hands out identification numbers for them. It is not part of Kafka’s brokers. Producers and consumers talk to it over a normal web connection. The most widely used registry is the one from Confluent, but similar services exist from other vendors and open-source projects, and cloud providers offer their own versions as well.
Think of a public library with a catalogue. Each book (schema) is given a unique catalogue number. When you send a friend a note, you do not copy the whole book into the note; you only write the catalogue number. Your friend goes to the library, finds the book by number and reads it. The Schema Registry is that library.
The main concepts
Schema ID
A unique number given to each distinct schema. Records carry this small number instead of the whole schema.
Subject
A named scope under which versions of a schema are stored. It is usually linked to a topic, for example the value schema of an orders topic.
Version
Each time a subject gets a changed schema that is accepted, it receives a new version number: 1, 2, 3 and so on.
Compatibility setting
A rule that decides which schema changes the registry will accept for a subject.
How a producer and consumer use it
Producer prepares the schema
Before sending its first record, the producer’s serializer either registers its schema with the registry or looks up the existing one.
Registry checks compatibility
If the schema is new, the registry compares it with earlier versions using the compatibility rule. Incompatible changes are rejected.
Registry returns a schema ID
The producer receives the number and remembers it in a local cache so that it does not need to ask again for every record.
Producer writes the record
The record value contains the schema ID followed by the encoded data. Kafka stores it as ordinary bytes.
Consumer reads the ID and fetches the schema
The consumer’s deserializer reads the number from the front of the record, asks the registry for that schema once, caches it, and then decodes the data.
sequenceDiagram
participant P as Producer
participant R as Schema Registry
participant K as Kafka Broker
participant C as Consumer
P->>R: Register or look up schema
R-->>P: Schema ID
P->>K: Record with schema ID and encoded data
K-->>C: Record with schema ID and encoded data
C->>R: Fetch schema by ID
R-->>C: Schema
C->>C: Decode data using schema
Because both sides cache schemas locally, the registry is contacted only when a new schema ID appears. In steady state, almost no extra network calls are made per record.
5What a Record Looks Like on the Wire
Understanding the byte layout of a record makes the whole system feel less magical.
When the registry-aware serializers from Confluent write a record value, they follow a simple layout, often called the wire format. First comes a single leading byte that acts as a marker, a kind of “magic number” that says this record follows the registry format. Next come four bytes that hold the schema ID. Everything after that is the actual data, encoded in the chosen format. A consumer reads the first five bytes, learns which schema was used, and decodes the rest.
flowchart LR
A[Marker byte] --> B[Four-byte schema ID]
B --> C[Encoded payload]
Picture a parcel with a small sticker on the front. The sticker says “Use instruction sheet number 42.” The recipient looks up sheet 42 and knows exactly how to open and assemble the contents. The sticker is tiny, but it makes the parcel self-explanatory.
Why this design is clever
Five extra bytes are a tiny cost compared with attaching an entire schema to every record, which could easily be larger than the data itself. The design also lets a single topic contain records written with several different schema versions at the same time. Each record tells its own story through its schema ID, so consumers can always decode old and new records correctly.
A small format-specific detail
Protobuf needs one more piece of information. A single Protobuf definition file can describe several message types, so the registry format for Protobuf adds a short list of numbers after the schema ID to point at the exact message type inside the file. Avro and JSON Schema do not need this extra list. As a user you almost never handle these bytes yourself, because the serializer and deserializer libraries take care of it.
Trying to read a registry-encoded topic with a plain string or plain JSON reader. The first few bytes are not part of the data, so the result looks like garbage. Always use the matching deserializer.
6Apache Avro: The Long-Time Favourite of Kafka
Avro is a compact binary format with a schema written in JSON. It was designed with data pipelines and change over time in mind.
What is Avro?
Apache Avro is a data serialization system that grew up in the Hadoop world. You describe your data once in a schema file that is itself written in JSON. The schema names a record, lists its fields, and gives each field a type. Avro then encodes actual data into a very small binary form. Because the schema is always needed to decode that binary form, Avro and schema registries became close partners, and Avro has long been the default format in many Kafka platforms.
Avro is like a coded shorthand notebook. The notes are extremely short because they leave out all the labels, but anyone holding the key to the shorthand can read them perfectly. The key is the schema, and the registry is where everyone finds the key.
The building blocks of an Avro schema
Primitive types
Simple values such as null, boolean, int, long, float, double, bytes and string.
Record
A group of named fields, like a row in a table or an order with several details.
Enum
A fixed list of allowed names, such as the status of an order.
Array and map
A list of items of one type, or a set of key and value pairs with text keys.
Union
A field that may be one of several types. A union of null and text is the usual way to say a field is optional.
Logical type
An extra label on a basic type, so that a number can mean a date, a timestamp or a decimal amount.
Why teams choose Avro
Strengths
- Very compact messages because field names are not stored in each record
- Strong, well-defined rules for schema evolution
- Default values and aliases make changes easier
- Works without generating code, using generic records
- Long history and wide tool support in data platforms
Limitations
- Binary data is unreadable without tools
- The schema is always required to decode the data
- Unions and nullable fields confuse beginners
- Less natural for direct calls between web services
7How Avro Works Inside: Writer and Reader Schemas
Avro’s cleverest idea is that the producer and consumer may use different schemas, and Avro reconciles them.
Two schemas in the room
When Avro data is written, it is encoded using the writer schema, which is the schema the producer had at that moment. When it is read, the consumer may hold a different version called the reader schema. Avro compares the two and translates the data on the fly. This process is called schema resolution, and it is what makes safe change over time possible.
Imagine a translator sitting between two people. One person speaks an older version of a language, and the other expects a newer version. The translator fills in words that did not exist before with polite defaults and quietly drops words the listener does not need. Avro’s schema resolution is that translator.
What resolution does in practice
Matching fields by name
Avro matches fields in the writer and reader schemas by their names, not by their position in the data.
New reader fields get defaults
If the reader expects a field the writer did not have, the default value from the reader schema is used. Without a default, resolution fails.
Extra writer fields are skipped
If the writer sent a field the reader does not know, the reader ignores it.
Certain type promotions are allowed
A small number of safe widenings are permitted, such as reading an integer as a long number.
Aliases for renames
Avro lets a field carry alternative names called aliases. If you rename a field, you can list the old name as an alias so that readers can still match old data. This is the tidy way to rename, but it must be used carefully, because tools do not all treat aliases in the same way.
Always give new optional fields a default value, and make the default a null option inside a union when the field may be missing. This single habit prevents most Avro compatibility failures.
Generic and specific records
Avro offers two styles of working. In the specific style, a tool generates classes from the schema, giving you typed objects in your language. In the generic style, you work with flexible records that are looked up by field name at run time. The generic style is popular in data platforms where the exact schema is not known ahead of time.
8Protocol Buffers (Protobuf): Numbered Fields and Strong Types
Protobuf comes from Google. It is compact, fast and strict, and it is the natural partner of many microservice systems.
What is Protobuf?
Protocol Buffers, usually shortened to Protobuf, is a binary serialization format created at Google. You describe your data in a definition file with a special simple language. The definition names each message, lists its fields, and gives every field a type and a unique number. A tool called the Protobuf compiler then generates ready-to-use classes for your programming language. Java, Python, Go, C++, C#, JavaScript and many others are supported, which makes Protobuf a good choice when many teams use different languages.
Protobuf is like a form where every box has a small printed number instead of a long label. When you fill in the form, you write “box 1: Alice, box 2: 30”. To read it back, you look up what box 1 and box 2 mean. The numbers are short, so the filled form stays small.
Field numbers: the heart of Protobuf
In Protobuf, each field carries a field number in the definition, and that number, not the name, is what appears in the encoded bytes. Names exist only for humans and for the generated code. This has an important consequence: you may rename a field freely without breaking anything on the wire, but you must never change or reuse a field number for a different meaning.
Message
The basic unit of structure, like an order or a customer, containing numbered fields.
Field number
The permanent identifier of a field in the encoded data. It should never be reused.
Repeated field
A field that can appear many times, forming a list.
Enum
A fixed set of named values, each with its own number.
Oneof
A group of fields where at most one can be set at a time.
Reserved
A declaration that certain numbers or names of removed fields must never be used again.
Why teams choose Protobuf
Strengths
- Very compact and fast to encode and decode
- Strong typing with generated classes in many languages
- Safe evolution through field numbers and reserved markers
- Unknown fields are tolerated, so old code can read new data
- Shared with gRPC, so the same definitions serve services and events
Limitations
- Binary data is unreadable without tools
- Missing fields and default values can look identical
- Requires a build step to generate code in most projects
- Less natural for very dynamic or schema-less processing
9How Protobuf Works Inside: Evolution by Numbers
Protobuf reaches safe change through a different road than Avro, and the difference is worth understanding.
Encoding without the schema at hand
A Protobuf-encoded message is a series of small pieces, each holding a field number, a type hint and a value. Because each piece is self-labelled with its number, a reader that does not know a field can still see that the piece exists and skip over it. Avro works differently: it needs the writer schema to know where each value starts and ends. This is why Protobuf readers are naturally tolerant of new fields they have never heard about.
Imagine a row of labelled jars, each with a numbered lid. If you only know jars 1, 2 and 3, and someone hands you a shelf with jars 1, 2, 3 and 7, you can safely ignore jar 7. In Avro’s shorthand notebook, an unknown extra entry would make you lose your place, unless you have the exact key.
The rules of safe change
Add new fields with new numbers
Old readers skip the new numbers, and new readers see nothing for old data, so they fall back to a default.
Never change a field’s number
Changing a number silently changes the meaning of existing data and causes wrong reads with no error.
Reserve removed fields
When you delete a field, mark its number and name as reserved so that nobody reuses them for something else later.
Avoid changing types in incompatible ways
Only a few type changes are safe. Turning a number into text, for example, is not.
The default value trap
In the third generation of the language, a field that was never set reads back as a standard zero-like value, such as zero for numbers, empty text for strings and false for booleans. That means a reader cannot tell whether the producer sent zero or forgot to send anything. If the difference matters, the field should be declared with the optional keyword, which lets the reader check whether the field was actually present.
Reusing the number of a deleted field for a brand-new field. Old records still in the topic will be decoded as the new field, producing confusing wrong values that no error message will warn you about.
10JSON Schema: Rules for Everyday JSON
JSON Schema does not change how data looks. It adds a rulebook so that ordinary JSON messages can be checked.
What is JSON?
JSON is a very popular text format for data. It uses curly braces for objects, square brackets for lists, and puts names in quotes before their values. Almost every programming language can read and write it, web browsers love it, and humans can read it easily. Because of this, a huge number of applications already produce JSON.
What is JSON Schema?
JSON Schema is a standard way to describe the expected shape of a JSON document. A JSON Schema is itself written as JSON. It says which properties an object may have, what type each property should be, which ones are required, which text patterns or number ranges are allowed, how long a list may be, and more. A validator can then take any JSON document and answer a simple question: does this document follow the rules?
JSON Schema is like the checklist an airport officer uses. The passenger’s bag (the JSON) stays exactly as it is, in plain sight. The officer only compares it against the checklist and says pass or fail.
Why teams choose JSON Schema in Kafka
Strengths
- Messages stay human-readable and easy to debug
- Fits naturally with web, mobile and REST-style systems
- Rich validation rules such as ranges, patterns and lengths
- Easy adoption for teams already sending plain JSON
- No special binary tools needed to inspect data
Limitations
- Larger messages because field names repeat in every record
- Slower to parse than compact binary formats
- Evolution rules are subtler and easier to get wrong
- Weaker fit for very high-volume, cost-sensitive streams
How it works with the registry
With a schema registry, the JSON Schema serializer checks each outgoing message against the registered schema, then writes the JSON text after the same marker byte and schema ID used by the other formats. On the reading side, the deserializer can validate the message again before handing it to your code. Bad messages can therefore be stopped at the door instead of poisoning a topic.
11The Open and Closed Content Model in JSON Schema
One subtle idea decides whether JSON Schema evolution is easy or painful: whether extra properties are allowed.
Open content model
By default, a JSON Schema allows an object to contain properties that the schema never mentioned. This is called an open content model. Consumers built on an open schema accept messages with extra fields, which makes it easy to add new fields later. It is friendly, but there is a catch: it also lets typos and unexpected fields slip through unnoticed.
Closed content model
A schema can also state that no additional properties are allowed. This is a closed content model. Everything that appears must be listed in the schema. This is strict and catches mistakes early, but it makes evolution harder, because adding a field means old consumers, whose schema does not list the new field, would reject the new messages.
| Aspect | Open Model | Closed Model |
|---|---|---|
| Extra fields in a message | Accepted | Rejected |
| Adding a new field later | Usually easy and safe for old readers | Risky for readers with the old schema |
| Catching typos and stray data | Weaker | Stronger |
| Best suited for | Evolving event streams | Strict, rarely changing contracts |
An open model is a suggestion box that accepts any note. A closed model is a form with fixed lines where anything written outside the lines is thrown away. Forms catch mistakes, but they must be reprinted every time you want a new line.
For event streams that will change over time, most teams prefer the open model for consumers, and use required fields and clear type rules to keep the important parts strict.
Why this matters for compatibility checks
Registries check whether a new schema is compatible with older ones, and for JSON Schema the answer depends heavily on the content model. The same change, adding one optional property, may be accepted under an open model and rejected under a closed one. Whenever a JSON Schema change is rejected and the reason is not obvious, check the content model first.
12Avro vs Protobuf vs JSON Schema: Side by Side
All three can protect a Kafka topic. They differ in size, speed, readability and the way they handle change.
| Aspect | Avro | Protobuf | JSON Schema |
|---|---|---|---|
| Data on the wire | Compact binary | Compact binary | Readable JSON text |
| How the schema is written | JSON document | Definition file with its own simple language | JSON document |
| Fields identified by | Name (matched during resolution) | Permanent field number | Property name |
| Message size | Very small | Very small | Larger |
| Human readability of data | Needs a tool | Needs a tool | Directly readable |
| Code generation | Optional | Common and expected | Rare |
| Optional fields | Union with null plus default | Fields are optional by nature, with default-value caveats | Properties not listed as required |
| Evolution style | Reader and writer schema resolution | Numbered fields, unknown fields tolerated | Depends on open or closed content model |
| Rename a field | Use aliases with care | Safe on the wire, since only numbers travel | Treated as remove plus add |
| Typical home | Data platforms and pipelines | Polyglot microservices and gRPC shops | Web, mobile and REST-oriented teams |
flowchart TD
Q[Choosing a format for a topic] --> A{Must humans read messages directly?}
A -- Yes --> J[JSON Schema]
A -- No --> B{Are services in many languages using generated classes or gRPC?}
B -- Yes --> P[Protobuf]
B -- No --> C{Data platform with heavy evolution and analytics?}
C -- Yes --> V[Avro]
C -- No --> D[Pick what your team already knows best]
There is no universally best format. The right choice depends on who reads your data, how large your volumes are, how often the shape changes and what your teams already use.
13Schema Evolution and Compatibility Modes
Data shapes change. Compatibility rules decide which changes are safe.
Why schemas must change
A business never stands still. Today an order has a customer and an amount. Next month someone wants a discount code, next year a delivery preference. If every change forced all producers and consumers to upgrade at the exact same moment, no large company could ever ship anything. Schema evolution is the practice of changing a schema in ways that keep old and new applications working together.
Think of a road that must be repaired while traffic keeps flowing. You cannot close it for a week. You build a new lane beside the old one, move cars gradually, and remove the old lane only when nobody uses it. Compatibility rules are the traffic plan for this repair.
Two directions of compatibility
Backward compatibility
A consumer using the new schema can still read data written with the old schema. New readers cope with old records. This is the default choice in many registries.
Forward compatibility
A consumer using the old schema can still read data written with the new schema. Old readers cope with new records.
Full compatibility
Both directions hold at once, so new readers can read old data and old readers can read new data.
The modes offered by a registry
| Mode | What it checks | Safe upgrade order |
|---|---|---|
| Backward | New schema can read data from the previous version | Upgrade consumers first, then producers |
| Backward transitive | New schema can read data from all earlier versions | Upgrade consumers first, then producers |
| Forward | Previous version can read data from the new schema | Upgrade producers first, then consumers |
| Forward transitive | All earlier versions can read data from the new schema | Upgrade producers first, then consumers |
| Full | Both directions against the previous version | Any order |
| Full transitive | Both directions against all earlier versions | Any order |
| None | No checks at all | Only by discipline and luck |
Forgetting that non-transitive modes compare a new schema only with the latest version. If a topic still holds records written with much older versions, and consumers may replay from the beginning, transitive modes protect you better.
Why the upgrade order matters
Under backward compatibility, the new consumer must be ready before any producer starts writing new-shaped data, or at least it must be able to handle both. That is why consumers are upgraded first. Under forward compatibility, old consumers must survive new data, so producers may go first. Getting the order wrong is a common cause of sudden failures during deployments, even when the schema change itself was legal.
14Safe and Unsafe Changes in Each Format
The same real-life change can be safe in one format and dangerous in another.
| Change | Avro | Protobuf | JSON Schema |
|---|---|---|---|
| Add an optional field | Safe if it has a default | Safe with a new field number | Safe under an open model; risky under a closed one |
| Add a required field | Breaks old data unless a default exists | Old data simply lacks it, so treat it as optional in practice | Breaks old data because old messages lack it |
| Remove a field | Safe if the old field had a default | Safe if the number is reserved | Depends on the content model and whether readers required it |
| Rename a field | Use aliases; otherwise breaks | Safe on the wire; generated code changes | Breaks, because names are the identifiers |
| Change a type | Only certain widenings are allowed | Only a few compatible changes are allowed | Usually breaks unless the new type accepts old values |
| Reorder fields | Safe, since names are matched | Safe, since numbers matter | Safe, since order is not meaningful |
A short story of a risky change
A team decides that the “amount” field of a payment, currently a whole number of cents, should become a decimal in dollars. It feels like a small tidy-up. In every format this is a type change, and in every format it is dangerous. Old records hold cents, new records hold dollars, and a consumer reading both will silently add mismatched numbers. The safest approach is to introduce a new field, such as “amount_decimal”, keep the old one for a while, let consumers migrate, and remove the old one much later.
Add, do not change. When you need a different meaning, create a new field and retire the old one slowly. The meaning of an existing field is part of the contract, not only its type.
Problem
A team changes the meaning of an existing field but keeps its name and type, for example switching a “price” field from including tax to excluding tax.
Why It’s Harmful
Registries can only check structure, not meaning. The change passes every compatibility test while silently corrupting reports and calculations downstream.
Correct Approach
Introduce a new field with a clear name for the new meaning, document it and retire the old field after all consumers have moved.
15Subjects and Naming Strategies
A subject is the shelf on which versions of a schema are kept. How you name the shelf decides how flexible your topics can be.
The three common strategies
Topic name strategy
The subject is based on the topic name, with separate subjects for keys and values. It is the usual default and means one schema lineage per topic.
Record name strategy
The subject is based on the fully qualified name of the record type, so the same type can appear in many topics under one lineage.
Topic record name strategy
The subject combines the topic name and the record name, allowing several record types in one topic with a separate lineage for each topic.
Imagine a library with three ways to shelve books. One shelf per room (topic), one shelf per book title across all rooms (record name), or one shelf per book title inside each room (topic record name). Each method suits a different kind of library.
When to use which
Topic name strategy
Best when each topic holds exactly one kind of event. It is simple, familiar and works well with tools that assume this layout.
Record name strategy
Best when the same event type is written to several topics and you want a single schema history for it.
Topic record name strategy
Best when one topic must carry several event types, such as all events of one customer in order, and you still want each type to evolve separately.
Putting several unrelated event types in one topic under the topic name strategy. All types then share one compatibility lineage, and the registry rejects changes that would be perfectly fine for each type on its own.
16Performance, Size and Cost
Format choice affects network use, disk use, processor time and how fast you can debug.
Message size
Binary formats leave out repeated labels. In JSON, every record repeats every field name as text, so a stream of a billion small records repeats the same names a billion times. Avro and Protobuf omit those names and use compact number encodings, so the same information takes noticeably less space. The exact saving depends on the data, but for records with many short fields the difference is easy to see, and it grows with volume.
Processing speed
Binary decoders read numbers directly, while text parsers must scan characters, handle quotes and convert digits to numbers. For very high-volume pipelines, this difference translates into less processor use and higher throughput. For moderate volumes, the difference may be too small to matter, and readability may be worth more than saved microseconds.
Compression changes the picture
Kafka producers can compress batches of records. Compression removes much of the repetition in JSON, which narrows the size gap with binary formats. It does not remove the parsing cost, and binary formats still compress further, but it is unfair to compare raw JSON with compressed binary. Compare like with like when making decisions, and measure with your own data.
The hidden cost: debugging
Costs are not only technical. When something goes wrong at three in the morning, an engineer who can open a message and read it immediately saves time. With binary formats you need consumer tools that understand the registry format to see the content. Good tooling removes most of this pain, but the team must have it in place before an incident, not during one.
Run a small test with your own real records in all candidate formats. Measure size, encode time and decode time. Numbers from your data beat general claims from any article, including this one.
17The Wider Ecosystem: Connect, Streams and Other Registries
Formats matter beyond your own producers and consumers, because the tools around Kafka speak them too.
Kafka Connect
Moves data between Kafka and databases, files and cloud services. It uses converters that turn data to and from Avro, Protobuf or JSON Schema, using the registry.
Kafka Streams
A library for processing streams in your application. It needs a SerDe for each topic it reads or writes, and registry-aware SerDes exist for all three formats.
ksqlDB
Lets you query streams with SQL-like statements. It can read schemas from the registry, so you do not have to declare every column by hand.
Other registries
Open-source and cloud-provided registries also exist, and several support more than one of these formats. Features and compatibility rules differ by product.
Formats are like electrical plug types. Your kitchen appliances (Connect, Streams, query tools) must fit the sockets your house provides. If everybody uses the same plug and a shared wiring plan, adding a new appliance is easy.
Mixing formats in one organisation
Large organisations often end up with several formats at once, for example Avro in the data platform, Protobuf in the microservices area and JSON Schema for a public-facing event feed. This is not necessarily wrong, but it raises the cost of training, tooling and governance. Where possible, agree on a default and allow exceptions only for a clear reason.
Assuming every tool supports every format equally well. Before choosing a format, check that your connectors, stream processing libraries and analytics tools support it at the versions you use.
18Security, Governance and Data Ownership
Schemas are important company assets, and the registry is a service that deserves proper protection.
Protecting the registry
The registry decides which data shapes are legal for your topics. Anyone who can register or delete schemas can change that. Protect it like any important internal service: encrypt connections, require authentication, and control who may read, register or delete schemas. In many companies, only automated deployment pipelines may register schemas in production, while developers only read them.
Sensitive information and schemas
Do not store secrets in schemas
Default values, documentation text and examples inside schemas are visible to everyone who can read the registry. Never put passwords, keys or real customer data there.
Mark personal data clearly
Use field descriptions or tags to mark which fields hold personal information. This helps privacy teams and lets tools apply masking or encryption to the right fields.
Think about who may read which topics
A schema is public knowledge inside the company, but the topic data itself is controlled by Kafka’s own permissions. Keep those two layers of access in mind and align them.
Governance and ownership
Each schema should have a clear owner, a team that answers questions and approves changes. Good practice includes code review of schema changes, automated compatibility checks in the build pipeline, and a written policy on how long old fields stay before removal. A shared catalogue of topics and their schemas lets new teams discover data without asking around.
Turn off automatic schema registration by producers in production. Register schemas through a reviewed pipeline instead, so that a bug or a typo in application code cannot quietly create a new schema version.
19Monitoring, Errors and Troubleshooting
Schema problems show up in a small number of recognisable ways. Knowing them saves hours.
What to watch
Registry availability
Whether the registry answers requests quickly. New producers and consumers cannot start without it, although running ones survive on their caches.
Deserialization errors
Counts of records that consumers failed to decode. A sudden rise usually means a schema or format mismatch.
Compatibility rejections
Registration attempts refused by the registry. They reveal risky changes before they reach production.
Schema growth
The number of subjects and versions over time. Very fast growth may mean uncontrolled auto-registration.
Dead letter volume
The number of bad records moved to a side topic for later inspection.
Consumer lag
A consumer stuck on a bad record stops advancing, and its lag grows.
Common problems and their usual causes
| Symptom | Likely cause | First thing to check |
|---|---|---|
| Consumer crashes on one offset repeatedly | Poison message that does not match the schema | Deserializer configuration and the record’s schema ID |
| Reading gives strange leading bytes or garbage | Wrong deserializer, ignoring the registry prefix | Which deserializer class the consumer uses |
| Registration refused | Change violates the compatibility mode | The mode and the difference against earlier versions |
| Unknown schema ID error | Consumer pointed to a different registry or environment | Registry address and environment settings |
| Values silently wrong after a release | Meaning change or reused Protobuf field number | Schema history and recent changes to numbers or meanings |
The dead letter idea
Instead of letting one bad record halt a consumer forever, many teams catch the decoding error, copy the bad record to a separate dead letter topic along with error details, and continue with the next record. Someone then examines the dead letters calmly and fixes the cause. This keeps the main flow healthy while preserving the evidence.
flowchart LR
T[Main topic] --> C[Consumer]
C --> D{Decoding succeeds?}
D -- Yes --> S[Process record]
D -- No --> L[Dead letter topic]
L --> R[Review and fix later]
20Best Practices and Anti-Patterns
Habits that keep schemas healthy for years, and traps that bite teams again and again.
Good habits
Design schemas with change in mind
Assume every schema will change. Prefer optional fields, sensible defaults and clear names from the start.
Choose a strict compatibility mode
Use backward or full compatibility, and consider the transitive versions when topics are replayed from the start.
Check compatibility in the build pipeline
Test proposed schema changes against the registry before merging, so mistakes are caught in review and not in production.
Document every field
Use description text to explain meaning, units and allowed values. Structure without meaning is only half a contract.
Use precise types for special data
Use proper date, timestamp and decimal types instead of loose text, so every consumer reads them the same way.
Plan the upgrade order
Write down whether consumers or producers go first for each change, and follow that order in every deployment.
Anti-patterns to avoid
Problem
Sending plain JSON strings to Kafka with no schema and no registry, hoping that all teams remember the structure.
Why It’s Harmful
Nothing stops a bad or changed message from being written. Consumers fail in surprising ways, and nobody has a trusted description of the data.
Correct Approach
Register a schema for every topic and use a registry-aware serializer, even if you continue to use JSON as the data format.
Problem
Setting the compatibility mode to none because a schema change was being rejected and the team wanted it to go through quickly.
Why It’s Harmful
The rejection was protecting consumers. Turning it off hides the risk until a consumer breaks in production, often at the worst possible time.
Correct Approach
Make the change compatible, for example by adding a new field with a default, or plan a coordinated migration to a new topic when a breaking change is truly needed.
Problem
Letting every producer register schemas automatically in production.
Why It’s Harmful
A small coding change can create unplanned schema versions, and different environments drift apart, which makes troubleshooting painful.
Correct Approach
Register schemas from a reviewed release pipeline and let applications only look up the versions they are meant to use.
Problem
Building one gigantic schema that holds every field any consumer might ever want.
Why It’s Harmful
Every team is coupled to every other team’s changes, messages grow heavy, and ownership becomes unclear.
Correct Approach
Keep schemas focused on one business event or entity, with a clear owner, and let consumers combine data as they need.
21Choosing a Format and Real-World Scenarios
Let us apply everything to realistic situations.
A data platform team feeding analytics and a warehouse
They manage hundreds of topics, change schemas often and connect many tools that expect a registry. Avro is a very natural fit here, because of its mature evolution rules, its small messages and its long-standing support across data tools.
A microservices company using several programming languages
Its services already talk to each other through gRPC, and each team wants typed, generated classes in its own language. Protobuf lets the same definitions serve requests and events, so it is often the smoothest choice.
A product team publishing events that web and mobile teams read directly
People need to look at messages in browsers and simple tools, and volumes are moderate. JSON Schema keeps the data readable while still enforcing structure at the door.
A high-volume telemetry pipeline where cost matters
Billions of small records arrive daily. A compact binary format, either Avro or Protobuf, reduces network, storage and processing costs, and the savings grow with every added zero of volume.
A team migrating from unstructured JSON
They cannot change everything at once. Starting with JSON Schema lets them keep existing producers and consumers, add checks gradually and later move critical topics to a binary format if needed.
A short decision checklist
Who reads the data?
If humans and simple tools must read it directly, lean towards JSON Schema. If only programs read it, binary formats are fine.
What do your teams already use?
Familiarity and existing tooling are powerful. A slightly worse format your teams know well often beats a perfect one nobody understands.
How large and how costly is the volume?
At large scale, size and speed differences turn into real money. At small scale, readability may win.
How often will the shape change?
Frequent change favours formats with well-understood evolution rules and strong tooling around compatibility checks.
What do your tools support?
Verify support in connectors, stream processing and analytics before committing.
22Frequently Asked Questions
No. Kafka stores and delivers bytes. The formats are handled by serializers and deserializers in your applications, usually together with a schema registry.
Technically no, because you can share schemas by other means. In practice a registry is strongly recommended, because it stores schemas centrally, hands out compact IDs and enforces compatibility rules.
There is no single best. Avro is a strong default for data platforms, Protobuf shines in polyglot service environments, and JSON Schema suits readable, web-facing events. Choose based on your readers, volume, change rate and tooling.
No. JSON is the data format. JSON Schema is a set of rules that describes and validates JSON data. With a registry, messages remain readable JSON, but they are checked against the schema.
The schema ID is a small number that identifies the exact schema used to write the record. Consumers use it to fetch the right schema from the registry and decode the data, which lets one topic safely hold records from several schema versions.
It depends on the format. In Protobuf, renaming is safe on the wire, since numbers travel, though generated code changes. In Avro, use aliases with care. In JSON Schema, a rename behaves like removing one property and adding another, so it is usually breaking.
It means that consumers using the new schema can still read data written with the old schema. Because of this, you normally upgrade consumers before producers.
Most likely the change breaks the compatibility mode set for that subject, for example adding a required field with no default or changing a type. Check the mode, compare with the earlier version, and make the change additive.
Yes, if you use a subject naming strategy based on the record name, such as the record name or topic record name strategy. With the default topic name strategy, all records in a topic share one schema lineage.
The overhead is small, because schemas are cached after the first lookup and only a few extra bytes are added to each record. Binary formats are often faster than plain JSON, and validation in JSON Schema adds some processing cost.
In production, most teams do. Schemas are registered through a reviewed pipeline, so that application bugs cannot quietly create new versions.
Without special handling, the consumer may fail repeatedly on that record. A common solution is to catch the error, send the bad record to a dead letter topic for later inspection, and continue with the next record.
23Summary and Key Takeaways
Kafka moves bytes and does not care what they mean, so applications need a shared, checkable agreement about the shape of their data. Schemas provide that agreement, a Schema Registry stores them centrally, and a short schema ID in each record lets consumers find the right schema. Avro offers compact binary data with strong evolution rules and is a long-time favourite in data platforms. Protobuf offers compact, strongly typed binary data based on permanent field numbers and fits polyglot service landscapes. JSON Schema keeps data as readable JSON while adding validation and suits web-facing and gradual adoption scenarios. Whichever you choose, the real safety comes from compatibility rules, careful evolution, good governance and healthy habits.
Key Takeaways
- Kafka is format-agnostic — it stores bytes, and the formats live in your serializers and deserializers.
- A schema is a data contract — it turns a loose agreement into rules that can be checked automatically.
- The registry stores schemas and issues IDs — each record carries a small marker and schema ID, not the whole schema.
- Avro matches fields by name — it resolves writer and reader schemas, so defaults are essential for safe change.
- Protobuf identifies fields by number — never change or reuse a number, and reserve removed ones.
- JSON Schema validates readable JSON — its open or closed content model strongly affects how easily it evolves.
- Compatibility modes and upgrade order matter — backward means consumers first, forward means producers first, full means any order.
- Add, do not change — introduce new fields for new meanings instead of altering existing ones.