What Is Serialization in Kafka? (Basics)
A beginner-friendly guide to how your data is turned into bytes before it enters Kafka and back into meaningful data when it comes out, and why the format you choose shapes speed, safety, and teamwork.
Imagine you want to mail a fully built toy model to a friend in another country. The postal service cannot carry a delicate model as it is. So you take it apart, pack the pieces flat in a box, and include an instruction sheet. Your friend opens the box, reads the instructions, and rebuilds the model exactly as it was. If you used a secret packing method that your friend does not know, they would receive a heap of pieces and no idea how to put them together. Apache Kafka works in much the same way. It cannot store or move your applications’ rich data directly. Data must be flattened into plain bytes on the way in and rebuilt on the way out. That flattening is called serialization, and the rebuilding is called deserialization. This guide explains both in simple words.
1Why Kafka Needs Serialization
To understand serialization, we first need to see what Kafka actually stores.
Kafka stores bytes, nothing more
Apache Kafka is a system for storing and moving streams of events, such as orders, payments, clicks, or sensor readings. Each event is stored as a record with a key and a value. To Kafka, both the key and the value are just a sequence of bytes. Kafka does not know whether those bytes represent a customer, a picture, or a temperature reading. It does not look inside. It only stores them, copies them to other servers, and hands them back.
Applications think in rich data
Your applications, however, work with meaningful things: an order with a number, a list of items, and a total price; a user with a name and an address; a sensor reading with a machine identity and a temperature. These live in programs as objects, structures, or records, in a form that only that programming language understands. They cannot be sent across a network or written to a disk as they are.
Serialization is the bridge
Serialization is the act of converting your data into a flat sequence of bytes that can be stored and transmitted. Deserialization is the reverse: converting the bytes back into the same meaningful data. Kafka sits in the middle and treats the bytes as a sealed parcel. The producer, the client that writes events, serializes before sending. The consumer, the client that reads events, deserializes after receiving.
Think of a translator between two people who speak different languages. Kafka only understands a universal alphabet of bytes. The serializer translates your application’s data into that alphabet, and the deserializer translates it back. Both translators must use the same dictionary, or the message is lost in translation.
flowchart LR
A["Producer application: rich data"] --> S["Serializer"]
S --> B["Bytes"]
B --> K["Kafka stores and copies bytes"]
K --> B2["Bytes"]
B2 --> D["Deserializer"]
D --> C["Consumer application: rich data again"]
Because Kafka treats data as opaque bytes, it works with any kind of content and any programming language. The price of this freedom is that you, not Kafka, are responsible for agreeing on the format.
2What Is Serialization? A Closer Look
Serialization sounds technical, but the idea is very human.
The simple definition
Serialization is turning structured data into a sequence of bytes using an agreed set of rules. Deserialization is applying the same rules in reverse. The agreed set of rules is called the format or the encoding. Any format will do, as long as both sides use the same one.
An example in words
Suppose an order has a number, a customer name, and a total. One possible format writes it as readable text, listing each field name and value in a fixed layout. Another format writes only the values in a fixed order, packed tightly with no names at all, relying on a separate description to say what each position means. Both carry the same information. The first is easier for people to read, and the second is smaller and faster for machines.
Three questions every format answers
How are values written?
Are numbers written as text or as compact binary? Are strings marked with their length or with an end marker?
How are fields identified?
Does each value carry its field name, a field number, or only its position?
How does the reader know the shape?
Is the shape described inside the data, or in a separate written schema that both sides share?
A recipe can be written as a full paragraph, “add two cups of flour, then one cup of sugar”, or as a compact table with only the numbers. The paragraph is friendly to read but long. The table is short, but you need the column headings to understand it. Formats make the same kind of choice.
Thinking that serialization is only about saving space. It is equally about agreement. A tiny format that nobody else can decode is worse than a large one that everybody understands.
3Where Serialization Happens: Keys and Values
Every Kafka record has two main parts, and each has its own serializer.
The key serializer
The key is an optional identity, such as a customer number, that helps decide which partition receives the record and keeps related events together. It is converted to bytes by the key serializer. Because the partition choice is calculated from the key’s bytes, the key format has a special importance, discussed later in this guide.
The value serializer
The value is the main content of the event, such as the details of an order. It is converted to bytes by the value serializer. Keys are usually simple, such as a number or a short text, while values are often larger structures. For this reason, teams often use different serializers for keys and values.
Where the work is done
The application creates the record
The application gives the producer a topic, a key, and a value in its own rich form.
The producer serializes both parts
The key serializer and the value serializer turn key and value into bytes inside the producer, before anything leaves the application.
The partitioner uses the key bytes
The producer chooses the partition from the bytes of the key, batches the record, and sends it.
Kafka stores the bytes as they are
The brokers store and replicate the bytes unchanged. They never translate them.
The consumer fetches the bytes
The consumer receives the same bytes that were stored.
The consumer deserializes
The key deserializer and the value deserializer turn the bytes back into meaningful data for the application.
On a parcel, the address label and the contents are packed differently. The label uses a standard, easily read format so every sorter can use it, while the contents can be packed in whatever way suits the item. The key is the label, and the value is the contents.
4The Built-in Basic Serializers
Kafka’s client library includes a few simple serializers for common needs, so you can start without any extra tools.
| Serializer Type | What It Handles | Typical Use |
|---|---|---|
| String | Plain text | Simple keys, text messages, and JSON written as text |
| Integer and Long | Whole numbers in compact binary | Numeric identities as keys |
| Double and Float | Decimal numbers | Simple measurements |
| Byte array | Raw bytes, unchanged | When you have already turned the data into bytes yourself |
| UUID | Universally unique identifiers | Identity keys |
| Void or null | No data at all | Records without a key |
Simple does not mean careless
These built-in serializers are perfect for keys and for very simple values. But they only handle single simple items. As soon as your event has many fields, lists, or nested parts, you need a format that can describe structure. That is where structured formats come in.
A simple envelope is fine for a single sheet of paper. If you must send a whole folder of documents with tabs and attachments, you need a proper binder with an index. Built-in serializers are envelopes. Structured formats are binders.
Many teams write their own serializers for special needs, for example to add compression or encryption. This is possible because a serializer is a small pluggable component with a very simple job: data in, bytes out.
5Popular Structured Formats
For real business events with many fields, teams usually choose among a small number of well-known formats.
Plain text
Fields written as readable lines or separated values. Easy to look at, but fragile, with no built-in structure or rules.
JSON
Readable text with named fields. Very popular, easy to debug, but larger and without enforced rules unless a schema is added.
Avro
A compact binary format driven by a schema. Field names are not repeated in every record, so events are small. Strong rules for safe change.
Protobuf
A compact binary format from Google that identifies fields by numbers. Fast and small, with strong support across many languages.
JSON with a schema
Readable JSON combined with a written description that can be checked, offering a middle path between readability and discipline.
Comparing them side by side
| Feature | Plain Text | JSON | Avro | Protobuf |
|---|---|---|---|---|
| Readable by people | Yes | Yes | No, needs a tool | No, needs a tool |
| Size on the wire | Large | Large | Small | Small |
| Speed to encode and decode | Moderate | Moderate | Fast | Fast |
| Strict structure | No | Optional | Yes | Yes |
| Safe rules for change | None | Weak by default | Strong | Strong |
| Ease of first use | Easiest | Very easy | Moderate | Moderate |
Plain text and JSON are like letters written in full sentences: anyone can read them, but they take more paper. Avro and Protobuf are like a compact form filled in with only the answers, together with a separate sheet that explains the questions. They are short and fast, but you need the question sheet.
6Schemas: The Shared Instruction Sheet
A schema is the key idea that makes compact formats safe and reliable.
What is a schema?
A schema is a written description of the shape of an event. It lists the fields, their names, their types, and which are required or optional. For example, an order schema might say that an event has an order number that is a whole number, a customer identity that is text, and a total price that is a decimal number.
Why a schema helps
It gives a shared contract
Producers and consumers agree in writing on what an event looks like. Nobody has to guess.
It catches mistakes early
If a producer tries to send an event that does not fit the schema, for example with a missing required field, the problem is caught before bad data enters Kafka.
It makes events small
With a schema shared in advance, field names need not be repeated in every record, so binary formats stay compact.
It documents your data
The schema is a living description that new team members can read to learn what flows through a topic.
A schema is like the blueprint of a building. Builders and inspectors both follow it. If a builder tries to put a door where the blueprint shows a wall, the inspector spots the difference at once.
Even when you use JSON, which does not require a schema, writing one down and checking events against it brings many of the same benefits.
7The Schema Registry
If schemas must be shared, someone has to keep them in one trusted place. That place is the schema registry.
What is a schema registry?
A schema registry is a separate service that stores and serves schemas. It gives every schema a unique number, called the schema identifier, and keeps every version of each schema over time. It also checks whether a new version of a schema is compatible with older ones, and refuses versions that would break readers.
How it works during sending and reading
The producer registers or looks up the schema
The first time it sends, the serializer asks the registry for the identifier of the schema. If the schema is new and acceptable, the registry stores it and returns a new identifier.
The serializer writes a small header plus the data
Each record starts with a tiny marker and the schema identifier, followed by the compact encoded data. Only a few extra bytes are added.
Kafka stores the bytes
The brokers keep the bytes as usual, unaware of the meaning of the identifier.
The consumer reads the identifier
The deserializer looks at the first bytes of each record to find which schema was used.
The consumer fetches the schema
It asks the registry for that schema, once, and keeps a copy in memory for later records.
The data is rebuilt
Using the schema, the deserializer turns the compact bytes back into meaningful data.
sequenceDiagram
participant P as Producer
participant R as Schema Registry
participant K as Kafka Cluster
participant C as Consumer
P->>R: Register or look up the schema
R-->>P: Schema identifier 42
P->>K: Bytes with identifier 42 and encoded data
C->>K: Fetch bytes
K-->>C: Bytes with identifier 42 and encoded data
C->>R: Fetch schema 42
R-->>C: The schema
Note over C: Consumer decodes the data using the schema
A library keeps an index card system. Every book carries a small catalogue number. When a reader picks up a book, the catalogue number tells them exactly where to find the reference notes explaining how to read it. The schema registry is the index, and the schema identifier is the catalogue number.
Treating the schema registry as an optional extra and then running it without care. If the registry is unavailable and consumers have never seen a schema before, they cannot decode those records. Run it reliably, and remember that clients cache schemas after the first use.
8Schema Evolution and Compatibility
Businesses change, and so do events. Schemas must be able to change without breaking everyone.
Why events change
Next year, your order events may need a new field for a discount code. Another field may become unnecessary. A number may need to become a decimal. These changes are called schema evolution. The danger is that old producers and new producers, and old consumers and new consumers, may all exist at the same time, especially in big organisations where teams deploy on their own schedules.
The compatibility rules
| Rule | What It Guarantees | Typical Safe Change |
|---|---|---|
| Backward compatible | New consumers can read data written with the older schema | Adding an optional field with a default value, or removing a field |
| Forward compatible | Old consumers can read data written with the newer schema | Adding a field that old readers simply ignore |
| Fully compatible | Both of the above hold at once | Adding or removing only optional fields with defaults |
| No compatibility check | Nothing is guaranteed | Any change, at the risk of breaking readers |
What is safe and what is dangerous
Usually Safe Changes
- Adding a new optional field with a default value
- Removing an optional field that has a default
- Adding documentation to a field
- Widening a number type in a way the format allows
Usually Dangerous Changes
- Renaming a field without an agreed alias
- Changing the type of a field to something incompatible
- Adding a required field with no default
- Reusing an old field number or name for a new meaning
Imagine a paper form used by a whole company. If you add an optional box at the bottom, old forms without it are still accepted, and new forms with it can still be read by clerks who ignore the extra box. If you change the meaning of an existing box, some clerks will misread the forms. Compatibility rules protect the meaning of the boxes.
flowchart LR
V1["Schema version 1: order number, customer, total"] --> V2["Schema version 2: adds optional discount code"]
V2 --> C1["Old consumer ignores discount code"]
V2 --> C2["New consumer reads discount code"]
V1 --> C3["New consumer reads old data with default"]
Choose the compatibility rule together with your consumers. If you can upgrade all consumers first, backward compatibility is enough. If you cannot control the consumers, full compatibility is the safer choice.
9Keys, Serialization, and Partitions
The way you serialize a key quietly decides where its events are stored.
The partition is calculated from bytes
The producer chooses a partition by running a fixed calculation on the bytes of the serialized key. The same bytes always give the same partition, as long as the partition count is unchanged. This means that two keys that look the same to a person, but produce different bytes, will go to different partitions.
How a small difference scatters events
Text versus number
One producer writes the customer number as text, another writes it as a binary number. The bytes differ, so the same customer’s events land in different partitions.
Extra spaces or different letter case
The keys “Alice” and “alice”, or “4711” and ” 4711″, produce different bytes and therefore different partitions.
Different serializer versions or libraries
Two applications using different libraries may encode the same value slightly differently, breaking the promise that the same key stays in the same partition.
If one sorting office writes postcodes with a space in the middle and another without, a machine may treat them as different codes and send the letters to different vans. Everyone must agree on the exact spelling.
Letting each team pick its own way of writing the same key. Agree on one key format for each topic, write it down, and use the same serializer everywhere so that order per key is preserved.
10When Deserialization Fails
Sometimes the consumer receives bytes it cannot understand. Knowing what to do about it saves whole nights of debugging.
Common reasons
Wrong deserializer
The consumer expects one format but the producer wrote another, such as expecting a compact binary format but receiving plain text.
Unknown schema
The record refers to a schema identifier that the consumer cannot fetch, perhaps because the registry is unreachable or the wrong registry is used.
Corrupted or malformed data
A buggy producer wrote bytes that do not follow the format at all.
Breaking schema change
A new schema version was accepted without a proper compatibility check, and old consumers cannot read it.
The poison pill problem
Consumers read a partition in order. If one record cannot be deserialized, the consumer throws an error. If the consumer simply retries the same record forever, the partition is blocked, and all events behind it wait. This troublemaker is called a poison pill. Meanwhile, lag grows, and healthy events are delayed.
Sensible ways to cope
Catch the error
Wrap deserialization so that a bad record raises a clear error instead of crashing the whole program.
Set the bad record aside
Copy the raw bytes, together with the topic, partition, offset, and error message, to a separate dead letter topic.
Move on
Commit the bookmark past the bad record and continue with healthy events.
Alert and investigate
Raise an alert so that people can find the cause, fix the producer, and replay the set-aside records if needed.
flowchart TD
R["Consumer receives a record"] --> Q{"Can it be deserialized?"}
Q -->|"Yes"| P["Process the event"]
Q -->|"No"| D["Send raw bytes and error details to dead letter topic"]
D --> A["Raise an alert"]
A --> N["Continue with the next record"]
P --> N
On a factory belt, one damaged item should not stop the whole line. A worker moves it to a problem bin, notes what is wrong, and lets the belt continue. The dead letter topic is that bin.
11Size, Speed, and Compression
The format you pick changes how much data moves and how hard your machines work.
Size affects everything
A larger event costs more in network traffic, more in disk space on every copy, and more in reading time. If an event grows from one kilobyte to three, and you have millions of events per hour and three copies of each, the difference becomes very large. Compact binary formats often make events several times smaller than the same content written as text.
Speed of encoding and decoding
Turning data into bytes and back costs processor time. Simple binary formats are usually faster than text formats that must be parsed character by character. For most applications this cost is small compared with the network and disk, but at very high volume it becomes noticeable.
Compression on top
Producers can also compress whole batches of records before sending. Text formats such as JSON compress very well, because they contain a lot of repetition, so much of their size disadvantage can be recovered. Binary formats compress less dramatically, since they are already compact. The best approach is to measure with your own data.
Sending clothes overseas, you can fold them loosely or vacuum-pack them. Vacuum-packing takes effort but saves space on every leg of the journey and every warehouse where the parcel is stored. Compact formats and compression are the vacuum-packing of data.
Keep events reasonably small. Very large payloads, such as whole images or long documents, are better stored elsewhere, with only a reference to them placed inside the Kafka event.
12Choosing a Format
There is no single best format. The right choice depends on your team, your data, and your future.
Who reads the data?
If many teams and languages consume it, a schema-driven format with a registry gives a clear shared contract.
How much data will flow?
At high volume, compact binary formats save real money on network and disk.
How often will events change?
If they change often, strong evolution rules protect you from breakage.
How important is human readability?
For quick debugging and learning, readable text is friendly, but tools can also display binary data.
What does your team already know?
A format the team understands well is safer than a clever one nobody has used.
| Situation | Good Choice | Reason |
|---|---|---|
| Learning Kafka, small experiments | Plain text or JSON | Easiest to read and test |
| Simple keys such as identifiers | String, integer, or UUID serializer | Small, stable, and easy to agree on |
| Business events shared by many teams | Avro or Protobuf with a schema registry | Compact, checked, and safe to evolve |
| Teams that love readable data but want discipline | JSON with a schema and registry | Readable and still checked |
| Very high volume telemetry | Compact binary with compression | Lowest cost per event |
Choosing plain text everywhere because it is easy on day one, and discovering a year later that thousands of events, dozens of consumers, and no shared rules make every change painful. Choose your format with the future in mind.
13Headers, Timestamps, and Other Parts of a Record
Besides key and value, a record has a few more parts that also need agreement.
Headers
Headers are optional labels that travel with the record, such as the application that produced it, a tracking number, or the version of the format used. Header names are text, and header values are bytes, so the same agreement is needed: both sides must know how to interpret them. Headers are a handy place to put information about the record without changing the value’s schema.
Timestamp
Every record carries a time. It is either set by the producer at creation, or added by the broker when the record is stored. It is stored as a number and is handled by Kafka itself, so it needs no serializer from you.
Header example: tracking
A tracing identifier placed in a header lets teams follow one request through many services and topics without touching the event’s main content.
Header example: format version
A header naming the version of the payload helps consumers choose the right way to decode during a gradual migration.
On a parcel, stickers such as “fragile” or “this way up” tell handlers how to treat it without opening the box. Headers are those stickers.
14Security and Privacy in Serialization
Turning data into bytes does not make it secret. Bytes can be read by anyone with access.
Serialization is not encryption
A compact binary format may look unreadable to a person, but any program with the schema can decode it easily. Serialization is only about representation, not protection. To keep data private, connections between clients and brokers should be encrypted, disks should be protected, and access to topics and to the schema registry should be controlled.
Sensitive fields
Some events contain sensitive personal data, such as identity numbers, addresses, or card details. Teams often handle these by leaving them out, by replacing them with tokens, or by encrypting specific fields before serialization so that only authorised readers can decrypt them. Remember that keys are stored openly with every record, so sensitive data should never be used as a key.
Protecting the registry
The schema registry is a valuable target. Anyone who can change a schema could break every consumer. Access to register new schemas should be limited, changes should follow a review process, and the compatibility rule should be enforced.
Writing a letter in a compact shorthand does not make it private, because anyone who knows the shorthand can read it. To keep it secret, you seal the envelope and lock the post box. Encryption and access control are the seal and the lock.
Believing that binary formats hide personal data. They do not. Treat every event as readable by anyone who can access the topic, and protect sensitive fields separately.
15Advantages and Trade-offs of Schema-Based Serialization
Structured, schema-driven formats bring discipline, and discipline has a price.
Advantages
- Small events that save network, disk, and cost
- A shared, written contract between producers and consumers
- Bad data is caught before it enters Kafka
- Safe rules for changing events over time
- Data is documented by its own schema
- Works across many programming languages
Disadvantages / Trade-offs
- An extra service, the registry, must be run and protected
- A learning curve for schemas and compatibility rules
- Binary data is unreadable without tools
- Changes need review and discipline instead of quick edits
- Consumers depend on the registry for schemas they have not yet cached
16Anti-Patterns and Best Practices
Most serialization problems come from the same handful of mistakes.
Problem
Every producer team invents its own event format, with no shared description.
Why It’s Harmful
Consumers must guess or ask, changes silently break readers, and bugs appear far from their cause.
Correct Approach
Agree on a schema for each topic, store it in a registry, and require producers to follow it.
Problem
Changing the meaning or type of an existing field without a compatibility check.
Why It’s Harmful
Old consumers misread or fail on new records, and historical data becomes inconsistent.
Correct Approach
Add new optional fields with defaults, retire old ones gradually, and let the registry enforce compatibility.
Problem
Mixing several different formats or event types in the same topic without any marker.
Why It’s Harmful
Consumers cannot tell what they are receiving and crash on the unexpected shapes.
Correct Approach
Keep one clear format per topic, or clearly label each type of event through the schema or a header.
Problem
Stopping a consumer forever on a record that cannot be decoded.
Why It’s Harmful
One poison pill blocks its partition, lag grows, and healthy events are delayed.
Correct Approach
Catch the error, send the raw bytes and error details to a dead letter topic, alert, and continue.
Problem
Putting huge payloads, such as entire documents or images, inside the event value.
Why It’s Harmful
Large records strain brokers, slow every reader, and multiply storage across replicas.
Correct Approach
Store big files elsewhere and place only a reference and key details in the event.
Best practices at a glance
One clear format per topic
Decide it early and write it down.
Use schemas and a registry
Make the contract explicit, versioned, and checked.
Enforce compatibility
Let the registry reject breaking changes.
Standardise key formats
Same serializer, same spelling, same case for every producer.
Plan for bad records
Use error handling and a dead letter topic for records that cannot be decoded.
Keep events small
Send what readers need and reference large files.
Protect sensitive data
Serialization is not secrecy. Encrypt or leave out sensitive fields.
Test changes before release
Check that old and new consumers can read new events.
17A Real-World Walk-Through
Let us follow the life of an order event in an online shop from creation to reading.
The team agrees on the contract
Order, payment, and email teams agree on a schema for order events: an order number, a customer identifier, a list of items, a total, and an optional discount code. The schema is stored in the registry with full compatibility enforced.
The producer serializes
When a customer buys something, the order service serializes the key, the customer identifier, as plain text in an agreed format, and the value in a compact binary format with the schema identifier at the front.
Kafka stores the bytes
The brokers store and replicate the record without caring what it means.
Three consumers decode
The email, fraud, and analytics services each read the record, fetch the schema once from the registry, cache it, and rebuild the order in their own programming languages.
The business changes
Marketing wants to record the campaign that led to each order. The order team adds an optional campaign field with a default value. The registry checks compatibility and accepts it.
Old and new work together
The email service, which has not been updated, ignores the new field and continues. The analytics service is upgraded later and starts using it. Nothing breaks.
A bad record arrives
A faulty test tool writes a badly formed record into the topic. Each consumer catches the failure, sets the record aside in a dead letter topic with the error details, alerts the team, and carries on with real orders.
flowchart LR
OS["Order Service"] -->|"key as text, value in compact binary with schema id"| K["Kafka: orders topic"]
OS --> R["Schema Registry"]
K --> EM["Email Service"]
K --> FR["Fraud Service"]
K --> AN["Analytics Service"]
EM --> R
FR --> R
AN --> R
K --> DL["Dead Letter Topic for undecodable records"]
18Frequently Asked Questions
It is the conversion of an application’s data into bytes before sending to Kafka, and deserialization is the conversion of those bytes back into meaningful data when reading.
No. The producer and consumer clients do it, using serializers and deserializers you configure. The brokers only store and move bytes.
Yes, and it is very common. Keys are often simple text or numbers, while values use a structured format such as Avro, Protobuf, or JSON.
No, but it is strongly recommended for shared business events. Without it, you must find another way to share schemas and to prevent breaking changes.
It is a fine choice for learning, low volume, and readability. For high volume or many consumers, a compact schema-driven format usually saves money and prevents surprises, though JSON with a schema is a decent middle path.
Decoding fails or produces nonsense. Consumers then throw errors on the affected records, which can block a partition unless the errors are handled properly.
It is possible, but hard. Producers and consumers must change together, or old and new formats must coexist for a while, for example in a new topic. It is much easier to choose carefully from the start.
No. It only changes the representation. Use encrypted connections, access control, and field-level protection for sensitive data.
Because the partition is calculated from the key’s bytes. Different encodings of the same key send events to different partitions and break the order for that key.
A record that a consumer cannot deserialize. If the consumer keeps retrying it, the partition stays blocked. Catch the error, set the record aside in a dead letter topic, and continue.
19Summary and Key Takeaways
We began with a toy model that must be taken apart, packed flat with an instruction sheet, and rebuilt by a friend who knows the same instructions. Serialization in Kafka is exactly that. Kafka stores only bytes, so producers serialize the key and the value on the way in, and consumers deserialize on the way out, with the brokers never looking inside. Simple built-in serializers cover keys and small values, while structured formats such as JSON, Avro, and Protobuf carry richer events. Schemas and a schema registry turn the format into a shared, versioned, and checked contract, and compatibility rules let events change without breaking old or new readers. The key format decides partition placement, bad records must be handled without blocking a partition, and serialization is never a substitute for security. Choose a format with the future in mind, agree on it in writing, and your data will keep its meaning as your systems grow.
Key Takeaways
- Kafka stores only bytes — producers serialize and consumers deserialize, while brokers treat the data as opaque parcels.
- Keys and values have their own serializers — keep keys simple and consistent, because the partition is calculated from the key’s bytes.
- Pick the format on purpose — plain text and JSON are easy to read, while Avro and Protobuf are compact, fast, and strongly structured.
- Schemas are the shared contract — a schema registry stores every version, gives each an identifier, and checks compatibility.
- Evolve safely — add optional fields with defaults, avoid renaming or retyping fields, and let compatibility rules protect old and new readers.
- Plan for undecodable records — catch the error, send the raw bytes to a dead letter topic, alert, and keep the partition moving.
- Serialization is not security — protect sensitive data with encryption and access control, and never put it in a key.