Understanding why enforcement exists starts with a surprising fact about Kafka itself.

Kafka stores bytes, not meaning

Apache Kafka stores streams of events in topics. Producers write events, consumers read them, and brokers, the servers that form the Kafka cluster, store and deliver them. A key design choice is that the broker treats every event as an opaque sequence of bytes. It does not know whether the bytes describe an order, a temperature or a photograph. This makes Kafka fast and flexible, but it also means Kafka, on its own, will happily accept any bytes from any producer.

What goes wrong without agreement

When producers and consumers are built by different teams, they must agree on the structure of the data: which fields exist, what type each field has and which fields are optional. Without a central place to record and check this agreement, several problems appear over time.

Two ideas form the foundation: the schema, which describes the data, and the registry, which stores and guards those descriptions.

A schema

A schema is a formal description of the structure of a piece of data. It lists the fields, names each one, states its type such as text, whole number or true and false, and says whether it is required or optional. For an order event, a schema may say that there is an order identifier, a customer identifier, a total amount and an optional coupon code.

A schema registry

A schema registry is a separate service that stores schemas, gives each one a unique number called a schema ID, keeps track of their versions over time, and checks whether a new version is allowed. It is not part of the Kafka brokers themselves. Applications talk to it over a simple web interface. The best known implementation is the Confluent Schema Registry, and there are alternatives from the open-source community and from cloud providers that follow similar ideas.

flowchart LR
  P["Producer"] -->|"1. register or look up schema"| R[("Schema Registry")]
  P -->|"2. send event with schema ID"| K["Kafka topic"]
  K -->|"3. deliver event"| C["Consumer"]
  C -->|"4. fetch schema by ID"| R
        

Where the registry keeps its data

The registry stores its schemas in an internal Kafka topic, so it inherits Kafka’s durability. Several registry instances can run together, with one acting as the leader that handles writes. Because schemas are small and change rarely, the registry is lightweight compared with the Kafka cluster.

Having a registry is not the same as enforcing it. Enforcement is the set of checks that make the rules unavoidable.

Schema Registry enforcement means making sure that only data that follows a registered, approved schema can be written to a topic, and that any change to a schema respects the compatibility rules the organisation has chosen. It answers two separate questions. First: is this event valid according to a known schema? Second: is this new version of the schema safe for everyone who depends on the older versions?

Enforcement is a stack of layers

No single switch enforces everything. Enforcement is built from several layers, each catching a different kind of mistake at a different moment. The earlier a mistake is caught, the cheaper it is to fix.

flowchart TB
  L1["Layer 1: Build pipeline checks, before code ships"] --> L2["Layer 2: Registry compatibility rules, when a schema is registered"]
  L2 --> L3["Layer 3: Client serializer checks, when an event is produced"]
  L3 --> L4["Layer 4: Broker schema validation, when an event arrives"]
  L4 --> L5["Layer 5: Governance and access control, who may change what"]
        

The trick that makes registries efficient is that events carry a tiny reference instead of the full schema.

The shape of an event on the wire

When a producer uses the registry-aware serializer, each event is written in a small envelope. The first byte is a fixed marker that tells readers this event follows the registry format. The next four bytes hold the schema ID. Everything after that is the actual data, encoded compactly using the chosen format. Because the field names and types are stored once in the registry, each event carries only values.

What the producer does

The serializer first checks its local memory to see whether it already knows the schema ID for this data shape. If not, it contacts the registry, either to register the schema or to look up an existing one, depending on configuration. It remembers the answer, so later events need no further calls. It then writes the marker, the ID and the encoded data, and sends the event to Kafka.

What the consumer does

The deserializer reads the marker and the schema ID from the first five bytes. If it has not seen that ID before, it asks the registry for the matching schema and remembers it. It then decodes the rest of the event using that schema, and, if the consumer application expects a slightly different version, resolves the differences between the writer’s schema and its own reader schema.

sequenceDiagram
  participant P as Producer
  participant R as Schema Registry
  participant K as Kafka
  participant C as Consumer
  P->>R: Register or look up schema
  R-->>P: Schema ID 17
  P->>K: Event with ID 17 and data
  K-->>C: Event with ID 17 and data
  C->>R: What is schema 17?
  R-->>C: Schema definition
  C->>C: Decode data using the schema
        

Schemas are organised into named groups so that each evolves independently and its history is kept.

Subjects

A subject is a named container in the registry that holds the history of one schema as it evolves. Each time a new, accepted version is registered under a subject, it receives the next version number, and a globally unique schema ID. Compatibility rules are applied per subject, comparing new versions against earlier versions in the same subject.

Versions versus IDs

People often mix up versions and IDs. A version is a counter within a subject: first, second, third. An ID is a number unique across the whole registry. The same schema text registered under two subjects shares one ID but is version one in each. Events carry the ID, while humans talk about versions.

Naming strategies

A naming strategy is the rule for choosing the subject name when producing. It decides how strictly a topic is tied to a schema.

The registry supports several ways of describing data, each with its own strengths.

Choosing a format

Pick the format that your teams already know and that fits your performance and tooling needs. Consistency across the organisation matters more than small differences, because every extra format multiplies the rules, libraries and training needed.

When someone registers a new version of a schema, the registry checks it against earlier versions using a chosen compatibility mode.

Two directions of compatibility

Compatibility is about who can read whose data. Backward compatible means that a consumer using the new schema can still read data written with the older schema. Forward compatible means that a consumer still using the older schema can read data written with the new schema. Full compatibility means both directions hold.

Compatibility modes are abstract until you see what real changes they allow or block.

Changes that are usually safe

Changes that usually break something

The order of upgrades

Compatibility also tells you who to upgrade first. Under backward rules, upgrade consumers first so they can already read both old and new data, then upgrade producers. Under forward rules, upgrade producers first, since old consumers can already read the new data. Under full rules, either order works.

flowchart LR
  subgraph B["Backward compatible"]
    B1["Upgrade consumers first"] --> B2["Then upgrade producers"]
  end
  subgraph F["Forward compatible"]
    F1["Upgrade producers first"] --> F2["Then upgrade consumers"]
  end
  subgraph U["Full compatible"]
    U1["Upgrade in any order"]
  end
        

The first line of defence is the serializer inside the producer application.

What the client checks

When the producer’s serializer receives an object, it converts it into bytes according to a schema. If the object does not match the schema, for example a required field is missing or a value has the wrong type, the serializer fails right there and the event is never sent. This stops malformed data at the source, which is the cheapest place to stop it.

Automatic registration

By default, many serializers automatically register the schema they use if it is not yet in the registry. This is convenient in development, but it means any application can create or change contracts simply by running. In production, many organisations switch automatic registration off, so that schemas are registered only through an approved process, and applications can only use schemas that already exist.

Using the latest approved version

Producers can be told to use the latest version registered for a subject, rather than deriving a schema from their own code. This keeps the code from silently drifting from the approved contract. If the application’s data does not fit that version, serialization fails, which alerts the team early.

The registry itself is the referee that decides whether a new schema version may exist at all.

Compatibility checks on registration

When a schema is registered under a subject, the registry compares it with the earlier versions according to the subject’s compatibility setting. If the new version breaks the rule, the registry refuses it and returns an error explaining why. The bad version never receives an ID, so no event can ever carry it. This is the single most important enforcement point for schema evolution.

Global and per-subject settings

A default compatibility level applies to the whole registry, and individual subjects may override it. A good practice is to set a strict global default and permit exceptions only with a clear reason and an owner.

Registry modes

The registry can also be put into different operating modes. In the normal read-write mode, schemas may be added. In read-only mode, no new schemas or changes are accepted, which is useful for locking down production or during migrations. An import mode allows schemas to be loaded with specific IDs when copying from another registry.

Immutability of registered schemas

A registered version is not edited in place. To change a contract, you register a new version. This immutability is what lets events safely carry an ID that always means the same thing.

To stop clients that bypass the serializer, the brokers themselves can check every incoming event.

What it is

Broker-side schema validation is a feature of the Confluent Server distribution of Kafka, which is a commercial platform edition and not part of the plain open-source Apache Kafka broker. When it is turned on for a topic, the broker connects to the registry, and for every event it receives it checks that the event carries a schema ID, and that this ID is registered under the subject that belongs to that topic according to the naming strategy. If not, the broker rejects the event and the producer receives an error.

What it does not do

It is important to understand the limits. The broker checks the schema ID and its association with the subject. It does not open the payload to confirm that the bytes truly match the schema. A client that stamps a valid ID on nonsense bytes would pass the broker check. Broker validation therefore stops raw, unregistered or wrongly-labelled data, but it is not a full content inspection.

flowchart TD
  E["Event arrives at broker"] --> Q1{"Has a schema ID?"}
  Q1 -- "No" --> X1["Reject event"]
  Q1 -- "Yes" --> Q2{"ID registered under the topic's subject?"}
  Q2 -- "No" --> X2["Reject event"]
  Q2 -- "Yes" --> OK["Accept and store event"]
        

The cheapest time to find a breaking change is before the code ever runs in production.

Schemas as code

Treat schema files like source code: keep them in version control, review changes with pull requests, and give each schema an owner. This gives a history of who changed what and why, and lets the team discuss a change before it is applied.

Compatibility tests in the pipeline

The registry offers a way to test whether a proposed schema would be accepted without actually registering it. A build pipeline can call this check for each changed schema. If the test says the change is incompatible, the build fails and the developer sees the problem within minutes, not after a production incident.

Controlled registration

After a change is reviewed and merged, a deployment step registers the schema in the registry using an account that has permission to do so. Applications themselves then only need permission to read. This separates the power to change contracts from the power to use them.

flowchart LR
  D["Developer edits schema"] --> PR["Pull request and review"]
  PR --> CT["Pipeline runs compatibility test"]
  CT -- "Incompatible" --> F["Build fails, fix the change"]
  CT -- "Compatible" --> M["Merge"]
  M --> RG["Deploy step registers schema"]
  RG --> APP["Applications use approved schema"]
        

Rules mean little if anyone can change them, so the registry needs its own security.

A schema says what shape data has. A data contract also says what values are acceptable and who is responsible.

The limit of structure alone

A schema can say that an age field is a whole number, but it cannot say that the number must be between zero and one hundred and fifty. A structurally valid event can still be meaningless. Recent versions of schema registry platforms extend schemas into data contracts by adding more information beside the structure.

What a data contract can carry

Handling invalid events gracefully

When a rule fails, throwing away the event or crashing the pipeline are both harsh choices. A common approach is to route the failing event to a dedicated error topic, called a dead letter topic, along with a reason. Teams can then inspect, fix and reprocess these events without blocking the good ones.

A worked example in words shows how the layers cooperate when a team adds a field.

Sometimes a real breaking change is unavoidable. The enforcement system does not forbid it. It forces you to do it carefully.

The new-topic approach

The most common safe method is to create a new topic with a new subject and a new, incompatible schema, then move producers and consumers across gradually. During the transition, a small translation application can read the old topic, convert events and write them to the new topic, so both sides continue to work. Once all consumers have moved, the old topic is retired.

flowchart LR
  OP["Old producers"] --> OT["Old topic, old schema"]
  OT --> TR["Translator application"]
  TR --> NT["New topic, new schema"]
  NP["New producers"] --> NT
  OT --> OC["Old consumers, migrating"]
  NT --> NC["New consumers"]
        

Migration rules

On platforms that support migration rules, a breaking change can be published as a new schema version together with a rule describing how to convert between the old and new shapes. Readers on either version then get data converted for them, allowing a smooth changeover without a separate topic.

Never bypass the check by weakening the rule

When the registry rejects a change, the tempting shortcut is to switch the compatibility level to none, register the change, and switch it back. This defeats the entire purpose and is how most schema-related outages begin. Use it only in a genuine emergency, with sign-off, and with a plan to repair the damage.

Adding a registry adds a dependency, so we must understand what happens if it is slow or down.

What still works when the registry is down

Because clients cache schemas, a producer or consumer that already knows the schemas it needs can continue working during a registry outage. Existing schema IDs resolve from local memory, and events flow as usual. This makes short outages much less harmful than they might seem.

What stops working

New producers or consumers starting up with empty caches cannot fetch schemas, and cannot start processing. A producer that needs to register or look up a new schema cannot proceed. If broker-side validation is on, the broker needs the registry to validate unfamiliar IDs, so brokers may reject events until the registry returns, unless their caches already cover the schemas in use.

Designing for resilience

Real organisations run development, test and production, and often several clusters. Schemas must travel with the data.

Keeping environments aligned

Each environment normally has its own registry. Because schema IDs are assigned by each registry, the same schema may have different IDs in different environments. Promoting a schema from test to production means registering it again in production through the same pipeline, not copying IDs by hand.

Replicating data across clusters

When events are copied from one Kafka cluster to another, the schema IDs inside them must still resolve on the other side. There are two approaches. One keeps a single shared registry that both clusters use. The other synchronises schemas between registries, preserving IDs, using features such as schema linking on platforms that provide it, or the import mode described earlier.

Schema contexts

Some registries support contexts, which are separate namespaces inside one registry. They allow schemas from different sources or environments to live side by side without their names or IDs clashing.

Schemas are meant to be permanent, so deletion needs special care.

Soft and permanent deletion

Registries usually offer two stages of deletion. A soft delete hides a version or a subject from normal listings but keeps the underlying schema so existing events can still be decoded. A permanent delete removes it completely, after which any event carrying that ID becomes unreadable. Permanent deletion should be extremely rare and follow a soft delete.

When deletion is justified

Deleting makes sense for experimental schemas that never carried real data, or when a topic has been fully retired and all its retained events have expired. It does not make sense while any topic still holds events written with that schema, including archives or backups.

Enforcement is only useful if you can see it working and notice when it does not.

Auditing who changed what

Keep logs of every registration, compatibility change, mode change and deletion, with the caller’s identity. Alert on changes to compatibility levels and modes in production, since these are the moves that weaken enforcement.

Dead letter topics as a signal

If invalid events are routed to a dead letter topic, its volume is a valuable quality indicator. A sudden rise usually points to a producer change or a broken upstream system.

Practices that keep contracts healthy, and the traps that quietly undo them.

Patterns worth adopting

Anti-patterns to avoid

Turning on every layer at once in a busy organisation causes friction. A staged approach works better.

flowchart TD
  A["Stage 1: Inventory topics and formats"] --> B["Stage 2: Adopt registry and serializers for new topics"]
  B --> C["Stage 3: Set compatibility levels and owners"]
  C --> D["Stage 4: Add pipeline checks and controlled registration"]
  D --> E["Stage 5: Disable auto-registration in production"]
  E --> F["Stage 6: Turn on broker validation for critical topics"]
  F --> G["Stage 7: Add rules, monitoring and audits"]
        

A checklist to keep close when designing or reviewing enforcement.

Five common situations show enforcement at work.

No. Brokers treat events as bytes. Structural checks come from serializers, the schema registry and optional broker-side validation on platforms that support it.

No. Each message carries a small marker and a schema ID. The schema itself lives in the registry and is fetched once and cached.

Backward is a common default. If consumers may replay old data, use a transitive version. If you cannot control upgrade order across many teams, full transitive gives the most freedom. Avoid none except for short-lived experiments.

The serializer fails and the event is not sent. This stops bad data at the source, before it reaches Kafka.

Yes, if nothing on the broker side checks. A client that writes raw bytes can send them to the topic. Broker-side validation and access control on producing reduce this risk.

No. It checks that the event carries a schema ID registered under the topic’s subject. It does not confirm that the payload truly follows the schema.

No. It is a feature of a commercial platform edition of the broker, not of the plain open-source Apache Kafka broker, so check what your distribution offers.

Running applications continue using their cached schemas. New starts, new schema registrations and validation of unfamiliar IDs are affected until it returns. Running several instances lowers the risk.

Only in limited ways allowed by the format’s rules, such as certain widenings. Most type changes break compatibility, and are best handled by adding a new field or a new topic.

Create a new topic and subject, bridge with a translator, migrate consumers, and retire the old topic. Some platforms also allow migration rules that convert between versions.

Convenient in development, but risky in production. Prefer registering through a reviewed pipeline and giving applications read-only access.

Not exactly. A schema is the structural part. A data contract adds ownership, metadata, value rules and handling of failures, and builds on the registry.

Kafka moves bytes without understanding them, so agreements about the shape of data must be recorded and enforced elsewhere. A schema registry stores schemas with versions and IDs, and lets producers and consumers refer to a contract by a tiny number. Enforcement is the practice of making those contracts unavoidable, through several layers: pipeline checks, registry compatibility rules, client serializers, broker-side validation and access control.

Good enforcement is strict where data is shared, automated where humans forget, and observable where things can go wrong. If you remember one sentence, let it be this: the registry keeps the contracts, but only the layers you switch on will make anyone obey them.