Before worrying about availability, understand what the registry does and why so many systems lean on it.

Apache Kafka stores and moves messages, which it calls records. Kafka itself treats the body of a record as a bag of bytes. It does not know or care whether those bytes describe a customer, an order, or a temperature reading. That freedom is powerful, but it creates a problem. The team writing records and the many teams reading them must agree on what the bytes mean. If the writer quietly renames a field or changes a number into text, every reader can break without warning.

A schema is a formal description of the structure of a record: which fields exist, what type each field has, and which fields are optional. Common schema formats include Avro, Protocol Buffers, and JSON Schema. The Schema Registry is a separate service that stores these schemas, gives each one a unique number, and checks new versions against rules to make sure changes do not break existing readers.

High availability is a promise about staying useful during failures. It helps to define exactly which promises matter for a registry.

High availability means that when a component fails, the system as a whole continues to serve its users, ideally without a human doing anything. For the registry, users fall into two groups with different needs. Readers are producers and consumers that look up schemas. Writers are engineers, deployment pipelines, and applications that register new schema versions. Most of the traffic is reading, and writing is rare and bursty.

The registry makes a deliberate choice to protect consistency above all. It guarantees that a schema ID is never reused for a different schema, even at the cost of refusing writes when it cannot be sure it is safe. Reads, on the other hand, are designed to be available from many servers at once. This asymmetry, safe writes through one gate and scalable reads through many, is the heart of its high availability design.

The registry keeps no database of its own. It uses a Kafka topic as its durable storage.

A Schema Registry server is a small web application. When you register a schema, the server does not write it to a local disk or an external database. Instead, it writes it as a record into a special Kafka topic, commonly named after an underscore and the word schemas. This topic is the single source of truth. Every registry instance reads the topic from the beginning when it starts, builds an in-memory copy of all schemas, and then keeps following the topic to stay current.

flowchart TB
    subgraph K["Kafka Cluster"]
        T["Internal schemas topic: compacted, single partition, replicated"]
    end
    subgraph SR["Schema Registry Instances"]
        P["Primary instance"]
        S1["Secondary instance 1"]
        S2["Secondary instance 2"]
    end
    P -->|"only the primary writes"| T
    T -->|"all instances read and follow"| P
    T -->|"all instances read and follow"| S1
    T -->|"all instances read and follow"| S2
    C1["Producer"] --> S1
    C2["Consumer"] --> S2
    C3["Admin or pipeline"] --> P
        

Why a topic makes a good store

A registry cluster has exactly one instance allowed to write at any moment. The rest serve reads and forward writes.

When several registry instances run together, they elect one of themselves as the primary. Older documentation and some setting names call this the leader or the master. The primary is the only instance that writes to the schemas topic. The other instances are secondaries, sometimes called followers. Secondaries serve read requests directly from their in-memory copy. When a secondary receives a write request, such as registering a new schema, it forwards the request to the primary over the network, and returns the answer to the client.

sequenceDiagram
    participant Cl as Client
    participant S as Secondary instance
    participant P as Primary instance
    participant K as Schemas topic
    Cl->>S: Register a new schema
    S->>P: Forward the request
    P->>P: Check compatibility and assign the next ID
    P->>K: Write the schema record
    K-->>P: Acknowledged by replicas
    P->>P: Read back own write to update memory
    P-->>S: Result with the new ID
    S-->>Cl: Result with the new ID
        

Why only one writer

Schema IDs must be unique and must grow in a strict order. If two instances could assign IDs at the same time, they might hand out the same number to two different schemas, and data written with that number would become unreadable or, worse, silently misread. A single writer avoids this entire class of problem without needing complicated agreement protocols between instances.

Electing a primary without any extra software is done by borrowing a feature Kafka already has: consumer group coordination.

Every registry instance is configured with the same group identifier. When instances start, each joins a group through the Kafka cluster’s group coordinator, exactly like ordinary consumers do. Kafka’s group protocol picks one member of the group as the group’s leader and tells everyone who it is. The registry uses that result to decide which instance is the primary. No separate coordination service is needed in modern versions, so one less system has to be kept healthy.

stateDiagram-v2
    [*] --> Joining
    Joining --> Secondary: Group formed and not selected
    Joining --> CatchingUp: Selected as primary
    CatchingUp --> Primary: Caught up with the topic
    Secondary --> CatchingUp: Previous primary lost and this one selected
    Primary --> Joining: Session lost or group rebalanced
    Secondary --> Joining: Session lost or group rebalanced
        

What happens when the primary dies

If the primary crashes or loses contact with Kafka, the group coordinator notices that its session has expired after a timeout and starts a new round of group formation. A new primary is selected among the eligible survivors. During the gap, which typically lasts for seconds, read requests continue to work on every surviving instance, but write requests fail or wait, because nobody is authorized to write.

Since the topic is the only place schemas live, protecting it is the most important availability decision you will make.

Running fifty registry instances would not help if the schemas topic sat on a single broker with one copy. The durability and availability of the registry can never exceed those of its topic. Treat this topic as the crown jewels of your data platform, with settings that are stricter than for ordinary topics.

The registry usually creates the topic itself

On first start, if the topic does not exist, the registry creates it using its own setting for replication factor. Many installations rely on the default, which may be lower than three on development clusters, and then copy that setup to production by accident. Always verify the topic’s actual replication factor and minimum in-sync setting after the first start, and correct them if needed.

Clients are surprisingly resilient, because they cache aggressively. Understanding the cache tells you what breaks and what does not.

The serializer inside a producer and the deserializer inside a consumer both remember the schemas they have already seen. A producer asks the registry for an ID once per schema and remembers it. A consumer asks the registry for the schema behind an ID once and remembers it. After that, the registry is not touched for that schema. In a stable system where schemas rarely change, the vast majority of messages never trigger a registry call at all.

flowchart TD
    A["Application needs a schema ID or schema"] --> B{"Already in the local cache?"}
    B -->|"Yes"| C["Use it immediately, no network call"]
    B -->|"No"| D["Call the registry"]
    D --> E{"Registry reachable?"}
    E -->|"Yes"| F["Store in cache and continue"]
    E -->|"No"| G["Retry, then fail the operation"]
        

The dangerous moments

The risky times are those when many caches go cold together. A mass restart of consumers, a large deployment, an autoscaling burst, or a cluster-wide rebalance can suddenly send thousands of new instances to the registry at once. If the registry is unavailable or slow at that moment, a stable system turns into a stalled one. Registry availability therefore matters most during change, not during calm.

Clients need a way to find a healthy instance. There are two common approaches, and many teams combine them.

The client setting that names the registry accepts a list of addresses separated by commas. When one address fails, the client tries the next. This is simple and needs no extra infrastructure. The alternative is to put a load balancer or a DNS name in front of all instances and give clients that single address. The load balancer checks the health of each instance and sends traffic only to healthy ones.

flowchart LR
    C1["Clients"] --> LB["Load balancer or service address"]
    LB -->|"health checked"| I1["Instance 1"]
    LB -->|"health checked"| I2["Instance 2"]
    LB -->|"health checked"| I3["Instance 3"]
    I1 --> T["Schemas topic in Kafka"]
    I2 --> T
    I3 --> T
        

What a good health check looks like

The registry exposes a simple web address that returns a success answer when the instance is running and able to serve. A good health check calls that address frequently and removes the instance from rotation after a few consecutive failures. It should also consider an instance unhealthy while it is still reading the schemas topic on startup, because serving reads from an incomplete copy would return misleading not-found errors.

Sticky or not

Balancers need not stick a client to one instance. Any instance can answer any read, and writes are forwarded internally. Simple round robin or least connections is enough. One exception is worth noting: because instances catch up with the topic independently, two instances can differ by a tiny moment in what they know, which the next chapter explains.

Because secondaries follow the topic, they can lag the primary by a few moments. That is usually harmless, but it explains some puzzling errors.

When the primary registers a schema, it writes to the topic and then reads its own write back before answering. Secondaries also read that record from the topic, but they do it independently and slightly later. For a brief window, perhaps tens or hundreds of milliseconds under normal conditions, a secondary does not yet know about the new schema. If a client asks that secondary for the new ID in that window, it may receive a not-found answer.

sequenceDiagram
    participant Pr as Producer
    participant P as Primary
    participant K as Schemas topic
    participant S as Secondary
    participant Co as Consumer
    Pr->>P: Register new schema
    P->>K: Write record
    P-->>Pr: New ID returned
    Pr->>Pr: Sends data using the new ID
    Co->>S: Ask for schema by the new ID
    S-->>Co: Not found yet, still catching up
    K-->>S: Record arrives shortly after
    Co->>S: Retry the lookup
    S-->>Co: Schema returned
        

Why this rarely hurts

In most systems, the time between a producer registering a schema and a consumer reading a message that uses it is much longer than the lag between instances. Messages first have to be written to Kafka, replicated, and then fetched by the consumer. The consumer’s lookup almost always happens after all instances have caught up. Occasional transient not-found errors are more likely under heavy load or when an instance is recovering.

The registry is light on resources. The main design questions are how many instances, and where to put them.

A registry instance is a small web service that holds all schemas in memory and answers mostly simple lookups. Even large organizations with thousands of schemas run it comfortably on modest machines. The bigger risk is not capacity but placement: running all instances on the same machine, the same rack, or the same zone.

A subtle requirement causes many real incidents: every instance must be able to reach the primary directly.

When a secondary forwards a write, it connects to the primary using the address the primary announced when it joined the group. That address comes from each instance’s own configuration for its advertised host name and listener. If the address is wrong, unreachable from other instances, or resolves to a private address that others cannot use, then forwarding fails, even though every instance can reach Kafka perfectly well.

Planned changes are the most common cause of registry trouble. A careful procedure removes almost all of it.

On startup an instance must read the entire schemas topic to build its in-memory copy. For a registry with a modest number of schemas this takes seconds. For very large registries, with hundreds of thousands of records, it can take longer, and the instance has a configurable time limit for this initial reading. If Kafka is slow or unreachable, the instance may fail to start within the limit and exit.

One cluster spread across zones handles most failures. Surviving the loss of an entire region needs a bigger design.

Inside one region, the standard design places three registry instances in three zones, backed by a Kafka cluster that also spans those zones. This survives any single instance, machine, or zone failure with no manual action. Surviving the loss of a whole region requires a second Kafka cluster and a second set of registry instances in another region, with schemas copied between them.

flowchart LR
    subgraph RA["Region A: active"]
        KA["Kafka cluster A"]
        RA1["Registry instances, eligible to be primary"]
        RA1 --> KA
    end
    subgraph RB["Region B: standby"]
        KB["Kafka cluster B"]
        RB1["Registry instances, not eligible to be primary"]
        RB1 --> KB
    end
    KA -->|"schemas replicated continuously"| KB
    AppsA["Applications in Region A"] --> RA1
    AppsB["Applications in Region B"] --> RB1
        

The eligibility setting

Each instance has a setting that says whether it may become primary. In a standby region, you set instances as not eligible, so they can never accidentally elect a local primary and start writing to their copy of the topic. They still serve reads to local applications. During failover, an operator flips the standby to eligible and points writers at it.

Two numbers define a disaster recovery plan: how much you can afford to lose, and how long you can afford to be down.

Recovery point objective, shortened to RPO, is the maximum amount of recent data you are willing to lose. Recovery time objective, shortened to RTO, is the maximum time you are willing to be without service. For schemas, the RPO question is: if the active region disappears at this instant, which recently registered schemas might not yet have reached the standby? The RTO question is: how long does it take to turn the standby into the active one?

The registry has operating modes that make copying and freezing schemas safe.

A registry can run in different modes, applied globally or per subject. In the normal read-write mode, clients can register and read schemas. In read-only mode, registrations and changes are refused while reads continue, which is useful during maintenance or in standby copies. In import mode, the registry accepts schemas together with their existing IDs instead of assigning new ones, which is what makes it possible to copy schemas between registries while preserving their numbers.

Container platforms make it easy to run several instances, and equally easy to run them badly.

Managed services

Many cloud providers and Kafka vendors offer a managed registry that handles instance counts, placement, upgrades, and storage for you. This removes most of the operational work described in this guide, but you still own client configuration, schema governance, access control, and your own disaster recovery expectations. Check the provider’s availability promise and how failover works, and confirm that the schema IDs are stable across provider-side maintenance.

Security settings interact with availability more than people expect. A locked door that no one can open is also an outage.

You cannot keep something available if you cannot see it becoming unavailable.

An alerting ladder

These arrangements have been proven again and again in production.

These mistakes appear in real incidents. Learn them so you do not repeat them.

Most registry incidents fall into a few families. Recognize the family and the fix is usually short.

A calm debugging routine

The same design plays out differently depending on the business.

A compact checklist you can use before, during, and after rollout.

A short readiness checklist

Short answers to the questions people ask most often when they start planning registry availability.

In a special Kafka topic. The registry instances keep an in-memory copy built from that topic, so the topic is the only durable store and its safety decides the registry’s safety.

No. Only the elected primary writes to the topic. Secondaries accept write requests but forward them to the primary, and they answer read requests themselves.

Instances join a group through Kafka’s group coordination, and the result of the group’s leader selection decides the primary. Instances marked as not eligible are never chosen.

Running producers and consumers with warm caches keep working for schemas they already know. New instances, new schemas, and anything that needs registration or a first lookup will fail until the registry returns.

Three is the practical standard, in three zones. Two survive a failure but leave no spare during maintenance, and one is only suitable for development.

No. Writes always go through the single primary. More instances add read capacity and resilience, which is where most of the traffic is.

The instance you asked may not yet have read the new record from the topic. Retrying after a short delay normally succeeds, and the answer is never wrong, only briefly missing.

Not strictly. Clients can be given a list of addresses and will try the next when one fails. A balancer adds convenience and health checks, but must itself be redundant.

Yes, usually as an active region and a standby region with schemas continuously copied. Standby instances should not be eligible to become primary, and promotion should be a controlled, rehearsed action.

The schemas topic and the stability of schema IDs. Lose the topic, or change what an ID means, and data already in Kafka may become unreadable.

It is safer not to. Register schemas through reviewed pipelines ahead of deployment, so registrations are controlled and predictable rather than a side effect of application startup.

Often a secondary cannot reach the primary’s announced address, so writes succeed only when they happen to land on the primary. Check advertised host names, listener protocols, and network rules between instances.

It removes much of the operational burden, but you still own client configuration, access control, schema governance, backups, and your recovery expectations. Confirm the provider’s availability and failover behavior.

The Schema Registry is a small service with a large responsibility: it holds the shared meaning of every record flowing through Kafka. Its high availability rests on a simple and elegant design. Durable storage lives in a replicated Kafka topic, many stateless instances serve reads from memory, and a single elected primary is the only writer so that schema IDs stay unique and ordered. Clients cache what they learn, which makes short outages survivable but makes cold starts and new schemas the fragile moments. Real availability comes from hardening the schemas topic, running several instances in separate zones, making sure they can reach each other, rolling changes one instance at a time, monitoring for a single healthy primary, and rehearsing regional failover with schema IDs preserved exactly. Follow these habits and the registry fades quietly into the background, which is exactly where infrastructure belongs.