Before learning the feature, understand the problem: copies are only useful when they do not fail together.

Apache Kafka is a platform that stores and moves streams of messages. A message is called a record. Records are grouped into named streams called topics. Each topic is split into smaller pieces called partitions, and each partition is an ordered log of records. A server that stores partitions is called a broker, and a group of brokers working together is called a cluster.

Servers break. Disks fail, machines crash, and networks drop. To survive this, Kafka keeps multiple copies of every partition. Each copy is called a replica. The number of copies is the replication factor. If the replication factor is three, then every partition exists on three different brokers. One replica acts as the leader, which handles all writes and, by default, all reads. The others are followers, which continuously copy the leader’s data.

The central idea is failure domains. Machines do not fail independently in real life. Servers in the same rack share the same power strip and the same network switch. Servers in the same room share cooling. Servers in the same cloud availability zone share the same data center building. If Kafka places all replicas of one partition in the same failure domain, a single event can remove every copy at once, and the partition becomes unavailable or, in the worst case, data is lost. Rack awareness is how you tell Kafka about these failure domains so that it can avoid this trap.

The word “rack” is a historical name. In practice it means any failure domain you choose to model.

In the early days of big data clusters, machines were literally stacked in metal cabinets called racks. A rack usually had one power feed and one top-of-rack network switch. When either failed, every machine in that rack went dark together. Engineers therefore taught their systems to place copies across different racks. The name stuck, even though modern clusters run in the cloud where there are no racks you can touch.

In Kafka, rack awareness is a feature in which each broker declares a label describing where it lives, and Kafka uses that label in two places. First, when placing replicas, Kafka spreads the copies of each partition across as many different labels as possible. Second, when consumers read data, Kafka can steer them toward a replica that has the same label as the consumer, so data does not have to travel far.

flowchart TB
    subgraph R1["Rack or Zone A"]
        B1["Broker 1"]
        B2["Broker 2"]
    end
    subgraph R2["Rack or Zone B"]
        B3["Broker 3"]
        B4["Broker 4"]
    end
    subgraph R3["Rack or Zone C"]
        B5["Broker 5"]
        B6["Broker 6"]
    end
    P["Partition 0 with replication factor 3"] --> B1
    P --> B3
    P --> B5
        

What a rack can represent

Rack awareness in Kafka has two separate uses that solve two separate problems. Keeping them apart avoids a lot of confusion.

The placement logic is simple to describe: interleave the racks, then deal out replicas round by round.

Every broker can be given a rack label in its configuration. When a new topic is created without an explicit placement, the cluster runs a placement routine. If all brokers have labels, the routine builds an ordered list of brokers in which the racks are alternated. It takes one broker from the first rack, one from the second, one from the third, then goes back to the first rack for the next broker, and continues in this way. The result is a list where neighbors tend to belong to different racks.

The routine then walks through the partitions. For each partition it picks a starting position in the alternating list and selects the required number of consecutive brokers as the replicas. Because neighbors are usually from different racks, the replicas of one partition naturally land in different racks. The starting position moves forward for each new partition, so the load of leaders and followers is spread evenly across brokers as well.

flowchart LR
    A["Brokers grouped by rack label"] --> B["Alternate racks to build an interleaved broker list"]
    B --> C["For each partition pick a starting spot in the list"]
    C --> D["Take the next replication-factor brokers as replicas"]
    D --> E["Replicas land in different racks whenever possible"]
    E --> F["Shift the starting spot for the next partition to balance load"]
        

What happens when there are more replicas than racks

Suppose you have three racks and a replication factor of five. Five copies cannot each sit in a unique rack. Kafka then spreads them as evenly as it can, so two racks hold two replicas each and one rack holds one. The rule is best effort: use as many distinct racks as possible, and when racks run out, reuse them as evenly as possible.

What happens when racks are unequal in size

If one rack has ten brokers and another has only two, the alternating pattern still gives each partition a replica in each rack, but the two small-rack brokers must carry a replica of a very large share of all partitions. They become overloaded in storage and traffic, while the big rack brokers are underused. Rack awareness ensures safety, but it can create imbalance when racks are lopsided. The remedy is to keep racks roughly equal in broker count.

The whole feature starts with one small configuration value on every broker.

Each broker has a setting that holds its rack label. It is a plain text value, such as the name of an availability zone or a rack identifier. The value is read when the broker starts and is registered with the cluster’s metadata, so every other component can learn it. Changing a rack label requires restarting that broker, because the label is part of its static identity.

Choosing good labels

Pick labels that correspond exactly to the failure domain you want to survive. In a cloud region with three availability zones, use the zone names. In your own data center, use a label per rack or per power group. Avoid labels that are too fine, such as one label per machine, because then Kafka would believe every machine is its own failure domain and rack awareness would give no benefit beyond ordinary replication. Avoid labels that are too coarse, such as one label for the entire cluster, because then no separation happens at all.

Numbers make the benefit concrete. The right combination of racks, replication factor, and minimum in-sync replicas decides how much failure you can survive.

Two topic settings interact with rack awareness. The replication factor says how many copies exist. The minimum in-sync replicas setting says how many copies must be up to date and reachable before Kafka accepts a write when the producer requests full acknowledgment. Together they define the balance between availability and durability.

The rule of thumb

Choose a replication factor equal to the number of racks or zones you want to span, commonly three. Set the minimum in-sync replicas to one less than the replication factor, commonly two. With producers that require full acknowledgment, this combination survives the loss of a whole rack without losing acknowledged data and without stopping writes.

Watching a failure unfold shows why every earlier choice matters.

sequenceDiagram
    participant P as Producer
    participant L as Leader in Rack A
    participant F1 as Follower in Rack B
    participant F2 as Follower in Rack C
    participant C as Cluster Controller
    P->>L: Write record
    L->>F1: Replicate
    L->>F2: Replicate
    F1-->>L: Acknowledged
    F2-->>L: Acknowledged
    L-->>P: Write confirmed
    Note over L: Rack A loses power
    C->>C: Detects leader is gone
    C->>F1: Elected as new leader from in-sync replicas
    P->>F1: Producer retries after metadata refresh
    F1->>F2: Replicate
    F1-->>P: Write confirmed
        

Spreading replicas across zones is safe, but in the cloud it turns internal traffic into billable traffic.

Most cloud providers charge for data that moves between availability zones inside the same region, while data that stays within one zone is free or much cheaper. When Kafka replicates across three zones, every record a producer writes is copied to two other zones. When consumers read from leaders located in other zones, those reads cross zones as well. For a busy cluster moving many megabytes per second, this becomes one of the largest line items in the bill.

flowchart LR
    subgraph ZA["Zone A"]
        PR["Producer"]
        LD["Leader"]
    end
    subgraph ZB["Zone B"]
        FB["Follower"]
        CB["Consumer"]
    end
    subgraph ZC["Zone C"]
        FC["Follower"]
    end
    PR -->|"write, same zone"| LD
    LD -->|"replication, crosses zones"| FB
    LD -->|"replication, crosses zones"| FC
    LD -->|"consumer read, crosses zones"| CB
        

The important insight is that replication traffic is the price of safety and cannot be avoided. Consumer read traffic, however, is often the largest part of the total because many consumer groups may read the same data. Reducing that portion is where nearest-replica reading pays off, and this leads us to the second half of rack awareness.

Since Kafka 2.4, consumers can fetch from a follower instead of the leader. This is often called follower fetching.

Originally, Kafka had a strict rule: all reads and all writes go to the leader. It kept the design simple and consistent, but it forced consumers to cross zones whenever the leader was elsewhere. A later improvement allowed consumers to read from any replica that has the data, including followers. When combined with rack labels, a consumer can be directed to the replica in its own rack, keeping traffic local.

sequenceDiagram
    participant C as Consumer in Zone B
    participant L as Leader in Zone A
    participant F as Follower in Zone B
    C->>L: Fetch request including my rack label
    L->>L: Selector compares consumer rack with replica racks
    L-->>C: Empty data plus preferred read replica is the follower in Zone B
    C->>F: Fetch request sent to the follower
    F-->>C: Records returned from the local zone
    Note over C,F: Future fetches keep going to the follower until told otherwise
        

Reading the flow

Reading from followers saves money, but it is not free of subtleties. Understanding them lets you decide with confidence.

Followers copy data from the leader with a small delay. A record becomes readable by consumers only after it has been replicated to all in-sync replicas and the leader has advanced a marker called the high watermark. The leader tells followers about the new watermark on their next fetch, so a follower learns that a record is safe to serve slightly after the leader does. As a result, a consumer reading from a follower may see new records a fraction of a second later than one reading from the leader.

Is it safe for data consistency?

Yes. Followers only serve data up to the high watermark, which covers records that all in-sync replicas already hold. That means a consumer never sees a record that could later disappear because of a leader change. Kafka also handles the case where a follower does not yet know about a newly committed offset by answering that the offset is out of range temporarily, and the consumer retries.

When to use it and when to skip it

A frequent surprise: rack awareness does not move producer traffic to the nearest broker.

All writes for a partition go to its leader, and only the leader. Rack awareness spreads replicas across racks but does not decide where leaders live in a way that matches any given producer. With three zones and evenly spread leaders, a producer is in the same zone as a given leader only about one time out of three. The remaining two thirds of its writes cross a zone boundary.

flowchart TD
    A["Producer sends a record"] --> B{"Which broker is the leader for this partition?"}
    B -->|"Same zone as the producer"| C["Local write, no cross-zone cost for the first hop"]
    B -->|"Different zone"| D["Cross-zone write, billed as inter-zone traffic"]
    C --> E["Leader replicates to followers in other zones"]
    D --> E
        

Preferred leaders and balance

Each partition has a preferred leader, which is the first replica in its replica list. Kafka periodically moves leadership back to the preferred replica after failures so that leaders stay spread evenly. Because the replica list is built from the alternating rack order, preferred leaders naturally rotate across racks and across brokers, which keeps the number of leaders in each zone roughly equal.

Adding rack labels to a running cluster does not rearrange anything by itself. Existing partitions stay where they were.

Placement decisions are made when a topic is created or when partitions are moved. If you add labels to brokers that already host topics, those topics keep their old replica layout, which may contain several replicas in one rack. Rack awareness only applies to new placements from then on. To fix old topics, you must trigger a reassignment that recomputes the placement using the rack information.

Capacity changes are the moments when rack awareness quietly breaks if nobody is watching.

When you add brokers, new brokers start empty. Kafka does not move existing data onto them automatically. You must reassign partitions to use them. If you add brokers to only one rack, that rack becomes larger, and the alternating pattern will later overload the smaller racks. If you remove brokers, the partitions on them must be moved first, and if the removal empties a rack or reduces racks below the replication factor, some partitions can no longer be spread perfectly.

flowchart LR
    A["Need more capacity"] --> B["Add the same number of brokers to every rack"]
    B --> C["Reassign partitions onto the new brokers"]
    C --> D["Check that every partition still spans distinct racks"]
    D --> E["Verify balance of storage and traffic per rack"]
        

Modern deployments rarely have physical racks. The same idea maps neatly onto zones, node labels, and managed services.

On container platforms, each worker node usually carries a label that names its availability zone. A Kafka operator can read that label and inject it as the broker’s rack value, so the rack label comes from the platform instead of a hand-edited file. To make this effective, the platform must also be told to spread broker pods across zones using anti-affinity or topology spread rules. Otherwise the scheduler might place several brokers in the same zone, and the labels would faithfully report a poorly spread cluster.

Managed Kafka services

Most managed Kafka offerings place brokers across several zones for you and set rack labels behind the scenes. Many also support follower fetching, but you may need to enable a broker-level setting through the provider’s configuration and set the rack label on the client yourself. The exact controls differ between providers, so consult their documentation before relying on the behavior.

The idea extends beyond plain consumers to the wider Kafka ecosystem.

flowchart TD
    L["Rack labels on brokers"] --> A["Replica placement"]
    L --> B["Follower fetching for consumers"]
    L --> C["Standby placement in stream processing"]
    L --> D["Rack goals in balancing tools"]
    L --> E["Monitoring and audits per rack"]
        

Rack awareness is not a security feature, but it touches security in a few important ways.

Also protect the configuration itself. Anyone who can change broker settings or run reassignment tools can move your data around or defeat your placement. Limit these rights to trusted administrators and audit their use.

A rack-aware design only stays rack-aware if you keep checking it.

An alerting ladder

These arrangements have been proven again and again in production.

These mistakes appear in real incidents. Learn them so you do not repeat them.

Most rack awareness problems fall into a handful of families. Recognize the family and the fix is usually short.

A calm debugging routine

The same feature serves different goals depending on the business.

A compact checklist you can use before, during, and after rollout.

A short readiness checklist

Short answers to the questions people ask most often when they start with rack awareness.

No. It is just a label for a failure domain. In the cloud it usually stands for an availability zone, and in other setups it can mean a room, a power group, or a whole data center.

No. Kafka trusts whatever you configure. Wrong labels quietly weaken your protection, so generate them from a reliable source and audit them.

Placement for new topics fails with an error, unless the administrator tells the tool to ignore rack information. Always label all brokers or none.

No. Labels influence new placements only. Existing partitions stay where they are until you run a reassignment that uses the rack information.

Three is the most common choice, matching a replication factor of three, so every partition has one replica per rack. More racks give finer spreading but add operational complexity.

Kafka spreads replicas as evenly as it can, so some racks will hold more than one replica of a partition. Safety against a single rack failure is reduced for those partitions.

No. Producers always write to the partition leader, wherever it is. Rack awareness affects where replicas live and, optionally, where consumers read.

It lets a consumer read from a copy of the data that is in its own zone instead of always contacting the leader, which may be in another zone. It saves network cost and can reduce latency.

Yes. Followers only serve records that are already committed, so consumers never see data that could later vanish. The only visible effect is a small extra delay before new records become readable.

The brokers also need the rack-aware replica selector enabled, and the client and broker versions must support the feature. Also confirm that some replica actually shares the consumer’s label.

Yes, with a restart of that broker. But existing replicas do not move automatically, so you may need a reassignment afterward to make placement match the new label.

No. Replication across zones is the cost of durability and stays the same. Rack awareness reduces only the consumer read portion, and only when follower fetching is enabled.

If every broker shares one failure domain, labels add nothing. You may still label by physical rack or power group inside the building to survive smaller failures.

Rack awareness turns Kafka’s replication from a simple copying mechanism into a real defense against correlated failures. By giving every broker an honest label for its failure domain, you let Kafka place the copies of each partition in different racks, zones, or sites, so a power cut, a switch failure, or a whole building outage still leaves healthy copies elsewhere. The same labels can also steer consumers to the nearest replica, cutting cross-zone network cost with only a small delay in freshness. The feature is simple to enable and easy to break: labels must be correct and complete, racks must stay balanced, existing topics must be reassigned, capacity must allow for a lost rack, and both halves of nearest-replica reading must be configured. Treat the rack label as critical configuration, audit it regularly, and rehearse failures, and this small setting quietly delivers a large amount of resilience.