Kafka’s design is resilient on paper. Only an experiment can show that your cluster and your applications are resilient in practice.

A quick picture of Kafka

Apache Kafka stores streams of events in named containers called topics, and each topic is split into partitions. The servers that hold partitions are brokers. Programs that write events are producers and programs that read events are consumers. To survive machine failures, every partition is copied to several brokers. These copies are called replicas. One replica is the leader and takes the reads and writes, and the others are followers that keep up by copying from the leader. The followers that are fully caught up form the in-sync replicas, or ISR.

Resilience is a chain, and chains break at the weakest link

Ask a team whether their Kafka setup is highly available and they will usually point to the replication factor of three. That is one link in a long chain. Are the three replicas really in three different zones? Does the producer refuse to accept a write when only one copy exists? Will the consumer application recover if a broker restarts in the middle of a message batch? Does the alert that should wake somebody actually reach a phone? Does the runbook still match the current cluster? A single weak link means the promise is broken, and it is usually the link that nobody has ever tested.

Failures you will meet whether you plan for them or not

At scale, failure is not rare. Machines crash, disks fill, network links flap, software is upgraded, certificates expire, cloud zones have bad hours and engineers make mistakes. The question is not whether these events will happen but whether you meet them for the first time during a drill or during a customer-facing outage. Chaos engineering moves that first meeting to the drill.

The hidden cost of never testing

Skipping resilience testing feels free, but the cost simply moves to a later and worse moment. A first failover that happens at three in the morning on a Saturday is run by tired people, without a rehearsed plan, while customers are watching. The same failover run on a Tuesday afternoon in a drill is run by rested people who have a plan and a stopwatch. The failure is identical. The outcome, and the price, are very different.

There is also a quiet cost in confidence. Teams that have never seen their system fail well tend to be afraid of every change. They delay upgrades, avoid restarts and keep old versions running, and that fear creates its own risk. Teams that practice failure develop calm. They know how the system behaves when it is hurt, so they are willing to improve it.

Before applying the ideas to Kafka, understand where they came from and what makes an experiment “chaos engineering” rather than just breaking things.

A short history

The practice began at Netflix. When Netflix moved from its own data centres to the cloud around 2010, engineers realized that cloud machines can disappear at any time. Instead of hoping for the best, they built a tool called Chaos Monkey that randomly terminated virtual machines during working hours, so that every team was forced to build services that survived losing a machine. Over time the tool family grew, and the discipline was described in a set of ideas known as the principles of chaos engineering and later in a book written by Netflix engineers. Today the approach is used across banks, retailers, streaming companies and cloud providers, and several tools and cloud services exist to support it.

The five principles in plain words

Chaos engineering is not random destruction

The word “chaos” sounds reckless, but the practice is careful and scientific. A scientist changes one variable, watches the results and compares them with a prediction. A chaos engineer does the same. Random breaking without a hypothesis, without measurements and without a stop button is simply an outage created by yourself. The difference between the two is discipline.

flowchart LR
    A["Define steady state with metrics"] --> B["Write a hypothesis"]
    B --> C["Plan scope, blast radius and abort rules"]
    C --> D["Inject the fault"]
    D --> E["Observe metrics, logs and client behavior"]
    E --> F{"Did the system stay within limits?"}
    F -->|Yes| G["Record success and raise the difficulty"]
    F -->|No| H["Abort, restore and analyze the weakness"]
    H --> I["Fix the design, settings or runbook"]
    I --> A
    G --> A

Related practices, so you can tell them apart

Objections you will hear, and honest answers

Chaos experiments test promises. So first list the promises Kafka makes, and the conditions each one depends on.

The main guarantees

Delivery semantics in simple words

Three terms describe what a consumer sees after a failure. At-most-once means a message may be lost but is never seen twice. At-least-once means a message is never lost but may be seen more than once. Exactly-once means each message affects the result exactly one time. The default and most common setup is at-least-once, and it means consumers must be safe against duplicates. This is a classic thing to check with chaos: kill a consumer at a bad moment and see whether the results are counted twice.

Where promises are set: a settings map

Many of Kafka’s promises are controlled by a small group of settings. Chaos experiments are the best way to prove that these settings are right for your needs.

Consistency and availability in plain words

A distributed system that copies data to several places faces a hard choice when the network breaks. It can keep accepting writes and risk the copies disagreeing, or it can refuse some writes and keep the copies consistent. This trade-off is often called the CAP theorem, and its practical meaning is simple: during a partition, you cannot have both perfect availability and perfect consistency. Kafka lets you choose a point on this line through its settings.

With strict settings, meaning acknowledgment from all in-sync replicas, a minimum ISR of two and no unclean leader election, Kafka favors consistency. During a partition, some writes are refused so that no acknowledged message is lost. With relaxed settings, Kafka favors availability. It keeps accepting writes, but acknowledged data may be lost when leaders change. Neither choice is universally right. A log of clicks for analytics can tolerate loss, while a ledger of payments cannot. Chaos experiments show you which choice your cluster actually makes, which is sometimes not the one you thought you configured.

A Kafka platform is more than a set of brokers. Draw the whole picture before choosing what to break.

flowchart LR
    PROD["Producer applications"] --> DNS["DNS and load balancer"]
    DNS --> B1
    DNS --> B2
    DNS --> B3
    subgraph AZA["Zone A"]
        B1["Broker 1"]
    end
    subgraph AZB["Zone B"]
        B2["Broker 2"]
    end
    subgraph AZC["Zone C"]
        B3["Broker 3"]
    end
    CTRL["Controller quorum"] --- B1
    CTRL --- B2
    CTRL --- B3
    B1 --> CONS["Consumer applications"]
    B2 --> CONS
    B3 --> CONS
    SR["Schema registry"] --- PROD
    SR --- CONS
    CON["Connect workers"] --- B2
    MON["Monitoring and alerting"] --- B1
    MON --- B2
    MON --- B3

A map of failure points

Think in terms of dependencies

A helpful exercise is to trace one business event through the whole system. Take “a customer places an order.” Which producer sends it? Through which network path? To which topic and partition? Who is the leader, where are the followers, and who reads it next? Where does the result go, and who depends on that result? Every arrow on that path is a dependency, and every dependency is a candidate for an experiment. Teams often discover in this exercise that a supposedly independent service silently depends on a single machine, a single DNS name or a single person.

A worked dependency trace

Take the event “customer places an order” in an imaginary shop. The web service creates the event and sends it through a load balancer to a producer library, which writes it to the orders topic. The leader of the chosen partition, say on broker 2 in Zone B, appends it and waits for followers in Zones A and C. A stream application reads the topic, enriches each order with customer data from a database, and writes the result to a second topic. A connector copies that topic into a warehouse, and a dashboard reads the warehouse. Count the components on this path: at least ten, spread over three zones and four teams. Each one can slow, stop or misbehave, and each arrow between them can break. Writing this path on a whiteboard, with a red dot on every arrow that has never been tested, usually produces a very long list of candidate experiments and a healthy sense of humility.

You cannot tell whether an experiment hurt the system unless you know precisely what a healthy system looks like.

Measure what users feel, not only what servers do

A common mistake is to define steady state only with server metrics such as processor use. Users and downstream applications do not care about processor use. They care whether their events are accepted quickly and whether results arrive on time. Good steady-state measures come in two groups. Outcome measures describe what applications and users experience. Health measures describe the internal condition of the cluster and help explain why an outcome changed.

Build a canary: a tiny producer and consumer that never stop

The best single steady-state signal is a canary. This is a tiny producer that writes a timestamped message to a special topic every second, paired with a tiny consumer that reads it and computes how long the trip took. The canary measures the whole path, including the broker, the network and the client libraries, from the point of view of an ordinary client. It gives you a clean line on a graph that jumps the moment something is wrong. Open-source tools exist for this idea, such as the Kafka monitor project that started at LinkedIn, and many teams write their own in an afternoon.

Know your natural variation

Every metric wiggles. Latency is a little higher at peak hour, and lag rises when a big batch job runs. If you do not know the normal wiggle, you will mistake normal variation for damage, or damage for variation. Study at least a couple of weeks of history and write down the normal range for each measure at the hour and day you plan to run the experiment. Experiments should compare against that range, not against a single perfect number.

Use service level objectives

A service level objective, or SLO, is a written target for a user-facing measure, for example “99.9 percent of events are consumed within two seconds.” SLOs give you an honest yardstick for experiments. If the hypothesis is “a broker failure does not breach our SLO,” you can decide clearly whether the experiment passed. SLOs also define an error budget, the small amount of allowed failure. Many teams spend a slice of that budget deliberately on chaos experiments, since the learning is worth more than the small risk.

An example service level objective set for a Kafka platform

These numbers are examples. Your own targets should come from what your business needs and what your architecture can honestly deliver. What matters is that they are written down, measurable and agreed, so that an experiment can be judged against them without argument.

A hypothesis turns “let us break something” into “let us learn something.” It also makes the result unambiguous.

The shape of a good hypothesis

A good hypothesis has four parts. It names the fault, it names the measure that should stay healthy, it gives a limit in numbers and it gives a time for recovery. For example: “When one broker in Zone B is forcibly stopped, the producer success rate stays above 99.9 percent, the 99th percentile end-to-end latency stays below one second after the first 30 seconds, no acknowledged message is lost, and all partitions return to full replication within ten minutes of the broker returning.”

The experiment card

Write every experiment on a one-page card, so anyone can understand and approve it. The card should be short enough to read in two minutes.

Ask “what will we do with the answer?”

Before running any experiment, ask what you will change if it fails and what you will conclude if it passes. If you cannot answer either, the experiment is probably not worth running. Experiments should guide decisions: change a setting, add a runbook, retire a risky design or add a monitor.

Choosing which experiment to run first

You cannot test everything at once, so rank candidate experiments. Score each one on three simple questions: how likely is this failure in real life, how bad would the impact be, and how uncertain are we about the outcome? Failures that are likely, harmful and poorly understood go to the top. A broker crash is likely and moderately harmful, so it comes early. A slow disk on one broker is fairly likely and can be very harmful because it is hard to detect, so it also ranks high. A total loss of a region is very harmful but rare, so it belongs on the yearly schedule. This ranking gives a simple, defensible plan for the first year.

The skill that separates chaos engineering from vandalism is control. You must always be able to stop.

Grow the blast radius in rings

The blast radius of an experiment is the amount of the system, and the number of users, that it can affect. Begin with the smallest radius that still teaches you something, and grow it in stages only when the earlier stage is understood.

flowchart TB
    R1["Ring 1: local or throwaway cluster, no real users"] --> R2["Ring 2: shared staging cluster with test traffic"]
    R2 --> R3["Ring 3: production canary topics or one low-risk consumer group"]
    R3 --> R4["Ring 4: one production broker during off-peak hours"]
    R4 --> R5["Ring 5: a full zone or a full failover, planned and announced"]

Abort conditions: the stop button

Every experiment needs written, measurable conditions under which it stops immediately. Examples are “produce success below 95 percent for 60 seconds,” “any offline partition on a critical topic,” “consumer lag on the payments group above five minutes” or “any customer-facing alert fires.” Wire these conditions into automation when you can, so that the fault is removed automatically even if no person is watching. Cloud fault-injection services support this idea with alarms that stop an experiment when triggered, and open-source tools can be scripted to do the same.

Safety checklist before every run

Protect data

Some faults can destroy data permanently, such as deleting a broker’s log directory or corrupting files. Run these only in environments where the data is disposable or fully backed up. In production, prefer faults that are reversible: stopping a process, adding latency, blocking a port or pausing a disk. If the answer you need can be found with a reversible fault, do not use an irreversible one.

Protect the chaos tools themselves

A tool that can stop machines and change networks is powerful, and therefore a security risk. Give it the least permissions it needs, limit its targets by tags or names, log every action it takes, and require approval for production use. An attacker or a careless user who controls the chaos tool controls the outage button.

Approvals and communication

Production experiments belong in the same change management process as any other risky change. Write a short notice that says what will happen, when, how long, what the expected impact is and who to contact. Send it to the owners of every application that uses the cluster, the on-call team and the support desk. On the day, post a message when the experiment starts, when it ends and when the system is confirmed healthy. This transparency turns a possible source of panic into an ordinary scheduled event.

Losing a broker is the most natural test for Kafka, and it teaches a great deal.

Experiment 1: stop a follower broker

Choose a broker that leads few partitions and stop its process abruptly. The cluster should keep serving all partitions, because other replicas exist. Leaders of the partitions on that broker do not change, but the in-sync set of each partition shrinks. Producers with strict acknowledgment settings still succeed as long as enough copies remain, given by the minimum ISR. Check that no under-replicated alert is missed, and that when the broker returns it catches up and rejoins every in-sync set.

Experiment 2: stop a leader-heavy broker

Now stop a broker that leads many partitions. This is a harder test. The controller must elect new leaders for all of them, and clients must find the new leaders through a metadata refresh. Expect a brief burst of errors and latency. Measure how long the burst lasts, how many messages the producers retried and whether any application gave up too soon. The result gives you the real number for “how long is our failover” instead of a guess.

sequenceDiagram
    participant X as Chaos Tool
    participant B as Broker 2 target
    participant C as Controller
    participant P as Producer
    participant M as Monitoring
    M->>M: Confirm steady state
    X->>B: Stop the process abruptly
    C->>C: Detect missing broker
    C->>C: Elect new leaders for its partitions
    P->>P: Errors, metadata refresh, retries
    P->>C: Ask for updated metadata
    P->>P: Sends succeed on new leaders
    M->>M: Check abort rules and record timings
    X->>B: Restart the process
    B->>C: Rejoin and catch up
    M->>M: Confirm full replication and balanced leaders

Experiment 3: repeated restarts (a rolling failure)

Real life often brings not one failure but several in a row: a rolling upgrade, an autoscaling event or a bad deployment. Restart brokers one after another with only a short pause between them, or restart the next before the previous one has fully caught up. This tests what happens when the cluster is still recovering while another broker leaves. It often reveals that the minimum ISR setting, or the amount of spare capacity, is tighter than assumed.

Experiment 4: kill the active controller

In a modern cluster with a controller quorum, terminate the active controller node. The remaining controllers elect a new one. The hypothesis is that brokers keep serving data during the change and that cluster-wide operations, such as creating a topic, recover within a short time. Then try the harder version: stop enough controllers to lose the majority. The expectation is that data traffic on existing partitions keeps flowing from the brokers’ local state, but no metadata changes are possible, no new leader can be elected, and any broker failure at that time causes lasting unavailability of its partitions. This experiment proves why the quorum should be spread across zones.

Experiment 5: controlled shutdown versus crash

Compare a gentle shutdown with a crash. In a gentle shutdown, the broker asks the controller to move its leaders away first, and clients see very little. In a crash, leaders vanish suddenly. Running both tells you how much your maintenance procedures protect users, and how much worse an unplanned failure is.

Experiment 5b: watching unclean leader election in a lab

A special experiment is worth running only in a disposable test cluster. Turn on unclean leader election for one test topic, and use a network block to make the followers fall behind while the leader keeps accepting writes. Then stop the leader. Because no in-sync follower remains, the cluster may promote an out-of-date replica as the new leader. Send verification messages before and after, and use the auditor to see how many acknowledged messages vanished. The result gives your team a memorable, first-hand picture of why this setting is usually kept off for important data, and why “availability at any cost” has a real cost.

Repeat the same run with the setting off. The partition should stay unavailable for writes until an in-sync replica returns, but no acknowledged message is lost. Seeing both runs side by side makes the trade-off concrete for engineers and managers alike.

Networks fail in messy, partial ways. Those partial failures are much harder on Kafka than a clean crash.

Why partial failures are harder than crashes

When a machine is completely dead, everybody agrees that it is dead. When a machine is slow, or when it can talk to some peers but not others, different parts of the system disagree about its state. One broker thinks a follower is fine, while the controller thinks it is gone, and a client sees timeouts. These disagreements cause flapping in-sync sets, repeated leader elections and confusing symptoms. Engineers call this a gray failure: not down, not healthy, just wrong.

Experiment 6: added latency between zones

Add 100 to 300 milliseconds of delay to traffic between two zones. Followers in the far zone now take longer to fetch, so replication lag grows and end-to-end latency rises for writes that wait for all in-sync replicas. Watch whether followers fall out of the in-sync set, which depends on the lag limit setting, and whether producers with strict settings slow down or fail. The hypothesis to test is that your latency budget can absorb realistic cross-zone slowdowns.

Experiment 7: packet loss

Drop a small percentage of packets, for example two to five percent, on the path between brokers or between clients and brokers. Loss forces retransmissions and makes every request slower and less predictable. The interesting result is not whether Kafka survives but how gracefully: do timeouts, retry settings and consumer session settings turn a slow network into a storm of rebalances and duplicate processing?

Experiment 8: network partition

Block traffic between groups of nodes completely. There are several shapes to try.

flowchart LR
    subgraph ZA["Zone A"]
        A1["Broker 1 leader"]
        AP["Producer"]
    end
    subgraph ZB["Zone B"]
        B1["Broker 2 follower"]
    end
    subgraph ZC["Zone C"]
        C1["Broker 3 follower"]
        CT["Controller"]
    end
    AP --> A1
    A1 -. "blocked link" .- B1
    A1 --> C1
    CT --> A1
    CT --> B1
    CT --> C1

What safe settings should show

With acknowledgment from all in-sync replicas, a minimum ISR of two out of three and unclean leader election turned off, a partition that loses contact with two of its three replicas should stop accepting writes rather than accept them on a single copy. Producers receive errors such as “not enough replicas.” That is a correct outcome: the system chose consistency over availability for that partition. Your job is to check that the application handles the errors well, buffers or backs off, and does not silently drop data.

How to inject network faults safely

On Linux machines, the traffic control features of the operating system can add delay, loss and bandwidth limits to a network interface, and firewall rules can block chosen addresses or ports. These are simple and powerful, and they are also dangerous because a mistake can cut off your own access to the machine. Three habits reduce the risk. First, always attach a timer so the fault removes itself, for example by scheduling the cleanup command at the same moment you start the fault. Second, apply the fault only to traffic between the specific addresses you are testing, never to the management interface you use to log in. Third, keep an out-of-band way into the machine, such as a console, in case the fault locks you out. In Kubernetes and cloud environments, chaos tools and fault services wrap these steps with scoping and automatic cleanup, which is one good reason to adopt them as the practice matures.

Brokers live on disks and depend on memory and processors. Stress these and you see how Kafka behaves when the machine, not the network, is the problem.

Experiment 9: slow disk

Kafka writes to disk all day, and it reads old data from disk whenever consumers fall behind. Slow the disk down, either with a tool that adds delay to file operations or, in the cloud, by pausing input and output on a volume for a short time. Watch the request latency at high percentiles, the time followers take to catch up and the speed at which lagging consumers can read. A slow disk on one broker often causes more harm than a dead broker, because the cluster keeps sending it work. This experiment reveals whether your monitoring can tell “slow” from “healthy.”

Experiment 10: disk full

Fill a data disk in a test environment until only a tiny amount of space is left. When a log directory can no longer be written, the broker can no longer serve partitions stored there. With several data directories, modern Kafka can mark the failing directory as offline and keep the other directories running. With a single directory, the broker typically shuts down. The experiment shows whether alerts fire early enough, whether retention settings free space in time and whether the runbook for freeing space is correct and safe to follow under stress.

Experiment 11: memory pressure and long pauses

Restrict the memory available to a broker, or induce long garbage collection pauses by stressing the heap. A paused broker looks dead to the cluster without actually being dead, and after the pause it returns with stale ideas about who leads what. Watch whether brokers get fenced, whether leader changes cause client errors and whether the returning broker recovers cleanly. The companion guide on diagnosing broker GC pauses goes deep into this class of faults.

Experiment 12: processor starvation

Consume most of the processor time on one broker with a stress tool. Request handler threads then wait for processor time, queues grow and latency rises. On virtual machines this imitates a noisy neighbor. The hypothesis to test is whether your cluster spreads work sensibly, whether alerts identify the struggling node and whether operators can drain leaders from it quickly.

Experiment 13: clock skew

Move a broker’s clock forward or back by minutes. Time affects retention decisions based on message timestamps, security certificates, token lifetimes and some timeouts. Clock skew experiments often reveal surprises in log deletion timing and in authentication. Run them in staging first.

Most of the data loss and duplication in real Kafka systems is caused by applications, not brokers. Test the applications too.

Producer experiments

Consumer experiments

Lag, retention and the point of no return

If a consumer group stays down longer than the topic’s retention, the data it needs is deleted before it reads it. The group then restarts from its reset policy, either the beginning or the end, and either reprocesses a huge amount or skips a huge amount. A useful experiment is to stop a non-critical consumer group for longer than expected and check that alerts warn about lag well before it reaches the retention limit, and that the recovery procedure is documented.

sequenceDiagram
    participant Q as Topic partition
    participant C1 as Consumer 1
    participant C2 as Consumer 2
    participant DB as Result store
    Q->>C1: Deliver message 41
    C1->>DB: Write result for message 41
    Note over C1: Crash before saving position
    Q->>C2: Group reassigns partition, deliver message 41 again
    C2->>DB: Write result again
    Note over DB: Safe only if writes are idempotent

Idempotence: the shield against duplicates

An operation is idempotent if doing it twice has the same effect as doing it once. Setting a customer’s address to a given value is idempotent. Adding ten dollars to a balance is not. Applications that consume Kafka should either be idempotent by design, for example by storing a unique event identifier and ignoring repeats, or should use Kafka’s transactional features where suitable. Chaos experiments on consumers are the fastest way to find the places where this design is missing.

Connect and stream processing applications

Two families of Kafka applications deserve their own experiments. Kafka Connect workers move data between Kafka and other systems, such as databases. Stop a worker, and the cluster of workers should reassign its connectors and tasks to the others. Check how long data flow pauses, whether the connector resumes from the right position and whether the external system receives duplicates. Also test a failure of the external system itself, such as a database that is unavailable for several minutes, to see if the connector retries, pauses safely or drops records.

Stream processing applications keep local state, such as counts or joins, and back it up to Kafka topics. Kill an instance and watch it rebuild its state on another machine. The rebuild can take long for large state, and during that time results are delayed. Measure this restore time and check that standby copies of the state are configured if the delay is too large. Then repeat with a broker failure during the restore, which is a harder and more realistic case.

Around every Kafka cluster sits a ring of helpers. When one of them fails, Kafka often fails with it.

Naming and addressing

Clients find brokers by name. Experiments on DNS reveal whether clients survive a name resolving to the wrong address, resolving slowly or not resolving at all. Some teams have discovered that their clients cached a single broker address forever, or that their bootstrap list contained only one broker. Try changing the address of a broker and see whether clients follow. Also test the load balancer in front of brokers, if you use one, because Kafka clients need to reach individual brokers and a misconfigured balancer can hide failures or send clients to the wrong node.

Schema registry and other services

If producers and consumers use a schema registry to encode and decode messages, the registry becomes a dependency on the critical path. Stop it for a few minutes. Well-built clients cache schemas, so existing traffic continues, but new schemas and restarted clients may fail. Find out which of your applications keep running and which stop. The same applies to Kafka Connect workers, stream processing applications, monitoring agents and secret stores that supply credentials.

Certificates and credentials

Expired certificates and rotated passwords are among the most common causes of real Kafka outages, since they arrive on a calendar date and everything fails at once. Test them deliberately: use a certificate that expires in a few minutes, rotate a service account’s password and see which clients reconnect automatically. Check that alerts fire well before expiry and that renewal is automated. If the platform has strict access rules, test that removing a permission by mistake produces a clear error and a fast fix, rather than a mysterious silent failure. The guide on fine-grained ACL design explains how to organize such rules.

Configuration and human error

A sample certificate-expiry drill

The most useful measurement is not whether the client failed, since you know it will. It is how many minutes passed before anyone knew, how many before someone understood, and how many before it was fixed.

The biggest promise a cloud architecture makes is that the loss of a whole zone will not stop the business. Prove it.

Rack awareness and zone-level thinking

Kafka can be told which rack, or in the cloud which Availability Zone, each broker lives in. With that knowledge it places the replicas of one partition in different zones. With three zones, a replication factor of three and a minimum ISR of two, losing one full zone should leave every partition with two replicas, still writable with strict acknowledgment. This design is only as good as its execution: a topic created before rack awareness was enabled, or a broker labeled with the wrong zone, silently breaks the guarantee.

Experiment 22: verify replica placement

Before injecting any fault, run a check. For every partition, list which zone each replica lives in and confirm that no partition has two replicas in the same zone. This inspection is itself a valuable experiment, since it often uncovers old topics with poor placement.

Experiment 23: lose a zone

Simulate the loss of one zone by stopping all brokers in it, or by blocking all network traffic to and from that zone. Cloud providers offer services that help; for example, AWS Fault Injection Service can stop instances by tag and can disrupt network connectivity for a subnet, and it supports stop conditions tied to monitoring alarms. Check the latest documentation for the actions available and the safety features. In the experiment, verify that producers and consumers continue, that client failover works when clients also sit in the lost zone, that capacity in the two remaining zones is enough for the whole load and that recovery after the zone returns does not overwhelm the network.

Experiment 24: cluster failover to a second region

Some organizations copy data to a second cluster in another region using a replication tool such as MirrorMaker 2. The copying is asynchronous, so the second cluster lags behind, and a real failover may lose the most recent events. A failover drill tests the whole procedure: stopping producers or switching their addresses, deciding where consumers resume given that offsets differ between clusters, and measuring the amount of data lost or repeated. It also tests the reverse process of returning to the primary region, which is often harder than failing over.

Managed Kafka services

If you run on a managed service such as Amazon MSK, the provider replaces failed brokers and patches them on a schedule, so some of the experiments above are not under your control. What remains fully testable is the client side and the architecture around the cluster: how your applications behave when brokers restart during maintenance, whether they follow leader changes, and whether your monitoring notices. The service offers an operation to reboot a broker, which can be used as a controlled fault, and the provider’s monitoring metrics can serve as your steady-state signals. Read the current documentation to see which faults are allowed.

Follower fetching and cross-zone traffic

By default, consumers read from the leader of each partition, and the leader may be in a different zone from the consumer. This creates cross-zone network traffic, which in many clouds is billed and adds latency. Newer Kafka versions let consumers read from a nearby follower if the client announces its zone. A related experiment is to disrupt a zone and observe where consumers read from. It shows whether the design keeps traffic local in normal times and switches cleanly during a failure, and whether the cost savings of local reads cost you anything in freshness, since followers can be slightly behind the leader.

Staying up is not enough. After every experiment, prove that no data was lost, duplicated or reordered.

The three integrity questions

A verification harness

Build a small test program that plays the role of a strict auditor. The writer part sends a stream of messages, each containing a key, an increasing sequence number and a checksum, and it records every message that Kafka acknowledged. The reader part reads the topic from the beginning after the experiment and checks three things: every acknowledged sequence number is present, the numbers for each key increase without going backward, and the number of repeats is counted. The comparison result is a clear verdict on data integrity. Testing tools for distributed systems use this same idea, and independent researchers have used similar checks to study Kafka’s behavior in the past.

flowchart LR
    W["Writer: numbered messages with keys and checksums"] --> K[("Kafka topic under test")]
    W --> L["Ledger of acknowledged messages"]
    X["Chaos tool injects faults"] --> K
    K --> R["Reader after the experiment"]
    R --> V{"Compare with ledger"}
    L --> V
    V --> O1["Missing messages: loss"]
    V --> O2["Repeated messages: duplicates"]
    V --> O3["Out-of-order per key: ordering breaks"]
    V --> O4["All clear: integrity holds"]

What counts as a failure?

Running the auditor cheaply

The integrity auditor does not have to be complex. A few hundred lines of code and a small dedicated topic are enough to begin with. The writer should use several keys so that ordering can be checked in many partitions, and it should write at a modest, steady rate so the results are easy to interpret. The reader should store what it saw in a simple file or table, so the comparison can be repeated later. Add a summary at the end that prints counts of missing, repeated and out-of-order messages. Start small, and extend it when the experiments demand it, for example with transactions or with several topics.

You do not need an expensive platform to start. But it helps to know what kinds of tools exist and what each one is good at.

A reference architecture

flowchart TB
    PLAN["Experiment card in Git"] --> CTRL["Chaos controller with approvals"]
    CTRL --> AGENT["Fault agents on hosts, pods or cloud APIs"]
    AGENT --> TARGET["Kafka brokers, controllers, clients, network"]
    TARGET --> METRICS["Metrics and logs"]
    CANARY["Canary producer and consumer"] --> METRICS
    AUDIT["Integrity auditor"] --> METRICS
    METRICS --> ALARM["Abort alarms"]
    ALARM -->|breach| CTRL
    METRICS --> DASH["Dashboards and reports"]
    DASH --> REVIEW["Review meeting and follow-up tickets"]

Kafka on Kubernetes: extra points

Operator-managed Kafka on Kubernetes adds its own behaviors. Pods that are killed are recreated by the platform, and the operator may perform its own rolling restarts. Chaos experiments here should also check the pod disruption budget, which limits how many brokers may be down at once, the spread of pods across nodes and zones, the behavior of persistent volumes when a pod moves and the effect of node drains. A frequent finding is that pods of one Kafka cluster land on the same node or zone because no spreading rule was set, so the replication factor gives less protection than expected.

Build, adopt or buy?

Teams often ask whether to write their own chaos scripts, adopt an open-source tool or buy a platform. A practical rule is to start with what you already have. Shell scripts and cloud command-line tools are enough for the first ten experiments. Adopt an open-source or cloud-native tool when you need repeatability, scheduling, stop conditions and audit trails. Consider a commercial platform when you want many teams to run experiments safely with central reporting. Whatever you pick, the same rules apply: hypotheses, small radius, measured results and follow-up.

A game day is a planned event where people and systems are tested together. It is the best way to begin.

Roles

The schedule

Blameless culture

People will only report surprises honestly if they are sure they will not be punished for them. The purpose of an experiment is to find weaknesses in the system, not in the people who built it. Praise the team that finds a flaw. A game day that ends with everyone saying “nothing went wrong” should make you suspicious that the experiment was too gentle or that observers missed something.

Surprise mode, later

Once the team is comfortable, you can run experiments where the response team does not know the exact fault in advance, as in a fire drill. This tests the alerting and the diagnosing skills that a real incident demands. Only do this after the standard game days have removed the obvious weaknesses, and always with a hidden safety person who can stop everything.

What observers should prepare

A single game day proves a moment in time. Automation keeps proving the system as it changes.

A maturity ladder

flowchart LR
    L0["Level 0: no experiments, hope only"] --> L1["Level 1: manual game days in staging"]
    L1 --> L2["Level 2: scripted experiments in staging, run before releases"]
    L2 --> L3["Level 3: scheduled experiments in production with guard rails"]
    L3 --> L4["Level 4: continuous, automated, part of every change"]

Put experiments into the delivery pipeline

Once experiments are scripted, they can become part of the process for changing Kafka. Before a rolling upgrade of brokers, run the broker-stop experiment on a staging cluster with the new version. Before changing a producer library, run the producer crash test against the new build. Before enabling encryption, test the impact under load and failure. A pass becomes a required gate, and a failure blocks the release until fixed.

Build a library of experiments

Store every experiment card and script in a repository, with clear names and versions. Tag them by fault type, scope and risk level, and record their history: when they ran, results, and links to follow-up work. Over time this library becomes an accurate map of what your platform has survived. It is also a training resource for new engineers, who learn a lot by reading how each failure was expected to behave.

Measure the programme itself

Guard rails for continuous experiments in production

The experiment is only half the work. The other half is understanding the result and changing something.

Three kinds of outcome

Every experiment ends in one of three ways. In a pass, the system stayed within limits, and you record the timings as a new baseline. In a fail with insight, the system missed the hypothesis and you now know why. In a blind result, you could not tell what happened because of missing monitoring, and the first fix is better observability. All three are valuable. Passing many times in a row is not a reason to stop, but a reason to raise the difficulty.

Findings that appear again and again

Write the result in plain words

A good report is short. It restates the hypothesis, shows a graph of steady-state measures across the run, lists the timeline of key events, records the measured numbers and ends with a list of actions with owners and dates. Send it to the teams that were affected and add it to the library. A report that nobody reads teaches nobody anything.

Re-run after fixing

A fix is a hypothesis too. After the change, run the same experiment again to confirm that the weakness is gone. Teams that skip this step often find later that the fix was incomplete, or that it created a new problem.

A simple debrief template

Keep the debrief blameless and short. The value is in the list of actions and in the promise to test again, not in the length of the meeting.

Chaos engineering costs time, tools and a little risk. Weigh them honestly.

Patterns that work

Common mistakes checklist

• Skipping the steady-state check and running on an already unhealthy cluster.
• Testing only brokers and forgetting clients, dependencies and people.
• Using irreversible faults when a reversible one would answer the question.
• Leaving a fault active after the experiment ends.
• Not verifying data integrity after the run.
• Making the test gentler after a failure instead of fixing the system.
• Running experiments once and never repeating them as the system changes.
• Giving the chaos tool broad permissions with no logging or approvals.

Best practices in one place

Know your steady state and build a canary. Write hypotheses with numbers. Keep the blast radius small and grow it in rings. Arm abort rules before injecting anything. Prove data integrity after every run. Cover clients and dependencies, not only brokers. Turn every finding into a ticket with an owner and re-run the experiment after the fix. Automate what works, and keep the human game day for learning how people respond.

Chaos engineering grew out of large streaming and internet companies, and the same ideas now serve many industries.

Lessons that repeat

Teams that benefit most share a few habits. They measure before they break. They pick faults that resemble real events. They include the human response in the drill. They act on every finding and re-run to confirm the fix. And they see chaos engineering not as a project with an end date but as a routine, like testing the fire alarm every month.