Before learning any tool, we need to accept an uncomfortable fact: in a large system, something is always broken. The question is whether anyone notices.

What a distributed system is

A distributed system is a program made of many smaller programs running on many computers that talk to each other over a network. A modern online shop is a good example. One service handles logins, another handles the product catalog, another handles the shopping cart, another handles payments, and so on. Kubernetes runs these services as containers spread across a cluster of machines.

Why things break

Each machine, disk and network cable has a small chance of failing. When you have thousands of them, small chances add up. If one machine fails once every three years, then in a fleet of 1,095 machines, you should expect about one failure every day. Beyond hardware, software has bugs, deployments go wrong, configuration is mistyped, certificates expire, and traffic spikes arrive without warning.

The famous false assumptions

Engineers at a large computer company once listed beliefs that newcomers to distributed systems tend to hold, and which cause painful surprises: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change, there is one administrator, transport cost is zero, and the network is the same everywhere. Every one of these is false, and every one can be tested by chaos experiments.

Failure is not always a clean stop

The worst failures are not the ones where a service dies. A dead service is easy to detect and easy to route around. The nasty ones are partial: a service answers, but ten times slower than usual. A network drops one packet in twenty. A disk accepts writes but takes seconds to confirm them. A clock jumps forward. These grey failures are exactly what ordinary tests miss, and what chaos tools simulate well.

Chaos engineering did not begin as a tidy discipline. It began as a nervous experiment at a company that had just moved to the cloud.

Where it started

Around 2010, Netflix was moving from its own data centers to the cloud. In the cloud, machines can vanish at any time. To make sure its services could cope, engineers built a tool called Chaos Monkey that randomly terminated servers during working hours. The rule was simple: if a service could not survive losing a machine, better to find out when engineers were at their desks than during a holiday weekend. Chaos Monkey grew into a family of tools often called the Simian Army, each simulating a different kind of trouble. Other companies, including Amazon with its practice of “game days,” built similar habits.

The principles

In the mid-2010s the practice was written down as a set of principles, published as the Principles of Chaos Engineering. They are short and worth remembering:

What chaos engineering is not

• It is not random destruction. Every experiment has a question, a plan, limits and a way to stop.
• It is not a replacement for testing. Unit and integration tests check that code is correct. Chaos experiments check that a running system copes with a hostile world.
• It is not only for giants. A team with three services and one cluster can learn a great deal from a single well-chosen experiment.

Every chaos experiment, whatever tool you use, follows the same loop. Learn it once, and you can design experiments for any system.

flowchart LR
  A["Define steady state with metrics"] --> B["Write a hypothesis"]
  B --> C["Choose a fault and limit the blast radius"]
  C --> D["Set abort conditions"]
  D --> E["Inject the fault"]
  E --> F["Observe metrics and behavior"]
  F --> G{"Did steady state hold?"}
  G -->|"yes"| H["Gain confidence, widen the scope"]
  G -->|"no"| I["Found a weakness - stop and recover"]
  I --> J["Fix the weakness"]
  J --> K["Re-run to verify the fix"]
  H --> A
  K --> A

Step by step

The value is in the hypothesis

Many beginners think the point of chaos engineering is the breaking. It is actually the predicting. Writing down what you expect forces the team to share assumptions, and the interesting results are the cases where reality disagrees with the prediction. A team that says “we have no idea what will happen” is not ready for the experiment, and a team that says “nothing will happen” is often wrong.

To design good experiments, you need to know what can fail. Here is a tour from the smallest piece to the whole region.

What Kubernetes protects you from, and what it does not

Kubernetes automatically handles the first rows of this table. If a Pod dies, a replacement starts. If a node fails, Pods move. That is why simple “kill a Pod” experiments often pass, which is reassuring but not very informative. The higher rows are where applications are weak, and where Kubernetes cannot help, since it does not know what your application considers healthy. Slow networks, slow dependencies and partial failures are where chaos experiments earn their keep.

The strongest objection to chaos engineering is fear: “what if the experiment causes a real outage?” Good practice answers this fear with careful limits.

Blast radius

The blast radius is the amount of the system, and the number of users, that an experiment can affect. You control it in several ways:

• Scope of targets. Choose one Pod, not a whole Deployment. Choose one namespace, not the cluster.
• Percentage of traffic. Affect a small share of requests, such as a canary group, before affecting all of them.
• Duration. Run for thirty seconds, not thirty minutes. Tools in this guide support automatic recovery when the duration ends.
• Severity. Add 100 milliseconds of delay before adding five seconds. Fill a disk to 70 percent before 100 percent.
• Environment. Begin in a development cluster, then staging, then a small slice of production.

Abort conditions and the big red button

Before every experiment, agree on measurable signals that mean “stop now,” and make stopping easy. With both tools in this guide, deleting or pausing the experiment object recovers the fault, and you can wire monitoring alerts to do this automatically. Everyone involved should know who has the authority to stop an experiment and how.

Prerequisites before any experiment

Now we meet the first of our two tools. Chaos Mesh brings a large, ready-made set of faults to Kubernetes and treats them like any other Kubernetes object.

Where it comes from

Chaos Mesh was created by PingCAP, the company behind the TiDB distributed database. The engineers there needed to test a database that must stay correct under every kind of failure, and built a toolkit for it. They released it as open source and donated it to the Cloud Native Computing Foundation (CNCF), where it is a hosted project at the incubating level. It runs on Kubernetes and is designed so that you describe an experiment in a configuration file and apply it, exactly as you would a Deployment.

The core idea: faults as Kubernetes objects

In Chaos Mesh, each kind of fault is a custom resource, a new type of Kubernetes object. There is a type for killing Pods, a type for network trouble, a type for disk trouble, a type for CPU and memory stress, and more. To start an experiment you create an object of the right type, saying what to break, where and for how long. To stop it, you delete the object. Everything fits the tools and habits teams already use: version control, code review and automation.

Why teams choose it

Chaos Mesh is made of three main programs. Understanding their roles tells you where things can go wrong and what permissions they need.

flowchart TB
  subgraph Users["Users and Automation"]
    ENG["Engineer or pipeline"]
    UI["Chaos Dashboard in browser"]
  end
  subgraph CM["chaos-mesh namespace"]
    DASH["chaos-dashboard - UI and API"]
    CTRL["chaos-controller-manager - controllers and webhooks"]
    DAE1["chaos-daemon on node 1"]
    DAE2["chaos-daemon on node 2"]
  end
  subgraph K8S["Kubernetes control plane"]
    API["API Server"]
  end
  subgraph N1["Node 1"]
    P1["Target Pod A"]
  end
  subgraph N2["Node 2"]
    P2["Target Pod B"]
  end
  ENG -->|"apply chaos object"| API
  UI --> DASH
  DASH -->|"create objects"| API
  API -->|"watch events"| CTRL
  CTRL -->|"gRPC request"| DAE1
  CTRL -->|"gRPC request"| DAE2
  DAE1 -->|"inject fault into namespaces and cgroups"| P1
  DAE2 -->|"inject fault into namespaces and cgroups"| P2
  CTRL -->|"update status"| API

The three main components

Why a privileged daemon?

Many faults need deep access to the machine. To add network delay inside a Pod, the daemon must enter that Pod’s network namespace and change its traffic rules. To stress CPU inside a container, it must run a process inside that container’s control group. These actions need elevated privileges on the node. This is the biggest security consideration of Chaos Mesh, and we cover it in Chapter 24.

Container runtime connection

The daemon finds the target container through the container runtime, such as containerd or CRI-O, using a socket on the node. Installation therefore asks you to tell Chaos Mesh which runtime and socket path your cluster uses. A wrong setting is a classic reason for experiments that never take effect.

Admission webhooks

The controller manager also runs webhooks. Some validate experiment objects, rejecting invalid ones with clear errors. One mutates Pods when needed, for example to add a helper sidecar for file system faults. These need valid certificates, which the installation sets up.

This is the menu of failures Chaos Mesh can serve. Each is a separate custom resource with its own set of actions.

Starter experiments that teach a lot

• Kill one Pod of a Deployment. Checks that traffic moves, replacements start quickly and no data is lost.
• Add 300 milliseconds of delay between a service and its database. Reveals missing timeouts and slow-query cascades.
• Partition two groups of Pods. Tests how clustered software behaves when members cannot talk to each other.
• Stress CPU in a container. Shows whether autoscaling reacts and whether limits protect neighbors.
• Return errors from DNS. Shows whether clients cache, retry, or crash.
• Shift the clock forward. Finds certificate and token handling problems.

The faults can seem like magic. In truth, they use a handful of well-known Linux features. Knowing them helps you predict what an experiment will and will not do.

Linux building blocks

Containers on Linux are ordinary processes, wrapped in two features. Namespaces give a process its own private view of things like the network, process list and file system. Control groups, or cgroups, limit how much CPU and memory a group of processes may use. Chaos Mesh’s daemon works by stepping into these same wrappers and changing things from inside.

Network faults

For delay, loss, duplication and bandwidth limits, the daemon uses the Linux traffic control facility inside the target Pod’s network namespace. Traffic control can place rules on outgoing network traffic that wait, drop or shape packets. To restrict the fault to specific destinations, such as only traffic to the database, the daemon uses packet-filtering rules and address sets. Partitions are built with firewall rules that drop packets between the chosen groups.

Pod faults

The simplest fault is a Pod kill, done through the Kubernetes API, as if an administrator had deleted the Pod. A container kill sends a signal to a specific container through the runtime. The “Pod failure” action works differently: it swaps the image of the containers for a deliberately broken placeholder for the duration, then restores them.

Stress faults

For CPU and memory stress, the daemon starts a stress-generating process inside the target container’s control group. Because it shares the container’s limits, the stress competes with your real application for the very resources the container is allowed to use. That is exactly what a noisy neighbor or a runaway thread would do.

Disk faults

IOChaos works by interposing on file access. Chaos Mesh injects a helper sidecar into the target Pod through a mutating webhook, and uses a user-space file system to intercept reads and writes to the chosen paths. The interception can then delay or fail the operations. This is why IO experiments require the sidecar to be present before the application starts using the files.

Time, DNS and HTTP faults

• Time: the daemon modifies how the target process obtains the current time, so the process sees a shifted clock while the rest of the machine does not.
• DNS: a special DNS server is deployed. Faults are applied by configuring cluster DNS to consult it for matching names.
• HTTP: the daemon redirects the Pod’s traffic through a transparent proxy that can modify, delay or abort messages that match the rule.

Recovery

Every injection has a matching recovery step, which the controller triggers when the duration ends, when the object is paused or when it is deleted. If the controller is down at that moment, the fault can persist until the controller returns, one reason that high availability of the chaos tools themselves matters. We return to this in Chapter 26.

An experiment object answers four questions: what to break, how much of it, for how long, and how often. These settings are your main safety controls.

Selectors: what to break

A selector describes which Pods, or nodes, are eligible. You combine several filters, and only targets that match all of them are chosen:

Modes: how much of it

Duration: for how long

Most fault types accept a duration. When the time is up, the controller removes the fault automatically. This is a vital safety net: even if you forget about the experiment, or the person who started it goes home, the system recovers by itself. Keep durations short at first, and extend them only when you understand the behavior.

Direction and targets for network faults

Network experiments add two more ideas. The direction says whether the fault applies to traffic leaving the Pod, entering it or both. The external target, or “to” group, narrows the fault to traffic going to particular Pods, addresses or domain names. For example, “add delay only to traffic from the cart service to the payments service” lets the rest of the system carry on normally and gives a very precise test.

Schedules: how often

A Schedule is another custom resource that wraps any fault type with a cron-style timetable, for example “every weekday at 11 a.m.” It also holds rules for history, which keeps a limited number of past runs, and for concurrency, which decides whether a new run may start while the previous one is still active. Scheduling turns chaos from a rare event into a steady habit.

Real outages are rarely a single event. Workflows let you build realistic scenarios from several steps.

Workflows: scenarios made of steps

A Chaos Mesh Workflow is a custom resource that describes a whole scenario as a graph of steps, much like a recipe. The step types are:

• Chaos steps that run a fault, such as a network delay.
• Serial steps that run their children one after another.
• Parallel steps that run their children at the same time.
• Suspend steps that wait for a fixed time, to let the system settle between faults.
• Task steps that run a container, for example a script that sends test traffic or checks a result.
• Status check steps that probe a target and fail the workflow if the system is unhealthy.

A scenario in words

Imagine you want to test the loss of a database replica during a traffic peak. The workflow could say: first start a load generator as a task. In parallel, wait two minutes, then kill one database Pod. After that, add 200 milliseconds of delay between the application and the remaining database for three minutes. Throughout, run a status check against the application’s health endpoint every ten seconds, and abort the whole workflow if two checks fail in a row. Such a scenario would be hard and risky to run by hand and is easy to repeat as a workflow.

Status checks as automatic guards

The status check feature lets an experiment test the health of the system itself. It can call an HTTP endpoint at an interval and judge success by the response. Used inside a workflow, it works as an abort condition. Used alongside a schedule, it can run as a “steady state verifier” that decides whether a scheduled experiment may run at all.

The dashboard

The dashboard offers a form-based way to create experiments and view their status, events and history. It is friendly for beginners and for game days, where a facilitator wants to start and stop faults while people watch. It can also create workflows through a visual editor. For long-term use, many teams save the generated definitions into version control and apply them through automation, so the dashboard remains a helper and not the source of truth.

Controlling who can do what

Because chaos can hurt, access must be controlled. The dashboard supports token-based access tied to Kubernetes RBAC, so a user can only create experiments in namespaces where their own account has permission. Configure this carefully, and do not expose the dashboard to the open internet. Chapter 24 covers governance in detail.

Now we turn to the second tool. Litmus takes a different route to the same goal, with a strong focus on shared experiments, platform features and measuring resilience.

Where it comes from

LitmusChaos was created by the team at MayaData, the company behind the OpenEBS storage project, because they wanted to test storage systems on Kubernetes. It joined the CNCF as a sandbox project in 2020 and later advanced to the incubating level. A company called Harness later took over the commercial work around it, and builds its chaos engineering product on Litmus, while the open source project continues in the CNCF.

The core idea: chaos as a shared, reusable catalog

Litmus organizes chaos into three kinds of objects and a public catalog. Experiments are the recipes: reusable definitions of a fault and how to run it. Engines connect an experiment to a particular application. Results record what happened. A public catalog called the ChaosHub holds dozens of ready-made experiments that you can pull into your own cluster. This catalog-and-engine structure grew from the idea that chaos tests should be a library that the whole community shares and improves.

Beyond injecting faults: judging the outcome

A defining feature of Litmus is that an experiment does not only cause trouble, it also checks whether the system coped. Experiments can include probes, small checks that run before, during or after the fault and decide the verdict. If the website kept answering with correct pages during the fault, the experiment passes. If not, it fails. This built-in judgment turns an experiment into an automated test that can run in a pipeline without a person watching.

Why teams choose it

Litmus splits its parts between a control plane that you use to plan and watch, and an agent side that runs inside each target cluster.

flowchart TB
  subgraph CC["ChaosCenter - control plane"]
    FE["Web portal"]
    GQL["GraphQL server"]
    AUTH["Authentication server"]
    DB[("Database")]
  end
  subgraph TC["Target cluster - chaos infrastructure"]
    SUB["Subscriber"]
    OP["Chaos operator"]
    RUN["Chaos runner"]
    EXP["Experiment job"]
    HELP["Helper pods when needed"]
    EXPT["Chaos exporter - metrics"]
    APP["Target application"]
  end
  HUB["ChaosHub - experiment catalog"]
  PROM["Prometheus"]
  FE --> GQL
  GQL --> AUTH
  GQL --> DB
  GQL |"secure connection"| SUB
  HUB -->|"experiment definitions"| GQL
  SUB -->|"create engine objects"| OP
  OP -->|"starts"| RUN
  RUN -->|"creates"| EXP
  EXP --> HELP
  EXP -->|"inject fault"| APP
  HELP -->|"inject fault"| APP
  EXPT -->|"results as metrics"| PROM

The control plane: ChaosCenter

ChaosCenter is the part you log into. It contains a web portal, a server exposing a GraphQL interface, an authentication service for users and projects, and a database that stores experiment definitions, schedules, results and user accounts. It can run in one cluster and manage experiments in many others, which suits organizations with many clusters and teams.

The chaos infrastructure in each target cluster

Standalone use

You do not have to use the portal. The operator and the custom resources work on their own. Teams can install only the operator, pull experiments from the hub, apply an engine and read the result, entirely with kubectl and pipeline tools. The portal adds convenience, multi-cluster control and team features.

Three objects tell the whole story of a Litmus experiment. Learn them, and you can read any Litmus configuration.

ChaosExperiment: the recipe

A ChaosExperiment is a reusable definition of one kind of fault. It names the container image that carries out the fault, the permissions needed, the default settings, and the list of adjustable values. Examples include deleting Pods, killing containers, adding network latency, burning CPU, draining a node or filling a disk. Experiments do not point at any application; they are general recipes.

ChaosEngine: the plan for one application

A ChaosEngine connects one application to one or more experiments. It holds:

• Application information: the namespace, the label and the kind of workload to target, such as a Deployment.
• The experiments to run and any overrides of their default values, such as the total duration, the interval between hits, the share of Pods affected, or the container to target.
• The service account that grants the experiment the rights it needs, and nothing more.
• Engine state: active, which lets it run, or stopped, which halts it.
• Clean-up policy: whether to keep or delete the experiment Pods after completion.
• Annotation check: an optional safety, requiring that the target workload carries an opt-in annotation before it can be touched.

ChaosResult: the verdict

A ChaosResult is created for each experiment in an engine. It records the verdict, the probe outcomes and the run history. The verdicts are:

The ChaosHub

The ChaosHub is a public Git repository of experiment definitions, grouped by category. There are generic Kubernetes experiments, ones for specific cloud providers, and ones for popular applications such as databases and message brokers. You can connect the portal to the public hub or to your own private hub, where your team curates its experiments. A private hub is a good way to share approved, well-tested experiments inside a company and to prevent people from running untested ones.

How the pieces relate

Let us follow one experiment from the moment you start it until the verdict is written.

sequenceDiagram
  autonumber
  participant U as User or Pipeline
  participant API as Kubernetes API
  participant OP as Chaos Operator
  participant RUN as Chaos Runner
  participant EXP as Experiment Pod
  participant APP as Target Application
  U->>API: Create ChaosEngine with state active
  API-->>OP: Engine event
  OP->>API: Validate target and permissions
  OP->>RUN: Launch chaos runner
  RUN->>API: Read ChaosExperiment definition
  RUN->>EXP: Create experiment job
  EXP->>APP: Run pre-chaos probes
  EXP->>APP: Inject fault for the chosen duration
  EXP->>APP: Run probes during chaos
  EXP->>APP: Remove fault and wait for recovery
  EXP->>APP: Run post-chaos probes
  EXP->>API: Write ChaosResult with verdict
  RUN->>API: Update engine status
  API-->>U: Read result and verdict

Step by step

Stopping an experiment

To stop a running experiment, set the engine state to stopped or delete the engine. The operator then tells the experiment to abort and revert the fault. Experiments are written to clean up after themselves when interrupted, but you should still rehearse stopping, since it is the move that matters most when something goes wrong.

Probes are what turn Litmus experiments into automated tests. They answer the question “was the system fine while we hurt it?”

What a probe is

A probe is a small check that runs as part of an experiment. It tests some aspect of the steady state, and its result contributes to the final verdict. A good experiment has probes that mirror what users care about, not only that the Pod came back.

The probe types

Probe modes: when to run

Tuning a probe

Probes have settings for how long to wait for each attempt, how many times to try, how long to pause between attempts, and whether to stop the whole experiment at the first failure. Set these thoughtfully. A probe with a too-short timeout fails for harmless reasons, and a probe that retries ten times may hide a real problem. Choose settings that match your service goals.

The resilience score

When you run several experiments together as a workflow, ChaosCenter combines their outcomes into a resilience score between 0 and 100. Each experiment can be given a weight showing its importance, and the score reflects how many probes passed, weighted accordingly. A score is a handy way to compare services, track improvement over time and set targets, for example “every critical service must score at least 90 before release.” Keep in mind that a score is only as meaningful as the experiments and probes behind it. A perfect score from weak experiments is false comfort.

The catalog is large. This chapter shows its main families and how to combine experiments into realistic scenarios.

Main experiment families

Common tuning values

Most experiments accept the same few settings. The total chaos duration sets how long the fault is active. The chaos interval sets the gap between repeated hits within that time. A force option decides whether Pods are deleted gracefully or immediately. A percentage setting decides what share of eligible Pods is affected. A target container setting narrows the fault to one container inside a Pod, which is wise when a Pod has sidecars you do not want to disturb.

Workflows: combining experiments

ChaosCenter lets you build workflows, series and groups of experiments with weights, schedules and shared settings. A typical workflow might run a Pod-delete experiment, then network latency, then a CPU stress test, one after another, and compute a resilience score at the end. Because workflows are saved definitions, they can be scheduled, copied between projects, versioned and shared.

Both tools break things on purpose, safely. They differ in style, and the right choice depends on your team, not on a winner.

The short comparison

Different shapes of safety

Chaos Mesh keeps a long-running privileged daemon on every node, which is always ready, so faults begin quickly, but that daemon is a powerful component you must guard well. Litmus tends to start short-lived Pods with specific rights for each experiment, which narrows what exists at any time but adds a little startup delay and more moving parts to understand.

How to choose

• If you want to explore many kinds of low-level faults quickly, and you already have strong monitoring, Chaos Mesh is a natural start.
• If you want automated pass-or-fail checks in release pipelines and shared, reviewed experiments, Litmus is a natural start.
• If you run many clusters and many teams, the project and hub features of Litmus help with organization, and the namespace-scoped controls of Chaos Mesh help with isolation. Evaluate both against your governance needs.
• If you are unsure, run a small pilot with each on a test cluster for a week, using the same simple question, and see which one your team understands faster.

Using both

Some organizations use both, with Litmus as the pipeline gate and Chaos Mesh for deep exploration during game days. This is allowed, but it doubles the amount of cluster-wide machinery, so make sure you have the people to look after both. Pick one for your first year if you are small.

Before you install anything, make sure the cluster is ready to be hurt without hurting people. A few preparations matter more than any tool setting.

A safe first environment

Begin in a cluster that mirrors production in structure but not in importance. It should run the same kinds of workloads, with the same settings for replicas, probes, limits and policies, but with test traffic only. The closer the match, the more your findings transfer to production. A tiny single-node playground teaches you the tool but not your system.

The readiness checklist

Installation in plain terms

Both tools are installed through their official Helm charts or manifests. The installation creates the custom resource definitions, the controller or operator, and the node-level or per-run components, plus the dashboard or portal. Put the tool in its own namespace, pin a specific version instead of “latest,” and read the release notes before upgrading, since custom resource formats occasionally change between versions. Keep the install settings in version control so that the cluster setup can be reproduced.

Minimizing the footprint

Install only what you need. Many Chaos Mesh features are optional, and you may switch off fault types you do not plan to use, which reduces the attack surface. For Litmus, install only the experiments from the hub that you will really run. A smaller catalog is easier to review and safer to hand to colleagues.

An experiment is only as useful as your ability to see its effects. Observability turns a fault into a lesson.

The three signals

Marking the experiment on the graph

The most useful habit is putting a visible marker on your dashboards when each experiment starts and ends. Both tools can help: Chaos Mesh publishes events and can integrate with Grafana through a data source and annotations, and Litmus provides an exporter that publishes experiment status as metrics, which you can overlay on application graphs. With a marker, anyone looking at a sudden spike can tell at once whether it came from the planned fault or from something else.

What to watch for each kind of fault

Do your alerts fire?

An overlooked benefit of chaos is testing your alerting. If you kill a database replica and no alert fires, that is a finding: either the system is fine and the alert is not needed, or the system is in trouble and nobody will know. Ask after every experiment: “Would the on-call person have been told, and would the message have been useful?”

Recording what you learn

Take screenshots of the dashboards, note the start and end times, write down what you expected and what happened. A short experiment report, including the hypothesis, the settings, the observed behavior and the follow-up actions, is the lasting product of the exercise. Without it, the knowledge lives in one person’s head.

The hypothesis is the heart of chaos engineering. A good one makes the experiment meaningful, and a vague one makes it theater.

Defining steady state

Steady state means “the system is behaving normally,” expressed in measurable terms that reflect user experience. Good steady-state measures are things like the share of successful checkouts, the 95th percentile response time, the number of orders processed per minute, or the age of the oldest message in a queue. Poor measures are things like “CPU is below 70 percent,” which describe machinery and not value.

Using SLOs

If your team already has service level objectives, such as “99.9 percent of requests succeed within 500 milliseconds,” they are the best steady-state definition. They state what users are promised. A chaos experiment then asks: “Can we keep this promise while the fault is active?” The error budget gives another useful angle, since a chaos run spends a bit of budget on purpose, and you can plan for how much you are willing to spend.

Writing a hypothesis

A good hypothesis has four parts, written in plain words:

• The fault: what exactly will we do?
• The scope: to which target, and for how long?
• The prediction: what do we expect to remain true?
• The abort rule: what result will make us stop at once?

Example: “When we kill one of the five product-catalog Pods in staging for a single event, the checkout success rate will stay above 99.5 percent and the replacement Pod will be ready within 40 seconds. We will stop if the success rate drops below 97 percent for more than 30 seconds.”

Predict first, then test

Ask the team to write down predictions before the run. Often people are surprised by how much they disagree. Disagreement is itself a finding: the system is not understood well enough, and the experiment shows who has the better model of it.

Good and weak hypotheses

Here is a ladder of experiments, from gentle to demanding. Each one has a plain question and a typical lesson. Climb it one step at a time.

How to use the ladder

Do not move up until the current step is boring: the system passes, the alerts are right and the team can run it without drama. Fix what you find at each rung and repeat the experiment to prove the fix worked. A passed experiment becomes a regression test that can run on a schedule forever.

The goal is not a few heroic tests, but a steady habit. Pipelines make the habit automatic.

Where chaos fits in delivery

Making experiments pipeline-friendly

• Deterministic enough: Results should not vary wildly between runs. Pin the target, fix the duration and use probes with sensible tolerances.
• Fast: A pipeline gate that takes an hour will be skipped. Keep gating experiments to a few minutes and run longer ones on a schedule.
• Clear exit codes: The pipeline needs a simple pass or fail from the result object, with a link to details.
• Self-cleaning: Every fault must be reverted even if the pipeline is cancelled, which is why durations and clean-up policies matter.
• Isolated: Run in a dedicated namespace or cluster, so parallel pipelines do not interfere with one another.

GitOps style

Because experiments are Kubernetes objects, they can be stored in Git and applied by the same tools that deploy your applications. A change to an experiment goes through review, just as code does. This makes it visible who changed the blast radius, and when, and it makes rollbacks trivial.

Handling flaky results

If an experiment fails sometimes and passes at other times, do not just rerun it until it passes. Investigate. Flakiness is often the true signal: the system is marginal, and the experiment sits right at the edge. Either fix the weakness or tune the experiment to a fairer threshold, but do both on purpose, with a note of why.

A tool that can kill Pods and cut networks is a powerful tool. Treat it like one.

The risks to understand

Chaos tools need elevated rights. The node-level agent of Chaos Mesh and the helper Pods of Litmus can reach into other containers’ namespaces, change network rules and consume resources. If an attacker or a careless user gained control of these, they could cause an outage far larger than any planned experiment, or use the privileges as a way to escape a container. Governance is about making sure that does not happen.

Controls to put in place

A protection against yourself

The tool’s own namespace and the cluster’s core components, such as the control plane, DNS and the networking layer, should be excluded from ordinary experiments. Deliberately testing them is a separate, advanced exercise with its own approvals, never a side effect of a loose label selector.

Approvals and change management

Match approval effort to risk. A small experiment in a test namespace needs no meeting. A production experiment needs a short written plan, an owner, an on-call contact informed, a time window, and an abort rule. A cross-team experiment needs the agreement of the owners of every service that might be touched. Write down these levels so people know what is expected without having to ask.

Compliance and customer trust

Some regulated industries expect evidence of resilience testing. Chaos experiment reports, with hypotheses and results, serve well as evidence. At the same time, do not run experiments that could expose or corrupt customer data without a careful review by the people responsible for data protection.

A game day is a planned session where a team deliberately breaks things and practices responding. It trains people as much as it tests systems.

Why people need practice

Systems are only half of resilience. The other half is people: Do they know where to look? Can they find the right dashboard in the first two minutes? Does the runbook still match reality? Do they know who to call? These questions are answered only by practicing under mild pressure.

The roles

A simple game day schedule

Measuring game day results

Useful numbers include the time from the start of the fault to the first alert, the time to the first person looking at the right dashboard, the time to diagnosis, and the time to recovery. Track them across game days. Falling numbers mean the team is getting stronger, even if the scenarios become harder.

The blameless rule

A game day must be safe for people. If someone makes a mistake during the exercise, the lesson is about the system that allowed the mistake, such as a confusing runbook, and not about the person. Teams that blame people during a drill soon stop volunteering for drills.

Stateless services are forgiving. Databases, queues and caches hold data that matters, so experiments need extra care and different questions.

Why state raises the stakes

When a stateless Pod dies, another identical one takes over. When a database Pod dies, the data it held, the position of the leader, and the consistency of replicas all matter. A badly planned experiment can lose or corrupt information. A well-planned one can prove that your replication, failover and backups really work.

Questions worth asking

Rules for stateful experiments

• Have verified backups first. Restore at least once to prove they work before you start breaking things.
• Use test data. Run first on copies or generated data, not customer records.
• Add integrity probes. Check not only availability but also correctness, for example by writing known records during the fault and reading them back afterwards.
• Move slowly. Prefer short, single faults to start, and understand the recovery behavior before combining faults.
• Involve the data owners. The people who understand the data’s meaning should review the plan.
• Prefer cluster-aware faults. Application-specific experiments, such as those in the Litmus catalog for popular databases, understand roles like leader and replica.

The cache stampede lesson

One of the most common discoveries is the cache stampede. When the cache is emptied or restarted, every request goes to the database at once. The database slows, requests time out, clients retry, and the overload deepens. The fix may be cache warming, request coalescing, limits, or jittered retries. A simple cache-kill experiment finds this safely before a real restart does.

Most real outages in distributed systems are about the network and about how services talk to each other. This is where chaos experiments often pay off the most.

The fallacies we forget

Designers of distributed systems often assume, without noticing, that the network is reliable, that delays are zero, and that bandwidth is infinite. None of these is true. Network chaos reminds the software of the real world by adding delay, jitter, loss, duplication, reordering, corruption, bandwidth limits and full partitions.

What each network fault teaches

The retry storm

Here is a classic pattern. A dependency becomes slow. Callers time out and retry. The retries add load to the already struggling dependency, so it becomes even slower, causing more timeouts and more retries. A small slowdown becomes a complete outage. A network delay experiment reveals this behavior quickly.

flowchart LR
  A["Dependency becomes slow"] --> B["Callers hit timeouts"]
  B --> C["Callers retry at once"]
  C --> D["Load on dependency rises"]
  D --> E["Dependency gets even slower"]
  E --> B
  F["Remedies: backoff with jitter, retry budgets, circuit breakers, load shedding"] -.->|"break the loop"| C

Reading the diagram

The loop on the right is a vicious circle, and each lap makes things worse. The dotted line shows where good engineering interrupts it. Exponential backoff spaces out retries. Jitter randomizes them so that clients do not retry in lockstep. A retry budget caps retries to a small fraction of total traffic. A circuit breaker stops calling a dependency that is clearly failing. Load shedding lets the dependency refuse excess work quickly instead of collapsing.

Service meshes and chaos

If you run a service mesh, you gain another route for fault injection. Meshes can inject delays and aborts at the proxy level for particular routes or percentages of requests, which is excellent for application-layer faults such as “make five percent of calls to the payments route return an error.” Chaos Mesh also offers an HTTP fault type for this style of testing. Network-layer faults and application-layer faults complement each other: the first tests the transport and the second tests your logic.

Be careful with shared infrastructure

Network faults can accidentally affect more than intended, especially when applied to Pods on a shared node. Scope by Pod and direction as tightly as possible. Test the experiment on a single Pod first and confirm in the graphs that only the intended path was affected.

Resources are shared, and sharing is where surprises hide. Stress experiments test the walls between neighbors and the speed of your growth reflexes.

What resource limits really do

A CPU limit throttles a container that tries to use more than its allowance, making it slower but keeping it alive. A memory limit is harsher: a container that exceeds it is terminated and restarted. Requests, on the other hand, decide how Pods are placed on nodes and what share of CPU each gets under contention. Many problems come from setting requests and limits by guesswork. Stress experiments replace guesswork with evidence.

Experiments in this family

Autoscaling under test

Horizontal autoscaling is a feedback loop. Metrics are collected, averaged, compared with a target, and the replica count is adjusted, with delays at each step. A stress experiment can show how long that loop really takes. Typical findings include:

• The metrics pipeline lags by a minute or more, so scaling begins late.
• New Pods take a long time to become ready because of image pulls or slow startup, so extra capacity arrives after the damage.
• The cluster has no spare node capacity, so new Pods sit pending until a node is added.
• The scaling target is based on CPU, but the true bottleneck is something else, such as a connection pool.
• The maximum replica count is set too low, so the autoscaler hits the ceiling.

Noisy neighbors and eviction

Run a memory stress experiment on one Pod and watch the others on the same node. If your neighbors suffer, your limits or priorities need attention. When a node runs low on memory, Kubernetes evicts Pods, and the order depends on quality-of-service class and priority. Experiments show whether the right Pods are evicted first, and whether critical services have the priority they deserve.

Testing capacity headroom

Combine a node drain with a load test and ask: “If we lose one node at peak hour, do we still serve everyone?” This tells you whether your headroom is adequate, or whether your cost-saving efforts have quietly removed the margin you need.

Eventually you will want to test bigger failures: a machine, a whole building’s worth of machines, or the cluster’s own brain.

Node failures

Nodes fail in many ways. They may crash outright, lose network, run out of disk or memory, or have a stuck container runtime. Kubernetes marks a silent node as not ready after a grace period, then evicts its Pods after a further delay, so rescheduling can take minutes. During that time, traffic may continue to flow to Pods that are on the lost node. Node experiments show how long the whole recovery chain really takes in your cluster, and whether your application survives the wait.

Zone failures

Cloud providers group machines into zones that fail independently. A good setup spreads replicas across zones using topology spread rules and anti-affinity. You can simulate a zone loss by draining or isolating every node in one zone, and verify that capacity remains and that data replicas are still reachable. This often exposes services that were accidentally placed together, or storage volumes that are tied to one zone.

Control plane concerns

The control plane decides where Pods run and keeps the records. Running Pods generally continue running while the control plane is briefly unavailable, but nothing new can be scheduled, scaled or repaired. Testing control plane faults is risky and often impossible on managed services, where the provider owns the machinery. The safer approach is to test what you control: does your application keep serving when it cannot reach the API server for a few minutes? Do your operators and controllers handle errors without crashing?

Component faults worth testing

The importance of dependencies you forgot

Draw a picture of everything your application needs in order to start, not just to run. Container registries, secret stores, configuration services, certificate authorities and DNS all appear on that picture. Many teams find they can survive losing a database but not losing a registry during a deployment.

This is an illustrative story, not a report about a real company. It shows how the ideas in this guide fit together in a small team.

The setting

A fictional online shop, “Corner Basket,” runs about twenty services on Kubernetes. Orders, payments, catalog, search and a recommendation service sit behind a single storefront. After a stressful holiday weekend where a slow payment provider caused a long outage, the team of eight decided to try chaos engineering.

Month one: foundations

They did not install a tool at first. They wrote down their steady state: checkout success above 99 percent, page response under one second for 95 percent of requests. They built a dashboard for it, checked that every service had readiness probes and more than one replica, and found that three services had only one replica. They fixed those, and had improved resilience before running a single experiment.

Month two: first experiments

They installed a chaos tool in their staging cluster, limited to the staging namespaces and protected by login. Their first experiment killed one catalog Pod. The checkout rate dipped to 96 percent for 25 seconds. The reason was that the catalog Pod accepted traffic before its cache had loaded, and the readiness check only tested that the port was open. They fixed the readiness check to wait for the cache, and the experiment then passed with no visible dip.

Month three: the dependency that was not optional

The team made the recommendation service fail completely. The storefront should simply have omitted the “you may also like” block. Instead, the whole page failed after a 30-second wait, because the code called the service without a timeout and treated its response as required. They added a short timeout and a fallback that showed popular items. A cheap experiment had found the exact weakness that had hurt them during the holiday, in a different dependency.

Month four: the payment provider

They added 3 seconds of delay to calls to the payment provider’s address. Checkout threads piled up, the order service ran out of workers, and unrelated order lookups started to fail. They introduced a separate worker pool for payments, a circuit breaker, and a clear message to the shopper when payment was slow. They repeated the test and saw that only checkout slowed, while everything else remained healthy.

Month five: automation and a game day

The team added four experiments with probes to their release pipeline, and a monthly game day for the on-call rotation. In the first game day, it took 11 minutes for anyone to find the right dashboard. They reorganized the dashboards and the runbook, and the next game day took 3 minutes.

Month six: production, carefully

After six months, with strong monitoring, small blast radii and an agreed stop rule, they ran a single Pod kill on a low-risk production service during business hours with the on-call engineer watching. It passed. They scheduled it monthly. Later, a change broke it, and the experiment caught the problem on its next run.

What changed

Learn from the mistakes others have already made. These are the most common ways that chaos engineering efforts fail or cause harm.

Chaos engineering grows in stages. Knowing where you are helps you choose a sensible next step.

flowchart LR
  S0["Stage 0: Foundations - monitoring, probes, replicas"] --> S1["Stage 1: Manual experiments in test"]
  S1 --> S2["Stage 2: Scheduled experiments with probes"]
  S2 --> S3["Stage 3: Pipeline gates and game days"]
  S3 --> S4["Stage 4: Careful production experiments"]
  S4 --> S5["Stage 5: Resilience as a shared culture"]

What each stage looks like

Reading the diagram

The chain runs from left to right and every box depends on the one before. Stage 0 comes first, because without foundations the other stages have nothing to stand on. The final stage is a state of mind, not a tool: people ask “what happens if this fails?” in design reviews, long before any experiment is run.

Measuring the program

• Share of critical services with at least one automated experiment.
• Number of weaknesses found and fixed, and the average time to fix.
• Time to detect and time to recover in game days.
• Incidents caused by failure types you have already tested, which should fall toward zero.
• Confidence reported by the on-call team.

Short answers to questions that come up again and again.

You have walked through the ideas, the tools, the habits and the traps. Here is the guide in a nutshell.

Your first week

Back to top