The only way to know your system survives a failure is to watch it survive one, ideally on a day you chose.

Every system fails eventually. Disks fill up, networks drop packets, machines vanish, and programs crash at the worst moment. Kubernetes was designed with this in mind. It restarts crashed containers, moves pods away from dead machines and spreads copies of your application across many nodes. But having these features is not the same as having a system that truly survives. A restart policy that has never been tested might not work the way you imagine. A backup service that has never been called might be pointed at a server that no longer exists.

Failure injection is the practice of causing small, controlled failures deliberately, so you can observe how your system reacts and fix weaknesses before real customers meet them. It is the heart of what the industry calls chaos engineering. The goal is not to cause damage. The goal is to replace hopeful guessing with real evidence.

Chaos engineering is a disciplined experiment, not random destruction.

The idea became famous when a large streaming company built a tool that randomly turned off servers in its production environment. The purpose was to force engineers to build services that did not depend on any single machine. Over time, the practice grew into a careful discipline with a few simple principles. Those principles are what separate chaos engineering from reckless breaking of things.

A few terms appear again and again. Learn them once and the rest of the topic becomes easy to follow.

Every good experiment follows the same loop, from a question to a fix and back to a new question.

The loop begins with a worry. Someone asks, “What happens to our checkout if the payment service becomes slow?” That worry becomes a hypothesis, which becomes a small experiment. After the experiment, the team compares what happened with what they expected, fixes any weakness, and then repeats the same experiment to prove the fix worked. Finally, the experiment is saved so it can be run again automatically in the future.

flowchart LR
  A["Worry or question"] --> B["Hypothesis about steady state"]
  B --> C["Design small experiment"]
  C --> D["Inject the fault"]
  D --> E["Observe and measure"]
  E --> F{"Steady state held?"}
  F -- "Yes" --> G["Grow the scope or automate"]
  F -- "No" --> H["Fix the weakness"]
  H --> C
  G --> A
        

Why a passing result is also valuable

When the steady state holds, the experiment has taught you something useful: your system really can handle that fault. That is real, recorded evidence, and it lets you move on to a harder fault. A team that only celebrates failures will start to fear passing results. A healthy team treats both outcomes as progress.

A cluster is built in layers, and every layer can fail in its own special way.

To choose good experiments, picture the cluster as a stack. At the bottom is the physical or virtual infrastructure: machines, disks and networks. Above it are the Kubernetes nodes and the control plane that manages them. Above that run the pods and containers. Above those are your applications and the services they depend on, and at the very top are the users. A failure at any level shows up differently at the levels above it.

flowchart TD
  A["Users and business outcomes"] --> B["Applications and dependencies"]
  B --> C["Pods and containers"]
  C --> D["Kubernetes nodes and kubelet"]
  D --> E["Control plane: API server, scheduler, etcd"]
  E --> F["Infrastructure: machines, disks, network, zones"]
        

The simplest and most common pattern: stop a pod or a container and watch what happens.

Killing a pod is the entry-level experiment. A pod is the smallest unit Kubernetes runs, and a controller such as a Deployment is responsible for keeping the desired number of copies alive. When you remove one, the controller should notice and create a replacement. The experiment teaches you how fast that happens, what users experience in between, and whether your application handles the abrupt end of a running process.

What this pattern typically reveals

When a whole machine, or a whole data center area, disappears, many pods vanish at once.

A node is a single machine in the cluster. Nodes fail for many reasons: hardware faults, kernel crashes, cloud providers reclaiming discounted machines, network cards failing or a bad update. When a node stops responding, Kubernetes marks it as not ready after a waiting period. After a further delay, pods on that node are marked for removal and their controllers create replacements elsewhere. This delay is a surprise to many teams, because the cluster does not react instantly.

sequenceDiagram
  participant N as Failed node
  participant C as Control plane
  participant K as Controller
  participant O as Other nodes
  N--xC: Stops sending heartbeats
  C->>C: Waits for grace period
  C->>C: Marks node not ready
  C->>K: Pods on node are unreachable
  K->>O: Create replacement pods
  O-->>C: New pods become ready
        

Ways to inject node failures

Zone failure: the bigger test

Cloud clusters often span several failure zones. A zone-failure experiment removes every node in one zone together. It asks the hardest questions: were replicas truly spread across zones, is there enough spare capacity in the remaining zones, do storage volumes tied to one zone block recovery, and does the load balancer stop sending traffic to the dead zone quickly?

Networks are the most common source of strange, partial failures in distributed systems.

A server that is completely dead is easy to detect. A network that is merely slow, lossy or sometimes unreachable is much harder, because everything appears to be working, just badly. Calls hang, retries pile up and timeouts fire at unpredictable moments. Network faults are therefore among the most valuable failures to inject, and they are available in nearly every chaos tool.

The partition and the split-brain problem

A network partition cuts the cluster into groups that cannot reach each other. This is the cruelest test for stateful systems such as databases and message brokers. Each side may believe the other is dead and try to take over. If both sides accept writes, the data diverges, a problem called split brain. A partition experiment shows whether your system chooses safety by refusing service on the smaller side, or chooses availability and risks inconsistency.

When names stop resolving, almost everything breaks at once, so it deserves its own experiments.

Inside Kubernetes, pods find each other by names. A small cluster service, usually CoreDNS, turns those names into addresses. Almost every call between services begins with a name lookup. If lookups fail or become slow, every service that depends on another service suffers, even though all of the actual programs are healthy. For this reason, DNS is a classic hidden single point of failure.

Typical discoveries

Starve a pod or a node of processor, memory, disk or input speed, and see who suffers.

Not every failure is a clean stop. A very common real-world problem is that a resource becomes scarce. A runaway program eats all the CPU. A memory leak slowly fills the machine. A log file fills the disk. A busy disk slows every read and write. Stress experiments create these conditions on purpose so you can watch how pods, neighbors and the node itself respond.

What stress experiments reveal

These experiments are the best way to test your resource requests, limits and quality-of-service choices. If a stressed pod without a limit takes down its neighbors, you have found a missing guardrail. If a pod with a proper limit is killed and restarts cleanly while the neighbors never notice, the guardrails work. They also show whether your alerts fire early enough, before users feel pain.

Sometimes the best way to test a dependency failure is to make the calls themselves misbehave.

Many outages begin when one service answers slowly or with errors, and its callers react badly. You can simulate this without touching machines or networks. A service mesh, which is a layer of small helper proxies sitting beside each pod, can intercept calls between services. Because the proxy sees every request, it can be told to delay a share of them or to answer with an error instead of passing them on. This is called fault injection at the application level, and it is precise, safe and easy to limit to certain routes, certain callers or a small percentage of traffic.

flowchart LR
  A["Caller service"] --> B["Proxy beside caller"]
  B -- "Most requests pass" --> C["Target service"]
  B -- "Small share delayed" --> C
  B -- "Small share answered with error" --> D["Error returned to caller"]
        

What happens to running applications when the brain of the cluster is slow or unreachable?

The control plane is the set of components that manage the cluster: the API server that accepts requests, the scheduler that places pods, the controller manager that keeps desired state, and the data store called etcd that holds everything. A common belief is that if the control plane fails, everything stops. In reality, pods already running on nodes usually keep running, because the node agents continue their work without constant instructions. What stops is change: no new pods are scheduled, no failed pods are replaced, no scaling happens and no updates roll out.

Hidden dependencies on the API server

Many applications quietly depend on the API server while running. Operators that watch resources, autoscalers, leader election mechanisms, and service mesh control components all talk to it. When the API server struggles, these can fail in surprising ways, such as losing leadership and restarting, or scaling incorrectly. Experiments reveal the dependencies you did not know you had.

Stateless pods are easy to replace. Stateful ones carry data, and data makes failure serious.

Databases, queues and caches keep important information on disks that outlive any single pod. Kubernetes handles this using persistent volumes, which are storage pieces that can be attached to a pod and re-attached after a restart. The tricky part is that a volume is often tied to one zone and can only be attached to one node at a time. When a node fails, the replacement pod may have to wait until the old attachment is released, and that wait can be long.

The questions that matter most

A large share of real outages are not hardware failures at all. They are changes that went wrong.

Surveys of incidents in many companies keep finding the same fact: most outages follow a change. A new version with a bug, a configuration edit with a typo, a certificate that expired, an image that could not be downloaded. These are failures of deployment and configuration, and Kubernetes has its own versions of them. You can rehearse these failures just as you rehearse hardware ones.

What good rollout safety looks like

Time is a hidden dependency. When clocks drift or certificates expire, strange failures follow.

Programs assume that the clock is correct. Security certificates have start and end dates. Tokens expire. Schedules depend on the time. Distributed databases use timestamps to order events. If the clock on one machine drifts by minutes, certificates may be judged as not yet valid or already expired, tokens may be rejected, and replicated data may be ordered wrongly. Because these faults are rare in daily life but nasty when they happen, injecting them is highly educational.

Kubernetes decides who receives traffic using health checks. Break the checks and you test the safety net itself.

Kubernetes uses three kinds of probes. A liveness probe asks whether a container is stuck and should be restarted. A readiness probe asks whether a container is ready to receive traffic. A startup probe gives slow starters extra time. These probes are the cluster’s eyes. If they are wrong, traffic goes to sick pods or healthy pods are restarted needlessly. Failure injection can make probes fail on purpose to see what the cluster does.

The shutdown dance

When a pod is told to stop, two things happen at the same time. The pod receives a termination signal, and the cluster begins removing the pod from the list of backends that receive traffic. Because these are separate and not perfectly synchronized, a pod can stop before every router has stopped sending it requests, and those requests fail. Applications that wait a few seconds before exiting, and finish in-flight work, avoid these errors. Experiments that kill pods under load expose the problem clearly.

Components that sit in the request path of the API server can themselves become the reason nothing works.

Many clusters run admission webhooks, which are programs that the API server calls before accepting a new or changed resource. They enforce policy, add defaults or inject helper containers. Because the API server waits for their answer, a slow or unavailable webhook can block deployments entirely, or, depending on its failure setting, allow requests through without protection. This makes webhooks a fascinating target for failure injection.

Autoscalers add safety, but they have delays and assumptions that can be tested.

Autoscaling features add pods when load rises and add nodes when pods cannot be placed. They are valuable but not instant. New pods take time to start, new nodes take minutes to join, and autoscalers depend on metrics that may themselves fail. An experiment can show what really happens during a sudden surge, or when the metrics system is down, or when the cloud has no more machines of the requested type available.

Failure injection is not testing Kubernetes. It is testing the protective habits your applications are supposed to have.

When you inject a fault, you are really checking whether the defenses built into your applications and platform work. Each defense has a common way of being wrong, and failure injection is the best way to find that out. Here are the most important defenses and what experiments reveal about them.

The retry storm

The most famous failure of a well-meant defense is the retry storm. A dependency slows down. Callers time out and retry. The retries add load to the already struggling dependency, which slows it further, which causes more timeouts and more retries. A small problem becomes a large outage that feeds on itself.

flowchart TD
  A["Dependency becomes slow"] --> B["Callers time out"]
  B --> C["Callers retry immediately"]
  C --> D["Extra load on dependency"]
  D --> A
  C --> E["Backoff and retry limits break the loop"]
        

You cannot tell whether an experiment broke something unless you agreed in advance what unbroken looks like.

The steady state is expressed with measurements that reflect what users experience, not what machines feel. CPU usage is a poor steady state signal because users do not care about it. A better signal is the share of requests that succeed, the time taken by the slowest few percent of requests, or the number of orders completed per minute. These are often written as service level indicators, with targets called service level objectives.

Error budgets and chaos

A service level objective implies an error budget, which is the small amount of failure the objective allows. Experiments spend a little of this budget on purpose, in return for knowledge. Teams that are already over budget should fix reliability first and avoid adding risk. Teams with plenty of budget can safely run more ambitious experiments. This link gives chaos work a natural pace and a business-friendly justification.

The skill in failure injection is not breaking things. It is breaking exactly as much as you intended, and no more.

Blast radius is the amount of damage an experiment could cause if it goes wrong. The golden rule is to begin with the smallest possible radius and grow it only as confidence grows. Kubernetes gives many natural ways to shrink the radius, because it has labels, namespaces, replicas and traffic routing that can all be used as boundaries around an experiment.

flowchart LR
  A["One pod in a test namespace"] --> B["One pod in staging"]
  B --> C["Several pods in staging"]
  C --> D["One pod in production with test traffic only"]
  D --> E["Small share of real traffic"]
  E --> F["Whole service or zone in production"]
        

Every experiment needs a way to stop immediately, and ideally a way to stop itself.

Even careful experiments can behave unexpectedly. Abort conditions are rules written before the experiment that define when to stop. They are tied to the same steady state signals you defined earlier. For instance: “If the success rate of checkout falls below ninety-eight percent for two minutes, stop immediately.” Good chaos tools can watch the signal and halt the experiment automatically, which removes the delay of a human noticing and reacting.

You can inject failures with simple built-in commands, with dedicated chaos projects, with service meshes or with cloud services.

There is no single correct tool. Many teams start with plain Kubernetes features because they need nothing extra, then adopt a dedicated chaos platform as their program grows. Several open source chaos projects for Kubernetes belong to the Cloud Native Computing Foundation, and cloud providers offer their own managed fault services. Tools change quickly, so check current documentation before choosing.

Understanding the machinery helps you trust it, secure it and debug it.

Most Kubernetes chaos platforms share a similar design. You describe an experiment as a Kubernetes resource, stating the fault type, the targets and the duration. A controller running in the cluster reads the resource and decides what must be done. For faults that need deep access to a node, such as network disruption or stress, a small helper program runs on each node and carries out the fault using features of the operating system. When the experiment ends or is deleted, the controller instructs the helper to undo the fault.

flowchart TD
  A["Engineer or pipeline creates experiment resource"] --> B["Chaos controller reads it"]
  B --> C["Selects targets by labels and namespace"]
  C --> D["Node helper receives instruction"]
  D --> E["Applies fault using operating system features"]
  E --> F["Timer or abort ends the experiment"]
  F --> G["Helper removes the fault"]
  G --> H["Status and results recorded"]
        

Why this design deserves respect

The node helper needs strong permissions because it changes network rules, uses system resources and can kill processes. That is powerful. If the chaos platform itself were compromised, an attacker could use it to cause damage at a large scale. Treating the chaos platform as high-privilege infrastructure, with careful access control, is therefore essential.

Who may break what, and when, is a governance question as much as a technical one.

In an organization with many teams, experiments on one team’s services can hurt another team’s users. A healthy program therefore sets rules. Services must opt in before they can be targeted. Only trained people may run experiments, and only certain fault types are allowed in each environment. Production experiments need extra approval. These rules protect trust, which is the most precious asset of a chaos program.

A game day is a rehearsal for a whole team, combining experiments with human practice.

Individual experiments teach systems. Game days teach people. In a game day, a group gathers, often for a few hours, to run planned failures while practicing how they would notice, diagnose and fix the problems. The group includes the engineers who own the services, someone to run the experiment, someone to watch the dashboards and someone to take notes. The best game days are friendly, structured and focused on learning.

sequenceDiagram
  participant F as Facilitator
  participant R as Responders
  participant S as System
  F->>R: Brief the plan and safety rules
  F->>S: Capture baseline metrics
  F->>S: Inject the first fault
  R->>S: Detect, diagnose and respond
  S-->>F: Signals and recovery time
  F->>S: Stop faults and verify recovery
  F->>R: Debrief and record actions
        

One-time experiments decay. Resilience stays real only when the experiments keep running.

A service that survived a pod kill in March may fail in September after a new feature, a new dependency or a changed configuration. To keep resilience from silently eroding, mature teams automate their experiments. Some run as part of the delivery pipeline, so every new version must pass a small set of fault tests before reaching production. Others run on a schedule in the background, at gentle intensity, all the time.

An experiment you cannot observe teaches you nothing. Your monitoring is the measuring instrument.

Before the first experiment, make sure you can see the system clearly. You need metrics for user-facing signals, logs that show errors, and traces that follow a request through many services. During an experiment, these views should be on a single shared dashboard, with the start and end times of the fault marked clearly so changes line up with causes.

Test your alerts too

Experiments are the best way to test alerting. If a fault that clearly hurts users does not trigger any alert, the monitoring has a gap. Record which alerts fired, how long they took and whether they pointed engineers in the right direction. Fixing alert gaps is one of the most valuable outcomes of any chaos program.

A one-page description for every experiment keeps everyone aligned and safe.

Before running any experiment, write a short card that answers the same questions every time. The card is reviewed by the service owner and anyone who could be affected. After the run, the card is updated with results. Over time, a collection of cards becomes your resilience library and training material for new engineers.

Organizations grow into chaos engineering in stages. Knowing the stages helps you pick the next step.

The hardest part of chaos engineering is rarely the technology. It is making people comfortable with finding problems.

If engineers fear that a discovered weakness will be used to criticize them, they will avoid experiments or hide results. A strong program is built on blameless learning: the assumption that people did their best with the information they had, and that weaknesses belong to the system. When a team finds a problem through an experiment, the right response from leaders is thanks, because the team just prevented a future outage.

The same ideas can test how the cluster behaves when part of it is misbehaving or compromised.

Failure injection is usually about accidents, but the same method can test security assumptions. What happens if a pod suddenly tries to connect to places it never should? What if a credential is revoked while a service is running? What if outbound internet access is blocked? These experiments check that your protective controls work and that your applications fail safely when those controls trigger.

If you are not sure where to begin, this ordered list moves from gentle to demanding.

Experienced teams keep meeting the same good and bad shapes. Learn them early.

Patterns that work

Patterns that hurt

A short checklist of habits that separate calm, valuable programs from risky ones.

Three questions before pressing the button

Even careful plans meet surprises. Knowing the usual problems makes them easy to handle.

Problem 1: The fault did not seem to do anything

First confirm that the targets were selected as expected. A typo in a label or a wrong namespace is the most common cause. Next check whether the tool reported the fault as applied. Then verify the fault from the inside, for example by measuring latency between the affected pods. Finally consider that the system may simply be resilient to this fault, which is a perfectly good result.

Problem 2: The fault would not stop

Use the manual stop button first, then delete the experiment resource. If effects remain, check the node helper programs on the affected nodes and restart them if needed. Inspect for leftover network rules, stress processes or edited settings. This is why every experiment needs a documented cleanup procedure and why faults should have short built-in durations.

Problem 3: The damage was larger than expected

Abort at once and restore service. Then run a blameless review. Ask which assumption was wrong: the dependency graph, the capacity estimate or the abort threshold. The lesson usually reveals a hidden coupling that was worth discovering, even if the way it was discovered was uncomfortable.

Problem 4: The results are confusing

Check whether something else changed at the same time, such as a deployment, a traffic spike or a scheduled job. Repeat the experiment under calmer conditions. Make sure your baseline was captured and that time clocks on dashboards line up with the experiment’s timeline.

Different industries focus on different failures, because their customers care about different things.

Applications depend on shared platform pieces such as ingress, load balancers and the network plugin, and those can fail too.

Between your users and your pods sit several shared components. An ingress controller receives requests from the outside and routes them to services. A load balancer in the cloud spreads traffic across nodes. A network plugin gives every pod an address and moves packets between them. A service mesh adds proxies and a control component that configures them. These pieces are run by a platform team, and a failure in one of them can hurt every application at once, so they deserve their own experiments.

The control-plane-versus-data-plane idea

Networking tools often have two halves. The data plane is the part that carries real traffic, such as proxies and routers. The control plane is the part that tells the data plane what to do. A well-designed system keeps carrying traffic even when its control plane is unavailable, because the data plane remembers its last instructions. A key question for many experiments is therefore whether the data plane really survives without its control plane.

Let us follow a single imaginary experiment through every step, so the ideas become concrete.

A small online shop runs a web front end, an order service, a payment service and a database in a Kubernetes cluster. The team is worried about the payment service. It is run by another team, and it has been slow a few times. They decide to find out what happens to orders when payments become slow.

To keep support and funding, show that failure injection makes the business measurably safer.

Leaders often ask whether chaos engineering is worth the effort. The answer lies in a few simple measurements tracked over time. Some measure the work done, some measure what was learned and some measure real outcomes in production. The most persuasive reports connect experiments to incidents that did not happen, or incidents that were shorter than they would have been.

Short, direct answers to the questions people ask most often when they first meet failure injection.

It can be, once you have a measured steady state, small blast radius, automatic abort rules, observability and a history of safe experiments in lower environments. Beginners should start in test and staging.

No. Deleting pods, draining nodes and scaling workloads need nothing extra. Dedicated tools add more fault types, schedules and safety controls when you outgrow the basics.

Ordinary tests check that features work. Failure injection checks how the whole running system behaves when parts fail, including real infrastructure, real networks and real human response.

User-facing signals first, such as success rate, latency and business outcomes, supported by internal signals such as the number of ready pods and error logs.

Start with a game day each quarter and move toward automated experiments in pipelines and on schedules, so that resilience is checked after every meaningful change.

That is a useful result. It records evidence that the system handles that fault, and it lets you move on to a harder one. Save it as a regression test.

The ones that worry you most and are most likely, such as losing a pod, losing a node, a slow dependency and a bad rollout. Use the starter catalog in this guide as a guide.

Yes. They need strong permissions to do their job. Restrict who can create experiments, which namespaces can be targeted and which fault types are enabled, and treat the platform as high-privilege infrastructure.

Begin with a small, safe experiment in a non-critical service and share the lessons. Link the work to outage cost, error budgets and customer trust, and invite skeptics to join the first game day.

No. It tests them. A backup or recovery plan that has never been exercised cannot be trusted, and experiments are how you turn a written plan into a proven one.

Ideally yes, with shared tooling, shared guidelines and central safety rules. Service owners understand their systems best, and shared standards keep everyone safe.

Writing a hypothesis and a measurable steady state before every experiment. It turns random breaking into learning.

Failure injection replaces hope with evidence. By breaking things deliberately, in small controlled ways, teams discover how their Kubernetes systems really behave when pods vanish, nodes die, networks slow down, names stop resolving, disks fill, configurations go wrong and control components stumble. The method is scientific: define a steady state, form a hypothesis, inject a realistic fault, measure the difference and fix what you learn. The craft lies in keeping the blast radius small, having automatic ways to stop, observing carefully and treating every finding as a shared lesson. Done well, it makes systems stronger, people calmer and outages shorter.