A game day is a planned event where a team practices handling failure on purpose, in a controlled way, and learns from the result.

The idea in plain words

Imagine telling your team: “On Thursday afternoon, between two and four, we will switch off one of the machines in our cluster, and we want to see what happens.” Everyone knows it is coming. Observers are watching charts, someone is taking notes, and a person with authority is ready to stop the exercise if anything goes badly. When the machine goes dark, the team watches whether workloads move elsewhere, whether alerts fire, whether users notice, and whether the people involved know what to do. At the end, the team sits down together and writes down what surprised them.

That is the entire concept. A game day combines three things that are each valuable alone and far more valuable together: a deliberate failure, careful observation, and honest conversation afterward.

What a game day is not

A game day is not a surprise attack on your own colleagues, and it is not an attempt to prove that someone made a mistake. It is not random destruction of production systems for entertainment. It is also not a replacement for good design. Game days reveal weaknesses; they do not repair them. The repair comes from the work that follows, and the main product of a game day is a list of well-understood improvements.

Why practice matters even for well-built systems

Teams often believe that a well-designed cluster does not need rehearsal. Design, though, only describes how a system should behave. Reality differs because of small configuration mistakes, settings copied from an old project, assumptions that were true last year, and dependencies nobody remembered. Rehearsal reveals the gap between the intended behavior and the actual behavior, and it does so when you can afford the discovery.

Complex systems rarely break in the way their designers imagined. Understanding why helps you choose what to practice.

Reliability and resilience are different

Reliability describes how often a system works as intended. Resilience describes how well it recovers and keeps serving when something goes wrong. A system can be reliable in calm conditions and still fragile, because it has never faced a hard day. Resilience is the ability to absorb a hit, adapt, and continue. Kubernetes was designed with resilience in mind: it restarts failed containers, moves workloads off failed machines, and keeps desired state in a replicated store. But those features only help if your applications and your configuration use them correctly.

The gap between a feature and a guarantee

Kubernetes offers many resilience features, but each depends on settings that you control. Running several copies of an application protects you only if those copies are not all placed on the same machine. A health check protects you only if it actually detects the failure. Automatic scaling helps only if capacity exists and the new copies can start quickly. A feature that is present but misconfigured gives a dangerous false sense of safety, and the only way to find out whether it truly works is to test it.

The practice grew from a simple discovery: the best way to trust a system is to watch it handle failure.

A short history

Large technology companies learned decades ago that waiting for failures to appear on their own was too costly. Some began rehearsing disasters on purpose. One well-known early example was a tool that randomly shut down servers in production during working hours, so engineers were forced to build services that survived such losses. Other organizations ran large exercises in which teams practiced recovering from simulated data center outages, and cloud providers popularized the term game day for hands-on events where teams work through realistic failures together. Over time, these ideas were collected into a discipline called chaos engineering, which has published principles that guide how experiments are designed.

The core principles in everyday language

flowchart LR
  A["Define steady state"] --> B["Form hypothesis"]
  B --> C["Inject a controlled failure"]
  C --> D["Observe what really happens"]
  D --> E{"Matches hypothesis?"}
  E -->|"Yes"| F["Confidence gained: widen scope next time"]
  E -->|"No"| G["Weakness found: fix it"]
  G --> A
  F --> A
        

Kubernetes gives you a toolbox for surviving failure. A game day checks that each tool is actually in place and working.

A failure domain is the set of things that fail together. Knowing your domains tells you which failures to rehearse.

What a failure domain is

Failures are rarely truly random. They tend to affect groups of things that share a common cause. All containers on one machine share that machine. All machines in one data center zone share that zone’s power and network. All workloads in a cluster share the cluster’s control plane. Each of these groupings is a failure domain, and resilience is largely about making sure that no single domain, when it fails, takes everything with it.

flowchart TD
  R["Region"] --> Z1["Zone A"]
  R --> Z2["Zone B"]
  R --> Z3["Zone C"]
  Z1 --> N1["Node"]
  Z1 --> N2["Node"]
  Z2 --> N3["Node"]
  Z3 --> N4["Node"]
  N1 --> P1["Pods"]
  N2 --> P2["Pods"]
  N3 --> P3["Pods"]
  N4 --> P4["Pods"]
  CP["Control plane and etcd"] --- R
  D["External dependencies: DNS, identity, databases, registries"] --- R
        

A ladder of domains

Several practices sound similar. Knowing how they differ helps you choose the right exercise for each goal.

A family of resilience exercises

Organizations use a range of exercises to build confidence in their systems, and each answers a different question. A game day is a collaborative, scheduled event focused on people and platform together. Automated chaos experiments run continuously in the background to catch regressions. Disaster recovery drills rehearse large-scale restoration, often with strict targets for time and data loss. Tabletop exercises walk through a scenario in conversation without touching any system. Load tests push a system with traffic to find capacity limits. These practices complement each other, and mature teams use all of them.

A game day is a team sport. Clear roles keep it calm, safe, and educational.

Who does what

Because a game day mixes deliberate disruption with careful observation, it needs a few clearly named roles. Without them, people either crowd around the same problem or assume someone else is watching safety. Assigning roles in advance, and telling everyone what each role means, is one of the cheapest ways to improve the quality of the exercise.

Separating the doers from the watchers

One principle matters above all: the people who cause the failure should not be the people who respond to it. When the same person both injects and fixes a problem, the exercise tests nothing about how the team would really cope. Many teams split participants into a planning group, who know everything, and a responding group, who know only that a game day is happening, and sometimes only the time window. This keeps the response realistic while the planning group maintains safety.

The best scenarios come from real risk, real history, and real worry, not from a list of things that are fun to break.

Where good scenarios come from

The most useful scenarios are drawn from your own world. Review your past incidents and ask which ones could happen again. Ask your engineers what keeps them awake at night. Look at your architecture for places where one component, if lost, would hurt many things. Read the postmortems of other companies, and ask whether the same story could happen to you. Finally, consider the changes you are about to make: a migration, an upgrade, a new region. Rehearsing before such changes catches problems early.

A catalog of failure scenarios for Kubernetes

Without a clear picture of normal and a clear prediction, a game day becomes mere destruction.

Steady state: the picture of normal

Steady state is described with a few numbers that reflect what users experience, not with internal details. Good examples include the percentage of requests that succeed, the time within which most requests complete, the number of orders processed per minute, and the length of important queues. Internal measures like processor usage matter for diagnosis but are poor definitions of health, because a system can use a lot of processor and serve users perfectly, or use little and be badly broken.

The shape of a good hypothesis

A strong hypothesis has four parts: the failure being introduced, the expected effect on users, the expected behavior of the platform, and the expected time to recover. For example: “If one worker machine disappears, user requests continue to succeed above the agreed rate, the lost workloads are rescheduled onto other machines, alerts fire within a minute, and everything returns to normal within five minutes.” Notice that this statement can be proved wrong, which is exactly what makes it useful.

A game day that harms real users has failed, however much was learned. Safety is designed in, not hoped for.

Blast radius: how far can the damage spread?

Blast radius is the portion of your system and your users that an experiment could affect if everything goes wrong. A good plan makes the blast radius deliberately small at first. You might target a single workload, a single machine, a copy of the application used only by internal staff, or a tiny share of traffic. As you gain trust in the process and in the system, you widen the radius step by step.

flowchart LR
  A["Step 1: Test environment, one workload"] --> B["Step 2: Test environment, whole service"]
  B --> C["Step 3: Production, internal users or tiny traffic share"]
  C --> D["Step 4: Production, one node or one zone in off-peak hours"]
  D --> E["Step 5: Production, wider scope during working hours with full readiness"]
        

Abort conditions: deciding the limits in advance

Before the exercise begins, the team agrees on exact conditions that trigger an immediate stop. These should be measurable and tied to user impact. Examples include the success rate falling below a stated number, response time exceeding a stated limit, error budget burning faster than a stated rate, or any sign of data loss. Deciding limits in calm conditions prevents the temptation to keep going “just a little longer” when the exercise is already causing harm.

Protecting data and people

Be especially careful with anything involving stored data, security controls, or regulated information. Prefer scenarios that test recovery using copies in a separate environment. Make sure that nothing in the exercise exposes secrets, weakens access controls beyond the test scope, or violates agreements with customers. Finally, remember that people are part of the system: schedule exercises at reasonable hours, avoid exhausting responders, and treat the exercise as work that deserves normal rest afterward.

Most of the success of a game day is decided before it starts.

Where should you run it?

A common question is whether to practice in a test environment or in production. Each has strengths. A test environment is safe and allows bold experiments, but it often differs from production in size, data, traffic, and configuration, so the lessons may not transfer perfectly. Production is the real thing, which makes the lessons truthful, but it carries real risk to real users. The usual answer is a progression: begin in a realistic test environment to learn the basics and polish the process, then move carefully to production with a small blast radius when the team is ready.

A readiness checklist

A game day has a rhythm. Following it keeps the event safe and the learning sharp.

flowchart LR
  A["Weeks before: choose scenarios, write hypotheses, approve plan"] --> B["Days before: announce, verify readiness, rehearse the undo"]
  B --> C["Kickoff: roles, rules, abort conditions"]
  C --> D["Scenario run: baseline, inject, observe, respond, restore"]
  D --> E["Hot debrief: first impressions while fresh"]
  E --> F["Days after: full review and action items"]
  F --> G["Weeks after: fix, retest, share findings"]
        

The structure of a single scenario

How long should it take?

For a first game day, a half-day is plenty: perhaps thirty minutes of briefing, three scenarios of about forty-five minutes each including debriefs, and a closing conversation. Resist the urge to cram. Fatigue reduces both safety and learning, and a rushed final scenario is more likely to cause mistakes than insight.

The simplest failures are also the most revealing. Start here.

What happens when a pod dies

When a container exits or a pod is removed, the controller responsible for that workload notices that fewer copies exist than requested and creates a replacement. The scheduler picks a machine, the node agent pulls the image if needed, starts the container, and runs its readiness checks. Only when those checks pass does traffic flow to the new copy. Every one of these steps takes time, and the total time is the window during which you have reduced capacity.

sequenceDiagram
  participant I as Injector
  participant P as Pod
  participant C as Controller
  participant S as Scheduler
  participant N as Node agent
  participant L as Load balancer
  I->>P: Terminate pod
  P-->>L: Stops answering
  L->>L: Removes pod from rotation
  C->>C: Notices missing replica
  C->>S: Request replacement
  S->>N: Place pod on healthy node
  N->>N: Pull image and start container
  N->>L: Readiness check passes
  L->>L: Adds new pod to rotation
        

Questions to answer during the exercise

Common findings

Machines fail, get patched, and get replaced constantly. Your platform should treat that as routine.

Three kinds of node events

A node can leave service in different ways, and each tests something different. A planned drain politely asks workloads to move away, respecting disruption budgets, and is what happens during maintenance and upgrades. An unplanned crash gives no warning: the node simply stops responding, and the control plane waits for a timeout before declaring it lost and rescheduling its workloads elsewhere. A partial failure leaves the node alive but unhealthy, such as a full disk or a stuck container runtime, which is often the hardest to detect and handle.

Capacity: the question behind every node failure

When a machine disappears, its workloads need somewhere to go. If the cluster has no spare room, replacement pods stay pending, and users feel the loss for as long as it takes to add capacity. Node failure exercises are, in essence, capacity exercises. They answer: how much headroom do we really have, how quickly can the autoscaler add machines, and do critical workloads win over less important ones when space is tight?

flowchart TD
  A["Node lost"] --> B["Workloads need new homes"]
  B --> C{"Spare capacity available?"}
  C -->|"Yes"| D["Pods scheduled quickly"]
  C -->|"No"| E["Pods stay pending"]
  E --> F["Autoscaler requests new nodes"]
  F --> G{"Quota and supply available?"}
  G -->|"Yes"| H["New node joins after provisioning delay"]
  G -->|"No"| I["Prolonged shortage: priorities decide who runs"]
  H --> D
        

Zone failures are rare, but they are the failure that multi-zone designs exist to survive.

What a zone outage tests

Cloud regions are divided into zones, each with independent power, cooling, and networking. Spreading a cluster across several zones is meant to survive the loss of one. But surviving requires that every layer cooperates: workloads must be spread across zones, enough capacity must exist in the remaining zones, data must be reachable or replicated, and the load balancing in front must stop sending traffic to the lost zone. A zone exercise tests all of these together, and it almost always finds a gap.

Networks rarely fail cleanly. They get slow, lossy, and confusing, which is far harder to handle than a clean break.

Why partial failure is worse than total failure

When a dependency is completely down, clients get an immediate error and can respond sensibly. When it becomes slow or drops some messages, clients wait, retry, wait again, and pile up. Threads fill with waiting requests, memory grows, and the slowness of one service spreads to everything that depends on it. This chain reaction, often called a cascading failure, is one of the most common ways large systems collapse.

flowchart LR
  A["Dependency becomes slow"] --> B["Callers wait longer"]
  B --> C["Waiting requests pile up"]
  C --> D["Callers run out of workers or memory"]
  D --> E["Callers become slow or fail"]
  E --> F["Their callers suffer in turn"]
  F --> G["Cascading failure"]
        

Network experiments worth running

The defensive habits these exercises test

What happens to the things you are running when the thing that manages them goes away?

A crucial distinction: running versus managing

The control plane decides what should run and records the plan. The machines, meanwhile, run the actual containers and handle actual traffic. If the control plane becomes unavailable, containers already running generally keep running, and traffic between them usually keeps flowing, because node agents and networking continue with the last instructions they received. What stops is change: no new deployments, no scaling, no replacement of failed pods, and no ability for humans to inspect or fix the cluster through the usual tools. This is a different failure from losing application capacity, and rehearsing it teaches your team how much it depends on the cluster’s management layer.

flowchart TD
  A["Control plane unavailable"] --> B["Running pods continue serving"]
  A --> C["No new pods are scheduled"]
  A --> D["Failed pods are not replaced"]
  A --> E["Autoscaling stops reacting"]
  A --> F["Humans cannot use normal management tools"]
  B --> G["Risk grows if a node also fails during the outage"]
  C --> G
  D --> G
        

Hidden dependencies on the control plane

Some applications unexpectedly depend on the cluster’s management layer while running. A workload that queries the cluster interface for configuration, a controller that must be alive to refresh credentials, or a leader election that stops when the lease cannot be renewed can all break when the control plane is unreachable. Rehearsing the outage finds these quiet dependencies.

Stateless workloads can be replaced. Stateful ones carry data, and data is not so easily recreated.

Why state raises the stakes

When a stateless web server dies, a fresh copy takes its place and nothing is lost. When a database dies, the question is whether its data survives and whether another copy can take over without losing recent writes. Kubernetes supports stateful workloads through persistent volumes and specialized controllers, but the safety of your data depends on how the underlying storage is configured, how replication is arranged, and how well your recovery steps have been practiced.

Two numbers that define recovery

Experiments for stateful systems

Many real outages are not caused by hardware. They are caused by changes, expirations, and honest mistakes.

The most common causes of real outages

Industry post-incident reviews repeatedly show that a large share of serious incidents follow a change made by people: a faulty release, a mistaken configuration edit, a certificate that expired unnoticed, or a secret that was rotated in one place but not another. These are among the most valuable things to rehearse, because they are frequent and because your detection and rollback abilities decide how long they last.

Shared clusters are shared fates. One greedy workload can hurt many innocent ones.

How resource pressure causes failure

Every workload on a machine shares its processor, memory, disk, and network with its neighbors. Kubernetes lets you declare how much each workload needs and the most it may use, and it uses those declarations to place workloads and to decide who to stop when resources run out. If the declarations are missing or wrong, a single misbehaving workload can starve others, push a machine into memory exhaustion, and trigger the termination of important neighbors.

What good protection looks like

You can start with almost no tools, and grow into a rich toolbox as your practice matures.

Start simple

Your first game days do not require special software. Deleting a pod, draining a node, scaling a workload to zero, and applying a restrictive network rule all use ordinary Kubernetes features and cloud console actions. Simple, manual injections are easy to understand, easy to undo, and impossible to misconfigure through a complicated tool. Many teams run several successful game days before adopting dedicated chaos tooling.

Categories of tools

Safety features to look for

What you cannot see, you cannot learn from. What you do not say, others cannot act on.

A game day dashboard

Prepare one view that shows everything needed to judge the exercise. It should display the user-facing steady-state numbers at the top, followed by workload health, node health, scheduling backlog, and the signals most relevant to the planned failure. Mark the moment of each injection on the charts, so that observers can see cause and effect plainly. A single shared screen for the room, or a shared link for remote participants, keeps everyone looking at the same facts.

Communication channels

Use a dedicated channel for the exercise, and label every message clearly so nobody confuses an exercise with a real incident. Keep three kinds of messages separate: control messages from the facilitator and safety owner, observations from the observers, and actions from the responders. Inform support teams and other stakeholders before the event, and tell them how to reach the safety owner if they see something that looks like real customer harm.

A resilient system needs resilient people. Game days are one of the few places where teams can practice the human side of an incident.

What good incident response looks like

When something breaks for real, the technical problem is only half of the challenge. The other half is coordination: noticing the problem, deciding who is in charge, dividing work, communicating with affected people, and keeping a record. Teams that have rehearsed this move calmly and efficiently. Teams that have not often make the same few mistakes: too many people investigating the same thing, nobody updating stakeholders, and nobody making decisions.

What to watch for during the response

Runbooks: tested, not just written

A runbook is a step-by-step guide for handling a known problem. Game days are the best test of whether runbooks work. Ask a responder to follow the runbook literally, without their own knowledge, and watch where they stumble. Steps that depend on tools nobody has access to, commands that no longer work, and instructions that assume knowledge only one person has are all common discoveries.

The failure was the easy part. The learning happens in the conversation afterward.

Why blameless matters

A blameless review focuses on how the system and the process allowed a problem to happen, not on who made a mistake. People make errors in every organization; the question is why the environment made the error easy to make and hard to catch. When reviews punish individuals, people hide information, and the organization loses the very facts it needs to improve. When reviews are blameless, people share openly, and the whole team gets smarter together.

Structure of a good review

Turning findings into improvements

A practice that improves nothing measurable will not survive the next budget meeting.

Metrics from the exercises themselves

Metrics from the follow-up work

You do not need to begin with elaborate exercises. Grow the practice in stages, and collect value at each step.

Choosing a rhythm

A common cadence is a larger game day each quarter, with smaller, targeted exercises monthly and automated experiments running continuously. Align the schedule with the rhythm of your organization: avoid your busiest seasons, and tie major exercises to significant changes such as new regions, large migrations, or platform upgrades. Consistency matters more than size; a small regular practice outperforms a grand event that happens once and is never repeated.

The hardest part of game days is often not technical. It is convincing an organization that breaking things on purpose is wise.

Common objections and honest answers

Building the business case

Most failed programs fail in familiar ways. Recognizing them early saves a great deal of effort.

These illustrative stories show the kinds of discoveries teams commonly make. Each follows the same path: hypothesis, surprise, fix, and retest.

Follow a realistic, illustrative game day for an online store to see how the pieces fit together.

The setup

An online store runs on a cluster spread across three zones. The team has never held a game day, so they plan a half-day in their production-like staging environment, with a plan to repeat the most valuable scenario in production later. They choose two scenarios: losing one worker machine, and slowing the connection to a payment provider. They name a facilitator, a safety owner, an injector, a scribe, and two responders who know only the time window. Observers include a product owner who wants to understand the risk.

Scenario one: losing a worker machine

The written hypothesis states that if one worker in the main pool disappears, the success rate of checkout stays above the agreed level, lost pods are replaced within three minutes, and an alert reaches the on-call responder within one minute. The baseline looks healthy. The injector powers off a machine, and the scribe records the time.

Within half a minute, the success rate dips noticeably, then recovers. Replacement pods appear quickly but take two minutes to become ready because the image is large. Observers notice that two of the three copies of a smaller internal service had been on the lost machine. The alert fires after four minutes instead of one, because it is configured with a long evaluation window. The responders restore normal service without needing escalation, but one of them says the dashboard did not show which machine was lost, so it took time to understand the pattern.

sequenceDiagram
  participant F as Facilitator
  participant I as Injector
  participant S as Cluster
  participant R as Responders
  participant O as Observers and scribe
  F->>O: Announce scenario one and confirm baseline
  I->>S: Power off one worker machine
  O->>O: Record injection time
  S->>S: Pods lost, replacements scheduled
  S-->>R: Alert fires after four minutes
  R->>S: Investigate and confirm recovery
  O->>O: Compare events with hypothesis
  F->>R: End scenario and hold hot debrief
        

Scenario two: a slow payment provider

The hypothesis states that if calls to the payment provider become slow, customers can still browse and fill their carts, and checkout shows a clear message within a few seconds instead of hanging. The injector adds delay to calls toward the provider. Within two minutes the safety owner sees the success rate for browsing falling too, far beyond the planned limit, and calls a stop. The delay is removed and the system recovers.

The investigation shows that the browsing service shares a pool of workers with the checkout service. Slow payment calls were holding workers, so browsing starved as well. The hypothesis was wrong in the most useful way possible, because the exercise revealed a dependency that nobody had mapped.

The review

Keep these short definitions handy while planning and reviewing exercises.

Quick answers to the questions teams ask most often when starting out.

It can be, when done carefully: small blast radius, written abort conditions, a named safety owner, a tested undo, and advance notice. Most teams start in a realistic test environment, then move to production gradually as their process and confidence mature.

A common rhythm is a larger event each quarter, smaller focused exercises monthly, and automated experiments running continuously. Consistency matters more than size, and exercises should follow major changes such as migrations and upgrades.

A first exercise can run with six to ten people: a facilitator, a safety owner, an injector, a scribe, a couple of responders, and a few observers. Larger events add more roles, but clarity of roles matters more than headcount.

No. Deleting pods, draining nodes, scaling workloads, and applying network rules are enough for the first several exercises. Dedicated tools help later with repeatability, scheduling, scoping, and safety controls.

Usually they know only that an exercise is happening and perhaps the time window. Advance knowledge of the exact failure makes the response unrealistically smooth. The planning group, including the safety owner, knows everything.

That is exactly what abort conditions and the safety owner are for. Stop immediately, remove the failure using the tested undo, restore service, and treat the event as a learning opportunity, including a review of why the safeguards did not prevent the impact.

A game day is a broad, collaborative exercise on how systems and people handle a failure. A disaster recovery drill focuses on restoring service after a major loss within strict recovery targets. They overlap, and mature teams run both.

Celebrate, then be curious. Perhaps the scenario was too gentle, the observation too shallow, or the hypothesis too vague. Increase the difficulty, widen the scope, or try a more unusual failure.

Yes. You can rehearse failures of your workloads, nodes, network, dependencies, and changes, and test how your tools behave when the provider’s interface is slow or briefly unavailable. You usually cannot break the provider’s control plane, and the provider handles its resilience.

Put them on a calendar, assign an owner for the program, track action items visibly, and retest fixed scenarios. Share success stories with leaders so the practice keeps its support.

Often a platform or reliability team, with participation from application teams. The leader should be someone who is trusted, who communicates clearly, and who cares more about learning than about proving anything.

Invite them as observers, explain the safety measures in plain terms, and share results as short stories linking findings to business risk and recovery targets. Understanding the process turns skeptics into supporters.

Kubernetes provides many tools for surviving failure, but a tool that has never been exercised is only a promise. A game day turns promises into evidence. By choosing realistic scenarios, writing clear hypotheses, limiting the blast radius, naming a safety owner, and watching carefully, teams discover the gaps between what they believe and what is true, at a time and place of their choosing. The real value arrives afterward, in the blameless review, the completed actions, and the retests that prove the fixes work. Practiced regularly, game days improve both the platform and the people who run it, so that real failures become smaller, shorter, and less frightening.