“Make it more reliable” is a wish. “Keep ninety-nine point nine percent of requests successful” is a plan.

The trouble with feelings

Without numbers, reliability becomes an argument. The product manager says the service feels slow, the engineer says it looks fine in the dashboard, and the on-call person says they have not slept in three days. Everyone is describing a different slice of the same reality, and nobody can say who is right. Decisions get made by whoever speaks loudest or was woken most recently.

Numbers change the conversation. When a team agrees in advance what “good enough” means, and measures it the same way every day, debates turn into questions with answers. Did we meet the target this month? How much room do we have left? Is this incident serious or just noisy? Those questions can be answered from data, not from mood.

Why 100 percent is the wrong target

It is tempting to aim for perfection. But perfection is impossible, and chasing it is extremely expensive. Every extra “nine” of reliability costs far more than the last, because you must remove ever-rarer failures, add redundancy and slow down changes. There is also a point beyond which your users cannot tell the difference: if their phone connection fails once in a hundred requests, making your service fail once in a million adds no real value to them.

The wise approach is to choose a target that is high enough that users are happy and low enough that you can still change the system. The gap between that target and perfection is the error budget: the amount of unreliability you are allowed to spend on deployments, experiments and the ordinary accidents of running software.

A story of two teams

Five short terms carry the whole idea. Learn them once and the rest of this guide will feel natural.

How the terms fit together

Why the SLO is stricter than the SLA

A sensible team keeps its internal SLO tighter than the promise it makes to customers. The gap gives early warning: you will notice trouble and fix it while you are still inside the contract. If the SLO and the SLA are the same number, the first alarm you receive will already be a breach, with money or reputation attached.

Kubernetes gives you powerful self-healing, and it also adds new layers where things can go wrong without any user noticing at first.

Many layers between the user and your code

A request from a customer passes through several layers before reaching your application and several more on the way back. There is the user’s network, a cloud load balancer, an ingress controller or gateway, possibly a service mesh, the Kubernetes service that spreads traffic, the pod itself, and then the dependencies your application calls. A failure in any layer looks the same to the user: the page did not load.

flowchart TD
    A["User or client"] --> B["Cloud load balancer"]
    B --> C["Ingress or gateway"]
    C --> D["Service mesh sidecars"]
    D --> E["Kubernetes service routing"]
    E --> F["Application pods"]
    F --> G["Databases, caches and other services"]
    F -.-> H["Nodes, network and storage underneath"]
        

Reading the diagram from top to bottom shows why SLOs must be measured from the user’s point of view. A dashboard that only watches pods can show everything green while an overloaded ingress drops requests before they ever arrive. The SLO belongs to the experience, not to any single component.

What Kubernetes gives you

What Kubernetes adds to worry about

Three levels where SLOs live

The best SLI is the one that moves when your users are unhappy and stays still when they are not.

Start from what users do

Before choosing any metric, list the few things users actually do with your service. For an online shop it might be browsing products, adding to the basket and paying. For a messaging app it might be sending and receiving messages. These are user journeys, and each deserves its own SLO, because a failure on the payment journey matters far more than a slow loading of a rarely used settings page.

The common families of SLIs

Matching SLI types to workloads

Prefer a few strong SLIs

A common mistake is to define dozens of indicators for every service. That produces dashboards that nobody reads and alerts that nobody trusts. A better rule is to pick one or two SLIs per user journey, usually availability and latency, and add others only when users clearly suffer in ways those two miss.

A simple ratio, carefully defined, is the foundation of everything else in this guide.

The universal formula

Almost every useful SLI can be written the same way: the number of good events divided by the number of valid events, expressed as a percentage. A good event is one the user would call a success, such as a request that returned a correct response quickly. A valid event is one that counts at all, which excludes things you do not want to hold the service responsible for, such as requests the user got wrong.

This shape has a lovely property. It always sits between zero and one hundred percent, it works for requests, jobs and records alike, and it makes the error budget a simple subtraction. Any measurement that can be turned into this ratio can become an SLI.

Deciding what counts as good and valid

Where to measure: the closer to the user, the better

Traps that make an SLI lie

Choosing the number is as much about people and money as it is about technology.

Reading the nines

Reliability targets are often described by the number of nines: ninety-nine percent is two nines, ninety-nine point nine is three nines, and so on. Each extra nine cuts the allowed failure by a factor of ten, and it also makes the engineering far more demanding. The table below shows how much full outage each target allows over a rolling twenty-eight-day window.

How to pick a target

Choosing the window

An SLO is measured over a time window, and the choice matters. A rolling window, such as the last twenty-eight days, moves forward continuously, so a bad day slowly leaves the record instead of vanishing at midnight on the first of the month. A calendar window resets on a fixed date, which matches how business reporting works but creates odd behaviour: a disaster on the thirtieth leaves almost no consequences, while one on the second hangs over the team for weeks.

Many teams prefer rolling windows of twenty-eight days, because that contains exactly four weeks and therefore the same number of weekends and weekdays every time. This avoids false changes in the numbers caused purely by the calendar.

The budget is only one subtraction away from the SLO, but doing it carefully makes every later decision easier.

Two ways to count the budget

The error budget can be stated in time or in events. The time version says how long the service may be completely broken, which is easy to talk about but only exact for simple on-or-off failures. The event version says how many requests, or jobs, may fail, which handles partial failures properly, because a problem that fails half the requests for ten minutes spends only half as much budget as total failure for ten minutes.

A worked example in words

Keeping the books

The remaining budget tells you where you stand. The burn rate tells you how quickly things are changing.

The definition in plain words

Burn rate is the current rate of errors divided by the error rate that the SLO allows. If the target allows one failure in a thousand and right now one request in a thousand is failing, the burn rate is one: you are spending the budget at exactly the pace that finishes it at the end of the window. If ten in a thousand are failing, the burn rate is ten, and the budget will be gone in a tenth of the window.

How burn rate translates into time

Why this matters more than the raw error rate

A raw error rate of one percent means different things for different services. For a service with a ninety-nine percent target it is exactly on pace, so it is not an emergency. For a service with a ninety-nine point nine percent target it is a burn rate of ten, a serious problem. Burn rate normalises errors against what each service can afford, so the same alerting rule works across services with different targets.

An alert should wake someone only when users are really being hurt and a human can really help.

Why simple alerts fail

The traditional approach is to alert when the error rate goes above a threshold for a few minutes. It has two problems. A short spike triggers a page even though it costs almost no budget, which trains people to ignore alerts. And a slow leak just under the threshold never triggers anything, yet quietly drains the whole budget over weeks. Neither behaviour matches what you actually care about, which is whether the SLO is in danger.

Multiple windows and multiple burn rates

The recommended pattern combines several alerts. Each alert watches a burn rate over a long window and over a short window at the same time, and fires only when both exceed the same threshold. The long window shows that the problem is real and significant. The short window shows that it is still happening right now, so the alert clears soon after recovery instead of staying red for hours.

flowchart TD
    A["Measure burn rate over several windows"] --> B{"Burn rate above 14.4 over 1 hour and 5 minutes?"}
    B -->|"Yes"| C["Page the on-call engineer now"]
    B -->|"No"| D{"Burn rate above 6 over 6 hours and 30 minutes?"}
    D -->|"Yes"| E["Page the on-call engineer"]
    D -->|"No"| F{"Burn rate above 1 over 3 days and 6 hours?"}
    F -->|"Yes"| G["Open a ticket for working hours"]
    F -->|"No"| H["No alert"]
        

The ladder reads from the most urgent condition to the least. A sudden severe outage trips the first rule within minutes. A moderate problem trips the second after a few hours. A slow, quiet leak trips the third as a ticket, giving the team time to fix it in working hours without waking anyone.

Rules that keep alerts healthy

The good news is that a typical cluster already produces most of the raw material you need.

Where the numbers come from

flowchart LR
    A["Ingress, mesh and application metrics"] --> D["Metrics store"]
    B["Kubernetes state and resource metrics"] --> D
    C["Synthetic probes"] --> D
    D --> E["Recording rules compute SLIs per service"]
    E --> F["SLO and budget calculations"]
    F --> G["Dashboards"]
    F --> H["Burn rate alerts"]
        

Raw metrics are collected in one place, and rules compute the SLI for each service as a ratio of good to valid events. Those ratios then feed budget calculations that power both human-friendly dashboards and automated alerts. Keeping one pipeline for everything means the dashboard and the alert can never disagree.

Practical points that save pain

A service that answers correctly after thirty seconds has, for most users, failed.

Why averages lie

Imagine a service where ninety-nine requests take one tenth of a second and one request takes ten seconds. The average is about two tenths of a second, which sounds wonderful. Yet one user in a hundred waited ten seconds, and if that user is performing a payment, they may leave forever. Averages smooth away exactly the experiences that cause complaints. Latency SLIs should look at the slow tail, not the middle.

The threshold-based latency SLI

The cleanest way to define a latency SLI is to use the same good-over-valid ratio as everywhere else. Choose a threshold, such as three hundred milliseconds, and count the valid requests answered faster than that. The SLI is the share of requests under the threshold. An SLO then says, for example, that ninety-five percent of requests must be under three hundred milliseconds and ninety-nine percent under one second.

Using two or three thresholds paints a better picture than one. The first describes the typical experience, the second protects against the slow tail, and together they stop teams from improving the average while quietly worsening the worst cases.

How histograms make this possible

Metrics systems usually record latency in histograms: a set of buckets that count how many requests took less than certain amounts of time. If a bucket boundary lines up exactly with your SLO threshold, the count in that bucket divided by the total count is the SLI directly, with no approximation. If the threshold falls between bucket boundaries, the result is an estimate, and the error depends on how wide the buckets are.

The most common Kubernetes workload is also the easiest place to start with SLOs.

Stateless services handle requests, keep no important data in the pod, and can run as many identical copies as needed. They fit SLOs naturally, because requests are discrete events that are easy to count. They also benefit most from Kubernetes features such as replication, rolling updates and autoscaling.

A starter set of SLOs

The usual enemies of stateless SLOs on Kubernetes

Not everything on Kubernetes answers requests. Different workloads need different promises.

Databases and other stateful services

Databases hold data that cannot be recreated by starting a new copy, so their SLOs must include more than request success. Availability and latency of queries still matter, but durability, replication health and recovery time matter just as much. A database that answers quickly but loses data has failed in the worst possible way.

Batch jobs and scheduled jobs

For a job that runs once a night, counting requests makes no sense. The better question is whether the work finished correctly and on time. Two SLIs cover most needs: the share of runs that complete successfully, and the freshness of the output, measured as the time since the last successful run. A freshness SLO such as “the report is never more than twenty-six hours old” protects the people who rely on the data, regardless of which individual run failed.

Streaming and queue-based systems

Systems that move messages between services care about delay and loss. The key SLIs are the share of messages processed within a time limit, which is end-to-end freshness, and the share delivered without loss or duplication. Consumer lag, meaning how far behind the readers are, is a useful signal, but the SLO should be about the delay a user would feel, not the raw size of the backlog.

If your applications sit on the platform, the platform’s reliability is a ceiling on theirs.

Platform teams run the shared pieces that every workload depends on: the cluster control plane, scheduling, networking, DNS, ingress and storage. Their customers are the application teams inside the company, and the same SLO thinking applies. The Kubernetes project itself defines scalability targets for things like API call latency and pod startup time, which shows that even the platform has measurable promises.

Making the platform a real service

Kubernetes ships with many tools for reliability. Most SLO failures come from using them half-way.

How each feature protects a specific SLI

The shutdown race that spends budget at every release

When a pod is told to stop, two things happen at roughly the same time: it receives a termination signal, and it is removed from the list of pods that receive traffic. These are not perfectly synchronised, so for a short moment requests may still arrive at a pod that has started shutting down. If the pod exits immediately, those requests fail. The fix is simple: add a short delay before shutdown begins, let the application finish in-flight requests when it receives the signal, and give the pod enough grace time to do both. Teams that fix this often recover a surprising share of their error budget.

Knowing the total budget is useful. Knowing what spends it is how you improve.

Every failure that users feel spends some budget. If you record, after each incident, how much it cost and why, a pattern emerges within a few months. Some causes are frequent and cheap, such as brief errors during rollouts. Others are rare and expensive, such as a zone outage. Both deserve attention, but they need different remedies, and the account of spending tells you where effort pays off most.

A habit that pays off: tag every budget loss

A chain is only as strong as its weakest link, and with microservices the chains are long.

Reliability multiplies down a chain

If a request must pass through several services in a row, and each must succeed for the request to succeed, the overall reliability is roughly the product of their individual reliabilities. Three services that are each ninety-nine point nine percent reliable give about ninety-nine point seven percent overall. Five such services give about ninety-nine point five percent. Every service you add to the critical path lowers the best reliability you can offer unless you protect against its failures.

flowchart LR
    subgraph S["Serial chain: every step must work"]
        A["Gateway 99.9%"] --> B["Orders 99.9%"] --> C["Payments 99.9%"] --> D["Database 99.9%"]
    end
    subgraph P["Redundant pair: either can work"]
        E["Replica A 99%"] --- F["Replica B 99%"]
    end
    S --> R1["About 99.6% overall"]
    P --> R2["About 99.99% overall"]
        

The diagram shows the two basic shapes. In a serial chain, each step adds a chance of failure, so four steps at three nines each allow about four times as much failure as any one of them alone, which is roughly ninety-nine point six percent overall. In a redundant arrangement, a request fails only if both copies fail at the same time, so two copies at ninety-nine percent each can reach roughly four nines, provided their failures are truly independent.

What this means for setting targets

Protecting yourself from dependencies

A budget with no consequences is just a chart. The policy is what makes it matter.

An error budget policy is a short document, agreed by product, engineering and operations, that says in advance what happens at different levels of budget. It removes arguments in the heat of the moment. When the budget is healthy, teams may move fast and take risks. When it is low, they shift effort toward reliability. When it is exhausted, new feature releases pause until the service recovers. Because everyone agreed in advance, the decision feels fair rather than personal.

flowchart TD
    A["Check remaining error budget"] --> B{"More than half remaining?"}
    B -->|"Yes"| C["Ship normally and allow planned experiments"]
    B -->|"No"| D{"Any budget left?"}
    D -->|"Yes"| E["Slow risky changes and prioritise reliability fixes"]
    D -->|"No"| F["Freeze feature releases and focus on reliability work"]
    F --> G["Resume when budget recovers and causes are fixed"]
        

The flow shows the idea in its simplest form. Real policies add detail, such as which kinds of changes are still allowed during a freeze, but the principle is the same: the remaining budget decides how much risk the team may take right now.

What a good policy contains

What the policy gives the business

The cheapest incident is the one you stop at five percent of traffic.

Deployments are the most common cause of budget loss, and they are also the one cause fully in your control. Progressive delivery reduces the damage by sending a new version to a small share of users first, comparing its SLIs with the old version, and promoting it only when it is at least as good. Tools for Kubernetes, such as Argo Rollouts and Flagger, automate this by analysing metrics during the rollout and rolling back when thresholds are crossed.

flowchart LR
    A["New version deployed"] --> B["Send 5% of traffic"]
    B --> C{"SLIs healthy compared with the baseline?"}
    C -->|"Yes"| D["Raise to 25%, then 50%, then 100%"]
    C -->|"No"| E["Roll back automatically"]
    D --> F{"Healthy at each step?"}
    F -->|"Yes"| G["Release complete"]
    F -->|"No"| E
        

The diagram shows the loop. Each step checks the same indicators that the SLO uses, such as error ratio and latency under threshold. A bad release then affects a small fraction of users for a few minutes, spending a tiny part of the budget instead of a large one.

Ways to connect releases to the budget

Writing the rules by hand for every service does not scale. Treat SLOs like any other configuration in your repository.

Why define SLOs in files

An SLO defined in a short, declarative file can be reviewed, versioned and reused. The same file can generate the recording rules, the alert rules and the dashboards, so they never drift apart. When a target changes, the change goes through code review, leaving a record of who agreed and why. This is the same discipline Kubernetes teams already apply to deployments, so it feels natural.

A sensible way to roll out the tooling

SLOs succeed or fail on people and habits at least as much as on tools.

Many organisations try to introduce SLOs for every service in one big push, and then give up when the work overwhelms the teams. A better approach is to start small, prove the value on one or two important services, and let success pull other teams in. The plan below spreads the work across about three months and is meant as a pattern, not a rigid schedule.

Roles that make it work

Teams that do well with SLOs repeat a few good habits. Teams that struggle repeat a few well-known mistakes.

Patterns that work

Anti-patterns to avoid

If you keep only one chapter for your next planning meeting, keep this one.

A quick health check for your SLO practice

The details vary by industry, but the shape of the practice is the same everywhere.

A short end-to-end walkthrough

An SLI is a measurement, an SLO is a target for that measurement over a time window, and an SLA is a contract with customers that states consequences if a promise is missed. SLOs are usually stricter than SLAs.

Subtract the SLO target from one hundred percent and apply the result to the number of valid events in the window. A ninety-nine point nine percent target over two hundred eighty million requests allows two hundred eighty thousand failures.

Burn rate is the current error rate divided by the error rate the SLO allows. A burn rate of one uses the budget exactly over the window, while a burn rate of ten empties it in a tenth of the time.

Alert on the danger of a breach, not the breach itself. Multi-window burn rate alerts warn early when the budget is draining fast enough to cause trouble, long before the SLO is actually violated.

Close to the user. The ingress or load balancer is a strong default for request-based services, combined with application metrics for correctness and synthetic probes for low-traffic journeys.

They are causes, not user experience. A service can have high CPU and happy users, or low CPU and failing requests. Use them to diagnose problems, and use user-facing ratios as the SLIs.

A rolling window of twenty-eight days is a common choice, because it holds exactly four weeks and avoids calendar effects. Some teams add shorter windows for alerting and longer ones for reporting.

Use synthetic probes to create steady traffic, use longer windows, and require a minimum number of events before an alert fires. A single failed request should not look like a catastrophe.

Use the success rate of runs and the freshness of the output, such as time since the last successful result. Also check the quality of the data, since a job that finishes with empty output is not really a success.

Either lower the target honestly, or add protection such as caching, fallbacks, queues and redundancy. You cannot reliably exceed a hard dependency’s reliability without such measures.

Usually yes, because users experience the failure regardless of the reason. If you want to exclude some maintenance, do it through a clear, documented rule, and prefer designing maintenance so that it causes no user-visible errors at all.

Follow the agreed policy. Typically feature releases pause, reliability work gets priority, and changes resume once the budget recovers and the main causes are fixed. Security fixes and reliability improvements are normally still allowed.

Yes. A budget that is always mostly unused may mean the target is too loose or the team is too cautious. Consider releasing faster, running experiments or tightening the target if users would benefit.

No. Probes tell Kubernetes whether a single pod should get traffic or be restarted. SLOs describe what users actually experience across the whole service, including problems that probes cannot see.

Keep it small. One or two per user journey, usually availability and latency, is plenty for most services. Add more only when users clearly suffer in ways the first ones miss.

Kubernetes workloads live in a world of constant change, shared infrastructure and long chains of dependencies. SLOs give that world a simple, shared language: choose what users care about, measure it as good events over valid events, set a target you can honestly meet, and treat the gap as a budget to spend wisely. Burn rate alerts warn you early without waking people for noise. A written policy turns the remaining budget into clear decisions about shipping or fixing. Kubernetes features such as probes, graceful shutdown, spread, replicas and progressive delivery protect that budget, and a ledger of causes shows where to invest next. Start small, keep it honest, and let the numbers replace the arguments.