Kubernetes moves and replaces your programs constantly. Without good measurement, you cannot tell whether the whole system is healthy or quietly failing.

A cluster that keeps changing

Kubernetes is a system that runs your applications, packaged as containers, across many computers. It starts new copies when traffic grows, restarts copies that crash, and moves them between machines during maintenance. This constant change is a great strength, but it also means that the thing you want to watch keeps moving. A program that was on one machine an hour ago may now be on another, with a different address and a different name.

In the old world, an administrator could log in to a single server and look at it. In Kubernetes, there may be hundreds of short-lived pods spread across dozens of machines. Nobody can log in to each one, and many will be gone before anyone could look. A monitoring system must therefore find things automatically, record what it sees, and let people ask questions afterwards.

What monitoring answers

Good monitoring answers a few simple questions. Is the application working right now? Is it getting slower over time? Is any machine running out of memory or disk space? Did the last release make things worse? Will we need more capacity next month? Each question is answered by numbers collected regularly and stored so they can be compared across time.

Metrics, logs, and traces

Teams usually look at their systems in three ways, sometimes called the pillars of observability. Each answers a different kind of question, and they work best together.

Prometheus collects and stores the numbers. Grafana draws them. Alertmanager calls for help. Each does one job well.

The idea in plain words

Prometheus is an open source monitoring system. It works by visiting a list of places on a regular schedule, reading the current numbers that each place offers, and storing them together with the time they were read. This is called scraping. Because Prometheus goes and asks, rather than waiting to be told, it is called a pull-based system.

Grafana is a separate open source program for drawing charts and dashboards. It does not collect anything itself. Instead, it connects to a data source such as Prometheus, asks it questions, and shows the answers as graphs, gauges, and tables. Alertmanager is a third program that receives warnings from Prometheus and decides who should be told and how.

flowchart LR
    A["Applications and exporters expose metrics"] -->|"scraped over HTTP"| P["Prometheus stores time series"]
    P -->|"queries with PromQL"| G["Grafana dashboards"]
    P -->|"alerts from rules"| AM["Alertmanager"]
    AM --> N["Chat, email, or paging tools"]
        

Read the figure from left to right. Applications and helper programs called exporters offer their numbers on a web address. Prometheus scrapes those addresses and stores the results. Grafana asks Prometheus questions, using a query language called PromQL, to draw dashboards. Meanwhile, rules inside Prometheus watch for trouble and send alerts to Alertmanager, which forwards them to the right people.

The vocabulary you will use

The monitoring stack is built from ordinary Kubernetes objects. Knowing them makes every later chapter easier.

Why labels matter so much here

In a cluster where pods come and go, Prometheus cannot keep a fixed list of addresses. Instead, it asks Kubernetes, “which pods and services exist right now?”, and then selects the ones it should scrape, using labels. The same labels are attached to the collected numbers, so that a chart can later show “requests for the checkout service in the production namespace”.

Permission to look

To see the pods and services of a cluster, Prometheus itself needs permission. It runs with a ServiceAccount, and access rules grant it the right to list and watch pods, services, and machines. Without these permissions, Prometheus cannot discover anything, and its target list stays empty, which is one of the commonest beginner problems.

Before wiring things together, it helps to understand what Prometheus actually stores and how it asks for it.

What a metric looks like

A metric is a name, a value, and a set of labels. For example, a metric that counts web requests might carry labels saying which service produced it, which method was used, and what status code came back. Every different combination of labels is a separate series of values over time, called a time series. Prometheus stores each one as a list of readings, each with a timestamp.

Labels are powerful, because they let you slice the same metric many ways. You can ask for all requests, only failed requests, or only failed requests from one service. But labels also have a cost, because each new combination creates a new series that must be stored. We return to this in the design chapters.

The four kinds of metric

How scraping works

Each target offers its current numbers as plain text at a web address, usually a path called metrics. Prometheus visits every target on a fixed interval, commonly every fifteen to thirty seconds. Each visit returns the current value of every metric that target offers. Prometheus adds a timestamp and appends the values to its store.

Prometheus also records a special metric for every target, showing whether the last scrape succeeded. This simple “up or down” value is the most useful health check of all, because it tells you at once whether a target is reachable.

Asking questions with PromQL

PromQL is the language for querying stored series. It can select series by name and labels, compute rates from counters, add up values across many pods, and calculate percentiles from histograms. A typical question in words might be: “For each service, what is the per-second rate of failed requests over the last five minutes, as a share of all requests?” The answer is a new series that can be drawn as a line or used in an alert.

A cluster has many layers, and each layer offers its own numbers. Knowing the main sources tells you which questions you can answer.

flowchart TD
    P["Prometheus"] --> K["kubelet and cAdvisor: container CPU and memory"]
    P --> NE["node-exporter: machine metrics"]
    P --> KS["kube-state-metrics: object state"]
    P --> AP["API server and control plane components"]
    P --> APP["Your applications, using client libraries"]
    P --> IN["Ingress controller and other add-ons"]
        

Two components with confusingly similar jobs

Beginners often confuse kube-state-metrics and the smaller metrics-server. They answer different questions. kube-state-metrics describes the desired and actual state of objects, for example that a deployment wants five replicas but only three are ready. It is a source of history for Prometheus. Metrics-server, in contrast, only supplies the current usage of pods and machines, to support autoscaling and quick checks. It keeps no history and offers no dashboards.

Grafana turns rows of numbers into pictures people can read in seconds. Understanding its main ideas helps you build dashboards that people actually use.

Data sources

Grafana holds no monitoring data itself. It connects to data sources, which are the systems that do. Prometheus is the most common one for metrics. Grafana can also connect to systems that store logs and traces, and even to ordinary databases. A single dashboard can mix panels from several data sources, which is very useful when you want metrics, logs, and traces of the same incident on one screen.

Dashboards, rows, and panels

A dashboard is a page. It is made of panels, and each panel shows the result of one query as a line chart, bar chart, gauge, table, or single number. Panels can be grouped into rows, and rows can be collapsed. A dashboard is stored as a text description, in a format called JSON, which means it can be saved, shared, and kept under version control.

Variables make one dashboard serve many

Without variables, you would need a separate dashboard for every service and every environment. With variables, one dashboard can show any namespace, any deployment, or any pod, chosen from a drop-down. The drop-down’s list is often filled automatically from the labels that Prometheus has seen, so new services appear without anyone editing the dashboard.

What to put on a dashboard

Two simple methods keep dashboards focused. The first is called the RED method and is used for services that handle requests. It asks for three things: the rate of requests, the number of errors, and the duration of requests. The second is called the USE method and is used for resources such as machines and disks. It asks for utilisation, saturation, and errors. Following these methods keeps dashboards useful and prevents endless walls of unrelated charts.

Now that each piece is familiar, let us see how they are placed together inside a real cluster, and why the pieces are arranged that way.

The pieces and where they live

In a typical cluster, the monitoring tools live together in their own namespace, often called monitoring. A namespace is simply a named room inside the cluster that keeps related things together and apart from everything else. Putting the monitoring tools in one room makes it easy to control who can enter and to see at a glance what belongs to monitoring.

Inside that room you will usually find a Prometheus server, one or more Alertmanager pods, a Grafana pod, a kube-state-metrics pod, and a node-exporter pod on every machine. You will also find an operator, which is a helper program that looks after Prometheus for you. We will meet the operator properly in a moment.

Why an operator helps

Setting up Prometheus by hand means writing a long settings file that lists every target. In a cluster where pods appear and vanish all day, that file would be out of date within minutes. The Prometheus Operator solves this by adding new kinds of objects to Kubernetes. These new kinds are called custom resources, and they are defined by a mechanism called a custom resource definition, or CRD.

With the operator installed, you can create a ServiceMonitor object that says, “please scrape every service that carries this label.” You can create a PrometheusRule object that holds alerting rules. The operator watches for these objects, rewrites Prometheus settings automatically, and tells Prometheus to reload. You describe what you want, and the operator makes it so. This is the same declarative idea that Kubernetes itself follows.

The kube-prometheus-stack bundle

Installing all of these pieces one at a time would be slow. Most teams instead use a Helm chart named kube-prometheus-stack, which installs the operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, a set of ready-made dashboards, and a set of ready-made alerting rules. Helm is a package manager for Kubernetes, and a chart is one package. One install gives you a working monitoring system in minutes.

Where the pieces should run

Monitoring must survive trouble, or it is useless exactly when needed. Many teams place the monitoring pods on machines that are less likely to be crowded, and give them priority so that Kubernetes does not evict them first when memory runs low. They also give Prometheus a persistent volume, which is a disk that keeps its data when the pod moves or restarts. Without it, every restart would erase the history.

The most surprising part of Prometheus in Kubernetes is that nobody tells it where the pods are. It finds them by asking Kubernetes itself.

Service discovery

Pods in Kubernetes come and go, and their addresses change every time. A fixed list of targets would be useless. Prometheus therefore uses a feature called service discovery. It connects to the Kubernetes API, the central source of truth for the cluster, and asks for a list of pods, services, or endpoints. It then keeps watching, so it hears about every new pod almost immediately and forgets pods that are gone.

Discovery only gives Prometheus a raw list. Which of those things should actually be scraped is decided by rules called relabeling. Relabeling looks at the labels attached to each discovered item, such as its namespace, its name, and any annotations, and decides whether to keep it, drop it, or rename its labels. This is how one Prometheus can watch hundreds of pods but only scrape the ones you care about.

sequenceDiagram
    participant Op as Operator
    participant K as Kubernetes API
    participant P as Prometheus
    participant Pod as Application pod
    Op->>K: Watch for ServiceMonitor objects
    K-->>Op: New ServiceMonitor for checkout
    Op->>P: Update scrape settings and reload
    P->>K: Ask for endpoints matching the label
    K-->>P: List of pod addresses
    loop Every scrape interval
        P->>Pod: Request the metrics page
        Pod-->>P: Current numbers as text
    end
    P->>P: Store samples with timestamps
        

Follow the figure from top to bottom. First, someone creates a ServiceMonitor. The operator notices, updates the Prometheus settings, and asks it to reload. Prometheus then asks the Kubernetes API which pod addresses match the label. Finally, on every scrape interval, Prometheus visits each address, reads the text of numbers, and stores them with the current time.

What a scrape looks like

A scrape is an ordinary web request. Prometheus asks a pod for a page, usually named metrics, and the pod replies with plain text. Every line has a metric name, optional labels, and a number. Prometheus reads each line, attaches extra labels that describe where it came from, such as the namespace and pod name, and stores the result.

Prometheus also records a special metric named up for every target. Its value is one when the scrape succeeded and zero when it failed. This one number is the fastest way to learn that a target has stopped answering, and many teams alert on it.

How the storage works

Prometheus keeps recent data in memory and writes it to disk in blocks that cover two hours each. Before data reaches a block, it is first written into a log called the write-ahead log, so that a crash does not lose the last few minutes. Older blocks are later merged into larger ones and finally deleted when they pass the retention limit.

The data is compressed very well, often to just one or two bytes per sample. Even so, the number of distinct series matters more than the number of samples, because every series needs an index entry and memory while it is active. This is why cardinality, the number of different label combinations, is the main thing that makes Prometheus grow.

How a query is answered

When Grafana sends a PromQL question, Prometheus finds all the series that match the names and labels in the question, reads their samples for the requested time range, and applies the maths, such as working out a rate per second. Results are sent back as lists of points. Because everything is time stamped, Prometheus can answer questions about any moment in the past that is still within retention.

Recording rules

A recording rule is a saved question whose answer Prometheus works out on a schedule and stores as a new metric. For example, instead of adding up thousands of series every time a dashboard opens, a rule can compute the total once a minute. The dashboard then reads one simple series. Recording rules make dashboards fast and make alert conditions easier to read.

Dashboards are for looking. Alerts are for when nobody is looking. Good alerting means the right person is told about the right problem at the right time, and no more.

Alerting rules

An alerting rule is a condition written in PromQL, together with a waiting time. For example, “the error rate has been above five percent for ten minutes.” Prometheus checks the rule at regular intervals. When the condition first becomes true, the alert is called pending. If it stays true for the full waiting time, the alert becomes firing and is sent to Alertmanager.

The waiting time is important. Without it, a brief blip would wake someone in the night. With it, only problems that last long enough to matter cause a message.

What Alertmanager does

Prometheus only decides that something is wrong. Alertmanager decides what to do about it. It has several jobs, each of which solves a real problem that appears once you have many alerts.

flowchart TD
    R["Alerting rule in Prometheus"] -->|"condition true"| PEND["Pending"]
    PEND -->|"still true after waiting time"| FIRE["Firing"]
    FIRE --> AM["Alertmanager"]
    AM --> DEDUP["Remove duplicates"]
    DEDUP --> GROUP["Group related alerts"]
    GROUP --> ROUTE{"Route by labels"}
    ROUTE -->|"severity critical"| PAGE["Paging service"]
    ROUTE -->|"severity warning"| CHAT["Team chat channel"]
    ROUTE -->|"team platform"| MAIL["Email"]
    PEND -->|"condition false again"| OK["Resolved"]
        

In the figure, an alert starts as pending and only becomes firing if the trouble lasts. Alertmanager then removes duplicates, groups related alerts, and uses labels to choose where each one goes. Critical problems may page a person, warnings may go to a chat channel, and some may simply become emails.

The Watchdog alert

What if the alerting system itself breaks? You would hear nothing, and silence would look like good news. To guard against this, the bundle includes a special alert named Watchdog that is always firing. It is sent to Alertmanager constantly. Some outside service is set up to expect this message and to raise the alarm if it ever stops arriving. This is a dead man’s switch, and it proves the whole alert path is alive.

Writing alerts people can act on

The best alerts describe symptoms that users feel, such as slow pages or failing requests, rather than every small cause, such as one busy processor. An alert should also say what is wrong, how bad it is, and where to look next. A link to a runbook, which is a short page of steps for handling that alert, turns a frightening midnight message into a manageable task.

Severity levels

Follow one number on its whole journey, from a moment inside an application to a line on a chart, and finally to deletion.

Stage by stage

What happens when a pod disappears

When a pod is deleted, Prometheus stops finding it in service discovery, and the scrapes stop. The old series is marked stale, which means it ends cleanly instead of showing as a flat line. The stored history for that pod remains on disk until retention removes it. This lets you investigate a pod that crashed an hour ago even though it no longer exists.

What happens when Prometheus restarts

On restart, Prometheus replays the write-ahead log to rebuild what was in memory. This can take a few minutes on a large server, and during that time scrapes may be missed. If the disk is lost, the history is lost too. This is why persistent volumes, and for important data, remote copies, matter.

Sending data elsewhere

Prometheus can forward copies of its samples to another system using a feature called remote write. The other system may be a long-term store, such as Thanos, Mimir, or a hosted service. This is how teams keep years of history without making one Prometheus server enormous, and how they combine several clusters into one view.

Clicking around to build a dashboard is fine for learning. In production, every dashboard should live in version control, so that it survives rebuilds and can be reviewed like any other change.

Why not just click?

If a dashboard exists only in the Grafana database, it can be lost when the pod is rebuilt, changed by anyone without a record, and difficult to copy to a second cluster. Teams that rely on hand-made dashboards often discover, at the worst moment, that the one dashboard everyone loved has vanished.

Provisioning

Grafana supports provisioning, which means loading data sources and dashboards from files when it starts. In Kubernetes, these files are stored in ConfigMaps. A ConfigMap is a Kubernetes object that holds small pieces of text, like settings or files, so that pods can read them without baking them into an image.

The bundle includes a small helper container, called a sidecar, that runs beside Grafana. The sidecar watches the cluster for ConfigMaps that carry a particular label. Whenever it finds one, it copies its content into the folder where Grafana looks for dashboards, and asks Grafana to reload. To add a dashboard, you simply create a labelled ConfigMap. To remove it, you delete the ConfigMap.

flowchart LR
    GIT["Dashboard files in version control"] --> CI["Pipeline applies to cluster"]
    CI --> CM["Labelled ConfigMap"]
    CM --> SC["Sidecar watches for labels"]
    SC --> GF["Grafana loads the dashboard"]
    GF --> USER["Team sees the same dashboard everywhere"]
        

The figure shows the path. Dashboard files are kept in version control. A pipeline applies them to the cluster as ConfigMaps. The sidecar sees the label and loads them into Grafana. Every cluster that runs the same pipeline ends up with the same dashboards.

Data sources as code

The same idea applies to data sources. Instead of adding Prometheus by hand in the Grafana screen, you provide a file that names it and gives its address. On startup, Grafana reads the file and connects. If the address changes, you change one file and redeploy.

Variables and reuse

Good dashboards use variables, such as namespace or pod, so that one dashboard serves every service. This reduces the number of dashboards to maintain. When you find that you are copying a dashboard and changing one name, add a variable instead.

Change control for dashboards

Because dashboards are files, changes can be reviewed by a colleague before they are applied. A pull request shows exactly which panels were added or which queries were changed. If a change turns out to be a mistake, it can be reverted in seconds. Grafana can be set to prevent edits from the screen for provisioned dashboards, so that the version-controlled file remains the single source of truth.

Monitoring knows a great deal about your system, and that knowledge is valuable to attackers. Protect it as carefully as the applications it watches.

Why monitoring needs protection

Metrics reveal names of services, versions of software, numbers of users, and the layout of the cluster. Labels sometimes carry sensitive information by mistake, such as email addresses or customer identifiers. Grafana can also hold credentials for many data sources. Someone who gains access could learn how to attack you or could read private business numbers.

Access to Kubernetes

Prometheus needs permission to read the Kubernetes API in order to discover pods. This is given through role-based access control, or RBAC, which is a system of roles that list exactly what an account may do. Prometheus should receive a role that can only read the object kinds it needs, never the ability to change things. The same is true of kube-state-metrics, which should have read-only rights.

Protecting the metrics pages

Every application that exposes metrics offers a page anyone on the network can read. Inside a cluster, that means any pod could read it unless you say otherwise. A NetworkPolicy is a Kubernetes rule that limits which pods may talk to which. Use it so that only Prometheus can reach the metrics pages, and only trusted users can reach Grafana and Alertmanager.

Where higher assurance is needed, metrics can also be served over encrypted connections, and Prometheus can be told to present a certificate or a token when it scrapes. Some cluster components, such as the kubelet, already require an authenticated request.

Logging in to Grafana

Grafana ships with a default administrator account, and its first password must be changed right away. Better still, connect Grafana to your company login system, using a standard such as OpenID Connect, so people sign in with their normal accounts. Then map groups to roles, so that most people can only view dashboards, some can edit them, and very few can administer the system.

Protecting the alert channels

Alertmanager holds secrets such as chat webhook addresses and paging keys. Store them in Kubernetes Secrets, restrict who can read those Secrets, and never put them in dashboards or in shared documents. Also make sure that alert messages do not contain private customer data, since chat channels are often widely read.

No tool is perfect. Knowing where Prometheus and Grafana shine, and where they struggle, helps you decide what to build around them.

The main trade-offs

Most decisions with this stack are trade-offs between detail and cost. More labels, more frequent scrapes, and longer retention all give a richer picture, but each one increases memory, disk space, and the time queries take. A good team decides deliberately how much detail each service deserves.

Comparing with other approaches

When this stack is a good fit

It is a strong choice for teams that run Kubernetes, want detailed numbers about services and machines, and are willing to learn a little. It is a weaker choice if you need only simple uptime checks, or if nobody has time to look after it. In those cases a managed service may cost less overall.

Patterns are ways of arranging things that many teams have found to work. Anti-patterns are habits that look sensible at first but cause pain later.

Helpful patterns

Anti-patterns to avoid

A short list of habits that keep the stack healthy, and the mistakes that most often cause trouble.

Best practices

Common mistakes

A gentle start plan

If your team is new to all this, begin in three steps. First, install the bundle and look at the ready-made dashboards for nodes and pods. Second, add monitors for your most important service and build one overview dashboard for it. Third, add just two or three alerts about user-facing symptoms, with clear owners. Grow from there only when real questions demand it.

Prometheus exports metrics about itself. Use them, because a monitoring system that fails quietly is worse than none at all.

Numbers to watch about Prometheus

Prometheus offers detailed numbers about its own work. A few of them tell you most of what you need to know. Watch how many series are active, how fast samples are arriving, how long scrapes and rule checks take, how full the disk is, and how much memory the process uses.

Alerts for the stack itself

The bundle already includes rules for common problems. They warn when Prometheus cannot reach a target, when it fails to send alerts, when its disk will fill soon, when a rule takes too long, and when the number of series jumps suddenly. Read those rules once so you know what they cover, and add your own for anything unique to your setup.

Logging

Prometheus, Alertmanager, and Grafana all write logs to their standard output, which Kubernetes collects. Send those logs to your normal log system. When something goes wrong, such as a rule failing to load or a data source refusing a connection, the log will usually state the reason in plain words.

When Prometheus grows too big

Signs of trouble include restarts because of memory, very slow dashboards, and gaps in the charts. There are several remedies, and you should choose the gentlest one that works.

Troubleshooting the usual suspects

Three short stories show how the ideas in this book work in practice. The details are typical rather than taken from any one company.

Example 1: A small team gets its first monitoring

Example 2: Finding a slow leak

Example 3: Many clusters, one view

What the examples share

In every story, the team began small, focused on symptoms that mattered to users, and kept history long enough to answer real questions. They treated dashboards and alerts as things that need care, not one-time setup. None of them tried to measure everything on the first day.

Short answers to questions that beginners often ask.

Prometheus alone can store data and show simple graphs, but its screen is basic. Grafana gives you much better dashboards and can combine several data sources. Most teams use both.

Pulling lets Prometheus know at once when a target is unreachable, keeps control of the scrape schedule in one place, and makes it easy to view a target’s metrics page by hand. For short jobs that end before a scrape, a helper called the Pushgateway exists, but it should be used sparingly.

Metrics-server only keeps the latest processor and memory numbers so that features like autoscaling and the top command work. It does not store history or offer queries. Prometheus stores history, supports queries, and drives alerts. Many clusters run both.

Cardinality is the number of distinct series a metric produces, which depends on the combinations of label values. Each series uses memory and disk. If a label can take millions of values, the number of series explodes and Prometheus can fail.

A ServiceMonitor finds targets through Kubernetes services and their endpoints. A PodMonitor finds pods directly, which is useful when pods have no service. Both tell Prometheus what to scrape without editing its main settings.

It depends on your questions. Two weeks is enough for most day-to-day investigation. If you compare with last month or last year, or need capacity planning, send data to a long-term store rather than making local disks huge.

Run two identical copies that scrape the same targets and send alerts to the same Alertmanager cluster. Alertmanager removes the duplicates. Note that each copy holds its own data, so a query may see slightly different values from each.

Yes. It can scrape any machine or service that exposes metrics, and exporters exist for databases, message queues, network gear, and more. Prometheus does not have to live in the same place as its targets, as long as it can reach them.

Prometheus handles numbers only. Logs and traces need other tools. Grafana can show all three side by side, which lets you jump from a spike on a chart to the related logs and traces.

Because that is what a counter does: it only increases, apart from resets when a program restarts. To see how fast it is changing, ask for its rate over a time window. That gives requests per second, errors per second, and so on.

They are a great starting point, but check them first. Confirm the queries match your label names, remove panels you do not need, and review any links or plugins. Treat them as drafts that you then own.

Usually not by itself. High processor use is often perfectly fine. Alert on effects users feel, such as slow responses or errors. Use processor use as supporting evidence on a dashboard, or alert only when it stays extremely high for a long time and threatens to cause harm.

A last look at the whole picture, in plain words.

A cluster with no monitoring is like a hospital ward with no charts, no monitors, and no nurse station. Things may go well for a while, but when they do not, nobody knows until it is too late. Prometheus, Grafana, and Alertmanager together give the cluster a memory, a face, and a voice: Prometheus remembers the numbers, Grafana shows them, and Alertmanager speaks up when something is wrong.

Prometheus finds pods by asking Kubernetes, scrapes their metrics pages on a schedule, and stores each number with a time. The operator turns simple Kubernetes objects into real settings, so monitoring can be described as configuration and kept in version control. Recording rules make heavy questions fast, alerting rules watch for trouble, and Alertmanager makes sure the right people hear about it, and only once.

Grafana turns those numbers into pictures. Dashboards built around clear methods, and stored as files, let a whole team see the same truth. Security matters at every step: read-only permissions, network rules, single sign-on, and care with labels. Finally, the stack must watch itself, because a quiet monitor may be a broken one.