Grafana is the window you look through. OpenTelemetry is everything that puts the view behind the glass.

What Grafana is

Grafana is an open-source tool for visualizing and exploring data. You connect it to places where data is stored, you write questions called queries, and Grafana draws the answers as charts, tables, maps, and gauges on pages called dashboards. It can also send you a message when the data crosses a limit you care about, which is called alerting. Grafana is made by a company called Grafana Labs, and there is also a hosted version, Grafana Cloud, that runs it for you.

What OpenTelemetry is

OpenTelemetry, often shortened to OTel, is an open standard and a set of tools for creating and moving telemetry. It gives your code a common way to describe what it is doing, and a common wire format, called OTLP, to send that description onward. It deliberately does not include a place to store the data or a screen to look at it.

So where does Grafana fit?

Grafana sits at the far end of the journey. OpenTelemetry produces and delivers. Storage systems keep. Grafana shows. In a typical setup, data created by the OpenTelemetry SDK travels through a Collector into storage systems, and Grafana then reads from those storage systems to draw your dashboards. Grafana never talks to your application directly.

Two tools exist because two different problems exist. Understanding the split makes every later decision easier.

Problem one: getting data out of programs

Before OpenTelemetry, each monitoring product had its own library for collecting data from your code. If you used Product A, your code was full of Product A calls. Switching meant rewriting. OpenTelemetry solves this: you instrument once, and you can send the data to almost any destination.

Problem two: understanding data

Even when data is collected, a person cannot read millions of raw records. They need charts that show trends, tables that rank problems, links that jump from a symptom to a cause, and alerts that wake the right person at the right time. This is the problem Grafana solves. It does not care how the data was made, as long as it can read it from a storage system.

The benefit of keeping them separate

Because the two are separate, you can mix and match. You can send OpenTelemetry data to Grafana-based storage today and to a different vendor tomorrow, and your instrumentation does not change. You can also show data from non-OpenTelemetry sources, such as cloud monitoring services and databases, on the same Grafana dashboard as your OpenTelemetry data.

A small set of words appears in every conversation about Grafana and OpenTelemetry. Learn them once, and the rest of the guide will be easy.

Three query languages you will hear about

The whole system fits in one picture. Learn it well, because every other chapter zooms into one corner of it.

flowchart LR
    APP["Your Application with OpenTelemetry SDK"] -- "OTLP" --> COL["Collector or Grafana Alloy"]
    COL --> TP["Tempo (traces)"]
    COL --> LK["Loki (logs)"]
    COL --> MM["Mimir or Prometheus (metrics)"]
    TP --> GF["Grafana"]
    LK --> GF
    MM --> GF
    GF --> USER["Engineer in a browser"]

Reading the picture

On the left, your application uses the OpenTelemetry SDK to create traces, metrics, and logs. It sends them using OTLP to a Collector, or to Grafana Alloy, which is Grafana’s own distribution of the Collector. The Collector sorts the data and sends each signal to the right backend. On the right, Grafana connects to those backends and draws what an engineer asks to see.

When people say “Grafana” they sometimes mean one thing and sometimes the whole family. Here is who is who.

The “LGTM” shorthand

People often say LGTM stack, which stands for Loki, Grafana, Tempo, and Mimir. It is a short way to describe the open-source set of tools for logs, dashboards, traces, and metrics. A ready-made single-container version for local learning exists, which is helpful if you want to try everything on a laptop.

Grafana is mostly a very good question-asker. Understanding data sources is the key to understanding what it can and cannot do.

What a data source is

A data source is a saved connection from Grafana to a backend. It holds the address, the login details, and settings such as which tenant to use. Once a data source exists, a panel can send queries to it and show the answer. One dashboard can use many data sources at the same time, so a single page can show metrics from Prometheus, logs from Loki, and database numbers side by side.

flowchart TB
    GF["Grafana Server"] -- "PromQL" --> PR["Prometheus or Mimir data source"]
    GF -- "LogQL" --> LO["Loki data source"]
    GF -- "TraceQL" --> TE["Tempo data source"]
    GF -- "Cloud API" --> CW["Cloud monitoring data source"]
    GF -- "SQL" --> DB["Database data source"]
    PR --> GF
    LO --> GF
    TE --> GF
    CW --> GF
    DB --> GF

The important consequence

Grafana does not keep your telemetry. It asks the backend for it every time a panel loads. This means that storage costs, retention, and speed are decided by the backend. It also means that if the backend is down, your dashboards are empty even though Grafana itself is fine.

Let us follow what happens between the moment someone opens a dashboard and the moment they see the charts.

sequenceDiagram
    participant U as Engineer Browser
    participant G as Grafana Server
    participant P as Prometheus Data Source
    participant L as Loki Data Source
    U->>G: Open dashboard Order Service Health
    G->>G: Load dashboard definition and variables
    G->>P: Query request rate as PromQL
    G->>L: Query error logs as LogQL
    P-->>G: Time series result
    L-->>G: Log lines result
    G-->>U: Panels drawn with fresh data
    Note over U,G: Repeats on every refresh interval

Step by step

First, the browser asks the Grafana server for the dashboard. The server loads the dashboard definition, which is a saved description of the panels, their queries, and the time range. It also fills in variables, which are drop-down choices such as environment or service name. Next, the server sends one query per panel to the matching data source. The data sources answer, and the server hands the results to the browser, which draws the charts. When auto-refresh is on, the whole round repeats every few seconds or minutes.

This explains a common surprise: a dashboard with forty panels sends forty or more queries every time it refreshes. If each query is heavy, the backend works hard, and the dashboard feels slow. Good dashboards keep queries light and the panel count reasonable.

Grafana can only show what the backends hold. So the real question is how OTLP data reaches Tempo, Loki, and a metrics store.

Option one: through a Collector or Alloy

The most common design places a Collector between your applications and the backends. Your SDKs send OTLP to the Collector. The Collector adds helpful labels, groups data into batches, and forwards each signal to its own backend. Grafana Alloy is a popular choice here, because it understands OTLP and also Prometheus-style scraping in the same program. A plain OpenTelemetry Collector works just as well, since the Grafana backends accept standard formats.

Option two: directly from the SDK

Modern Grafana backends accept OTLP directly. Tempo accepts OTLP traces. Loki can accept OTLP logs through a dedicated endpoint. Prometheus-compatible stores such as Mimir, and recent Prometheus versions when enabled, can accept OTLP metrics. So a small system could skip the Collector. It is simple, but you lose a central place to filter, protect, and reroute data, so most production systems keep the Collector.

flowchart LR
    S1["Service A SDK"] -- "OTLP" --> AL["Alloy or Collector"]
    S2["Service B SDK"] -- "OTLP" --> AL
    S3["Service C SDK"] -- "OTLP" --> AL
    AL -- "batch and add labels" --> AL
    AL -- "traces" --> T["Tempo"]
    AL -- "logs" --> L["Loki"]
    AL -- "metrics" --> M["Mimir"]

A trace is the story of one request. Grafana, with Tempo behind it, turns that story into a picture you can read in a minute.

What you see

A trace is a group of spans. Each span is one timed step, such as “handle checkout” or “query the database”. Spans have parents and children, which show which step caused which. Grafana draws them as a waterfall: a list of horizontal bars where the left edge shows when a step started and the bar length shows how long it lasted. A long bar is a slow step. A bar with many small children shows a step that makes many calls.

Finding a trace

There are three common ways. You can paste a trace ID, which you might have copied from a log line or an error message. You can search with TraceQL, for example by asking for traces from one service where the duration is over a limit or the status is an error. Or you can follow a link from a log line, a metric point, or a service map. Good teams practice the third way the most, because it starts from a symptom and lands in a cause.

flowchart TB
    R["Span: POST checkout - 820 ms"] --> A["Span: validate cart - 25 ms"]
    R --> B["Span: charge payment - 700 ms"]
    R --> C["Span: send receipt - 60 ms"]
    B --> B1["Span: call bank API - 640 ms"]
    B --> B2["Span: write payment row - 40 ms"]

In the tree, the root span took 820 milliseconds. The biggest child, charging the payment, took 700. Inside it, calling the bank took 640. So the slowness is in the bank call, not in your own database. A waterfall makes this obvious in seconds, while a pile of logs might take an hour.

Service graph and span metrics

Tempo can also be set up to produce extra data from the spans it sees, using a component often called the metrics generator. It creates span metrics, which are request counts, error counts, and duration summaries per service and operation, and a service graph, which is a map of which services call which. These appear in Grafana as charts and a node map, and they let you start from the big picture before diving into one trace.

Metrics are numbers measured again and again. They are the cheapest signal to store and the best one for spotting trends.

The three metric shapes you will chart

Why percentiles matter

An average hides pain. If ninety-nine requests take one tenth of a second and one takes ten seconds, the average looks fine, but one user is miserable. A percentile such as the ninety-fifth tells you the time under which ninety-five percent of requests finish. Histograms make these charts possible, which is why OpenTelemetry histograms are valuable.

A note on names

OpenTelemetry names metrics with dots, such as http.server.request.duration. Prometheus-style stores prefer underscores and often add unit suffixes. When OTLP metrics enter such a store, names and some labels are usually translated. So the name you wrote in code may look slightly different in Grafana. Settings control this behavior, and defaults have changed across versions, so check what your own system does before building dashboards.

Logs are the detailed diary of your system. Loki and Grafana make that diary searchable without making it expensive.

How Loki thinks

Many log systems index every word, which is powerful but costly. Loki takes a lighter path. It indexes only a small set of labels, such as service name, environment, and namespace, and stores the log text in compressed chunks. When you search, you first pick labels to narrow down the streams, and then filter the text inside them. This keeps storage cheap, but it means that good labels are essential.

Labels versus the log body

A label should have few distinct values, like a service name or a level. A value that is unique for every request, such as a user ID or a trace ID, must not become a label, because it creates a huge number of tiny streams and hurts performance. Such values belong inside the log line or in what Loki calls structured metadata, which is stored beside the line without being indexed as a label.

What OpenTelemetry adds

OpenTelemetry log records carry fields such as severity, body, resource attributes, and, when logged inside a span, the trace ID and span ID. When logs arrive in Loki through OTLP, a chosen set of resource attributes become labels and the rest become structured metadata. The trace ID that rides along is what makes the jump from a log line to a trace possible, which is the topic of the next chapter.

The greatest power of Grafana with OpenTelemetry is not any single chart. It is the ability to jump from one signal to another about the same event.

Three links worth setting up

flowchart LR
    M["Metric chart shows slow spike"] -- "exemplar link" --> T["Trace waterfall"]
    T -- "trace to logs" --> L["Logs of the slow span"]
    L -- "derived field on trace ID" --> T
    T -- "trace to metrics" --> M

What makes it work

All three links depend on shared identifiers and shared names. The trace ID must be present in the log record. The service name must be spelled the same way in traces, logs, and metrics, so that Grafana can narrow a log search to the right service. And the time must be roughly the same across systems. If one service names itself “checkout” in traces and “checkout-svc” in logs, the link finds nothing.

Anyone can make a dashboard. Making one that helps at three in the morning is a skill.

Start with a question

Every dashboard should answer one clear question, such as “Is the checkout service healthy?” or “Where is the money spent?” A dashboard with no question becomes a wall of charts that nobody trusts. Write the question at the top, in the title or a text panel.

Order from general to specific

Put the most important health indicators at the top: request rate, error rate, and slow requests. Below them put the supporting detail, such as resource use and dependency health. Place links at the bottom or side that lead to deeper dashboards and to trace and log exploration.

Use variables and consistency

Variables let one dashboard serve every service and environment. Instead of copying a dashboard thirty times, you choose a service from a drop-down. Keep colors, units, and time ranges consistent, so that a person can read any dashboard in your company the same way.

Keep dashboards as managed files

Dashboards are stored as structured text, which means they can be kept in version control, reviewed, and deployed automatically. This avoids the situation where someone edits a production dashboard by hand and nobody knows why it changed. Many teams provision dashboards from files so that the source of truth is clear.

Dashboards need a person to look at them. Alerts look for you, which is why they are the part that protects your sleep.

The pieces of Grafana alerting

Alert states

An alert instance moves through states. It is Normal when the condition is not met. It becomes Pending when the condition is met but has not lasted long enough, and Alerting when it has lasted for the required time. Special states exist for when data is missing or when the query itself failed. The waiting period prevents a single brief spike from paging someone.

stateDiagram-v2
    [*] --> Normal
    Normal --> Pending: "condition met"
    Pending --> Alerting: "held for the waiting time"
    Pending --> Normal: "condition cleared"
    Alerting --> Normal: "condition cleared"
    Alerting --> NoData: "query returns nothing"
    NoData --> Normal: "data returns"

What to alert on

Alert on symptoms that users feel, such as high error rates or slow responses, rather than on every internal number. A high processor use may be fine, while a slow checkout is never fine. Each alert should point to a dashboard or a runbook, which is a written guide for what to do.

You can use Grafana without running a server, or you can run everything yourself. Each choice trades effort against control.

An AWS example

On AWS, a common pairing is Amazon Managed Grafana for the screens and Amazon Managed Service for Prometheus for metrics. The Collector sends metrics to the managed Prometheus service using a remote write exporter with AWS request signing, which proves the sender’s identity using its cloud role. Grafana can also read directly from CloudWatch and X-Ray as data sources, which is useful when some of your data already lives there. Features and names evolve, so check the current AWS documentation when you design this.

flowchart LR
    APP["Apps on containers"] -- "OTLP" --> COL["Collector"]
    COL -- "remote write with signing" --> AMP["Managed Prometheus"]
    COL -- "traces" --> XR["X-Ray or Tempo"]
    AMP --> AMG["Managed Grafana"]
    XR --> AMG
    CW["CloudWatch"] --> AMG

Here is a made-up story to show every piece working together. The company, names, and numbers are invented for teaching.

In under ten minutes, the team moved from a vague symptom to a cause, using one tool and four connected views. Without correlation, they might have searched logs by hand across four services.

Small naming choices made in your SDK decide whether Grafana feels magical or messy.

What resource attributes are

A resource in OpenTelemetry describes the thing producing telemetry: the service name, version, environment, host, and so on. These descriptions, called resource attributes, are attached to every trace, metric, and log from that service. They are the common thread across signals.

How Grafana backends use them

The backends turn some resource attributes into labels. In Prometheus-style stores, a special info-style metric can hold the full set while a few key ones become labels, depending on settings. In Loki, a configured list of attributes becomes index labels. In Tempo, attributes are searchable by TraceQL. Because the same service name is used everywhere, selecting “checkout” in a drop-down variable filters every panel the same way.

Storing every trace of every request gets expensive quickly. Sampling is the art of keeping what you need.

Two moments to decide

Head sampling decides when a trace begins, for example by keeping one in every ten. It is cheap and simple, but it cannot know yet whether the request will turn out to be slow or broken. Tail sampling waits until the trace is complete and then decides, so it can keep every error and every slow trace and only a small share of normal ones. It needs a Collector that holds traces in memory for a while, which costs more and is harder to scale.

Metrics keep the full picture

A useful trick is to generate span metrics before sampling happens, so that rate, error, and duration charts count every request even though only some traces are kept. The charts stay accurate, and the stored traces are the interesting examples.

Telemetry often contains sensitive details, and Grafana is the window to all of it. Guard the window.

Sign-in and permissions

Grafana supports local accounts and sign-in through company identity systems. Prefer your company’s single sign-on so that people who leave lose access automatically. Grafana uses roles, such as viewer, editor, and administrator, and folders with permissions, so that a team sees only its own dashboards. Enterprise and cloud offerings add finer controls, such as team-based data source permissions, so check which features your edition has.

Protecting the data path

• Encrypt in transit. Use TLS between SDKs, Collectors, backends, and Grafana.
• Authenticate senders. Backends should accept data only from known senders, using tokens or signed requests.
• Separate tenants. Tempo, Loki, and Mimir support multiple tenants, so teams or environments can be isolated.
• Protect credentials. Data source passwords and API tokens belong in a secrets manager, not in dashboard files.

Keeping secrets out of telemetry

The best protection is not to collect sensitive data in the first place. Do not put passwords, card numbers, or personal details in span attributes, log lines, or labels. The Collector can remove or mask known sensitive fields before they reach storage, which gives you a safety net.

Observability can quietly become one of the largest bills in a company. Knowing what drives cost lets you steer it.

What drives cost

Practical controls

• Remove labels you never filter by, especially anything unique per request.
• Drop noisy, low-value logs in the Collector before they are stored.
• Choose retention by signal: keep metrics longer than detailed traces and logs.
• Use recording rules or pre-aggregation so common heavy queries are computed once.
• Review the most expensive dashboards and queries every quarter.

Your observability stack is itself a system that can fail. If it fails quietly, you will be blind exactly when you need sight.

What to watch

A heartbeat for alerting

A useful trick is an alert that is always firing on purpose, called a heartbeat or watchdog. It is sent to an outside service. If that service stops receiving the heartbeat, you know that your alerting pipeline has broken. Otherwise, a broken pipeline looks identical to a calm night.

Where the monitors should live

If your monitoring runs only inside the system it watches, a major outage can take both down together. For important production, keep a small independent path, such as a separate small stack or an external check, that can tell you the main one is gone.

Grafana is popular for good reasons, but it is not free of costs. A fair look at both sides will help you decide.

Key trade-offs

Teams keep rediscovering the same good habits and the same bad ones. Here are the ones that matter most.

Good patterns

Anti-patterns

If you remember only a few pages of this guide, make them these.

Best practices

• Use OpenTelemetry for instrumentation and keep your code free of vendor-specific calls.
• Agree on resource attribute names, especially service name and environment, before building dashboards.
• Send all signals through a Collector so you can filter, protect, and reroute them in one place.
• Set up correlation links early, since they reward you on your first incident.
• Start every service dashboard with rate, errors, and duration, then add detail.
• Manage dashboards and alert rules as files in version control.
• Watch the health of the observability stack itself, with a heartbeat alert.
• Review costs and slow queries on a regular schedule.

Common mistakes

The most common support question is “I send data, but Grafana shows nothing.” Follow the path from left to right and you will find the break.

flowchart TB
    Q["Panel shows no data"] --> A{"Is the time range right?"}
    A -- "no" --> A1["Widen or fix the range"]
    A -- "yes" --> B{"Does the data source test pass?"}
    B -- "no" --> B1["Check address and login and tenant"]
    B -- "yes" --> C{"Does the same query work in Explore?"}
    C -- "no" --> C1["Fix the query or label names"]
    C -- "yes" --> D{"Is the backend receiving data?"}
    D -- "no" --> E{"Is the Collector sending?"}
    E -- "no" --> E1["Check exporter settings and errors"]
    E -- "yes" --> E2["Check backend limits and auth"]
    D -- "yes" --> F["Check dashboard variables and filters"]

Explaining the checklist

Begin with the simplest causes, because they are the most common. A time range that is too narrow or in the wrong place hides data. A data source with a wrong address, a missing tenant header, or an expired token fails the connection test. If the connection works, copy the query into the exploration screen and run it there. If it fails, the problem is the query, perhaps because a label was renamed during translation, such as a dot becoming an underscore. If the query works in Explore, then the dashboard variables or filters are likely too narrow.

If the backend truly has no data, move left. Ask whether the Collector is sending it, and look at the Collector’s own logs and metrics for refused or failed sends. If it is sending but the backend still has nothing, look at ingestion limits, authentication, and rate rules at the backend.

The following stories are invented to show typical situations. They are not reports about real companies.

Traces, metrics, and logs are the three classic signals. Grafana also helps you look at two extra kinds of evidence that fill common blind spots.

Continuous profiles

A profile shows which functions in your program use the most processor time or memory. A trace tells you that a step was slow. A profile tells you which lines of code inside that step were busy. Pyroscope stores profiles over time, and Grafana draws them as a flame graph, a stack of wide and narrow blocks where wide blocks are the hungry functions. When profiles and traces share labels such as service name, Grafana can offer a link from a slow span to the profile of the same time window. OpenTelemetry has been working on its own profiling signal, and its status may have changed since this was written, so check the current state before depending on it.

Browser and mobile data

Back-end telemetry only describes what happens on your servers. Customers feel the full path, including the network and their own device. A browser library such as Faro, or the OpenTelemetry browser tools, records page load times, errors, and user actions, and can pass the trace context to back-end calls. In Grafana, one trace can then begin in the browser and continue through your services, so you can tell whether slowness came from the page or from the server.

You do not need to build everything at once. A staged path lowers risk, and each stage gives value on its own.

Signs you are ready for the next stage

Move forward when the earlier stage is boring. If people still argue about service names, do not add sampling yet. If dashboards are not trusted, do not add more alerts yet. A stable base makes each new layer easier, and an unstable base makes every layer worse.

Grafana is the screen, and OpenTelemetry is the camera and the cable. OpenTelemetry creates and carries traces, metrics, and logs in a common format. Backends such as Tempo, Loki, and Mimir or Prometheus keep that data. Grafana asks the backends questions and turns the answers into dashboards, exploration views, and alerts. Understanding this split keeps you from mixing up responsibilities, and it gives you freedom to change one part without rewriting another.