The Profiling Signal in OpenTelemetry: Continuous Profiling Explained
Learn what the newest OpenTelemetry signal is, how it shows exactly which lines of code burn CPU and memory in production, how it connects to traces, metrics and logs, and what is mature, what is still alpha and how to plan for it.
Picture a car dashboard. The speedometer tells you how fast you are going, the trip log tells you where you have been, and the warning lights tell you when something is wrong. All of that is useful, but when the engine starts to feel sluggish, none of it tells you which part of the engine is struggling. For that, a mechanic holds a stethoscope against the engine and listens to each moving part in turn. Now imagine the stethoscope never leaves the engine: it listens quietly all day, every day, and keeps a recording you can replay after any problem. That is the idea behind continuous profiling.
Software teams already have three well-known signals that describe running systems: metrics, traces and logs. OpenTelemetry is adding a fourth, the profiling signal, which tells you which functions and lines of code consume the processor, the memory and the waiting time. This tutorial explains what profiling is, what makes it continuous, how OpenTelemetry represents and moves profiles, how they connect to traces, and where the project honestly stands today. It is written for readers who have never profiled anything, yet it goes deep enough for architects planning a platform.
1Why the Three Signals Are Not Enough
Metrics, traces and logs tell you that something is wrong and roughly where. Profiles tell you why the code is slow.
A familiar story
An online store notices that its checkout service has become slower and more expensive to run. Metrics show that processor use has doubled over two weeks. Traces show that checkout requests now spend more time inside the service itself, not waiting on other services. Logs show nothing unusual. Everyone agrees the service is the problem, but nobody can say which part of the service is responsible. Engineers start guessing: maybe it is the new discount feature, maybe a library upgrade, maybe a growing data set.
Guessing is slow and costly. What the team needs is a view inside the service that says, for example, “thirty percent of all processor time is spent converting prices to text inside one particular function”. With that sentence, the fix takes an afternoon. Producing that sentence is the job of profiling.
Imagine a restaurant owner who sees that the kitchen is slower than before. The till shows fewer meals per hour (metrics). Order tickets show which tables waited longest (traces). The complaint book lists unhappy comments (logs). But only by watching what each cook is doing, second by second, does the owner discover that one cook spends half the evening hunting for a missing knife. Profiling is watching what each cook is doing.
The question each signal answers
| Signal | Best question it answers | Where it stops |
|---|---|---|
| Metrics | Is something wrong, and how big is it? | Gives numbers, not causes |
| Traces | Which service or step in a request is slow? | Stops at the boundary of a step; cannot see inside it |
| Logs | What happened, in words? | Only shows what developers chose to write down |
| Profiles | Which functions and lines of code are consuming resources? | Shows resource use, not business meaning |
flowchart LR
M["Metrics: something is wrong"] --> T["Traces: it is in this service"]
T --> P["Profiles: it is in this function"]
L["Logs: here is what was written"] -.-> T
P --> F["Fix the code"]
Many people think profiling is something developers do on their laptop, once, when a program feels slow. That is only one way to profile. The more powerful idea is profiling production systems all the time, because many performance problems only appear under real traffic and real data.
2What Profiling Is: The Basics
Before continuous profiling and OpenTelemetry, a few core ideas need to be crystal clear.
Resources and where they go
A running program uses resources: processor time, memory, time spent waiting for locks or disks, and more. A profile is a record of where a program spent those resources. OpenTelemetry describes a profile as a collection of samples with associated metadata that shows where applications consume resources during execution. Instead of one number for the whole program, you get a breakdown by piece of code.
Stacks: the call chain
Programs are built from functions that call other functions. At any instant, a running thread is inside one function, which was called by another, which was called by another, all the way back to the program’s entry point. That chain is the call stack. It answers not only “which function is running right now” but also “who called it and why”.
A call stack is like a trail of sticky notes on a desk. The top note says “currently writing the invoice”, the one underneath says “because the customer clicked buy”, the one under that says “because the web server received a request”. Reading from top to bottom explains the whole chain of reasons.
Sampling: the clever shortcut
Recording every single function call in a busy program would slow it to a crawl and produce mountains of data. Sampling is the shortcut used by almost all production profilers. Many times per second, the profiler briefly looks at what each thread is doing and records its call stack. It does not record everything, only a steady series of snapshots.
The logic is statistical. If a function appears in thirty percent of the snapshots, then the program was probably inside that function about thirty percent of the time. With enough snapshots, the picture becomes accurate even though no single snapshot is important.
Imagine walking through a large office once every few minutes and noting what each person is doing. After a hundred rounds, you would know quite reliably that people spend about a third of their day in meetings, even though you never watched anyone for a full hour. Sampling profilers work the same way.
Flame graphs
The most popular way to look at a profile is a flame graph. Each box is a function. Boxes are stacked so that a function sits on top of the function that called it. The width of a box shows the share of samples in which that function, or something it called, was present. Wide boxes are where the time goes, and a wide box near the top is a prime suspect. The name comes from the way large graphs resemble flames.
Width means share
A wider box means that function accounted for more of the measured resource.
Height means depth
Tall stacks show deep chains of calls; height alone is not a problem.
Order is not time
Left to right is usually alphabetical or merged, not chronological, so do not read it as a timeline.
Look for plateaus
A wide, flat top is code doing real work itself, which is often the best place to optimize.
Different kinds of profiles
| Profile type | What it measures | Question it helps answer |
|---|---|---|
| CPU (on-CPU) | Time code actually spends running on the processor | Which code burns the most computing power? |
| Wall-clock or off-CPU | Time spent waiting, such as for locks, disks or the network | Why is the program idle when it should be busy? |
| Memory allocation | Where memory is requested | Which code creates the most garbage or pressure? |
| Heap or live memory | What memory is still held | Why is memory growing, and what is holding it? |
| Lock contention | Where threads wait for each other | Which shared resource is limiting concurrency? |
The documentation uses on-CPU profiles as its main example, and the initial focus of the project’s tooling has been on processor profiles. Other profile types fit the same data model in principle, but check the current documentation and your tools to see which are actually supported.
3What Makes Profiling Continuous
Profiling once is a snapshot. Profiling always is a time machine.
The old way: profile after the fire
Traditionally, teams attached a profiler when a problem was already happening, for example in a test environment or briefly on one production machine. This has serious weaknesses. The slow behavior may be intermittent and gone by the time the profiler attaches. The test environment may not have the same traffic or data. Attaching a heavy profiler to production may itself slow the system down, which is risky precisely when it is already struggling.
The continuous way
Continuous profiling means sampling production workloads all the time, at a low enough rate that the cost is small, and storing the results with timestamps. Because the data is always there, you can ask questions about the past: what did the service look like at three in the morning last Tuesday, and how does it differ from today?
A photograph taken after a car accident shows the wreck. A dashboard camera that records every journey shows how the accident happened. Continuous profiling is the dashboard camera for your code’s resource use.
Three properties of continuous profiling
Always on
Profiles are collected from running production systems without needing someone to start a session, so evidence exists before anyone knows it is needed.
Low overhead
Sampling rates are kept modest, and collection methods are chosen to minimize interference, so that running the profiler does not distort what it measures. Exact overhead depends on the tool and workload, so measure it in your own environment.
Stored over time
Profiles are kept with timestamps and labels, so they can be compared across time, across versions and across machines.
What time unlocks
Regression detection
Compare the profile before and after a release. If a function grew wider in the flame graph, you can see exactly which change cost more resources.
Intermittent problems
Rare slow periods that vanish before a human can react are captured automatically and can be examined afterward.
Capacity and cost insight
Aggregating profiles across thousands of machines reveals which functions cost the most overall, guiding where optimization saves real money.
Continuous profiling adds some processing overhead, adds network traffic and adds storage. The design question is never “can we afford to profile?” but “at what rate, for which workloads and for how long do we keep the data?” Later chapters return to this trade-off.
4Where Profiles Fit Among OpenTelemetry Signals
OpenTelemetry’s aim is to make all signals share context, so that an engineer can move smoothly between them.
The fourth signal
OpenTelemetry already standardizes traces, metrics and logs. Profiles are being added as the fourth signal. The project’s own documentation describes them as complementing the others: traces show how requests flow across services, while profiles show precisely where inside the code resources are consumed.
Traces
The journey of one request across services, as a tree of timed steps.
Metrics
Numbers sampled over time that summarize health and performance.
Logs
Timestamped records of events, text or structured.
Profiles
Samples of call stacks with measured values, showing where code spends resources.
Shared context is the real prize
The strength of OpenTelemetry is not any one signal but the shared vocabulary between them. All signals can carry the same resource attributes, which describe where the data came from, such as the service name, the Kubernetes pod and the cloud region. Profiles join that family. A profile from a given pod can therefore be lined up with that pod’s metrics, logs and traces without any special glue.
Think of four different reports about the same patient, from the heart monitor, the blood test, the nurse’s notes and the X-ray. If every report carries the same patient number and timestamp, the doctor can read them together. Shared resource attributes are the patient number.
A different shape of data
Profiles are not like the other signals in how they are produced. Traces and metrics are normally created by libraries inside the application, using instrumentation the application’s own code or an agent adds. Profiles, in the main design OpenTelemetry has been developing, are produced by an external agent that observes programs from outside, often from the operating system kernel, rather than by an SDK inside the program. This difference shapes almost everything that follows: how you deploy it, what permissions it needs and which languages it can see.
flowchart TB
subgraph Inside["Produced inside the application"]
TR["Traces"]
ME["Metrics"]
LO["Logs"]
end
subgraph Outside["Produced by an external agent"]
PR["Profiles"]
end
Inside --> OTLP["OTLP"]
Outside --> OTLP
OTLP --> COL["Collector"]
COL --> BE["Backends"]
5Maturity: An Honest Status Check
Before designing anything, know exactly how mature the pieces are, because this signal is young.
Where the project stands
The OpenTelemetry project announced the profiles signal as entering public alpha in March 2026. In the announcement, the project says the signal is not yet ready for critical production workloads and encourages teams to test it while understanding the limits of alpha software. Alpha means that the specification and implementations can still change, sometimes in ways that break compatibility. You should read the official documentation for the current status before committing to a design, since this is a fast-moving area.
What exists at alpha
A wire format in OTLP
A defined profiles data model with deduplicated stacks and dictionary tables for compact encoding, supporting aggregated data and timestamped events.
Trace correlation fields
Samples can carry trace and span identifiers so a profile can point at the request it was working on.
A pprof translator
Conversion to and from the widely used pprof format, designed to avoid information loss.
Collector building blocks
A pprof receiver, Kubernetes metadata enrichment and support for the transformation language, behind an explicit feature gate.
An eBPF profiler
A whole-system profiler for Linux that can run as a Collector receiver and covers several major language runtimes.
Validation tooling
A conformance checker to confirm that data follows the specification.
What is still missing or unsettled
Working Today
- Collecting profiles with an external eBPF-based agent on Linux
- Receiving pprof files into a Collector profiles pipeline
- Enriching profiles with Kubernetes metadata
- Forwarding profiles over OTLP to compatible backends
Not Yet Settled
- Production-ready, widely adopted backends for storing and querying profiles
- A standard approach to symbolization, which turns addresses into function names
- Sharing process and thread context between agents and applications
- Language SDK interfaces for producing profiles from inside applications
Treat everything in this tutorial as a guide to the direction and the ideas, and verify specifics against the current project documentation. Pilot profiling in non-critical environments first, expect breaking changes between releases and avoid building business-critical dependencies on alpha interfaces.
Even at alpha, the concepts, the data model and the deployment shapes are stable enough to learn, and teams that understand them early can adopt the mature version faster and influence what it becomes. The older, separate profiling ecosystem is also already usable today, as later chapters explain.
6How Profilers Collect Data
There are several ways to observe a running program, and each choice affects overhead, coverage and effort.
Approach 1: Language-runtime profilers
Many languages ship a profiler as part of their runtime or ecosystem. A Go program can expose profile endpoints, a Java program can use flight-recording facilities of the Java runtime, and Python and other languages have profiling libraries. These profilers understand their runtime deeply and can offer rich detail such as memory allocation profiles. The downside is that each language needs its own tool, its own integration and sometimes code or startup changes.
Approach 2: Instrumenting profilers
These profilers insert measurement code around functions to record every call. They give exact counts but add significant overhead, which makes them unsuitable for always-on production use. They remain handy for focused experiments on a developer’s machine.
Approach 3: Sampling from outside, using the kernel
The third approach, and the one OpenTelemetry’s reference profiler uses, is to sample from outside the program using facilities of the operating system. On Linux, a technology called eBPF allows small, sandboxed programs to run inside the kernel. A profiler can use it to take stack snapshots of every process on a machine many times per second, without changing those processes and without restarting them.
A language-runtime profiler is like each cook keeping a personal diary of what they did. An eBPF profiler is like a single manager standing in the kitchen doorway with a stopwatch, glancing at every cook on a regular rhythm. The diaries are richer for each cook, but the manager covers the entire kitchen with one person and no cook needs to change habits.
| Approach | Strength | Weakness | Fit for continuous use |
|---|---|---|---|
| Language-runtime profilers | Deep, language-aware detail | One tool per language, may need code or startup changes | Good where the language offers a low-cost mode |
| Instrumenting profilers | Exact call counts | High overhead | Poor |
| Kernel-based sampling with eBPF | Whole machine, no code changes, many languages | Linux only, needs privileges, harder to decode some runtimes | Very good |
Because OpenTelemetry can translate to and from the widely used pprof format, data from language-runtime profilers can enter the same pipeline as data from the eBPF profiler. A platform does not have to choose only one source.
7The OpenTelemetry eBPF Profiler
This is the reference collection agent for the profiling signal, so it is worth understanding how it works and what it needs.
What it is
The OpenTelemetry eBPF profiler is a whole-system profiler. It samples every process on a host from within the kernel, across many languages at once. It can run as a standalone agent or be embedded into a Collector as a receiver, so that profiles flow directly into a Collector pipeline. It grew out of technology originally developed and contributed by Elastic, and several vendors and open-source projects now participate in it.
How it works, in plain steps
A timer fires
The kernel triggers a sampling event at a regular rate on each processor core.
A small kernel program runs
The profiler’s eBPF program wakes up, looks at which process and thread are currently running, and begins to read that thread’s stack.
The stack is walked
The program follows the chain of calls back to the entry point, using knowledge of how the language runtime lays out its stack. This step is called unwinding, and it is one of the hard parts.
Raw data is handed to user space
The compact record, which contains addresses rather than names, is passed to the profiler’s regular program running outside the kernel.
Aggregation and decoration
The agent aggregates identical stacks, attaches process and container information, and prepares data for export, often resolving what it can into function names.
Export through the pipeline
When run as a Collector receiver, the data enters a profiles pipeline and is forwarded over OTLP to a destination.
flowchart LR
subgraph Kernel["Linux kernel"]
TM["Sampling timer"] --> EB["eBPF program reads stacks"]
end
subgraph Host["User space on the same host"]
AG["Profiler agent or Collector receiver"]
AG --> AGG["Aggregate and decorate"]
end
EB -->|"Raw stack records"| AG
APP1["Process A"] -.->|"Observed"| EB
APP2["Process B"] -.->|"Observed"| EB
AGG -->|"OTLP profiles"| NEXT["Collector pipeline"]
What it supports
The profiler targets Linux and understands both natively compiled code and many managed runtimes. Documentation from the project and from vendors lists languages such as C, C++, Rust, Go, Java on the HotSpot virtual machine, Python, Ruby, PHP, Perl, Node.js, .NET and Erlang-family runtimes. The exact versions covered, and the processor architectures supported, change between releases, so confirm the list for the version you plan to run.
What it needs
Linux
The profiler depends on kernel features, so it does not run on other operating systems.
A modern kernel
Minimum kernel versions depend on the architecture and the release. Check the current documentation rather than assuming.
Elevated privileges
Loading eBPF programs normally needs root or specific capabilities, which has security consequences discussed later.
Per-node deployment
On Kubernetes it is typically run as one instance on every node, using a daemon-style deployment.
Some managed container platforms restrict kernel-level access or privileged containers. If you run on such a platform, confirm that your environment allows the profiler before planning a rollout, since the whole approach depends on that access.
8The Hard Parts: Unwinding and Symbolization
Two technical problems explain why profiling is trickier than it first appears, and why standardization takes time.
Unwinding: reading the call chain
Unwinding is the act of reconstructing the chain of function calls from raw memory. For a program compiled with the right settings, this is straightforward: each function leaves a clear pointer to its caller. But many programs are built in ways that make the trail faint, and managed runtimes keep their own call information in their own formats. Reading the right information for each language is a large piece of the profiler’s engineering.
Imagine tracing a parcel’s route using the labels left at each depot. In some depots, workers follow a common standard and the trail is easy. In others, each uses a different style of label, and some leave no label at all. An unwinder is the detective who must understand every depot’s labelling habit.
Symbolization: turning addresses into names
The kernel and the profiler’s low-level code deal in memory addresses, which are long numbers such as locations in a program. Humans need function names and ideally file names and line numbers. The conversion from address to name is called symbolization. It needs symbol information that maps address ranges to names, typically shipped inside the program’s files or in separate debug files.
Symbolization is difficult in practice for several reasons. Many production programs are stripped of their names to save space. Different builds of the same program have different address layouts, so the right symbols must be matched to the exact build. Just-in-time compiled runtimes create code on the fly, so names must be captured while the program runs. And symbols can be large, so storing and distributing them is a cost.
On the host
The agent resolves names where the program runs. Fast and simple, but it needs symbol access on every machine, and it costs some processing there.
In a central service
The agent sends addresses with build identifiers, and a symbol service resolves them later. This keeps agents light but needs a place to store debug files for every build.
At build time
The continuous integration system uploads symbol files whenever a build is produced, so the central service always has what it needs.
Runtime-provided names
Managed runtimes expose their own function names, which the agent reads directly, avoiding the need for separate files.
Where OpenTelemetry stands
The project’s alpha announcement lists symbolization standardization as unfinished work: an interface, a storage format and a reference implementation are planned. The eBPF profiler already performs some symbolization on the host, and the announcement mentions automatic on-target symbolization for Go programs, but a full, interoperable story across languages and backends is still being designed.
Whatever tools you choose, decide early how symbols will reach your analysis system. Make uploading symbols part of your build pipeline, keep them for as long as you keep profiles, and test that a freshly released build produces readable flame graphs before you need them in an incident.
9The Profiles Data Model
The data model is how OpenTelemetry writes profiles down, and its design choices explain how efficient continuous profiling can be.
The core ingredients
An OpenTelemetry profile is built from a small set of ideas. They are best understood through the question each one answers.
Sample
One observation: a call stack together with a measured value, such as the amount of processor time attributed to it.
Stack
An ordered list of locations, from the function that was running up to the program’s entry point.
Location and function
A place in code, and the function it belongs to, with names and, where available, file and line details.
Mapping
The binary or library where the code lives, which helps locate symbols for it.
Attributes
Key and value labels that describe a sample or a profile, such as thread name or process details.
Link
A reference to a trace and span, so a sample can say which request it was working on.
Deduplication with dictionaries
Profiles contain an enormous amount of repetition. The same function appears in thousands of stacks, and the same stack appears in thousands of samples. Writing out every function name every time would be wasteful. The OpenTelemetry format therefore stores each distinct item, such as a function, a location, a stack or a string, once in a dictionary table and refers to it by a small number everywhere else.
Think of a book that has a glossary. Rather than repeating the full definition of a technical term each time it appears, the text simply says “see entry 17”. The reader gets the same information, and the book is far shorter. Dictionary tables are the glossary for profiles.
flowchart LR
S["Sample with value and timestamps"] -->|"refers to"| ST["Stack table"]
ST -->|"lists"| LO["Location table"]
LO -->|"points to"| FU["Function table"]
LO -->|"points to"| MA["Mapping table"]
FU -->|"names in"| STR["String table"]
S -->|"labels from"| AT["Attribute table"]
S -->|"may reference"| LK["Link table with trace and span ids"]
Aggregated data and timestamped events
The data model supports two styles. Aggregated profiles summarize many observations into totals per stack over a period, which is compact and perfect for flame graphs. Timestamped data keeps individual events with their own times, which preserves detail such as the exact moment of a sample and enables linking with specific traces. Choosing between them is a trade-off between storage cost and precision.
Compatibility with pprof
A widely used profile format called pprof already exists, especially in the Go community, and many tools can read it. OpenTelemetry provides a translator that converts to and from its own format, designed so that no information is lost in the round trip. This lets existing tools and data sources join the new ecosystem and gives teams a gentle migration path.
| Question | pprof | OpenTelemetry profiles |
|---|---|---|
| Purpose | A file format for profile data used by many tools | A signal in a wider telemetry standard, carried over OTLP |
| Context sharing | Limited labels | Shares resource attributes and trace links with other signals |
| Transport | Files or tool-specific endpoints | OTLP, with Collector pipelines |
| Conversion | Source of data for the translator | Round-trip conversion designed to be lossless |
During alpha the exact table layout has changed and may change again. Do not store raw wire-format data for the long term, and prefer conversion tooling provided by the project when moving data between versions.
10Correlating Profiles with Traces
Correlation turns profiles from a separate tool into part of an investigation flow.
The idea
A trace says “this request spent 800 milliseconds inside the checkout service”. A profile says “the checkout service spends much of its time in this function”. Correlation joins the two: if a sample can record the trace and span it was running on behalf of, then you can open a slow span and ask what code the service was executing during that span.
Imagine a security camera in a warehouse that stamps each recording with the number of the delivery truck currently being unloaded. If a delivery was slow, you can pull out only the footage from that truck’s unloading time and see precisely what the workers were doing.
How it can work
The application starts a span
Instrumentation creates a span in the usual way and the current trace and span identifiers become part of the thread’s context.
The identifiers are made visible to the profiler
Because the profiler observes from outside, the application or its agent must expose the current identifiers in a place the profiler can read, such as a designated memory area tied to the thread.
The profiler samples and reads the context
When it takes a stack sample, the profiler also reads the identifiers for that thread and attaches them to the sample as a link.
The backend joins the signals
A backend that understands both can show the profile restricted to a single span, or jump from a hot function to example traces that were running it.
sequenceDiagram
participant App as Application thread
participant Ctx as Shared thread context
participant Prof as Profiler agent
participant BE as Backend
App->>Ctx: Publish current trace and span ids
Prof->>App: Take stack sample
Prof->>Ctx: Read trace and span ids
Prof->>BE: Send sample with stack and link
BE->>BE: Join with trace data by span id
BE-->>App: Engineer sees hot code for this span
The open design work
Agreeing on exactly how applications publish this thread context, in a way that works across languages and agents and is cheap enough to do on every request, is one of the pieces the project lists as still under development. Until it is settled, trace correlation may need vendor-specific setup or may be unavailable for some languages. Check your chosen profiler’s documentation for the current state.
Without correlation
You can see which functions are hot across the whole service, but you cannot tell which of them mattered to the slow request a customer complained about.
With correlation
You open the slow trace, select the slow span and view a flame graph made only from samples taken during that span. The answer is the code that ran, not the code that usually runs.
11The Collector Profiles Pipeline
The Collector treats profiles like its other signals: receivers, processors and exporters connected into a pipeline.
A pipeline for a new signal
Readers who know the OpenTelemetry Collector will recognize the model. Each pipeline is declared for one signal type, and profiles now have their own pipeline type next to traces, metrics and logs. Data enters through receivers, passes through processors in order and leaves through exporters. The same ideas of memory protection, batching and queues apply, though the available components are fewer than for the older signals.
Components available at alpha
| Component type | Example | What it does for profiles |
|---|---|---|
| Receiver | eBPF profiler receiver | Samples all processes on a Linux host from the kernel and emits profiles |
| Receiver | pprof receiver | Ingests profiles in the pprof format, for example from language-runtime profilers |
| Receiver | OTLP receiver | Accepts profiles sent by other agents or Collectors over OTLP |
| Processor | Kubernetes attributes processor | Adds namespace, pod and workload information to profiles |
| Processor | Transformation support | Lets operators change or filter profile data using the Collector’s transformation language |
| Exporter | OTLP exporter | Sends profiles on to another Collector or a compatible backend |
flowchart LR
R1["eBPF profiler receiver"] --> P1["Memory limiter"]
R2["pprof receiver"] --> P1
R3["OTLP receiver"] --> P1
P1 --> P2["Kubernetes attributes"]
P2 --> P3["Transform and filter"]
P3 --> P4["Batch"]
P4 --> X1["OTLP exporter to gateway"]
A feature gate protects early adopters
Because the signal is in alpha, Collector support is switched off by default behind an explicit feature gate that must be enabled when the Collector starts. Feature gates are the project’s way of letting adventurous users try new behavior while protecting everyone else from changes that may still break. Expect the gate name and behavior to evolve, and read release notes when upgrading.
A feature gate is like the “staff only” rope in front of a new exhibit still being installed. Curious visitors who ask can go in and look, but the museum does not promise the exhibit is finished or safe for everyone.
What processors can do for profiles
Enrichment
Adding Kubernetes metadata so that every profile can be grouped by namespace, workload and pod, in exactly the same way as traces and metrics.
Filtering
Dropping profiles from workloads that should not be profiled, such as noisy system processes or sensitive namespaces, before they consume storage.
Attribute hygiene
Removing or renaming labels that would create too many distinct series, which keeps storage and queries manageable.
If your organization already runs Collectors as an observability pipeline platform, profiles can reuse the same operating model: node-level agents, gateway tiers, configuration as code and self-monitoring. You are adding a signal to a platform, not building a second one.
12Deployment Topologies on Kubernetes
Profiling agents are tied to nodes, which makes the deployment shape different from application-level telemetry.
One agent per node
Because the eBPF profiler watches everything on a machine, the natural unit of deployment is the node. On Kubernetes, that means a daemon-style deployment that places exactly one profiler instance on every node. One instance covers all the pods on that node, without touching any of them, and new pods are covered automatically as they appear.
Think of a lifeguard at each swimming pool. You do not give each swimmer a personal lifeguard. One watcher per pool covers everyone who jumps in, and arrivals are covered the moment they enter the water.
A typical flow
flowchart LR
subgraph NodeA["Node A"]
PA1["Pod 1"]
PA2["Pod 2"]
AGA["Profiler agent"]
end
subgraph NodeB["Node B"]
PB1["Pod 3"]
AGB["Profiler agent"]
end
AGA --> GW["Gateway Collector pool"]
AGB --> GW
GW --> BE["Profile backend"]
GW --> AR["Optional cold storage"]
Placement choices
Profiler as a Collector receiver on each node
The node-level Collector already running for logs and host metrics also hosts the profiler receiver. This reuses existing deployment and configuration, at the cost of coupling the profiler’s privileges to that Collector.
A dedicated profiler Collector
A separate node-level deployment runs only the profiler. Privileges are isolated from other telemetry, and failures cannot affect other signals, but there is one more component to run.
Standalone profiler with a downstream Collector
The profiler runs as its own agent and sends over OTLP to a Collector. This suits cases where the profiler is managed by a different team or provided by a vendor.
Separation of duties
Since profiling needs privileges that most workloads should never have, it is wise to isolate it from application-facing Collectors. Application telemetry paths can then remain unprivileged, and only the profiling path carries elevated rights. A gateway tier downstream does not need those rights, because it only receives data over the network.
Short-lived processes may finish between samples, very small functions may rarely be caught, and workloads on nodes that do not run the agent are invisible. Profiling coverage is statistical and node-based, so verify which nodes and workloads are actually represented.
13Backends and Analysis
OpenTelemetry defines how profiles travel. Storing and exploring them is up to backends, and that part is still maturing.
What a profile backend must do
Store efficiently
Hold huge volumes of repeated stacks cheaply, using deduplication and compression.
Query by labels and time
Select profiles by service, pod, version or time range and merge them into one view.
Visualize
Draw flame graphs, tables of top functions and comparisons between two selections.
Symbolize
Resolve addresses into names using stored symbol files where the agent did not.
Link to other signals
Jump between a hot function and the related traces, metrics or logs.
The landscape
Several open-source and commercial systems already store and analyze profiles. Some accept pprof, some are adding or have added support for OpenTelemetry’s profile format, and some offer only their own protocols. The alpha announcement itself describes production-ready backends for the new format as not yet emerged, and one profiling desktop tool is explicitly described as not being a real production backend. Because support changes quickly, check what your chosen backend can ingest today.
Reading a profile in practice
Choose the scope
Pick a service, a version and a time window that match the question, such as the hour when latency rose.
Find the wide plateaus
Look for wide boxes near the top of the flame graph, which show code that spends real resources itself.
Walk down the stack
Read which callers lead to that function. The cause is often a caller invoking it far too often.
Compare with a baseline
Use a difference view against an earlier period or version to confirm what changed, rather than blaming code that was always expensive.
Verify after the fix
After a change is deployed, check that the wide box really shrank, which closes the loop with evidence.
A flame graph on its own can mislead, because the widest box may simply be the code that always does the main work. A comparison between two moments shows what changed, which is usually what you are hunting.
14Use Cases and Workflows
Profiles pay for themselves when they shorten investigations and reduce running costs.
Finding the code behind a processor-use spike
Metrics show a service suddenly using twice the processor. A profile comparison between before and after the spike reveals a single function that grew wide, perhaps because a new feature calls an expensive conversion inside a loop. The fix is targeted and verifiable.
Catching regressions at release time
After each deployment, compare the new version’s profile against the previous one for the same traffic level. A function that grew noticeably becomes a review item before customers feel it.
Reducing cloud spend
Aggregate profiles across a fleet. If a small number of functions account for a large share of total processor use, optimizing them lowers the number of machines needed. Profiles turn “reduce cost” from a vague hope into a ranked list of candidates.
Hunting memory growth
Allocation and live-memory profiles show which code creates or retains memory. This is valuable for leaks and for runtimes where garbage collection pressure causes pauses.
Understanding waiting, not just working
Off-processor profiles reveal threads waiting for locks, disks or network replies, explaining slow requests even when processor use is low.
Explaining a slow span
With trace correlation, an engineer opens a slow span and sees the code that ran during it, closing the gap between “which step” and “which code”.
Evaluating libraries and upgrades
Before and after profiles around a library upgrade or a language-runtime update show whether the change is cheaper or costlier in practice.
Optimizing without a profile is like trimming a household budget by guessing. With a profile, you have the bank statement: you see that most of the money goes to two large payments and can focus your effort there.
flowchart TD
A["Alert or complaint"] --> B["Metrics show which resource"]
B --> C["Traces show which service and span"]
C --> D["Profile shows which function"]
D --> E["Change code"]
E --> F["Deploy"]
F --> G["Compare profiles before and after"]
G -->|"Improved"| H["Close the loop"]
G -->|"Not improved"| D
15Cost, Overhead, Sampling and Retention
Continuous means always, and always means volume, so cost deserves its own plan.
Where the cost comes from
Agent overhead
Sampling and stack reading use a little processor time and memory on every node. The amount depends on the sampling rate and the workload.
Network
Profiles leave each node regularly, adding traffic that grows with the number of nodes and processes.
Storage
Retaining profiles with fine time resolution and many labels is the largest long-term cost.
Symbols
Storing debug files for every build, and processing to resolve them, adds a quiet but real bill.
Levers you control
| Lever | Effect | Risk |
|---|---|---|
| Sampling frequency | Lower rates cut overhead and data volume | Rare or short code paths become harder to see |
| Aggregation window | Longer windows shrink storage | Less precise timing of events |
| Selective scope | Profile only chosen namespaces or workloads | Blind spots outside the scope |
| Label discipline | Fewer distinct label values keep queries fast | Less slicing detail |
| Retention tiers | Keep recent data fine, older data coarse | Old incidents have less detail |
Statistics tell you how much is enough
Because profiling is sampling, more samples give a more accurate picture, but with sharply diminishing returns. For widely used functions, even a modest rate across many machines yields very clear results. For rare code paths, you may need longer time windows or targeted sessions. A useful habit is to ask how many samples a conclusion rests on before trusting it.
An opinion poll of a thousand people is enough to describe a country’s mood, but not enough to describe a tiny village within it. Profiling is the same: great for the busy parts of your code, weaker for the quiet corners.
Published figures for profiler overhead are generally low, but they depend on workload and settings. Run a controlled comparison of latency and processor use with the profiler on and off in your own environment before wide rollout, and watch for unusual cases such as very high thread counts.
16Security, Privacy and Compliance
A profiler that can see every process on a machine is powerful, and anything powerful needs careful boundaries.
The privilege question
To observe other programs from the kernel, the eBPF profiler needs elevated rights: typically running as root or holding specific kernel capabilities, and often running as a privileged container on Kubernetes. That is a much higher level of access than ordinary telemetry agents need. If such an agent were compromised, an attacker would gain a strong foothold on the node. The profiler therefore deserves the same protection as other node-level security-sensitive agents.
A building’s master key opens every door. It is enormously convenient, but the person who holds it must be vetted, the key must be logged and stored safely, and nobody else should be able to copy it. A node-level profiler is the master key for observability.
Reducing the risk
Narrow capabilities
Where the platform allows it, grant only the specific capabilities required rather than full privileged mode.
Isolate the agent
Run it in its own namespace with strict access rules, separate from application-facing Collectors.
Trusted images
Mirror, scan and pin the agent image, and limit who can change its deployment.
Restricted egress
Allow the agent to send data only to the intended Collector address, and encrypt that traffic.
Audit and review
Keep configuration in version control, require review, and alert on unexpected changes to the agent.
What a profile reveals
A profile mainly contains function names, library names, file paths and counts. It does not normally contain the values held in variables or the content of requests, which makes it less directly sensitive than logs or traces with captured payloads. But it is not harmless. Function and library names can reveal internal architecture, proprietary algorithms or the exact software versions in use, which helps an attacker plan. Labels attached to profiles, such as process arguments or environment details, may also contain sensitive values if not filtered.
Process command lines and similar details sometimes carry secrets or personal data. Review which attributes your profiler attaches, and use pipeline filtering to remove anything that should not leave the node.
Compliance and multi-tenancy
In shared clusters, profiles from one team’s workloads may be visible to platform operators or to other teams if access controls are loose. Apply the same tenant boundaries you use for other signals: route by namespace or team, use per-tenant access in the backend and limit retention for regulated workloads. Where data residency applies, keep profiles inside the same boundaries as other telemetry from those workloads.
17Platform Governance and Rollout
Profiling becomes sustainable when a platform team treats it as a managed service with clear ownership.
The platform view
In organizations that run an observability pipeline as a platform, profiling is simply another signal that the platform provides. Application teams do not install profilers; they receive profiling coverage as a property of running on the platform. The platform team owns the agents, the pipeline, the symbol process, the backend choice, the access rules and the cost reporting.
Define the goals
Decide which questions profiling should answer, such as cost reduction, regression detection or incident analysis, so that scope and success measures are clear.
Choose the pilot scope
Select a small set of nodes or a few services with a known performance question, preferably in a non-critical environment, given the alpha status.
Prepare the supporting pieces
Confirm node privileges, kernel compatibility, symbol upload from the build system, a backend that can ingest the data and a Collector pipeline with the feature gate enabled.
Measure overhead and quality
Compare latency and resource use with and without the agent, and check that flame graphs are readable and complete.
Widen gradually
Expand by cluster or team in waves, with a documented opt-out and with volume and cost reports for each tenant.
Operate and improve
Monitor the agents, track coverage, keep up with the fast-moving project and review the value delivered, adjusting rates and retention accordingly.
Ownership split
| Area | Platform team | Application team |
|---|---|---|
| Agents and pipeline | Owns and operates | Consumes automatically |
| Symbol upload | Provides the tooling | Ensures builds include symbols or use the shared pipeline |
| Access and retention | Defines policy by tenant | Requests exceptions with a reason |
| Analysis and fixes | Provides dashboards and guidance | Reads profiles and optimizes own code |
| Cost | Reports usage per tenant | Acts on high-cost findings |
Profiling is easy to dismiss until it solves a real problem. Record each case where a profile found a regression or saved resources, with numbers, and share them. A short list of concrete wins is the best argument for continued investment.
18Advantages, Disadvantages and Trade-offs
An honest view of both sides helps decide when, where and how far to go.
Advantages
- Shows exactly which code consumes resources, closing the gap between a slow service and the cause
- Whole-host coverage across many languages without code changes, using the eBPF approach
- Historical data lets teams compare versions, times and machines
- Shares resource attributes and trace links with the other signals
- Identifies concrete optimization targets that directly reduce infrastructure cost
- Vendor-neutral format with conversion to and from pprof
Disadvantages / Trade-offs
- The signal is in alpha, so formats and components can change and are not for critical production reliance
- The reference profiler is Linux-only and needs elevated privileges
- Symbolization and stack decoding are hard and not yet standardized across tools
- Backend choices that ingest the new format are still limited
- Sampling is statistical, so rare code paths can be missed
- Adds overhead, network traffic and storage that must be budgeted
How profiles compare with the other signals in practice
| Aspect | Traces | Metrics | Profiles |
|---|---|---|---|
| Granularity | One request at a time | Aggregated numbers | Code-level resource use |
| Produced by | Instrumentation in the application | Instrumentation or the platform | Usually an outside agent |
| Maturity in OpenTelemetry | Stable | Stable | Public alpha |
| Typical question | Which step was slow? | Is the system healthy? | Which code is expensive? |
| Main cost driver | Request volume | Series cardinality | Number of nodes, rate and retention |
When profiling may not be the right next step
If your services lack basic metrics, traces and logs, profiling is not the first investment, because those signals tell you where to look. If your environment cannot run privileged node agents, the eBPF route is blocked and you may rely on language-runtime profilers instead. If your performance issues are mostly waiting on other services, traces and dependency analysis will help more than processor profiles.
19Design Patterns and Anti-Patterns
Lessons from early adopters, collected so that you do not have to repeat them.
Patterns that work
One agent per node
Use a daemon-style deployment so that every pod on a node is covered automatically, including new ones.
Privilege isolation
Keep the privileged profiler separate from unprivileged application-facing Collectors.
Symbols from the build
Upload symbol files from the continuous integration pipeline for every release so names are always resolvable.
Shared resource attributes
Enrich profiles with the same Kubernetes attributes as other signals so investigations can pivot freely.
Compare, do not just look
Use difference views between versions or time windows as the default way to investigate regressions.
Pilot behind a gate
Enable the feature gate in a small, non-critical environment first and expand only after measuring overhead.
Anti-patterns to avoid
Problem
A team treats the alpha profiling signal like a stable production feature and builds critical dashboards, alerts and automations directly on it.
Why It’s Harmful
Formats, component names and behavior may change between releases, so an upgrade can silently break the very tooling that people rely on during incidents.
Correct Approach
Use profiling as a supporting, investigative aid for now. Keep critical alerting on stable signals, pin component versions and test upgrades in a staging environment.
Problem
The profiler is deployed on every node at maximum sampling rate with full privileges and no review, because “more data is better”.
Why It’s Harmful
Overhead, network traffic and storage grow without added insight, and the privileged agent widens the attack surface of every node.
Correct Approach
Start with a moderate rate on selected nodes, measure the benefit, narrow privileges where possible and expand only where the data proves useful.
Problem
No plan exists for symbols, so flame graphs are filled with raw addresses or unnamed frames, and nobody can read them.
Why It’s Harmful
Engineers lose trust in profiling after a few useless sessions and stop using it, even though the underlying data was fine.
Correct Approach
Make symbol handling part of the build and release process, retain symbols as long as profiles, and verify readability after each release.
Problem
Profiles carry unfiltered labels such as full command lines, pod names with random suffixes and per-request identifiers.
Why It’s Harmful
Label combinations multiply, storage and query costs balloon, and secrets in command lines may leak into shared systems.
Correct Approach
Keep a small, agreed set of labels, group by workload rather than by individual pod where possible, and filter sensitive attributes in the Collector.
Problem
Optimization work starts from a single flame graph of one machine for a few seconds, and decisions are made on that alone.
Why It’s Harmful
Small samples are noisy, one machine may not represent the fleet, and the widest box may simply be the code that always does the main job.
Correct Approach
Aggregate across machines and time, compare against a baseline, and check that conclusions rest on enough samples before changing code.
20Real-World Scenarios and a Maturity Roadmap
The scenarios below are composites drawn from common patterns, followed by a staged path for adoption.
Scenario 1 — The mystery cost increase
A cloud bill climbs steadily while traffic stays flat. A fleet-wide profile shows that a logging library formatting messages that are never printed accounts for a large share of processor time. Changing a configuration setting removes the waste, and the following month’s bill drops visibly, with a before-and-after profile as proof.
Scenario 2 — The release that quietly got slower
After a routine release, response times creep up by a small amount. A difference view between the old and new versions highlights a serialization function that widened sharply. The cause is a library upgrade that changed default behavior, and the team pins a setting before most customers notice.
Scenario 3 — Slow requests with idle processors
Latency is high, but processor use is low. A profile that includes waiting time shows many threads blocked on a lock around a shared cache. The team splits the cache into independent sections, and the waiting disappears.
Scenario 4 — The slow span with a hidden cause
A trace shows one span taking far longer than expected, but the span’s code looks simple. With trace correlation enabled, the profile restricted to that span reveals time spent in a regular-expression routine that behaves badly for certain inputs. A small change to the pattern removes the stall.
Scenario 5 — A cautious pilot on a locked-down platform
A regulated company cannot run privileged containers broadly. It enables the profiler on one dedicated node pool with strict network rules, profiles only a handful of services, and uses language-runtime profiles converted from pprof for the rest. The platform team revisits the decision as the project matures.
Scenario 6 — Memory that never comes back
A service restarts every few days after growing in memory. Allocation and live-memory profiles point to a cache that never expires entries. After adding a limit, restarts stop, and the profile confirms flat memory over the following weeks.
Roadmap
Stage 1 — Learn and experiment
Run the profiler and a viewer in a test cluster, produce flame graphs for a sample workload and learn to read them.
Stage 2 — Wire into the pipeline
Enable the Collector’s profiles support, add enrichment and filtering, and send data to a backend that can ingest it.
Stage 3 — Pilot on real services
Profile a few non-critical production workloads, measure overhead, fix symbol handling and collect first success stories.
Stage 4 — Harden and govern
Reduce privileges, isolate the agents, define retention and access policy, and write runbooks and cost reports.
Stage 5 — Correlate and scale
Add trace correlation where supported, widen coverage in waves and make profile links part of standard incident workflows.
Stage 6 — Track the standard as it stabilizes
Follow project releases, adopt stable components as they arrive, retire workarounds and fold profiling fully into the platform.
flowchart LR
A["Stage 1 Learn"] --> B["Stage 2 Wire to pipeline"]
B --> C["Stage 3 Pilot"]
C --> D["Stage 4 Harden and govern"]
D --> E["Stage 5 Correlate and scale"]
E --> F["Stage 6 Track stability"]
21Best Practices and Common Mistakes
A short checklist of habits that make profiling useful instead of noisy.
Best Practices
- Begin with a clear question, such as cost, regressions or incident analysis
- Pilot behind the feature gate in non-critical environments
- Isolate and minimize the privileges of the profiling agent
- Upload symbols from the build for every release
- Enrich profiles with the same resource attributes as other signals
- Compare profiles across versions and time rather than reading one in isolation
- Measure overhead before and after, in your own environment
- Follow release notes and pin versions of every component
Common Mistakes
- Relying on alpha interfaces for critical workflows
- Ignoring symbol handling until the first incident
- Giving the agent broad privileges without review
- Using very high sampling rates by default
- Letting label sets grow without limits
- Drawing conclusions from tiny samples
- Assuming all nodes and languages are covered equally
- Treating profiling as a replacement for metrics, traces and logs
Reading results with care
Profiles measure where resources go, not whether that is a problem. A function that dominates the graph may simply be the main job of the service. The useful questions are whether the share changed, whether it is larger than expected for the work done, and whether a cheaper way exists. Combine profile findings with business context before deciding what to optimize.
Keeping up with a young signal
Because the signal is in alpha, details change quickly: the data model’s layout, component names, feature gates, supported languages and backend support. Subscribe to the project’s release notes and announcements, keep a small test environment that tracks the latest releases and treat the official documentation as the source of truth for anything that may have changed since this tutorial was written.
Record which versions, kernels, languages and settings your pilot depends on. When a future release changes something, you will know immediately which assumption broke instead of rediscovering it the hard way.
22Frequently Asked Questions
Quick answers to the questions that come up first.
It is the fourth OpenTelemetry signal, alongside traces, metrics and logs. It represents profiles: samples of call stacks with measured values that show which functions and lines of code consume processor time, memory and waiting time.
Continuous profiling samples production workloads all the time at low overhead and stores the results with timestamps. That lets you investigate past incidents, compare versions and see long-term trends, rather than only catching problems while a profiler happens to be attached.
It entered public alpha in March 2026, and the project says it is not yet ready for critical production workloads. It is suitable for testing, pilots and non-critical use, with the expectation that interfaces may change. Check the official documentation for current status.
The main approach is an external agent, the OpenTelemetry eBPF profiler, which samples every process on a Linux host from the kernel. Profiles can also come from language-runtime profilers in the pprof format, which a Collector receiver can ingest.
Not for basic whole-host profiling with the eBPF profiler, which observes from outside. Linking profiles to specific traces needs the application or its instrumentation to expose context in a way the profiler can read, and that part is still being standardized.
No. The reference profiler targets Linux and needs a suitable kernel and elevated privileges. Some managed container platforms restrict that kind of access. Language and version coverage also varies, so verify support for your runtimes.
pprof is an established profile file format used by many tools. OpenTelemetry provides a translator that converts to and from its own format with the aim of no information loss, so existing profile sources and tools can take part in the new ecosystem.
Symbolization converts raw memory addresses into function names, files and lines. Without it, flame graphs are unreadable. It needs symbol information matched to the exact build, which is why many teams upload symbols from their build pipeline.
It is designed to be low, but the exact figure depends on the tool, the sampling rate and the workload. Measure it in your own environment with the profiler on and off before rolling it out widely.
In a backend that can ingest profiles. Several systems exist, but production-ready support for the new OpenTelemetry format is still emerging, so check what your chosen backend accepts today and whether conversion from OTLP or pprof is needed.
In the official OpenTelemetry documentation for the profiles signal, the project’s announcements and the release notes for the Collector and the eBPF profiler. Details such as the feature gate, supported languages and data model layout can change between releases.
23Summary and Key Takeaways
The profiling signal gives OpenTelemetry the missing zoom lens. Metrics tell you that a resource is being consumed, traces tell you which service and step are slow, and profiles tell you which code is responsible. Continuous profiling keeps that lens on all the time, so the evidence already exists when a problem appears, and so that versions, hours and machines can be compared. Like the stethoscope that never leaves the engine in the opening picture, it listens quietly and lets you replay what it heard.
The signal is young. It is in public alpha, the reference profiler is Linux-only and privileged, symbolization and trace correlation are still being standardized, and backend support is growing. That is a reason to pilot thoughtfully, not a reason to ignore it. Teams that learn the model now, keep their pipeline platform ready and treat profiling as a managed, governed capability will be well placed as the pieces stabilize.
Key Takeaways
- Profiles show where code spends resources — they sample call stacks and turn them into flame graphs that point to the functions responsible for processor, memory and waiting costs.
- Continuous means always on, low overhead and stored over time — that is what enables regression detection, incident forensics and fleet-wide cost insight.
- Profiles are the fourth OpenTelemetry signal — they share resource attributes with the others and can carry trace and span links for correlation.
- The main collector is an external eBPF profiler — it observes whole Linux hosts from the kernel, needs elevated privileges and is usually deployed once per node.
- The Collector gains a profiles pipeline — with receivers such as eBPF and pprof, enrichment, transformation and OTLP export, currently behind a feature gate.
- The hard parts are unwinding, symbolization and correlation — plan symbols as part of your build process and treat trace linking as maturing functionality.
- It is public alpha, so pilot carefully — avoid critical dependencies, isolate privileges, measure overhead, govern cost and follow the project’s documentation for the current state.