Good observability starts with data, and data starts with instrumentation. The hardest part is often getting that instrumentation into place.

What Instrumentation Means

Instrumentation is the act of making a program report on itself. A program that is instrumented creates traces, which follow a single request across many services; metrics, which are numbers measured over time such as requests per second; and logs, which are timestamped records of events. Together these are called telemetry. OpenTelemetry is the open-source project that defines standard ways to create, name and send it.

The usual way to instrument is to add a library to your code and call it at the interesting moments. That works well when you own the code, know the language, and have time. In practice, one or more of those conditions often fails.

Three Common Obstacles

What Teams Want From a Solution

• Fast coverage. See something useful for every service within days, not quarters.
• No code changes. Do not edit, rebuild or redeploy the programs being watched.
• Language independence. One approach that works the same way for any language.
• Standard output. Data in OpenTelemetry’s standard format, so that existing tools and dashboards keep working.
• Low overhead. Little extra load on already busy production machines.

Before looking at eBPF, it helps to see where it sits among the other options.

Option One: Manual Instrumentation With an SDK

An SDK, short for software development kit, is the OpenTelemetry library that lives inside your program. With manual instrumentation, developers call it directly: start a span here, record a metric there, attach the customer tier as an attribute. It gives the richest, most meaningful data, because the people who wrote the code decide exactly what matters. It also needs the most effort and must be repeated for every language.

Option Two: In-Process Auto-Instrumentation

Many languages let an agent attach to a running program and add instrumentation automatically to popular libraries. The Java agent, which hooks into the Java runtime, is the best-known example, and similar tools exist for Python, Node.js and .NET. Developers change no code, but they must add the agent to every service, and each language has its own agent with its own quirks. The agent lives inside the program’s process and shares its memory and CPU.

Option Three: eBPF-Based Instrumentation

With eBPF, a separate program runs next to the application, often one per machine, and installs tiny probes in the operating system’s kernel. These probes watch system events, such as a network read, and the moments when certain functions in the application are called. The separate program turns those observations into standard OpenTelemetry spans and metrics. The application is unaware and unchanged.

The name sounds technical, but the idea is simple: run small, checked programs inside the operating system without modifying it.

A Short History

The story starts in 1992, when researchers designed the BSD Packet Filter, known as BPF. Its job was narrow: let a tool like tcpdump tell the kernel which network packets it wanted, so that the kernel could discard the rest early and cheaply. BPF was a tiny virtual machine inside the kernel, and its design proved sound for decades.

Around 2014, engineers extended it dramatically for Linux. The new version, extended BPF or eBPF, supports far more than packet filtering. Programs can be attached to many kinds of events, can remember information between events, and can pass data to ordinary programs. Despite the name, eBPF today is a general-purpose way to safely extend the kernel, and the letters are treated as a name, not an abbreviation.

What the Kernel Is, and Why Programs Want to Watch It

The kernel is the core of the operating system. It manages memory, schedules which program runs, talks to disks and network cards, and mediates every request an application makes of the machine. Every file opened, every network connection made and every byte sent passes through it. That makes the kernel a perfect observation post: if you can watch it, you can see what every program on the machine is doing.

Historically, extending the kernel meant either changing its source code, a slow process involving the whole operating system community, or loading a kernel module, a plug-in with full power that can crash the entire machine if it has a bug. eBPF offers a third path.

Safety Through the Verifier

The key to eBPF is the verifier, a checker built into the kernel. Before any eBPF program is allowed to run, the verifier analyzes every possible path through it. It rejects programs that could loop forever, read memory they should not, or crash the system. A program that passes is compiled into native machine code by a just-in-time compiler, so it runs at close to full speed.

The price of this safety is restriction. eBPF programs must be small and must finish quickly. They cannot wait for a disk or sleep. They cannot call arbitrary kernel functions, only a limited set of approved helpers. That limited power is exactly what makes them safe to run in production.

Where eBPF Is Used Today

Observability is only one use. eBPF powers networking, where projects such as Cilium route and filter traffic inside the kernel, and large-scale load balancers and attack filters process packets at very high speed. It powers security tools that watch for suspicious system calls. Performance engineers have used it for years to answer questions like “why is this machine slow?” without installing heavy tools. OpenTelemetry’s use of it is a natural extension of that story.

Following one probe from creation to data delivery explains most of the vocabulary you will meet.

flowchart TD
  A["Developer writes a small program
in restricted C or Rust"] --> B["Compiler produces eBPF bytecode"]
  B --> C["User-space loader asks the kernel
to load the program"]
  C --> D{"Verifier checks safety"}
  D -->|"Rejected"| X["Load fails with an error"]
  D -->|"Accepted"| E["Compiled to native code"]
  E --> F["Attached to a hook
for example a function call or a network read"]
  F --> G["Event happens and the program runs"]
  G --> H["Results written to a map or ring buffer"]
  H --> I["User-space agent reads the data"]
    

The Steps in Plain Words

Maps and Ring Buffers

eBPF programs cannot hold information in ordinary variables between runs, because each run is a fresh, short event. Instead they use maps, shared tables managed by the kernel. A program can record “request 42 started at this time” in a map when a call begins, and look it up when the call ends to compute the duration. Maps are also how user-space and kernel programs exchange settings, such as which processes to watch.

To send a stream of observations out, programs write to a ring buffer, a fixed-size circular area that the agent reads. If the agent falls behind and the buffer fills, new events are dropped. That is a design trade-off to remember: the kernel side never waits, so a slow reader loses data instead of slowing the system.

Compile Once, Run Everywhere

Kernels differ in the exact layout of their internal data. A program written for one kernel version may look in the wrong place on another. The solution, called CO-RE for compile once, run everywhere, relies on a description of the kernel’s own data layout, called BTF. At load time the loader adjusts the program to match the machine it runs on. This is why modern eBPF agents can ship one build for many kernels, and why they usually require a kernel that exposes BTF.

An eBPF program does nothing until it is attached to an event. The choice of hook decides what the instrumentation can see.

The Main Hook Types

User-Space Probes and Their Fragility

User-space probes are the tool that lets eBPF understand application behaviour. A probe can be attached to a specific function in a program, so that each time the function runs, the eBPF code can read its arguments. To place such a probe, the agent must know where the function lives in the program’s file, usually by reading its symbol table, the list of function names and addresses.

This creates dependencies. If a binary is stripped of symbols to save space, probes cannot find their targets. If a new version of a library renames or reorganizes a function, probes written for the old layout may fail or read the wrong data. The instrumentation must be updated whenever supported libraries change enough to move things around. This is the main reason that probes into application code need constant maintenance, while probes on stable kernel events change rarely.

Kernel Probes and Stable Surfaces

Probes at the kernel level, such as those watching network reads and writes, are more portable across applications because every program uses the same system interfaces. A web server in any language eventually calls the kernel to read bytes from a connection. That common surface is what lets a single agent observe programs written in many languages. The agent then examines the bytes to recognize protocols such as HTTP, which is the topic of later chapters.

An eBPF instrumentation tool is not just a few kernel probes. It is a small system with a kernel half and a user-space half, each doing the jobs it is best at.

flowchart LR
  subgraph Node["One machine"]
    APP["Applications
any language"] --> K["Kernel with eBPF probes"]
    K --> RB["Ring buffers and maps"]
    RB --> AG["User-space agent
builds spans and metrics"]
  end
  AG -->|"OTLP"| COL["Collector"]
  COL --> BE["Backend"]
    

The Kernel Half

The kernel half is a set of small eBPF programs attached to hooks. Their job is deliberately narrow: notice that something happened, capture a few facts, and hand them over quickly. They record things like the time, the process that triggered the event, the connection involved and the first bytes of a message. They do not try to understand what they see, because they are limited in size and cannot do heavy work.

The User-Space Half

The user-space half is an ordinary program, often written in Go or Rust, running as a normal process. It loads and attaches the probes, reads the ring buffers, and does the heavy thinking that the kernel side may not do. It parses protocol bytes into requests and responses, pairs them up to compute durations, builds OpenTelemetry spans and metrics with standard names, enriches them with information such as the Kubernetes pod name, and exports them using OTLP, the OpenTelemetry Protocol, usually to a Collector.

Why Split the Work This Way

Doing as little as possible inside the kernel keeps programs inside the verifier’s limits and keeps the risk low. If the user-space half crashes, the application keeps running, because it never depended on the agent. If a parser has a bug, it affects the agent’s output, not the stability of the machine. And updating parsing logic is as easy as updating an ordinary program, with no kernel changes required.

Several projects bring eBPF into the OpenTelemetry family. They differ in what they watch and in how mature they are.

Go Automatic Instrumentation

The OpenTelemetry Go automatic instrumentation project targets programs written in Go. Go programs are compiled into a single native file with no runtime agent mechanism like Java’s, so a classic in-process agent is not possible. The project attaches user-space probes to well-known functions in popular Go libraries, such as the standard web server and client, database access, and common messaging and RPC libraries. When those functions run, the probes read their arguments and produce spans. It usually targets a single process identified by the path of its executable, and it can be deployed in Kubernetes through the OpenTelemetry Operator or as a companion container.

OpenTelemetry eBPF Instrumentation

The second effort is a language-independent tool. It began as Beyla, a project from Grafana Labs, and as far as is publicly known it was contributed to the OpenTelemetry project and continues there as OpenTelemetry eBPF Instrumentation, often abbreviated OBI. It watches kernel-level activity to recognize web, RPC and database traffic from programs in many languages and produces spans and request metrics. It can discover which processes to watch using listening ports, executable names or Kubernetes labels, and it understands Kubernetes metadata. Because the project is evolving quickly, check its current documentation for supported protocols and kernel requirements.

The eBPF Profiler

The third project is a continuous profiler, contributed to OpenTelemetry from a commercial profiling agent. Instead of watching requests, it samples what code is running on each CPU many times per second across the whole machine and builds a picture of where time is spent. It is notable for walking the call stacks of native code and several language runtimes, directly from eBPF, without needing special compile options in the programs. Profiles are a newer OpenTelemetry signal that is still being defined, so expect changes.

Network Observability and the Wider Ecosystem

Other work focuses on network observability: watching connections, bytes and retransmissions between workloads to build maps of who talks to whom. Beyond OpenTelemetry’s own repositories, an ecosystem of tools uses eBPF and can export OTLP. Examples include Kubernetes-focused observability platforms and distributions that mix eBPF for some languages with in-process agents for others, plus offerings from commercial vendors. Each has its own strengths, and their behaviour does not always match the OpenTelemetry projects exactly.

The magic has a mechanism. Understanding it explains both the strengths and the limits of the approach.

Recognizing Protocols From Bytes

When a program receives an HTTP request, the bytes arrive through the kernel. A probe on the kernel function that delivers those bytes can see the first part of each message. The user-space agent examines it. HTTP requests begin with a recognizable pattern: a method, a path and a version. The agent can tell that a request arrived, which method and path it used, and, when the program writes its reply, which status code it sent. The time between the two is the request’s duration. This works for any language, because any web server must read and write through the kernel.

The same idea extends to other protocols: HTTP/2 and gRPC, several database protocols, caches and message brokers. Each needs a parser that understands that protocol’s message layout, so supported protocols are a key feature to check in any tool.

Seeing Inside Encrypted Traffic

Most production traffic is encrypted. At the kernel level, bytes on the connection are scrambled and cannot be read. The workaround is to attach user-space probes to the functions in the encryption library, such as the widely used OpenSSL or the Go standard library’s encryption, at the point where plain data is handed over for encryption or has just been decrypted. This yields readable requests without breaking encryption on the wire. It depends on the application using a library the agent knows how to hook, which is one more compatibility question to ask.

Hooking Language-Level Functions

For some languages, especially Go, user-space probes can be attached directly to functions in common libraries. The probe reads the function’s arguments and sees structured information, such as the parsed request object, rather than raw bytes. That gives a more exact view, at the cost of depending on library versions and internal data layouts.

From Events to Standard Telemetry

A single request, traced step by step, shows exactly where the kernel probes and the agent do their work.

sequenceDiagram
  participant C as Caller
  participant K as Kernel probes
  participant A as Application
  participant G as eBPF agent
  participant O as Collector
  C->>A: HTTP request arrives
  K->>K: Note read event and start time
  A->>A: Handles the request
  A->>C: Response is written
  K->>G: Event with timing and first bytes
  G->>G: Parse request and build span
  G->>O: Send span and metrics over OTLP
    

The Story in Eight Steps

What Is Missing From the Story

Notice what the agent knows and what it does not. It knows the method, path, status, timing and the endpoints of each connection. It does not know the customer’s identity, the order’s value, which branch of the code ran, or why the downstream call happened. It also has to decide how the server span and the client span relate, and how this trace connects to the caller’s trace on another machine. That last question is difficult enough to need its own chapter.

A trace is only a trace if its pieces connect. Making them connect from outside the application is the hardest thing eBPF instrumentation does.

What Context Propagation Is

When service A calls service B, they need to share an identifier saying “this call belongs to trace number 4711, and I am span number 9”. That information travels in a standard HTTP header called traceparent, defined by the W3C Trace Context standard. Service B reads the header and continues the same trace. This hand-off is called context propagation. Without it, each service creates its own separate, unconnected trace.

Reading Is Easy, Writing Is Hard

Reading the incoming header is straightforward. The bytes are already present in the request, so the agent can read the trace identifier and attach its spans to the caller’s trace. Writing the header into an outgoing request is much harder. The bytes of the outgoing request belong to the application. To add a header, something must change those bytes before they are sent, and the safety rules that make eBPF trustworthy deliberately restrict how much a program may change.

How Tools Approach It

The details differ between tools and change as projects mature, so check what your chosen tool supports on your kernel version and protocols.

What Happens When Propagation Fails

When the chain breaks, each service still gets accurate spans and metrics, but traces no longer cross service boundaries. You see many short traces instead of one long one. Service maps built from connection data still work, because they depend on who connected to whom, not on trace identifiers. Latency by service is still correct. What you lose is the ability to follow one specific request from front door to database.

The promise of “any language” is real at one level and limited at another. The difference depends on how each language runs.

Two Levels of Visibility

At the protocol level, the agent sees traffic crossing the kernel: requests, responses, timing and connection details. This works for almost any language, because every language eventually calls the same kernel functions. At the function level, the agent sees inside the application, such as the arguments of a specific library call. This depends on how the language is compiled and executed, and works only where probes can find stable function addresses.

The Concurrency Puzzle

To compute a request’s duration, the agent must match its start to its end. In old-style servers, one thread handles one request at a time, so matching by thread is easy. Modern programs handle many requests on the same thread, or move one request between threads: event loops in Node.js, goroutines in Go, async code in many languages. Matching must therefore rely on the connection a request arrived on, not on the thread. This works well for simple request-response protocols, and less well when many requests share one connection, as in HTTP/2 and gRPC, where the agent must also track stream identifiers.

When Depth Matters, Add an In-Process Agent

For Java and similar runtimes, a common pattern is to let the eBPF agent provide baseline coverage for everything, and attach the language’s own in-process agent only to the services where deeper detail repays the extra setup. The protocol-level data from eBPF still gives you a consistent view across the fleet, even for services that have no in-process agent.

Kubernetes is where eBPF instrumentation is most often used, because one agent per machine can cover every pod on it.

The Node-Level Agent

Kubernetes organizes work into pods, each holding one or more containers, running on machines called nodes. A DaemonSet is a Kubernetes object that runs exactly one copy of a program on every node. Deploying the eBPF agent as a DaemonSet gives complete node coverage: when a new pod lands on a node, the local agent notices and begins watching it, with no change to the pod.

flowchart TD
  subgraph N1["Node 1"]
    P1["Pod A"] -.->|"observed by"| AG1["Agent pod from the DaemonSet"]
    P2["Pod B"] -.->|"observed by"| AG1
  end
  subgraph N2["Node 2"]
    P3["Pod C"] -.->|"observed by"| AG2["Agent pod from the DaemonSet"]
  end
  AG1 -->|"OTLP"| GW["Collector gateway"]
  AG2 -->|"OTLP"| GW
  GW --> BE["Backend"]
    

What the Agent Needs

• Visibility into other containers’ processes. By default a container sees only its own processes. The agent usually runs with access to the host’s process list so it can see and probe programs in neighbouring pods.
• Elevated permissions. Loading eBPF programs needs special capabilities, covered in the next chapter. Many deployments run the agent as a privileged container, and security-conscious ones list only the exact capabilities required.
• Access to kernel interfaces. The agent mounts a few host paths, such as the filesystem that holds eBPF objects and the file describing kernel data layouts.
• Read access to Kubernetes metadata. To label telemetry with pod and workload names, the agent asks the Kubernetes API, which needs a role granting read-only access.

Choosing What to Watch

Watching every process on every node is rarely what you want. Agents let you narrow scope by listening port, executable name, namespace or labels. Starting narrow, for example one namespace, lets you check data quality and measure overhead before expanding. Excluding noisy infrastructure processes also keeps costs and cardinality under control.

Sidecar and Operator Options

Some projects, especially the Go instrumentation, can run beside a single application instead of per node. The OpenTelemetry Operator for Kubernetes can inject such helpers when a pod carries the right annotations, and shared process visibility inside the pod lets the helper see the application’s process. The per-pod approach gives tighter control and clearer ownership, at the price of more copies of the agent and more resource requests.

eBPF is powerful because it works at a low level, and that brings requirements that SDKs never have.

The Requirements Checklist

Capabilities and Privilege

Linux divides administrator power into small pieces called capabilities. Newer kernels have capabilities specifically for loading eBPF programs and for performance monitoring, so agents no longer need full administrator power. Typical agents still ask for a short list that includes permission to load programs, to observe other processes and to read certain system files. The tool’s documentation lists the exact set. Granting only that list, instead of a blanket privileged mode, is a straightforward security improvement.

The Compatibility Matrix

Large organizations run a mix of kernels: new cloud images, older long-term-support enterprise distributions with features back-ported, and vendor-specific kernels. Build a simple matrix of operating system image, kernel version, BTF availability and security policy for each cluster. Test the agent on one machine from each row before a broad rollout. The results often show a few special cases, such as an older node pool that needs an upgrade or an exception.

Where eBPF Agents Cannot Run

Some environments simply do not allow it. Serverless container platforms that give you no access to the underlying kernel cannot host a node-level agent. Some managed Kubernetes offerings restrict privileged workloads or hide the nodes. Local development on laptops with other operating systems usually runs a Linux virtual machine, which may or may not expose the features required. In these places, in-process instrumentation remains the practical choice.

An observation of “a request took 80 milliseconds” is useless until you know which service it belongs to. eBPF sees processes and sockets; people think in services and teams.

The Identity Gap

What eBPF sees at the kernel level is a process identifier, a binary name and a network address. What dashboards need is a service name, a namespace, a version, a deployment environment and an owner. Bridging this gap is the job of the agent’s metadata enrichment, and doing it well decides whether the data is usable.

Where the Names Come From

Aligning With SDK-Produced Data

If some services use SDKs and others use eBPF, the same service should carry the same name in both worlds. Otherwise a dashboard for “checkout” shows half the story, and totals are wrong. Establish one naming rule, such as “the service name is the Kubernetes workload name unless an explicit label overrides it”, and apply it in the agent, in SDK configuration and in the Collector. Write the rule down and check it with a periodic report of names that appear in only one source.

Kubernetes Cost and Permissions

Enrichment from the Kubernetes API requires the agent to watch pods and other objects. On large clusters, hundreds of agents each watching everything can create real load on the API. Efficient agents cache data, watch only the objects on their own node, or delegate enrichment to a central Collector. Whichever design you choose, grant read-only roles, and test behaviour during large rollouts, when many pods change at once.

The agent is the first stop. The OpenTelemetry Collector is where shared policy is applied before data reaches the backend.

Where to Send the Data

The agent exports using OTLP to a Collector, either one running on the same node or a shared gateway pool. A Collector is a program with receivers that accept data, processors that change it, and exporters that send it onward. Sending through a Collector, rather than straight to a backend, lets you apply consistent rules to data from both eBPF agents and SDKs: filtering, renaming, sampling, redaction and routing.

A Sensible Processor Order

The Cardinality Problem Is Bigger With Protocol Parsing

An agent that reads request paths from raw bytes may see paths like /orders/8841, /orders/8842 and so on, with a different identifier each time. If these become metric attributes, each identifier creates a new time series, and the backend’s cost balloons. Good agents try to infer route templates, such as /orders/{id}, and good Collector pipelines add a normalization step as a safety net. Watch the number of distinct values of the route attribute during the first days of a rollout.

Mind the Edition of the Names

The agent emits attribute and metric names from some edition of OpenTelemetry’s semantic conventions, the shared dictionary of names. Your SDK-based services may emit a different edition. Check the schema URL, the label that identifies the edition, and normalize in the Collector where editions differ. A short comparison of the agent’s output against your registry or rule sheet, done before broad rollout, prevents surprises in dashboards and alerts.

eBPF is often described as low overhead. That is true in design, but the actual cost depends on how many events your machines produce and how much work the agent does with each one.

Where the Cost Comes From

A Back-of-the-Envelope Estimate

The following numbers are illustrative assumptions, not measurements. Suppose a node serves 5,000 requests per second, and each request produces about six probe events: one for the inbound read, one for the outbound response write, and two each for a downstream call. That is 30,000 events per second. If a probe takes about one microsecond to run, the kernel-side cost is 30 milliseconds of CPU per second, roughly three percent of one core spread across the application’s own request paths.

Now consider the agent. If parsing and span building take around ten microseconds per event, the agent needs about 300 milliseconds of CPU per second, close to a third of one core. The lesson is that the user-space half usually costs more than the kernel half. Efficient parsing, caching and batching matter more than shaving nanoseconds from probes.

Sharing Across Processes

An in-process agent runs inside every service, with its own memory and CPU use, multiplied by the number of service instances. An eBPF agent runs once per node, so its fixed costs are paid once for all the pods on the machine. For nodes that host dozens of small services, this amortization is a real advantage. For a node running a single large service, the benefit is smaller.

How to Measure It Yourself

A handful of settings decide whether an agent is quiet and useful or noisy and costly across hundreds of nodes.

Narrow the Scope

The cheapest event is one that is never observed. Restrict the agent to the namespaces, ports or workloads you care about. Exclude infrastructure noise and test traffic. Disable protocols you do not use, so parsers for them are not even loaded. Scope is the biggest single lever, and the easiest to adjust gradually.

Mind the Ring Buffer

Remember that the ring buffer is a fixed-size shared area. If events arrive faster than the agent drains them, new events are dropped. Suppose each event carries 300 bytes and the node produces 30,000 events per second. That is about nine megabytes per second flowing into the buffer. A buffer of eight mebibytes would hold less than one second of events. If the agent stalls for longer than that, because of CPU starvation or a garbage-collection pause, data is lost. Larger buffers buy time, at the cost of kernel memory, and sufficient CPU for the agent buys more.

A Table of Levers

Treat Rollout as an Experiment

Start with one cluster and one namespace. Record three baseline numbers: agent CPU, drop counts and number of distinct route values. Expand in steps, such as ten percent of nodes, then half, then all, and check the same numbers at each step. Keep a documented way to disable the agent quickly, such as a configuration flag, so that a problem found at three in the morning can be contained in minutes.

Traces tell you which request was slow. Profiles tell you which lines of code were busy. eBPF makes the second kind of data cheap enough to collect all the time.

What a Profile Is

A CPU profile is a statistical picture of where a program spends its processing time. A profiler periodically interrupts the processor, notes the call stack, the chain of functions currently running, and moves on. After thousands of samples, the functions that appear most often are the ones using the most CPU. Results are usually drawn as a flame graph, where wider boxes mean more time.

Why eBPF Suits Profiling

A performance event hook lets an eBPF program run at a fixed rate on each CPU, for example some tens of times per second. At each tick, the program records the current stack. Because it runs in the kernel and sees every process, a single agent per machine profiles all programs at once, without restarting them or adding special flags. At such a low sampling rate, overhead stays small enough to leave switched on permanently, which is what makes continuous profiling practical.

The hard part is reading stacks. Native programs may omit the information that makes unwinding easy. Interpreted and just-in-time-compiled languages keep their own structures. OpenTelemetry’s eBPF profiler contains logic to unwind native code and several popular language runtimes directly from eBPF, which is a large part of its value.

Cost of the Data

Take a machine with 64 CPU cores and a sampling rate of 20 samples per second per core. That is 1,280 stacks per second. Many repeat the same few paths, so the agent aggregates them into counts before export, and the data is small compared with traces. Turning raw addresses into readable function names, called symbolization, is often done later on a server, using debug information, to keep the node light.

What Questions It Answers

Sometimes the question is not what the code did but how the network behaved between services. eBPF sits right where that information lives.

What the Kernel Knows About Connections

The kernel tracks every network connection: who opened it, to whom, how many bytes moved, how many packets were lost and retransmitted, and how long round trips take. Probes and socket hooks can sample these facts without touching the application. Combined with Kubernetes metadata, they turn raw addresses into statements like “the checkout pods sent 40 gigabytes to the inventory pods yesterday, with a rising rate of retransmissions”.

Typical Network Signals

Service Maps From Connections

Because each connection has a source and a destination, you can draw a map of which services talk to which, without any trace headers. This map is built from facts, not from declared configuration, so it reveals surprising dependencies such as a forgotten service still being called every minute. Even where full traces cannot be connected, these maps remain accurate.

Beware of Flow Cardinality

A raw list of every connection includes ephemeral ports and short-lived pod addresses, so its cardinality, the number of unique combinations, is enormous. Good network tools aggregate by workload, namespace or zone before export. Keep the unit of reporting at the level that people reason about, such as “service A to service B”, and drop port and address details from long-term metrics. Otherwise, the backend cost grows with every pod restart.

An agent that can see every request on a machine is powerful, and anything powerful must be handled with care.

What the Agent Can See

At the protocol level, an eBPF agent can read the beginning of every message crossing the machine. That includes URLs, headers and sometimes parts of bodies. Those may contain things that should never be stored: access tokens in headers, personal identifiers in paths, secrets in query strings, or fragments of database statements with real values. Even if the agent never exports these fields, it has had access to them, which matters for risk assessment and compliance.

Principles for Safe Use

• Capture the minimum. Limit how many bytes per message are captured and which fields are kept. Do not capture bodies unless you have a specific, reviewed need.
• Redact early. Remove or mask sensitive values in the agent, and again in the Collector, before anything is stored.
• Exclude sensitive workloads. Keep payment, health or identity systems out of scope until their owners have reviewed what will be observed.
• Get consent from owners. In shared clusters, tell teams what is being observed and how to opt out or adjust.
• Minimize privileges. Grant exactly the capabilities the tool documents, not a blanket privileged mode.

The Agent Is a Valuable Target

Because the agent has elevated permissions on every node, compromising it would be attractive to an attacker. Reduce that risk like you would for any privileged component: pull images only from trusted registries and verify signatures where available, pin versions, restrict who can modify the DaemonSet, keep the agent’s own network access narrow, and review its permissions after each upgrade. Log changes to its configuration, so that unusual changes are visible.

Verified Is Not the Same as Harmless

The kernel’s verifier guarantees that an eBPF program cannot crash the system or read memory outside its allowed areas. It does not guarantee that the program’s purpose is acceptable. A perfectly verified program can still read sensitive data it is allowed to see. Trust therefore comes from knowing who wrote the program, reviewing its source or its vendor’s attestations, and controlling who can load new programs on your nodes.

Honest tools have honest limits. Knowing them in advance prevents disappointment and wrong decisions.

The Business Meaning Gap

An eBPF agent sees that a request to /checkout took 230 milliseconds and returned a success code. It does not see that the customer was on a premium plan, that the cart held eleven items or that a discount rule fired. Those facts live inside the application’s memory and logic, and never cross a hook in a form the agent can interpret. Anything that requires knowing the meaning of the data, not just its shape, needs in-code instrumentation.

Errors That Look Like Successes

Many applications return a normal success status even when the work failed, placing the real error inside the body of the response. GraphQL services are a classic example. A protocol-level agent that reads only status codes will report these as healthy requests. If your error signals live in message bodies, you need the application to expose them as attributes or metrics.

A Table of Gaps

Service Meshes and Double Counting

In clusters that use a service mesh, a sidecar proxy sits beside each application and handles its network traffic. An eBPF agent on the same node can see both the application’s traffic to the proxy and the proxy’s traffic to the network, producing two sets of spans for one call. Decide which layer is the source of truth for each service, and filter the other, so that you do not double the volume or the apparent request count.

No approach wins every comparison. The table below sets them side by side so you can see the trade-offs at a glance.

Questions That Decide the Choice

• Can you change and redeploy the code? If not, eBPF may be your only practical option.
• Do you need business detail? If so, plan for SDK instrumentation on key services.
• Can your platform host privileged node agents? If not, in-process agents are the realistic path.
• How many languages do you run? The more there are, the more attractive a single node-level approach becomes.
• How important are complete end-to-end traces? If very, test propagation carefully with your actual traffic.

The most successful teams treat eBPF as the floor of their observability and SDKs as the ceiling.

The Floor

The floor is the minimum level of visibility every service must have, no matter who owns it or what language it uses: request rate, errors, duration, dependencies and basic resource use. eBPF is ideal for establishing this floor quickly, because it requires nothing from service teams. Once it is in place, nobody is invisible, and unknown services stop being a surprise during incidents.

The Ceiling

The ceiling is the richest view you can get, built by people who know the code. Critical services receive manual spans around key business steps, custom metrics for things like orders and payments, and attributes that identify customers, plans and features. This costs effort, so it is applied where the value is highest.

flowchart TD
  A["Can you change and redeploy the code?"] -->|"No"| B["Use eBPF as the main source"]
  A -->|"Yes"| C{"Need business detail
or custom spans?"}
  C -->|"Yes"| D["Use SDK or in-process agent
and add eBPF for gaps"]
  C -->|"No"| E{"Can the platform run
privileged node agents?"}
  E -->|"Yes"| F["Use eBPF for fast broad coverage"]
  E -->|"No"| G["Use in-process agents"]
    

Avoid Duplicates

When a service has both SDK instrumentation and an eBPF agent watching it, both may create a server span for the same request. The result is doubled counts and confusing traces. There are three ways to avoid it. Exclude SDK-instrumented services from the eBPF agent’s scope. Filter one source out in the Collector for those services. Or use an agent that detects existing instrumentation and steps back, if your chosen tool offers that behaviour. Whichever you choose, test with one service and compare counts against your load generator.

One Vocabulary for Both

Combined data is only useful if both sources speak the same language. Agree on one edition of the semantic conventions, one rule for service names, and one set of resource attributes. Use the Collector to normalize any differences. Dashboards then work across the whole fleet, and a service can graduate from eBPF-only to SDK instrumentation without breaking charts.

A Typical Journey

When data is missing or strange, a short list of usual causes explains most cases.

A Debugging Routine

Teams that adopt eBPF instrumentation successfully share a few habits. Teams that struggle share a few mistakes.

Habits That Work

Mistakes That Teach

The same technology gets used in quite different ways depending on the organization. These illustrative scenarios show the range.

Industry Context

eBPF’s reputation rests on years of production use. Large operators have publicly described using it for high-speed load balancing and for filtering attack traffic, and well-known performance engineers helped popularize eBPF tracing tools for finding bottlenecks on busy servers. OpenTelemetry’s contribution is to turn that raw capability into standard telemetry that any compatible tool can consume.

Adoption grows in stages. Knowing which stage you are in tells you what to do next, and what to avoid doing too early.

Do Not Climb Faster Than Your Platform

A company whose clusters run old kernels or locked-down managed services may stay at level one for a while, until platform upgrades catch up. That is a reasonable outcome. The right move is to use in-process agents where eBPF cannot run, and plan the platform changes that would make eBPF possible, instead of forcing a fragile deployment.

When you are ready to try this in your own environment, follow this order. Each step reduces risk for the next.

A quick reference for the terms used in this guide.

Short answers to the questions that come up most often.

eBPF-based instrumentation lets you observe programs from the operating system’s side, producing standard OpenTelemetry data without touching application code.

You began with a company that could not instrument all its code, and learned how eBPF offers a safe, language-independent way to watch programs from the kernel. You saw how probes and a user-space agent turn low-level events into spans, metrics, profiles and network signals, how the OpenTelemetry projects package those ideas, and where the approach struggles, especially with context propagation, business meaning, privileges and platform limits.