Eight words carry the entire topic. Learn them here and the rest of the guide reads easily.

Many modern applications are built as a collection of small programs called microservices. One handles logins, another handles payments, another recommends products. They talk to each other over the network all day long. As the number of services grows into the dozens or hundreds, a new problem appears: every service needs the same network chores done, such as encrypting traffic, retrying failed calls and measuring how long calls take. Writing that code again and again in every service is slow and error-prone.

Popular meshes include Istio, Linkerd, Consul and Cilium’s service mesh. They differ in design, but they share the idea that a layer outside your application code handles network behavior. That layer is in a unique position to watch every request, which is why meshes produce so much telemetry for free.

Without writing a single line of monitoring code, a mesh already produces three kinds of telemetry.

Because every request passes through a proxy, the proxy can note it down. Nobody has to change the application. That is the great attraction of mesh telemetry: it works uniformly across every service, whatever language it was written in, including old programs nobody wants to touch.

The golden metrics

Istio, for example, produces a standard set of metrics inside the proxy itself. They count requests, measure how long requests took, and measure the size of requests and responses. Each is labeled with who called, who answered, the response code and more. Together they give you the classic health indicators of a service: how much traffic, how many errors, and how slow. Engineers sometimes call these the RED measures, for rate, errors and duration.

Spans without application code

The proxy can also start and finish spans. Each proxy creates a span for the request it handles, recording the time it took and details such as the destination and the status. Envoy, the proxy under several meshes, supports sending these spans out using the OpenTelemetry protocol, and Istio can be told to use an OpenTelemetry provider for tracing.

Logs of every call

An access log is a line written for each request. It is the oldest form of network telemetry, and it is rich: it includes the response code, how long the call took, the bytes moved and a short code called a response flag that explains unusual endings such as a timeout or a refused connection.

If the mesh already records so much, why bring OpenTelemetry in at all?

The mesh sees only the edges of each service. It knows a request entered the payment service and left 90 milliseconds later. It cannot know what happened inside those 90 milliseconds: whether the time went into a database query, a call to a bank or a slow loop. Only code inside the service can say that, and OpenTelemetry is the standard way for that code to speak.

Four things OpenTelemetry provides

• SDKs and instrumentation that live inside your application and create detailed spans, metrics and logs.
• A common data protocol, OTLP, so telemetry from any source travels in one format.
• Semantic conventions, which are agreed names for common attributes, such as the HTTP method or the database name, so that data from different tools lines up.
• The Collector, a standalone program that receives telemetry from many sources, changes it and forwards it to storage.

The comparison

Each side covers the other’s blind spot. A wise team does not choose one. It joins them, and the rest of this guide explains how.

The simplest and most reliable design sends both streams to one OpenTelemetry Collector.

Here is the overall shape of a healthy integration. The application, using an OpenTelemetry SDK, sends its telemetry out. The proxy next to it sends its own telemetry out. Both go to a Collector, which tidies, labels and combines them, and then forwards everything to the backend where people look at dashboards.

flowchart LR
A["Application with OTel SDK"] -->|"local traffic"| P["Mesh proxy beside the app"]
A --> C["OpenTelemetry Collector"]
P --> C
CP["Mesh control plane"] -.-> P
C --> B["Backend for traces metrics and logs"]
    

Five jobs of the integration

The most valuable integration is also the most delicate one. A trace is only as good as its weakest link.

A trace is a tree of spans that share one trace identifier, a long random number created when the request first arrives. Every service that touches the request adds its own span to the tree. To do that, it must know the trace identifier, and the only way to learn it is from the request itself. The identifier is carried in a small set of HTTP headers. The web standard for this is called W3C Trace Context, and its main header is named traceparent.

sequenceDiagram
participant U as User
participant G as Ingress Gateway Proxy
participant PA as Proxy of Service A
participant A as Service A App
participant PB as Proxy of Service B
participant B as Service B App
U->>G: Request
Note over G: Creates the root span
G->>PA: Request with trace headers
PA->>A: Request with trace headers
Note over A: App reads the headers and keeps the trace id
A->>PA: Outgoing call with the same trace headers
PA->>PB: Request with trace headers
PB->>B: Request with trace headers
    

The part the mesh cannot do alone

A proxy sees an incoming request and, a moment later, an outgoing request. It cannot tell that the outgoing request was caused by the incoming one. Only the application knows that. So the application must copy the trace headers from the incoming request onto every outgoing request. This is called context propagation. If the application forgets, the outgoing call starts a brand new trace, and the story splits in two.

Two easy ways to pass the baton

• Use an OpenTelemetry SDK or automatic agent. It reads the incoming headers and writes them onto outgoing calls without extra effort. This also adds the application’s own spans inside the same trace.
• Copy the headers by hand. For tiny services without an SDK, forward the trace headers your mesh uses. This keeps the trace connected but adds no inner detail.

How mesh spans and application spans nest

In a healthy joined trace, each hop appears as a pair of mesh spans around the application’s own work. The calling proxy records a client span, which measures the call from the caller’s side, including time on the wire. The receiving proxy records a server span, which measures the request from the moment it arrived. Inside the server span, the application adds its own spans for the work it does. Reading from the top, the trace therefore shows: caller proxy, receiving proxy, application code, and then the next call out. The gaps between neighboring spans are time spent in transit or in queues, which is exactly the information you cannot get from application spans alone.

Everybody must speak the same header dialect, and someone must be first to speak.

Trace headers come in more than one dialect. The modern standard is W3C Trace Context. An older and still common format, B3, came from the Zipkin tracing project. Other formats exist for particular vendors. A propagator is the small piece of code that reads and writes one of these formats. If the proxy writes B3 headers and the application only understands W3C headers, the application sees no trace at all and starts a new one.

Pick one format and use it everywhere

The cleanest choice is W3C Trace Context in both the mesh and the applications. If a migration is slow, configure the application SDK with both propagators at once, so it can read whichever the proxy sent and write both when calling onward. Check your mesh’s documentation for how to set the format, because the setting differs between products and versions.

Who is the root?

The first proxy a request meets, usually the ingress gateway at the edge of your cluster, decides whether a trace exists. If the incoming request has no trace headers, the gateway creates the identifier and the first span, called the root span. Every later hop becomes a child. If the caller outside your system already sent trace headers, such as a partner’s system, you can choose to continue their trace or start your own. Starting your own is safer when the caller is untrusted, because outsiders could otherwise influence your sampling or flood your storage.

Baggage: a small passenger on the same ride

Baggage is a companion standard that carries small name and value pairs along with the request, such as a customer tier or a test-traffic flag. Proxies generally pass baggage along untouched, so a service far down the chain can read what the front door noted. It is handy for labeling, but remember that baggage travels in clear headers, so never put secrets in it.

Recording everything is expensive. The danger is that the mesh and the application each decide differently.

Storing a trace for every request would drown most systems, so engineers record only a fraction. This is called sampling. In a mesh, the proxies can sample, and the applications can sample too. If they do so independently, you get half-finished stories: the proxy kept the trace, but the application dropped it, or the other way around.

flowchart TB
R["Request arrives at the gateway"] --> D{"Is there already a sampling decision in the headers?"}
D -- "Yes" --> F["Every proxy and app follows that decision"]
D -- "No" --> M["Gateway rolls the dice at the configured rate"]
M --> K{"Sampled?"}
K -- "Yes" --> S["Mark the trace as sampled and record spans"]
K -- "No" --> N["Mark as not sampled and skip spans"]
    

The rule that keeps stories whole

The trace headers include a sampled flag. The first component to see the request decides and writes the flag. Everyone downstream should follow the parent’s decision. In OpenTelemetry SDKs this behavior is called a parent-based sampler. When the application uses it, the application records spans exactly when the gateway did, and the trace is complete or absent, never half there.

A note on defaults

Meshes usually ship with a low default sampling rate, since tracing every request on a busy mesh is costly. Istio, for instance, commonly defaults to about one percent. That is wise for production, and it surprises people who try the system with a single test request and see nothing. In test environments, raise the rate to a hundred percent. In production, raise it only as far as your budget and storage allow.

Choosing a rate: a quick calculation

Suppose the gateway receives 1,000 requests per second. At one percent sampling, ten requests per second are kept, which is 864,000 traces per day. At a hundred percent, it would be 86,400,000 traces per day, a hundred times more. Because every trace holds many spans, one for each hop and each internal step, the span count is higher still. A sensible plan keeps a low rate for ordinary traffic and finds a way to keep the interesting requests, such as errors and slow ones, through smarter sampling in the Collector.

Mesh metrics speak a Prometheus dialect. OpenTelemetry has its own conventions. The Collector translates between them.

A mesh like Istio exposes its metrics in the Prometheus format, a plain-text layout that is read by periodically visiting a web address on each proxy, an act called scraping. The OpenTelemetry Collector has a Prometheus receiver that does exactly this, turning the scraped values into OpenTelemetry metrics and passing them along with everything else.

Same idea, different names and units

Notice the unit difference. The mesh reports milliseconds, and the OpenTelemetry convention uses seconds. A call that takes 250 milliseconds is 0.25 seconds. If you plot the two on one chart without converting, one line is a thousand times larger than the other. The Collector can rename metrics and convert units, so decide on one convention and let the Collector apply it.

The double counting trap

If both the proxy and the application measure the same request, you now have two request counts for one request. Adding them together doubles every number. The usual remedy is to choose a source of truth for each question. Use the mesh metrics for traffic between services, since they cover every service uniformly. Use the application metrics for business meaning and internal work. Keep them in separate, clearly named families, and never sum across the two.

Metrics and traces answer different questions

A metric tells you that something is wrong and how much, for all traffic, cheaply. A trace tells you where and why, but only for the requests you recorded. Because the mesh counts every request in its metrics, even when traces are sampled at one percent, mesh metrics are an unbiased measure of traffic. Use them to notice problems, and use traces to explain them. Teams that rely on sampled traces to count traffic end up undercounting by a factor of a hundred, while teams that rely on metrics alone know something is broken but cannot say where.

Linking numbers to stories

OpenTelemetry supports exemplars, small samples that attach a trace identifier to a metric point. With them, a spike on an error chart can link straight to a trace that shows a failing request. Support depends on your tools, but when present, it makes the integration of metrics and traces feel seamless.

An access log line is far more useful when it carries the trace identifier. This chapter shows how to get there.

By default, many meshes write access logs to the proxy’s standard output as plain text, which the platform collects like any other container log. That works, but it leaves the log as a separate island. Two improvements bring it into the OpenTelemetry world. First, the mesh can be told to send access logs through OpenTelemetry’s protocol directly to the Collector, as structured records rather than text. Istio’s configuration includes an OpenTelemetry access log provider for this, although the exact setup differs between versions. Second, the log should include the trace identifier, so a person looking at a slow span can jump to the exact log line for that request, and the other way around.

What an access log adds that spans do not

• Every request, not a sample. Traces are usually sampled at a low rate, but access logs can cover all requests, so no failed call goes unrecorded.
• Response flags. Short codes that say why a request ended unusually, for example no healthy server was available, the upstream timed out or the connection was reset.
• Network facts. Source and destination addresses, bytes sent and received, and the time spent in each phase of the call.

Volume is the catch

An access log line is typically a few hundred bytes. If one service receives 1,000 requests per second, that is 86.4 million lines per day. At about 500 bytes each, that is 43.2 billion bytes, or roughly 43 gigabytes, per day from just one busy service. A mesh with hundreds of services can produce terabytes. The good news is that the Collector can help. You can log only errors, drop health-check traffic, or keep a small random share of successes while keeping every failure.

If the mesh calls a service one thing and the application calls it another, your charts will show two services where there is only one.

Every piece of OpenTelemetry telemetry carries resource attributes, which describe who produced it. The most important is service.name. When the application’s SDK says “this is the checkout service” and the mesh proxy says “this came from a workload named checkout-v2 in the shop namespace,” a naive backend treats these as two different things. Traces then show a mesh node and an application node side by side instead of one merged service.

The cure: normalize in the Collector

The Collector sits in the middle and can fix this in one place. Its Kubernetes attributes processor labels every record with the pod, namespace and deployment it came from, using information from the cluster. Its attributes and transform processors can rename a value, remove a suffix or set service.name from a trusted source, such as a label on the workload. Choose a single rule, for example “the service name is the value of the app label on the pod,” and apply it to application and mesh data alike.

Other labels worth aligning

• Version. Mesh metrics often include a workload version label, which is perfect for comparing a new release with the old one during a gradual rollout.
• Environment. Production, staging and test, in one spelling.
• Team or owner. Useful for routing alerts and attributing cost.
• Cluster and region. Important for multi-cluster meshes, where the same service name may exist in several places.

Every proxy needs a nearby place to send its telemetry. Where you put the Collector decides how reliable and how costly the whole system is.

Hundreds of proxies, each sending spans, logs and metrics, can overwhelm a single destination. The usual design uses two layers, shown below. A small node Collector runs on every machine, as a daemon-style program that Kubernetes places once per node. Proxies and applications on that machine send to it, over a fast local hop. The node Collectors then forward to a central gateway tier of larger Collectors that do the heavy work and talk to the backend.

flowchart LR
subgraph Node1["Node 1"]
A1["Pods with proxies"] --> D1["Node Collector"]
end
subgraph Node2["Node 2"]
A2["Pods with proxies"] --> D2["Node Collector"]
end
D1 --> G["Gateway Collector tier"]
D2 --> G
G --> B["Backend"]
    

A loop to avoid

If the Collector itself runs inside the mesh with a proxy beside it, the traffic that delivers telemetry is also traced by the mesh. Now the act of delivering spans creates new spans, which must be delivered, which creates more spans. This feedback loop wastes resources and fills your storage with telemetry about telemetry. Prevent it by keeping Collector pods out of the mesh, or by turning off tracing and access logging for their traffic.

Sizing and keeping the Collector reliable

The Collector is now part of your production path, so it needs the same care as any service. Give it memory and processor limits, and enable its memory limiter so it refuses data politely instead of crashing when overloaded. Run several gateway replicas, and put a load balancer that understands request traffic in front of them, so a single slow replica does not hold up many proxies. Watch its own metrics: queue size, dropped items and export failures. If the Collector falls behind, proxies and applications drop telemetry silently, and you lose the very evidence you need during an incident.

The proxy is the only witness for a whole family of events. Without its telemetry, they are invisible.

Many important events happen in the network layer, before a request ever reaches your code or after it leaves. The application cannot report them because it never knew. Only the proxy can.

Identity you can trust

When the mesh uses mutual TLS, each proxy proves who it is to the other using a certificate tied to the workload. The proxy can therefore record not just “a request came from this address,” but “a request came from this verified workload identity.” Addresses change as pods come and go, but identities stay meaningful. Recording the verified source and destination in metrics and logs makes security reviews far easier, because you can answer “which services actually call the payment service” from evidence rather than from a diagram that may be out of date.

Response flags are a gift

Envoy marks each request with short response flags. A few examples: one means no healthy upstream server was available, another means the upstream connection failed, another means the upstream request timed out, and another means no route matched. When an engineer sees a burst of errors, the flag turns “something is wrong” into “the destination had no healthy servers.” Carrying these flags into your logs and, where possible, into span attributes is one of the most useful pieces of integration work.

The other half of the story is hidden from the proxy. Only your own code can tell it.

From the proxy’s point of view, a service is a closed box. A request goes in, a response comes out, and some time passes. The proxy has no idea what happened inside the box. OpenTelemetry in the application opens the box.

Business meaning

A proxy knows the address of a request. Your application knows what the request meant: which customer, which product, which feature flag, which plan. Adding such labels to spans, as attributes, turns the mesh’s anonymous call graph into a story that product and support teams can use, such as “orders from premium customers are slowest on the payment step.”

Where the two meet in a trace

In a joined trace, the mesh span for the call to the payment service has the application’s own span nested inside it, and the database and bank calls hang under that. The outer mesh span shows the time on the wire, the inner spans show what the code did. If the outer span is 300 milliseconds and the inner code span is 100 milliseconds, then 200 milliseconds were spent outside your code, in the network, in queues or in retries. That subtraction is one of the great pleasures of joined telemetry.

Not every mesh puts a proxy next to every service. The integration idea stays the same, but the details move.

The classic design puts one sidecar proxy in every pod. It is simple to reason about, and it costs memory and processor time for every pod. Newer designs try to reduce that cost.

flowchart LR
subgraph Sidecar["Sidecar model"]
S1["App"] --- S2["Proxy in every pod"]
end
subgraph Ambient["Ambient model"]
A1["App"] --- A2["Node level agent for connections"]
A2 --- A3["Optional shared request proxy"]
end
    

Ambient mode

Istio’s ambient mode, which has become a supported option in recent releases, removes the sidecar. A small agent on each node, named the zero-trust tunnel, handles secure connections for every pod on that node. It sees connection-level facts such as bytes and connections, and it can emit connection metrics. Where you need request-level detail, such as paths, status codes and spans, you add a shared waypoint proxy for a namespace or service. Request-level traces and metrics then come from the waypoint, so the integration rules from the earlier chapters apply to it.

Other meshes

• Linkerd uses its own lightweight proxy rather than Envoy and exposes Prometheus metrics. It can send spans to a Collector through its tracing add-on, although the details vary by version.
• Cilium’s service mesh uses kernel-level technology and a per-node Envoy, which again moves the telemetry source from the pod to the node.
• Consul typically runs Envoy sidecars, so Envoy-based guidance applies.

Telemetry is not free. Both the mesh and OpenTelemetry cost something, and together they can cost more than expected.

Every proxy uses memory and processor time, and each hop through a proxy adds a small delay. Producing telemetry adds more. Spans must be built, logs formatted, metrics counted and everything sent over the network. At a handful of services this does not matter. At hundreds, it is a real line on the budget.

Spans multiply

A request that passes through five services creates more spans than people expect. Each service can produce a span in the calling proxy, a span in the receiving proxy and one or more spans in the application code. Ten or more spans for a single request is normal. At 1,000 requests per second with ten spans each, that is 10,000 spans per second before sampling. This is why the sampling decision from Chapter 7 matters so much.

Telemetry describes who talked to whom and what they said. Treat it as sensitive data.

A mesh often exists to improve security, with encrypted connections and identity checks between services. It would be ironic if its telemetry then leaked the very information it protects.

The Collector as a privacy filter

Because all telemetry passes through the Collector, it is the natural place to enforce privacy rules. Its processors can delete sensitive attributes, mask parts of a value and drop whole records before anything reaches storage. Writing these rules once, centrally, is far safer than trusting every team to remember them in every service.

Most integration problems show themselves in a handful of ways. Learn the symptoms and the fix is usually short.

When the integration is wrong, the trace view looks wrong in recognizable ways. The flow below turns the most common symptoms into a quick path to the cause.

flowchart TB
S["A trace looks broken"] --> Q1{"Do app spans and mesh spans share a trace id?"}
Q1 -- "No" --> A1["Check header forwarding in the app and the propagation format"]
Q1 -- "Yes" --> Q2{"Is a hop missing?"}
Q2 -- "Yes" --> A2["Check sampling flags and proxy tracing settings for that workload"]
Q2 -- "No" --> A3["Check the export path to the Collector and the backend"]
    

A test that finds most problems

Send one request with a known trace identifier through the system, with the sampling flag set so it is recorded. Then search for that identifier in the backend. You should find the gateway span, each proxy span and every application span, connected as one tree. Wherever the tree breaks, the break is your bug. Keep this as an automatic check after every mesh or Collector upgrade.

Eight habits of teams that run mesh and OpenTelemetry together successfully, and the choices they weigh.

Trade-off: mesh-only tracing, or full application instrumentation?

Trade-off: sampling at the proxy, or in the Collector?

The usual compromise is a modest sample at the edge, so volume stays manageable, with extra rules in the Collector layer that preserve the high-value traces after the outcome is known. That second step needs all spans of a trace to reach the same Collector, which is a design constraint worth planning from the start.

Four habits that look sensible and quietly break the picture.

Other frequent mistakes

• Running the Collector inside the mesh without excluding its own traffic.
• Leaving the demo sampling rate in production, or the production rate in testing.
• Forgetting that queues and background jobs need trace headers copied into messages.
• Logging authorization headers in access logs.
• Upgrading the mesh without re-running the end-to-end trace check.

A practical order that delivers value early and avoids the traps above.

Five illustrative stories showing how the integration pays off.

These scenarios are illustrative composites. The numbers are chosen for clarity and are not measurements from any named company. For background, Envoy began at Lyft, Istio was started by Google, IBM and Lyft, and Linkerd, from the company Buoyant, is credited with popularizing the term “service mesh.”

Short answers to the questions people ask most often.