A mesh moves the hard parts of networking out of your code and into a layer that every service shares.

A service mesh is a layer of software that manages how services talk to each other. It has two halves. The control plane is the brain: it decides the rules, such as who may talk to whom and how many retries are allowed. The data plane is the muscle: it is the set of proxies that actually carry every request and enforce those rules. This guide is about the muscle.

A proxy is a program that stands in the middle of a conversation. A client talks to the proxy, and the proxy talks to the server on its behalf. Because the proxy is in the middle, it can inspect, change, retry, or refuse requests. In a mesh, there is one proxy per application instance, which is why the pattern is called a sidecar, like the small passenger car attached to the side of a motorcycle.

Why this design is so popular

Before service meshes, every team had to build networking features into their own code: retry logic, timeouts, encryption, and metrics. Different teams used different languages and libraries, so the behavior was inconsistent. A mesh moves these features into one proxy that behaves the same way for every service, whatever language it is written in. A security team can change a rule once and it takes effect across hundreds of services.

When one application becomes fifty small services, the network between them becomes the hardest part.

In a monolith, one part of the program calls another with a simple function call. It either works instantly or crashes in a clear way. When the program is split into many microservices, each function call becomes a network call. Networks are unreliable. Messages are delayed, lost, or duplicated. A service can be slow, dead, or half-alive. Suddenly every service needs to answer hard questions: how long should I wait, should I retry, who is this caller, can I trust them, and how do I know what happened?

Library approach versus proxy approach

Twelve short words that you will meet in every service mesh discussion.

North-south versus east-west

Two phrases describe the direction of traffic. North-south traffic enters or leaves the cluster, such as a user on the internet calling your website. East-west traffic moves between services inside the cluster, such as the checkout service calling the payment service. Gateways manage the first kind. Sidecars mostly manage the second.

One brain, many proxies, and a steady flow of instructions between them.

flowchart TB
  K8S["Kubernetes API server"] -->|"watch Services, Pods, policies"| ISTIOD
  subgraph CP["Control plane"]
    ISTIOD["istiod - config, certificates, xDS server"]
  end
  subgraph PA["Pod A"]
    AA["App container A"]  EA["Envoy sidecar A"]
  end
  subgraph PB["Pod B"]
    AB["App container B"]  EB["Envoy sidecar B"]
  end
  ISTIOD -->|"xDS config and certificates"| EA
  ISTIOD -->|"xDS config and certificates"| EB
  EA |"mTLS traffic"| EB
    

Look at Figure 1. The control plane, named istiod in Istio, watches the Kubernetes API server. It learns which Services and Pods exist, and which rules the operators have written. It then turns all this into proxy configuration and pushes it to every sidecar. When Pod A calls Pod B, the request leaves A’s application, passes through A’s Envoy, crosses the network encrypted, enters B’s Envoy, and only then reaches B’s application. The control plane is not in the path of any request, which is important: if istiod stops for a short time, existing traffic keeps flowing with the last configuration the proxies received.

You write a Pod with one container. By the time Kubernetes stores it, it has more. Here is how.

Nobody wants to edit hundreds of deployment files to add a proxy to each one. Instead, Kubernetes has a feature called an admission webhook. When you create any object, the API server passes it through a series of checks before saving it. A mutating admission webhook is allowed to change the object during this journey. The mesh registers one such webhook for Pods. Whenever a new Pod is created in a chosen namespace, the API server sends the Pod to the webhook, and the webhook replies with a patch that adds the proxy.

sequenceDiagram
  participant U as User or Deployment controller
  participant API as API Server
  participant WH as Injection webhook in istiod
  participant S as Scheduler
  participant K as Kubelet
  U->>API: Create Pod with one app container
  API->>WH: Admission review request
  WH-->>API: Patch adds init container and sidecar
  API->>API: Validate and store the patched Pod
  API-->>S: Watch event for unscheduled Pod
  S->>API: Bind Pod to node
  API-->>K: Watch event for Pod on this node
  K->>K: Run init container to set traffic rules
  K->>K: Start Envoy sidecar and app container
    

Deciding which Pods get a sidecar

Injection is controlled by labels and annotations, which are small tags on Kubernetes objects. The common pattern is to label a whole namespace so that every new Pod inside it is injected. Individual Pods can opt out, or opt in, with a Pod-level annotation. For safe upgrades, Istio uses revision labels: you install a new version of the control plane next to the old one, and move namespaces over one at a time by changing the label.

A look at the extra parts, one by one.

Two programs in one container

The proxy container does not simply run Envoy. It runs a small helper called pilot-agent, which in turn starts Envoy. The agent has several jobs. It creates the first, minimal configuration that Envoy needs to start. It asks the control plane for a certificate for the Pod and passes it to Envoy. It forwards configuration updates. It answers health checks. It restarts Envoy if it crashes. This division of labor keeps Envoy focused on moving traffic and leaves housekeeping to the agent.

Resource cost per Pod

Every sidecar uses memory and CPU. The default resource requests in Istio are around one tenth of a CPU core and 128 mebibytes of memory per proxy, though real use depends on traffic and on how large the mesh is. Multiply this by hundreds or thousands of Pods and the cost is real. Later chapters show how to reduce it. Treat the sidecar as a small tax on every Pod, in exchange for the features it brings.

The application believes it is talking directly to another service. The kernel quietly sends the bytes through the proxy instead.

The Pod shares one network space, called a network namespace, among all its containers. This means the application and the sidecar see the same network interfaces and the same local addresses. The init container uses this fact. It installs rules with a Linux tool called iptables, which sits inside the kernel and can rewrite where a packet is going. The rules say: any outgoing connection from this Pod, except those made by the proxy itself, should be redirected to the proxy’s outbound port. Any incoming connection to this Pod should be redirected to the proxy’s inbound port.

flowchart LR
  subgraph PA["Client Pod network namespace"]
    APP1["App sends request to a service"] --> IPT1["iptables redirect"]
    IPT1 --> OUT["Envoy outbound listener 15001"]
  end
  subgraph PB["Server Pod network namespace"]
    IPT2["iptables redirect"] --> IN["Envoy inbound listener 15006"]
    IN --> APP2["App receives request on its own port"]
  end
  OUT -->|"mTLS over the network"| IPT2
    

Well-known ports of the sidecar

Avoiding the loop

If the proxy’s own outbound connections were captured, they would be redirected back to the proxy, forever. To prevent this, the rules look at the user ID of the process making the connection. The proxy runs as a special user (ID 1337 in Istio), and connections from that user are left alone. This also explains an odd rule of thumb: never run your own application as that user ID, or its traffic will bypass the mesh.

Envoy is built like an assembly line. A connection enters at one end and passes through stations until it reaches the destination.

flowchart LR
  L["Listener accepts connection"] --> LF["Listener filters - detect protocol"]
  LF --> FC["Filter chain match"]
  FC --> TLS["TLS termination"]
  TLS --> HCM["HTTP connection manager"]
  HCM --> HF["HTTP filters - auth, stats, fault, router"]
  HF --> RT["Route table - match host and path"]
  RT --> CL["Cluster - upstream service"]
  CL --> LB["Load balancer picks endpoint"]
  LB --> UP["Upstream connection pool"]
    

Threading model

Envoy uses a small number of threads and never blocks. One main thread handles configuration updates and housekeeping. Several worker threads handle traffic. When a new connection arrives, it is given to one worker thread, and that worker handles the connection for its whole life. Each worker runs an event loop: it waits for things to happen on many connections and reacts quickly. Because a connection never moves between threads, workers rarely need locks, which is a main reason Envoy is fast. Istio limits the number of workers per sidecar to a small number by default, to avoid wasting CPU in small Pods.

The main building blocks

Protocol awareness

Envoy understands many protocols, including HTTP/1.1, HTTP/2, gRPC, and plain TCP. For HTTP and gRPC, it reads each request, so it can route by path and report per-request metrics. For other protocols, it treats the connection as a stream of bytes and offers connection-level features only. This matters in practice: if a port is mistakenly treated as plain TCP, you lose request-level routing and metrics for that service. Naming Kubernetes service ports by protocol is therefore a small but valuable habit.

Envoy does not read a big file. It subscribes to a live feed of changes.

Services in Kubernetes come and go constantly. Pods start, die, move, and scale. A static configuration file would be out of date in seconds. So Envoy uses xDS, a set of discovery APIs where the “x” stands for a family. The control plane streams updates, and Envoy applies each update live with no restart and no dropped connections.

sequenceDiagram
  participant E as Envoy proxy
  participant P as pilot-agent
  participant I as istiod
  E->>P: Open xDS stream on local socket
  P->>I: Forward stream to istiod over mTLS
  I-->>E: Clusters CDS
  I-->>E: Endpoints EDS
  I-->>E: Listeners LDS
  I-->>E: Routes RDS
  E-->>I: ACK each accepted update
  Note over I: A Service or policy changes
  I-->>E: Push only the changed resources
  E-->>I: ACK or NACK with reason
    

Acknowledge or reject

Each time Envoy receives a configuration, it checks it. If it is valid, Envoy applies it and replies with an acknowledgement (ACK). If it is invalid, Envoy keeps its old working configuration and replies with a rejection (NACK) that carries an error message. This is a safety feature: a bad configuration never replaces a working one. A common debugging step is therefore to ask the control plane whether any proxy is in a stale or rejected state.

Order matters

If routes pointed to a cluster that did not exist yet, requests would fail during the gap. The aggregated stream makes sure that clusters arrive before the routes that use them, and that old resources are removed only after nothing refers to them. This careful ordering is why configuration changes are smooth, even in a busy mesh.

Why config size is a problem

By default, the control plane tells every sidecar about every service in the mesh. In a mesh with thousands of services, each Envoy holds a very large configuration and uses much memory. It also receives a flood of updates for services it will never call. The answer is to limit each proxy’s view to only what it needs, using a scoping resource called Sidecar in Istio. We return to this in the performance chapter.

This is where the proxy earns its keep, by turning an unreliable network into a more dependable one.

Client-side load balancing

The usual Kubernetes Service balances at the connection level inside the kernel, which can leave long-lived connections stuck on one Pod. Envoy balances at the request level, on the client side. It knows every endpoint and can send each request to a different one. Common policies are round robin, random, and least request, which sends the next request to the endpoint with the fewest requests in flight. Recent Istio versions use least request by default. Envoy can also prefer endpoints in the same zone, to save cost and delay, falling back to others when needed.

Retries and timeouts

A retry sends the request again if the first try fails in a way that is probably temporary, such as a refused connection. A timeout sets how long the proxy waits for an answer before giving up. Istio’s defaults include a small number of retries for connection-level failures. Retries are powerful but dangerous: if every layer of a call chain retries three times, one failure can multiply into dozens of calls and overload the struggling service. This is called a retry storm. Good practice is to retry in one place, set retry budgets, and always set timeouts shorter at the edges than in the depths.

Circuit breaking in plain words

An electrical circuit breaker cuts power when current is too high, protecting the house. A software circuit breaker cuts traffic to a service that is failing or overloaded, protecting both the callers and the service. In Envoy, this takes the form of connection and request limits plus outlier detection. When an endpoint keeps failing, it is ejected for some time, with the base time often thirty seconds, and ejection time grows with repeated failures. The result is that clients fail fast and the sick Pod gets room to recover.

Inside a cluster, an IP address means almost nothing, because Pods come and go. Identity must come from something stronger.

Normal TLS, the lock icon in your browser, proves only the server’s identity to the client. In mutual TLS (mTLS), the client also proves its identity with a certificate. Both sides check the other’s certificate before any data is exchanged, and the connection is encrypted. In a mesh, the sidecars do all of this on behalf of the applications, so the application can speak plain traffic to its local proxy.

sequenceDiagram
  participant PA as pilot-agent
  participant I as istiod CA
  participant E as Envoy
  PA->>PA: Create private key and request
  PA->>I: Send CSR with Pod service account token
  I->>I: Validate token and identity
  I-->>PA: Signed certificate chain
  PA-->>E: Deliver certificate through SDS
  Note over PA,E: Renewed before expiry with no restart
    

Identity based policy

Because each sidecar presents a certificate with a clear workload identity, rules can be written in terms of identity instead of IP addresses. For example, “only the checkout workload may call the payment workload.” The receiving Envoy checks the caller’s certificate and applies the policy before the request ever reaches the application. This is far stronger than IP-based firewalls, which break whenever Pods are rescheduled.

Permissive versus strict

When a mesh is first introduced, not every Pod has a sidecar. To avoid breaking traffic from Pods without proxies, Istio’s default mode accepts both encrypted and plain traffic. This is called permissive mode. Once every workload is in the mesh, you should switch to strict mode, where only mTLS traffic is accepted. Staying permissive forever leaves a door open for unencrypted traffic.

Because every request passes through a proxy, the proxy is the perfect place to measure it.

Without a mesh, each team must add measurement code to every service. With a mesh, the sidecar records the same standard facts for every request, whatever the language. The data falls into three groups, often called the three pillars of observability.

The golden signals from the proxy

Operators look at four numbers for every service: how many requests arrive (traffic), how many fail (errors), how long they take (latency), and how busy the service is (saturation). The mesh provides the first three automatically, per service pair, without code changes. This is why a mesh dashboard can show a map of which services call which, with error rates on each line, within minutes of installation.

Reading Envoy response flags

The short flags in access logs are the fastest way to understand a failure. Learn a few by heart.

Two containers in one Pod must start and stop in a sensible order. Kubernetes did not guarantee that for a long time.

stateDiagram-v2
  [*] --> InitRules
  InitRules --> ProxyStarting: Traffic rules installed
  ProxyStarting --> ProxyReady: Config received from istiod
  ProxyReady --> AppRunning: App container starts
  AppRunning --> Draining: Pod deletion begins
  Draining --> [*]: Proxy exits after app finishes
    

Startup race

All normal containers in a Pod start at about the same time. If the application starts faster than Envoy, its first outgoing calls are redirected to a proxy that is not ready yet, and they fail. Applications that call other services right at startup, for example to load configuration, often crash and restart a few times until the proxy catches up. A setting often named hold application until proxy starts makes the platform start the proxy first and wait until it is ready before starting the app.

Shutdown race

When a Pod is deleted, Kubernetes tells all containers to stop at once. If the sidecar exits first, the application can no longer make outgoing calls while finishing its last requests, and in-flight requests may be cut. The proxy therefore has a drain period, during which it stops accepting new connections but keeps serving existing ones. In Istio it defaults to a few seconds, and for slow applications it should be raised, so the proxy outlives the application’s final work.

Jobs and CronJobs

A batch Job’s Pod is finished when its main container exits. But a sidecar container keeps running forever, so the Pod never completes, and the Job never finishes. Teams used to solve this by calling a special endpoint to tell the proxy to quit once the work was done, or by adding wrapper scripts. It was an awkward chore that every team had to learn.

Native sidecar containers

Kubernetes now has a built-in answer. A special kind of init container, marked to restart always, runs as a true sidecar: it starts before the main containers, stays running during the Pod’s life, is stopped after the main containers have finished, and does not block a Job from completing. This feature moved through preview stages and became generally available in Kubernetes 1.33. Recent versions of service meshes can use it for injection, which removes most of the awkward workarounds.

The sidecar is the classic design, but it is not the only one.

CNI plugin instead of the init container

The Container Network Interface (CNI) is the plugin system the kubelet uses to set up Pod networking. A mesh can supply a CNI plugin that installs the traffic redirection rules while the Pod’s network is created. Then the Pod needs no privileged init container. This helps in clusters with strict security policy, because ordinary application namespaces no longer need elevated network permissions.

Ambient mode: no sidecar in the Pod

Sidecars cost memory in every Pod and require restarts to upgrade. Istio’s ambient mode splits the data plane into two layers. First, a small node-level agent called ztunnel, written in Rust, runs once per machine and handles identity and mTLS for all Pods on that machine, using a tunnel protocol on port 15008. Second, optional waypoint proxies, which are Envoy instances shared by a namespace or service, handle the richer request-level features only where needed. Ambient mode reached general availability in Istio 1.24.

Linkerd: a different proxy

Linkerd, also a graduated CNCF project, keeps the sidecar model but uses its own small proxy written in Rust. It aims for simplicity and a low resource cost, with fewer configuration options than Envoy-based meshes. Its injection also uses a webhook and an annotation, and it offers a CNI plugin as well.

Proxyless gRPC

Some gRPC libraries can speak xDS themselves. Then the application gets routing and load balancing directly from the control plane, without a sidecar for those requests. This removes the extra network hop for the most latency-sensitive services, at the price of tying the feature to a library and language.

The mesh adds work to every request. Knowing where the cost lives lets you control it.

Each request passes through two extra proxies, one on each side. Each proxy reads the request, applies filters, and writes it out again. In practice this adds a small amount of delay, often around a millisecond or a few, depending on the features enabled and the load. Many applications never notice it. Latency-sensitive paths, such as calls that happen dozens of times inside one user request, can feel it more, because delays add up along the chain.

Where the time goes

The data plane is on the path of every request, so its failures matter. Here is what happens when each part breaks.

The webhook failure policy decision

When the injection webhook cannot be reached, the API server must decide what to do. If the policy is fail, the Pod creation is rejected, so you never get a Pod without a proxy, but a webhook outage blocks all new Pods in the injected namespaces. If the policy is ignore, Pods are created without a proxy, so deployments continue, but those Pods bypass the mesh silently, which may violate security rules. Both choices have costs, and the right one depends on whether availability or security is more important for the namespace.

The mesh adds security, and it also adds new things that need protecting.

Where the mesh does not protect you

A mesh protects traffic between proxies. It does not protect against a compromised application, a stolen database password, or a vulnerable library. Traffic that bypasses the proxy, for example excluded ports or processes running as the proxy’s user, gets no mesh protection. The loopback link between the application and its own sidecar inside the Pod is also unencrypted, since it never leaves the Pod. And because the sidecar shares the Pod with the application, an attacker who breaks into the application container may be able to reach the sidecar’s local interfaces, so the admin port must be kept local and the Pod hardened.

When a request fails inside a mesh, there are more places to look. A good order saves hours.

Metrics for the data plane and control plane

A calm debugging order

Every design has a price. Here is an honest look at the sidecar mesh.

Do you need a mesh at all?

A mesh is a strong tool, but not a free one. A cluster with a handful of services, one language, and modest security needs may be better served by simple libraries, ingress rules, and network policies. The mesh pays off when there are many services, many teams, strong security or compliance needs, and a need for uniform control. Adopt it for a reason you can state in one sentence.

A few habits keep a mesh helpful. A few others turn it into a burden.

Patterns worth copying

Anti-patterns to avoid

A short checklist to read before you add a mesh to a cluster.

Mistakes beginners make

• Labeling a namespace and expecting running Pods to change. Injection only happens at Pod creation.
• Forgetting that the sidecar counts in resource requests. Cluster capacity shrinks after injection.
• Blaming the mesh for every failure. Many errors are real application errors that the mesh is now able to show you.
• Setting timeouts too short or retries too many. Both cause more trouble than they solve.
• Treating a Job like a long-running service. Without a native sidecar or a quit step, it never finishes.
• Skipping protocol naming on ports. The proxy falls back to plain TCP handling, and metrics disappear.
• Enforcing strict mTLS without a readiness plan. Health probes and old clients start failing.

The same ideas appear in many production environments.

Short answers to the questions beginners ask most often.

You started with an assistant sitting beside every employee and finished with a full picture of how a service mesh data plane works. A webhook slips a proxy into every Pod. Network rules quietly redirect all traffic through it. Envoy processes each request through listeners, filters, routes, and clusters, guided by a live feed of configuration. Short-lived certificates give every workload an identity, so traffic can be encrypted and controlled by who is calling. The price is extra resources and extra moving parts, and the reward is consistent security, reliability, and visibility across every service.