The words “zero trust” are used everywhere, often loosely. Before we touch Kubernetes, we need a precise and simple meaning.

The core idea in one sentence

Zero trust is a security approach in which no user, device, or program is trusted automatically just because of where it is on the network. Every request must be identified, checked against a rule, and allowed only if the rule says yes. The phrase was popularized in the early 2010s by an analyst named John Kindervag, and the United States standards body NIST later published a detailed reference, called Special Publication 800-207, that describes zero-trust architecture in formal terms.

Four questions behind every request

A zero-trust system asks the same four questions each time one program tries to talk to another.

Three guiding principles

To fix a problem, you must first see it clearly. Kubernetes was designed to make networking simple for developers, and that simplicity is exactly what makes it risky.

The Kubernetes networking promises

Kubernetes sets down a few simple rules. Every pod receives its own IP address. Every pod can reach every other pod using that address, without any address translation in between. Nodes can talk to all pods and pods can talk to all nodes. These rules make life easy, because developers never need to think about ports and tunnels. The cost is that, by default, the cluster is one giant open room.

What a flat network allows

In a flat network, a pod running a small public website can connect directly to the payments database in another namespace, because nothing forbids it. The network does not know that the website has no business talking to the database. It only knows that both have IP addresses.

flowchart LR
  I["Internet user"] --> W["Public web pod"]
  W --> A["Orders API pod"]
  A --> D["Payments database pod"]
  W -. "default cluster: also allowed" .-> D
  W -. "default cluster: also allowed" .-> S["Internal admin pod"]
  W -. "default cluster: also allowed" .-> K["Cloud metadata service"]
    

Lateral movement: the real danger

Attackers rarely win by breaking into the most valuable system directly. They usually break into the weakest system first, such as an outdated web component, and then move sideways from machine to machine until they reach what they want. Security people call this lateral movement. A flat network makes sideways travel trivial. Many widely reported breaches, including the 2013 retail breach in which attackers entered through a third-party vendor and then crossed an internal network, followed this pattern.

The attack surface inside a cluster

Zero trust is applied to the network, so you need a clear picture of how traffic actually flows in a cluster. We will build the picture one piece at a time.

Pods and their addresses

A pod is the smallest unit Kubernetes runs: one or more containers that share a network identity. Each pod gets one IP address from a range assigned to its machine. Pods come and go constantly, so their addresses change. This is the first reason zero-trust rules cannot be written around IP addresses alone: the address you trusted a minute ago may now belong to a completely different program.

Services: a stable name in front of moving pods

A Service gives a group of pods one stable name and one stable virtual address. Clients call the Service, and Kubernetes spreads the calls across the healthy pods behind it. A component called kube-proxy, or an equivalent built into the network plugin, performs this spreading by rewriting the destination address of each packet to the address of a real pod.

DNS: names instead of numbers

Cluster DNS translates Service names, such as orders.shop.svc.cluster.local, into virtual addresses. Almost every application call begins with a DNS lookup, so DNS is both essential and a favorite place for attackers to hide. Remember this, because it matters when we write egress rules later.

The network plugin

Kubernetes itself does not move packets between machines. It delegates the job to a CNI plugin, where CNI stands for Container Network Interface. Popular choices include Calico, Cilium, and on Amazon EKS the AWS VPC CNI. The plugin gives out pod addresses and wires up the connections. Very importantly for us, the plugin is also the part that enforces network rules. If your plugin does not enforce rules, the rules you write do nothing.

Ingress and egress

Security people use two simple words. Ingress means traffic coming into a pod. Egress means traffic going out of a pod. They also use compass words: north-south traffic crosses the cluster boundary, such as a customer visiting your website. East-west traffic stays inside, such as the website calling the orders service. Zero trust cares about both, but east-west traffic is the part most teams forget.

flowchart TD
  U["Customer on the internet"] -->|"north-south"| LB["Load balancer"]
  LB --> ING["Ingress or gateway"]
  ING --> WEB["Web pods"]
  WEB -->|"east-west"| API["Orders API pods"]
  API -->|"east-west"| DB["Database pods"]
  API -->|"north-south egress"| EXT["External payment provider"]
  WEB -.-> DNS["Cluster DNS"]
  API -.-> DNS
    

No single feature gives you zero trust. You build it from layers, each answering one of the four questions from the first chapter.

flowchart TD
  L1["Layer 1: Default-deny network rules (NetworkPolicy)"] --> L2["Layer 2: Workload identity (service accounts, SPIFFE)"]
  L2 --> L3["Layer 3: Encrypted traffic (mutual TLS)"]
  L3 --> L4["Layer 4: Request-level authorization (who may call what)"]
  L4 --> L5["Layer 5: Controlled entry and exit (gateways, egress control)"]
  L5 --> L6["Layer 6: Cloud and node boundaries (security groups, metadata protection)"]
  L6 --> L7["Layer 7: Observability and detection (flow logs, audit, alerts)"]
    

Reading the stack

Why layers instead of one big wall

Each layer covers a weakness of the others. Network rules are fast and simple but only understand addresses and ports. Identity and encryption understand who is calling but depend on a healthy certificate system. Request authorization understands individual API calls but needs a proxy in the path. Cloud boundaries protect against mistakes inside the cluster. Observability tells you when any of the other layers has failed. This design philosophy is called defense in depth.

Kubernetes gives you one built-in object for controlling pod traffic. It is called NetworkPolicy, and it is the foundation of everything else in this post.

What a NetworkPolicy is

A NetworkPolicy is a rule that lives inside one namespace. It selects a group of pods using labels, and then states which incoming connections (ingress) and which outgoing connections (egress) those pods may have. The “allowed” side can be described by pod labels, by namespace labels, or by ranges of IP addresses called CIDR blocks, together with port numbers.

The most important behavior to understand

NetworkPolicy has one behavior that surprises almost every beginner. Pods start with everything allowed. The moment any policy selects a pod for a given direction, that pod becomes isolated in that direction, and only what the policies explicitly allow is permitted. Policies can only add permissions. They cannot subtract them. There is no “deny” rule in the standard object. If two policies select the same pod, the pod gets the combined allowances of both.

The three ways to name “who”

The enforcement catch

Plugins that enforce standard network policies include Calico, Cilium, and Antrea. On Amazon EKS, the AWS VPC CNI can enforce them as well when the feature is switched on. Simple plugins that only provide connectivity do not enforce them.

Typical limits of the standard object

The recipe for a zero-trust namespace is short: close everything first, then open only what the application truly needs. This chapter shows the pattern step by step.

Step one: the default-deny policy

A default-deny policy selects every pod in a namespace and lists no allowed connections. The result is that all ingress, and optionally all egress, is blocked for the whole namespace. Because allowances accumulate, you then add small policies that open exactly the paths you want. Most teams create two default-deny policies per namespace: one for incoming traffic and one for outgoing traffic.

Step two: do not forget DNS

This is the most common mistake. If you block all egress, pods can no longer ask cluster DNS to translate names, and every application call breaks with a confusing “name not found” message. The fix is a small policy that allows pods to send DNS queries, on port 53 over both UDP and TCP, to the cluster DNS pods in the system namespace. Add this as part of the default egress setup so that nobody ever has to debug it twice.

Step three: allow each real path

For every legitimate connection, add one small policy named after what it does. A good habit is to write each rule from the point of view of the receiving pod: “the orders API accepts connections from the web pods on port 8080”. Rules that read like plain sentences are easier to review in code reviews.

flowchart LR
  ING["Ingress gateway"] -->|"allow port 8080"| WEB["Web pods"]
  WEB -->|"allow port 9090"| API["Orders API"]
  API -->|"allow port 5432"| DB["Database"]
  API -->|"allow port 443 to payment provider"| EXT["External provider"]
  WEB -. "blocked" .-> DB
  ING -. "blocked" .-> API
    

A policy catalog you can reuse

Namespace labels make policies simple

Kubernetes automatically adds a label to every namespace that holds the namespace name. You can use it in namespace selectors to say “pods from the gateway namespace” without inventing labels yourself. For team-level rules, add your own labels, such as the owning team or the environment, and select on those.

Services versus pods in rules

Network policies are checked against the pods behind a Service, not the Service name. When one pod calls a Service, the packet is rewritten to a real pod address first, and the rule is evaluated against that pod. In practice this means you write rules about pod labels and ports, and the port you allow is the pod’s real container port, which may differ from the Service port. Mixing these up is a classic reason a rule appears correct but still blocks traffic.

Standard policies are a strong start, but real systems soon need more: explicit denies, domain names, and rules about web requests. Two popular network plugins add these powers.

Calico: richer policy and priorities

Calico extends the policy model with its own objects. It supports explicit allow and deny rules, rules ordered by priority, policies that apply to the whole cluster rather than one namespace, and policies that protect the machines themselves. This lets a security team set cluster-wide guardrails that application teams cannot override, such as “no pod may ever reach the cloud metadata address”.

Cilium: identity and eBPF

Cilium takes a different approach. Instead of tracking IP addresses, it gives each group of pods with the same labels a numeric security identity, and writes rules in terms of those identities. It enforces them inside the Linux kernel using a technology called eBPF, which lets small, safe programs run directly in the kernel’s packet path. Because the rules follow identities rather than addresses, they keep working correctly as pods come and go, and the checks are very fast.

What identity-aware plugins add

A standard emerging for cluster-wide rules

The Kubernetes networking community has been developing additional policy objects that let cluster administrators set rules above the namespace level, including baseline defaults and explicit denies. These efforts are still maturing and support differs by plugin, so check what your plugin and version support before relying on them.

Network rules based on addresses and labels are a good start, but labels can be edited and addresses can be spoofed in some situations. True zero trust needs identities that can be proven with cryptography.

The problem with trusting labels

A label is just a piece of text attached to a pod. Anyone with permission to create or edit pods can write any label they like. If your network rules say “allow pods labeled app equals billing”, then an attacker who can create a pod with that label can borrow the permission. Strong zero trust therefore pairs label-based rules with identities that cannot be forged.

Kubernetes service accounts

Every pod runs as a service account, which is Kubernetes’ built-in way to name a workload. Modern clusters issue each pod a short-lived, signed token tied to its service account, delivered through a mechanism called a projected volume. These tokens expire, are bound to a specific audience, and are rotated automatically by the kubelet. They are good proof of identity to the Kubernetes API server and to any system that can verify them.

SPIFFE: a universal name tag

SPIFFE stands for Secure Production Identity Framework for Everyone. It is an open standard that defines a name format for workloads, called a SPIFFE ID, which looks like a web address with the pattern spiffe://trust-domain/ns/namespace/sa/serviceaccount. It also defines how a workload receives a short-lived certificate proving it owns that name. SPIRE is the most well-known program that implements the standard. Companies such as Uber have publicly described adopting SPIFFE and SPIRE for workload identity across their systems.

flowchart TD
  P["Pod starts"] --> A["Local agent checks pod facts: namespace, service account, node"]
  A --> S["Identity server verifies the facts"]
  S --> C["Short-lived certificate issued with the workload's SPIFFE ID"]
  C --> W["Workload presents the certificate when it connects"]
  W --> R["Receiver verifies the certificate chain and the ID"]
    

Why short lifetimes matter

A stolen password works until someone notices and changes it. A stolen certificate that lives for only a day, or even an hour, becomes useless quickly. Zero-trust systems therefore issue certificates with short lifetimes and renew them automatically in the background. Service meshes typically rotate workload certificates about once a day by default, with no human involved.

Cloud identity for pods

Pods also need identities when they call cloud services such as storage buckets and queues. On Amazon EKS you can link a service account to a cloud role, so each pod receives only the permissions that role grants. This is the same least-privilege idea applied to cloud calls, and it prevents a compromised pod from using the machine’s broad permissions.

Network rules decide who may connect. Mutual TLS decides who is really on the other end, and makes sure nobody can eavesdrop or tamper along the way.

Start with ordinary TLS

When you open a secure website, your browser uses TLS, short for Transport Layer Security. The website shows a certificate that proves its identity, and the two sides agree on secret keys to scramble the conversation. In ordinary TLS, only the server proves who it is. The browser stays anonymous.

Now make it mutual

In mutual TLS, usually written mTLS, both sides show certificates. The caller proves who it is, and the receiver proves who it is. After the handshake, the receiver knows with cryptographic certainty which workload is calling, and both know the conversation is private. This is exactly the “verify explicitly” principle applied to service-to-service traffic.

sequenceDiagram
  participant C as Client workload
  participant S as Server workload
  C->>S: Hello and request to connect
  S-->>C: Server certificate with its identity
  C->>C: Verify server certificate against trusted authority
  C->>S: Client certificate with its own identity
  S->>S: Verify client certificate and read caller identity
  C-->>S: Agree on session keys
  C->>S: Encrypted application request
  S-->>C: Encrypted response
    

What mTLS gives you

The problem mTLS creates

Asking every application team to add certificate handling to every program would be slow, error-prone, and different in every language. Certificates must be requested, renewed, and checked correctly everywhere. This is the reason most organizations hand the job to a shared layer. That layer is a service mesh, or increasingly, the network plugin itself.

A service mesh is a layer of infrastructure that handles secure communication for every service, so developers can focus on business logic.

What a mesh actually is

A service mesh places a small network proxy next to each workload. All traffic into and out of the workload passes through its proxy. The proxies together form the data plane, which does the real work of encrypting, checking identity, and applying rules. A separate set of components called the control plane hands out certificates and distributes policy to the proxies.

flowchart LR
  subgraph PodA["Pod A"]
    AppA["App container"] --- PxA["Proxy"]
  end
  subgraph PodB["Pod B"]
    PxB["Proxy"] --- AppB["App container"]
  end
  PxA == "mutual TLS" ==> PxB
  CP["Mesh control plane: certificates and policy"] -.-> PxA
  CP -.-> PxB
    

Meet the popular meshes

Sidecars and the new sidecar-free designs

The original mesh design puts one proxy container, called a sidecar, inside every pod. It works well but uses extra memory and CPU in every pod and adds some startup complexity. Newer designs move the work out of the pod. Istio’s ambient mode uses a small shared agent on each node to handle encryption and identity, and adds optional shared proxies only for workloads that need request-level rules. Cilium uses the kernel itself. These designs reduce overhead, though they are younger, so evaluate their maturity for your version.

Permissive and strict modes

You cannot flip a whole cluster to encrypted-only in one night. Meshes therefore offer a permissive mode, in which a service accepts both encrypted and plain connections, and a strict mode, in which only authenticated mTLS connections are accepted. The usual path is to start permissive, watch which callers are still using plain traffic, fix them, and then switch to strict one namespace at a time.

Once identity is proven, you can write rules far more precise than “this pod may connect to that pod”. You can say which identity may perform which action.

From connections to requests

A network rule says “the web pods may reach the orders API on port 8080”. An authorization rule says “the web workload may read orders, but only the billing workload may create refunds”. The second rule needs a component that understands web requests, namely the mesh proxy or a gateway. It can examine the caller’s identity, the HTTP method, the request path, and even specific headers before deciding.

Example rule ideas, in words

End users, not only services

Zero trust also covers the human on the other side of the request. A common pattern is for the user to log in through an identity provider and receive a signed token, often a JSON Web Token. The mesh or gateway can validate the token’s signature, check its expiry and audience, and pass the verified claims to the application. Combined with workload identity, the service sees both which service is calling and which person it is acting for.

The edge of the cluster is where the outside world meets your workloads. Zero trust treats this boundary with extra care, but it does not stop there.

Ingress and the Gateway API

An ingress controller or a gateway receives traffic from outside and routes it to the right Service. Kubernetes has two ways to describe such routes. The older Ingress object is simple and widely supported. The newer Gateway API is more expressive and separates responsibilities: infrastructure teams own the gateway itself, while application teams own the routes attached to it. This separation fits zero trust well, because it limits who can expose what.

What the edge should do

Do not let the gateway become a trust shortcut

A common mistake is to treat everything arriving from the gateway as automatically trusted. In a zero-trust design, the gateway is just another workload with its own identity. The backend still authenticates it, still checks the user’s token, and still applies request-level rules. The gateway is allowed to talk only to the services it truly fronts, not to the entire cluster.

Cloud load balancers and the layers behind them

On a cloud such as AWS, a managed load balancer usually sits in front of the gateway. It adds its own protections: network access rules, optional web application firewall, and managed certificates. These cloud features complement the in-cluster rules and form the outer layers of the stack.

flowchart LR
  USER["Customer"] --> WAF["Cloud firewall and load balancer"]
  WAF --> GW["Gateway with its own identity"]
  GW -->|"mTLS and token check"| SVC["Backend service"]
  SVC -->|"mTLS and identity rule"| DATA["Data service"]
    

Most teams worry about attackers getting in. Zero trust also worries about what gets out. Outgoing traffic is how stolen data leaves, and how a captured pod calls home to its master.

Why egress is often wide open

Developers want their pods to download packages, call partner APIs, and reach cloud services. Locking this down feels like friction, so most clusters allow all outgoing traffic. That means an attacker who compromises one pod can download hacking tools, contact a remote control server, and upload your customer data to any address on the internet, with no resistance.

The layers of egress control

The egress gateway pattern

An egress gateway is a dedicated workload through which all traffic to the outside must pass. It gives you one place to log, inspect, and allow destinations. It also lets your partners allow-list a single stable address for your whole cluster. Applications cannot reach the internet except through it, because network policies block every other route.

flowchart LR
  APP["Application pods"] -->|"allowed"| EG["Egress gateway"]
  APP -. "direct internet blocked" .-> NET["Internet"]
  EG -->|"approved destinations only"| PART["Partner API"]
  EG --> LOG["Central logs and alerts"]
    

DNS as a hidden exit

Even when ordinary connections are blocked, attackers have been known to hide stolen data inside DNS queries, because DNS is almost always allowed. To limit this, make sure pods use only the cluster DNS service rather than public resolvers, and monitor for unusual volumes of strange-looking name lookups.

The most valuable targets in a cluster are not your applications. They are the Kubernetes control plane and the machines running your pods. Zero-trust networking must protect both.

The API server

The API server is the front door to the entire cluster. Anyone who can talk to it with enough permissions can create pods, read secrets, and change rules. Zero trust applies in three ways. First, limit who can reach it at the network level: keep the endpoint private where possible, or restrict it to approved address ranges. Second, require strong authentication for every caller. Third, use fine-grained role-based access control so that each person and program has only the permissions it needs.

The kubelet

Every node runs an agent called the kubelet, which has its own network port. If that port accepts unauthenticated requests, an attacker who reaches it can run commands in pods. Modern, hardened clusters turn off anonymous access to the kubelet and require the API server to authenticate to it. Managed services configure this securely, but self-managed clusters need a deliberate check.

The data store

Cluster state, including secrets, lives in a data store called etcd. It should accept connections only from the API server, require mutual TLS, and encrypt sensitive data at rest. On managed services the provider handles this for you.

Pods that call the API server

Some pods, such as controllers and operators, must talk to the API server. Treat them as high-value targets. Give each its own service account with narrow permissions, and use network policies to confirm that only those pods can reach the API server address at all. Most ordinary applications never need to call it, so blocking them closes a major path.

Everything cryptographic in zero trust rests on a chain of trust that begins with a certificate authority. If that foundation is shaky, the whole structure is.

What a certificate authority does

A certificate authority, or CA, is the trusted party that signs certificates. When a workload presents a certificate, the receiver checks whether it was signed by a CA that it trusts. If yes, the identity inside is believed. If the CA is compromised, an attacker can mint a certificate for any identity, so protecting the CA is the single most important job in the system.

A sensible hierarchy

flowchart TD
  ROOT["Root CA: kept offline and heavily protected"] --> INT["Intermediate CA: used by the cluster"]
  INT --> W1["Workload certificate: orders service"]
  INT --> W2["Workload certificate: billing service"]
  INT --> W3["Workload certificate: gateway"]
    

Good practice keeps the root authority offline and uses an intermediate authority for day-to-day signing. If the intermediate is ever compromised, you replace it without having to rebuild trust from scratch. Meshes and identity systems can be set up to use an intermediate issued by your organization’s own root, which lets workload identities connect cleanly to the rest of your company’s trust system.

Automating certificates

Certificates for human-facing websites are usually handled by a Kubernetes add-on called cert-manager, which requests, renews, and installs certificates automatically from public or private authorities. Certificates for workload-to-workload mTLS are typically handled by the mesh or identity system itself. The goal in both cases is the same: no human ever copies a certificate by hand, and nothing expires unnoticed.

Secrets are part of the network story

Database passwords, API keys, and tokens are credentials that grant network access. Keep them out of container images and plain configuration files. Use a dedicated secret store with access control and audit logging, and prefer short-lived credentials over long-lived ones. Where possible, replace passwords with identity-based access so there is nothing to steal at all.

Your cluster sits inside a cloud network, and the cloud has its own firewalls and identities. The strongest designs make the cloud layer and the Kubernetes layer agree with each other.

Network policy on the VPC CNI

The AWS VPC CNI assigns pods addresses from your virtual network, and recent versions can enforce standard Kubernetes network policies directly. This means the same policy objects you write for any cluster can control pod traffic on EKS once you enable the feature. As always, confirm enforcement with a real test, and check which policy features your version supports.

Security groups for pods

AWS lets you attach a cloud security group to individual pods on supported machine types. This gives a pod the same kind of firewall rule as any other cloud resource. A typical use is letting only the orders pods reach the database service, with the database security group itself referring to the pod security group. It provides a second, independent barrier outside Kubernetes, which is valuable if someone misconfigures a Kubernetes policy.

Blocking the metadata service

Every cloud machine has a special internal address, called the instance metadata service, that hands out information about the machine, including temporary credentials for the machine’s role. If a pod can reach it, an attacker who takes over the pod may steal those credentials and gain the machine’s broad cloud permissions. On AWS, require the session-based version of the service, known as IMDSv2, and set its hop limit to one so that pod traffic cannot reach it. Also use per-pod roles so pods never need the machine’s role at all.

Pod-level cloud identities

Private connections to cloud services

Instead of sending traffic to cloud services over the public internet, use private endpoints that keep traffic on the provider’s internal network. Combine them with endpoint policies that state which buckets, tables, or queues may be reached from your network. This turns “can this pod reach the service” into a decision made in two places: Kubernetes and the cloud.

Choosing a mesh on AWS

AWS has announced the retirement of its older managed mesh product, App Mesh, so new designs generally choose an open-source mesh such as Istio or Linkerd, a network-plugin approach such as Cilium, or a service-networking offering from AWS called VPC Lattice. Verify the current status of any managed product before committing to it.

A rule that you cannot test or observe is only a hope. Zero trust includes continuous watching, so that failures and attacks become visible quickly.

Three sources of truth

What to look for

Discovery before enforcement

Observability has a second, practical use: it tells you what rules to write. Run in an observing mode for a few weeks, collect the flows that really occur, and turn them into the first draft of your policies. Tools exist that read flow data and propose network policies automatically. Always review their suggestions by hand, since they will happily approve a connection that was itself a mistake.

The biggest risk of zero trust is not the attacker. It is the well-meaning engineer who turns on strict rules on a Friday afternoon and takes down checkout. A careful rollout plan avoids that.

The phased path

Manage policy as code

Treat every network policy, mesh rule, and authorization policy as source code. Store it in version control, review it like any other change, test it, and deploy it through an automated pipeline. This approach is often called GitOps. It gives you history, rollback, and a clear answer to the question “who allowed this and why”.

Test the rules, not just the application

Add automated checks that prove the policies work. A good test suite includes both positive checks, confirming that required connections succeed, and negative checks, confirming that forbidden connections fail. Run them after every policy change and on a schedule, because drift and plugin upgrades can quietly change behavior.

Let us apply everything to one imaginary application, step by step, and watch the open cluster turn into a zero-trust one.

The application

An online store runs in a cluster with five parts. A web service serves the shop pages. An orders service manages carts and orders. A payments service charges cards through an outside provider. A catalog service holds product data. A database stores orders. An ingress gateway receives all customer traffic, and a monitoring system scrapes metrics from every service.

Step one: draw the communication map

Step two: close everything

Create a default-deny policy for both ingress and egress in the store namespace. Add the DNS allowance so names still resolve. At this moment nothing works, which is exactly the safe starting point.

Step three: open each mapped path

Add one small policy for each row of the table. For example, the orders service accepts connections only from the web workload, and the database accepts connections only from orders. The web service has no path to the database or payments at all, so even a fully compromised web pod cannot reach them.

Step four: add identity and encryption

Enable the mesh in the namespace and move from permissive to strict mode. Now every connection in the table is encrypted and authenticated, and each service knows the verified identity of its caller.

Step five: add request-level rules

On the payments service, allow only the orders identity to call the charge path, and nobody to call administrative paths. On the orders service, allow the web identity to read and create orders, but not delete them.

Step six: control the exit

Allow the payments pods to reach only the payment provider’s name through the egress gateway. Every other pod in the namespace has no route to the internet, apart from DNS.

flowchart TD
  CUST["Customers"] --> GW["Gateway"]
  GW -->|"mTLS, port 8080"| WEB["Web"]
  WEB -->|"mTLS, read and create only"| ORD["Orders"]
  WEB -->|"mTLS, read only"| CAT["Catalog"]
  ORD -->|"mTLS, charge path only"| PAY["Payments"]
  ORD -->|"mTLS, port 5432"| DB["Database"]
  PAY -->|"via egress gateway"| EXT["Payment provider"]
  MON["Monitoring"] -.->|"metrics port only"| WEB
  MON -.-> ORD
  MON -.-> PAY
    

The result in numbers

Theory feels abstract until you watch a realistic attack play out twice: once against an open cluster, once against a zero-trust one.

The attack begins

An attacker finds a flaw in a library used by the web service and gains the ability to run commands inside one web pod. This is the starting point in both stories.

Story one: the flat cluster

Story two: the zero-trust cluster

Side-by-side comparison

Zero trust is powerful, but it is not free. Understanding the costs lets you spend wisely and answer honestly when someone asks “is it worth it”.

Performance overhead

Encrypting every connection and checking identity takes a little time. Modern processors perform encryption very quickly, and long-lived connections pay the handshake cost only once. In practice, the added delay per call is usually small compared to the time applications spend doing their own work, but the effect grows with the number of hops in a request. A request that passes through ten services pays the overhead ten times. Measure with your own traffic before and after, rather than trusting a published number.

Resource overhead

Sidecar proxies consume memory and CPU in every pod. With hundreds of pods, this adds up to real money. Sidecar-free designs lower this cost, as do shared agents per node. Include proxy resources when you size pods, and remember that a proxy without enough CPU becomes a source of latency.

Operational complexity

A mesh and an identity system are two more large systems to upgrade, monitor, and debug. When a call fails, the cause might now be an application bug, a network policy, a certificate problem, or a mesh rule. Teams need good tooling and training to find it fast. Without them, engineers lose trust in the platform and ask for exceptions.

What happens when the control plane fails?

A key design question is how the data plane behaves if the mesh control plane or certificate system becomes unavailable. A well-designed system keeps existing connections and cached policies working for a while, and certificate lifetimes are long enough to ride out brief outages. But if the outage outlasts those lifetimes, new certificates cannot be issued and connections start to fail. Plan for this by running the control plane in several zones, by testing failure drills, and by choosing certificate lifetimes that balance security against survivability.

Is it worth it?

For a small internal tool with no sensitive data, perhaps a lighter setup is enough: default-deny policies and a locked-down egress. For systems that handle money, personal data, or regulated information, the answer is almost always yes, because the cost of one serious breach far exceeds the cost of running these controls. A sensible compromise is to apply the full stack to the most sensitive namespaces first and keep the baseline everywhere.

Most zero-trust examples show web requests. Real systems also move data through databases, message queues, caches, and custom protocols. These need slightly different handling.

Not everything speaks HTTP

A web request has a method and a path, which makes request-level rules easy to write. A database connection or a message queue speaks its own language. A mesh proxy can still encrypt and authenticate these connections as plain streams of data, which is called opaque TCP handling, but it cannot read inside them to say “this user may read this table”. For that kind of rule, you rely on the data service’s own permissions.

Two layers of protection for data stores

Treat every database and queue as protected twice. The first layer is the network and identity layer: only the approved workloads can even open a connection, and the connection is encrypted. The second layer is the data service’s own login and permissions. If either layer is misconfigured, the other still holds. Never rely on “only internal apps can reach it” as the only protection for a data store.

Stateful applications and stable names

Databases and brokers often run as StatefulSets, where each pod has a stable name and its own storage. Pods inside such a cluster talk to each other directly, for example to copy data between replicas. Zero-trust rules must therefore allow replica-to-replica traffic on the replication ports, and block everything else. Write these rules by label, so that a new replica automatically gains the same access.

Protocol quirks to watch

Large organizations rarely run just one cluster. They run several for different regions, teams, and environments. Zero trust must work across all of them without becoming a tangle.

Why multiple clusters exist

Teams create several clusters to place workloads close to customers, to survive the loss of an entire region, to separate production from testing, or to give sensitive workloads their own boundary. Each cluster has its own control plane, its own network, and its own certificate authority unless you design otherwise.

Two ways to build trust across clusters

flowchart TD
  ROOT["Organization root CA, kept offline"] --> ICA["Intermediate CA for cluster A"]
  ROOT --> ICB["Intermediate CA for cluster B"]
  ICA --> WA["Workloads in cluster A"]
  ICB --> WB["Workloads in cluster B"]
  WA |"mTLS through gateways"| WB
    

Connecting the networks safely

Clusters can be connected in several ways. Some meshes provide gateways that carry encrypted traffic between clusters over the internet or private links. Some network plugins offer a built-in multi-cluster feature that gives workloads in different clusters a shared view of identities. Whichever you choose, the same rules apply: the link must be encrypted, each side must authenticate the other, and the policies must still describe which workload may call which, even across the cluster boundary.

Hybrid and on-premises systems

Many companies still run older systems outside Kubernetes, such as legacy databases or mainframes. Zero trust treats these exactly like any other workload: they get identities where possible, their connections are limited to what is needed, and the traffic between them and the cluster is encrypted and logged. Where the older system cannot do mutual TLS, place a small proxy in front of it that does the work on its behalf.

Security that you cannot demonstrate is hard to defend. Zero trust gives you both stronger protection and clearer evidence for the people who check your work.

What auditors usually ask

Whatever the regulation, auditors tend to ask the same few questions. Is sensitive data separated from everything else? Is data encrypted while traveling across the network? Who can access the system, and is each access recorded? Can you prove the rules are the ones you say they are? Zero-trust controls map directly onto these questions.

Standards and benchmarks that help

Continuous compliance

Instead of preparing for audits once a year, automate the checks. Run scanners that compare your live cluster to your benchmark every day. Fail deployments that would create a namespace without default-deny rules, using admission controls that reject non-compliant objects. Keep logs for as long as your regulations require. When the auditor arrives, you simply hand over the reports that already exist.

flowchart LR
  GIT["Policy in version control"] --> CI["Automated tests"]
  CI --> ADM["Admission control blocks non-compliant objects"]
  ADM --> LIVE["Live cluster"]
  LIVE --> SCAN["Daily compliance scans"]
  SCAN --> REP["Reports and alerts for auditors and teams"]
    

Teams adopting zero trust tend to stumble in the same places. Learn these ahead of time and save months of rework.

These questions come up again and again when teams begin their zero-trust journey.