The classic tools worked well until clusters became large, busy and constantly changing.

Kubernetes makes a bold promise about networking. Every Pod gets its own IP address, every Pod can talk to every other Pod without clumsy address translation, and Services give groups of Pods a stable name and address that survives restarts. Behind that friendly promise, something has to actually move the packets, balance the traffic across healthy Pods and enforce the rules about who may talk to whom.

For many years, the answer relied heavily on a Linux feature called netfilter and its command-line companion iptables. A component named kube-proxy watched the cluster and wrote iptables rules so that traffic sent to a Service address was redirected to one of the Pods behind it. Network policies were turned into more rules. This approach is simple and battle tested, but it was never designed for a world where thousands of Services and tens of thousands of Pods appear and disappear every hour.

A quick tour of the pieces that any networking add-on has to provide.

Before diving into Cilium, it helps to be clear about what Kubernetes itself expects from the network. The rules are short and elegant. Every Pod receives a unique IP address. Pods on the same machine and on different machines can reach each other directly using those addresses. Agents on a node can reach all the Pods on that node. Kubernetes does not build this network itself. It delegates the job to a plugin that follows a standard called the Container Network Interface, usually shortened to CNI.

The jobs of a network plugin

When the kubelet starts a Pod, it asks the container runtime to create an isolated network space for it. The runtime then calls the CNI plugin, which has three tasks. It gives the Pod an address, it connects the Pod’s network space to the rest of the node and cluster, and it removes everything again when the Pod is deleted. Cilium is a CNI plugin, but as you will see, it does far more than these three basics.

Services and who does the load balancing

Pods are temporary and their addresses change. A Service solves this by offering one stable virtual address that fronts a changing set of Pods. Something on every node must translate that virtual address into the address of a real, healthy Pod. In most clusters that something is kube-proxy. In a Cilium cluster, it can be Cilium itself, using eBPF, which is why you will often hear the phrase kube-proxy replacement.

Network policy

By default, every Pod can talk to every other Pod, which is convenient and dangerous. Kubernetes offers an object called NetworkPolicy so that teams can say which Pods may talk to which. The API only defines the rules. A plugin must actually enforce them, and not every plugin does. Cilium enforces them, and offers richer rules of its own.

A safe way to teach the Linux kernel new tricks without rebuilding it.

The kernel is the core of the operating system. It manages memory, schedules programs and moves every network packet. For decades, adding new behaviour to the kernel meant either changing its source code, which takes years to reach users, or loading a kernel module, which is powerful but risky, because a bug in a module can crash the entire machine.

eBPF offers a third path. The name once stood for extended Berkeley Packet Filter, though today it is simply used as a name. It lets you write small programs that run inside the kernel in a sandbox. Before a program is allowed to run, the kernel inspects it with a component called the verifier. The verifier proves the program cannot loop forever, cannot read memory it should not touch and cannot crash the system. Only programs that pass are accepted, then translated into fast machine code and attached to a hook, which is a specific moment in the kernel’s life when the program should run, such as the instant a packet arrives.

flowchart LR
    A["Developer writes a small program"] --> B["Compiled to eBPF bytecode"]
    B --> C["Kernel verifier checks safety"]
    C -->|Rejected| D["Program is refused"]
    C -->|Accepted| E["Translated to machine code"]
    E --> F["Attached to a kernel hook"]
    F --> G["Runs on every matching event"]
    G  H["Shared eBPF maps hold state"]
        

Three ideas to hold on to

First, eBPF programs are event driven. They do nothing until something happens, such as a packet arriving or a connection starting. Second, they are small and fast, because they run in the kernel without switching back and forth to ordinary programs. Third, they can remember things through eBPF maps, which are tables shared between the kernel programs and the ordinary programs that control them.

Where the little programs attach, and where they keep their notes.

The power of eBPF for networking comes from the number of places where a program can be attached. Each place sees traffic at a slightly different moment, and Cilium picks the best one for each job.

Why maps matter so much

Maps are the memory of the datapath. When Cilium learns that a Pod belongs to a certain identity, it writes that fact into a map, and the kernel programs read it at full speed. When a Service gains a new backend Pod, the agent updates one entry in a map, instead of rewriting a long list of rules. This is a big part of why eBPF scales so smoothly: small updates to hash tables replace large rewrites of ordered lists.

An open source project that turns eBPF into a complete Kubernetes networking platform.

Cilium is an open source project, created originally by the company Isovalent and now developed by a wide community. It is a graduated project of the Cloud Native Computing Foundation, which is the foundation’s highest maturity level and signals broad adoption and healthy governance. Several managed Kubernetes offerings from cloud providers build on Cilium for their networking layer, which is one reason it has become so widely used.

At its core, Cilium is a CNI plugin that uses eBPF for its datapath, which means the path that packets take. Around that core sits a growing family of capabilities, all built on the same foundation. It is helpful to see these capabilities as layers that you can switch on as needed.

Features come from one foundation

A subtle benefit of this design is consistency. Because routing, policy and observation all use the same identity information and the same kernel programs, they agree with each other. The flow that Hubble displays is the same flow the policy engine judged. In a stitched-together setup of separate tools, those views can disagree, which makes troubleshooting painful.

The programs, objects and tables that make up a Cilium installation.

A Cilium installation has a small number of moving parts. Most of the real work happens in one agent that runs on every node. Understanding what each part does makes it much easier to troubleshoot later.

flowchart TD
    K["Kubernetes API server"] --> OP["Cilium operator: cluster-wide tasks"]
    K --> AG1["Cilium agent on node 1"]
    K --> AG2["Cilium agent on node 2"]
    AG1 --> BPF1["eBPF programs and maps on node 1"]
    AG2 --> BPF2["eBPF programs and maps on node 2"]
    BPF1 --> HU1["Hubble observer on node 1"]
    BPF2 --> HU2["Hubble observer on node 2"]
    HU1 --> HR["Hubble Relay"]
    HU2 --> HR
    HR --> UI["Hubble UI and command line"]
    AG1 --> EN1["Envoy proxy for layer seven"]
        

Custom objects that describe the state

Cilium stores much of its knowledge as Kubernetes custom resources. There are objects for security identities, for endpoints, for node information and for the richer policies that Cilium adds. Because they live in the Kubernetes API, they can be listed, inspected and managed with the same tools as everything else, and they fit naturally into configuration stored in version control.

The idea that makes Cilium policy stable even when addresses keep changing.

The most important concept in Cilium is that security is based on identity, not address. Traditional firewalls say “allow traffic from address 10.0.4.17 to address 10.0.9.3.” In a cluster, that is almost meaningless, because the Pod at 10.0.4.17 may be replaced a minute from now by a different Pod, and the address could be handed to something unrelated.

Cilium instead looks at the labels attached to a Pod. Labels are the tags that Kubernetes already uses to organise things, such as which application a Pod belongs to and which environment it runs in. Cilium takes the set of relevant labels, gives that set a number called a security identity, and assigns that identity to every Pod that carries the same labels. All the replicas of the same application share one identity, no matter how many exist or where they run.

sequenceDiagram
    participant P as New Pod
    participant A as Cilium agent
    participant K as Kubernetes API
    participant M as eBPF maps
    P->>A: Pod starts, networking requested
    A->>K: Read the Pod's labels
    A->>A: Work out the identity for this label set
    A->>K: Create or find the matching identity object
    A->>M: Record this Pod address with its identity
    Note over M: Every node learns the address to identity mapping
        

How identity travels with traffic

Every node keeps a table, often called the IP cache, that maps Pod addresses to identities. When a packet arrives, the datapath looks up the sender’s address and learns its identity almost instantly. When traffic travels between nodes through an overlay tunnel, the sender’s identity can even be carried inside the tunnel header, so the destination does not need to look it up. Policy is then checked using identities: is the sender’s identity allowed to reach the receiver?

Why this scales

A policy that says “the frontend identity may reach the payments identity on one port” is just one small rule, however many Pods exist. Adding fifty replicas adds no new rules, because they share the identity. The number of identities depends on the number of distinct label combinations, not the number of Pods, which keeps policy tables compact.

Following one connection from a Pod on one node to a Pod on another.

The best way to understand a datapath is to follow a packet. Suppose a frontend Pod wants to talk to a backend Pod. Here is the story in the order it happens.

sequenceDiagram
    participant F as Frontend Pod
    participant N1 as Node 1 eBPF
    participant N2 as Node 2 eBPF
    participant B as Backend Pod
    F->>N1: Packet leaves the Pod
    N1->>N1: Identity lookup, policy check, connection tracking
    N1->>N2: Forward across the network
    N2->>N2: Learn sender identity, apply receiving policy
    N2->>B: Deliver packet
    B-->>N2: Reply
    N2-->>N1: Reply travels back using the tracked connection
    N1-->>F: Reply delivered
        

How Pods get addresses and how their traffic crosses the physical network.

Where Pod addresses come from

Cilium offers several ways to hand out Pod addresses, called IP address management modes. The default gives each node a slice of a cluster-wide range, and the operator manages the slices. Other modes follow the Kubernetes node settings, or integrate with a cloud provider’s own network interfaces, so that Pods receive real addresses from the cloud network. Choosing a mode is mainly a question of how much you want the Pod network to blend with the surrounding network.

Two ways to cross the network

Pods on different nodes must reach each other over the physical network. Cilium supports two broad styles. In encapsulation mode, also called overlay, each packet is wrapped inside another packet that carries node addresses, using a tunnel format such as VXLAN or Geneve. The physical network only needs to know how to deliver traffic between nodes and never has to understand Pod addresses. In native routing mode, Pod packets travel unwrapped, and the surrounding network, or a routing protocol, must know how to reach every Pod range.

Replacing long rule lists with fast table lookups.

Services are where eBPF delivers some of its most visible benefits. In the classic design, kube-proxy turns each Service into a set of iptables rules. A packet sent to a Service address walks down a chain of rules until one matches, and then another set of rules picks a backend. With thousands of Services, this chain is long, and every change to a Service means rewriting rules.

Cilium replaces this with eBPF maps. One map holds the Services and their backends, and a lookup finds the answer in roughly constant time, however many Services exist. When a backend appears or disappears, the agent changes a single entry. The behaviour is the same from the application’s point of view, but the cost of getting there is much lower and much more predictable.

flowchart LR
    subgraph OLD["Classic kube-proxy with iptables"]
        direction TB
        A1["Packet to Service address"] --> A2["Walk rule 1"]
        A2 --> A3["Walk rule 2"]
        A3 --> A4["...thousands of rules..."]
        A4 --> A5["Match found, pick a backend"]
    end
    subgraph NEW["Cilium with eBPF maps"]
        direction TB
        B1["Connection or packet to Service address"] --> B2["One hash lookup"]
        B2 --> B3["Backend chosen"]
    end
        

Socket-level load balancing

For traffic between Pods inside the cluster, Cilium can pick the backend at the moment the application connects. The program attached to the socket rewrites the destination from the Service address to a real Pod address before the first packet is created. After that, packets flow straight to the chosen Pod with no per-packet address translation. The application never knows, and no extra translation work is done for the life of the connection.

External traffic and NodePorts

Traffic that arrives from outside the cluster, for example through a NodePort or a load balancer, is handled in the packet path. Cilium selects a backend, and may use a technique called direct server return, in which the reply travels straight from the backend to the client instead of passing back through the node that first received the request. It can also use consistent hashing, so that a change in the set of backends moves as few existing connections as possible.

Controlling who may talk to whom, using identity instead of addresses.

Cilium enforces the standard Kubernetes NetworkPolicy and adds its own richer policy objects. A policy chooses a group of Pods by labels and then lists what traffic is allowed in or out. Those rules can refer to other groups of Pods by their labels, to whole namespaces, to address ranges outside the cluster, or to special named entities such as the nodes themselves, the Kubernetes API server or the outside world.

How the default changes

Without any policy, Pods accept all traffic. The moment a policy selects a Pod for a given direction, that Pod switches to deny by default for that direction, and only the traffic that some policy explicitly allows is let through. This is a common source of surprise: adding a first small policy can suddenly block traffic that used to work, because the Pod has moved from open to closed. Planning the policy as a whole, rather than one rule at a time, avoids outages.

flowchart TD
    A["Packet arrives with sender identity"] --> B{"Does any policy select the receiving Pod for this direction?"}
    B -->|No| C["Allow"]
    B -->|Yes| D{"Is there an explicit deny rule that matches?"}
    D -->|Yes| E["Drop and record the reason"]
    D -->|No| F{"Is there an allow rule that matches identity, port and protocol?"}
    F -->|Yes| G["Allow"]
    F -->|No| E
        

Going beyond addresses and ports to understand what applications are actually saying.

Layers three and four describe where traffic goes and on which port. Layer seven describes what the application is asking for. Two Pods may both talk to the same service on the same port, yet one only reads data while the other tries to delete it. A policy that cannot tell these apart is too blunt. Cilium can enforce rules that understand protocols such as HTTP, allowing a particular method and path while blocking others.

How layer seven rules are applied

Understanding application protocols requires parsing the traffic, which is more work than checking an identity and a port. When a policy contains such rules, Cilium steers the matching traffic through an Envoy proxy that runs on the node. The proxy examines each request, applies the rules and passes allowed requests along. Traffic that needs no such inspection never touches the proxy and stays on the fast eBPF path.

DNS-aware policy

Many workloads must reach outside services by name, such as a payment provider or a package repository. The addresses behind those names change often, so writing the addresses into policy is hopeless. Cilium offers rules based on domain names. A DNS proxy watches the lookups that Pods make, notes which addresses were returned for allowed names, and permits traffic to just those addresses for a limited time. Pods can therefore be allowed to reach one named service and nothing else on the internet.

Protecting traffic in transit and proving who is on the other end.

Inside a cluster, traffic between Pods normally travels in plain form across the node network. Anyone who can watch that network, such as a compromised machine or a careless tap, could read it. Cilium offers transparent encryption between nodes, meaning applications need no changes at all. The nodes simply encrypt traffic as it leaves one and decrypt it as it arrives at the other.

Proving identity, not just hiding traffic

Encryption keeps eavesdroppers out, but it does not prove that the other end is who it claims to be. Cilium has been developing mutual authentication, in which workloads prove their identity to each other using standard workload identity documents before traffic is allowed. This links the idea of Cilium identity to cryptographic proof. It is an area that has been evolving quickly, so check the current status and limits in the documentation before relying on it for critical needs.

Turning the datapath into a window that engineers can look through.

One of the most loved parts of Cilium is Hubble, its observability layer. Because eBPF programs already look at every packet to make decisions, they can also report what they saw at almost no extra cost. Hubble collects those reports, enriches them with Kubernetes information such as Pod names, namespaces and labels, and presents them in a way people can understand.

flowchart LR
    A["eBPF programs see each flow"] --> B["Event buffer on the node"]
    B --> C["Hubble observer adds Pod and label names"]
    C --> D["Hubble Relay combines all nodes"]
    D --> E["Command line tool"]
    D --> F["Web interface with service map"]
    C --> G["Metrics for Prometheus"]
        

What you can see

For every flow, Hubble can show who talked to whom, using names rather than raw addresses, on which port and protocol, and what the verdict was. A verdict says whether the traffic was forwarded or dropped, and a dropped flow comes with a reason, such as being denied by policy. For supported protocols, Hubble also shows the application level details, like the HTTP method and response status, and the DNS names that were looked up.

A troubleshooting story

A team sees that a new service cannot reach its database. Without good tools, they would read policies, compare labels and guess. With Hubble, they filter flows from the service to the database and immediately see a dropped verdict with the reason “denied by policy.” The missing rule is obvious. What might have taken an afternoon takes a few minutes. This is the practical value of observability that is built into the datapath.

Getting mesh-like features without adding a helper container to every Pod.

A service mesh adds features such as smart traffic routing, retries, secure service-to-service connections and detailed metrics. The traditional way to provide them is to inject a sidecar proxy into every Pod. That works, but it multiplies the number of containers, adds memory and processor cost to each Pod, and complicates upgrades because every Pod must be restarted to pick up a new proxy.

Cilium offers a different approach. Because eBPF already steers traffic in the kernel, it can send only the traffic that needs layer seven treatment to a shared proxy on the node, instead of to a proxy in every Pod. The same node-level proxy also serves ingress traffic entering the cluster, and Cilium implements both the older Ingress API and the newer Gateway API, which is the modern Kubernetes standard for describing how external traffic reaches Services.

Joining several clusters so they behave like parts of one network.

Real organisations rarely run a single cluster. They run several, for different regions, teams, environments or failure domains. Out of the box, each cluster is an island, with its own Services and its own policies. Cilium Cluster Mesh connects clusters so that Pods in one can reach Pods and Services in another directly, with identity-based policy that works across them.

flowchart LR
    subgraph CA["Cluster A"]
        A1["Frontend Pods"]
        A2["Mesh API server"]
    end
    subgraph CB["Cluster B"]
        B1["Backend Pods"]
        B2["Mesh API server"]
    end
    A2  B2
    A1 -->|"Direct Pod to Pod traffic"| B1
        

Global Services

The headline feature is the global Service. A Service with the same name and namespace in several clusters can be marked as global, and Cilium load balances across the backends in all of them. If the backends in one cluster fail, traffic flows to another cluster automatically. Teams can also prefer local backends and use remote ones only as a fallback, which keeps traffic close and costs low during normal times.

A tour of capabilities that solve specific, practical problems.

When the extras pay off

On bare metal and in private data centres, the combination of BGP and load balancer address pools replaces separate load balancer products for many teams. For regulated industries, a predictable egress address and a host firewall address common audit requirements. For shared clusters with many tenants, bandwidth limits keep neighbours fair. Each feature is optional, and each deserves a short experiment before it goes into production.

Where eBPF helps most, and where you still need to do your homework.

eBPF networking is often described as faster than the older approach, and in many situations that is true. Understanding why helps you predict when it will matter for you and when it will not.

The places to be careful

Maps have fixed sizes. If the connection tracking table fills up, new connections may be dropped, and if the policy table is too small for the number of rules, some policies cannot be loaded. Cilium exposes how full the maps are, and operators should size them for peak load and alert well before they fill. Kernel version also matters: newer kernels unlock faster paths and more features, while older ones force fallbacks. Finally, layer seven inspection and encryption cost extra processing, and should be measured rather than assumed to be free.

A powerful component deserves careful protection.

Cilium sits in a very privileged position. It loads programs into the kernel and controls the network of every Pod. That power is what makes it useful, and it is also why the security of Cilium itself deserves attention.

Policy as code

Because policies are ordinary Kubernetes objects, teams can keep them in version control, review changes like code, test them in non-production clusters and roll them out with the same pipelines as applications. This turns network security from a pile of hand-edited rules into a reviewed and auditable history. A common pattern is a default-deny policy for each namespace, with explicit allow rules owned by the teams that understand their own traffic, and cluster-wide guardrails owned by a platform or security team.

What to watch, how to upgrade and how to find the cause when something breaks.

Installing Cilium is the easy part. Running it well for years means watching a few key signals, upgrading carefully and having a calm troubleshooting routine.

Upgrades

Upgrade Cilium the same careful way you would upgrade anything that carries all your traffic. Read the release notes for changes in defaults and removed features, test in a non-production cluster first, upgrade nodes gradually, and verify connectivity after each step. Because the kernel programs keep running while the agent restarts, well-planned upgrades are usually smooth, but surprises do happen, and a rehearsed rollback plan is worth having.

A calm troubleshooting routine

Habits that experienced teams build into their Cilium clusters.

A safe rollout recipe for policy

Five habits that often lead to trouble, and what to do instead.

A balanced view before you commit.

How it compares with other networking choices

The honest summary is that Cilium offers the most integrated eBPF-based networking story, but integration comes with a learning curve. Small clusters with simple needs may be perfectly happy with a simpler plugin. Large, security-sensitive or fast-changing clusters are where the extra capability pays for itself. Several other plugins also offer eBPF modes, so compare current features against your actual requirements rather than choosing by reputation.

Where teams use these ideas in practice.

Short answers to questions beginners ask most often.

It is a way to run small, verified programs safely inside the Linux kernel, attached to events such as packet arrival, so that networking, security and observability can be done fast and without changing the kernel itself.

It can be. Cilium can handle Service load balancing with eBPF so that kube-proxy is not needed. This is optional, and it depends on kernel support and your preferences.

Yes. Cilium enforces the standard NetworkPolicy objects and also adds its own richer policy objects for features such as deny rules, protocol-aware rules and domain-name rules.

Pod addresses change constantly, so rules based on addresses go stale. Identity comes from labels and stays the same for every Pod of an application, making policy stable and compact.

Not necessarily. Cilium covers secure connectivity, policy, observability and ingress, and offers sidecar-free mesh features. If you need advanced traffic management that only a dedicated mesh provides, evaluate that mesh as well.

Existing connectivity generally continues, because the eBPF programs and maps remain in the kernel. The agent reconnects to that state and reconciles it when it comes back.

It is possible with careful migration procedures, but it is delicate. Many teams prefer to adopt Cilium when building a new cluster and move workloads over gradually.

No, it is optional, but it is one of the most valuable parts of the platform, because it makes dropped traffic and policy problems easy to see.

The kernel verifier checks every program for safety before it runs. The agent that loads programs is privileged, so it must be protected carefully, but the programs themselves are constrained by the verifier.

Not always by a large amount. Benefits are biggest in large clusters with many Services and heavy churn. Smaller clusters may see little difference, so measure with your own workloads.

Traditional Kubernetes networking leaned on long chains of firewall rules and on IP addresses that change every time a Pod moves. eBPF lets Cilium put small, verified programs directly inside the Linux kernel, where they can look up Services in hash tables, recognise workloads by label-based identity, enforce policy at layers three to seven, and report every flow to Hubble. The result is networking that scales more smoothly, security that survives churn, and visibility that turns long debugging sessions into short ones. The price is a dependence on modern kernels, a broad feature set to learn, and a few new operational habits such as sizing maps and watching drop reasons. Teams that start small, observe first and add features deliberately tend to get the most from it.