To understand CNI plugins we first need a handful of simple words: pod, node, network, address and plugin.

So what exactly is a CNI plugin?

CNI stands for Container Network Interface. It is a small, published standard that describes how a program that runs containers should ask another program to set up the network for those containers. The second program is the CNI plugin. When a new pod is created, the machinery on the node calls the plugin and says, in effect: “Here is a new pod. Please give it an address and connect it to the network.” When the pod is deleted, the machinery calls the plugin again and says: “Please take that address back and clean up.”

The important idea is separation. Kubernetes defines what must be true of the pod network. The CNI plugin decides how to make it true. Because the interface is standard, you can choose a simple plugin for a small cluster or a sophisticated one for a big cluster, without changing anything else in Kubernetes.

The three promises of the Kubernetes network

Kubernetes makes a few simple promises about the pod network. Any CNI plugin you choose must keep them. Almost everything else in this guide follows from these promises.

A short history

Before container platforms agreed on a standard, every tool invented its own way of wiring up networks. Each new network product had to write separate glue for each container platform. The CNI standard, created in the middle of the 2010s and later adopted by the Cloud Native Computing Foundation, replaced this tangle with one small rulebook. Today Kubernetes and most other container platforms use it. Because it is only a few pages long, writing a plugin is straightforward, which is one reason the ecosystem grew so rich.

Containers are isolated by design. That is wonderful for safety and terrible for connection. Someone has to build the bridges.

Containers live in sealed rooms

Each pod runs inside its own private network space, a sealed room with its own doors, its own address list and its own view of the network. This isolation is a feature: one pod cannot accidentally listen to another pod’s traffic or clash over the same port number. But a sealed room is useless until you cut a door in the wall and connect it to a corridor.

What people did before

flowchart LR
  subgraph W["Without a CNI plugin"]
    A1["Pod on node 1"] -.->|"No road"| B1["Pod on node 2"]
  end
  subgraph V["With a CNI plugin"]
    A2["Pod on node 1"] -->|"Address and road provided"| B2["Pod on node 2"]
  end
    

Reading Figure 1

The left half shows two pods on two nodes with a dotted arrow marked “No road”. Each pod exists, but nothing connects them, so a message from one never arrives at the other. The right half shows the same pods after a CNI plugin has done its work. The plugin has given each pod an address and prepared a path, so the solid arrow succeeds. The pods are identical in both halves. Only the plugin makes the difference.

A few building blocks of networking appear again and again. Learn them once, and every plugin will make sense.

Two ways to build cross-node roads

The hardest job of a plugin is getting traffic from a pod on one node to a pod on another. Two broad styles exist, and nearly every plugin uses one or both.

Where do pod addresses come from?

When the cluster is created, the administrator reserves a large address range for pods. That range is split into smaller slices, and each node receives one slice. When a pod is created on a node, the plugin picks an unused address from that node’s slice. Because slices never overlap, no two pods anywhere in the cluster can receive the same address. Some plugins let the cluster hand out slices; others manage pools themselves. The result is identical: unique addresses, no clashes.

Network policy: the rulebook and the enforcer

A network policy says things like “only pods labelled as the web tier may reach pods labelled as the database, and only on this port”. Kubernetes stores the policy, but Kubernetes does not enforce it. The enforcement is done by the CNI plugin. This is a crucial fact: on a plugin that does not implement network policy, the policy objects can be created without error and silently do nothing. We return to this in Chapter 9.

Several pieces cooperate to give a pod its network. It helps to see who talks to whom, from the moment Kubernetes decides to start a pod.

flowchart TB
  API["Kubernetes control plane"] -->|"Pod placed on this node"| KL["Kubelet on the node"]
  KL -->|"Create the pod sandbox"| RT["Container runtime"]
  RT -->|"Call the plugin: add this pod"| CNI["CNI plugin program"]
  CNI -->|"Ask for an address"| IPAM["Address manager"]
  CNI -->|"Create virtual cable and routes"| NS["Pod network space"]
  AG["Plugin agent on the node"] -->|"Keeps routes and rules up to date"| NS
  AG -->|"Learns about pods and policies"| API
  NS --> APP["Application containers"]
    

Reading Figure 2 piece by piece

Follow the arrows from the top. The control plane, which is the brain of Kubernetes, decides that a pod should run on a particular node. The kubelet on that node notices and asks the container runtime to create the pod. The runtime first creates an empty sealed room, called the pod sandbox, and then calls the CNI plugin program with a request to add this pod to the network.

The plugin program does the quick, one-time work: it asks the address manager for a free address, creates the virtual cable, places one end in the pod’s room, gives it the address and sets up the routes so packets know where to go. Then it reports success, and the runtime goes on to start the application containers.

Below and beside this flow sits the second half of most plugins: a long-running node agent. It is not part of the one-time set-up. Instead it runs all the time as a small program on every node, watching the cluster for changes such as new pods, moved pods, new nodes and new policies, and keeping the node’s routes and rules current. The plugin program is the quick installer; the agent is the caretaker.

The parts, one by one

Where the files live

Two folders on every node matter. One holds the plugin programs. The other holds the configuration file that says which plugin to use. When you install a plugin, its installer typically drops the program and the configuration file into these folders on every node. The runtime looks in the configuration folder, picks the first file in alphabetical order, and calls the plugin it names. That is why files in that folder are often given numbered names.

Now let us open the box. We will look at the small conversation between the runtime and the plugin, and then at how a packet travels from one pod to another.

The conversation: four simple commands

The CNI standard defines a tiny vocabulary. The runtime sends the plugin one of a few commands, together with the details it needs, and the plugin replies with a result. The four commands are easy to remember.

The details of each request, such as the pod’s identifier, the location of its private room and the network name, travel with the command. The plugin’s answer includes the address it assigned and the interfaces it created. Everything is plain, structured text, which is why plugins can be written in almost any language.

Chaining: several plugins in a row

The standard lets a configuration name several plugins in a list. The runtime calls them in order, passing the result of each to the next. A common arrangement uses a main plugin to create the connection and give the address, followed by small helper plugins that adjust bandwidth limits, enable port mappings or tune settings. Chaining is how features are added without rewriting the main plugin. Some advanced setups also let a pod have more than one network, using a special multiplexing plugin that calls several others.

flowchart LR
  P1["Pod A on node 1"] --> V1["Virtual cable and node bridge"]
  V1 --> Q{"Is the target pod on this node?"}
  Q -->|"Yes"| P2["Pod B on the same node"]
  Q -->|"No"| R{"How does this plugin cross nodes?"}
  R -->|"Overlay"| T["Wrap packet and send through tunnel"]
  R -->|"Routed"| RT["Send packet unwrapped along learned routes"]
  T --> N2["Node 2 unwraps and delivers"]
  RT --> N2
  N2 --> P3["Pod C on node 2"]
    

Reading Figure 3

A message begins in Pod A. It leaves through the virtual cable and arrives at the node. The node asks the first question: is the target on this very node? If yes, the node simply hands it to the neighbour, and nothing leaves the machine. If no, the plugin’s style decides what happens. An overlay plugin wraps the packet in an outer packet addressed to the other node and sends it through a tunnel. A routed plugin sends it as it is, relying on routes that were previously taught to the network. Either way, the second node ends up holding the packet, removes any wrapping and delivers it to Pod C.

Enforcement inside the node

Besides connecting pods, a plugin usually installs rules that decide which packets are allowed. Older plugins do this with the operating system’s classic packet-filtering tables. Newer ones, like Cilium, attach small programs to the kernel’s network path through a technology called eBPF, which lets safe, sandboxed programs run inside the operating system without changing it. We will meet the difference in Chapter 7.

Let us follow one pod from birth to death and see what the network does at each stage.

sequenceDiagram
  participant K as Kubelet
  participant R as Runtime
  participant C as CNI plugin
  participant I as Address manager
  participant A as Node agent
  K->>R: Create pod sandbox
  R->>C: ADD this pod
  C->>I: Give me a free address
  I-->>C: Address granted
  C-->>R: Cable created, address set
  R-->>K: Sandbox ready
  K->>R: Start application containers
  A->>A: Update routes and rules for the new pod
  K->>R: Remove pod sandbox later
  R->>C: DEL this pod
  C->>I: Release the address
  A->>A: Remove routes and rules
    

Reading Figure 4

The story runs from the top. The kubelet asks the runtime for a sandbox. The runtime asks the plugin to add the pod. The plugin asks the address manager for a free number, gets it, builds the virtual cable, and reports back. Only then does the kubelet start the application containers, which is why an application never starts without a network. Meanwhile the node agent quietly updates routes and rules so other nodes learn about the newcomer. At the end of life, the sequence reverses: the runtime calls DEL, the address returns to the pool and the agent removes the routes.

Six stages of a pod’s network life

How services fit into the picture

Pods come and go, and their addresses change, so Kubernetes offers a stable virtual address called a service. A service is a fixed number in front of a changing group of pods. When a pod calls a service, something must translate the fixed number into the address of one healthy pod. Traditionally a component on every node called the service proxy does this, by installing rules that pick a pod at random. The CNI plugin’s job is different: it makes sure that once the real pod address is chosen, the packet can actually reach it. Some plugins, notably Cilium, can take over the service proxy job as well, so one system handles both the choosing and the delivery.

Understanding this split helps troubleshooting. If a service address fails but direct pod addresses work, look at the service proxy or its replacement. If direct pod addresses fail as well, look at the plugin. Two layers, two sets of suspects.

What if something goes wrong?

If the plugin fails during ADD, the runtime reports the failure to the kubelet, and the pod sandbox is torn down and retried. The pod shows a status such as container creating for a while, and the events mention that the network could not be set up. If cleanup fails during DEL, an address can stay reserved, called a leak, until a housekeeping process reclaims it. A cluster that churns through many short-lived pods can run out of addresses on a node if leaks accumulate, so good plugins track and reclaim these.

All three keep the promises from Chapter 1. They differ in philosophy, features and how they move packets. Let us meet each in turn.

flowchart TB
  subgraph F["Flannel: simple overlay"]
    F1["Pod"] --> F2["Node bridge"] --> F3["VXLAN tunnel"] --> F4["Other node"]
  end
  subgraph C["Calico: routed or overlay, plus rules"]
    C1["Pod"] --> C2["Node routes"] --> C3["Learned routes or light tunnel"] --> C4["Other node"]
  end
  subgraph L["Cilium: kernel programs"]
    L1["Pod"] --> L2["eBPF programs in the kernel"] --> L3["Routing or tunnel"] --> L4["Other node"]
  end
    

Reading Figure 5

Each row shows the road a packet takes. Flannel’s path is short and easy to picture: the pod, a node bridge, a tunnel and the other node. Calico’s path uses ordinary routing, with routes learned from its own agent, and can fall back to a light tunnel when the underlying network will not carry pod routes. Cilium replaces much of the classic path with small programs running inside the kernel, deciding where packets go and whether they may pass.

Flannel: the simple one

Flannel was created to do one job with minimal fuss: give every node its own slice of pod addresses and connect the nodes with a simple overlay. Its default tunnel wraps pod packets in ordinary network packets, using a technology called VXLAN. It also offers other modes, such as routing directly between nodes on the same network, or encrypted tunnels using WireGuard.

Calico: the routing and policy specialist

Calico came from a networking-first mindset: treat pods as ordinary citizens of the network and route them like any other address. Its node agent, often called Felix, programs routes and rules on every node. A routing daemon shares pod ranges between nodes using a well-known internet routing protocol called BGP, so the real network learns where pods live and no wrapping is needed. When the network cannot carry those routes, Calico can switch to a light overlay.

Calico’s greatest fame is its network policy engine. It supports the standard Kubernetes policies and adds richer ones of its own, such as cluster-wide rules, ordering, explicit deny and rules based on more kinds of matching. It can enforce with classic packet filtering, or with an eBPF mode on modern kernels.

Cilium: the kernel-powered one

Cilium is built on eBPF, the technology that lets small safe programs run inside the operating system’s kernel. Instead of long chains of packet-filtering rules, Cilium attaches programs directly to the network path. This can make lookups faster and gives Cilium deep visibility into traffic. It can also take over the job of the service proxy that normally handles Kubernetes service addresses, and it offers a rich observability tool called Hubble that shows who talked to whom.

A distinctive idea in Cilium is identity-based policy. Rather than tracking addresses, which change constantly, it gives each group of similar pods a numeric identity derived from their labels, and writes policy in terms of identities. It can also understand some application-level traffic, letting a rule say that a pod may call one web path but not another. Extra features include transparent encryption between nodes and connecting multiple clusters together.

A single table cannot decide for you, but it shows the shape of the differences at a glance.

Other names you will meet

Cloud-native plugins versus portable plugins

A cloud plugin integrates tightly with one provider. Pods get real addresses from the virtual network, security groups and flow logs can see pod traffic, and no overlay is needed. The price is portability and limits: the number of addresses available may bound how many pods fit on a node, and moving to another cloud means changing the plugin. Portable plugins like the big three work everywhere, at the price of running their own network layer on top of the cloud’s. Many managed clusters offer a choice, and some teams run a portable plugin in a chained or overlay arrangement on top of the cloud’s own.

By default every pod can talk to every other pod. That is convenient and dangerous. Network policy is how you close the doors that should never have been open.

Open by default

A fresh cluster behaves like an open-plan office with no walls. A compromised pod in a low-importance application can reach the database of an important one, simply because the network allows it. Attackers rely on this ease of movement, called lateral movement. Network policy turns the open office into rooms with locked doors.

How policies work in plain words

A policy selects a group of pods by their labels and then lists what is allowed to reach them, called ingress rules, and what they are allowed to reach, called egress rules. The key behaviour is this: once any policy selects a pod for a direction, everything not explicitly allowed in that direction is denied. Policies only add permissions; they never subtract, so several policies on one pod combine into one wider set of allowed traffic.

A worked policy example in words

Imagine a shop with three tiers: a web tier, an orders tier and a database tier, each in its own group of pods with its own label. First, a default-deny policy closes every direction for all pods in the shop’s namespace. Second, a policy allows the outside world to reach the web tier on its public port. Third, a policy allows only the web tier to call the orders tier on its service port. Fourth, a policy allows only the orders tier to reach the database on its port. Fifth, a small egress policy lets every tier reach the cluster’s naming service, and lets the orders tier call one named payment provider outside. The result is a clear picture: traffic can flow down the chain and nowhere else. If a pod in the web tier is compromised, it cannot talk to the database directly, because no policy opens that door.

Notice how readable this is. Each policy is one sentence in plain language, tied to labels. When a new engineer asks how the shop is protected, the answer is a short list of sentences, not a maze of addresses and ports.

Other security topics

There is no best plugin, only the best fit for your platform, your people and your needs. A few questions get you most of the way.

flowchart TD
  A["Which plugin should I choose?"] --> B{"Do I need network policy?"}
  B -->|"No, learning or tiny lab"| C["Flannel is enough"]
  B -->|"Yes"| D{"Do I want deep traffic visibility and identity rules?"}
  D -->|"Yes, and my kernel is modern"| E["Cilium"]
  D -->|"No, mature policy and routing are enough"| F["Calico"]
  A --> G{"Am I locked to one cloud and want native pod addresses?"}
  G -->|"Yes"| H["Consider the cloud provider plugin"]
    

Reading Figure 6

Begin with the top question about network policy. If you need no policy, for instance in a learning lab or a tiny personal cluster, the simplicity of Flannel is hard to beat. If you do need policy, the next question is about depth. If you want detailed visibility, identity-based rules and advanced features and you run a modern kernel, Cilium fits. If a mature, proven policy engine with flexible routing is enough, Calico fits. A separate branch reminds you that if you live in one cloud, the provider’s own plugin deserves a look for its native pod addresses.

The questions behind the chart

Networking sits on the critical path of every request. Small inefficiencies multiply when millions of packets flow each second.

Where the time goes

Ways to go faster

Routed modes avoid wrapping, so where the underlying network allows it, routing is usually faster than tunnelling. Kernel-program approaches like eBPF replace long rule lists with quick lookups, which helps in clusters with many services and policies. Taking over the service proxy function in the same layer removes another set of rules. Using the largest packet size the network supports, and setting the plugin’s packet size correctly, prevents fragmentation. Finally, sizing address slices sensibly avoids running out of addresses on busy nodes.

Scaling out

When a cluster grows from ten nodes to a thousand, the shared state that all agents watch becomes a bottleneck. Plugins respond with caching proxies between agents and the store, with route reflectors that avoid every node talking to every other, and with careful batching of updates. Plan your address ranges for the size you may become, not the size you are, because changing them later means rebuilding.

Every choice buys something and costs something. Here is an honest ledger for the CNI approach and for the three plugins.

Trade-offs in plain words

The first trade-off is simplicity against features. Flannel is easy to grasp and hard to break, but gives you no policy and little visibility. Cilium gives a great deal, but you must understand and operate a more complex system. Calico sits in the middle. The right point on this line depends on how much you need and how much complexity your team can carry.

The second is overlay against routed. An overlay works everywhere and asks nothing of the network, at a modest performance cost. Routing is faster and easier to inspect but needs cooperation from the network, which may be hard in cloud or corporate environments.

The third is portability against integration. A cloud plugin integrates neatly but ties you to one provider. A portable plugin runs anywhere but adds its own layer. Decide how likely you are to move, and how much you value native integration today.

The fourth is maturity against novelty. Older techniques such as classic packet filtering are proven and widely understood. Newer kernel programs promise speed and insight but demand modern systems and newer skills. Neither is wrong; they suit different teams at different times.

When the pod network breaks, everything above it breaks with it. A calm, layered routine finds the fault quickly.

Symptoms and where to look

A layered troubleshooting routine

Work from the bottom up. First, check the nodes: are they ready, and can they reach one another on the node network? Second, check the plugin’s node agents: is one healthy on every node, and do its logs show errors? Third, test the same-node path, then the cross-node path, using a simple test pod in each place. Fourth, test name lookups separately, because failures there look like network failures. Fifth, review policies that could select the affected pods. Sixth, use your plugin’s own diagnostic and visibility tools, which often say plainly which rule dropped a packet.

What to monitor all the time

Track agent health and restarts on every node, the number of free addresses per node, dropped packet counters, policy denials, connection table use, tunnel health and latency between nodes. Alert when agents crash, when free addresses run low, and when drops jump suddenly. Ship agent logs to a central place so that you can investigate after a node has gone. Plugins such as Cilium and Calico expose metrics that fit well into common dashboards, and Cilium’s flow tool shows conversations directly.

A pattern is a proven way of solving a repeated problem. An anti-pattern is a tempting habit that keeps causing pain.

Patterns worth adopting

Anti-patterns to avoid

This chapter gathers the advice into checklists you can keep beside your desk.

Best practices

Common mistakes

A note on migrating between plugins

Changing the plugin of a running cluster is one of the riskiest operations in Kubernetes. Pods already have addresses and connections created by the old plugin, and the old and new plugins may disagree about ranges, tunnels and rules. Some projects document staged migration paths that run both plugins side by side for a while, moving nodes over in groups. Most teams find it safer to build a new cluster with the new plugin and move workloads across. Whichever route you take, rehearse it, plan a rollback and schedule downtime windows for critical systems.

These generic stories, based on common patterns rather than any specific company, show how the choices play out.

Three worked scenarios: thinking like a designer

A short mental checklist for a new cluster

Before creating a cluster, pause and walk through a short list. State your needs in one sentence: policy, visibility, cloud integration or simplicity. Check the kernel versions your machine images provide. Choose pod and service ranges that overlap with nothing. Decide how nodes will reach each other and which ports the plugin needs open. Decide how you will test that policy really blocks. Decide how you will monitor agents and free addresses. Finally, write down how you would migrate away if you had to. Ten minutes on this list prevents most of the problems in this article.

Quick answers to the questions beginners ask most often.

This post was written without live web research. Please verify version-sensitive details, such as default modes, kernel requirements and feature availability, against each project’s official documentation.

Let us close the loop, returning to the new city from the start of this article.

Just as a growing city needs someone to number the houses, lay the roads and decide who may visit whom, a Kubernetes cluster needs a CNI plugin. It gives each pod a unique address, connects pods across nodes, tears everything down cleanly when a pod ends, and, in most modern plugins, enforces rules about which pods may talk.

You met the problem of sealed rooms, the core building blocks, and the architecture of plugin program, node agent and address manager. You followed a pod’s network life and a packet’s journey, compared overlay and routed styles, and met Flannel, Calico and Cilium along with cloud-native alternatives. You learned about policy, performance, trade-offs, troubleshooting and the mistakes that cost teams the most.

Version drift and things to verify

Kubernetes networking changes steadily. Default modes, supported kernels, policy features and the maintenance status of individual plugins have all shifted over the years. Because this article was written without live web research, treat version-specific statements as a starting point. Before relying on a feature in production, read the official documentation for the exact version you will run, try the behaviour in a test cluster, and record the version in your team’s notes so upgrades prompt a re-check. Managed platforms add their own rules, so check what yours permits.

A small glossary to keep nearby

Key takeaways