Start with why Kubernetes needs a switchboard at all.

Pods are temporary

A Pod is the smallest unit that runs your containers, and every Pod gets its own IP address. But Pods are disposable. They are replaced during updates, rescheduled when a node fails, and created or removed as load changes. Each replacement gets a brand new IP address. If a front-end program stored the address of a back-end Pod, it would break the next time that Pod was replaced, which in a busy cluster could be minutes later.

Many Pods behind one name

Usually a Service fronts several identical Pods, so the same request can be answered by any of them. That means somebody must choose which Pod receives each new connection. This is load balancing. The choice must be fast, because it happens for every connection, and it must follow changes quickly, because Pods appear and disappear all the time.

Where the choice can be made

The vocabulary you need before looking inside the switchboard.

What a Service is

A Service is a Kubernetes object that gives a group of Pods a stable name and a stable virtual IP address. It picks its Pods using labels. The virtual address is called the ClusterIP. It is virtual because it is not attached to any real network card. No machine answers at that address. It exists only as a rule in the kernel’s packet handling.

Endpoint slices

The list of real Pod addresses behind a Service is stored in objects called EndpointSlices. A controller keeps them up to date as Pods become ready, fail or disappear. Large Services are split across several slices, which keeps each object small and makes updates cheap. kube-proxy reads these slices to know where traffic may be sent.

What DNS adds

People rarely type the virtual address. Cluster DNS gives each Service a name, so a program can connect to a friendly name that resolves to the ClusterIP. DNS itself is handled by a separate component and has nothing to do with kube-proxy modes. DNS answers “what is the address of this Service,” and kube-proxy’s work starts when a packet is sent to that address.

A small program with one job: translate cluster objects into kernel instructions.

The job in one sentence

kube-proxy runs on every node, watches Services and EndpointSlices through the API server, and programs the node’s kernel so that traffic sent to a Service address gets redirected to one of the healthy Pods behind it. It is a translator between the Kubernetes world of objects and the Linux world of network rules.

What it does not do

A short history

The earliest design really did proxy traffic through a user-space program, which was slow because every packet had to cross between kernel and program. It was replaced by the rule-based approach, first with iptables, then with IPVS as an alternative, and more recently with nftables as a modern successor. The old user-space mode has been removed from current releases. Separately, some network plugins now take over the whole job using eBPF and no longer run kube-proxy at all. These are the engines we compare in this post.

Whatever the mode, the flow from a Kubernetes change to a kernel change looks the same.

Following one change

Suppose a new Pod for a Service becomes ready. The endpoint controller adds its address to an EndpointSlice. kube-proxy on every node has a watch open on EndpointSlices, so each instance receives the change within moments. It updates its internal picture, notes that the Service’s rules are now out of date, and schedules a rule update. Updates are deliberately batched, so a burst of changes leads to one rule sync rather than hundreds.

sequenceDiagram
  participant P as New Pod
  participant E as Endpoint controller
  participant A as API Server
  participant K as kube-proxy on each node
  participant N as Node kernel
  P->>A: Pod becomes ready
  E->>A: Add Pod address to EndpointSlice
  A-->>K: Watch event: EndpointSlice changed
  K->>K: Update local picture and mark Service dirty
  K->>K: Wait for batching interval
  K->>N: Program updated rules
  Note over N: New connections can now reach the new Pod
        

Batching and timing

kube-proxy has two timing ideas. One is a minimum gap between syncs, so that a storm of changes does not cause continuous reprogramming. The other is a periodic full sync, which re-applies the entire desired state even if no event arrived, repairing any damage done by something else. These intervals trade freshness against cost. Short gaps make new Pods reachable sooner, long gaps save CPU on busy clusters.

Readiness matters

Only Pods that report themselves as ready receive traffic. A Pod that is still starting, or that has failed its readiness check, is left out of the rules. This is why readiness checks are so important to a smooth rollout: they decide when the switchboard starts sending calls to a new agent. Pods that are shutting down are treated specially in newer versions, so existing connections can finish while no new connections arrive.

The long-standing default. It works almost everywhere and is simple to reason about, up to a point.

What iptables is

iptables is a Linux tool for managing the kernel’s packet-filtering rules. Rules are organised into tables and chains, and each packet walks through the chains and is matched against rules in order. A rule can accept a packet, drop it, or change its destination. Changing the destination is called destination NAT, and it is the trick that makes Services work: a packet addressed to the virtual IP has its destination rewritten to a real Pod address.

How the rules are organised

kube-proxy creates a hierarchy of custom chains. A top-level chain holds one entry per Service address and port. Each of those jumps to a per-Service chain, and the per-Service chain holds one entry per Pod behind it. The per-Pod entry performs the actual address rewriting. To spread load, the per-Service chain uses a random-probability rule: the first Pod gets a chosen probability, the next one a larger share of what is left, and the last one takes the remainder, which produces an even random split.

flowchart TD
  P["Packet to Service IP and port"] --> T["Top-level services chain"]
  T --> M{"Matches a Service entry?"}
  M -- "No" --> X["Continue normal handling"]
  M -- "Yes" --> S["Per-Service chain"]
  S --> R{"Random pick by probability"}
  R --> E1["Per-Pod rule 1: rewrite to Pod 1"]
  R --> E2["Per-Pod rule 2: rewrite to Pod 2"]
  R --> E3["Per-Pod rule 3: rewrite to Pod 3"]
        

The return path and connection memory

Once a connection’s first packet has been rewritten, the kernel’s connection tracking table remembers the choice. Every later packet of that connection follows the same mapping without walking the chains again, and reply packets are automatically rewritten back so the client believes it is talking to the Service address. This memory is why changing the rules does not break connections that are already open.

Instead of a long list of rules, use the kernel’s own virtual server feature, built for exactly this job.

What IPVS is

IPVS, short for IP Virtual Server, is a load-balancing feature that lives inside the Linux kernel. It was designed for large-scale load balancing long before Kubernetes existed. It keeps a table of virtual services, each with a list of real servers behind it, and looks them up using hash tables, so finding the right virtual service takes about the same time whether there are ten Services or ten thousand.

How kube-proxy uses it

In IPVS mode, kube-proxy creates one virtual server for each Service address and port, and adds one real server for each ready Pod. The Service addresses are attached to a dummy network interface on the node, so the kernel treats them as local addresses and hands matching packets to IPVS. A small number of packet-filter rules and address sets still exist alongside IPVS to handle tasks such as marking packets that need address masquerading. So IPVS mode is not entirely rule-free, but the number of rules stays small and mostly constant regardless of how many Services exist.

flowchart TD
  P["Packet to Service IP"] --> D["Service IP is local on a dummy interface"]
  D --> V["IPVS virtual server table lookup by hash"]
  V --> SCH{"Scheduling algorithm"}
  SCH --> R1["Real server: Pod 1"]
  SCH --> R2["Real server: Pod 2"]
  SCH --> R3["Real server: Pod 3"]
        

Choice of scheduling algorithm

One real advantage of IPVS is that it offers several ways to choose the real server, not just a random pick. The choice is configurable in kube-proxy.

Requirements and caveats

IPVS needs specific kernel modules loaded on every node. Missing modules are one of the commonest reasons for failures when switching modes, and some minimal or locked-down operating systems do not include them. IPVS also keeps its own connection table with its own timeouts, separate from the general connection tracking, which means that some long-idle connections may be dropped by timeout settings that differ from what teams are used to. Tuning these timeouts is a known part of running IPVS well.

A newer in-kernel option that keeps the rule-based idea but fixes its biggest scaling weakness.

What nftables is

nftables is the modern replacement for iptables in the Linux kernel. It offers a cleaner way to describe rules and, crucially, supports sets and maps: data structures that let one rule say “if the destination is in this set, jump to the matching target found in this map.” Looking up an entry in a map takes about the same time however large the map is.

What changes for kube-proxy

In nftables mode, kube-proxy keeps its familiar job but expresses Services as map entries instead of long chains. Updates are applied as atomic transactions, so the kernel switches from the old rules to the new ones in one step, with no window in which rules are half-written. This gives performance in large clusters that is much closer to IPVS, without IPVS’s extra kernel modules and its separate connection table. The mode matured through the recent releases and is meant as the long-term in-kernel path on Linux, so check your version’s notes for its current status.

Why mention it in an iptables-versus-IPVS-versus-eBPF post

Because for many teams in the coming years the real question is no longer “iptables or IPVS,” but “stay with kube-proxy in its newer nftables form, or move to an eBPF-based network plugin that replaces kube-proxy.” Knowing that nftables exists keeps the comparison honest and up to date.

The third approach is different in kind. It does not tune kube-proxy, it retires it.

What eBPF is

eBPF is a technology that lets small, safe programs run inside the Linux kernel at specific hook points, such as when a packet arrives or when a program opens a connection. The kernel checks each program for safety before running it, so it cannot crash the system. Network plugins use eBPF to handle packets with custom logic, without the long rule chains of iptables.

Not a kube-proxy mode, a replacement

This point is important. eBPF is not one of the values you can set in kube-proxy’s mode field. Instead, a network plugin that uses eBPF can take over Service handling completely. Popular plugins offer a kube-proxy replacement feature, and with it enabled you do not run kube-proxy at all. The plugin watches the same Services and EndpointSlices, but keeps its own eBPF tables, usually hash maps, and the eBPF programs consult those maps.

Where the load balancing happens

flowchart LR
  subgraph IN["Traffic from inside the node"]
    C["Pod calls connect to Service IP"] --> H1["eBPF hook at connect"]
    H1 --> M1[("eBPF service map")]
    M1 --> T1["Destination swapped to a real Pod IP"]
  end
  subgraph OUT["Traffic from outside the cluster"]
    X["Packet arrives at node port"] --> H2["eBPF hook at network device"]
    H2 --> M2[("eBPF service map")]
    M2 --> T2["Packet forwarded to chosen Pod"]
  end
        

Put the three approaches side by side by following the same request through each.

The scenario

A front-end Pod on node A opens a connection to a back-end Service. The Service has three ready Pods, one on node A, one on node B and one on node C. We follow the first packet and what happens next, in each engine.

sequenceDiagram
  participant F as Front-end Pod on node A
  participant K as Node A kernel
  participant B as Back-end Pod on node B
  Note over F,K: iptables, IPVS or nftables
  F->>K: Packet to Service IP
  K->>K: Match Service, pick Pod, rewrite destination
  K->>B: Packet to Pod IP on node B
  B-->>K: Reply from Pod IP
  K->>K: Reverse rewrite to Service IP
  K-->>F: Reply appears to come from Service IP
  Note over F,K: eBPF connect-time translation
  F->>K: connect to Service IP
  K->>K: Hook swaps in Pod IP before sending
  F->>B: Packets go straight to Pod IP
  B-->>F: Replies need no reverse rewrite
        

What the comparison teaches

All approaches pick a Pod and rewrite a destination. The difference is when and how. The rule-based engines do it as the packet passes through, and rely on connection tracking to remember the choice and to undo the rewrite on replies. The connect-time eBPF approach does it once at the start for in-node clients, so the rest of the connection is plain. For traffic that arrives from outside the node, all engines must rewrite as the packet passes, so the advantage of connect-time translation applies mainly to in-cluster, same-node origins.

One table that collects the differences that matter in day-to-day decisions.

Two kinds of cost

It helps to separate two costs when comparing. The data path cost is how long it takes to handle each packet or connection. The control path cost is how long and how much CPU it takes to apply a change. A cluster with few, long-lived connections but frequent Pod changes cares mostly about the control path. A cluster with stable Pods but millions of short connections cares more about the data path. The engines differ in both, so identify which one hurts you before choosing.

Where each engine’s costs show up, and what you can do before changing the engine.

Where problems appear first

In rule-based modes, the first visible symptom is usually slow rule programming rather than slow packets. As the rule set grows, each sync takes longer, and during busy periods syncs queue up. The effect is that new Pods take longer to become reachable and removed Pods keep receiving traffic a little longer. CPU use of kube-proxy on every node climbs, and in extreme cases the table update itself becomes a source of short pauses for traffic.

Tuning before switching

Locality: keeping traffic close

Performance is not only about CPU. Sending a connection to a Pod in another zone adds latency and, on many clouds, cross-zone transfer charges. Newer Kubernetes versions let a Service prefer endpoints in the same zone, or on the same node, and kube-proxy modes honour those hints when programming rules. This can cut both latency and cost, but it must be used with enough Pods in each zone, otherwise a few Pods get overloaded while others sit idle.

The options that change which Pod is chosen and what address the Pod sees.

External traffic policy

For Services reachable from outside, such as NodePort and LoadBalancer types, there is a setting that controls how a node treats incoming traffic. In the Cluster setting, a node may forward the connection to a Pod on any other node, spreading load evenly but adding a hop and usually masking the original client address. In the Local setting, a node only sends traffic to Pods running on itself, which keeps the client address intact and avoids an extra hop, but can create uneven load if Pods are not spread evenly across nodes.

Why the source address matters

Applications sometimes need the real client address for logging, rate limiting or access rules. Address masquerading replaces it with the node’s own address, which hides the client. The Local policy, or an eBPF data plane that avoids masquerading, keeps the original address. This is one practical reason teams move to eBPF or adopt the Local policy for externally exposed Services.

Session affinity

A Service can ask that connections from the same client address keep going to the same Pod for a period of time. All modes support this, but they implement it differently, and it is based only on the client’s network address, not on cookies. Clients behind a shared address, such as a large office network or another proxy, will all stick to one Pod, which can overload it.

Health checks for load balancers

When the Local policy is used with a cloud load balancer, kube-proxy exposes a small health-check port on each node. It answers positively only when the node has a ready local Pod for the Service. The cloud load balancer uses that answer to stop sending traffic to nodes that cannot serve it. Without this, the Local policy would drop connections on nodes without Pods.

What happens to connections that are already open when a Pod goes away, and why some mistakes only appear with UDP.

The connection tracking table

The kernel keeps a table of active connections, called the conntrack table. In the rule-based modes it holds the mapping from a Service address to the chosen Pod, so replies can be rewritten back. The table has a fixed size. If it fills up, new connections are dropped, which shows up as mysterious timeouts on busy nodes. Monitoring its usage is as important as monitoring CPU.

Stale entries and UDP

When a Pod behind a Service is removed, entries pointing to it may remain in the table. For TCP, the connection simply breaks and the client reconnects, which picks a live Pod. For UDP, there is no connection to break, so the client’s packets may keep going to the removed Pod’s address until the entry expires, producing a silent black hole. For this reason kube-proxy actively clears stale entries for UDP Services when endpoints change. DNS is a common UDP Service, which is why conntrack problems often show up as intermittent DNS failures.

Graceful termination

When a Pod is told to stop, it should keep serving the connections it already has, while no new connections arrive. Modern Kubernetes marks such endpoints as terminating, and kube-proxy modes cooperate: they stop choosing the terminating Pod for new connections but leave existing mappings alone until the Pod is gone. In IPVS mode this is done by lowering the real server’s weight to zero, so it receives nothing new but keeps its current connections.

Practical guidance for different cluster shapes.

Changing the engine under live traffic is possible, but it rewards careful, gradual steps.

Why migration is delicate

The Service switchboard sits under every connection in the cluster. A mistake can break internal calls between applications, DNS or access from outside. Switching modes replaces one set of kernel structures with another, and leftover rules from the old mode may linger unless they are cleaned up. A staged approach limits the blast radius.

Special care for eBPF replacement

Moving to an eBPF plugin that replaces kube-proxy is a larger step than switching modes, because it is often combined with changing or reconfiguring the network plugin itself. Plan it as a network migration project, with a tested rollback, a maintenance window for the riskiest stages, and a clear plan for removing kube-proxy only after the replacement has proven itself. Expect to retrain operators on new tools.

The component that rewrites every Service connection deserves careful protection.

Privileges of the node agent

kube-proxy edits kernel networking state, so it runs with high privileges on every node, usually with network administration rights. A compromised kube-proxy Pod could redirect traffic, which is powerful. Protect it by restricting who can change its configuration, using trusted images, and limiting its access to the API server to only what it needs. Its identity is normally bound to a narrow role for reading Services and EndpointSlices.

Exposure through NodePorts and external addresses

A NodePort Service opens a port on every node, often across a wide range, whether or not the node runs any of the Service’s Pods. This enlarges the attack surface. Limit where nodes are reachable from with network firewalls and security groups, and restrict which address ranges NodePorts bind to when you do not need them on every interface. For load balancer Services, use source range restrictions so that only expected networks can connect.

Policy is a different layer

Because kube-proxy does not enforce network policy, a cluster without a policy-capable network plugin has no Pod-to-Pod restrictions at all, however carefully the Services are set up. Network policy and Service translation interact: policy is evaluated against real Pod addresses after translation in rule-based modes, while eBPF plugins implement both together and apply identity-aware rules. Test policies with your chosen engine, since subtle differences exist in how they treat Service traffic.

Information exposure

What to watch, and how to find the problem when a Service does not respond.

Metrics worth tracking

A troubleshooting path for an unreachable Service

Typical symptoms and causes

Habits that keep Service networking boring, which is exactly what you want.

Patterns that work

Failures seen again and again in real clusters.

A final balance sheet for the whole kube-proxy approach and its alternatives.

Connection-level balancing is not request-level

A subtle limit worth repeating: kube-proxy engines balance new connections, not individual requests. If a client opens one long-lived connection, such as with some modern protocols that multiplex many requests, all those requests go to the same Pod. Teams that need request-level balancing, retries or fine-grained traffic shaping turn to ingress controllers, gateways or service meshes on top of Services, not instead of them.

Choosing between the options

Questions that beginners and experienced engineers keep asking.

In the modes used today, no. kube-proxy only programs the kernel, and the kernel moves the packets. If kube-proxy stops, existing rules keep working with their last known state, though updates stop until it returns.

No. eBPF-based Service handling comes from a network plugin that replaces kube-proxy. You enable the plugin’s replacement feature and do not run kube-proxy on those nodes.

Because the address is virtual and the rules usually match only the specific protocol and port of the Service. A ping uses a different protocol, so nothing translates it. Test with a real connection to the Service port instead.

Generally yes, since each node translates traffic that leaves it, which is why gradual migration by node pool works. It does make troubleshooting harder, so keep mixed periods short and well labelled.

They can. Mappings live in kernel tables owned by the old engine, so switching may drop or orphan some connections. Roll nodes gradually, drain them when possible, and expect clients to reconnect.

Not always. In small clusters the difference is usually invisible. The advantage appears when the number of Services is large and rule programming time dominates. Benchmark your own workload.

Under the Cluster traffic policy, nodes mask the client address when forwarding to a Pod on another node. Use the Local policy or an eBPF data plane that avoids masquerading if you need real client addresses.

In rule-based modes, usually rule programming time and CPU use of kube-proxy, which shows up as slow reachability of new Pods. Packet speed degrades later. Connection tracking limits are another early casualty on busy nodes.

No. Headless Services have no virtual address, so DNS returns Pod addresses directly and clients connect to Pods without any Service translation.

No. It balances new connections. Many requests on one connection all go to the same Pod. For request-level balancing, use an ingress, gateway or service mesh layer.

Start from what your platform supports and what the current release notes recommend. Many teams begin with the default, then consider nftables or an eBPF plugin if they need more scale or visibility. Avoid building new plans around a mode that your version’s notes mark as deprecated.

Usually yes, because existing rules stay in the kernel while it restarts. The risk is the gap during which Service changes are not applied, and the full resync on startup, so restart nodes gradually rather than all at once.

Pods are temporary, so Kubernetes gives groups of them a stable virtual address through Services, and keeps the list of ready Pods in EndpointSlices. kube-proxy watches those objects on every node and writes instructions into the kernel so that traffic to a Service address is redirected to a healthy Pod. In the rule-based modes the kernel does the rewriting, kube-proxy only programs it, and connection tracking remembers each choice. The iptables mode is simple and universal but slows as the number of Services grows. IPVS uses hash tables and offers several balancing algorithms but needs extra kernel pieces and has an uncertain future. nftables keeps the in-kernel model but uses maps for constant-time lookups and atomic updates. eBPF replacement, provided by network plugins, removes kube-proxy and translates Services at connect time or at the network device.

Come back to the helpline at the start. The number stays fixed, the agents come and go, and the switchboard keeps its list current. What differs between the engines is how that switchboard is built: a long checklist, an indexed machine, a phone book or a smart gadget at the door. The best choice is the simplest one that your measurements show you can live with, adopted gradually, watched carefully and understood well enough to debug at three in the morning.