Before anything else, let us pin down exactly what a node is and what it is not.

The one-sentence answer

A node is a single machine, either a physical server or a virtual machine, that belongs to a Kubernetes cluster and does the real work of running your applications. Every time you deploy something on Kubernetes, that something ends up running on a node. A worker node is simply the everyday name for such a machine. People use the word “worker” to tell it apart from the machines that manage the cluster.

Let us define the words in that sentence. A cluster is a group of machines that are managed together so they behave like one big computer. A container is a lightly packaged box that holds one program plus everything the program needs, such as its libraries and settings. A pod is the smallest unit Kubernetes schedules. It wraps one or more containers that must live side by side and share the same network address and storage. A pod always lives on exactly one node. It can never be split across two.

Where do nodes come from?

A node can be almost any computer that runs a supported operating system, which is usually Linux. In the cloud, a node is normally a virtual machine, such as an Amazon EC2 instance in Amazon EKS, a Compute Engine VM in Google GKE, or a virtual machine in Azure AKS. In a company data center, a node might be a physical server in a rack. Hobbyists even build small clusters from Raspberry Pi boards. Kubernetes does not care about the brand of the hardware. It only cares that the machine has the right software installed and can reach the rest of the cluster over a network.

Older documents call worker nodes “minions” and the management machines “masters”. Modern Kubernetes prefers “worker node” and “control plane node”. If you read an old tutorial, mentally translate the words.

A cluster has two kinds of machines with two very different jobs. Mixing them up is one of the most common sources of confusion.

The control plane is the brain of the cluster. It stores what you asked for, decides where things should run, and watches for anything that drifts from the plan. The worker nodes are the muscles. They follow instructions and run the actual containers. Your own application code normally runs only on worker nodes.

The control plane has four main parts. The kube-apiserver is the front door; every request goes through it. etcd is the database that stores the desired state of the cluster. The kube-scheduler chooses a node for every new pod. The kube-controller-manager runs many small loops that keep reality matching the plan, for example noticing that a node has gone silent.

flowchart TB
    Dev["Developer using kubectl"] --> API
    Users["End users"] --> LB["Cloud Load Balancer"]
    subgraph CP["Control Plane"]
        API["kube-apiserver"]
        ETCD[("etcd datastore")]
        SCH["kube-scheduler"]
        CTRL["kube-controller-manager"]
        API --- ETCD
        SCH --> API
        CTRL --> API
    end
    subgraph W1["Worker Node 1"]
        K1["kubelet"]
        R1["Container runtime"]
        X1["kube-proxy"]
        POD1["Pods"]
        K1 --> R1 --> POD1
    end
    subgraph W2["Worker Node 2"]
        K2["kubelet"]
        R2["Container runtime"]
        X2["kube-proxy"]
        POD2["Pods"]
        K2 --> R2 --> POD2
    end
    K1 --> API
    K2 --> API
    LB --> X1
    LB --> X2
    

Read the diagram from the top. A developer sends a request through kubectl, the command-line tool, to the API server. The API server saves it in etcd. The scheduler and controller manager watch the API server and react. Notice that the control plane never talks directly to your containers. Instead, the kubelet on each worker node reaches up to the API server, asks what it should be running, and does it. End-user traffic takes a separate path: it arrives at a cloud load balancer, which forwards it to the nodes, where kube-proxy steers it to the right pod.

A node is more than a computer. It is a computer plus a small team of software helpers that connect it to the cluster.

On top of its operating system, every worker node runs a handful of components. Each has one clear responsibility. Understanding who does what makes almost every Kubernetes problem easier to debug.

flowchart LR
    API["kube-apiserver (control plane)"]
    subgraph NODE["Worker Node"]
        KUBELET["kubelet"]
        CADV["cAdvisor inside kubelet"]
        CRI["Container runtime such as containerd"]
        PROXY["kube-proxy"]
        CNI["CNI plugin"]
        CSI["CSI node plugin"]
        KERNEL["Linux kernel rules: iptables or IPVS"]
        subgraph PODS["Pods"]
            PA["Pod A"]
            PB["Pod B"]
        end
        KUBELET -->|"CRI calls"| CRI
        CRI --> PODS
        KUBELET -->|"asks for pod network"| CNI
        KUBELET -->|"asks for volumes"| CSI
        KUBELET --- CADV
        PROXY -->|"programs traffic rules"| KERNEL
    end
    KUBELET |"watch and report"| API
    PROXY -->|"watch Services"| API
    

Look at how the kubelet sits in the middle. It is the only component that constantly talks to the control plane about pods. When a pod arrives, the kubelet asks the container runtime to create it, asks the CNI plugin to wire up its network, and asks the CSI plugin to mount its storage. Meanwhile kube-proxy works separately: it watches Services in the API server and translates them into rules that the Linux kernel enforces. The kernel itself moves the packets, so kube-proxy is not sitting in the traffic path like a traditional proxy server would.

There is one more small helper worth knowing: the pause container. Whenever a pod starts, the runtime first creates a tiny, do-nothing container whose only job is to hold the pod’s network space open. The real containers then join that space and share it. This is why all containers in a pod can talk to each other over localhost.

If you remember only one node component, make it the kubelet. It is the link between the cluster’s plan and the machine’s reality.

The kubelet is a background program that runs on every worker node. It does not run inside a container; it runs directly on the machine so that it keeps working even when containers misbehave. Its work can be described as a loop that repeats forever: look at what should be running, look at what is actually running, and fix any difference.

The three kinds of probes

A probe is a routine check, like a nurse asking “are you okay?” at regular intervals. A liveness probe asks whether the container is stuck; if the answer is no, the kubelet restarts it. A readiness probe asks whether the container is ready to receive traffic; if the answer is no, the pod is quietly removed from the list of destinations for its Service until it recovers. A startup probe gives slow-starting programs extra time before the other two begin. When a container keeps crashing, the kubelet waits longer between each restart, up to five minutes, which you will see as the status CrashLoopBackOff.

Static pods

Normally pods arrive from the API server. There is one exception: static pods. These are described by files placed in a special folder on the node itself, and the kubelet starts them without asking anyone. Tools such as kubeadm use this trick to launch control plane components, because at the moment a cluster is born there is no API server yet to give instructions.

Because the kubelet is so central, its own health matters. If it crashes or freezes, the node stops sending heartbeats, and the control plane will eventually declare the node lost even though the containers may still be running. The kubelet listens on port 10250, which is why securing that port is such an important task, as you will see later.

The kubelet gives the orders. The container runtime is the hands that do the work.

A container runtime is software that downloads container images, unpacks them, and starts them as isolated processes on the machine. An image is the frozen, ready-to-run package of your program, similar to an installer file. The runtime pulls it from a registry, which is an online storage place for images such as Docker Hub or Amazon ECR.

How does a container stay separated from its neighbours on the same machine? The Linux kernel provides two tools. Namespaces give each container its own private view of the system, so it sees only its own processes and network. Cgroups (control groups) limit how much CPU and memory the container may use. Together they create the illusion of a small private computer without the cost of running a full extra operating system.

The Container Runtime Interface

The kubelet does not know how any specific runtime works. Instead, both sides agree on a common language called the Container Runtime Interface, or CRI. Any runtime that speaks CRI can be plugged in. This is a good example of a clean design: because the contract is fixed, teams can swap the engine underneath without changing the rest of Kubernetes.

That last row causes a lot of worry, so let us clear it up. In version 1.24, Kubernetes removed a built-in bridge called dockershim that let the kubelet talk to Docker Engine. This did not break Docker-built images. Images follow an open standard called OCI, and containerd runs them happily. What changed is only which program on the node starts them.

Pods are born and die constantly. Networking is what keeps the rest of the system able to find them.

Kubernetes follows a simple networking promise: every pod gets its own IP address, and any pod can reach any other pod without tricks such as address translation. Making that promise true is the job of the CNI plugin (Container Network Interface). When the kubelet creates a pod, the CNI plugin picks an address, creates a virtual network cable between the pod and the node, and adds the routes needed so other nodes know where to send packets. Different plugins do this differently. Calico uses routing rules, Cilium uses a modern kernel technology called eBPF, and the AWS VPC CNI hands pods real addresses from your private cloud network.

There is a problem, though. Pod addresses change every time a pod is replaced. If a website pod needs to call a database pod, it cannot memorize an address that may vanish in an hour. Kubernetes solves this with a Service: a stable name and virtual address that stands in front of a changing group of pods.

What kube-proxy actually does

The Service address is virtual: no machine owns it. kube-proxy makes it real. It watches the API server for Services and for the list of healthy pods behind each one. Then it writes rules into the node’s kernel that say: “any packet sent to this Service address should be redirected to one of these pod addresses.” Because every node has its own kube-proxy, every node can route Service traffic locally without asking anyone.

Services come in several flavours. A ClusterIP Service is reachable only from inside the cluster. A NodePort Service opens the same port number on every node so outside traffic can enter. A LoadBalancer Service asks the cloud provider to create a public load balancer in front of the nodes. Names such as “orders.shop.svc” are turned into Service addresses by CoreDNS, a DNS server that itself runs as pods on your nodes.

A node has a fixed amount of CPU, memory, and disk. Kubernetes must share them fairly without letting any one pod ruin the party.

Every node reports how much it owns. This total is its capacity. But not all of it is available to your pods, because the operating system and Kubernetes components need some for themselves. What is left after those reservations is called allocatable. The scheduler only ever plans against the allocatable number.

Requests, limits, and what happens when they are crossed

A request is a promise the scheduler honours: “this pod needs at least this much.” A limit is a ceiling the node enforces. The two resources behave differently when the ceiling is hit. CPU is compressible: a container that wants more CPU than its limit is simply slowed down, called throttling. Memory is incompressible: you cannot take back memory that a program is already holding, so a container that exceeds its memory limit is killed and reports OOMKilled, short for out of memory.

Quality of Service classes

Based on the requests and limits you set, Kubernetes puts every pod into one of three classes. These decide who is sacrificed first when a node runs short.

By default, the kubelet begins evicting pods when available memory falls below 100 MiB or when free disk space on the main filesystem drops under 10 percent. It also caps each node at 110 pods, a number you can change but should think about carefully, because more pods on a node means more networking and bookkeeping work.

Let us follow one request from the moment you press Enter until your container is serving traffic.

Suppose you ask Kubernetes for a Deployment, which is a request such as “keep three copies of my web app running.” Behind the scenes, controllers turn that into three pod records that have no node assigned yet. Then the real journey to a node begins.

sequenceDiagram
    participant U as Developer
    participant A as kube-apiserver
    participant E as etcd
    participant S as kube-scheduler
    participant K as kubelet on the node
    participant R as Container runtime
    participant C as CNI plugin
    U->>A: Submit Deployment
    A->>E: Save desired state
    Note over A,S: Controllers create Pod records with no node assigned
    S->>A: Watch for unassigned pods
    S->>A: Bind pod to the chosen node
    A->>E: Save node assignment
    K->>A: Watch for pods on my node
    K->>R: Create pod sandbox
    R->>C: Set up pod network
    K->>R: Pull image and start containers
    K->>A: Report status Running
    

The scheduler does the choosing, but it does not touch the node. It first filters out nodes that cannot host the pod, for instance those without enough allocatable memory or those carrying a taint the pod does not tolerate. Then it scores the survivors and picks the best. It records its decision by binding the pod to that node in the API server. From then on, the story belongs to the kubelet.

How a pod leaves

Shutdown is polite by design. When a pod is deleted, Kubernetes first removes it from Service lists so no new traffic arrives. The kubelet then sends the container a gentle stop signal, SIGTERM, giving the program time to finish current work and close connections. By default it waits 30 seconds. If the container is still alive after that, the kubelet forcibly kills it. Well-behaved applications listen for SIGTERM, finish in-flight requests, and exit cleanly, which is what makes rolling updates invisible to users.

Machines fail. Kubernetes is built on the assumption that they will, and it has a clear routine for noticing and recovering.

Each node keeps a list of conditions, which act like a health report card. The kubelet updates them and the control plane reads them.

Heartbeats

How does the control plane know a node is alive? The kubelet sends two kinds of heartbeat. The lightweight one is a Lease, a tiny record in a special namespace called kube-node-lease that the kubelet refreshes about every 10 seconds. The heavier one is the full node status, sent whenever something changes and at least every five minutes. Leases were introduced because thousands of large status updates were putting needless load on the API server and etcd.

What happens when a node goes silent

flowchart TB
    A["Node stops sending heartbeats"] --> B["Node controller waits about 40 seconds"]
    B --> C["Node marked NotReady or Unknown"]
    C --> D["Taints added to the node"]
    D --> E["Pods wait out a 5 minute default tolerance"]
    E --> F["Pods on the node are evicted"]
    F --> G["Controllers create replacement pods"]
    G --> H["Scheduler places them on healthy nodes"]
    

Follow the chain. After the grace period of about 40 seconds without a heartbeat, the node controller marks the node’s Ready condition as Unknown. It then adds taints, which are “keep away” labels, telling the cluster that the node is unreachable. Pods normally carry a built-in patience of 300 seconds, so they are not thrown out at the first hiccup. If the node does not return within that time, the pods are evicted. Their parent controllers notice that fewer copies are running than requested and create replacements, which the scheduler places on healthy nodes.

Many clouds add their own repair layer on top. An auto-scaling group or a managed node group can notice that a machine failed its health check, terminate it, and launch a fresh one that joins the cluster automatically. This is why treating nodes as replaceable, rather than precious, works so well.

Not every pod belongs on every node. Kubernetes gives you a small toolbox to steer pods toward or away from specific machines.

Real clusters are rarely made of identical machines. Some nodes have fast SSD disks, some have expensive GPUs, some run in a particular data center zone, and some are cheap short-lived spot machines. The following tools let you describe those differences and act on them.

Drain respects a safety device called a PodDisruptionBudget, which lets an application say “never take more than one of my copies down at a time.” It also skips DaemonSet pods, which are meant to run on every node anyway. Without these rules, a routine maintenance drain could accidentally remove every copy of a service at once.

The node model is powerful, but every design choice has a price. Knowing both sides helps you decide well.

The classic trade-off: big nodes or small nodes?

When you design a cluster you must pick machine sizes. There is no universal answer, only a balance.

A node holds real credentials, real data, and a shared kernel. Protecting it protects everything on it.

A node is only as safe as the weakest workload running on it.

Network rules matter too. By default, any pod can talk to any other pod. NetworkPolicy objects, enforced by the CNI plugin on each node, let you say which pods may talk to which, so that a breach in one place cannot spread freely. Finally, remember the shared-kernel point from Chapter 5: for code you do not trust, use a sandboxed runtime or put it on its own dedicated nodes.

You cannot fix what you cannot see. Nodes generate a steady stream of signals, and reading them early prevents late-night surprises.

What to watch

Container logs live as files on the node itself, under the /var/log/pods folder, and the kubelet rotates them, by default at 10 MiB per file with five files kept. That means logs vanish when a node is replaced. This is exactly why a log agent should copy them off the machine continuously. Alert on symptoms that affect users, such as rising error rates, and use node metrics to explain the cause.

Teams that run nodes well tend to repeat the same habits. Teams that struggle tend to repeat the same mistakes.

Patterns that work

Anti-patterns to avoid

A short checklist you can apply to almost any cluster, followed by the traps that catch newcomers most often.

Best practices

Common mistakes

These stories show how the ideas in this guide look at large scale.