If you only remember one paragraph from this article, make it the next one.

The control plane is the group of programs that manages a Kubernetes cluster. It stores the cluster’s records, accepts requests from people and tools, decides where applications should run, and constantly works to make the real state of the cluster match the state you asked for. It does not run your applications itself. Your applications run on other machines called worker nodes, and the control plane simply directs them.

The five parts in one glance

Why it has this name

The words “control plane” come from computer networking. In a network router, the control plane decides where traffic should go, while the data plane actually moves the traffic. Kubernetes borrowed the idea. The control plane decides and records, and the worker nodes do the real work of running containers. In older Kubernetes articles you will see the control plane machines called master nodes. The community moved away from that word, so today the correct name is control plane node.

Three common misunderstandings

“The control plane runs my containers.” It does not. Containers run on worker nodes, started by the kubelet. The control plane only decides and records.

“If the control plane stops, my website goes offline.” Usually not. Existing containers keep serving visitors. What you lose is the ability to change, heal, or scale anything until the brain returns.

“The control plane is one program.” It is a small team of programs, each with one clear responsibility. This separation is intentional, because small focused programs are easier to secure, scale, and replace than a single giant one.

Before opening the control plane, let us look at the whole building it lives in.

A container is a sealed package holding an application and everything it needs to run. Running one container is easy. Running thousands across many computers is hard, and that is the job of Kubernetes, a system that manages containers at scale. A Kubernetes installation is called a cluster, which means a group of computers cooperating as one. Each computer in the cluster is a node. Nodes come in two kinds: control plane nodes, which host the decision-making programs, and worker nodes, which host your applications inside small wrappers called pods.

flowchart TB
    ENG["Engineer using kubectl"] --> LB["Load Balancer in front of API servers"]
    CI["CI pipeline and other tools"] --> LB
    LB --> API
    subgraph CP["Control Plane"]
        API["kube-apiserver"]
        ETCD["etcd"]
        SCH["kube-scheduler"]
        KCM["kube-controller-manager"]
        CCM["cloud-controller-manager"]
        API  ETCD
        SCH --> API
        KCM --> API
        CCM --> API
    end
    subgraph W1["Worker Node 1"]
        KL1["kubelet"]
        PX1["kube-proxy"]
        POD1["Pods running your apps"]
    end
    subgraph W2["Worker Node 2"]
        KL2["kubelet"]
        PX2["kube-proxy"]
        POD2["Pods running your apps"]
    end
    KL1 --> API
    KL2 --> API
    KL1 --> POD1
    KL2 --> POD2
    CCM --> CLOUD["Cloud Provider API"]

Who does what

The two helpers that live on every worker node

Two small programs on each worker node deserve a mention, because they are the control plane’s hands and ears. The kubelet is an agent that introduces the node to the cluster, listens for pods assigned to it, tells the container runtime to start or stop containers, and sends back regular health reports. Without the kubelet, a node is just a silent computer as far as Kubernetes is concerned. The kube-proxy keeps network rules up to date so that traffic sent to a Service name reaches a healthy pod behind it.

Notice how these helpers relate to the control plane: they never make cluster-wide decisions. They obey the records stored by the control plane and report facts back. This one-way habit of “decisions flow down, facts flow up” is a good mental picture for the whole system. It also explains why a worker node can keep running its containers even when it temporarily loses contact with the brain.

Every part of the control plane is built on one simple but powerful idea. Once you see it, Kubernetes stops feeling like magic.

Say what you want, not what to do

In Kubernetes you do not give step-by-step orders. Instead you write a description of the situation you want, for example “three copies of my shopping website, each using this package, each allowed this much memory”. This description is called the desired state. The situation that really exists at this moment is called the actual state. The whole job of the control plane is to notice any gap between the two and close it.

The loop that never stops

The tool that closes the gap is the control loop, also called a reconciliation loop. It is a small program that repeats four steps forever.

Why this design is so tough

Because the loop compares states rather than reacting to single events, it is very forgiving. Imagine a message saying “a pod died” gets lost in the network. In an event-driven system, that pod would be forgotten forever. In Kubernetes, the next round of the loop simply notices that two pods exist while three are wanted, and it fixes the problem anyway. Engineers call this style level-based rather than edge-based. In plain words: the system trusts the current picture more than the news of changes.

Also notice that there is not one giant controller. There are many small ones, each responsible for one kind of object. One cares about copies of pods, another about nodes, another about batch jobs. Small focused loops are easier to build, test, and fix than one huge brain.

Every single conversation in Kubernetes passes through one program. Understanding it explains most of the system.

The API server, named kube-apiserver, is a web service that exposes the Kubernetes API, which stands for application programming interface, a set of agreed rules for how programs ask each other for things. Engineers reach it using a command-line tool named kubectl. Automated pipelines, dashboards, and even the other control plane parts all use the same door. Nobody is allowed to walk around the back. Not even the scheduler edits the database directly.

What the API server does

Useful facts about the API server

The API server is stateless, meaning it keeps no important data of its own because everything lives in etcd. This makes it easy to run several copies side by side behind a load balancer, which is how production clusters become highly available. It listens on port 6443 by default and always uses encrypted connections. The API is organised into groups and versions, such as the core group for pods and services and the apps group for Deployments, so that Kubernetes can improve features without breaking old tools. The server also offers health addresses named /livez and /readyz so that monitoring tools can ask whether it is alive and ready.

If the API server is the front door, etcd is the vault behind it. Losing it is the worst thing that can happen to a control plane.

etcd is a small, fast, distributed database that stores data as simple key and value pairs, like a giant labelled filing cabinet. Every object in Kubernetes, including every pod, Deployment, Service, Secret, and setting, is saved there. It first appeared in 2013 at a company called CoreOS and is now a project of the Cloud Native Computing Foundation. Kubernetes uses it as the one and only source of truth. If a fact is not in etcd, then as far as the cluster is concerned, it does not exist.

How etcd keeps everyone in agreement

Production clusters run several etcd copies, called members, on different machines so that one failure does not lose data. But copies can disagree, so etcd uses a voting method called Raft. In Raft, the members elect one leader. All changes go to the leader, who sends them to the others. A change is officially saved only when a majority, also called a quorum, confirms it. The formula for a majority is half the members plus one.

This table explains why the number of members is almost always odd. Adding a fourth member does not let you survive more failures than three, yet it makes the vote slower and costs more.

What etcd needs to stay healthy

Backup and restore in plain words

A snapshot is a frozen copy of everything in etcd at one moment. Taking one is quick and does not stop the cluster. Good teams take snapshots on a schedule, for example every hour, and copy them to a safe place outside the cluster, such as a storage bucket in a different region. Old snapshots are kept for a few days, so that even a mistake noticed late can be undone.

To restore, the team stops the control plane, loads the chosen snapshot into a fresh etcd, and starts the control plane again. The cluster then wakes up believing it is back at the moment of the snapshot. Any change made afterwards, such as a Deployment created ten minutes later, will be missing, so the team must re-apply it. Worker nodes and running containers are not stored in etcd, so they are simply reconciled against the restored records. Because restoring is stressful, engineers write a step-by-step checklist called a runbook and practise it before disaster ever strikes.

When a new pod is born, it has no home. The scheduler finds one.

When you create a pod through a Deployment, the record saved in etcd has an empty field for the node name. The scheduler watches for exactly such homeless pods. For each one, it works out which worker node is the best place to run it, and then writes that choice back through the API server. This decision is called binding. The scheduler does not start anything. It only decides. The kubelet on the chosen node then does the real work.

Two phases: filter, then score

What influences the decision

If no node passes the filter, the pod stays in the Pending state and the scheduler keeps trying as the cluster changes. Kubernetes can also use priority classes. A high-priority pod that cannot fit may cause the scheduler to evict lower-priority pods to make room, which is called preemption.

A worked example

Suppose your cluster has three worker nodes. Node one has plenty of free memory but sits in Zone A alongside two other copies of your shopping app. Node two has less free memory but sits in Zone B and runs no copy of the app. Node three is in Zone C with good free memory, but carries a taint reserved for machine learning jobs, and your pod has no matching toleration.

The scheduler first filters. Node three is removed at once because of the taint. Nodes one and two both have room, so both survive. Next comes scoring. Node one earns points for free memory, but loses points because putting a fourth copy in Zone A would leave the app poorly spread. Node two earns points for spreading across zones. The spread rule counts heavily, so node two wins and the pod is bound there. Because the decision follows clear rules, an engineer can always reconstruct why the scheduler chose as it did.

Real clusters are built with a flexible scheduling framework in which each rule is a plug-in. Companies with unusual needs, like batch computing or graphics processor sharing, replace or add plug-ins, or run a second custom scheduler alongside the default one. Pods choose which scheduler they trust by naming it in their description.

The control loops from Chapter C live here. This is where the real work of keeping the cluster tidy happens.

The kube-controller-manager is one program that bundles dozens of separate control loops, each called a controller. Packing them into one program simply makes them easier to deploy. Each controller watches one kind of object through the API server and reacts to gaps between desired and actual state.

Some controllers you should know

A closer look at the node controller

Every kubelet sends a heartbeat to the control plane regularly. If the node controller stops hearing from a node for about 40 seconds, which is the default grace period, it marks that node as unknown. Kubernetes then adds a taint to the node. By default, pods tolerate that taint for another five minutes before they are evicted, and only then does the ReplicaSet controller notice missing pods and create replacements elsewhere. These waiting times protect you from panic over a short network hiccup, and you can tune them for faster recovery.

The cloud-controller-manager

When Kubernetes runs on a cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure, some tasks need the cloud’s own tools. The cloud-controller-manager holds the cloud-specific controllers, so the main code stays neutral. Its node controller asks the cloud whether a machine that went silent has been deleted. Its service controller creates a real cloud load balancer when you ask for a Service of type LoadBalancer. Its route controller sets up network routes between nodes. On a laptop cluster or a bare-metal data centre, this component may not exist at all.

Now we connect all the parts. Let us follow a single request, “run three copies of my web app”, through the control plane.

sequenceDiagram
    participant E as Engineer with kubectl
    participant A as kube-apiserver
    participant D as etcd
    participant C as Controller Manager
    participant S as kube-scheduler
    participant K as kubelet on chosen node
    E->>A: Create Deployment with 3 replicas
    A->>A: Authenticate, authorize, run admission checks
    A->>D: Save Deployment
    A-->>E: Created
    C->>A: Watch sees new Deployment
    C->>A: Create ReplicaSet
    C->>A: Create 3 Pods with no node yet
    S->>A: Watch sees Pods without a node
    S->>A: Bind each Pod to a chosen node
    K->>A: Watch sees Pod assigned to me
    K->>K: Pull image and start containers
    K->>A: Report Pod status Running
    A->>D: Save updated status

Step by step in plain words

Inside the API server: the request pipeline

Every request to the API server passes through the same series of gates, like passengers at an airport moving through ticket check, security, and passport control. The order matters, and a failure at any gate stops the request.

flowchart LR
    R["Incoming request"] --> AU["Authentication: who are you?"]
    AU --> AZ["Authorization: are you allowed?"]
    AZ --> MA["Mutating admission: adjust the object"]
    MA --> VS["Schema validation"]
    VS --> VA["Validating admission: final policy check"]
    VA --> ST["Persist in etcd"]
    ST --> RS["Response and watch notifications"]
    AU -.->|"Fails"| X["Rejected with an error"]
    AZ -.->|"Fails"| X
    VA -.->|"Fails"| X

Admission controllers deserve a special mention. Mutating admission can change a request, for example by adding a default setting or a helper container. Validating admission can only accept or reject it, and it is where policies such as “no container may run as root” are enforced.

The other direction: what happens when something is deleted or breaks

The same loops handle unhappy events. Suppose a worker node loses power. Its kubelet stops sending heartbeats. After the grace period of about 40 seconds, the node controller marks the node unknown and adds a taint. After a further wait, the pods on that node are marked for eviction. The ReplicaSet controller now sees fewer pods than wanted and creates new pod records. The scheduler places them on healthy nodes, and their kubelets start the containers. No human touched anything, and the desired state of three copies is restored.

Deleting works in a similar polite way. When you delete a pod, the API server first marks it as terminating and gives it a grace period, thirty seconds by default. The kubelet sends the containers a gentle stop signal so they can finish current requests and close connections. Only if they refuse to leave in time are they forced to stop. Finally the pod record is removed from etcd. This graceful goodbye is why well-written applications lose no customer requests during updates.

How can dozens of programs stay in sync without constantly shouting at each other? The answer is a few clever tricks inside the API machinery.

Watch instead of ask

Imagine a hundred controllers each asking “has anything changed?” every second. The API server would drown. So Kubernetes offers a watch: a program opens one long-lived connection and says “tell me whenever pods change”. The API server then streams each change as it happens. Under the hood, etcd itself supports watching, and the API server passes those events along after its own checks.

Versions and local caches

Every object carries a counter called the resource version, which increases whenever the object changes. Controllers use it to say “start telling me about changes after version 4,502”. If a controller was disconnected for a moment, it can catch up from where it left off rather than starting from scratch. Most controllers also keep a local copy of the objects they care about, called a cache, kept fresh by a helper called an informer. Reading from the cache is instant and puts no burden on the API server. Work items are placed in a work queue, so a controller handles them one by one, retries failures, and slows down when errors repeat.

Preventing two writers from clashing

What if two people edit the same object at the same time? Kubernetes uses optimistic concurrency. Each update must include the resource version it was based on. If the object has changed since, the API server rejects the update with a conflict message, and the caller reads the fresh copy and tries again. No locks are needed, and nobody silently overwrites someone else’s work.

A brain that can fail is a risky brain. Production clusters therefore run more than one copy of everything.

The shape of a highly available control plane

A production control plane usually has three control plane nodes, sometimes five. Each runs its own API server, scheduler, and controller manager. A load balancer sits in front of the API servers and sends each request to a healthy one. Because the API server is stateless, all copies are equally active. Scheduler and controller manager copies are running too, but only the elected leader of each does work at any moment.

flowchart TB
    USERS["kubectl, CI tools, and worker node kubelets"] --> LB["Load Balancer virtual address"]
    subgraph CPN1["Control Plane Node 1 in Zone A"]
        A1["kube-apiserver"]
        S1["scheduler standby"]
        M1["controller manager LEADER"]
        E1["etcd member"]
    end
    subgraph CPN2["Control Plane Node 2 in Zone B"]
        A2["kube-apiserver"]
        S2["scheduler LEADER"]
        M2["controller manager standby"]
        E2["etcd member"]
    end
    subgraph CPN3["Control Plane Node 3 in Zone C"]
        A3["kube-apiserver"]
        S3["scheduler standby"]
        M3["controller manager standby"]
        E3["etcd member"]
    end
    LB --> A1
    LB --> A2
    LB --> A3
    A1 --> E1
    A2 --> E2
    A3 --> E3
    E1  E2
    E2  E3
    E1  E3

Stacked versus external etcd

What happens when the control plane fails?

This is one of the most surprising facts for beginners: your running applications usually keep running. The kubelet on each worker node already knows which pods it should run, and the containers themselves do not need the control plane to serve visitors. Network rules already programmed on the nodes also keep working. But everything that needs a decision stops.

Whoever controls the control plane controls every application in the cluster. Protecting it is the first duty of any platform team.

Key protections

Network exposure and certificates

Permissions deserve a closer look. RBAC works with four simple ideas. A role lists actions, such as “read pods in the shop area”. A binding attaches that role to a person, a group, or a program. A service account is the identity a program running inside a pod uses when it calls the API server. Every action is checked against these rules, and anything not allowed is refused by default. Good teams start with narrow roles, for example “can read pods but not secrets”, and review the list of powerful roles regularly.

The API server should be reachable only from trusted networks. Many teams place it on a private address and reach it through a secure tunnel or a bastion machine. The etcd port must never be open to the wider network. Certificates also expire. On clusters created with the common kubeadm tool, the certificates are valid for one year by default and are renewed when you upgrade. An unnoticed expiry can suddenly stop the cluster from talking to itself, so put a reminder in the calendar or use automation.

A cautionary tale

In 2018 security researchers discovered that a Kubernetes administration console belonging to Tesla was open to the internet with no password. Attackers had used that door to run hidden cryptocurrency mining inside the company’s cloud account. The lesson is simple: management doors must never face the public internet without strong authentication, because they lead straight to the control plane.

You cannot fix what you cannot see. The control plane politely publishes plenty of information about its own health.

Where the information comes from

Each control plane program exposes a /metrics address in a format that Prometheus, the popular monitoring system, can collect. The API server also has health addresses named /livez and /readyz. Programs write logs to standard output, and on control plane nodes these are collected by a log agent and shipped to a central store. In managed services, the provider often lets you switch on control plane logs, delivered to a service such as Amazon CloudWatch.

The numbers worth watching

flowchart LR
    API["kube-apiserver metrics and audit log"] --> PR["Prometheus"]
    ETCD["etcd metrics"] --> PR
    SCH["scheduler metrics"] --> PR
    KCM["controller manager metrics"] --> PR
    API --> LA["Log agent on control plane node"]
    ETCD --> LA
    LA --> LS["Central log store"]
    PR --> GF["Grafana dashboards"]
    PR --> AM["Alertmanager"]
    AM --> OC["On-call engineer"]
    LS --> INV["Investigation and audit review"]

A helpful starting rule comes from the four golden signals in Google’s well-known reliability book: latency (how long requests take), traffic (how many arrive), errors (how many fail), and saturation (how full the resources are). Apply them to the API server first, then add etcd disk speed, and you already cover most early warnings.

The control plane is not a closed box. Its greatest strength is that you can teach it new tricks using the same rules it uses for itself.

Custom resources and operators

Kubernetes ships with objects such as pods and Deployments. But suppose your company runs a database and wishes it could say “give me a database with three replicas and a daily backup” and have the cluster handle everything. You can add a brand-new kind of object by registering a CustomResourceDefinition, usually shortened to CRD. The API server immediately learns to accept, check, and store that new object type, exactly as it does for built-in ones.

A CRD alone is only a form to fill in. To make something happen, you write a custom controller that watches those objects and runs a control loop, just like the built-in controllers. A custom controller that carries the knowledge of a human expert, such as a database administrator, is called an operator. The idea was introduced by CoreOS in 2016, and today many tools, including cert-manager for certificates and Prometheus Operator for monitoring, work this way.

Other useful patterns

Anti-patterns to avoid

These habits come from real outages and long nights. Copy them early and you will skip the worst of the pain.

Habits that keep a control plane healthy

Common mistakes and their fixes

You do not have to build and run this brain yourself. Many teams rent it.

Two ways to get a control plane

The three big managed services are Amazon EKS, Google Kubernetes Engine, and Azure Kubernetes Service. In each, the provider hides the control plane nodes from you. You see only the API address and your worker nodes. Amazon says EKS runs the control plane across multiple availability zones, and Google’s regional clusters likewise replicate the control plane across zones. In exchange, you lose direct access to etcd and to some control plane settings, and you usually pay a small hourly fee per cluster.

Stories from the real world

How big should a control plane be?

Size depends on how many nodes, pods, and objects the cluster holds, and on how busy the clients are. A small cluster of a few dozen nodes runs happily on three modest control plane machines. As the number of nodes and objects grows into the thousands, the API server needs more memory and processor time, and etcd needs faster disks and more memory to keep its data cached. Watching the metrics from Chapter L is the reliable way to know when to grow, rather than guessing in advance.

It also helps to remember that the count of objects matters as much as the count of machines. A cluster with a few nodes but hundreds of thousands of stored records, from leftover jobs or too many custom resources, can strain etcd more than a large cluster with tidy records. Regular clean-up is a cheap way to delay costly upgrades of the control plane hardware.

Short, plain answers to the questions beginners ask most.

Back to the airport tower for one last look.

The control plane is the tower of a Kubernetes airport. It takes requests at the front desk, records every decision in a reliable ledger, matches each new pod to a suitable machine, and runs a crowd of small watchers that never stop comparing what you wanted with what exists. It does not carry passengers, but nothing moves in an organised way without it. Understand these five parts and the simple loop that binds them, and the rest of Kubernetes becomes far easier to learn.