What Is the Controller Manager in Kubernetes?
Kubernetes keeps promises: if you ask for three copies of a program, you keep getting three. The Controller Manager is the part of the cluster that keeps those promises. This guide explains, from zero, what it is, what lives inside it, and how it quietly repairs your cluster all day long.
Think about a city’s maintenance department. Nobody in the city asks for the streetlights to be checked every night, yet they are. Broken bulbs are replaced, fallen trees are cleared, and overflowing bins are emptied. Different teams look after different jobs, but they all follow one idea: the city should look the way the rules say it should, and any difference must be fixed.
Kubernetes has an equivalent department, and it is called the Controller Manager. It does not make headlines and you rarely see it, but almost every automatic thing Kubernetes does, from replacing a crashed Pod to cleaning up after a deleted application, traces back to it. This article explains it slowly, in plain words, with everyday pictures to help each idea stick.
1The Problem: Who Keeps the Promises?
Writing down a goal is easy; making the world match it, forever, is the hard part.
When you use Kubernetes, you describe what you want. For example, you might say, “I want three copies of my shopping program running at all times.” Kubernetes stores that wish carefully. But a stored wish does nothing by itself. Somebody must notice whether three copies really exist, start new ones if only two remain, and remove extras if there are four.
Computers fail, programs crash, people delete things by accident, and demand changes minute by minute. A one-time action is never enough. What is needed is a worker that keeps watching and keeps correcting, without ever getting bored or tired.
Imagine a swimming pool with a rule that the water must stay between 26 and 28 degrees. Writing the rule on a sign does not heat the water. You need a device that measures the temperature all day and switches the heater on or off as needed. In Kubernetes, the device that measures and corrects is a controller, and the box that holds many controllers is the Controller Manager.
So the honest answer to “who keeps the promises?” is: the controllers, and the Controller Manager is the home where they live and run.
2A Quick Tour of Kubernetes First
You cannot understand one gear without seeing the clock.
Kubernetes runs programs packed in containers across a group of computers called a cluster. Each computer in the cluster is called a node. The smallest unit Kubernetes runs is a Pod, a small wrapper around one or more containers that always live and die together.
The cluster is split into two halves. The worker nodes are where your Pods actually run. The control plane is the thinking half, the brain of the cluster, which makes decisions about what should happen. The Controller Manager lives in the control plane, alongside a few other important parts.
API server
The front door of the cluster. Every request, from people and from other parts of the system, passes through it.
State store
A very reliable database, usually etcd, that remembers everything the cluster knows and wants.
Scheduler
Picks which node each new Pod should live on, like a hotel clerk assigning rooms.
Controller Manager
Runs the controllers that keep the real cluster matching the wishes stored in the state store.
Kubelet
The agent on each worker node that actually starts and watches the containers of Pods assigned there.
The API server is the only part that talks directly to the state store. Everything else, including the Controller Manager, does its work by talking to the API server.
3So, What Is the Controller Manager?
The short version: one program that runs many small watchers.
The Controller Manager, whose real program name is kube-controller-manager, is a control plane component that runs the core controllers of Kubernetes. A controller is a loop that watches part of the cluster and takes action to bring the real situation in line with the desired situation. There are many kinds of controllers, each with its own narrow job.
Logically, every controller is a separate program with a separate purpose. To keep things simple to run, Kubernetes bundles the core controllers together and runs them as one single process. That bundle is the Controller Manager. It is like a building that houses many small departments under one roof, sharing the same entrance and electricity.
Think of a shopping mall. Inside there is a pharmacy, a bakery, a bank and a tailor, each running its own business. But they share one building, one security desk and one parking area. The mall is the Controller Manager, and the shops are the individual controllers.
What it does not do
Borders matter, so let us draw them clearly. The Controller Manager does not choose which node a Pod runs on; that is the scheduler’s job. It does not start containers; that is the kubelet’s job. It does not store anything permanently; that belongs to the state store. Its job is to notice differences and to create, update or delete objects through the API server so that the difference disappears.
Some beginners imagine the Controller Manager as one giant controller. It is really a host for many small, independent controllers. Each one cares about a single type of thing, such as nodes, Deployments or Jobs.
4Core Concepts and Vocabulary
Ten simple words that unlock every controller conversation.
Before we open the boxes, here are the words you will meet again and again. They are simpler than they look.
Controller
A loop that watches some objects and acts to fix any difference between wanted and actual.
Desired state
What you asked for, such as “three copies”. It is the goal written in the cluster’s memory.
Current state
What is really happening right now. The controller compares this with the desired state all the time.
Reconciliation
The act of closing the gap between current and desired state, over and over again.
Resource or object
Any thing Kubernetes stores and manages, such as a Pod, a Deployment, a Node or a Service.
Watch
A live connection to the API server that announces changes as they happen, so nobody has to keep asking.
Work queue
A to-do list inside each controller of objects that need attention, handled one by one.
Owner reference
A note saying “this object was created by that object”, forming a family tree used for cleanup.
Leader election
A way for several copies of the Controller Manager to agree that only one of them is in charge at a time.
Replica
One identical copy of a Pod. “Three replicas” means three identical copies sharing the work.
If you keep only two words, keep desired state and current state. Almost every controller is just a different way of comparing those two and acting on the difference.
5Where It Sits in the Architecture
A working relationship with the API server, and with nobody else.
The Controller Manager lives on the control plane computers. It has a single main partner: the API server. It reads the current situation from the API server, and when it needs a change, it asks the API server to make it. It does not reach out to individual nodes or kubelets directly. This keeps the system tidy, because every change goes through the same front door and leaves the same records.
flowchart TD
U["You: write down what you want"] --> API["API server: the front door"]
API STORE[("State store: the memory")]
CM["Controller Manager: many controllers"] API
SCH["Scheduler: picks a node for each Pod"] API
API KL1["Kubelet on Node 1"]
API KL2["Kubelet on Node 2"]
KL1 --> P1["Pods on Node 1"]
KL2 --> P2["Pods on Node 2"]
Imagine a company where every request must be written on a form and handed to one reception desk. The maintenance team reads the forms, notices what is missing, and files new forms asking for repairs. The scheduler files forms about room assignments. Nobody whispers instructions in corridors. The reception desk is the API server, and the maintenance team is the Controller Manager.
On many self-built clusters, the Controller Manager runs as a special kind of Pod managed directly by the kubelet on the control plane computers. On managed cloud services, the provider runs it for you and you never see it.
6The Heart of It: The Control Loop
Watch, compare, act, repeat.
Every controller, no matter its job, follows the same simple rhythm. First it watches the objects it cares about. Second it compares what it sees with what is wanted. Third, if there is a difference, it acts to reduce the difference. Then it goes back to watching. This is called a control loop, and it never finishes.
A home thermostat is the classic example. You set the wanted temperature. The thermostat keeps measuring the room. If it is too cold, it turns the heater on. If it is warm enough, it turns it off. It does not remember the past or worry about why the temperature changed. It only looks at the gap between now and wanted.
Level-based, not event-based
Here is a subtle but wonderful design choice. Controllers do not rely only on being told “something changed”. They look at the whole current picture and compare it with the goal. If a notification is lost or delayed, nothing breaks, because the next comparison will notice the gap anyway. Engineers call this level-triggered behaviour, and it makes the system resilient to missed messages.
Notice a change
The controller hears from the API server that something it cares about was created, changed or deleted.
Add it to the work queue
The affected object is put on the controller’s to-do list so nothing is forgotten, even during a busy moment.
Compare wanted and actual
When its turn comes, the controller reads the goal and the reality for that object from its local memory.
Act through the API server
If the two differ, the controller creates, updates or deletes objects by sending requests to the API server.
Report and repeat
The controller records what it did in the object’s status, and the loop starts again with the next change or the next scheduled check.
Good controllers are written so that doing their job twice is harmless. If the goal is already met, a second pass simply finds nothing to do. This property makes retries and restarts safe.
7Inside a Single Controller
A small machine made of three simple pieces.
Although each controller has a different purpose, they are built from the same three ingredients. Knowing them makes any controller easy to understand.
Informer
The watcher. It keeps a live connection to the API server and receives announcements of changes as they occur.
Local cache
A private, up-to-date copy of the objects the controller cares about, so it can read quickly without bothering the API server each time.
Work queue and workers
A to-do list of objects that need attention and a few workers that pick them up, one at a time, and run the compare-and-fix step.
Picture a librarian who keeps a small notebook of the books on their shelf. A doorbell rings whenever a book is added or removed, and the librarian notes it in the notebook. When there is a free moment, the librarian works through a list of shelves to tidy. The doorbell is the informer, the notebook is the cache, and the list of shelves is the work queue.
Why the cache is so important
A cluster can hold hundreds of thousands of objects. If every controller asked the API server for fresh details every time, the front door would be swamped. By keeping a local, constantly updated copy, controllers make almost all their reads for free and only speak to the API server when they need to change something.
Being polite when things go wrong
If an attempt to fix something fails, for example because another change happened at the same moment, the controller puts the object back on the queue and tries again after a short wait. The wait grows if failures continue, which prevents a struggling controller from hammering the system. This patient retrying is called backoff.
Several parts of the cluster can touch the same object. Kubernetes uses version numbers on objects, so an update based on an outdated picture is rejected. The controller then reads the fresh version and tries again. This is safe and expected, not an error to fear.
8Meet the Built-in Controllers
A tour of the most important tenants of the mall.
The Controller Manager hosts many controllers. You do not need to memorise them all. Here are the ones you will meet most often, each with its everyday picture.
Deployment controller
Manages rollouts and updates of your applications. It works by creating and adjusting ReplicaSets, like a project manager directing team leaders.
ReplicaSet controller
Keeps the right number of identical Pods alive, like a lifeguard who makes sure the correct number of swimmers is always visible.
StatefulSet controller
Runs Pods that need stable names and lasting storage, like assigning permanent named lockers to specific staff.
DaemonSet controller
Makes sure one copy of a helper runs on every node, like placing a fire extinguisher on every floor.
Job controller
Runs tasks that must finish, and tracks how many have succeeded, like a teacher counting completed homework.
CronJob controller
Starts Jobs on a schedule, like an alarm clock that launches a task every morning.
Node controller
Watches node health and reacts when a node stops reporting, like a receptionist noticing an employee who never arrived.
EndpointSlice controller
Keeps a live list of which Pods stand behind each Service, like a phone directory that updates itself.
Namespace controller
Cleans out everything inside a namespace when it is deleted, like clearing a room after the guest checks out.
Garbage collector
Removes objects whose owners are gone, like clearing away leftovers when the party is over.
Persistent volume controllers
Match storage requests with available disks and manage their life, like a housing office matching tenants to flats.
Autoscaling and quota controllers
Adjust replica counts to match demand and enforce usage limits, like a manager who adds cashiers at rush hour and keeps departments within budget.
The list also includes controllers for service accounts, certificates, cleaning up finished Pods and Jobs, and more. The pattern is always the same: watch a kind of object and fix the difference.
9Controllers Working as a Team: The Deployment Chain
Small, simple jobs combine into powerful behaviour.
One of the loveliest things about controllers is that none of them needs to understand the whole picture. Each does its narrow job, and the combined result feels intelligent. The best example is what happens when you create a Deployment.
You tell Kubernetes you want an application with three copies. The Deployment controller notices the new Deployment and creates a ReplicaSet for it. The ReplicaSet controller notices the new ReplicaSet and sees it should have three Pods, but has none, so it creates three Pod objects. At this point the scheduler notices Pods without a home and assigns each one to a node. Finally, the kubelet on each chosen node starts the containers.
sequenceDiagram
participant You
participant API as API server
participant DC as Deployment controller
participant RC as ReplicaSet controller
participant SCH as Scheduler
participant KL as Kubelet
You->>API: Create a Deployment with 3 copies
API-->>DC: New Deployment noticed
DC->>API: Create a ReplicaSet
API-->>RC: New ReplicaSet noticed
RC->>API: Create 3 Pods
API-->>SCH: Pods without a node noticed
SCH->>API: Assign each Pod to a node
API-->>KL: Pod assigned to your node
KL->>KL: Start the containers
Think of building a house. The architect approves the plan, the site manager orders the workers, the workers request materials, the delivery team brings them, and the bricklayers lay the bricks. Nobody controls every step, yet the house appears. Each person reacts to what the previous person produced.
Why this design is so strong
Because each controller reacts only to the state of objects, they do not need to call each other directly. You can add a new controller without changing the old ones. If one controller is briefly unavailable, the others keep doing their jobs, and the chain resumes when it returns.
Each object in the chain carries an owner reference to the one that created it: Pods point to their ReplicaSet, and ReplicaSets point to their Deployment. This family tree lets the cluster clean up automatically and lets controllers know which Pods are theirs.
10Self-Healing in Action
The moment when the design proves its value.
Let us see what happens when things break. Suppose one of your three Pods disappears, perhaps because its node failed. The ReplicaSet controller is watching, and it notices that only two Pods exist while the goal is three. It creates a new Pod object. The scheduler places it on a healthy node, and the kubelet starts it. Nobody had to wake up, log in or type a command.
The same principle repairs many situations. If someone deletes a Pod by mistake, it returns. If someone changes the number of copies in the goal, the controller adjusts reality to match. If a rollout is misconfigured, the Deployment controller can pause it or revert to an earlier version.
A crashed Pod
The kubelet restarts failed containers inside the Pod. If the whole Pod is lost, the ReplicaSet controller creates a fresh one elsewhere.
A dead node
The node controller notices missing heartbeats and marks the node unhealthy. Pods on it are eventually removed, and their controllers create replacements on healthy nodes.
An accidental deletion
Someone removes a Pod that belongs to a Deployment. The controller sees the gap between wanted and actual and simply makes a new one.
A changed goal
You raise the replicas from three to six. Controllers create three more Pods. Lower it again, and extras are removed.
Controllers repair the machinery, not the mistakes in your program. If a new version crashes as soon as it starts, controllers will keep replacing it faithfully, but it will never work. Testing and health checks are still needed.
11The Node Controller and Failing Computers
How Kubernetes decides that a computer has gone missing.
One of the most important controllers deals with the health of nodes. Every kubelet regularly sends a small “I am alive” message, called a heartbeat, to the API server. The node controller inside the Controller Manager keeps an eye on these heartbeats.
If heartbeats stop for longer than a grace period, the node controller marks the node as unhealthy, showing it as Unknown or NotReady. It also places a special mark, called a taint, on the node, which tells the cluster that new Pods should avoid it and that existing Pods should leave after a waiting time. Pods that cannot leave in time are removed, and their controllers create replacements elsewhere.
flowchart TD
HB["Kubelet sends heartbeats"] --> OK["Node controller: node looks healthy"]
HB -. "heartbeats stop" .-> LATE["Grace period passes"]
LATE --> NR["Node marked NotReady or Unknown"]
NR --> TAINT["Node receives a taint: keep new Pods away"]
TAINT --> WAIT["Pods wait for a tolerated period"]
WAIT --> EVICT["Pods are removed from the node"]
EVICT --> REPL["ReplicaSet controller creates replacements on healthy nodes"]
Imagine a mountain expedition that checks in with base camp every hour. If one team stays silent, base camp does not panic at the first missed call. It waits a reasonable time, then marks the team as missing, stops sending new supplies their way, and finally sends a replacement team along another route.
The waiting periods exist so a brief network hiccup does not trigger a storm of unnecessary Pod movements. They can be tuned, but very short waits risk unnecessary disruption.
12Cleaning Up: Garbage Collection and Finalizers
Good housekeeping is part of the promise.
Creating things is only half the story. When you delete an application, all the things created for it must go too, or the cluster would slowly fill with forgotten leftovers. The garbage collector controller uses the owner references we met earlier. When an owner disappears, the objects it owned are removed as well.
You can choose how deletion behaves. In one style, the owner is removed at once and the leftovers are cleaned up in the background. In another, the owner waits until everything it owns has been removed first. A third option leaves the owned objects in place and simply cuts the ownership link.
Finalizers: a note that says “not yet”
Sometimes an object must not vanish until some cleanup is finished, such as releasing a cloud disk or a load balancer. Kubernetes handles this with finalizers. A finalizer is a small marker on the object that means “someone still needs to do something before this can be deleted.” The responsible controller performs the cleanup, removes its marker, and only then does the object disappear.
When you move out of a rented flat, the landlord does not hand back your deposit immediately. There is a checklist: return the keys, check the walls, settle the final bill. Only when each item is ticked off does the tenancy officially end. Finalizers are those tick-boxes.
If an object stays in a deleting state for a long time, a finalizer is often waiting for a controller that is missing or unhealthy. The right fix is to restore that controller, not to strip the marker without understanding why it is there.
13High Availability and Leader Election
Two chefs shouting orders in one kitchen would ruin the meal.
Production clusters run several copies of the control plane so that a single failure does not stop the cluster from healing itself. But there is a puzzle: if three copies of the Controller Manager all acted at once, they would fight, creating duplicate Pods and deleting each other’s work.
Kubernetes solves this with leader election. All the copies start, but they compete for a small shared record in the cluster, called a lease. The copy that holds the lease is the leader and does all the work. The others wait quietly as standbys and keep trying to take the lease. The leader must renew it regularly. If the leader dies and stops renewing, one of the standbys takes over within seconds.
| Situation | What happens | Result |
|---|---|---|
| Three copies start together | One wins the lease; two wait | Only one set of controllers acts |
| The leader crashes | Lease expires; a standby takes it | Healing resumes within moments |
| The leader is slow or cut off | It cannot renew, so it steps down | Avoids two leaders acting at once |
| Only one copy exists | It is the leader by default | Works, but no backup if it fails |
Picture a ship with three officers trained to steer. Only the one holding the ship’s wheel gives orders. If that officer falls ill, the wheel is passed to another. The wheel is the lease, and passing it smoothly prevents the ship from being pulled in two directions.
Although standby copies do no work most of the time, they are the reason your cluster keeps healing itself while a control plane computer is being upgraded or repaired.
14The Cloud Controller Manager
A cousin that deals with the outside world of cloud providers.
Kubernetes runs on many different clouds and data centres. Some tasks depend on which provider you use, such as creating a load balancer, attaching a cloud disk, or discovering the details of a virtual machine. Early on, this provider-specific code lived inside the main Kubernetes program, which made it heavy and tied releases together.
To fix this, Kubernetes separated the cloud-specific controllers into their own component called the Cloud Controller Manager. Cloud providers maintain their own version of it. The regular Controller Manager keeps the general, provider-independent controllers, while the Cloud Controller Manager handles work like checking whether a node still exists in the cloud, setting up network routes, and creating cloud load balancers for Services.
| Feature | Kube Controller Manager | Cloud Controller Manager |
|---|---|---|
| Purpose | Core, provider-independent controllers | Controllers that talk to a cloud provider |
| Examples | Deployments, ReplicaSets, Jobs, namespaces | Node existence in the cloud, load balancers, routes |
| Maintained by | The Kubernetes project | The cloud provider, following a shared framework |
| Needed on your own hardware | Yes | Usually not, unless you integrate with a provider |
A company has a general facilities team for cleaning and lighting, and a separate liaison who deals with the landlord of the building, arranging parking and utilities. The liaison knows the landlord’s particular rules. The Cloud Controller Manager is that liaison.
15Custom Controllers and Operators
The same pattern, extended to things Kubernetes never knew about.
The control loop idea is so useful that Kubernetes lets anyone build their own controllers. First, Kubernetes can be taught new kinds of objects, called custom resources. For example, a team could invent an object type called “Database”. Then a custom controller watches those objects and does whatever is needed to make them real, such as creating storage, starting Pods, and arranging backups.
When a custom controller captures the expert knowledge of running a particular piece of software, such as a database or a message system, it is often called an operator. The operator behaves like a skilled human administrator who never sleeps: it knows how to install, upgrade, back up and repair the software it manages.
A database operator
You ask for a database with three members. The operator creates the storage, starts the members in the right order, handles failover if one dies, and takes scheduled backups.
A certificate operator
You declare which website needs a security certificate. The operator requests it, installs it, and renews it before it expires.
A policy or platform controller
A platform team defines a simple “application” object. A controller expands it into all the Deployments, Services and settings that the team’s standards require.
Custom controllers usually run as ordinary Pods in the cluster, separate from the built-in Controller Manager. They follow the same loop and use the same API server, but they are installed and updated on their own.
16Security Around the Controller Manager
Because it can create and delete almost anything, it must be guarded carefully.
The Controller Manager holds considerable power. It can create Pods, remove objects and read some sensitive information in order to do its work. That power makes it an attractive target, so a few simple layers of protection are essential.
Least privilege identities
Each controller can use its own identity with only the permissions its job needs, so one flaw cannot open every door.
Secure connections
Traffic between the Controller Manager and the API server is encrypted and authenticated with certificates.
Protected endpoints
Its health and metrics endpoints should be reachable only by trusted tools, and require authentication.
Locked-down control plane
The computers that run the control plane should be tightly firewalled, with limited login access and careful patching.
Audit trails
Every action goes through the API server, which can record who did what, so unexpected changes can be traced.
Protected keys
The keys used to sign service account tokens and certificates need special care, since anyone holding them can impersonate others.
In a large company, the maintenance team has keys to many rooms. Sensible companies give each team member only the keys their job requires, log every use of a master key, and keep the key cabinet in a locked office. The same instincts protect a cluster.
Giving every application in the cluster wide administrator rights makes the controllers’ careful permissions meaningless. Follow the rule of fewest permissions everywhere.
17Monitoring, Logging and Metrics
If the healer is unwell, nothing else gets healed.
Because so much automation depends on the Controller Manager, watching its health is important. If it stops or slows down, your cluster will still run existing Pods, but it will stop repairing, scaling and cleaning up. Problems can stay hidden for a long time unless you are watching.
Metrics
The Controller Manager publishes numbers about its work queues, such as how long items wait and how many retries occur.
Logs
Its written diary records which controller did what and why. This is essential when a rollout stalls or a Pod never appears.
Events
Short messages attached to objects, such as “created Pod” or “scaled up the replica set”, that tell the story of what controllers have done.
Signals worth watching first
Beginners can start with a few simple questions. Is the leader lease being renewed regularly? Are the work queues short, or are items waiting a long time? Are there frequent errors talking to the API server? Are Deployments reaching their wanted number of ready Pods in a reasonable time?
A hospital monitors its own emergency generators as carefully as its patients, because if the generators fail, everything else is at risk. Watching the Controller Manager is like testing those generators regularly.
When something is not happening, the events on the affected object often give the fastest clue. They usually name the controller involved and say what it tried to do.
18Reading the Signs: Common Situations
What you see, and which controller is probably involved.
You can often work out which part of the cluster is responsible just by looking at the symptom. Here are some common situations and how to think about them.
| What you see | Plain meaning | Where to look |
|---|---|---|
| Deployment shows fewer ready Pods than wanted | Pods are being created but are not becoming ready | Pod events, readiness checks, resources on nodes |
| No new Pods appear at all | The controller chain is not creating them | Controller Manager health, leader lease, logs |
| Pods stay Pending | Pods exist, but no node has been chosen | Scheduler and available capacity |
| A namespace stays in Terminating | Something inside it cannot be cleaned up | Finalizers and the controllers they wait for |
| Old Pods remain after a rollout | Update is waiting or has been paused | Deployment status and rollout settings |
| Pods on a dead node linger | Waiting periods have not yet expired | Node controller settings and taints |
Think of a doctor listening to symptoms. A fever points one way and a cough another. The skill is not in memorising every disease but in knowing which body system each symptom belongs to. In the same way, a stuck Pod tells you whether to look at controllers, the scheduler or the kubelet.
Walk from the top of the family tree downwards: Deployment, then ReplicaSet, then Pod, then node. The first place where reality stops matching the goal is where the trouble starts.
19Advantages and Trade-offs of the Controller Design
A brilliant idea, with a few things to respect.
Building a system from many small control loops shapes almost everything about Kubernetes. It has clear strengths and a few costs worth knowing about.
Advantages
- Automatic self-healing without human effort
- Small, focused controllers are easier to build, test and understand
- Loose coupling means parts can fail and recover independently
- Level-based comparison survives missed messages and restarts
- Extensible: anyone can add new controllers for new kinds of objects
- Leader election gives high availability without duplicate work
Disadvantages / Trade-offs
- Eventual, not instant, results: the cluster needs a moment to catch up
- Chains of controllers can be hard to trace when something goes wrong
- Two controllers fighting over the same object can cause endless flapping
- A busy or unhealthy Controller Manager slows down the whole cluster
- Waiting periods for failed nodes mean recovery is not immediate
“Eventual” is a friendly word here. In normal conditions the gap closes in moments, but it is good to remember that Kubernetes promises to get there, not that it is already there at the instant you look.
20Patterns to Follow and Anti-patterns to Avoid
Lessons other teams already paid for.
Helpful patterns
Declare the goal, then let controllers work
Describe what you want in written files and apply them. Let the controllers work out the steps instead of doing them by hand.
Use higher-level objects
Create Deployments rather than bare Pods. A bare Pod has no controller watching over it, so if it dies, nobody replaces it.
Run several control plane copies
Keep more than one Controller Manager with leader election so healing continues during maintenance and failures.
Keep goals in version control
Written goals reviewed by colleagues give you history, accountability and an easy way to undo mistakes.
Anti-patterns
Problem
Manually editing or deleting Pods that belong to a Deployment or ReplicaSet to fix a problem.
Why It’s Harmful
The controller sees a gap and immediately recreates or reverts what you changed. Your effort is undone, and confusion grows.
Correct Approach
Change the goal on the owning object, such as the Deployment, and let the chain of controllers carry it out.
Problem
Two separate tools or scripts manage the same object at the same time, for example a controller and a person running automated jobs on the same replica count.
Why It’s Harmful
Each keeps undoing the other’s changes, causing constant flapping, wasted resources and hard-to-explain behaviour.
Correct Approach
Give each object a single clear owner. If autoscaling manages replicas, stop setting the count by hand.
Problem
Removing finalizers by force to make stuck objects disappear, without checking why cleanup did not finish.
Why It’s Harmful
The external resources the finalizer protected, such as cloud disks or load balancers, may be left behind, costing money and creating security risks.
Correct Approach
Find out which controller should complete the cleanup, fix or restore it, and let the process finish naturally.
21Best Practices and Beginner Mistakes
Small habits that keep the healer healthy.
Prefer managed control planes when possible
Cloud providers run and upgrade the Controller Manager for you, removing a lot of operational work for beginners and small teams.
Keep control plane versions consistent
Upgrade the components in the supported order and stay within the allowed version differences to avoid surprising behaviour.
Watch the leader lease and work queues
These two signals reveal problems early, long before users notice missing repairs.
Set health checks and resource settings on Pods
Controllers can only repair what they can measure. Clear readiness signals let rollouts proceed safely.
Tune waiting periods thoughtfully
Node failure timings balance fast recovery against needless disruption. Change them only with a clear reason and testing.
Practise failure on purpose
Deliberately stop a node or a control plane copy in a safe environment, and watch the controllers repair things. Confidence comes from seeing it work.
A Pod created on its own has no controller responsible for replacing it. If its node dies, it is simply gone. Always wrap important Pods in a controller-managed object.
After a change, controllers need a little time to react. Refresh and wait a moment before assuming something is broken.
A huge cluster with an undersized control plane leads to long queues and slow repairs. Size and monitor it as you would any critical service.
22Real-World Examples
Where controllers quietly protect services you use every day.
You never see the Controller Manager, but on any Kubernetes-based service it is doing steady work. Here are everyday pictures of what that looks like.
An online store during a sale
Traffic climbs, so an autoscaling controller raises the wanted number of copies. The ReplicaSet controller creates the extra Pods, and the scheduler and kubelets bring them to life. When the sale ends, the process runs in reverse.
A streaming service after a hardware failure
A computer in the data centre dies during the evening peak. The node controller notices, and controllers create replacement Pods on healthy computers. Most viewers never notice.
A bank releasing a new version
The Deployment controller moves traffic to the new version step by step, keeping the old version until the new one proves healthy. If problems appear, it can roll back to the previous version.
A nightly report
A CronJob controller starts a Job every night. The Job controller tracks it until it succeeds and retries if it fails, then finished Pods are cleaned up automatically.
A team deleting a test environment
Someone removes a namespace. The namespace controller and garbage collector clear out every Pod, Service and setting inside it, leaving no leftovers.
A platform team offering databases as a service
Developers request a database with a simple object. An operator, which is a custom controller, creates storage, starts members, and handles backups and failover.
23Frequently Asked Questions
Quick answers to questions beginners ask most.
No. A controller is a single control loop for one kind of job. The Controller Manager is the program that hosts many controllers together in one process.
Controllers create and maintain objects, such as Pods, so that the cluster matches your goals. The scheduler only chooses which node an unassigned Pod should run on. They are separate components that cooperate through the API server.
No. It talks only to the API server. The kubelets on nodes also talk to the API server, so information flows between them without a direct link.
Pods already running keep running, because the kubelets look after them. But automatic repairs, scaling, rollouts and cleanup pause until the Controller Manager returns or another copy becomes leader.
A single copy is a single point of failure for all automatic healing. With several copies and leader election, another one takes over quickly if the leader fails.
No. Only the leader acts. The others stand by, which avoids conflicting actions such as creating duplicate Pods.
It runs the controllers that depend on a specific cloud provider, such as creating load balancers and checking whether a virtual machine still exists. It keeps provider-specific code separate from core Kubernetes.
Yes. You can teach Kubernetes new kinds of objects and then write a controller that watches them and acts. Such controllers usually run as ordinary Pods and are often called operators when they encode expert knowledge of a product.
Waiting periods are built in on purpose, so short network glitches do not trigger unnecessary movement of Pods. The delays can be tuned, but very short ones increase the risk of needless disruption.
Understand Pods, Deployments and the idea of desired versus current state. Then learn how ReplicaSets and Deployments relate, and trace what happens when a Pod dies. Once that feels natural, the rest of the controllers are variations on the same theme.
24Summary and Key Takeaways
The Controller Manager is the part of the Kubernetes control plane that runs the built-in controllers, the small loops that keep the real cluster matching the goals you wrote down. Each controller watches one kind of object, compares wanted with actual, and acts through the API server to close any gap. Together they build rollouts, replace lost Pods, notice failing nodes, clean up leftovers and much more, all without human hands. Learn the idea of desired state, the control loop and the chain of owners, and you will understand why Kubernetes feels like it looks after itself.
Key Takeaways
- It hosts the core controllers — the Controller Manager bundles many independent control loops into one control plane program.
- Every controller follows the same loop — watch, compare wanted with actual, act through the API server, and repeat forever.
- It talks only to the API server — it never starts containers or contacts nodes directly; the scheduler and kubelets do those jobs.
- Small controllers combine into big behaviour — Deployment, ReplicaSet, scheduler and kubelet form a chain that turns one goal into running containers.
- Self-healing is a side effect of the design — lost Pods, dead nodes and deleted objects are repaired because the gap from the goal is noticed.
- Leader election prevents chaos — several copies can run for safety, but only one acts at a time.
- It is extensible and must be protected — custom controllers and operators reuse the pattern, while permissions, encrypted connections and monitoring keep the control plane trustworthy.