One simple habit, repeated forever: look at the world, compare it with the goal, and take a small step toward the goal.

A reconciliation loop is a small piece of software that runs the same three steps over and over. First it observes what is really happening. Second it compares that with what is supposed to be happening. Third it acts to close the gap. Then it goes back to the first step and does it all again. The word “reconcile” simply means “make two things agree,” just like reconciling a bank statement with your own notebook of spending.

The program that runs this loop is called a controller. The design idea behind it is called the controller pattern. Kubernetes has a controller for almost every kind of object you can create. There is one for Deployments, one for Jobs, one for Nodes, one for Services, and many more. Each controller is responsible for one small area, and each one follows the same habit.

Where this idea comes from

The idea is much older than Kubernetes. Engineers who build machines, such as factory robots, aircraft autopilots, and the cruise control in a car, have used feedback loops for a very long time. Cruise control does not press the gas pedal a fixed number of times. It keeps measuring the speed and adjusts the engine until the speed matches the number you chose.

Kubernetes inherited the idea from the systems that came before it inside Google, especially the cluster manager called Borg. Borg kept huge fleets of machines running by constantly comparing what jobs should exist with what actually existed. When Google designed Kubernetes as an open-source cousin of Borg, the loop became the heart of the whole design.

There are two ways to give instructions to a computer system, and the choice between them explains the whole design.

The first way is the imperative style. You give a list of commands, and the system runs them in order. “Start server A. Start server B. Start server C.” The second way is the declarative style. You describe the final result you want, and the system works out the commands by itself. “I want three servers.”

Imperative instructions look simple, but they break easily in real life. Suppose you order three servers and one of them crashes ten minutes later. Your original commands are already finished. Nobody is watching, so nobody replaces the crashed server. You have to notice the problem yourself, decide what to do, and type new commands. At the scale of thousands of machines, this is impossible to do by hand.

The declarative style needs a worker that never sleeps and never forgets the goal. That worker is the controller. Without controllers, a declaration would be a wish written on paper. With controllers, a declaration becomes a promise that the system keeps working to fulfil.

This is also why people say Kubernetes is a “self-healing” system. Self-healing is not magic. It is simply the loop noticing a difference between what you asked for and what exists, and then fixing it. A crashed container, a lost machine, and an accidental deletion are all just differences waiting to be noticed.

Six short words that you will see in every Kubernetes conversation about controllers.

Objects, spec, and status

Everything you create in Kubernetes is stored as an object. A Deployment is an object. A Pod is an object. A Service is an object. Each object has some identity information, such as a name and a namespace, a spec section, and a status section. Think of an object as a form. You fill in the “what I want” box, and the controller fills in the “what I see” box.

Generation and observed generation

Every time you change the spec, Kubernetes increases a counter on the object called metadata.generation. When a controller finishes reacting to a change, it writes the number it handled into status.observedGeneration. If the two numbers differ, the controller has not caught up yet. This small trick lets tools such as kubectl rollout status say honestly, “I am still waiting for the controller to notice your change.”

Conditions

Controllers also report health in a list called conditions. A condition has a type such as Available or Progressing, a status of True, False, or Unknown, a short reason, a human-readable message, and the time it last changed. Conditions are the polite way for a controller to say, “here is how I am feeling about this object,” and they are what monitoring tools and humans read first when something looks wrong.

Controllers do not talk to each other directly. They all talk to one shared notice board.

To see where controllers fit, we need to meet the main parts of a Kubernetes cluster. The cluster has a control plane, which is the brain, and many worker nodes, which are the machines that run your applications. The control plane holds the shared notice board and the controllers. The nodes hold the real containers.

flowchart TB
  U["User or CI pipeline"] -->|"kubectl apply"| API["kube-apiserver"]
  API  ETCD[("etcd datastore")]
  CM["kube-controller-manager running many controllers"] -->|"watch and update"| API
  SCH["kube-scheduler"] -->|"watch and bind pods"| API
  KL["kubelet on each worker node"] -->|"watch pods and report status"| API
  KL --> CR["Container runtime"]
  OP["Custom operator controllers"] -->|"watch and update"| API
    

Look carefully at Figure 1. The user sends a request to the API server. The API server saves it in etcd. The controller manager, the scheduler, the kubelets, and any custom operators are all watching the API server for changes. None of them call each other. They coordinate by reading and writing objects on the shared notice board.

This arrangement is called choreography, as opposed to orchestration. In orchestration, one boss tells everyone what to do. In choreography, each dancer listens to the music and reacts to what the others are doing. Choreography has a big advantage: if one dancer falls over, the others do not freeze. They keep dancing, and the fallen dancer rejoins when ready.

A controller is built from a handful of small, well-defined parts. Once you know them, every controller looks familiar.

If every controller asked the API server “what is the state of everything?” many times each second, the API server would collapse. So controllers use a smarter design, found in the Kubernetes client libraries, that keeps the traffic low. Let us walk through each part in the order the data flows.

flowchart LR
  A["API server"] -->|"list once then watch"| I["Informer"]
  I --> C["Local cache with indexes"]
  I -->|"event handlers add keys"| Q["Work queue of object keys"]
  Q --> W["Worker threads"]
  W -->|"read current state"| C
  W -->|"create, update, delete"| A
  W -->|"on failure requeue with backoff"| Q
    

This is the single most important idea in the whole pattern, and it is easy to understand with a doorbell.

There are two ways a system can react to the world. An edge-triggered system reacts to events, meaning the moment something changes. A level-triggered system reacts to the current level, meaning how things are right now. Kubernetes controllers are deliberately level-triggered.

Why does this matter so much? In a large cluster, messages are sometimes delayed, dropped, or delivered twice. Networks glitch. Controllers restart during upgrades. If a controller depended on seeing every single event, one lost message could leave your cluster wrong forever. A level-triggered controller does not care. When it wakes up, it simply looks at everything again and fixes whatever is wrong.

A second benefit is collapsing. Imagine you scale a Deployment from 3 to 4, then to 5, then to 6, all within one second. An edge-triggered controller would process three separate events. A level-triggered controller may only wake up once and see “the goal is 6, I have 3,” and then create three Pods in a single pass. It never needs to know about the intermediate numbers.

The safety net called resync

As an extra safety net, informers can periodically replay every object in their cache back to the controller, even if nothing changed. This is called a resync. It does not ask the API server again, it just reminds the controller to take another look. If a bug or a temporary failure caused the controller to miss something, the next resync gives it another chance. Many modern operators also use a “requeue after” feature to wake themselves on a timer for things that Kubernetes cannot announce, such as a certificate that is about to expire.

Let us follow one small change all the way from your keyboard to running containers.

Suppose a Deployment named “checkout” currently asks for three Pods, and you change the number to five. This looks like one action, but it triggers a relay race across several controllers. Each runner only does its own small job and then passes the baton by writing a change to the API server.

sequenceDiagram
  participant U as User
  participant API as API Server
  participant DC as Deployment Controller
  participant RC as ReplicaSet Controller
  participant S as Scheduler
  participant K as Kubelet
  U->>API: Change replicas from 3 to 5
  API-->>DC: Watch event for the Deployment
  DC->>API: Update ReplicaSet desired count to 5
  API-->>RC: Watch event for the ReplicaSet
  RC->>API: Create 2 new Pod objects
  API-->>S: Watch event for unscheduled Pods
  S->>API: Bind each Pod to a node
  API-->>K: Watch event for Pods on this node
  K->>API: Report Pod status as Running
  API-->>RC: Watch event for Pod status change
  RC->>API: Update ReplicaSet status
    

The decision flow inside a single reconcile

Each runner in this relay follows the same small flowchart. It is worth memorizing, because every well-written controller looks like this one.

flowchart TD
  S["Take a key from the queue"] --> G["Read desired state from the cache"]
  G --> E{"Does the object still exist?"}
  E -->|"No"| D["Clean up leftovers and stop"]
  E -->|"Yes"| O["Observe the actual state"]
  O --> X{"Does actual match desired?"}
  X -->|"Yes"| ST["Update status and stop"]
  X -->|"No"| A["Take one step toward desired state"]
  A --> R{"Did the action succeed?"}
  R -->|"Yes"| ST
  R -->|"No"| B["Requeue with backoff"]
  B --> S
    

Notice that the flow never says “remember what happened last time.” Every pass begins fresh. Notice also that the controller takes one step toward the goal and then reports. If more steps are needed, the next pass takes them. This keeps each pass short and keeps the system responsive.

You have probably been relying on reconciliation loops for years without noticing them.

A closer look at the Node controller

The Node controller shows how patient a reconciliation loop can be. Every kubelet sends a small heartbeat to the API server by renewing a tiny object called a Lease. If the Node controller does not see a heartbeat for about forty seconds, which is the default grace period, it marks the node’s Ready condition as Unknown and adds a taint that says the node is unreachable.

The Pods on that node usually carry a built-in toleration that allows them to stay for five minutes after the taint appears. Only after that does the system evict them so that their controllers can create replacements on healthy machines. The delay is deliberate. A short network hiccup should not cause every Pod on a machine to be thrown away. This is a good example of reconciliation including patience, not only speed.

Creating things is easy. Cleaning up properly is where the real care is needed.

Owner references: who made this?

When a controller creates a child object, it stamps the child with an owner reference that points back to the parent. A ReplicaSet owns its Pods. A Deployment owns its ReplicaSets. A Job owns its Pods. This tiny label serves two purposes. It tells the controller which objects are really its own, so it does not accidentally touch Pods created by someone else. It also tells Kubernetes what to clean up when the parent is deleted.

The garbage collector

A built-in controller called the garbage collector watches these owner links. When a parent is deleted, it removes the children that are left behind. There are three styles of deletion. In background deletion, which is the default, the parent disappears first and the children are cleaned up afterward. In foreground deletion, the parent stays visible, marked as being deleted, until all its children are gone. In orphan deletion, the children are left alone and simply lose their owner link.

Finalizers: please wait, I am not done

Sometimes deleting an object in Kubernetes is not enough, because it created something outside the cluster. Imagine a controller that creates a cloud load balancer or a storage disk for each object. If the object vanished instantly, the load balancer would be left behind, silently costing money. A finalizer solves this. It is a text marker in the object’s metadata that says, “do not truly delete me until the controller named here has finished its cleanup.”

When you delete an object that has a finalizer, the API server does not remove it. It only stamps a deletion timestamp on it. The owning controller sees the timestamp, performs the cleanup, and then removes its finalizer. Once the last finalizer is gone, the API server finally deletes the object.

stateDiagram-v2
  [*] --> Active
  Active --> Terminating: User deletes the object
  Terminating --> CleanupRunning: Deletion timestamp set and finalizer present
  CleanupRunning --> Removed: Controller finishes cleanup and removes finalizer
  Removed --> [*]
    

The reconcile function looks small, but it has to survive a messy world full of crashes, duplicates, and collisions.

Idempotency: safe to repeat

An action is idempotent if doing it twice has the same result as doing it once. Switching a light to the “on” position is idempotent. Pressing a toggle button is not, because the second press turns it off again. Because Kubernetes may call your reconcile function many times for the same object, it must be idempotent. Instead of “create a Pod,” the logic should be “make sure a Pod exists, and create it only if it does not.”

Conflicts and optimistic concurrency

Many actors can edit the same object. To prevent silent overwriting, every object carries a resourceVersion, which is like a page edition number. When you update an object, you must say which edition you read. If someone else changed it in the meantime, the API server rejects your update with a conflict error. This approach is called optimistic concurrency, because it assumes conflicts are rare and handles them only when they appear.

The right response to a conflict is not to panic. The reconciler simply stops, returns an error or a requeue request, and tries again. On the next pass it reads the fresh version from the cache and decides again from scratch. Because the logic is level-triggered, nothing is lost.

Retries with exponential backoff

When a reconcile fails, the key goes back into the queue, but not immediately. The queue applies a rate limiter that starts with a very short delay, a few milliseconds, and roughly doubles it after each consecutive failure, up to a maximum of about a thousand seconds. This is called exponential backoff. A broken object slows down gradually and does not overload the controller or the API server. When a pass finally succeeds, the failure count for that key is forgotten.

Three possible endings of a reconcile

Status updates and the self-trigger loop

A reconciler normally writes the object’s status at the end of every pass. But writing the status changes the object, and that change creates a new watch event, which wakes the same controller again. If the controller is not careful, this can create an endless loop where it keeps waking itself. The fix is to react only to meaningful changes, for example only when the generation number changes, and to avoid writing the status when nothing about it has actually changed.

The controller pattern is not locked inside Kubernetes. You can use it for anything you can describe as a goal.

Kubernetes lets you invent your own object types using a feature called a Custom Resource Definition, or CRD. A CRD tells the API server, “there is now a new kind of object called PostgresCluster, and here is the shape of its spec.” After that, people can create PostgresCluster objects just like Deployments. But on its own, the object does nothing, because it is only a record.

To bring it to life, you write a custom controller that watches PostgresCluster objects and creates the real Pods, Services, volumes, and backups that match the spec. A custom resource together with its controller is called an operator. The name comes from the idea of packaging the knowledge of a human operator, such as a database administrator, inside software.

What operators typically automate

Controllers beyond the cluster

Projects like Crossplane and the AWS Controllers for Kubernetes extend the pattern to cloud resources themselves. You describe a database or a storage bucket as a Kubernetes object, and a controller creates and maintains the real cloud resource. Kubernetes becomes a general-purpose goal-keeping engine, not only a container launcher.

A controller that runs on a single machine is a single point of failure. Kubernetes solves this without giving up simplicity.

In a production cluster, the control plane usually runs on three or more machines so that no single failure stops it. But if three copies of the same controller all acted at the same time, they would trip over each other, creating duplicate Pods and fighting over the same fields. The answer is leader election. Many copies run, but only one is the active leader. The others stay on standby, watching, ready to take over.

Leader election in Kubernetes is built on a small object called a Lease. The leader keeps renewing the lease, like keeping its hand raised. By default the lease lasts about fifteen seconds, the leader tries to renew it for about ten seconds, and standby copies check every couple of seconds. If the leader crashes and stops renewing, the lease expires, and one standby grabs it and becomes the new leader.

What happens when each part fails

Look at the “recovers” column. The recovery method is nearly always the same: read the current state and carry on. There is no special failure code for every case. The ordinary loop is the recovery plan. This is the quiet superpower of the pattern, and it is why Kubernetes clusters can survive upgrades, crashes, and network blips with so little drama.

A loop that runs forever must be cheap, or the whole cluster pays the price.

The official Kubernetes guidance for a single cluster says it is designed for up to about five thousand nodes, around one hundred and fifty thousand Pods in total, and no more than one hundred and ten Pods per node. These numbers are possible largely because of the careful design you saw in Chapter 5. Let us look at the techniques that keep the load manageable.

The thundering herd after a restart

When a controller starts, its first list brings in every object, and every object is placed in the queue once. A controller that manages fifty thousand objects therefore has fifty thousand pieces of work ready at the same moment. This surge is called a thundering herd. If each reconcile makes expensive calls to the API server or to an outside service, the start-up can overload the very systems it depends on. Careful controllers keep reconciles cheap when nothing needs changing, use caches for reads, and apply their own rate limits to external calls.

Reconcile time matters

Each worker handles one key at a time, so a slow reconcile blocks that worker. If a pass waits ten seconds for an outside service, and there are only a few workers, the queue can pile up. Good reconcilers avoid waiting inside the function. Instead of sleeping until a database becomes ready, they record the current state and ask to be requeued shortly, freeing the worker for other keys.

A controller can create, change, and delete things automatically, so it must hold only the keys it truly needs.

Every controller talks to the API server using an identity, usually a ServiceAccount. The permissions of that identity are controlled by RBAC, which stands for role-based access control. RBAC is a list of rules that says which actions, such as get, list, watch, create, update, or delete, are allowed on which kinds of objects in which namespaces.

Why a compromised controller is dangerous

A controller with broad rights is a high-value target. If an attacker takes control of an operator that can read all Secrets and create Pods anywhere, they can read passwords and run their own programs on your nodes. This is called privilege escalation. Reducing the permissions of every controller limits the damage if one is ever tricked or taken over.

Trust the cluster, but validate the input

Users can write anything into a custom resource spec. A good operator treats the spec as untrusted input. It checks values, rejects unsafe combinations, and never uses spec text to build commands or file paths directly. The CRD itself can carry a schema that makes the API server reject malformed objects before the controller sees them.

A controller works quietly in the background, so you need instruments to know that it is healthy.

Because a controller has no screen and no user, you learn about its health from the numbers it publishes. Controllers built with the standard libraries expose metrics in a format that Prometheus can collect. The most useful ones tell you how busy the queue is, how long work takes, and how often work fails.

Logging with context

Good controllers write structured logs that always include the object’s kind, namespace, and name, so you can follow one object’s story from start to finish. They log decisions, such as “creating Pod because only 2 of 3 exist,” not just errors. They avoid logging secrets and large payloads, and they use levels so that noisy detail can be switched on only when you are debugging.

Events and conditions: talking to humans

Kubernetes has a built-in way for controllers to leave short, human-friendly messages called Events. When you run a describe command on an object and see lines like “Scaled up replica set” or “Failed to pull image,” you are reading events. Combined with status conditions, they form the first place to look when something seems stuck. A controller that never writes events or conditions is hard to trust, because it fails silently.

Every design has a price. Knowing it helps you decide when the pattern is the right tool.

Eventual consistency in plain words

“Eventually consistent” means that if you stop making changes, the system will become correct, but perhaps not at this very second. When you scale a Deployment, you will not see five ready Pods the moment you press Enter. You will see the number rise over a few seconds as each controller does its part. For almost all workloads this delay is harmless, but you should never write scripts that assume instant results. Wait for the status to report success instead.

A few good habits make controllers reliable. A few bad ones make them dangerous.

Patterns worth copying

Anti-patterns to avoid

A short checklist to read before you build or install any controller.

Mistakes beginners make as cluster users

• Fixing symptoms by hand. If you delete a Pod that a ReplicaSet owns, it will simply come back. Change the owner’s spec instead.
• Editing child objects. Changes made directly to a Pod owned by a Deployment will be overwritten or lost at the next rollout. Edit the Deployment.
• Expecting instant results. Wait for the rollout to report success instead of assuming it is done after a second.
• Ignoring status and events. They are the controller’s explanation of what it is doing. Read them before guessing.
• Force-removing finalizers. It unblocks the screen but may leave real resources running and billing you.
• Letting two tools manage the same object. For example, a GitOps tool and a person editing live objects. The tool will keep reverting the person.

The pattern is not a classroom idea. It runs the infrastructure of some of the largest engineering teams in the world.

Short answers to the questions that beginners ask most often.

You started with a thermostat and finished with a deep view of how Kubernetes keeps thousands of moving parts in order. The same small habit explains the whole picture: look at reality, compare it with the goal, take a small step, and repeat. Each controller is simple, but many simple loops working together on a shared notice board create a system that heals, scales, and upgrades itself.