Before opening the machinery, we need to see the problem. Why does Kubernetes need operators at all?

What Kubernetes does well

Kubernetes is a system that runs containers across many computers. You describe what you want, for example “run three copies of my website,” and Kubernetes makes it true and keeps it true. If a copy crashes, Kubernetes starts another. If a machine fails, Kubernetes moves the copies elsewhere. This works beautifully for stateless programs, which are programs that keep no important data of their own and can be replaced freely.

What Kubernetes does not know

Now think about a database cluster with three members. One is the leader and two are followers. To replace a failed follower, you must copy the data from the leader, tell the cluster about the new member, and wait until it is in sync. To upgrade, you must update followers first, then hand leadership to one of them, then update the old leader. To back up, you must pause writes for a moment or use a special snapshot command. Kubernetes has no idea about any of this. It only knows how to start and stop containers.

The human operator

Traditionally, a person called an operator, usually a site reliability engineer or database administrator, held this knowledge in their head and in a pile of runbooks. They were paged at night, followed the steps and fixed things. This is slow, expensive and error-prone, and does not scale to hundreds of clusters.

The software operator

The operator pattern was introduced by CoreOS in 2016. The idea is to take the knowledge of the human operator and write it as software that runs inside the cluster, watches the system, and acts. The software operator uses the same tools Kubernetes itself uses, so it feels native. Today, operators run databases, message brokers, certificate managers, monitoring stacks, storage systems and even entire clusters.

The control loop is the single most important idea in Kubernetes. If you understand it, operators will feel natural. If you skip it, nothing else will make sense.

Declarative: say what, not how

In Kubernetes you rarely give orders such as “start this container now.” Instead, you declare the state you want, such as “there should be three copies of this program,” and store that declaration in the cluster. Declarative means you describe the destination, not the route. The declaration is saved in a database called etcd, reached through the API server, which is the front door of the cluster.

Desired state and actual state

Every object in Kubernetes has two faces. The desired state, called the spec, is what you asked for. The actual state, called the status, is what really exists right now. A big part of Kubernetes is the work of noticing when these two differ and fixing the difference.

Controllers

A controller is a program that runs a never-ending loop with three steps. First, observe the world. Second, compare it with what is wanted. Third, act to reduce the difference. Then it starts again. Kubernetes has dozens of built-in controllers. The Deployment controller makes sure the right number of Pods exist. The Node controller notices machines that stop responding. The Job controller runs tasks to completion.

Level-triggered, not edge-triggered

This is a subtle but very important idea. An edge-triggered system reacts to events: “a Pod was just deleted.” A level-triggered system reacts to the current level: “there are two Pods and we want three.” Kubernetes controllers are level-triggered. They do not trust that they saw every event. They look at the whole picture each time and act on the difference. This makes them robust. If the controller crashes for an hour and misses a hundred events, it simply looks at the world when it restarts and fixes whatever is wrong.

To build an operator you first need a place to write down what you want. Custom resources are that place.

Resources and kinds

Everything you create in Kubernetes is an object of some kind, such as Pod, Service or Deployment. Each kind is served through the API under a name, for example “pods”. These named collections are called resources. Kubernetes ships with dozens of built-in kinds, but you can add your own.

The CustomResourceDefinition

A CustomResourceDefinition, or CRD, is an object that teaches the API server about a new kind. Once you create a CRD describing, say, a kind called PostgresCluster, the API server starts accepting, storing and serving objects of that kind, with the same commands you use for built-in objects. Objects of your new kind are called custom resources, or CRs. From the point of view of the cluster user they look and feel like native Kubernetes objects.

What a CRD contains

Spec and status, again

Custom resources follow the same convention as built-ins. The spec holds what the user wants. The status holds what the operator reports. Because the status subresource is a separate endpoint, users can edit the spec but should not be able to edit the status, and the operator can update the status without disturbing the spec. A very useful side effect is that changing only the status does not increase the object’s generation counter, which lets controllers tell the difference between “a person changed the request” and “someone updated the report.”

With the control loop and custom resources in hand, we can now describe an operator fully and see where Kubebuilder and the Operator SDK fit.

An operator is two things

• A custom resource definition. The vocabulary: the new kinds of objects a user can create to say what they want.
• A controller. A program running in the cluster, usually as a Deployment, that watches these objects and makes the real world match them.

A story: the database operator

Suppose a user creates a PostgresCluster object saying “three replicas, version 16, 100 gigabytes of storage each, backups every night.” The operator notices this object. It creates storage volumes, a StatefulSet for the database Pods, Services for clients to reach them, a Secret holding the generated password and a scheduled Job for backups. It then waits for Pods to be ready, configures replication, and writes into the status “cluster healthy, leader is pod-0.” Later the user changes the version to 17. The operator notices and carries out a careful rolling upgrade, one Pod at a time, followers first. If a Pod dies, the operator notices that the real world no longer matches the spec and repairs it.

Where the tools come in

Writing a controller from scratch means writing code to talk to the API server, cache objects, receive change notifications, queue work, retry failures, elect a leader among several copies, handle certificates for webhooks, expose metrics, and package everything for installation. That is a lot of plumbing that has nothing to do with the actual knowledge about your database. Kubebuilder and the Operator SDK exist so you only have to write the knowledge, and the plumbing is generated or provided by a shared library.

The names are confusing, because there are several overlapping projects. This chapter sorts them out.

The cast of characters

How Kubebuilder and Operator SDK relate

The two projects began separately, with similar aims. Over time they converged. Today, the Go support in the Operator SDK is built through a plugin system on top of Kubebuilder’s scaffolding, so a Go project made by either tool looks very similar inside. The Operator SDK then adds extras: commands for bundle creation and OLM integration, a scorecard to check an operator against best practices, and the Helm and Ansible operator types for teams that prefer not to write Go. The Operator Framework, which the SDK belongs to, is a CNCF project.

Which should you choose?

When you run the init command, a whole project appears. Knowing what each part is for removes the mystery.

The commands that create things

You usually begin with three commands. One initializes a new project, choosing a domain name and a repository. A second creates an API, meaning a new kind with its type definitions and a controller. A third creates a webhook for that kind. Each command adds files and updates a small file called PROJECT that records what has been generated, so later commands know the state of the project.

A tour of the layout

The two generation steps

Two make targets matter more than any others. The first generates the “boilerplate” Go code your types need, such as the functions that copy objects safely. The second generates the YAML manifests: the CRDs, the RBAC roles and the webhook configurations. Both read the markers you wrote in your code comments. You run them every time you change your types or markers, and you commit the results. Forgetting to regenerate is the most common source of “my change has no effect” confusion.

Now we enter the engine room. controller-runtime is a library, and understanding its main pieces explains how every operator behaves.

flowchart TB
  subgraph Process["Operator process"]
    MGR["Manager"]
    SCH["Scheme - type registry"]
    CACHE["Shared Cache - informers"]
    CLI["Client - reads from cache, writes to API"]
    subgraph CTRL["Controller"]
      SRC["Source and event handlers"]
      PRED["Predicates"]
      Q["Rate-limited work queue"]
      W["Worker goroutines"]
      REC["Reconciler - your logic"]
    end
    WH["Webhook server"]
    LE["Leader election"]
    MET["Metrics and health endpoints"]
  end
  API["Kubernetes API Server"]
  CACHE -->|"list and watch"| API
  API -->|"change notifications"| CACHE
  CACHE --> SRC
  SRC --> PRED
  PRED --> Q
  Q --> W
  W --> REC
  REC --> CLI
  CLI -->|"reads"| CACHE
  CLI -->|"writes"| API
  API -->|"admission calls"| WH
  MGR --- CACHE
  MGR --- CTRL
  MGR --- WH
  MGR --- LE
  MGR --- MET
  LE -->|"lease"| API

The main characters

Things the manager also gives you for free

• Leader election so only one copy of the operator acts at a time, even when several are deployed.
• A metrics endpoint with counters and timers for reconciles and queues.
• Health and readiness endpoints that Kubernetes can use to check on the operator.
• Graceful shutdown that stops workers and releases the leader lock on a termination signal.

How an operator starts, and in what order, explains many mysterious start-up behaviors.

Setting the stage

When the operator program starts, its main function does a fixed series of steps. It builds a scheme and registers the built-in Kubernetes types and your own custom types. It reads settings from flags, such as the metrics address and whether to use leader election. It loads the connection details for the cluster, either from the cluster itself when running inside a Pod or from a configuration file when running on a developer’s laptop. It then creates the manager with these options.

Registering controllers and webhooks

Next, the program asks each of your controllers to set itself up with the manager. During setup, a controller says which kinds it reconciles and which other kinds it watches. Webhooks are registered too. Nothing is running yet; these steps only build the plan.

Starting

Finally, the program asks the manager to start. The manager starts its parts in a careful order:

Why the cache sync matters

The wait for cache synchronization is a critical safety feature. Imagine a controller that starts reconciling before its cache has loaded the existing Pods. It would think no Pods exist and try to create all of them again. Waiting until the cache reflects the real world prevents this kind of confusion.

Shutdown

When the Pod receives a termination signal, the manager stops accepting new work, lets running reconciles finish within a grace period, releases the leader lease if it holds it, and exits. A fast hand-over of the lease lets another replica take over quickly.

The cache is the most important performance feature of an operator, and also the source of its subtlest bugs. Here is how it truly works.

The problem the cache solves

A controller reads objects constantly. If every read went to the API server, a busy operator would flood it with requests and slow down the whole cluster. Instead, the operator keeps its own local copy of the objects it cares about and keeps it up to date by listening for changes.

List and watch

The Kubernetes API supports two operations that make this possible. A list returns all objects of a kind right now, along with a version marker called the resource version. A watch opens a long-running connection that streams every change after a given resource version: added, modified or deleted. A component that lists once and then watches can maintain a perfect local copy with very little network traffic.

flowchart LR
  API["API Server"] -->|"list then watch"| REF["Reflector"]
  REF --> DF["Delta queue"]
  DF --> INF["Informer loop"]
  INF --> IDX[("Indexer - local store")]
  INF --> H1["Event handler 1"]
  INF --> H2["Event handler 2"]
  H1 --> WQ1["Work queue of controller A"]
  H2 --> WQ2["Work queue of controller B"]
  R["Reconciler"] -->|"Get and List"| IDX

The pieces of an informer

• Reflector. Performs the initial list, then the watch, and pushes every change into a queue. If the watch breaks, it re-lists and resumes.
• Delta queue. A short buffer of changes in the order they arrived.
• Indexer. The local store, which holds the latest copy of each object and also supports fast lookups by custom keys, such as “all Pods in this namespace” or “all Pods that belong to this owner.”
• Event handlers. Small functions that are called when an object is added, updated or deleted. In an operator, they decide which reconcile requests to put on the work queue.

Shared informers

If two controllers both watch Pods, it would be wasteful to open two watch connections and keep two copies. controller-runtime uses shared informers: one informer per kind, with one local store, shared by every controller that needs it. Each controller attaches its own event handlers.

Periodic resync

Informers can also replay all objects to handlers at a regular interval, called the resync period, as a safety net against missed events. In controller-runtime the default is long, about ten hours, and a level-triggered design means the correctness of a well-written controller should not depend on it.

The staleness catch

The cache is eventually consistent, which means it can be a moment behind the truth. Imagine your operator creates a Pod and immediately reads the list of Pods from the cache. The new Pod may not be there yet. A careless controller might then create a second Pod. Good controllers account for this: they use deterministic names so a duplicate create fails harmlessly, they requeue to check again, or they rely on the next event rather than assuming that their write is instantly visible.

Between “something changed” and “your code runs” sits a small but clever data structure. It explains how operators avoid doing the same work twice and how they retry failures politely.

Why not just run code on every event?

Imagine an object changes five times in one second. If your code ran once per event, it would run five times, mostly doing the same work. Worse, two copies could run at once for the same object and trip over each other. A queue solves both problems.

Keys, not events

The work queue does not store events. It stores keys, and a key is simply the namespace and name of an object, like “shop/orders-db.” When an event arrives, the handler puts the key in the queue. Your reconcile function later receives only the key, and then looks up the current state of that object itself. This is level-triggered design in action: the queue says “go look at this object,” and does not say “this changed in this particular way.”

The three guarantees of the queue

Rate limiting and back-off

When a reconcile returns an error, the key goes back into the queue, but not immediately. The default rate limiter combines two ideas. The first is exponential back-off per item: the first retry waits a few milliseconds, then the delay doubles each time, up to a ceiling of about a thousand seconds. The second is an overall rate limit across all items, a token bucket that allows a certain number of retries per second with some burst. The system uses whichever is slower. A stubborn failure on one object backs off quickly, while a mass failure across many objects cannot create a retry storm.

Workers and concurrency

Each controller runs a number of worker threads that pull keys from the queue. The default is one worker, which means reconciles happen strictly one after another. You can raise it so that different objects are reconciled in parallel. The queue still guarantees that the same key is never processed by two workers at once, but your code must be safe when different keys run together, for example when they share an external system.

The reconcile function is where your knowledge lives. Everything else in this guide exists to call it at the right time with the right input.

What gets called

The reconciler has one required function, usually called Reconcile. It receives a context, which carries cancellation signals and logging, and a request, which holds just the namespace and name of the object. It returns a result and an error. Nothing else.

sequenceDiagram
  autonumber
  participant U as User
  participant A as API Server
  participant C as Cache
  participant Q as Work Queue
  participant R as Reconciler
  U->>A: Create or update custom resource
  A-->>C: Watch event
  C->>Q: Add key namespace and name
  Q->>R: Hand over key
  R->>C: Get custom resource
  alt Not found
    R-->>Q: Done, object was deleted
  else Found
    R->>C: List child objects
    R->>R: Compare desired and actual
    R->>A: Create or update children
    R->>A: Update status
    alt Error
      R-->>Q: Return error, retry with back-off
    else Wait for something
      R-->>Q: Return result with requeue after
    else All good
      R-->>Q: Return success, forget the key
    end
  end

A typical reconcile, in plain steps

The three ways to return

Idempotency is the golden rule

Your reconcile might run twice for the same state, might be interrupted halfway by a crash, and might see a slightly stale cache. For all these reasons, every step must be safe to repeat. “Create a Service” must really mean “make sure this Service exists.” Counting on “this step has surely not been done yet” leads to duplicates and conflicts.

An operator does not only care about its own custom resources. It also needs to wake up when the things it created change. This chapter explains how it decides what to watch.

Primary and secondary resources

The kind your controller is responsible for is called the primary resource. In our database example it is PostgresCluster. The kinds it creates and manages, such as StatefulSets, Services and Secrets, are secondary resources. A change to a secondary resource can matter. If someone deletes the Service that the operator created, the operator should notice and recreate it.

Three ways of watching

Event handlers

Each kind of watch uses an event handler, a small helper that translates “object X was added, changed or deleted” into “put these keys on the queue.” The standard handlers are one that enqueues the object itself, one that enqueues the owner of the object, and one that uses your own mapping function. Together they cover almost every case.

Predicates: filtering out noise

Not every change is interesting. The status of an object changes constantly, often caused by the operator’s own updates. If every status write triggered another reconcile, the operator would chase its own tail. Predicates are filters that sit between the handler and the queue and can drop events.

• Generation changed. Passes an event only if the object’s generation number increased, which happens when the spec changes but not when only the status or metadata changes. This is the classic filter for primary resources.
• Label or annotation changed. Passes events only when specific labels or annotations changed.
• Custom filters. You can write your own, for example “only objects in namespaces we manage.”

The generated setup code reads almost like a sentence. This chapter explains how that sentence turns into running machinery.

Fluent setup

controller-runtime offers a builder that is used in a chain of short calls. Read in words, the typical setup says: “A controller, managed by this manager, for this kind, which owns these other kinds, with these options, named this, completed with this reconciler.” Each part has a purpose.

What happens behind the scenes

When the builder completes, it creates a controller object with a name, a work queue and a reconciler. For each watch, it asks the manager’s cache for the informer of that kind, creating it if needed, and attaches the right event handler and predicates. It then registers the controller with the manager so the manager can start it at the right moment. No data flows yet. All of this is plumbing, assembled up front.

One controller per kind

A common and healthy rule is one controller for each primary kind. If a single controller tries to manage several kinds, its reconcile function grows complicated, and the queue keys become ambiguous. When several kinds are related, such as a Cluster and its Backups, give each its own controller and let them coordinate through the objects they create and read.

The client looks simple, with its Get, List, Create, Update, Patch and Delete. But beneath the surface, it makes several design choices that you must understand to avoid subtle bugs.

Reads from the cache, writes to the server

The default client sends reads to the local cache and sends writes directly to the API server. This gives fast, cheap reads and correct writes. It also means a read right after a write may not see the write yet, the staleness problem from Chapter 9. When you truly need the latest data, controller-runtime offers a second reader that bypasses the cache and goes straight to the API server. Use it sparingly, since it costs a real API call.

Structured, unstructured and metadata-only

Objects can be handled in three forms. Structured objects use typed Go structures and are cached. Unstructured objects are generic maps, useful for kinds you do not have types for, and by default are read directly from the server rather than the cache. Metadata-only objects fetch just names, labels and owner references, which is much lighter on memory if you only need to know that something exists.

Optimistic concurrency

Every object carries a resource version. When you update an object, the server checks that the version you send still matches the stored one. If someone else changed it in between, your update is rejected with a conflict error. This is called optimistic concurrency, because it assumes conflicts are rare and checks only at the moment of writing. The standard reaction to a conflict is to return the error, so the item is retried with fresh data from the cache.

Update, patch and server-side apply

Writing status separately

When the status subresource is enabled, the status has to be written through its own endpoint, using the status writer of the client. Writing the spec does not change the status, and writing the status does not change the spec. This keeps users and the operator from stepping on each other.

Creating things is half the job. Cleaning up properly when a resource is deleted is the other half, and Kubernetes gives you two tools for it.

Owner references: automatic cleanup inside the cluster

An owner reference is a small note on an object that says “I belong to that other object.” When you create a child, such as a StatefulSet, you set the custom resource as its owner. Kubernetes has a built-in garbage collector, a controller that watches for objects whose owners no longer exist and deletes them. So deleting the custom resource automatically removes all its children. Owner references also tell the Owns watch which owner to wake when a child changes. A rule to remember: owners and children must live in the same namespace, or the owner must be cluster-scoped.

Finalizers: cleanup that Kubernetes cannot do

Owner references only handle objects inside the cluster. What if your operator created a cloud database, a DNS record or a storage bucket outside Kubernetes? Garbage collection knows nothing about them. Finalizers solve this. A finalizer is a string added to an object’s metadata. While any finalizer remains, Kubernetes will not delete the object. When someone asks to delete it, Kubernetes only sets a deletion timestamp and leaves the object in place. The operator sees the timestamp, cleans up the external resource, removes its own finalizer, and only then does Kubernetes truly delete the object.

flowchart TB
  A["Reconcile sees new resource"] --> B["Add finalizer to the resource"]
  B --> C["Create external and child resources"]
  C --> D["User deletes the resource"]
  D --> E["API sets deletion timestamp, object stays"]
  E --> F["Reconcile sees deletion timestamp"]
  F --> G["Clean up external resources"]
  G --> H{"Cleanup succeeded?"}
  H -->|"yes"| I["Remove finalizer"]
  H -->|"no"| J["Return error and retry with back-off"]
  J --> F
  I --> K["API deletes the object"]
  K --> L["Garbage collector removes owned children"]

Rules for safe finalizers

• Add the finalizer before creating anything external. Otherwise a crash between creating and recording leaves an orphan.
• Make cleanup idempotent. It may run several times. “Delete the bucket” must succeed if the bucket is already gone.
• Remove the finalizer only after cleanup succeeds. Never remove it to “get unstuck” without understanding why cleanup failed.
• Do not add finalizers lightly. Every finalizer is a promise that deletion will wait for your operator. If the operator is uninstalled first, the objects can hang in “terminating” forever.

A good operator explains itself. The status is the voice of the operator, and there are conventions for using it well.

What the status is for

The status answers the user’s questions: Is it working? What is it doing? What went wrong? It should be a summary of observed facts, written by the operator and read by people and tools. It is not the place to store the operator’s private working data, since status can be lost or edited in surprising ways, and the controller should be able to rebuild any decision from the real world.

Conditions

The Kubernetes convention is a list of conditions, each with a type, a true or false or unknown status, a reason written in a short code-like word, a human-readable message, and the time it last changed. Typical types are Ready, Progressing, Degraded and Available. This gives every tool a common way to ask “is this healthy?” without learning your specific fields. Kubernetes tools such as kubectl wait can wait on a named condition.

Observed generation

A small but valuable field is the observed generation: the generation number of the spec that the operator last acted on. If the spec’s generation is 7 and the observed generation in the status is 6, readers know the operator has not yet processed the latest change, so the status is out of date. Without it, a user cannot tell whether “Ready: True” refers to the old spec or the new one.

Printer columns and events

You can choose a few status fields to be displayed when users list the objects, such as phase and ready replicas. In addition, operators emit Kubernetes events, short messages attached to an object that appear in its description, like “Created StatefulSet” or “Backup failed: timeout.” Events are temporary and best used for human-friendly breadcrumbs, not for state that tools must rely on.

Reconcile loops react after an object is stored. Webhooks act at the door, before storage, and also help old and new versions of an API live together.

Admission webhooks

An admission webhook is a web address that the API server calls whenever someone creates or updates a matching object. A mutating webhook can change the object, for example by filling in defaults that the schema cannot express. A validating webhook can reject the object with a clear message, for example “a cluster with three replicas must have at least one backup location.” Validation at admission is far kinder to users than a failure minutes later inside the reconcile loop. Kubebuilder scaffolds both kinds with a simple interface and handles the server and registration.

Certificates

The API server only talks to webhooks over secure connections, so the webhook needs a certificate that the API server trusts. The scaffold usually integrates with cert-manager, a popular tool that issues and renews certificates automatically and injects the trust information into the webhook configuration. Without a certificate story, webhooks simply fail to be called.

Versioning and conversion

APIs evolve. A custom resource might begin as v1alpha1, with fields that turn out to be badly designed, and later become v1 with a cleaner layout. Users with existing objects cannot all change at once, so the API server serves several versions at the same time. Only one version, the storage version, is written to etcd. When a client asks for a different version, the API server must convert. For simple changes this can be automatic. For anything else, you provide a conversion webhook.

This hub-and-spoke design means that when you add a version, you write one new pair of conversions instead of one for every existing version. Your reconciler works with a single version, the hub.

A lot of an operator’s files are written by machines, not by people. The secret behind this is a set of special comments called markers.

What a marker is

A marker is a normal-looking comment in your Go code that starts with a recognizable prefix. To the Go compiler it is just a comment. To the generator tool, called controller-gen, it is an instruction. By placing markers on your type definitions and on your reconcile function, you describe things that cannot be expressed in plain Go, such as “this field must be at least 1” or “this controller needs permission to create Services.”

Kinds of markers

What the generators produce

• Deep-copy functions. Kubernetes objects must be copyable without sharing hidden pointers, otherwise two parts of the program could change the same data by accident. The generator writes these functions for every type.
• CRD manifests. The complete definition, schema included, ready to install.
• RBAC manifests. The Role or ClusterRole that grants exactly the rights that the markers list.
• Webhook manifests. The registration objects that tell the API server where to send calls.

Why this design is good

The type definition becomes the single source of truth. The Go structure, the schema that users see and the documentation all come from the same place, so they cannot drift apart. Permissions live next to the code that needs them, so a reviewer sees “this function creates Services” and “the role grants create on Services” in the same file.

Operators talk to a live API server, which makes them harder to test than ordinary programs. A layered approach keeps tests fast and honest.

Layer one: unit tests with a fake client

controller-runtime provides a fake client that keeps objects in memory and behaves like a simple API server. It is perfect for testing small pieces of logic: given this spec and these existing children, does the function produce the right child objects? These tests run in milliseconds and need no cluster. Their weakness is that the fake does not enforce everything a real server does, such as schema validation, defaulting, admission or garbage collection.

Layer two: envtest, a real control plane without the rest

envtest starts a real etcd and a real Kubernetes API server on the test machine, using downloaded binaries. Your controller runs against it exactly as in production, with real watches, real conflict errors and real CRD validation. What envtest does not start are the other built-in controllers or any node agents. So a Deployment created in envtest will never produce Pods, a deleted owner will not trigger garbage collection of its children, and a Pod will never become ready. Tests must either create those effects by hand, for example by writing a status onto an object, or avoid depending on them.

Layer three: end-to-end tests on a real cluster

For tests that need the complete system, such as an upgrade of a real database, teams use a lightweight cluster running in containers, with a tool such as kind. These tests are slower and need more resources but give the highest confidence. The scaffold includes a small end-to-end suite that installs the operator and checks that it starts and exposes metrics.

Writing reliable asynchronous tests

Because controllers work in the background, a test cannot check the result the instant it creates an object. It has to wait until a condition becomes true. The scaffolded tests use a testing library that provides “eventually” and “consistently” checks, which poll until a condition holds or a timeout passes. A poorly written test that sleeps for a fixed time is slow when it passes and flaky when the machine is busy.

Production operators should survive the loss of a machine. But running two copies of a controller at once causes chaos, so they need a way to agree on who is in charge.

The problem with two copies

If you run two replicas of an operator for safety, both receive the same events and both would try to create the same children and update the same status. They would fight, and external actions such as “failover the database” could be executed twice. We want one active copy, called the leader, and the others standing by.

How leader election works

Kubernetes provides a simple building block: a small object called a Lease. The replicas compete to write their identity into the Lease. The one that succeeds is the leader. It must renew the Lease regularly to prove it is still alive. If it fails to renew within a deadline, for example because it crashed or lost network, the other replicas notice that the Lease has expired and one of them takes over.

What runs where

Not everything is limited to the leader. Components that need to be available on every replica, such as the health endpoints, the metrics endpoint and the webhook server, run on all replicas so that admission calls are served by any copy. Controllers and anything that changes the world run only on the leader. The cache is usually running on every replica so a new leader starts with warm data and is ready to work immediately.

Safe hand-over

If the operator is shut down gracefully, it can release the Lease at once, instead of waiting for the expiry, which makes upgrades faster. The setting for this must be used with care: it requires that the program really stops all work before the lease is released, otherwise two leaders could briefly overlap.

Designing for hand-over

• Never keep important state only in memory. A new leader begins with nothing but the cluster state.
• Make every action idempotent, since the old leader may have done half of an action before it died.
• Test failover on purpose. Kill the leader Pod during a reconcile and verify that the new leader finishes the job correctly.

An operator is only useful once it runs in a cluster. Here is how the scaffolded project turns source code into a deployed, supported product.

From code to container

The Dockerfile in the scaffold uses a multi-stage build: the first stage compiles the program with the full Go toolchain, and the second copies only the resulting binary onto a very small base image. The result is a tiny image with no shell and few tools, which reduces attack surface and download time. Container images are built and pushed with the Makefile targets, using whichever container tool your team prefers.

From manifests to cluster

The config folder is organized for Kustomize, a tool that builds final manifests by layering patches over a base. A default layer ties together the CRDs, the RBAC roles, the manager Deployment, the webhook configuration and the certificates. Developers can install the CRDs, deploy the operator, and remove it again with a few make commands. Overlays let you change settings per environment without editing the originals.

Other ways to ship

Order of installation

Always install the CRDs before the operator, and the operator before any custom resources. If custom resources arrive first, the API server rejects them as an unknown kind. If the operator arrives before its CRDs, it fails to start its watches. The scaffold’s make targets and the OLM bundle handle this order for you.

On top of the Kubebuilder foundation, the Operator SDK provides tools for distribution, quality and teams that do not want to write Go.

Operators without Go

Not every team has Go developers. The Operator SDK supports two other styles. In a Helm-based operator, you supply a Helm chart, and the operator creates a release whenever a custom resource appears, mapping the custom resource’s spec onto chart values. In an Ansible-based operator, you supply playbooks or roles, and the operator runs them whenever the custom resource changes. These are quick to build and fine for simple installation and upgrade logic. They are less suited to complex, stateful, event-heavy logic, where a Go operator has more control.

Bundles and the Operator Lifecycle Manager

The Operator Lifecycle Manager, or OLM, is a cluster component that works like a package manager for operators. The SDK generates a bundle, a package that contains the CRDs and a ClusterServiceVersion, a description of the operator: its version, the permissions it needs, the kinds it owns, how to upgrade from earlier versions and metadata for display. Bundles are collected into catalogs. A cluster administrator then subscribes to an operator in a catalog, and OLM installs it, resolves dependencies and applies upgrades along a defined path.

The scorecard

The SDK includes a scorecard that runs a series of checks against a bundle on a test cluster. It verifies, for example, that custom resources have a status block, that the bundle metadata is valid, and that the example resources can actually be created. It gives teams a repeatable quality gate.

Capability levels

The Operator Framework defines five levels to describe how mature an operator is. They give users a quick sense of what to expect.

An operator often has broad rights in a cluster, because creating databases and rolling upgrades requires it. That makes it a juicy target, and a risky one if built carelessly.

RBAC: ask only for what you need

The permissions of an operator come from the markers described in Chapter 18. It is tempting to write broad markers, such as “all verbs on all resources.” Resist this. Each marker should list the specific kinds and verbs the code needs. Ask for secrets only if the operator really reads or writes secrets, and prefer to limit access to a namespace instead of the whole cluster. A namespace-scoped operator, watching only the namespaces it serves, has a much smaller blast radius than a cluster-wide one.

Protecting the process

• Run as a non-root user with a read-only root filesystem, dropped capabilities and a restricted security profile. The scaffold sets sensible defaults.
• Use a minimal image with no shell, and pin the base image and dependencies.
• Keep secrets out of logs, events and status. A password written into a status field is readable by everyone who can read the custom resource.
• Protect the metrics endpoint. Recent scaffolds serve metrics over a secured connection with authentication and authorization. Older ones used a separate proxy container for this. Whatever the version, do not leave metrics open to the whole network.
• Use network policies to limit what can talk to the operator and what the operator can talk to.

Who may create custom resources?

An operator turns a short custom resource into powerful actions. Anyone who can create a PostgresCluster can cause storage to be provisioned and Pods to run. Use RBAC to limit who may create, edit and delete custom resources. Validate the contents with schema rules and webhooks, so users cannot request dangerous settings, such as mounting host paths or using unapproved images.

Privilege escalation through the operator

A subtle danger: if a user can create a custom resource that causes the operator to create Pods or Roles, the user is indirectly using the operator’s permissions. Without checks, a user with permission only to create a custom resource could get the operator to run a Pod with settings they could never create directly. The operator must treat the custom resource spec as untrusted input, and apply the same limits that Kubernetes would.

Supply chain

Operators are installed with high privileges, so verify where they come from. Prefer signed images, pinned versions and reviewed manifests. Scan images and dependencies regularly, and update promptly when vulnerabilities are announced.

When an operator misbehaves at night, you want answers fast. Good observability is what turns a mystery into a five-minute fix.

Logging

controller-runtime uses a structured logging interface, so each log line carries named fields rather than only free text. The reconcile function receives a logger already tagged with the controller name, the kind and the namespace and name of the object being worked on. This means you can search logs for one particular object and see its entire story. Log decisions and surprises, such as “creating Service” or “waiting for replica to be ready,” at an information level, and keep very chatty detail at higher verbosity.

Metrics

Without extra work, the manager exposes Prometheus metrics. The key families are:

You can also add your own metrics, such as the number of managed clusters by phase, or the time it took to complete a backup. Names vary slightly between versions, so check the library version you use.

Events

The manager can supply an event recorder. Events are short messages attached to an object, visible when users describe it. Use them for milestones and problems that the user should see: “Backup completed,” “Failed to provision storage: quota exceeded.” They expire after a short time, so never rely on them for permanent records.

Tracing and profiling

For deep investigations, you can add distributed tracing to follow a reconcile through calls to external systems, and enable the built-in profiling endpoint to see where CPU and memory go. Use profiling on a protected port only.

Alerts worth setting

• Reconcile error rate above a small percentage for several minutes.
• Work queue depth growing continuously or older than an acceptable age.
• Operator Pod restarts, or no leader holding the lease.
• Custom resources with a Ready condition false for longer than a set time.
• Rising number of conflict errors, which can signal competing writers.

An operator that works with ten custom resources may crawl with ten thousand. These are the dials that matter and the traps to avoid.

Memory: the cache is the big cost

Most operator memory problems come from the cache. By default, if your code reads a kind through the client, the whole cluster’s worth of that kind is cached. A cluster with tens of thousands of Pods or Secrets can push an operator’s memory up by gigabytes. Fix it by narrowing what is cached:

• Restrict namespaces. Tell the cache which namespaces to watch, if the operator does not need the whole cluster.
• Use label and field selectors. Cache only objects that carry a particular label, such as ones the operator created itself.
• Trim objects. A transform function can strip bulky fields, like managed-field metadata, before objects are stored.
• Use metadata-only watches. If you only need to know that something exists or who owns it, avoid caching its full body.
• Bypass the cache for rare reads. A one-off read of a large kind straight from the server can be cheaper than caching it forever.

Concurrency and throughput

With the default of one worker, each reconcile waits for the previous one. If a reconcile takes two seconds and 1,000 objects change, the last one waits more than half an hour. Raise the number of workers so objects are reconciled in parallel, and make sure your code and any external systems can handle that. Better still, make reconciles fast, by doing slow external work asynchronously and returning with a delayed requeue instead of waiting.

Be kind to the API server

• Avoid unnecessary writes. Compare the desired and actual state, and write only if they differ. Writing an identical status on every pass creates update events, which cause new reconciles, which cause more writes.
• Use indexes instead of scanning. If you often need “all children of this owner” or “all resources referencing this Secret,” register a field index in the cache so lookups are direct.
• Respect client rate limits. The client has its own limits on queries per second. Heavy operators may need to raise them, but only after understanding the load they add.
• Watch API priority and fairness. Kubernetes can throttle noisy clients. An operator that hammers the API can be slowed, and can slow others.

Many resources, many operators

For very large fleets, consider sharding: run several operator instances, each responsible for a subset of resources, chosen by label or namespace. Also consider splitting heavy duties into separate controllers or even separate operators, so an expensive backup process does not delay urgent failover decisions.

Measure first

Use the work queue metrics from Chapter 24 to know whether you are limited by worker count, by slow reconciles or by API latency, and use profiling to find memory hogs. Guessing leads to tuning the wrong thing.

Operators live in a messy world of slow networks, restarting Pods and external systems that fail at random. Reliable ones follow a few simple habits.

Classify errors

Not all errors are equal. A conflict means somebody changed the object, so retrying with fresh data will usually work. A “not found” for something you expected can mean the cache has not caught up. An invalid configuration in the spec will never succeed by retrying, so retrying with back-off only creates noise. For the latter, set a condition explaining the problem and stop, waiting for the user to fix the spec. Handle each class on purpose, and avoid a single catch-all that returns every error.

Prefer waiting over failing

Many situations are not errors, just waiting: the database is not ready, the volume is still being created, the backup is running. Returning an error pollutes metrics and triggers back-off. Instead, set a status such as “Progressing,” and return a result that asks to be called again after a fixed delay. Better yet, watch the thing you are waiting for, so you wake up exactly when it changes.

Timeouts and cancellation

Every call to an external system must have a timeout. A reconcile that blocks forever on a hung connection occupies a worker permanently, and with one worker, stops the whole controller. Use the context passed to reconcile, which is cancelled when the operator shuts down, and add deadlines to outbound calls.

Drift and out-of-band changes

Because of the level-triggered approach, the operator can detect and repair changes made behind its back, such as someone editing a child object by hand. Decide your policy: do you revert manual changes, or do you allow certain fields to be changed by others? With server-side apply and clear field ownership, you can own only the fields you care about and leave others alone.

Careful about irreversible actions

Some actions are dangerous to repeat or to run based on stale information, such as deleting a volume or promoting a new database leader. Before performing them, re-read the freshest state, use the non-cached reader if needed, record your intent in the status first so a restart can continue properly, and require explicit confirmation signals for the most destructive operations.

After years of operators in the wild, the community has learned which designs age well and which cause pain.

Patterns that work

Patterns that hurt

A short checklist you can paste into your team’s playbook.

The most common mistakes, explained

• Forgetting to regenerate. You edit a marker or a type, but the CRD or role on the cluster is the old one. Regenerate and redeploy after any change.
• Missing RBAC for watched kinds. To watch a kind, the operator needs list and watch rights. Without them, the cache never syncs and the controller silently waits.
• Conflict storms. Updating the whole object from stale data causes constant conflicts. Use patches or server-side apply, and fetch fresh data.
• Infinite reconcile loops. A controller that changes something it also watches, such as an annotation or a timestamp, triggers itself forever.
• Cross-namespace owners. Setting an owner in a different namespace is invalid and the child will be garbage collected or never linked.
• Hard-coded names that collide. Two custom resources in different namespaces produce children with the same name in a shared namespace. Derive names from the owner’s identity.
• Ignoring upgrade paths. The new operator version meets old custom resources and old child objects. Test upgrades with data from the previous release.

When an operator does not behave, work through these questions in order. Most faults sit in the first few layers.

Problem: nothing happens when I create a custom resource

• Is the operator running and the leader? Check that the Pod is up and that the logs show controllers starting. If leader election is on, check who holds the lease.
• Is the CRD the one you expect? An outdated CRD may lack the field you set, and the API server silently drops unknown fields unless validation is strict.
• Did the cache sync? Logs that say a cache failed to sync usually mean missing permissions to list or watch a kind.
• Is a predicate filtering the event? A too-strict filter can hide the very change you made.

Problem: reconcile runs in a tight loop

Look at the reconcile rate in metrics and at what changes between runs. Common causes are status writes that always differ, such as a timestamp, a watch on a kind that the controller itself keeps modifying, or a requeue that returns immediately. Add a predicate, write only on real change, and give requeues a sensible delay.

Problem: lots of conflict errors

Two writers are changing the same object, or the code updates using stale data. Switch from full updates to patches or server-side apply, re-read before writing, and make sure only one controller manages each field.

Problem: a resource is stuck terminating

Check its finalizers. If the operator that owns the finalizer is gone or its cleanup keeps failing, the object waits forever. Fix the cleanup or restore the operator. Only as a last resort, and after verifying that external resources are cleaned up, remove the finalizer manually.

Problem: webhook calls fail

Check that the webhook server is running on every replica that can be called, that the Service points at the right port, and that the certificate is valid and trusted by the API server. A failed webhook with a fail-closed policy blocks writes, so look at API server error messages that name the webhook.

Problem: the operator uses too much memory

Look at which kinds are cached, using profiling and the informer list. Narrow by namespace, label selector or metadata-only watches, as described in Chapter 25.

Problem: it works in tests but not in the cluster

Remember that envtest has no built-in controllers or node agents. Behavior that depends on garbage collection, Pod scheduling or readiness will differ. Reproduce the issue in a kind cluster before assuming it is a bug in the logic.

The patterns in this guide are not academic. They power some of the most widely used software in the Kubernetes world.

This is an illustrative story assembled from common experiences. The company is fictional, but the decisions and the mistakes are realistic.

The situation

BrokerWorks is a mid-sized software company with 60 product teams. Each team wanted its own message broker cluster, and the platform team was spending half its time on tickets: create a cluster, resize it, rotate its passwords, upgrade it, restore it. They decided to build an operator that offered a single custom resource called MessageBroker, with just five spec fields: size, version, storage class, backup schedule and an optional network exposure setting.

The build

What went wrong

Three problems taught the team lasting lessons. First, early versions wrote the status on every pass with a timestamp, which caused an endless loop of events and pushed CPU use high. They fixed it by writing only on real change and filtering out status-only updates. Second, in a busy cluster the operator’s memory grew to several gigabytes because it cached all Secrets cluster-wide. Restricting the cache to Secrets with the operator’s own label cut memory by more than 90 percent. Third, a cleanup bug in the finalizer left a team’s namespace stuck terminating when the cloud load balancer had already been removed by hand; the cleanup treated “already gone” as an error. Making cleanup idempotent solved it.

The outcome

An operator is a long-lived product. Its API, its dependencies and the Kubernetes underneath it all keep changing, so you need a plan for change.

Evolving the custom resource API

• Adding optional fields with defaults is safe and does not need a new version.
• Renaming, removing or restructuring fields requires a new version and a conversion path.
• Deprecation first. Mark old versions as deprecated so users get warnings, announce a timeline, and keep serving them until usage is gone.
• Storage version migration. When you change which version is stored, existing objects remain in the old form until rewritten. A migration step reads and writes each object so that all are stored in the new version, after which the old one can be dropped from the CRD’s list of stored versions.

Upgrading the scaffold and libraries

Kubebuilder evolves its layout between major versions, and controller-runtime and the Kubernetes libraries move forward every few months, with occasional breaking changes in function names and defaults. A healthy habit is to upgrade regularly, in small steps, rather than once every two years in a painful jump. Read the release notes, and compare your project layout with a freshly generated one to see what changed. The Kubebuilder project documents migration guides between its versions, and has tooling that helps re-create a project’s scaffolding on a newer layout.

Supporting Kubernetes versions

Each controller-runtime release lines up with a Kubernetes release. Decide which Kubernetes versions you promise to support, test against the oldest and newest in your matrix, and watch for deprecated APIs that your operator uses to create child objects. The operator is itself a client and can break if the API server removes a version it relies on.

Upgrading the operator in a live cluster

Short answers to the questions people ask most often.

If you remember only a handful of ideas from this guide, make them these.