Operators in Kubernetes: Teaching the Cluster to Run Your Software
A complete, plain-English guide to the Kubernetes Operator pattern: what operators are, why they exist, how their control loops really work inside, how to design them safely, and where they shine in production.
Imagine you hire a very capable robot butler for your house. Out of the box, it knows how to keep the lights on, restock the fridge when milk runs out, and lock the doors at night. But it knows nothing about your grandmother’s sourdough starter, which must be fed twice a day, kept at the right temperature, and split when it grows too big. You could write the instructions on sticky notes and do the feeding yourself every day, or you could teach the butler the sourdough rules once and let it handle the starter forever. A Kubernetes Operator is exactly that lesson: it is how you teach your cluster the special care rules of a particular piece of software, so that humans no longer need to perform that care by hand.
In this guide we will start from the very basics. We will first recall what Kubernetes already does well, then see where it runs out of knowledge, and then build up the operator idea one small step at a time. By the end, you will understand custom resources, control loops, reconciliation, finalizers, capability levels, security, monitoring, design patterns, and the mistakes that trip up even experienced teams.
1What Is a Kubernetes Operator?
An operator is a piece of software that runs inside the cluster and acts like a tireless human expert for one specific application.
A Kubernetes Operator is a custom controller paired with one or more custom resource types. The custom resource type lets you describe, in the language of Kubernetes, the thing you want, for example “a three-node database cluster with daily backups”. The controller is a program that constantly watches those descriptions and does whatever is needed in the real world to make them true. When something drifts, such as a node dying or a version being changed, the operator notices and repairs or updates the system without waiting for a human.
The word “operator” comes from the human role of the same name. Every serious system has a human operator, an engineer who knows the application’s quirks: how to bring up a new replica, how to take a safe backup, how to upgrade without losing data, and what to do when the primary node fails. The operator pattern captures that knowledge as software. The term was introduced by the engineering team at CoreOS in 2016, and it has since become one of the most important extension patterns in the Kubernetes ecosystem.
Think of a regular Kubernetes Deployment as a general-purpose apartment caretaker who can fix a leaking tap or replace a bulb. An operator is a specialist hired for one tricky machine, such as a pipe organ. The caretaker is useful everywhere, but only the specialist knows how to tune each pipe, what to do when humidity changes, and when to call for repairs.
It is important to notice what an operator is not. It is not a new Kubernetes component that the project ships with the core. It is not a special kind of container. It is an ordinary application, usually packaged as a Deployment, that talks to the Kubernetes API just like any other client, but with a very particular job: to watch certain resources and keep reality aligned with them.
2Why Operators Exist
Kubernetes is brilliant at running stateless apps, but complex stateful systems need extra human knowledge.
Kubernetes was designed so that you tell it what you want, and it makes it happen. For a simple web server, “run five copies of this container” is enough. If one copy dies, Kubernetes starts another. Every copy is identical and interchangeable, so the platform does not need to know anything about what is inside the container. This is the world of stateless applications, and Kubernetes handles it beautifully with its built-in objects.
Now think about a database cluster. The copies are not interchangeable. One is the primary and others are followers. Each has its own data on its own disk. Adding a new member requires copying data from an existing one and registering it in the cluster membership. Removing a member requires care so that you do not lose a quorum. Upgrading requires a specific order, followers first and the primary last. None of this is something Kubernetes can guess, because the rules belong to the database, not the platform.
Kubernetes knows how to run containers. It does not know how to run your database, your message broker, or your search cluster. Operators fill that knowledge gap.
The Runbook Problem
Before operators, teams wrote runbooks: long documents that said things like “if the primary fails, promote the follower with the most recent data, then update the connection address”. Runbooks are slow, error-prone, and depend on a person being awake at three in the morning. They also get out of date as the system changes. An operator turns that runbook into running code that reacts in seconds and behaves the same way every single time.
Where the Pain Showed Up First
Stateful data services
Databases and queues need ordered startup, stable identity, backups, and careful failover that no generic object understands.
Day-two operations
Installing is the easy part. Upgrades, scaling, certificate rotation, and recovery are where teams burn time.
Tribal knowledge
Expert steps live in one person’s head or a stale document, so quality varies from person to person.
Self-service at scale
Hundreds of teams asking for databases cannot each wait for a specialist. Operators let them ask in a single short request.
3Kubernetes Foundations You Need First
A short refresher on the building blocks that operators stand on.
To understand operators, you need only a handful of Kubernetes ideas. Let us define each one the first time it appears, in simple words.
API Server
The front desk of the cluster. Every request to create, read, update, or delete anything goes through it.
etcd
The cluster’s memory, a reliable key-value store where all object definitions are saved.
Resource
A kind of thing Kubernetes manages, such as a Pod, Service, or Deployment. Each stored item is an object.
Spec and Status
Spec is what you want. Status is what currently exists. The gap between them is the work to be done.
Controller
A program that watches objects and acts to move reality toward the spec. The Deployment controller is one example.
Namespace
A named room inside the cluster that groups objects and helps separate teams or environments.
The API server is the reception desk of a hotel, etcd is the guest register behind it, the spec is the guest’s room request, the status is the room’s real condition, and a controller is the housekeeper who walks the corridors making sure each room matches what the guest asked for.
Declarative Versus Imperative
Kubernetes is declarative. Instead of giving orders step by step, such as “start a container, then another”, you declare an end state, such as “I want three copies”, and let the system find the path. Imperative approaches are like giving driving directions turn by turn. Declarative approaches are like giving a taxi driver the destination address. If a road is closed, the driver finds another route. Operators extend this same comfortable declarative style to applications that Kubernetes never heard of.
The key observation is that Kubernetes itself is already a big collection of controllers. The ReplicaSet controller keeps the right number of Pods alive. The Node controller notices missing machines. The Job controller runs tasks to completion. An operator is simply another controller, written by you or by an application’s maintainers, living in the same family.
4The Control Loop: Heart of Every Operator
Observe, compare, act, repeat. That is the whole secret.
Every controller, including every operator, is built around a control loop. The loop has three simple steps. First, observe the current state of the world. Second, compare it with the desired state described in the spec. Third, act to reduce any difference. Then the loop begins again. It never ends, which is exactly the point: the system is always nudging reality toward what you asked for.
flowchart LR
A[Desired state in spec] --> C{Compare}
B[Observed state in cluster] --> C
C -->|Same| D[Do nothing and wait]
C -->|Different| E[Take corrective action]
E --> B
D --> B
This diagram shows that there are two inputs to every decision: what you want and what exists. When they match, the controller rests. When they differ, it takes action, and the result of that action becomes part of what is observed next time around. Notice that there is no “finish” box. A controller is not a script that runs once; it is a permanent watchdog.
A home thermostat is a perfect control loop. You set the desired temperature, the thermostat measures the room, and if the room is colder it turns on the heater. When the room is warm enough, it turns the heater off. It does not care why the room got cold, whether a window opened or the sun set. It only cares about the gap.
Level-Triggered Versus Edge-Triggered
There are two ways a system can react to change. An edge-triggered system reacts to events, such as “a Pod was deleted”. If it misses the event, perhaps because it was restarting, it never learns about the change. A level-triggered system looks at the current state and asks “does this match what I want?” regardless of how it got there. Kubernetes controllers are designed to be level-triggered. Events are only hints that something may have changed; the real decision is always based on a fresh comparison of desired and actual state.
Many beginners think the operator must handle “created”, “updated”, and “deleted” events separately. In a healthy design, it does not. It receives a signal that something about an object changed and then simply reconciles everything from the current state.
5Custom Resources and Custom Resource Definitions
Before an operator can manage a new kind of thing, Kubernetes must be taught that the new kind exists.
Kubernetes ships with a fixed vocabulary: Pod, Service, Deployment, and so on. There is no built-in word for “PostgresCluster” or “KafkaTopic”. A Custom Resource Definition, usually shortened to CRD, is a registration form you submit to the API server that says “a new kind of object called PostgresCluster now exists, and here is the shape of its data”. Once the CRD is accepted, the API server immediately starts accepting, validating, and storing objects of that kind, with no changes to Kubernetes itself.
An individual object of that new kind is called a Custom Resource, or CR. If the CRD is the blank form, the CR is the filled-in form. A team might create a custom resource named “orders-database” that asks for three replicas, a particular version, and nightly backups. The CR does nothing by itself. It is only a record of intent. The operator reads it and makes the intent real.
A CRD is like a new order form that a restaurant prints for a special catering service. A CR is one customer’s completed form. The form itself does not cook anything. The kitchen staff, which is the operator, reads each form and prepares the meal.
What Lives Inside a Custom Resource
Metadata
Name, namespace, labels, annotations, owner references, and finalizers, the same as any Kubernetes object.
Spec
The user’s wishes, written in domain language such as version, size, storage class, and backup schedule.
Status
The operator’s report of reality: current phase, ready replicas, observed generation, and conditions.
Schemas, Validation, and Versions
A good CRD includes a schema, which is a structured description of every allowed field, its type, and its limits. With a schema, the API server rejects bad input immediately, such as a negative replica count or a misspelled field, long before the operator sees it. This gives users fast, clear feedback and protects the operator from nonsense.
CRDs also support multiple versions. As your API grows, you may rename fields or restructure the spec. You can serve version one and version two side by side, and a conversion step translates between them so that older clients continue to work. Planning versions early matters because your custom resource becomes a public API for your platform users.
Subresources: Status and Scale
Two special subresources are commonly enabled. The status subresource separates the part of the object that users write (spec) from the part that the operator writes (status), so that they cannot accidentally overwrite each other, and so that permissions can differ. The scale subresource lets standard tools, such as autoscalers and the scale command, adjust replica counts on a custom resource just as they would for a Deployment.
Design your spec for the user’s intent, not for your implementation. A user should say “storage of 100 gigabytes” rather than listing volume claim details. The operator can translate intent into the low-level objects.
6Architecture and Components of an Operator
An operator is a small family of cooperating pieces, not a single block of code.
Although every operator looks a little different, most share a common architecture. The pieces below fit together to form a complete system that understands a custom resource and keeps it healthy.
flowchart TD
U[User or GitOps tool] -->|creates custom resource| API[Kubernetes API Server]
API --> ETCD[(etcd)]
API -->|watch events| OP[Operator Manager]
subgraph OP_BOX [Operator Pod]
OP --> CACHE[Shared cache]
CACHE --> QUEUE[Work queue]
QUEUE --> REC[Reconciler logic]
end
REC -->|creates and updates| OWNED[Owned objects: Pods, Services, Volumes, Secrets]
REC -->|writes status| API
REC -->|talks to| EXT[External systems and application APIs]
Custom Resource Definition
The registered schema that teaches the cluster about the new kind of object and enforces validation.
Manager
The main process of the operator. It starts caches, controllers, health endpoints, metrics, and handles leader election.
Cache and Informers
A local, always-fresh copy of the objects the operator cares about, so it does not hammer the API server with repeated reads.
Work Queue
A line of “please look at this object” requests that removes duplicates and retries failures with increasing delays.
Reconciler
The brain. It receives one object name at a time, compares spec to reality, and takes the necessary actions.
Webhooks (optional)
Extra checkpoints that can validate, adjust defaults, or convert versions of your custom resource before it is stored.
Picture a busy repair shop. The reception desk takes requests (API server). A whiteboard lists open jobs (work queue). Each mechanic picks one job at a time (reconciler). A parts shelf nearby saves trips to the warehouse (cache). The shop manager keeps the lights on and makes sure only one shift lead is in charge (manager and leader election).
7Internal Working: Informers, Queues, and Reconcilers
Let us open the hood and follow what happens, step by step, from a change in the cluster to an action by the operator.
Step One: Watching Through Informers
An informer is a component that opens a long-lived watch connection to the API server for one kind of object. First it lists all existing objects of that kind and stores them in a local cache. After that it receives a continuous stream of changes, such as added, modified, or removed, and updates the cache. If the connection breaks, the informer reconnects and re-synchronizes so that it never loses its place. Because many parts of the operator can share the same cache, the number of requests to the API server stays low even when thousands of objects exist.
Step Two: Queuing Work
When the cache changes, the informer does not call the reconciler directly. Instead, it drops the name and namespace of the changed object into a work queue. The queue has a few clever properties. It merges duplicates, so if an object changes ten times in a second, the reconciler is asked to look at it only once. It ensures the same object is never processed by two workers at the same moment. And it supports rate-limited retries, so that failing items wait longer each time rather than flooding the system.
Step Three: Reconciling
A worker takes one name from the queue and runs the reconciler. The reconciler reads the object from the cache, reads related objects it owns, decides what needs to change, performs the changes through the API server, and updates the status. At the end it tells the framework one of a few outcomes: finished successfully, try again soon because something is not ready, try again after a specific delay, or failed with an error that should trigger a retry with backoff.
sequenceDiagram
participant API as API Server
participant INF as Informer and Cache
participant Q as Work Queue
participant R as Reconciler
API->>INF: Stream of change events
INF->>Q: Add object name and namespace
Q->>R: Hand over one item
R->>INF: Read desired and current state
R->>API: Create or update owned objects
R->>API: Update custom resource status
R-->>Q: Result: done or requeue
Why Idempotency Matters
Because the reconciler may run many times for the same object, sometimes after partial failures, it must be idempotent. That means running it once or running it ten times with the same inputs must give the same final result. A reconciler that blindly “creates a Service” on every run will fail on the second run. A well-written one first checks whether the Service exists, creates it only if missing, and updates it only if it differs from what is wanted.
Storing hidden state in the operator’s memory, such as “I already created the backup”. If the operator restarts, that memory is gone. All important facts must live in the cluster, either in the object’s status or in owned objects, so any fresh copy of the operator can continue seamlessly.
Leader Election
For high availability, teams often run two or three copies of the operator. But two controllers acting on the same object at the same time would cause chaos. Leader election solves this: the copies compete for a lock, held as a small object in the cluster, and only the winner reconciles. The others stand by. If the leader crashes or loses connectivity, the lock expires and another copy takes over within seconds.
8Data Flow and Lifecycle of a Custom Resource
From the moment a user submits a request to the moment it is deleted, a custom resource travels through predictable stages.
Submission and admission
The user submits the resource. The API server checks permissions, applies defaults, runs validation against the schema, and consults any admission webhooks.
Persistence
If accepted, the object is saved in etcd and a change event is broadcast to anyone watching.
Detection and queueing
The operator’s informer sees the new object and queues it for reconciliation.
First reconciliation
The reconciler builds everything the application needs: configuration, secrets, storage, services, and workloads. It marks each as owned by the custom resource.
Convergence
As owned objects become ready, they change, which triggers more reconciliations. The status moves from pending to progressing to ready.
Steady state and drift repair
Day to day, the operator watches for drift, such as a deleted Service or a failed replica, and heals it. Spec edits by the user trigger rolling changes.
Deletion and cleanup
When the user deletes the resource, the operator performs any cleanup tasks, then releases the object so that it and its dependents can be removed.
stateDiagram-v2
[*] --> Pending: User submits resource
Pending --> Provisioning: Operator starts work
Provisioning --> Ready: All parts healthy
Ready --> Updating: Spec changed
Updating --> Ready: Change finished
Ready --> Degraded: Part fails
Degraded --> Ready: Operator heals
Ready --> Deleting: User deletes
Deleting --> [*]: Cleanup complete
Many operators publish this phase information in the status so that users can see at a glance what is happening. However, good designs go beyond a single phase word and also publish conditions, which we cover next.
9Owner References, Finalizers, Status, and Conditions
Four small features make operators tidy, honest, and safe to delete.
Owner References
A label on a child object saying “this belongs to that parent”. When the parent is deleted, Kubernetes automatically garbage-collects the children.
Finalizers
A named marker that blocks final deletion until the operator finishes cleanup that Kubernetes cannot do itself.
Status
The operator’s written report of reality, including observed generation so users know which spec version it has processed.
Conditions
A list of named checks such as Ready, Available, or Degraded, each with true, false, or unknown, a reason, and a message.
Owner References in Plain Words
When your operator creates a Service and a StatefulSet for a database resource, it marks both as owned by that resource. This has two benefits. First, deleting the database resource automatically removes the Service and StatefulSet without the operator writing any deletion code. Second, the operator can ask the framework to wake it up whenever an owned object changes, so if someone deletes the Service by accident, the operator is notified and recreates it quickly.
Finalizers: The Polite “Wait, Not Yet”
Owner references only clean up things inside Kubernetes. What about things outside, such as a cloud storage bucket, a DNS record, or a backup archive? A finalizer handles that. When a user deletes a resource that carries a finalizer, Kubernetes does not remove it immediately. Instead, it stamps a deletion time on it and waits. The operator sees the deletion time, performs the external cleanup, removes its finalizer name, and only then does Kubernetes finish the deletion.
A finalizer is like the checkout desk at a hotel. You cannot hand back your key and vanish; the staff first checks the minibar, collects any keys, and settles the bill. Only after every desk signs off are you officially checked out.
If the operator is removed from the cluster before the resources it guards are deleted, those resources can hang forever in a “terminating” state, because nobody is left to remove the finalizer. Always plan the order of removal, and keep the operator running until its resources are cleaned up.
Conditions Over Phases
A single phase word, such as “Ready”, cannot express nuance. A database might be reachable but missing a backup, or healthy but mid-upgrade. Conditions let you publish several independent truths at once. They also follow a common convention, which means standard tools and dashboards can interpret them without special knowledge of your operator. Always set the observed generation so users can tell whether the status reflects their latest change or an older one.
10Operator Capability Levels
Not every operator is equally clever. A five-level maturity model helps you judge how much a given operator really does for you.
| Level | Name | What the Operator Can Do |
|---|---|---|
| 1 | Basic Install | Automated provisioning and configuration of the application from a custom resource. |
| 2 | Seamless Upgrades | Patch and minor version upgrades for both the application and the operator itself. |
| 3 | Full Lifecycle | Backup, restore, failure recovery, and storage lifecycle management. |
| 4 | Deep Insights | Metrics, alerts, log processing, and workload analysis exposed for observability. |
| 5 | Auto Pilot | Horizontal and vertical scaling, automatic tuning, anomaly detection, and self-healing decisions. |
The ladder is helpful in two ways. If you are choosing an operator to adopt, it tells you what to expect: a level one operator saves you from writing installation manifests but leaves backups and upgrades to you. If you are building one, it gives you a roadmap. Most teams start at level one, reach level two quickly, and spend the most effort on level three, because backup, restore, and failure recovery are where application knowledge is hardest to encode.
Do not chase level five on day one. A reliable level two operator that never loses data is far more valuable than an ambitious level five operator that sometimes makes surprising decisions.
11Ways to Build an Operator
You can build an operator with many tools. The right choice depends on your team’s skills and how much logic the operator needs.
| Approach | Best For | Strengths | Limits |
|---|---|---|---|
| Helm-based operator | Wrapping an existing chart | Fastest start, reuses packaging you already have | Limited to install and upgrade, little day-two logic |
| Ansible-based operator | Teams fluent in automation playbooks | Readable steps, good for orchestration across systems | Slower, harder to test deeply, less fine-grained control |
| Go with Kubebuilder or Operator SDK | Complex, production-grade operators | Full power, strong typing, rich libraries, large community | Steeper learning curve and more code to maintain |
| Other languages (Java, Python, Rust, .NET) | Teams with strong skills in that language | Use familiar tools and libraries | Smaller ecosystems and fewer shared patterns |
| Higher-level compositions | Platform teams assembling cloud resources | Describe compositions declaratively with little custom code | Less suited to deep application-specific logic |
Scaffolding and Frameworks
Frameworks such as Kubebuilder and the Operator SDK generate the skeleton of a project: the manager, the CRD generation, the RBAC permissions, the packaging, and the test environment. They are built on a shared library called controller-runtime, which provides the informers, caches, queues, and leader election we described earlier. Using a framework means you can concentrate on the unique part, the reconcile logic, rather than rebuilding plumbing that thousands of projects already share.
Start by asking how much the application needs beyond install and upgrade. If the answer is “not much”, a Helm-based approach may be enough. If failover, backup, and data safety are involved, invest in a full programming language so you can test every decision thoroughly.
12Packaging, Distribution, and Lifecycle Management
Operators are software too. They need installing, upgrading, and trusting.
An operator has two parts to install: the CRDs and the operator workload itself. Teams commonly install them through Helm charts, plain manifests applied by a GitOps tool, or a dedicated operator manager. One of the best-known managers is the Operator Lifecycle Manager, which is itself an operator. It reads catalog entries describing available operators, installs them with their dependencies, handles permissions, and performs upgrades along a defined path. Public catalogs, such as OperatorHub, make it easy to discover community and vendor operators.
The Upgrade Puzzle
Upgrading an operator is more delicate than upgrading an ordinary app. CRDs are cluster-wide, so a new CRD version affects every namespace. The operator may need to migrate existing resources to a new schema version. And the new operator version may roll out changes to the applications it manages, such as restarting database pods. Careful teams read release notes, test in a staging cluster, upgrade CRDs before the operator, and confirm that the operator is the only version running.
Read the release notes
Look for schema changes, removed fields, and any automatic restarts the new version may trigger.
Rehearse in staging
Upgrade a non-production cluster with realistic custom resources and watch what the operator changes.
Update the CRDs first
Make sure the new schema is available so the new operator can read and write the fields it expects.
Roll out the operator
Replace the operator workload and confirm that the leader has changed and reconciliation is healthy.
Verify the managed applications
Check status conditions and application health across all custom resources before declaring success.
13Real-World and Industry Examples
Operators are not a theory. Many of the most popular cloud-native projects are delivered as operators.
Prometheus Operator for Monitoring
Instead of hand-editing monitoring configuration files, teams create small resources that describe which services to scrape and which alert rules to apply. The operator generates the configuration and keeps the monitoring servers running and reloaded. This lets every application team own its own monitoring definitions without touching a central file.
Strimzi for Apache Kafka
Running a Kafka cluster involves brokers, coordination, certificates, topics, and users. Strimzi lets teams declare clusters, topics, and users as resources, and handles rolling restarts, certificate renewal, and scaling in a safe order so that messages keep flowing.
CloudNativePG and Postgres Operators
PostgreSQL operators manage primary and replica instances, automatic failover, continuous backup to object storage, point-in-time recovery, and rolling minor upgrades. A developer asks for a database in a few lines and receives a production-grade cluster.
cert-manager for Certificates
Teams request a certificate by creating a resource. The controller talks to the certificate authority, proves domain ownership, stores the issued certificate in a Secret, and renews it before it expires, removing a classic source of outages.
Rook for Storage
Rook turns the Ceph distributed storage system into something that is declared and healed by the cluster itself, handling disks joining and leaving, daemons being replaced, and storage pools being created.
Elastic Cloud on Kubernetes
The operator provisions search clusters, coordinates rolling upgrades, manages security settings, and scales node groups, so that search teams do not run these tasks by hand.
A pattern appears across all of these examples. The user writes a short, domain-flavoured request. The operator, written by people who deeply understand the application, expands it into dozens of lower-level objects and keeps them correct for the life of the system.
14Advantages, Disadvantages, and Trade-offs
Operators are powerful, but they are another piece of software you must trust and maintain.
Advantages
- Captures expert knowledge as repeatable, testable software.
- Gives users a simple, declarative way to request complex systems.
- Heals failures automatically and consistently, day or night.
- Works naturally with GitOps, access control, and audit tooling because everything is a Kubernetes object.
- Enables self-service platforms where teams do not wait for specialists.
- Reduces configuration drift between environments.
Disadvantages and Trade-offs
- Operators have bugs, and a bug in a controller can damage many resources quickly.
- They require high privileges, which increases the security impact if compromised.
- Building and maintaining one demands strong engineering skills and good testing.
- Too many operators in one cluster add resource use, upgrade work, and mental load.
- Hidden automation can surprise people who do not understand what the operator will do.
Operators Compared with Other Tools
| Tool | Main Job | Acts After Install? | Knows Application Rules? |
|---|---|---|---|
| Plain Deployment or StatefulSet | Run containers with basic self-healing | Only basic restarts | No |
| Helm chart | Template and install a set of objects | No, it runs only when you run it | Only as encoded in the template |
| GitOps tool | Keep the cluster matching a Git repository | Yes, but only for manifests | No |
| Operator | Continuously manage an application’s lifecycle | Yes, always watching | Yes, that is its purpose |
These tools are not rivals. A common healthy setup is for a GitOps tool to apply a small custom resource from Git, and for an operator to turn that resource into a running, healthy application. Helm may even be used to install the operator itself.
If your application is stateless and identical across replicas, a standard Deployment with good health checks is usually enough. Build an operator only when there is real operational knowledge worth encoding.
15Security Considerations
An operator is a trusted robot with keys. Treat those keys with care.
Because operators create and modify other objects, they need permissions. Kubernetes controls permissions through role-based access control, shortened to RBAC. A common and dangerous shortcut is to give the operator full cluster administrator rights so that “everything just works”. If the operator is compromised or has a bug, the attacker or the bug can then do anything in the cluster. The principle of least privilege says to give the operator only the specific verbs on only the specific resources it truly needs.
Least-privilege RBAC
List every resource the operator touches and grant only those. Prefer namespace-scoped roles over cluster-wide ones when possible.
Separate user and operator rights
Users who create custom resources should not automatically be able to read the secrets the operator generates for them.
Secure webhooks
Admission webhooks must use trusted certificates and fail safely. A broken webhook can block all changes to its resource kind.
Supply chain hygiene
Pin operator images, verify signatures, scan for vulnerabilities, and review community operators before adopting them.
Hardened pods
Run the operator as a non-root user, with a read-only filesystem and restricted capabilities, and limit its network paths.
Secret handling
Generate strong credentials, store them in Secrets, rotate them on schedule, and never place them in status fields or logs.
Multi-Tenancy Concerns
CRDs are cluster-wide, which means that all tenants share the same definition. A single operator watching all namespaces serves everyone, but one tenant’s bad resource could affect the shared operator’s performance. Many organizations limit the namespaces an operator watches, set quotas on how many custom resources a team can create, and consider running separate operator instances for strongly isolated tenants.
If an operator can create Pods with any service account, then anyone who can create its custom resources may indirectly gain those powers. Always ask: “What is the most powerful thing a user can trick this operator into doing?”
16Monitoring, Logging, and Metrics
You must be able to see what the watcher is doing, or you will not know when the watcher itself is in trouble.
Operators run silently in the background, so observability is essential. There are two layers to watch: the health of the operator program, and the health of the applications it manages. A well-built operator exposes information for both.
Reconcile metrics
How many reconciliations happen, how many fail, and how long they take. A rising failure count is an early warning.
Queue depth and age
How many items are waiting and how long they have waited. A growing queue means the operator cannot keep up.
API client metrics
Request rates and errors against the API server, which reveal throttling or permission problems.
Resource-level status
Counts of custom resources by phase or condition, so dashboards show how many databases are degraded right now.
Events
Kubernetes events attached to the custom resource that tell humans in plain language what the operator did and why.
Structured logs
Log lines that include the resource name, namespace, and a reconciliation identifier, making it easy to follow one story through the noise.
Health Probes and Alerts
The operator should expose liveness and readiness endpoints so Kubernetes can restart it if it hangs. Alerts should fire for repeated reconcile errors on the same object, a leader missing for too long, queue backlogs, and custom resources stuck in a non-ready condition beyond a reasonable time. Without these alerts, a stuck operator can look perfectly healthy while quietly doing nothing.
Write events and conditions for humans, not just for machines. A message such as “Waiting for primary to accept connections” tells an on-call engineer far more than a bare error code.
17Design Patterns for Good Operators
Experienced teams keep returning to the same handful of patterns.
Single responsibility per resource
Each custom resource kind should describe one clear thing, such as a cluster, a backup, or a user, rather than one giant resource doing everything.
Small, ordered steps
Break complex tasks, such as upgrades, into small steps whose results are recorded in status, so the work can safely pause and resume.
Owned objects with owner references
Every child object points back to its parent so cleanup and change notifications work automatically.
Status as the source of progress
Record milestones in status so any new operator instance can pick up exactly where the last stopped.
Operations as separate resources
Model one-time actions, like a backup or a restore, as their own short-lived resources rather than flags on the main resource.
Pause and dry-run switches
Provide a simple way for humans to pause reconciliation of one resource while they investigate, without deleting anything.
Anti-Patterns to Avoid
Problem
The operator reacts to event types separately, with different code paths for created, updated, and deleted, and assumes it will never miss an event.
Why It’s Harmful
Events can be missed during restarts or network breaks. The system then drifts silently, and the different code paths produce inconsistent behavior.
Correct Approach
Use a single reconcile path that always compares desired and actual state, regardless of what triggered it.
Problem
The operator holds important progress information only in its memory, such as which step of an upgrade it has completed.
Why It’s Harmful
A restart or a leader change wipes that knowledge, which can lead to repeated or skipped steps, and possibly data loss.
Correct Approach
Persist progress in the custom resource status or in annotations on owned objects so any instance can resume safely.
Problem
Teams build an operator for an application that does not need one, or create one giant operator that manages many unrelated systems.
Why It’s Harmful
Extra automation adds risk and maintenance cost without benefit, and a sprawling operator becomes hard to test, secure, and reason about.
Correct Approach
Build focused operators only where operational knowledge is real, and keep each one small and clearly scoped.
Problem
The operator reconciles in a tight loop, re-queuing itself instantly when something is not ready, or rewrites objects even when nothing changed.
Why It’s Harmful
It floods the API server, wastes resources, and can trigger endless update cycles with other controllers.
Correct Approach
Update objects only when they truly differ, wait for events when possible, and use delayed re-queues with sensible backoff.
18Best Practices and Common Mistakes
A checklist of habits that separate dependable operators from fragile ones.
Best Practices
- Make every reconcile step idempotent and safe to repeat.
- Validate input with a strong schema and webhooks where needed.
- Use owner references and finalizers for clean deletion.
- Publish conditions, observed generation, and helpful events.
- Test with failures injected: killed pods, slow networks, API errors.
- Version your API and plan conversions before users depend on it.
- Document what the operator will and will not do automatically.
Common Mistakes
- Granting cluster administrator rights for convenience.
- Ignoring conflicts when two writers update the same object.
- Never testing upgrades of the operator itself.
- Putting secrets or large data in status fields.
- Deleting resources without considering data that must be kept.
- Letting the operator manage things humans also edit by hand, leading to tug-of-war.
Protecting Data During Deletion
One of the most important design decisions is what should happen to data when a custom resource is deleted. Deleting a database resource by accident should not instantly erase the only copy of the data. Many mature operators default to retaining storage volumes, or require an explicit setting, or take a final backup before cleaning up. Make destructive behavior opt-in and clearly documented.
Handling Conflicts and Shared Ownership
When two actors try to change the same object, such as a human editing a Service and the operator rewriting it, each keeps undoing the other’s work. A healthy design decides clearly who owns which fields. Modern Kubernetes supports field-level ownership tracking so that an operator can manage only the fields it cares about and leave the rest alone. When in doubt, let the operator own the object entirely and have humans change the custom resource instead.
Testing Strategy
Operators deserve a layered testing approach. Small unit tests verify decision logic on its own. Tests against a lightweight API server verify how the operator behaves with real Kubernetes objects. End-to-end tests in a real cluster verify full flows such as creating a cluster, killing a member, upgrading, and restoring from backup. Because the cost of an operator bug is high, the time invested in thorough testing pays back many times over.
19Frequently Asked Questions
Almost. Every operator contains at least one controller, but the word operator usually means a controller that manages a specific application and comes with its own custom resource types. Built-in controllers, such as the one for Deployments, do not use the word operator.
No. To use an existing operator, you install it and then create custom resources. You only write code when you build a new operator.
The applications it already created keep running, because they are ordinary Kubernetes objects. What stops is the automation: no healing, no scaling, no handling of changes, and no processing of deletions that rely on finalizers. That is why the operator itself should be highly available and monitored.
An operator can watch one namespace, a chosen list, or all namespaces in its cluster. Managing multiple clusters is possible but usually done by running an operator in each cluster, coordinated by a higher-level tool.
Helm renders templates and installs objects once, when you run it. An operator keeps watching after installation and reacts to changes, failures, and upgrades continuously. Helm is a package manager; an operator is a resident expert.
Most often a finalizer is still present and the operator is either not running, lacks permission to finish cleanup, or is failing on an external step. Check the operator’s logs, events on the resource, and permissions before considering any manual removal of the finalizer.
Expose the settings users genuinely need and give everything else safe defaults. A small, well-named spec is easier to learn, easier to validate, and easier to keep compatible as the operator evolves.
No. They are used for monitoring stacks, certificates, networking, storage, machine learning platforms, cloud resources, and internal company platforms. Any time a system has repeatable operational knowledge, an operator can capture it.
There is no fixed number, but each operator adds a workload to maintain, permissions to review, and CRDs to upgrade. Treat each one as a dependency with a cost, and prefer well-maintained ones with active communities.
Yes, which is why safe defaults, backups, retained volumes, careful testing, and limited permissions matter. Choose operators with a clear track record, and rehearse recovery before you need it.
20Summary and Key Takeaways
An operator is Kubernetes learning a new skill. Kubernetes already knows how to compare what you want with what exists and to keep correcting the difference. Operators extend that habit to databases, message brokers, certificates, storage, and any other system whose care requires specialized knowledge. You describe the goal in a custom resource, and the operator’s control loop does the rest, again and again, without tiring.
To use operators well, remember that they are real software with real power. Give them only the permissions they need, watch their health as carefully as you watch the applications they manage, test their behavior under failure, and design them so that all important facts live in the cluster rather than in memory. Done properly, operators turn fragile runbooks into reliable, self-service platforms.
Key Takeaways
- Operator = custom resource plus controller — the resource describes intent, and the controller makes reality match it.
- Everything is a control loop — observe, compare, act, repeat, always based on current state rather than remembered events.
- Reconcile must be idempotent — running it many times must be safe, and all progress must be stored in the cluster.
- Use owner references, finalizers, and conditions — they give you clean deletion, honest status, and tools that understand your resources.
- Security is not optional — apply least-privilege RBAC, secure webhooks, harden the pod, and review every operator you adopt.
- Observe the operator itself — track reconcile errors, queue depth, leader health, and resource conditions, with alerts that humans can act on.
- Build only when it earns its keep — choose operators for real operational complexity, and start with a reliable basic level before adding clever automation.