Operator Maturity Model in Kubernetes
From a simple installer to a self-driving system: the five capability levels of Kubernetes Operators, what each one really means, how to climb the ladder safely, and how to judge where your own operators stand, explained from zero.
Imagine you hire someone to look after your house while you travel. On the first day, a beginner shows up, unlocks the doors, and turns on the lights. That is helpful, but the moment a pipe leaks, you get a phone call. A better caretaker also repairs and upgrades things without moving you out. A still better one handles emergencies, keeps backups of your valuables, and sends you a weekly report. The best one notices a small crack in the wall before it becomes a leak, and fixes it before you ever hear about it.
Software systems have the same kind of caretakers, and in Kubernetes they are called Operators. An Operator is a program that knows how to run one particular application, such as a database or a message queue, the way an expert human would. But not all Operators are equally capable. Some only install software. Others can heal, back up, tune, and scale it with no human involved. The Operator Maturity Model, officially called the capability levels, is a simple five-step ladder that describes how much an Operator can really do. This post explains that ladder from the very beginning, assuming you know nothing yet. By the end, you will understand every level, how to build and test an Operator that reaches it, how to judge the Operators you depend on, and how to talk about all of this confidently in interviews and design reviews.
1What Is an Operator, and Why Does It Exist?
Before we measure maturity, we need to be clear about what is being measured. Let us build the idea from the ground up.
The problem Kubernetes cannot solve alone
Kubernetes is excellent at running stateless programs, which are programs that keep no important data inside themselves. If a web server pod dies, Kubernetes simply starts another one, and nothing is lost. But some applications are stateful. A database holds your customers’ orders. A message queue holds messages that have not been delivered yet. These programs have strict rules: which copy is the leader, which copies are followers, in what order they must start, and how data is copied between them. Kubernetes, by itself, knows none of this. It sees only containers.
Where the human expert used to fit in
Traditionally, a skilled database administrator or site reliability engineer carried that knowledge in their head. They knew how to create a replica, how to promote a replica when the leader died, how to take a safe backup, and how to upgrade without losing data. Every time something went wrong at three in the morning, a human woke up and ran a careful sequence of commands.
The Operator idea
In 2016, engineers at a company called CoreOS proposed a clever idea: write that human expert’s knowledge as software and run it inside the cluster. They called the result an Operator, because it does the work of a human operator. Their first examples were Operators for etcd and Prometheus. The idea spread quickly, and today there are Operators for databases, message brokers, monitoring systems, storage systems, and even cloud services.
Think of a self-service coffee machine. A plain machine only heats water; you must do everything else. A good machine grinds beans, tamps, brews, and cleans itself. An Operator is like a machine that has absorbed the knowledge of a skilled barista: you say “one large latte”, and it handles every step, including what to do when the milk runs out.
The two ingredients
Custom Resource
A new kind of object added to Kubernetes, such as “PostgresCluster”, in which you describe the outcome you want in plain settings.
Controller
A program that watches those objects and keeps working until the real world matches what you described.
Production example: Prometheus Operator
The Prometheus Operator is one of the best-known examples. Instead of hand-writing monitoring configuration files, teams create small Kubernetes objects that say “monitor this service” or “alert on this condition”, and the Operator builds and reloads the real configuration automatically. It turned a tedious expert task into a few lines of declared intent, which is exactly the promise of the whole Operator pattern.
An Operator is not a magic tool. It is ordinary software that happens to run inside the cluster and speak the Kubernetes language. Its quality depends entirely on how much of the expert’s knowledge was captured in it. The maturity model is a way to measure that.
2The Building Blocks: Custom Resources and Controllers
To understand why some Operators are more capable than others, you need to see the machinery inside every one of them.
Declared state and real state
Kubernetes works on a simple principle. You describe what you want in an object, which is called the desired state. The cluster looks at what actually exists, called the actual state. A program called a controller compares the two and takes actions to close the gap. This continues forever. If something breaks and the actual state drifts away, the controller notices and fixes it. This is the foundation of everything Kubernetes does.
Custom Resource Definitions
Kubernetes comes with built-in object types such as Pod, Service, and Deployment. A Custom Resource Definition, shortened to CRD, lets you teach the cluster a brand-new object type. Once the CRD is installed, users can create objects of that type just like any built-in one, use the same command-line tools, and apply the same access rules. The new object is called a custom resource.
The reconciliation loop
Inside every Operator there is a reconcile loop. Each time something related to a custom resource changes, the loop runs. It reads the desired state, looks at the actual state, decides what to do, does one small step, and reports the result back in the object’s status. Because the loop can run many times, every step must be safe to repeat. Engineers call this property idempotent: doing it twice has the same effect as doing it once.
flowchart TD
U["User creates or edits a custom resource"] --> API["Kubernetes API server stores the object"]
API --> W["Controller is notified of the change"]
W --> R["Reconcile loop starts"]
R --> C["Compare desired state with actual state"]
C --> D{"Any difference?"}
D -- "No" --> S["Update status and wait for the next event"]
D -- "Yes" --> A["Take one safe action: create, update, or repair"]
A --> S
S --> W
Why “level-triggered” matters
A well-built controller does not rely on remembering the exact event that woke it up. Instead, it looks at the full current picture every time. Engineers call this level-triggered design, in contrast with edge-triggered design, which reacts only to the change itself. Level-triggered controllers recover gracefully from missed events, restarts, and crashes, because the next run simply re-examines everything. This is one of the main reasons the Kubernetes model is so resilient.
An edge-triggered thermostat turns the heater on only at the instant someone presses a button, and forgets everything after. A level-triggered thermostat keeps checking the room temperature all the time and keeps adjusting. If the power cuts out for an hour and comes back, the second thermostat fixes the room by itself. The first would sit there confused.
The helper machinery
| Piece | What it is | Why Operators need it |
|---|---|---|
| Informer and cache | A local, always-updated copy of the objects the controller cares about | Avoids hammering the API server with repeated reads |
| Work queue | A line of objects waiting to be reconciled, with retries and delays | Handles bursts of events and failed attempts gracefully |
| Owner references | Links from created objects back to the custom resource that owns them | Lets Kubernetes clean up everything automatically when the owner is deleted |
| Finalizers | Markers that pause deletion until cleanup work is done | Allows safe teardown of external resources such as cloud disks |
| Status and conditions | A section of the object where the Operator reports progress and health | Gives users and tools a clear picture of what is happening |
| Leader election | A way for several copies of an Operator to agree that only one acts at a time | Prevents two copies from fighting over the same object |
A controller is the general mechanism, and an Operator is a controller that carries application-specific operational knowledge. The built-in Deployment controller is a controller, but nobody calls it an Operator, because it knows nothing special about any particular application.
3Why We Need a Maturity Model
Walk through any Operator catalog and you will find hundreds of entries, all described with the same friendly words. A maturity model helps you see through the marketing.
The honesty problem
Two Operators may both claim to “manage PostgreSQL”. One may merely generate the starting configuration and then step away. The other may monitor replication, promote a new leader during a failure, take verified backups, restore them on demand, and tune its own memory settings. From the outside, both look like a single custom resource. Without a shared scale, you cannot tell them apart until a crisis arrives.
What the model gives you
A shared vocabulary
Teams can say “we need at least Level 3 for production databases” and everyone understands what that means.
A roadmap
Authors can plan which abilities to add next instead of building features at random.
A way to compare
You can compare two Operators for the same application on equal terms.
A staffing guide
The lower the level, the more human work your team still carries.
Where the model comes from
The five capability levels were introduced by the Operator Framework, an open-source project that also provides the Operator SDK for building Operators and the Operator Lifecycle Manager for installing and updating them. Public catalogs, such as OperatorHub, display a capability level for many listed Operators so that users can see at a glance how far each one goes.
The level shown for an Operator is usually declared by its authors, not independently audited. Treat it as a starting claim and verify it against your own needs, especially for the abilities that matter most to you, such as backup and recovery.
What the model does not measure
The levels describe capability, meaning what the Operator can do. They do not measure code quality, security, performance, community health, or how well the Operator is documented. A Level 5 Operator that is poorly tested can be more dangerous than a well-tested Level 2 one. Throughout this post, we will keep these separate and give you checklists for the other qualities too.
What the model is good for
- Comparing Operators on feature depth.
- Planning a roadmap for your own Operator.
- Explaining automation goals to managers.
What it cannot tell you
- Whether the Operator is reliable or secure.
- Whether it fits your specific workload.
- Whether anyone still maintains it.
4The Five Levels at a Glance
Here is the whole ladder on one page. The next five chapters visit each step in depth.
flowchart BT
L1["Level 1: Basic Install - automated setup and configuration"] --> L2["Level 2: Seamless Upgrades - patch and minor version updates"]
L2 --> L3["Level 3: Full Lifecycle - backup, recovery, failure handling"]
L3 --> L4["Level 4: Deep Insights - metrics, alerts, logs, analysis"]
L4 --> L5["Level 5: Auto Pilot - scaling, tuning, self-healing, anomaly detection"]
| Level | Name | Plain-English meaning | Human still needed for |
|---|---|---|---|
| 1 | Basic Install | It can set the application up and configure it from a declared description | Upgrades, backups, failures, tuning, everything else |
| 2 | Seamless Upgrades | It can upgrade the application and itself without losing data or causing long outages | Backups, failure recovery, tuning |
| 3 | Full Lifecycle | It handles storage, backup, restore, and recovery from failure | Understanding trends, tuning, capacity decisions |
| 4 | Deep Insights | It exposes rich metrics, alerts, and logs, and can analyze the workload | Acting on the insights |
| 5 | Auto Pilot | It scales, tunes, and heals the application on its own, and spots abnormal behavior | Setting goals and handling truly unusual events |
The ladder is cumulative
Each level assumes the abilities of the levels below it. An Operator cannot honestly claim Level 3 if it cannot upgrade safely, because safe upgrades come at Level 2. Similarly, automatic tuning at Level 5 depends on the metrics gathered at Level 4. This is why you should climb in order rather than jump to the exciting top steps first.
Learning to drive works the same way. First you learn to start the car and steer (Level 1). Then you learn to change gears and maintain it (Level 2). Then you learn what to do in a breakdown (Level 3). Then you learn to read the dashboard and warning lights (Level 4). Finally the car drives itself using what it senses (Level 5). Nobody can safely skip straight to self-driving.
A shortcut to remember the levels
Install
Can it put the application in place correctly?
Upgrade
Can it move the application to a newer version safely?
Protect
Can it back up, restore, and survive failures?
Observe
Can it tell you what is happening inside?
Act
Can it adjust and heal by itself using what it observes?
Install, Upgrade, Protect, Observe, Act. If you can recite those five words and explain each with one example, you can answer most questions about the maturity model.
5Level 1: Basic Install
The first step is the simplest: turn a short description into a correctly running application. Many Operators begin and sometimes end here.
What Level 1 means
At Level 1, the Operator provisions the application and applies its configuration automatically. You create a custom resource that says, for example, “I want a three-member message cluster with these settings”. The Operator creates the pods, the services, the storage claims, the configuration files, and the secrets, and wires them together so the application starts correctly. It also keeps the configuration in line with what you declared: if someone edits a setting in the custom resource, the Operator applies it.
What it saves you
Without an Operator, installing a clustered application often means filling in dozens of files and running several steps in the correct order. A single forgotten setting can cause a quiet failure that shows up weeks later. Level 1 replaces that pile of steps with one declaration, and, just as importantly, makes the installation repeatable. The same description produces the same result in a test cluster, a staging cluster, and production.
Level 1 is the caretaker who unlocks the house, turns on the water, switches on the lights, and sets the thermostat exactly as you wrote on the note. The house works on day one. But when winter comes and a pipe freezes, the caretaker has no instructions.
What a Level 1 Operator typically does
Builds the pieces
Creates the workloads, services, storage, and settings the application needs.
Applies settings
Turns the simple fields of the custom resource into the application’s real configuration files.
Rejects bad input
Refuses nonsense settings early, with a clear message, rather than failing later.
Shows status
Writes a basic status, such as “creating”, “ready”, or “failed”, into the custom resource.
What it does not do
Strengths of Level 1
- Fast, repeatable installs.
- Fewer configuration mistakes.
- Easy to understand and to build.
- Works well for stateless or easily replaced applications.
Gaps at Level 1
- No safe upgrade path guaranteed.
- No backups or recovery.
- Little or no health logic beyond restarting pods.
- Humans still handle every real incident.
Production example: Helm-style operators
Many Operators created from Helm charts sit at this level. They convert a chart into a custom resource so that installing and changing settings is a Kubernetes-native action. This is genuinely useful for platform consistency, but a chart on its own knows nothing about how to promote a replica or restore a backup, so such Operators rarely climb far beyond the first two steps without extra work.
When evaluating a Level 1 Operator, ask: “If I delete the custom resource and recreate it with the same settings, do I get the same running system?” A trustworthy answer is yes. This is the repeatability that makes Level 1 valuable.
6Level 2: Seamless Upgrades
Installing is the easy part. The real test of any operational tool is what happens a year later, when the software needs to move to a newer version.
What Level 2 means
At Level 2, the Operator can upgrade the application it manages, including patch releases and minor version changes, with minimal downtime and without losing data. It also supports upgrading itself. You change the version in the custom resource, and the Operator carries out the careful sequence needed to get there.
Why upgrades are hard
Think about a database with one leader and two followers. If you restart them all at once, the service goes down. If you restart the leader first, you may lose recent writes. The safe approach is a rolling upgrade: update one follower, wait until it has caught up, update the next, and only then switch the leader role to an upgraded copy and update the old leader last. Each step must check that the system is healthy before moving on. Different applications have different rules, and some versions require special data migrations that must run exactly once.
sequenceDiagram
participant U as User
participant O as Operator
participant F1 as Follower 1
participant F2 as Follower 2
participant L as Leader
U->>O: Change version in the custom resource
O->>F1: Upgrade and restart
F1-->>O: Healthy and caught up
O->>F2: Upgrade and restart
F2-->>O: Healthy and caught up
O->>L: Hand leadership to an upgraded member
O->>L: Upgrade the old leader
L-->>O: Healthy
O-->>U: Status shows upgrade complete
The upgrade promises
Check before changing
Confirm the current system is healthy and the target version is a supported jump.
Go one step at a time
Upgrade members in a safe order, never all at once, so the service stays available.
Verify after each step
Wait for health signals, not just a timer, before moving on.
Stop if something looks wrong
Pause and report instead of pushing ahead into a broken state.
Upgrading the Operator itself
An Operator is also software with versions. A new Operator version may change how it manages existing applications. A mature Operator must be able to take over applications that an older version created, without recreating them. The Operator Lifecycle Manager helps by describing which Operator versions can replace which, so that the cluster only follows supported upgrade paths.
Upgrading the Operator can accidentally trigger changes to the applications it manages, such as a rolling restart caused by a changed default. Good Operators avoid this, and good teams test Operator upgrades in a copy of production before applying them for real.
Level 2 is a caretaker who can renovate the kitchen while you still live in the house. They replace the stove one day, the sink another day, and never leave you without water or a place to cook. They also know that some changes, like rewiring, must wait for the right moment.
Production example: managed Kafka on Kubernetes
Operators for Apache Kafka, such as Strimzi, are well known for coordinating rolling restarts of brokers in a safe order during upgrades and configuration changes, waiting for partitions to be fully in sync before touching the next broker. This kind of careful sequencing is the hallmark of Level 2 behavior.
7Level 3: Full Lifecycle
Level 3 is where an Operator starts to earn the trust of people running important data. It covers everything that happens between birth and retirement of the application, including the unhappy parts.
What Level 3 means
At Level 3, the Operator manages the whole application lifecycle and its storage. That includes taking backups, restoring from them, recovering from failures, and handling data safely when the application is changed or removed. A human no longer has to run special scripts to protect the data.
The four big abilities
Backups
Schedules regular backups, stores them somewhere safe such as object storage, and tracks which ones exist.
Restore
Creates a new application instance from a chosen backup, or rewinds an existing one to a point in time.
Failure handling
Detects a failed member and replaces or repairs it, including promoting a new leader when needed.
Clean shutdown
Removes the application in a controlled way, deciding what happens to its data, using finalizers.
Failure handling in detail
Failures come in many sizes. A pod may crash and restart, which Kubernetes handles alone. A whole machine may disappear, taking the pod and its local storage with it. A network split may leave two parts of the cluster each believing the other is dead. The Operator must tell these cases apart and react correctly. Wrongly promoting a second leader during a network split can cause two copies of the data to diverge, a dangerous problem often called split brain. Mature Operators use careful rules, often relying on a trusted coordination mechanism, to make sure only one leader exists at a time.
flowchart TD
F["Operator detects an unhealthy member"] --> Q{"Is it a pod crash, a lost node, or a network split?"}
Q -- "Pod crash" --> R1["Let Kubernetes restart it and watch recovery"]
Q -- "Lost node" --> R2["Create a replacement member and resync data from healthy copies"]
Q -- "Network split" --> R3["Keep the side with a majority, fence the other side"]
R1 --> OK["Update status: healthy"]
R2 --> OK
R3 --> OK
A backup is only real if you can restore it
A famous saying in operations is that nobody has backups; they only have restores. A backup that has never been restored might be empty, corrupted, or missing a key. A strong Level 3 Operator does not stop at creating backups. It also provides a tested restore path, and ideally verifies backups automatically by restoring them into a throwaway copy and checking the result.
What Level 3 gives you
- Real protection for your data.
- Recovery from many failures without a human.
- Clear, repeatable restore procedures.
- Confidence to run important data on Kubernetes.
What Level 3 still lacks
- Limited visibility into how the application behaves.
- No automatic tuning for load.
- Humans still decide when to scale.
Level 3 is a caretaker who keeps copies of your important papers in a safe deposit box, knows exactly what to do when a pipe bursts, and can rebuild a damaged room from the original plans. You can sleep better, even though the caretaker still does not tell you how the house is aging.
Production example: PostgreSQL operators
Several PostgreSQL Operators, including those from Zalando, Crunchy Data, and the CloudNativePG project, are widely used in production. They typically manage automatic failover between a primary and replicas, scheduled backups to object storage, and point-in-time recovery. Teams choose among them partly by how thoroughly each handles these Level 3 abilities.
8Level 4: Deep Insights
Protecting the application is not enough if you cannot see how it behaves. Level 4 is about visibility: turning the Operator into a source of understanding, not just action.
What Level 4 means
At Level 4, the Operator provides rich metrics, alerts, log processing, and workload analysis. It does not merely run the application; it measures it and explains what it sees. The goal is to give humans and, later, automated systems enough information to make good decisions.
The four kinds of insight
| Insight | What it contains | Example for a database |
|---|---|---|
| Metrics | Numbers measured over time | Connections in use, replication lag, cache hit ratio, disk growth |
| Alerts | Rules that warn when something is wrong or about to be | Replica falling behind, disk will be full in two days, backup missed |
| Logs | Event records, gathered and organized | Slow queries collected and summarized |
| Analysis | Interpretation of the workload | Recommended indexes, unusual traffic patterns, capacity forecasts |
Good insight is application-aware
Kubernetes already shows generic numbers like CPU and memory. Those are not enough. A database can have perfectly normal CPU while its replicas silently fall hours behind. Only an Operator that understands the application knows which signals matter. Level 4 Operators export application-specific metrics in a standard format, ship ready-made dashboards and alert rules, and expose the health of each member in the custom resource status.
flowchart LR
APP["Managed application"] --> EXP["Operator-provided metrics and health checks"]
EXP --> MON["Monitoring system"]
MON --> DASH["Dashboards for humans"]
MON --> ALERT["Alerts to on-call team"]
EXP --> STAT["Custom resource status and events"]
STAT --> HUMAN["Engineers using kubectl or consoles"]
Insight about the Operator itself
A mature Operator also tells you how it is doing. How many reconcile loops ran, how many failed, how long they took, how many objects are waiting in the queue? Without these numbers, you cannot tell whether a stuck application is the application’s fault or the Operator’s. We return to this in the chapter on observing the Operator.
Does the Operator ship dashboards and alert rules out of the box? Does the custom resource status explain problems in plain words? Can you tell from the status alone which member is lagging and why? If the answers are yes, you are looking at real Level 4 behavior.
Level 4 is a caretaker who not only fixes problems but also sends you a monthly report: water use is up, the roof has a small leak forming, the boiler is five years old. You are not asked to do anything automatically, but you now see the house clearly and can plan.
Production example: monitoring-aware operators
Operators for systems such as Elasticsearch, Kafka, and Ceph commonly integrate with the Prometheus monitoring stack by exposing metrics and shipping alert rules. Teams that adopt them gain consistent dashboards across many clusters, which makes the on-call experience far more predictable than when every cluster is monitored by hand.
9Level 5: Auto Pilot
The top of the ladder is the most exciting and the easiest to get wrong. At Level 5, the Operator does not just show the problem; it acts on it.
What Level 5 means
At Level 5, the Operator performs horizontal and vertical scaling, automatic configuration tuning, abnormality detection, and scheduling tuning. It uses the insights gathered at Level 4 to adjust the application without waiting for a human. The aim is a system that runs itself, with people setting goals and boundaries rather than pushing buttons.
The auto-pilot abilities
Horizontal scaling
Adds or removes members when load changes, for example adding read replicas during a busy season.
Vertical scaling
Gives members more memory or processor power when they are running short.
Automatic configuration
Adjusts application settings, such as memory pools, to match the workload it observes.
Anomaly detection
Notices abnormal patterns, like a sudden jump in errors, and responds or raises an alarm.
Smart scheduling
Chooses where members run, spreading them across zones and avoiding noisy neighbors.
The feedback loop
Auto pilot works through a loop with four steps: observe, decide, act, and check the result. The Operator measures the system, compares the measurements with goals such as “keep response time under 50 milliseconds”, decides on an adjustment, applies it carefully, and then measures again to confirm that it helped.
flowchart LR
O["Observe: metrics and health"] --> D["Decide: compare with goals and limits"]
D --> A["Act: scale, tune, or heal in small steps"]
A --> C["Check: did the change help?"]
C --> O
Why Level 5 is dangerous when rushed
An automated system can make mistakes much faster than a human. A badly tuned autoscaler can add machines in a loop and run up a huge bill. An overeager tuning rule can set a memory value that crashes the database. Two automatic systems, such as the Operator and a cluster autoscaler, can fight each other. For these reasons, Level 5 abilities should always come with guard rails: minimum and maximum limits, slow and gradual changes, a way to pause automation, and clear logs of every decision.
Benefits of auto pilot
- Faster reaction than any human on call.
- Lower cost through right-sizing.
- Fewer middle-of-the-night pages.
- Consistent behavior across many clusters.
Risks of auto pilot
- Mistakes repeat at machine speed.
- Harder to understand why something changed.
- Conflicts with other automation.
- Requires excellent Level 3 and 4 foundations.
Level 5 is a possible ceiling, not a requirement. Many excellent Operators deliberately stop at Level 3 or 4, because automatic tuning of that particular application is risky or not worth the complexity. A higher level is not always better; the right level depends on the application and the risk you can accept.
Production example: controllers that act on their own
Kubernetes has well-known controllers that behave in the auto-pilot spirit. The Horizontal Pod Autoscaler adjusts replica counts based on measured load, and Karpenter on AWS watches pending pods and chooses new machines automatically. They show both the power of closed feedback loops and the need for careful limits, since a runaway loop here means real money spent.
10Choosing How to Build: Helm, Ansible, or Go
The tool you pick to write an Operator quietly decides how high it can climb. Understanding this saves teams from painful rewrites later.
The three classic paths
The Operator SDK, part of the Operator Framework, lets you create an Operator in three main ways. Each trades ease against power.
| Approach | How it works | Strength | Typical ceiling |
|---|---|---|---|
| Helm-based | Wraps an existing Helm chart so that a custom resource fills in its values | Fastest start; reuse what you already have | Level 1 to 2 |
| Ansible-based | Runs automation playbooks in response to custom resource changes | Good for teams that already automate with Ansible; flexible steps | Up to Level 5 in principle, with effort |
| Go-based | A full program written with Kubernetes controller libraries | Maximum control, speed, and testability | Level 5 |
Why Helm tops out early
A Helm chart is a template that renders Kubernetes objects once. It has no built-in way to watch a running database, decide that a replica fell behind, take a backup, or coordinate a careful sequence. A Helm-based Operator can reapply the chart when settings change, but it cannot reason about the live system. That is why such Operators fit Levels 1 and 2 and rarely go further.
Other tools you will meet
Kubebuilder and controller-runtime
Widely used libraries and scaffolding for writing controllers in Go. The Operator SDK’s Go path is built on them.
Java Operator SDK
A framework for teams who prefer to write Operators in Java.
Kopf and similar
Lightweight options for simple Operators or quick prototypes in Python.
Metacontroller, KUDO
Frameworks that let you describe controller behavior with less code, with their own trade-offs.
How to choose
Decide your target level honestly
If you only need repeatable installs, Helm may be enough. If you need failover and backups, plan for code.
Look at your team’s skills
An Operator nobody on the team can maintain becomes a liability, however clever it is.
Think about testing
Real code is easier to unit test and to simulate failures against than a long chain of templates.
Plan for the long run
Choose a tool that can carry you to the level you will need in two years, not only today.
Teams often start with a quick Helm-based Operator, then discover they need failover logic that Helm cannot express. Moving to Go at that point means redesigning the custom resource and migrating live systems. If you suspect you will need Level 3 or higher, begin with a tool that can support it.
11Designing the API: Your Custom Resource Is a Promise
The custom resource is the front door of your Operator. Once people start using it, changing it is expensive, so a little careful design here pays off for years.
Spec and status
Almost every custom resource has two main sections. The spec is what the user writes: the desired outcome. The status is what the Operator writes: the observed reality. Keeping them separate is a golden rule. Users never edit status, and the Operator never edits spec. This division keeps responsibilities clear and prevents the two sides from overwriting each other.
Describe intent, not steps
A good spec says what the user wants, not how to do it. “Three replicas, version 15, daily backups at two in the morning” is intent. “Run these four commands in this order” is a script. The Operator is the one who knows the steps. A spec that exposes too many internal knobs becomes hard to evolve and easy to misuse.
Signs of a good API
- Few, meaningful fields with sensible defaults.
- Clear names that a newcomer can guess.
- Validation that rejects mistakes immediately.
- Status that explains what is happening.
Signs of a weak API
- Dozens of raw configuration options passed through.
- Fields that only make sense in a certain order.
- Silent acceptance of invalid values.
- Status that just says “ok” or “error”.
Status conditions
Instead of a single word, mature Operators report a list of conditions, each describing one aspect of health, such as “Ready”, “BackupHealthy”, or “UpgradeInProgress”. Each condition has a true or false value, a short reason, a human-readable message, and a timestamp of the last change. Tools and dashboards can then show a precise picture, and automation can wait for a particular condition to become true.
Validation
CRDs can carry a schema that checks every incoming object. Simple rules, such as numbers within a range or text matching a pattern, can be enforced by the cluster itself before the Operator even sees the object. More complex checks can be done by a validating webhook, a small service the API server asks before accepting a change. Early, clear validation is far kinder to users than a mysterious failure hours later.
Versioning the API
APIs change. Fields are added, renamed, or removed. Kubernetes lets a CRD serve several versions at once, such as v1alpha1, v1beta1, and v1, and use a conversion webhook to translate objects between them. The names carry meaning: alpha means early and subject to change, beta means mostly stable, and v1 means users can rely on it. Mature Operators promise not to break existing objects when they release new versions.
flowchart LR
A["v1alpha1: experimental, may break"] --> B["v1beta1: mostly stable, tested widely"]
B --> C["v1: stable, compatibility promised"]
C -.-> D["Conversion webhook translates between served versions"]
The custom resource is the menu of a restaurant. A good menu lists dishes by what they are, not by cooking instructions, stays consistent from month to month, and tells you when a dish is sold out. A bad menu lists raw ingredients and leaves the cooking to you, or changes every week so regulars cannot order.
Write down how a new user will create their first object before you finalize the API. If the example is longer than a screen, the API probably exposes too much. Defaults should make the shortest example work.
12Reconciliation Done Right
A large part of the difference between a toy Operator and a trustworthy one is how carefully the reconcile loop is written. These habits show up at every level of the maturity model.
Habit one: be idempotent
Every pass of the loop must be safe to run again. Instead of “create a volume”, write “make sure a volume exists”. If it already exists, do nothing. If a previous run stopped halfway, the next run finishes the job. This single habit makes crashes, restarts, and retries harmless.
Habit two: take small steps
Do a little, record progress in the status, and return. Long operations that block the loop for minutes make the Operator slow to react to other objects. Instead, start the operation, return, and come back to check on it. Operators that move one step at a time are easier to debug, easier to pause, and safer during upgrades.
Habit three: own what you create
Set an owner reference on every object the Operator creates, pointing back to the custom resource. When the custom resource is deleted, Kubernetes automatically removes everything it owns. Without owner references, deleted applications leave orphaned storage, services, and settings behind.
Habit four: clean up with finalizers
Some things cannot be cleaned up by Kubernetes alone, such as a cloud disk, a DNS record, or a backup in external storage. A finalizer is a label on the object that tells Kubernetes: “do not fully delete this until I say so”. When a user deletes the custom resource, the Operator sees the deletion request, performs the external cleanup, removes its finalizer, and only then does the object disappear. If the Operator forgets to remove the finalizer, the object hangs in a deleting state forever, so this logic must be well tested.
User deletes the object
Kubernetes marks it for deletion but keeps it because a finalizer is present.
Operator notices
The reconcile loop sees the deletion timestamp and switches into cleanup mode.
Cleanup runs
It deletes external resources, perhaps taking a final backup first if the user asked for it.
Finalizer removed
Kubernetes can now finish deleting the object and its owned children.
Habit five: handle conflicts and retries
Several actors may change the same object at once: the user, the Operator, other controllers. Kubernetes protects against lost updates by rejecting a write that is based on an outdated version, called a conflict. A well-written Operator simply re-reads and tries again. Failures should also be retried with growing delays, called exponential backoff, so that a broken dependency is not hammered with requests.
Habit six: report honestly
Write real status after each meaningful step, including failures and their reasons. Emit Kubernetes events for notable actions. An Operator that fails silently looks fine right up until the moment it is not.
If an Operator updates the object’s status in a way that triggers another reconcile, which updates the status again, it can spin forever, consuming CPU and flooding the API server. Only write when something actually changed, and ignore changes to the parts of the object the Operator itself writes.
| Habit | Question to ask during code review |
|---|---|
| Idempotent | If this step ran twice, would anything break? |
| Small steps | Can the loop return quickly and resume later? |
| Ownership | Does every created object have an owner reference? |
| Finalizers | What happens if cleanup fails halfway? |
| Conflicts | What if the object changed while we were working? |
| Honest status | Would a stranger understand the status without reading the code? |
13Packaging and Lifecycle Management
An Operator that lives only on its author’s laptop helps nobody. Level 2 maturity applies to the Operator too: it must be installable, upgradable, and removable in a clean way.
The Operator Lifecycle Manager
The Operator Lifecycle Manager, usually called OLM, is a component of the Operator Framework that installs Operators, keeps them updated, and manages their permissions. Think of it as an app store and update service for Operators inside the cluster. It understands which versions can replace which, and it can pause automatic updates when you want to approve them manually.
The key objects
ClusterServiceVersion
A description of one Operator version: its deployment, permissions, the custom resources it owns, and its documentation.
CatalogSource
A source of available Operators, like a shelf in the store.
Subscription
A request to follow one Operator through a chosen update channel.
InstallPlan
The concrete list of steps OLM will perform, which can require human approval.
Update channels and approval
Operators publish versions into channels, such as a stable channel for tested releases and a fast channel for newer ones. A subscription chooses a channel and an approval mode: automatic, where OLM applies updates as they appear, or manual, where an administrator reviews each install plan first. Production clusters often choose manual approval for critical Operators so that upgrades happen at planned times.
flowchart LR
CAT["Catalog of Operator versions"] --> SUB["Subscription: chosen channel and approval mode"]
SUB --> PLAN["Install plan: steps OLM will take"]
PLAN --> APPR{"Manual approval needed?"}
APPR -- "Yes" --> ADMIN["Administrator reviews and approves"]
APPR -- "No" --> RUN["OLM installs or upgrades the Operator"]
ADMIN --> RUN
Other packaging routes
OLM is not the only way. Many Operators are also installed with Helm charts or plain manifests, which is simple and works on any cluster. Some managed platforms provide their own catalogs of add-ons. Whichever route you pick, the same questions apply: how are CRDs upgraded, how are permissions reviewed, how do you roll back, and how do you remove the Operator without destroying the applications it manages?
Helm installs CRDs once and does not upgrade them the way it upgrades ordinary objects. If you rely on Helm, plan a separate step for updating CRDs, or you may run a new Operator against an old schema.
Removing an Operator safely
Deleting an Operator should not accidentally delete your data. If the custom resources are removed along with the CRD, the Operator’s finalizers and owner references may cascade into deleting databases and storage. Mature Operators and good runbooks make the safe order clear: first decide what happens to the managed applications, then remove the Operator, and only at the very end consider removing the CRDs.
14Security: Operators Are Powerful Programs
An Operator can create pods, read secrets, and delete storage. That power is exactly why it must be built and installed with care.
Why Operators are attractive targets
Operators often run with broad permissions, sometimes across the entire cluster, so they can manage applications in many namespaces. If an attacker compromises an Operator, they may gain the ability to read secrets, start pods, or modify any application it manages. The Operator’s identity is therefore a high-value asset.
Principles for safe Operators
Least privilege
Grant only the permissions the Operator truly needs, on only the object types it manages. Avoid cluster-wide administrator rights.
Scope narrowly
Where possible, watch only specific namespaces rather than the whole cluster.
Harden the Operator pod
Run as a non-root user, drop extra capabilities, use a read-only filesystem, and restrict its network access.
Protect the supply chain
Use signed images from trusted sources, pin versions, scan for vulnerabilities, and keep a software bill of materials.
Secure webhooks
Webhooks need valid certificates. Manage them automatically, for example with cert-manager, and plan what happens if they expire.
Protect what the Operator creates
An Operator also decides the security settings of the pods it builds. A careless Operator may create pods that run as root, mount sensitive paths, or expose open network ports. A mature Operator applies safe defaults, supports network policies, uses encryption between members where the application allows, and keeps generated passwords in secrets with tight access.
Healthy security posture
- Narrow roles reviewed before installation.
- Pods built with restricted settings by default.
- Generated credentials stored in secrets, never in logs.
- Clear process for reporting vulnerabilities.
Warning signs
- Installation instructions that say “grant cluster-admin”.
- Images pulled from unknown registries with floating tags.
- Passwords visible in status or events.
- No security policy or maintainer response.
Before installing any third-party Operator, read its role definitions. Ask: which resources can it read, which can it change, and does it really need to touch secrets in every namespace? A few minutes of reading prevents many future problems.
Production example: scoping Operators per team
Platform teams in larger companies often install one instance of an Operator per team namespace, each with permissions limited to that namespace, instead of one powerful instance for the whole cluster. This reduces the blast radius if one instance is compromised or misbehaves, and lets teams upgrade on their own schedule.
15Testing an Operator
The higher an Operator climbs the maturity ladder, the more it can damage when it is wrong. Testing is what lets you climb safely.
Why Operators are hard to test
An Operator interacts with a live cluster, with real storage, and often with a stateful application that has its own quirks. Failures are timing-dependent: a bug may only appear if a node dies during an upgrade. Simple tests that check one function at a time are not enough. A good test strategy uses several layers, each catching a different kind of mistake.
The testing pyramid for Operators
| Layer | What it checks | Speed | Typical tools |
|---|---|---|---|
| Unit tests | Individual decisions, such as “given this state, what is the next action?” | Very fast | Ordinary test frameworks |
| Control-plane tests | The reconcile loop against a real API server without real nodes | Fast | envtest, a helper in the controller libraries |
| End-to-end tests | A real application on a real small cluster | Slow | kind or minikube clusters, test frameworks |
| Upgrade tests | Moving from an old Operator and application version to a new one | Slow | Scripted end-to-end runs |
| Chaos tests | Behavior while pods, nodes, or networks are deliberately broken | Slowest | Fault-injection tools |
Test the level you claim
A very practical idea is to write tests that match the level you advertise. If you claim Level 2, you need automated upgrade tests across every supported path. If you claim Level 3, you need tests that take a backup, destroy the application, and restore it, checking that the data matches. Level 4 needs tests proving that metrics and alerts really fire. Level 5 needs tests showing that scaling and tuning stay within their guard rails under changing load.
Create and change
Create the object, check everything appears, edit settings, check they are applied, delete, check all is cleaned up.
Upgrade paths
Upgrade across each supported version pair while sending traffic, and check there is no data loss.
Break and restore
Kill members, partition networks, corrupt data, and verify recovery and restore results.
Signals fire
Induce known problems and verify the right alerts and metrics appear.
Limits hold
Apply load patterns and verify that scaling respects minimum, maximum, and rate limits.
The Operator SDK includes a scorecard tool that runs a set of basic checks on an Operator’s packaging and behavior. It is not a substitute for deep testing, but it catches common mistakes early and gives teams a shared minimum standard.
Testing an Operator is like testing a hospital’s emergency plan. Reading the plan is not enough. You must run drills where the power goes out, a patient arrives during a shift change, and the main generator fails. Only drills reveal the gaps, and the more life-critical the hospital, the more often it drills.
16Observing the Operator Itself
Level 4 is usually described in terms of the application. But the Operator is a running program too, and it deserves the same visibility.
Why the Operator needs its own monitoring
When an application is unhealthy, the cause might be the application, the cluster, or the Operator itself. If the Operator is stuck, crash-looping, or silently failing to reconcile, nothing gets repaired, and no alarm rings because the application’s own metrics may still look normal for a while. Watching the Operator closes this blind spot.
The signals that matter
| Signal | What it tells you | Alert when |
|---|---|---|
| Reconcile count and errors | How often the loop runs and how often it fails | Error rate stays above normal for several minutes |
| Reconcile duration | How long each pass takes | Passes become much slower than usual |
| Work queue depth | How many objects are waiting | The queue keeps growing and never drains |
| Work queue retries | How many items keep failing and returning | The same items retry repeatedly |
| API request errors and latency | Whether the cluster API is healthy from the Operator’s view | Many throttled or failed calls |
| Leader election state | Whether exactly one copy is active | No leader, or frequent leader changes |
| Restarts and memory use | Whether the Operator is crashing or leaking | Restarts increase or memory climbs steadily |
Logs that tell a story
Operator logs should be structured, so that tools can filter them by object name and namespace. Each reconcile pass should log which object it handled and what decision it made, without printing secrets. When something goes wrong at night, the on-call engineer should be able to search the logs for a single application name and read a clear, ordered story of what the Operator did to it.
Events for humans
Kubernetes events attached to the custom resource are the friendliest form of insight. “Created backup job”, “Promoting replica two to leader”, and “Upgrade paused: member not healthy” show up when users describe the object. They turn the Operator from a black box into a narrator.
flowchart TD
OP["Operator pod"] --> M["Metrics: reconcile counts, durations, queue depth"]
OP --> L["Structured logs per object"]
OP --> E["Events on each custom resource"]
OP --> S["Status conditions on each custom resource"]
M --> DASH["Operator health dashboard"]
M --> AL["Alerts for stuck or failing reconciles"]
L --> SEARCH["Searchable log platform"]
E --> USERS["Users running describe commands"]
S --> TOOLS["Dashboards and automation waiting on conditions"]
An Operator that crashes is easy to notice. An Operator that runs but has stopped making progress, perhaps stuck on one object that blocks the queue, is far more dangerous. Alert on “no successful reconcile in the last N minutes” in addition to error counts.
17Safety Rails: How Operators Go Wrong
The more an Operator can do, the more carefully it must be fenced in. Understanding common failure modes helps you demand the right safety features.
The classic ways Operators fail
Fighting with other controllers
Two controllers each try to set the same field to different values, and the object flips back and forth endlessly.
Mass restarts after an Operator upgrade
A new Operator version changes a default, and every managed database restarts at once.
Acting on wrong information
The Operator misreads a temporary glitch as a real failure and deletes or replaces healthy members.
Runaway scaling
A feedback loop keeps adding resources because its measurement is wrong or delayed.
Cascading deletes
Removing a custom resource or CRD triggers deletion of data that nobody intended to remove.
Safety features worth demanding
Pause switch
A simple way, such as an annotation, to tell the Operator “stop acting on this object for now”, so humans can intervene.
Rate and size limits
Maximum changes per hour and bounds on scaling, so mistakes cannot grow explosively.
Dry run
Ability to show what the Operator would do before it does it, especially for upgrades and deletes.
Delete protection
Storage and backups kept by default when the custom resource is removed, unless the user explicitly chooses otherwise.
Gated steps
No destructive step unless health checks pass and a quorum of healthy members exists.
Decision logs
Every automated action recorded with the reason, so changes can be explained afterward.
Budgets for disruption
Kubernetes provides an object called a PodDisruptionBudget that limits how many members of an application may be unavailable at the same time during voluntary actions such as node maintenance. A mature Operator creates and maintains one for each application it manages, so that routine cluster operations cannot take down too many members together.
Think of a surgeon with a checklist. Even the most skilled surgeon uses a pause before the first cut to confirm the patient, the procedure, and the site. These small rails do not slow down expertise; they prevent the rare catastrophic mistake. Operators need the same discipline, and the higher the level, the more it matters.
An Operator you can trust
- Documents exactly what it will change automatically.
- Provides pause, limits, and delete protection.
- Fails safe: when unsure, it stops and reports.
An Operator to be wary of
- Acts instantly on any signal without checks.
- Deletes storage when the custom resource is removed.
- Cannot be paused without uninstalling it.
When an automated system is unsure, the safest action is to do nothing and ask a human. Great Operators are conservative about destructive steps and generous about explaining themselves.
18Worked Example: A Database Operator Climbs the Ladder
To make the levels concrete, let us follow one imaginary Operator for a replicated database as it grows from a simple installer into an auto-pilot system. The details are illustrative, but the pattern is realistic.
Version one: Level 1
The team writes a custom resource called a “DatabaseCluster”. The user states the version, the number of replicas, and the storage size. The Operator creates the pods, the storage claims, a service for clients, and a configuration file. It marks the first replica as primary and the rest as followers. It works, and installs now take minutes instead of a day. But when the primary pod’s machine dies, the cluster stays without a primary until a human acts.
Version two: Level 2
The team adds careful upgrade logic. The Operator updates followers first, waits for each to catch up, switches the primary role to an upgraded follower, and updates the old primary last. It refuses to upgrade across unsupported versions and pauses if any member is unhealthy. Upgrade night becomes a calm event: the engineer changes one field and watches the status.
Version three: Level 3
Next comes protection. The Operator takes scheduled backups to object storage, records them, and offers a restore field that creates a new cluster from a chosen backup or a point in time. Failure handling arrives: if the primary is lost, the Operator checks that a safe majority is healthy, promotes the most up-to-date follower, and creates a replacement member. The team runs a drill by deleting the primary pod, then the whole node, and finally restoring to a fresh cluster, and measures how long each takes.
Version four: Level 4
The team adds visibility. A small exporter reports replication lag, connection counts, disk growth, and backup age. The Operator ships dashboards and alert rules, writes conditions like “ReplicaLagging” into the status, and emits events for every promotion and backup. Now on-call engineers see trouble coming, such as a disk predicted to fill in three days, instead of only reacting to outages.
Version five: Level 5
Finally, the team uses the data. The Operator can add read replicas when connection load stays high, expand storage before it fills, and adjust memory settings within safe bounds. Every automatic change obeys minimum and maximum limits, is rate limited, and is logged with its reason. A pause annotation lets humans freeze automation during sensitive events.
flowchart LR
V1["v1: install and configure"] --> V2["v2: rolling upgrades"]
V2 --> V3["v3: backups, restore, failover"]
V3 --> V4["v4: metrics, alerts, dashboards"]
V4 --> V5["v5: guarded scaling and tuning"]
| Version | New ability | Human work removed | New risk introduced |
|---|---|---|---|
| 1 | Declarative install | Manual setup steps | Wrong defaults at scale |
| 2 | Rolling upgrades | Hand-run upgrade runbooks | Bad upgrade logic affects every cluster |
| 3 | Backups and failover | Emergency recovery work | Wrong failover can cause data loss |
| 4 | Rich insight | Guessing what is wrong | Alert fatigue if poorly tuned |
| 5 | Guarded automation | Routine scaling and tuning | Automated mistakes at machine speed |
Every step removes some human work and introduces a new kind of risk. The art of Operator engineering is to add each ability only when the tests and safety rails for it are ready. The ladder is a sequence of promises, and each promise must be kept before the next is made.
19Real Operators and Where They Fit
The best way to feel the maturity model is to look at real Operators. The table below describes well-known examples at a general level. Treat the capability notes as approximate, because projects add features over time and declare their own levels.
| Operator | Manages | Notable strengths |
|---|---|---|
| Prometheus Operator | Monitoring stacks | Declarative monitoring configuration; strong integration with alerting rules |
| Strimzi | Apache Kafka | Careful rolling updates, topic and user management, security setup |
| CloudNativePG, Zalando Postgres Operator, Crunchy PGO | PostgreSQL | Failover, backups to object storage, point-in-time recovery, monitoring |
| Rook | Ceph storage | Deploys and manages a distributed storage system, including health and scaling of storage daemons |
| Elastic Cloud on Kubernetes | Elasticsearch and related tools | Orchestrated upgrades, node roles, security configuration |
| cert-manager | Certificates | Automatic issuing and renewal of certificates through custom resources |
| Percona Operators | MySQL, MongoDB, PostgreSQL | Backups, restore, scaling, monitoring integration |
What the examples teach
Databases demand Level 3
Almost every successful database Operator invests heavily in backup, restore, and failover, because data loss is unforgivable.
Not every Operator needs Level 5
Many beloved Operators stop at the point where more automation would add more risk than value.
Controllers can be Operators too
Tools like cert-manager apply the same pattern to a non-database task and show it generalizes beyond data stores.
Ecosystems matter
Operators that plug into standard monitoring, backup, and security tools are far easier to run in production.
How to evaluate a candidate Operator
Read the declared level
Note the claim, then test whether the abilities behind it really exist in the documentation.
Try the failure scenarios
In a test cluster, kill a member, restore a backup, and run an upgrade. Watch what really happens.
Check the project health
Look at release frequency, open issues, number of maintainers, and how security problems are handled.
Read the permissions
Review the roles the Operator requests and consider whether they are justified.
Plan the exit
Ask how you would move the data elsewhere if the project were abandoned.
Production example: competing PostgreSQL Operators
Several communities offer PostgreSQL Operators with different philosophies. Some lean on external tools for failover, others build failover into the Operator itself. Companies commonly run a short proof of concept with two candidates, deliberately breaking each one, before choosing. The comparison is exactly the kind of structured judgment the maturity model supports.
20Maturity Is Also About Your Organization
The five levels measure one Operator. But a company adopting Operators goes through its own growth, and that journey has stages too. Understanding both keeps expectations realistic.
Why organizational maturity matters
A Level 5 Operator in the hands of a team that has never run a stateful application on Kubernetes can still cause trouble. The team may not know how to read its status, rehearse its restores, or tell when automation is misbehaving. Platform maturity is the ability of people and processes to use the tool well. The best results come when Operator capability and team capability grow together.
Four stages of adoption
Experimenting
A few engineers try Operators in test clusters. Nothing important depends on them yet. The goal is learning.
Standardizing
The platform team picks approved Operators, writes runbooks, sets up monitoring, and defines how they are installed and upgraded.
Self-service
Application teams request databases and queues by creating custom resources, and the platform handles the rest within policy limits.
Optimizing
The organization measures cost, reliability, and toil, retires weak Operators, and contributes fixes upstream.
Matching levels to stages
| Adoption stage | Sensible Operator level for important data | Why |
|---|---|---|
| Experimenting | Any, in test environments only | Learning is the goal, so risk must stay low |
| Standardizing | Level 3 or higher for production data | Backup and failover must exist before real data arrives |
| Self-service | Level 3 or 4 with strong guard rails | Many teams depend on it, so visibility and safety matter most |
| Optimizing | Level 4 or 5 where automation is proven safe | Cost and toil savings now outweigh the added risk |
Buying a high-performance sports car does not make you a racing driver. Your skill and the car’s power need to match. A team that is still learning is often safer with a capable but conservative Operator, and then moves to more automated ones as its confidence and tooling grow.
A practical way to judge progress is to count the hours your team spends each month on manual tasks around the application: upgrades, restores, failovers, resizing. As Operator maturity and team maturity rise together, that number should fall steadily.
21Build, Buy, or Do Not Use an Operator at All
The most important decision about Operators is sometimes whether to use one. Writing your own is a long-term commitment, and the best Operator is often the one you did not have to build.
Three honest options
Adopt an existing Operator
Cheapest to start, backed by a community, but you inherit its choices, its release schedule, and its limits.
Write your own
Perfect fit for your needs, but you own the code, the tests, the upgrades, and the on-call forever.
Use a managed cloud service
Let a provider run the database or queue. You trade control and sometimes cost for far less operational work.
A decision flow
flowchart TD
S["You need to run a stateful application"] --> M{"Is a managed cloud service acceptable for cost, control, and compliance?"}
M -- "Yes" --> USEM["Use the managed service and avoid operating it yourself"]
M -- "No" --> E{"Does a well-maintained Operator at a suitable level already exist?"}
E -- "Yes" --> ADOPT["Adopt it, test it, and contribute fixes if needed"]
E -- "No" --> B{"Is the application important enough and your team strong enough to maintain an Operator?"}
B -- "Yes" --> BUILD["Build an Operator, planning tests and safety from day one"]
B -- "No" --> SIMPLE["Use simpler tooling such as charts and runbooks"]
When writing your own makes sense
Good reasons to build
- An internal system with unique operational rules that no public Operator covers.
- A platform product where custom resources are the main interface for developers.
- A team with strong skills in Go or Ansible and the time to maintain the code.
Weak reasons to build
- “It would be fun” or “it will look good on a resume”.
- A public Operator exists that is only slightly different from what you want.
- Nobody is assigned to maintain it after the first release.
Building an Operator is the cheap part. Maintaining it through Kubernetes version changes, application version changes, security patches, and growing feature demands is the expensive part. Count several years of ownership when you estimate the cost.
Operators for your own platform
Not every Operator manages someone else’s software. Many companies write Operators for their own internal concepts: a “Team” object that creates namespaces, quotas, and access rules; a “Service” object that sets up deployments, monitoring, and alerts together. These are often simple Level 1 or 2 Operators, and they can be wonderfully valuable. They turn the company’s standards into a few lines of declared intent and make the right way the easy way.
22Operators and the AWS World
Many readers run Kubernetes on Amazon EKS. This chapter links the maturity ideas to what you will meet on AWS.
Controllers for AWS services
The Operator pattern is not limited to software running inside the cluster. AWS Controllers for Kubernetes, often called ACK, lets you manage AWS services such as storage buckets, queues, and databases through custom resources. You create a Kubernetes object, and a controller creates and maintains the matching cloud resource. Other projects, such as Crossplane, take a similar approach with a wider range of providers. In maturity terms, these tools often focus on the first levels: declarative creation, configuration, and drift correction, while the cloud service itself handles backups and scaling.
Karpenter as an auto-pilot example
Karpenter, an open-source node provisioning project started by AWS, watches for pods that cannot be scheduled and decides which machines to launch, then removes machines that are no longer needed. It is a good everyday illustration of a Level 5-style loop: it observes demand, decides, acts, and checks. It also shows why guard rails such as limits on total capacity are essential.
Choosing between self-run Operators and managed services
| Question | Operator on EKS | Managed AWS service |
|---|---|---|
| Who handles backups and failover? | The Operator, which you must test and monitor | The provider, within documented limits |
| Who patches the engine? | You, through the Operator’s upgrade feature | The provider, on a maintenance schedule you can influence |
| How much control over tuning? | Very high | Limited to exposed parameters |
| Portability across clouds? | High | Lower, tied to provider features |
| Operational burden? | Higher | Lower |
| Cost model? | Pay for compute and storage plus your team’s time | Pay a service premium and save team time |
Practical AWS tips for Operators
Give Operators their own cloud roles
When an Operator needs to reach object storage for backups, use pod-level cloud identities with narrow permissions rather than machine-wide roles.
Plan volumes and zones
Block storage volumes live in one zone. Operators must place members in sensible zones and handle volumes that cannot follow a pod to another zone.
Use durable object storage
Send backups to a bucket with versioning and, for critical data, copies in another region.
Treat Operators like platform add-ons
Version, test, and upgrade them on a schedule that matches your cluster upgrades.
Many teams mix both worlds: managed services for the most critical relational data where the provider’s track record is valuable, and Operators for message brokers, caches, and internal platform objects where flexibility and portability matter more.
23Scoring Yourself: A Practical Assessment
Here is a simple checklist you can use to judge any Operator, yours or someone else’s. It turns the five levels into questions you can answer with evidence instead of impressions.
How to use the checklist
For each level, answer every question with yes or no, and ask to see it work rather than reading a claim. An Operator earns a level only when all the questions for that level and the ones below it are answered yes.
| Level | Questions to answer with evidence |
|---|---|
| 1 | Can I create a working application from a short custom resource? Are settings changes applied? Is deletion clean, with no orphaned objects? Does status show progress and errors? |
| 2 | Can I upgrade the application with no data loss and minimal downtime? Does it check health between steps? Does it reject unsupported version jumps? Can the Operator itself be upgraded without disturbing running applications? |
| 3 | Are backups automatic and stored outside the cluster? Have I restored one successfully? Does it handle the loss of a member or a node? Does it avoid split-brain situations? |
| 4 | Does it expose application-specific metrics? Does it ship alert rules and dashboards? Do status and events explain problems clearly? Does it report its own health? |
| 5 | Does it scale and tune automatically within safe limits? Can automation be paused? Are all automatic actions logged with reasons? Does it detect abnormal behavior? |
Beyond the five levels: the extra score
Because the model ignores quality, add a second, simple scorecard for the things it leaves out. Give each a rating from weak to strong.
Testing depth
Does the project run upgrade, failure, and restore tests automatically for every release?
Permissions and supply chain
Are roles narrow, images signed, and vulnerabilities handled promptly?
Maintenance health
Are there several active maintainers, regular releases, and responsive issue handling?
Documentation and runbooks
Can a new engineer follow the guides to recover from a failure without reading the source code?
Match to your needs
Does it support your cloud, storage, and compliance needs, and its version policy match yours?
flowchart LR
A["Capability level: what it can do"] --> Z["Overall confidence"]
B["Quality and testing: how well it does it"] --> Z
C["Security: how safely it does it"] --> Z
D["Community: how long it will keep doing it"] --> Z
A project with a high capability level, a single part-time maintainer, and no automated upgrade tests is a risk, however good its feature list looks. Combine the level with the extra scorecard before making decisions.
24A Roadmap for Climbing the Ladder
If you are building an Operator, the path upward is long. This roadmap suggests an order of work that keeps risk under control at every step.
Write down the human runbook first
Collect the exact steps an expert follows for install, upgrade, failover, backup, and restore. An Operator can only automate what is understood.
Design the custom resource
Choose a small, intent-based spec and a rich status with conditions. Plan the version strategy from day one.
Reach Level 1 with tests
Build idempotent reconciliation, ownership, finalizers, and validation. Prove them with automated tests before moving on.
Reach Level 2 with upgrade tests
Implement ordered, health-gated upgrades and test every supported upgrade path, including upgrading the Operator itself.
Reach Level 3 with drills
Add backups, restore, and failover. Run regular restore drills and failure drills, and publish how long recovery takes.
Reach Level 4 with real dashboards
Export meaningful metrics, ship alert rules, and improve events and conditions, guided by what on-call engineers actually need.
Approach Level 5 cautiously
Start by recommending actions, then automate one low-risk action at a time behind a feature switch, with limits and a pause control.
Habits that keep the roadmap healthy
Do
- Release small changes often with clear notes.
- Keep a compatibility table of Operator, application, and Kubernetes versions.
- Invite users to share failure stories and add tests for each one.
Avoid
- Adding automation before you can observe the system.
- Breaking the custom resource API without a migration path.
- Declaring a higher level than your tests can demonstrate.
25Operators Compared with Helm, GitOps, and Other Tools
Newcomers often ask whether Operators replace the tools they already use. They do not. Each tool answers a different question, and the best platforms combine them.
Four tools, four different jobs
| Tool | Main question it answers | When it works | What it does not do |
|---|---|---|---|
| Helm | How do I package and install a set of Kubernetes objects with settings? | At install and upgrade time, when a person or pipeline runs it | It does not watch the running system or react to failures |
| GitOps tools such as Argo CD and Flux | Does the cluster match what is stored in Git? | Continuously, comparing cluster objects with the repository | They do not understand application internals such as replication or backups |
| Operators | How do I run this specific application correctly, day after day? | Continuously, inside the cluster | They do not manage your Git workflow or package unrelated applications |
| Infrastructure tools such as Terraform and Crossplane | How do I create the cloud resources underneath the cluster? | At provisioning time, or continuously in the case of Crossplane | They do not manage the application running on top |
Think of building and running a restaurant. Helm is the delivery truck that brings the kitchen equipment in a labeled box. GitOps is the inspector who checks that the kitchen matches the approved floor plan. Terraform is the builder who constructs the building itself. The Operator is the head chef who runs the kitchen every day and knows what to do when the oven breaks mid-service.
How they work together
A very common and healthy design uses several of these at once. The infrastructure tool creates the cluster and its cloud resources. A GitOps tool keeps the Operator itself installed and up to date, usually by deploying a Helm chart or a set of manifests from Git. Developers then commit small custom resources, such as a database request, to Git. The GitOps tool applies them, and the Operator takes over and does the real operational work. Git remains the source of truth for intent, while the Operator holds the source of expertise.
flowchart TD
TF["Infrastructure tool creates cluster and cloud resources"] --> K["Kubernetes cluster"]
GIT["Git repository holds Operator install and custom resources"] --> GO["GitOps tool syncs Git to the cluster"]
GO --> K
K --> OP["Operator reads each custom resource"]
OP --> APP["Running application: install, upgrade, backup, heal"]
APP -.-> OBS["Metrics and status flow back to dashboards"]
A common point of friction
One practical problem appears when a GitOps tool and an Operator both want to control the same field. For example, the Operator may update the status or a generated field, while the GitOps tool sees the difference from Git and tries to revert it. The result is a tug of war. The usual solution is to configure the GitOps tool to ignore fields that the Operator owns, and to keep only the user-facing spec in Git. Clear ownership of each field removes the conflict.
Good division of responsibility
- Git holds only what humans decide, such as spec fields.
- The Operator owns status and anything it generates.
- The GitOps tool is told to ignore Operator-owned fields.
Confused ownership
- Generated values stored back into Git by hand.
- Two tools fixing the same difference repeatedly.
- Nobody sure which tool is responsible for a change.
Operators versus plain scripts and scheduled jobs
Another comparison is with ordinary scripts run on a schedule. A scheduled job can take a nightly backup, and for a small system that may be entirely enough. What a script lacks is continuous awareness. It does not notice a failure at two in the afternoon, does not coordinate with an upgrade in progress, and does not report its progress in a form the rest of the platform can read. An Operator turns the same knowledge into a long-lived, cluster-native, observable process. If your needs are small and rare, a script may be the wiser and cheaper choice. As the number of applications and the cost of mistakes grow, the Operator starts to pay for itself.
Use packaging tools for things you install, GitOps for things you want to stay in sync with a repository, infrastructure tools for things outside the cluster, and Operators for things that need ongoing, application-specific care.
Production example: a layered platform
Many platform teams run exactly this layering. A GitOps tool installs a curated set of Operators from a repository, such as one for databases and one for monitoring. Application teams then request databases by committing a short custom resource. The platform team reviews changes to the Operators themselves, while application teams control only their own requests, giving fast self-service with clear guard rails.
26Common Mistakes and Anti-Patterns
The same errors appear again and again in Operator projects and in teams adopting them. Knowing them in advance can save months.
What people do
They mark an Operator as Level 3 or 4 because a feature exists on paper, without ever testing a restore or a failure.
Why it hurts
Users trust the badge and discover the gap during an outage, when it is too late.
Better approach
Tie every claimed level to automated tests and public drills, and publish the results.
What people do
They rush to Level 5 scaling and tuning without solid metrics, alerts, and logs.
Why it hurts
Automatic actions based on thin data cause strange changes that nobody can explain or reverse.
Better approach
Build Level 4 first, run in recommendation mode, and only then allow actions.
What people do
They schedule backups and assume they work.
Why it hurts
Backups can be empty, incomplete, or unreadable. The failure is only found when the data is already gone.
Better approach
Automate restore tests into a throwaway copy and verify the data regularly.
What people do
They give an Operator cluster-administrator rights because it is the easiest way to make it work.
Why it hurts
One compromise or bug can affect the whole cluster.
Better approach
Define narrow roles, scope to specific namespaces, and review them like any other security-sensitive code.
What people do
They pass hundreds of raw application settings through the custom resource.
Why it hurts
The API becomes a thin, confusing wrapper that is hard to validate and impossible to evolve safely.
Better approach
Expose a small set of intent-level fields, plus a controlled way to supply advanced settings when truly needed.
What people do
They build an Operator that cannot be told to stop acting on one application.
Why it hurts
During an incident, humans cannot safely intervene because the Operator keeps undoing their changes.
Better approach
Provide a simple, documented pause control and respect it in every reconcile pass.
What people do
They install a third-party Operator and nobody is responsible for upgrading, monitoring, or understanding it.
Why it hurts
Versions drift, bugs go unpatched, and the first real incident is also the first time anyone reads the documentation.
Better approach
Assign a clear owner, schedule upgrades, and run drills before production depends on the Operator.
27Frequently Asked Questions
These are the questions people most often ask when they first meet the maturity model.
28Summary and Key Takeaways
What to remember
- An Operator encodes expert knowledge as software. It pairs a custom resource with a controller that keeps reality in line with what you declared, for one specific application.
- The maturity model has five cumulative levels. Basic Install, Seamless Upgrades, Full Lifecycle, Deep Insights, and Auto Pilot. Remember them as install, upgrade, protect, observe, act.
- Higher is not always better. Each step removes human work and adds a new kind of risk. Choose the level that matches the application and your team, and stop where more automation stops paying off.
- Build quality matters as much as capability. Idempotent reconciliation, ownership, finalizers, honest status, and clean API design show up at every level.
- Test the level you claim. Upgrade tests for Level 2, restore and failure drills for Level 3, alert tests for Level 4, and guard-rail tests for Level 5.
- Add safety rails and least privilege. Pause controls, limits, delete protection, narrow permissions, and a trustworthy supply chain protect you as Operators become more powerful.
- Decide wisely whether to use one at all. Compare managed services, existing Operators, and building your own, count the cost of long-term ownership, and always assign a clear owner.