Operator Maturity Model in Kubernetes

Operator Maturity Model in Kubernetes

From a simple installer to a self-driving system: the five capability levels of Kubernetes Operators, what each one really means, how to climb the ladder safely, and how to judge where your own operators stand, explained from zero.

Imagine you hire someone to look after your house while you travel. On the first day, a beginner shows up, unlocks the doors, and turns on the lights. That is helpful, but the moment a pipe leaks, you get a phone call. A better caretaker also repairs and upgrades things without moving you out. A still better one handles emergencies, keeps backups of your valuables, and sends you a weekly report. The best one notices a small crack in the wall before it becomes a leak, and fixes it before you ever hear about it.

Software systems have the same kind of caretakers, and in Kubernetes they are called Operators. An Operator is a program that knows how to run one particular application, such as a database or a message queue, the way an expert human would. But not all Operators are equally capable. Some only install software. Others can heal, back up, tune, and scale it with no human involved. The Operator Maturity Model, officially called the capability levels, is a simple five-step ladder that describes how much an Operator can really do. This post explains that ladder from the very beginning, assuming you know nothing yet. By the end, you will understand every level, how to build and test an Operator that reaches it, how to judge the Operators you depend on, and how to talk about all of this confidently in interviews and design reviews.

1What Is an Operator, and Why Does It Exist?

Before we measure maturity, we need to be clear about what is being measured. Let us build the idea from the ground up.

The problem Kubernetes cannot solve alone

Kubernetes is excellent at running stateless programs, which are programs that keep no important data inside themselves. If a web server pod dies, Kubernetes simply starts another one, and nothing is lost. But some applications are stateful. A database holds your customers’ orders. A message queue holds messages that have not been delivered yet. These programs have strict rules: which copy is the leader, which copies are followers, in what order they must start, and how data is copied between them. Kubernetes, by itself, knows none of this. It sees only containers.

Where the human expert used to fit in

Traditionally, a skilled database administrator or site reliability engineer carried that knowledge in their head. They knew how to create a replica, how to promote a replica when the leader died, how to take a safe backup, and how to upgrade without losing data. Every time something went wrong at three in the morning, a human woke up and ran a careful sequence of commands.

The Operator idea

In 2016, engineers at a company called CoreOS proposed a clever idea: write that human expert’s knowledge as software and run it inside the cluster. They called the result an Operator, because it does the work of a human operator. Their first examples were Operators for etcd and Prometheus. The idea spread quickly, and today there are Operators for databases, message brokers, monitoring systems, storage systems, and even cloud services.

Everyday analogy

Think of a self-service coffee machine. A plain machine only heats water; you must do everything else. A good machine grinds beans, tamps, brews, and cleans itself. An Operator is like a machine that has absorbed the knowledge of a skilled barista: you say “one large latte”, and it handles every step, including what to do when the milk runs out.

The two ingredients

Ingredient 1

Custom Resource

A new kind of object added to Kubernetes, such as “PostgresCluster”, in which you describe the outcome you want in plain settings.

Ingredient 2

Controller

A program that watches those objects and keeps working until the real world matches what you described.

2016
year the Operator idea was introduced
5
capability levels in the maturity model
1
goal: encode expert knowledge as software

Production example: Prometheus Operator

The Prometheus Operator is one of the best-known examples. Instead of hand-writing monitoring configuration files, teams create small Kubernetes objects that say “monitor this service” or “alert on this condition”, and the Operator builds and reloads the real configuration automatically. It turned a tedious expert task into a few lines of declared intent, which is exactly the promise of the whole Operator pattern.

i
Remember this

An Operator is not a magic tool. It is ordinary software that happens to run inside the cluster and speak the Kubernetes language. Its quality depends entirely on how much of the expert’s knowledge was captured in it. The maturity model is a way to measure that.

2The Building Blocks: Custom Resources and Controllers

To understand why some Operators are more capable than others, you need to see the machinery inside every one of them.

Declared state and real state

Kubernetes works on a simple principle. You describe what you want in an object, which is called the desired state. The cluster looks at what actually exists, called the actual state. A program called a controller compares the two and takes actions to close the gap. This continues forever. If something breaks and the actual state drifts away, the controller notices and fixes it. This is the foundation of everything Kubernetes does.

Custom Resource Definitions

Kubernetes comes with built-in object types such as Pod, Service, and Deployment. A Custom Resource Definition, shortened to CRD, lets you teach the cluster a brand-new object type. Once the CRD is installed, users can create objects of that type just like any built-in one, use the same command-line tools, and apply the same access rules. The new object is called a custom resource.

The reconciliation loop

Inside every Operator there is a reconcile loop. Each time something related to a custom resource changes, the loop runs. It reads the desired state, looks at the actual state, decides what to do, does one small step, and reports the result back in the object’s status. Because the loop can run many times, every step must be safe to repeat. Engineers call this property idempotent: doing it twice has the same effect as doing it once.

flowchart TD
  U["User creates or edits a custom resource"] --> API["Kubernetes API server stores the object"]
  API --> W["Controller is notified of the change"]
  W --> R["Reconcile loop starts"]
  R --> C["Compare desired state with actual state"]
  C --> D{"Any difference?"}
  D -- "No" --> S["Update status and wait for the next event"]
  D -- "Yes" --> A["Take one safe action: create, update, or repair"]
  A --> S
  S --> W
    
Figure 1: The reconcile loop that sits at the heart of every Operator

Why “level-triggered” matters

A well-built controller does not rely on remembering the exact event that woke it up. Instead, it looks at the full current picture every time. Engineers call this level-triggered design, in contrast with edge-triggered design, which reacts only to the change itself. Level-triggered controllers recover gracefully from missed events, restarts, and crashes, because the next run simply re-examines everything. This is one of the main reasons the Kubernetes model is so resilient.

Everyday analogy

An edge-triggered thermostat turns the heater on only at the instant someone presses a button, and forgets everything after. A level-triggered thermostat keeps checking the room temperature all the time and keeps adjusting. If the power cuts out for an hour and comes back, the second thermostat fixes the room by itself. The first would sit there confused.

The helper machinery

PieceWhat it isWhy Operators need it
Informer and cacheA local, always-updated copy of the objects the controller cares aboutAvoids hammering the API server with repeated reads
Work queueA line of objects waiting to be reconciled, with retries and delaysHandles bursts of events and failed attempts gracefully
Owner referencesLinks from created objects back to the custom resource that owns themLets Kubernetes clean up everything automatically when the owner is deleted
FinalizersMarkers that pause deletion until cleanup work is doneAllows safe teardown of external resources such as cloud disks
Status and conditionsA section of the object where the Operator reports progress and healthGives users and tools a clear picture of what is happening
Leader electionA way for several copies of an Operator to agree that only one acts at a timePrevents two copies from fighting over the same object
!
Common confusion

A controller is the general mechanism, and an Operator is a controller that carries application-specific operational knowledge. The built-in Deployment controller is a controller, but nobody calls it an Operator, because it knows nothing special about any particular application.

3Why We Need a Maturity Model

Walk through any Operator catalog and you will find hundreds of entries, all described with the same friendly words. A maturity model helps you see through the marketing.

The honesty problem

Two Operators may both claim to “manage PostgreSQL”. One may merely generate the starting configuration and then step away. The other may monitor replication, promote a new leader during a failure, take verified backups, restore them on demand, and tune its own memory settings. From the outside, both look like a single custom resource. Without a shared scale, you cannot tell them apart until a crisis arrives.

What the model gives you

Buyers

A shared vocabulary

Teams can say “we need at least Level 3 for production databases” and everyone understands what that means.

Builders

A roadmap

Authors can plan which abilities to add next instead of building features at random.

Risk

A way to compare

You can compare two Operators for the same application on equal terms.

Operations

A staffing guide

The lower the level, the more human work your team still carries.

Where the model comes from

The five capability levels were introduced by the Operator Framework, an open-source project that also provides the Operator SDK for building Operators and the Operator Lifecycle Manager for installing and updating them. Public catalogs, such as OperatorHub, display a capability level for many listed Operators so that users can see at a glance how far each one goes.

!
Read the badge carefully

The level shown for an Operator is usually declared by its authors, not independently audited. Treat it as a starting claim and verify it against your own needs, especially for the abilities that matter most to you, such as backup and recovery.

What the model does not measure

The levels describe capability, meaning what the Operator can do. They do not measure code quality, security, performance, community health, or how well the Operator is documented. A Level 5 Operator that is poorly tested can be more dangerous than a well-tested Level 2 one. Throughout this post, we will keep these separate and give you checklists for the other qualities too.

What the model is good for

  • Comparing Operators on feature depth.
  • Planning a roadmap for your own Operator.
  • Explaining automation goals to managers.

What it cannot tell you

  • Whether the Operator is reliable or secure.
  • Whether it fits your specific workload.
  • Whether anyone still maintains it.

4The Five Levels at a Glance

Here is the whole ladder on one page. The next five chapters visit each step in depth.

flowchart BT
  L1["Level 1: Basic Install - automated setup and configuration"] --> L2["Level 2: Seamless Upgrades - patch and minor version updates"]
  L2 --> L3["Level 3: Full Lifecycle - backup, recovery, failure handling"]
  L3 --> L4["Level 4: Deep Insights - metrics, alerts, logs, analysis"]
  L4 --> L5["Level 5: Auto Pilot - scaling, tuning, self-healing, anomaly detection"]
    
Figure 2: The five capability levels, each built on the one below
LevelNamePlain-English meaningHuman still needed for
1Basic InstallIt can set the application up and configure it from a declared descriptionUpgrades, backups, failures, tuning, everything else
2Seamless UpgradesIt can upgrade the application and itself without losing data or causing long outagesBackups, failure recovery, tuning
3Full LifecycleIt handles storage, backup, restore, and recovery from failureUnderstanding trends, tuning, capacity decisions
4Deep InsightsIt exposes rich metrics, alerts, and logs, and can analyze the workloadActing on the insights
5Auto PilotIt scales, tunes, and heals the application on its own, and spots abnormal behaviorSetting goals and handling truly unusual events

The ladder is cumulative

Each level assumes the abilities of the levels below it. An Operator cannot honestly claim Level 3 if it cannot upgrade safely, because safe upgrades come at Level 2. Similarly, automatic tuning at Level 5 depends on the metrics gathered at Level 4. This is why you should climb in order rather than jump to the exciting top steps first.

Everyday analogy

Learning to drive works the same way. First you learn to start the car and steer (Level 1). Then you learn to change gears and maintain it (Level 2). Then you learn what to do in a breakdown (Level 3). Then you learn to read the dashboard and warning lights (Level 4). Finally the car drives itself using what it senses (Level 5). Nobody can safely skip straight to self-driving.

A shortcut to remember the levels

1

Install

Can it put the application in place correctly?

2

Upgrade

Can it move the application to a newer version safely?

3

Protect

Can it back up, restore, and survive failures?

4

Observe

Can it tell you what is happening inside?

5

Act

Can it adjust and heal by itself using what it observes?

i
Interview-ready summary

Install, Upgrade, Protect, Observe, Act. If you can recite those five words and explain each with one example, you can answer most questions about the maturity model.

5Level 1: Basic Install

The first step is the simplest: turn a short description into a correctly running application. Many Operators begin and sometimes end here.

What Level 1 means

At Level 1, the Operator provisions the application and applies its configuration automatically. You create a custom resource that says, for example, “I want a three-member message cluster with these settings”. The Operator creates the pods, the services, the storage claims, the configuration files, and the secrets, and wires them together so the application starts correctly. It also keeps the configuration in line with what you declared: if someone edits a setting in the custom resource, the Operator applies it.

What it saves you

Without an Operator, installing a clustered application often means filling in dozens of files and running several steps in the correct order. A single forgotten setting can cause a quiet failure that shows up weeks later. Level 1 replaces that pile of steps with one declaration, and, just as importantly, makes the installation repeatable. The same description produces the same result in a test cluster, a staging cluster, and production.

Everyday analogy

Level 1 is the caretaker who unlocks the house, turns on the water, switches on the lights, and sets the thermostat exactly as you wrote on the note. The house works on day one. But when winter comes and a pipe freezes, the caretaker has no instructions.

What a Level 1 Operator typically does

Create

Builds the pieces

Creates the workloads, services, storage, and settings the application needs.

Configure

Applies settings

Turns the simple fields of the custom resource into the application’s real configuration files.

Validate

Rejects bad input

Refuses nonsense settings early, with a clear message, rather than failing later.

Report

Shows status

Writes a basic status, such as “creating”, “ready”, or “failed”, into the custom resource.

What it does not do

Strengths of Level 1

  • Fast, repeatable installs.
  • Fewer configuration mistakes.
  • Easy to understand and to build.
  • Works well for stateless or easily replaced applications.

Gaps at Level 1

  • No safe upgrade path guaranteed.
  • No backups or recovery.
  • Little or no health logic beyond restarting pods.
  • Humans still handle every real incident.

Production example: Helm-style operators

Many Operators created from Helm charts sit at this level. They convert a chart into a custom resource so that installing and changing settings is a Kubernetes-native action. This is genuinely useful for platform consistency, but a chart on its own knows nothing about how to promote a replica or restore a backup, so such Operators rarely climb far beyond the first two steps without extra work.

i
Good question to ask

When evaluating a Level 1 Operator, ask: “If I delete the custom resource and recreate it with the same settings, do I get the same running system?” A trustworthy answer is yes. This is the repeatability that makes Level 1 valuable.

6Level 2: Seamless Upgrades

Installing is the easy part. The real test of any operational tool is what happens a year later, when the software needs to move to a newer version.

What Level 2 means

At Level 2, the Operator can upgrade the application it manages, including patch releases and minor version changes, with minimal downtime and without losing data. It also supports upgrading itself. You change the version in the custom resource, and the Operator carries out the careful sequence needed to get there.

Why upgrades are hard

Think about a database with one leader and two followers. If you restart them all at once, the service goes down. If you restart the leader first, you may lose recent writes. The safe approach is a rolling upgrade: update one follower, wait until it has caught up, update the next, and only then switch the leader role to an upgraded copy and update the old leader last. Each step must check that the system is healthy before moving on. Different applications have different rules, and some versions require special data migrations that must run exactly once.

sequenceDiagram
  participant U as User
  participant O as Operator
  participant F1 as Follower 1
  participant F2 as Follower 2
  participant L as Leader
  U->>O: Change version in the custom resource
  O->>F1: Upgrade and restart
  F1-->>O: Healthy and caught up
  O->>F2: Upgrade and restart
  F2-->>O: Healthy and caught up
  O->>L: Hand leadership to an upgraded member
  O->>L: Upgrade the old leader
  L-->>O: Healthy
  O-->>U: Status shows upgrade complete
    
Figure 3: A rolling upgrade where the Operator checks health before every step

The upgrade promises

1

Check before changing

Confirm the current system is healthy and the target version is a supported jump.

2

Go one step at a time

Upgrade members in a safe order, never all at once, so the service stays available.

3

Verify after each step

Wait for health signals, not just a timer, before moving on.

4

Stop if something looks wrong

Pause and report instead of pushing ahead into a broken state.

Upgrading the Operator itself

An Operator is also software with versions. A new Operator version may change how it manages existing applications. A mature Operator must be able to take over applications that an older version created, without recreating them. The Operator Lifecycle Manager helps by describing which Operator versions can replace which, so that the cluster only follows supported upgrade paths.

!
Hidden risk

Upgrading the Operator can accidentally trigger changes to the applications it manages, such as a rolling restart caused by a changed default. Good Operators avoid this, and good teams test Operator upgrades in a copy of production before applying them for real.

Everyday analogy

Level 2 is a caretaker who can renovate the kitchen while you still live in the house. They replace the stove one day, the sink another day, and never leave you without water or a place to cook. They also know that some changes, like rewiring, must wait for the right moment.

Production example: managed Kafka on Kubernetes

Operators for Apache Kafka, such as Strimzi, are well known for coordinating rolling restarts of brokers in a safe order during upgrades and configuration changes, waiting for partitions to be fully in sync before touching the next broker. This kind of careful sequencing is the hallmark of Level 2 behavior.

7Level 3: Full Lifecycle

Level 3 is where an Operator starts to earn the trust of people running important data. It covers everything that happens between birth and retirement of the application, including the unhappy parts.

What Level 3 means

At Level 3, the Operator manages the whole application lifecycle and its storage. That includes taking backups, restoring from them, recovering from failures, and handling data safely when the application is changed or removed. A human no longer has to run special scripts to protect the data.

The four big abilities

Protect

Backups

Schedules regular backups, stores them somewhere safe such as object storage, and tracks which ones exist.

Recover

Restore

Creates a new application instance from a chosen backup, or rewinds an existing one to a point in time.

Heal

Failure handling

Detects a failed member and replaces or repairs it, including promoting a new leader when needed.

Retire

Clean shutdown

Removes the application in a controlled way, deciding what happens to its data, using finalizers.

Failure handling in detail

Failures come in many sizes. A pod may crash and restart, which Kubernetes handles alone. A whole machine may disappear, taking the pod and its local storage with it. A network split may leave two parts of the cluster each believing the other is dead. The Operator must tell these cases apart and react correctly. Wrongly promoting a second leader during a network split can cause two copies of the data to diverge, a dangerous problem often called split brain. Mature Operators use careful rules, often relying on a trusted coordination mechanism, to make sure only one leader exists at a time.

flowchart TD
  F["Operator detects an unhealthy member"] --> Q{"Is it a pod crash, a lost node, or a network split?"}
  Q -- "Pod crash" --> R1["Let Kubernetes restart it and watch recovery"]
  Q -- "Lost node" --> R2["Create a replacement member and resync data from healthy copies"]
  Q -- "Network split" --> R3["Keep the side with a majority, fence the other side"]
  R1 --> OK["Update status: healthy"]
  R2 --> OK
  R3 --> OK
    
Figure 4: A simplified failure decision tree inside a Level 3 Operator

A backup is only real if you can restore it

A famous saying in operations is that nobody has backups; they only have restores. A backup that has never been restored might be empty, corrupted, or missing a key. A strong Level 3 Operator does not stop at creating backups. It also provides a tested restore path, and ideally verifies backups automatically by restoring them into a throwaway copy and checking the result.

What Level 3 gives you

  • Real protection for your data.
  • Recovery from many failures without a human.
  • Clear, repeatable restore procedures.
  • Confidence to run important data on Kubernetes.

What Level 3 still lacks

  • Limited visibility into how the application behaves.
  • No automatic tuning for load.
  • Humans still decide when to scale.
Everyday analogy

Level 3 is a caretaker who keeps copies of your important papers in a safe deposit box, knows exactly what to do when a pipe bursts, and can rebuild a damaged room from the original plans. You can sleep better, even though the caretaker still does not tell you how the house is aging.

Production example: PostgreSQL operators

Several PostgreSQL Operators, including those from Zalando, Crunchy Data, and the CloudNativePG project, are widely used in production. They typically manage automatic failover between a primary and replicas, scheduled backups to object storage, and point-in-time recovery. Teams choose among them partly by how thoroughly each handles these Level 3 abilities.

8Level 4: Deep Insights

Protecting the application is not enough if you cannot see how it behaves. Level 4 is about visibility: turning the Operator into a source of understanding, not just action.

What Level 4 means

At Level 4, the Operator provides rich metrics, alerts, log processing, and workload analysis. It does not merely run the application; it measures it and explains what it sees. The goal is to give humans and, later, automated systems enough information to make good decisions.

The four kinds of insight

InsightWhat it containsExample for a database
MetricsNumbers measured over timeConnections in use, replication lag, cache hit ratio, disk growth
AlertsRules that warn when something is wrong or about to beReplica falling behind, disk will be full in two days, backup missed
LogsEvent records, gathered and organizedSlow queries collected and summarized
AnalysisInterpretation of the workloadRecommended indexes, unusual traffic patterns, capacity forecasts

Good insight is application-aware

Kubernetes already shows generic numbers like CPU and memory. Those are not enough. A database can have perfectly normal CPU while its replicas silently fall hours behind. Only an Operator that understands the application knows which signals matter. Level 4 Operators export application-specific metrics in a standard format, ship ready-made dashboards and alert rules, and expose the health of each member in the custom resource status.

flowchart LR
  APP["Managed application"] --> EXP["Operator-provided metrics and health checks"]
  EXP --> MON["Monitoring system"]
  MON --> DASH["Dashboards for humans"]
  MON --> ALERT["Alerts to on-call team"]
  EXP --> STAT["Custom resource status and events"]
  STAT --> HUMAN["Engineers using kubectl or consoles"]
    
Figure 5: How a Level 4 Operator turns raw application behavior into usable insight

Insight about the Operator itself

A mature Operator also tells you how it is doing. How many reconcile loops ran, how many failed, how long they took, how many objects are waiting in the queue? Without these numbers, you cannot tell whether a stuck application is the application’s fault or the Operator’s. We return to this in the chapter on observing the Operator.

i
Look for these signs

Does the Operator ship dashboards and alert rules out of the box? Does the custom resource status explain problems in plain words? Can you tell from the status alone which member is lagging and why? If the answers are yes, you are looking at real Level 4 behavior.

Everyday analogy

Level 4 is a caretaker who not only fixes problems but also sends you a monthly report: water use is up, the roof has a small leak forming, the boiler is five years old. You are not asked to do anything automatically, but you now see the house clearly and can plan.

Production example: monitoring-aware operators

Operators for systems such as Elasticsearch, Kafka, and Ceph commonly integrate with the Prometheus monitoring stack by exposing metrics and shipping alert rules. Teams that adopt them gain consistent dashboards across many clusters, which makes the on-call experience far more predictable than when every cluster is monitored by hand.

9Level 5: Auto Pilot

The top of the ladder is the most exciting and the easiest to get wrong. At Level 5, the Operator does not just show the problem; it acts on it.

What Level 5 means

At Level 5, the Operator performs horizontal and vertical scaling, automatic configuration tuning, abnormality detection, and scheduling tuning. It uses the insights gathered at Level 4 to adjust the application without waiting for a human. The aim is a system that runs itself, with people setting goals and boundaries rather than pushing buttons.

The auto-pilot abilities

Scale out

Horizontal scaling

Adds or removes members when load changes, for example adding read replicas during a busy season.

Scale up

Vertical scaling

Gives members more memory or processor power when they are running short.

Tune

Automatic configuration

Adjusts application settings, such as memory pools, to match the workload it observes.

Detect

Anomaly detection

Notices abnormal patterns, like a sudden jump in errors, and responds or raises an alarm.

Place

Smart scheduling

Chooses where members run, spreading them across zones and avoiding noisy neighbors.

The feedback loop

Auto pilot works through a loop with four steps: observe, decide, act, and check the result. The Operator measures the system, compares the measurements with goals such as “keep response time under 50 milliseconds”, decides on an adjustment, applies it carefully, and then measures again to confirm that it helped.

flowchart LR
  O["Observe: metrics and health"] --> D["Decide: compare with goals and limits"]
  D --> A["Act: scale, tune, or heal in small steps"]
  A --> C["Check: did the change help?"]
  C --> O
    
Figure 6: The closed feedback loop that defines an auto-pilot Operator

Why Level 5 is dangerous when rushed

An automated system can make mistakes much faster than a human. A badly tuned autoscaler can add machines in a loop and run up a huge bill. An overeager tuning rule can set a memory value that crashes the database. Two automatic systems, such as the Operator and a cluster autoscaler, can fight each other. For these reasons, Level 5 abilities should always come with guard rails: minimum and maximum limits, slow and gradual changes, a way to pause automation, and clear logs of every decision.

Benefits of auto pilot

  • Faster reaction than any human on call.
  • Lower cost through right-sizing.
  • Fewer middle-of-the-night pages.
  • Consistent behavior across many clusters.

Risks of auto pilot

  • Mistakes repeat at machine speed.
  • Harder to understand why something changed.
  • Conflicts with other automation.
  • Requires excellent Level 3 and 4 foundations.
!
Not every Operator should reach Level 5

Level 5 is a possible ceiling, not a requirement. Many excellent Operators deliberately stop at Level 3 or 4, because automatic tuning of that particular application is risky or not worth the complexity. A higher level is not always better; the right level depends on the application and the risk you can accept.

Production example: controllers that act on their own

Kubernetes has well-known controllers that behave in the auto-pilot spirit. The Horizontal Pod Autoscaler adjusts replica counts based on measured load, and Karpenter on AWS watches pending pods and chooses new machines automatically. They show both the power of closed feedback loops and the need for careful limits, since a runaway loop here means real money spent.

10Choosing How to Build: Helm, Ansible, or Go

The tool you pick to write an Operator quietly decides how high it can climb. Understanding this saves teams from painful rewrites later.

The three classic paths

The Operator SDK, part of the Operator Framework, lets you create an Operator in three main ways. Each trades ease against power.

ApproachHow it worksStrengthTypical ceiling
Helm-basedWraps an existing Helm chart so that a custom resource fills in its valuesFastest start; reuse what you already haveLevel 1 to 2
Ansible-basedRuns automation playbooks in response to custom resource changesGood for teams that already automate with Ansible; flexible stepsUp to Level 5 in principle, with effort
Go-basedA full program written with Kubernetes controller librariesMaximum control, speed, and testabilityLevel 5

Why Helm tops out early

A Helm chart is a template that renders Kubernetes objects once. It has no built-in way to watch a running database, decide that a replica fell behind, take a backup, or coordinate a careful sequence. A Helm-based Operator can reapply the chart when settings change, but it cannot reason about the live system. That is why such Operators fit Levels 1 and 2 and rarely go further.

Other tools you will meet

Go

Kubebuilder and controller-runtime

Widely used libraries and scaffolding for writing controllers in Go. The Operator SDK’s Go path is built on them.

Java

Java Operator SDK

A framework for teams who prefer to write Operators in Java.

Python

Kopf and similar

Lightweight options for simple Operators or quick prototypes in Python.

Generic

Metacontroller, KUDO

Frameworks that let you describe controller behavior with less code, with their own trade-offs.

How to choose

1

Decide your target level honestly

If you only need repeatable installs, Helm may be enough. If you need failover and backups, plan for code.

2

Look at your team’s skills

An Operator nobody on the team can maintain becomes a liability, however clever it is.

3

Think about testing

Real code is easier to unit test and to simulate failures against than a long chain of templates.

4

Plan for the long run

Choose a tool that can carry you to the level you will need in two years, not only today.

!
The rewrite trap

Teams often start with a quick Helm-based Operator, then discover they need failover logic that Helm cannot express. Moving to Go at that point means redesigning the custom resource and migrating live systems. If you suspect you will need Level 3 or higher, begin with a tool that can support it.

11Designing the API: Your Custom Resource Is a Promise

The custom resource is the front door of your Operator. Once people start using it, changing it is expensive, so a little careful design here pays off for years.

Spec and status

Almost every custom resource has two main sections. The spec is what the user writes: the desired outcome. The status is what the Operator writes: the observed reality. Keeping them separate is a golden rule. Users never edit status, and the Operator never edits spec. This division keeps responsibilities clear and prevents the two sides from overwriting each other.

Describe intent, not steps

A good spec says what the user wants, not how to do it. “Three replicas, version 15, daily backups at two in the morning” is intent. “Run these four commands in this order” is a script. The Operator is the one who knows the steps. A spec that exposes too many internal knobs becomes hard to evolve and easy to misuse.

Signs of a good API

  • Few, meaningful fields with sensible defaults.
  • Clear names that a newcomer can guess.
  • Validation that rejects mistakes immediately.
  • Status that explains what is happening.

Signs of a weak API

  • Dozens of raw configuration options passed through.
  • Fields that only make sense in a certain order.
  • Silent acceptance of invalid values.
  • Status that just says “ok” or “error”.

Status conditions

Instead of a single word, mature Operators report a list of conditions, each describing one aspect of health, such as “Ready”, “BackupHealthy”, or “UpgradeInProgress”. Each condition has a true or false value, a short reason, a human-readable message, and a timestamp of the last change. Tools and dashboards can then show a precise picture, and automation can wait for a particular condition to become true.

Validation

CRDs can carry a schema that checks every incoming object. Simple rules, such as numbers within a range or text matching a pattern, can be enforced by the cluster itself before the Operator even sees the object. More complex checks can be done by a validating webhook, a small service the API server asks before accepting a change. Early, clear validation is far kinder to users than a mysterious failure hours later.

Versioning the API

APIs change. Fields are added, renamed, or removed. Kubernetes lets a CRD serve several versions at once, such as v1alpha1, v1beta1, and v1, and use a conversion webhook to translate objects between them. The names carry meaning: alpha means early and subject to change, beta means mostly stable, and v1 means users can rely on it. Mature Operators promise not to break existing objects when they release new versions.

flowchart LR
  A["v1alpha1: experimental, may break"] --> B["v1beta1: mostly stable, tested widely"]
  B --> C["v1: stable, compatibility promised"]
  C -.-> D["Conversion webhook translates between served versions"]
    
Figure 7: A typical life of a custom resource API, with conversion keeping old objects working
Everyday analogy

The custom resource is the menu of a restaurant. A good menu lists dishes by what they are, not by cooking instructions, stays consistent from month to month, and tells you when a dish is sold out. A bad menu lists raw ingredients and leaves the cooking to you, or changes every week so regulars cannot order.

i
Design tip

Write down how a new user will create their first object before you finalize the API. If the example is longer than a screen, the API probably exposes too much. Defaults should make the shortest example work.

12Reconciliation Done Right

A large part of the difference between a toy Operator and a trustworthy one is how carefully the reconcile loop is written. These habits show up at every level of the maturity model.

Habit one: be idempotent

Every pass of the loop must be safe to run again. Instead of “create a volume”, write “make sure a volume exists”. If it already exists, do nothing. If a previous run stopped halfway, the next run finishes the job. This single habit makes crashes, restarts, and retries harmless.

Habit two: take small steps

Do a little, record progress in the status, and return. Long operations that block the loop for minutes make the Operator slow to react to other objects. Instead, start the operation, return, and come back to check on it. Operators that move one step at a time are easier to debug, easier to pause, and safer during upgrades.

Habit three: own what you create

Set an owner reference on every object the Operator creates, pointing back to the custom resource. When the custom resource is deleted, Kubernetes automatically removes everything it owns. Without owner references, deleted applications leave orphaned storage, services, and settings behind.

Habit four: clean up with finalizers

Some things cannot be cleaned up by Kubernetes alone, such as a cloud disk, a DNS record, or a backup in external storage. A finalizer is a label on the object that tells Kubernetes: “do not fully delete this until I say so”. When a user deletes the custom resource, the Operator sees the deletion request, performs the external cleanup, removes its finalizer, and only then does the object disappear. If the Operator forgets to remove the finalizer, the object hangs in a deleting state forever, so this logic must be well tested.

1

User deletes the object

Kubernetes marks it for deletion but keeps it because a finalizer is present.

2

Operator notices

The reconcile loop sees the deletion timestamp and switches into cleanup mode.

3

Cleanup runs

It deletes external resources, perhaps taking a final backup first if the user asked for it.

4

Finalizer removed

Kubernetes can now finish deleting the object and its owned children.

Habit five: handle conflicts and retries

Several actors may change the same object at once: the user, the Operator, other controllers. Kubernetes protects against lost updates by rejecting a write that is based on an outdated version, called a conflict. A well-written Operator simply re-reads and tries again. Failures should also be retried with growing delays, called exponential backoff, so that a broken dependency is not hammered with requests.

Habit six: report honestly

Write real status after each meaningful step, including failures and their reasons. Emit Kubernetes events for notable actions. An Operator that fails silently looks fine right up until the moment it is not.

!
Danger: hot loops

If an Operator updates the object’s status in a way that triggers another reconcile, which updates the status again, it can spin forever, consuming CPU and flooding the API server. Only write when something actually changed, and ignore changes to the parts of the object the Operator itself writes.

HabitQuestion to ask during code review
IdempotentIf this step ran twice, would anything break?
Small stepsCan the loop return quickly and resume later?
OwnershipDoes every created object have an owner reference?
FinalizersWhat happens if cleanup fails halfway?
ConflictsWhat if the object changed while we were working?
Honest statusWould a stranger understand the status without reading the code?

13Packaging and Lifecycle Management

An Operator that lives only on its author’s laptop helps nobody. Level 2 maturity applies to the Operator too: it must be installable, upgradable, and removable in a clean way.

The Operator Lifecycle Manager

The Operator Lifecycle Manager, usually called OLM, is a component of the Operator Framework that installs Operators, keeps them updated, and manages their permissions. Think of it as an app store and update service for Operators inside the cluster. It understands which versions can replace which, and it can pause automatic updates when you want to approve them manually.

The key objects

Package

ClusterServiceVersion

A description of one Operator version: its deployment, permissions, the custom resources it owns, and its documentation.

Catalog

CatalogSource

A source of available Operators, like a shelf in the store.

Choice

Subscription

A request to follow one Operator through a chosen update channel.

Plan

InstallPlan

The concrete list of steps OLM will perform, which can require human approval.

Update channels and approval

Operators publish versions into channels, such as a stable channel for tested releases and a fast channel for newer ones. A subscription chooses a channel and an approval mode: automatic, where OLM applies updates as they appear, or manual, where an administrator reviews each install plan first. Production clusters often choose manual approval for critical Operators so that upgrades happen at planned times.

flowchart LR
  CAT["Catalog of Operator versions"] --> SUB["Subscription: chosen channel and approval mode"]
  SUB --> PLAN["Install plan: steps OLM will take"]
  PLAN --> APPR{"Manual approval needed?"}
  APPR -- "Yes" --> ADMIN["Administrator reviews and approves"]
  APPR -- "No" --> RUN["OLM installs or upgrades the Operator"]
  ADMIN --> RUN
    
Figure 8: How OLM turns a subscription into a controlled Operator upgrade

Other packaging routes

OLM is not the only way. Many Operators are also installed with Helm charts or plain manifests, which is simple and works on any cluster. Some managed platforms provide their own catalogs of add-ons. Whichever route you pick, the same questions apply: how are CRDs upgraded, how are permissions reviewed, how do you roll back, and how do you remove the Operator without destroying the applications it manages?

!
CRD upgrades are special

Helm installs CRDs once and does not upgrade them the way it upgrades ordinary objects. If you rely on Helm, plan a separate step for updating CRDs, or you may run a new Operator against an old schema.

Removing an Operator safely

Deleting an Operator should not accidentally delete your data. If the custom resources are removed along with the CRD, the Operator’s finalizers and owner references may cascade into deleting databases and storage. Mature Operators and good runbooks make the safe order clear: first decide what happens to the managed applications, then remove the Operator, and only at the very end consider removing the CRDs.

14Security: Operators Are Powerful Programs

An Operator can create pods, read secrets, and delete storage. That power is exactly why it must be built and installed with care.

Why Operators are attractive targets

Operators often run with broad permissions, sometimes across the entire cluster, so they can manage applications in many namespaces. If an attacker compromises an Operator, they may gain the ability to read secrets, start pods, or modify any application it manages. The Operator’s identity is therefore a high-value asset.

Principles for safe Operators

1

Least privilege

Grant only the permissions the Operator truly needs, on only the object types it manages. Avoid cluster-wide administrator rights.

2

Scope narrowly

Where possible, watch only specific namespaces rather than the whole cluster.

3

Harden the Operator pod

Run as a non-root user, drop extra capabilities, use a read-only filesystem, and restrict its network access.

4

Protect the supply chain

Use signed images from trusted sources, pin versions, scan for vulnerabilities, and keep a software bill of materials.

5

Secure webhooks

Webhooks need valid certificates. Manage them automatically, for example with cert-manager, and plan what happens if they expire.

Protect what the Operator creates

An Operator also decides the security settings of the pods it builds. A careless Operator may create pods that run as root, mount sensitive paths, or expose open network ports. A mature Operator applies safe defaults, supports network policies, uses encryption between members where the application allows, and keeps generated passwords in secrets with tight access.

Healthy security posture

  • Narrow roles reviewed before installation.
  • Pods built with restricted settings by default.
  • Generated credentials stored in secrets, never in logs.
  • Clear process for reporting vulnerabilities.

Warning signs

  • Installation instructions that say “grant cluster-admin”.
  • Images pulled from unknown registries with floating tags.
  • Passwords visible in status or events.
  • No security policy or maintainer response.
i
Review before you install

Before installing any third-party Operator, read its role definitions. Ask: which resources can it read, which can it change, and does it really need to touch secrets in every namespace? A few minutes of reading prevents many future problems.

Production example: scoping Operators per team

Platform teams in larger companies often install one instance of an Operator per team namespace, each with permissions limited to that namespace, instead of one powerful instance for the whole cluster. This reduces the blast radius if one instance is compromised or misbehaves, and lets teams upgrade on their own schedule.

15Testing an Operator

The higher an Operator climbs the maturity ladder, the more it can damage when it is wrong. Testing is what lets you climb safely.

Why Operators are hard to test

An Operator interacts with a live cluster, with real storage, and often with a stateful application that has its own quirks. Failures are timing-dependent: a bug may only appear if a node dies during an upgrade. Simple tests that check one function at a time are not enough. A good test strategy uses several layers, each catching a different kind of mistake.

The testing pyramid for Operators

LayerWhat it checksSpeedTypical tools
Unit testsIndividual decisions, such as “given this state, what is the next action?”Very fastOrdinary test frameworks
Control-plane testsThe reconcile loop against a real API server without real nodesFastenvtest, a helper in the controller libraries
End-to-end testsA real application on a real small clusterSlowkind or minikube clusters, test frameworks
Upgrade testsMoving from an old Operator and application version to a new oneSlowScripted end-to-end runs
Chaos testsBehavior while pods, nodes, or networks are deliberately brokenSlowestFault-injection tools

Test the level you claim

A very practical idea is to write tests that match the level you advertise. If you claim Level 2, you need automated upgrade tests across every supported path. If you claim Level 3, you need tests that take a backup, destroy the application, and restore it, checking that the data matches. Level 4 needs tests proving that metrics and alerts really fire. Level 5 needs tests showing that scaling and tuning stay within their guard rails under changing load.

Level 1

Create and change

Create the object, check everything appears, edit settings, check they are applied, delete, check all is cleaned up.

Level 2

Upgrade paths

Upgrade across each supported version pair while sending traffic, and check there is no data loss.

Level 3

Break and restore

Kill members, partition networks, corrupt data, and verify recovery and restore results.

Level 4

Signals fire

Induce known problems and verify the right alerts and metrics appear.

Level 5

Limits hold

Apply load patterns and verify that scaling respects minimum, maximum, and rate limits.

i
Use the scorecard

The Operator SDK includes a scorecard tool that runs a set of basic checks on an Operator’s packaging and behavior. It is not a substitute for deep testing, but it catches common mistakes early and gives teams a shared minimum standard.

Everyday analogy

Testing an Operator is like testing a hospital’s emergency plan. Reading the plan is not enough. You must run drills where the power goes out, a patient arrives during a shift change, and the main generator fails. Only drills reveal the gaps, and the more life-critical the hospital, the more often it drills.

16Observing the Operator Itself

Level 4 is usually described in terms of the application. But the Operator is a running program too, and it deserves the same visibility.

Why the Operator needs its own monitoring

When an application is unhealthy, the cause might be the application, the cluster, or the Operator itself. If the Operator is stuck, crash-looping, or silently failing to reconcile, nothing gets repaired, and no alarm rings because the application’s own metrics may still look normal for a while. Watching the Operator closes this blind spot.

The signals that matter

SignalWhat it tells youAlert when
Reconcile count and errorsHow often the loop runs and how often it failsError rate stays above normal for several minutes
Reconcile durationHow long each pass takesPasses become much slower than usual
Work queue depthHow many objects are waitingThe queue keeps growing and never drains
Work queue retriesHow many items keep failing and returningThe same items retry repeatedly
API request errors and latencyWhether the cluster API is healthy from the Operator’s viewMany throttled or failed calls
Leader election stateWhether exactly one copy is activeNo leader, or frequent leader changes
Restarts and memory useWhether the Operator is crashing or leakingRestarts increase or memory climbs steadily

Logs that tell a story

Operator logs should be structured, so that tools can filter them by object name and namespace. Each reconcile pass should log which object it handled and what decision it made, without printing secrets. When something goes wrong at night, the on-call engineer should be able to search the logs for a single application name and read a clear, ordered story of what the Operator did to it.

Events for humans

Kubernetes events attached to the custom resource are the friendliest form of insight. “Created backup job”, “Promoting replica two to leader”, and “Upgrade paused: member not healthy” show up when users describe the object. They turn the Operator from a black box into a narrator.

flowchart TD
  OP["Operator pod"] --> M["Metrics: reconcile counts, durations, queue depth"]
  OP --> L["Structured logs per object"]
  OP --> E["Events on each custom resource"]
  OP --> S["Status conditions on each custom resource"]
  M --> DASH["Operator health dashboard"]
  M --> AL["Alerts for stuck or failing reconciles"]
  L --> SEARCH["Searchable log platform"]
  E --> USERS["Users running describe commands"]
  S --> TOOLS["Dashboards and automation waiting on conditions"]
    
Figure 9: Four channels through which a well-behaved Operator makes its work visible
!
Silent stalls are the worst failure

An Operator that crashes is easy to notice. An Operator that runs but has stopped making progress, perhaps stuck on one object that blocks the queue, is far more dangerous. Alert on “no successful reconcile in the last N minutes” in addition to error counts.

17Safety Rails: How Operators Go Wrong

The more an Operator can do, the more carefully it must be fenced in. Understanding common failure modes helps you demand the right safety features.

The classic ways Operators fail

1

Fighting with other controllers

Two controllers each try to set the same field to different values, and the object flips back and forth endlessly.

2

Mass restarts after an Operator upgrade

A new Operator version changes a default, and every managed database restarts at once.

3

Acting on wrong information

The Operator misreads a temporary glitch as a real failure and deletes or replaces healthy members.

4

Runaway scaling

A feedback loop keeps adding resources because its measurement is wrong or delayed.

5

Cascading deletes

Removing a custom resource or CRD triggers deletion of data that nobody intended to remove.

Safety features worth demanding

Control

Pause switch

A simple way, such as an annotation, to tell the Operator “stop acting on this object for now”, so humans can intervene.

Limits

Rate and size limits

Maximum changes per hour and bounds on scaling, so mistakes cannot grow explosively.

Preview

Dry run

Ability to show what the Operator would do before it does it, especially for upgrades and deletes.

Data

Delete protection

Storage and backups kept by default when the custom resource is removed, unless the user explicitly chooses otherwise.

Health

Gated steps

No destructive step unless health checks pass and a quorum of healthy members exists.

Trace

Decision logs

Every automated action recorded with the reason, so changes can be explained afterward.

Budgets for disruption

Kubernetes provides an object called a PodDisruptionBudget that limits how many members of an application may be unavailable at the same time during voluntary actions such as node maintenance. A mature Operator creates and maintains one for each application it manages, so that routine cluster operations cannot take down too many members together.

Everyday analogy

Think of a surgeon with a checklist. Even the most skilled surgeon uses a pause before the first cut to confirm the patient, the procedure, and the site. These small rails do not slow down expertise; they prevent the rare catastrophic mistake. Operators need the same discipline, and the higher the level, the more it matters.

An Operator you can trust

  • Documents exactly what it will change automatically.
  • Provides pause, limits, and delete protection.
  • Fails safe: when unsure, it stops and reports.

An Operator to be wary of

  • Acts instantly on any signal without checks.
  • Deletes storage when the custom resource is removed.
  • Cannot be paused without uninstalling it.
i
Golden rule

When an automated system is unsure, the safest action is to do nothing and ask a human. Great Operators are conservative about destructive steps and generous about explaining themselves.

18Worked Example: A Database Operator Climbs the Ladder

To make the levels concrete, let us follow one imaginary Operator for a replicated database as it grows from a simple installer into an auto-pilot system. The details are illustrative, but the pattern is realistic.

Version one: Level 1

The team writes a custom resource called a “DatabaseCluster”. The user states the version, the number of replicas, and the storage size. The Operator creates the pods, the storage claims, a service for clients, and a configuration file. It marks the first replica as primary and the rest as followers. It works, and installs now take minutes instead of a day. But when the primary pod’s machine dies, the cluster stays without a primary until a human acts.

Version two: Level 2

The team adds careful upgrade logic. The Operator updates followers first, waits for each to catch up, switches the primary role to an upgraded follower, and updates the old primary last. It refuses to upgrade across unsupported versions and pauses if any member is unhealthy. Upgrade night becomes a calm event: the engineer changes one field and watches the status.

Version three: Level 3

Next comes protection. The Operator takes scheduled backups to object storage, records them, and offers a restore field that creates a new cluster from a chosen backup or a point in time. Failure handling arrives: if the primary is lost, the Operator checks that a safe majority is healthy, promotes the most up-to-date follower, and creates a replacement member. The team runs a drill by deleting the primary pod, then the whole node, and finally restoring to a fresh cluster, and measures how long each takes.

Version four: Level 4

The team adds visibility. A small exporter reports replication lag, connection counts, disk growth, and backup age. The Operator ships dashboards and alert rules, writes conditions like “ReplicaLagging” into the status, and emits events for every promotion and backup. Now on-call engineers see trouble coming, such as a disk predicted to fill in three days, instead of only reacting to outages.

Version five: Level 5

Finally, the team uses the data. The Operator can add read replicas when connection load stays high, expand storage before it fills, and adjust memory settings within safe bounds. Every automatic change obeys minimum and maximum limits, is rate limited, and is logged with its reason. A pause annotation lets humans freeze automation during sensitive events.

flowchart LR
  V1["v1: install and configure"] --> V2["v2: rolling upgrades"]
  V2 --> V3["v3: backups, restore, failover"]
  V3 --> V4["v4: metrics, alerts, dashboards"]
  V4 --> V5["v5: guarded scaling and tuning"]
    
Figure 10: One Operator’s journey, with each version unlocking the next
VersionNew abilityHuman work removedNew risk introduced
1Declarative installManual setup stepsWrong defaults at scale
2Rolling upgradesHand-run upgrade runbooksBad upgrade logic affects every cluster
3Backups and failoverEmergency recovery workWrong failover can cause data loss
4Rich insightGuessing what is wrongAlert fatigue if poorly tuned
5Guarded automationRoutine scaling and tuningAutomated mistakes at machine speed
i
Notice the pattern

Every step removes some human work and introduces a new kind of risk. The art of Operator engineering is to add each ability only when the tests and safety rails for it are ready. The ladder is a sequence of promises, and each promise must be kept before the next is made.

19Real Operators and Where They Fit

The best way to feel the maturity model is to look at real Operators. The table below describes well-known examples at a general level. Treat the capability notes as approximate, because projects add features over time and declare their own levels.

OperatorManagesNotable strengths
Prometheus OperatorMonitoring stacksDeclarative monitoring configuration; strong integration with alerting rules
StrimziApache KafkaCareful rolling updates, topic and user management, security setup
CloudNativePG, Zalando Postgres Operator, Crunchy PGOPostgreSQLFailover, backups to object storage, point-in-time recovery, monitoring
RookCeph storageDeploys and manages a distributed storage system, including health and scaling of storage daemons
Elastic Cloud on KubernetesElasticsearch and related toolsOrchestrated upgrades, node roles, security configuration
cert-managerCertificatesAutomatic issuing and renewal of certificates through custom resources
Percona OperatorsMySQL, MongoDB, PostgreSQLBackups, restore, scaling, monitoring integration

What the examples teach

Lesson

Databases demand Level 3

Almost every successful database Operator invests heavily in backup, restore, and failover, because data loss is unforgivable.

Lesson

Not every Operator needs Level 5

Many beloved Operators stop at the point where more automation would add more risk than value.

Lesson

Controllers can be Operators too

Tools like cert-manager apply the same pattern to a non-database task and show it generalizes beyond data stores.

Lesson

Ecosystems matter

Operators that plug into standard monitoring, backup, and security tools are far easier to run in production.

How to evaluate a candidate Operator

1

Read the declared level

Note the claim, then test whether the abilities behind it really exist in the documentation.

2

Try the failure scenarios

In a test cluster, kill a member, restore a backup, and run an upgrade. Watch what really happens.

3

Check the project health

Look at release frequency, open issues, number of maintainers, and how security problems are handled.

4

Read the permissions

Review the roles the Operator requests and consider whether they are justified.

5

Plan the exit

Ask how you would move the data elsewhere if the project were abandoned.

Production example: competing PostgreSQL Operators

Several communities offer PostgreSQL Operators with different philosophies. Some lean on external tools for failover, others build failover into the Operator itself. Companies commonly run a short proof of concept with two candidates, deliberately breaking each one, before choosing. The comparison is exactly the kind of structured judgment the maturity model supports.

20Maturity Is Also About Your Organization

The five levels measure one Operator. But a company adopting Operators goes through its own growth, and that journey has stages too. Understanding both keeps expectations realistic.

Why organizational maturity matters

A Level 5 Operator in the hands of a team that has never run a stateful application on Kubernetes can still cause trouble. The team may not know how to read its status, rehearse its restores, or tell when automation is misbehaving. Platform maturity is the ability of people and processes to use the tool well. The best results come when Operator capability and team capability grow together.

Four stages of adoption

1

Experimenting

A few engineers try Operators in test clusters. Nothing important depends on them yet. The goal is learning.

2

Standardizing

The platform team picks approved Operators, writes runbooks, sets up monitoring, and defines how they are installed and upgraded.

3

Self-service

Application teams request databases and queues by creating custom resources, and the platform handles the rest within policy limits.

4

Optimizing

The organization measures cost, reliability, and toil, retires weak Operators, and contributes fixes upstream.

Matching levels to stages

Adoption stageSensible Operator level for important dataWhy
ExperimentingAny, in test environments onlyLearning is the goal, so risk must stay low
StandardizingLevel 3 or higher for production dataBackup and failover must exist before real data arrives
Self-serviceLevel 3 or 4 with strong guard railsMany teams depend on it, so visibility and safety matter most
OptimizingLevel 4 or 5 where automation is proven safeCost and toil savings now outweigh the added risk
Everyday analogy

Buying a high-performance sports car does not make you a racing driver. Your skill and the car’s power need to match. A team that is still learning is often safer with a capable but conservative Operator, and then moves to more automated ones as its confidence and tooling grow.

i
Measure toil

A practical way to judge progress is to count the hours your team spends each month on manual tasks around the application: upgrades, restores, failovers, resizing. As Operator maturity and team maturity rise together, that number should fall steadily.

21Build, Buy, or Do Not Use an Operator at All

The most important decision about Operators is sometimes whether to use one. Writing your own is a long-term commitment, and the best Operator is often the one you did not have to build.

Three honest options

Use

Adopt an existing Operator

Cheapest to start, backed by a community, but you inherit its choices, its release schedule, and its limits.

Build

Write your own

Perfect fit for your needs, but you own the code, the tests, the upgrades, and the on-call forever.

Skip

Use a managed cloud service

Let a provider run the database or queue. You trade control and sometimes cost for far less operational work.

A decision flow

flowchart TD
  S["You need to run a stateful application"] --> M{"Is a managed cloud service acceptable for cost, control, and compliance?"}
  M -- "Yes" --> USEM["Use the managed service and avoid operating it yourself"]
  M -- "No" --> E{"Does a well-maintained Operator at a suitable level already exist?"}
  E -- "Yes" --> ADOPT["Adopt it, test it, and contribute fixes if needed"]
  E -- "No" --> B{"Is the application important enough and your team strong enough to maintain an Operator?"}
  B -- "Yes" --> BUILD["Build an Operator, planning tests and safety from day one"]
  B -- "No" --> SIMPLE["Use simpler tooling such as charts and runbooks"]
    
Figure 11: A simple path for deciding between managed services, existing Operators, and your own

When writing your own makes sense

Good reasons to build

  • An internal system with unique operational rules that no public Operator covers.
  • A platform product where custom resources are the main interface for developers.
  • A team with strong skills in Go or Ansible and the time to maintain the code.

Weak reasons to build

  • “It would be fun” or “it will look good on a resume”.
  • A public Operator exists that is only slightly different from what you want.
  • Nobody is assigned to maintain it after the first release.
!
The hidden cost

Building an Operator is the cheap part. Maintaining it through Kubernetes version changes, application version changes, security patches, and growing feature demands is the expensive part. Count several years of ownership when you estimate the cost.

Operators for your own platform

Not every Operator manages someone else’s software. Many companies write Operators for their own internal concepts: a “Team” object that creates namespaces, quotas, and access rules; a “Service” object that sets up deployments, monitoring, and alerts together. These are often simple Level 1 or 2 Operators, and they can be wonderfully valuable. They turn the company’s standards into a few lines of declared intent and make the right way the easy way.

22Operators and the AWS World

Many readers run Kubernetes on Amazon EKS. This chapter links the maturity ideas to what you will meet on AWS.

Controllers for AWS services

The Operator pattern is not limited to software running inside the cluster. AWS Controllers for Kubernetes, often called ACK, lets you manage AWS services such as storage buckets, queues, and databases through custom resources. You create a Kubernetes object, and a controller creates and maintains the matching cloud resource. Other projects, such as Crossplane, take a similar approach with a wider range of providers. In maturity terms, these tools often focus on the first levels: declarative creation, configuration, and drift correction, while the cloud service itself handles backups and scaling.

Karpenter as an auto-pilot example

Karpenter, an open-source node provisioning project started by AWS, watches for pods that cannot be scheduled and decides which machines to launch, then removes machines that are no longer needed. It is a good everyday illustration of a Level 5-style loop: it observes demand, decides, acts, and checks. It also shows why guard rails such as limits on total capacity are essential.

Choosing between self-run Operators and managed services

QuestionOperator on EKSManaged AWS service
Who handles backups and failover?The Operator, which you must test and monitorThe provider, within documented limits
Who patches the engine?You, through the Operator’s upgrade featureThe provider, on a maintenance schedule you can influence
How much control over tuning?Very highLimited to exposed parameters
Portability across clouds?HighLower, tied to provider features
Operational burden?HigherLower
Cost model?Pay for compute and storage plus your team’s timePay a service premium and save team time

Practical AWS tips for Operators

Identity

Give Operators their own cloud roles

When an Operator needs to reach object storage for backups, use pod-level cloud identities with narrow permissions rather than machine-wide roles.

Storage

Plan volumes and zones

Block storage volumes live in one zone. Operators must place members in sensible zones and handle volumes that cannot follow a pod to another zone.

Backups

Use durable object storage

Send backups to a bucket with versioning and, for critical data, copies in another region.

Add-ons

Treat Operators like platform add-ons

Version, test, and upgrade them on a schedule that matches your cluster upgrades.

i
A balanced approach

Many teams mix both worlds: managed services for the most critical relational data where the provider’s track record is valuable, and Operators for message brokers, caches, and internal platform objects where flexibility and portability matter more.

23Scoring Yourself: A Practical Assessment

Here is a simple checklist you can use to judge any Operator, yours or someone else’s. It turns the five levels into questions you can answer with evidence instead of impressions.

How to use the checklist

For each level, answer every question with yes or no, and ask to see it work rather than reading a claim. An Operator earns a level only when all the questions for that level and the ones below it are answered yes.

LevelQuestions to answer with evidence
1Can I create a working application from a short custom resource? Are settings changes applied? Is deletion clean, with no orphaned objects? Does status show progress and errors?
2Can I upgrade the application with no data loss and minimal downtime? Does it check health between steps? Does it reject unsupported version jumps? Can the Operator itself be upgraded without disturbing running applications?
3Are backups automatic and stored outside the cluster? Have I restored one successfully? Does it handle the loss of a member or a node? Does it avoid split-brain situations?
4Does it expose application-specific metrics? Does it ship alert rules and dashboards? Do status and events explain problems clearly? Does it report its own health?
5Does it scale and tune automatically within safe limits? Can automation be paused? Are all automatic actions logged with reasons? Does it detect abnormal behavior?

Beyond the five levels: the extra score

Because the model ignores quality, add a second, simple scorecard for the things it leaves out. Give each a rating from weak to strong.

Quality

Testing depth

Does the project run upgrade, failure, and restore tests automatically for every release?

Security

Permissions and supply chain

Are roles narrow, images signed, and vulnerabilities handled promptly?

Community

Maintenance health

Are there several active maintainers, regular releases, and responsive issue handling?

Docs

Documentation and runbooks

Can a new engineer follow the guides to recover from a failure without reading the source code?

Fit

Match to your needs

Does it support your cloud, storage, and compliance needs, and its version policy match yours?

flowchart LR
  A["Capability level: what it can do"] --> Z["Overall confidence"]
  B["Quality and testing: how well it does it"] --> Z
  C["Security: how safely it does it"] --> Z
  D["Community: how long it will keep doing it"] --> Z
    
Figure 12: Capability is only one of four inputs to your confidence in an Operator
!
Do not trust a number alone

A project with a high capability level, a single part-time maintainer, and no automated upgrade tests is a risk, however good its feature list looks. Combine the level with the extra scorecard before making decisions.

24A Roadmap for Climbing the Ladder

If you are building an Operator, the path upward is long. This roadmap suggests an order of work that keeps risk under control at every step.

1

Write down the human runbook first

Collect the exact steps an expert follows for install, upgrade, failover, backup, and restore. An Operator can only automate what is understood.

2

Design the custom resource

Choose a small, intent-based spec and a rich status with conditions. Plan the version strategy from day one.

3

Reach Level 1 with tests

Build idempotent reconciliation, ownership, finalizers, and validation. Prove them with automated tests before moving on.

4

Reach Level 2 with upgrade tests

Implement ordered, health-gated upgrades and test every supported upgrade path, including upgrading the Operator itself.

5

Reach Level 3 with drills

Add backups, restore, and failover. Run regular restore drills and failure drills, and publish how long recovery takes.

6

Reach Level 4 with real dashboards

Export meaningful metrics, ship alert rules, and improve events and conditions, guided by what on-call engineers actually need.

7

Approach Level 5 cautiously

Start by recommending actions, then automate one low-risk action at a time behind a feature switch, with limits and a pause control.

Habits that keep the roadmap healthy

Do

  • Release small changes often with clear notes.
  • Keep a compatibility table of Operator, application, and Kubernetes versions.
  • Invite users to share failure stories and add tests for each one.

Avoid

  • Adding automation before you can observe the system.
  • Breaking the custom resource API without a migration path.
  • Declaring a higher level than your tests can demonstrate.
A level is not a trophy you declare. It is a promise you can demonstrate on demand.

25Operators Compared with Helm, GitOps, and Other Tools

Newcomers often ask whether Operators replace the tools they already use. They do not. Each tool answers a different question, and the best platforms combine them.

Four tools, four different jobs

ToolMain question it answersWhen it worksWhat it does not do
HelmHow do I package and install a set of Kubernetes objects with settings?At install and upgrade time, when a person or pipeline runs itIt does not watch the running system or react to failures
GitOps tools such as Argo CD and FluxDoes the cluster match what is stored in Git?Continuously, comparing cluster objects with the repositoryThey do not understand application internals such as replication or backups
OperatorsHow do I run this specific application correctly, day after day?Continuously, inside the clusterThey do not manage your Git workflow or package unrelated applications
Infrastructure tools such as Terraform and CrossplaneHow do I create the cloud resources underneath the cluster?At provisioning time, or continuously in the case of CrossplaneThey do not manage the application running on top
Everyday analogy

Think of building and running a restaurant. Helm is the delivery truck that brings the kitchen equipment in a labeled box. GitOps is the inspector who checks that the kitchen matches the approved floor plan. Terraform is the builder who constructs the building itself. The Operator is the head chef who runs the kitchen every day and knows what to do when the oven breaks mid-service.

How they work together

A very common and healthy design uses several of these at once. The infrastructure tool creates the cluster and its cloud resources. A GitOps tool keeps the Operator itself installed and up to date, usually by deploying a Helm chart or a set of manifests from Git. Developers then commit small custom resources, such as a database request, to Git. The GitOps tool applies them, and the Operator takes over and does the real operational work. Git remains the source of truth for intent, while the Operator holds the source of expertise.

flowchart TD
  TF["Infrastructure tool creates cluster and cloud resources"] --> K["Kubernetes cluster"]
  GIT["Git repository holds Operator install and custom resources"] --> GO["GitOps tool syncs Git to the cluster"]
  GO --> K
  K --> OP["Operator reads each custom resource"]
  OP --> APP["Running application: install, upgrade, backup, heal"]
  APP -.-> OBS["Metrics and status flow back to dashboards"]
    
Figure 13: How infrastructure tools, GitOps, and an Operator divide the work

A common point of friction

One practical problem appears when a GitOps tool and an Operator both want to control the same field. For example, the Operator may update the status or a generated field, while the GitOps tool sees the difference from Git and tries to revert it. The result is a tug of war. The usual solution is to configure the GitOps tool to ignore fields that the Operator owns, and to keep only the user-facing spec in Git. Clear ownership of each field removes the conflict.

Good division of responsibility

  • Git holds only what humans decide, such as spec fields.
  • The Operator owns status and anything it generates.
  • The GitOps tool is told to ignore Operator-owned fields.

Confused ownership

  • Generated values stored back into Git by hand.
  • Two tools fixing the same difference repeatedly.
  • Nobody sure which tool is responsible for a change.

Operators versus plain scripts and scheduled jobs

Another comparison is with ordinary scripts run on a schedule. A scheduled job can take a nightly backup, and for a small system that may be entirely enough. What a script lacks is continuous awareness. It does not notice a failure at two in the afternoon, does not coordinate with an upgrade in progress, and does not report its progress in a form the rest of the platform can read. An Operator turns the same knowledge into a long-lived, cluster-native, observable process. If your needs are small and rare, a script may be the wiser and cheaper choice. As the number of applications and the cost of mistakes grow, the Operator starts to pay for itself.

i
Rule of thumb

Use packaging tools for things you install, GitOps for things you want to stay in sync with a repository, infrastructure tools for things outside the cluster, and Operators for things that need ongoing, application-specific care.

Production example: a layered platform

Many platform teams run exactly this layering. A GitOps tool installs a curated set of Operators from a repository, such as one for databases and one for monitoring. Application teams then request databases by committing a short custom resource. The platform team reviews changes to the Operators themselves, while application teams control only their own requests, giving fast self-service with clear guard rails.

26Common Mistakes and Anti-Patterns

The same errors appear again and again in Operator projects and in teams adopting them. Knowing them in advance can save months.

ANTI-PATTERN 01 · CLAIMING A LEVEL YOU CANNOT SHOWAvoid
What people do

They mark an Operator as Level 3 or 4 because a feature exists on paper, without ever testing a restore or a failure.

Why it hurts

Users trust the badge and discover the gap during an outage, when it is too late.

Better approach

Tie every claimed level to automated tests and public drills, and publish the results.

ANTI-PATTERN 02 · AUTOMATION BEFORE VISIBILITYAvoid
What people do

They rush to Level 5 scaling and tuning without solid metrics, alerts, and logs.

Why it hurts

Automatic actions based on thin data cause strange changes that nobody can explain or reverse.

Better approach

Build Level 4 first, run in recommendation mode, and only then allow actions.

ANTI-PATTERN 03 · UNTESTED RESTORESAvoid
What people do

They schedule backups and assume they work.

Why it hurts

Backups can be empty, incomplete, or unreadable. The failure is only found when the data is already gone.

Better approach

Automate restore tests into a throwaway copy and verify the data regularly.

ANTI-PATTERN 04 · OVER-PRIVILEGED OPERATORSAvoid
What people do

They give an Operator cluster-administrator rights because it is the easiest way to make it work.

Why it hurts

One compromise or bug can affect the whole cluster.

Better approach

Define narrow roles, scope to specific namespaces, and review them like any other security-sensitive code.

ANTI-PATTERN 05 · EXPOSING EVERY KNOBAvoid
What people do

They pass hundreds of raw application settings through the custom resource.

Why it hurts

The API becomes a thin, confusing wrapper that is hard to validate and impossible to evolve safely.

Better approach

Expose a small set of intent-level fields, plus a controlled way to supply advanced settings when truly needed.

ANTI-PATTERN 06 · NO WAY TO PAUSEAvoid
What people do

They build an Operator that cannot be told to stop acting on one application.

Why it hurts

During an incident, humans cannot safely intervene because the Operator keeps undoing their changes.

Better approach

Provide a simple, documented pause control and respect it in every reconcile pass.

ANTI-PATTERN 07 · ADOPTION WITHOUT OWNERSHIPAvoid
What people do

They install a third-party Operator and nobody is responsible for upgrading, monitoring, or understanding it.

Why it hurts

Versions drift, bugs go unpatched, and the first real incident is also the first time anyone reads the documentation.

Better approach

Assign a clear owner, schedule upgrades, and run drills before production depends on the Operator.

27Frequently Asked Questions

These are the questions people most often ask when they first meet the maturity model.

Q1What exactly is an Operator?
An Operator is a program running in Kubernetes that combines a custom resource, where you describe what you want, with a controller that works continuously to make reality match it, using built-in knowledge of how to run one specific application.
Q2What are the five maturity levels?
Level 1 is Basic Install, Level 2 is Seamless Upgrades, Level 3 is Full Lifecycle, Level 4 is Deep Insights, and Level 5 is Auto Pilot. A handy way to remember them is: install, upgrade, protect, observe, act.
Q3Is a higher level always better?
No. A higher level means more capability, but also more automation that must be trusted. Many excellent Operators stop at Level 3 or 4 because more automation would add risk. Choose the level that fits your application and your team.
Q4Can a Helm chart reach Level 5?
Not by itself. A chart renders objects but does not watch a running system or make decisions. A Helm-based Operator typically reaches Level 1 or 2. Higher levels need real logic, usually written in Go or driven by automation playbooks.
Q5Who decides an Operator’s level?
Usually the Operator’s authors declare it, and public catalogs display it. It is not independently audited in most cases, so verify the abilities yourself before relying on the badge.
Q6What is the difference between a controller and an Operator?
A controller is the general mechanism that reconciles desired and actual state. An Operator is a controller that carries operational knowledge about a specific application, such as how to fail over a database safely.
Q7Why do finalizers matter?
They let the Operator clean up things Kubernetes cannot remove alone, such as cloud disks or external backups, before the object disappears. Without them, deletions can leave orphaned resources behind.
Q8Should I run databases on Kubernetes with an Operator?
It can work very well with a mature Operator, tested backups, and a team that understands it. If you do not want to own that responsibility, a managed database service is often the better choice. The answer depends on control, cost, skills, and compliance needs.
Q9How do I upgrade an Operator safely?
Read the release notes, test the upgrade in a copy of production, check that it will not restart managed applications unexpectedly, take fresh backups, and use manual approval for critical Operators so upgrades happen at planned times.
Q10What happens to my data if I delete the Operator?
Deleting the Operator alone usually leaves running applications in place but unmanaged. Deleting the custom resources or the CRD can trigger cleanup that removes applications and, depending on settings, their storage. Always read the removal guidance and keep backups first.
Q11How is Karpenter or an autoscaler related to this model?
They are controllers that apply the same observe, decide, act loop. They illustrate the Level 5 idea of automated adjustment with guard rails, even though they manage infrastructure rather than a specific application.
Q12Where should a beginner start?
Install a well-known Operator in a test cluster, create a custom resource, and watch what it builds. Then break something on purpose, such as deleting a pod, and observe the recovery. After that, try an upgrade and a restore. Hands-on experience makes the levels real.

28Summary and Key Takeaways

What to remember

  • An Operator encodes expert knowledge as software. It pairs a custom resource with a controller that keeps reality in line with what you declared, for one specific application.
  • The maturity model has five cumulative levels. Basic Install, Seamless Upgrades, Full Lifecycle, Deep Insights, and Auto Pilot. Remember them as install, upgrade, protect, observe, act.
  • Higher is not always better. Each step removes human work and adds a new kind of risk. Choose the level that matches the application and your team, and stop where more automation stops paying off.
  • Build quality matters as much as capability. Idempotent reconciliation, ownership, finalizers, honest status, and clean API design show up at every level.
  • Test the level you claim. Upgrade tests for Level 2, restore and failure drills for Level 3, alert tests for Level 4, and guard-rail tests for Level 5.
  • Add safety rails and least privilege. Pause controls, limits, delete protection, narrow permissions, and a trustworthy supply chain protect you as Operators become more powerful.
  • Decide wisely whether to use one at all. Compare managed services, existing Operators, and building your own, count the cost of long-term ownership, and always assign a clear owner.