Custom Resource Definitions (CRDs) in Kubernetes
Learn how to teach a Kubernetes cluster brand-new words, how the API server stores and serves them, and how well-known projects from CoreOS, AWS, Intuit and the wider community build whole platforms on top of this one idea.
Imagine you move into a new apartment. The landlord says, “You can use the kitchen, the bedroom and the bathroom, but you cannot build a new room.” Kubernetes, straight out of the box, feels a little like that. It gives you Pods, Deployments, Services and a few dozen other building blocks, and they are very good. But sooner or later you want the cluster to understand a new idea of your own, such as a “Database”, a “Certificate” or a “Website”, as if it had always known it.
Custom Resource Definitions, usually shortened to CRDs, are the door that lets you add that new room. They are one of the most important reasons Kubernetes grew from a container runner into a general platform for building platforms. In this guide you will learn what a CRD is, how the cluster handles it inside, how it is versioned, secured and monitored, and how large projects use it in production. You will also learn the mistakes that hurt teams most often, so you can skip them.
1What Is a Custom Resource Definition?
A CRD is a way to teach Kubernetes a brand-new word, so the cluster can store and manage things it did not know about on day one.
Kubernetes is a system that runs and looks after software for you. Everything inside it is described as a resource. A Pod is a resource. A Deployment is a resource. A Service is a resource. A resource is simply a named kind of object that the cluster stores, watches over and lets you create, read, update and delete.
A Custom Resource Definition is a special object that says: “From now on, this cluster also understands a new kind of resource, and here is what it looks like.” Once you create it, you can create objects of that new kind with the same tools you already use for built-in ones, such as kubectl, YAML files and access rules. Those objects are called custom resources, or CRs for short.
Think of a restaurant that only sells burgers, fries and shakes. One day the owner prints a new section on the menu called “Salads” and tells the kitchen what a salad is. The menu change is the CRD. Every salad a customer orders afterwards is a custom resource. The restaurant did not need to be rebuilt; it only needed a new menu page.
A short history in plain words
Kubernetes began at Google and was released as open source in 2014. It was inspired by an internal Google system called Borg, which ran huge numbers of jobs across data centers. Version 1.0 arrived in 2015. Very quickly, people wanted to add their own object types, and the first answer was a feature called ThirdPartyResources.
ThirdPartyResources were awkward and had many rough edges, so they were replaced. CRDs arrived as a beta feature in Kubernetes 1.7 in 2017, and they became a fully stable feature in Kubernetes 1.16 in 2019 under the API group apiextensions.k8s.io/v1. Around the same time, the team at CoreOS popularised the idea of an “operator”, which is a program that uses custom resources to run complex software such as Prometheus. Today a large share of the Kubernetes ecosystem is built on this mechanism.
Breaking the name into three parts
Custom means it is yours, not part of the standard Kubernetes list. Resource means a type of object the cluster stores. Definition means the rule book that describes the type: its name, the fields it may have and the rules those fields must follow.
A CRD on its own only teaches the cluster to store and serve data. It does not make anything happen. To make things happen you add a small program called a controller, which you will meet in Chapter 9.
2A Quick Refresher: Resources, the API Server and etcd
To understand CRDs you need to know three things: who answers requests, where data is kept, and who does the work.
A Kubernetes cluster has a central front desk called the API server. An API (application programming interface) is just a set of web addresses that programs can call to ask for things. Every action in Kubernetes goes through this front desk. When you run kubectl, it simply sends web requests to the API server on your behalf.
Behind the front desk is a database called etcd. It is a small, very reliable key-value store, which means it keeps information as labelled pieces, much like a filing cabinet with named folders. The API server is the only component that talks to etcd directly. Everything else asks the API server.
The third piece is the group of controllers. A controller is a program that watches the cluster and tries to make reality match what you asked for. If you say “I want three copies of my web server” and only two are running, a controller notices and starts one more.
Picture a public library. The front desk is the API server: you hand over a request and it answers. The archive room is etcd: it holds the records. The librarians who walk around fixing shelves and returning books are the controllers. You never walk into the archive yourself, and the librarians also ask the front desk for information.
Desired state and actual state
Kubernetes works in a “declare, then reconcile” style. You write down the desired state, which is what you want. The cluster looks at the actual state, which is what really exists. Controllers keep closing the gap between the two. This style is called declarative, and it is the heart of everything that follows.
How resources are named
Every resource type lives in an API group and has a version and a kind. For example, Deployments live in the group apps, at version v1, with the kind Deployment. The oldest, most basic types such as Pods have no group name at all and are called the core group. On the web, built-in resources are found under /api/v1 for the core group and under /apis/GROUP/VERSION for all the other groups. CRDs use exactly the same layout, which is why they feel native.
flowchart LR
U["User with kubectl"] --> A["API Server"]
A --> E[("etcd Database")]
C1["Built-in Controllers"] --> A
C2["Your Custom Controller"] --> A
A --> N["Nodes running Pods"]
Look at the picture again and notice that your own custom controller sits in the same position as the built-in ones. It speaks to the same front desk with the same rules. That equal footing is what makes CRDs so powerful.
3CRD and Custom Resource: Rule Book and Filled-In Forms
People mix up these two words all the time. The difference is simple: one is the template, the other is the thing made from it.
A CRD describes a type. A custom resource is one actual object of that type. If you know databases, the CRD is like a table definition and each custom resource is like a row in that table. If you know programming, the CRD is like a class and the custom resource is like an instance.
A school prints a blank “Leave Request” form. The design of that form, with its boxes for name, date and reason, is the CRD. Every form a student fills in and hands to the office is a custom resource. The school designs the form once, and hundreds of students use it.
A real example: cert-manager
cert-manager is a popular open-source project that automatically gets and renews security certificates for your applications, for example from Let’s Encrypt. When you install it, it creates several CRDs, including Certificate, Issuer and ClusterIssuer. That is the rule book part, done once by whoever installs the tool.
After that, any team can create a Certificate object that says “I need a certificate for this website name.” Each of those objects is a custom resource. The cert-manager controller sees it, talks to the certificate authority, and stores the finished certificate in a Kubernetes Secret. The team never types a single certificate command.
| Question | CRD | Custom Resource |
|---|---|---|
| What is it? | The definition of a new type | One object of that type |
| Who usually creates it? | Platform team or tool installer | Application teams and automation |
| How many exist? | One per type | Anywhere from a few to many thousands |
| Is it namespaced? | No, it is always cluster-wide | Depends on the scope chosen in the CRD |
| Example | Certificate (the type) | Certificate for shop.example.com |
Deleting a CRD deletes every custom resource of that type, everywhere in the cluster. It is like tearing out a table from a database along with all its rows. Treat CRD deletion as a dangerous action and protect it carefully.
4The Anatomy of a CRD
A CRD is just a small document with a handful of parts. Once you know each part, any CRD in the wild becomes easy to read.
A CRD is itself a Kubernetes object, so it has the usual parts: a name, some labels and a body called the spec. Inside the spec, a few fields do all the important work. Here they are one by one.
Group
A web-domain style name that keeps your types separate from everyone else’s, such as a domain your company owns. It avoids two projects accidentally using the same name.
Kind
The type name written in the object, like Certificate or Backup. It is written in singular, with a capital first letter.
Plural and Singular
The lowercase words used in web addresses and kubectl commands. The plural is the one you use most, as in “get certificates”.
Short Names and Categories
Optional nicknames such as “cert” for Certificate, and group labels such as “all” so the type shows up in broad listings.
Scope
Either Namespaced or Cluster. Namespaced objects live inside one namespace, like a room in a building. Cluster-scoped objects belong to the whole building.
Versions List
One or more versions such as v1alpha1 or v1. Each version says whether it is served to users and whether it is the one stored in etcd.
Schema
A description of every allowed field, its type and its limits, written in the OpenAPI v3 format. This is the quality gate for incoming data.
Subresources and Printer Columns
Optional parts that add a separate status area, a scale handle, and custom columns in the kubectl table output.
The naming rule that surprises beginners
The name of the CRD object must be exactly the plural name, a dot, and the group. So a type with the plural backups in the group storage.example.com must be named backups.storage.example.com. If the name does not match, the API server refuses it. This rule makes sure the name is unique across the whole cluster.
Choosing between namespaced and cluster scope
Most custom resources should be namespaced. Namespaces are how teams share one cluster safely, because access rules, quotas and cleanup can all be applied per namespace. Choose cluster scope only for things that truly belong to the entire cluster, such as a cluster-wide certificate issuer or a node pool definition. You cannot casually switch from one scope to the other later, so decide with care.
Always use a domain you control for your group. Groups that end in k8s.io or kubernetes.io are reserved for the Kubernetes project itself, and the API server asks for a special approval annotation before it accepts them.
5Internal Working: What the API Server Does With a CRD
Creating a CRD feels instant, but a careful set of steps happens inside the API server before your new type goes live.
The Kubernetes API server is built from several layers. One built-in layer is called the apiextensions API server, and its job is to handle CRDs. When you submit a new CRD, it goes through the usual gates of authentication (who are you?), authorization (are you allowed?) and admission (do the policies agree?). Then the apiextensions layer takes over.
The CRD is saved
The definition is validated and written to etcd like any other object.
Names are checked
A naming controller makes sure no other type already uses the same plural, singular, kind or short names. If there is a clash, the CRD is marked as not accepted.
The schema is checked
Another controller checks that the schema is “structural”, a strict shape that Chapter 6 explains. If not, a warning condition is raised.
The type is established
An establishing controller sets the Established condition to true. This is the signal that the new endpoints may start serving traffic.
Endpoints appear live
The API server builds the new web addresses on the fly, with no restart. Discovery information and the published schema are updated, so kubectl can find the type and kubectl explain can describe it.
flowchart TD
A["Client submits CRD"] --> B["Authentication and Authorization"]
B --> C["Admission Control"]
C --> D["apiextensions API Server"]
D --> E["Naming check"]
D --> F["Schema check"]
E --> G["Established condition set"]
F --> G
G --> H["New REST endpoints served"]
D --> I[("etcd stores the CRD")]
Imagine a city hall that opens a new service counter. First someone checks that the counter’s name does not clash with another one. Then the rules for the forms are approved. Only after that is the sign switched on and citizens can queue up. No part of the building has to be closed during the process.
How custom resources are stored
When someone creates a custom resource, the request follows the same path as any built-in object: authentication, authorization, admission, validation and then storage. Custom resources are stored in etcd as JSON, under a path based on the group and the plural name. Built-in types often use a more compact format, so custom resources are a little heavier to store and to transfer.
The API server also adds standard metadata automatically. This includes a unique ID, a creation time, a resource version that changes on every update, and a generation number that changes when the spec changes. These small fields are what make controllers and caches work correctly, and you will see them again in later chapters.
6Schema Validation: Keeping the Data Clean
A good schema is like a good form: it stops mistakes before they enter the system.
Every version of a CRD carries a schema, which lists the fields a custom resource may have. The format is OpenAPI v3, a widely used way to describe data shapes. For each field you say what type it is (text, number, true or false, list, or nested group), and you can add limits such as a minimum number, a maximum length, a fixed list of allowed words or a text pattern.
An online shopping form will not let you type letters into the phone number box, will not accept a missing email address, and gives you a drop-down list for the country. A schema does the same for custom resources, but it lives on the server, so nobody can bypass it.
The “structural schema” rule
In the stable v1 API, every CRD must have a structural schema. In simple terms, this means every field must clearly declare its type, and the rules must be written in a plain, predictable way. This strictness has a big payoff: it lets the API server safely prune, default and convert objects, and it lets kubectl explain work well.
Pruning and defaulting
Pruning means the server quietly drops any field that is not in the schema. If a user mistypes a field name, it will not be stored, which keeps garbage out of etcd. If you truly need to keep unknown fields in one area, you can mark that area with a special preserve-unknown-fields flag, but use it rarely.
Defaulting means the schema can supply a value when the user leaves a field empty. For example, a “replicas” field can default to 3. Defaults are applied when data is read from etcd and when a request arrives, so older objects gain new defaults automatically.
Rules written with CEL
Simple type checks are not always enough. You may want to say “the end date must be after the start date” or “if backups are enabled, a schedule must be set.” For this, newer Kubernetes versions let you write validation rules in CEL, the Common Expression Language, directly inside the schema. CEL rules reached stable status in Kubernetes 1.29. They run inside the API server, so you do not need to build or host a separate webhook service for many checks.
| Feature | What it does | Plain-words example |
|---|---|---|
| Type and format | Checks the basic kind of value | “Port must be a whole number” |
| Enum | Allows only listed values | “Tier must be gold, silver or bronze” |
| Minimum and maximum | Limits ranges and lengths | “Replicas between 1 and 50” |
| Required fields | Rejects objects missing a field | “A database name is compulsory” |
| Pruning | Drops unknown fields | “Typo field is silently removed” |
| Defaulting | Fills empty fields | “No storage size given, use 10 GB” |
| CEL rules | Cross-field and custom logic | “End date must be after start date” |
Make your schema strict from the very first version. It is easy to relax a rule later, but very hard to tighten one after hundreds of objects already exist with loose data.
7Versioning and Conversion: Changing Your Type Without Breaking Users
Your first design is never your last one. Versions let a CRD grow while old users keep working.
A CRD can list many versions at the same time. A common journey is v1alpha1 (early and experimental), then v1beta1 (mostly stable), and finally v1 (stable and promised not to break). Each version has two switches that beginners often confuse.
- Served means users can read and write objects using that version’s web address.
- Storage means this is the version actually saved in etcd. Exactly one version must be marked as the storage version.
Think of a hotel that keeps its guest records in one master format, but front-desk staff can show a guest the record in an old or a new layout depending on who asks. The master format is the storage version. The layouts shown to guests are the served versions.
Conversion between versions
When a user asks for a version that is different from the stored one, the API server must convert the object. There are two strategies. The first is None, which only changes the version label in the object and works only when the shapes are identical. The second is a conversion webhook, a small web service you run that knows how to translate between versions field by field. Conversion webhooks became stable in Kubernetes 1.16.
A widely used design is called hub and spoke. You pick one version as the hub and write conversions between the hub and every other version, instead of writing conversions between all pairs. With four versions this means three conversion pairs rather than six, and it keeps the logic much easier to test.
flowchart LR
A["v1alpha1 spoke"] H["v1 hub"]
B["v1beta1 spoke"] H
H S[("Stored in etcd")]
Retiring an old version safely
Dropping an old version is not as simple as deleting it from the list. Objects saved in etcd may still be in that old format. The CRD keeps a list called storedVersions in its status to track every version that has ever been used for storage. Before removing a version, you must rewrite all stored objects into the new storage version, then clean up that list. Tools such as the Kubernetes storage version migrator can help with the rewrite.
You can also mark a version as deprecated and give a message. The API server then sends a warning to every user who still calls the old version, which gives people time to move on.
Teams often change the storage version without migrating existing objects, then remove the old version later. The cluster can then hold objects it can no longer read. Always migrate first, and verify before you delete.
8Spec, Status, Subresources and Printer Columns
A well-designed custom resource separates what the user wants from what the system reports back.
Almost every Kubernetes object has two main sections. The spec is the wish: what the user wants to be true. The status is the report: what the controller observed and did. A good custom resource follows this same split. Users write the spec. Controllers write the status.
When you order a pizza by app, your order form is the spec: size, toppings, address. The tracking bar that says “preparing, baking, out for delivery” is the status. You would never want the customer to edit the tracking bar, and you would never want the kitchen to change your order silently.
The status subresource
A subresource is a separate web address attached to an object that updates only one part of it. When you turn on the status subresource in a CRD, two good things happen. First, a normal update to the main object ignores any changes to the status section. Second, controllers update the status through its own address, which has its own access rules. This stops users from faking a status, and it stops a status update from accidentally overwriting a user’s spec.
The scale subresource
The scale subresource lets tools treat your custom resource like something with a replica count. You tell the CRD which field holds the desired count and which holds the observed count. After that, the standard kubectl scale command and the Horizontal Pod Autoscaler can scale your custom resource without knowing anything else about it.
Conditions: the common way to report health
Instead of one long text field, mature projects report health as a list of conditions. Each condition has a type such as Ready, a status of True, False or Unknown, a short machine-friendly reason, a human message and a last-change time. Many also record observedGeneration, the generation of the spec that the controller last handled. If it is lower than the object’s current generation, readers know the controller has not caught up yet.
Printer columns
By default, kubectl get shows only the name and age of a custom resource, which is not very useful. Additional printer columns let the CRD say “also show the Ready condition, the version and the endpoint.” This tiny feature makes custom resources far easier to use day to day.
Always enable the status subresource for any CRD that a controller manages. Without it, spec and status share one write path, and races between users and controllers become very hard to debug.
9Controllers and the Operator Pattern: Making Things Happen
A CRD stores the wish. A controller grants it. Together they form the operator pattern, the most important idea in this guide.
A controller is a program that runs a never-ending loop. It looks at the desired state in your custom resources, looks at the actual state of the world, and takes small steps to make them match. When a controller is paired with a CRD to run a complicated piece of software, the combination is called an operator. The name comes from the idea of a human operator, a person who knows how to run a database or a message queue, written down as code.
A home thermostat is a perfect controller. You set the wish: 22 degrees. The thermostat measures the room, and if it is too cold it switches the heater on; if it is warm enough it switches the heater off. It never gets tired of checking. A custom resource is the dial you turn, and the controller is the thermostat’s brain.
The reconcile loop
The heart of every controller is a function called reconcile. It is given the name of one object and does four simple things. It reads the object. It checks what exists in the cluster for that object. It creates, changes or deletes things to close any gap. Finally it writes a status report back. If anything fails, it returns an error and the controller tries again later with a growing delay.
A good reconcile function is idempotent, which is a fancy word meaning “running it twice gives the same result as running it once.” Controllers are also level-triggered. This means they react to the current state of the world rather than to a single event. Even if a controller misses a notification because it was restarting, it will still notice the gap on its next check and repair it.
flowchart TD
W["Watch events from API server"] --> Q["Work queue of object names"]
Q --> R["Reconcile one object"]
R --> G["Read desired state from custom resource"]
G --> O["Observe actual state in cluster"]
O --> D{"Do they match?"}
D -- "No" --> A["Create, update or delete child objects"]
D -- "Yes" --> S["Update status and finish"]
A --> S
S -- "Error or change" --> Q
Inside a controller: informers, caches and queues
A controller does not ask the API server for everything every few seconds, because that would overload it. Instead it opens a long-lived watch, which is like subscribing to a news feed. The API server pushes a message whenever an object changes. A helper called an informer keeps a local copy of the objects in memory, called a cache, so reads are fast. Changes are added as object names to a work queue, which removes duplicates and applies retry delays.
Most controllers run several copies for safety, but only one copy should act at a time. They choose a leader using a small shared object called a Lease. If the leader dies, another copy takes over within seconds.
Tools for building operators
You rarely write all this from scratch. The popular tools are Kubebuilder and the Operator SDK, which both build on a library called controller-runtime. They generate the CRD from your Go types, set up the informers and queues, and give you a clean place to write only the reconcile logic. Other languages have helpers too, such as the Java Operator SDK and Kopf for Python.
When a controller creates child objects, such as Pods or Services, it should mark the custom resource as their owner. Then, when the custom resource is deleted, Kubernetes garbage collection removes all the children automatically. Without owner references you leave orphans behind.
10Data Flow and Lifecycle of a Custom Resource
Follow one custom resource from the moment a user applies it to the moment it is gone.
To see the whole picture, imagine a made-up type called Website. A user wants a small website with three copies behind a web address. They write one short Website object and send it to the cluster. Here is everything that happens next.
sequenceDiagram
participant U as User
participant A as API Server
participant E as etcd
participant C as Website Controller
participant K as Kubernetes Objects
U->>A: Apply Website object
A->>A: Authenticate, authorize, admit, validate
A->>E: Store the object
A-->>C: Watch event: new Website
C->>A: Read Website and current children
C->>A: Create Deployment and Service
A->>K: Pods start running
C->>A: Write status Ready
U->>A: Get Website
A-->>U: Spec plus status
Submit
The user sends the object. The API server checks identity and permissions, runs admission policies, applies defaults, prunes unknown fields and validates against the schema.
Store
The object is saved in etcd with a resource version and a generation number.
Notify
Watchers, including the controller, receive an event saying a new Website exists.
Reconcile
The controller creates the child objects it needs, with the Website marked as their owner.
Report
The controller writes status conditions and the observed generation, so users can see progress.
Change or delete
If the user edits the spec, the generation grows and the loop runs again. If the user deletes the object, the deletion flow below begins.
Finalizers: cleaning up before goodbye
Sometimes deleting a custom resource must also clean up something outside Kubernetes, such as a cloud storage bucket or a DNS record. For this, controllers use finalizers. A finalizer is a label on the object that says “do not fully remove me until my cleanup is done.” When a user deletes the object, Kubernetes only stamps it with a deletion time. The controller sees the stamp, performs the cleanup, removes its finalizer, and only then does the object truly disappear.
Moving out of a rented flat is not finished the moment you hand over the keys. The landlord keeps a checklist: meter readings, cleaning, returned remote controls. Only when every item is ticked does the tenancy officially end. A finalizer is that checklist item.
If a controller is removed from the cluster before it clears its finalizers, the custom resources can be stuck in a Terminating state forever. Always remove controllers after the resources they manage, not before.
11Admission Control: Policies Beyond the Schema
Schemas check the shape of one object. Admission control checks the rules of your whole organisation.
Admission control is a step in the API server that happens after authorization and before the object is saved. It can change a request or reject it. For custom resources, you have three main tools.
Mutating Webhook
A web service the API server calls to edit an incoming object, for example to add default labels or fill a field that needs a calculation.
Validating Webhook
A web service that approves or rejects an object. Use it for rules that need outside knowledge, such as looking at other objects in the cluster.
ValidatingAdmissionPolicy
Rules written in CEL and kept as Kubernetes objects, with no extra server to run. It became stable in Kubernetes 1.30 and is a great fit for organisation-wide rules.
CEL Rules in the CRD
Rules that live inside the schema itself and travel with the CRD. Best for rules about the object’s own fields.
At an airport, the boarding-pass scanner checks that your ticket is shaped correctly. That is the schema. A security officer checks that you follow rules the ticket cannot show, such as banned items. That is admission control. A customs officer may even stamp your passport. That is a mutating step.
The danger of webhooks
Webhooks are powerful but they add a new moving part to your cluster. If a validating webhook is down and its failure policy is set to Fail, every request it covers is rejected. If the webhook protects the very namespace where it runs, the cluster can deadlock: the webhook cannot restart because it is blocked by itself.
To stay safe, keep timeouts short, run at least two replicas spread over different nodes, exclude the webhook’s own namespace and the system namespaces from its scope, and choose the failure policy by asking “which is worse, a blocked request or an unchecked one?” Prefer CEL rules in the schema or in a ValidatingAdmissionPolicy whenever they can express your rule, because they run inside the API server and cannot go offline separately.
Use the schema for field shape, CEL for cross-field rules, a policy for organisation rules, and a webhook only when you truly need logic or data that CEL cannot reach.
12Security: Who May Create, Read and Change Custom Resources
Custom resources are first-class citizens, so they need first-class protection.
Kubernetes uses RBAC, short for role-based access control, to decide who may do what. A role lists verbs such as get, list, create, update and delete, and the resources they apply to. For a custom resource, you name the API group and the plural resource name, for example “the group storage.example.com, resource backups.” Users and programs then receive that role through a binding.
RBAC is the key-card system of an office building. A role is a type of key card, such as “visitor” or “facilities”. A binding is handing a card to a particular person. A CRD is a new door added to the building, and you must decide which cards open it.
The main security points
Creating CRDs is cluster-level
CRDs are cluster-wide objects. Only trusted platform administrators should be allowed to create, edit or delete them.
Per-resource roles
Give application teams roles scoped to their namespace and to specific verbs. Separate read access from write access.
Aggregated roles
Add labels to ClusterRoles so that the built-in view, edit and admin roles automatically include your new resource.
Least privilege
Give your controller only the permissions it needs. A controller with cluster-admin rights becomes a prime target for attackers.
No secrets in specs
Custom resource specs are readable by anyone with get access. Store passwords in Secrets and reference them by name instead.
Webhook TLS
Webhooks must serve over TLS with certificates the API server trusts. Automate rotation, often with cert-manager.
Supply chain and audit
Installing a third-party operator usually means applying many CRDs and a controller with broad permissions. Treat this like installing a program on a server: read what it asks for, pin the version, and keep the manifests in your own repository. Also turn on API server audit logging for your custom groups, so you can answer the question “who changed this object and when?” after an incident.
If a custom resource lets a user say “run this container with these permissions,” then permission to create that resource is as powerful as permission to create Pods. Review every field that can lead to code running in the cluster.
13Performance and Scalability
CRDs are cheap to create but they are not free. Every type and every object costs memory, storage and network.
Each CRD adds a new set of endpoints to the API server. The server must hold the schema in memory, include it in discovery documents and publish it in the schema listing. A cluster with a handful of CRDs will never notice. A cluster with hundreds of CRDs, as can happen when many operators are installed, may see slower start-up, larger discovery responses and higher API server memory use.
Each CRD is like adding a new department to a company directory. One more department costs almost nothing. But if the directory grows to hundreds of departments, every phone call to the reception desk takes longer because the receptionist has more pages to flip.
Where the load really comes from
- Number of objects. etcd has a limited database size, and every custom resource uses some of it. Large clusters should watch the database size and set sensible limits.
- Size of each object. A single request to etcd is limited to roughly 1.5 MB by default. Large blobs of data belong in object storage or a ConfigMap reference, not inside a custom resource.
- Frequency of change. Every write is a new version in etcd and a notification to every watcher. A status field that changes every second for thousands of objects can overwhelm the whole API server.
- Number of watchers. Each controller replica holds a full cache and a watch. Many controllers watching the same large type multiply the cost.
- Large lists. Asking for every object in one call is expensive. Use pagination and label selectors.
| Problem | Symptom | Typical fix |
|---|---|---|
| Hot status writes | High write rate, slow API server | Update status only when it truly changes; batch updates |
| Huge objects | Requests rejected as too large | Store payloads outside, keep only references |
| Too many CRDs | Slow discovery, high memory | Remove unused operators; consolidate related types |
| Controller overload | Growing work queue | Tune workers; add rate limits; avoid needless reconciles |
| Full relists | Spikes in memory and traffic | Use watch with resource versions; use pagination |
High availability
The CRD data itself is as available as the cluster control plane, because it lives in the same etcd. What you must design for is the controller. Run at least two replicas with leader election, spread them across zones, and set resource requests and limits. Remember that if the controller is down, existing workloads keep running, but no new changes are processed until it returns. Also plan for the webhook and conversion services, which are in the request path.
14Monitoring, Logging and Metrics
You can only fix what you can see. A good operator tells you what it is doing in three ways: status, metrics and logs.
The first level of monitoring is the status you already designed. Anyone can run kubectl get or kubectl describe and read the conditions. Good conditions turn a vague “it is broken” into a precise “Ready is False because the storage class does not exist.”
The second level is metrics, which are numbers collected over time, usually by Prometheus. The controller-runtime library exposes ready-made metrics for every controller, including how many reconciles ran, how many failed, how long they took and how deep the work queue is. A growing queue is the clearest sign that a controller cannot keep up. The API server also reports request counts and latency per resource, so you can see which custom types are the busiest.
The third level is logs and events. Controllers should write structured logs that always include the object’s namespace and name. They should also emit Kubernetes Events, short human-readable messages that show up under kubectl describe, for important milestones and failures.
Reconcile errors
The count of failed reconciles. A steady rise means a bug, a missing permission or a broken dependency.
Work queue depth
How many objects are waiting. If it never drains, the controller is too slow or stuck.
Reconcile duration
How long each loop takes. Long loops often mean slow calls to an outside service.
Resources by condition
How many objects are Ready versus not Ready. Tools such as kube-state-metrics can turn custom resource fields into metrics.
API request latency
Slow reads and writes for your group point to heavy objects or an overloaded control plane.
Deprecated API calls
The API server counts requests to deprecated versions. Use it to find clients that must upgrade before you remove a version.
Alert people when users are hurt, for example “many Websites have been not Ready for ten minutes,” rather than on tiny internal details. Keep detailed metrics for investigating, not for waking someone at night.
15Failure Scenarios and Recovery
Every system fails sometimes. Knowing the common failures in advance turns a midnight panic into a calm checklist.
The controller crashes
Running workloads keep running because Pods do not depend on the controller to stay alive. New and changed custom resources simply wait. When the controller returns it reads the current state and catches up, because it is level-triggered. Recovery: restart or reschedule, and check that leader election moved to a healthy replica.
The CRD is deleted by mistake
All custom resources of that type are removed, and in the worst case the controller then deletes the real things they described, such as databases. Recovery: restore from backups of the CRDs and objects, which is why GitOps repositories and backup tools matter. Prevention: strict RBAC, and finalizers or admission policies that block deletion of important CRDs.
A conversion webhook is down
Any request that needs conversion fails, and so can listing objects across versions. Even controllers can break. Recovery: restore the webhook quickly; run several replicas; and avoid conversion in the hot path by keeping the stored version and the most-used version the same.
A validating webhook blocks everything
An outage in the webhook rejects all new writes under its scope. Recovery: delete or patch the webhook configuration to restore writes, then fix the service. Prevention: failure policy chosen deliberately, namespace exclusions and replicas.
Objects stuck in Terminating
A finalizer is waiting for a controller that no longer exists or cannot reach an outside system. Recovery: bring the controller back so it can finish cleanup. Only as a last resort, remove the finalizer by hand and clean up the outside resource yourself.
Reconcile storms
A bug causes a controller to update an object, which triggers its own watch, which causes another update, and so on forever. The API server and etcd are flooded. Recovery: stop the controller, fix the loop so it writes only when something truly changed, and restart. Prevention: compare before writing, and use predicates to ignore changes that do not matter.
An etcd backup protects the whole cluster, but teams often forget to back up the cloud resources behind the custom resources. Remember that an operator’s real data may live outside Kubernetes, so test restores end to end.
16CRDs Compared With the Alternatives
CRDs are not the only way to add configuration or behaviour to a cluster. Choosing the right tool saves months of pain.
Before you create a CRD, ask a simple question: “Does this thing need to be a first-class object that people create, list, watch and control with access rules?” If the answer is no, a lighter tool may do. If the answer is yes, a CRD is usually the right choice. Here are the common options side by side.
| Option | Best for | Strengths | Weak points |
|---|---|---|---|
| ConfigMap | Plain settings read by one application | Simple, built in, no extra install | No schema, no validation, no status, no versioning |
| Helm values | Packaging and installing an application | Templating, easy sharing of charts | Acts only at install time; does not keep watching or repairing |
| Custom Resource Definition | New object types with a controller | Native API, schema, RBAC, watch, status, versions | You must build and run a controller; shares the same etcd |
| Aggregated API server | Types needing custom storage or special behaviour | Full control over storage and request handling | You must write, run and secure a whole API server |
| Plain script or CI job | One-off or occasional tasks | Easy to start | No continuous reconciliation; drift goes unnoticed |
A ConfigMap is a sticky note on the fridge: quick, but nobody checks it. Helm is a moving company that sets up your furniture once and leaves. A CRD with a controller is a live-in housekeeper who keeps looking at the house and putting things right every day. An aggregated API server is building your own house from the foundation up.
When an aggregated API server is worth it
An aggregated API server is your own server that plugs into the main API server and handles a whole API group. You choose it when you need something CRDs cannot do, such as storing data in a different database, serving huge numbers of objects with special performance needs, or adding unusual actions like streaming logs. The metrics server is a well-known example. For nearly everything else, CRDs are far easier, because Kubernetes gives you storage, validation, versioning, watches and access control for free.
17Advantages, Disadvantages and Trade-offs
CRDs give you a lot for little effort, but they also bring new responsibilities.
Advantages
- Your new type feels native: same kubectl, same YAML, same access rules.
- Free storage, watches, versioning and audit logging from the API server.
- Declarative by nature, so teams can keep everything in Git and review changes.
- Schema validation rejects bad data before it can cause harm.
- Huge ecosystem: tools for building, testing and packaging operators.
- Lets a platform team offer simple self-service building blocks to developers.
- No API server restart needed to add or change a type.
Disadvantages
- A CRD alone does nothing; you must write and run a reliable controller.
- Everything shares the cluster’s etcd, so heavy use can hurt the whole cluster.
- Cluster-wide CRDs are shared, so versions and upgrades need cluster-level coordination.
- Version changes need careful conversion and migration planning.
- Webhooks add new points of failure in the request path.
- Too many CRDs from many operators can slow the API server and confuse users.
- Debugging a controller needs skills beyond writing YAML.
The main trade-offs, in plain words
- Power versus simplicity. A rich custom API can hide complexity from users, but the complexity does not vanish. It moves into the controller your team must maintain.
- Strict versus flexible. A tight schema gives safety and good documentation. A loose schema is quicker to change but allows mistakes to pile up.
- Shared versus separate. Putting many tenants on one cluster saves money, but one tenant’s CRD problems may affect all.
- Declarative versus immediate. Controllers work in the background and take time. If a user needs an instant answer, a custom resource may feel slow.
18Design Patterns and Anti-patterns
Experienced teams keep repeating the same good shapes, and keep avoiding the same bad ones.
Patterns worth copying
Operator
One CRD per application, with a controller that encodes the expert’s knowledge: install, upgrade, back up and heal. Examples are the Prometheus Operator and Strimzi for Kafka.
Spec and Status Split
Users write the spec; the controller writes the status. This keeps the contract clear and avoids write conflicts.
Declarative Wrapper
Represent an outside system as a custom resource. AWS Controllers for Kubernetes lets teams manage cloud services such as storage buckets and databases using Kubernetes objects.
Composition and Claims
A platform team defines a high-level type, such as a “Database”, that expands into many low-level resources. Developers create a small claim. Crossplane popularised this idea.
Role-Oriented API
Split a large concern into several types owned by different roles. The Gateway API separates the class of gateway, the gateway itself and the routes, so infrastructure and application teams each own their part.
Hub-and-Spoke Versions
One internal hub version with conversions to every other version, which keeps upgrades manageable.
Anti-patterns to avoid
What it looks like
Using custom resources as a general-purpose database for application data such as user profiles, orders or logs.
Why it hurts
etcd is meant for small, important cluster state, not for millions of business records. Heavy use slows the whole control plane and risks hitting storage limits that affect every workload.
Do this instead
Keep business data in a real database. Use custom resources only for configuration and desired state, and reference external data by name or address.
What it looks like
Treating a custom resource as a command, such as an object named “RunBackupNow” that must be created each time to trigger an action.
Why it hurts
Kubernetes is built around states, not commands. Command objects pile up, are hard to replay and confuse tools that expect to reapply the same file safely.
Do this instead
Describe the desired result, such as a backup schedule or a “latest backup requested” field, and let the controller decide when and how to act. For true one-off actions, use a Job.
What it looks like
One giant CRD with hundreds of optional fields that tries to describe every use case, often copying the full Pod specification inside it.
Why it hurts
The schema becomes huge and hard to read, validation is weak, and every change in Kubernetes itself forces a change in your CRD.
Do this instead
Offer a small, opinionated set of fields for the 90 percent case, and provide one clearly named escape hatch for advanced users.
19Best Practices and Common Mistakes
These habits come from years of running operators in real clusters.
Best practices
Design the API like a product
Pick clear names, keep fields few and well described, and write the schema with descriptions so that kubectl explain teaches users how to fill it in.
Start small and alpha
Release as v1alpha1, learn from real users, then promote to beta and v1 once the shape settles.
Always use status with conditions
Report Ready, the reason and the observed generation, so humans and tools can tell what is going on.
Make reconcile idempotent
Safe to repeat, safe to interrupt, and writing only when something changed.
Use owner references and finalizers wisely
Owner references for in-cluster children, finalizers only for outside cleanup, and always remove finalizers once cleanup is done.
Manage CRDs as code
Keep CRDs in Git, review changes, test upgrades in a staging cluster, and apply them before the controller version that needs them.
Test the full lifecycle
Create, update, delete, upgrade and failure. Run tests against a real API server, not only against mocks.
Common mistakes
- Forgetting that Helm does not upgrade CRDs. Helm 3 installs CRDs placed in a special crds folder, but it does not upgrade or delete them later. Many teams ship CRDs in a separate chart or apply them with a dedicated step.
- Starting the controller before the CRD is ready. The controller fails because the type does not exist yet. Wait for the Established condition.
- Making breaking schema changes in the same version. Renaming or removing a field in v1 breaks users. Add a new version instead.
- Letting the controller edit the spec. This fights with users and with GitOps tools. Write results to status instead.
- Ignoring pruning. Fields missing from the schema are silently dropped, which looks like data loss to a confused user.
- Treating all errors as retryable. A permanently invalid object retried forever wastes resources. Report it in status and stop retrying until it changes.
- No limits on controller resources. A busy operator without memory limits can take down its node.
Use kubectl get crds to list every type in the cluster and kubectl explain followed by the resource name to read its documentation. These two commands answer most beginner questions about an unfamiliar operator.
20Real-World and Industry Examples
Some of the best-known tools in the Kubernetes world are really just CRDs plus a controller.
Prometheus Operator: monitoring as objects
Born at CoreOS and one of the first famous operators. It adds types such as Prometheus, Alertmanager, ServiceMonitor and PrometheusRule. A team creates a ServiceMonitor to say “scrape this service”, and the operator rewrites the monitoring configuration. Nobody edits a config file by hand.
cert-manager: certificates on autopilot
A CNCF project that adds Certificate, Issuer and ClusterIssuer. It requests, renews and stores certificates, for example from Let’s Encrypt. Teams across the industry use it so they never again face an expired certificate at 3 a.m.
Istio: traffic rules as custom resources
The Istio service mesh uses types such as VirtualService, DestinationRule and Gateway. Engineers describe how traffic should be split between versions of a service, and the mesh does the rest. This allows gradual rollouts at big companies without changing application code.
Argo CD: GitOps with Application objects
Started at Intuit and now a CNCF project. It adds Application, ApplicationSet and AppProject. An Application says “make the cluster match this Git folder”, and Argo CD keeps checking and syncing. It is a good example of a controller that manages other controllers’ inputs.
AWS Controllers for Kubernetes and Karpenter: the cloud as objects
AWS Controllers for Kubernetes provides CRDs for AWS services such as S3 buckets, RDS databases and DynamoDB tables, so infrastructure can be created with the same tools as applications. Karpenter, which came from AWS, adds types such as NodePool, EC2NodeClass and NodeClaim, and uses them to add and remove EC2 machines based on what Pods need.
Gateway API: a standard born as CRDs
Built by the Kubernetes networking community, the Gateway API adds GatewayClass, Gateway and HTTPRoute as CRDs. Because they are CRDs rather than core types, they can be released and improved on their own schedule, independent of Kubernetes releases. Several vendors and cloud providers implement it.
Strimzi and Zalando Postgres Operator: databases on demand
Strimzi gives you Kafka, KafkaTopic and KafkaUser types for running Apache Kafka. The Postgres Operator from Zalando offers a PostgreSQL type that sets up a replicated database cluster with automatic failover. In both cases, a few lines of YAML replace days of manual administration.
Crossplane: build your own platform
Crossplane lets a platform team define its own high-level APIs. It uses a CompositeResourceDefinition that itself generates new CRDs. Developers ask for a “Database” and get the right cloud resources, network and secrets behind the scenes, with company rules baked in.
21Frequently Asked Questions
Short answers to the questions beginners ask most.
22Summary and Key Takeaways
A Custom Resource Definition is a small document with a big effect: it turns Kubernetes from a fixed set of tools into a platform you can extend.
You started with a simple picture, a menu with a new page added, and ended with a detailed view of how the API server serves, stores, validates, converts, secures and monitors your own types. The big idea is that Kubernetes treats your custom objects exactly like its own. The big responsibility is that you must design them well, because real users and real systems will depend on them.
Key Takeaways
- A CRD defines a type; a custom resource is one object of that type. Deleting a CRD deletes every object made from it.
- A CRD only stores and serves data. A controller, paired with the CRD as an operator, makes things happen by reconciling desired state with actual state.
- Design the schema strictly and early. Use structural schemas, defaults and CEL rules, and keep spec for users and status for controllers.
- Plan versions from day one. Use served and storage versions, conversion where needed, and migrate stored objects before retiring any version.
- Protect and observe your types. Apply least-privilege RBAC, treat webhooks as critical services, and watch reconcile errors, queue depth and status conditions.
- Respect the shared etcd. Keep objects small, avoid hot status writes and never use custom resources as an application database.
- Pick the lightest tool that fits. Use a ConfigMap for simple settings, a CRD with a controller for continuous management, and an aggregated API server only when CRDs truly cannot do the job.