OpenAPI and CRD Schema Validation in Kubernetes

OpenAPI and CRD Schema Validation in Kubernetes

How Kubernetes decides which custom objects are allowed through the front door, and how you write rules that stop bad data before it can hurt your cluster.

Picture a small bakery that takes orders on blank pieces of paper. One customer writes: “Cake, size: purple, candles: minus four, deliver: yesterday.” The baker is confused, the delivery driver is stuck, and nobody gets a cake. Now picture the same bakery with a printed order form. Size must be small, medium or large. Candles must be a number between 0 and 50. The delivery date must be today or later. Bad orders are stopped at the counter, before anyone turns on the oven.

Kubernetes faces exactly this problem. Thousands of people, scripts and tools send it instructions every minute, and many of those instructions describe brand-new kinds of objects that the Kubernetes authors never saw. Something has to check every one of those instructions against a printed form. That printed form is called a schema, the language it is written in comes from the OpenAPI specification, and the way you attach it to your own object types is the CustomResourceDefinition, or CRD.

In this guide you will learn what a schema is, how to read and write one, what the strict “structural” rules are and why they exist, how Kubernetes prunes and fills in fields, how to write smart cross-field rules with the Common Expression Language (CEL), and how all of this fits with admission webhooks and policy engines. Every new word is explained the first time it appears, so you can start from zero.

1Why Validation Exists in the First Place

Before we touch a single keyword, we need to understand the problem that schema validation solves. It is simpler than it sounds.

Kubernetes in sixty seconds

A cluster is a group of computers that work together to run your applications. Inside the cluster there is a central program called the API server (its real name is kube-apiserver). Think of it as the front desk of a hotel. Every request, whether it comes from a person typing kubectl or from another program, goes through this one desk. Nobody is allowed to walk into the back rooms.

Behind the front desk sits etcd, a small, reliable database that stores the official record of everything in the cluster. Around it work many controllers, which are programs that constantly compare “what the record says should exist” with “what actually exists” and then fix any difference. If the record says “run three copies of this web server” and only two are running, a controller starts a third.

This style is called declarative. You do not give step-by-step orders. You describe the end result you want, store that description, and let the controllers do the work. It is like leaving a note for a house cleaner that says “the kitchen should be spotless” instead of standing next to them giving directions.

What goes wrong without a form

Because controllers trust whatever is in the record, a bad record is dangerous. Imagine someone stores a database object with a replica count of “many” instead of a number. The controller reads it, cannot understand it, and either crashes or silently ignores it. If the controller crashes in a loop, every other object it manages also stops getting attention. The mistake spreads from one typo to a whole team’s outage.

The longer a bad value sits in storage, the harder it is to clean up. The best moment to catch a mistake is the moment it is submitted, when the person who made it is still looking at their screen. This idea is often called fail fast. Validation at the front desk is the cheapest form of failing fast, because the error message appears right in the terminal, naming the exact field that is wrong.

Everyday analogy

Think of airport check-in. The agent at the counter does not care about your holiday plans. They only check that your passport is real, your name matches the ticket, and your bag is under the weight limit. Catching a problem here costs two minutes. Catching the same problem at the boarding gate, or worse, after the plane has taken off, costs hours. Schema validation is the check-in counter for your Kubernetes objects.

Where the idea came from

Kubernetes was open-sourced by Google in 2014, shaped by lessons from the internal cluster systems the company had run for years. From the start, the built-in object types such as Pods and Deployments were checked by hand-written code inside the API server. That worked well for the types the core team wrote. It did not work for everyone else who wanted to teach Kubernetes about their own things, such as certificates, databases, message queues or machine-learning jobs.

The answer was to let users add new object types without changing the API server’s code. Early on this was done with a feature called ThirdPartyResources, which was awkward and was soon replaced. Its successor, the CustomResourceDefinition, became the standard way to extend Kubernetes. A CRD says, in effect, “from now on this cluster also understands objects of this new kind.” Along with the new kind comes a schema, so the API server can do the same checking for custom objects that it does for built-in ones.

1.16
release where the stable CRD API (apiextensions.k8s.io/v1) arrived
1.29
release where CEL validation rules became stable for CRDs
1.22
release that removed the old v1beta1 CRD API completely
1

ThirdPartyResources (early days)

The first, rough way to add custom types. It lacked good validation and was replaced.

2

CustomResourceDefinitions arrive (around 1.7)

A cleaner extension point. Schemas were optional at first, so many CRDs shipped with no checks at all.

3

OpenAPI v3 validation (around 1.9 onward)

CRDs could carry an OpenAPI v3 schema, and the API server began enforcing it on create and update.

4

The v1 API and structural schemas (1.16)

Schemas became mandatory and had to follow strict “structural” rules, which unlocked pruning and defaulting.

5

CEL rules and ratcheting (1.25 onward)

Rules that compare fields to each other arrived, and later a gentle mode that avoids breaking old objects.

i
Good to know

The API server rejects an invalid object with an HTTP status of 422, which means “I understood your request but the content is not acceptable.” The response body lists every failing field, so one attempt often shows you all the problems at once.

2Core Concepts: The Vocabulary You Need

A handful of words appear again and again in this topic. Learn them once and the rest of the article will feel easy.

TermWhat it meansEveryday picture
ResourceA type of object the API serves, such as Pods or Services.A category of form at the front desk.
CustomResourceDefinition (CRD)An object that teaches the cluster a brand-new resource type.The blueprint for a new kind of form.
Custom resource (CR)One actual object of that new type.One filled-in copy of the form.
SchemaA description of what fields an object may have and what values are allowed.The printed boxes and instructions on the form.
OpenAPIA widely used standard for describing APIs and their data shapes.The common language all form designers agree on.
ValidationChecking a submitted object against the schema and rules.The clerk checking your form before stamping it.
AdmissionThe stage where the API server can change or reject a request after login checks.The security gate between the front desk and the back office.
Controller / operatorA program that acts on stored objects to make reality match them.The kitchen staff cooking from the order slips.

What is OpenAPI, really?

OpenAPI is a public standard for describing how an API behaves and what data it sends and receives. It started life as a project called Swagger and was later donated to a neutral industry group, the OpenAPI Initiative, which is part of the Linux Foundation. Today it is used by companies of every size to produce documentation, client libraries and automatic checks.

The part of OpenAPI that matters for Kubernetes is the schema object. A schema object is a small description of a piece of data: “this is a string no longer than 63 characters,” or “this is a list of numbers, at least one and at most ten.” It is based on JSON Schema, a separate standard for describing JSON data. Kubernetes uses version 3 of the OpenAPI schema language, which is why you will see the field name openAPIV3Schema inside a CRD.

Kubernetes does not support every feature of OpenAPI. It takes a carefully chosen subset and adds its own extra keywords, all starting with x-kubernetes-. The standard allows custom extensions with an x- prefix, and Kubernetes uses that door to add behaviours such as “treat this list like a map.” We will meet them in Chapter 7.

Everyday analogy

A restaurant has three layers. The front desk takes the order slip (the API server). The filing cabinet keeps every slip safe (etcd). The kitchen reads the slips and cooks (controllers). A CRD is like adding a new section to the menu: you also have to print a new order slip layout so the front desk knows how to check slips for the new dish. The schema is that slip layout.

?
Don’t confuse these

“CRD” is the definition of a type. “Custom resource” is one object of that type. A cluster might have one CRD for certificates and ten thousand certificate objects. Validation rules live on the CRD, and they are applied to every custom resource of that type.

3Architecture: The Anatomy of a CRD

A CRD is just another Kubernetes object, and it has a handful of important parts. The schema sits inside one of them.

When you create a CRD, you are talking to a built-in component of the API server called the apiextensions server. It watches for new CRDs. As soon as one is accepted, it starts serving a brand-new set of web addresses for the new type, with no restart and no new code. The new type then behaves like any built-in type: you can create, read, update, delete, list and watch it, and kubectl understands it.

flowchart TD
  A["CustomResourceDefinition"] --> B["group and names"]
  A --> C["scope"]
  A --> D["versions list"]
  D --> E["served flag"]
  D --> F["storage flag"]
  D --> G["openAPIV3Schema"]
  D --> H["subresources"]
  D --> I["additionalPrinterColumns"]
  G --> J["properties and types"]
  G --> K["validation keywords"]
  G --> L["x-kubernetes-validations"]
    
Figure 1: The main parts of a CRD. The schema lives inside each entry of the versions list.

Read the diagram from top to bottom. The CRD has a name and group at the top, a scope, and a list of versions. Each version carries its own schema, so version one of your type can have different rules from version two. Inside the schema you find the plain shape (properties and types), the standard validation keywords, and the Kubernetes-specific CEL rules.

Identity

Group and names

The group is a domain-style prefix such as example.com that avoids clashes. The names block gives the kind, plural, singular and optional short names used on the command line.

Reach

Scope

Namespaced means each object lives inside a namespace, like a folder. Cluster means one set for the whole cluster, like a shared library card catalogue.

Evolution

Versions

A list such as v1alpha1, v1beta1, v1. Each has a served flag (can clients use it?) and a storage flag (is this the form saved in etcd?). Exactly one version is the storage version.

Rules

The schema

The openAPIV3Schema block describes every allowed field. It is required in the stable v1 API and must be structural.

Extras

Subresources

Optional add-ons such as status (a separate section controllers write to) and scale (a standard way to resize). They change who is allowed to edit what.

Friendliness

Printer columns

Extra columns shown by kubectl get, such as a “Ready” flag, so people can see health at a glance.

Spec and status: two halves of one object

Most custom resources follow a convention borrowed from built-in types. The spec section holds what the user wants. The status section holds what the controller observed. Users write the spec, controllers write the status. When you enable the status subresource, the API server enforces this split: updating the main object ignores changes to status, and updating the status address ignores changes to spec. That separation means a user cannot accidentally (or deliberately) forge a controller’s report.

Validation applies to both halves. A good schema constrains the spec tightly, because it is human input, and also describes the status carefully, so tools and dashboards know what to expect.

Scope and version flags in practice

The served flag lets you retire an old version gracefully. Setting it to false stops clients from using that version while the data in storage is still readable through newer versions. The storage flag decides which version’s shape is written to etcd. Changing the storage version later does not rewrite old data on its own; objects are converted when they are next written or when you run a migration. We return to this in Chapter 10.

!
Watch out

Pick the group and kind with care. Once real objects exist in thousands of clusters, renaming either one is a painful migration, because every stored object and every client is tied to those names.

4OpenAPI v3 Building Blocks You Will Use Daily

A schema is built from a small toolbox. Once you know each tool, you can read almost any CRD in the wild.

Types: the six basic shapes

Every piece of data in a schema has a type. There are six you need to know. A string is text. An integer is a whole number. A number is any number, including decimals. A boolean is true or false. An object is a bundle of named fields, like a form with labelled boxes. An array is an ordered list of items, like a shopping list.

Objects and arrays are special because they contain other things. An object lists its fields under properties, and each property has its own schema. An array describes what each entry looks like under items. This nesting is how a tiny schema grows into a description of a very large object, one small piece at a time.

The validation toolbox

KeywordApplies toWhat it enforcesEveryday picture
requiredobjectsListed fields must be present.A box on a form marked with a red star.
enumanyValue must be one of a fixed list.A drop-down menu: small, medium or large.
minimum, maximumnumbersValue must stay inside a range.A speed limit sign.
minLength, maxLengthstringsText must not be too short or too long.A tweet-style character cap.
patternstringsText must match a regular expression.A phone number box that accepts only digits.
formatstringsText must look like a known kind, such as a date-time, uri, ipv4 or uuid.An email box that rejects text without an at-sign.
minItems, maxItemsarraysList must have a sensible number of entries.“Name between one and five guests.”
uniqueItemsarraysNo duplicates. Not allowed in CRDs, see Chapter 5.A raffle where each ticket number is used once.
defaultanyValue to fill in when the user leaves the field out.A pre-ticked checkbox.
nullableanyWhether an explicit null is accepted.Allowing a box to be marked “not applicable”.
descriptionanyHuman-readable help text shown by kubectl explain.The small grey help text under a box.

Required versus optional, and why it matters

By default, every field in an object is optional. You turn a field into a mandatory one by adding its name to the object’s required list. This is a decision with real consequences, because once a field is required, every existing tool that creates the object must supply it. Adding a new required field to a version people already use will break their automation the next time it runs. As a rule of thumb, start with fewer required fields and use defaults, then tighten later only in a new version.

Closed lists with enum

The enum keyword is the simplest and most powerful tool for catching typos. If a field called tier can only be “free”, “standard” or “premium”, then writing “premuim” fails immediately with an error that lists the allowed values. Without an enum, the typo would be stored happily, and the controller would either guess or fail much later.

Strings need limits too

An unlimited string is an open invitation for trouble. A name that is a million characters long could slow down the API server, bloat etcd and break tools that print it. Setting maxLength and, where it makes sense, a pattern keeps values sensible. There is also a hidden reason to set limits that we cover in Chapter 8: CEL rules need to know how big a value can get, and without a limit the API server assumes the worst case.

Everyday analogy

Think of a customs form at a border. It has boxes for your name (text, limited length), your age (a whole number, between 0 and 130), your reason for travel (pick one from a list), and a tick-box for “carrying food”. Officers do not need to understand your life story. They only check that each box holds the right kind of thing. Schema keywords do the same job for objects.

Maps: when the field names are not known in advance

Sometimes you cannot list every field name ahead of time. Labels on a Pod are a good example: users invent their own keys. For this the schema uses additionalProperties, which describes the shape of every value in a “free-keyed” object, such as “each value must be a string.” This is the schema’s way of saying “a dictionary of names to values.”

An object can use either properties (a fixed list of named fields) or additionalProperties (a dictionary), but not both on the same level. Choosing between them is a design decision: fixed fields give you strong checking and nice documentation; dictionaries give flexibility but weaker checks.

i
Tip

Always fill in description for every field. The API server publishes these descriptions, and kubectl explain shows them to your users. A schema without descriptions is like a form with unlabelled boxes.

5Structural Schemas: The Strict House Rules

OpenAPI is flexible. Kubernetes deliberately narrows it. Understanding why makes the rules feel logical instead of annoying.

In the stable CRD API, a schema must be structural. That is a fancy word for a simple idea: the schema must describe the real shape of the data clearly enough that the API server can work with it automatically. The server needs to know, for every piece of data, exactly what type it is and where it lives. Only then can it safely remove unknown fields, fill in defaults and run CEL rules.

Everyday analogy

Imagine a warehouse robot that sorts parcels. If every parcel has a clear label with its size and destination, the robot works alone, all day, without mistakes. If some parcels are labelled “it depends” or “see the other label”, the robot freezes. A structural schema is the clear label that lets the API server act as the robot.

The rules, in plain words

1

Every node declares a type

The root, each property, each array item and each dictionary value must say what type it is. The only exceptions are nodes marked with the special int-or-string or preserve-unknown-fields extensions.

2

Describe fields outside the logic keywords

If you use allOf, anyOf, oneOf or not, every field they mention must also be described in the normal place. Those keywords may add conditions but may not introduce new structure.

3

Keep certain keywords out of the logic keywords

Inside allOf, anyOf, oneOf and not you may not set description, type, default, additionalProperties or nullable.

4

No references or definitions

OpenAPI lets you define a shape once and point to it elsewhere with $ref. CRDs do not allow this. Every shape must be written out fully where it is used.

5

A few keywords are banned outright

For example uniqueItems: true and additionalProperties: false are not allowed, because they make pruning and merging ambiguous.

6

Metadata is special

You may not describe the metadata block in detail. The API server owns it. At most you may restrict the object’s name or generated-name prefix.

Why so strict?

Each rule protects a feature. The type rule makes pruning possible, because you cannot decide whether a field is “unknown” if you do not know what is known. The no-reference rule keeps the schema fast and simple to walk through at high speed, which matters because the API server checks it on every single write. The ban on uniqueItems exists because checking uniqueness over big lists can be slow and expensive, so Kubernetes offers list-type markers and CEL as safer alternatives.

For the machines, the benefits are huge. A structural schema can be turned into accurate documentation, into client code, into a form in a web console, and into the OpenAPI document that kubectl downloads. In other words, the strict rules are what make the whole tool ecosystem work smoothly around your CRD.

ANTI-PATTERN 01Avoid
What people do

They copy a schema from a general OpenAPI project, complete with references, definitions and “one of these shapes” blocks, and paste it into a CRD.

Why it hurts

The API server refuses the CRD because it is not structural. Worse, if someone works around it by loosening the schema to “any object”, the CRD accepts anything, and you lose every benefit of validation.

Do this instead

Write the shapes out in full where they are used. Many teams generate their CRDs from typed code with a tool such as controller-gen, which flattens everything correctly and keeps the schema in step with the code.

!
Watch out

Very old CRDs created under the deprecated v1beta1 API could have non-structural schemas. Those CRDs had to be fixed before upgrading past version 1.21 of Kubernetes, because the v1beta1 API was removed in 1.22. If you inherit an old cluster, check for it.

6Pruning, Defaulting and Unknown Fields

Validation is not only about saying no. Because the schema is structural, the API server can also tidy up and fill in your objects automatically.

Pruning: trimming what the schema does not know

Pruning means that any field the schema does not describe is quietly removed from the object before it is stored. If your schema knows about replicas and a user sends replcas (a typo), the misspelled field is not part of the shape, so the API server drops it. The object is stored without it.

Why would Kubernetes do something that sounds so silent? Because stored data should always match a known shape. Without pruning, old and new versions of a controller could end up arguing over mystery fields that nobody documented, and a malicious or careless user could stash arbitrary data in your objects. Pruning keeps etcd clean and keeps the contract between the schema and the controller honest.

Pruning happens at two moments. It is applied to every incoming request, and it is also applied when objects are read back from etcd. The second part matters when you change the schema: if you remove a field from a version, the old data for that field disappears from what clients see immediately, and it is gone from storage the next time the object is written.

Everyday analogy

Pruning is a gardener trimming a hedge to a template. Anything that sticks out beyond the outline is cut. If the template says the hedge is a rectangle and a branch grows to the left, the branch is trimmed, no matter how healthy it was. The outline is the schema, and a stray branch is an unknown field.

The silent-typo trap and how to avoid it

Silent pruning is a double-edged sword. It protects storage, but it can hide a typo from the person who made it. To fix this, Kubernetes supports field validation on the server. A client can ask the server to be Strict (reject objects with unknown or duplicate fields), Warn (accept, but send back a warning message) or Ignore (prune silently). Current versions of kubectl ask for strict checking by default, so a typo normally produces a clear error. Other tools, such as a custom script, might not ask, so the safe habit is to make your own clients request strict validation too.

When you really do want unknown fields

Sometimes a field is meant to hold free-form data that you cannot describe. A plugin configuration blob is a classic example. For these cases the schema can add the marker x-kubernetes-preserve-unknown-fields set to true on that one node. Pruning then leaves everything under that node alone. Use it sparingly and on the smallest possible node, because inside that node you lose most of the checking.

Defaulting: filling in the blanks

Defaulting means that when a field is missing, the API server inserts the value you named in the default keyword. A user who leaves out replicas gets 1; a user who leaves out protocol gets “TCP”. The stored object then always contains an explicit value, so controllers never have to guess.

The built-in Deployment object is a familiar example. If you do not say how many copies to run, you get one. If you do not set an update strategy, you get a rolling update that replaces pods gradually, and the revision history kept for rollbacks is limited to a sensible number. Users enjoy short manifests, and the system enjoys complete records. Defaults on a CRD give your custom types the same comfort.

Defaulting runs at three moments: on the incoming request, after any mutating admission step that changed the object, and again when reading from etcd. The last one is very handy. If you add a new field with a default in a later version of your CRD, old objects that were stored without the field will appear to have it, without anyone rewriting them.

!
Common gotcha

A default for a nested field applies only if its parent exists. If spec itself is missing, then the defaults inside spec never run. The usual fix is to give the parent its own default of an empty object. Then the parent appears, and the defaults inside it fall into place.

What about null?

JSON allows an explicit null, which means “I deliberately set this to nothing.” In a CRD, a field that does not declare nullable: true treats null as if the field were absent: the null is pruned away, and then the default, if any, is applied. If you do want to allow null as a real value, mark the field nullable. Be careful with this choice, because many languages and tools handle null in surprising ways.

StepWhat it doesWhen it runsCan it reject?
PruningRemoves fields the schema does not know.On requests and when reading from etcd.Only if strict field validation is requested.
DefaultingAdds values for missing fields.On requests, after mutating admission, and when reading from etcd.No.
ValidationChecks types, limits, formats and CEL rules.On create and update.Yes, with a 422 error.
i
Tip

Default values are themselves checked against the schema when the CRD is created. A default that breaks its own field’s rules will make the CRD itself invalid, which is a good early warning.

7Kubernetes Extensions to OpenAPI

Plain OpenAPI cannot express a few things that Kubernetes needs. The x-kubernetes- keywords fill those gaps.

Flexible type

x-kubernetes-int-or-string

Lets a field hold either a whole number or text. Built-in objects use it for values like a port (a number or a named port) or a percentage (“25%” or 3).

Free-form

x-kubernetes-preserve-unknown-fields

Turns off pruning for one node so that arbitrary content under it is kept. Use for plugin blobs and nothing else.

Nested objects

x-kubernetes-embedded-resource

Says that a field holds a complete Kubernetes object. The server then checks that it has a valid API version, kind and metadata.

Merging

x-kubernetes-list-type

Explains how to treat a list when several people edit it: as one unit (atomic), as a set of unique values (set), or as a map keyed by a field (map).

Merging

x-kubernetes-list-map-keys

Names the field or fields that act as the key when a list is treated as a map, such as the container name.

Merging

x-kubernetes-map-type

Says whether an object can be edited field by field (granular) or must be replaced as one unit (atomic).

Rules

x-kubernetes-validations

Holds the CEL rules that check relationships between fields. This is the subject of the next chapter.

Int-or-string in action

Think of a form box that says “How much of the fleet can be offline during an update?” Some people want to answer “2 machines”, others want to answer “10 percent”. The int-or-string marker allows both answers. When you use it, you typically also add a pattern to limit the text form, so that only values such as “10%” are accepted and random words are not.

Lists as sets and maps: why merging matters

Kubernetes has a modern editing system called server-side apply. It lets many different tools and people each own different parts of the same object. For this to work, the server must understand how to merge changes to lists. A plain list is ambiguous: if two tools each add an entry, is that two entries or a clash?

The list-type marker removes the doubt. A list marked atomic is treated as one indivisible value, so whoever writes last replaces the whole list. A list marked set holds unique simple values, such as a list of finalizer names, and the server merges them like a bag of tokens. A list marked map holds objects that each have a key, such as the containers in a Pod, which are identified by name. The server merges them entry by entry using that key.

These markers also deliver validation as a bonus. For a set, the API server rejects duplicate values. For a map, it rejects two entries with the same key. This is the safe, built-in replacement for the banned uniqueItems keyword. When you choose the map style, the key fields must be either required or have defaults, so every entry can be identified.

Everyday analogy

Imagine a shared shopping list on a fridge. If the list is “atomic”, it is a single sheet: whoever writes last throws the old sheet away. If it is a “set”, it is a pile of unique tokens: add the same token twice and it is still one. If it is a “map”, every line starts with a name tag, so two housemates can each edit their own named line without fighting.

Production example: Pod containers

In every Kubernetes cluster, the list of containers inside a Pod is treated as a map keyed by container name. That is why two different tools, say a deployment pipeline and a sidecar injector, can each edit their own container without erasing the other’s work. Your own CRDs can use the same pattern for any list of named items.

Embedded resources

Some custom types carry a full Kubernetes object inside them. A typical case is a template, such as “when you create one of these, also create a Pod that looks like this.” Marking such a field as an embedded resource tells the server to confirm the inner object has the three standard identity fields: apiVersion, kind and metadata. It also prevents pruning of that identity and validates the metadata like any other object.

8CEL Validation Rules: Checking Fields Against Each Other

Schema keywords check one field at a time. Real-world rules usually compare fields. This is where the Common Expression Language shines.

The gap that CEL fills

Suppose your CRD has minReplicas and maxReplicas. Each is a number between 1 and 1000, and the plain keywords check that. But nothing in plain OpenAPI says “minimum must not be larger than maximum.” Another case: “if the storage type is set to disk, then a disk size is required.” These rules look across fields, and the standard schema keywords cannot say them.

For years, people solved this by running a separate program, an admission webhook, that the API server would call on every write. Webhooks work, but they are extra servers to deploy, secure, monitor and keep available. If the webhook is down, writes can fail for the whole cluster. CEL moves these rules into the CRD itself, so the API server runs them directly with no network call.

What is CEL?

The Common Expression Language is a tiny, fast language that Google designed for evaluating rules safely. It is used in many places outside Kubernetes, for example in network proxies and security-rule engines. CEL is deliberately limited. It has no loops that can run forever, no ability to write files or call the network, and every expression is guaranteed to finish. That makes it safe to run inside a shared, critical program like the API server.

A CEL expression reads a bit like a sentence in a spreadsheet formula: you compare values, combine conditions with “and” and “or”, test whether text starts with something, count items in a list, and so on. The expression must come out as true or false. True means “the object is fine”; false means “reject it with my message.”

How the rules are attached

Rules are written in a list called x-kubernetes-validations, placed on any node of the schema. Each entry in the list has these parts:

FieldPurpose
ruleThe CEL expression. It must evaluate to true for the object to be accepted.
messagePlain-text error shown when the rule fails.
messageExpressionOptional CEL expression that builds the error message, so it can include the bad value.
reasonMachine-readable category such as a forbidden value, an invalid value, a required value or a duplicate.
fieldPathOptional pointer to the exact field the error should be attached to, instead of the whole node.
optionalOldSelfLets a transition rule run even on create, when there is no old value.

Inside a rule, the name self means “the value at the node where this rule is placed.” If you put the rule on the spec node, then self is the whole spec, and the rule can reach into its fields. If you put it on a single field, self is just that field’s value. This is helpful because the closer the rule sits to the data, the clearer the error message.

Transition rules: remembering the past

CEL has one more trick. On an update, a rule can use oldSelf, which means “the value of this node before the change.” Rules that use it are called transition rules, because they judge the move from old to new. They are the standard way to build immutability. A rule that says “the new value must equal the old value” makes a field settable at creation and frozen forever after. Another rule might say “a status can only move forward, never backward”, or “the storage size may grow but never shrink.”

Transition rules only run on updates, because a create has no old object. If you want a rule to run on create too, with the old value treated as optional, that is what optionalOldSelf is for.

Everyday analogy

Plain schema keywords are a spell-checker: they notice a misspelled word. CEL rules are an editor reading the whole page: “You said the event ends at 3 pm but starts at 5 pm. That does not add up.” Transition rules are an editor with the previous draft in hand: “Last time you said the venue was fixed. Why did it change?”

Examples of rules, in words

Ordering

Min must not exceed max

Compare two number fields in the same object and refuse the object when the lower bound is larger than the upper bound.

Conditional

If A, then B is required

When the type field says “disk”, demand that the size field is present. Otherwise allow it to be absent.

At least one

Email or phone

Require that at least one of two optional contact fields is set, which plain required lists cannot say.

Immutability

Frozen after creation

A transition rule that rejects any update which changes the field’s old value.

Format

Name must end with a suffix

A string check that a name ends with a required label, giving a friendlier message than a long regular expression.

Size

List-wide limits

Rules over a whole list, such as “every port number in this list must be unique and below 65536.”

The cost budget: why unbounded fields cause errors

Because CEL runs inside the API server, Kubernetes must make sure no rule can eat too much time. When you create a CRD, the server estimates the worst-case cost of every rule before accepting it. The estimate depends on how big the data could possibly be. If a string has no maxLength, the server must assume it could be enormous, and if a list has no maxItems, it must assume it could hold a huge number of entries. A rule that loops over such a list can then be judged too expensive, and the CRD is rejected with an error saying the estimated rule cost exceeds the budget.

The cure is almost always the same: add sensible limits to the strings and lists the rule touches. This is a nice side effect, because limits are good practice anyway. There is also a separate runtime limit, so even if a rule passes the estimate, a request that makes it do too much work is stopped. And there is a total budget per CRD, so many expensive rules cannot add up to a problem.

Pros of CEL rules

  • No extra server to run, patch or secure.
  • Rules travel with the CRD, so they apply in every cluster that installs it.
  • Very fast, because the API server runs them in-process.
  • Safe by design: no loops without end, no outside access.
  • Errors show up right in kubectl output.

Cons of CEL rules

  • Can only see the object being checked, not other objects in the cluster.
  • Need bounds on strings and lists to pass the cost estimate.
  • A new language for the team to learn.
  • Complex rules are hard to read and test.
  • Cannot call outside services such as an inventory system.

Production example: Gateway API

The Gateway API project, which defines standard resources for routing traffic into Kubernetes clusters, moved many of its cross-field checks into CEL rules in its CRDs. This reduced the need to run a separate validating webhook for the common cases, so installing it became simpler and more reliable for the platform teams who adopt it.

i
Tip

Write the failure message as if speaking to a tired colleague at 2 a.m. “maxReplicas must be at least minReplicas” is useful. “Validation failed” is not.

9Internal Working: The Life of One Request

Let us follow a single request from your keyboard to the database, and see exactly where each check happens.

flowchart LR
  A["Client request"] --> B["Authentication"]
  B --> C["Authorization"]
  C --> D["Decode, prune, apply defaults"]
  D --> E["Mutating admission"]
  E --> F["Schema and CEL validation"]
  F --> G["Validating admission"]
  G --> H["Write to etcd"]
  H --> I["Response to client"]
  F -.->|invalid| X["422 error to client"]
  G -.->|denied| Y["Denied error to client"]
    
Figure 2: The path of a create or update request for a custom resource.
1

Authentication

The API server finds out who you are, using a certificate, a token or an identity provider. Without a valid identity the request stops here.

2

Authorization

The server asks whether you may create or update this type of object in this namespace. This is role-based access control at work. Having permission on Pods tells you nothing about permission on your custom type.

3

Decode, prune and default

The body is parsed. Unknown fields are pruned (or flagged, if strict field validation was requested) and missing fields get their defaults.

4

Mutating admission

Any mutating webhooks may change the object, for example to add a label or inject a sidecar. If they change it, defaulting runs again on the result.

5

Schema and CEL validation

The final shape of the object is checked against the OpenAPI schema and all CEL rules. This is the core of this article. Failure means a 422 error listing every problem.

6

Validating admission

Last-chance checks run: validating webhooks and ValidatingAdmissionPolicies. They see the final, defaulted, mutated object and can still say no.

7

Persist

The object is written to etcd, and watchers such as controllers are notified. Only now does the world change.

The order is not an accident. Validation happens after mutating admission so that it judges the final shape of the object, not an earlier draft. If validation ran first, a mutating webhook could then turn a valid object into an invalid one and nobody would notice. Validating webhooks run after schema validation so that they can assume a well-formed object and spend their effort on business rules.

How the schema is applied so quickly

The API server handles a very large number of writes in a busy cluster, so it cannot afford to re-read the CRD from scratch every time. When a CRD is created or changed, the apiextensions server compiles the schema into an internal validator and keeps it in memory. The same is done for CEL rules: each expression is parsed and checked once, and the compiled form is reused. A request then simply runs the prepared checks. This is why even complicated schemas add only a small delay to each request.

When a CRD itself is bad

The CRD is itself validated when you create it. If its schema is not structural, the object reports a condition named NonStructuralSchema in its status. Conditions are small status notes, a bit like a traffic-light panel on a dashboard. You will also see conditions such as NamesAccepted (are the names free and valid?) and Established (is the new API address live?). When something goes wrong, running kubectl describe on the CRD and reading these conditions usually points straight to the cause.

Everyday analogy

A package passes through a delivery depot. First the depot checks who is sending it and whether they are allowed to (authentication and authorization). Then staff tidy and label it (pruning and defaulting). Then special handling may add stickers (mutating admission). Then an inspector checks the contents against the shipping manifest (validation). Finally security scans it one last time (validating admission), and only then is it put on the truck (etcd).

10Data Flow and Lifecycle: Evolving Schemas Without Breaking People

A schema is not written once and forgotten. It must change as your product grows, and old objects are still sitting in storage.

Versions and conversion

A CRD can serve several versions at once, for example v1beta1 and v1. Clients can read and write whichever version they like, but the server stores each object in just one: the storage version. When a client asks for a version different from the stored one, the server converts the object on the fly.

Conversion comes in two flavours. With the default strategy, called None, the server simply changes the apiVersion label and leaves the fields alone. This works only if the shapes are identical. With the Webhook strategy, the server calls a conversion service you run, which translates field by field, such as turning a single host field into a list of hosts. Because a conversion webhook sits on the path of reads and writes, it must be extremely reliable, so many teams try hard to avoid needing one by making only additive changes.

Safe and unsafe schema changes

ChangeRiskWhy
Add an optional fieldLowOld objects stay valid. Add a default if controllers need a value.
Add a new enum valueLowExisting values still pass. Older controllers may not understand the new one, so update them first.
Loosen a limitLowEverything valid before is still valid.
Tighten a limit or patternMediumStored objects might now break the rules. Updates to them can start failing.
Add a required fieldHighEvery existing tool and stored object lacks it. Use a default or a new version.
Remove or rename a fieldHighPruning drops the old data. Clients still sending it lose information silently.
Change a field’s typeVery highOld stored values no longer fit. Needs a new version and conversion.

Validation ratcheting: forgiving the past

Imagine you tighten a rule: names must now be at most 30 characters, and some old objects have 40. Without special help, the next time anyone updates such an object, say to change an unrelated label, validation would fail, because the old name no longer passes. The user would be forced to fix a field they never touched.

Validation ratcheting solves this. A ratchet is a gear that turns one way only. On an update, if validation finds errors, the server compares the failing fields with the old object. If the failing parts have not changed, those errors are ignored, because the object was already in that state before. Only errors in fields the user actually changed cause a rejection. It became available as a beta feature turned on by default from Kubernetes 1.30, and it has since moved forward toward stable status. In practice it means schema authors can add stricter rules with much less fear of stranding existing objects.

flowchart TD
  A["Update request arrives"] --> B["Validate new object against schema"]
  B -->|no errors| OK["Accept and store"]
  B -->|errors found| C["Compare failing fields with the old object"]
  C -->|failing fields unchanged| OK
  C -->|failing fields changed| R["Reject with 422"]
    
Figure 3: How ratcheting decides whether an update with validation errors is still allowed.

Read the diagram from the top. A request arrives and the new object is validated. If there are no errors the update is accepted. If there are errors, the server looks at where they are. If they sit in parts that are identical in the old object, the update is allowed anyway. If the user changed one of the failing fields, the update is rejected. Ratcheting has a few limits: it generally does not forgive errors on lists that cannot be matched item by item, and it does not rescue a rule that was always broken for a field the user is now editing.

Migrating stored data

Changing the storage version only affects objects as they are written. Objects nobody touches stay in the old format in etcd. Over a long time, you may want to rewrite all of them, so that you can safely stop serving the old version. The common approach is to read and write every object once, using a storage version migration tool, and then remove the old version from the CRD’s list of stored versions. Until you do, the CRD keeps a record of every version that may still exist in storage, and the API server will refuse to drop a version still listed there.

!
Watch out

Never delete and recreate a CRD to “fix” its schema in a live cluster. Deleting a CRD deletes every custom resource of that type, and there is no undo. Update the CRD in place instead.

11Built-in Types, Published Schemas and Client-Side Checks

The schema is not only for the API server. It is also published, so every tool in your pipeline can use it.

Built-in resources versus CRDs

The built-in types, such as Pods, Services and Deployments, are validated by hand-written code in the API server. The rules for them are compiled into the program. Custom resources do not have that luxury; the API server has no custom code for them, so everything comes from the schema and the CEL rules you supply. From the user’s point of view the experience is meant to feel identical: the same error format, the same status codes and the same field-level messages.

Publishing: how kubectl learns your schema

The API server collects the schemas of all built-in types and all installed CRDs and serves them as a single OpenAPI document at well-known web addresses. This is the reason kubectl explain can show help for a type that was installed five minutes ago. It downloads the published document and prints the descriptions from your schema. Editors and IDE plugins use the same document to offer auto-complete and red underlines as people write manifests, which is validation before the request is even sent.

Three places to catch errors

Earliest

In the editor

Plugins that read the published schema highlight mistakes as you type. Cheap and instant, but advisory only.

Before merge

In your pipeline

Tools such as kubeconform check manifest files against schemas during a pull request. Failures block the merge and no cluster is touched.

Authoritative

On the server

The real API server applies the real rules. A server-side dry run sends your object through the whole path but stops before saving.

Dry run: rehearsing a request

A server-side dry run asks the API server to process the request fully, including pruning, defaulting, webhooks and validation, and then to return the result without storing anything. It is the most trustworthy way to test a manifest because it uses the true rules of the true cluster, including any extra webhooks that clients do not know about. A common pipeline step is to dry-run every changed manifest against a staging cluster before deploying to production.

Field validation modes at a glance

ModeUnknown or duplicate fieldsBest for
StrictRequest is rejected with an error.Pipelines and humans. Catches typos early.
WarnRequest succeeds, a warning is returned.Gradual rollout when old tooling still sends extra fields.
IgnoreUnknown fields are silently dropped.Legacy clients only. Hides mistakes.
i
Tip

If you publish a CRD for others to use, ship the schema in your documentation site too, and encourage people to run a pipeline check. The earlier the feedback loop, the fewer support questions you will receive.

12Beyond the Schema: Webhooks, Policies and Controllers

Schema validation is the first line of defence, not the only one. Knowing where it ends helps you choose the right tool for each rule.

A schema can only judge one object, in isolation, at the moment it is written. It cannot know that the name you picked is already used by something else, that your team has run out of its quota, or that the image you named is not on the approved list. For rules like these, Kubernetes offers more layers, and each layer trades power for cost.

flowchart TD
  A["Incoming object"] --> L1["Layer 1 Structural schema: shape and types"]
  L1 --> L2["Layer 2 CEL rules: relations between fields"]
  L2 --> L3["Layer 3 ValidatingAdmissionPolicy: cluster wide policy"]
  L3 --> L4["Layer 4 Admission webhooks: external logic"]
  L4 --> L5["Layer 5 Controller checks: runtime reality"]
    
Figure 4: Layers of defence, from cheapest and fastest at the top to most powerful and most costly at the bottom.

The diagram shows five layers. The first two are part of your CRD and travel with it. The third is a cluster-level policy that an administrator installs. The fourth is an outside program the API server calls over the network. The fifth is your own controller, which discovers problems that are only visible when it tries to act, such as “the cloud account you pointed me to does not exist.”

ValidatingAdmissionPolicy: CEL for the whole cluster

A ValidatingAdmissionPolicy, which became stable in Kubernetes 1.30, takes the idea of CEL rules and applies it to any resource, built-in or custom, from outside the CRD. A cluster administrator writes a policy such as “no workload may use the latest tag for its image,” then attaches it to a set of resources with a binding. The binding also decides what happens on a violation: deny the request, warn the user but allow it, or only record it in the audit log. Policies can read an extra parameter object, so one policy can be reused with different limits per team.

This is a perfect fit for organisation-wide rules owned by platform teams, while the CRD’s own CEL rules stay in the hands of the CRD author. The two do not conflict; both must pass.

Admission webhooks: the heavy hammer

A validating admission webhook is a small web service that the API server calls with the proposed object. The service replies with allow or deny. Because it is ordinary code, it can do anything: look up other objects, call an external database, check a company-wide naming registry. That freedom is also the cost. The service must be highly available, because a failing webhook can block writes; it must be secured with certificates; and it adds delay to each request. Teams therefore choose them only when schema, CEL and policies are not enough.

Policy engines

Projects such as Kyverno and OPA Gatekeeper package the webhook approach into ready-made policy engines. Teams write policies in a simpler language, and the engine handles the web service, reporting and audit. They are popular for compliance rules like “every namespace must have an owner label” or “only images from our registry.” They sit in layers three and four of the diagram.

LayerWho owns itCan see other objects?Extra servers?Best for
Schema keywordsCRD authorNoNoneTypes, limits, formats, required fields.
CEL in the CRDCRD authorNoNoneCross-field and immutability rules.
ValidatingAdmissionPolicyCluster adminOnly a parameter objectNoneOrganisation-wide guardrails.
Admission webhookPlatform or vendorYes, by any meansYesRules needing outside data.
ControllerCRD authorYesAlready existsRuntime conditions, reported in status.
Everyday analogy

Think of buying a house. The form you fill in has boxes with limits (schema). The clerk checks that the numbers add up (CEL). The bank applies its lending rules (policy). A lawyer checks records held elsewhere, such as who really owns the land (webhook). And only when you move in do you find out whether the roof leaks (controller). Each step catches different kinds of problems, and skipping the early ones makes the later ones more expensive.

i
Rule of thumb

Push each rule as far up the list as it can go. If a plain keyword can express it, do not write CEL. If CEL can express it, do not write a webhook. Every step down adds moving parts that can fail.

13Security, Monitoring, Logging and Metrics

Validation is a security feature and an operational signal at the same time. Treat it as both.

Schema as a security tool

Every limit you write is a small shield. A maxLength on a name stops someone from filling etcd with giant strings. A maxItems on a list stops a flood of entries. An enum stops free-form input where only a handful of values make sense. These limits matter because etcd has a cap on how big a single request can be (around 1.5 megabytes by default), and a very large object slows down every controller that watches it.

Regular expressions in a pattern are checked by Go’s standard engine, which runs in time proportional to the input and so avoids the “catastrophic” slowdowns some other engines suffer. Even so, keep patterns short and simple. Anything that needs a long pattern is often easier to read as a CEL rule.

Who may create a CRD?

Creating or editing a CRD is a very powerful permission. It can add new web addresses to the cluster’s API and, through conversion webhooks, route traffic to outside services. Grant it only to cluster administrators and trusted automation. Meanwhile, give users access to the custom resources themselves through ordinary role-based access control, with fine-grained rules for who may read, write or touch the status.

Validation is not authorization

A schema checks that a value is well formed, not that the caller is allowed to ask for it. A perfectly valid request to give an application cluster-admin rights is still dangerous. Always combine schema checks with roles, and use the status subresource so that only your controller can write the status.

Keep secrets out of custom resources

Custom resources are stored like any other object. Passwords and keys should live in Secret objects, and your CRD should hold only a reference by name. If you must keep sensitive data in a custom resource, make sure encryption at rest is set up for it on the cluster.

What to watch

Metrics

Request counts by status

The API server exports request totals labelled by result code. A sudden rise in 422 responses for one resource usually means a new client sending bad data, or a schema that changed.

Metrics

Request latency

Request duration histograms show if validation or webhooks have become slow. Compare custom resources with built-in ones as a baseline.

Metrics

Webhook rejections

Counters of admission webhook rejections reveal which webhook is saying no, and how often.

Logs

Audit logs

Audit entries record who sent each request and what status came back, so you can trace a rejected write to a person or pipeline.

Status

CRD conditions

Conditions such as NamesAccepted, Established and NonStructuralSchema show whether the definition itself is healthy.

Warnings

Client warnings

In warn mode, the server returns warning headers. Collecting them in your pipeline shows who is still sending deprecated fields before you switch to strict.

Good error messages are an operations tool

When validation fails at 3 a.m. during a deployment, the error message is the first and often only clue an engineer sees. A message that names the field, the bad value and the allowed range lets them fix the problem in a minute. A vague message turns a one-minute fix into a long search. Treat your message text as part of your product, and review it with the same care as the rules.

!
Watch out

Be careful with messages that echo the user’s input back. A message built from a value can leak data into logs. Avoid including anything sensitive in messageExpression.

14Design Patterns and Anti-Patterns

Experienced CRD authors keep returning to the same handful of good habits, and the same handful of traps.

Patterns worth copying

Pattern

Closed vocabularies

Use enum for any field with a fixed set of values. Add new values carefully and update controllers first.

Pattern

Spec, status and conditions

Keep desired state and observed state apart. Report health through a list of conditions with a type, status, reason and message.

Pattern

Immutability by rule

Use a transition rule to freeze fields that cannot safely change after creation, such as a storage class or a region.

Pattern

Reference by name

Point to other objects, such as a Secret, with a small name-and-namespace reference instead of copying their content.

Pattern

Defaults at every level

Give both the parent object and its child fields defaults so that short manifests produce complete records.

Pattern

Generate from types

Write your data types in code, mark them with validation comments, and generate the CRD from them. The schema and the code never drift apart.

Traps to avoid

ANTI-PATTERN 02Avoid
What people do

They mark the root of the schema as “preserve unknown fields” so that anything is accepted, usually because they are in a hurry and plan to tighten it later.

Why it hurts

“Later” rarely arrives. Users start relying on whatever they happened to send, and tightening the schema now breaks real objects. You also lose pruning, so typos are stored forever.

Do this instead

Start with a small, strict schema and widen it as real needs appear. Widening a rule is easy and safe; tightening it is painful.

ANTI-PATTERN 03Avoid
What people do

They write a webhook for every rule, including simple ones like “minimum must not exceed maximum.”

Why it hurts

Each webhook is one more service that can go down and block all writes to that resource. It also needs certificates, deployment, monitoring and upgrades.

Do this instead

Use keywords first, then CEL. Reserve webhooks for rules that truly need outside information.

ANTI-PATTERN 04Avoid
What people do

They skip schema validation and let the controller check everything after the object is stored, reporting errors in the status.

Why it hurts

The user gets no immediate feedback. Bad objects pile up in etcd, and every controller restart must re-examine them. Pipelines report success for objects that will never work.

Do this instead

Reject what you can at the door. Let the controller report only what it genuinely could not know in advance.

ANTI-PATTERN 05Avoid
What people do

They put a whole application’s configuration into one huge free-form field, such as a big block of text holding another file format.

Why it hurts

The API server cannot check, default, document or merge anything inside it. Errors are found only at runtime, and server-side apply cannot track who owns which part.

Do this instead

Model the settings as real fields with real types. A bit more upfront work gives you validation, documentation and clean merging for free.

15Best Practices and Common Mistakes

A short checklist you can keep next to your keyboard, followed by the errors that appear most often in real clusters.

Best practices

1

Bound everything

Give every string a maxLength, every list a maxItems, and every number a minimum and maximum. This protects storage and keeps CEL costs low.

2

Describe every field

Write helpful descriptions. They are your documentation inside kubectl explain.

3

Prefer additive change

Add optional fields with defaults. Reserve new versions for breaking changes, and keep old versions served until clients have moved.

4

Write friendly messages

Name the field and say what is expected. Use fieldPath to point to the exact box that is wrong.

5

Test on a real API server

Unit tests on your Go types do not run the real schema. Run integration tests against a real control plane, with both good and bad objects, to confirm that rules accept and reject what you expect.

6

Test upgrades, not only fresh installs

Create objects with the old CRD, apply the new CRD, then update the old objects. This is the only way to see ratcheting, defaulting and pruning behave as they will in production.

7

Roll out strictness in stages

For existing clients, begin with warnings, watch the logs, then switch to rejection once the warnings fade.

Common mistakes and quick fixes

MistakeWhat you seeFix
Using references or definitions in the schemaCRD rejected as non-structural.Write shapes out in full, or generate them.
No limits on strings or lists used by CELError that estimated rule cost exceeds the budget.Add maxLength and maxItems.
Default set on child but not parentDefaults never appear.Give the parent object a default of empty.
Field name typoSetting silently ignored.Use strict field validation in all clients.
Added a required field to a live versionOld automation suddenly fails.Use a default or introduce a new version.
Removed a field from the schemaStored values vanish on next write.Deprecate first, remove later, with a migration.
Forgot the status subresourceUsers can edit status, or spec edits wipe it.Enable the status subresource.
Very large CRD applied with the client-side methodError about an annotation being too long.Use server-side apply, which skips that annotation.
Deleted and recreated a CRDAll custom resources of that type are gone.Always update in place, and back up first.
?
About that annotation error

The older client-side apply method stores a copy of your manifest in an annotation, and annotations are limited to about 256 kilobytes in total. A big CRD can exceed that. Server-side apply tracks ownership inside the server instead, so it avoids the problem entirely.

Trade-offs at a glance

Advantages of schema validation

  • Mistakes caught at submit time with a clear message.
  • No custom code in the API server, and no extra services.
  • Free documentation, editor support and pipeline checks.
  • Pruning and defaulting keep stored data tidy.
  • Works the same way on every conformant cluster.

Disadvantages and limits

  • Strict structural rules limit what OpenAPI features you can use.
  • Cannot see other objects or outside systems.
  • Tightening rules later can strand old objects.
  • Silent pruning hides typos from careless clients.
  • Large schemas increase CRD size and review burden.

16Real-World and Industry Examples

The best way to see why this matters is to look at the popular projects that rely on it every day.

cert-manager: certificates as objects

cert-manager, a widely used tool for issuing and renewing security certificates, defines a Certificate type. The schema insists that each certificate names its issuer, lists the domain names, and gives durations in the expected format. A typo in the issuer reference shape is stopped at the door, so the failure appears in the engineer’s terminal instead of as an expired certificate on a Saturday night.

Prometheus Operator: telling monitoring what to watch

The Prometheus Operator introduced types such as ServiceMonitor, which describe how to find and scrape metrics from a group of services. Schema rules on the endpoints and selectors mean that a monitoring change that is malformed is rejected immediately, rather than quietly leaving a service unmonitored.

Crossplane and platform teams: safe, simple forms

Crossplane lets a platform team define a new type, for example “Database”, and it generates the matching CRD with a schema. The platform team can expose only a few safe choices, such as size small, medium or large, and hide the dozens of cloud settings behind it. Application developers get a short form, and the schema is what stops them from choosing something the company never intended to allow.

AWS Controllers for Kubernetes: cloud resources as custom resources

AWS Controllers for Kubernetes lets you manage AWS services, such as storage buckets or databases, by creating Kubernetes objects. The CRDs for these services are generated from the published API models of the AWS services, so the allowed fields, types and enumerated values in the schema mirror what the AWS service itself accepts. A bad value is rejected by the cluster before any cloud call is made. On a managed offering such as Amazon EKS, the API server is run for you, but schema behaviour is the standard upstream behaviour described in this article.

Gateway API: standard routing for everyone

As noted earlier, Gateway API ships rich CEL rules in its CRDs. Because the rules travel with the definitions, every cluster that installs the CRDs gets the same checks, regardless of which vendor’s controller handles the traffic. This is the real strength of schema-level validation: portable, shared rules.

The schema is the contract between the people who write objects and the programs that act on them.

A typical journey at a growing company

A small team begins with a single CRD for its internal deployment tool. At first the schema is loose, and engineers send whatever works. After a few incidents caused by typos, the team adds enums, limits and required fields. A migration is planned using defaults and ratcheting, so existing objects keep working. Later, a platform group introduces ValidatingAdmissionPolicies for organisation-wide rules such as naming and ownership labels. Only a few rules that need external data, such as checking a budget system, remain in a webhook. By then the schema, the policies and the controller each have a clear job, and incidents from malformed objects become rare.

17Frequently Asked Questions

Q1Is a schema required for a CRD?
In the stable v1 API, yes. Every version must have a structural schema. If you truly need to accept anything, you can mark the root as an object that preserves unknown fields, but this gives up most of the benefits described here.
Q2Does validation check the objects that already exist?
No. Validation runs only when an object is created or updated. Existing objects are not re-checked when you change the schema. They will meet the new rules the next time someone writes to them, which is why ratcheting and careful schema changes matter.
Q3Why did one of my fields disappear?
Almost certainly pruning. The field is missing from the schema for that version, perhaps because of a typo or because the field exists only in a different version. Check the spelling, the version, and whether the schema describes the field at that exact position.
Q4Can a rule check another object, such as whether a referenced Secret exists?
Not in the CRD. Schema rules and CEL in the CRD see only the object being written. Use a webhook, a policy engine, or let your controller check and report the result in the status.
Q5What is the difference between a 400 and a 422 error?
A 400 means the request itself was badly formed, for example broken JSON. A 422 means the request was understood but the object failed validation. Seeing many 422 responses usually points to schema or rule problems.
Q6Do CEL rules run on the status section?
Yes. Rules can be placed anywhere in the schema, including in the status. They run when the status is written, usually by the controller. Be careful, though, because a rule that rejects a status update can stop your own controller from reporting.
Q7Which Kubernetes version do I need for CEL rules?
They started as an alpha feature in 1.23, became beta and on by default in 1.25, and are stable from 1.29. Ratcheting arrived as beta in 1.30. On any recent managed cluster you can rely on both.
Q8Why does my CRD fail with a “rule cost exceeds budget” message?
The server assumes unbounded strings and lists can be huge. Add maxLength and maxItems to the fields the rule touches, and simplify the rule if it loops over large lists.
Q9Can one CRD reference a shared definition from another CRD?
No. References are not allowed, so shared shapes must be repeated in each schema. Most teams solve this by generating the schemas from shared code.
Q10What is the difference between a required field and a non-nullable field?
Required means the field must be present. Nullable means an explicit null is accepted as a value. A field can be optional but non-nullable, which means it may be absent, but sending null is treated as absent and may receive a default.
Q11How big can a custom resource be?
The practical limit comes from etcd, which restricts request size to around 1.5 megabytes by default. Very large objects also slow down watchers, so keep objects small and move bulky data to external storage with a reference.
Q12How do I test my schema before releasing?
Run a real API server in your tests, install the CRD, then create a set of valid and invalid objects and confirm the outcomes. Add upgrade tests with objects created under the old schema. Finally, use a server-side dry run in staging for real manifests.
Q13Do I still need webhooks now that CEL exists?
Sometimes. CEL covers most cross-field and immutability rules, and ValidatingAdmissionPolicy covers many cluster-wide rules. Webhooks remain useful when a rule needs data from outside the object, or when you need to mutate objects, which CEL validation cannot do.

18Summary and Key Takeaways

Schema validation is the printed order form of Kubernetes. It lets the API server say “no” quickly and clearly, before a mistake reaches storage and the controllers that trust it. The OpenAPI v3 schema describes the shape of each custom object, the strict structural rules make that description precise enough for the server to prune, default and check automatically, and CEL adds the cross-field and immutability rules that plain keywords cannot express. Around that core sit policies, webhooks and controllers, each handling the checks that only they can do.

Key Takeaways

  • The schema is the front-desk form. A CRD’s openAPIV3Schema tells the API server what each custom object may contain, and bad objects are rejected with a 422 error before they are stored.
  • Be structural. Every node needs a type, references are not allowed, and logic keywords may only add conditions. These rules are what make pruning, defaulting and CEL possible.
  • Know what the server does for you. Unknown fields are pruned, missing fields get defaults, and nested defaults need a default on the parent. Ask for strict field validation so typos become errors.
  • Use CEL for relationships. Rules with self compare fields, and rules with oldSelf freeze or guide changes. Add maxLength and maxItems so the cost estimate passes.
  • Change schemas gently. Add optional fields with defaults, avoid new required fields in live versions, rely on ratcheting for tightened rules, and never delete a CRD in a live cluster.
  • Choose the lowest layer that works. Keywords first, then CEL, then ValidatingAdmissionPolicy, then webhooks, then controller checks. Each step down adds moving parts.
  • Treat validation as security and operations. Limits protect etcd, status stays with the controller, and good error messages plus 422 metrics and audit logs help you find problems fast.