Before comparing the two doors, we need to understand why anyone wants to open a door. The answer lies in what Kubernetes really is.

Kubernetes is an API first

Most people meet Kubernetes as a tool that runs containers. But under the surface, Kubernetes is mainly a very well-designed API, a standard way for programs to ask for things and to learn what exists. Every action, from creating a pod to scaling a deployment, is just a request to this API. Every piece of cluster state is just an object stored behind it. Even the built-in components, such as the scheduler and the kubelet, are simply clients of the same API that you use.

This design has a powerful consequence. If you can teach the API a new kind of object, you gain a bundle of features for free: a consistent way to create, read, update and delete it, a way to watch for changes, permission control, audit logs, command-line tooling and an ecosystem of other tools that already know how to talk to the API. Instead of building a whole new management system, you extend the one that your teams already know.

The declarative idea

Kubernetes objects are declarative. You do not give step-by-step instructions. You write down what you want, such as “I want a database with three replicas”, and store it as an object. A program called a controller then watches the object and works to make reality match the description. This split between a stored wish and a tireless worker is the foundation of the operator pattern, and both CRDs and aggregated APIs exist to support it.

What teams build on top

Two official extension mechanisms

The Kubernetes project documents exactly two ways to add new API types that look and feel native. These are the subject of this guide.

To understand how to extend the API, we need a clear picture of what the API looks like and how requests move through it.

Resources, groups, versions and kinds

Every object type in Kubernetes has a few names that identify it. Learning these words makes everything else easier.

• A resource is a type of object that the API can serve, such as pods or deployments. It appears in web addresses in its plural form.
• A kind is the singular, formal name of the type, such as Pod or Deployment. It is the word you write at the top of a manifest.
• A group is a family that the resource belongs to, such as the apps group or the batch group. The oldest types live in a nameless “core” group. Groups prevent name clashes, so two projects can both define something called “Backup” without confusion.
• A version marks the maturity and shape of the type, such as v1beta1 or v1. Types can have several versions at once, so that old clients keep working while new ones adopt the improved shape.

Together, group, version and resource form the web address of a type. A custom resource gets its own address in the same way as a built-in one, which is why tools treat them equally.

Verbs: what you can do

The API is organized around a small set of actions called verbs: get, list, watch, create, update, patch and delete. Permissions are expressed in these terms. A role might say “may list and watch databases, but not delete them”. Because every type uses the same verbs, one permission system covers everything.

The watch superpower

The watch verb is what makes Kubernetes feel alive. Instead of asking repeatedly “has anything changed?”, a client opens a long-lived connection and the server pushes every change as it happens. Controllers depend on this. An extension that cannot support watching efficiently is a poor citizen, so both extension mechanisms must provide it.

Where the data lives

Kubernetes stores its state in a small, consistent key-value database called etcd. The API server is the only component that talks to etcd directly. Everything else asks the API server. The API server validates each request, applies rules, and then saves the object. When you create a custom resource using a CRD, it is saved here, right beside pods and secrets. When you use an aggregated API, you decide where your data lives.

flowchart LR
    U["kubectl, controllers, CI tools"] -->|"HTTPS requests"| API["Kubernetes API Server"]
    API --> A["Authentication - who are you?"]
    A --> Z["Authorization - are you allowed?"]
    Z --> AD["Admission - rules and defaults"]
    AD --> V["Validation of the object"]
    V --> E["etcd - the cluster database"]
    E -.->|"change events"| W["Watchers and controllers"]

The Kubernetes API server looks like one program. Inside, it is actually a chain of three, and that chain is the secret behind both extension mechanisms.

The delegation chain

The main Kubernetes API server binary is built from three cooperating servers stacked in a line. A request enters at the front and, if the first server does not own the answer, it is handed down the line.

flowchart TB
    REQ["Incoming API request"] --> AGG["Aggregator - front desk"]
    AGG -->|"type owned by an APIService"| EXT["External extension API server"]
    AGG -->|"not owned - delegate"| CORE["Core API server - pods, services, nodes"]
    CORE -->|"not a built-in type - delegate"| CRDS["API extensions server - custom resources from CRDs"]
    CRDS -->|"unknown - delegate"| NF["Not found"]
    CORE --> ETCD["etcd"]
    CRDS --> ETCD
    EXT --> OWN["Its own storage - your choice"]

What this tells us

This picture already reveals the heart of the comparison.

• With a CRD, your new type is served by the third link in the chain, a part of the Kubernetes API server itself. You write no server code. You only describe the type, and the existing machinery does the rest.
• With an aggregated API, you write and run a completely separate API server. You tell the aggregator at the front desk to forward certain requests to it. You gain full control, and you take on full responsibility.

Many teams reach for a new API type when something far simpler would do. A good architect asks this question first.

A ladder of options

Think of extension options as steps on a ladder, from lightest to heaviest. Always start at the bottom and climb only as far as you must.

Signs you need a real API type

• You want users to manage the thing with the same tools as other Kubernetes objects.
• You want permissions to be controlled per object type, and per namespace.
• You want to watch for changes and react automatically.
• You want the object to appear in listings, with status that shows its health.
• You want a controller to turn declared intent into real-world actions.

Signs you do not

• Your program just reads a few settings at startup. Use a ConfigMap.
• The data is large, changes constantly or is queried in complex ways. A real database is better, and an API object can just point to it.
• Only one team and one tool will ever use the data, and nobody needs Kubernetes permissions or watches.
• You are building a general-purpose business application. Kubernetes objects are best for describing infrastructure and platform intent, not for storing customer records.

A Custom Resource Definition is the lightest and most popular way to extend Kubernetes. It is a description of a new type, written as a normal Kubernetes object.

The two words, carefully

People often mix up two similar terms. The CustomResourceDefinition, or CRD, is the blueprint: a Kubernetes object that describes a new type. A custom resource, or CR, is an actual instance of that type. The CRD is like the design of a form. A custom resource is a filled-in copy of that form. Once you apply a CRD, the API server immediately starts accepting custom resources of that type.

What a CRD describes

Why the schema matters so much

In the current stable API for CRDs, every custom resource type must have a structural schema. This is a strict, well-defined way of describing fields so that the API server can understand every part of the object. The benefits are large:

• Validation. A malformed object is rejected immediately with a clear message, instead of failing mysteriously inside a controller hours later.
• Pruning. Fields that are not in the schema are removed before saving, so typos cannot silently sneak into the database.
• Defaulting. Missing fields can be filled in automatically with sensible values.
• Documentation. Tools can read the schema to show help text and offer autocompletion.

Rules written as expressions

Beyond basic types, modern Kubernetes lets CRD authors write validation rules using the Common Expression Language, known as CEL. These are small, safe expressions that run inside the API server. A rule can say “the maximum must be greater than or equal to the minimum”, or “this field cannot be changed after creation”. Because these rules run inside the server, you often do not need to build and run a separate validation web service, which removes a major source of complexity and failure.

The magic of CRDs is that no restart and no new program is needed. Let us watch what happens inside the server when a CRD is applied.

From CRD to working API

sequenceDiagram
    participant U as User
    participant A as API Server
    participant X as Extensions server inside
    participant E as etcd
    U->>A: Create CRD
    A->>E: Save CRD
    E-->>X: CRD change event
    X->>X: Build REST handlers and schema
    X-->>A: Type is Established
    U->>A: Create custom resource
    A->>A: Authenticate, authorize, admit
    A->>X: Validate, default and prune with schema
    X->>E: Save custom resource
    E-->>U: Success and watch events

Features you receive for free

• Status subresource. A separate section for what the controller observes, kept apart from what the user wants. Users edit the wish, the controller edits the status, and neither can overwrite the other by accident.
• Scale subresource. A standard interface that lets tools like the Horizontal Pod Autoscaler scale your custom type just like a deployment.
• Printer columns. Extra columns in listings, so a database object can show its version and readiness at a glance.
• Finalizers. Markers that stop an object from being fully deleted until a controller has cleaned up the outside resources it created.
• Owner references and garbage collection. A parent object can own children, so deleting the parent automatically cleans up everything it created.
• Watch with caching. The server supports efficient change streams out of the box.
• Permissions, auditing and tooling. Role-based access control, audit logs, kubectl and GitOps tools all work immediately.

Every API changes over time. A CRD must support old and new shapes at the same time, and this is where real planning begins.

Why versions exist

Imagine your database type starts with a simple field for the size. Later you realize you need to specify the storage and the compute separately. You cannot just rename fields, because thousands of stored objects and many tools use the old shape. Instead, you add a new version of the type, and keep the old one working until everyone has moved.

Served versus stored

A CRD can list several versions. Two flags control their roles. A version marked served can be used by clients. Exactly one version is marked storage, and this is the shape in which objects are written to etcd. When a client uses a different version than the storage version, the API server must convert the object on the way in and out.

Two conversion strategies

Moving stored data to a new storage version

Changing the storage version does not automatically rewrite old objects. They stay in etcd in the old shape until they are next written. Before you can safely stop serving an old version, you must make sure no stored object still uses it. A storage version migration, which simply reads and re-writes every object, takes care of this. The project provides tooling for such migrations, and many operators run their own.

A CRD on its own does nothing except store data. The real value appears when a controller reads the objects and acts on them.

What a controller does

A controller is a program, usually running as a pod in the cluster, that watches custom resources and makes the real world match what they describe. It follows the reconciliation loop you may know from other Kubernetes parts: observe the desired state, observe the actual state, take the smallest action that moves them closer, then repeat. A controller bundled with a custom resource type, designed to manage one application or system, is called an operator. The word suggests a human operator whose knowledge has been captured in software.

The reconciliation cycle

flowchart TB
    W["Watch custom resources"] --> Q["Work queue of changed objects"]
    Q --> R["Reconcile one object"]
    R --> O["Read desired spec and actual world"]
    O --> D{"Do they match?"}
    D -->|"yes"| S["Update status: healthy"]
    D -->|"no"| A["Create, update or delete child resources"]
    A --> S2["Update status: progressing"]
    S --> W
    S2 --> W

Good controller habits

• Idempotence. Running the same reconcile twice must be safe. Controllers can be restarted at any time and will see the same objects again.
• Level-based thinking. React to the current state, not to a list of past events. Events can be missed, but the state is always there to read.
• Owner references. Mark everything the controller creates as owned by the custom resource, so cleanup is automatic.
• Finalizers. Use them to clean up outside resources, such as cloud disks, before the object disappears.
• Status conditions. Report progress in a standard way that humans and tools can read.
• Leader election. Run several copies for availability, with only one active at a time.

Controllers work with both mechanisms

Notice that a controller is independent of how the API type is served. The same controller pattern works whether the objects come from a CRD or from an aggregated server, because it simply talks to the Kubernetes API. The choice between the two doors affects how the objects are stored and served, not how controllers behave. Many operator frameworks make building controllers much easier, and they are written mostly with CRDs in mind.

The second door is heavier but much more flexible. Instead of describing a type to the existing server, you bring your own server.

The big idea

With API aggregation, you write and run your own API server, called an extension API server. It could be a pod inside the cluster or even something running elsewhere. You then register it with the main API server by creating a small object called an APIService. This object says, in effect: “Any request for group X and version Y should be forwarded to this service.”

From then on, the main API server acts as a front door and a bouncer. It checks who the caller is and whether they are allowed to proceed, and then passes the request along to your server. Your server answers, and the reply travels back through the front door to the client. To the user, it feels like one single Kubernetes API.

What an APIService contains

Availability is reported

The APIService object also carries a status. The main API server regularly checks that the extension server is reachable and healthy, and records an Available condition. If it is not available, requests for that address fail with an error saying the service is unavailable. This honest reporting is helpful, but it also reveals a risk: the extension server is now on the critical path for its part of the API.

Real examples you may already use

• Metrics Server registers the metrics group so that kubectl top and the Horizontal Pod Autoscaler can ask for live CPU and memory. The data comes from memory, not from the database.
• Custom metrics adapters serve the custom and external metrics groups, translating questions into queries against monitoring systems.
• KubeVirt, which runs virtual machines on Kubernetes, uses CRDs for its main objects but also adds an aggregated API for special actions such as opening a console to a running machine.
• Service Catalog, an earlier project for provisioning external services, was built as an aggregated API server. It has since been retired, a reminder of the long-term cost of this approach.

The trickiest part of aggregation is trust. The extension server must trust the main API server, and the main API server must trust the extension server. Let us see how this is arranged.

The journey of one request

sequenceDiagram
    participant C as Client
    participant M as Main API Server with aggregator
    participant X as Extension API Server
    participant S as Its own storage or data source
    C->>M: Request for aggregated type
    M->>M: Authenticate and authorize caller
    M->>X: Proxy request with front-proxy certificate and identity headers
    X->>X: Verify proxy certificate and read identity
    X->>M: Ask permission check for this user
    M-->>X: Allowed
    X->>S: Read or write
    S-->>X: Data
    X-->>M: Response
    M-->>C: Response

The trust machinery

• Front-proxy client certificate. The main API server presents a special certificate when it forwards requests. The extension server accepts requests only from a caller holding such a certificate signed by the expected authority.
• Request header authentication. The identity of the original caller is carried in headers, such as the user name and group names. Because only the trusted front door can set them, the extension server can believe them.
• Published settings. The main API server publishes the details that extension servers need, such as the certificate authority and the header names, in a well-known configuration object in the system namespace. Extension servers read it on startup.
• Delegated authorization. Rather than maintaining its own permission rules, the extension server asks the main API server to check whether this user may perform this action. This keeps all permission logic in one place, and means ordinary Kubernetes roles work for aggregated types too.
• CA bundle in the APIService. This lets the main API server verify the extension server’s own certificate in the other direction.

With a CRD, you write a description. With aggregation, you write a real server. It helps to see the long list of things that become your job.

Responsibilities you inherit

Tools that help

You are not starting from nothing. The Kubernetes project publishes a generic API server library that contains most of the machinery, including request handling, authentication, authorization, watch and storage on etcd. A sample extension server in the project shows how to put the pieces together, and community frameworks exist to generate boilerplate. Still, even with these tools, you take on a project of significant size, written in the same language and style as Kubernetes itself, and you must keep it up to date with every Kubernetes release.

Effort at a glance

Now that you know both mechanisms from the inside, here is a direct comparison across the dimensions that matter in practice.

flowchart LR
    subgraph CRD["CRD path"]
        C1["Client"] --> C2["API Server"]
        C2 --> C3["Built-in extensions server"]
        C3 --> C4["Cluster etcd"]
    end
    subgraph AGG["Aggregated path"]
        A1["Client"] --> A2["API Server with aggregator"]
        A2 --> A3["Your extension API server"]
        A3 --> A4["Your own storage or live data"]
    end

The short version

A CRD gives you a free, fast, inherited API with limits on what it can do. An aggregated API gives you total freedom with total responsibility. In nearly every case, the free option is enough, and the freedom of the second option goes unused.

There are real situations where a CRD simply cannot do the job. Knowing them helps you avoid both over-engineering and under-engineering.

1. Data that does not live in etcd

A CRD always stores objects in the cluster’s etcd. If your data is huge, changes many times per second, already lives in another system, or is computed from live sources, putting it into etcd is unwise or impossible. An aggregated API can read from a database, a monitoring system or even memory. Metrics Server is the classic case: every number is fresh, short-lived and never saved.

2. Custom actions on objects

Kubernetes pods have special actions like opening a shell, streaming logs and forwarding ports. These are not plain create-or-update operations. They are subresources with their own behavior. CRDs allow only two subresources, status and scale. If you need an action such as “open a console to this virtual machine” or “stream the output of this job”, an aggregated server can provide it. This is why KubeVirt, which mostly uses CRDs, still runs an aggregated server for its console and similar actions.

3. Different storage rules

Perhaps you need strong history, data in a relational database, special indexes, or storage with a different consistency model. Or perhaps you need an object that exists only virtually, without being saved at all. Aggregation lets you design this.

4. Complex or non-standard logic on writes

CRDs support schema rules, expression rules and, with extra webhooks, custom checks. An aggregated server can perform any logic inside the request itself: look up outside systems, generate values, or reject changes based on arbitrary conditions, without separate webhook calls.

5. Performance at a different scale

If you must serve a huge number of objects, or huge individual objects, you may need your own storage, caches and indexes tuned to your workload, rather than sharing the cluster database.

6. Managing a different system as if it were Kubernetes

Sometimes a team wants Kubernetes-style access, tooling and permissions in front of a completely different system. An aggregated server can present such systems through familiar API conventions, using its own backing data.

The reason CRDs dominate real-world use is not only that they are simple. It is how much comes bundled with them without any effort.

The free bundle

The ecosystem effect

Because custom resources look like ordinary Kubernetes objects, the entire world of Kubernetes tooling treats them as first-class. Configuration management and GitOps tools can apply them from a repository. Policy engines can validate them. Dashboards can display them. Backup tools can include them. Command-line completion can suggest their fields. A CRD plugs into all of this on day one.

This is why so many well-known projects are built primarily on CRDs.

• cert-manager defines certificates and issuers as custom resources, and a controller keeps certificates valid.
• Prometheus Operator defines monitoring servers and scrape rules as custom resources.
• Argo CD and Argo Workflows describe applications and pipelines as custom resources.
• Istio and the Gateway API describe traffic routing and gateways as custom resources.
• Cluster API describes entire Kubernetes clusters as custom resources, which controllers create and manage.
• Crossplane describes cloud resources and platform building blocks as custom resources.
• Knative describes serverless services and revisions as custom resources.

Limits you should expect

• You cannot choose where or how data is stored.
• You are limited to the standard set of actions, plus the status and scale subresources.
• Complex checks that need outside information require admission webhooks, which are separate services that add moving parts.
• Object size and count are bounded by what etcd and the API server comfortably handle.
• Very complex version changes require a conversion webhook, with its own care and risk.

Both mechanisms can scale well, but each has its own pressure points. Knowing them keeps surprises away.

The figures above are the default limits of a standard etcd setup, and managed Kubernetes services may configure them differently. Treat them as signs of scale, not guarantees.

Pressure points with CRDs

• Shared database. Custom resources share etcd with everything else. A controller that writes constantly, or objects that are large, can slow down or even endanger the entire cluster. If etcd hits its storage quota, writes stop for everyone.
• Object size. Very large objects, such as those carrying embedded files or long lists, are heavy to store, transfer and cache. Keep objects small and store bulk data elsewhere.
• Object count. Hundreds of thousands of objects of one type can strain listing and caching in the API server, which holds recent data in memory for fast watches.
• Number of CRDs. Each CRD adds handlers, schemas and cached discovery data to the API server. Very large numbers of CRDs can increase memory use and slow discovery for every client.
• Write frequency. Controllers that update status many times per second create heavy write load and heavy watch traffic. Batch updates and avoid pointless writes.

Pressure points with aggregated APIs

• The extra hop. Every request passes through the main API server and is then proxied to your server. This adds a little latency and some load to the front door, which is rarely a problem but should be measured for very busy APIs.
• Your own caching and watch. If your server does not cache and manage watches as efficiently as the main one, many controllers watching your types can overload it.
• Your own storage. You are now responsible for the scalability of whatever holds your data.
• Connection handling. Long-lived watch connections and streams all pass through the proxy and need to be handled carefully, including timeouts and reconnection.

flowchart TB
    subgraph SHARED["CRD - shared resources"]
        CR["Many custom resources"] --> ETCD["Cluster etcd"]
        PODS["Pods, secrets, nodes"] --> ETCD
        ETCD --> RISK["One busy neighbor can slow everyone"]
    end
    subgraph ISO["Aggregated - isolated resources"]
        AR["Your objects"] --> OWN["Your own storage"]
        OWN --> SAFE["Cluster etcd is protected, but you operate more systems"]
    end

Every design should be examined by asking one question: what breaks when this part fails? The two mechanisms fail in very different ways.

When an aggregated server goes down

If your extension server is unreachable, requests for its types fail. That is expected. But there are side effects that surprise many teams.

• Discovery errors. Tools that list all available APIs may print warnings or even fail, because one group could not be reached. Users see confusing messages about being unable to retrieve the complete list of server APIs.
• Stuck namespace deletion. When a namespace is deleted, Kubernetes must look through every type to remove objects in it. If one aggregated API is unavailable, the check cannot complete and the namespace may remain in a terminating state indefinitely.
• Controller trouble. Controllers and tools that scan all types, such as garbage collection and backup tools, may stall or log errors.
• Autoscaler impact. If the unavailable server is the metrics provider, the autoscalers that depend on it cannot make decisions.

When something fails with a CRD

• The CRD is deleted. All its custom resources are deleted with it. This is the most dangerous CRD mistake.
• The conversion webhook is down. Reads and writes that need conversion between versions can fail, which can break controllers and tools for that type.
• An admission webhook is down. Depending on its failure setting, writes are either blocked or silently unchecked.
• The controller is down. The API still works, and objects can still be created, but nothing acts on them. Status goes stale, and users are left waiting. This is a softer failure than a missing API.
• A bad schema change. A stricter schema can cause existing stored objects to become invalid, which makes updating them fail until they are fixed.

Protecting yourself

• Run extension servers with multiple replicas, spread across zones, with proper health checks and disruption budgets.
• Monitor APIService availability and alert quickly when it flips.
• Automate the rotation of certificates well before they expire.
• Keep conversion and admission webhooks as few and as simple as possible, and design their failure policy on purpose.
• Treat CRDs as precious data: limit who can delete them, and back up both CRDs and their instances.
• Test the failure of every component in a staging environment before relying on it.

Both mechanisms widen the cluster’s attack surface. The question is who carries the burden of protecting that surface.

Shared protections

• Role-based access control. Permissions work on custom types just as on built-in ones. You can grant read-only access to a type, restrict creation to a team, or limit access to specific namespaces.
• Audit logging. Both kinds of types appear in the cluster audit trail, so you can see who created or changed what.
• Admission policies. Cluster-wide policies, whether written in CEL or run by policy engines, can examine custom objects.

CRD-specific points

Aggregation-specific points

Practical hardening

• Use network policies so that only the main API server and the necessary clients can reach the extension server.
• Run it with the least privileges possible, as a non-root user, with a read-only file system where feasible.
• Always rely on delegated authorization so that permissions stay defined in one place.
• Never use the option to skip certificate checks.
• Scan and sign the images of extension servers and operators, and verify them at admission.
• Encrypt sensitive fields at rest. Remember that secrets must not be placed in plain custom resource fields; reference a Secret instead.

When an extension misbehaves, you need to know where to look. Both kinds expose clear signals if you know their names.

What to look at

A troubleshooting walk-through

Here is a simple method for choosing between the two doors. Ask the questions in order, and stop at the first clear answer.

flowchart TB
    S["New API need"] --> Q1{"Is a ConfigMap or plain settings enough?"}
    Q1 -->|"yes"| CM["Use a ConfigMap"]
    Q1 -->|"no"| Q2{"Must data live outside etcd or be computed live?"}
    Q2 -->|"yes"| AGG["Use an aggregated API"]
    Q2 -->|"no"| Q3{"Need custom verbs or subresources beyond status and scale?"}
    Q3 -->|"yes, essential"| HYB["CRD for objects plus aggregated API for the special actions"]
    Q3 -->|"no"| Q4{"Can schema, CEL rules and webhooks express your validation?"}
    Q4 -->|"no"| AGG
    Q4 -->|"yes"| Q5{"Small objects, moderate counts, modest write rate?"}
    Q5 -->|"no"| AGG
    Q5 -->|"yes"| CRD["Use a CRD with a controller"]

The questions in words

• Is a plain setting enough? If yes, stop. Use a ConfigMap and avoid a new API.
• Must the data live outside etcd, or be computed live? Metrics, giant datasets and external records say yes. That points to aggregation.
• Do you truly need special actions? Consoles, streams and other custom verbs cannot be done by a CRD. A hybrid might serve you best.
• Can your validation rules be written as schema and expressions? If your rules need unusual logic inside the request, aggregation gives the freedom.
• Is the scale reasonable for shared storage? Huge or hot data harms the cluster database. If your numbers are high, move away from etcd.
• Do you have a team to run a server forever? If not, that alone is a reason to pick a CRD and live with its limits.

Scoring your situation

Experienced platform teams reuse a small set of patterns. They also share a list of traps they once fell into.

Patterns that work

Anti-patterns to avoid

Here is a checklist to take into your next design review.

Best practices for CRDs

• Own your group name. Use a domain you control, so that names never clash with other projects.
• Write a strict structural schema with types, limits, defaults and descriptions for every field.
• Use expression rules for cross-field checks, rather than adding a webhook that needs to be run and kept alive.
• Enable the status subresource, and keep user intent and observed state apart.
• Add printer columns so that a simple listing tells users what they need to know.
• Plan versions early. Start with an alpha version, use conversion only when needed and migrate stored data before removing old versions.
• Protect the CRD. Tight permissions for create, edit and delete, and backups of both the CRD and its instances.
• Prefer namespaced scope unless the thing is genuinely cluster-wide.

Best practices for aggregated APIs

• Justify it in writing. Document the capability that a CRD cannot provide, and revisit it at each major Kubernetes release as CRDs gain features.
• Use the official libraries rather than inventing your own handling of watch, versions and authentication.
• Delegate authentication and authorization to the main API server.
• Verify certificates in both directions and automate their renewal.
• Run it highly available and monitor the APIService condition.
• Plan the data layer for backups, encryption, scaling and recovery.
• Keep up with upgrades. Assign a team to follow Kubernetes releases and update the server’s dependencies regularly.

Common mistakes

• Reaching for aggregation because it feels more serious, when a CRD would do.
• Treating the cluster database as an application database.
• Putting secrets directly into custom resource fields.
• Deleting a CRD without realizing that all its objects disappear.
• Forgetting finalizers, leaving outside resources such as cloud disks behind.
• Leaving finalizers on objects after the controller has been removed, which makes them impossible to delete.
• Building conversion that loses information, so objects change when read through another version.
• Skipping alerts on APIService availability and controller health.
• Making many tiny CRDs with overlapping meaning, instead of a clear small set.

The balance of use in the wild shows how the guidance plays out in practice: a great many CRDs, and a small number of carefully justified aggregated servers.

Quick answers to the questions that come up most often.