Before any detail, let us build a clear picture of the idea in one sitting.

Kubernetes is controlled entirely through an API. An API (Application Programming Interface) is simply a menu of requests that a program is allowed to make, such as “show me all Pods” or “create a Service”. In Kubernetes, that menu is served over HTTPS by a component called the kube-apiserver. Whether you use the kubectl command line, a dashboard, a CI pipeline or an internal controller, they all talk to this one server.

Out of the box, the kube-apiserver knows about the built-in resource types: Pods, Services, Deployments, Nodes, Secrets and so on. But real companies need more. They want to ask Kubernetes for “the current CPU use of every Pod”, or “a virtual cluster”, or “a database backup”. Those things are not built in.

The API aggregation layer is the part of the kube-apiserver that lets you plug in your own separate API servers and make them appear as if they were part of Kubernetes itself. A user keeps using the same address, the same login and the same kubectl. Behind the scenes, the kube-apiserver quietly forwards the request to your extra server, waits for the answer, and passes it back.

The one-line definition

The aggregation layer lets Kubernetes serve extra API groups from separate, independently run servers, while presenting them to users under one front door with one set of login rules.

Where the name comes from

The word “aggregation” means gathering many pieces into one. Several different API servers, each owning its own group of resources, are gathered together into one combined API surface. The internal component that does the gathering is called kube-aggregator, and in modern clusters it runs inside the same process as the kube-apiserver. You do not install it separately.

Kubernetes was designed from day one to be extended. Its creators knew that no single project could ever predict every resource that people would want to manage. So they built several extension points. The aggregation layer is one of the two main ways to add new resource types to the API. The other is the CustomResourceDefinition, usually called a CRD, which we compare in detail later.

What life looked like before

In the early days, if you wanted a new kind of resource, you had two bad choices. You could change the Kubernetes source code itself, rebuild the whole API server and run your own fork. That meant every upgrade became a painful merge exercise. Or you could build a completely separate tool with its own address, its own login and its own client, and ask your users to learn yet another interface.

The needs that pushed the design

A real example that made it famous

The best-known case is resource metrics. People wanted the cluster to answer “how much CPU and memory is each Pod using right now?” That data is not stored as a normal object. It is measured live on every node and changes every few seconds. Storing it as ordinary records in the main database would be wasteful and slow. Instead, a small server called metrics-server gathers the numbers, keeps them in memory, and answers through the aggregation layer under the group metrics.k8s.io. Tools like kubectl top and the Horizontal Pod Autoscaler then read it as if it were a native part of Kubernetes.

Six terms carry almost the whole topic. Learn them once and the rest becomes easy.

API group and version

Every resource in Kubernetes lives in an API group and carries a version. Think of the group as a surname and the version as an edition number. Deployments belong to the group apps at version v1. The metrics data belongs to the group metrics.k8s.io at version v1beta1. Groups keep names from colliding, and versions let an API evolve (for example from alpha to beta to stable) without breaking old clients.

Extension API server

An extension API server is a normal program, usually running as Pods inside the cluster, that speaks the Kubernetes API conventions. It owns one or more API groups. It knows how to list, get, create, update, delete and watch its own resource types. It is also called an aggregated API server.

APIService object

The APIService is a small Kubernetes object that acts like an entry in the mall directory. It belongs to the group apiregistration.k8s.io. It says: “the group G at version V is served by the Service named S in namespace N.” Creating an APIService is the single act that tells the aggregator to start forwarding requests.

The aggregator

The aggregator is the code inside the kube-apiserver that reads APIService objects, keeps a live routing table, and proxies matching requests. It also checks whether each extension server is healthy and records the result on the APIService.

Delegated authentication and authorization

The extension server does not keep its own list of users. Instead it delegates the questions “who is this person?” and “may they do this?” back to the main API server. This is how one login system covers every API. We cover the exact flow in chapter 6.

Local versus remote APIService

Built-in groups such as apps and batch are also registered as APIService objects, but with no service reference. These are called local entries because the same process answers them. Only entries with a service reference are remote and get proxied. This detail explains why a fresh cluster already shows many APIService objects before you add anything.

The kube-apiserver is not one block. It is a small chain of servers, each passing along what it cannot handle.

Inside the control plane process there are three API servers joined together in a chain, a design pattern known as delegation. Each server tries to handle a request. If the path does not belong to it, it hands the request to the next server in the chain. The last link returns a “not found” answer.

flowchart TB
  C["Client: kubectl or a controller"] --> LB["Load Balancer in front of control plane"]
  LB --> AGG["Aggregator Server"]
  AGG -->|"built-in groups"| KAS["Core Kube API Server"]
  KAS -->|"path not known"| CRD["API Extensions Server for CRDs"]
  CRD -->|"still not known"| NF["Not Found Handler"]
  AGG -->|"remote APIService matched"| PX["Proxy Handler"]
  PX --> SVC["Kubernetes Service"]
  SVC --> E1["Extension Server Pod A"]
  SVC --> E2["Extension Server Pod B"]
  KAS --> ETCD[("Cluster etcd")]
  CRD --> ETCD
  E1 --> ST[("Extension Storage or Memory")]
  E2 --> ST
    

Reading the picture step by step

The request enters from the top. The aggregator server sits at the front of the chain because it must see every request first. It looks at the URL path, such as /apis/metrics.k8s.io/v1beta1, and checks its routing table of APIService objects.

• If the path matches a remote entry, the proxy handler forwards the request to the extension server.
• If the path matches a local entry, the request is delegated to the core kube-apiserver, which serves built-in resources from etcd.
• If the core server does not know the path, it delegates to the API extensions server, which serves resources defined by CRDs.
• If nobody knows the path, the final handler answers with “404 not found”.

The main building blocks

How the aggregator reaches your Pods

The control plane often runs on nodes where the usual in-cluster networking helper, kube-proxy, is absent. In that case the cluster Service address may not be reachable from the aggregator. A setting called --enable-aggregator-routing tells the aggregator to resolve the Service to its endpoint addresses and connect to Pod addresses directly. This is a common reason an otherwise correct setup reports “unavailable”, and we revisit it in the troubleshooting chapter.

Follow a single request from the moment you press Enter until the answer appears on your screen.

Suppose a platform engineer runs kubectl top pods. Behind that short command, a precise relay race takes place. The diagram below shows every runner in order.

sequenceDiagram
  autonumber
  participant U as User or Controller
  participant A as Aggregator inside kube-apiserver
  participant S as Service and Endpoints
  participant E as Extension API Server
  participant D as Data Store or Memory
  U->>A: GET pods under metrics.k8s.io
  A->>A: Authenticate the user and check path permission
  A->>S: Proxy request with identity headers over TLS
  S->>E: Forward to a healthy Pod
  E->>E: Verify the proxy certificate against request header CA
  E->>A: Ask SubjectAccessReview for the original user
  A-->>E: Allowed or denied
  E->>D: Read or write the data
  D-->>E: Result
  E-->>A: Response
  A-->>U: Response returned to the user
    

Stage by stage

Why two permission checks?

The first check is a coarse gate at the front door. The second check is the fine-grained one, performed by the server that actually understands the resource. The extension server knows which verbs, namespaces and subresources exist in its own API, so it is best placed to judge them. Having both means a mistake in one place does not open the whole system.

Long-lived requests

Many Kubernetes clients use watch requests, which keep a connection open so the server can push changes as they happen. The aggregator supports proxying these streaming connections. This is important because controllers depend on watches to react quickly. It also means a restart of the extension server briefly cuts those streams, and well-written clients reconnect automatically.

Security is the heart of the design, so let us slow down and see exactly who trusts whom.

Step one: the extension server must trust the sender

When the aggregator proxies a request, it adds identity headers such as the user name, the group list and optional extra attributes. Anyone on the network could try to forge such headers, so the extension server must be certain the request really came from the main API server. This is done with a client certificate.

• The kube-apiserver is configured with a proxy client certificate and key.
• That certificate is signed by a certificate authority named in the setting called requestheader-client-ca-file.
• The extension server reads that authority from a ConfigMap named extension-apiserver-authentication in the kube-system namespace, which the control plane publishes for exactly this purpose.
• It also reads the list of allowed certificate names and the exact header names to trust, so a stranger with a valid certificate from another source is still rejected.

Step two: the extension server asks about the person

Once the sender is trusted, the extension server uses two standard Kubernetes review requests, sent to the main API server:

Because answers come from the main server, a RoleBinding that lets Alice read metrics.k8s.io works exactly like a RoleBinding that lets her read Pods. Nothing new to learn and no second user database to maintain.

The permissions the extension server itself needs

For this delegation to work, the extension server’s own service account needs two things:

• Permission to create TokenReview and SubjectAccessReview objects, commonly granted by binding the built-in ClusterRole named system:auth-delegator.
• Permission to read the extension-apiserver-authentication ConfigMap, commonly granted by binding the built-in Role named extension-apiserver-authentication-reader in kube-system.

Step three: the main server must trust the extension server

Trust must also flow the other way. When the aggregator connects, it validates the extension server’s serving certificate. The APIService object carries a CA bundle, which is the certificate authority that signed the extension server’s certificate. The aggregator verifies the server’s certificate against this bundle and checks that it was issued for the Service’s DNS name. This prevents an impostor Pod from answering in place of the real one.

There is an escape switch called insecure skip TLS verify. It exists for quick experiments on a laptop. In production it removes the guarantee that you are talking to the right server, so it should never be used there.

Caching of decisions

Asking the main server about every single request would be expensive. Extension servers built with the standard libraries cache recent answers for a short time. This keeps latency low, but it also means a permission change can take a few seconds to be reflected.

Discovery: how clients learn what exists

Clients like kubectl do not have a hard-coded list of resources. They ask the server. A request to the path /apis returns every group, and a request to /apis/GROUP/VERSION returns the resource types inside that group. This process is called discovery. The aggregator answers the top-level request itself using the APIService objects, and for the detailed listing it asks the extension server.

In large clusters with many extension groups, old-style discovery meant dozens of separate calls every time a client started. Newer Kubernetes releases added aggregated discovery, which lets a client fetch all groups and resources in just two requests, one for the core group and one for all named groups. This feature became stable in Kubernetes 1.30 and makes tools noticeably quicker to start on busy clusters.

Priorities: who wins when names clash

Two numbers on each APIService decide ordering:

Use clear, deliberate numbers. A newly added group with an accidentally high priority can change what existing tools show first, which surprises users.

Availability: the health badge on every APIService

The aggregator runs an availability controller that regularly calls each remote extension server and updates a condition called Available on the APIService. This is the health badge you see in kubectl get apiservices.

While an entry is marked not available, requests to that group receive a “service unavailable” style error. Requests to every other group continue normally, which is a nice isolation property: one broken extension does not take down the rest of the API.

An extension API has a life of its own, from first build to final retirement. Here is the full journey.

The order matters when retiring

Deleting the Pods while the APIService still exists leaves a broken entry that makes the whole cluster noisy. Removing the APIService first makes the group disappear cleanly. Only then is it safe to tear down the workload behind it.

Versioning the API itself

Like every Kubernetes API, an aggregated API passes through maturity stages. An alpha version may change or vanish. A beta version is more careful about changes. A stable version is a promise. Serving two versions at once requires the extension server to convert between them, and that conversion code is entirely your responsibility, which is a real part of the cost we discuss in chapter 10.

The aggregation layer forwards requests. It does not store anything for you. Your extension server decides where its data lives.

This freedom is one of the biggest reasons people choose an aggregated API. A native Kubernetes resource is always a record in the cluster’s main database, called etcd. An extension server can choose differently depending on what the data is like.

Why not just use the main etcd?

Giving an extra server direct access to the main cluster database widens the blast radius. A bug or a flood of writes from that server could slow down every other resource in the cluster, including the Pods and Nodes that keep the business running. Most teams therefore keep the extension’s data separate, or choose a design with no persistent data at all. Teams who only need durable custom objects often step back and use a CRD, which is stored and protected by the main server itself.

Consistency and watches

Kubernetes clients expect each object to carry a version marker that always moves forward, so that a watch can resume from “where it left off”. If your storage cannot provide ordered change numbers, supporting watches correctly becomes difficult. Extensions backed by their own etcd get this ordering almost for free. Extensions that translate from other systems must create an ordering themselves or declare that watches are unsupported.

The most valuable skill here is knowing when not to use the aggregation layer.

A CustomResourceDefinition lets you teach Kubernetes a new resource type by simply describing its shape. You write the description, apply it, and the built-in API extensions server immediately serves a full REST API for it, storing objects in etcd. No new server to run. For the large majority of needs, such as “an Application object” or “a Backup request”, this is the right answer.

A simple decision path

• Do you just need new kinds of objects that controllers will manage? Choose a CRD.
• Do you need data that is computed, streamed or lives in another system, with the API looking native? Consider aggregation.
• Do you need custom subresources like a live log stream or an interactive session? Aggregation fits.
• Do you need strict protection of the main database from your own load? Aggregation with separate storage helps.
• Is your team small and busy? Prefer the CRD and revisit later only if a real limit appears.

The clearest production use of aggregation is the path that feeds autoscaling.

flowchart LR
  K["Kubelet on every node"] -->|"resource usage"| MS["metrics-server"]
  MS --> API1["APIService metrics.k8s.io"]
  API1 --> AGG["Aggregator in kube-apiserver"]
  AGG --> HPA["Horizontal Pod Autoscaler controller"]
  AGG --> TOP["kubectl top"]
  HPA -->|"change replica count"| DEP["Deployment"]
  ADP["Metrics adapter or vendor agent"] --> API2["APIService custom or external metrics"]
  API2 --> AGG
    

Walking through the flow

Every node runs a program called the kubelet. It already measures CPU and memory for each container. The metrics-server collects these numbers from all nodes at a regular interval, which defaults to about fifteen seconds, and keeps only the latest values in memory. It then registers an APIService for metrics.k8s.io. When the Horizontal Pod Autoscaler wants to decide “do we need more replicas?”, it asks the main API server, which proxies to metrics-server. The answer comes back, the controller does a small calculation, and the Deployment is scaled. The autoscaler repeats this loop about every fifteen seconds.

Who uses this in production

Performance: the cost of one extra hop

Every proxied request crosses the network twice more than a native one: main server to extension server, and the review call back. For occasional calls this is invisible. For a controller that polls constantly, the delay adds up. Keep responses small, support filtering by labels and fields where it makes sense, and avoid design where clients must list everything to find one item.

Your extension server should also use the same caching tricks the main server uses. A shared in-memory cache fed by a watch on the underlying data means a read never has to touch the slow store.

Scalability: protect both servers

High availability: never run just one copy

A single Pod means any restart, node drain or crash turns the APIService unavailable. The standard recipe is simple and effective:

• Run at least two or three replicas, spread across failure zones using anti-affinity or topology spread rules.
• Add a PodDisruptionBudget so planned maintenance never removes all replicas at once.
• Use readiness checks so a starting Pod receives no traffic until it is truly ready.
• Set CPU and memory requests so the scheduler places Pods sensibly and the autoscaler for the extension itself has data.
• Make the storage layer highly available too, otherwise the front tier is redundant but the back tier is a single point of failure.

Failure scenarios and recovery

An extension server sits inside the trust boundary of the cluster, so it deserves the same care as the control plane.

Threats worth naming

The protection checklist

• Treat APIService write access like cluster-admin. Redirecting a group sends all of its traffic, including user identities, to wherever the entry points.
• Always validate certificates. Provide a real CA bundle and never enable the skip-verification switch outside a lab.
• Restrict the network. Allow only the control plane and required clients to reach the extension port. Block everything else with network policies.
• Grant minimum rights. The service account needs the two delegation bindings plus whatever the logic truly requires, and nothing more.
• Plan certificate rotation. Short-lived certificates reduce damage from leaks, but only if renewal is automated and monitored.
• Harden the workload. Run as a non-root user, use a read-only filesystem where possible, pin and scan container images, and keep them patched.
• Protect stored data. Encrypt data at rest and in transit, and restrict who can reach the storage layer.
• Audit. The main server records which user called which path, including aggregated groups. Also keep the extension server’s own logs, because it sees details the main server does not.

What to watch

Alerts that earn their keep

• Any APIService unavailable for longer than a few minutes.
• Error ratio above a threshold for requests to your group.
• Ninety-ninth percentile latency rising above an agreed limit.
• Certificates within thirty days of expiry.
• Fewer ready replicas than the minimum you promised.

Troubleshooting path

flowchart TD
  A["API call fails or tool shows warnings"] --> B{"Is the APIService Available?"}
  B -->|"No"| C["Read the reason and message on the APIService"]
  C --> D["Check Service name, port and Endpoints"]
  D --> E["Check Pod readiness and logs"]
  E --> F["Check certificates and CA bundle"]
  F --> G["Check network policy and aggregator routing"]
  B -->|"Yes"| H["Check delegation role bindings"]
  H --> I["Check latency and timeouts"]
  I --> J["Check storage layer health"]
    

Reading common symptoms

Logging advice

Log each request with the user name, verb, resource and result, using a request identifier that you also pass along. During an incident, the ability to follow one request from the main server’s audit record into your own log lines saves hours. Keep log volume sensible so a busy API does not drown the storage that holds the logs.

Patterns that work

Anti-patterns to avoid

Best practices in one place

Mistakes seen again and again

• Forgetting the delegation bindings. Pods are healthy, yet every request fails to authorize.
• Certificate names that do not match the Service. The aggregator rejects the server, and the only hint is a failed discovery check.
• Blocking the control plane with network rules. Everything works from inside the cluster but not through the aggregator.
• Leaving dead APIService objects behind. Namespace deletion and discovery pay the price for years.
• Assuming the main server runs admission rules for the extension. It does not, so invalid data can slip in unless the extension validates.
• Ignoring the extra latency. A design that needs many round trips per user action feels slow even if each call is fast.
• Skipping backups because “it is just an API”. If the data is not rebuildable, it must be backed up.

Now that the big picture is clear, let us open the hood and look at the small controllers that keep the routing table honest.

The aggregator is not a single piece of code that just forwards traffic. It is a group of cooperating loops, each with one narrow job. Understanding them explains almost every strange behaviour you will meet in production.

The routing table and its handler

When an APIService is created, changed or deleted, a controller notices and updates an in-memory table that maps each group and version to a handler. For a remote entry the handler is a reverse proxy. For a local entry the handler simply passes the request along the delegation chain. The table lives in memory, which is why lookups cost almost nothing: matching a URL path to a handler takes a tiny fraction of a millisecond, far smaller than the network time that follows.

Because this table is in the memory of each kube-apiserver process, every replica of the control plane keeps its own copy. They all build the same picture from the same stored APIService objects, so after a change there is a very short window where one replica already knows about a new entry and another does not. Clients that hit the slower replica receive a “not found” answer for a moment, and a retry succeeds. This is normal and usually lasts only a second or two.

The availability controller

This loop is the health inspector. For every remote APIService it performs a lightweight call to the extension server and records the outcome on the object as the Available condition. It also reacts to changes in the Service and Endpoints objects, so if the last ready Pod disappears the entry turns unavailable quickly rather than waiting for the next scheduled check. When the entry flips to unavailable, the proxy handler stops sending real user traffic there and returns an error right away. This protects the extension server from a pile-up of requests while it is recovering, and it protects users from long waits.

The discovery and OpenAPI loops

Besides routing, the aggregator maintains two published documents that clients read.

The merged schema is what powers features such as client-side validation, the explain command and code generators. If an extension server publishes a broken or enormous schema, the merge can become slow or even fail for that one group. Keep your published schema accurate and reasonably small, and test that it merges cleanly before shipping.

Automatic entries for CRD groups

Here is a subtle fact that surprises many people. When you create a CRD, the cluster also creates an APIService entry for its group, automatically. These entries carry a special label that marks them as managed by the system, and they are local entries because the API extensions server in the same process answers them. If you edit or delete one of these managed entries by hand, the controller that owns them will usually restore it. Do not fight it. Manage the CRD and the entry follows.

How the proxy handles connections

The proxy opens secure connections to the extension server and reuses them for many requests, which saves the cost of repeating the handshake each time. For watch requests it keeps the connection open and streams events as they arrive. It also carries upgrade-style connections, which are needed for features like interactive sessions into containers when an extension offers them. When the extension server restarts, these long connections break, and clients are expected to reconnect and resume from the last version they saw. If your extension cannot resume cleanly, clients are forced to fetch the full list again, which is costly in a big cluster. Plan resumption support from the start.

An extension server is a real Kubernetes-style API server. It must keep the same promises as the built-in one, or the tools around it will behave strangely.

Kubernetes clients make a number of quiet assumptions about how a server behaves. A good extension honours all of them. The following ideas are the ones that matter most.

Resource version: the ever-moving bookmark

Every object carries a resource version, an opaque marker that changes whenever the object changes. Clients treat it as a bookmark. A watch can say “tell me everything that happened after this bookmark”. An update can say “change this object only if it still matches this bookmark”. The second use is called optimistic concurrency. It prevents two people from silently overwriting each other, because the later writer is told “your copy is stale, fetch it again”. An extension that ignores this rule will lose data under concurrent writes.

List, watch and consistency

Controllers usually begin with a full list and then switch to a watch starting from the list’s bookmark. To make this reliable the server must return a consistent snapshot and then deliver every change after it, in order and without gaps. If the server can no longer remember changes from an old bookmark, it must say so clearly so the client lists again. This is the foundation of how controllers stay correct for months without drifting.

Subresources

A subresource is an extra address hanging off an object, such as its status or an action. The standard ones are status and scale. An aggregated API can invent others, for example a path that streams a live view of something, or a path that runs an operation on the object. Because each subresource has its own address, it can have its own permission rule. A person may be allowed to read an object but not trigger its action.

Defaulting, validation and admission

When a request creates or changes an object, a well-built server runs it through a pipeline. First it fills in missing fields with defaults. Then it checks the values are valid. Then it runs admission steps, which can approve, reject or adjust the request. Extension servers built on the official libraries include a place to plug these steps in and can call external webhooks too. Remember that the main server’s own admission rules do not cover your group, so any policy that must apply to your objects has to be wired into your server.

Finalizers and owner references

Kubernetes cleans up related objects through two small fields. A finalizer is a note saying “do not fully delete this until I have finished some cleanup”. An owner reference is a note saying “this object belongs to that object, so remove me when it goes away”. The garbage collector reads these notes across all groups, including yours. If your objects use them correctly, deletion behaves like it does for native resources. If they are wrong, cleanup can hang forever.

Conversion between versions

If you serve two versions, a client can ask for either one, and the server must translate in both directions without losing information. A common technique is to keep one internal form and convert to and from each public version. Round-trip tests that convert a value there and back and compare the result are cheap and catch most mistakes.

Why the official libraries matter

The Kubernetes project provides a generic API server library that already implements the behaviours above: authentication and authorization delegation, request handling, watch support, schema publishing, fairness controls and health endpoints. Building on it means you spend your effort on your own resources and rules rather than reinventing the plumbing. Teams that write an API server from scratch usually discover, one outage at a time, which of the quiet assumptions they forgot.

Most real incidents with aggregated APIs are plumbing problems, not logic problems. This chapter is about the plumbing.

Names and addresses

Inside a cluster, every Service gets a DNS name of the form service-name dot namespace dot svc. The serving certificate of your extension must be valid for exactly that name, because the aggregator checks the name it is connecting to against the certificate. A certificate for a different name, or one that only lists the short name, will fail the check. A neat habit is to include the short name, the namespace-qualified name and the full name in the certificate so that every route works.

The control plane may live in another network

On managed services the control plane often runs in a network that you do not control. Traffic from that network to your Pods must be explicitly permitted.

The port question

An APIService names a port on the Service. If you leave it out, the standard secure port is assumed. The extension must listen with TLS on the matching target port. A frequent beginner error is to start the extension on a plain, non-secure port because it seemed simpler. The aggregator always speaks TLS to remote entries, so the connection fails and the symptom is the same vague discovery failure.

Certificates: three different jobs

There are three kinds of certificate in this story, and confusing them is the number one source of mistakes.

• The proxy client certificate belongs to the main server and proves its identity to the extension. It is managed by whoever manages the control plane.
• The serving certificate belongs to the extension and proves its identity to the main server. You manage it, and its authority goes into the CA bundle on the APIService.
• The client authentication authority is what the extension uses to recognise ordinary client certificates, if it accepts them directly. It comes from the published ConfigMap.

Rotation without downtime

When the authority behind the serving certificate changes, the CA bundle on the APIService must be updated, and during the switch both the old and the new authority should be trusted for a time. A neat pattern is to publish a bundle containing both, roll out the new serving certificates, and only later remove the old authority. Tools such as certificate management controllers can inject the bundle into the APIService automatically, which removes a manual step that people forget at three in the morning.

The word “aggregation” appears in many places in software. Let us make sure it is not confused with its neighbours.

Not an API gateway

In the wider world, an API gateway is a front door for many services that handles login, rate limiting and routing for web traffic. Some gateways also “aggregate” by combining several backend answers into one reply. The Kubernetes aggregation layer is related only in spirit. It does not merge answers from several backends into one response. It routes each request to exactly one owner of that group. Its purpose is to extend one specific API, the Kubernetes API, and not to front a company’s web services.

Compare the extension tools

How they work together

These tools are partners, not rivals. A very common production design looks like this: a CRD describes what the user wants, an operator watches those objects and builds the real thing, an admission webhook enforces company policy on them, and an aggregated API exposes live status and metrics about the result. Each tool does the job it is best at. The user sees a single, tidy Kubernetes experience.

What the aggregation layer is not for

• It is not a place to run heavy business logic that has nothing to do with Kubernetes resources.
• It is not a substitute for a message queue or a general database.
• It is not needed just to react to cluster events. A controller watching existing resources does that without any new API.
• It is not a way to bypass permission rules. Everything still passes through the same login and role checks.

An extension lives alongside a cluster that keeps moving. Planning for change is part of the design, not an afterthought.

Two things that change at once

Your extension has its own releases, and the cluster has its own too. Kubernetes publishes new minor releases several times a year, and managed services upgrade the control plane on a schedule you may not fully control. Both sides must keep speaking the same language during the upgrade, when some control plane replicas run the old version and some run the new one.

Compatibility rules of thumb

A safe upgrade sequence for your extension

Deprecating your own API

Treat your API like a product. Mark old versions as deprecated in the documentation and, where the libraries allow, return warnings to clients that still use them. Track which clients call the old version using request metrics labelled by version. Only when that number reaches zero, or an agreed small figure, is it safe to stop serving it. Deleting an old version while an unknown tool still calls it creates the kind of outage that is hard to diagnose, because the failing tool might be a forgotten cron job.

Three short stories show how the ideas fit together when real things go right and wrong. The teams in these stories are illustrations, not specific companies.

Scenario one: the autoscaler that stopped scaling

A retail platform notices at peak hour that its web Pods are not increasing even though traffic doubled. The on-call engineer checks the autoscaler and sees a message saying that the metrics for the Pods are unavailable. Rather than guess, the engineer lists the APIService objects. The entry for resource metrics shows not available, with the reason that no endpoints exist. A quick look at the Pods shows that the metrics-server Pod was evicted from a node under memory pressure and the replacement was stuck waiting for a place to run because only one replica existed and its resource request was too large for the remaining space.

The fix was immediate: free space so the Pod could schedule. The lasting improvements were more valuable. The team ran two replicas spread across zones, added a disruption budget, lowered the resource request to a realistic figure, and created an alert on the unavailable APIService signal so that the next occurrence pages a human within minutes rather than being discovered by customers.

Scenario two: the namespace that would not die

A platform team deletes a test namespace and it stays in the terminating state for hours. Everything inside seems gone. The team reads the namespace conditions and finds a message that discovery failed for one API group. That group belongs to an experimental extension installed months earlier by a team that has since left. Its Pods were removed during a cleanup, but the APIService entry was left behind and now points to a Service that no longer exists. The namespace controller must ask every group whether anything remains, and one group never answers.

Deleting the stale APIService made the namespace finish terminating within seconds. Afterward, the team wrote a short housekeeping rule: whenever an extension is retired, its APIService must be removed first, and a weekly check lists any entries that are not available and have been so for more than a day.

Scenario three: the certificate that quietly expired

A company runs an extension that exposes live cost data per team. One Monday morning every cost command fails, while the rest of the cluster is perfectly healthy. The APIService shows failed discovery check. The Pods are running and ready. The engineer inspects the serving certificate and finds it expired over the weekend. It had been created manually a year earlier with a one-year life, and the person who made it had changed teams.

Reissuing the certificate and restarting the Pods fixed the outage. To prevent a repeat, the team moved to an automated certificate controller with short lifetimes, configured it to update the CA bundle on the APIService itself, and added a dashboard and a warning alert for any certificate within thirty days of expiry.

Running an extension is a long-term commitment. A little structure now saves many bad nights later.

What to test before launch

Who owns it?

Every extension needs a named owning team. That team answers alerts, renews certificates, reviews who has access, upgrades libraries and retires the API when its time comes. When ownership is vague, the extension becomes the thing everyone depends on and nobody maintains. Write the owner, the escalation path and the documentation link into the description of the APIService or a nearby document so a stranger can find them at night.

A compact runbook

Capacity planning in plain terms

Start with three numbers: how many requests per second you expect at peak, how many watchers will stay connected, and how big the typical object is. Multiply requests by the time each takes to find how many requests are in flight at once, then give each replica comfortable headroom, such as running at no more than half of its tested limit during normal peaks. Add replicas for failure tolerance on top of that, so that losing one zone still leaves enough capacity. Revisit the numbers each quarter, because a successful API tends to attract more callers than anyone predicted.

Technology is only half the story. In a company with many teams, rules and costs decide whether an extension helps or hurts.

Who is allowed to add an API?

Because a new APIService can change what the whole cluster appears to offer, mature organisations treat it as a controlled change. A typical approach is a short review that asks the same questions every time. What problem does this solve that a CRD cannot? Who owns it? How is it secured? How will it be monitored and upgraded? What is the plan if the owning team disappears? Teams that answer these questions in writing before launch rarely face the orphaned extension problem later.

Naming and group hygiene

Choose group names from a domain your organisation controls, so there is never a clash with a project or vendor. Keep a simple catalogue of all groups in use, with the owner next to each. When two teams want similar names, the catalogue settles the question in minutes instead of weeks. Avoid names that end in the reserved suffixes used by the Kubernetes project itself, because those are managed by the community.

The true cost of an extension

Shared clusters and fairness

In a cluster shared by many teams, one team’s busy extension can affect everyone through the shared front door. Set fair limits on how much traffic any one caller may send, watch the request metrics by group, and talk to heavy users early. If a team truly needs heavy and unpredictable traffic, a separate cluster for that workload is often cleaner than trying to protect everyone else with tuning.

Documentation people actually read

Publish a one-page guide for every extension: what it is for, example requests, the permissions needed, the owning team, known limits and how to ask for help. Keep an example for each resource and subresource. A user who can copy a working example in under a minute will use your API well. A user who has to read the source code will send you questions at midnight.

A quick reference to every important word in this guide, in plain language.

On clusters created with common installers and on managed cloud services, yes. The relevant settings for the proxy certificate and request headers are usually configured for you. On a hand-built cluster you must set them yourself.

No. The aggregator watches APIService objects and updates its routing table on the fly. The first-time setup of the proxy certificate settings does need a restart, but adding or removing individual APIServices does not.

Yes, an APIService can point to an external address instead of a cluster Service. The same trust rules still apply, so certificates and network reachability from the control plane must be correct. Running inside the cluster is simpler for most teams.

Requests to your group fail with an unavailable error. Other groups keep working. The exception is the side effect on tasks that list every group, such as namespace cleanup, which is why a down extension should be fixed or removed promptly.

No. A webhook is called by the main server to approve or modify requests for resources it already serves. An aggregated API is a whole separate API surface with its own resources.

Yes. Because discovery tells kubectl which resources exist, commands such as get, describe and delete work immediately. Table-style output needs your server to return column information, otherwise the output is plain.

Yes. You create one APIService per group and version, and all of them can point to the same Service.

Only if your extension stores them in that etcd, which is generally discouraged. Data in separate storage must be backed up separately.

Start with a CRD. Move to aggregation only for needs a CRD cannot meet, such as live computed data, special storage or custom subresources.

Only highly trusted cluster administrators or automation they control, because an APIService can redirect traffic for an entire group.

It can. Keep your libraries reasonably close to the cluster version, test against new releases, and read upgrade notes about API changes, since your server speaks to the main API for its review calls.