The API server is the front door of a Kubernetes cluster. Everyone and everything that wants to read or change the cluster must go through it.

First, what is an API?

An API (Application Programming Interface) is a menu of requests that a program is willing to answer, written in a strict format so that other programs can use it without human help. Think of a restaurant menu: you cannot walk into the kitchen and grab ingredients, but you can order any dish on the menu and the kitchen will respond in a predictable way. Kubernetes publishes its own menu, called the Kubernetes API, and it includes requests such as “create a Deployment,” “show me all Pods,” “delete this Service,” or “tell me whenever a Pod changes.”

And what is a server?

A server is simply a program that waits for requests and answers them. Put the two ideas together and the kube-apiserver is the program that receives requests written in the Kubernetes API format, checks them, carries them out, and returns an answer. It listens on a network port (6443 by default) and talks over an encrypted connection, so nobody on the network can read or tamper with the messages.

The most important rule

Here is a rule that will simplify everything else you learn: in Kubernetes, every component talks to the API server, and the API server is the only one that talks to the database. Your command-line tool talks to it. The scheduler talks to it. The controllers talk to it. The kubelet on every worker machine talks to it. Even the Pods running inside the cluster can talk to it if you allow them. No component gives orders directly to another component. Instead they all read from and write to the shared record kept by the API server, and each one reacts to what it sees there.

flowchart LR
    KC["kubectl and other tools"] --> API
    CI["CI/CD pipelines"] --> API
    UI["Dashboards"] --> API
    SCH["Scheduler"] --> API
    CM["Controller Manager"] --> API
    KUB["kubelets on worker nodes"] --> API
    PODS["Pods with permission"] --> API
    API["kube-apiserver - the front door"] --> ETCD[("etcd - the cluster database")]

To see why one front door is a clever design, imagine the alternative.

The chaotic alternative

Suppose every part of the cluster could write straight into the database. The scheduler might save a decision while a controller saves a different one at the same moment. Different programs could store data in slightly different formats. Anyone with a network connection to the database could read passwords and change settings. There would be no single place to check permissions, no single place to record who did what, and no way to upgrade one part without breaking the others.

The clean solution

Kubernetes solves this by placing one gatekeeper in front of the database. That gatekeeper gives the whole system four valuable gifts.

A design that keeps components simple

A hidden benefit is that the other components stay small and easy to replace. The scheduler does not need to know how to talk to a database, how to authenticate people, or how to store history. It only needs to know how to ask the API server for unscheduled Pods and how to tell it which node was chosen. Because of this design, you can even swap in a different scheduler, or run several at once, without changing anything else. The API is the shared contract that lets the whole cluster be built from independent, replaceable parts.

Before following a request through the API server, you need five small ideas.

Resources and objects

A resource type is a kind of thing the API knows about, such as Pods, Deployments, Services, Nodes, or Namespaces. An object is one specific instance of a resource type, such as “the Deployment named checkout in the shop namespace.” The API server stores objects and lets clients create, read, change, and delete them.

Spec and status: the two halves of every object

Almost every object has two main sections. The spec is what you want: “three copies of this container image.” The status is what actually exists right now: “two copies are running, one is starting.” You write the spec. The cluster’s components write the status. The gap between the two is what controllers work to close, which is the desired-state idea at the heart of Kubernetes.

Verbs: what you can do

API groups and versions

With hundreds of resource types, the API needs an orderly filing system. Resource types are placed in API groups, such as apps for Deployments or batch for Jobs, and each group has versions like v1 or v1beta1. When you write a configuration file, its first lines name the group and version, for example apps/v1. The oldest and most basic resources, such as Pods and Services, live in the “core” group, which has no name and is simply written as v1.

Namespaces

Many objects live inside a namespace, a named room in the cluster that keeps teams or environments tidy. Some objects, like Nodes, belong to the whole cluster and have no namespace. The API server uses the namespace as part of an object’s address and in its permission checks.

Although it is described as one program, the API server is built from several layers that each handle a different kind of API.

flowchart TB
    IN["Incoming request"] --> SEC["Security chain - authentication, authorization, admission"]
    SEC --> AGG["Aggregation layer - the traffic director"]
    AGG --> CORE["Built-in API - Pods, Deployments, Services"]
    AGG --> EXT["Extension API - custom resource definitions"]
    AGG --> ADD["Add-on API servers - for example metrics"]
    CORE --> STORE["Storage layer with watch cache"]
    EXT --> STORE
    STORE --> ETCD[("etcd")]

The parts, one by one

Stateless by design

The API server keeps no permanent data of its own. Everything important lives in etcd, and whatever the API server holds in memory can be rebuilt from etcd in moments. This is a huge advantage: you can run several API servers side by side behind a load balancer, stop one for an upgrade, or lose one to a crash, and the cluster keeps working.

Every write request follows the same road. Learning this road once explains most of Kubernetes behavior.

sequenceDiagram
    participant C as Client
    participant A as Authentication
    participant Z as Authorization
    participant M as Mutating Admission
    participant V as Validating Admission
    participant S as Storage Layer
    participant E as etcd
    C->>A: Send request with credentials
    A->>Z: Identity confirmed
    Z->>M: Permission granted
    M->>V: Defaults and changes applied
    V->>S: Object is valid and allowed
    S->>E: Save the object
    E-->>S: Saved
    S-->>C: Success response

Why the order matters

The checks are arranged from cheap to expensive and from general to specific. There is no point examining the details of an object if the caller is not even allowed to be there, so identity and permission come first. Mutating plugins run before validating plugins so that the rules judge the object in its final form, including anything the mutating plugins added. And nothing reaches etcd unless it survives every checkpoint.

Kubernetes does not keep a list of human users. Instead it trusts outside systems to vouch for people and checks their proof.

This surprises many beginners. There is no “create user” request in the Kubernetes API. Identities come from somewhere else, such as a certificate authority, a company sign-in system, or the cluster’s own service account mechanism. The API server simply supports several ways of checking proof, and a cluster can enable more than one at the same time.

Groups and identities

After authentication, the API server knows a user name and often a list of groups. It does not remember these. They arrive with every request and are used in the next step, authorization, to decide what is allowed. Because identities live outside Kubernetes, removing someone from your company sign-in system also removes their access, which is far safer than managing passwords inside the cluster.

Authentication says who you are. Authorization says what you are allowed to do. They are different questions.

The default is “no”

Kubernetes authorization is based on a friendly rule: everything is forbidden unless a rule says it is allowed. Rules only ever add permissions and never take them away, which keeps the system easy to reason about. The API server can use several authorization methods together and allows a request as soon as one of them says yes.

RBAC in plain English

Role-Based Access Control has four simple building blocks. A Role is a list of allowed actions inside one namespace, such as “may read Pods and Services.” A ClusterRole is the same idea but can cover the whole cluster or resources that have no namespace, such as Nodes. A RoleBinding hands a Role to a person, group, or service account inside a namespace. A ClusterRoleBinding hands a ClusterRole out across the entire cluster.

How a decision is made

The API server builds a small question from every request: who is asking, which verb, which resource, in which namespace, and which specific object. It then searches the RBAC rules for a match. If it finds one, the request moves on. If not, the caller receives a “forbidden” answer, and that denial is recorded in the audit log. Because the check uses the identity established one step earlier, the two steps work together as a pair.

A request can have a proven identity and full permission and still be a bad idea. Admission control is where the cluster applies its own common sense.

What admission control is

Admission controllers are plugins inside the API server that examine, and sometimes change, requests that would create, update, or delete objects. They run after authentication and authorization, and before anything is saved. They are the reason a cluster can enforce rules like “every container must declare its memory needs,” “no container may run as an administrator,” or “every Pod must carry a team label.”

flowchart LR
    A["Request passes authentication and authorization"] --> B["Mutating admission - may change the object"]
    B --> C["Schema validation - is it well formed"]
    C --> D["Validating admission - may reject the object"]
    D --> E[("Saved in etcd")]
    D -.->|Rejected| X["Error returned to the caller"]

Two kinds: mutating and validating

Built-in and custom controllers

Kubernetes ships with many admission controllers already switched on. The LimitRanger applies default resource sizes. The ResourceQuota controller refuses to let a namespace exceed the total amount of resources it was granted. The Pod Security admission checks Pods against the built-in Pod Security Standards. The NamespaceLifecycle controller stops new objects from being created inside a namespace that is being deleted.

When these are not enough, you can add your own in two ways. Admission webhooks let the API server call a program you write, over the network, before deciding. This is how service meshes inject their helper containers and how policy tools such as Kyverno and Open Policy Agent Gatekeeper enforce company rules. Newer clusters also support ValidatingAdmissionPolicy, which lets you write simple rules in a small expression language that runs inside the API server itself, with no outside program needed. That makes it faster and removes a possible point of failure.

The API server does not just answer questions. It also tells everyone else, instantly, when something changes.

The problem with asking again and again

Imagine a controller that wants to know about new Pods. It could ask the API server every second, “Any new Pods?” With one controller this is harmless. With hundreds of controllers, thousands of nodes, and millions of objects, it would bury the API server in useless questions. Kubernetes solves this with the watch.

How a watch works

A client first asks for a full list of the objects it cares about. The API server answers and includes a resource version, which is a marker for “the state of the world at this moment.” The client then opens a long-lived connection and says, “Tell me everything that changes after this marker.” From then on the API server pushes a tiny message whenever an object is added, modified, or deleted. If the connection breaks, the client reconnects from the last marker it saw, so it never misses a change.

flowchart LR
    ETCD[("etcd")] --> WC["API server watch cache"]
    WC -->|list then watch| INF["Informer inside the controller"]
    INF --> LC["Local in-memory copy"]
    INF --> Q["Work queue of changed objects"]
    Q --> CTRL["Controller logic"]
    CTRL -->|write results| API["API server"]
    API --> ETCD

Informers and the local copy

Programs that watch the API use a helper called an informer. The informer performs the list, keeps the watch alive, and stores a private copy of the objects in memory. When the controller needs to look something up, it reads its own copy instead of calling the API server. When something changes, the informer puts the object’s name in a work queue, and the controller handles it in its own time. This is the reason one API server can serve an entire cluster full of hungry watchers.

Optimistic concurrency: two writers, one winner

What if two components try to change the same object at the same time? The API server uses the resource version again. Every update must include the version the writer last saw. If the object has changed since then, the API server refuses the update with a conflict error, and the writer must read the fresh copy and try again. This “check the version before you write” method prevents accidental overwrites without slow locks. It is also why you sometimes see the message that an object “has been modified” when editing configuration by hand.

The API server and etcd are a pair. One is the spokesperson and the other is the memory.

The only door to the database

etcd is a small, reliable database that stores the full state of the cluster. Only the API server is allowed to read from it or write to it. Everything else, from the scheduler to the kubelets, asks the API server. This means the security of etcd depends on the API server being the sole guardian, and that you can change how data is stored, encrypted, or backed up in one place without touching the other components.

How data is stored

Each object is saved under a path-like key that includes its type, namespace, and name. The API server converts the object into a compact stored form, usually binary, and writes it. When Kubernetes releases a new API version, the API server can read old stored objects and convert them on the fly, so a cluster survives upgrades without a painful data migration.

The watch cache: protecting the database

etcd is precise but not built for thousands of clients watching at once. To protect it, the API server keeps its own in-memory cache of recent objects and changes. Most list and watch requests are answered from this cache. Only writes, and the occasional read that must be perfectly fresh, go all the way to etcd. This layer is one of the main reasons that Kubernetes clusters can scale to thousands of nodes on a fairly small database.

Encryption at rest

By default, secrets are stored in etcd only encoded, not truly encrypted. Anyone who could read the etcd files would see them. The API server can be configured to encrypt selected resources, especially Secrets, before storage, using keys that you control or that are protected by a cloud key management service. This protects against stolen disks and backups.

One of the smartest features of Kubernetes is that you can teach its API new words.

Custom resources

Suppose your company runs databases and wants to say, in the same simple style as a Deployment, “I want a database called orders with three replicas and daily backups.” With a CustomResourceDefinition, you register a new resource type such as Database. From then on the API server treats it like a built-in type: it validates it, stores it in etcd, checks permissions for it, and lets people list and watch it. The API server does not know what a database is. It only stores the wishes.

Controllers and operators

To make the wishes come true, you add a custom controller, a program that watches those custom objects through the API server and takes action, such as creating the Pods, disks, and backup jobs needed. When such a controller encodes the knowledge of a human expert, for example how to safely upgrade a particular database, it is called an operator. Popular tools like Argo CD, cert-manager, Istio, and the Prometheus operator all work this way: they add resource types to the API and controllers that react to them.

The aggregation layer

Sometimes you need an entirely separate API server with its own logic and storage. The aggregation layer lets you register such an extra server so that requests to certain paths are forwarded to it, while clients still see one unified API. The metrics server uses this to provide resource usage information through the normal Kubernetes API.

The API is a promise. Kubernetes has clear rules for how and when that promise can change.

Alpha, beta, and stable

Deprecation: retiring old versions

When a newer version of an API replaces an older one, the old version is first marked deprecated, and the API server starts returning a warning whenever it is used. After a defined number of releases it is removed, and any configuration that still uses it stops working. A famous example was the removal of several older beta versions in the release of 2022, which broke many tools that had not been updated. The lesson is to read upgrade notes carefully, scan your configuration files for deprecated versions before every upgrade, and keep your tools current.

Conversion behind the scenes

Because several versions of one type can exist at once, the API server can convert between them. A client may write using one version and another client may read using a different version of the same object. The API server keeps one storage version in etcd and converts on the way in and out. This is what allows old and new clients to coexist during upgrades and makes the API dependable.

The API server is on the path of nearly every action in the cluster, so it must be fast and it must not be a single point of failure.

Running several copies

Because the API server is stateless, the standard way to make it dependable is to run two or three copies, ideally in different availability zones, behind a load balancer. Clients connect to the load balancer’s address, which forwards each request to a healthy copy. All the copies are active at the same time, share the same etcd, and produce identical answers. Losing one simply shifts traffic to the others.

flowchart TB
    CLIENTS["kubectl, controllers, kubelets, pipelines"] --> LB["Load balancer with one stable address"]
    LB --> A1["API server 1 in Zone A"]
    LB --> A2["API server 2 in Zone B"]
    LB --> A3["API server 3 in Zone C"]
    A1 --> E1[("etcd member 1")]
    A2 --> E2[("etcd member 2")]
    A3 --> E3[("etcd member 3")]

Protecting the server from overload

What if a buggy program sends the API server a flood of requests? Older versions used a crude cap on the number of simultaneous requests, which meant one noisy client could crowd out important ones. Modern clusters use API Priority and Fairness. Requests are sorted into priority levels, such as system-critical traffic from the control plane and ordinary traffic from users, and within each level they are shared fairly between callers. When the server is busy, a runaway script waits in its own queue while the scheduler and kubelets carry on. Clients that are turned away receive a “too many requests” answer and are expected to slow down and retry.

Habits that keep the API server fast

• Use watches instead of repeated lists. Programs that list everything every few seconds are the number one cause of API server overload.
• Ask for pages. Large lists can be fetched in chunks, which avoids sudden memory spikes.
• Filter on the server. Label and field selectors let the API server send only what you need.
• Keep objects small. Huge objects and endless numbers of tiny ones both cost memory and etcd space.
• Limit extension sprawl. Every webhook and every custom type adds work to the request path.
• Size the machines properly. Bigger clusters need more memory and processor for the API servers, and fast disks for etcd.

The single front door is a brilliant idea, and like every idea it has costs.

Key trade-offs

Whoever controls the API server controls the cluster, so it deserves the strongest protection you can give.

A layered defense

A real lesson from the field

Security researchers have repeatedly found clusters whose management interfaces were reachable from the open internet with no password. In one widely reported case in 2018, attackers found an exposed Kubernetes console at a large company and used it to run cryptocurrency mining software on the company’s cloud account. The attack did not require clever hacking. It only needed a door that had been left open, and it is a perfect reminder that the first layer of defense is simply not exposing the API server carelessly.

Managed services and shared responsibility

On Amazon EKS, Google GKE, and Azure AKS, the provider runs and patches the API servers for you, and it offers options for private endpoints and for sending audit logs to its logging service. You are still responsible for who has access, what those people can do, and which policies are enforced. The provider secures the building; you decide who gets a key.

Because the API server sits in the path of everything, its health is the cluster’s health. Watch it closely.

Is it alive? Health endpoints

The API server exposes simple health-check addresses. The livez endpoint answers “is the process alive?”, the readyz endpoint answers “is it ready to serve traffic?”, and both can show the result of each internal check. Load balancers use these answers to decide which copies should receive requests, and cloud providers use them to replace unhealthy control plane machines.

How is it doing? Metrics

The API server publishes detailed measurements in a format that tools like Prometheus can collect. A handful of them tell most of the story.

Audit logs: the cluster’s black box

Audit logging records a trail of requests passing through the API server. You write an audit policy that says how much detail to record for each kind of request, from just the basic facts (who, what, when) to the full request and response bodies. A common approach is to record light information for most calls and detailed information for sensitive ones, like changes to permissions or access to Secrets. Audit logs answer questions such as “who deleted this Deployment last Tuesday?” and “which account has been trying to list Secrets?”

Application logs and events

The API server also writes its own text logs, which are crucial when a request is unexpectedly refused or a webhook misbehaves. On managed services, the provider offers these as control plane logs that you can turn on. Do not forget Kubernetes events, the short notes that the cluster writes about itself. They are stored through the API server too, and they are the first place to look when a Pod will not start.

Patterns are habits that healthy teams share. Anti-patterns are shortcuts that feel smart today and hurt later.

Patterns that work

Anti-patterns to avoid

These lessons are usually learned painfully. You can learn them here for free.

Best practices

• Run at least two, preferably three, API servers across zones behind a load balancer, or use a managed control plane that does this for you.
• Keep Kubernetes current. Upgrade one version at a time, test in staging, and read the notes about removed API versions first.
• Enable audit logging with a sensible policy, and ship logs to a protected, separate system.
• Use short-lived credentials. Prefer sign-in through an identity provider and bound service account tokens over long-lived files.
• Apply least privilege. Review RBAC regularly and remove old bindings.
• Turn on encryption for Secrets at rest and protect the keys.
• Tune client behavior. Make your own programs use watches, pagination, and back-off when told to slow down.
• Watch the numbers. Track request latency, error rates, queue depth, and etcd health, and set alerts.
• Test your extensions. Load-test webhooks and controllers, and decide deliberately whether each should fail open or closed.
• Practice recovery. Rehearse losing one API server, then the load balancer, then restoring etcd from a backup.

Common mistakes

These examples come from what organizations and projects have shared publicly. Details may have changed since.