Before following a request, we need to know who is receiving it and why it is the center of the cluster.

Kubernetes is a system that runs and manages containers across many computers. Behind the scenes it keeps a long list of things that should exist: which applications should run, how many copies of each, which network rules apply, which secrets are stored, and so on. The component that guards and serves this list is the kube-apiserver, usually called the API server.

The API server is a web server. It speaks HTTP, mostly using a style called REST, which simply means that things are named by addresses and you use standard actions such as read, create, change and delete. A Deployment, a Pod, a Service or a Secret is each called a resource. When you ask Kubernetes to run an application, you are really asking the API server to create a resource that describes what you want.

Why a single door is a good idea

No other Kubernetes component is allowed to talk to the cluster database directly. The scheduler, the controller manager, the kubelet on each machine, kubectl, your CI pipeline and your monitoring tools all talk to the API server instead. Because of this, there is exactly one place where identity, permissions, rules and recording happen. If you want to know how Kubernetes stays safe and consistent, you start by understanding this door.

The API server is also stateless. It does not keep the master record in its own memory or disk. The record lives in a separate database called etcd. This means you can run several API servers side by side behind a load balancer, and any of them can answer any request.

A request passes through a fixed series of gates, and failing any gate stops it right there.

When a request reaches the API server it does not go straight to the database. It travels through a pipeline. Each stage asks one question, and the request continues only if the answer is acceptable. At a high level the questions are: who are you, are you allowed to do this, is this request healthy enough to be processed right now, do the cluster rules allow this change, is the object itself well formed, and can it be safely stored?

flowchart LR
  A["Client: kubectl, controller, kubelet"] --> B["TLS and HTTP parsing"]
  B --> C["Authentication"]
  C --> D["Audit and impersonation"]
  D --> E["Priority and fairness"]
  E --> F["Authorization"]
  F --> G["Mutating admission"]
  G --> H["Schema validation"]
  H --> I["Validating admission"]
  I --> J["Write to etcd"]
  J --> K["Response to client"]
        

Notice the shape of the diagram. Early stages are about the caller: identity and permission. Middle stages are about the change: rules and correctness. The last stage is about durability: making sure the change is saved. Read requests, such as listing Pods, skip the admission and write stages, but they still go through identity, permission and flow control.

Before anyone checks your name, a secure channel must exist and the request must be understood.

Every conversation with the API server begins with a network connection. In a normal cluster the API server listens on a secure port, commonly 6443, and speaks HTTPS. Older clusters once offered an insecure local port, but that was removed many versions ago, so today you should assume everything is encrypted.

What TLS gives you

TLS, short for Transport Layer Security, does two jobs. First, it encrypts the traffic so that nobody on the network can read or change it. Second, it lets the client confirm that it is really talking to the genuine API server and not an imposter. The API server shows a certificate, and the client checks that this certificate was signed by an authority it trusts. That trusted authority information is usually stored in the client’s kubeconfig file.

TLS can also work in the other direction. The server can ask the client to show its own certificate. This is called mutual TLS, and it is one of the ways a client proves who it is. We will return to this in the next chapter.

HTTP and the shape of the address

Once the secure channel is ready, the client sends an HTTP request. Modern clients use HTTP/2, which allows many requests to share one connection. This matters because controllers and kubelets keep connections open for a long time and send many requests through them.

The address in the request tells the API server which resource is wanted. Core resources such as Pods live under one kind of path, while resources that belong to named groups, such as Deployments in the apps group, live under another. The path also includes the version of the API, for example v1, and optionally the namespace and the name of the object. The HTTP method then maps to a Kubernetes verb: reading one object is get, reading a collection is list, a long-running read is watch, a POST creates, a PUT replaces, a PATCH changes part of an object, and a DELETE removes.

The request information step

Early in the pipeline the API server works out a tidy summary of the request: the verb, the API group, the version, the namespace, the resource, the subresource and the name. Every later stage reuses this summary. That is why authorization rules can talk about “list Pods in the team-a namespace” so naturally, because the server has already split the raw address into those parts.

Authentication answers one question only: who is making this request?

Kubernetes does not have a built-in list of human users. There is no “create user” command. Instead, the API server trusts proof that comes from outside and turns that proof into an identity. The result of successful authentication is a small package: a username, a set of groups, and sometimes extra information. Everything after this stage uses that package.

The authenticators

The API server can be configured with several authentication methods at once. It tries them one after another. The first method that recognizes the credentials wins. If none of them recognize the request, the server answers with 401 Unauthorized, unless anonymous access is enabled.

Service account tokens in more detail

Applications inside the cluster need an identity too. Kubernetes gives each Pod a service account, and the kubelet places a token for it inside the Pod. In current versions this token is short-lived, tied to the specific Pod and to an intended audience, and is refreshed automatically before it expires. This is much safer than the old style of long-lived tokens stored in Secrets, which are no longer created automatically.

Anonymous requests

If a request carries no credentials and anonymous access is turned on, the API server treats the caller as the user system:anonymous in the group system:unauthenticated. Authorization rules then decide what, if anything, such a caller may do. Many clusters keep a tiny number of public paths for health checks, and nothing else.

Authentication also decides one more thing that is easy to miss: it removes the credentials from the request. After this stage, later stages see the identity but no longer see the raw token. This stops secrets from leaking deeper into the system by accident.

The API server wraps its real work in layers of filters, a bit like layers of an onion.

Inside the API server code, a request does not move through the stages by magic. Each stage is a small piece of code called a filter or handler, and the filters are wrapped around each other. The request enters the outermost layer, passes inward through each one, and finally reaches the code that actually reads or writes the resource. The collection of layers is known as the handler chain.

The outer layers handle general safety. One recovers from crashes so that a bug in one request does not bring down the whole server. One attaches a time limit so that ordinary requests cannot hang forever; the default is about a minute for normal requests, while long-running ones such as watch are treated differently. One records the time the request arrived. Others add headers and measure latency. The exact order can shift a little between Kubernetes versions, but the idea stays the same.

Audit recording starts

Once the caller is known, the audit filter begins writing a record of what is happening. The audit system can capture the identity, the time, the verb, the address, and, depending on how it is set up, the request body and the response body. We look at audit in more detail in a later chapter, but it is important to know that it begins early so that even rejected requests leave a trace.

Impersonation

Kubernetes allows one identity to act as another. This is called impersonation. A request can carry special headers saying “I am user A, but please treat me as user B”. The API server first checks that user A is allowed to impersonate user B. If so, the identity is swapped, and every later stage sees user B.

Impersonation is useful for administrators who want to test what a limited user can do without asking for that user’s password. It is also used by some platforms that sit in front of the cluster and act on behalf of their own users. Because it is so powerful, the permission to impersonate should be handed out very carefully.

Even permitted requests must wait their turn when the server is busy, and some turns are more important than others.

An API server can be overwhelmed. A broken script that lists every Pod in a huge cluster in a tight loop can eat up so much capacity that the scheduler and the controllers cannot get their own requests through. When that happens the whole cluster stops reacting, even though nothing is technically broken. To prevent this, the API server includes a feature called API Priority and Fairness, often shortened to APF.

How APF thinks

APF sorts every incoming request into a group using rules called flow schemas. Each flow schema points at a priority level, which is a pool of capacity. Important traffic, such as leader election by control plane components and requests from nodes, is placed in high-priority pools. Ordinary workloads from users and applications go into general pools. Within a pool, requests are separated into flows, for example by user, so that one noisy caller cannot crowd out everyone else.

If there is room, the request runs immediately. If not, it waits in a queue. APF uses a careful method of picking queues so that a noisy caller is likely to fill only its own queues, leaving other callers unaffected. If the queue is full or the wait is too long, the API server rejects the request with status 429 Too Many Requests and a hint about how long to wait before trying again.

flowchart TD
  R["Incoming request"] --> F["Match a flow schema"]
  F --> P["Choose a priority level"]
  P --> Q{"Spare capacity?"}
  Q -->|Yes| X["Run now"]
  Q -->|No| W["Wait in a fair queue"]
  W --> T{"Queue full or too slow?"}
  T -->|No| X
  T -->|Yes| E["Reject with 429 and retry hint"]
        

Older clusters used a simpler approach with two global limits on how many requests may be in flight at the same time, one for reads and one for writes. Those limits still exist as an outer safety net, but APF is the modern and fairer mechanism. Long-running requests such as watches do not hold a slot for their entire life. They are counted mainly while the initial data is being sent.

Knowing who you are is not enough. The server must also decide whether you may do this exact thing.

Authorization takes the identity from authentication and the request summary from earlier, and checks them against a set of rules. It looks at the user, the groups, the verb, the API group, the resource, the subresource, the name and the namespace. If the rules allow the action, the request moves on. If not, the answer is 403 Forbidden.

Authorization modes

A cluster administrator picks one or more authorization modes, and the API server asks them in order. Each mode can say allow, deny, or “no opinion”. The first mode that gives a clear allow or deny decides the outcome. If every mode has no opinion, the request is denied. The most common set is Node plus RBAC.

How RBAC works

RBAC has four main objects. A Role lists permissions inside one namespace. A ClusterRole lists permissions that can apply cluster-wide or to resources that are not in any namespace, such as Nodes. A RoleBinding attaches a role to people or programs inside a namespace. A ClusterRoleBinding attaches a cluster role everywhere.

RBAC is purely additive. There are no deny rules. A caller gets the combined permissions of everything bound to them, and nothing more. This keeps the model simple but means you must be careful about what you add.

Paths that are not resources

Some addresses on the API server are not Kubernetes resources, for example health checks, version information and metrics. These are called non-resource URLs, and RBAC can allow or deny them with separate rules.

One subtle point is that the permission to create Pods is more powerful than it looks. A person who can create Pods may be able to mount Secrets or run privileged containers, which gives indirect access to far more than Pods. When designing roles, think about what a permission allows in practice, not only what it is named.

The request is from a known caller who is allowed to ask, but the cluster still gets to inspect and adjust what is being requested.

Authentication and authorization look at the caller. Admission control looks at the content. It runs only for requests that would change something: create, update, delete, and a few special connect actions such as opening a shell into a Pod. Plain reads never go through admission.

Admission control is made of many small pieces called admission controllers. Some are built into the API server. Others are added by you using webhooks or policy objects. They run in two phases, and the order is important: first the mutating phase, then validation of the object’s shape, then the validating phase.

flowchart LR
  A["Authorized write request"] --> B["Mutating admission: may change the object"]
  B --> C["Object schema validation"]
  C --> D["Validating admission: may only accept or reject"]
  D --> E["Persist to etcd"]
        

The mutating phase

Mutating controllers are allowed to edit the object. They fill in missing values, add labels, inject extra containers, or attach default settings. For example, if you create a PersistentVolumeClaim without naming a storage class, a default-storage-class controller can add the cluster’s default one. The mutating controllers run one after another, so each one sees the changes made by the one before it.

The validating phase

After the object has its final shape and has passed schema checks, validating controllers get their turn. They are not allowed to change anything. They only say yes or no. Because they cannot change the object, they can safely run at the same time, which makes this phase faster. If any one of them says no, the whole request is rejected.

Some built-in admission controllers

Webhooks and policy objects let you add your own rules to the pipeline without changing Kubernetes itself.

Built-in controllers cannot know your company’s special rules. Maybe every Pod must have an owner label, or images may only come from an approved registry, or every container must define memory limits. Kubernetes gives you two main ways to add such rules.

Admission webhooks

A webhook is a small web service that you run. You register it with the API server and tell it which resources and verbs you care about. When a matching request arrives, the API server sends the object to your service inside a message called an AdmissionReview. Your service replies with allow or deny, and, if it is a mutating webhook, a description of the changes to apply.

Because the API server waits for the webhook, your service becomes part of the request path. This brings several settings that matter a great deal:

Built-in policy rules

Running a webhook server just to enforce a simple rule is a lot of effort. Newer Kubernetes versions offer validating admission policies, where the rule is written as an expression inside a Kubernetes object, using a small expression language called CEL. The API server evaluates the rule itself, with no extra network call. This is faster, simpler and more reliable. A similar approach for mutations is maturing in recent versions.

The same object can be written in several API versions, so the server must translate and double-check it.

Kubernetes APIs evolve. A resource may start as a beta version, become stable, and later have an older version retired. The API server can understand several versions of the same resource at once, so that old tools keep working while new tools use new features. To make this possible it uses a trick: it keeps one internal form of every object, and converts to and from the version that the client used.

Decoding and defaulting

When the body of a request arrives, the API server decodes it from JSON, YAML or a compact binary format called protobuf. It then converts it to the internal form and fills in default values for fields that the client did not set. For instance, a Pod without a restart policy gets the standard default. This is part of the reason that an object you read back often looks larger than what you submitted.

Validation

Next comes validation. The API server checks that required fields are present, that names follow the allowed format, that numbers are in range, that fields that cannot change are not being changed, and that fields that belong together make sense together. For custom resources, defined by CustomResourceDefinitions, the API server checks the object against the schema supplied by the definition. If a field is unknown or has the wrong type, the request can be rejected with status 422 Unprocessable Entity. Newer versions can also warn about or reject unknown fields, which catches spelling mistakes in field names that would otherwise be silently ignored.

Storage version and conversion

Although many versions can be served, each resource has one storage version, which is the version actually written into etcd. When the object is read through a different version, the server converts it on the way out. For custom resources that have several versions, the definition can say how conversion happens: either simply by changing the version label, or by calling a conversion webhook that you provide.

After every check has passed, the object is finally written to the cluster’s memory.

The object is now complete and approved. The API server hands it to its storage layer, which turns it into bytes and stores it in etcd. Built-in resources are usually stored in the compact protobuf format, while custom resources are stored as JSON. Each object is saved under a key that looks like a file path, grouped by resource type and namespace, so that the server can find a whole collection quickly.

Why etcd is trusted with this job

etcd is a distributed database. It runs on several machines, commonly three or five, and uses an agreement method called Raft. A write is only considered done when a majority of the etcd members have recorded it. This means the cluster record survives the loss of a machine, and two members can never disagree about what the latest truth is.

Encryption at rest

By default, data in etcd is stored without encryption. The API server can be told to encrypt chosen resources, such as Secrets, before they are written, using a key that is kept in its configuration or in an external key management service. Anyone who steals a backup of etcd then sees only scrambled data. Encryption at rest does not replace access control, but it adds a valuable second wall.

When many clients edit the same object, the server needs a fair way to avoid lost changes.

Every stored object carries a field called resourceVersion. It comes from the revision counter of etcd and changes every time the object changes. Clients should treat it as an opaque label and never do arithmetic on it. Its purpose is to enable optimistic concurrency.

Optimistic concurrency

Suppose two people read the same object, and both change it. When the first person saves, all is fine. When the second person saves, they are still holding the old resourceVersion. The API server notices that the stored version has moved on and refuses the update with status 409 Conflict. The second person must read the fresh object, re-apply their change, and try again. This stops one editor from silently overwriting the other.

Patching and server-side apply

Replacing a whole object is risky when several tools manage different fields. Kubernetes therefore supports patching, which changes only chosen fields. A newer approach, called server-side apply, lets the API server track which tool owns which field. When two tools try to control the same field with different values, the server reports a conflict instead of letting them fight silently.

Deleting is a process

Deleting an object is not always instant. If the object has finalizers, which are labels that mean “someone must clean up first”, the API server only sets a deletion timestamp. Controllers that own the finalizers do their cleanup and then remove their label. When no finalizers remain, the object is truly removed. Pods also have a graceful stop period, so applications can finish what they are doing. Objects can have owners, and when an owner is deleted, a garbage collector removes the objects that depended on it, according to the chosen deletion policy.

The journey ends with a reply to the caller and a permanent note in the logbook.

Once etcd confirms the write, the API server builds a response. For a create, the answer is status 201 Created along with the saved object, now holding generated fields such as the unique identifier, the creation time and the resourceVersion. For a read or update, the answer is 200 OK. For a delete that still has cleanup to do, the answer may be 202 Accepted. If anything went wrong earlier, the response is an error with a small structured body that explains the reason.

Audit logging

In parallel with all of this, the audit system has been recording the request. Audit records are written at defined moments, called stages: when the request is received, when the response starts for long-running requests, when it completes, and if a panic occurs. An audit policy decides how much detail to keep for each kind of request, using four levels.

Audit records can be written to a file on the control plane machine or sent to an external service. They answer the questions that matter after an incident: who deleted that namespace, from which address, and at what time.

Most traffic to the API server is not writing. It is controllers and tools asking what changed.

A cluster contains many programs that need to react quickly to changes. The scheduler needs to know the moment a new unscheduled Pod appears. The kubelet needs to know when a Pod is assigned to its machine. Asking again and again, “anything new?”, would flood the API server. Kubernetes solves this with watch.

List, then watch

A client first lists the current objects and remembers the resourceVersion of that list. It then opens a watch starting from that point. The API server keeps the connection open and streams small messages, each saying that an object was added, modified or deleted. Occasional bookmark messages let the client keep track of its position. If the client falls too far behind and its position is no longer available, the server answers with 410 Gone, and the client must list again.

The watch cache

If every client’s read went all the way to etcd, the database would quickly become the bottleneck. The API server therefore keeps its own in-memory copy of each resource type, kept up to date by watching etcd. This is called the watch cache. Many reads and all watches are served from it. Clients can choose to accept slightly older data from the cache, which is fast, or ask for the very latest data, which may be sent to etcd for confirmation.

Pagination

Listing a huge collection in one answer can use a great deal of memory on both sides. Clients can ask for results in pages by setting a page size and following a continue token. Well-behaved tools do this by default for large lists.

sequenceDiagram
  participant C as Controller
  participant A as API Server
  participant W as Watch Cache
  participant E as etcd
  E->>W: Stream of changes
  C->>A: List Pods
  A->>W: Read current state
  W-->>A: Objects and resource version
  A-->>C: Full list
  C->>A: Watch from that version
  E->>W: New Pod written
  W-->>A: Event added
  A-->>C: Stream event
        

Client libraries package this pattern in a helper often called an informer. An informer lists, watches, stores a local copy, and tells the controller about each change. Because the local copy answers most questions, the controller rarely has to ask the API server again.

Not every request follows the plain read-and-write path. Some open tunnels, and some are handed to other servers.

Subresources

Some resources have extra addresses attached to them, called subresources. A Pod has a status subresource, a scale subresource exists for Deployments, and Pods have logs, exec and port-forward subresources. Each is authorized separately. This allows, for example, a monitoring tool to read logs without being able to open a shell into a container.

Logs, exec and port-forward

Logs and shells are not stored by the API server. When you ask for them, it acts as a middleman. It connects to the kubelet on the right machine and carries the traffic back to you. For interactive sessions the connection is upgraded into a long-lived streaming connection. For this reason the API server must be able to reach every node’s kubelet, and the kubelet must trust the API server.

Custom resources and aggregation

Kubernetes can be extended with your own resource types. There are two ways. With CustomResourceDefinitions, you describe your new type and its schema, and the built-in extensions server begins to serve it with the same authentication, authorization, admission and storage behavior as built-in types. With the aggregation layer, you run your own API server and register it, and the main server forwards matching requests to it after checking identity and permission. The metrics API is a well-known example of this second style.

Inside the main process, three servers are chained like a relay. The aggregator gets the request first. If the address belongs to a registered extension server, it forwards it. If not, it passes the request to the core API server, which serves built-in resources. If that does not know the address either, the request passes to the extensions server for custom resources. If nobody knows it, the answer is 404 Not Found.

flowchart LR
  R["Request"] --> G["Aggregator"]
  G -->|Registered extension| X["Extension API server"]
  G -->|Otherwise| K["Core API server"]
  K -->|Not a built-in type| C["Custom resources server"]
  C -->|Unknown| N["404 Not Found"]
        

Let us follow one real action from your keyboard all the way to a running container.

You run a command to create a Deployment named web that asks for three copies of an application. Here is what happens, in order.

sequenceDiagram
  participant U as You
  participant A as API Server
  participant E as etcd
  participant D as Deployment Controller
  participant S as Scheduler
  participant K as Kubelet
  U->>A: Create Deployment
  A->>A: Authenticate, authorize, admit, validate
  A->>E: Store Deployment
  A-->>U: 201 Created
  A-->>D: Watch event for new Deployment
  D->>A: Create ReplicaSet and Pods
  A->>E: Store Pods
  A-->>S: Watch event for unscheduled Pods
  S->>A: Bind Pods to nodes
  A-->>K: Watch event for assigned Pods
  K->>A: Report Pod status
        

Notice that you received your reply at step four, long before any container was running. The API server’s promise is only that your wish has been safely recorded. Making the wish come true is the work of controllers, and that work happens afterwards, step by step. This is why a command can succeed and yet the application may still be starting for a while.

Each stage of the lifecycle is also a security control, and each can be tightened.

Because the API server is the one door to the cluster, protecting it is the most valuable security work you can do. A useful way to think is to walk the same path a request takes and ask, at each stage, what could go wrong.

If you cannot see how the front door is behaving, you will only learn about problems when users complain.

The API server exposes metrics that describe its own health, in a format that common monitoring systems understand. A few of them tell you most of what you need.

Reading the signals

A sudden rise in 429 responses points to flow control being triggered, so look for a noisy client. Slow writes with healthy reads point to etcd or to a slow admission webhook. Many 401 or 403 responses from one source can mean a broken credential or an intruder trying doors. A rapid growth in the count of one object type, such as Events or Jobs, often means a controller is misbehaving.

Tracing and client-side clues

Recent versions can produce distributed traces of a request as it moves through the server, including calls to etcd and to webhooks. On the client side, turning up the verbosity of kubectl shows the exact addresses, headers and response codes that were used, which is often the quickest way to understand a failure from the user’s point of view.

These mistakes appear again and again in real clusters, and each one harms the request lifecycle in a different way.

A single front door is simple and safe, but it must be built to handle a crowd.

Because the API server is stateless, the usual way to scale is to run several copies behind a load balancer. Reads and watches are served from each copy’s own memory, which spreads the load nicely. Writes still all end up in etcd, so etcd is often the real limit on how fast a very large cluster can change. Fast disks, low network delay between etcd members, and keeping object sizes small all help.

Cluster size also matters. The number of objects, the number of watchers, and the rate of change all affect memory use and latency. Many teams therefore split very large environments into several clusters instead of growing one without limit.

The response code tells you which stage of the journey rejected the request.

Because each stage has its own typical error, you can often locate a problem just by reading the number. This turns a confusing failure into a short list of places to look.

A calm way to investigate

Start with the code. Then look at the message body, because Kubernetes errors usually name the reason and often the exact field. Next, check the audit log to confirm which identity made the request. For 403 errors, an authorization check tool can show whether RBAC allowed the action, which helps you separate permission problems from admission denials. For slow or failing writes, look at webhook and etcd latency metrics before blaming the application.

The same lifecycle powers very different kinds of teams and platforms.

Short answers to questions that beginners often ask about the request lifecycle.

No. Admission runs only for requests that create, change or delete objects, and for special connect actions such as opening a shell. Reads still pass authentication and authorization.

Authentication decides who you are. Authorization decides what that identity is allowed to do. A person can be authenticated perfectly and still be forbidden.

Validators should judge the final form of the object. If mutation ran afterwards, a change could slip in that no validator had seen.

Technically possible for administrators, but it bypasses authentication, authorization, admission, validation and audit, and can corrupt the cluster. It should not be part of normal operations.

They keep running, because the kubelet on each machine continues to manage the containers it already has. New changes, scaling and replacements cannot happen until the API server returns.

Success means the object was stored. Scheduling, image downloads and container start happen afterwards through controllers and kubelets, and can take time or fail separately.

Someone else changed the object after you read it, so your copy is out of date. Read it again, reapply your change and retry. Server-side apply can help when several tools manage the same object.

A dry run goes through all the checks without saving the object. Webhooks that declare side effects must not act on dry runs, which is why that declaration matters.

Each resource has a chosen storage version. Requests in other versions are converted to it before saving, and converted back when read.

Watch delivers changes the moment they happen and uses far fewer resources than repeated questions. It makes the whole system react quickly and keeps the API server calm.

RBAC has no deny rules, it only adds allowed actions. To block something, you remove the permission, or use admission policy or a webhook authorizer that can deny.

Every action in Kubernetes begins as a request to the API server, and every request follows the same disciplined path. A secure connection is made, identity is proven, the request is placed fairly in line, permission is checked, the content is adjusted and judged by admission rules, the object is validated and converted, and finally it is saved in etcd and announced to every watcher. Understanding this path turns Kubernetes from a mysterious system into a predictable one, because each error, delay and security control maps to a specific stage you can inspect.