A cluster full of teams needs rules that are enforced automatically, not rules that live in a wiki.

The problem with trust alone

Kubernetes is very permissive by design. If a person has the right to create a Pod, the cluster will happily accept almost any Pod they describe. It does not care whether the container image comes from a trusted registry, whether the Pod asks for full root powers, or whether the team forgot to add a label that finance uses to split the monthly bill. Access control answers the question “who are you and what are you allowed to do”. It does not answer “is the thing you are asking for a good idea”.

In a small cluster with three engineers, people simply agree on conventions. In a large company with hundreds of engineers, conventions get forgotten. One missing resource limit can let a single application eat all the memory on a machine. One privileged container can become a doorway for an attacker. Reviewing every change by hand does not scale, so the rules need to be enforced by software at the moment a request arrives.

Two jobs, one doorway

Webhooks in Kubernetes do two related jobs. The first job is to reject requests that break your rules. The second job is to change requests so they follow your rules automatically, for example by adding a missing label or a default setting. Both jobs happen at the same doorway, which is the Kubernetes API server, and both happen before anything is written to the cluster’s database.

To understand webhooks you first need to know where they sit in the line.

Every change goes through the API server

The API server is the front door of the Kubernetes control plane. When someone runs a command to create a Deployment, or when a tool applies a file, or when another program calls the Kubernetes API, the request lands on the API server first. Nothing talks to the cluster’s database (called etcd) directly except the API server itself. This single front door is what makes central rule enforcement possible.

Each request passes through a fixed series of stages. Understanding the order is the most important thing in this whole tutorial, because most confusing webhook behavior comes from not knowing which stage a webhook lives in.

flowchart LR
  A[Client request] --> B[Authentication]
  B --> C[Authorization]
  C --> D[Mutating admission]
  D --> E[Schema validation]
  E --> F[Validating admission]
  F --> G[Saved in etcd]
        

Why mutating comes before validating

The order is not an accident. If validation ran first, a policy could approve an object that a later mutation then changed into something forbidden. By letting every mutation finish first and only then running validation, Kubernetes makes sure the validators look at the final version of the object, the exact thing that will be stored. Validators get the last word.

Webhooks are one member of a larger family called admission controllers.

What an admission controller is

An admission controller is a piece of code inside the API server that can accept, reject, or modify a request after it has been authenticated and authorized. Kubernetes ships with many of them already compiled in. Some fill in default values, some enforce limits, and some protect the cluster from dangerous settings.

Why “dynamic” matters

The built-in controllers are fixed. Changing them means rebuilding and restarting the API server, which ordinary cluster users and even most platform teams cannot do, especially on managed cloud services. Dynamic admission control solves this. The two webhook plugins are already built in and switched on, and they read a list of external addresses from ordinary Kubernetes objects. You add, change, or remove rules at runtime by editing those objects, with no restart.

This is the reason webhooks became the main extension point for policy in Kubernetes. They let a platform team add organization specific rules to a cluster that they do not fully control, using nothing more than a service running inside the cluster and a configuration object.

Strip away the jargon and a webhook is just a web server that answers questions from the API server.

An ordinary HTTPS service

A webhook is a small program that listens on a network address and speaks HTTPS. When the API server wants an opinion, it sends a request containing a description of the object being created or changed. The webhook examines the description and sends back an answer. That is all. There is no special Kubernetes magic inside the program. It can be written in any language, and it can run anywhere the API server can reach.

Most teams run webhooks as a Deployment inside the cluster, fronted by a Service. The configuration object then points to that Service by name. Some teams instead point to an external address, which is useful when the logic lives outside the cluster, but this brings extra network and trust concerns that we will cover later.

The conversation in one picture

sequenceDiagram
  participant U as User or tool
  participant A as API server
  participant M as Mutating webhook
  participant V as Validating webhook
  participant E as etcd
  U->>A: Create object
  A->>M: AdmissionReview request
  M-->>A: Allowed with patch
  A->>A: Schema validation
  A->>V: AdmissionReview request
  V-->>A: Allowed or denied
  A->>E: Store object
  A-->>U: Success or error
        

Notice that the webhook never talks to the user directly. It only answers the API server. If the webhook says no, the API server turns that refusal into an error message that the user sees in their terminal or tool.

A mutating webhook is allowed to rewrite the object before it is saved.

What mutation means

To mutate something means to change it. A mutating webhook receives the object, decides it needs adjustment, and replies with a list of edits. The API server applies those edits to the object and carries on. The user who made the request often never notices that their object was altered, which is both the power and the danger of the technique.

Typical things mutating webhooks do

The two rules of good mutation

The first rule is that a mutation must be idempotent. That is a fancy word meaning “doing it twice gives the same result as doing it once”. If a webhook adds a sidecar every time it is called, and it is called twice, the Pod ends up with two sidecars. A well written mutating webhook first checks whether its change is already present and only adds it when missing.

The second rule is that a mutation must be predictable. Surprising edits make debugging painful, because the object a user applied is not the object that ended up in the cluster. Good teams document every mutation, keep the list short, and expose a clear marker, such as an annotation, showing that a webhook touched the object.

A validating webhook never changes anything. It only says yes or no.

Pure judgement

A validating webhook receives the final version of the object and answers with an allow or a deny. When it denies, it can include a human readable message explaining why. The API server then rejects the request and shows that message to the person who made it. Because validators cannot edit the object, they are simpler to reason about and safer to run than mutating webhooks.

Typical things validating webhooks check

Good error messages are part of the product

When a validating webhook rejects a request, the message it returns becomes the user’s whole experience of your policy. A vague message such as “denied by policy” leaves developers frustrated and sends them to the platform team for help. A good message names the rule, explains the reason, and says exactly how to fix the object. Platform teams that invest in clear messages see far fewer support requests.

Validating webhooks can also return warnings. A warning does not block the request. It appears in the user’s terminal as a gentle note, which is useful when you are introducing a new rule and want people to see it before you start enforcing it.

The two types look alike on the surface, but they behave quite differently.

Why running validators in parallel matters

Because validating webhooks only give opinions, the API server asks all of them at once and waits for every answer. If any single one says no, the request is rejected. This design keeps the total waiting time close to the slowest webhook instead of the sum of all of them, which is an important performance property when many policies are installed.

Mutating webhooks cannot work this way. Each one must see the result of the previous one, so they run in a line. That is one more reason to keep the number of mutating webhooks small.

A webhook only gets called once a configuration object announces it.

Two configuration kinds

Kubernetes has two object kinds for this purpose. One is named MutatingWebhookConfiguration and the other is named ValidatingWebhookConfiguration. Each holds a list of webhook entries. Each entry says what kinds of requests it wants to see, where to send them, and how the API server should behave if the webhook is slow or unavailable. These objects are cluster wide, which means they are not owned by any one namespace.

Think of the configuration as the sign-up sheet at the security desk. It tells the guard which specialist to call, for which kinds of visitors, and what to do if the specialist does not pick up the phone.

The fields that matter most

How rules narrow the traffic

The rules section is your first and most important line of defense against unnecessary load. Each rule names operations, API groups, API versions, and resource names. It can also name subresources, such as the status or scale of an object, and it can say whether it applies to namespaced objects, cluster wide objects, or both. A webhook that only cares about Pods should say so explicitly instead of asking to see every object in the cluster.

Wildcards are allowed, but they are dangerous. A rule that matches everything will send every change in the cluster to your webhook, including busy internal objects that change many times per second. Always list only the resources you truly need.

Requests and responses travel inside one shared envelope called AdmissionReview.

What the webhook receives

The API server sends a JSON message shaped as an AdmissionReview object. Inside the request part, the webhook finds everything it needs to make a decision. Here are the pieces that matter most in daily work.

What the webhook must answer

The response carries the same uid it received, plus an allowed flag set to true or false. When the answer is no, the webhook should add a status with a code and a clear message. For mutating webhooks, the response may also carry a patch and a patch type. The only patch type Kubernetes currently accepts is JSON Patch. Webhooks may also add a list of warnings and a set of audit annotations, which are notes that appear in the cluster’s audit log for that request.

flowchart TD
  A[Request arrives at webhook] --> B{Understand the object?}
  B -- No --> C[Return denied with clear message]
  B -- Yes --> D{Follows the rules?}
  D -- No --> C
  D -- Yes --> E{Needs changes?}
  E -- No --> F[Return allowed]
  E -- Yes --> G[Return allowed plus patch]
        

A mutating webhook does not send back a whole new object. It sends a short list of edits.

JSON Patch in plain words

JSON Patch is a small standard for describing changes to a JSON document. Each edit in the list names an action and a location. The common actions are add, replace, and remove. The location is a path that walks down through the object, like a street address that goes from country to city to house number.

For example, adding a label to an object means adding a new entry at the path that points to the labels of that object. Removing a setting means pointing at that setting’s path and asking for removal. The webhook packs the list of edits into text, encodes it in a transport friendly format called base64, and places it in the response.

Where patches go wrong

Validating webhooks must never return patches. If one does, the API server ignores or rejects it, depending on the version, and the confusion this causes in practice is entirely avoidable.

When several webhooks touch the same object, small details start to matter.

Why order is unreliable

In a real cluster you may have a service mesh webhook, a secrets webhook, a cost labeling webhook, and a few others, all wanting to modify the same Pod. The API server runs them one by one, but the order in which they are called is not guaranteed to stay the same, and you should never build logic that depends on it.

This creates a puzzle. Imagine webhook A runs first and adds a container. Webhook B runs second and adds a setting to every container, including the one A just added. Now flip the order. B runs first and misses A’s container. The final object differs depending on order. That is the exact problem the reinvocation policy exists to solve.

Reinvocation

A mutating webhook can set its reinvocation policy to “IfNeeded”. This asks the API server to call it a second time if any other mutating webhook changed the object after the first call. The webhook then gets another chance to react to the newly modified object. The default is “Never”, which means the webhook is called once and that is the end of it.

Reinvocation is the reason idempotency is non-negotiable. A webhook that gets called twice must produce the same final result as if it had been called once. If it blindly appends a container every time, the second call makes a mess.

Because webhooks can change or block everything, they must be treated as part of the control plane’s trust boundary.

Encrypted and verified

The API server only talks to webhooks over HTTPS. This protects the contents of requests, which can include sensitive data such as the specification of workloads, and it makes sure the API server is really talking to your webhook and not an impostor. To check identity, the API server needs to trust the certificate the webhook presents. The configuration object carries a certificate authority bundle for exactly this purpose, and the webhook’s certificate must be signed by that authority and must be valid for the address used to reach it.

For a webhook reached through an in-cluster Service, that means the certificate must name the Service address in the form used inside the cluster. Mismatched names are among the most common reasons a freshly installed webhook fails with a certificate error.

Certificate lifecycle

Certificates expire. A webhook certificate that quietly expires at three in the morning can break every deployment in the cluster, because the API server can no longer verify the webhook and the failure policy kicks in. Mature teams automate the entire lifecycle of these certificates, either with a certificate management tool running in the cluster or with the webhook’s own logic to rotate its certificates and refresh the authority information in the configuration.

Who can call the webhook

By default the webhook accepts calls from anyone who can reach it over the network. In many setups that is acceptable, because the content is not secret and the decision is advisory. However, a hardened environment should restrict access. Network policies can limit incoming traffic to the control plane’s address range, and the API server can be configured with credentials it presents to webhooks, so the webhook can verify that the caller really is the API server.

Who can change the configuration

The webhook configuration objects are extremely powerful. Whoever can edit them can redirect every request in the cluster to a server of their choosing, read the contents, change them, or block everything. Access to create or modify these objects should be limited to a very small group of trusted administrators and ideally changed only through reviewed, automated pipelines.

A webhook sits on the critical path of every matching request, so its health becomes the cluster’s health.

Fail closed or fail open

The failure policy answers a hard question: what should the cluster do when the webhook cannot be reached, returns an error, or takes too long? Setting “Fail” is called failing closed. Requests are refused until the webhook returns. This is safe from a policy point of view because nothing slips past unchecked, but it makes the webhook a single point of failure for the cluster. Setting “Ignore” is called failing open. Requests are allowed through when the webhook is broken. The cluster keeps working, but your rules are silently not enforced during the outage.

There is no universally correct answer. Choose based on the harm of a missed check compared with the harm of a blocked cluster, and decide per webhook, not once for all of them.

The self-blocking deadlock

Here is the nastiest failure story in the webhook world. A webhook runs as Pods inside the cluster, and its configuration says to intercept all Pod creations and fail closed. One day the webhook’s Pods crash. Kubernetes tries to create replacement Pods. The API server calls the webhook to approve the new Pods. But the webhook has no running Pods, so the call fails. The failure policy rejects the request. The replacement Pods can never start. The webhook cannot recover because it is blocking its own recovery.

flowchart TD
  A[Webhook Pods crash] --> B[Controller creates replacement Pods]
  B --> C[API server calls the webhook]
  C --> D[No webhook Pod is running]
  D --> E[Call fails and policy is Fail]
  E --> F[New Pod creation is rejected]
  F --> B
        

The cure is to design the loop away. Exclude the webhook’s own namespace from its rules using a namespace selector. Exclude the system namespaces that run core cluster components. Run several replicas of the webhook spread across different machines and zones. Use a disruption budget so maintenance cannot take all of them down at once. And keep a documented emergency procedure for removing or relaxing the configuration object quickly.

Timeouts and latency

Every call to a webhook adds waiting time to the user’s request. If a webhook takes five seconds, every matching operation takes five seconds longer. If several webhooks are in the path, the delays stack up for mutating ones and overlap for validating ones. Aim for webhook answers in tens of milliseconds, set the timeout as low as is safe, and never let a webhook make slow outside calls in the middle of its decision.

Also remember that the total time the API server will wait for all webhooks on a single request is bounded, so a slow chain of webhooks can make the whole request fail even though no individual webhook crossed its own limit.

The cheapest webhook call is the one that never happens.

Narrow the scope on purpose

Every request the API server forwards costs network time, uses CPU in your webhook, and adds a chance of failure. The best way to improve both speed and reliability is to send fewer requests. Use these filters in layers.

Avoid noisy resources

Some Kubernetes objects change constantly. Leases used for leader election are renewed every few seconds. Events pour in during any incident. Node status updates arrive continuously. A webhook that intercepts these will be flooded, and a flooded webhook slows or breaks things for the whole cluster. If you must see such objects, make the webhook extremely fast and extremely simple.

A webhook should answer questions, not take actions in the outside world.

What a side effect is

A side effect is any change a webhook makes outside the object it is judging. Examples include writing a record to an external database, reserving an IP address, charging a quota counter, or sending a notification. Side effects are risky because the webhook answers before the object is really saved. If a later webhook rejects the request, or the save fails, the side effect has already happened even though the object never existed.

Dry run requests

Kubernetes lets users send a request in dry run mode. The whole path executes, including admission, but nothing is saved at the end. This is a handy way to ask “would this be accepted?”. Because dry runs pass through webhooks, a webhook with side effects would corrupt its external state on every trial. To make this safe, configuration objects must declare their side effect behavior. A webhook declared as having no side effects is called during dry runs normally. A webhook declared as having no side effects only during dry runs is called but must honor the dry run flag. A webhook that cannot promise either is not allowed in the current API version at all.

Webhooks power many tools that platform teams install without ever writing one themselves.

The pattern in all of these is the same. A shared concern that every team would otherwise implement differently is pulled into one place and enforced at the doorway. The application developers get consistency without extra effort, and the platform team gets enforcement without chasing people.

Webhooks are powerful, but they are not always the lightest tool for the job.

In-process policies

Running and securing a separate web service is real work. Kubernetes has therefore added policy features that run entirely inside the API server, with no external service to maintain. The most established of these is ValidatingAdmissionPolicy, which became stable in Kubernetes 1.30. It lets administrators write validation rules in a compact expression language called CEL and bind them to resources through ordinary configuration objects. Since the rules are evaluated inside the API server, there is no network hop, no certificate to rotate, and no webhook to crash.

A matching feature for mutation, called MutatingAdmissionPolicy, has been progressing through the maturity stages in recent releases. Check the documentation for your cluster version to learn whether it is available and enabled before planning around it.

How to choose

Start by asking whether the rule can be written as a simple check on the fields of the object. If yes, an in-process policy is usually the best first choice. If the rule needs to compare against other objects, call a company directory, or perform complex logic, a webhook or a policy engine is more fitting. If you only need to detect problems and not prevent them, a controller or a periodic audit may be enough.

You cannot protect what you cannot see, and webhooks sit in a place where failures are costly.

Signals from the API server

The API server publishes metrics about every webhook it calls. These show how many requests were sent, how many were rejected, how long the answers took, and how many times a failure was ignored because the policy was set to fail open. Watching these numbers gives early warning that a webhook is slowing down, rejecting unexpectedly, or silently failing open.

Signals from the webhook itself

Your webhook should log each decision with the object name, namespace, operation, requesting user, decision, and reason. It should expose its own metrics for request rate, error rate, and response time, and these should feed dashboards and alerts. Include the request identifier so a log line can be tied back to a specific API call.

Audit trails

The cluster’s audit log records requests and, with audit annotations, can also record why a webhook made its decision. Together with the webhook’s own logs, this answers the common question “who tried to do what, and why was it blocked or changed?”. Auditors and security teams value this record highly.

Teams that run webhooks happily for years tend to follow the same handful of habits.

Patterns to avoid

A short checklist that saves a lot of late night debugging.

A practical checklist

When deployments suddenly fail, work from the outside in.

Read the error message first

Most webhook problems announce themselves in the error text shown to the user. The message usually names the webhook that was involved and describes whether the failure was a refusal, a timeout, a connection problem, or a certificate problem. Resist the urge to start changing things before understanding which of these you are dealing with.

A calm recovery routine

No. Admission webhooks run only for requests that change state, which means create, update, delete, and special connect actions. Reading, listing, and watching bypass them entirely.

Mutating webhooks run first. After they finish, the object is checked against its schema, and only then do the validating webhooks run on the final version.

Not as a single registration. Kubernetes has separate configuration kinds for each. However, one program can serve both purposes by registering itself under both configuration kinds with different addresses.

A delete request carries the existing object as the old version and no new version. A validating webhook can allow or refuse the delete. Changing an object being deleted is not a meaningful use of mutation.

They are applied one after the other, so the later edit wins. Because order is not guaranteed, avoid having two webhooks own the same field. Give each field a single owner.

Fail closed for strict security rules where unchecked objects are unacceptable, and fail open for convenience features. In both cases invest in high availability so the choice rarely matters.

Yes, if your rules match them. That is why excluding system namespaces is so important. A fail closed webhook that matches system Pods can prevent core components from starting.

No. The two webhook configuration kinds are exempt, so administrators can always repair a broken setup.

Usually network reachability. In managed or private clusters, the control plane must be allowed through firewalls to the port the webhook listens on. Certificate names and the listening port are the other usual suspects.

No. A webhook answers once at request time and then forgets. A controller keeps watching saved objects and keeps working toward a goal. Operators often include both.

Often less than before. Simple field checks fit in-process policies well. Webhooks remain valuable when logic needs outside data, comparison across objects, or capabilities the expression language does not offer.

Use warnings instead of denials, send dry run requests, or apply the webhook only to a labeled test namespace first. Watch the metrics and logs before widening the scope.

Admission webhooks give Kubernetes administrators a precise and flexible way to enforce standards and automate repetitive work at the exact moment every change arrives. Mutating webhooks shape objects so they follow your conventions. Validating webhooks act as the final referee over the finished object. Together they let a platform team keep a large, shared cluster safe, consistent, and cost aware without relying on memory or manual reviews. That power comes with responsibility. Because webhooks sit on the critical path of every matching request, they must be narrowly scoped, securely certified, highly available, well monitored, and carefully rolled out. Treated with that discipline, they are among the most valuable tools in a Kubernetes platform.