Before learning the tools, we need to understand the gap they fill. This chapter explains what Kubernetes already protects, and what it quietly leaves open.

What Kubernetes is, in one paragraph

Kubernetes is a system that runs software packaged in containers across many computers. You tell it what you want, for example “run five copies of my shopping website,” and it makes that happen. You describe what you want in files called manifests, and you send them to the Kubernetes API server, which is the front door of the cluster. Everything that happens in a cluster starts as a request to that front door.

What RBAC does and does not do

Kubernetes already has a built-in permission system called RBAC, which stands for Role-Based Access Control. RBAC answers one question: “Is this person allowed to perform this action on this kind of object?” For example, it can say that the payments team may create Pods in the payments namespace. A namespace is simply a named section of the cluster, like one floor of our apartment building.

But RBAC only looks at the kind of action and the kind of object. It does not look inside the object. If the payments team is allowed to create Pods, RBAC accepts a safe Pod and a dangerous Pod with the same enthusiasm. It never asks whether the Pod runs as the root user, whether it uses the “latest” image tag, or whether it pulls images from a trusted registry.

The kinds of mistakes policies prevent

Real clusters collect the same mistakes again and again. A short list of the most common ones shows why rules matter:

• Privileged containers. A container that runs with extra powers can often break out and take over the machine it runs on.
• Untrusted images. Pulling software from a random public registry may bring in malware or unpatched flaws.
• Missing limits. A container without CPU and memory limits can eat the whole machine and starve its neighbors.
• Open doors to the internet. A Service of type LoadBalancer can expose an internal tool to the world with one line of configuration.
• Missing labels. Without an owner label, nobody knows who to call at 3 a.m. when a workload misbehaves, and cost reports cannot be split by team.

The scale problem

You can review ten manifests by hand. You cannot review ten thousand. Large companies run hundreds of clusters, with thousands of developers changing things all day. Human review does not scale, and checklists in a wiki are quickly forgotten. The only dependable approach is to teach the cluster itself to check every request automatically, every time, in the same way. That idea is called policy enforcement, and the rest of this guide is about doing it well.

Here we meet the first of our two tools. OPA is the “brain” that makes decisions. It is useful far beyond Kubernetes, which is why it is worth understanding on its own.

What “policy as code” means

A policy is a rule written down, such as “every container image must come from our company registry.” Policy as code means you write that rule in a text file that a computer can read, store it in Git like any other source code, and let machines enforce it. This gives you the same benefits developers already enjoy with software: you can review changes, see who changed what and when, test rules before using them, and roll back a bad rule in seconds.

What OPA is

Open Policy Agent, usually shortened to OPA and pronounced “oh-pa,” is an open source, general-purpose policy engine. It is written in the Go language and became a graduated project of the Cloud Native Computing Foundation (CNCF) in early 2021, which is the foundation’s highest maturity level. “General-purpose” means OPA was not built only for Kubernetes. Companies use it to decide whether an API call is allowed, whether a Terraform plan is safe, whether a user may open a record in an application, and much more.

OPA works like a very fast question-answering machine. You hand it three things and it returns an answer:

Separating deciding from enforcing

The most important design idea in OPA is that it only decides. It never blocks anything by itself. Some other system must ask OPA for a decision and then act on it. This separation is powerful because you can change your rules without touching the application that enforces them.

Who uses OPA in production

Publicly listed adopters of OPA include Netflix, Capital One, Goldman Sachs and Pinterest, among many others. They use it for authorizing service-to-service calls, controlling who can see what data, and checking cloud infrastructure and Kubernetes configuration. The common thread is that one engine and one language can serve many different places, so rules can be shared and taught once.

To enforce rules, we need a place in Kubernetes where every request can be inspected. That place already exists, and it is called admission control.

The journey of a request

When you run a command such as kubectl apply, nothing is created immediately. Your request travels through a series of steps inside the API server. First, the server checks who you are (authentication). Next, it checks what you are allowed to do (authorization, usually RBAC). Then comes the stage that matters most for us: admission control. Only after all of those checks pass does Kubernetes save the object into its database, which is called etcd.

flowchart LR
  A["kubectl or CI/CD pipeline"] --> B["API Server"]
  B --> C["Authentication"]
  C --> D["Authorization - RBAC"]
  D --> E["Mutating Admission"]
  E --> F["Schema Validation"]
  F --> G["Validating Admission - Gatekeeper webhook"]
  G --> H[("etcd storage")]
  G -.->|"deny"| X["Error returned to the user"]

Two kinds of admission

Admission has two phases. In the mutating phase, controllers may change the request. For example, a controller may automatically add a default label or inject a helper container. In the validating phase, controllers can only say yes or no. They cannot change anything. Mutating runs first, so validating controllers see the final shape of the object. This order matters: a rule that checks for a label should run after the system that adds the label has finished its work.

Webhooks: asking an outside expert

Kubernetes ships with a few built-in admission controllers, such as one that applies resource quotas and one that enforces the Pod Security Standards. But no built-in list can cover every company’s rules. So Kubernetes offers dynamic admission webhooks. A webhook is simply a web address that the API server calls. For every matching request, the API server sends a package called an AdmissionReview, containing the object, and waits for a reply that says “allowed” or “denied,” with an optional message.

Important limits to remember

• Admission runs only when something is created, updated or deleted. It does not run when someone reads data.
• Admission never looks back. Objects that already existed before a rule was written are not rechecked at admission time. This gap is the reason Gatekeeper also has an audit feature, covered in Chapter 11.
• Each webhook has a failure policy. If the webhook server is down, “Fail” rejects the request, and “Ignore” lets it through. This choice has big consequences for both safety and availability, and we discuss it in Chapter 13.

Gatekeeper is the bridge between OPA’s general decision engine and the specific needs of a Kubernetes cluster.

The early days: OPA with a sidecar

The first way to use OPA in Kubernetes was to run it next to a helper program called kube-mgmt. Policies were stored in ConfigMaps, which are simple Kubernetes objects that hold text. This worked, but it had real problems. There was no standard way to pass settings into a rule, so teams copied and pasted near-identical rules. There was no built-in report of which existing objects already broke the rules. Error messages and status were hard to see. Managing many rules felt like managing loose pieces of paper.

Gatekeeper version 3

Gatekeeper was created to fix these problems. It began as a joint effort between engineers from Google, Microsoft, Red Hat and Styra (the company that created OPA), and it lives as a sub-project of OPA in the CNCF. Version 3 made policy a native part of Kubernetes. Instead of ConfigMaps, rules and their settings are stored as Custom Resource Definitions (CRDs). A CRD is a way to teach Kubernetes about a brand-new kind of object. After Gatekeeper is installed, your cluster understands new object kinds such as ConstraintTemplate, and you manage them with the same kubectl commands you already know.

What Gatekeeper gives you

Now we open the box. Gatekeeper is made of a few small parts, and each part has one clear job.

The big picture

Gatekeeper installs itself into its own namespace, usually called gatekeeper-system. Inside, it runs two main groups of Pods. One group handles live requests from the API server. The other group performs the periodic scans. Both groups carry a copy of the OPA engine inside their own program, so there is no separate OPA server to manage and no extra network hop between Gatekeeper and its rule engine.

flowchart TB
  subgraph Authors["Policy Authors"]
    Dev["Platform Engineer"]
    Git["GitOps Repository"]
  end
  subgraph Cluster["Kubernetes Cluster"]
    API["API Server"]
    ETCD[("etcd")]
    subgraph GK["gatekeeper-system namespace"]
      CM["Controller Manager Pods - webhook server, controllers, OPA engine"]
      AUD["Audit Pod - OPA engine"]
    end
  end
  PROM["Prometheus"]
  Dev --> Git
  Git -->|"apply templates and constraints"| API
  API  ETCD
  API -->|"AdmissionReview over HTTPS"| CM
  CM -->|"allow or deny"| API
  CM -->|"watch templates and constraints"| API
  AUD -->|"list existing resources"| API
  AUD -->|"write violations to constraint status"| API
  PROM -->|"scrape metrics"| CM
  PROM -->|"scrape metrics"| AUD

Component by component

Why two sets of Pods?

The live path and the scanning path have very different needs. The live path must answer in milliseconds and must be highly available, so it runs several replicas that can share the load. The audit path is heavy, because it reads a lot of objects, but it can take its time. Keeping them separate means a slow scan can never make a live request wait.

This is the heart of how you use Gatekeeper day to day. It is a two-layer idea, and once you see it, everything else falls into place.

Layer one: the ConstraintTemplate

A ConstraintTemplate is a reusable recipe for a type of rule. It contains three things. First, the logic itself, written in Rego. Second, a description of what settings the rule accepts, called parameters. Third, the name of a new object kind that this template will create. For example, a template might define a rule type for “required labels” that accepts a list of label names as a setting.

When you apply a template, something clever happens. Gatekeeper reads it, compiles the logic, and automatically registers a brand-new CRD with the kind name you chose. If your template says its kind is K8sRequiredLabels, your cluster now understands that kind. By convention these names are written in CamelCase and often start with K8s.

Layer two: the Constraint

A Constraint is a single use of a template. It says where the rule applies and which settings to use. A constraint has three important parts:

• Match. Which objects the rule applies to: which kinds (such as Pods or Namespaces), which namespaces, and optionally which labels.
• Parameters. The settings passed to the template, such as “the required labels are owner and cost-center.”
• Enforcement action. What to do when a violation is found: deny, warn, or just record it. Chapter 10 covers this in detail.

A worked example in words

Suppose your finance team needs every namespace to carry an owner label and a cost-center label so cloud bills can be split by team. You would take the required-labels template, either from the public library or written yourself. Then you would create one constraint of that kind. Its match section would say “apply to Namespaces.” Its parameters would list the two label names. Its enforcement action would start as record-only. From that moment on, any new namespace missing those labels is reported, and once you switch the action to deny, it is blocked with a message that names the missing labels.

Rego is the language inside templates. You do not need to be a programmer to understand how it thinks, and this chapter explains the ideas without any code.

A different way of thinking

Most programming languages are written as instructions: do this, then do that. Rego is declarative, which means you describe the conditions under which something is true and let the engine figure out how to check them. A Rego rule reads like a sentence: “A violation exists when the object is a Pod, and one of its containers uses an image that does not start with an approved prefix.”

The pieces you must know

An important safety property

In Gatekeeper, a rule that cannot find the field it needs simply produces no result. This is a common source of surprise. If a rule looks for a field that is missing from the object, the rule is silent, not angry. A policy author must think carefully about the case where the field is absent, because “missing” often means “unsafe.” For example, a container that has no security settings at all is usually more dangerous than one that sets them explicitly.

Let us follow a single Pod creation request from the developer’s keyboard to the final yes or no, step by step.

sequenceDiagram
  autonumber
  participant U as Developer
  participant A as API Server
  participant W as Gatekeeper Webhook
  participant O as OPA Engine
  participant E as etcd
  U->>A: kubectl apply Pod
  A->>A: Authenticate and authorize
  A->>W: AdmissionReview request
  W->>W: Find constraints that match
  W->>O: Evaluate templates with object and parameters
  O-->>W: List of violations
  alt Violations found with deny action
    W-->>A: allowed false with message
    A-->>U: Error: request denied
  else No violations
    W-->>A: allowed true
    A->>E: Persist object
    A-->>U: pod created
  end

Step by step

Why it is fast

Three design choices keep decisions quick. The OPA engine is embedded, so there is no extra network call. Templates are compiled once when applied, not on every request. And constraint matching filters out most rules before any Rego logic runs. In well-sized clusters, a typical decision takes only a few milliseconds up to a few tens of milliseconds, depending on how many policies match and how complex they are.

A rule has a life of its own: it is written, tested, reviewed, delivered, enforced, and finally improved. Seeing the whole loop helps you build a process that is safe and repeatable.

flowchart LR
  A["Author writes template and constraint"] --> B["Test locally with the gator CLI"]
  B --> C["Pull request and peer review"]
  C --> D["Merge to main branch"]
  D --> E["GitOps tool syncs to cluster"]
  E --> F["Template is compiled and constraint kind registered"]
  F --> G["Constraint created in dry-run mode"]
  G --> H["Webhook and audit evaluate"]
  H --> I["Violations reviewed by owners"]
  I --> J["Switch to warn then deny"]
  J --> A

Where the data lives

Gatekeeper keeps its important information in three places, and it helps to know which is which:

• In etcd, as Kubernetes objects. Templates, constraints, and configuration are normal objects stored by the API server. They survive restarts, and they can be backed up like anything else in the cluster.
• In memory, inside each Gatekeeper Pod. Compiled rules and any replicated cluster data live in memory so that evaluation is fast. If a Pod restarts, it rebuilds this memory from the API server.
• In status fields and metrics. Results of audits are written onto the constraint objects, and counters are exposed for monitoring tools.

What happens when a policy changes

When you edit a template, each Gatekeeper Pod notices the change by watching the API server, recompiles the logic, and swaps in the new version. If the new logic has an error, the template reports the problem in its status and the old version is not silently replaced by a broken one. When you change a constraint, the very next admission request uses the new settings. There is no restart and no downtime, which is one of the big wins over older approaches.

A good rule applied in the wrong place or at the wrong strength causes outages. This chapter shows how to control how hard a rule pushes and where it reaches.

Enforcement actions

Newer versions of Gatekeeper also allow different actions at different enforcement points, for example deny during live admission but only record during audit. This is useful when you want strict control on new changes but do not want old objects to look like emergencies.

Scoping with match

Every constraint has a match section that narrows its reach. The main options are:

Exemptions: the safety valve

Some namespaces must never be blocked by your own rules. System namespaces such as kube-system run the components that keep the cluster alive. If a policy accidentally blocks one of those components from restarting, the cluster can enter a state that is hard to repair. Gatekeeper supports excluding namespaces in three ways: per constraint, through a cluster-wide configuration object that can exempt namespaces from the webhook, audit, or other processes, and through a special label that tells the webhook to ignore a namespace entirely.

Admission only guards the front door. Audit walks the halls looking at what is already inside, and replicated data lets rules compare one object with others.

How audit works

The audit Pod wakes up on a timer, with a default of every 60 seconds. It lists the objects in the cluster, checks each one against every matching constraint, and writes any violations onto the constraint object itself. You can then read the violations with a simple kubectl describe on the constraint. By default each constraint stores up to 20 violations, which keeps the object small, while the total violation count is still shown in metrics.

flowchart LR
  T["Timer fires every audit interval"] --> L["Audit lists existing resources"]
  L --> M["Match against each constraint"]
  M --> V["Evaluate with the OPA engine"]
  V --> S["Write violations to constraint status"]
  S --> P["Export metrics and logs"]
  P --> T

Why audit matters

There are three situations where audit is the only thing that catches a problem. First, the object existed before the rule did. Second, the rule was in dry-run or warn mode when the object was created. Third, the webhook was unavailable and failing open, so the request slipped through. Audit is your safety net for all three, and it is also the feature that compliance teams love, because it gives an always-fresh report of how the cluster measures against its rules.

Replicated data for referential rules

Some rules cannot be decided by looking at one object alone. For example, “no two Ingress objects may use the same hostname” requires knowing about all the other Ingress objects. To support this, Gatekeeper has a configuration object where you list which kinds of objects should be copied into the engine’s memory. Rules can then look through this copy, called the inventory, while making a decision.

Gatekeeper can do more than say no. It can quietly fix common mistakes, understand objects that create other objects, and consult outside systems.

Mutation: fixing instead of blocking

Blocking a developer for forgetting something small is annoying, and it creates support tickets. Mutation lets Gatekeeper change the object on its way in. It uses its own set of object kinds. One kind (Assign) can set any field, for example to set a default pull policy. One (AssignMetadata) can add labels and annotations. One (ModifySet) can add or remove items from a list, such as extra arguments to a container. Newer versions also include a kind for rewriting image references, which is handy for redirecting images to an internal mirror.

Expansion: seeing the Pods hidden inside a Deployment

Remember the trap from Chapter 7: a rule that checks Pods does not fire when a Deployment is created, because the Pods do not exist yet. Expansion templates solve this. They tell Gatekeeper how a higher-level object, such as a Deployment, turns into Pods. Gatekeeper then builds an imaginary Pod, tests it against Pod rules, and can warn or reject the Deployment right away. Developers get fast feedback at the moment they apply the Deployment, not minutes later when Pods fail to appear.

External data: asking another system

Sometimes the facts needed to decide are not inside the cluster. For example, “has this image been scanned and signed?” is a question for an image-signing service. Gatekeeper’s external data feature lets a policy call a registered provider service during evaluation. Use this carefully. Every outside call adds time to the request, and a slow provider can slow every deployment in the cluster. Providers should be fast, highly available, and called only when truly needed.

A guard who can be tricked, bribed or switched off is not a guard. This chapter covers how to make sure Gatekeeper cannot be bypassed, and how to keep it from becoming a danger of its own.

The fail-open versus fail-closed decision

Every webhook has a failure policy that decides what happens when Gatekeeper cannot be reached. With “Ignore,” also called fail-open, the request is allowed. With “Fail,” also called fail-closed, the request is rejected. The standard Gatekeeper installation uses the first option, because rejecting everything whenever Gatekeeper has trouble can bring a cluster to a halt, including the very Pods that Gatekeeper needs in order to recover.

Many mature teams choose fail-open for general use and fail-closed only for a narrow set of high-risk resources, combined with several replicas, Pod disruption budgets and spreading Pods across zones.

Closing the common bypass paths

• Protect the policy objects. Use RBAC so that only the platform team can create, change or delete templates, constraints and Gatekeeper configuration. Anyone who can delete a constraint can switch off the rule.
• Protect the namespace. Restrict who can edit anything inside gatekeeper-system, including the webhook configuration objects.
• Guard the ignore label. The label that makes a namespace skip the webhook is powerful. Write a rule, enforced by Gatekeeper itself, that only authorized users may add this label.
• Cover all entry points. Policies should apply to Pods created through Deployments, Jobs, CronJobs and direct Pods. Remember that ephemeral containers and Pod updates are also paths worth checking.
• Keep the webhook secure. Allow only the API server to reach the webhook with network policies where possible, and keep certificate rotation healthy.
• Trust your supply chain. Install Gatekeeper from verified images, pin versions, and review any third-party template before using it. A template is code that runs inside your control plane path.

Relation to the Pod Security Standards

Kubernetes has a built-in feature, Pod Security Admission, that enforces three broad levels of Pod safety: privileged, baseline and restricted. It replaced the older PodSecurityPolicy, which was removed in Kubernetes 1.25. Gatekeeper is a good companion to it: the built-in feature covers the standard Pod safety baseline cheaply, and Gatekeeper handles all the company-specific rules that the built-in one cannot express.

You cannot trust what you cannot see. Gatekeeper gives you three windows into its behavior: status fields, logs and metrics.

Window one: status on the objects

The easiest place to look is the constraint itself. Its status shows the time of the last audit, the total number of violations, and the list of the first few violating objects with their messages. The template status shows whether each Gatekeeper Pod accepted the template, and surfaces compile errors. When something seems wrong, always check these two statuses first.

Window two: logs

Gatekeeper logs in a structured format. The webhook can log every denied request, and the audit Pod logs each violation it finds along with the audit run it belongs to. In a busy cluster, denied-request logs can be noisy, so send them to a central logging system and search by constraint name or namespace. Logs are also where you learn that the webhook is rejecting requests for technical reasons, such as certificate trouble.

Window three: metrics

Gatekeeper exposes metrics in the Prometheus format, a widely used monitoring standard. Exact names can differ slightly between versions, but the useful families are:

Alerts worth setting

• Admission latency crossing a safe fraction of the webhook timeout.
• No successful audit in several audit intervals.
• A sudden jump in denied requests, which may mean a bad rule or a bad deployment pipeline.
• Gatekeeper Pods restarting, being out of memory, or unavailable.
• Templates in error state.

Gatekeeper sits on the critical path of every deployment. If it is slow, everything is slow. This chapter shows how to keep it fast at the scale of large organizations.

What affects admission speed

• Number of matching constraints. Each match means running Rego. Narrow matches, such as by kind and namespace, keep this small.
• Rule complexity. Rules that loop over big lists, or compare every object against every other object, cost more than simple field checks.
• Replicated data size. Referential rules that scan large inventories get slower as the cluster grows.
• External data calls. Each outside call adds network time and a new way to fail.
• Machine resources. Too little CPU makes evaluation queue up. Too little memory causes restarts.

Scaling the live path

Run multiple replicas of the controller manager, typically three or more in production, and spread them across failure zones. Requests from the API server are balanced across them. Give the Pods proper CPU and memory requests so Kubernetes places them on healthy machines. Use Pod disruption budgets so that a node upgrade never takes down all replicas at once. For very high request rates, add replicas before adding bigger machines, since evaluation of separate requests is independent and spreads well.

Scaling the audit path

Audit cost grows with the number of objects and constraints. For clusters with hundreds of thousands of objects, tune the audit interval to a longer period, lower the chunk size so memory stays bounded, and restrict audited kinds to those you really care about. Avoid enabling audit for kinds that change constantly and add little compliance value.

The fastest way to lose trust in a policy system is to break production with a surprise. A calm, repeatable rollout process is what separates mature platform teams from the rest.

Test before you ship

Gatekeeper provides a command-line tool called gator. It lets you run templates and constraints against sample objects on your own computer or in your build pipeline, with no cluster needed. It can also run formal test suites that state which sample objects must pass and which must fail, so that every future change to a template is checked automatically. A good test suite includes at least one passing example, one failing example, and one tricky edge case, such as an object with the checked field missing.

A safe rollout path

Deliver policies with GitOps

GitOps means a tool inside your cluster continuously makes the cluster match what is stored in Git. Templates and constraints are ideal for this. A pull request shows exactly what is changing, reviewers can approve it, and a revert is a single click. One ordering detail is worth knowing: a constraint can only be created after its template has been accepted and its kind registered. Good pipelines apply templates first and constraints second, or retry briefly until the new kind appears.

Once you move past the policy library, you will meet a handful of Rego ideas that explain most of the behavior you see. Here they are in plain language.

Helper rules and shared libraries

A real template rarely has just one rule. Authors split the work into small helper rules, each with a clear name, such as “this container is privileged” or “this image comes from an approved registry.” The main violation rule then reads like a short paragraph that combines the helpers. This makes the logic easier to read, easier to test one piece at a time, and easier to reuse. Gatekeeper templates can also include shared library code, so the same helper can serve several templates without being copied and pasted. When a shared helper is fixed, every template that uses it gets the fix.

Negation: the careful word “not”

Rego lets you say that something must not be true, for example “there is no security setting that forbids privilege escalation.” Negation is very useful, but it behaves in a particular way. It succeeds when the thing it checks cannot be found. That is exactly what you want when you are looking for a missing setting, and exactly what you do not want when a typing mistake makes a field name wrong. A misspelled field name can make a “not” check pass for every object. This is one of the strongest reasons to test rules against both good and bad samples.

Sets, lists and collecting results

Rego has simple collections. A set is a group of unique items, like a bag of different label names. Policies often build a set of the labels that are required, build another set from the labels that are present on the object, and then subtract one from the other. Whatever remains is the list of missing labels, which makes a perfect error message. This way of working, collecting and comparing groups, is called a comprehension, and it replaces many loops you would write in other languages.

Default values for missing fields

Because Kubernetes objects leave many fields empty, rule authors often use a helper that returns a default when a field is absent. For example, a rule can ask for the “allow privilege escalation” setting and treat a missing value as “true,” because that is how Kubernetes behaves when nothing is set. Writing the default explicitly protects you from the silent-pass behavior described in Chapter 7.

Writing messages people can act on

The text that a violation returns is what developers read when they are blocked. A message such as “denied” helps nobody. A message such as “container web uses image from registry example.org, allowed registries are registry.internal and mirror.internal” tells the developer exactly what to change. The best messages name the object, name the field, state the rule, and say what is allowed. Many teams also add a short link to an internal page that explains the policy and how to request an exception.

Why some rules are slow

Rego is fast when the engine can quickly narrow down which data to look at. It is slow when a rule must examine large collections without a clear starting point. The classic slow pattern is a referential rule that walks every object of a kind in the replicated inventory to compare against the one being created. Better rules look up a specific namespace or a specific name directly, which lets the engine jump to the answer. When a rule is slow, look first for loops over big inventories and for repeated work that a helper could compute once.

Most teams start with the same dozen rules. Knowing what each one protects against helps you decide which to switch on first.

Which should you turn on first?

Start with rules that are cheap to follow and expensive to ignore. Required labels cost developers almost nothing and pay back at once in ownership and billing. Approved registries and no-latest-tag protect your supply chain and make releases repeatable. Blocking privileged containers and host paths removes the most dangerous escape routes. Probes and resource limits improve reliability more than security, and they can come right after.

One cluster is easy. Fifty clusters across three clouds is a different problem, and policy is one of the first things that drifts.

The drift problem

When each cluster owner installs Gatekeeper by hand and picks their own rules, you quickly end up with fifty slightly different security postures. Some clusters block privileged containers, some only warn, and some never installed the rule at all. An auditor asking “are all production clusters compliant?” gets fifty different answers. The cure is to treat policy like any other fleet-wide configuration: define it once, deliver it everywhere, and measure it centrally.

flowchart TB
  REPO["Central policy repository"] --> BASE["Baseline bundle - applies to every cluster"]
  REPO --> PROD["Production bundle - strict, deny actions"]
  REPO --> DEV["Development bundle - relaxed, warn actions"]
  BASE --> C1["Cluster A"]
  BASE --> C2["Cluster B"]
  BASE --> C3["Cluster C"]
  PROD --> C1
  PROD --> C2
  DEV --> C3
  C1 --> MET["Central metrics and compliance dashboard"]
  C2 --> MET
  C3 --> MET

Layering bundles

A practical approach is to build policy in layers. The baseline bundle holds rules every cluster must follow, such as no privileged containers and required ownership labels. An environment bundle adds stricter actions for production and gentler ones for development. A team bundle can add special rules for sensitive workloads, such as payment systems. Each cluster subscribes to the layers that fit it, and a GitOps tool keeps every cluster in line with the repository.

Measuring compliance across the fleet

Each cluster exposes violation and audit metrics. A central monitoring system collects them and shows, for each rule, how many clusters follow it and how many objects break it. Leaders can then see trends: violations going down means teams are fixing things, while a sudden increase after a release points to a problem worth investigating. Managed cloud policy services and multi-cluster tools from the cloud providers do the same job with ready-made dashboards.

Staged delivery to the fleet

Never push a new deny rule to every cluster at once. Start with a small canary group of low-risk clusters, watch the denial and violation metrics for a few days, then widen. If something goes wrong, rolling back a single tagged change in the repository fixes all clusters together.

When something goes wrong with a policy system, a calm, ordered checklist saves hours. These are the problems you will meet most often, and where to look first.

Problem: my template does not work

Look at the template’s status. If it has compile errors, the engine could not understand the logic, and the message usually names the line and the problem. If the status shows that some Gatekeeper Pods have not created the template, wait a moment for them to catch up, then check their logs. A common cause is a syntax error copied from an old example.

Problem: I cannot create my constraint

The error usually says the kind is not known. This means the template is not yet ready, or the kind name in the constraint does not match the name in the template. Check spelling and capital letters, since kind names are case sensitive. In pipelines, make sure templates are applied first and that the pipeline waits until the new kind is registered.

Problem: my rule never fires

Work through the layers from outside to inside:

• Is the constraint matching the object? Check kinds, namespaces and selectors in the match section. A typo in an API group is enough to match nothing.
• Is the namespace exempt? Check the configuration object and any ignore label.
• Is the webhook receiving the request? Confirm the webhook configuration covers the operation, such as create and update, and the resource kind.
• Is the action dry-run? A dry-run constraint never blocks, so you will only see its results in audit.
• Is the logic silent for a missing field? Test the rule with gator against the exact object that should fail.

Problem: requests are timing out

Check the latency metric and the Pod resources. Slow decisions usually come from too few replicas, too little CPU, a heavy referential rule, or a slow external data provider. Remember that the default timeout is short. If the failure policy is fail-open, timeouts quietly let requests through, which is why you should alert on latency before it reaches the limit.

Problem: audit shows nothing, or looks stale

Check that the audit Pod is running, that its last-run timestamp is recent, and that the kinds you care about are not excluded by the audit settings. If the audit Pod is being restarted for lack of memory, reduce the chunk size or the number of audited kinds.

Problem: Gatekeeper Pods keep restarting

Look at memory first. Replicating large kinds into memory is the most common cause. Next, check the certificates, since the webhook needs a valid serving certificate. Finally, check whether the Pods can reach the API server.

Gatekeeper sits in the middle of every deployment, so changing it deserves the same care as changing the API server itself.

Upgrading Gatekeeper

Treat each Gatekeeper release like any critical platform upgrade. Read the release notes for changes in default behavior, removed features, and changes to the custom resource definitions. Upgrade a test cluster first, run your policy tests, and watch the latency and audit metrics. Then move through your environments in order. Pin the version in your deployment tooling so that nobody gets a surprise upgrade by accident.

Kubernetes upgrades and compatibility

Each Gatekeeper version supports a range of Kubernetes versions. Before upgrading Kubernetes, check that your Gatekeeper release supports the target version. Webhook behavior and API versions occasionally change between Kubernetes releases, and a mismatch can cause strange admission failures.

Keeping policies healthy over time

• Version your bundles. Use tags in Git so you can say exactly which version of the rules each cluster runs.
• Review on a schedule. Every quarter, look at which rules fire most, which are never triggered, and which exemptions are still needed.
• Retire what no longer helps. A rule that has no violations and no purpose is a maintenance cost.
• Track deprecations. Kubernetes retires old API versions. A rule that targets a removed API version silently stops matching anything.

Backup and recovery

The good news is that Gatekeeper has very little state of its own. Templates, constraints and configuration live in Git, which is your source of truth, and also in the cluster database. Audit results are rebuilt automatically on the next scan. To recover a lost cluster, you reinstall Gatekeeper, apply the policy bundle from Git, and wait for the audit to repopulate the reports. The only thing worth backing up separately is anything you changed by hand, which is a good reason to never change policies by hand.

For many organizations, the real reason to adopt policy enforcement is not curiosity. It is that someone has asked them to prove their systems follow the rules.

Turning frameworks into policies

Security and compliance frameworks, such as the CIS Kubernetes Benchmark, PCI DSS for payment data, SOC 2 for service providers, and many national regulations, contain requirements that sound like policy statements: restrict privileged access, use approved software, encrypt traffic, keep an inventory of assets, and show who is responsible for each system. Many of those statements can be translated directly into constraints. The mapping is simple to describe: each framework control points to one or more rules, each rule has an owner, and the audit report becomes the evidence that the rule is working.

Evidence without screenshots

Traditionally, audit season meant weeks of collecting screenshots and spreadsheets. With policy as code, the evidence is automatic. The Git history shows who proposed and approved each rule. The audit results show the current state of every object. The metrics show trends over time. An auditor can see all of that without anyone preparing a special report.

A healthy exception process

No rule fits every case. A legacy payment system may need a host path that nobody else may use. Rather than weakening the rule for everyone, build a clear path for exceptions:

Technology is only half of the story. A policy system lives or dies by how people feel about it, and by who is responsible when it breaks.

Who owns what

Clear ownership prevents most disputes. A typical split looks like this:

Treat the policy system like a service

Because the webhook sits on the path of every deployment, it deserves service-level goals like any critical system. Many teams set targets such as a high percentage of admission requests answered within a short time, and an agreed maximum time for an audit cycle. The targets are tracked on a dashboard, and misses trigger a review. An outage in the policy system is a platform outage, and it should be handled with the same urgency.

Making life easy for developers

Developers accept rules they understand and can follow quickly. Several habits make a big difference:

• Shift left. Run the same policies in the build pipeline and on the developer’s own computer with gator, so problems appear minutes after the code is written, not at deployment time.
• Explain every rule. Keep a short page for each policy with the reason, a good example, and how to ask for an exception.
• Fix by default. Use mutation for harmless defaults so developers are not blocked over small things.
• Give visibility. Let every team see its own violations in a dashboard without asking the platform team.
• Listen. Hold regular office hours, track the most common denials, and improve the messages or the rules that cause the most pain.

This is an illustrative story, assembled from patterns that many real teams follow. The company is fictional, but the steps and the lessons are real.

The situation

ShopSphere is an online marketplace with 300 developers, 40 product teams, and 12 Kubernetes clusters across two regions. In one quarter they suffered three incidents: a container with host access was used to read data from a neighboring workload in a test environment, an unlabeled batch job ran for a month and cost the company a large cloud bill before anyone found its owner, and an image pulled from a public registry turned out to contain a known vulnerability. Leadership asked the platform team to make sure these could not happen again.

The plan, month by month

What went wrong along the way

Not everything was smooth. In the second month, a referential rule that compared every Ingress with every other Ingress slowed admission noticeably on the largest cluster. The team rewrote it to look up hostnames directly instead of scanning the whole inventory, and latency returned to normal. In the fourth month, a deny rule briefly blocked a node agent because its namespace had been missed in the exemption list. Because the rule had already been tested in warn mode, the problem showed up as a warning first and was fixed before enforcement.

The results

Gatekeeper is not the only way to enforce rules in Kubernetes. Knowing the alternatives helps you choose, and often lets you combine them.

A quick decision guide

If your team already uses OPA elsewhere, or needs advanced rules that look across objects, Gatekeeper is a natural fit. If your team prefers writing everything in the same style as other Kubernetes files, Kyverno may feel more comfortable. If your rules are simple field checks and you want zero extra components, the built-in policy feature is attractive. Many organizations mix them: built-in Pod security for the baseline, plus Gatekeeper for the organization’s own policy library. Newer Gatekeeper releases are also adding support for the CEL language and the built-in policy engine, so the boundaries between these options keep moving. Check current release notes before deciding.

After watching many teams adopt policy engines, the same successful habits and the same painful mistakes appear again and again.

Patterns that work

Patterns that hurt

A short checklist you can copy into your team’s playbook.

The most common mistakes, explained

• Locking yourself out. A rule that blocks the cluster’s own system components can stop recovery. Always exempt system namespaces before turning on broad rules.
• Applying constraints before templates. The constraint kind does not exist yet, so the apply step fails. Order your pipeline correctly.
• Ignoring init containers and ephemeral containers. A rule that checks only the main containers leaves other container lists unprotected.
• Checking only Pods. A Deployment can look fine and then produce Pods that fail later. Use expansion or add rules at the Deployment level.
• Assuming audit is instant. Audit runs on a timer. A new violation may take a full interval to appear.
• Trusting the default violation limit. If a constraint shows exactly 20 violations, there may be many more. Check the metrics for the true total.

Gatekeeper is not only a lab tool. It sits underneath major cloud offerings and enterprise platforms.

Short answers to the questions people ask most often.

If you remember only a handful of ideas from this guide, make them these.