An audit log is a trustworthy record of actions. It answers questions that no other tool can.

The four questions

Almost every audit or investigation boils down to four questions: who did it, what did they do, when did it happen and from where did the request come? Add a fifth, was it allowed, and you have the core of what a good audit trail provides.

Three reasons teams need it

What happens without it

Many clusters run for years with no audit logging at all, because the feature is not always switched on by default in self-managed installations. The first time anyone notices is during an incident or an audit, when the question “who deleted that namespace?” has no answer. The records cannot be recreated after the fact. A log you did not turn on is a log you will never have.

Compliance is not the same as security

Passing an audit and being secure overlap but are different. An audit checks that you have the controls you promised and can show evidence for them. Security asks whether attackers can actually get in. A good audit logging program serves both: it satisfies the auditor and gives defenders real visibility. A weak one satisfies neither: it produces mountains of data that nobody reads and that would not help in a real investigation.

Kubernetes produces several kinds of records. Mixing them up is the first source of confusion.

What is special about the audit log

Application logs are written by programs and describe what they chose to say. The audit log is written by the API server about every request it receives, regardless of what the caller wanted. The caller cannot opt out. That independence is what makes it useful as evidence: the record of the action is made by the gatekeeper, not by the person acting.

What the audit log does not show

• Anything done inside a container after someone gets a shell. The audit log shows that a shell was opened (an exec request), but not the commands typed inside.
• The contents of your application data, such as database rows.
• Actions that never touch the API server, for example a process on a node talking directly to the container runtime, or one container talking to another over the network.
• Changes in the cloud account outside Kubernetes, such as editing a firewall rule, which are covered by the cloud provider’s own logs.

A complete picture therefore combines several sources, and later chapters show how.

Many standards touch Kubernetes. They use different words, but they ask similar things about logging.

Common frameworks and what they want from logs

Themes that repeat

• Record the right events: logins, access to sensitive data, changes to permissions and configuration, and failures.
• Include the right details: who, what, when, where and the outcome.
• Protect the records: from change, from deletion and from unauthorized reading.
• Keep them long enough: retention periods are often a year or more, with some period readily available for quick search.
• Review them: someone or something must actually look, regularly, and act on findings.
• Keep clocks accurate: timestamps must be reliable across systems to reconstruct a sequence.
• Show it works: be ready to provide evidence on request.

Shared responsibility

Standards usually apply to your whole environment, and cloud providers have their own certifications for their part. A managed Kubernetes service may handle the control plane’s logging infrastructure, but you must still enable, configure, retain and review those logs for your own workloads. Responsibility for what runs in the cluster always remains yours.

A note on specifics

Requirements, retention numbers and wording vary between versions of each standard and between assessors. Treat the table as a general guide, and confirm exact requirements with your compliance team or assessor before designing retention and review processes.

To understand audit logs, follow a single request from a person to the cluster’s memory.

flowchart LR
  U["Person, tool or program"] -->|"request"| AUTHN["1. Authentication: who are you?"]
  AUTHN --> AUDIT["Audit layer: records the request"]
  AUDIT --> AUTHZ["2. Authorization: are you allowed?"]
  AUTHZ --> MUT["3. Mutating admission: modify request"]
  MUT --> VAL["4. Validation and policy admission"]
  VAL --> ETCD[("5. Stored in etcd")]
  ETCD --> RESP["Response to caller"]
  RESP --> AUDIT2["Audit layer: records the result"]
  AUDIT --> BACKEND["Audit backend: log file or webhook"]
  AUDIT2 --> BACKEND

Reading the diagram

Every request to the cluster, whether from a human typing a command or a program calling the API, passes through the same series of gates. First, authentication decides who the caller is. Then, authorization checks whether that caller may do this action. Next, admission steps may change the request or reject it on policy grounds. If everything passes, the object is stored in the cluster’s memory, etcd, and a response is returned.

The audit layer sits near the front. It notes the request as soon as the caller’s identity is known, and again when the request completes. That placement is important: even requests that are denied at authorization or rejected by policy are recorded. The failed attempts are often the most interesting part of an investigation.

What is and is not captured

Who calls the API server

Not only people. The kubelets on every node, the controllers inside the control plane, the scheduler, your operators, your CI tools and your monitoring agents all make requests, thousands per minute. The great majority of audit events come from these system components, not from people. A good audit policy separates the routine chatter from the human and sensitive activity.

A single request can produce more than one audit event, one for each stage of its life.

The four stages

Why stages matter for volume and clarity

If you record every stage of every request, the log doubles or triples in size and each action appears several times. Most teams omit the RequestReceived stage, since the ResponseComplete event already includes everything needed. The omit-stages setting in the audit policy controls this. Keep the first stage only if you specifically need to prove that requests that never finished were still seen.

A unique thread: the audit ID

Every request gets a unique audit identifier, and all events from the same request share it. If you do keep several stages, that identifier ties them together. It also lets you correlate across systems, for example by seeing the same identifier in a webhook’s logs or a proxy’s logs.

Timing information

Each event carries timestamps for when the request was received and when the stage finished. The difference shows how long the request took. Unusually slow requests can point to performance problems or to heavy use such as someone listing every Secret in the cluster.

For each request, you choose how much of it goes into the log. The choice balances usefulness, volume and the risk of recording sensitive data.

The four levels

What each level lets you answer

The Secret trap

If you log request or response bodies for Secrets, the log will contain the secrets themselves: passwords, tokens, keys. That turns your audit log into a rich target for attackers, and defeats the purpose of controlling who can read secrets. The standard rule is to record Secrets, ConfigMaps that may hold tokens, and authentication-related objects at Metadata level only. You still learn who touched them and when, without copying the contents.

Choosing levels sensibly

• Metadata for most things: a good default for reads and routine changes.
• Request or RequestResponse for key configuration changes: RBAC objects, admission policies, network policies and workload definitions in sensitive namespaces, where the exact change matters.
• None for pure noise: health endpoints, the routine lease renewals of system components and node status updates.
• Metadata for anything with secret content: Secrets, service account tokens, token review and certificate signing requests.

The audit policy is the rulebook that decides, request by request, what level to record. Writing it well is the single most important step.

How the policy works

A policy is an ordered list of rules. Each rule describes a kind of request, matching on things such as the user, the user group, the verb (get, create, delete and so on), the resource type, the namespace or a URL pattern, and assigns a level. When a request arrives, the rules are checked from the top, and the first rule that matches decides its level. If no rule matches, the request is not recorded at all, so a policy should end with a catch-all rule.

The order matters

Because the first match wins, specific rules must come before general ones. If a general rule saying “record everything at Metadata” sits at the top, none of the later, more specific rules will ever be reached. A well-ordered policy places the exclusions of noise first, then the special handling of sensitive objects, then rules for important changes, and finally a catch-all.

A sensible order, described in words

Choosing what to exclude

The omit-stages and omit-managed-fields settings

Policies can also drop the RequestReceived stage, as discussed, and omit managed fields, which are bookkeeping data about who set each field and make request bodies much larger without adding evidence value. Using both of these cuts volume significantly with little loss.

Start from a known example, then adapt

The Kubernetes project and cloud providers publish example policies. Cloud-managed clusters often come with a fixed or semi-fixed policy, and some allow configuration of what is sent. For self-managed clusters, begin with a published example and adapt it with your own namespaces, custom resources and sensitive objects. Test the policy in a non-production cluster, measure the volume and review a sample of events before rolling it out.

Treat the policy as code

Keep the audit policy in version control, review changes like any other security change and record who approved them. The policy itself is audit evidence: auditors will ask what you log and why, and a reviewed document with history is a strong answer.

Once an event is created, it must be sent somewhere. The API server offers two built-in destinations.

Log file backend

The API server writes events as lines of structured text to a file on the control plane machine. You control the path, the maximum age of files, the number of old files kept and the maximum size before rotation. A log shipper running on the machine then reads the file and sends it onward. This method is simple and reliable, since the API server does not depend on anything remote to keep running, but it ties you to the control plane machine’s disk and requires running and securing the shipper there.

Webhook backend

The API server sends events over the network to a service you operate, in batches. The service receives them and does what you decide: store, forward, filter or alert. It removes the need to run an agent on the control plane machine, but it creates a dependency: if the receiver is slow or down, the API server may buffer events, drop them, or in blocking mode slow down. Several settings control batching, buffer size, retry behavior and whether sending blocks the request.

Batching and blocking modes

Backends can run in batch mode, where events are collected in memory and sent in groups, or in blocking mode, where each event must be written before the request continues. Blocking gives the strongest guarantee that an action is recorded, since no action completes without a record, but it adds delay to every request and makes the API server depend on the backend. Batch mode is faster, with a small risk that events are lost if the server crashes or the buffer overflows. Most clusters choose batch mode with generous buffers and monitoring of dropped events.

What happens when the backend struggles

• With a full buffer in batch mode, the API server drops events and counts the drops in its own metrics. You must watch this metric.
• With blocking mode, requests slow down or fail until the backend recovers.
• With the file backend, a full disk can cause trouble for the control plane machine. Size and rotate the files, and monitor the disk.

The shipping path

Whatever backend you choose, the goal is the same: get the events off the control plane quickly into a protected, centralized place. A record that stays only on the machine where it was written can be tampered with by anyone who compromises that machine, and it vanishes if the machine does. Later chapters cover the full pipeline.

On managed services you cannot edit the API server’s settings directly, so audit logging works through the provider’s own features.

The general pattern

The provider runs the control plane and collects its audit events. You decide whether to switch on delivery of those logs into your account, where they land, and how long they are kept. The policy is largely chosen by the provider, so you accept what is recorded, with some control over what you retain and alert on.

Always check these things

• Is audit log delivery switched on? On some services it is off by default for the Kubernetes audit category, and you must enable it deliberately.
• How long are logs kept by default? Often much shorter than compliance requires. Export to long-term storage with your own retention.
• What is in the events? Check whether the cloud identity appears in the user field, or only a mapped role name. This affects how well you can attribute an action to a person.
• What levels does the provider record? Providers may record at Metadata or Request level, and may filter certain requests. Read the documentation so you know what evidence you actually have.
• Who can read, change or delete them? Apply tight access controls on the log destination.

Two layers, not one

On managed services, two audit trails exist and both matter. The Kubernetes audit log records requests to the cluster’s API. The cloud audit trail records actions on the cloud resources: creating or deleting the cluster, changing node groups, altering network rules, modifying permissions of the cloud accounts. Someone could weaken the cluster’s security without ever touching the Kubernetes API, for example by changing a firewall or granting a cloud role. Collect and review both.

Features change

Each provider adds and renames audit features regularly. The table is a map of the general approach, not a substitute for current documentation. Before an audit, re-read the provider’s current guidance and confirm what your configuration really records.

An audit event is a small structured record. Learning to read it quickly is the core skill of every investigator.

The fields that matter most

Reading one event, in words

Suppose an event says: verb “delete,” resource “deployments” in namespace “payments,” name “checkout,” user “alice@example.com” in group “platform-admins,” source address from the office network, user agent a command-line tool, response code 200, and an annotation saying that authorization allowed it through a particular role binding. In plain English: Alice, from the office, using the standard tool, deleted the checkout Deployment in the payments namespace, and she was allowed to because of her membership in the platform admins group. In one line, you have the four questions answered, plus the fifth.

Response codes worth recognizing

Annotations as explanations

The authorization annotations record the decision made and the reason, often naming the role or binding that granted access. This is gold for compliance: it tells an auditor not only that an action was allowed, but which rule allowed it. Admission-related annotations, depending on your configuration, record which policies were applied or which webhooks modified the request.

Group and extra fields

The user’s groups explain much of their access. The extra attributes may hold information supplied by the identity provider, such as the cloud identity behind a mapped role. Look at these to see through generic names to the actual person.

An audit log is only as good as the identities in it. If everyone shares the same account, the log says “admin did it,” which proves nothing.

How Kubernetes identifies callers

Kubernetes has no built-in user database. It accepts identities from several sources and records whatever name the source supplies:

• Client certificates: the name inside the certificate becomes the user, and the organization fields become groups.
• Bearer tokens from an identity provider (OIDC): the user and groups come from claims in the token issued by your company’s login system.
• Service account tokens: programs running in the cluster, named after their namespace and service account.
• Cloud provider authenticators: cloud identities mapped to cluster users.
• Webhook token authentication and proxies: custom systems that vouch for a caller.
• Anonymous requests: callers with no credentials, if allowed at all.

The shared-account problem

Many clusters start with an administrator certificate in a shared file, used by several people. Every action then appears under the same name. The audit log cannot say which person did it. Compliance frameworks usually require individual accountability, so shared credentials are a common audit finding.

Service accounts: the non-human callers

Programs in the cluster use service accounts. Each should have its own, with only the access it needs, so that the audit log shows which program did what. A single default service account used by dozens of applications blurs attribution and encourages excess permissions. Newer versions favor short-lived, automatically rotated tokens bound to a Pod, rather than permanent tokens stored in secrets, which are both safer and easier to attribute.

Impersonation

Kubernetes lets an authorized user act as another user or group. This is useful for testing permissions, but it can also hide the true actor. The audit event records the impersonated identity in a separate field. Auditors and defenders should watch for impersonation, restrict who may do it, and make sure the real actor remains visible in the log.

Break-glass access

Sometimes administrators need emergency access that bypasses normal approvals. Handle this with a documented break-glass process: a special, strongly protected account or role, granted for a short time after approval, with every use alerted and reviewed afterwards. The audit log is the evidence that break-glass access was used only when justified.

The special administrator group

Kubernetes treats members of a special built-in administrators group as all-powerful: authorization checks are bypassed for them, so even a missing role binding does not stop them. Credentials for this group, for instance the initial administrator certificate of a self-managed cluster, must be locked away, rarely used, and every use should raise an alert.

Role-based access control decides who may do what. Audit logs show who actually does what. Comparing the two is a powerful compliance practice.

The permission gap

Most users and service accounts have far more permission than they ever use. Roles are granted for convenience, “just give them admin for now,” and never trimmed. The gap between granted and used permission is unneeded risk. Audit logs reveal the used side: for each identity, which verbs on which resources they have really exercised in the past months.

A review cycle

Permissions to scrutinize first

Reading authorization decisions

The authorization annotations in audit events name the rule that allowed or denied the request. Use them to find which bindings are doing the real work, and which are idle. A role that never appears as a reason for an allow decision in months is a candidate for removal.

Tools that help

Several open-source tools can analyze roles and bindings, show who can do what, and suggest policies based on audit data. They differ in features and maturity, so evaluate them in a test cluster. The principle is the same whichever you pick: compare declared access to observed use, and trim the difference.

Logging is also a risk. Logs can leak the very data you are trying to protect, and privacy laws apply to logs too.

The sensitive things that creep in

Principles for safe logging

• Metadata first. Use Metadata level wherever the content is not needed to answer a real question.
• Exclude by resource type. Treat Secrets, tokens, certificates and any custom resource that holds personal data as Metadata-only.
• Reduce at the source, in the policy, rather than hoping to clean the data later.
• Scrub in the pipeline for anything that slips through: filters that mask patterns such as tokens and card numbers.
• Control who reads the logs, since even metadata reveals who accessed what and when.
• Apply retention limits so personal data in logs is not kept forever without need.

Privacy law and logs

Under data protection laws such as the European GDPR, logs that contain personal data are themselves personal data. This means they need a lawful basis, protection, limited retention, and sometimes a way to respond to a person’s request. Security logging is generally accepted as a legitimate purpose, but it should be proportionate: record what you need for security and accountability, and not more. Discuss specifics with your privacy officer.

Application logs are the bigger risk

The most common source of leaks is not the audit log but application logs, where programs print request details, tokens, user data or error messages containing sensitive values. Developers should be guided to avoid logging secrets and personal data, and pipelines should detect and mask them. Scan logs periodically for patterns that look like credentials, and treat a found secret as a security incident: rotate it.

The audit log sees the front door. Much of what an attacker or an insider does happens inside, where the API server never looks.

The layers of evidence

Why the node layer matters

Anyone who obtains access to a node, whether by logging in or by escaping from a container, can act outside the reach of the Kubernetes audit log. They may read other containers’ files, steal credentials or tamper with processes. Node-level logging and file integrity monitoring are the safety nets. Limit direct logins to nodes, and treat any login as an event to review.

Tying the evidence together

To follow an incident across layers, you need common keys: timestamps that agree, Pod and namespace names, node names, container identifiers, user names and source addresses. Enrich logs with Kubernetes metadata at collection time, such as labels, namespace and Pod name, so that an event on a node can be connected to the workload that caused it.

Application-level audit

Standards that care about data access, such as those for health and payment data, want to know who viewed or changed a specific customer record. The Kubernetes audit log cannot tell you that; only the application can. A mature program therefore asks application teams to produce their own audit events, with user identity, action and object, in a structured format that flows into the same pipeline.

Runtime security tools fill the gap that audit logs leave. They watch behavior inside running workloads.

What they see

Runtime tools use operating system features, such as kernel hooks, to observe system calls made by containers: starting programs, opening files, making network connections, changing permissions. They compare behavior against rules or learned baselines and emit events when something looks wrong.

Typical detections

How they pair with the audit log

The two sources complement one another well. Imagine an attacker exploits a web application, opens a shell in the container, steals the Pod’s service account token and uses it to list secrets. The runtime tool shows the shell and the file read. The audit log shows the service account’s unusual list request from that Pod’s address. Together they give the full story and a clear timeline.

Operating a runtime tool well

• Start in observation mode, reviewing alerts for a few weeks to learn normal behavior and tune out false positives.
• Send events to the same central store as audit logs, with Kubernetes metadata attached.
• Write rules that fit your workloads. A rule that suits a web server may flood alerts on a build runner.
• Decide response in advance: alert only, isolate the Pod, or kill it, and who decides.
• Protect the tool itself, since attackers will try to disable it.

Performance and trade-offs

Kernel-level sensors add some overhead and need elevated privileges on nodes. Choose a mature tool, test it on representative workloads, and weigh its coverage against its cost.

Records are only useful if they reach a safe place, in order, in full, and can be searched when it matters.

flowchart LR
  subgraph SRC["Sources"]
    A1["API server audit events"]
    A2["Node and system logs"]
    A3["Runtime security events"]
    A4["Application logs"]
    A5["Cloud audit trail"]
  end
  subgraph COL["Collect and enrich"]
    C1["Agents on nodes and webhook receiver"]
    C2["Add cluster, namespace and workload labels"]
    C3["Mask secrets and sensitive patterns"]
  end
  subgraph TRN["Transport"]
    T1["Buffered, encrypted delivery"]
  end
  subgraph STO["Storage"]
    S1[("Hot: searchable, recent")]
    S2[("Archive: immutable, long term")]
  end
  subgraph USE["Use"]
    U1["Dashboards and search"]
    U2["Detection rules and alerts"]
    U3["Compliance reports and evidence"]
  end
  SRC --> COL --> TRN --> STO --> USE

Reading the diagram

Read it from left to right. On the left are the different sources. The next stage gathers events, labels them with the cluster, namespace and workload they came from, and masks anything sensitive. A transport layer carries them to storage, buffering when the destination is unavailable and encrypting data in transit. Storage has two tiers: a hot tier where recent events can be searched quickly, and an archive tier for long-term, protected retention. On the right, the data is used for dashboards, for detection and alerting and for compliance reports.

Design principles

• Get logs off the machine fast. The shorter the time an event lives only on a node, the smaller the window for loss or tampering.
• Buffer and retry. Networks fail. The pipeline should hold events and deliver them later, without gaps.
• Separate duties. The team that runs the cluster should not have unchecked power to delete the logs that record their own actions.
• Centralize across clusters, so one search covers the whole fleet, with fields identifying each cluster and environment.
• Structure the data. Keep audit events as structured records with consistent field names, so that searches and rules are reliable.
• Monitor the pipeline. Alert when events stop arriving, when delays grow, or when the API server reports dropped events. A silent log is as worrying as a noisy one.

Choosing storage

Heartbeat checks

Add a periodic, known test action, for example a harmless object created and deleted by an automated job, and verify that its audit events appear in the central store within a short time. This end-to-end test proves the whole pipeline works, and its failure triggers an alert long before anyone needs the logs in an emergency.

A log that can be quietly edited or deleted is worthless as evidence. The value of an audit trail rests on trust that nobody tampered with it.

Threats to the record

• An attacker covering tracks by deleting or editing entries after gaining access.
• An insider with administrator rights erasing evidence of their own actions.
• Accidents, such as a wrongly configured retention rule that removes data early.
• Loss through failure, such as a full disk or a deleted bucket.
• Clock errors that make the order of events uncertain.

Protection measures

Retention

Retention is how long you keep logs. It must satisfy the longest requirement among the rules that apply to you, but not exceed what privacy law and business need justify. A common structure keeps recent logs immediately searchable for several months and the full set archived for a year or more. Write the policy down, apply it with automated lifecycle rules and test that data is neither deleted early nor kept forever.

Legal hold

If an investigation or legal matter begins, normal deletion must pause for the relevant logs. Have a way to place a hold, and a person who can authorize it, before you need it.

Proving integrity to an auditor

Be ready to explain: where logs are stored, who can access them, what prevents alteration, how you would detect tampering, and how you know that no gaps exist. Evidence such as the storage lock configuration, access reports and the heartbeat test results turns these answers into proof.

Collecting logs is only half the work. The other half is noticing the few events that matter among millions.

High-value detections from audit logs

Good alerts are specific and actionable

• An alert should tell the responder who, what, where and why it matters, with a link to the full event.
• It should have an owner and a documented first response.
• Alerts that fire constantly and are ignored train people to ignore alerts. Tune or remove them.
• Combine signals. “Shell opened in a container” alone is a weak alert; “shell opened in a container, followed by a Secret listing from the same Pod’s address” is a strong one.

Tiers of response

Baselines

Alerts are most precise when they know what is normal. Learn the usual activity of each system account and each team: which namespaces they touch, at what times, from which networks. Then deviations become obvious. Start with simple, high-value rules, and add baselining as your data grows.

When an alert fires or someone reports a problem, a calm, structured search turns scattered events into a story.

The investigation flow

Questions the audit log helps answer

• Which account created this suspicious Pod, and from which address?
• Did the same account read Secrets before creating it?
• Which role binding gave it permission to do so, and who created that binding?
• Were other namespaces touched?
• Did the actor try things that were denied before finding something that worked?
• When did the activity start, and is it still going on?

An illustrative chain

An investigator starts from a cryptocurrency-mining Pod found in a development namespace. The audit log shows it was created by a CI service account at 02:14 from an unfamiliar address. Earlier that night, the same account listed Secrets in three namespaces, and before that, a request from the same address used a stolen token to authenticate. The identity provider shows the token was created for a build job two weeks earlier. The fix: rotate the token, delete the Pod, tighten the CI account’s permissions, and add alerts for CI accounts making requests from outside the build network.

Preserve evidence

Before cleaning up, make copies of relevant logs and object definitions, note times and commands, and keep a record of every action you take. Cleaning up too soon can erase evidence that you need later. If legal action is possible, involve your legal and security response teams early and follow their handling rules.

Prepare before it happens

Write a playbook for common scenarios, practice it in an exercise, and have saved searches ready that retrieve an identity’s activity or an object’s history in seconds. During a real incident, nobody enjoys inventing queries.

Logs tell you what happened. Policies prevent bad things from happening at all, and their decisions are evidence too.

Preventive and detective controls

Auditors speak of two kinds of controls. Preventive controls stop a violation before it happens, such as a rule that blocks privileged containers. Detective controls notice a violation after the fact, such as an alert on a privileged container that slipped through. A strong program uses both, and audit logging is the backbone of the detective side.

Admission policy engines

Policy engines hook into admission and check each request against rules written by your security team: images must come from approved registries, containers must not run as the administrator user, every workload must have an owner label, namespaces must have resource quotas, and so on. Built-in policy features in newer Kubernetes versions offer similar checks without a separate service.

Policy decisions as evidence

Modes of enforcement

• Audit mode: the policy records violations but does not block. Useful when introducing a rule, to see its impact first.
• Warn mode: the user sees a message but the request proceeds.
• Enforce mode: violating requests are rejected.

The usual path is to start in audit mode, fix the findings, warn for a while, then enforce. Each mode produces its own evidence of progress.

Built-in pod security standards

Kubernetes includes standard security profiles for Pods, from permissive to restricted, which can be applied per namespace in audit, warn or enforce modes. Their audit-mode violations show up as annotations on audit events, making them a handy, low-cost compliance signal.

Protecting the guardrails

Because policies are only as good as their integrity, treat changes to them as sensitive. Restrict who may edit policy objects and webhook configurations, send every change to the audit log at Request level and alert on edits. Attackers frequently try to disable policy enforcement before acting.

Where changes come through pull requests, the repository history tells the “why” that audit logs cannot.

Two views of the same change

The audit log says that a service account applied a change to a Deployment at 11:03. The Git history says that Priya proposed the change, Omar reviewed and approved it against ticket 4812, and the automation applied it. The first is a technical fact; the second is a business justification with approvals. Auditors want both, and a link between them.

How GitOps helps compliance

Connecting the dots

• Have the automation add annotations to the objects it applies, such as the commit identifier and the pull request, so audit events and cluster objects point back to the source.
• Use a consistent identity for the automation, so its actions are easy to recognize and separate from human ones.
• Alert on changes made by humans in environments that should be changed only by automation. Those are either emergencies, which should be documented, or violations.
• Keep the repository protected: branch rules, required reviewers, signed commits where possible and tight access to the settings.

Emergency changes

Real life includes emergencies when someone must change the cluster directly. Allow it through the break-glass process described earlier, but demand that the change be recorded afterwards, brought back into the repository and reviewed. The audit log shows the direct change; the follow-up ticket shows the justification.

Segregation of duties

Many standards require that the person who writes a change is not the only person who approves and deploys it. Pull request approvals and separate deployment automation naturally create this separation, and the logs show that it was respected.

Protecting sensitive data is a core part of most standards. Showing that you protect it requires evidence at several levels.

Where secrets live and what can leak

By default, Kubernetes Secrets are stored in etcd in an encoded but not encrypted form. Anyone who can read etcd or its backups, or who is allowed to read Secrets through the API, can see them. Several layers protect them:

• Access control: few identities may read Secrets, and every read is logged at Metadata level.
• Encryption at rest: the API server can encrypt Secrets, and optionally other resources, before writing them to etcd, using keys it holds or keys held by an external key management service.
• External secret stores: sensitive values live in a dedicated secrets manager, and the cluster fetches them when needed, so that long-lived copies are not stored in the cluster.
• Rotation: secrets and keys are replaced regularly, limiting the value of any stolen copy.
• Backups: etcd snapshots must be encrypted and access-controlled, because they hold everything.

Evidence to collect

Key use logs

When an external key management service is used, it keeps its own log of each use of a key: which identity asked to decrypt, when and why. These logs, combined with the Kubernetes audit log, show the whole story of a secret’s access: the API read request on one side and the key use on the other.

Audit questions to prepare for

• Who can read Secrets in production, and how do you know that list is complete?
• When was each important secret last rotated?
• If a developer’s laptop is stolen, what can the thief reach?
• Can you show every read of this secret in the last ninety days?

Standards care about who can reach sensitive systems and data. Several records show whether network boundaries really exist.

Network policy as segmentation

By default, Pods in a cluster can talk to one another freely. Network policies restrict that, allowing only the connections that are needed. For standards that require separation of sensitive environments, such as payment data, network policies and the network plugin’s enforcement are primary controls. Evidence includes the policies in version control, test results proving that blocked paths are really blocked, and logs of denied connections where the plugin supports it.

Flow logs

Many network plugins, service meshes and cloud networks can record connections: source, destination, port, bytes, and often an allow or deny verdict. Flow logs answer questions such as “which workloads talked to the database last week?” and “did anything outside the payment namespace reach the payment service?” They can be voluminous, so sample or aggregate them where appropriate, and keep the sensitive segments in full.

Ingress and egress

Data access evidence

For systems holding regulated data, the question “who looked at this record?” belongs to the application and database. Database audit features record queries, logins and changes. Combine them with the cluster evidence to show that only authorized workloads and people could reach the data store, and that every access was logged.

Access to nodes and the control plane

• Keep the API server endpoint private or limited to known address ranges, and record the configuration.
• Use bastion hosts or identity-aware proxies for administrator access to nodes, with session recording.
• Disable direct login to nodes where possible, and treat the exceptions as logged, approved events.

Auditors do not ask for logs for their own sake. They ask whether a control works and want evidence. A mapping from controls to evidence makes audits predictable.

A sample map

How to use the map

• Keep it as a living document in version control, owned by a named person.
• For each row, link to the real artifacts, such as the saved query that produces a report, the dashboard, the policy file.
• Where possible, generate evidence automatically on a schedule so that it is always fresh.
• Mark gaps honestly, with a plan and a date. Auditors respect known and managed gaps far more than hidden ones.

Benchmarks and scanners

Tools that check a cluster against the CIS Kubernetes Benchmark or similar standards produce reports that map neatly to many controls. Run them on a schedule, track failures to resolution and keep the reports as evidence of continuous checking.

Audit stress comes from scrambling. Teams that treat compliance as a daily habit find audits a routine event.

Continuous compliance

Instead of collecting evidence once a year, build automation that checks controls all the time and records the results. Examples: a nightly job that verifies the audit policy matches the approved version, a weekly report of new administrator bindings, a monthly test that archives are immutable, and a dashboard showing the pipeline’s health. When the auditor arrives, the evidence already exists.

The audit calendar

What auditors commonly ask for

• A list of users and service accounts with access, and the justification for privileged ones.
• Samples of audit events for specific actions chosen by the auditor, such as “show me who changed this role last March.”
• Proof that logs are retained for the required time, and cannot be altered.
• Evidence of regular log review, with examples of alerts and what was done about them.
• Change records with approvals, and their link to what was deployed.
• Records of access reviews and of removal of departed staff.
• Results of incident exercises.

Be able to answer from the logs quickly

Prepare saved queries for common requests: all actions by a named user between two dates, all changes to roles and bindings in a year, all Secret reads in production, all use of break-glass accounts. Test that they run fast against the archive as well as the hot store. Pulling a year of data from cold storage can take hours; know this ahead of time.

Culture

The best audit posture is an honest one. Teams that explain their controls clearly, show their evidence and admit gaps with a plan build trust. Teams that hide problems or invent stories lose it. Encourage engineers to see auditors as colleagues who test the system from another angle.

The picture gets more complex when clusters are shared by several teams, or when there are many clusters across environments and regions.

Multi-tenant clusters

When several teams share a cluster, the audit log shows all of their actions mixed together. Each team should see its own events, but not others’. A good setup tags events with the namespace and team, filters access by those tags and gives platform administrators the full view. Tenants should be unable to see audit data about other tenants, which may reveal names and activity.

Multi-cluster logging

Ownership

Clarify who owns what: a platform team owns the audit policy, the pipeline and the archive; security owns detection rules and investigation; application teams own application audit events and respond to alerts about their workloads; compliance owns the mapping and the audit relationship. Write this down. Gaps in ownership are where logs quietly break.

Consistency pays off

Fifty clusters each with a different logging setup are fifty separate audit problems. Provision clusters from a common template, which bakes in audit settings, shipping and retention, so that a new cluster is compliant on day one. Then verify with automated checks that no cluster has drifted.

Separate accounts and environments

Keep logs for production in an environment that development staff cannot administer. Many organizations send all security logs to a dedicated, locked-down account owned by the security team, which is read-only for everyone else.

Audit logging is not free. A busy cluster can produce gigabytes per day, and the cost shows up in performance, storage and licensing.

Where the volume comes from

Most events are generated by machines, not people. Controllers, kubelets, schedulers and operators make a constant stream of reads, watches and status updates. A cluster with a thousand nodes may generate millions of events per hour. If the policy records all of them at a detailed level, the log becomes unmanageable and the signal drowns in noise.

What drives the size

Impact on the API server

Audit logging uses CPU and memory in the API server, especially when request and response bodies are recorded or when batching is poorly sized. A blocking backend adds delay to every request. In large clusters, measure API server latency before and after turning on a new policy, and watch the server’s audit metrics for dropped or delayed events.

Controlling cost without losing value

• Spend detail where risk is: sensitive namespaces, privilege changes, access control objects.
• Spend little where risk is low: routine machine chatter.
• Pre-aggregate high-volume signals before storing them, for instance counts per user per hour, while keeping raw events for sensitive classes.
• Compress and move older data to cheap storage after the search period passes.
• Review the top ten talkers monthly and keep trimming.

Do not save money by dropping the wrong things

The temptation to cut costs by shortening retention or excluding categories is strong. Always check with compliance first, and never exclude events that would be needed to reconstruct a security incident or to answer a standard audit question. Savings that cost you an audit finding are not savings.

Estimating before you build

Measure the event rate for a day in a representative cluster with the proposed policy, then multiply by retention. Include the extra copies created by replication and by archives. This estimate guards against sticker shock when the first bill arrives.

Audit logging fails silently. Nothing crashes; the records simply stop being complete. Knowing the typical failures lets you test for them.

Test, do not assume

Because failures are quiet, build tests that prove the system works. A scheduled job performs a known action, such as creating and deleting an object with a distinctive name, and checks that the corresponding audit events arrive in central storage within a time limit. Another job compares each cluster’s audit configuration with the approved baseline. A third attempts a harmless forbidden action and verifies that the denial is recorded.

Upgrades and audit logging

Kubernetes upgrades can change audit event formats, add new fields, add or rename resources and change defaults. After each upgrade, run the end-to-end tests, review a sample of events and confirm that detection rules still match. Add this to the upgrade checklist.

The dangerous gap

The worst failure is the one noticed only when needed: an incident occurs and the logs for the critical hours do not exist. If you cannot afford that, invest in monitoring the monitoring system.

An audit log records the behavior of real people. That brings legal duties and ethical questions that responsible teams think about openly.

Purpose limitation

Records of people’s activity should be used for the stated purposes: security, accountability, compliance and operations. They should not become a general tool to watch employees’ productivity or personal behavior. State the purposes in policy, tell staff that logging occurs and what it is used for, and restrict access to those who need it.

Principles in practice

• Transparency: staff know that administrative actions are logged, and why.
• Minimization: collect what is needed. Do not capture the contents of personal communications or unrelated data.
• Access control: reading logs about people is itself sensitive and should require a reason and be recorded.
• Retention limits: keep identifiable logs only as long as required.
• Proportionate investigation: investigations of individuals follow a defined process, with approval from HR or legal where appropriate.
• Fairness: logs show actions, not intentions. A single event should not be taken as proof of bad faith without context.

Regional rules

Laws differ by country. Some regions give employees rights about how their data is handled, require consultation with worker representatives before monitoring systems are introduced, or restrict moving personal data across borders. Customer data inside application logs adds further obligations. Involve your privacy officer, and design to the strictest rules that apply to you.

Protecting people from log misuse

Service providers

If logs are stored with a third-party service, treat that provider as a data processor. Check where data is held, who at the provider can access it, how it is secured, and what happens at contract end. The convenience of a hosted logging service does not remove your accountability.

This is an illustrative story, not a report about a real company. It shows how the ideas in this guide combine over a year.

The situation

“Paylark,” a fictional payments startup with a team of thirty engineers, ran its services on a managed Kubernetes cluster. A large customer demanded a security audit within six months. The platform lead, Dana, opened the cluster’s settings and found that control plane audit logs were not delivered anywhere, three engineers shared one administrator credential, and nobody had ever reviewed who had access to Secrets.

Month one: get visibility

Dana first switched on control plane audit log delivery, and sent it to a separate security account. She exported cloud account audit trails to the same place. She did not yet write alerts; she simply wanted the data to start flowing, since logs cannot be collected backwards in time. A heartbeat job was created to prove events were arriving.

Month two: identity

The team connected the cluster to the company’s identity provider. Every engineer now logged in with a personal, short-lived credential and multi-factor authentication. The shared administrator credential was locked away as a break-glass secret, with an alert on any use. CI pipelines received their own service accounts with narrow roles. Immediately, the audit log began showing individual names, and the volume of unexplained “admin” activity vanished.

Month three: reduce noise, protect secrets

The first week of data was overwhelming: millions of events from system components. Dana studied the top talkers and shaped a policy that dropped routine machine chatter, recorded Secrets and tokens at Metadata level only, and captured changes to roles, bindings, policies and workloads in production at Request level. Volume fell by about eighty percent, and the log finally became readable. A review of Secret reads revealed that a monitoring tool had been listing all Secrets every minute, which was replaced with a narrower permission.

Month four: detect

The team wrote a short list of alerts: use of the administrator group, new bindings to wide roles, exec into production Pods, privileged Pod creation, anonymous success and any change to audit or policy settings. They tested each one in a staging cluster. In the first week, exec alerts fired on a developer debugging production at night. This led to a conversation, a clearer on-call process and a request-and-approve mechanism for production access.

Month five: evidence and review

Paylark built an evidence map, stored in version control, linking each control to its proof. Quarterly access reviews compared roles with actual use from the logs, and removed a dozen unused permissions. Archives moved into immutable storage with a one-year lock, and a legal hold procedure was written. A mock audit by an outside consultant found two gaps: no record of an exercise of the incident playbook, and unclear ownership of application audit events. Both were fixed.

Month six: the audit

The auditor asked for the list of people with access to production Secrets, samples of audit events for chosen role changes and proof that logs could not be altered. Each request was answered within an hour, using saved queries and the evidence map. The auditor noted one finding, a missing review of node-level logs, which the team added to its roadmap. The customer deal went ahead.

The aftermath

Three months later, the alerts caught a real problem: a CI token leaked in a public repository was used to list Secrets from an unfamiliar address. The audit log gave the team the exact timeline within minutes, and they rotated the token and confirmed that nothing else had been touched.

These common mistakes recur in many organizations. Recognizing them is the first step toward avoiding them.

You do not need everything at once. This staged path shows how to grow from nothing to a confident, continuous program.

flowchart LR
  S0["Stage 0: Blind - no audit logs"] --> S1["Stage 1: Collecting - audit logs on and shipped off the cluster"]
  S1 --> S2["Stage 2: Accountable - personal identities and a shaped policy"]
  S2 --> S3["Stage 3: Protected - immutable storage, retention, access control"]
  S3 --> S4["Stage 4: Watching - tested detections and review"]
  S4 --> S5["Stage 5: Continuous - automated evidence and fleet-wide consistency"]

Reading the diagram

Each box is a stage that builds on the one before it. The path begins in the dark, with no audit logs at all, and moves first to simply collecting them, then to making them meaningful with real identities and a sensible policy, then to protecting them, then to using them actively and finally to producing evidence continuously across every cluster.

What each stage looks like

Questions to place yourself

• If someone deleted a namespace yesterday, could we tell who, within ten minutes?
• Could any administrator delete the evidence of their own actions?
• How long do we keep audit logs, and why that long?
• Which detections have we actually tested?
• How would we know if our log shipping stopped today?
• When did we last perform an access review?

Short answers to questions that come up again and again.

You have walked from the first idea of a diary for the cluster to a full, continuous compliance program. Here it is in a nutshell.

Your first month

Back to top