To understand external secrets management we first need a handful of simple words: secret, cluster, vault, identity and lease.

So what exactly is external secrets management?

External secrets management means storing your secrets in a dedicated system outside the cluster, and letting your applications receive them from there when they need them. The cluster does not hold the master copy. It only carries a short-lived, narrowly permitted copy to the pod that needs it, and the vault keeps the record of who asked and when.

The important idea is separation of duties. Kubernetes is excellent at running applications. A vault is excellent at guarding secrets. Each does the job it was built for, and a small piece of glue connects them. That glue can be a helper container, an operator that copies values into the cluster, or a driver that mounts secrets as files. We meet all of them in later chapters.

The three promises of a good secrets system

A short history

In the early days of containers, teams put passwords in configuration files, environment settings and even inside container images. Kubernetes introduced its own Secret object to give these values a proper home. Later, as companies grew and auditors asked harder questions, teams needed central control, rotation and audit logs that the built-in object was never designed to provide. Dedicated secret managers matured, and integration tools grew up around them. Today the pairing of a cluster with an external manager is a common pattern in production platforms.

Kubernetes Secrets look safe, but they hide a few uncomfortable facts. Understanding them is the reason the whole topic exists.

What is wrong with sticky notes?

A built-in Kubernetes Secret is stored in the cluster’s database, called etcd. By default the values there are only encoded, not encrypted. Encoding is like writing a message in a different alphabet that anyone can read back with a simple tool. It hides nothing from a determined reader. Encryption at rest can be switched on, but it is a setting someone must remember, and it protects only the disk, not the people and programs that are allowed to ask for the secret.

What people did before

flowchart LR
  subgraph W["Built-in secrets only"]
    A1["Password typed into a file"] --> B1["File stored in version control"]
    B1 --> C1["Copied to the cluster database"]
    C1 --> D1["Readable by many people and tools"]
  end
  subgraph V["With an external vault"]
    A2["Password created in the vault"] --> B2["Pod proves its identity"]
    B2 --> C2["Vault checks policy and hands over a short-lived copy"]
    C2 --> D2["Every access is logged"]
  end
    

Reading Figure 1

The left half shows how a password travels when only built-in secrets are used. Someone types it into a file, the file goes into version control, and it is copied into the cluster’s database, where many people and tools may read it. Each step creates another copy that must be guarded. The right half shows the vault approach. The password is created in the vault and stays there. When a pod needs it, the pod proves its identity, the vault checks the policy and hands over a short-lived copy, and the access is logged. There is one master copy, and every use leaves a trail.

A few building blocks appear again and again in Vault and in every integration. Learn them once and everything else will make sense.

Static and dynamic secrets

Vault can hand out two very different kinds of secrets. A static secret is a fixed value stored in Vault, such as an API key issued by a partner. Vault keeps it safe and controls access, but the value itself stays the same until someone changes it. A dynamic secret is created on demand. When an application asks for database access, Vault connects to the database, creates a brand-new user with a random password, hands the credentials to the application, and deletes the user automatically when the lease ends. Every application instance gets its own login, and stolen credentials expire on their own.

Three ideas to keep straight

The sealed vault

Vault stores its data encrypted. The key that unlocks it is itself protected, and when Vault starts it is sealed: it cannot read anything until it is unsealed. This is a deliberate safety feature. A stolen copy of Vault’s storage is useless without the unseal keys. In many setups, unsealing is automated using a cloud key service, so that restarts do not need a human to type keys at three in the morning.

Several pieces cooperate between a pod and a stored secret. It helps to see who talks to whom.

flowchart TB
  POD["Application pod with a service account"] -->|"Shows identity token"| AUTH["Vault auth method for Kubernetes"]
  AUTH -->|"Asks the cluster: is this token genuine?"| API["Kubernetes API server"]
  API -->|"Yes, this pod belongs to this service account and namespace"| AUTH
  AUTH -->|"Issues a short-lived Vault token with policies"| POD
  POD -->|"Requests a secret using the token"| ENG["Secrets engine"]
  ENG --> STORE["Encrypted storage"]
  ENG --> AUD["Audit log"]
  OP["Helper: injector, operator or CSI driver"] -.->|"Does the fetching for the pod"| POD
    

Reading Figure 2 piece by piece

Follow the arrows from the top left. An application pod carries an identity token issued by Kubernetes. It shows the token to Vault’s Kubernetes auth method. Vault does not simply trust the token. It asks the cluster’s own API server whether the token is genuine and which service account and namespace it belongs to. When the cluster confirms, Vault looks up the matching role, issues its own short-lived token carrying the right policies, and hands it back.

With that token, the pod can ask a secrets engine for a value. The engine reads from encrypted storage and writes an entry to the audit log. In practice, the pod’s application rarely does all this itself. A helper, drawn with a dotted line, does the fetching on its behalf. That helper may be an injector, an operator or a storage driver, and we compare them in Chapter 7.

The parts, one by one

Where Vault runs

Vault can run outside the cluster on dedicated machines, inside the same cluster, or in a separate management cluster. Running it outside the workload cluster is the classic choice: the vault does not depend on the thing it protects, and a problem in the cluster cannot take the vault down with it. Running it inside is simpler to operate for small teams but needs extra care, because the cluster now depends on a component that runs on the cluster. Many organisations use a dedicated platform cluster or machines for Vault and let many workload clusters connect to it.

Now let us open the box. We will follow the login step by step, then see how policies, leases and renewals work.

The login: proving a pod is who it says it is

Every pod in Kubernetes runs under a service account, and the cluster gives the pod a signed token for that account. The token contains claims such as the account’s name, its namespace and an expiry time. When the pod shows this token to Vault, Vault passes it to the cluster’s token review service, which answers whether the token is valid and who owns it. Only after a positive answer does Vault look for a matching role.

flowchart TD
  S["Pod presents its service account token"] --> A{"Does the cluster confirm the token is genuine and not expired?"}
  A -->|"No"| X["Login refused"]
  A -->|"Yes"| B{"Does a Vault role name this service account?"}
  B -->|"No"| X
  B -->|"Yes"| C{"Is the pod's namespace allowed by the role?"}
  C -->|"No"| X
  C -->|"Yes"| D["Vault issues a short-lived token with the role's policies"]
  D --> E["Pod can read only what those policies allow"]
    

Reading Figure 3

The flow has three checkpoints, each one a chance to say no. First, the cluster itself must confirm the token is genuine and unexpired, so a forged or stale pass fails immediately. Second, a Vault role must exist that names this exact service account, so a genuine pod with no matching role gets nothing. Third, the pod’s namespace must be one the role allows, so a service account with the same name in a different namespace is turned away. Only when all three pass does Vault issue a token, and that token carries only the policies attached to the role.

Roles bind identity to permissions

A role is the bridge between the cluster’s world and Vault’s world. It says, in effect: “pods running as this service account in this namespace may log in, and while logged in they receive these policies and this token lifetime”. A well-designed role is narrow. It names specific accounts and specific namespaces, rather than allowing everyone, and attaches policies that reach only the secrets that application really needs.

Why this design resists theft

An attacker who steals a service account token still needs the cluster to confirm it, so an expired token fails. An attacker who steals a Vault token gets only the policies of that role, and only until the token expires. An attacker who steals a dynamic database credential holds something that will delete itself. Each layer limits the damage of the layer before it, which is the whole point of the design.

Let us follow one secret from the moment a pod starts to the moment the pod is gone.

sequenceDiagram
  participant P as Pod helper
  participant K as Kubernetes API
  participant V as Vault
  participant D as Database
  P->>V: Log in with service account token
  V->>K: Is this token genuine?
  K-->>V: Yes, account and namespace
  V-->>P: Short-lived Vault token
  P->>V: Request database credentials
  V->>D: Create a temporary user
  D-->>V: User created
  V-->>P: Credentials with a lease
  P->>P: Write credentials for the application
  P->>V: Renew the lease before it ends
  P->>V: Pod ends, credentials revoked
  V->>D: Remove the temporary user
    

Reading Figure 4

The story begins with the helper in the pod logging in to Vault with the service account token. Vault confirms the token with the cluster and returns its own short-lived token. The helper then asks for database credentials. Vault connects to the database and creates a temporary user, returns the credentials with a lease, and the helper writes them where the application can read them. While the pod lives, the helper renews the lease before it runs out. When the pod ends, the credentials are revoked and Vault removes the temporary user from the database, leaving nothing behind.

Six stages of a secret’s life

Three ways to handle rotation

Rotation means changing a secret on a schedule or after an incident. There are three broad strategies, and each suits different applications.

Whichever strategy you use, rehearse it. A rotation that has never been tested is an outage waiting for a calendar date.

What if something goes wrong?

If Vault is unreachable at start-up, the helper cannot fetch secrets and the pod usually waits or fails, depending on how the helper is set up. If Vault becomes unreachable later, the application keeps using the values it already has until they expire. Long leases therefore give resilience, and short leases give safety. If a secret is rotated while pods are running, applications that read the value only at start-up keep the old one until they restart or reload. This is why rotation needs a plan for how running applications learn about new values.

There are several ways to carry a secret from Vault into a pod. Each has its own personality, strengths and traps.

flowchart TB
  subgraph I["Pattern A: Agent injector"]
    I1["Webhook adds helper containers to the pod"] --> I2["Helper logs in and fetches secrets"] --> I3["Writes files to a shared memory folder"]
  end
  subgraph O["Pattern B: Operator that syncs"]
    O1["Operator watches custom resources"] --> O2["Fetches from the vault"] --> O3["Creates or updates a cluster secret"]
  end
  subgraph C["Pattern C: CSI driver"]
    C1["Pod declares a secrets volume"] --> C2["Driver fetches at pod start"] --> C3["Mounts secrets as files"]
  end
  subgraph D["Pattern D: Application talks to Vault"]
    D1["Application uses a client library"] --> D2["Logs in and reads directly"]
  end
    

Reading Figure 5

Each box shows a different road. In pattern A, a special component watches for new pods and quietly adds helper containers to those that ask for secrets. The helpers log in, fetch the values and write them as files into a shared memory folder that the application reads. In pattern B, an operator copies values from the vault into ordinary cluster secrets, and pods use them the usual way. In pattern C, the pod asks for a special volume, and a storage driver fetches the secrets at start-up and mounts them as files. In pattern D, the application itself talks to the vault using a client library.

Pattern A: the agent injector

The injector uses a hook that watches pods as they are created. A pod that carries certain labels or notes is modified: an init container is added to fetch secrets before the application starts, and a sidecar container keeps them fresh afterwards. Secrets are written as files into a shared in-memory folder. The application knows nothing about Vault; it simply reads files. The cost is one or two extra containers in every pod that uses it. This pattern echoes what we saw with init containers: a helper prepares the pod before the application begins.

Pattern B: operators that sync to cluster secrets

An operator watches special custom resources that describe which secrets to fetch, from where, and how often to refresh. It then creates or updates ordinary cluster secrets with the fetched values. Two well-known families exist: operators from the vault vendor, and general-purpose operators that support many secret managers behind a common interface. Pods consume the resulting cluster secrets exactly as they always have, through environment values or mounted files.

Pattern C: the CSI driver

The secrets store CSI driver lets a pod declare a volume whose contents come from an external store. When the pod starts, the driver logs in to the vault, fetches the requested secrets and mounts them as files. Optionally it can also create a cluster secret from them. Because the fetch happens as part of mounting, a pod cannot start until its secrets are ready, which is a useful safety property.

Pattern D: the application talks to Vault directly

The application includes a Vault client library, logs in with its service account token, and reads what it needs. This gives the most control: it can renew leases, react to rotation instantly and request dynamic secrets exactly when needed. The price is that Vault-specific code enters every application, which teams must write, test and maintain, and which ties the application to Vault.

A single table cannot decide for you, but it shows the shape of the differences at a glance.

Other names you will meet

Secrets that are not passwords

Secrets management is not only about passwords. Vault can also act as a private certificate authority, issuing short-lived certificates so that services can prove their identity to each other and encrypt their traffic. It can offer encryption as a service, where an application sends data to be encrypted or decrypted and never holds the key itself. It can hand out temporary cloud access credentials so that no long-lived cloud keys sit in a cluster. These features show why many teams choose a full vault rather than a simple store: one policy model, one audit trail and one set of habits can cover passwords, certificates, keys and cloud access together.

Managed secret stores versus self-run Vault

A managed store removes the burden of running the vault yourself: no servers to patch, no unseal to plan, and the provider handles availability. The price is tighter ties to one provider and fewer advanced features. Running Vault yourself, or using a managed Vault offering, gives richer features such as dynamic secrets across many systems, and works the same across clouds and on your own servers, at the price of operating a critical service. Many organisations run one central Vault and connect all clusters to it, and some mix in a managed store for simple cases.

A vault protects secrets only as well as the arrangement around it. This chapter covers the weak points and how to close them.

The weak points people forget

The secret zero problem

Every secret system faces a puzzle: to fetch secrets you need an identity, and an identity looks a lot like a secret. If a pod needs a password to talk to Vault, where does that password come from? Kubernetes’ built-in identity solves it neatly. The cluster already knows each pod and issues a signed, short-lived token automatically, so no human-typed password is needed for the first step. Vault trusts the cluster’s answer about who the pod is. The chain of trust therefore rests on the cluster’s own identity system, and protecting the cluster’s service accounts and API access becomes essential.

Multi-tenancy and separation between teams

When many teams share one vault, separation becomes a design question. Vault offers ways to divide it into isolated areas so that one team’s administrators cannot see or change another’s secrets, and different editions provide different tools for this. At a minimum, use clear path prefixes per team and environment with policies that never cross them. Give each team’s administrators rights only in their own area. Keep production and non-production apart, ideally with separate vaults or separate isolated areas and separate administrators, so that a mistake in testing can never touch a production secret.

Separate audit trails per team are valuable too. When a security review asks who touched a payment credential, the answer should come from a log that is easy to filter and hard to alter.

Practical hardening checklist

There is no single best pattern, only the best fit for your platform, your applications and your people. A few questions get you most of the way.

flowchart TD
  A["How should my pods receive secrets?"] --> B{"Can I change the application code?"}
  B -->|"Yes, and I want fine control"| C["Application talks to the vault directly"]
  B -->|"No, or I prefer not to"| D{"Do I need dynamic secrets and lease renewal?"}
  D -->|"Yes"| E["Agent injector"]
  D -->|"No, mostly static values"| F{"Must the pod refuse to start without its secrets?"}
  F -->|"Yes"| G["CSI driver"]
  F -->|"No, and my apps already read cluster secrets"| H["Sync operator"]
    

Reading Figure 6

Begin with the top question: can you change the application? If yes, and you want the finest control, letting the application talk to the vault directly is an option, at the cost of library code. If you prefer not to touch the application, ask whether you need dynamic secrets and lease renewal. If yes, the agent injector supports them best. If your secrets are mostly static, ask whether the pod must refuse to start without them. If so, the CSI driver gives that property naturally. If not, and your applications already read cluster secrets, the sync operator is the least disruptive choice.

The questions behind the chart

Once every pod depends on the vault, the vault becomes part of your critical path. Plan for its speed and its failures.

Where the time goes

Ways to go faster and stay safer

Use caching where the pattern allows it, so that repeated reads within a short time do not each hit the vault. Choose sensible lease lengths: longer leases mean fewer renewals but weaker safety. Spread mass restarts so that hundreds of pods do not log in at once. Run several Vault servers for availability and consider read replicas or performance standby servers where your edition provides them. Scale the helpers’ resource requests to realistic values, and keep secret sizes small.

Many clusters, one vault

Large organisations often connect dozens of clusters to one central Vault. Each cluster gets its own login path, so that a service account named “orders” in one cluster is not confused with an identically named account in another. The vault trusts each cluster separately, and roles are defined per cluster. This keeps clusters isolated from each other while giving the security team one place to see and control access. The price is that the vault serves a much larger population of pods, so capacity planning, high availability and careful rate management matter more.

Latency also matters. A pod that must cross a slow or distant network to reach the vault will start slowly, and the login and fetch steps are sensitive to delay. Placing the vault close to the clusters that use it, or using replicated read capacity near each region where your edition allows it, keeps start-up times reasonable.

Designing for a vault outage

Assume the vault will be unavailable sometimes: for upgrades, for network problems, for a bad day. Decide what your applications do in that moment. Applications that already hold their secrets should keep running, so use leases and token lifetimes long enough to ride out short outages. New pods that cannot fetch secrets should wait with clear messages and retry, instead of starting with defaults. Practise the failure in a test environment, including a full loss of a zone, and keep a documented recovery procedure with restore tests of Vault’s backups.

Every choice buys something and costs something. Here is an honest ledger for external secrets management.

Trade-offs in plain words

The first trade-off is security against operational burden. A central vault raises security and visibility, and at the same time it becomes something you must run well. For a small team with a handful of static secrets, the built-in objects with encryption at rest and tight access rules may be enough, and the burden of a vault may not be worth it yet.

The second is transparency against control. Injectors and operators hide the vault from applications, which is easy for developers, but they limit fine control over renewal and rotation. Direct access gives control, and puts vault code into every application.

The third is safety against convenience of start-up. Making pods wait for their secrets is safe, and can slow or block starts during outages. Letting them start and fail later is faster, and creates confusing partial failures.

The fourth is freshness against stability. Short leases and frequent refresh limit the life of leaked secrets but create more traffic and more chances for something to go wrong. Longer lifetimes are calmer and weaker. Pick lifetimes that match how sensitive each secret is.

When a secret does not arrive, the cause lies in one of three stages: identity, authorisation or delivery. A calm routine finds it quickly.

Symptoms and where to look

A calm troubleshooting routine

First, identify the stage. Did the pod fail to log in, fail to read, or fail to receive? Second, check the pod: which service account does it use, and in which namespace? Third, compare with the role: does a role name exactly that account and namespace? Fourth, compare the policy with the exact path the application requests, remembering that engines may add path prefixes that trip people up. Fifth, read the helper’s or operator’s logs, which usually quote the vault’s reply. Sixth, check Vault’s audit log for the failed request; it shows precisely which identity asked for which path and why it was refused. Seventh, check the vault’s own health: sealed, overloaded or unreachable.

Testing the integration before you trust it

A short experiment beats a long debate. In a test cluster, create a test service account, a narrow role and a policy for one test secret. Start a pod with that account and confirm it can read the secret. Then start a second pod with a different service account and confirm it is refused, and check that the refusal appears in the audit log with the right identity and path. Change the secret and watch how long the running pod takes to see the new value. Stop the vault, or block its network, and observe what happens to a running pod and to a new pod. Finally, revoke a lease and confirm that the credential really stops working. Keep the results in your team’s notes, together with the versions used, and rerun the same experiment after every upgrade. Ten minutes of testing here saves hours of guessing during a real incident.

What to monitor all the time

Track Vault’s availability and sealed state, request rates and latency, failed logins, permission denials, lease counts, token counts, storage health and audit device health. Alert on a sealed vault immediately, on sudden spikes of denials, which may mean an attack or a broken deployment, and on secrets close to expiry without renewal. On the cluster side, watch helper container restarts, operator sync errors and pods waiting on secrets for longer than a few minutes.

A pattern is a proven way of solving a repeated problem. An anti-pattern is a tempting habit that keeps causing pain.

Patterns worth adopting

Anti-patterns to avoid

This chapter gathers the advice into checklists you can keep beside your desk.

Best practices

Common mistakes

A platform guide for your teams

If you run a platform for other teams, write a one-page guide and keep it current. It should say which integration pattern is standard, how to request a role and a secret path, how rotation works, what to do during a vault outage, and who to contact. Include a short decision chart like the one in Chapter 10 and a few sentences on the most common traps. Developers who can read a page like this rarely need to ask for help, and the platform team spends far less time on access tickets.

Make the safe path the easy path. Provide templates that already contain a narrow role, a sensible lease length and a correct pattern, so that teams start secure by default and adjust from there.

These generic stories, based on common patterns rather than any specific company, show how the ideas play out.

Three worked scenarios: thinking like a designer

A short mental checklist for a new application

Before connecting a new application to the vault, pause and walk through a short list. State which secrets it needs and whether each could be dynamic. Give it its own service account. Write a role that names that account and namespace only. Write a policy for exactly the paths it needs. Choose the delivery pattern and decide where the values will land. Decide how rotation will reach running instances. Decide what the application does if the vault is unavailable. Check that audit logs will show its access. Ten minutes on this list prevents most of the problems described in this article.

Quick answers to the questions beginners ask most often.

This post was written without live web research. Please verify version-sensitive details, such as feature names, defaults and supported providers, against the official documentation of each tool you plan to use.

Let us close the loop, returning to the hotel key room from the start of this article.

Just as a hotel is safer with a guarded key room, a logbook and keys that expire on their own, a cluster is safer when its secrets live in a proper vault outside the cluster. Pods prove who they are, receive only what a policy allows, and every access leaves a trail. Dynamic secrets limit the harm of any leak by expiring on their own.

You met the problem of sticky-note secrets, the building blocks of engines, auth methods, roles and policies, and the architecture that connects pods to Vault. You followed a secret’s life, compared the four integration patterns, learned about security, performance, trade-offs and troubleshooting, and saw the mistakes that trap teams most often.

Version drift and things to verify

This area of Kubernetes changes quickly. Tool names, installation methods, supported providers, defaults and even the recommended patterns have shifted over the years, and several projects have merged, split or been renamed. Because this article was written without live web research, treat version-specific statements as a starting point. Before relying on a feature in production, read the official documentation for the exact versions you will run, try the behaviour in a test cluster, and record the versions in your team’s notes so upgrades prompt a re-check. Also verify licence and edition differences for Vault features, since some capabilities depend on the edition you use.

A small glossary to keep nearby

Key takeaways