Before Kubernetes enters the picture, we need to agree on what “secret” means in software.

The plain-English definition

A secret is any small piece of data that gives power to whoever holds it. A password lets you log in. An API key lets you spend someone’s money or read someone’s data. A private certificate key lets you pretend to be a trusted website. If a stranger gets one of these, they can act as if they were you, which is why we protect them more carefully than ordinary settings.

Ordinary settings, such as the colour of a button or the name of a country, are not secret. If they leak, nothing bad happens. The whole idea of a Secret in Kubernetes is to give sensitive values a separate home, with separate rules, away from the ordinary settings.

Why not just put secrets in the code?

Beginners often ask why they cannot simply type the password into the program. The trouble is that code travels. It gets copied to laptops, uploaded to shared repositories, packed into images, and read by many colleagues. A password written into code follows it everywhere, and you can never fully take it back once it has been copied.

Changing a password also becomes painful. If the password is baked into the program, you must rebuild and redeploy the whole application just to change one word. Keeping secrets outside the program means you can change them without touching the program at all.

A quick tour of the building blocks you need before Secrets make sense.

Kubernetes is a system that runs and manages applications packed into small, portable boxes called containers. Instead of you starting programs by hand on individual computers, you tell Kubernetes what you want, and it keeps making that true. If a program crashes, Kubernetes restarts it. If traffic grows, Kubernetes can start more copies.

Keep the last two blocks in mind. Nearly every security question about Secrets comes back to who can talk to the API server and who can peek inside etcd.

A Secret is a Kubernetes object designed to hold a small amount of sensitive data.

The core idea

A Kubernetes Secret is a named object stored in the cluster that holds one or more key and value pairs. The keys might be “username” and “password”, and the values are the actual sensitive data. Applications running in pods can be given access to these values without the values ever being written into the application’s image or its deployment description.

Kubernetes was built with a clear division of labour. The person who knows the password creates the Secret. The person who writes the application simply says “this pod needs that Secret”. The real value never has to be copied around by hand between them.

Why Secrets exist as a separate object

Kubernetes already has another object for ordinary settings, called a ConfigMap. Secrets are a sibling of ConfigMaps, but they are treated with extra care. Secrets can be given tighter access rules, are kept away from casual logs and printouts, and are handled more carefully on the nodes that use them.

What a Secret looks like inside

A Secret has a name, lives in a namespace, has a type that describes its purpose, and carries a bundle of data entries. The data values are stored in an encoded form called base64. Beginners nearly always misunderstand this last point, so it deserves its own chapter later. For now, remember only this: base64 is a way of writing data, not a way of hiding it.

Kubernetes labels each Secret with a type so that tools know what to expect inside it.

The type tells Kubernetes and other tools what shape of data the Secret should contain. Some types are checked by Kubernetes itself, for example by insisting that certain keys are present. Others are simply labels to help humans and tools.

A closer look at the TLS type

The TLS type is one of the most common in daily life. Whenever you visit a website with a padlock in the browser, a certificate and a private key are working behind the scenes. In Kubernetes, the certificate and key are kept in a TLS Secret, and the component that handles incoming web traffic reads them from there. Because the private key lets anyone impersonate your website, this Secret is among the most valuable in the cluster.

A closer look at the registry type

Companies rarely publish every container image to the public internet. Private images sit in private registries that demand a login. The Docker Config type stores that login, and Kubernetes uses it when a node needs to pull an image. Without it, the pod would be stuck waiting, unable to fetch its own software.

Creating a Secret is only half of the story. The other half is getting it safely to the application that needs it.

There are three main ways a pod can receive the contents of a Secret. Each has strengths and weaknesses, and choosing well is one of the first real security decisions you will make.

flowchart LR
    S["Secret object in the cluster"] --> E["Environment variable"]
    S --> F["Mounted file"]
    S --> R["Registry login for image pull"]
    E --> A["Application in the pod"]
    F --> A
    R --> N["Node downloads private image"]
    N --> A
        

The diagram shows that one Secret can feed an application in different ways. The registry route is a little different because the node, not the application, is the one using the Secret, but the end result is the same: the application is able to start and do its work.

Let us follow a Secret from the moment it is created to the moment an application reads it.

Step one: the Secret is stored

When someone creates a Secret, the request goes to the API server. The API server checks who is asking, checks whether that person is allowed to create Secrets, and then writes the Secret into etcd, the cluster’s database. From this moment, the Secret exists as a permanent record.

Step two: the Secret is scheduled to a node

When a pod that needs the Secret is placed on a node, the kubelet, which is the small manager program running on every node, notices. The kubelet asks the API server for the Secret. The API server again checks permissions, and this time the answer is limited: a node may only receive the Secrets needed by pods actually running on it.

Step three: the Secret is delivered to the pod

The kubelet places the Secret’s data into the pod. For mounted files, the kubelet stores the data in memory-backed storage on the node rather than on the hard disk. This design means that the secret values do not get written into permanent disk storage, and they vanish when the pod is removed.

sequenceDiagram
    participant Admin as Administrator
    participant API as API Server
    participant DB as etcd
    participant Kubelet as Kubelet on Node
    participant Pod as Application Pod
    Admin->>API: Create Secret
    API->>API: Check identity and permissions
    API->>DB: Store Secret
    Kubelet->>API: Request Secret for a scheduled pod
    API->>Kubelet: Send Secret if node is authorised
    Kubelet->>Pod: Provide as file in memory or as variable
    Pod->>Pod: Application reads the value
        

Automatic updates for mounted files

When a Secret is changed, the kubelet periodically checks for updates and refreshes the mounted files. There is a short delay, often up to a minute or so, before the new value appears. Applications that reread their files can pick up the change without a restart, which makes password rotation far smoother.

Every secret goes through stages, from birth to retirement. Good teams plan for all of them.

Most beginners think only about the first four steps. Experienced engineers spend just as much time on the last two, because a secret that can never be replaced or retired is a risk that grows every year.

The single most important lesson for anyone new to Kubernetes Secrets.

When you look at a Secret’s stored data, the values look like a jumble of letters and numbers. It is natural to think they have been scrambled for safety. They have not. The jumble is base64, a method of writing any data using only ordinary letters, digits and a few symbols, so that it can travel safely inside text files.

Anyone can turn base64 back into the original text in a second, with no password and no special tool. It is like writing a message in a different alphabet that everybody in the world happens to know.

So why does Kubernetes use it?

Secret values can be binary data, such as certificates and keys, which do not fit neatly into text files. Base64 lets Kubernetes carry any kind of data inside its text-based descriptions. It is a convenience for transport, not a protection.

Since Secrets are not safe by default, security is built in layers around them.

Layer 1: Control who can read Secrets

Kubernetes uses a system called role-based access control, usually shortened to RBAC. It decides who may do what to which objects. The golden rule is least privilege: give each person and each program only the exact access they need, and nothing more.

Be especially careful with permissions to list or watch Secrets. Someone who is allowed to list all Secrets in a namespace can read every value, even if they were never told the names. Also remember that anyone allowed to create a pod in a namespace can point that pod at any Secret in the same namespace and read it. Pod creation rights are therefore a hidden form of Secret access.

Layer 2: Encrypt Secrets at rest

Kubernetes can be configured so that the API server encrypts Secret data before writing it into etcd. With this turned on, a stolen copy of etcd is unreadable without the encryption key. The strongest form uses an external key management service, so the key that unlocks the Secrets is never stored next to the Secrets themselves.

Layer 3: Protect traffic

Secrets travel across the network between the API server, etcd and the nodes. Every one of these paths should use encrypted connections. Kubernetes components normally do this by default, but it is worth verifying in any environment you build yourself.

Layer 4: Separate with namespaces and nodes

Namespaces divide Secrets into groups, and RBAC can be applied to each group separately. For highly sensitive workloads, teams also pin them to dedicated nodes, so that a compromise of a lower-value application cannot reach the memory of the machine holding critical secrets.

Layer 5: Watch everything

Audit logs record who read which Secret and when. Reviewing them, and raising alarms on unusual reads, turns silent theft into visible events. We look at this in more detail in a later chapter.

flowchart TD
    A["Secret data"] --> B["Layer 1: RBAC limits who can read"]
    B --> C["Layer 2: Encryption at rest in etcd"]
    C --> D["Layer 3: Encrypted network traffic"]
    D --> E["Layer 4: Namespace and node separation"]
    E --> F["Layer 5: Audit logs and alerts"]
        

Many companies decide that the cluster should not be the true home of their most valuable secrets.

An external secret manager is a dedicated service built only for storing, protecting, rotating and auditing secrets. Examples include HashiCorp Vault, and the managed offerings of the major cloud providers. These services offer stronger features than a plain Kubernetes Secret: automatic rotation, detailed history, short-lived credentials and fine control of access.

The challenge is bringing those secrets to the applications inside the cluster. Several popular approaches exist.

Changing secrets safely is just as important as storing them safely.

Why rotate at all?

Every day a secret exists, the chance that it has leaked grows a little. People leave companies, laptops get lost, and logs sometimes capture things they should not. Rotation limits the damage by making sure that a stolen secret expires and becomes useless.

Rotation should be routine, not a panic reaction. A team that rotates its secrets every few weeks will handle a real emergency calmly, because the process is already tested and familiar.

How rotation reaches the application

How smoothly a rotation goes depends on how the Secret is delivered. Mounted files can refresh on their own, and an application that rereads its files will pick up the new value. Environment variables are locked in at startup, so those pods must be restarted. Good applications are designed to reload their credentials, or to fail gracefully and retry, when a password changes.

Immutable Secrets

Kubernetes allows a Secret to be marked immutable, meaning its contents can never be edited once created. This sounds restrictive, but it brings real benefits. It prevents accidental changes that could break every pod at once, and it reduces the load on the API server because the kubelet no longer needs to keep checking for updates. To change an immutable Secret, you create a new one with a new name and switch your pods over to it, which also gives you a clean, reversible rollout.

You cannot protect what you cannot see.

Even with strong locks, a careful team assumes that something may eventually go wrong and prepares to notice it quickly. For Secrets, this means keeping a record of every access and watching that record for strange behaviour.

Audit logs

The API server can record every request it receives: who asked, what they asked for, and whether it was allowed. For Secrets, the important recording level captures the fact that a read happened without writing the secret value itself into the log. A poorly configured audit log that copies the secret data would simply create a second, less protected copy of the secret.

What to watch for

Keeping secrets out of ordinary logs

Applications sometimes print their whole configuration when they start, or dump everything during an error. If a secret is in that output, it now sits in the logging system, where many more people can read it. Teams should review what their applications print, and log collection tools can be set to mask values that look like passwords or keys.

Patterns are proven habits that work. Anti-patterns are popular mistakes that keep hurting teams.

Patterns worth adopting

Anti-patterns to avoid

An honest look at what built-in Kubernetes Secrets give you, and what they do not.

The takeaway is not that Kubernetes Secrets are bad. They are a useful foundation. But they are a foundation, not a finished building. The security of your secrets depends on what you build around them.

A checklist of habits that separate careful teams from careless ones.

Best practices

• Turn on encryption at rest — ideally with a key held in an external key management service, so the lock and the key are never stored together.
• Apply least privilege everywhere — narrow RBAC roles, per-namespace permissions and no casual cluster-admin access.
• Prefer mounted files over environment variables — files are easier to protect and can refresh without a restart.
• Keep secrets out of code, images and repositories — use sealing tools or external managers for anything that must be version controlled.
• Rotate on a schedule — practise rotation regularly so it is never a stressful, untested event.
• Use separate credentials per application — so a single leak cannot open every door.
• Audit and alert — record reads of Secrets, and raise alarms on unusual patterns.
• Prefer short-lived and identity-based access — the best secret is the one that does not need to be stored.

Common mistakes

How the ideas in this guide show up in everyday engineering.

Only to the extent that you protect it. By default the values are merely base64 encoded and are stored unencrypted in etcd. With encryption at rest, tight access rules and auditing, a Secret becomes properly protected.

Both hold configuration for applications. A ConfigMap is meant for ordinary, non-sensitive settings. A Secret is meant for sensitive values and receives extra handling, such as memory-backed storage on nodes and separate access rules.

Mounted files are usually the safer choice. They are less likely to appear in logs and crash reports, they can carry strict permissions, and they can refresh without restarting the pod. Environment variables are simpler but leak more easily.

A pod can only use Secrets from its own namespace. However, someone with broad permissions across namespaces can still read Secrets in all of them, which is why access rules matter so much.

Not always. For small or low-risk setups, well-configured built-in Secrets can be enough. As the number of applications, the sensitivity of the data or regulatory demands grow, an external manager brings rotation, short-lived credentials and better audit trails.

The Secret object stays in the cluster. Only the copy delivered to that pod, including the memory-backed files on the node, is removed along with the pod.

There is no single answer, but a common approach is every few weeks to a few months for long-lived secrets, and immediately after any suspected leak or staff departure. Short-lived credentials that expire on their own reduce the need for manual rotation.

Only if the backups themselves are encrypted and restricted. A backup of etcd contains all the Secrets, so it must be guarded with the same care as the live cluster.

Secrets are stored in etcd and sent through the API server. Very large objects would slow down the whole cluster, so each Secret is limited to about one mebibyte. Secrets are meant for small values like keys and passwords, not for large files.

Start by tightening who can read Secrets and turning on encryption at rest. These two steps close the most common gaps with modest effort.

Kubernetes Secrets give applications a dedicated, separate place to receive passwords, keys and tokens, so that these values need not live in code or images. They travel from the API server and etcd, through the kubelet, into pods as files or environment variables, and they are treated with more care than ordinary settings. But out of the box, they are only encoded, not encrypted, and real safety comes from the layers you add: strict access rules, encryption at rest, careful delivery, rotation, monitoring and, for the most sensitive cases, an external secret manager.

Treat secrets the way a good restaurant treats its safe: known location, few keys, regular code changes and a record of every opening. Do that, and Kubernetes becomes a trustworthy home for the private parts of your software.