Immutable ConfigMaps and Secrets in Kubernetes
How one small switch turns settings and passwords into sealed, unchangeable objects, why that protects you from silent mistakes and heavy cluster load, and how to roll out changes safely with versioned names, explained from zero.
Picture a restaurant kitchen with a whiteboard that lists today’s recipe: how much salt, how much sugar, which oven temperature. Any cook can walk up and rub out a number and write a new one. One busy afternoon, a cook mistypes the sugar amount. Within minutes, every dish leaving the kitchen tastes wrong, and nobody can tell who changed the board or when.
Now imagine that instead of a whiteboard, the restaurant prints each recipe on a laminated card and numbers it: Recipe card version 7, version 8. Nobody can edit a printed card. To change the recipe, you print version 9 and hand it to the cooks, who switch to it on purpose. If version 9 turns out to be wrong, you go back to version 8, which is still sitting in the drawer.
Kubernetes has exactly this choice for its configuration objects. A ConfigMap holds ordinary settings, and a Secret holds sensitive values like passwords. By default, both are like whiteboards: anyone with permission can edit them at any time, and running applications may see the change. With a single field, you can make them like laminated cards instead. In this guide you will learn what immutable ConfigMaps and Secrets are, why they exist, how they behave inside the cluster, how to roll out changes with versioned names, and how to run the pattern safely in production. Every new word is explained the first time it appears.
1What Are Immutable ConfigMaps and Secrets?
Two familiar objects plus one simple switch.
A ConfigMap is a Kubernetes object that stores settings as key and value pairs, for example “page color equals blue” or “log level equals info”. Applications read these settings when they start or while they run, so that the settings live outside the application’s container image.
A Secret is almost the same idea, but meant for sensitive values such as passwords, tokens and keys. It is handled a little more carefully by Kubernetes, though, as we will see in the security chapter, it is not a magic vault by default.
The word immutable means “cannot be changed”. Both objects have an optional field named immutable. When you set it to true, Kubernetes seals the object. Its data can never be edited again. To make a change, you must delete the object and create a new one.
A normal ConfigMap is a page in a notebook written in pencil: easy to erase and correct, but also easy to smudge by accident. An immutable ConfigMap is a page written in ink and then laminated. If you want different words, you write a new page. The old page is not damaged, and everyone can see exactly which page each person is reading.
2The Problem: Why Mutable Configuration Is Risky
Being able to edit live configuration sounds convenient. In production, that convenience has sharp edges.
Silent blast radius
One edit to a shared ConfigMap can reach every Pod that mounts it, all at once, with no rollout, no health check and no chance to stop. A typo becomes a fleet-wide outage.
Inconsistent behavior
Files mounted from a ConfigMap update slowly, but environment variables never update until restart. Some Pods see the new value and others still hold the old one, and nobody notices.
Broken rollbacks
You roll a Deployment back to the previous version, but the ConfigMap still holds the new settings. The old code now runs with the wrong configuration.
Weak audit trail
When an object is edited in place, the previous value is gone from the cluster. Working out what changed and when means digging through logs.
Load on the control plane
Every machine keeps watching the ConfigMaps and Secrets its Pods use. In a big cluster, tens of thousands of such watches put steady pressure on the central API server.
Immutability answers all five. A sealed object cannot spread an accidental edit, cannot cause half-updated fleets, keeps old versions available for rollback, leaves a clear trail because changes are new objects, and lets the cluster stop watching it.
3Foundations: How Pods Use ConfigMaps and Secrets
To see what immutability changes, first see how applications receive their settings.
A Pod is the smallest unit Kubernetes runs: one or more containers sharing a network address. A Pod can receive settings from a ConfigMap or Secret in three ways.
| Method | How it looks to the app | Updates while running? | Best for |
|---|---|---|---|
| Environment variables | Named values in the process environment | No. Fixed at container start. | Simple single values |
| Mounted volume | Files in a folder, one file per key | Yes, slowly, if the object is mutable and mounted as a whole folder | Config files, certificates |
| Mounted with a sub-path | A single file placed inside an existing folder | No. Sub-path mounts do not receive updates. | Adding one file to a folder without hiding others |
| Command arguments | Values substituted into the start command | No. Fixed at container start. | Passing options to the program |
Where the data lives
ConfigMaps and Secrets are stored in the cluster’s database, called etcd, and served by the API server, which is the front desk of Kubernetes. On every machine, an agent called the kubelet fetches the objects that its Pods need and places them where the containers can read them.
Size and shape
Each object may hold up to about one mebibyte of data in total. Keys must use simple characters. A ConfigMap can also hold binary data, and a Secret always stores values in an encoded form. These limits are the same whether the object is mutable or not.
Environment variables are instructions whispered to a cook at the start of the shift; they are never updated until the next shift. A mounted folder is a notice board in the kitchen that a runner refreshes now and then. A sub-path mount is a sticky note glued in a corner that the runner forgets to replace.
4Core Concepts and Vocabulary
The words you will meet again and again in this topic.
Immutable
An object whose data cannot be changed after creation. In Kubernetes, you seal a ConfigMap or Secret with the immutable field.
Mutable
The default: the object can be edited at any time, and mounted copies eventually reflect the edit.
Versioned name
Naming each new configuration differently, for example with a version number or a short hash, such as “app-config-v7”. Each version is its own object.
Content hash
A short fingerprint computed from the data. If any character changes, the fingerprint changes. Used to build names automatically.
Rolling update
Replacing Pods a few at a time with new ones, so the application stays available while it changes version.
Watch
A long-lived connection where a client asks the API server “tell me whenever this object changes”. Watches keep clients up to date but cost the server memory and effort.
Hot reload
An application re-reading its configuration while running, without restarting. Depends on mutable, mounted configuration.
Kustomize
A built-in Kubernetes configuration tool that can generate ConfigMaps and Secrets with automatic hash suffixes and update every reference to them.
GitOps
Managing cluster state from files in a Git repository, with a tool that keeps the cluster in sync. Every change is a reviewed commit.
Garbage collection
Removing objects that are no longer used. With versioned names, old ConfigMaps pile up unless something cleans them.
5Architecture and Components
Which parts of Kubernetes touch a ConfigMap or Secret, and what does sealing change for each?
flowchart TB
YOU["You or your pipeline"] --> API["API server: validates and stores objects"]
API --> ETCD[("etcd: cluster database")]
API --> ADM["Validation: rejects edits to immutable data"]
subgraph NODE["Each machine"]
KL["Kubelet: fetches config for its Pods"]
VOL["Files or variables inside containers"]
POD["Pods"]
KL --> VOL
VOL --> POD
end
API -->|watch or fetch| KL
The parts
API server
Receives your request to create, update or delete the object. For a sealed object, it rejects any attempt to change the data and replies that the field is immutable.
etcd
Stores the object. Every version you create takes space here, which is why old versions should eventually be cleaned up.
Kubelet
For each Pod on its machine, fetches the ConfigMaps and Secrets the Pod refers to. For mutable objects it keeps checking for changes. For sealed objects it can stop checking, since nothing can change.
Deployment controller
Creates Pods from a template. When the template refers to a new ConfigMap name, the template changes, so a rolling update starts naturally.
Your delivery tooling
Kustomize, Helm or a GitOps tool that generates the new object and rewrites the references. This is where immutability becomes convenient or painful.
The check lives in the API server, so it applies no matter which tool you use to send the request, whether a command line, a dashboard or a pipeline. Nothing on the client side can bypass it.
6How Mutable Configuration Behaves
Understanding the default behavior shows you exactly what immutability fixes.
sequenceDiagram
participant U as You
participant A as API server
participant K as Kubelet
participant C as Container
U->>A: Edit mutable ConfigMap
A-->>K: Watch event: object changed
Note over K: Waits for its next sync and cache refresh
K->>C: Updates mounted files after a delay
Note over C: Environment variables stay old
Note over C: App must notice new files itself
The three surprises
- It is not instant. The kubelet refreshes mounted files periodically and uses a cache, so the change appears on different machines at different times, often within a minute or two but not guaranteed.
- Environment variables never update. A Pod that read its settings through variables keeps the old values until the container restarts. Half of your fleet may be on new settings while the other half holds the old ones.
- The app must cooperate. Even when the file changes, the program must re-read it. Many programs read configuration only once at startup and never look again.
Mixed fleets
Because of these three surprises, a single edit can leave your Pods in different states: some with new files and old memory, some with old files, some restarted by chance and fully new. Debugging “why does this Pod behave differently?” then becomes very hard, since the object looks correct now but was different an hour ago.
Editing a mutable ConfigMap is like a manager changing a rule on the office intranet. Some employees refresh the page immediately, some see the change hours later, some never open the page again, and those who printed the old rule keep following it. Everyone thinks they are following the current policy.
Outages caused by editing shared configuration in place are a well-known category in operations. The edit is small, the reach is huge, and the first sign of trouble is many Pods misbehaving at once. This is the pattern immutability is designed to prevent.
7What Immutability Actually Changes
The exact rules, so there are no surprises.
| Question | Answer |
|---|---|
| How do I make an object immutable? | Set the immutable field to true when creating it, or edit an existing mutable object once to set it to true. |
| Can I edit its data afterwards? | No. Attempts to change the data are rejected by the API server. |
| Can I turn it back to mutable? | No. Sealing is one-way. To get a mutable version, create a new object. |
| How do I change the values? | Delete the object and create a new one, usually with a new name. |
| Does it affect other fields like labels? | The protection is on the data fields and the immutable flag itself. Descriptive fields such as labels are generally still editable, but check your version’s behavior before relying on it. |
| What happens to running Pods if I delete it? | Running containers keep the copy they already have. New Pods that need it cannot start until an object with that name exists again. |
| Which versions support it? | The feature became generally available in Kubernetes 1.21, so it is present in essentially all supported clusters today. |
| Does it encrypt anything? | No. It controls changes, not visibility. Encryption and access control are separate topics. |
The name reuse trap
If you delete a sealed ConfigMap and immediately create a new one with the same name but different data, Pods that start later see the new data, while Pods that started earlier still hold the old one. You have recreated the mixed fleet problem by another route. This is why immutable objects should get new names for new content rather than reusing the old name.
A laminated recipe card cannot be corrected with a pen. You can throw it away and print another one, but if you give the new card the same number as the old, cooks who kept the old card will never know it changed. A new number for a new recipe avoids that confusion.
8Internal Working: Why the Cluster Gets Lighter
Immutability is not only about safety. It also removes work from the control plane.
flowchart LR
subgraph MU["Mutable objects"]
M1["Kubelet on every machine"] -->|keeps a watch open per object| M2["API server keeps many watches"]
M2 --> M3["Higher memory and CPU on API server and etcd"]
end
subgraph IM["Immutable objects"]
I1["Kubelet fetches once"] --> I2["No watch needed"]
I2 --> I3["Lower load on API server"]
end
How the kubelet reads objects
By default, the kubelet uses a watch on each ConfigMap and Secret that its Pods use. That is efficient for one object, but multiplied across thousands of machines and many objects per machine, the API server ends up maintaining a very large number of open watches, each of which consumes memory and processing time.
When the kubelet sees an object marked immutable, it knows the data can never change, so it fetches it once and stops watching. The saving grows with cluster size. Kubernetes’ own scaling work highlighted this as a real bottleneck in very large clusters, which is one of the reasons the feature was introduced.
What you gain
- Fewer open watches, so lower API server memory and processing load.
- Less traffic between machines and the control plane.
- More predictable behavior during control-plane restarts, when all clients reconnect at once.
- Headroom for other, more important objects to be watched.
How much does it matter?
For a small cluster with a handful of ConfigMaps, the load saving is negligible. The safety benefits are what count. For clusters with thousands of machines and many thousands of mounted ConfigMaps and Secrets, the load saving becomes significant and is a proper scalability tool.
Production example: a large fleet
A company running several thousand nodes mounts a shared certificate bundle and a shared settings object into most Pods. Marking both as immutable removed a large share of the watches that the API server had to maintain, and reduced the load spike when control-plane components restarted. The team now treats immutability as the default for widely shared configuration.
9The Versioned-Name Workflow
If you cannot edit an object, how do you change your settings? By creating a new version and pointing to it.
sequenceDiagram
participant D as Developer or pipeline
participant A as API server
participant DP as Deployment controller
participant P as Pods
D->>A: Create new ConfigMap named app-config-v8, sealed
D->>A: Update Deployment to refer to app-config-v8
A-->>DP: Pod template changed
DP->>P: Start new Pods with v8, one batch at a time
P-->>DP: New Pods ready
DP->>P: Stop old Pods that used v7
Note over A: app-config-v7 kept for rollback
Create the new version
Make a new ConfigMap with a new name that changes when the content changes, for example with a version number or hash. Seal it with the immutable field.
Point the workload at it
Change the Deployment’s Pod template to refer to the new name. Because the template changed, Kubernetes treats this as a new version of the workload.
Roll out gradually
The rolling update replaces Pods a few at a time. New Pods use the new settings, and old Pods keep the old settings until they are replaced. Health checks guard every step.
Watch and decide
If metrics look good, the rollout finishes. If not, roll back, and the old Pods return with the old settings that they still refer to.
Clean up later
After the change has been stable for a while, delete configuration versions that no workload refers to any more.
Why this is powerful
The configuration change now travels through the same safe machinery as a code change: gradual, observable and reversible. There is no longer a special, dangerous path for “just changing a setting”. Both code and configuration are versioned things that move together.
Handing out version 8 of the recipe card one kitchen station at a time, watching how the dishes taste, and keeping version 7 in the drawer is very different from erasing a whiteboard in front of the whole kitchen at once.
A worked walkthrough
Let us follow a small story. A team runs a Deployment named “storefront” with six Pods. The Pods mount a sealed ConfigMap named “storefront-config-v7”, which contains a setting for how long to wait for the payment service before giving up: five seconds. Customers complain that payments sometimes time out, so the team decides to raise the wait to eight seconds.
A developer edits the settings file in Git
The file now says eight seconds. A reviewer approves the change, and the merge starts the pipeline.
The pipeline generates a new object
The generator calculates a fingerprint of the new content and creates “storefront-config-v8”, sealed. Version 7 is untouched and still in the cluster.
The Deployment template changes
The pipeline rewrites the reference from v7 to v8. Kubernetes sees a new Pod template and starts a rolling update, creating a second ReplicaSet.
Pods change a few at a time
Two new Pods start with v8. Their readiness checks pass. Two old Pods are stopped. The team watches the payment error rate and sees it drop on the new Pods.
The rollout finishes
All six Pods now use v8. The old ReplicaSet remains with zero Pods, still pointing at v7, as a bookmark for rollback.
Weeks later, cleanup
After several more releases, the old ReplicaSet that pointed to v7 is pruned by Kubernetes, and the cleanup job then deletes “storefront-config-v7” because nothing refers to it any more.
Notice what never happened: nobody edited a live object, no Pod saw a setting change under its feet, and at every moment the team knew exactly which Pods had which timeout. If the eight-second timeout had caused a new problem, such as slow checkouts piling up, one rollback command would have brought back both the old ReplicaSet and, with it, the five-second setting.
10Rollouts and Rollbacks Done Right
The best argument for immutability is what happens on a bad day, when you need to go back.
flowchart TD
D["Deployment"] --> RS1["Old ReplicaSet: template refers to app-config-v7"]
D --> RS2["New ReplicaSet: template refers to app-config-v8"]
RS1 --> P1["Pods with settings v7"]
RS2 --> P2["Pods with settings v8"]
CM7[("ConfigMap v7: sealed")] -.-> RS1
CM8[("ConfigMap v8: sealed")] -.-> RS2
Rollbacks with mutable objects
A Deployment remembers its old Pod templates as old ReplicaSets, and a rollback simply reactivates the old one. But if the ConfigMap has been edited in place, the old template points at the same object, which now holds the new data. Rolling back the workload does not roll back the settings. The old code starts up with the new configuration, often with confusing results.
Rollbacks with immutable, versioned objects
Each ReplicaSet points to a specific versioned ConfigMap, and that version is frozen. Rolling back reactivates the old ReplicaSet, whose Pods mount the old, unchanged settings. Code and configuration return together, exactly as they were.
| Situation | Mutable, edited in place | Immutable, versioned |
|---|---|---|
| Rollout of a bad setting | Affects Pods as the edit propagates; no gradual control | Affects only Pods in the current batch; halts on failed checks |
| Rollback | Workload returns to old code but keeps new settings | Workload and settings both return to the old version |
| Which settings does Pod X have? | Depends on timing and how it consumes them | The name in its template tells you exactly |
| Audit | Overwritten values leave no copy in the cluster | Every version exists as its own object until cleaned |
Rollback only works while the old ConfigMap still exists. Deployments remember a limited number of old ReplicaSets, by default ten. Do not delete a configuration version while an old ReplicaSet still refers to it, or the rollback will fail with a missing-object error.
11Tooling: Making Versioned Names Automatic
Nobody wants to invent version numbers and edit references by hand. Common tools do it for you.
Kustomize generators
Kustomize can generate a ConfigMap or Secret from files or literal values and append a hash of the content to the name. It then rewrites every reference in your workloads to the new name. Change one character of data and the name, the references and therefore the rollout all follow automatically. Generators can also mark the result immutable through their options.
Helm
Helm templates can compute a hash of the configuration and put it in the ConfigMap name, or in a Pod annotation that changes when the config changes. Either forces a rollout. Chart authors decide whether to mark objects immutable.
GitOps controllers
Tools such as Argo CD and Flux apply the files in Git. They must be told to prune, meaning delete objects no longer present in Git, or old versions will accumulate. Some tools have special handling for immutable field errors.
Custom pipelines
A simple script can calculate a hash, create the object, patch the Deployment and later delete old versions. Good enough for small teams.
Operators
Some operators generate configuration objects for the software they manage and version them themselves, sparing you the work.
Policy engines
Admission policies can require the immutable field on certain objects, so no team forgets it. We cover this in the governance chapter.
The hash suffix idea
Hashing is the heart of automation. The tool reads the content, computes a short fingerprint, and adds it to the name, giving names like “app-config-9tgm2k”. Same content always gives the same name, so re-applying an unchanged file changes nothing. Any change to the content gives a different name, so the reference changes, and a rollout starts. Humans never need to think about version numbers.
Whichever tool you use, make sure your process regularly deletes unused versions. Automated creation without automated cleanup slowly fills the cluster with hundreds of stale objects.
12Immutable Secrets and Credential Rotation
Secrets follow the same rules as ConfigMaps, with a few extra things to think about.
An immutable Secret cannot have its values changed. That fits secrets well, because a changed password or key is a significant event that deserves a controlled rollout.
Rotating a credential safely
Create the new credential at the source
For example, create a second database user or a new API key. Keep the old one working for now.
Create a new sealed Secret
Give it a new name that reflects the new version.
Roll out the reference
Update the workload to use the new Secret. A rolling update replaces Pods gradually.
Confirm nothing uses the old credential
Check logs and metrics on the service that owns it, to be sure no client still uses the old one.
Retire the old credential and Secret
Disable the old credential at the source, then delete the old Secret object.
Special considerations
- Immutability is not encryption. A sealed Secret is still just encoded, not encrypted, unless you enable encryption at rest in the cluster and control who can read it.
- External secret systems. Tools that sync secrets from a vault into Kubernetes often want to update values in place. Check whether your tool supports creating new versioned objects, or whether sealing will conflict with its update behavior.
- Frequent rotation. If a secret changes every few hours, immutable objects would create many versions. In that case, mounting from an external secrets provider directly into the Pod may be a better fit.
- Token-like secrets. Short-lived tokens are better handled through projected service account tokens or an identity system than by any stored object.
Rotating a building’s door lock safely means cutting new keys, handing them to everyone, checking that nobody still uses the old ones, and only then changing the lock. You would never change the lock while half the staff are still outside holding old keys.
13Choosing: Mutable or Immutable?
Immutability is a strong default for production, but it is not right for everything. Here is how to decide.
flowchart TD
Q1{"Does the app re-read its config while running and you rely on live edits?"}
Q1 -->|Yes| M1["Mutable mounted volume, with care and a reload mechanism"]
Q1 -->|No| Q2{"Is the object shared by many Pods or read on many nodes?"}
Q2 -->|Yes| I1["Immutable with versioned names"]
Q2 -->|No| Q3{"Is it production configuration where a bad edit is costly?"}
Q3 -->|Yes| I2["Immutable with versioned names"]
Q3 -->|No| M2["Mutable is fine for tests and experiments"]
| Situation | Better choice | Why |
|---|---|---|
| Production application settings loaded at startup | Immutable | Changes should come through rollouts |
| Widely shared certificates or bundles | Immutable | Big scalability benefit and safe rotation |
| Feature flags flipped many times a day, read live | Mutable or an external flag service | Versioning would create endless objects |
| Config for a tool that watches the file and reloads | Mutable, if you accept the trade | Live editing is the point |
| Development and test environments | Mutable | Fast iteration matters more |
| Database passwords | Immutable, with a rotation process | Rotation deserves a controlled rollout |
| Configuration that changes with every release anyway | Immutable | It already travels with the Deployment |
A shop keeps its printed price list laminated, updated once a season, and its daily specials on a chalkboard by the door. Both have a place. The mistake is writing the price list in chalk, or laminating the daily specials.
14Performance and Scalability
Where immutability helps the cluster, and where the new habits can add cost.
Fewer watches
Each sealed object removes a per-node watch from the API server. In very large clusters, this measurably reduces memory and processing load on the control plane.
Calmer restarts
When API servers restart, thousands of kubelets reconnect and re-establish their watches at once. Fewer watches make that storm smaller.
More objects in etcd
Every version you create is a separate object in the database. Hundreds of generations of large ConfigMaps add up. Regular cleanup keeps etcd small and fast.
More rollouts
Config changes now cause Pod replacements. Frequent configuration changes across many workloads mean many rolling updates, which cost scheduling effort and image pulls.
Startup time
New Pods must fetch the new object. Large objects, close to the size limit, slow Pod startup, so keep them lean and split big files across objects when sensible.
Keeping objects small
Kubernetes limits each object to about one mebibyte. Very large objects also use more etcd space and more network on every fetch. Store big files, such as large data sets or models, in dedicated storage rather than in a ConfigMap, and keep only the settings that truly belong there.
Group settings that change together into one object, and settings that change at different rhythms into different objects. That way a small change does not force a rollout of unrelated configuration, and a rarely changing shared object stays stable and cheap.
15Advantages, Disadvantages and Trade-offs
A clear look at what you gain and what you give up.
Advantages
- Protects from accidental or careless in-place edits.
- Configuration changes go through the same gradual rollout as code.
- Rollbacks restore both code and settings.
- Clear answer to “which settings does this Pod have?”.
- Lower load on the API server and etcd watches.
- Better audit trail: each version is its own object.
- Fits GitOps, where the Git history is the record of truth.
Disadvantages
- Cannot edit; every change means create, update reference, and clean up.
- Needs tooling or discipline to generate names and update references.
- Old versions accumulate without cleanup.
- Every change restarts Pods, even tiny ones.
- Sealing is one-way, so a mistaken seal means a new object.
- Not suited to live-reload workflows or frequently changing flags.
- Some tools that update objects in place need adjusting.
The design trade-offs
Safety versus agility. A mutable object gives immediate change, at the risk of immediate mistakes. A sealed object adds a step that forces review and a rollout. Choose safety for production and agility for experiments.
Restarts versus live reload. Immutability trades in-place updates for Pod restarts. If restarts are cheap and rollouts fast, this is a bargain. If your application takes ten minutes to warm up its caches, frequent configuration rollouts have a real cost.
Object count versus clarity. More objects mean more to store and clean, but each is a clear, frozen record. Automation and retention rules keep the count in check.
Cluster load versus operational effort. The scalability gain is largest exactly where clusters are big, and teams there can afford the tooling. Small clusters gain mostly safety, and should weigh the extra process against the benefit.
16Security
Immutability helps security in some ways and does nothing in others. Be clear about which is which.
Tamper resistance
An attacker or a careless insider with edit rights cannot quietly change a sealed object’s data. To alter it, they must delete and recreate it, which is more visible in audit logs and breaks references.
Change control
Every configuration change becomes a new object and a rollout, which pipelines can review, approve and record.
Delete still works
Anyone allowed to delete and create objects can still replace a sealed object. Immutability does not replace access control.
Not encryption
Secrets are stored only encoded by default. Enable encryption at rest for the cluster database, and consider an external key management service.
Role-based access
Give people the narrowest rights: reading Secrets is a powerful right and should be rare. Separate the right to create objects from the right to update Deployments.
Limit who can read
Anyone who can create a Pod in a namespace can usually mount any Secret in that namespace. Keep sensitive Secrets in tightly controlled namespaces.
Policy enforcement
Use admission policies to require the immutable field on production objects, and forbid ConfigMaps from holding credentials.
Audit logging
Turn on API audit logs for ConfigMap and Secret creation, deletion and reads, so replacements and unusual reads leave a record.
Never commit raw secrets
Do not keep plain secret values in Git. Use sealed or encrypted secret formats, or external secret managers, so the repository is safe to share.
Delete old secrets promptly
Old versions keep old credentials alive in the cluster. After rotation, remove them, and disable the old credential at its source.
ConfigMaps are meant for non-sensitive settings and are often less protected. Sealing a ConfigMap does not make it a safe place for passwords.
A tamper-evident seal on a box tells you if someone opened it, but it does not stop a determined thief from taking the whole box. Locks, guards and cameras still matter. Immutability is the seal, not the lock.
17Monitoring, Logging and Metrics
How to see whether the pattern is working, and catch trouble early.
Rollout status
Watch the progress of each rollout after a config change: how many new Pods are ready, how many are failing. A config change is now a deployment, and should be monitored as one.
Pod startup errors
A Pod that refers to a missing ConfigMap or Secret fails to start, with a container-configuration error or a failed-mount event. Alert on these.
API server rejections
Attempts to change sealed objects are rejected with a message that the field is immutable. A spike shows a tool or a person trying to edit in place.
Object counts and size
Track the number of ConfigMaps and Secrets per namespace and their total size, plus etcd database size. Growth without matching deployments means stale versions are piling up.
Control plane load
Watch the number of open watches and API server memory. Compare before and after sealing shared objects, to prove the benefit.
Application behavior after change
Record which configuration version each Pod runs, as a label or annotation, and chart error rates and latency by version. Regressions then point straight to the config that caused them.
Audit events
Log deletions and creations of configuration objects, who made them, and from which pipeline.
Alarms worth setting
- Pods failing to start because a referenced ConfigMap or Secret is missing.
- A rollout stuck for longer than expected after a config change.
- Configuration objects without the immutable flag in production namespaces.
- Configuration versions older than a retention limit with no workload using them.
- Sudden rise in the count of ConfigMaps or in etcd size.
18Lifecycle, Cleanup and Governance
Creating versions is easy. Keeping them tidy takes a plan.
Retention window
Keep at least as many old configuration versions as your Deployments keep old ReplicaSets, so any rollback that Kubernetes still offers will find its settings.
Prune from Git
Configure your GitOps tool to delete objects that are no longer in the repository, but also to respect a delay so rollbacks stay possible.
Ownership links
Where tools allow it, set an owner relationship so a version is removed automatically when its owner is deleted. Be careful that ownership does not delete a version that an older ReplicaSet still needs.
Labels
Label every generated object with the application, the version and the creation source, so cleanup jobs can find stale ones safely.
Cleanup job
Run a scheduled job that lists configuration objects, checks whether any Pod template refers to each, and removes the unused ones older than the retention window.
Admission policy
Require the immutable field in production namespaces, and require the version label, so nothing enters without a way to be tracked.
Production example: a tidy cleanup policy
A platform team keeps the last ten versions of every generated configuration object, matching the ten old ReplicaSets that each Deployment remembers. A weekly job removes anything older that no Pod template refers to. In two years the cluster never hit an object-count problem, and every rollback that was needed found its configuration waiting.
19Design Patterns and Anti-patterns
Reliable recipes and the habits to avoid.
Patterns that work
Pattern: Hash-named, sealed configuration
Generate names from content, seal every object, and let the tooling rewrite references. Change flows only through rollouts.
Pattern: Config and code released together
Keep the application’s configuration in the same repository and pipeline as its deployment files, so a single commit describes exactly what runs.
Pattern: Shared base plus per-environment overrides
A base configuration is shared, and environments add overrides through generators. Each environment gets its own sealed, hash-named object.
Pattern: Separate fast-changing from slow-changing settings
Stable settings in sealed objects; rapidly changing flags in a dedicated flag service or a small mutable object with a reload sidecar.
Pattern: Canary a configuration change
Roll a new configuration to a small share of Pods or one region first, compare error rates, then continue. Only possible when versions coexist, which is what sealed, versioned objects provide.
Traps to avoid
What it looks like
Deleting and recreating a sealed object under the same name.
Why it hurts
New Pods get the new data while existing Pods keep the old, and the Deployment sees no change, so no rollout occurs. You have a mixed fleet again.
Better approach
Give each new content its own name, so the Pod template changes and a rollout follows.
What it looks like
Sealing an object that a tool must update in place, such as one managed by a secret-syncing controller.
Why it hurts
The controller fails on every update with an immutable-field error, and the values silently go stale.
Better approach
Use a mechanism that supports versioned objects, or leave that specific object mutable and protect it another way.
What it looks like
Generating versions with no cleanup.
Why it hurts
Thousands of stale objects slow listings, bloat etcd, and make it hard to see what is in use.
Better approach
Add retention rules and a cleanup job from day one.
What it looks like
Deleting old versions immediately after a rollout.
Why it hurts
A rollback then fails, because the old ReplicaSet points to a ConfigMap that no longer exists.
Better approach
Keep versions for at least as long as old ReplicaSets are retained.
What it looks like
Sealing objects that hold values changing many times a day.
Why it hurts
Endless new objects and constant rollouts overwhelm the cluster and the team.
Better approach
Move fast-changing values to a feature flag service or a purpose-built dynamic configuration system.
20Best Practices and Common Mistakes
A short list to keep beside you when you adopt the pattern.
Habits that pay off
- Seal production configuration by default and make mutability the documented exception.
- Let tools generate names from content hashes, and let them rewrite references.
- Create the new object before updating the reference so Pods never point to something that does not exist yet.
- Keep old versions until the rollback window passes, then remove them automatically.
- Mount whole folders, not sub-paths, if you ever need live updates on mutable objects.
- Prefer files to environment variables for anything you might want to rotate, and always plan for restart.
- Test your rollback in a staging cluster, including the configuration side.
- Record the config version on each Pod, so incidents point at the change.
- Use admission policies to require the flag in the namespaces that matter.
Mistakes beginners make
Trying to edit a sealed object
The API server refuses. The fix is a new object with a new name and an updated reference.
Forgetting to update the reference
A new ConfigMap exists, but the Deployment still points at the old one, so nothing changes.
Updating the reference before creating the object
New Pods cannot start because the object is missing, and the rollout stalls.
Expecting instant change
Assuming a mutable edit takes effect at once, when files update slowly and environment variables never update.
Sub-path surprises
Mounting a single key with a sub-path and wondering why edits never arrive.
Treating sealed as secure
Assuming a sealed Secret is encrypted or hidden. It is neither, unless the cluster is configured that way.
Sealing too early
Making an object immutable in the middle of experiments, then having to recreate it repeatedly.
Oversized objects
Stuffing large files into one ConfigMap until it hits the size limit or slows Pod starts.
21Real-World and Industry Examples
The same idea in many settings.
The Kubernetes scaling story
The feature was introduced largely because of scale. Large clusters were spending significant control-plane effort on watches for objects that almost never changed. Making immutability an option lets operators trade the ability to edit for lower load, and Kubernetes itself recommends it for exactly that reason.
Kustomize hash suffixes
Many teams using Kustomize get this pattern for free. Their generated ConfigMaps carry a hash in the name, and every workload reference is rewritten, so a change to a settings file triggers a rolling update without any extra scripting. Teams that add the immutable option gain the scalability benefits as well.
Helm checksum annotations
Chart authors often place a checksum of the configuration in a Pod annotation. When the settings change, the annotation changes, so Pods are replaced and pick up the new values. It reaches the same goal by a different route, even when the object itself is mutable.
GitOps at an enterprise
A large enterprise stores all configuration in Git. Each merge produces new, hash-named, sealed objects, and the GitOps controller applies and prunes them. Auditors can trace any running Pod to a commit, a reviewer and a time.
Certificate bundle distribution
A company distributes its trusted certificate authorities to thousands of Pods through one shared sealed ConfigMap. When a certificate must be added, they create a new version and roll it out cluster by cluster, watching for connection failures at each step.
A typical online shop
The shop’s checkout service reads its settings from a sealed, hash-named ConfigMap and its payment credentials from a sealed Secret. On Friday, a developer raises the timeout for the payment service. The pipeline creates a new ConfigMap version, the checkout Deployment rolls out a few Pods at a time, and error rates stay flat. The following Monday, the team discovers a rare issue and rolls back with one command. The old Pods return with the old timeout, exactly as before.
22Troubleshooting
Most problems come from a handful of causes. Follow the path of the reference.
flowchart TD
T1["Problem after a config change"] --> T2{"Did the update command fail with an immutable field message?"}
T2 -->|Yes| F1["Create a new object with a new name and update the reference"]
T2 -->|No| T3{"Are new Pods failing to start?"}
T3 -->|Yes| F2["Check the referenced object exists in the same namespace and the name is spelled correctly"]
T3 -->|No| T4{"Did any Pods restart at all?"}
T4 -->|No| F3["Deployment template unchanged: reference or name not updated"]
T4 -->|Yes| T5{"Behavior still old?"}
T5 -->|Yes| F4["Check which object each Pod mounts, and whether the app re-reads its config"]
T5 -->|No| F5["Compare error rates by config version and roll back if needed"]
| Symptom | Likely cause | What to do |
|---|---|---|
| Update rejected: field is immutable | Tried to edit a sealed object | Create a new object with a new name; update references |
| New Pods stuck with a configuration error | Referenced object missing or in another namespace | Create it, or fix the name and namespace |
| Failed mount event on a Pod | Volume source object does not exist | Check the name; create the object before the Deployment |
| Rollback brings back the old code but new settings | Mutable object was edited in place | Adopt versioned, sealed objects going forward |
| Rollback fails: object not found | Old version was cleaned up too early | Recreate it from Git; lengthen the retention window |
| Changed the ConfigMap but nothing happened | Deployment still refers to the old name, or app never re-reads files | Update the reference; restart the app if needed |
| Some Pods old, some new, same name | Object deleted and recreated with the same name | Roll all Pods, and use unique names in future |
| GitOps tool reports permanent out-of-sync error | Tool tries to patch a sealed object in place | Use hash-named generation, or set the tool to replace on this kind of change |
| Cluster slow with thousands of ConfigMaps | No cleanup of old versions | Run cleanup; add retention automation |
When in doubt, ask two questions. Which ConfigMap name does the Pod’s template actually refer to? And does that object have the content I think it has? Most mysteries end at one of those two answers.
23Frequently Asked Questions
Quick answers to common questions.
24Summary and Key Takeaways
You started with a whiteboard in a busy kitchen. Now you know how to swap it for numbered, laminated recipe cards.
Key Takeaways
- The switch: Setting immutable to true on a ConfigMap or Secret seals its data. It cannot be edited, and the sealing cannot be undone. Change means a new object.
- Why it helps safety: No accidental in-place edits spreading across every Pod, no half-updated fleets, and rollbacks that restore both code and settings.
- Why it helps scale: The kubelet stops watching sealed objects, reducing load on the API server, which becomes important in very large clusters.
- The workflow: Create a new versioned, ideally hash-named, sealed object; point the Deployment at it; let a rolling update replace Pods; keep old versions for rollback and clean them up later.
- Tooling makes it painless: Kustomize generators, Helm checksums and GitOps pruning automate naming, references and cleanup.
- Know the limits: Immutability is not encryption, does not stop someone who can delete and recreate, and does not suit fast-changing live settings.
- Govern and observe: Enforce the flag with policy in production, monitor rollouts and missing-object errors, record the config version on each Pod, and rehearse rollbacks before you need them.