Before comparing tools and techniques, it helps to pin down exactly what makes a piece of data a “secret” in the first place, and why ordinary configuration handling is not good enough for it.

A secret is any piece of information that would cause real harm if it were read by someone who should not have it — a database password, an API key that grants access to a paid cloud service, a private cryptographic key used to sign or decrypt data, a token that authenticates one internal service to another, or a certificate’s private half. This stands in direct contrast to ordinary configuration: values like which port an application should listen on, which environment it is running in, or which logging level to use. Configuration is meant to be visible, inspectable, and often even checked into version control alongside the application’s own source code. A secret is the opposite on every one of those points: it must be invisible by default, inspectable only by the specific people and systems authorized to use it, and it must never sit in version control at all.

Why does this distinction matter so specifically inside Docker? Because, as established when discussing how images are built, a Docker image is a stack of frozen, immutable layers, and anything written into one of those layers during the build process becomes a permanent, extractable part of that image’s history — readable by anyone who can pull the image, regardless of whether a later instruction appears to delete it. A configuration value baked into an image is a minor inconvenience if it ever needs to change. A secret baked into an image is a security incident waiting to be discovered, because every copy of that image, everywhere it has ever been pulled, cached, or archived, now permanently carries a credential that may need to be treated as compromised the moment anyone realizes it is there.

It is worth being precise about one more distinction beginners often blur: a secret’s value versus its reference. The name of a secret, or a pointer telling an application where to go fetch it from, is not itself sensitive and can be treated like ordinary configuration — it is perfectly fine for a service definition to openly state “this application needs a secret called database-password.” What must never be treated like ordinary configuration is the actual plaintext value that secret resolves to once fetched. This distinction is what makes it possible to keep infrastructure definitions themselves readable, version-controlled, and reviewable, while still keeping the sensitive values those definitions point to completely separate and protected.

Several distinct mechanisms exist for getting a value into a running container, and only some of them are actually appropriate for secrets. Knowing the full map makes the right choice obvious.

Docker’s own native answer to this problem, available specifically in Swarm mode (Docker’s built-in orchestration system for running containers across a cluster of machines), is called simply Docker secrets. A secret created this way is encrypted, stored centrally by the Swarm manager nodes, and only ever transmitted to a worker node over an encrypted connection, at the exact moment a container that has been explicitly granted access to that secret is scheduled to run there. Crucially, the secret’s value is never written to that worker node’s regular disk at all — it is mounted directly into the container’s filesystem as a file, backed by memory rather than persistent storage, at a fixed, predictable location: a directory named /run/secrets/, followed by the secret’s own name.

flowchart TB
    subgraph Manager["Swarm Manager Nodes"]
        Store["Encrypted Secret Store (Raft log)"]
    end
    subgraph Worker["Worker Node"]
        Mem["In-Memory tmpfs Mount"]
        Container["Running Container"]
    end
    Store -->|"Encrypted transfer, only to authorized services"| Mem
    Mem -->|"Mounted at /run/secrets/<name>"| Container
        

Beyond Docker’s own built-in mechanism, a large and mature ecosystem of dedicated external secret managers exists specifically to solve this problem at a broader scale — systems designed to store, version, rotate, and audit access to secrets across an entire organization’s infrastructure, not just a single Swarm cluster. These systems typically expose secrets to a container through one of two patterns: an initialization step that fetches secrets before the main application starts, or a lightweight companion process running alongside the main application that keeps a secret’s value continuously refreshed on disk or in memory as needed.

A fourth architectural component, easy to overlook, sits underneath every one of the mechanisms above: the encryption and authentication layer that protects a secret both while it sits in storage and while it travels across a network to reach the container that needs it. A secret store that only encrypts data at rest but transmits it in plain text over the network has simply moved the weak point rather than closing it; a properly designed system encrypts a secret at every stage of its journey, and requires the receiving side to prove its identity before any value is handed over at all, so that even an attacker positioned somewhere on the network between the store and the container gains nothing usable.

Understanding the mechanics of delivery explains exactly why some methods are safe and others quietly are not.

When a secret is delivered through Docker’s native Swarm secrets feature, the flow works roughly as follows. First, an operator creates the secret once, submitting its value to the Swarm manager, which encrypts it and stores it as part of the cluster’s internal, replicated configuration state. Second, a service definition — the specification for a particular containerized application running across the cluster — explicitly declares which named secrets it is authorized to use. Third, whenever the Swarm scheduler places a new instance of that service onto a worker node, the manager transmits the secret’s decrypted value to that specific node over an already-encrypted management connection, and only for the duration that a container needing it is actually running there. Fourth, on the worker node, the container engine mounts that secret as a single file inside a special in-memory filesystem, made visible only inside the container’s own filesystem namespace at the fixed path under /run/secrets/, readable by the application process the same way it would read any ordinary file.

A related, newer mechanism worth knowing about operates purely at build time. Modern build engines support mounting a secret into a single build step for the duration of that step only, without that secret’s value ever being written into the resulting committed layer at all — even though the build step itself, running inside its own temporary container, can read and use it normally, for example to authenticate to a private package repository while installing dependencies. This solves a real, common problem: some builds genuinely need a credential during the build process itself, not just at runtime, and this mechanism provides a safe way to supply one without repeating the classic mistake of a secret leaking into a frozen layer.

The mechanics of why this build-time mount is safe are worth spelling out, because it looks, on the surface, similar to a build argument, yet behaves completely differently underneath. A build argument’s value becomes part of the instruction that used it, and if that instruction’s effect gets committed as a layer, the value can end up captured inside that layer’s metadata or contents. A build-time secret mount, by contrast, is deliberately excluded from the layer-commit process entirely: the build engine makes the secret available inside the temporary build container’s filesystem for that one step, then ensures nothing from that mount point is ever included when the resulting layer is captured and hashed. The credential is present exactly where and when it is needed, and structurally absent from everything that gets kept.

A secret’s full life has more stages than most people assume, and skipping any one of them is where real incidents come from.

The lifecycle begins with creation, where a secret’s value first comes into existence — generated by a system, issued by a third-party service, or set by a human operator — and is immediately handed to a secret store rather than being typed into a chat message, a ticket, or a plaintext file. Next comes storage, where the secret sits encrypted at rest, accessible only through an authenticated, audited interface, never as a plain file sitting on a shared drive. Then comes distribution, the moment covered in the previous chapter, where the secret’s value is delivered only to the specific, authorized workload that needs it, only for as long as that workload is running. Then comes consumption, where the application itself reads the secret’s value from wherever it was delivered — a mounted file, an environment variable, or a direct call to a secret manager’s own retrieval interface — and uses it to authenticate or decrypt whatever it needs to.

sequenceDiagram
    participant Op as Operator or System
    participant Store as Secret Store
    participant Node as Worker Node
    participant App as Application Process
    Op->>Store: Create secret (encrypted at rest)
    Store->>Node: Deliver only to authorized container, on demand
    Node->>App: Mount as in-memory file or inject at startup
    App->>App: Consume secret to authenticate or decrypt
    Store-->>Store: Rotation replaces old value on schedule
    Store-->>Node: Revocation removes access immediately if needed
        

Two final stages are the ones most often neglected, and they are exactly the ones that turn a good secrets-management setup into a genuinely resilient one. Rotation is the deliberate, scheduled process of replacing a secret’s value with a new one before the old one has any chance to become stale or over-exposed, ideally without requiring any downtime for the applications consuming it. Revocation is the ability to immediately invalidate a secret the instant it is suspected of being compromised, cutting off whatever access it granted before any further damage can occur, independent of whatever the normal rotation schedule would have otherwise been.

Rotation itself comes in two distinct flavors worth telling apart. Scheduled rotation happens on a predictable, routine cadence purely as good hygiene, whether or not anything is suspected to be wrong — the same instinct behind routinely changing a physical lock even without evidence of a break-in. Emergency rotation happens immediately, outside any schedule, the moment a secret is suspected of being exposed, and its entire value depends on how quickly it can actually be carried out. A team that has only ever practiced scheduled rotation, on a calm, planned timeline, may discover during a real incident that rotating a credential without breaking every service depending on it is a much harder operational problem under pressure than it looked on a routine Tuesday. This is precisely why mature teams rehearse emergency rotation deliberately, before they ever need it for real.

Every method of getting a secret into a container involves a real trade-off. Knowing the specific weakness of each one is what lets a team choose deliberately instead of by habit.

It is worth noting that this trade-off table is not an argument for perfectionism paralysis. A small internal tool with a low-sensitivity credential and a team just getting started with containers may reasonably choose runtime environment variables today, deliberately, as a pragmatic first step, while planning a path toward file-based or native secret delivery as the system matures. What matters is that the choice is made deliberately, with the specific weaknesses in the table above understood and consciously accepted for now, rather than adopted by default simply because it was the first method anyone tried and nobody ever revisited the decision once the stakes quietly grew.

Almost every real-world Docker secrets incident traces back to one of a small number of well-understood leak paths. Knowing them by name is most of the defense.

Closing these vectors follows directly from understanding them. Baked-in layer extraction is closed by never writing a secret into any Dockerfile instruction, and by using build-time secret mounts for anything a build genuinely needs. Public registry exposure is closed by strict, actively audited registry access controls and by treating every image as if it might one day be exposed, which naturally reinforces never baking secrets in regardless of how private a registry currently appears to be. Build log capture is closed by disciplined logging practices and by secret managers that automatically mask known secret values wherever they might appear in output. Process list inspection is closed by preferring file-based or environment-based delivery over command-line arguments for anything sensitive. Version control commits are closed by pre-commit scanning tools that detect likely secret patterns before they are ever pushed, combined with treating any historically committed secret as permanently compromised and rotating it, rather than assuming a later deletion made it safe. Over-broad access grants are closed by the principle covered next in Chapter 8: granting each workload access only to the specific secrets it actually needs, and nothing more.

One further, more subtle vector deserves its own mention: secrets sprawl across shared infrastructure. A secret placed in a general-purpose configuration system alongside ordinary, non-sensitive settings — rather than in a system purpose-built for secrets — tends to inherit that general system’s much wider circle of administrative access. A configuration management tool that dozens of engineers can reasonably read for entirely legitimate, everyday reasons is not, by that same fact, a safe home for a production database password, even though nothing about storing it there looks obviously wrong at first glance. This is why the centralized-store pattern from Chapter 8 insists specifically on a system built for secrets, with access governed independently from an organization’s general configuration and documentation tooling, rather than simply “somewhere the team already keeps shared settings.”

Secrets management is one of the rare areas where the most important “metric” is not a number trending up or down, but a clear, continuously answered yes-or-no question: who has access to what, and when did they last use it.

The first practical signal worth tracking is secret age — how long it has been since each individual secret’s value was last rotated, compared against a defined maximum acceptable lifetime for that class of secret. A dashboard that surfaces secrets approaching or past their rotation deadline turns rotation from an easily-forgotten manual chore into a visible, trackable operational item, the same way a team already tracks certificate expiry dates.

The second is access auditing — a durable record of exactly which service, and ideally which specific container instance, actually retrieved a given secret’s value, and when. This is different from, and more valuable than, simply knowing which services are theoretically authorized to access it. An access log lets a team answer the single most urgent question during a suspected incident: if this secret may be compromised, exactly which systems have actually been using it, and therefore exactly what blast radius does a rotation or revocation need to cover.

The third is automated secret-pattern scanning, run continuously against source repositories, build logs, and even built container images themselves, looking for text patterns that resemble common credential formats. Treating a scanner hit as a real, urgent finding — triggering immediate rotation of whatever credential was found, rather than just deleting the exposed text and moving on — is what actually closes the exposure, since, as covered earlier, a value that ever appeared in a version-controlled or logged location must be treated as compromised regardless of whether it was later removed from view.

A fourth, quieter signal worth tracking is unused standing access — secrets a service was granted permission to read at some point in the past but has not actually fetched in a long time. This pattern often shows up naturally as systems evolve: a service that once depended on a particular external integration is refactored to no longer need it, but the original access grant is simply never revoked, because removing access nobody is actively complaining about rarely feels urgent. Periodically reviewing and revoking exactly this kind of stale, unused access is a low-effort, high-value habit, since it shrinks the total blast radius of the system over time without requiring any new tooling beyond the access logs a team should already be collecting.

These are the proven, repeatable patterns experienced platform and security teams reach for, roughly in order of impact.

Together, these four patterns describe a coherent operational posture: grant only what is needed, deliver secrets through a channel designed for them, rotate on a schedule rather than reactively, and route everything through one accountable system. None of them requires exotic tooling to begin adopting — they are disciplines a team can start applying immediately, even before a full external secret manager is in place.

It is worth noting that these four patterns reinforce one another rather than operating independently. Least-privilege access is far easier to enforce consistently once every secret already flows through one centralized store, because a single system can apply one consistent policy engine to every grant, rather than each team implementing its own access checks separately. Automated rotation, in turn, is far easier to roll out safely once secrets are delivered as files or through a native mechanism rather than environment variables, because a file-based mount can often be refreshed in place without requiring the consuming application to be restarted, while many environment-variable-based setups require a full container restart to pick up a new value. Adopting one pattern well tends to make the next one meaningfully easier, which is part of why teams that start with centralization first, even in a modest form, tend to have an easier time layering the remaining three patterns on top of it later.

Every one of these anti-patterns is the exact mirror image of a pattern from Chapter 8, which is why recognizing the pattern makes the anti-pattern obvious.

A condensed, practical checklist — what to consistently do, and where teams most often slip.

Secrets management failures and fixes are among the most publicly documented lessons in the entire container ecosystem — because the failures tend to become very visible, very quickly.

They are not automatically catastrophic, and they are extremely widely used in practice, but they carry real, specific risks compared to file-based or native secret delivery: visibility in configuration inspection, accidental capture by crash reporters, and inheritance by child processes. Where an application can be configured to read a mounted file instead, that is the safer default; where environment variables are unavoidable, minimizing what logs and tooling can see them becomes especially important.

Yes. Because image layers are immutable and append-only, the earlier layer that wrote the secret still contains it in full, regardless of what a later instruction appears to do. The only safe approaches are to never write the secret into any layer at all, using build-time secret mounts for anything a build step genuinely needs, or to keep secrets entirely outside the image and inject them only at runtime.

Docker’s native secrets feature, available in Swarm mode, is a genuinely solid, encrypted, memory-backed mechanism and is enough for many teams running entirely on Swarm. An external secret manager becomes valuable once an organization needs centralized policy and auditing across multiple platforms and environments, advanced rotation for many different credential types, or integration with a broader identity and access-management system beyond what a single orchestrator’s built-in feature covers.

There is no single universal number, and the right interval depends on the sensitivity of what the secret protects and how automatable rotation is for that specific credential type. What matters more than any specific interval is that a defined interval exists at all, is actually enforced by tooling rather than left to memory, and that rotation can happen quickly and without downtime whenever it is triggered outside the normal schedule by a suspected compromise.

The exposed credential should be treated as compromised immediately and rotated as the very first action, not as a follow-up step after removing the exposed copy. Removing the visible copy without rotating the underlying credential leaves the actual access that credential grants completely unaffected, since anyone who already saw or copied it retains working access until the value itself is changed.

It is safer than hardcoding, but not automatically safe on its own. A build argument’s value can still end up captured into a committed layer if a build step writes it to a file that gets copied into the image, or prints it to output that a build log then stores. The genuinely safe pattern for a credential a build step needs is a dedicated build-time secret mount, designed specifically so the value is available only during that one step and is never written into any committed layer at all.