Before rotation makes sense, it’s worth understanding exactly what a Swarm secret is and why it behaves so differently from an ordinary environment variable.

What a Docker Swarm secret actually is

A Swarm secret is a blob of sensitive data — a password, an API key, a TLS certificate, an SSH key — that is registered once with the Swarm cluster’s manager nodes, stored encrypted, and then made available only to the specific services that have been explicitly granted access to it. Unlike an environment variable, which is often visible through simple process inspection or accidentally logged, a Swarm secret is delivered to a container as a file mounted inside an in-memory filesystem, never written to the container’s actual disk layer, and never passed as a plaintext environment variable at all.

Why secrets are immutable by design

A detail that surprises many engineers the first time they encounter it: an existing Swarm secret’s content cannot be edited in place once created. This is a deliberate design choice, not a missing feature. If a secret’s underlying value could be silently changed while services were already referencing it by name, there would be no clear, auditable moment at which the change took effect, and no way to safely stage a transition where old and new values briefly coexist. Instead, rotation in Swarm works by creating an entirely new secret object with a new name, and then updating the service definition to reference that new secret instead of the old one — immutability at the secret level is precisely what makes the rotation process at the service level safe and predictable.

Scope: Swarm mode only

It is worth being precise about scope: this entire secrets and rotation system is a feature of Docker Swarm mode specifically, requiring a cluster of manager and worker nodes, and is not available to standalone containers run outside of Swarm. Organizations running plain Docker Compose without Swarm mode enabled, or a different orchestrator entirely, rely on different secret-delivery mechanisms, even though the underlying goals — encryption at rest, in-memory-only delivery, controlled rotation — are broadly similar across the container ecosystem.

Several distinct components across the Swarm cluster cooperate to store, distribute, and rotate secrets securely.

flowchart TB
    subgraph MGR["Swarm Manager Nodes"]
        RAFT["Encrypted Raft Log
(secret storage)"]
    end
    subgraph WRK["Worker Node"]
        AGENT["Swarm Node Agent"]
        subgraph TASK["Service Task Container"]
            TMPFS["/run/secrets/ (tmpfs, in-memory)"]
            APP["Application Process"]
        end
    end
    RAFT -->|encrypted delivery over TLS| AGENT
    AGENT -->|mounts at container start| TMPFS
    APP -->|reads file| TMPFS
    

How secrets differ architecturally from Swarm configs

Swarm also offers a related but distinct object type called a config, intended for non-sensitive configuration data rather than credentials. The two share much of the same underlying delivery architecture — replicated storage on manager nodes, targeted delivery to specific services — but configs are not specially protected against being read by anyone with appropriate cluster access in the way secrets are, and configs are typically mounted onto the container’s normal filesystem rather than exclusively an in-memory one. Understanding this distinction matters because using a config where a secret was appropriate is a common and meaningful security mistake.

A closer look at what actually happens, step by step, when a secret is created, delivered, and later rotated.

From creation to delivery

When a new secret is created, its content is sent to a Swarm manager node, encrypted, and committed to the cluster’s replicated Raft log, which ensures the secret’s data survives the loss of any single manager node without ever existing in plaintext on manager disk. When a service that references that secret is scheduled onto a worker node, the manager delivers the secret’s decrypted content to that specific worker over the cluster’s mutually authenticated TLS connections, and the worker’s own node agent mounts it into the target container as a file under an in-memory tmpfs filesystem, immediately before that container’s process starts, so the application can read it as an ordinary file from the very first moment it runs.

What actually happens during a rotation

Rotation begins with creating a brand-new secret object containing the new credential value, entirely independent of the old one. The service definition is then updated to reference this new secret, typically in place of, or alongside, the old one. Swarm’s rolling update engine then begins replacing the service’s running tasks one at a time, or in whatever batch size the service’s update configuration specifies: for each task being replaced, a new task is started with the updated secret already mounted, health-checked to confirm it started successfully, and only once healthy is the corresponding old task — still running with the old secret — stopped and removed. This continues until every task in the service has been transitioned, with the service as a whole remaining available to traffic throughout the entire process.

sequenceDiagram
    participant Op as Operator
    participant Mgr as Swarm Manager
    participant Old as Old Task (secret v1)
    participant New as New Task (secret v2)

    Op->>Mgr: create new secret (v2)
    Op->>Mgr: update service to reference secret v2
    Mgr->>New: schedule replacement task with secret v2
    New-->>Mgr: health check passes
    Mgr->>Old: stop and remove old task
    Note over Old,New: service remains available
to traffic throughout
    Mgr->>Mgr: repeat for each remaining task
    

Why the old secret must survive until rotation completes

A crucial internal-working detail: the old secret object must remain available in the cluster for as long as any task still referencing it is running, which is why deleting the old secret immediately upon creating the new one is a common and serious mistake. Because rotation proceeds gradually across a service’s tasks — not instantaneously across all of them at once — there is necessarily a window during which some tasks are still using the old secret while others have already moved to the new one, and removing the old secret prematurely would cause the not-yet-updated tasks to fail outright the next time they needed to be rescheduled.

Tracing a secret’s complete journey from creation through active use, rotation, and eventual safe removal.

Coordinating rotation with the credential’s source of truth

In practice, a full rotation lifecycle usually needs to be coordinated with whatever external system actually issues or validates the credential — a database’s own user-password store, a certificate authority, an external API provider’s key management console. The safest sequencing generally introduces the new credential at the source first (so both old and new values are simultaneously valid there), then rotates the Swarm secret and lets tasks transition, and only invalidates the old credential at the source once the rotation across the entire Swarm service has fully completed — mirroring, at the external system, the same overlap window Swarm itself creates internally during the rolling update.

Swarm’s approach to secrets rotation trades some operational complexity for strong safety guarantees — a trade-off worth examining explicitly.

The overlap window as the central trade-off

The single most important trade-off to internalize is this: safe, zero-downtime rotation absolutely requires a period during which both the old and new credential values are simultaneously valid, both within Swarm’s own rolling update and at whatever external system issued the credential in the first place. Organizations sometimes resist this on the assumption that a credential should be invalidated the instant a replacement exists, but insisting on zero overlap is fundamentally incompatible with zero downtime — one of those two goals must yield, and in the overwhelming majority of production systems, a brief, carefully bounded overlap window is the far more acceptable trade.

Secrets rotation exists specifically as a security practice, and understanding its guarantees — and its limits — matters as much as understanding the mechanics.

Why rotation matters even without a known compromise

Regularly rotating credentials, even in the complete absence of any known or suspected compromise, is a defense-in-depth practice that limits the value of a credential to an attacker who may have obtained it through a channel that was never detected — a leaked log line from years ago, a misconfigured backup, a departed employee’s still-valid local copy. A credential that is rotated on a predictable schedule has a naturally bounded useful lifetime for anyone who acquired it improperly, regardless of whether the legitimate operators ever learn that acquisition happened at all.

What rotation does not protect against

It is worth being precise about the limits of this mechanism. Secrets rotation protects the credential itself from having unbounded useful life, but it does nothing to protect against an attacker who has already compromised a running container and can simply read whatever secret is currently mounted at that exact moment — rotation changes which secret is valid over time, not whether a currently-compromised process can read the currently-valid one. Similarly, rotation does not substitute for restricting which services are granted access to a given secret in the first place; a secret rotated regularly but overexposed to services that never actually need it is still a meaningfully larger attack surface than necessary.

Confirming a rotation genuinely completed, and catching a failed or stalled one, requires watching the right signals throughout the process.

What to watch during an active rotation

The most important signal during an active rotation is straightforward: the count of a service’s tasks still referencing the old secret versus the new one, which should monotonically move toward zero old-secret tasks as the rolling update proceeds. A rotation that stalls partway — some tasks successfully transitioned, others stuck — typically indicates a task repeatedly failing its health check after being started with the new secret, which is a strong signal that the new credential itself may be invalid or that the application cannot yet authenticate with it, rather than a Swarm-level problem.

Confirming completion before cleanup

Before deleting the old secret object, the safest practice is explicitly confirming, not assuming, that zero running tasks anywhere in the cluster still reference it — a check that should be automated as part of any rotation pipeline rather than performed on faith after a fixed waiting period. Because Swarm distributes tasks across potentially many worker nodes, a manual, ad-hoc check on a single node is not sufficient evidence that rotation has genuinely completed cluster-wide.

Recurring approaches teams use to make secrets rotation a routine, low-risk operation, and the recurring mistakes that turn it into an incident instead.

Concrete, practical guidance for rotating Swarm secrets confidently and repeatedly, without turning routine maintenance into a production incident.

Secrets rotation discipline shows up wherever organizations run stateful services that must never fully halt for routine credential maintenance.

Swarm secrets rotation turns credential replacement from a risky, disruptive event into a routine, verifiable, zero-downtime operation.