Raft Consensus in Docker Swarm: How Managers Agree on the Truth
When several manager nodes are all responsible for the same cluster, someone has to decide what's actually true if they disagree — Raft is the algorithm Docker Swarm relies on to make that decision safely, even when nodes crash or the network misbehaves.
Imagine a company with five regional offices, all of which must agree on the exact same company policy at all times, with zero tolerance for two offices operating on conflicting versions of the rulebook. Phone lines between offices sometimes drop. Sometimes an office’s power goes out entirely. Yet somehow, whenever a policy change is approved, every office that’s currently reachable ends up with the exact same updated rulebook, in the exact same order, and if the head office suddenly goes silent, the remaining offices are able to elect a new head office and carry on without ever adopting two different rulebooks at once. This is precisely the problem Raft solves for Docker Swarm’s manager nodes — keeping a cluster’s entire state consistent and agreed-upon, even through crashes, restarts, and unreliable networks.
1What Is Raft Consensus?
Before looking at how Docker Swarm uses it, let’s build a clear picture of what a consensus algorithm actually does.
The Core Problem: Agreement Under Failure
Whenever more than one machine is responsible for the same critical data — in Swarm’s case, the entire desired state of the cluster: which services should exist, how many replicas each should have, what configuration they use — those machines need a reliable way to agree on that data, even when some of them crash, restart, or temporarily lose network connectivity. A consensus algorithm is a formally proven method for a group of machines to agree on a single, consistent sequence of decisions, guaranteeing that even in the presence of failures, they never end up disagreeing about what actually happened.
Raft: Consensus Designed to Be Understandable
Raft is a consensus algorithm specifically designed, from the ground up, to be easier to understand and implement correctly than older alternatives like Paxos, without sacrificing the same rigorous safety guarantees. Docker Swarm adopted Raft as the algorithm behind its manager nodes precisely because of this balance — strong, provably correct consistency guarantees, implemented in a way that’s tractable for real production software to get right.
Think of a jury of several people who must reach a unanimous verdict, but where jurors occasionally step out of the room and might miss part of the deliberation. Raft is like a strict set of jury-room rules: exactly one juror is designated the foreperson at any time, only the foreperson can propose a new verdict item, every juror present must acknowledge receiving each item before it’s considered official, and if the foreperson ever goes silent for too long, the remaining jurors hold a new election to pick a replacement foreperson before deliberation continues. These rules guarantee the group never ends up with two different “official” verdicts floating around at once.
Consensus
The process by which a group of machines agree on a single, consistent version of shared data, despite failures.
Manager Node
A Swarm node responsible for maintaining the cluster’s overall state and making scheduling decisions, as opposed to a worker node, which only runs containers.
Quorum
The minimum number of manager nodes that must be reachable and in agreement for the cluster to safely make new decisions.
Replicated Log
The ordered sequence of state-changing operations that every manager node keeps an identical copy of, forming the basis for how Raft keeps everyone in sync.
2Architecture: Manager Nodes and Their Roles
Raft assigns every manager node one of three roles at any given moment, and understanding these roles is central to understanding the whole system.
graph TD
A[Leader Manager Node] -->|Replicates log entries| B[Follower Manager Node 1]
A -->|Replicates log entries| C[Follower Manager Node 2]
A -->|Replicates log entries| D[Follower Manager Node 3]
A -->|Replicates log entries| E[Follower Manager Node 4]
style A fill:#fef2f2,stroke:#dc2626
style B fill:#fafafa,stroke:#171717
style C fill:#fafafa,stroke:#171717
style D fill:#fafafa,stroke:#171717
style E fill:#fafafa,stroke:#171717
Leader
The single manager node, at any given time, responsible for accepting all new state changes and replicating them out to the other manager nodes.
Follower
A manager node that passively receives and acknowledges log entries from the current Leader, and stands ready to become a Candidate if the Leader disappears.
Candidate
A temporary role a Follower takes on when it believes the Leader has failed, during which it requests votes from other manager nodes to become the new Leader.
Term
A logical, ever-increasing counter that identifies a particular period of leadership, used by Raft to detect and resolve conflicting or outdated information.
Why Manager Count Should Be Odd
Raft’s quorum requirement is based on a strict majority of manager nodes — for example, in a five-manager cluster, at least three managers must be reachable and agree for the cluster to keep making decisions. This is precisely why Swarm deployments are conventionally sized with an odd number of managers: an odd count maximizes fault tolerance for a given number of nodes, since adding an even-numbered manager (going from three to four, for instance) doesn’t actually improve how many failures the cluster can tolerate, while still adding operational overhead.
3Internal Working: Leader Election
One of Raft’s most important mechanics is how a new Leader gets elected when the current one becomes unreachable.
Detecting a Missing Leader
Every Follower expects to periodically hear from the current Leader through regular heartbeat messages. If a Follower goes too long without hearing from the Leader — a randomized timeout designed specifically to reduce the chance of multiple nodes triggering an election at the exact same moment — it assumes the Leader has failed and transitions itself into the Candidate role to start a new election.
sequenceDiagram
participant L as Old Leader
participant F1 as Follower 1
participant F2 as Follower 2
participant F3 as Follower 3
Note over L: Leader crashes / becomes unreachable
F1->>F1: Election timeout expires
F1->>F1: Becomes Candidate, increments Term
F1->>F2: RequestVote (new Term)
F1->>F3: RequestVote (new Term)
F2-->>F1: Vote granted
F3-->>F1: Vote granted
Note over F1: F1 receives majority of votes
F1->>F1: Becomes new Leader
F1->>F2: Heartbeat (asserts leadership)
F1->>F3: Heartbeat (asserts leadership)
Voting Rules That Guarantee Safety
Each manager node votes for at most one Candidate per term, and Raft’s voting rules ensure a Candidate can only win an election if its log is at least as up-to-date as a majority of the cluster’s nodes. This prevents a node that missed recent updates from ever becoming Leader and accidentally overwriting or losing already-agreed-upon state — a critical safety guarantee for a system managing something as important as a production cluster’s configuration.
If every Follower waited exactly the same amount of time before starting an election, multiple nodes could become Candidates simultaneously, splitting the vote and requiring repeated election rounds. By randomizing each node’s timeout slightly, Raft makes it statistically likely that one node notices the missing Leader first and wins the election cleanly, before others even start their own timeout.
4Data Flow: How a State Change Gets Committed
Beyond leader election, Raft’s other central job is safely replicating every state change — like deploying a new service — across the whole manager cluster.
A client request arrives at the Leader
For example, a `docker service create` command is routed to whichever manager currently holds the Leader role — even if the command was issued against a different manager, Swarm forwards it to the Leader.
The Leader appends the change to its own log
The new entry — representing the desired new state — is added to the Leader’s local replicated log, but is not yet considered official.
The Leader replicates the entry to all Followers
The Leader sends this new log entry out to every Follower manager node in the cluster, asking each to append it to their own local log.
Followers acknowledge receipt
Each Follower that successfully appends the entry sends an acknowledgment back to the Leader.
Once a majority acknowledges, the entry is committed
As soon as the Leader has heard back from enough Followers to form a majority of the whole cluster, the entry is considered permanently committed — safe, official, and durable.
The Leader applies the change and confirms to the client
The Leader updates its own understanding of cluster state to reflect the change and reports success back to whoever issued the original command.
Raft only requires acknowledgment from a majority of nodes, not every single one, specifically so the cluster can continue making progress even if a minority of manager nodes are temporarily unreachable — waiting for unanimous agreement would make the entire cluster grind to a halt the moment even one manager went offline.
5Advantages and Trade-offs
Raft gives Swarm strong guarantees, but those guarantees come with real, deliberate costs.
Advantages
- Guarantees the cluster never disagrees about its own state, even through node crashes and network partitions
- Automatically elects a new Leader without manual intervention if the current one fails
- Provides strong durability — a committed change survives even if the node that originally accepted it later fails
- Designed to be more understandable and implementable correctly than older consensus algorithms
- Tolerates the loss of a minority of manager nodes while continuing to operate normally
Disadvantages / Trade-offs
- Every state change must wait for majority acknowledgment, adding latency compared to a single-node system
- Losing a majority of manager nodes at once halts all cluster state changes until quorum is restored
- More manager nodes generally means more replication overhead, not necessarily more performance
- Election churn — repeated, unnecessary leader elections — can occur under poor network conditions, temporarily disrupting cluster responsiveness
- Manager node placement (spreading across failure domains) requires careful planning to actually deliver the fault tolerance Raft promises
6Security Implications
Because manager nodes hold and agree on the cluster’s entire desired state, the Raft layer itself is a sensitive piece of Swarm’s overall security posture.
Communication between Swarm manager nodes for Raft log replication is encrypted by default, protecting the cluster’s sensitive configuration and secrets data as it’s replicated between managers over the network.
Because any manager node participates fully in the Raft cluster and holds a complete replica of the cluster’s state, compromising even a single manager node’s underlying host is a significantly more serious event than compromising a worker node, and manager nodes should be secured and access-restricted accordingly.
An attacker (or an unplanned outage) that manages to take down a majority of manager nodes simultaneously doesn’t just cause a performance issue — it halts the cluster’s ability to make any new decisions at all, making manager node availability itself a meaningful part of the cluster’s overall security and resilience posture.
7Manager Count and Fault Tolerance
A concrete look at how many manager node failures a Swarm cluster can tolerate, depending on how many managers it has.
| Total Managers | Quorum Required | Failures Tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
8Design Patterns and Anti-patterns
Operational experience running Swarm clusters has produced some clear, well-established dos and don’ts around manager topology.
Problem
Running only a single manager node in a production cluster.
Why It’s Harmful
A single manager provides zero fault tolerance under Raft’s quorum model — if that one node fails, the entire cluster loses its ability to make any state changes, and recovery may require manual intervention.
Correct Approach
Run at least three manager nodes in any production Swarm cluster, so the loss of a single manager doesn’t halt the cluster’s ability to reach quorum.
Problem
Placing all manager nodes in the same physical rack, availability zone, or power domain.
Why It’s Harmful
Raft’s fault tolerance only helps against failures it can actually survive — if every manager shares the same underlying infrastructure, a single infrastructure-level failure can take down a majority of managers all at once, despite the cluster technically having “enough” managers on paper.
Correct Approach
Spread manager nodes across genuinely independent failure domains, so no single infrastructure event can take down a majority of them simultaneously.
Problem
A cluster needs strong fault tolerance without excessive replication overhead as it scales.
Why It Works
Keeping the manager count deliberately small — typically three or five — while scaling worker node count independently gives the cluster strong Raft fault tolerance without paying the replication cost of having every added node participate in consensus.
Correct Approach
Add capacity by adding worker nodes, not manager nodes, and only adjust the manager count when fault-tolerance requirements genuinely change.
9Monitoring and Key Metrics
Because Raft’s health is foundational to the entire cluster’s ability to function, it deserves dedicated, ongoing observation.
Current Leader stability
Frequent, unexpected leader elections are a strong signal of network instability or resource pressure among manager nodes, even when the cluster otherwise appears to be functioning.
Manager node reachability
Continuously tracking how many manager nodes are currently reachable helps catch a cluster drifting dangerously close to losing quorum before it actually happens.
Log replication lag
Measuring how far behind a Follower’s log is from the Leader’s helps identify a struggling manager node before it becomes a bottleneck for committing new changes.
State-change commit latency
Tracking how long it takes for new state changes to be committed provides an early warning sign of replication or network issues across the manager cluster.
10Real-World and Industry Examples
Raft’s role in Swarm reflects a much broader pattern across distributed infrastructure software.
Docker Swarm Production Clusters
Organizations running Swarm in production almost universally follow the three- or five-manager convention specifically to align with Raft’s quorum-based fault tolerance model, rather than choosing manager counts arbitrarily.
Other Infrastructure Tools Built on Raft
Raft’s design has proven influential well beyond Docker — several other widely used distributed systems tools rely on the same consensus algorithm to keep their own cluster state consistently agreed upon across multiple nodes, a testament to how well the algorithm balances correctness with real-world implementability.
Multi-Region Deployments
Teams operating Swarm clusters that span multiple data centers pay particularly close attention to manager placement and network latency between regions, since Raft’s commit latency is directly tied to how quickly a majority of managers can acknowledge new log entries.
11Frequently Asked Questions
A single manager would be a single point of failure — if it went down, the entire cluster would lose its ability to make decisions. Raft lets multiple manager nodes safely share that responsibility, so the cluster keeps functioning even if some managers fail.
The cluster can no longer commit new state changes — existing running services generally keep running, but you can’t deploy new ones, scale existing ones, or make other cluster-state changes until enough manager nodes become reachable again to restore a majority.
Because quorum is based on a strict majority, an odd manager count maximizes fault tolerance for a given number of nodes — adding an even-numbered manager doesn’t improve how many failures the cluster can tolerate, while still adding replication overhead.
No. Only manager nodes participate in the Raft consensus process. Worker nodes simply execute the containers they’re assigned and don’t hold a replica of the cluster’s Raft-managed state.
A Follower that stops receiving heartbeats from the Leader within a randomized timeout becomes a Candidate and requests votes from the other managers; if it receives votes from a majority, it becomes the new Leader.
Yes, by default, communication between Swarm manager nodes for Raft log replication is encrypted, helping protect the cluster’s sensitive state as it’s replicated across the network.
12Summary and Key Takeaways
Raft is the quiet foundation underneath every Swarm cluster’s sense of a single, shared reality. Every service you deploy, every scaling decision, every configuration change has to pass through Raft’s leader election and log replication machinery before it becomes official — and it’s precisely this discipline that lets a Swarm cluster survive manager node crashes, network hiccups, and restarts without ever splitting into two conflicting versions of the truth. Understanding Raft isn’t just academic: it directly explains why manager counts matter, why quorum loss is dangerous, and why manager node placement deserves the same careful thought as any other critical piece of infrastructure.
Key Takeaways
- Raft solves distributed agreement — it guarantees Swarm manager nodes never disagree about the cluster’s state, even through failures.
- Three roles, one Leader at a time — Leader, Follower, and Candidate roles coordinate to keep exactly one node in charge of accepting new changes.
- Majority, not unanimity — changes are committed once a majority of managers acknowledge them, letting the cluster tolerate some node failures.
- Odd manager counts maximize fault tolerance — three or five managers is the common production convention, balancing resilience and overhead.
- Quorum loss halts state changes — losing a majority of managers doesn’t crash running services, but it stops the cluster from making new decisions.
- Manager placement matters as much as manager count — spreading managers across real failure domains is essential to actually realize Raft’s fault tolerance.
- Manager security is high-stakes — since every manager holds a full replica of cluster state, compromising one is far more serious than compromising a worker node.