Lock and Milestone Steps in Jenkins
A complete, plain-English guide to the two steps that keep concurrent pipeline runs from trampling each other: how to make builds take turns with locks, how to cancel outdated builds with milestones, and how to combine them for safe deployments.
Picture a small restaurant with one oven. Three cooks arrive at the same moment, each with a tray of bread. If all three shove their trays in at once, the trays collide, the temperature is wrong for everyone, and the bread is ruined. The sensible fix is a simple rule: only one cook uses the oven at a time, and the others wait their turn. That rule is a lock. Now imagine that while the second cook waits, the head chef changes the recipe and the first cook’s tray is already out of date. It would be silly to bake the old recipe after the new one. So the head chef adds a second rule: when a newer order passes the oven door, older orders still waiting are thrown out. That rule is a milestone.
Jenkins pipelines run into exactly these two problems every day. Many builds can be running at the same time, and they often need to touch the same shared thing: a test server, a database, a deployment target. This guide teaches, from zero, how the lock step and the milestone step solve those problems, how they work inside, and how to use them together so that only the latest change deploys, and only one deployment happens at a time.
1What Are the Lock and Milestone Steps?
We begin with the simplest picture: what each step does, and how they differ.
Jenkins is a tool that automates building, testing, and delivering software. Each time code changes, Jenkins starts a new build of a pipeline. If developers push changes quickly, several builds of the same pipeline can be running at once. Most of the time this is fine, because each build works in its own folder. Trouble starts when builds reach out to something they must share.
The lock step gives a build exclusive use of a named resource for the duration of a block of work. If another build already holds the lock, the newcomer waits in a line until it is released. The lock step comes from a plugin usually called Lockable Resources.
The milestone step marks a point in a pipeline that builds must pass in order, oldest first. When a newer build passes a milestone, older builds that have not yet passed it are cancelled, because they are outdated. The milestone step comes from a plugin usually called Pipeline Milestone Step.
A single-lane bridge has a traffic light. Only one car crosses at a time, and the others wait. That light is the lock. Now imagine a rule at the front of the queue: “If a newer delivery van arrives with updated goods, the older van carrying outdated goods is sent home.” That rule is the milestone.
The two steps at a glance
Take turns
Only one build at a time may run a protected block. Others wait, and nothing is cancelled.
Newest wins
Builds pass in order. When a newer build passes, older builds that are behind are cancelled.
Protects shared things
Servers, databases, devices, licences, or any resource that cannot be used by two builds at once.
Protects freshness
Stops an old build from finishing after a newer one and overwriting its results.
Both steps belong to plugins that may need to be installed on your Jenkins. Their exact options and wording can differ between plugin versions, so check the Pipeline Syntax help page on your own server before relying on any option named in this guide.
2Why These Steps Exist
To see why these steps matter, imagine a busy afternoon without them.
Problem one: two builds, one shared thing
A team has one test server. Build 41 starts deploying its version to the server. A minute later, Build 42 starts and deploys its version to the same server. Now the server holds a mixture of two versions, tests fail in strange ways, and nobody knows which build caused the failure. The builds were each correct. The collision made them wrong.
flowchart TD
A["Build 41 starts and deploys"] --> C["Both write to the same test server"]
B["Build 42 starts and deploys"] --> C
C --> D["Files are mixed and test results are confused"]
Problem two: the old build that finishes last
Build 41 is slow because it runs a long test suite. Build 42, started later, finishes faster because it has fewer tests to run. Build 42 deploys. Then, a few minutes later, Build 41 finishes its tests and deploys too. The server now runs the older code, and the newer fix is gone. This is called an out-of-order deployment, and it is one of the most confusing bugs in delivery pipelines, because everything looks green.
Problem three: wasted work on outdated builds
A developer pushes five small commits in ten minutes. Five builds queue up. Only the last one matters, since it contains all the others. Running the first four to completion wastes machine time and delays the one that counts.
A newspaper printing room has one press. If two editions were printed on it at once, the pages would mix. If an old edition finished printing after the new one, readers would receive yesterday’s news. The printing room needs one rule for turns and another for freshness. Those are the lock and the milestone.
3Concepts You Need First
A few ideas from computing come up in every lock and milestone discussion. Learn them once here.
Concurrency
Several things happening during the same period of time. In Jenkins, several builds, or several branches of one build, running together.
Race condition
A bug that appears only when two things happen at nearly the same moment, so the result depends on who is faster.
Mutual exclusion
A rule that only one party may use something at a time. A lock is the usual way to enforce it.
Critical section
The part of the work that must not overlap with anyone else’s. The lock wraps exactly this part.
Deadlock
A stand-off where two parties each hold something the other needs, and both wait forever.
Ordering
A rule about which work is considered earlier or later. For builds, the build number gives a natural order.
A few Jenkins words too
The controller is the main Jenkins server that plans the work and keeps track of locks. An agent is a worker machine that runs the commands. An executor is a slot on an agent that runs one task at a time. A pipeline is the ordered list of stages in a Jenkinsfile. A build is one run of a pipeline, with its own number. Newer builds have higher numbers, which is how Jenkins knows which is “older”.
A fitting room in a clothes shop is a critical section. Only one person may be inside at a time, so there is a lock on the door and a queue outside. A deadlock would be two people each holding a door key that the other needs to leave. The shop avoids it with simple rules about who takes what first.
Before adding a lock, name the exact thing that cannot be shared. If you cannot name it, you probably do not need the lock, and adding one will only make builds wait for no reason.
4The Lock Step in Detail
Now we look at the shape of the lock step, and the options that control it.
The basic shape
The lock step wraps a block of work. You give it the name of a resource. When the pipeline reaches the step, Jenkins tries to give the build that resource. If it is free, the build takes it and runs the block. When the block ends, whether it finished well, failed, or was aborted, the lock is released automatically. If the resource is already held, the build waits in a queue until its turn.
| Option | What it does | When to use it |
|---|---|---|
resource | The name of the single resource to lock | A specific server, database, or device |
label | Lock any resource that carries this label | A pool of similar resources, where any one will do |
quantity | How many resources with the label to take | When a build needs several from the pool at once |
variable | Names an environment variable that will hold the name of the locked resource | When the block must know which pool member it received |
inversePrecedence | Serve the newest waiting build first instead of the oldest | Deployments, where only the latest matters |
skipIfLocked | Skip the block if the resource is busy instead of waiting | Optional work that is not worth waiting for |
extra | A list of additional resources to acquire together with the first | When a block needs several specific resources at once |
priority | A number that lets more important builds jump the queue, where supported | Urgent fixes that must not wait behind routine builds |
A car-sharing service has a pool of cars. You can ask for one specific car by its number plate (a resource), or for any car of a certain type (a label). You can ask for two cars at once (a quantity). If none is free, you wait in line, or, if you say “only if one is free now”, you go away and try later (skip if locked).
Resources are named things
A resource in the lock step is only a name. Jenkins does not know or care what the name stands for. It could be a test server, a database schema, a physical phone, or a licence seat. The lock works only because every pipeline that uses the thing agrees to use the same name, and to always ask for the lock first. A lock is a polite agreement between pipelines, not a wall around the real resource.
Where resources come from
- Defined by an administrator on the Lockable Resources page, with a name, a description, and optional labels.
- Created on the fly the first time a pipeline asks for a name that does not exist. Such automatic resources are temporary, and are usually removed when no longer in use. Depending on your version and setup, this behaviour may be limited.
A typo in a resource name can quietly create a new, separate resource. Two pipelines using “staging-server” and “staging-sever” will not lock each other out at all. Agree on resource names centrally, and prefer resources defined by an administrator.
5Using Lock in Declarative and Scripted Pipelines
Where you place the lock decides how many executors are wasted while builds wait.
Two places to put it
In a Scripted Pipeline, or inside a script section, you wrap the protected steps in a lock block. In Declarative Pipeline, you can also use the lock as an option of a stage. The stage-level option is usually better, and the reason is executors.
| Placement | Order of events | Executor held while waiting? |
|---|---|---|
| Lock as a stage option, with an agent in the stage | Acquire the lock first, then allocate the agent, then run | No, the build waits for the lock without holding an executor |
| Lock inside a block that already has an agent | Agent allocated first, then wait for the lock inside it | Yes, the executor sits idle while waiting |
| Lock around an agent block in Scripted style | Acquire the lock, then allocate the agent inside it | No, if written in this order |
You can queue at the restaurant door and be led to a table only when the kitchen is ready. Or you can sit at a table first and then wait for the kitchen, taking up a seat that someone else could use. The first way keeps tables free. The second way fills the restaurant with people who are waiting.
Why executor use matters
Build agents have a limited number of executors. If ten builds each grab an executor and then wait for the same lock, ten executors are wasted while one build does the real work. Other unrelated builds may then be unable to start. Getting the order right, lock first and agent second, keeps executors free for useful work.
In Declarative Pipelines, put the lock in the stage’s options, and set the pipeline’s top-level agent to none, with agents named inside the stages. The pipeline then waits for the lock on the lightweight controller side, and takes an executor only when the lock is granted.
Keep the locked block small
The lock should wrap only the part that truly needs exclusive use. If a stage compiles code, runs unit tests, and then deploys to a shared server, only the deploy step needs the lock. Locking the whole stage makes every build wait for compilation and tests that could have run in parallel with others.
The exact syntax and behaviour of stage-level lock options depend on your Lockable Resources plugin version. Check the Pipeline Syntax help on your server, and test with two builds started together to see the order in which events happen.
6Labels, Quantities, and Several Locks at Once
Sometimes one named resource is not enough. Here is how to lock pools, and how to avoid the classic trap of locking several things.
Lock by label: any one of a pool
Suppose you have five identical test phones. A test build needs one, but does not care which. You give each phone the label “phone” in the Lockable Resources settings. A build then asks for a lock by label. Jenkins picks any free phone that has the label, and can tell the build which one it received through the variable option. If all phones are busy, the build waits.
Quantity: several from the pool
A build that needs three phones at once can ask for a quantity of three. Jenkins waits until three are free and hands them all over together. Taking them all together, and not one at a time, matters, as the next section explains.
The deadlock trap
Imagine two builds. Build A locks the database and then asks for the test server. Build B locks the test server and then asks for the database. Each holds what the other needs, and each waits forever. This is a deadlock, and no timeout in the world will fix it unless one build gives up.
Two people meet in a narrow corridor. Each politely refuses to step back until the other does. Neither moves, and the corridor is blocked. Locks acquired in different orders behave the same way.
How to avoid deadlock
Take everything at once
Use the extra option, or a quantity, to request all needed resources in one go, so a build never holds some while waiting for others.
Use one order everywhere
If you must nest locks, always take them in the same order, for example alphabetically by name.
Add timeouts
Wrap the lock in a time limit, so a build gives up and fails clearly instead of waiting all night.
Prefer fewer locks
Sometimes one bigger resource, such as “the whole staging environment”, is simpler and safer than several small ones.
Coarse locks (fewer, bigger)
- Simple to understand and hard to deadlock.
- Fewer names to keep consistent.
Fine locks (many, small)
- More builds can run at the same time.
- More names, more chances for deadlock or typos.
7The Lock Queue: Who Goes Next?
When several builds want the same lock, Jenkins puts them in a line. The rules of that line matter more than most people expect.
sequenceDiagram
participant B1 as Build 1
participant B2 as Build 2
participant B3 as Build 3
participant L as Lock for the test server
B1->>L: Request the lock
L-->>B1: Granted
B2->>L: Request the lock
L-->>B2: Wait in the queue
B3->>L: Request the lock
L-->>B3: Wait in the queue
B1->>L: Release the lock
L-->>B2: Granted next in order
B2->>L: Release the lock
L-->>B3: Granted
The default: first come, first served
By default, builds get the lock in the order they asked for it. This is fair, like a queue at a shop till. It is a good rule for work where every build matters, for example a resource that each build needs to test its own results.
Newest first: inverse precedence
For deployments, fairness is often the wrong rule. If five builds are waiting to deploy, only the newest one matters, since it contains the work of the others. The inversePrecedence option flips the queue so that the newest waiting build goes first. Combined with a milestone, which cancels the older waiting builds, this gives the behaviour most teams want: the latest change deploys, and stale ones vanish.
| Queue rule | Who gets the lock next | Best for |
|---|---|---|
| Default | The build that has waited longest | Work where every build must run |
| Inverse precedence | The newest build that is waiting | Deployments and other “latest wins” work |
| Skip if locked | Nobody waits. Busy means skip | Optional work, such as a nightly cleanup that can wait for tomorrow |
| Priority, where supported | The build with the highest priority number | Urgent fixes that must jump ahead |
At a hospital, patients are usually seen in order of arrival. But an emergency case goes first. In a print shop taking orders for the same poster, only the latest revision of the artwork is worth printing, so earlier revisions still waiting are dropped. Choosing a queue rule is choosing which kind of waiting room you run.
Limits on waiting
A build that waits for a lock does not use up a real executor, when the lock is placed correctly, but it still occupies a place in the queue and a place in the developer’s patience. Wrap the lock in a timeout, so that a build that has waited far longer than any normal run gives up with a clear message. Otherwise, one stuck build holding the lock can silently block every build behind it for hours.
Choose the timeout by asking what the longest sensible wait is. If a deployment takes twenty minutes, and at most three builds usually queue, a wait of an hour or so is generous. Anything longer suggests a stuck lock, and a person should look at it.
8The Milestone Step in Detail
Now the second step. The milestone step is small, but it changes how concurrent builds behave in a big way.
The basic idea
A milestone is a marker that you place at a point in a pipeline. It is like a checkpoint on a race course. Jenkins keeps track, for each pipeline, of which builds have passed each checkpoint. The rule it enforces is simple: builds must pass a milestone in order, oldest first, and a newer build passing it makes older builds that are still behind it obsolete.
When an older build is made obsolete, Jenkins aborts it. This happens even if that build is in the middle of some other work, because the whole point is that its result is no longer wanted.
Imagine a relay of newspaper vans, all heading to the printing plant with the day’s copy. At the plant gate stands a checkpoint. When the van with the newest copy passes the gate, every older van still on the road is called back, since its copy is out of date. The gate is the milestone.
What the step accepts
| Option | Meaning | Notes |
|---|---|---|
| None | Jenkins gives the milestone the next number automatically, in the order the pipeline meets them | Enough for most pipelines |
ordinal | An explicit number for this milestone | Rarely needed. Do not mix with automatic numbering |
label | A human-readable name shown in logs | Helpful for people reading the build page |
Ordinals
Each milestone in a pipeline has an ordinal, a number that says which checkpoint it is: first, second, third, and so on. Jenkins works these out automatically from the order in which the pipeline meets the milestone steps. That means the first milestone step reached is number one, the second is number two, and so on. Builds are compared at each ordinal separately.
Adding, removing, or reordering milestone steps in a pipeline changes their ordinals. While old builds are still running with the previous numbering, the mix can produce surprising cancellations. Change milestone layouts when few builds are active, and test the change with two overlapping builds.
Where milestones are usually placed
- At the end of the build and test phase, just before anything that publishes or deploys.
- Just before a lock, so that outdated builds are cancelled instead of queuing for it.
- Inside a lock, as a second check, so that an outdated build which got the lock first is stopped before it does harm.
- Between environments, such as before staging and before production.
Milestones do not do anything special for the first build of a job, since there is no older build to cancel. They only start to matter when builds overlap.
9How Milestone Ordering Really Works
The rules are simple, but they have two sides. Let us look at both.
Rule one: a newer build passing cancels older builds behind it
When a build reaches a milestone and passes it, Jenkins looks at older builds of the same pipeline. Any older build that has already passed the previous milestone but has not yet passed this one is aborted. Such builds are, by definition, running behind the build that just went through, and are therefore outdated.
Rule two: an older build cannot pass after a newer one
If an older build finally reaches a milestone after a newer build has already passed it, the older build is not allowed through. It is aborted at the milestone. This closes the out-of-order gap: Build 41 can never finish after Build 42 and overwrite its results.
sequenceDiagram
participant B40 as Build 40
participant B41 as Build 41
participant M as Milestone Two
B40->>B40: Still running long tests
B41->>M: Reaches the milestone first
M-->>B41: Passes
M-->>B40: Aborted because a newer build already passed
Note over B40,M: An older build can never pass after a newer one
The ordering, step by step
Build 40 and Build 41 are both running
Build 41 was triggered later. Both have passed the earlier stages, and both are heading for the second milestone.
Build 41 reaches the milestone first
It may have fewer tests to run. Jenkins lets it pass, and records that build 41 is now the newest build past this point.
Jenkins aborts Build 40
It is older and has not passed the milestone. The console shows that it was cancelled because of a newer build.
Build 41 continues alone
The work that follows the milestone is now done by only the newest build.
In a photo-finish race with a rule that “only the latest entry counts”, if a runner from a later heat crosses the line first, the earlier heat’s stragglers are told to stop. They cannot finish afterwards and claim the place. Both rules are enforced at the finish line.
What “aborted” looks like
An aborted build is not a failed build. It is marked with the aborted status, which most teams show in grey, and it does not count as a test failure. Your notification rules should treat it that way, or people will be alerted about builds that were simply superseded. The console log of such a build usually contains a short message saying that a newer build has passed the milestone.
Tell your team early that aborted builds after a milestone are expected and healthy. Otherwise, the first developer to see one will wonder if something is broken.
10Combining Lock and Milestone: The Safe Deployment Pattern
Used together, the two steps solve the deployment problem completely. Here is the pattern most teams end up with.
The aim is easy to state: only the latest build deploys, and only one deployment happens at a time. The lock gives the “one at a time”. The milestone gives the “only the latest”. But the order in which you place them, and a couple of details, matter a lot.
sequenceDiagram
participant N as Newest Build
participant O as Older Builds
participant M1 as Milestone Before the Lock
participant L as Deploy Lock
participant M2 as Milestone Inside the Lock
participant D as Deploy Step
N->>M1: Pass the first milestone
M1-->>O: Cancel older builds still waiting
N->>L: Ask for the lock
L-->>N: Granted
N->>M2: Pass the second milestone
M2-->>O: Cancel any older build that slipped in
N->>D: Deploy safely
D->>L: Release the lock
The recipe in five moves
Build and test in parallel with others
Building and testing does not touch shared resources, so many builds may do it at once with no lock at all.
Place a milestone before the lock
When a newer build passes, older builds that are waiting for the lock are cancelled instead of standing in the queue.
Lock the deployment target, with newest first
Use inverse precedence, so when the lock frees up, the newest waiting build gets it and not the oldest.
Place a second milestone inside the lock
This catches the rare case where an older build gained the lock before a newer one arrived. As soon as it is inside, a milestone check stops it if a newer build has already passed.
Deploy, and let the lock release itself
When the block ends, for any reason, the lock is freed, and the next waiting build proceeds.
Think of a bakery delivery dock with one loading bay. The dock manager first turns away any van that carries yesterday’s bread while today’s van has already arrived (the first milestone). Only one van may use the bay at a time (the lock), and the newest waiting van goes in first (inverse precedence). Once inside the bay, the driver does a final check: “Is there a fresher van behind me?” (the second milestone). Then the loading happens.
Why two milestones and not one
The first milestone cancels builds that are still waiting. The second protects against a timing gap: an older build might have acquired the lock a moment before a newer build reached the first milestone. Without the inside check, the older build would deploy, and then the newer one would deploy on top of it. With the inside check, the older build is caught as soon as the newer one passes, or is stopped when it tries to pass the inside milestone.
Exact behaviour, especially at the boundaries between waiting, holding the lock, and passing a milestone, depends on plugin versions. Test this pattern on a non-critical pipeline first, start three builds close together, and read the logs of each one.
11How They Work Internally
This chapter lifts the lid. Knowing where the state lives explains behaviours that otherwise feel like magic.
The lock: a shared ledger on the controller
The Lockable Resources plugin keeps a central record on the controller. For each resource, it knows whether the resource is free, who holds it, and who is waiting. When a pipeline reaches a lock step, the pipeline engine asks this record for the resource. If it is free, the record marks it as held by that build and lets the build continue. If not, it adds the build to a waiting list, and the build pauses until the lock is handed over.
Because the record is on the controller, and not on any agent, the lock works across all agents, all branches, and all pipelines on that Jenkins. It does not, however, work across different Jenkins servers. Two separate Jenkins installations cannot see each other’s locks.
A meeting-room booking board hangs in the reception of one office building. Everyone in that building checks the board. It does not help people in another building, who have their own board and know nothing about this one. The lock ledger is that board.
Release is automatic
The lock is tied to the block of work, and not to a step that must be remembered. When the block ends by success, failure, or abort, the plugin releases the lock and offers it to the next waiting build. This is the great advantage over hand-made locking schemes, which often leave a lock behind when something goes wrong.
Persistence and restarts
The plugin saves its state to disk on the controller, so that locks and queues can survive a controller restart, and pipelines resumed after the restart can carry on holding what they held. The detailed handling of unusual cases, such as a build that vanished while holding a lock, is what the management page’s tools are for.
The milestone: a per-job scoreboard
The milestone plugin keeps, for each job, a small record of the highest build number to have passed each milestone ordinal. When a build reaches a milestone, the plugin compares its build number with the recorded number. If the build is newer, it passes, the record is updated, and older builds that are behind are told to stop. If the build is older, it is stopped. Because the record is per job, milestones in one job never affect another job.
| Question | Lock step | Milestone step |
|---|---|---|
| What does it record? | Who holds each resource, and who is waiting | The newest build to pass each milestone in each job |
| Where does it apply? | Across all jobs on this Jenkins that use the same resource name | Within one job only |
| What happens to the loser? | It waits, and later runs | It is aborted |
| Is anything cancelled? | No, unless the pipeline adds a timeout | Yes, older builds that are behind |
| Does it survive a restart? | Designed to | Designed to, as part of the job’s saved state |
Details of these internals differ between plugin versions. The picture above is the general design as documented and commonly understood. For a specific server, the Lockable Resources page and the build logs show what is really happening at any moment.
12Lock, Milestone, and the Other Concurrency Tools
Jenkins offers several ways to control overlapping builds. Choosing the right one saves a lot of trouble.
| Tool | What it controls | What happens to extra builds | Best for |
|---|---|---|---|
| Disable concurrent builds option | Whether two builds of the same job may run at once | They wait in the queue | Jobs that must never overlap at all |
| Disable concurrent builds with abort previous, in newer versions | The same, but a new build aborts the running one | Older running builds are aborted | Simple “latest wins” jobs with no shared resource |
| Throttle concurrent builds plugin | How many builds of a category may run together | They wait in the queue | Limiting load on a shared service across jobs |
| Lock step | Exclusive use of a named resource within a block | They wait for the lock | Shared servers, devices, and environments |
| Milestone step | Order of builds at a checkpoint | Older builds are aborted | Stopping outdated builds from finishing after newer ones |
| Executor limits | How many builds can run on machines at all | They wait for a free slot | Protecting machine capacity |
A theatre can limit the total audience with a seat count (executors), cap how many people may enter per minute (throttle), allow one person at a time into a fitting room (lock), and turn away ticket holders for last night’s show once tonight’s has begun (milestone). Each rule fixes a different problem.
Choosing by the problem
- “Two runs of this job must never overlap.” Disable concurrent builds.
- “Only the newest run of this job matters.” Use the abort-previous form if your version has it, or milestones for finer control.
- “Different jobs share a server.” Use a lock with the same name in each of them.
- “Too many jobs hit one service at once.” Use throttling, or a lock with a quantity.
- “An old build must not deploy after a new one.” Use a milestone before the deploy.
Prefer the simplest tool that solves the problem. A single option on the job is easier to understand than a lock and two milestones, if it does what you need.
13Managing Resources: The Lockable Resources Page
Locks are not only used by pipelines. People can see and manage them on a special page in Jenkins.
The Lockable Resources plugin adds a page that lists every known resource, its labels, its description, whether it is free, and which build holds it or is waiting. This page is useful for operating your delivery system day to day.
Reserve
A person marks a resource as reserved. Builds cannot take it until it is unreserved. Useful for manual testing or maintenance on a shared server.
Unreserve
Releases a personal reservation, so builds can use the resource again.
Steal or reassign
An authorised user takes over a reservation from someone else, or gives it to another person. Handle with care.
Unlock
Forcefully frees a lock held by a build that is stuck. A last resort, since the build may still be using the resource.
A shared meeting room can be booked by people for their own use. A caretaker can put a “closed for repairs” sign on the door, so nobody may enter. The reserve action is that sign. Removing it opens the room again.
Labels and descriptions
Give every resource a clear description saying what it is, who owns it, and what it is safe to do with it. Use labels to group resources into pools. For example, all test devices might carry the label “android-device”, so pipelines can ask for any one of them without naming a particular device.
Manual work and pipelines together
Because a person can reserve a resource, a pipeline waiting for it might sit in the queue for a long time without any technical fault. When the console says the build is waiting for a lock, check whether someone has reserved the resource by hand. Ask them to unreserve it, or agree on a policy, for example that a reservation lasts no longer than a working day.
Anyone with the right to steal or unlock can break the protection that a lock gives. Grant those rights only to a small group who understand the consequences, and record when they are used.
14Performance and Scalability
Locks trade speed for safety. This chapter explains what that costs and how to keep the cost small.
What a lock costs
A lock turns parallel work into serial work for the locked block. If a deployment takes twenty minutes and the lock allows one build at a time, then at most three deployments finish per hour, no matter how many builds you have. The lock is a bottleneck by design. The question is whether it is a bottleneck you can afford.
These numbers are only an illustration, but they show the shape of the problem. Six queued builds, at twenty minutes each, mean a two-hour wait for the last one. If five of those builds are outdated, the wait is pure waste. With milestones cancelling the outdated ones, the newest build waits only for the deployment in progress, at most twenty minutes.
A single ticket window for a busy railway station handles one traveller at a time. If each traveller takes two minutes, thirty travellers mean an hour of queue. Opening more windows, or sending travellers with old tickets away without waiting, shortens the queue.
Ways to keep the cost small
Lock only the critical part
Build, compile, and unit test outside the lock. Lock only the step that touches the shared resource.
Shorten the locked block
Move slow checks before or after the lock, and keep the locked part to setup, deploy, and the essential tests.
Add more resources
If one server is a bottleneck, create a pool of servers with a label, and lock by label. Throughput grows with the pool.
Cancel what does not matter
Use milestones so that builds that are already outdated leave the queue.
Create environments on demand
Where possible, give each build its own temporary environment, so no lock is needed at all.
Watch the wait times
Track how long builds wait for each lock. A rising trend tells you when to add capacity.
Scale considerations
On a very busy Jenkins with hundreds of builds waiting on many locks, the controller does bookkeeping work for every waiting build. This is normally light, but it grows with the number of waiting builds and resources. Keep the number of long-waiting builds low, and do not use locks as a general throttle for large fleets when a proper throttling plugin or a scheduling change is a better fit.
15Reliability: Stuck Locks, Restarts, and Timeouts
A lock is a promise that something will be released. Reliability is about what happens when that promise is broken.
| Situation | What happens | Protection |
|---|---|---|
| The locked block fails | The lock is released automatically and the next build proceeds | Nothing extra needed |
| The build is aborted while holding the lock | The lock is released when the block unwinds | Make cleanup steps safe to run on abort |
| The build hangs inside the block | The lock is held, and everyone behind waits | A timeout around the locked block |
| The agent dies mid-block | The block fails and the lock is released, once Jenkins notices | Reliable agents, and timeouts |
| The controller restarts | The pipeline resumes and the lock state is restored | Test restarts, and use the management page for unusual leftovers |
| A person reserves and forgets | Builds wait indefinitely | A reservation policy, and timeouts on waiting |
| Deadlock between two builds | Both wait forever | Acquire all resources at once, or in one agreed order |
A hotel room key card expires at check-out time, even if the guest forgets to return it. The hotel does not depend on people remembering. Locks with timeouts work the same way: they do not depend on every build behaving perfectly.
Two timeouts, not one
- A timeout on waiting for the lock. If a build cannot get the lock within a sensible time, it gives up with a clear message, so a person can investigate.
- A timeout on the locked work. If the block runs much longer than usual, it is stopped, and the lock is released for the others.
Cleanup inside the locked block
A locked block often changes the shared resource. It might load test data, reserve a port, or start a service. If the block is aborted, that state may be left behind. Design the block so that the start of the next locked run cleans up any mess left by the previous one, or use cleanup actions that run even on failure. Otherwise, a single aborted build can leave the shared resource in a state that makes every following build fail.
Make the first step inside a locked block a reset: bring the shared resource to a known, clean state. It costs a little time and prevents a whole family of mysterious failures.
16Security Considerations
Locks and milestones are safety tools for correctness. They are not security walls. Here is what they can and cannot protect.
A lock is an agreement, not a barrier
A lock works only because every pipeline that uses the shared resource agrees to ask for it first. Nothing stops a pipeline, or a person, from using the real server without asking for the lock. Anyone who can edit a Jenkinsfile can remove the lock step, use a different name, or ignore locks altogether. So a lock protects against honest mistakes and accidental collisions. It does not protect against a hostile user.
A “please take a ticket” sign at a deli counter keeps honest customers in an orderly line. It does not stop someone from pushing to the front. Real protection needs a staff member checking tickets, which in Jenkins means permissions and reviews.
Removing the lock
A change to the Jenkinsfile can drop the lock step, and two deployments will collide again. Review pipeline changes that touch locks or milestones.
Untrusted pull requests
A build from an outside contributor could take a lock and hold it, blocking your whole team. Do not let untrusted builds use shared resources that others depend on.
Manual stealing and unlocking
People with the rights to reserve, steal, or unlock can disrupt deployments or hide mistakes. Limit these rights.
Denial of service by waiting
A build that never releases a lock, or a person who reserves everything, can stop delivery. Use timeouts and reservation policies.
Cancelled builds hiding problems
Milestones abort older builds silently. If an important security scan is in an older build, it may be cancelled before it finishes. Place milestones only after scans that must complete.
Resource names leaking information
Names such as “prod-payments-db” appear in logs and on the resources page. Do not put secrets or sensitive details in names.
Habits that protect you
- Use Jenkins permissions for the resources page. Only trusted people may reserve, steal, reassign, or unlock.
- Protect the real resource as well. The shared server itself should have its own access rules, so that the lock is a helpful convention on top of real controls.
- Review pipeline changes. A change to a deploy stage is as important as a change to production code.
- Restrict who can deploy. Use folders, branch rules, and approvals, in addition to locks.
- Keep untrusted builds away from the locks that guard shared and valuable resources.
Do not describe a lock as “security” in your own documentation. Doing so leads people to relax real controls, because “it is locked”. Call it what it is: a way to keep honest builds from colliding.
17Monitoring, Logging, and Troubleshooting
When a build waits too long, or vanishes as aborted, here is how to find the reason calmly.
What Jenkins shows you
- The console log. A build waiting for a lock prints a message naming the resource. A build stopped by a milestone prints a message about a newer build.
- The Lockable Resources page. Shows which resources are free, which are held and by whom, which are reserved by hand, and which builds are waiting.
- The pipeline steps view. Shows the lock step as a running step, so you can see how long it has been waiting.
- The build history. Shows aborted builds in grey, which after milestones is a normal sight.
- Metrics and audit plugins. Can record lock wait times and manual actions, for trends and for accountability.
flowchart TD
A["A build is waiting or was aborted unexpectedly"] --> B{"Does the console say waiting for a lock"}
B -->|Yes| C{"Who holds the lock"}
C -->|A running build| D["Wait, or check that build for a hang and add timeouts"]
C -->|Reserved by a person| E["Ask the person or unreserve it on the resources page"]
C -->|Nobody yet still waiting| F["Check resource and label names for typos and check the queue order"]
B -->|No| G{"Was it aborted because of a newer build"}
G -->|Yes| H["A milestone cancelled it so this is expected"]
G -->|No| I["Check timeouts, manual aborts, and fail fast settings"]
Common symptoms and causes
| Symptom | Likely cause | What to do |
|---|---|---|
| Build waits for a lock for hours | A hung build holds it, or a person reserved it | Check the resources page, then abort the hung build or unreserve |
| Two builds deploy at the same time | Different resource names, or the lock is missing from one pipeline | Compare names character by character and confirm every pipeline has the lock |
| Old build deploys after a new one | No milestone, or the milestone is placed before the slow part | Place a milestone right before the deploy, and one inside the lock |
| Builds aborted right after being started | A newer build passed the milestone, or a milestone layout changed | Read the log message. If unexpected, check ordinals and ordering |
| All builds hang on a lock with nobody holding it | A stale record after a crash or restart | Use the management page to unlock, after confirming nothing is using the resource |
| Executors all busy but nothing runs | Lock placed inside an agent block, so builds wait while holding executors | Move the lock outside the agent, or use the stage option |
| Two pipelines deadlock | Locks acquired in different orders | Acquire together with the extra option, or use one fixed order |
| Deployment order is oldest first | Default queue order is first come, first served | Use inverse precedence for deployments |
Numbers worth tracking over time
Track the average and worst wait time for each lock. Track how many builds are aborted by milestones each week, since a sudden rise may point to a layout mistake. Track how often someone unlocks or steals by hand. And track how long each locked block takes, since it sets the maximum speed of the whole delivery line.
18Design Patterns and Anti-patterns
Patterns are habits that keep delivery calm. Anti-patterns are habits that create mysterious waits and lost builds.
Patterns that help
Build free, deploy locked
Build and test in parallel with no lock. Lock only the step that touches the shared environment.
Milestone before deploy
Cancel outdated builds before they queue for the lock or deploy.
Newest first
Use inverse precedence on deployment locks, so the latest build wins the next turn.
Lock outside the agent
Acquire the lock before an executor is taken, so waiting builds do not hold slots.
Pools by label
Group identical resources under a label, so builds take any free one and throughput scales with the pool.
Reset at the start
The first step inside a lock returns the shared resource to a clean, known state.
Anti-patterns to avoid
Context
A team wraps the whole pipeline, from checkout to deploy, in one lock, to be safe.
What goes wrong
Only one build at a time can even compile or test. Builds queue for an hour before doing work that never needed exclusive access, and developers wait for feedback they could have had in minutes.
Better decision
Lock only the deploy and the tests that depend on the shared environment. Let everything else run in parallel.
Context
A lock is used with no milestone, and builds pile up behind it in first-come, first-served order.
What goes wrong
Outdated builds deploy one after another, each one overwriting the last, until the newest finally deploys. Time is wasted, and for a while the environment runs old code.
Better decision
Add a milestone before the lock, use inverse precedence, and add a second milestone inside the lock.
Context
A lock is placed inside an agent block, and no timeout is set.
What goes wrong
Each waiting build holds an executor while doing nothing. A single hung deployment stalls the lock, and soon all executors are held by waiting builds, so even unrelated jobs cannot start.
Better decision
Acquire the lock before allocating the agent, and add timeouts on both the wait and the locked work.
Context
A milestone is placed at the very start of the pipeline, before the slow tests and security scans.
What goes wrong
Each new commit cancels the previous build before it has completed its tests. On a busy branch, no build ever finishes, and nothing gets verified.
Better decision
Place milestones after the checks that must finish, and just before the steps that must be ordered, such as publishing and deploying.
19Advantages, Trade-offs, Best Practices, and Common Mistakes
A balanced view of the two steps, and the habits that keep them working for you.
Advantages
- Prevents collisions on shared servers, databases, and devices.
- Stops outdated builds from deploying after newer ones.
- Saves machine time by cancelling obsolete builds.
- Locks are released automatically, even on failure or abort.
- Small, readable steps in the pipeline file.
- Pools and labels let capacity grow.
Disadvantages and trade-offs
- A lock turns parallel work into serial work.
- Names must be agreed and kept consistent.
- Deadlocks are possible with several locks.
- Milestone ordinals are fragile when layouts change.
- Neither step is a security control.
- Locks work within one Jenkins only.
Best practices in one place
- Lock only the critical part. Keep everything else outside the lock.
- Acquire the lock before the agent, so waiting builds do not hold executors.
- Use inverse precedence for deployments, so the newest build goes first.
- Add a milestone before the lock and another inside it.
- Place milestones after checks that must finish, never before them.
- Add timeouts for waiting and for the locked work.
- Take multiple resources together, or in one fixed order, to avoid deadlock.
- Agree on resource names centrally, and define important ones on the resources page.
- Reset the shared resource at the start of the locked block.
- Limit who can reserve, steal, and unlock, and treat aborted builds after milestones as normal in your alerts.
- Test with three overlapping builds before trusting a new layout.
The mistakes seen most often
Mistake 1: different names for the same thing
Two pipelines lock “staging” and “Staging”. Resource names are case sensitive, so they do not exclude each other, and deployments collide. The fix is one agreed name, defined once.
Mistake 2: milestone in the wrong place
A milestone sits before the slow tests, so every new commit aborts the build that was still testing. The fix is to move it after the tests.
Mistake 3: the forgotten reservation
A person reserves a server “for a minute” and leaves for a holiday. Every build waits. The fix is a reservation policy, plus timeouts on waiting.
Mistake 4: alarms for aborted builds
Notifications treat aborted builds as failures, and the team is spammed each time a milestone cancels an outdated run. The fix is to exclude aborted status from failure alerts.
Mistake 5: relying on locks for security
Documentation says “the deploy server is locked”, and access rules on the server itself are left loose. The fix is to protect the server directly, and use the lock only for ordering.
20Worked Scenario: Taming a Busy Staging Server
Let us follow one imaginary team as it fixes a messy staging environment, step by step, using both steps.
The starting point
A team of fifteen developers shares one staging server. Their pipeline builds, tests, deploys the result to staging, and runs a set of end-to-end tests there. The deployment and tests together take about twenty minutes. On a busy afternoon, six commits arrive within an hour. All six builds reach the deploy stage at nearly the same time. The results are chaos: files from two versions are mixed on the server, end-to-end tests fail for no visible reason, and sometimes an older build finishes last, leaving old code on staging.
Name the shared thing
The team agrees the shared thing is “the staging environment”, and creates a single resource for it on the Lockable Resources page with a clear description and an owner.
Add a plain lock around deploy and end-to-end tests
Collisions stop at once. But now the six builds queue in arrival order. The last one waits about two hours, since each locked block takes twenty minutes, and old versions deploy in turn.
Move the lock outside the agent
They use the stage option so that waiting builds no longer hold executors. Other unrelated jobs on the same agents start running normally again.
Add milestones and inverse precedence
A milestone before the lock cancels outdated waiting builds. The lock serves the newest waiting build first. A second milestone inside the lock catches any older build that slipped in.
Add timeouts and a reset step
The wait is limited to an hour, the locked work to thirty minutes, and the first step inside the lock resets the staging server to a clean state.
Set policies for people
Only two team leads may steal or unlock. Manual reservations must be released by the end of the working day. Alerts ignore builds aborted by milestones.
| Version | Collisions | Wait for the newest build | Outdated builds deploy? | Machine time on outdated builds |
|---|---|---|---|---|
| No lock | Frequent | None, but results unreliable | Sometimes, out of order | Wasted on all six |
| Plain lock, first come first served | None | Up to about 2 hours for the sixth | Yes, one after another | About 100 minutes of locked time wasted on five builds |
| Lock, inverse precedence, two milestones | None | At most about 20 minutes | No, they are cancelled | Near zero for the cancelled builds |
It is like moving from a chaotic market stall where everyone shouts at once, to a stall with a single serving window, a “newest order first” sign for the special case of amended orders, and a rule that old, replaced orders are torn up. The stall serves fewer orders, but every one that is served is the right one.
The numbers here are made up to show the pattern. Introduce the changes one at a time on a quiet day, test with three overlapping builds, and read the log of each to confirm the behaviour before relying on it.
21Real-World Usage Patterns
Here is how the two steps tend to appear in real delivery pipelines. These are illustrative patterns in general terms, not claims about the private setups of any particular company.
Jenkins is one of the most widely used automation servers, and teams of every size run into the same two problems: shared environments and overlapping builds. What tends to be common is a habit of locking only what is truly shared, cancelling what is obviously outdated, and watching wait times. The patterns below show that habit in practice.
Pattern: one shared staging environment
A product team locks its single staging environment during deploy and end-to-end tests, with a milestone before and inside the lock, so only the newest change reaches staging.
Pattern: a pool of test devices
A mobile team keeps a rack of phones, each tagged with the same label. Test pipelines lock by label, take any free phone, and release it when finished. The pool size sets how many test runs can happen at once.
Pattern: infrastructure changes one at a time
A platform team uses a lock so that only one pipeline at a time can apply changes to a given cloud environment, avoiding two changes racing and leaving the environment in a half-changed state.
Pattern: licence seats as resources
A team with a limited number of seats for a commercial tool creates one resource per seat. Pipelines that need the tool lock by label, so they never exceed the number of licences they own.
Pattern: release branches with ordered promotion
A team uses a milestone before each promotion stage, so a slow build of an older commit can never be promoted after a newer one has already moved on.
Jenkins is often used together with Docker, Kubernetes, and cloud platforms that can create fresh environments on demand. Where each build can have its own temporary environment, no lock is needed at all. Locks and milestones matter most when a shared, long-lived environment cannot be avoided.
22Frequently Asked Questions
23Summary and Key Takeaways
Key Takeaways
- The lock step makes builds take turns. It gives one build at a time exclusive use of a named resource for a block of work, and releases it automatically when the block ends.
- The milestone step makes newer builds win. Builds pass a checkpoint in order, and a newer build passing cancels older builds that are behind it.
- Use both for safe deployments. Place a milestone before the lock, use inverse precedence so the newest waits first, and add a second milestone inside the lock.
- Lock as little as possible. Build and test outside the lock, lock only the critical step, and acquire the lock before the agent so waiting builds do not hold executors.
- Prevent deadlock and stuck locks. Take several resources together or in one fixed order, and add timeouts on both waiting and locked work.
- Place milestones with care. Put them after the checks that must finish and before the steps that must be ordered, never at the very start of a pipeline.
- Remember what they are not. Neither step is a security control, and locks work within one Jenkins only, so protect real resources directly and review pipeline changes.