Pipeline Checkpoints and Resuming: How Jenkins Never Forgets Where a Build Was
A complete, beginner-friendly guide to how Jenkins survives restarts mid-build, saves its place with checkpoints, and lets long pipelines pick back up exactly where they left off.
Imagine reading a thousand-page novel with no bookmark, no dog-eared pages, and a habit of occasionally losing power to the reading lamp mid-chapter. Every time the lights flicker back on, you’d have to start again from page one, because nothing recorded exactly where you were. Long-running Jenkins pipelines face an equivalent risk: a build might run for hours, spanning a controller restart, a plugin update, or simply a planned maintenance window, and losing all that progress every time would make long, multi-stage pipelines painfully fragile. Jenkins solves this with a genuinely clever mechanism — it doesn’t just remember that a pipeline was running, it remembers exactly which line it was on, ready to pick up mid-sentence the moment things come back online. This guide explains how that persistence works, what a “checkpoint” actually is, and how to design pipelines that resume safely rather than dangerously.
1Why Pipelines Need to Survive Interruption
Before looking at checkpoints specifically, it helps to understand the underlying problem Jenkins pipelines were designed to solve.
The controller is a single point that sometimes needs to restart
The Jenkins controller — the central process coordinating every build — occasionally needs to restart: applying a security update, upgrading a plugin, or recovering from an unexpected crash. In older, simpler automation tools, any build running at that exact moment would simply be lost, forcing someone to notice the failure and start the entire process over from scratch, however long it had already been running.
An old-fashioned tool that loses all progress on restart is like a video game with no save feature at all — a single power outage means replaying the entire game from the very beginning, no matter how far you’d gotten.
Why this matters more as pipelines grow longer
A short, five-minute build losing its progress to a restart is merely annoying. A pipeline that runs for several hours — perhaps waiting on a long-running test suite, an external approval, or a slow deployment across many servers — losing all of that progress to an untimely restart is a genuinely serious problem, wasting significant time and potentially leaving infrastructure in a half-finished, inconsistent state.
Controller
The central Jenkins process that schedules and coordinates builds, and which occasionally needs to restart itself.
Durability
Jenkins’ general capability to preserve a running pipeline’s progress and state across an interruption.
Checkpoint
A deliberately saved point in a pipeline’s progress that a later run can be restarted from, skipping everything before it.
CPS (Continuation-Passing Style)
The technique Jenkins’ Pipeline engine uses internally to represent a running script in a form that can be paused and later resumed.
Solving this problem required Jenkins to fundamentally rethink how a running pipeline script is represented internally, which is exactly the subject of the next chapter.
2The CPS Engine — Making a Running Script Pausable
Surviving a restart requires the pipeline’s exact execution position to be captured in a form that can be written to disk and later reconstructed.
Ordinary scripts can’t be paused mid-line
A typical script, once running, holds its current position purely in the computer’s temporary memory — the exact line it’s on, the values of its variables — with no mechanism to write that position down and later reconstruct it elsewhere. If that memory is wiped out by a process restarting, the script’s position is gone permanently, along with any variables it was holding.
Continuation-Passing Style as the solution
Jenkins Pipeline scripts are internally transformed into a special form called Continuation-Passing Style, or CPS, specifically so that the entire state of execution — which step is currently running, what values local variables hold, which stage and branch of logic is active — can be serialized: converted into a form that can be written to disk as ordinary data, and later read back and reconstructed exactly as it was.
An ordinary script is like reading a book purely from memory, with nothing written down — lose your train of thought, and you’ve lost your place entirely. A CPS-transformed pipeline is like reading with a detailed bookmark that notes not just the page, but the exact sentence, which character just spoke, and what you were thinking at that exact moment — detailed enough that anyone could pick the book back up and continue exactly as if no time had passed.
Where this serialized state actually lives
Jenkins periodically writes this serialized execution state to a file on disk associated with the running build, updating it as the pipeline progresses through its steps. When the controller restarts, it reads this file back for any build that was in progress, reconstructs the CPS execution state from it, and continues running from that exact reconstructed position.
This persistence mechanism is a built-in, automatic feature of Jenkins Pipeline — a pipeline author doesn’t need to write any special code to benefit from surviving a controller restart; it happens as a natural consequence of how Pipeline scripts are executed internally.
With the general durability mechanism understood, the next chapter looks at architecture and components more closely, distinguishing this automatic restart survival from the more deliberate, explicit idea of a checkpoint.
3Architecture & Components
Several distinct pieces cooperate to provide both automatic restart survival and deliberate, explicit resuming.
CPS Execution Engine
Transforms and runs pipeline scripts in a form whose execution state can be serialized and later reconstructed.
Durability Setting
A configurable option controlling how frequently and thoroughly Jenkins writes a running pipeline’s state to disk, trading persistence robustness against performance overhead.
Build State File
The on-disk file holding a specific build’s serialized execution state, read back after a restart to resume it.
Restart From Stage / Checkpoint Feature
A separate, more deliberate capability letting someone explicitly re-run a completed or failed pipeline starting from a specific named stage, rather than from the very beginning.
Two distinct kinds of “resuming”
It’s worth clearly separating two related but different ideas that this guide’s title brings together. The first is automatic restart survival — a still-running build picking up exactly where it was after an unplanned or planned controller restart, handled entirely by the CPS engine and durability settings. The second is deliberately restarting a pipeline that has already finished or failed, from a specific named point, rather than from scratch — this is the explicit “checkpoint” or “restart from stage” capability, aimed at saving time when only a later part of a long pipeline needs to run again.
flowchart TD
A[Running Pipeline] --> B[CPS Engine Serializes State Periodically]
B --> C[Build State File on Disk]
D[Controller Restarts] --> E[Reads Build State File]
E --> F[Pipeline Resumes Automatically]
G[Pipeline Finishes or Fails] --> H[Explicit Checkpoint / Restart-From-Stage Feature]
H --> I[New Run Starts From Chosen Stage, Reusing Earlier Results]
Automatic restart survival is like a video game’s autosave quietly kicking in every few minutes so a sudden power cut doesn’t erase your progress. A deliberate checkpoint is more like a save file you consciously create right before a difficult boss fight, specifically so you can retry just that fight without replaying the entire level leading up to it.
With these two concepts distinguished, the next chapter walks through exactly what happens internally in each case, starting with the automatic restart-survival path.
4Internal Working — Surviving an Unplanned Restart
A closer look at what actually happens when the controller running a live pipeline goes down and comes back.
Step 1 — Ongoing state persistence during normal execution
While a pipeline runs normally, the CPS engine periodically writes its current serialized execution state to the build’s state file, capturing which step is active, what stage the pipeline is in, and the values of relevant variables at that moment — this happens continuously, not just when a shutdown is anticipated.
Step 2 — The controller shuts down or crashes
Whether triggered by a planned restart for maintenance or an unexpected crash, the controller process stops running, along with every currently in-progress build’s live execution — but the most recently written state file for each build remains safely on disk.
Step 3 — The controller starts back up
As Jenkins starts again, it scans for builds that were marked as still in progress at the time of the shutdown, and for each one, reads back that build’s persisted state file from disk.
Step 4 — Execution state is reconstructed
The CPS engine deserializes the saved state, reconstructing the exact execution position the pipeline was at — which step was running, what stage it belonged to, what variables were in scope — as faithfully as the last successfully written save point allows.
Step 5 — The pipeline resumes
With state reconstructed, the pipeline continues running from that point onward, exactly as if the restart had never interrupted it, aside from the connected agent possibly needing to reconnect if it was running a step at the exact moment of interruption.
sequenceDiagram
participant PE as CPS Engine
participant SF as Build State File
participant CTRL as Jenkins Controller
PE->>SF: Periodically write serialized execution state
CTRL->>CTRL: Restart (planned or unplanned)
CTRL->>SF: Read persisted state for in-progress builds
SF-->>CTRL: Last saved execution state
CTRL->>PE: Reconstruct execution position
PE->>PE: Resume pipeline from reconstructed state
Restart survival reconstructs the pipeline script’s own execution position — it does not, and cannot, undo the real-world side effects of a step that was already partway through running when the interruption happened, such as a deployment command that had started but not finished.
Restart survival handles unplanned interruptions to a live build, but the deliberate, explicit checkpoint capability serves a different purpose entirely — restarting an already-completed or failed pipeline from a chosen point — which the next chapter covers.
5Data Flow & Lifecycle of an Explicit Checkpoint
The deliberate checkpoint capability follows its own distinct lifecycle, separate from automatic restart survival.
A Long Pipeline Runs Through Several Stages
Early stages complete successfully, producing results — build artifacts, test reports — that would be expensive to regenerate unnecessarily.
A Later Stage Fails, or Someone Wants to Re-Run Part of It
Perhaps a flaky test failed in a later stage, or a person simply wants to re-run only the deployment portion of an already-successful pipeline.
A Restart-From-Stage Is Initiated
Rather than starting an entirely new pipeline run from scratch, Jenkins is instructed to start a new run beginning at a chosen, later stage.
Earlier Stage Results Are Reused
The new run reuses the workspace and results left behind by the earlier stages of the original run, rather than recomputing them.
Execution Continues From the Chosen Stage Onward
The new run proceeds through the chosen stage and everything after it as a fresh execution, while everything before it is simply inherited rather than repeated.
Why this saves real time on long pipelines
A pipeline with an hours-long build stage followed by a much shorter deployment stage benefits enormously from being able to retry just the deployment stage after a transient failure, rather than repeating the entire lengthy build all over again purely to get back to the point that actually needs re-running.
This is like re-recording just the final verse of a song after a small mistake, using the already-recorded earlier verses as-is, rather than scrapping the entire recording session and starting the whole song over from the first note.
Because restarting from a chosen stage skips real work that already happened, it’s worth understanding exactly which kinds of pipeline setups this can safely apply to — the subject of the next chapter’s comparison.
6Types: Automatic Resume, Restart From Stage, and Milestones
Jenkins offers a few related but distinct mechanisms in this space, each suited to a different situation.
Automatic Restart Survival
Requires no special pipeline design — every running build benefits from it automatically, provided the controller’s durability settings are configured to persist state, resuming a live, in-progress build after an unplanned or planned restart.
Restart From Stage (Explicit Checkpoint)
Applies to a completed or failed pipeline, letting someone deliberately re-run it starting from a chosen, later stage, reusing whatever the earlier stages already produced rather than recomputing them.
Milestone Step
A related but distinct concept used mainly to manage concurrent builds of the same job — marking a point in a pipeline that, once reached, cancels any older, still-running build of the same job that hasn’t yet reached that same point, keeping only the newest build progressing past it.
| Mechanism | Applies To | Purpose |
|---|---|---|
| Automatic restart survival | A currently running build | Continue after an unplanned or planned controller interruption |
| Restart from stage | A completed or failed build | Re-run only from a chosen later point, saving redundant work |
| Milestone step | Multiple concurrent builds of the same job | Ensure only the newest build proceeds past a given point |
A well-designed long pipeline commonly uses all three together — automatic restart survival protects against interruptions during a live run, restart-from-stage saves time when only a later part needs retrying, and milestones prevent an older, superseded run from wastefully continuing alongside a newer one.
Whichever combination a pipeline relies on, it’s worth weighing the broader advantages and disadvantages of resumability as a design goal — the focus of the next chapter.
7Durability Levels and the Trade-off With Performance
Persisting execution state to disk isn’t free — Jenkins offers configurable durability levels that trade robustness for performance.
Writing to disk more often means more overhead
The most robust durability setting writes a pipeline’s execution state to disk after essentially every step, maximizing how little progress could ever be lost to an interruption, but adding real, measurable overhead to every single step of every pipeline, since each one now triggers a disk write.
Less frequent persistence means faster builds, more at risk
A less robust durability setting writes state less often, trading some risk of losing a small amount of recent progress during a truly unlucky, badly-timed interruption for noticeably better overall pipeline performance, particularly valuable for high-volume, short-lived builds where the odds of an interruption hitting during any single build are already low.
Maximum Durability
- Minimal risk of lost progress from any interruption
- Best suited to long-running, high-value pipelines
- Adds measurable overhead to every step
Reduced Durability
- Noticeably better overall build performance
- Well suited to short, frequent, low-risk builds
- A badly-timed interruption could lose a small amount of recent progress
Choosing a durability level is like deciding how often to save a document while writing it. Saving after every single keystroke guarantees you’ll never lose more than a fraction of a second’s work, but slows you down constantly. Saving every few minutes is far less disruptive, at the cost of possibly losing a small amount of recent typing if the power happens to cut out at just the wrong moment.
This trade-off between robustness and performance carries directly into the broader advantages and disadvantages of relying on resumability at all, which the next chapter examines.
8Advantages, Disadvantages & Trade-offs
Resumability solves a real, painful problem, but it also introduces subtleties a pipeline author needs to respect.
Advantages
- Long-running builds survive planned maintenance restarts without losing hours of progress
- Unplanned crashes are far less costly, since builds resume rather than restart entirely
- Restarting from a chosen stage saves significant time on long pipelines needing only a partial re-run
- Configurable durability lets teams balance robustness against performance for their specific needs
- Milestones keep concurrent builds of the same job from wastefully racing each other
Disadvantages / Trade-offs
- Persisting execution state adds real overhead, more pronounced at higher durability settings
- Restarting from a stage can be genuinely unsafe if that stage’s earlier steps caused real-world side effects that shouldn’t be repeated
- A step already partway through running at the moment of an interruption may not resume cleanly, depending on what it was doing
- Long-lived build state files consume disk space that needs periodic cleanup for old, finished builds
- Adds a layer of behavior that can be confusing to reason about for engineers unfamiliar with how CPS persistence works
That last caveat — designing stages that are safe to re-run — touches directly on both security and correctness, which the next chapter addresses.
9Security and Correctness Considerations
Resumability’s biggest risk isn’t a typical security vulnerability — it’s the danger of accidentally repeating a real-world action that should only ever happen once.
Idempotency is the central concern
A step is called idempotent if running it multiple times produces the same end result as running it once — checking out source code, for example, is naturally idempotent. A step that sends a one-time notification, charges a payment, or triggers an external, irreversible action is not naturally idempotent, and restarting or resuming a pipeline that includes such a step risks triggering that action a second, unwanted time.
Problem
A deployment stage that pushes a new version live to production is included in the range of stages a team routinely re-runs using the restart-from-stage feature whenever a later, unrelated stage fails.
Why It’s Harmful
Re-running the deployment stage triggers an actual second deployment, potentially duplicating side effects like sending release notifications twice, or, in more serious cases, causing inconsistent or conflicting states in production infrastructure.
Correct Approach
Design deployment and other side-effecting stages to be genuinely idempotent wherever possible, or explicitly exclude the deployment stage from the range that gets replayed, restarting only from a later, clearly re-runnable point instead.
Persisted state and sensitive data
Because the CPS engine serializes a pipeline’s variables to disk as part of its persistence mechanism, any sensitive value held in a plain variable during execution could, in principle, end up written into that state file. Jenkins’ credential-handling mechanisms are specifically designed to avoid this by masking and specially handling secret values, which is exactly why pipeline authors should always use proper credential bindings rather than storing secrets in ordinary variables.
Before enabling or relying on restart-from-stage for a given pipeline, explicitly review every stage that could be skipped past for hidden non-idempotent side effects — the risk is easy to overlook because the pipeline “just works” most of the time, until the one time it doesn’t.
Understanding which stages are safe to re-run connects directly to knowing what actually happened during a resumed or restarted build, which the monitoring chapter that follows addresses.
10Monitoring, Logging & Observability
Knowing exactly when and why a build resumed, or was restarted from a stage, matters for both debugging and trust.
Build Console Log Resume Marker
Jenkins typically notes in the console output when a build’s execution has resumed following a controller restart, making the event visible after the fact.
Restart-From-Stage Build Records
A build started via restart-from-stage is recorded as linked to the original run it restarted from, preserving traceability back to what was originally executed.
Controller System Logs
Jenkins’ own administrative logs record controller restarts themselves, useful for correlating an unusual build behavior with an infrastructure event that happened around the same time.
Disk Usage of Build Records
Tracking how much disk space accumulated build state files consume helps catch a growing storage problem before it becomes serious.
Why traceability matters especially here
Because a resumed or restarted build’s history can be more complex than a simple, single, uninterrupted run, clear traceability back to exactly what happened — which stages were newly executed versus reused from an earlier run — is especially valuable when investigating an unexpected outcome, such as a deployment appearing to have happened twice.
A well-annotated resume marker in a build’s log is like a clear editorial note in a manuscript that’s been revised in stages — it tells a later reader exactly which parts are original and which were added or redone, rather than leaving them to guess.
Good visibility into resume and restart events naturally leads into recognizing which structural patterns make a pipeline genuinely safe to resume, and which quietly create risk — the subject of the next chapter.
11Design Patterns & Anti-Patterns
Some pipeline structures embrace resumability safely; others quietly undermine it.
Idempotent Deployment Steps
Structuring a deployment so that applying it again produces the same end state rather than a duplicated or conflicting one, often by describing the desired final state rather than issuing a one-time imperative action.
Isolating Non-Repeatable Actions Into Their Own Stage
Keeping any genuinely one-time action — like sending a release announcement — in its own small, clearly labeled stage, making it easy to consciously exclude that specific stage from any restart-from-stage range.
Problem
A pipeline mixes build, test, notification, and deployment logic all together within a single large stage, rather than separating them into distinct, individually named stages.
Why It’s Harmful
There is no way to restart from a point that skips just the notification or deployment portion without also either repeating or entirely skipping the other logic bundled into that same large stage, removing the flexibility that restart-from-stage is meant to provide.
Correct Approach
Break a pipeline into smaller, clearly named stages along natural boundaries — particularly separating side-effecting actions from purely computational ones — so that restart-from-stage decisions can be made precisely.
These principles turn into a concrete, practical checklist in the next chapter.
12Best Practices & Common Mistakes
A short, practical checklist for anyone designing pipelines that need to survive interruption or restart cleanly.
Design Every Stage With Re-Run Safety in Mind
Ask, for each stage, what happens if this runs twice — and design accordingly, especially for anything touching real infrastructure.
Isolate One-Time Actions
Keep genuinely non-repeatable actions in their own small, clearly named stage rather than bundled with other logic.
Match Durability Settings to Pipeline Value
Use higher durability for long, high-value pipelines, and accept lighter durability for short, frequent, low-risk builds.
Use Milestones for Frequently Re-Triggered Jobs
Add milestone steps to pipelines that commonly get triggered again before a previous run finishes, so only the newest run proceeds.
Restarting a pipeline from a stage without first checking whether that stage or anything after it contains a non-idempotent action, discovering only after the fact that something real happened twice.
Assuming maximum durability settings are always the right default, without weighing the real, cumulative performance cost across a high volume of short, low-stakes builds.
With the checklist in place, it’s worth seeing how organizations actually rely on this capability in practice, which the next chapter explores.
13Real-World & Industry Examples
Resumability shows up most visibly wherever pipelines are long, expensive, or need to coexist with regular maintenance windows.
Overnight, Multi-Hour Build and Test Pipelines
Organizations running extensive overnight test suites rely on restart survival so a scheduled maintenance window overlapping with an in-progress run doesn’t force the entire multi-hour process to start over the next morning.
Large-Scale Infrastructure Rollouts
Pipelines that gradually roll a change out across many servers or regions, one at a time, use restart-from-stage to retry a failed later region without re-touching regions that already succeeded earlier in the same run.
High-Traffic Continuous Integration Systems
Busy CI systems handling many rapid, successive commits to the same branch commonly rely on milestone steps to ensure that only the build for the very latest commit continues past a certain point, letting older, now-outdated runs be safely abandoned.
Frequently Patched Jenkins Environments
Organizations that patch and restart their Jenkins controllers on a regular schedule for security reasons depend on durability settings tuned appropriately for their typical build lengths, so routine maintenance doesn’t become a recurring source of wasted build time.
Before closing out, it’s worth addressing the questions that come up most often once engineers start relying on resumability day to day.
14Frequently Asked Questions
No — automatic restart survival is a built-in behavior of the CPS engine that applies to ordinary pipeline scripts without any special code, though the controller’s durability setting determines how robust that survival is.
Behavior here depends on the step and the agent connection — some steps can reconnect and continue cleanly, while others may need to be re-run, which is exactly why designing steps to tolerate being re-run safely matters even for automatic restart survival, not just explicit restart-from-stage.
This capability is generally available for declarative pipelines meeting certain structural requirements; scripted pipelines and some more complex declarative structures may have more limited support, so it’s worth checking a specific Jenkins installation’s documentation for exact requirements.
No — a milestone step manages competition between multiple concurrent builds of the same job, ensuring only the newest one proceeds past a given point, while a checkpoint or restart-from-stage is about resuming a single build’s own progress after it finishes or fails.
The relative overhead is more noticeable on short, high-frequency pipelines than on long ones, simply because the fixed cost of persistence represents a larger fraction of a short build’s total runtime — which is exactly why many teams choose lighter durability specifically for their fastest, most frequent jobs.
No — restarting from a stage typically creates a new, separate build record that references the original run it restarted from, preserving both the original attempt and the new one for later inspection.
15Summary and Key Takeaways
Jenkins’ ability to remember exactly where a pipeline was — whether surviving an unplanned restart mid-build, or letting someone deliberately re-run only from a chosen later stage — turns interruptions and partial failures from costly full restarts into brief, manageable pauses. This is made possible by the CPS engine’s ability to serialize a pipeline’s exact execution state to disk, configurable durability settings that trade robustness against performance, and a separate, deliberate restart-from-stage capability for re-running completed or failed pipelines efficiently. The real discipline this capability demands isn’t technical complexity so much as careful design: treating every stage’s potential to be skipped, resumed, or re-run as a real possibility, and making sure nothing genuinely irreversible happens without deliberate, explicit protection.
Key Takeaways
- Two distinct mechanisms exist — automatic restart survival for a live, in-progress build, and explicit restart-from-stage for re-running a finished or failed one.
- The CPS engine makes it possible — pipeline scripts are transformed so their exact execution state can be serialized to disk and later reconstructed.
- Durability settings are a real trade-off — more frequent persistence means less risk of lost progress, at the cost of measurable overhead on every step.
- Idempotency is the central design concern — any stage that might be skipped past, resumed, or re-run needs to tolerate that safely, especially anything touching real infrastructure.
- Milestones solve a related but different problem — managing competition between concurrent builds of the same job, not resuming a single build’s own progress.
- Traceability matters for resumed and restarted builds — clear logs and linked build records make it possible to reason about exactly what ran and when.
- Good pipeline structure is the real enabler — small, clearly named stages, with side-effecting actions isolated from purely computational ones, make safe resuming genuinely possible.