Parallel Stages in Jenkins
A complete, plain-English guide to running pipeline work side by side: how parallel stages are written, how Jenkins really runs them, and how to make builds faster without making them fragile.
Imagine a kitchen preparing a birthday dinner. One cook chops vegetables, then boils the pasta, then bakes the cake, then sets the table. Everything happens one after another, and dinner is ready after three hours. Now imagine four cooks. One chops, one boils, one bakes, and one sets the table, all at the same moment. Dinner is ready when the slowest job finishes, perhaps after one hour. Nothing about the recipes changed. Only the way the work was arranged in time changed.
That is what parallel stages do for a Jenkins pipeline. A build that runs unit tests, integration tests, and code checks one after another may take forty minutes. If those three jobs do not depend on each other, they can run side by side and finish in the time of the slowest one. This guide explains the idea from zero, shows exactly how Jenkins handles it, and warns you about the traps that make parallel builds slower, flakier, or more expensive than the sequential ones they replaced.
1What Are Parallel Stages?
We start with the simplest picture: what a parallel stage is, and what it changes about a pipeline.
Jenkins is a tool that automates building, testing, and delivering software. You describe the work in a file called a Jenkinsfile. That file holds a pipeline, an ordered list of stages such as Build, Test, and Deploy. By default, stages run one after another, in the order you wrote them.
A parallel stage breaks that rule on purpose. Inside one stage, you place a parallel block that contains several smaller stages, often called branches. Jenkins starts all the branches at the same time, waits until every one has finished, and only then moves on to the next stage. Work that used to take the sum of all its parts now takes about as long as the longest part.
Think of a road that splits into three lanes for a short stretch and then merges again. Cars can travel in all three lanes at once, but nobody moves on until the road has merged. The split is the start of the parallel block. The merge is the moment Jenkins waits for every branch. Only the merged road continues to the next stage.
The two moments that matter
The fork
Jenkins reaches the parallel block and starts every branch at once, each on its own machine or on the same one, depending on how you set it up.
The join
Jenkins waits until all branches have finished. If any branch failed, the parent stage is marked as failed, and the build usually stops there.
Parallel stages are a Declarative Pipeline feature. Scripted Pipeline has its own parallel step that does a similar job in a different style. We compare the two in Chapter 10.
2Why Parallel Execution Exists
Parallel work is an old idea. Here is the problem it solves and how it came to Jenkins.
The problem: slow feedback
A developer saves a change and waits for Jenkins to say whether it is safe. If the answer takes an hour, the developer starts something else, forgets the details of the first change, and returns later to fix a problem that would have taken five minutes to fix while it was fresh. Long builds also queue up, so other developers wait behind them. Slow feedback quietly taxes every person on the team.
As test suites grow, pipelines grow with them. Unit tests, integration tests, browser tests, code style checks, security scans, and documentation builds all pile up. Running them one after another is the simplest arrangement, and also the slowest.
The idea: use more workers at once
Computers have many processor cores, and companies often own many build machines, yet a sequential pipeline uses only one thing at a time. Parallel execution puts the idle capacity to work. This idea is much older than Jenkins. Supercomputers, factory assembly lines, and databases all use it. In Jenkins, the parallel step appeared in the early Pipeline plugins around 2016, when Pipeline became a built-in feature of Jenkins 2.0. Declarative Pipeline followed in 2017, and support for parallel stages, and later for sequential stages inside each parallel branch, was added in subsequent releases.
Marking exam papers is faster when three teachers each mark one third of the pile than when one teacher marks all of it. The pile is the same size. The wall-clock time is much shorter, provided the teachers do not need each other’s answers to mark their own share.
The catch
Parallel is not free. More workers means more machines, more cost, and more chances for two jobs to step on each other’s toes. The rest of this guide is about getting the speed without the mess.
3Jenkins Basics You Need First
A few Jenkins words come up in every parallel discussion. Learn them once here.
Controller
The main Jenkins server. It reads your Jenkinsfile, plans the work, and keeps track of progress. It was formerly called the master.
Agent (node)
A worker machine, virtual machine, or container where the real commands run. The words agent and node mean nearly the same thing.
Executor
A slot on an agent that can run one piece of work at a time. An agent with four executors can run four things at once.
Workspace
The folder on the agent where the code for a run is placed and where commands do their work.
Label
A tag on an agent, such as “linux” or “windows”, so a stage can ask for the right kind of machine.
Queue
The waiting line for work that needs an executor. When all executors are busy, new work waits here.
Doing work versus waiting
Two kinds of tasks behave very differently when run in parallel. A compute-heavy task, such as compiling code, uses the processor hard. Two such tasks on the same machine compete for the same cores and may not go much faster together than one after the other. A waiting task, such as calling a remote service and waiting for the reply, uses almost no processor while it waits. Many of these can overlap happily on one machine. Knowing which kind you have decides how many branches make sense and where they should run.
An executor is not a processor core. It is a Jenkins slot. An agent with many executors and few cores can accept a lot of work, then run all of it slowly. Match executors to what the machine can really handle.
4Anatomy of a Parallel Stage
Now the shape of the block: where it goes, what it holds, and the rules Jenkins enforces.
The shape
In a Declarative Pipeline, a stage that wants to run work in parallel contains a parallel block instead of a list of steps. Inside that block sit the named branches, each one written as its own stage with its own steps. The stage that holds the block acts as a container. It is a folder with several drawers, and each drawer is worked on at the same time.
flowchart TD
A["Checkout code"] --> B["Build"]
B --> C["Unit tests"]
B --> D["Integration tests"]
B --> E["Static analysis"]
C --> F["Join point"]
D --> F
E --> F
F --> G["Package"]
The rules
A stage has one job: steps, stages, parallel, or matrix
A stage that contains a parallel block cannot also have its own plain list of steps. The container holds branches only.
Each branch has a name
The name appears on the build screen, so choose clear words such as “Unit Tests” rather than “Branch A”.
Each branch may have its own settings
A branch can name its own agent, environment values, conditions, options, and post actions, just like an ordinary stage.
All branches start together
Jenkins starts them at the same time, though a branch that needs an agent may have to wait for a free executor.
The next stage waits for all of them
The join happens when every branch is finished, whether by success, failure, or being skipped.
Nesting depth is limited. A branch inside a parallel block can hold a short sequence of stages, but you cannot keep placing parallel blocks inside parallel blocks forever. Exact limits depend on your Declarative Pipeline plugin version, so check the syntax help on your own server.
A relay race has runners in a fixed order. A team sprint is different: four runners start together and the team is timed by its last runner. A parallel block is the team sprint. The stage after it is the medal ceremony, which cannot begin until the last runner crosses the line.
5Agents and Workspaces in Parallel
This chapter covers the single most important practical detail: where each branch actually runs, and which folder it works in.
Two ways to arrange branches
If the pipeline names an agent at the very top, for example “any available agent”, Jenkins reserves one executor on one machine for the whole pipeline. Any parallel branch that does not name its own agent then runs inside that same machine and that same workspace. All the branches share one folder and one set of files.
If instead each branch names its own agent, Jenkins gives each branch its own executor. The branches may land on different machines, each with its own private folder. The top of the pipeline is then usually written to say “no agent”, so that no machine is held while the branches wait.
| Arrangement | Where branches run | Files | Risk |
|---|---|---|---|
| One agent at the top, none per branch | Same machine and executor | Shared folder | Branches can overwrite each other’s files |
| No agent at the top, one agent per branch | Separate executors, maybe separate machines | Private folders | Each branch must fetch its own code and inputs |
| Agent at the top, agent per branch | Top agent is held while branches run elsewhere | Mixed | An executor is wasted, and shortage can cause deadlock |
Three students working on one shared sheet of paper will smudge each other’s writing. Three students each with their own sheet can write freely, but each must copy the question onto their own sheet first. Shared folders are the shared sheet. Separate workspaces are the private sheets.
What happens with the same label
If two branches ask for the same kind of agent and Jenkins puts both on one machine, it does not give both the same folder. Jenkins creates an extra workspace folder for the second branch, commonly with an at-sign and a number added to the folder name. This keeps files apart, but it also means each branch starts with an empty folder and must fetch its own code.
The executor starvation trap
Imagine a build farm with four executors in total. A pipeline starts with an agent at the top, using one executor for the whole run. It then reaches a parallel block with four branches, each asking for its own agent. Only three executors remain free, so the fourth branch waits. If several such pipelines start together, each can hold one executor while waiting for others that other pipelines are holding. Nobody can finish. This is called starvation or, in the worst case, deadlock.
Put “no agent” at the top of the pipeline and ask for agents only in the stages and branches that truly need them. Executors are then held only while real work happens.
When branches share a workspace, two branches that write to the same build output folder, temporary file, or local database can corrupt each other’s results. The failures look random, because they depend on timing. Give each branch a private folder or a private agent.
6Failure Behaviour and failFast
What happens when one branch breaks while others are still running? Jenkins gives you a choice.
The default: let everyone finish
By default, when one parallel branch fails, the other branches keep running until they finish on their own. Only then does Jenkins mark the parent stage as failed and stop the pipeline. This is useful when you want a complete report. If three test suites run side by side and one fails, you still learn the results of the other two in the same run.
The alternative: stop the rest early
Inside the stage that holds the parallel block, you can add an option called failFast and set it to true. Now, as soon as one branch fails, Jenkins asks all the other branches to stop. They are shown as aborted. This saves time and machine cost, because there is little point in finishing three more branches when the whole stage is already doomed. There is also a pipeline-wide option, named parallelsAlwaysFailFast, which applies the same behaviour to every parallel block in the file.
Let everyone finish (default)
- You see every failure in one run.
- Better for test suites where each result is valuable.
- Fewer reruns, since one run tells the whole story.
Stop early with failFast
- Saves time and cost on runs that are already broken.
- Frees executors for other builds sooner.
- You only see the first failure, so you may fix one problem and hit another on the next run.
A group of hikers walking on different trails hears that one trail is closed by a landslide. A cautious guide calls everyone back at once. That is failFast. A different guide says, “Finish your walk, and report what you saw.” That is the default. Which is better depends on what you want to learn from the trip.
Failing softly with partial results
Sometimes a failing branch should not fail the whole build. A slow experimental check, for example, might be allowed to fail while the build is marked as unstable rather than failed. Jenkins offers a step called catchError that lets a branch report a problem but continue, and lets you choose the resulting build and stage status. Use this sparingly, because a check that is allowed to fail is soon a check that nobody reads.
A good rule is to use failFast on the pull request pipeline, where speed matters most, and to let everyone finish on the nightly pipeline, where a complete report matters most.
7Sequences, Conditions, and Post Actions Inside Branches
A branch is not just one command. It can be a small pipeline of its own.
Sequential stages inside a branch
A branch can hold a short list of stages that run one after another, while the branches themselves run side by side. Imagine a branch for Linux that compiles, then tests, then packages, and another branch for Windows that does the same. Each branch is a mini pipeline. The two mini pipelines run at the same time, and the join waits for both to complete.
flowchart LR
S["Start of parallel block"] --> L1["Linux compile"]
S --> W1["Windows compile"]
L1 --> L2["Linux test"]
L2 --> L3["Linux package"]
W1 --> W2["Windows test"]
W2 --> W3["Windows package"]
L3 --> J["Join"]
W3 --> J
What each branch can have of its own
Agent
A machine or container suited to that branch, such as a Windows machine for the Windows branch.
Environment
Named values used only inside that branch, such as a test database name that no other branch may touch.
When condition
A gate so that the branch runs only when needed. A slow browser test branch might run only on the main branch.
Options
Limits such as a time limit or retries, so one stuck branch cannot hold the join forever.
Post actions
Cleanup and reporting that run after that branch ends, whether it passed or failed, such as collecting its test report.
Tools
Specific tool versions, such as a particular Java version for one branch and another for a second branch.
Skipped branches
If a branch has a when condition and the condition is false, that branch is skipped, while the others run. The join treats a skipped branch as finished. Remember that a branch with its own agent will, by default, start that agent before checking the condition. Set the beforeAgent option to true inside the when block if you want the condition checked first, so no machine is started for a branch that will be skipped.
A school sports day has several events running at once on different fields. Each field has its own referee, its own equipment, and its own rules. Some events, like the swimming gala, only happen if the weather is warm. The head teacher waits until every event is finished before the awards ceremony.
Post actions inside a branch run when that branch ends. Post actions on the parent stage run when the whole parallel block ends. Keep branch-specific cleanup in the branch, and keep summary work, such as merging test reports, in the parent or a later stage.
8Sharing Data Between Branches and Stages
Parallel branches are like people working in separate rooms. To share results, they need a proper way to pass papers between rooms.
Why sharing is tricky
If branches run on different agents, they have different folders on different machines. Files written by one branch are simply not present for another. Even when they run on the same machine, using the same folder is dangerous, as Chapter 5 showed. And values held in variables inside one branch’s script code do not automatically appear in another branch.
The tools Jenkins gives you
| Tool | What it does | Best for | Limits |
|---|---|---|---|
stash and unstash | Saves files on the controller under a name, and restores them on any agent later | Passing compiled output from a build stage to test branches | Meant for small to medium file sets, and heavy use loads the controller |
archiveArtifacts | Stores files with the build so people and later stages can download them | Keeping final outputs such as packages and reports | Storage grows with every build unless cleaned up |
| External storage | Uses an artifact repository or cloud bucket outside Jenkins | Large build outputs and long-term storage | Needs credentials and network access |
| Environment variables | Passes short text values into each branch | Version numbers and target names | Text only, and set before the fork |
| Fresh checkout in each branch | Each branch fetches the code itself | Independent branches that only need source code | Repeated download time in every branch |
Before a group project, the leader photocopies the instructions and hands a copy to each team member, who then works in a separate room. Stash and unstash are the photocopier. When the work is done, each member puts their result in a shared box, which is the archive. Nobody works on the leader’s original page.
The classic fan-out pattern
Build once
A single stage compiles the code and stashes the compiled output.
Fan out
A parallel block starts several test branches, each on its own agent. Each branch restores the stash, so all test the same build.
Fan in
Each branch publishes its own test report. After the join, a later stage collects the results.
Release
A final stage packages and publishes only if the join was successful.
Avoid having several branches assign to the shared global environment object at the same time. They can overwrite each other in confusing ways. If each branch must produce a value, give each a unique name, such as one that includes the branch name.
9Matrix Stages and Dynamic Parallelism
Sometimes the same work must run for many combinations, or for a list that changes from run to run. Jenkins has tools for both.
The matrix directive
Suppose you want to test your program on three operating systems and two language versions. That is six combinations. Writing six branches by hand is tedious and easy to get wrong. The matrix directive lets you list axes, such as “platform” with three values and “version” with two values. Jenkins builds every combination for you, each as a parallel cell, and runs the same stages in each cell. Each cell can read its own axis values as environment variables. You can also list excludes to remove combinations that make no sense, such as a version that a platform does not support.
A pizza shop offers three sizes and four toppings. Instead of writing twelve separate recipes, the shop writes one recipe and a grid of sizes and toppings. The kitchen cooks every square in the grid. A matrix is that grid.
Dynamic parallel branches
When the number of branches is not known in advance, a fixed list is not enough. A monorepo might have twenty services today and twenty-five next month. For this, you use a script section that builds the set of branches while the pipeline runs, usually by looping over a list and creating one branch per item, and then calls the Scripted parallel step with that set. This is the most flexible approach, and also the one that needs the most care, since you are back in ordinary Groovy code.
| Approach | Best for | Strength | Weakness |
|---|---|---|---|
| Hand-written parallel branches | A few different jobs, such as tests, lint, and docs | Clear and readable | Repetitive when jobs look alike |
| Matrix | The same work over combinations of values | Concise and consistent | Combinations grow quickly, so cost can surprise you |
| Dynamic branches from a script section | A list that changes, such as changed services | Very flexible | Harder to read and to secure |
Matrix sizes multiply. Four platforms, five versions, and three databases makes sixty cells. Sixty parallel cells may need sixty executors, and can cost far more than the time saved is worth. Exclude combinations you do not truly support, and consider testing the full grid only on a nightly schedule.
Pattern: test only what changed
A team with many services in one repository builds a list of changed folders, then creates one parallel branch per changed service. A change to one service runs one branch, while a change to a shared library runs all of them.
10Declarative Parallel Versus the Scripted Parallel Step
Jenkins offers two ways to write parallel work. Knowing the difference helps you pick the right one.
The Declarative style uses a fixed layout: a stage containing a parallel block of named stages. It is readable, checked for mistakes before the run, and shown nicely on the build screen. The Scripted style uses a step named parallel that takes a map of names to blocks of code. It is flexible, since you can build the map with any logic you like, but it gives you fewer safety checks and more freedom to make errors.
| Feature | Declarative parallel stages | Scripted parallel step |
|---|---|---|
| Readability | High, fixed structure | Depends on the author |
| Checked before running | Yes, syntax is validated up front | Only when the code runs |
| Per-branch agent, environment, conditions, and post | Built in | You write them yourself in code |
| Fail-fast setting | Option in the stage, or pipeline-wide option | A setting passed along with the map |
| Number of branches known in advance | Needed | Can be decided at run time |
| Display of branches | Shown as stages on the build screen | Shown as branches, with less structure |
Declarative is a printed form with boxes to fill in. It is easy for everyone to read and hard to get wrong. Scripted is a blank sheet where you may write anything. It can express things the form cannot, but nobody can be sure you followed the rules.
Use Declarative parallel stages by default. Reach for a script section only for the small part that must be dynamic, and keep that part short.
11How Jenkins Runs Parallel Branches Internally
This chapter lifts the lid. Understanding what the controller and the agents each do explains almost every behaviour you will meet.
Two layers: planning and doing
A Jenkins pipeline has two layers of activity. The planning layer runs on the controller. It reads your Jenkinsfile, keeps track of which stage is active, and decides what starts next. The doing layer runs on the agents. It runs the real commands: compilers, test tools, and scripts.
When the planning layer reaches a parallel block, it creates one flow of control for each branch. These are not heavy operating system threads. The pipeline engine manages them cooperatively, switching between them as they wait or produce output. Each branch then asks the build queue for an executor, if it needs one. When an executor is free, the queue assigns it, and the real work begins on that agent. Because the real work runs on separate agents or executors, it truly happens at the same time.
sequenceDiagram
participant Ctl as Controller
participant Q as Build Queue
participant A1 as Agent One
participant A2 as Agent Two
Ctl->>Ctl: Reach the parallel block and create branches
Ctl->>Q: Request an executor for the Unit branch
Ctl->>Q: Request an executor for the Integration branch
Q->>A1: Assign the Unit branch
Q->>A2: Assign the Integration branch
A1-->>Ctl: Stream logs and status
A2-->>Ctl: Stream logs and status
A1-->>Ctl: Unit branch finished
A2-->>Ctl: Integration branch finished
Ctl->>Ctl: Join the branches and choose the next stage
A building site manager with a clipboard (the controller) sends three crews to three different rooms. The manager does not lay bricks. The manager tracks who has arrived, who has finished, and when the next phase can begin. The crews (the agents) do the physical work at the same moment.
What the controller still pays for
Even though the commands run on agents, the controller has work to do for every branch. It tracks state, receives the log output streaming back, and saves progress to disk so that the pipeline can survive a restart. Each extra branch adds a little load. With a handful of branches this is invisible. With hundreds of branches in many busy pipelines, the controller can become the slowest part of the system. Many teams therefore keep branch counts modest and put the heavy lifting inside the agents.
The role of the queue
If a branch asks for an agent and none is free, it waits in the queue. The build screen shows a message that it is waiting for the next available executor. The join cannot happen until that branch eventually runs and finishes. This means that a parallel block is only as parallel as your available capacity allows. Ask for eight branches on a farm with three free executors, and Jenkins will run three at first and the rest as executors free up.
Saved state and restarts
Jenkins saves the progress of a running pipeline as it goes. If the controller restarts in the middle of a run, the pipeline can resume. Long-running shell commands on agents are designed to keep going during a short controller restart, and the pipeline reconnects to them afterwards. There are limits, and a restart at a bad moment can still cost you a branch, so treat this as a safety net and not a promise.
Log lines from different branches arrive interleaved on the main console, each marked with its branch name. Modern Jenkins views also let you open the log of one branch at a time, which is far easier to read.
12Performance, Scalability, and Cost
Parallel work should make builds faster. Here is how much faster, what limits the gain, and what it costs.
The slowest branch decides
The time for a parallel block is roughly the time of its longest branch, plus overhead. Overhead includes waiting for an agent, fetching the code, restoring stashed files, and starting tools. Suppose a pipeline has a five-minute build, three test suites of ten, eight, and twelve minutes, and a three-minute packaging step. In sequence, that is 5 + 10 + 8 + 12 + 3, or thirty-eight minutes. In parallel, it is 5 + 12 + 3, or twenty minutes. The speed-up is real, but it is less than three times, because the build and packaging steps are still sequential.
This effect has a famous name, Amdahl’s law: the parts that cannot be run in parallel put a ceiling on the total speed-up, no matter how many workers you add.
A wedding cake needs three tiers baked at once in three ovens, but the icing can only start after all the tiers are done, and only one person can ice it. Adding a fourth oven does not help the icing step at all. The unshareable part sets the limit.
Balancing the branches
In the example, the twelve-minute branch is the bottleneck. If the work were split more evenly, say ten minutes each, the parallel part would take ten minutes instead of twelve. Splitting one large test suite into several similar-sized shards, sometimes based on the timing data from earlier runs, is one of the most effective ways to speed up a parallel pipeline.
What it costs
Machine hours
Three machines for twelve minutes is more machine time than one machine for thirty, due to overhead. You trade money for speed.
Startup overhead
Each branch may fetch code, restore files, and warm up tools. A one-minute setup in five branches is five minutes of machine time.
Queue pressure
Wide parallel blocks can occupy many executors at once, making other teams’ builds wait.
Controller load
More branches mean more logging and state for the controller to manage.
Scaling out
To scale, teams often use elastic agents: machines or containers that are created when needed and removed afterwards. A Kubernetes-based setup, for example, can start a fresh pod for each branch. This keeps the fleet small when idle and large when busy, and gives each branch a clean environment. The trade-off is startup time for each new pod, so it pays off most for branches that run for several minutes.
Measure before and after. Record the total build time and the machine time for a week before you parallelise, and again for a week afterwards. Numbers settle arguments faster than opinions.
13Reliability and Handling Flaky Work
Parallel builds fail in new and creative ways. This chapter shows how to keep them dependable.
Shared resources: the hidden enemy
Two branches that use the same test database, the same network port, the same temporary folder, or the same external account can interfere. A test that passes alone may fail when another branch is running. These failures are hard to reproduce because they depend on timing. The cure is isolation: give each branch its own database name, its own port, its own folder, or its own container.
Two people trying to use one telephone line at the same time will cut each other off. Giving each person their own phone solves it. Parallel branches that each need “the phone” should each get their own.
When isolation is not possible
Sometimes there is exactly one of something, such as a shared test environment or a licence. Jenkins has plugins that provide locks on named resources, so that only one branch at a time may use them while the others wait. Locks are useful, but they turn parallel work back into sequential work for that resource. Use them only for what really cannot be duplicated.
Tools for steadier branches
| Tool | What it protects against | Caution |
|---|---|---|
| Timeout option | A stuck branch holding the join forever | Set it generously above normal run time |
| Retry option | Occasional infrastructure hiccups, such as a network blip | Retrying hides real flakiness, so track how often it happens |
| Fail-fast option | Wasting resources after a certain failure | You see only the first problem |
| Unique names per branch | Collisions on databases, ports, and folders | Clean up afterwards to avoid leftovers |
| Locks on shared resources | Two branches using one scarce resource | Reduces parallelism |
| Post cleanup per branch | Leftover files or containers | Must run even on failure |
Availability of agents
If one agent goes offline during a run, the branch running on it fails. Having spare capacity, using elastic agents, and keeping branches short all reduce the chance that a single machine problem ruins a whole build. Keep branches small enough that a rerun of one branch is cheap.
A retry that silently makes a flaky test pass teaches the team to ignore flakiness. Track how many branches needed a retry each week. A rising number is a warning that the pipeline is slowly rotting.
14Security Considerations
More branches and more agents mean more places where something can go wrong. These are the security ideas to keep in mind.
Secrets in every branch
If a credential is made available at the top, every branch receives it, even branches that do not need it. Bind secrets only inside the branch that uses them.
Leftovers on shared agents
Files, tokens, and caches from one job can remain on an agent and be read by the next. Prefer fresh agents for sensitive work, and clean the workspace afterwards.
Untrusted pull requests
A stranger who can edit the Jenkinsfile can add branches that steal secrets or use up your machines. Limit who can change the pipeline for builds that can reach secrets.
Runaway cost
A change that adds a huge matrix can consume your build capacity or cloud budget. Review pipeline changes like code, and set limits on the number of agents.
Unsafe dynamic names
If branch names are built from outside input, such as file names or branch labels, clean them first so odd characters cannot cause trouble.
Interleaved logs
Secrets printed by one branch appear in the shared log. Masking helps, but the safest habit is never to print secrets.
Habits that protect you
- Least privilege. Give each branch only the credentials and permissions it needs.
- Ephemeral agents. Use containers or machines that are destroyed after the job.
- Review pipeline changes. A change that adds parallel branches deserves the same care as a code change.
- Limit capacity per team. Quotas keep one pipeline from starving all others.
- Keep Jenkins updated. Security advisories for Jenkins and its plugins are published regularly.
A school trip gives each supervisor only the keys they need for their own group. Handing the master key to everyone because “it is simpler” turns a single lost key into a school-wide problem. Branch-level credentials follow the same idea.
15Monitoring, Logging, and Troubleshooting
When a parallel build is slow, stuck, or strange, here is how to find the cause calmly.
What Jenkins shows you
- The pipeline graph. Branches are drawn side by side, so you can see which one is still running, which failed, and which is waiting.
- Per-branch logs. You can open the log for one branch, which avoids the confusion of interleaved lines.
- The build queue. Shows branches waiting for an executor and why.
- Timing data. Stage durations reveal the slowest branch, which is the one to optimise.
flowchart TD
A["A parallel build is slow or broken"] --> B{"Is a branch waiting for an executor"}
B -->|Yes| C["Check labels and free executors, or add capacity"]
B -->|No| D{"Does the failing branch pass when run alone"}
D -->|Yes| E["Look for shared files, ports, or databases"]
D -->|No| F{"Was the branch aborted by another failure"}
F -->|Yes| G["Check failFast and find the first real failure"]
F -->|No| H{"Is one branch much slower than the rest"}
H -->|Yes| I["Split that branch into smaller shards"]
H -->|No| J["Check agent startup time and stash or checkout overhead"]
Common symptoms and causes
| Symptom | Likely cause | What to do |
|---|---|---|
| Branch waits a long time to start | Not enough executors with the right label | Add capacity, or use elastic agents |
| Tests pass alone but fail together | Shared database, port, or folder | Isolate resources per branch |
| Branch shows as aborted | Another branch failed while fail-fast was on | Find the first failed branch |
| Pipeline hangs with no progress | Executor deadlock from an agent held at the top | Use no agent at the top and per-branch agents |
| Files missing in a branch | Branch runs on a different agent and nothing was stashed | Stash and unstash, or check out again |
| Speed-up is small | One branch dominates, or overhead is large | Balance the branches and cut setup time |
| Test reports are incomplete | Each branch published to the same file name | Use unique report names and merge after the join |
Signals worth tracking over time
Track queue wait time per branch, since it shows whether you have enough capacity. Track the duration of the slowest branch each week. Track how often branches are retried or aborted. And track total machine time per build, so that a drop in wall-clock time is not bought with an unnoticed rise in cost.
16Design Patterns and Anti-patterns
Patterns are habits that keep parallel pipelines calm. Anti-patterns are habits that keep causing puzzles at two in the morning.
Patterns that help
Build once, test many
Compile one time, stash the output, and fan out several test branches that all use the same build.
No agent at the top
Ask for agents only where work happens, so no executor sits idle while branches wait.
Sharded tests
Split a large suite into similar-sized pieces so no branch is a bottleneck.
Fast lane, slow lane
Run cheap checks with fail-fast on pull requests, and the full grid on a nightly schedule.
Isolated by design
Give each branch its own container, database name, and port range, so branches never fight.
Merge results after the join
Each branch publishes its own report. A later stage merges and summarises them.
Anti-patterns to avoid
Context
A pipeline names an agent at the top and then runs several branches with no agents of their own.
What goes wrong
All branches share one folder on one machine. They overwrite each other’s output, and results become random. The team blames flaky tests, when the real cause is shared files.
Better decision
Give each branch its own agent or its own workspace, and use no agent at the top.
Context
A team parallelises everything it can, with dozens of tiny branches, to make the diagram look impressive.
What goes wrong
Setup overhead in each tiny branch is larger than the work itself. Total time barely improves, machine cost rises, and the controller struggles with the log volume.
Better decision
Parallelise the few branches that are long enough to be worth the overhead, and measure the result.
Context
Branches are made to depend on each other, for example a branch that waits for a file another branch will create.
What goes wrong
The branches are no longer independent. Timing decides whether the file exists, and the pipeline deadlocks or fails at random.
Better decision
If one piece of work needs another’s output, put them in sequence. Only run independent work in parallel.
17Advantages, Trade-offs, Best Practices, and Common Mistakes
A balanced view of parallel stages, and the habits that keep them working for you.
Advantages
- Shorter waiting time for developers.
- Better use of idle machines and executors.
- Each branch can have its own agent, environment, and conditions.
- Clear side-by-side view of results on the build screen.
- Matrix support removes repetitive copy and paste.
- Fail-fast saves resources on broken runs.
Disadvantages and trade-offs
- Higher machine cost and more overhead.
- Shared resources cause timing-dependent failures.
- More load on the controller and the queue.
- Harder to read interleaved logs.
- Speed-up limited by the sequential parts and the slowest branch.
- More moving parts to secure and monitor.
Best practices in one place
- Only parallelise independent work. If one piece needs another’s output, keep them in sequence.
- Use no agent at the top. Request agents where work actually happens.
- Give each branch its own workspace or agent, and its own ports, databases, and folders.
- Balance the branches. The slowest branch sets the time, so split it into shards.
- Set timeouts so a stuck branch cannot hold the join forever.
- Choose failFast on purpose. Use it where speed matters, and skip it where a full report matters.
- Stash only what is needed, and archive or upload large outputs instead.
- Bind secrets inside the branch that uses them.
- Name branches clearly so the build screen tells a story.
- Measure time and cost before and after each change.
The mistakes seen most often
Mistake 1: shared folder, shared trouble
A team names an agent at the top and forgets that branches without their own agent share its workspace. Two test suites write to the same output folder and produce corrupted reports. The fix is per-branch agents or private folders.
Mistake 2: the held executor
The pipeline holds an executor at the top and requests more for its branches. Under load, the pipelines wait on each other. The fix is no agent at the top.
Mistake 3: matrix explosion
A matrix of three axes produces sixty cells, and the build farm is swamped. The fix is to exclude unsupported combinations and run the full grid only at night.
Mistake 4: expecting the shell to share variables
One branch sets a variable in a shell command, and another expects to see it. Each shell has its own copy, so the value never arrives. The fix is to pass values through stash, files, or values set before the fork.
18Worked Scenario: Speeding Up a Slow Pipeline
Let us take one imaginary pipeline and improve it in steps, watching the time fall.
The starting point
A small team runs an online shop. Their pipeline has these stages in order: Checkout (1 minute), Build (5 minutes), Unit Tests (10 minutes), Integration Tests (8 minutes), Browser Tests (12 minutes), Static Analysis (4 minutes), and Package (3 minutes). That totals 43 minutes for every change.
Find the independent work
Unit Tests, Integration Tests, Browser Tests, and Static Analysis all need only the built output. None needs another’s results. They can run side by side.
Build once and stash
The Build stage stashes its output. Each test branch restores the stash on its own agent, so no branch depends on another’s folder.
Fan out the four branches
The time is now 1 + 5 + 12 + 3 = 21 minutes, plus roughly two minutes of overhead per branch for agent start and restore, so about 23 minutes.
Balance the slowest branch
The team splits Browser Tests into two shards of six minutes each. The longest branch is now Unit Tests at ten minutes, so the parallel block takes about twelve including overhead. Total is about 21 minutes.
Add fail-fast on pull requests only
On pull requests, if Static Analysis fails after four minutes, the other branches stop at once, saving machine time and freeing executors.
| Version | Approx. total time | Machines busy at once | Comment |
|---|---|---|---|
| Fully sequential | 43 minutes | 1 | Simple, slow |
| Four parallel branches | About 23 minutes | Up to 4 | Big win for little effort |
| Five branches with sharded browser tests | About 21 minutes | Up to 5 | Diminishing returns |
| Same, with fail-fast on pull requests | Shorter on failing runs | Up to 5 | Saves cost on broken changes |
Moving house with four friends is much faster than moving alone. Moving with twenty friends is only slightly faster, because the lift and the front door become the bottleneck. Notice the same shape in the table: the first few branches give the big win, and later ones give less.
The numbers here are made up to show the pattern. Use your own timing data. Jenkins shows stage durations, and a few days of history is enough to find the bottleneck.
19Real-World Usage Patterns
Here is how parallel stages tend to appear in real delivery pipelines. These are illustrative patterns in general terms, not claims about the private setups of any particular company.
Jenkins is one of the most widely used automation servers, and large engineering organisations have publicly described running it at scale, often with fleets of elastic agents. What tends to be common is a habit of splitting long checks into independent pieces, isolating each piece, and measuring the result. The patterns below show that habit in practice.
Pattern: cross-platform builds
A desktop software vendor builds and tests its product for Windows, macOS, and Linux at the same time. Each branch runs on a matching agent and produces its own installer, and a final stage gathers all three.
Pattern: sharded test suites
A team with a large browser test suite splits it into a set of equal shards using timing data from previous runs. Each shard runs on its own container, and the whole suite finishes in the time of one shard.
Pattern: independent microservices
A company with many services in one repository builds the list of changed services, and creates one parallel branch for each. A small change runs quickly, while a change to a shared library runs everything.
Pattern: multi-version compatibility
A library maintainer tests against several language versions with a matrix. The full grid runs nightly, while pull requests test only the oldest and newest versions.
Pattern: quality gates alongside tests
Security scans, style checks, and license checks run in their own branches next to the tests. The join waits for all of them, and the release stage runs only if every branch passed.
Parallel stages are often paired with Docker and Kubernetes, which can give every branch a clean, disposable environment. This pairing solves many of the shared-resource problems described in this guide.
20Frequently Asked Questions
failFast, the other branches are stopped as soon as one fails.parallel step. It is flexible, but harder to read and to secure.when block. Set beforeAgent to true in it if you do not want an agent to start for a branch that will be skipped.21Summary and Key Takeaways
Key Takeaways
- Parallel stages run independent work side by side. Jenkins forks at the start of the block and joins when every branch has finished.
- The slowest branch decides the time. Sequential parts and startup overhead limit the speed-up, so balance and shard your branches.
- Agents and workspaces matter most. Use no agent at the top, and give each branch its own agent or folder so branches never overwrite each other.
- Choose failure behaviour deliberately. Use
failFastwhere speed matters, and let everyone finish where a complete report matters. - Share data through proper channels. Use stash, archives, or external storage, not shared folders or shell variables.
- Isolate shared resources. Give each branch its own ports, databases, and names, and use locks only for what cannot be duplicated.
- Measure both time and cost. Watch queue waits, slowest branch, retries, and total machine time, and secure each branch with least-privilege credentials.