Executors in Jenkins
A complete, plain-English guide to the slots that decide how much work Jenkins can do at once: what executors are, how the queue hands them out, how many you really need, and how to stop builds from waiting in line for no good reason.
Picture a busy bank with ten customers standing in line and three tellers behind the counter. Each teller serves one customer at a time. It does not matter how many customers arrive: only three can be served at the same moment, and everyone else waits. If the tellers are slow, or if one teller goes to lunch, the line grows. If the bank hires two more tellers, the line shrinks. Nothing about the customers changed. Only the number of tellers, and how well they were used, changed.
In Jenkins, the tellers are called executors. The customers are your builds. Almost every complaint of the form “my build has been waiting for twenty minutes” is really a story about executors: too few of them, the wrong kind of them, or the right ones held by someone who is not using them. This guide teaches the topic from zero. By the end you will know what an executor is, how Jenkins decides who gets one, how to choose how many you need, and how to keep your pipelines from quietly hogging them.
1What Is an Executor?
We begin with the simplest possible picture: what an executor is, where it lives, and what it does.
Jenkins is a tool that automates building, testing, and delivering software. The actual work happens on machines called nodes or agents. Each node offers a certain number of executors. An executor is a slot on a node that can run one piece of work at a time. When a build needs to run its commands, it must first get hold of an executor. When the build finishes, the executor is released and can serve the next build.
So the number of executors on all your nodes added together is the number of builds Jenkins can run at the same moment. Ten executors means ten simultaneous builds, or ten simultaneous parts of builds. The eleventh has to wait.
Think of a restaurant kitchen with several cooking stations. Each station can cook one dish at a time. A station is an executor. The kitchen is a node. The waiters bringing orders are the build triggers. A restaurant with one kitchen and four stations can cook four dishes at once, no matter how many orders come in. More stations means more dishes at once, but also a more crowded kitchen.
What an executor is not
It is not a processor core
An executor is a Jenkins slot. A machine with two cores can be configured with ten executors. Whether that is wise is a different question.
It is not a machine
One machine can offer many executors, and one agent can offer just one. The machine is the node, and the slots on it are executors.
It is not a build
A build is a run of a job. An executor is the seat the build sits in while it runs. Seats are permanent, and builds come and go.
It is not always needed
Some pipeline work, such as the planning done by the controller, uses a special lightweight slot that does not count against your executors. Chapter 7 explains this.
You will often see the Jenkins sidebar show a panel called “Build Executor Status”. It lists each node and its executors, marking which are idle and which are busy. It is the quickest way to see the current state of your capacity.
2Why Executors Exist
To understand why Jenkins counts slots at all, imagine the world without them.
The problem: unlimited work on limited machines
A build server is a real computer with a limited amount of processor power, memory, and disk. If Jenkins started every requested build the instant it arrived, then a busy morning with fifty commits would launch fifty compilations at once. The machine would slow to a crawl, run out of memory, and possibly crash. Every build would take longer, and some would fail for reasons that have nothing to do with the code.
Executors are the brake. They put a firm cap on how much work each machine will accept at once. Work beyond the cap is not refused. It waits in a line, called the build queue, and starts as soon as a slot frees up.
A short history
Jenkins began in 2011 as a fork of a project called Hudson, which had been written years earlier by Kohsuke Kawaguchi. From the start, the design had one main server and optional worker machines, with a configurable number of executors on each. The main server used to be called the “master” and workers were called “slaves”. Modern Jenkins uses the words controller and agent instead. The core idea of the executor has stayed steady through all these years, even as pipelines, containers, and cloud agents changed how Jenkins is used.
A car park has a fixed number of spaces. The barrier at the gate lets a new car in only when a space is free. Cars that arrive when the park is full wait outside in a line. Without the barrier, the park would jam and nobody could move. Executors are the spaces, the queue is the line outside, and Jenkins is the barrier.
3Jenkins Basics You Need First
A few Jenkins words come up in every executor discussion. Learn them once here.
Controller
The main Jenkins server. It stores job settings and credentials, keeps the queue, and decides which build goes where. The Jenkins web pages come from it.
Node and agent
A machine, virtual machine, or container that can run builds. The words node and agent mean nearly the same thing. The controller itself also appears as a node, often named “Built-in Node”.
Job and build
A job is the saved description of work. A build is one run of that job, with its own number and result.
Label
A tag on a node, such as “linux” or “gpu”, used to ask for the right kind of machine.
Workspace
The folder on a node where the code for a build is placed and where its commands work.
Pipeline and stage
A pipeline is a build described in a Jenkinsfile as an ordered list of stages, such as Build, Test, and Deploy.
Two kinds of waiting
People often say “my build is waiting” without saying what for. There are two very different cases. In the first, a build is waiting for an executor because everything is busy. In the second, a build has an executor and is waiting for something else, such as a person to click “approve”, a server to answer, or a timer. The first case is a shortage of capacity. The second case is a waste of it, because the waiting build is sitting on a slot that another build could have used. Keeping these two apart will save you hours of confusion.
When someone says “Jenkins is slow”, ask a single question first: is the build waiting for a slot, or is it running slowly on a slot? The fixes are completely different.
4Anatomy of a Node and Its Executors
Every node has a small set of settings that decide how its executors behave. Here they are, one by one.
| Setting | What it means | Why it matters |
|---|---|---|
| Number of executors | How many builds this node may run at once | The main capacity control for the node |
| Labels | Tags describing the node, separated by spaces | Lets jobs ask for the right machine |
| Usage | Use this node as much as possible, or only for jobs whose label matches | Keeps special machines free for special work |
| Remote root directory | The folder on the node where Jenkins keeps its files and workspaces | Needs enough disk space for all executors together |
| Launch method | How the controller connects to the agent | Decides how reliably the node stays online |
| Availability | Keep online always, bring online on demand, or follow a schedule | Controls cost and readiness |
| Node properties | Extra environment variables and tool locations for that node | Gives builds machine-specific settings |
The number of executors
This is the setting most people think of first. Set it to one, and the node runs one build at a time. Set it to four, and up to four builds can run at once on that node, each in its own workspace folder. Later chapters explain how to choose the number. For now, remember that each executor is a promise the node makes to Jenkins: “I can handle this much work at once.” If the promise is too big, builds compete for the machine’s real resources and everything slows down.
Labels and usage
A label is how a job says which kind of machine it needs. A node with the label “linux docker” can serve jobs that ask for linux, or for docker, or for both. The usage setting controls whether a node accepts any job, or only jobs that name one of its labels. Setting a special machine, such as one with an expensive graphics card, to accept only matching jobs stops ordinary builds from wasting its time.
A hospital has general wards and a special heart unit. The heart unit has a limited number of beds and special equipment. Patients only go there if their case needs it, otherwise the beds would be full of people who could have used any ward. Labels and the usage setting are how Jenkins keeps the special unit for special cases.
The remote root directory must hold the workspaces of all executors at once. Ten executors, each checking out a five gigabyte repository, need at least fifty gigabytes. Running out of disk is a common reason a node goes offline in the middle of a busy day.
5Executor States and Lifecycle
An executor is always in one of a few simple states. Learning them helps you read the Jenkins screens at a glance.
Idle
The executor is free and waiting. It shows as “Idle” in the sidebar. This is the state you want to see when the queue has work, because it means Jenkins can start something immediately.
Assigned
The scheduler has matched a queued item to this executor. There is a brief hand-off moment while the executor prepares to run it.
Busy
The executor is running a build, or a block of a pipeline. The sidebar shows the job name, the build number, and a progress bar.
Released
The work has finished, whether by success, failure, or being aborted. The executor cleans up and returns to idle. The next queued item can use it.
Offline
The whole node is unavailable, so none of its executors can run work. Reasons include a lost network connection, an administrator taking the node offline, or Jenkins automatically taking it offline because of low disk space.
A hotel room can be empty and ready, booked and about to be occupied, occupied by a guest, being cleaned, or closed for repairs. Executor states follow the same cycle. A closed room, like an offline node, is useless to every guest in the queue.
Temporarily offline versus disconnected
A node can be offline in two quite different ways. Temporarily offline means an administrator deliberately marked it so, usually for maintenance, and the connection remains. Running builds finish, but no new ones start there. Disconnected means the connection to the machine is lost, whether by a network problem, a restart of the machine, or a crashed agent process. Builds running on it at that moment usually fail. Knowing which you are looking at tells you whether to wait or to investigate.
Before restarting or updating a node, mark it temporarily offline and wait for its busy executors to finish. This drains the node gracefully, and no build is cut off in the middle.
6The Build Queue
Between “a build was requested” and “a build is running” lies the queue. Understanding its stages explains most waiting messages.
When something triggers a build, such as a commit, a timer, a person clicking a button, or another job finishing, Jenkins does not start it at once. It places an item in the build queue. The item then passes through a few stages before it reaches an executor.
flowchart LR
A["Trigger arrives"] --> B["Waiting during the quiet period"]
B --> C{"Blocked by another rule"}
C -->|Yes| D["Blocked"]
D --> C
C -->|No| E["Buildable"]
E --> F{"Free executor with a matching label"}
F -->|No| E
F -->|Yes| G["Pending hand-off"]
G --> H["Running on the executor"]
H --> I["Finished and executor freed"]
| Queue state | Meaning | Typical message |
|---|---|---|
| Waiting | The item is in its quiet period, a short delay that lets several quick commits merge into one build | In the quiet period, with a countdown |
| Blocked | Something prevents it from starting, such as another build of the same job that must finish first | Waiting for a previous build, or blocked by another job |
| Buildable | It is ready to run and only needs an executor | Waiting for the next available executor |
| Pending | An executor has been chosen and the hand-off is in progress | Very brief |
The quiet period
Jenkins waits a few seconds, five by default, before releasing a new item from the queue. If another trigger for the same job arrives during that time, the two are merged into one build. This prevents a burst of commits from producing a burst of builds. A longer quiet period gives more merging, at the cost of slower feedback.
Reading the waiting message
When a build sits in the queue, Jenkins gives a reason. The most common message says it is waiting for the next available executor on a named node or label. If the message names a label that no online node has, the build will wait forever, because no executor anywhere is able to take it. This is not a shortage of capacity. It is a mismatch, and adding more executors elsewhere will not help.
A restaurant host takes your name and asks you to wait a moment while they see if two nearby parties can share a table. That short pause is the quiet period. If the only free table is a booth and you asked for a window seat that does not exist, no amount of waiting will seat you. That is a label mismatch.
Jenkins also lets you cancel items in the queue, and shows how long each has been waiting. A queue that grows all day and never empties is the classic sign that demand exceeds the executors you have.
7Pipelines and Lightweight Executors
Not every part of a pipeline needs a full executor. Knowing which parts do is the key to using your slots wisely.
Two kinds of work in a pipeline
A pipeline run has two layers. The first is the planning layer, running on the controller. It reads the Jenkinsfile, remembers where the build has got to, and decides what happens next. The second is the doing layer, running on agents. It runs your real commands, such as compilers, tests, and scripts.
The planning layer runs in a special lightweight way, often called a flyweight executor. It does not take one of your configured slots. It is cheap, because it mostly waits and makes small decisions. On the controller’s executor list it may appear as a small lightweight entry. This is why a pipeline that is doing nothing but waiting for approval does not, by itself, use up one of your real executors.
The doing layer is different. Whenever a pipeline reaches a step that must run commands on a machine, it needs a real executor on a node. In Declarative syntax this happens when a stage or the whole pipeline names an agent. In Scripted syntax it happens inside a node block. The executor is held from the start of that block to its end.
| Pipeline activity | Uses a real executor? | Comment |
|---|---|---|
| Planning and deciding what runs next | No, lightweight slot on the controller | Cheap, does not count against your configured executors |
| Stage with an agent | Yes, for the whole stage | Held until the stage ends |
| Whole pipeline with an agent at the top | Yes, for the entire run | Held even during stages that do nothing on the machine |
| Running a shell command | Yes, it must run inside an agent block | Cannot run without one |
| Waiting for human approval inside an agent block | Yes, still held while waiting | A big source of waste |
| Waiting for approval outside any agent block | No | Recommended |
| Each branch of a parallel block with its own agent | Yes, one per branch | Branches wait if there are too few slots |
At a hospital, a coordinator sits at a desk deciding which patient goes to which room. The coordinator does not occupy a treatment room. A patient waiting for test results in a treatment room, however, blocks that room for the next patient. The coordinator is the flyweight. The treatment room is the real executor. Do your waiting in the waiting room, not the treatment room.
An approval step, a long sleep, or a wait for an outside system placed inside an agent block keeps the executor tied up doing nothing. If ten pipelines all wait for approval at the same moment, ten executors are idle in name but unavailable in practice. Move waiting out of agent blocks.
8Asking for Executors in a Jenkinsfile
The Jenkinsfile is where you tell Jenkins what kind of slot you need, and for how long.
The agent directive
In Declarative Pipeline, the agent section says where the work runs. It has a few common forms.
Any
Take any available executor. Simple, but gives you no control over the kind of machine.
Label
Take an executor on a node with the given label, such as one that has the right operating system or tools.
None
Do not take an executor at this level. Each stage below must then name its own agent, if it needs one.
Container or pod
Start a container or a Kubernetes pod for the work, which is an elastic executor that disappears afterwards.
Agent at the top or in each stage?
If you write an agent at the very top, Jenkins reserves one executor for the entire pipeline run. It is held through every stage, including the ones that only wait. If instead you write “none” at the top and name agents inside the stages that need them, an executor is held only while each stage runs. The second way is almost always more economical on a shared system. Its cost is that each stage may get a different machine and a fresh workspace, so files must be passed along with stash and unstash, or by fetching the code again.
| Style | Executor held | Workspace | Best when |
|---|---|---|---|
| Agent at the top | Whole run | Same folder for all stages | Short pipelines where every stage needs the same files |
| None at the top, agent per stage | Only during each stage | Each stage gets its own | Longer pipelines, or pipelines with waiting, approvals, or parallel work |
Renting a workshop for a whole week when you only need it on Tuesday and Friday means it stands empty for five days while other people who need it are turned away. Renting it by the day costs a little more effort each time you set up, but frees it for others when you are not using it.
Label expressions
A label can be more than a single word. You can combine labels with “and”, “or”, and “not” to describe exactly what you need. A job could ask for “linux and docker”, or for “windows or macos”, or for “linux and not gpu”. Jenkins matches the expression against every node’s labels, and only nodes that satisfy it are candidates. A well-chosen label scheme is like a clear address system, and a messy one leads to confusing waits.
Approval without holding a slot
A common design for release pipelines is to put the approval in its own stage, with no agent. The pipeline plans on the lightweight controller, asks a human to approve, and only after approval starts a stage that names an agent. Nobody waits with a slot in their pocket.
Use the same small set of labels across the whole company, and document them. Labels such as “linux”, “windows”, “docker”, and “large” are easy to remember and easy to search for when a build is stuck.
9The Controller and Its Executors
The controller is also a node, and it can have executors. Whether it should is one of the most important decisions in a Jenkins setup.
The controller is the brain of Jenkins. It holds job definitions, credentials, keys, and the history of every build. It also appears in the node list under a name such as “Built-in Node”, and it can be given executors like any other node. Older installations often ran builds on it by default.
Why most teams set it to zero
Security
A build running on the controller runs on the same machine, and often as the same operating system user, as Jenkins itself. Code in that build could read secrets and settings that belong to the whole Jenkins installation.
Stability
A heavy build can use up processor time and memory, slowing every web page and every other job. If it crashes the machine, the whole Jenkins goes down with it.
Clarity
With zero executors on the controller, its job is clear: plan, schedule, and display. All real work runs on agents that you can replace easily.
Scalability
Adding capacity means adding agents, which is easier than growing one big machine.
The manager of a bank sits in an office with the keys to the vault. A good manager does not also work as a teller at the counter, where customers, noise, and mistakes can reach the vault. Tellers work at the counter. The manager manages. Setting the controller to zero executors is the same separation.
The small exceptions
A tiny personal Jenkins on a laptop, used only for experiments, may reasonably run builds on the controller. A test installation used to learn Jenkins does the same. The advice is aimed at any Jenkins that holds real secrets, serves a team, or runs untrusted code. Jenkins also displays warnings when the controller has executors, and current documentation recommends setting the number to zero for such installations. The exact wording of the screen varies between versions.
Before you set the controller’s executors to zero, make sure at least one working agent exists with the labels your jobs need. Otherwise every build will sit in the queue with nowhere to run.
10How Executors Work Internally
This chapter lifts the lid. Seeing the mechanism makes many puzzling behaviours easy to explain.
A slot is a thread and a promise
For each executor slot, the controller keeps an object backed by a thread. This is true even for slots that belong to remote agents. When there is nothing to do, the thread sleeps. When the scheduler assigns work to it, the thread wakes up, prepares a workspace, and runs the job. If the job’s commands must run on an agent machine, the thread sends them over the connection to the agent and waits for the output and the exit status. So the thread on the controller is a lightweight “handle” for the work, and the heavy lifting is done by processes on the agent.
It follows that an executor is not a process, not a core, and not memory. It is a permission slip. The number on the settings page is the number of permission slips the node has handed out.
sequenceDiagram
participant Q as Build Queue
participant S as Scheduler
participant E as Executor Slot
participant A as Agent Process
participant W as Workspace Folder
Q->>S: Buildable item with a label
S->>S: Match the label against idle executors
S->>E: Hand the item to a free executor
E->>W: Allocate a workspace folder
E->>A: Send the steps over the agent connection
A-->>E: Stream output and exit status
E->>W: Release the workspace
E-->>S: Executor is idle again
The scheduling loop
Jenkins runs a scheduling routine again and again. Each time, it looks at the buildable items in the queue and the idle executors on online nodes. For each item, it checks the label expression against the nodes. It also considers a few preferences, such as favouring the node where the job last ran, because that node may still hold the workspace and caches from the previous build. If a match is found, the item is handed to that executor. If not, the item stays in the queue and Jenkins tries again on the next round.
A taxi dispatcher looks at the waiting passengers and the free taxis. If a free taxi is near a passenger, the dispatcher sends it. If the passenger asked for a wheelchair-accessible taxi and none is free, the passenger waits, even if ordinary taxis are sitting idle. The dispatcher tries again a moment later.
Preferring the same node
Reusing the same node is a gentle optimisation. A repeat build on the same node may find the code already downloaded and the dependency cache warm, which makes it faster. It is only a preference, not a rule. If the favoured node is busy, Jenkins will not wait forever and will use another node.
What happens when the agent dies
If the connection to an agent is lost while an executor is busy, the thread on the controller notices that the connection has gone. The build running there is normally marked as failed or aborted, and the executor disappears with its node. Pipeline builds can sometimes survive short interruptions, since long-running shell steps are designed to keep running on the agent while the controller reconnects, but a machine that has truly gone will take its running builds with it.
Details of these internals vary between Jenkins versions. The description above matches how Jenkins is generally documented and understood. For a specific installation, the Jenkins system information and thread dump pages show what each executor thread is doing at that moment.
11How Many Executors Do You Need?
This is the question everyone asks. There is no magic number, but there is a sensible way to reason about it.
Start from demand
A simple rule from queueing theory, known as Little’s law, says that the average number of things being served equals the arrival rate multiplied by the average time each one takes. Applied to Jenkins: the average number of busy executors equals the number of builds per hour multiplied by the average build time in hours.
Ten is only the average. Real demand comes in waves: mornings, after lunch, just before a release. If the peak is double the average, you need about twenty executors to avoid a queue at the peak. Many teams add headroom of a third to a half above the average, then adjust after watching the queue for a few weeks. This is a starting rule of thumb, and not a law of nature.
A coffee shop serves sixty customers an hour, and each takes ten minutes to be served at a table. On average, ten tables are occupied. But if the lunchtime rush doubles the customers, twenty tables are needed. A shop with only twelve tables will have a queue at lunch, even though the average day looks fine.
Then check the machine
The number of executors that the demand asks for must also fit the machine. Ask what the typical build does to the machine.
| Build type | What limits it | Executors per machine, as a starting point |
|---|---|---|
| Compile-heavy, multi-threaded tools | Processor cores | A small fraction of the core count, since each build already uses many cores |
| Single-threaded scripts and tests | One core each | About one per core |
| Memory-hungry builds | Available memory | Total memory divided by the peak memory of one build |
| Builds that mostly wait on networks or services | Almost nothing on the machine | More than the core count is often fine |
| Builds with big checkouts or images | Disk space and disk speed | Disk space divided by workspace size, and watch disk speed |
Watch for the noisy neighbour
When several executors share one machine, they compete for the same processor, memory, disk, and network. One heavy build can slow the others, and builds that ran in eight minutes alone may take fourteen when three of them run together. If the times vary wildly with load, the node has too many executors for its real resources.
Few executors per machine
- Each build gets more of the machine and runs faster.
- Builds are better isolated from each other.
- Behaviour is predictable.
Many executors per machine
- Fewer machines to manage.
- Idle capacity is shared, so use is efficient.
- Noisy neighbours, memory pressure, and disk contention.
Change one thing at a time and measure. Raise the executors on one node by one, watch average build time and queue wait for a few days, and then decide. Guessing is cheaper than measuring only until the first outage.
12Workspaces and Concurrency
When several executors work at once, they must not trample each other’s files. Here is how Jenkins keeps them apart, and where it needs your help.
One folder per running build
Each build gets its own workspace folder on the node. When two builds of the same job run at the same time on the same node, Jenkins gives the second one a separate folder, often the same name with an at-sign and a number added. This keeps their files apart. The price is that each starts with an empty folder, so it must download its code and dependencies again.
Two students in a library both need the same reference book and a work desk. The librarian gives each their own desk and their own copy of the book. They do not share notes by accident, but each had to go and fetch a copy.
Controlling concurrent builds of one job
Sometimes two builds of the same job must not run together, for example when both would deploy to the same server. Pipelines have an option to disable concurrent builds, so a second build of the same job waits for the first to finish. That wait appears in the queue as a blocked item. Note that a blocked build does not use an executor, but it does delay the answer to the developer, so use this option only where overlapping is truly harmful.
Limiting use of shared resources
| Need | Tool | Effect |
|---|---|---|
| Only one build of a job at a time | Disable concurrent builds | Later builds wait in the queue |
| Limit how many builds of a category run together | Throttling plugins | Caps a group of jobs, even if executors are free |
| One user of a scarce resource at a time | Lockable resources plugin | Builds wait for a named lock |
| Keep heavy jobs off busy nodes | Labels and usage settings | Steers work to suitable machines |
Disk hygiene
Workspaces pile up. A node with many executors and many jobs can accumulate hundreds of workspace folders, and fill its disk. Jenkins can take a node offline automatically when disk space runs low, which removes all of its executors at once. Clean workspaces at the end of builds when you can, and schedule periodic cleanup for old ones.
Tests that use a fixed port, a fixed database name, or a fixed temporary file will collide when two run together on one node, even with separate workspaces. Give every build unique names for these, or run each build in its own container.
13Elastic and Cloud Agents
Fixed machines waste money when idle and cause queues when busy. Elastic agents create executors on demand and remove them when they are no longer needed.
The idea
With a static agent, you set up a machine once and keep it running. Its executors are always there, whether or not there is any work. With an elastic agent, Jenkins creates a fresh machine, virtual machine, or container when work is waiting, runs the build, and throws the machine away afterwards. Popular choices include container platforms such as Kubernetes and Docker, and cloud virtual machines. Each such agent is typically given a single executor, since it exists for just one build.
A company can buy a fleet of delivery vans, and pay for them even on quiet days. Or it can hire vans by the hour when parcels pile up. The first is a static agent. The second is an elastic agent. Hiring is more flexible, but each van takes a few minutes to arrive.
| Feature | Static agents | Elastic agents |
|---|---|---|
| Start of a build | Immediate, if a slot is idle | Delay while the agent is created |
| Cost when idle | Paid whether used or not | Close to zero when nothing is running |
| Environment | Accumulates leftovers over time | Clean every time |
| Peak capacity | Fixed by the number of machines | Grows as needed, up to a limit you set |
| Caches | Warm and reused | Cold unless you arrange shared caches |
| Maintenance | Patching and cleanup by you | Rebuild the image, and every agent is fresh |
Retention and cleanup
A node can be told when to go away. A retention strategy might say “keep the node forever”, “shut it down after it has been idle for ten minutes”, or “use it once and remove it”. Idle timeouts save money. Use-once agents give the cleanest environments and the strongest isolation between builds.
Limits you should set
- A maximum number of agents. Without a cap, a burst of builds, or a mistake in a matrix, can create hundreds of paid machines.
- Resource requests per agent. On container platforms, state how much processor and memory each build needs, so the platform can place them well.
- Start-up time targets. Use small, pre-built images so agents start in seconds and not minutes.
- Shared caches. Keep dependency caches in shared storage so fresh agents do not download everything each time.
An elastic setup hides capacity problems until the bill arrives. A pipeline that starts many agents for tiny tasks can cost more in start-up time and money than the work is worth. Measure cost per build as well as time per build.
14Performance, Scalability, and Reliability
A healthy executor setup is fast when busy, cheap when quiet, and calm when something breaks. Here is what to watch.
Where the time goes
The time between a commit and a result has three parts: waiting in the queue, setting up (getting an executor, fetching code, starting tools), and doing the work. Adding executors reduces the first part. Caches, small images, and warm agents reduce the second. Faster tests and better parallelism reduce the third. Before you buy more capacity, find out which part is the biggest.
Add capacity
More executors or agents shorten the queue. This helps only if builds are actually waiting for slots.
Release slots early
Remove waiting from agent blocks, add timeouts, and clean up stuck builds so slots return to the pool sooner.
Shorten builds
Use caches, incremental builds, and gated stages. A build that takes half the time halves the demand on executors.
Spread the load
Move heavy scheduled jobs away from the busiest hours, so peaks are smaller.
Scaling out and up
To grow, you can make nodes bigger with more executors, or you can add more nodes. Bigger nodes are simpler to manage but risk noisy neighbours. More small nodes give better isolation and a smaller blast radius when one fails. Most teams settle on many modest agents, or elastic agents that each run one build.
The controller also has limits. It tracks every executor, streams every log, and saves the state of every pipeline. Very large fleets and very chatty builds can make the controller the bottleneck, so its own memory and disk need attention. Very large organisations often run several controllers, each serving a group of teams.
When things fail
| Failure | Effect on executors | Protection |
|---|---|---|
| An agent machine crashes | Its executors vanish, and its running builds fail | Several agents per label, so others can take over new work |
| Network loss between controller and agent | Node shows as disconnected | Reliable launch methods and automatic reconnection |
| Disk fills up | Jenkins may take the node offline | Workspace cleanup and disk monitoring |
| A build hangs forever | One executor is lost until someone aborts it | Timeouts on stages and on the whole pipeline |
| Controller restarts | Queue and pipeline state are saved and resume | Planned restarts with a drain: stop new builds and wait |
| Cloud quota reached | New elastic agents cannot start | Set limits below the quota and alert when near it |
A supermarket with one giant checkout and one cashier is fast when quiet, and stops entirely when the cashier is ill. A supermarket with six ordinary checkouts loses only a sixth of its capacity when one cashier is away. Several modest agents behave like the six checkouts.
Put a timeout on every pipeline. A build that would normally take ten minutes should not be allowed to hold an executor for ten hours. The timeout turns a silent loss of capacity into a visible failure that you can fix.
15Security Considerations
An executor is a place where code runs. That makes it a place where things can go wrong, so it deserves careful handling.
Builds on the controller
Code running on the controller can reach its files and secrets. Keep the controller’s executors at zero on any shared or important installation.
Leftovers between builds
Static agents keep files, caches, and sometimes tokens from earlier builds. A later job, possibly from another team, could read them. Clean workspaces and prefer use-once agents for sensitive work.
Untrusted code
A pull request from a stranger can change the Jenkinsfile and run anything on an executor. Run such builds on isolated agents with no access to secrets.
Shared agents across teams
Many executors on one machine share its user account and its disk. Use separate agents for teams that must not see each other’s files.
Over-powerful agents
An agent that has broad cloud permissions gives every build that runs there the same power. Give agents the smallest permissions that work.
Resource abuse
A build that starts a crypto miner, or a matrix that starts hundreds of agents, turns your executors into someone else’s benefit. Review pipeline changes and set quotas.
Habits that protect you
- Separate roles. The controller plans. Agents work. Do not blur them.
- Use labels to draw lines. Put trusted, production-deploying jobs on labelled agents that untrusted jobs cannot select.
- Least privilege. Agents get the fewest credentials and cloud permissions needed.
- Disposable agents. The fewer things that live on an agent, the fewer things can leak from it.
- Secure the connection. Use authenticated, encrypted connections between controller and agents.
- Keep agents patched. Update the operating system, the agent software, and the tools on a schedule.
A shared kitchen in an office should be wiped and cleared after each person. If one person leaves their lunch bag with their wallet inside, the next person can see it. A disposable agent is a kitchen that is reset to spotless after every user.
16Monitoring, Logging, and Troubleshooting
When builds wait too long, here is how to find out why, calmly and step by step.
What Jenkins shows you
- The Build Executor Status panel. Shows every node’s executors as idle or busy, with the job each one is running.
- The Build Queue panel. Lists waiting items and the reason each is waiting.
- The nodes page. Shows each node’s state, its disk space, its response time, and whether it is online.
- Load statistics. A graph over time of total executors, busy executors, and queue length, for the whole system or per label. This is the best single chart for judging capacity.
- Metrics plugins. Monitoring plugins can export executor and queue numbers to your usual dashboards, so you can alert on them.
flowchart TD
A["Builds are waiting in the queue"] --> B{"Are all executors busy"}
B -->|No| C{"Does the queue message name a label"}
C -->|Yes| D["No online agent has that label so fix the label or bring an agent online"]
C -->|No| E["Check blocked reasons such as concurrency limits or locks"]
B -->|Yes| F{"Are busy executors doing useful work"}
F -->|No| G["Find stuck builds and held executors then add timeouts"]
F -->|Yes| H{"Is the wait only at peak times"}
H -->|Yes| I["Add elastic capacity or spread the schedule"]
H -->|No| J["Add executors or agents and shorten builds"]
Common symptoms and causes
| Symptom | Likely cause | What to do |
|---|---|---|
| Queue message names a label and never clears | No online node has that label | Check the label spelling, or bring a matching agent online |
| Executors show busy, but nothing seems to run | Builds waiting for approval or a sleep inside agent blocks | Move the waiting out of agent blocks and add timeouts |
| Node suddenly offline | Low disk space, lost connection, or a crash | Check disk, the agent log, and the network |
| Queue only grows at the same time each day | Scheduled jobs all start together | Stagger the schedules |
| Build times vary a lot | Too many executors on one machine | Lower the number, or move to larger or more nodes |
| Pipelines seem to freeze under load | Executor starvation, where pipelines hold slots while waiting for more | Use no agent at the top, and agent per stage |
| New elastic agents do not appear | Quota or limit reached, or the image cannot start | Check the cloud limits, the agent template, and the platform events |
Numbers worth tracking over time
Track average and worst-case queue wait time, since it is what developers feel. Track executor use over the day, to see the peaks. Track how many executors are held by builds waiting for something other than work. And track cost per build for elastic agents. Review the charts weekly, and note what changed when the numbers moved.
17Design Patterns and Anti-patterns
Patterns are habits that keep capacity healthy. Anti-patterns are habits that quietly waste it.
Patterns that help
Controller plans, agents work
Zero executors on the controller, and all builds on agents.
Agent per stage
No agent at the top, and an agent only in stages that run commands, so slots are held only while working.
Small label vocabulary
A short, documented list of labels, used consistently across teams.
Use-once elastic agents
Fresh containers for each build, with a cap on the total number.
Timeouts everywhere
Every pipeline and every risky stage has a maximum run time.
Capacity reviews
A regular look at the load chart, queue times, and cost, and a small adjustment each time.
Anti-patterns to avoid
Context
A release pipeline names an agent at the top, and includes a manual approval that often waits for hours.
What goes wrong
The executor is held the whole time, doing nothing. A handful of waiting release pipelines can take most of the team’s capacity, and unrelated builds queue up.
Better decision
Use no agent at the top, put the approval in its own stage without an agent, and start an agent only for the stages that run commands.
Context
To reduce queue times, an administrator raises the number of executors on every node to twenty, without looking at the machines.
What goes wrong
Twenty builds fight over four processor cores and a slow disk. Each build takes three times as long, tests time out, and the queue is no shorter. Failures increase, and everyone blames the code.
Better decision
Set executors from the real resources of the machine, measured by watching load. Add machines when demand exceeds what one machine can handle.
Context
Every job asks for the same single node by name, because “it always worked there”.
What goes wrong
One node is overloaded while others sit idle. When it fails or needs maintenance, every job stops at once.
Better decision
Ask for labels that describe what the job needs, not a machine name. Let Jenkins choose among all matching nodes.
18Advantages, Trade-offs, Best Practices, and Common Mistakes
A balanced view of the executor model, and the habits that keep it working for you.
Advantages
- A simple model: one slot runs one piece of work.
- A firm cap protects machines from overload.
- The queue absorbs bursts without dropping work.
- Labels steer work to the right kind of machine.
- Easy to scale by adding nodes or elastic agents.
- Clear numbers to monitor and plan around.
Disadvantages and trade-offs
- A slot can be held while doing nothing.
- The number of slots does not reflect real machine resources.
- Too many per machine causes noisy neighbours.
- Label mismatches create endless waits.
- Elastic agents add start-up delay and cost surprises.
- Static agents collect leftovers and need care.
Best practices in one place
- Keep the controller at zero executors on any shared or important Jenkins.
- Use no agent at the top of a pipeline, and name agents in the stages that need them.
- Never wait inside an agent block. Approvals, long sleeps, and outside waits belong outside.
- Put timeouts on pipelines and risky stages, so a hung build cannot keep a slot forever.
- Choose executor counts from real resources, then adjust after watching the load.
- Use labels that describe needs, not machine names.
- Prefer several modest agents to one giant one.
- Cap elastic agents so a burst or a mistake cannot create a huge bill.
- Clean workspaces and watch disk space on every node.
- Watch the load chart weekly, including queue wait and cost.
The mistakes seen most often
Mistake 1: confusing executors with cores
An administrator sets sixteen executors on a four-core machine “to be safe”. Builds slow to a crawl and tests time out. The fix is to base the number on measured load, not on hope.
Mistake 2: the held slot
A pipeline names an agent at the top, then waits for approval. Executors are used up by builds that are doing nothing. The fix is to move waiting out of agent blocks.
Mistake 3: the label that no one has
A job asks for a label with a small typo. It waits in the queue forever, and adding executors elsewhere does not help. The fix is to read the queue message and compare it to the node labels.
Mistake 4: builds on the controller
A team leaves the controller with executors because it is convenient. A heavy build slows the whole system and untrusted code runs next to the secrets. The fix is zero executors on the controller.
19Worked Scenario: The Queue That Never Empties
Let us take one imaginary team and fix their queue in steps, using the ideas from this guide.
The starting point
A small team runs Jenkins with two static agents, each with four executors, so eight in total. About forty-eight builds arrive each hour, and each build holds an executor for an average of twelve minutes. By Little’s law, that is 48 times 0.2 hours, or 9.6 executors busy on average. The team has only eight. The queue grows all day and never fully clears.
Look at what the executors are doing
The team’s load chart shows all eight busy. Opening the executor panel, they find several pipelines waiting for approval inside an agent block. About a quarter of the held time is pure waiting.
Release the waiting slots
They move approvals into their own stages without an agent, and change the top of each pipeline to use no agent. The average hold time falls from twelve minutes to nine. Demand is now 48 times 0.15, or 7.2 busy executors.
Shorten the builds
They add dependency caches and skip stages that do not matter for small changes. The hold time falls to seven minutes, and demand drops to about 5.6 busy executors.
Plan for the peak
At the lunchtime rush the demand doubles, to about 11.2. Eight static executors cannot cover it. The team adds an elastic pool capped at six extra agents, giving a peak capacity of fourteen.
Protect the controller
They set the controller’s executors to zero, so no build can run beside the secrets, and add timeouts so a hung build cannot hold a slot for hours.
| Version | Average busy executors needed | Capacity | Result |
|---|---|---|---|
| Start | 9.6 | 8 | Queue grows all day |
| Waiting moved out of agent blocks | 7.2 | 8 | Almost full, so waits still spike |
| With caches and gated stages | 5.6 | 8 | Comfortable on average |
| Lunchtime peak at double demand | 11.2 | 14 with elastic burst | Queue stays short |
A road that is nearly full moves slowly, and one small hold-up creates a long jam. A road that is half full flows freely. The same is true of executors. Keeping use well below the full mark, with an elastic overflow lane for the rush, keeps waits short.
These numbers are made up to show the pattern. Use your own build counts and durations. The important lesson is the order of the fixes: release wasted slots first, shorten builds second, and only then buy more capacity.
20Real-World Usage Patterns
Here is how executors tend to be arranged in real Jenkins installations. These are illustrative patterns in general terms, not claims about the private setups of any particular company.
Jenkins is one of the most widely used automation servers, and large engineering organisations have publicly described running it with large fleets of agents, often created on demand. What tends to be common is a habit of separating planning from work, labelling machines by purpose, and measuring queue times. The patterns below show that habit in practice.
Pattern: a small team with two agents
A team of ten developers keeps a controller with zero executors and two modest agents with a few executors each. Most builds start at once, and a short queue at peak times is accepted as a fair price for a small bill.
Pattern: Kubernetes pods as executors
A platform team lets Jenkins create a pod for every build, each with one executor and its own tools. When the queue is empty, there are no build pods running at all. A cap on the number of pods protects the cluster.
Pattern: special machines by label
A company with mobile apps keeps a few macOS machines for iOS builds and marks them to accept only matching jobs. The scarce machines are never wasted on ordinary Linux builds.
Pattern: separate controllers per group
A very large organisation runs several Jenkins controllers, one per business unit, so that one team’s heavy pipelines cannot slow another’s, and each can set its own security rules.
Pattern: nightly work on cheap capacity
A team runs heavy nightly test suites at a time when the daytime agents are idle, or on temporary cheaper machines, so developers’ daytime builds never wait behind them.
Jenkins is often used together with Docker, Kubernetes, and cloud platforms. These tools provide machines that appear and disappear on demand, which makes the executor model both easier to scale and easier to overspend on. Set limits, and watch the bill.
21Frequently Asked Questions
22Summary and Key Takeaways
Key Takeaways
- An executor is a slot, not a core. It lets one piece of work run at a time on a node, and the total number of slots is the number of builds Jenkins can run at once.
- The queue is the waiting line. Items pass through waiting, blocked, and buildable states, and a label with no matching online node makes a build wait forever.
- Hold slots only while working. Use no agent at the top of a pipeline, name agents in the stages that need them, and keep approvals and other waiting outside agent blocks.
- Keep the controller at zero executors. Plan on the controller, and run all real work on agents that you can isolate and replace.
- Size from real resources and demand. Use Little’s law to estimate average busy executors, add headroom for peaks, and check that each machine can really carry its share.
- Use elastic agents with limits. They save money when idle and add capacity at peaks, but need caps, small images, and cost tracking.
- Watch, then adjust. Track queue wait, executor use, held slots, and cost, and fix wasted slots before buying more capacity.