The build queue is the waiting line where Jenkins keeps every build request until a machine is ready to run it.

First, the basics

Jenkins is a free automation server that builds, tests, and delivers software. A build is one run of a job: fetch the code, compile it, run the tests, and report the result. Builds do not run on the main Jenkins server. They run on worker machines called agents. Each agent can run a limited number of builds at the same time, and each slot that can run one build is called an executor.

What is the queue?

When a build is requested but no executor is free, Jenkins cannot start it immediately. It places the request in the build queue. The queue appears on the left side of the Jenkins home page, listing the jobs that are waiting and the reason each one waits. As soon as a suitable executor becomes free, Jenkins takes the next eligible item out of the queue and starts the build.

Who puts items in the queue?

• People who click the build button.
• Code changes reported by webhooks or found by polling.
• Schedules that start jobs at fixed times.
• Other jobs that trigger downstream jobs after they finish.
• Automation that calls the Jenkins web interface from scripts and tools.

Why queue management matters

Every minute a build waits is a minute a developer waits. Long queues slow feedback, encourage people to batch changes into larger and riskier ones, and create pressure to skip tests. Managing the queue well is one of the most direct ways to make a whole engineering team faster.

Any system with limited workers and unlimited requests needs a queue. Build servers are no exception.

The early days

The first Jenkins installations were small. One server ran a handful of jobs, and each build started the moment it was requested. As teams added more jobs, more developers, and more tests, builds began to overlap. To keep the server from collapsing under too many simultaneous builds, Jenkins introduced executors, which cap how many builds run at once, and the queue, which holds the rest.

The move to many agents

Soon one machine was not enough. Jenkins gained the ability to distribute builds across many agents, each with its own tools and operating system. That created a new question: not only “is there a free executor?” but “is there a free executor on a machine that can run this particular job?” Labels and matching rules grew out of that need.

The cloud era

Later, agents became disposable. Instead of a fixed set of machines, Jenkins can create agents on demand in the cloud or in a container platform, and remove them when idle. The queue then also became a signal: a growing queue tells Jenkins to create more agents, and an empty queue tells it to remove them.

Real-world motivation

Companies that run large build farms describe the queue as their most important health signal. Teams with hundreds of developers watch queue wait time the way a website team watches page load time, because both directly shape how people feel about the tool.

A handful of ideas explain almost everything about queues.

Vocabulary table

Requests flow in from several sources, meet the queue on the controller, and leave for the agents that can run them.

The players

The triggers produce build requests. The controller keeps the queue and runs the scheduler, the part that decides which item goes to which executor. Agents provide executors. A cloud provisioner, when used, watches the queue and creates new agents if the line grows. Monitoring observes queue length, wait times, and utilisation.

flowchart LR
    T1["Webhook from code host"] --> Q["Build Queue on Controller"]
    T2["Scheduled trigger"] --> Q
    T3["User clicks build"] --> Q
    T4["Upstream job finishes"] --> Q
    Q --> SCH["Scheduler matches items to executors and labels"]
    SCH --> A1["Static agents with executors"]
    SCH --> A2["Elastic cloud agents"]
    Q -->|"Queue length signal"| PROV["Cloud Provisioner"]
    PROV --> A2
    A1 --> RUN["Builds run"]
    A2 --> RUN
    Q --> MON["Metrics for queue time and length"]
    A1 --> MON
    A2 --> MON

Reading the diagram

Requests arrive from webhooks, schedules, people, and upstream jobs, and all of them land in the same queue on the controller. The scheduler looks at each item, checks which agents can run it, and hands it to a free executor. If the queue grows and cloud agents are configured, the provisioner is told to create more of them. Metrics record how long items wait and how busy agents are, giving the team the numbers that guide decisions.

Where the queue lives

The queue is held in the memory of the controller and is saved to disk regularly and at shutdown, in a file in the Jenkins home directory. If the controller restarts, it reads the file and restores the waiting items, so requests are not lost. The controller is therefore the single place where waiting is managed, which is also why an overloaded controller slows the whole system.

One queue, many rules

Although there is one queue, many rules shape how items leave it: labels, concurrency limits, locks, throttles, priorities, and quiet periods. The rest of this guide explains each rule, so that when an item is stuck, you know which rule to suspect.

A queue item is not simply “waiting.” It passes through several states, and each state has its own reasons and its own fixes.

The states, in plain words

Why the distinction matters

Blocked and buildable items look alike on the page, because both are “waiting.” But their causes and cures differ completely. A buildable item needs more capacity or better matching. A blocked item needs someone to understand which rule holds it back. Adding more agents will not unblock an item held by a lock.

Follow one build from the moment code is pushed to the moment an executor starts it, including the case where a new cloud agent must be created first.

sequenceDiagram
    participant D as Developer
    participant H as Code Host
    participant Q as Build Queue
    participant S as Scheduler
    participant P as Cloud Provisioner
    participant A as Agent Executor
    D->>H: Push commit
    H->>Q: Webhook creates queue item
    Q->>Q: Wait out the quiet period and merge repeats
    Q->>S: Item becomes buildable
    S->>S: Look for free executor with matching label
    S->>P: No free agent so signal demand
    P->>A: Start new agent
    A-->>S: Agent connects and offers executor
    S->>A: Assign the build
    Q-->>Q: Item leaves the queue
    A->>A: Run the build
    A-->>D: Result reported on the commit

The lifecycle in plain steps

Where the time goes

Total wait is the sum of several small waits: the quiet period, any blocked time, the search for a free executor, and the provisioning of a new agent. Reducing wait time starts with knowing which of these dominates. A team that shortens the quiet period to zero but ignores a five-minute agent start-up has fixed the wrong thing.

The queue drains only as fast as executors free up. Understanding executors is the key to understanding capacity.

What is an executor?

An executor is a slot on a machine that can run one build at a time. A machine with four executors can run four builds together. Jenkins uses the word node for any machine that provides executors, including the controller itself, which is called the built-in node in current versions.

The controller should not build

Current guidance is to set the number of executors on the built-in node to zero. The controller has an important job: it keeps the queue, serves web pages, stores results, and talks to agents. Heavy builds on the same machine compete with that work and can slow everyone. There is a security reason too. Build code running on the controller can reach its files and secrets.

How many executors per agent?

There is no universal number, but a helpful rule of thumb is to match the executors to what the builds need:

Many small agents or few large ones?

Not every build can run on every agent. Labels are how jobs say what they need and agents say what they offer.

What a label is

A label is a short tag attached to an agent. Typical labels describe the operating system, the installed tools, the hardware, or the purpose: “linux,” “windows,” “docker,” “gpu,” “large,” or “release.” A job or a pipeline stage asks for a label, and Jenkins runs it only on agents that carry it.

Label expressions

Jobs can ask for more than one label using simple logic. A job may need “linux and docker,” or “windows or mac,” or “not slow.” The expression describes the set of acceptable agents. The richer the expression, the smaller the pool that can run the job, and the greater the chance that the item waits.

Pools as a queue design tool

The matching trap

Labels create a subtle problem: a big pool of idle agents does not help an item whose label matches none of them. An item waiting for the label “gpu” will wait even if fifty general agents are idle. The queue message says so, for example “waiting for next available executor on gpu.” When you see long waits with idle capacity elsewhere, check labels first.

Jenkins deliberately delays and combines requests to avoid doing the same work twice.

The quiet period

When a build is requested by a trigger such as a code push, Jenkins does not start it at once. It waits for a short time, five seconds by default, called the quiet period. If another request for the same job arrives during that time, Jenkins folds it into the same queue item. The result is one build that sees all the changes, instead of several builds racing each other.

Merging of identical requests

If a job is already in the queue and another request arrives with the same parameters, Jenkins usually does not add a second item. It notes the extra cause on the existing item. This protects the queue from being flooded by repeated clicks or repeated triggers. If the requests carry different parameter values, they are different requests, and each one gets its own item.

Choosing the quiet period

The trade-off

A longer quiet period saves capacity, because fewer builds run, but it delays feedback by that many seconds. A shorter period gives faster feedback but may run more builds. For busy repositories, a moderate delay often produces both less load and a better overall experience.

Parameters and duplicates

Parameterised builds deserve special care. If a form has a field that changes on every request, such as a timestamp, no two requests are ever identical, and merging never happens. A few automated callers can then flood the queue with near-identical items. Keep parameters meaningful, and avoid adding values that change every time for no reason.

Jenkins tells you why each item waits. Learning to read the messages solves most mysteries in seconds.

The common messages

A step-by-step investigation

When the message is silent

Occasionally an item shows no clear reason, or stays in the handover state for a long time. This often points to trouble with an agent, such as a broken connection or a full disk. Check the agent’s status page and log, and consider disconnecting and reconnecting it.

A simple idea from mathematics explains why queues seem fine for months and then suddenly become terrible.

The surprising shape of waiting

Imagine ten executors and builds that each take ten minutes. The system can finish sixty builds per hour if it is perfectly busy. If requests arrive at thirty per hour, executors are about half busy, and builds start almost at once. At fifty per hour, executors are busy most of the time, and waits are noticeable. At fifty-eight per hour, waits are long. At sixty or above, the queue grows without limit, because work arrives faster than it can be done.

The lesson: do not aim for full utilisation

Waiting time does not rise in a straight line as the system fills up. It stays low for a long time, and then it rises steeply as the system approaches full use. A system running at ninety-five percent looks efficient on a chart, but its users experience long waits. Many teams aim for utilisation of around sixty to seventy percent at peak, leaving room for bursts.

Bursts matter more than averages

Averages hide the real trouble. If most builds arrive in the two hours after lunch, or right at the start of the working day, the system must handle those peaks, not the daily average. A system that is quiet all night and overloaded from nine to eleven has a peak problem, whatever the average says.

Long jobs hurt everybody

A single build that takes three hours blocks an executor for the whole time. A handful of such jobs can occupy a large part of a small pool. Mixing quick jobs and long jobs in one pool is one of the most common causes of slow queues. Separate pools for long jobs, or limits on how many may run at once, protect the quick ones.

Sometimes the goal is to make builds wait on purpose, so that they do not trample each other or the systems they touch.

Why limit overlap?

Two builds of the same job that run at the same time can interfere. They may write to the same test database, deploy to the same environment, or fight over the same files. Limiting how many run together protects shared resources, even though it lengthens the queue.

The options in Jenkins

Waiting versus cancelling

Which fits which job

Several plugins give finer control than the built-in settings. Use them with care, because every rule is another reason for an item to wait.

Throttle Concurrent Builds

The Throttle Concurrent Builds plugin limits how many builds may run at once, either for one job or for a named group of jobs called a category. It can also limit how many builds of a category may run on a single node. It is ideal for protecting a limited outside resource, for example “no more than three builds may use the shared test database at once,” or “no more than two heavy jobs per agent.”

Build Blocker

The Build Blocker plugin lets a job refuse to start while certain other jobs are running, chosen by name patterns. Use it when two jobs conflict but do not share a named resource, for example a backup job that must not overlap with a big deployment.

Lockable Resources

The Lockable Resources plugin lets you declare named resources and ask for them in a pipeline. A build holds the lock while its block runs, and others queue for it. Locks can also pick one resource from a labelled group, so several identical test environments can be shared fairly among builds.

Choosing the right tool

The hidden cost

Locks and throttles hold executors in some designs. A pipeline that takes an agent first and then waits for a lock keeps that agent idle while it waits. If many builds do this, they can fill the pool with builds that are only waiting, and starve the builds that could have run. Whenever possible, acquire locks before taking an agent, so that waiting costs nothing.

By default, the queue is roughly first come, first served. Sometimes that is not what a company wants.

The default order

Without extra rules, items are taken in the order they became ready, subject to label matching. That is simple and fair in the sense of arrival time, but it treats a one-minute check and a three-hour test suite alike, and it treats a production hotfix like a routine nightly build.

Priority plugins

Plugins such as Priority Sorter let administrators give jobs numeric priorities, often by folder, view, or job name pattern. Higher-priority items are taken first when executors free up. Common uses are letting release and hotfix jobs jump ahead of routine builds, and letting quick feedback jobs go before long ones. Check the plugin’s current maintenance status before adopting it.

The risk of priorities

Priorities can cause starvation. If high-priority jobs keep arriving, low-priority jobs may never start. A team that marks everything “high priority” achieves nothing, because the top level becomes the new ordinary. Use few levels, and reserve the top for the truly urgent.

Fairness between teams

Priorities decide the order of items, but they do not divide capacity fairly. If one team runs a huge test matrix, it can fill the pool while the others wait. The usual answers are:

• Separate pools with their own labels, so each team has guaranteed capacity.
• Throttle categories so no team can use more than a share of the pool.
• Chargeback or visibility, where each team sees its own usage and cost, which gently encourages efficient use.

How you write a pipeline decides how many executors it occupies and for how long. Careless pipelines waste capacity quietly.

Do not hold an executor while waiting

A pipeline can pause for human approval with the input step, wait for a condition with waitUntil, or sleep. If these steps run inside an agent block, the executor is held the whole time, doing nothing. A build that waits overnight for a manager’s approval occupies an executor overnight. Pipelines have a special ability: steps outside any agent block run on a lightweight slot on the controller, which does not consume a normal executor. So the advice is: take an agent only for the steps that need one, and put waiting steps outside it.

Use an agent per stage

In Declarative Pipeline, setting the top-level agent to none and choosing agents per stage means each stage takes and releases an executor as needed. The approval stage runs with no agent, and the build stages take one only when they run.

Beware of nested agent blocks

If a pipeline takes an executor and, inside it, asks for another agent block, it holds the first while waiting for the second. If many pipelines do this at the same time, and the pool is small, every executor may be held by a build that is waiting for another executor. Nobody can proceed. This is a classic deadlock.

flowchart TD
    A["Pipeline takes executor 1"] --> B["Inside it asks for a second agent block"]
    B --> C{"Is a second executor free"}
    C -->|"Yes"| D["Runs and finishes, releases both"]
    C -->|"No, pool full of similar pipelines"| E["Waits while holding executor 1"]
    E --> F["Every executor is held by a waiting pipeline"]
    F --> G["Deadlock and queue grows forever"]

Parallel branches and fan-out

The parallel step starts several branches at once. If each branch asks for its own agent, a single pipeline may demand many executors at the same moment. Twenty parallel branches from one build can occupy a small pool completely. Limit the number of parallel branches, use failFast to stop the rest when one fails, and consider throttle categories so that one pipeline cannot take over.

Right-size each stage

Stages that only send a message or check a small condition do not need a heavy agent. Ask for the lightest label that will do the job. This keeps expensive agents free for work that truly needs them.

Half of queue management is controlling what goes into the queue in the first place.

Polling versus webhooks

Jenkins can learn about code changes by polling, which means asking the repository at intervals whether anything changed, or through webhooks, which are messages sent by the repository the moment something happens. Polling many repositories every minute creates load and delays. Webhooks are faster and cheaper, with polling kept only as a backup.

Scheduled builds and the thundering herd

Schedules that all say “midnight” start every job at exactly the same moment, creating a wall of demand. Jenkins schedules support a special symbol, written as the letter H, which spreads start times by a hash of the job name. Using it turns one big spike into a gentle spread across the hour.

Downstream fan-out

A job that triggers many downstream jobs when it finishes creates a burst of items at once. Ten upstream jobs each triggering ten downstream jobs generate a hundred items in minutes. Control this with throttle categories, with sensible dependency design, and with quiet periods that merge repeated triggers.

Multibranch scans and pull requests

Multibranch pipelines scan repositories and create jobs for branches and pull requests. A new repository with many branches, or a large rebase, can create dozens of builds at once. Filter which branches are built, cancel superseded builds, and avoid building stale branches automatically.

Automation that hammers the queue

Scripts and tools that call Jenkins can create builds in loops. A misbehaving script can fill the queue with thousands of items. Use API tokens per tool, so you can tell who is responsible, and consider rate limits at a proxy in front of Jenkins.

Instead of guessing how many agents you need, let demand create them, and let idleness remove them.

How it works

Cloud and container plugins teach Jenkins to create agents on demand. When items wait in the queue with a label that a cloud can provide, Jenkins asks the cloud to start an agent. When the agent has been idle for a set time, Jenkins removes it. The queue is both the signal and the customer.

Common platforms

The settings that matter

• Maximum agents: a hard cap that protects your budget and your cloud quota. Without it, a burst or a bug can create hundreds of machines.
• Idle time before removal: too short, and you pay start-up delays again and again. Too long, and you pay for idle machines.
• Start-up time: the time from demand to a ready agent. Prepared images with tools installed start faster than machines that install everything at boot.
• Warm capacity: a small number of agents kept ready all the time, so common builds start immediately.
• Templates per label: each label maps to a template that describes size, tools, and limits.

Strengths and limits

When queue times are too long, the cause could be capacity, matching, rules, or wasteful builds. A short guide points to the right fix.

flowchart TD
    S["Queue time is too long"] --> Q1{"Are items blocked or buildable"}
    Q1 -->|"Blocked"| B1["Find the rule, lock or throttle holding them"]
    Q1 -->|"Buildable"| Q2{"Is there idle capacity with other labels"}
    Q2 -->|"Yes"| M["Fix labels or rebalance pools"]
    Q2 -->|"No"| Q3{"Are executors doing useful work"}
    Q3 -->|"No, waiting or hung"| W["Move waits outside agents and add timeouts"]
    Q3 -->|"Yes"| Q4{"Is demand wasteful"}
    Q4 -->|"Yes, superseded or duplicate builds"| C["Cancel old builds and merge requests"]
    Q4 -->|"No"| Q5{"Can builds be made faster"}
    Q5 -->|"Yes"| F["Cache, parallelise, trim tests"]
    Q5 -->|"No"| A["Add capacity or elastic agents"]

Why this order

The path goes from the cheapest and most likely explanation to the most expensive. Rules and labels cost nothing to fix. Wasteful demand is usually cheaper to remove than to serve. Speeding up builds raises capacity without buying machines. Adding capacity comes last, because it treats the symptom and can hide the real cause.

Examples of the path in action

The queue is a shared resource, and shared resources attract accidents and abuse.

Permissions

Jenkins has fine-grained permissions. The right to start a build, the right to cancel a build or a queue item, and the right to configure jobs are separate. A sensible setup gives most developers the right to build and cancel their own team’s jobs, keeps configuration rights narrower, and reserves administrator rights for a small group. Otherwise anyone might cancel another team’s release build, or reconfigure a job that all others depend on.

The queue as a target

Flooding the queue with expensive items is a simple way to cause trouble, whether by a careless script or by malice. Every item costs memory on the controller, and every started build costs agent time, and money if the agents are elastic. Attackers can also use builds to run their own code on your machines, for example to mine cryptocurrency.

Defences

Separate the sensitive pools

Release and deployment builds hold powerful credentials. Give them their own agents, with their own labels, so that ordinary or untrusted builds can never run on the same machines. Then a flood of pull request builds can neither delay a release nor reach its secrets.

Cancelling and visibility

Cancelling is a power. Log who cancels what. A queue item that vanishes, or a build that is aborted, should leave a trail. Audit logs help resolve arguments between teams and help spot suspicious activity.

Capacity planning turns guessing into arithmetic. You do not need complex tools to do it well.

The basic calculation

Capacity in builds per hour is roughly the number of executors multiplied by sixty, divided by the average build time in minutes. Ten executors and ten-minute builds give sixty builds per hour. Demand is the number of build requests per hour at peak. If demand approaches capacity, waits explode, as we saw in the theory chapter.

A simple planning routine

Levers that raise capacity

The controller is a bottleneck too

A controller with too many jobs, too many agents, or too many plugins can slow down scheduling itself. Symptoms include a sluggish web page, and a queue that fills even though executors are free. When that happens, consider splitting into several controllers, each with its own set of teams and agents. This limits the blast radius and spreads the scheduling work.

Do not forget the outside systems

More executors mean more simultaneous downloads and uploads. The package registry, the code host, and the artifact store all have limits. If builds run faster than those systems can serve them, the queue moves to another place, and failures start showing up as timeouts.

The queue must survive restarts, lost agents, and busy days, or teams will lose work and trust.

What happens on a restart

The controller saves the queue to disk in its home directory and restores it at start-up. Items that were waiting return to the queue. Builds that were running are handled according to the type of job and the durability settings: many pipelines can resume, while others must be restarted. Plan restarts for quiet times, and use the quiet-down feature before maintenance so running builds can finish first.

What happens when an agent disappears

If an agent is lost while running a build, that build fails or, for suitable pipelines, can be retried on another agent. Elastic agents are removed by the platform from time to time, for example by reclaiming a cheaper machine, so pipelines should expect and survive agent loss for infrastructure reasons.

Failure modes to plan for

A safety net of permanent agents

Even a mostly elastic setup benefits from a small number of always-on agents. They serve urgent builds while cloud agents warm up, and they keep the system running if the cloud provider has a problem. It costs a little, and it buys calm.

Stuck items

Items that wait for an impossible reason, such as a label that no longer exists, can sit in the queue for weeks and confuse everyone. Set an alert for items older than a limit, and clean up such items on a schedule.

You cannot manage a queue you cannot see. A few well-chosen numbers reveal almost everything.

The essential metrics

More measures worth tracking

• Blocked versus buildable items, which shows whether rules or capacity are the cause.
• Time to provision an agent, from demand to a connected executor.
• Agents online and offline, by pool.
• Long-running builds, those that exceed a sensible duration.
• Executor idle time, to spot wasted, expensive machines.
• Cancelled and superseded builds, to see how much work is being avoided.
• Retries and failures caused by infrastructure, which hint at agent trouble.

Where the numbers come from

Jenkins exposes queue and executor information through its web interface and its programmatic interfaces. Monitoring plugins can publish the numbers to popular monitoring systems, where they can be charted and alerted on. Metric names vary by plugin and version, so check the documentation of the one you install. Even a simple scheduled script that records the queue length every minute gives you a history to learn from.

Alerts that help

Set a target

Agree on a target for queue time, such as “ninety percent of builds start within two minutes.” A target turns arguments about “slow” into measurable facts and gives the team a shared goal.

Queue rules are code in disguise. Test them before a busy day tests them for you.

Why test?

Locks, throttles, priorities, and elastic agent settings interact in surprising ways. A rule that looks harmless can block a whole team, and an elastic setting that looks safe can create a large bill. Testing on purpose reveals these problems while they are cheap.

Kinds of tests

Use a test Jenkins

Do these experiments on a separate test controller with its own small pool, not on production. A test environment built from the same code as production, using configuration as code, makes the tests realistic and repeatable.

Measure and record

For each test, record what you expected and what happened: how long items waited, how many agents appeared, how the messages read. Keep the results with the rules, so the next person understands why a setting was chosen.

Game days

Once or twice a year, hold a rehearsal in which you simulate a bad day: a cloud quota limit, a lost pool, or a flood of builds. Watch how the alerts and the team respond. The rehearsal will expose gaps in alerts and runbooks far more effectively than reading documents.

Queue settings are part of your infrastructure. Keep them in code, so they can be reviewed, tested, and rebuilt.

What belongs in code

Configuration as Code

The Jenkins Configuration as Code plugin reads a text file describing the controller’s settings, including nodes, clouds, and many plugin options. Together with a repository and reviews, it makes capacity changes as safe as code changes. A change from ten to twelve executors becomes a small, reviewed edit, not a click that nobody remembers.

Shared libraries for standard behaviour

A shared library can offer standard steps that apply queue-friendly habits automatically: cancelling superseded builds, taking an agent only for the necessary block, applying sensible timeouts, and limiting parallelism. Teams then get good behaviour by default, without each writing it again.

Container platforms

On Kubernetes, pod templates define what each label means: image, resources, and limits. The platform’s own autoscaling adds nodes when pods cannot be scheduled, which creates a second queue underneath the Jenkins one. Watch both. A Jenkins queue that looks fine can hide a cluster that is short of capacity, and the reverse.

Cost controls

Active queue management brings real benefits, and every rule has a cost.

Balancing choices

Balance simplicity and control

Every rule is a promise that someone will remember why it exists. Add rules only in response to measured problems, and remove them when the problem is gone. A small team with a modest pool needs almost none. A company with hundreds of developers needs many, and the discipline to keep them documented.

There is no universal answer. Match the design to your size, your budget, and your patience for complexity.

A pattern is a solution many teams discovered independently. An anti-pattern is a trap many teams fell into.

Patterns worth copying

Anti-patterns to avoid

These lessons are usually learned the hard way. You can learn them the easy way.

Best practices

• Keep the controller free of builds, with zero executors on it.
• Read the queue reason first whenever a build waits.
• Design pools by workload, and keep the label vocabulary small and documented.
• Leave headroom: plan for peaks, and aim for about sixty to seventy percent utilisation at peak.
• Cancel superseded builds and merge repeated requests.
• Do not hold agents while waiting for approvals or slow steps.
• Use webhooks, and spread schedules with the hash symbol.
• Cap everything: parallel branches, elastic agents, build durations.
• Protect sensitive pools from untrusted and routine builds.
• Measure queue time, set a target, and alert on stuck items.
• Keep queue settings in code and test them with bursts.
• Speed up builds before you buy machines.

Common mistakes and their fixes

Let us walk through a full improvement for a fictional company, step by step, without writing any code.

The situation

Larkspur Media has one Jenkins controller with twelve agents in a single pool, each with four executors. About a hundred and twenty developers use it. Every morning between nine and eleven, developers complain that builds wait fifteen to thirty minutes. Management’s first reaction is to buy twelve more agents.

Step by step

The result after one month

Many Jenkins installations start with a single pool where everything runs on everything. Here is how to move to managed pools without disruption.

What you may find

• Every agent has the same generic label, or none at all.
• Long jobs, quick jobs, and releases share the same executors.
• Executors were set to a round number years ago and never reviewed.
• Nobody knows which team uses how much capacity.

A safe migration path

The reward

After migration, quick jobs no longer wait behind long ones, releases are protected, and each pool has an owner who can see and manage its own numbers.

A queue that nobody owns slowly fills with rules, exceptions, and mysteries. Ownership and agreements keep it healthy.

Who owns what

Agreements to write down

• A queue-time target, such as ninety percent of builds starting within two minutes.
• A label policy: the approved labels, what they mean, and how to request a new one.
• A priority policy: who may use high priority, and for what.
• A pipeline standard: no waiting inside agent blocks, timeouts on long steps, limits on parallel branches.
• A rule for new rules: every lock or throttle needs a description, an owner, and a review date.

Regular reviews

Every quarter, look at the capacity sheet, the queue-time chart, the list of rules, and the ten biggest consumers of executor time. Retire old rules, fix the worst pipelines, and adjust capacity. Small, regular reviews prevent the slow drift into a confusing setup.

Visibility and accountability

Show each team its own usage: builds, executor hours, and waiting time. When teams can see their consumption, they take care of it. Some organisations also show the cost, which encourages efficiency without forcing central control.

These scenarios show how organisations manage their Jenkins queues. They are illustrative, so you can map them to your own situation.

What these examples share

In every good outcome, the team measured first, found a specific cause, and fixed it with a small, targeted change. In the bad outcomes, the queue was treated as a mystery, and the reaction was to add capacity or rules without understanding what was happening.

Every automation system has to match jobs to limited workers. Comparing them shows what is essential.

Features change quickly, so check each tool’s current documentation before relying on this table.

What Jenkins does differently

In Jenkins the controller pushes work to agents, and one central scheduler decides. In some other systems, agents pull work from named queues. Jenkins gives you the freedom to compose rules from plugins and settings, and the responsibility to keep them understandable.

What is universal

The lessons apply everywhere: measure queue time, keep utilisation below saturation, separate long jobs from short ones, cancel superseded work, protect sensitive pools, cap elastic capacity, and make the reason for every wait visible.

Once the basics feel comfortable, these ideas help larger and more cost-conscious organisations.

Predictive and scheduled scaling

Instead of waiting for the queue to grow, scale capacity ahead of known peaks. If demand rises every weekday at nine, raise the warm pool at eight forty-five. Some teams use history to predict demand and pre-start agents, trading a little idle cost for near-instant starts.

Cheaper, interruptible machines

Many clouds offer machines at a discount if you accept that they may be taken away with little warning. Builds are good candidates, because they can be retried. Combine such machines with retry rules for lost agents, and keep a base of stable machines for critical work.

Build caching and remote execution

The cheapest capacity is the build you do not run. Shared caches, and build systems that reuse results of unchanged parts, can cut build times dramatically. Some organisations go further and execute builds remotely on large pools of workers, which changes the shape of capacity planning.

Multiple controllers

Very large installations run several controllers, each with its own queue, agents, and teams. This limits the effect of any single failure and keeps each scheduler fast. It needs a shared standard and a central view of queue health across all controllers.

Custom queue behaviour

Jenkins allows plugins to influence how the queue is sorted and how items are assigned to nodes. Most teams never need custom logic, and the built-in features plus a few well-known plugins go a very long way. Consider custom work only after measurement shows a clear, recurring problem that nothing else solves.

Service level objectives

Treat queue time as a service level objective: a promise such as “ninety-five percent of builds start within three minutes,” with an error budget and a review process. This turns queue management into an engineering discipline with clear feedback.

When something feels wrong, walk through these questions in order.

Every important word from this guide in one place, explained in plain language.

Short answers to the questions people ask most.

Managing the build queue means matching demand to capacity, removing waste, and making every wait explainable.

You began with a busy coffee shop and finished with a clear picture of how Jenkins receives requests, holds them, matches them to executors and labels, and starts them. Along the way you learned the states of a queue item, how to read its reason, why waits explode near full utilisation, how to control concurrency, locks, and priorities, how to design pipelines that do not waste executors, how elastic agents fit in, and how to plan capacity, monitor, test, and govern the whole system. The main rules are simple: measure first, fix cheap causes before buying capacity, keep headroom, separate long work from short, and never hold an executor while waiting.