We begin with the simplest picture: what a throttle is, and what it protects.

Jenkins is a tool that automates building, testing, and delivering software. Every build it runs consumes resources: processor time and memory on agents, slots on a test database, calls to an outside service, and money on a cloud account. A resource is only useful if it stays healthy. A database that receives twenty simultaneous test runs when it can handle three does not simply run slowly. It falls over, and every test fails.

Throttling means deliberately limiting how much work is allowed to use a resource. A throttle can limit the number of builds running at the same moment, the number of builds started in a period of time, or the amount of a machine that a single build may take. Work that exceeds the limit is not thrown away. It waits in the queue and starts when there is room.

flowchart LR
  A["Many builds want to run"] --> B["The throttle decides how many may proceed"]
  B --> C["Allowed builds use the shared resource"]
  B --> D["The others wait in the queue"]
  D --> B
    

What a throttle is not

To see why throttles matter, imagine a busy afternoon with none.

Problem one: shared things with a hard limit

A team keeps one integration test database. It was built to serve three test runs at the same time. On a busy afternoon, eight pipelines reach the integration test stage together. The database is overloaded, queries time out, tests fail for no reason connected to the code, and developers waste hours chasing false alarms. The builds were each correct. Their number made them wrong.

Problem two: bursts of demand

A large merge, a change to a shared library, or a scheduled job can trigger dozens of builds within a minute. The queue fills, all agents are busy, and a developer’s urgent one-line fix waits behind a hundred less important runs.

Problem three: outside limits

Many services outside your control set their own limits. Code-hosting services cap the number of requests per hour. Public container registries limit how many images an anonymous user may pull. If your builds ignore these limits, they get blocked, and builds fail for reasons that have nothing to do with your code.

Problem four: money

Cloud agents cost money by the minute. A mistake in a pipeline, such as a huge matrix or a runaway trigger, can start hundreds of machines. Without a cap, the first sign of trouble may be the invoice.

A few ideas come up in every throttling discussion. Learn them once here.

The three kinds of limits

Before any plugin, Jenkins already has a built-in throttle: the number of executors.

An executor is a slot on a build agent that runs one piece of work at a time. If your agents together offer ten executors, then at most ten builds, or ten parts of builds, can run at the same moment. All others wait in the build queue. This is the coarsest, and most important, throttle you have. It protects the agents themselves from being overloaded.

What executors control and what they do not

Setting the number sensibly

The number of executors on a node should follow what the machine can genuinely carry. Compute-heavy builds need few executors per machine, because each already uses many cores. Builds that mostly wait on networks can share a machine in greater numbers. A machine with too many executors slows every build on it, and one with too few wastes its capacity. Measure the load, change the number a little at a time, and watch what happens to build times.

The controller and executors

The controller, the main Jenkins server, can also have executors. Most teams set this to zero, so that all builds run on agents and the controller stays free for its real job of planning and serving pages. That is itself a throttle: it stops builds from competing with Jenkins for its own machine.

When the limit belongs to a shared thing, not to a machine, you need a throttle that spans jobs and agents. The best-known tool for this is the Throttle Concurrent Builds plugin.

This plugin lets an administrator define categories. A category is a name, such as “integration-database”, together with a set of limits. Jobs, or parts of jobs, then declare that they belong to the category. Jenkins counts how many builds in the category are running and refuses to start new ones once the limit is reached. They wait in the queue until a running one finishes.

Total limit and per-node limit

The two numbers protect different things. The total limit protects the shared resource, such as the database or a licence. The per-node limit protects the agent, by stopping several throttled builds from landing on the same machine. For a database that can serve three test runs, you might set the total to three. If each test run is heavy on the agent as well, you might also set the per-node limit to one, so that the three runs spread over three agents.

Where categories are defined

Categories are defined once, in the system configuration of Jenkins, by an administrator. This keeps the limits central and consistent. Job authors then only choose the category name. A typo in a category name in a Jenkinsfile can mean the job is not throttled at all, or is throttled by the wrong group, so agree on a list of names and document what each one protects.

Where you apply a throttle decides exactly what is counted, and how many executors are wasted while builds wait.

Two ways in a pipeline

In a Pipeline, the throttle plugin offers two places to declare a category. The first is a job property, set for the whole job. The whole run is then counted against the category, from start to finish, and a new run waits in the queue before it even begins. The second is a step that wraps a block of work, usually the block that takes an agent. Only the wrapped block is counted, and other parts of the pipeline run freely.

Keep the throttled part small

A pipeline that compiles code, runs unit tests, runs integration tests against the shared database, and then packages the result should throttle only the integration test stage. Compiling and unit testing use only the agent and can run with as many others as executors allow. A whole-job throttle would hold back all of that work for no reason.

Matrix jobs

A matrix, or multi-configuration, job runs the same work for many combinations, such as several operating systems and versions. Each combination is its own build. A single trigger can therefore create dozens of builds at once. Matrix-specific throttles let you limit how many combinations of one matrix run together, so that a single commit cannot swamp your agents.

Combining with parallel branches

Parallel branches inside one pipeline each want an executor, and may each be counted separately by a step-level throttle. If a pipeline has six branches that use the database, and the category allows three, then three branches run and three wait, even within the same build. Design with this in mind, and consider whether so many branches make sense for that resource.

Sometimes the problem is not how many builds run together, but how fast new ones keep starting. Rate limits answer that.

The quiet period

Jenkins waits a few seconds, five by default, between a trigger and the moment a build leaves the queue. If more triggers for the same job arrive during that time, they are merged into one build. This short pause absorbs bursts of commits, and every job benefits from it without any extra setup. A longer quiet period merges more triggers into fewer builds, at the cost of slower feedback for the first commit.

Limiting builds per time period

Jenkins also offers ways to limit how many builds of a job may start in a period. A freestyle job has a built-in option to throttle builds, where you state a number of builds and a time period, such as one build per hour. Newer Pipeline versions provide a similar option that you can set as a job option, often with a “user boost” feature so that a build started by a person by hand skips the limit, while automatic triggers respect it. Availability and names vary by version, so check the help page on your server.

Polling and triggers

Many jobs learn about new commits by polling, which means asking the source-control server every few minutes, “Is there anything new?”. Hundreds of jobs polling every minute can put a heavy load on both Jenkins and the source-control server. Webhooks, where the server sends a message when something changes, are much lighter. Where polling is unavoidable, spread the schedules so that jobs do not all ask on the same minute.

Jenkins offers several tools that limit concurrency. Each fits a different problem. Here they are side by side.

Throttle category or lock?

This is the most common choice. A lock treats each resource as a single item that one build holds at a time. It is ideal when you have named things, such as one staging server or five test phones. A throttle category treats the resource as a pool with a count. It is ideal when the resource can serve several builds at once, and you simply want to cap the number, such as a database that handles three parallel runs, and you do not care which “slot” each build gets.

When agents are created on demand, throttling protects your budget and your cluster. And inside each agent, size limits protect the builds from one another.

Caps on dynamic agents

Cloud and container plugins create agents when builds need them. Without limits, a burst of builds can create hundreds of machines. Most such plugins offer caps. A container platform plugin usually offers a limit on the total number of build pods in a cloud, and a limit on the number of instances of each pod template. Virtual machine plugins offer a maximum number of instances. Set these below your account quotas and your budget, so that a runaway pipeline hits a Jenkins limit before it hits a cloud bill.

Size limits inside an agent

On container platforms, each build pod can declare requests, the amount of processor and memory it expects to need, and limits, the amount it may never exceed. Requests help the platform place pods sensibly. Limits stop one greedy build from starving its neighbours. A build that goes over its memory limit is stopped by the platform, and the failure is loud and clear, which is better than a silent slowdown of everything else.

Your builds do not only use your own machines. They also call outside services with limits of their own. Good pipelines respect those limits.

Where outside limits appear

• Source-control services cap the number of requests per hour for each account or application. Jenkins plugins that scan repositories and update commit statuses can use up the allowance.
• Container registries may limit the number of image pulls, especially for anonymous users or shared addresses.
• Package registries and mirrors may block clients that make too many requests too quickly.
• Cloud provider interfaces limit how fast you may create, list, or delete resources.
• Notification and chat services limit the number of messages per minute.

How the pieces of Jenkins help

sequenceDiagram
  participant P as Pipeline Step
  participant S as External Service
  P->>S: Send a request
  S-->>P: Too many requests so please wait
  P->>P: Wait a short time
  P->>S: Try again
  S-->>P: Still busy
  P->>P: Wait twice as long
  P->>S: Try again
  S-->>P: Success
    

Why growing waits matter

If fifty builds all fail at once and all retry after exactly five seconds, the outside service gets another burst of fifty requests at the same moment, and everything fails again. Growing waits, plus a small random variation, spread the retries out over time. Jenkins provides retry steps and waiting steps that can be combined for this, and the wait step even has a built-in growing delay option in many versions.

Throttles create waiting. How the waiting line behaves decides whether the system feels fair.

flowchart TD
  A["A queued build is ready"] --> B{"Is an executor with the right label free"}
  B -->|No| C["Stay in the queue"]
  B -->|Yes| D{"Would a throttle rule be broken"}
  D -->|Yes| E["Stay in the queue as blocked by a throttle"]
  D -->|No| F["Start on the executor"]
  C --> A
  E --> A
    

Why builds wait

A ready build can wait for several different reasons. No executor with the right label is free. Or a throttle category is full. Or a lock is held. Or another run of the same job must finish first. Jenkins shows the reason for each waiting item in the queue. Reading it is the fastest way to know which kind of limit is holding the build.

Fairness and priority

By default, the queue is mostly first come, first served. That is fair, but not always sensible. An urgent production fix may wait behind a hundred routine runs. Plugins exist that let administrators assign priorities to jobs, so that important work is chosen first when a slot frees up. This can help, but it also introduces a risk: low-priority work may wait forever if higher-priority work never stops arriving, which is called starvation. Use priorities sparingly, with a few clear levels, and watch the waiting time of the lowest level.

This chapter lifts the lid. Knowing how the queue asks questions explains many puzzling delays.

The queue is a loop of questions

Jenkins runs a scheduling routine again and again, many times a minute. Each time, it looks at every waiting build and asks whether it is allowed to start. It asks several parties. The scheduler checks whether an executor with a matching label is free. Then plugins that add rules, including the throttle plugin and the lock plugin, are consulted. Each rule can answer “yes, go ahead” or “no, wait, because of this reason”. Only if everyone agrees does the build begin. The reason from the first “no” is the one shown in the queue.

How counting works

The throttle plugin keeps track of which builds belong to each category, by looking at the builds that are currently running. When a queued build asks permission, the plugin counts the running builds in its category, on the whole system and on each node, and compares the counts with the limits. If a limit is reached, it says no. When a running build finishes, the count drops, and on the next round the waiting build may pass.

Job-level and step-level counting

A job-level throttle is checked when the build sits in the queue, before it starts. A step-level throttle is checked when the running pipeline reaches the wrapped block. The build has already started, but its progress pauses at that point until it may enter. This difference explains why step-level throttles appear as “waiting” inside a running build, while job-level throttles appear as waiting in the queue.

Where the state lives

Category definitions live in the system configuration on the controller. Counts are worked out from the builds that are running, so they are not stored as separate numbers that could drift. Rate limits use recorded start times of recent builds. This design means restarts do not usually corrupt counts, but a build that vanished unexpectedly, for example after an agent crash, may keep counting until Jenkins notices it is gone.

A throttle set too loose does not protect. A throttle set too tight wastes time. Here is a simple way to choose numbers.

Start from the resource, not from the builds

Ask what the shared thing can handle. A database that runs three test suites in parallel without slowing down has a capacity of three. A service that allows sixty requests an hour, with each build making ten, supports six builds an hour. A licence server with five seats supports five users. The limit you set should be at or slightly below that capacity, never above it.

Then check the demand

A simple rule from queueing theory, known as Little’s law, says that the average number of builds in progress equals the arrival rate multiplied by the time each takes. If thirty builds arrive each hour, and each holds the throttled resource for six minutes, which is a tenth of an hour, the average number holding the resource is thirty times a tenth, which is three. If your limit is three, you are running at the edge, and waits will be long at busy times.

Three ways to reduce the queue

Throttles trade speed for safety. This chapter explains what that trade costs, and how to keep the system dependable when limits are in place.

What a throttle costs

Every throttle turns some parallel work into waiting. If a category allows three builds and thirty arrive in an hour, each holding the resource for six minutes, the resource is busy all the time, and the wait for the next build depends on how many are already in line. The lower the limit compared with demand, the longer developers wait. The right comparison is not “is there a wait?” but “is the wait shorter than the failures and crashes it prevents?”.

Ways to keep the cost small

• Throttle only what needs it. Wrap the exact stage that uses the shared resource, not the whole pipeline.
• Shorten the hold time. Prepare data and build artifacts before entering the throttled block, and clean up after leaving it.
• Increase the capacity behind the limit. Add another instance, and raise the limit with it.
• Keep hot paths free. Quick unit tests and static checks should not depend on any throttled resource.
• Reserve capacity for urgent work. Use a separate label or pool.
• Use elastic capacity where safe. For stateless work, let cloud agents expand within a cap.

Scalability of the throttles themselves

The scheduling loop consults every rule for every waiting build. With hundreds of builds waiting on many categories, this bookkeeping grows. It is normally light, but a very large queue of long-waiting builds, or a pile of complicated rules, can add load to the controller. Keep the number of categories modest, the number of long-waiting builds low, and avoid using throttles as a substitute for adding capacity.

Reliability: when the count is wrong

Throttles are safety tools for volume. They also help against misuse, though they are not security walls.

Volume as an attack

Not all overload is an accident. A user with the right to start builds can, deliberately or not, start thousands of them. A pull request from a stranger can add a huge matrix, or a loop that starts endless jobs. Even without theft of data, this can exhaust agents, fill the queue, run up the cloud bill, and stop the team from working. This is called a denial of service, and cost caps are one of the best defences.

Habits that protect you

• Set hard caps at the outer edge. Cloud and container caps below quotas and budget are your last line of defence.
• Give untrusted builds their own small pool. Use labels and separate caps for outside contributions.
• Restrict who can change throttles. Only administrators should define categories and limits.
• Review pipeline changes. A change that touches throttles, matrices, or triggers deserves the same care as a change to production code.
• Alert on cost and queue growth. A sudden rise is the first sign of a runaway.
• Use timeouts. They limit how long any build can occupy a limited resource.

When builds wait too long, or a protected resource still overloads, here is how to find the reason calmly.

What Jenkins shows you

• The build queue. Lists waiting builds with the reason each is waiting, and how long it has waited.
• The executor status panel. Shows which executors are busy and what they are running.
• Load statistics. A chart over time of total executors, busy executors, and queue length.
• The console log. A build waiting inside a step-level throttle prints a message naming the category.
• Metrics plugins. Export queue size, waiting time, and executor use to your normal dashboards, so you can alert on them.
• Cloud and cluster dashboards. Show agent counts, quotas, and spending.

flowchart TD
  A["Builds wait longer than expected"] --> B{"Does the queue reason mention a throttle or category"}
  B -->|Yes| C{"Is the count of running builds in the category correct"}
  C -->|No| D["Look for stuck or orphaned builds and clear them"]
  C -->|Yes| E["The limit is working so raise it or add capacity"]
  B -->|No| F{"Are all executors busy"}
  F -->|Yes| G["Add capacity or shorten builds"]
  F -->|No| H["Check labels, locks, and disabled concurrent builds"]
    

Common symptoms and causes

Numbers worth tracking over time

Track the average and worst waiting time in the queue, split by reason. Track how full each throttle category is over the day. Track how many builds needed retries because of outside limits. Track agent counts and cost. A weekly look at these charts shows whether your limits are too loose, too tight, or well placed.

Patterns are choices that keep throttles helpful. Anti-patterns are habits that turn limits into obstacles.

A simple way to choose

flowchart TD
  A["What are you protecting"] --> B{"A shared service or environment"}
  B -->|Yes| C{"Must builds run strictly one at a time"}
  C -->|Yes| D["Use a lock on the resource"]
  C -->|No| E["Use a throttle category with a maximum"]
  B -->|No| F{"Machine or cloud capacity"}
  F -->|Yes| G["Set executor counts and cloud caps"]
  F -->|No| H{"Too many builds per hour"}
  H -->|Yes| I["Use rate limiting or a longer quiet period"]
  H -->|No| J["Reconsider whether a throttle is needed"]
    

Patterns that help

Anti-patterns to avoid

A balanced view of throttling, and the habits that keep it working for you.

Best practices in one place

1. Find the real bottleneck first. Throttle the shared thing that fails, not the whole system.
2. Set limits from measured capacity, at or slightly below what the resource can handle.
3. Keep throttled blocks small, and do preparation and cleanup outside them.
4. Place throttles around the agent block, so waiting builds do not hold executors.
5. Add timeouts to every throttled block.
6. Name categories after the resource, and document the true capacity behind each number.
7. Cap dynamic agents below cloud quotas and budget.
8. Use proxies, caches, and backoff for outside systems.
9. Reserve an urgent lane for hotfixes.
10. Watch queue waiting times and category fullness, and review limits regularly.
11. Test with overlapping builds before trusting a new throttle.

The mistakes seen most often

Let us follow one imaginary team as it tames an overloaded test database, and learns why the first fix is rarely the last one.

The starting point

A team of twenty developers shares one integration test database. It can serve three test runs at once without slowing down. About thirty builds arrive each hour. A whole pipeline takes about fifteen minutes, and its integration test stage, which is the only part that touches the database, takes about six. On busy afternoons, eight pipelines reach the integration stage together. The database is overloaded, queries time out, and about a third of the runs fail for reasons unrelated to their code. Developers rerun builds, which adds more load.

Here is how throttling tends to appear in real Jenkins installations. These are illustrative patterns in general terms, not claims about the private setups of any particular company.

Jenkins is one of the most widely used automation servers, and teams of every size meet the same limits: shared environments, outside rate limits, and cloud bills. What tends to be common is a habit of throttling the real bottleneck, capping dynamic capacity, and watching the queue. The patterns below show that habit in practice.