A plain-language first look before any details.

Jenkins is a free, open-source automation server that builds, tests and delivers software. Its work is written as a pipeline, a recipe stored in a text file called the Jenkinsfile. A pipeline is divided into stages, such as Build, Test and Deploy, and each stage contains steps, the single actions Jenkins performs.

A distributed build setup splits Jenkins into two kinds of machines. The controller is the manager: it stores settings, schedules work, shows the web pages and keeps history. The agents are the workers: they run the commands that compile code, run tests and package software. Many agents can work at the same time, so many builds can run in parallel.

Distribution brings three gifts. Speed: many builds run at once, and one build can even split into parallel pieces. Variety: some agents run Linux, some Windows, some have special hardware, so each job runs where it fits. Isolation: a misbehaving build damages only its own agent, not the manager. The price is more moving parts to run, monitor and secure, which is what this guide is about.

The problem it solves and how the idea grew.

The problem: one machine has limits

Every machine has a fixed number of processor cores, a fixed amount of memory and a fixed disk speed. When many builds run at once, they compete for these resources and all slow down. Worse, one runaway build can consume everything, including the resources Jenkins itself needs to stay responsive. The web page slows, the queue stalls, and people cannot even see what is wrong.

There is also a variety problem. A mobile team needs machines with phone tools. A data team needs machines with graphics cards. A company that ships on Windows and Linux needs both. One machine cannot be all of these.

A handful of small ideas that make everything else easy.

Controller

The main Jenkins server. It holds configuration, credentials, job definitions and build history, and decides which agent runs what.

Agent

A worker machine, virtual machine or container that runs build steps. It connects to the controller and waits for work.

Executor

A slot on an agent that can run one build at a time. An agent with four executors can run four builds at once. Executors are the unit of capacity in Jenkins.

Label

A tag attached to an agent, such as linux or gpu. Pipelines ask for a label, and Jenkins picks an agent that carries it.

Build queue

The waiting list of builds that have work to do but no free executor yet. Queue length and waiting time are the best health signals of a distributed setup.

Workspace

A folder on an agent where a build downloads code and writes output.

Cloud (provisioner)

A plugin that can create and destroy agents automatically, for example by asking a cloud provider for a virtual machine or a Kubernetes cluster for a pod.

The parts of a distributed Jenkins setup and how they fit together.

flowchart LR
    DEV["Developers push code"] --> SCM["Source Control Server"]
    SCM -->|"webhook"| CTRL["Jenkins Controller"]
    CTRL --> Q["Build Queue"]
    Q --> SCH["Scheduler matches labels"]
    SCH --> P1["Static agent pool: Linux"]
    SCH --> P2["Static agent pool: Windows"]
    SCH --> P3["Elastic pool: containers or cloud VMs"]
    P1 --> ART["Artifact storage"]
    P2 --> ART
    P3 --> ART
    P1 --> CACHE["Shared caches and package proxies"]
    P3 --> CACHE
    P3 -->|"provision and remove"| CLOUD["Cloud or Kubernetes cluster"]
    CTRL --> MON["Monitoring and alerts"]

Read the diagram from left to right. A push triggers the controller through a webhook. Work goes into the build queue. The scheduler matches each item to an agent that carries the right label, drawing from static pools of always-on machines or an elastic pool that expands through a cloud or a cluster. Agents fetch dependencies from shared caches and package proxies, and publish results to artifact storage. Monitoring watches the queue and the machines and raises alerts.

Protecting the one component that everything depends on.

In a distributed setup, the controller is the single place where configuration, credentials and history live. If it slows down, every developer feels it. If it fails, no build can start. Keeping it healthy is the first rule of scaling.

Set the built-in executors to zero

By default the controller has a built-in node that can run builds. On a shared installation you should set its number of executors to zero, so nothing runs there. This has two benefits: performance is protected, and security improves, because untrusted build code can no longer reach the machine that holds all the secrets.

The hidden work in pipelines

Even when all steps run on agents, the pipeline’s own logic, the small program that decides what happens next, runs on the controller. Simple pipelines make this cost trivial. But a pipeline that loops through thousands of items, parses large files or does heavy calculation in its own script can burden the controller. The rule is to let agents do the heavy work through ordinary commands, and keep the pipeline script itself thin.

Two ways for a worker to join the team.

An agent must connect to the controller before it can receive work. There are two main styles, and the choice affects networking, security and automation.

flowchart TD
    subgraph SSHM["Controller starts the connection"]
        C1["Controller"] -->|"opens secure shell connection"| A1["Agent machine"]
        A1 --> R1["Agent program launched by controller"]
    end
    subgraph INB["Agent starts the connection"]
        A2["Agent machine"] -->|"connects outward with secret"| C2["Controller"]
        C2 --> R2["Agent stays connected and waits for work"]
    end

Which to choose at scale

For large, automatically created fleets, inbound agents are the natural fit. A newly created container or virtual machine starts, connects outward to the controller and announces itself. The controller does not need to know its address beforehand. For a small, fixed group of long-lived machines, secure shell is simple and well understood.

The number of slots decides how much competition happens on one machine.

Every executor on an agent can run one build. More executors mean more builds at once on the same machine, but also more competition for the processor, memory and disk. Too many executors make every build slower. Too few waste the machine.

The one-build-per-agent idea

Many large installations settle on one executor per agent, and scale by adding more small agents. This gives perfect isolation: a build cannot steal memory or disk from a neighbor, cannot leave files that disturb the next build unnoticed, and behaves the same every time. It suits containers and cloud virtual machines, which are cheap to start.

How to tune

• Watch processor and memory use during busy hours. If they sit near their limits, reduce executors.
• Watch idle time. If agents are mostly idle while builds queue for a different label, the labels or pools are wrong.
• Change one thing at a time, and compare average build duration before and after.

The addressing system that sends each build to a suitable worker.

A label is a short word attached to an agent. A pipeline asks for a label, and Jenkins picks an agent that has it. Labels can also be combined in label expressions using words such as and, or and not, for example “linux and docker” or “windows and not gpu”.

flowchart TD
    ROOT["All agents"] --> OS["By operating system"]
    ROOT --> SZ["By size"]
    ROOT --> CAP["By capability"]
    ROOT --> TR["By trust level"]
    OS --> L["linux"]
    OS --> W["windows"]
    SZ --> S1["small"]
    SZ --> S2["large"]
    CAP --> D["docker"]
    CAP --> G["gpu"]
    TR --> T1["trusted"]
    TR --> T2["untrusted"]

Good labeling rules

• Label by capability, not by project. A label such as docker stays useful for years. A label such as payments-team-agent multiplies pools and wastes capacity.
• Keep dimensions independent. Operating system, size, capability and trust level combine freely through expressions.
• Keep the list short. Every new label splits the fleet into smaller, less flexible groups.
• Document each label, so people know what it promises.

Why too many labels hurt

Suppose you have 40 agents. If everything shares one label, any build can use any agent, so the fleet behaves like one big pool and queues stay short. If you split them into ten pools of four, one pool can be full while another sits idle. Fine-grained labels cost you the natural efficiency of sharing.

The queue, the scheduler and the moment of assignment.

flowchart TD
    A["Build requests an agent with a label"] --> B["Item enters build queue"]
    B --> C{"Free executor with matching label?"}
    C -->|"Yes"| D["Scheduler picks a node"]
    D --> E["Build starts on that agent"]
    C -->|"No"| F{"Cloud can provision more?"}
    F -->|"Yes"| G["Request new agent from cloud"]
    G --> H["New agent connects"]
    H --> D
    F -->|"No"| I["Build waits in queue"]
    I --> C

Reading the queue

The queue page tells you why each item waits: “waiting for next available executor,” “all nodes of label X are offline,” or “waiting for an agent to be provisioned.” These reasons are the fastest diagnosis of a scaling problem. If many items wait for a label with no agents online, the fix is capacity or connectivity. If items wait even though agents look idle, suspect labels or throttling rules.

Permanent machines, their strengths and their weaknesses.

A static agent is a machine that is created once and stays online. It might be a physical server in an office, a virtual machine in a data center or a long-running cloud instance. Jenkins connects to it and reuses it for build after build.

Keeping static agents healthy

• Clean workspaces at the end of builds, or on a schedule.
• Watch disk space. Jenkins can take an agent offline automatically when free space drops below a threshold, which is a useful safety net.
• Rebuild machines regularly from a known image, rather than patching them forever.
• Standardize tools so every agent with a given label behaves the same.

Workers that exist only for the length of one build.

An ephemeral agent is created for a build and destroyed afterward. It might be a container started in a cluster, or a virtual machine created by a cloud provider on request. Nothing survives from one build to the next, which makes every build start from a clean, known state.

flowchart LR
    Q["Build waits in queue"] --> R["Cloud plugin requests an agent"]
    R --> S["Container or VM starts from an image"]
    S --> C["Agent connects to controller"]
    C --> B["Build runs on clean environment"]
    B --> D["Build ends"]
    D --> X["Agent destroyed"]
    X --> F["Resources returned, billing stops"]

Why ephemeral agents dominate at scale

They turn cost into a function of usage: no builds, no agents, no bill. They make every build reproducible, because the environment is defined by an image, not by the history of a machine. And they make security simpler, because a compromised build cannot leave anything behind for the next one.

The feedback loop that keeps capacity close to demand.

Autoscaling means adding agents when work is waiting and removing them when they are idle. Jenkins does this through cloud plugins that watch the queue, ask a provider for more agents, and later shut down the ones that are no longer needed.

flowchart LR
    A["Queue grows"] --> B["Cloud plugin sees waiting builds"]
    B --> C["Provision new agents up to a limit"]
    C --> D["Agents connect and take work"]
    D --> E["Queue shrinks"]
    E --> F["Agents become idle"]
    F --> G{"Idle longer than the retention time?"}
    G -->|"Yes"| H["Terminate agent"]
    G -->|"No"| F
    H --> I["Capacity matches demand"]
    I --> A

The key settings

Common surprises

• Provisioning delay. Creating a virtual machine can take a minute or several. Containers start faster, but still need to pull an image.
• Thundering herd. At nine in the morning, hundreds of builds start at once. Autoscaling reacts after the queue grows, so the first minutes are slow. Warm capacity or scheduled scaling before peak hours smooths this.
• Quota walls. Providers limit how many machines you may create. Hitting the wall makes builds wait even though the cloud plugin is trying.
• Flapping. If idle retention is very short and builds arrive in bursts, agents are created and destroyed constantly, wasting time.

Using a container cluster as an agent factory.

Kubernetes is a system that runs and manages containers across many machines. The Jenkins Kubernetes plugin lets Jenkins ask a cluster for a pod, which is one or more containers running together, for each build. The pod becomes the agent, runs the build steps, and is deleted when the build ends.

The pod template

A pod template describes what the agent looks like: which container images it contains, how much processor and memory each may use, which storage to attach and which labels it carries. Pipelines can either refer to a template by label or define one inline.

Benefits and cautions

When work spreads across machines, files need a way to travel.

In a distributed pipeline, one stage may run on one agent and the next on a different one. Files created in the first workspace do not follow automatically. Jenkins offers several ways to move them, each suited to a different size and purpose.

flowchart TD
    A["Agent 1: build stage"] --> S["Stash small files through controller"]
    A --> AR["Archive artifacts"]
    A --> EX["Upload to external artifact repository"]
    S --> B["Agent 2: test stage unstashes"]
    AR --> H["Kept with the build record"]
    EX --> D["Agent 3: deploy stage downloads"]
    EX --> R["Long-term storage and sharing"]

The controller is not a file server

Stash and archive normally send data through the controller. A few large builds doing this at the same time can saturate its network and disk. At scale, send big files straight from the agent to an artifact repository or object storage, and pass only a reference between stages.

Workspace hygiene

• Clean workspaces when builds finish, or use ephemeral agents so this happens by design.
• Use unique workspace paths when several builds of the same job can run together, so they do not overwrite each other.
• Never leave secrets in a workspace that a later build could read.

Most of a build’s time is often spent fetching things, not building them.

Before a build can compile anything, it must download the source code and the libraries it depends on. On a fresh ephemeral agent, this happens every time. Across hundreds of builds, it adds up to hours of waiting and heavy network traffic. Caching attacks that waste.

The trade-off with clean builds

Caches make builds faster, but stale caches can hide problems: a build passes because of a leftover file that should not exist. Use caches for downloads and compiled dependencies, keyed by the exact versions, and keep the final build steps clean. Regularly run a full clean build, for example nightly, to confirm that nothing depends on hidden leftovers.

Making one slow build faster by running its parts side by side.

So far we have used many agents to run many builds at once. Distribution can also make a single build faster, by splitting its work into pieces that run on different agents at the same time. Jenkins pipelines do this with the parallel step.

flowchart TD
    B["Build stage finishes"] --> SP["Split test suite into four groups"]
    SP --> T1["Agent A: group 1"]
    SP --> T2["Agent B: group 2"]
    SP --> T3["Agent C: group 3"]
    SP --> T4["Agent D: group 4"]
    T1 --> J["Join: collect all results"]
    T2 --> J
    T3 --> J
    T4 --> J
    J --> G["Gate and report"]

Ways to split

• By kind of test: unit, integration and browser tests in separate branches.
• By operating system or version: the same tests on several platforms at once, often called a matrix.
• By test groups: divide a large suite into equal parts by past duration. A plugin for parallel test execution can compute balanced groups from previous run times.

Costs to keep in mind

How many executors do you really need?

A useful rule from queueing theory, known as Little’s law, says that the average number of things in a system equals the rate at which they arrive multiplied by the average time each one spends there. For builds, this gives a simple estimate of how many executors are busy on average.

flowchart LR
    A["Arrival rate: builds per hour"] --> M["Multiply"]
    B["Average build duration in hours"] --> M
    M --> C["Average executors busy"]
    C --> D["Multiply by peak factor of 2 to 3"]
    D --> E["Executors needed at peak"]
    E --> F["Compare with fleet size and ceiling"]

Beyond the average

The average tells you the minimum. Queues appear when arrivals bunch up. To keep waiting times low, plan for the peak, not the average, and track the real queue wait time. A useful goal is that most builds start within a minute of being queued.

Keeping the manager fast while the workers multiply.

As the fleet grows, the controller’s job grows too. It tracks more agents, more running builds, more logs streaming in, more history and more web requests. Its limits are memory, disk speed, network and the number of plugins and jobs it must manage.

Signs the controller is struggling

• The web interface is slow to load, even when few people use it.
• Builds start late even though agents are idle.
• Agents disconnect and reconnect at random.
• Long pauses in the controller’s memory cleanup.
• Disk usage or disk wait times climb steadily.

When one manager is no longer enough.

A single controller cannot grow forever. Its memory, its disk and the number of plugins and jobs it can handle are finite, and it is a single point of failure: if it stops, nobody builds. Very large organizations therefore run several controllers, each serving a group of teams.

flowchart TD
    ORG["Organization"] --> C1["Controller: payments teams"]
    ORG --> C2["Controller: mobile teams"]
    ORG --> C3["Controller: data teams"]
    C1 --> A1["Agent pool 1"]
    C2 --> A2["Agent pool 2"]
    C3 --> A3["Agent pool 3"]
    C1 -.-> SH["Shared services: artifact repository, package proxy, secret manager, monitoring"]
    C2 -.-> SH
    C3 -.-> SH

Making many controllers manageable

• Define everything as code. Use configuration as code so each controller can be recreated from files, and made identical.
• Standardize a base image with a tested set of plugins, and update it centrally.
• Share what can be shared: artifact storage, package proxies, secret managers and monitoring.
• Give teams ownership, within guardrails set by a platform team.

In a large fleet, some machine is always about to break.

With hundreds of agents, failures are not exceptions but routine: a machine runs out of disk, a network connection drops, a cloud provider reclaims a cheap instance, a container is killed for using too much memory. A reliable setup expects these events and limits the harm.

flowchart TD
    A["Agent lost during build"] --> B{"Cause"}
    B -->|"Machine or network problem"| C["Build marked failed or aborted"]
    B -->|"Cloud reclaimed instance"| C
    C --> D{"Retry allowed for agent loss?"}
    D -->|"Yes"| E["Retry on a new agent"]
    D -->|"No"| F["Report and alert"]
    E --> G["Build completes"]
    F --> H["Cleanup and investigate"]

Defenses

The controller as a single point of failure

The open-source controller is one active instance. Protect it with frequent backups of its home folder, configuration as code that can rebuild it quickly, and a tested recovery plan. Practice restoring it, because an untested backup is only a hope.

Scale should not mean an unpleasant bill.

Distributed builds turn engineering time into cloud spending. The goal is not the lowest bill but the best balance between developer waiting time and money. An hour of a developer’s time usually costs more than an hour of a build machine, so cutting wait times is often worth paying for.

Make cost visible

Tag every agent with the team or project it serves, and report the cost of builds per team. When teams see what their builds cost, they choose faster tests and smarter triggers without being told. Without visibility, costs grow quietly until a large bill forces a crisis.

More machines mean more places where things can go wrong.

Every agent runs code, some of it written by people you do not fully control. Every connection between agent and controller is a path that must be protected. Security in a distributed Jenkins rests on a few clear principles.

You cannot scale what you cannot see.

A distributed system fails in many small ways. Good monitoring turns vague complaints, such as “Jenkins feels slow,” into specific findings, such as “the Windows pool has been full since ten.” Watch the system at three layers: the queue, the agents and the controller.

Tools

Monitoring plugins can expose Jenkins numbers to systems such as Prometheus, and dashboards, meaning wall screens of graphs, make the health of the fleet visible to everyone. The cluster or cloud provider adds its own metrics for machines and containers. Combine them so that a queue spike on the Jenkins side can be matched to a provisioning delay on the cloud side.

Alerts that matter

• Queue wait above target for more than a few minutes.
• Agents offline beyond a threshold, or a pool with zero agents.
• Controller disk above a safe level.
• Provisioning failures from the cloud, which often mean a quota was reached.
• Sharp increase in aborted or lost builds.

Making the fleet reproducible instead of hand-crafted.

At scale, nobody can click through the web interface to set up every agent, cloud and setting. Two practices make large installations manageable: describing Jenkins itself in files, and building agents from prepared images.

Jenkins Configuration as Code

The Configuration as Code plugin lets you describe the controller’s settings, clouds, agent templates, security rules and credentials sources in text files stored in version control. Applying the files recreates the same configuration. A new controller can be built in minutes, changes are reviewed like any other code, and a mistake can be rolled back.

Image baking

An image is a saved snapshot of a machine or container with all its tools installed. Baking means preparing that image ahead of time, so that new agents start already complete, instead of installing tools at start-up. Baked images make agents start faster and behave identically every time.

flowchart LR
    R["Repository of image definitions"] --> B["Automated image build"]
    B --> T["Test the image"]
    T --> S["Scan for vulnerabilities"]
    S --> P["Publish to registry or image store"]
    P --> C["Cloud templates reference new image version"]
    C --> A["New agents start from it"]
    A --> M["Old images retired on a schedule"]

Proven ways of arranging a distributed Jenkins.

Habits that look fine at first and hurt later.

A checklist to apply to any Jenkins setup today.

Mistakes people make when they are new

Scaling machines before fixing the queue’s cause. Look at why builds wait before buying more capacity.

Ignoring start-up time. Ephemeral agents are clean, but a three-minute start-up on every build annoys people. Use warm pools and small images.

Forgetting the cloud quota. Autoscaling stops at the provider’s limit, and builds wait quietly. Watch quotas and request increases before you need them.

Underestimating log volume. Chatty builds add up. Reduce noise, or send logs elsewhere.

Skipping the recovery drill. A controller backup that has never been restored is not yet a backup.

An honest look at what distribution does well and where it strains.

Trade-offs to weigh

There is no universally right answer. The right design depends on your load pattern, your budget, your security needs and your team’s skills. Choose on purpose, measure the results, and write down why.

How the ideas look in the kinds of organizations that run Jenkins at scale.

Follow one imaginary team through three stages of growth.

At each step the team measured first and changed second. They never needed a heroic redesign. Distribution grew with their needs, one understood improvement at a time.

Symptoms, likely causes and calm fixes.

A calm debugging routine

1. Open the queue and read the reason each item waits.
2. Check the pool for the label: how many agents, how many free, how many offline.
3. Check the provider for provisioning errors and quotas.
4. Check the controller for memory, disk and response time.
5. Change one thing at a time and compare queue wait before and after.

Every important word in this guide in one place.

Short answers to the questions that come up most often.

The whole story in a few lines.

Distributed builds let Jenkins scale by separating thinking from doing. A small, well-protected controller schedules and records. Many agents, ideally short-lived containers or cloud machines, do the work. Labels route jobs to suitable machines, autoscaling matches capacity to demand, caches and artifact stores keep data moving quickly, and parallel splitting shortens slow builds. Simple arithmetic guides capacity, monitoring reveals bottlenecks, configuration as code and baked images keep everything reproducible, and careful separation of trust levels keeps it safe. The best setups grow one measured improvement at a time.