A plain-language first look before any details.

Jenkins is a free, open-source automation server that builds, tests and delivers software. Its work is written as a pipeline, a recipe stored in a text file called the Jenkinsfile. A pipeline is divided into stages, such as Build, Test and Deploy, and each stage contains steps, the single actions Jenkins performs.

An intermediate mistake is a habit that works well enough while a system is small, and quietly causes trouble as it grows. It is different from a beginner mistake in one important way: nothing breaks at once. The pipeline is green. The team is happy. The cost is deferred.

The mistakes fall into a few families: where work runs, how it is configured, how secrets are handled, how waiting and failure are treated, how the system is maintained and how people are given access. Each chapter takes one mistake and follows the same pattern, so you can read them in any order.

The plateau between “it works” and “it lasts.”

The plateau

Learning Jenkins has two steps. First you learn to make things work: create a job, write a pipeline, get a green build. That takes weeks. Then you must learn to make things last: keep them safe, fast, recoverable and understandable by other people. That takes years, mostly because the lessons arrive slowly, through incidents. Many people stop learning after the first step, because everything appears to be fine.

Common causes

A handful of small ideas that make every later chapter easier.

Blast radius

The blast radius of a mistake is how much damage it can do when it goes wrong. A mistake in one test job has a small blast radius. A mistake on the controller has a large one, because everything depends on it. Good habits keep blast radii small.

Configuration drift

Drift is what happens when things that should be identical slowly become different, such as two agents with different tool versions, or a job whose settings were edited by hand until nobody knows what it does.

Single point of failure

A single point of failure is one component whose failure stops everything. The Jenkins controller is one. Knowing this makes backups and protection of the controller a priority.

Reproducibility

A build is reproducible if running it again on the same code gives the same result. Floating versions, unpinned tools and leftover files break reproducibility.

Least privilege

Least privilege means giving each person, job and machine only the access it needs. It limits what a mistake or an attacker can reach.

Controller and agent

The controller is the main Jenkins server that schedules work and stores settings. Agents are the worker machines that run builds.

Six layers of a Jenkins setup, each with its own typical mistakes.

flowchart TD
    P["People and process: permissions, reviews, ownership"] --> C["Controller: disk, plugins, backups, security"]
    C --> A["Agents: labels, executors, drift, cleanup"]
    A --> PL["Pipeline code: scripts, shell steps, errors, timeouts"]
    PL --> S["Secrets and credentials: storage, printing, scope"]
    S --> D["Delivery: triggers, pull requests, deployments"]
    D --> O["Observability: logs, metrics, alerts"]

Read the diagram from top to bottom. People and process decide who can do what. The controller holds settings and history. Agents run the work. Pipeline code describes it. Secrets let it reach outside systems. Delivery connects it to source code and production. Observability lets you see it all. Mistakes tend to cluster by layer, and the layers depend on each other: careless permissions (top) can lead to a leaked secret (middle) which leads to a production incident (bottom).

The most common mistake, and one of the most dangerous.

What it looks like

A new Jenkins installation can run builds on the controller itself. It is convenient: no agents to set up. Many teams never change this, and builds keep running on the same machine that holds all the settings, history and secrets.

Why it hurts

flowchart LR
    B["Build code runs on the controller"] --> R1["Can read credentials and settings"]
    B --> R2["Can use the controller's memory and processor"]
    B --> R3["Can fill the controller's disk"]
    R1 --> X1["Secret theft by hostile or careless code"]
    R2 --> X2["Slow web pages and stuck queue"]
    R3 --> X3["Controller crash and no builds for anyone"]

• Security: build code runs next to every secret. A hostile pull request, a compromised dependency or a careless script can reach files that should be protected.
• Performance: heavy builds compete with the software that schedules everything, so the web page slows and queues stall.
• Stability: a runaway build can fill the disk or exhaust memory, and the controller goes down with it.

The fix

Set the number of executors on the built-in node to zero, and run everything on agents. Even one small agent is better than none. Agents can be simple virtual machines or short-lived containers.

Clicking in the web page is easy. Reproducing it is not.

What it looks like

A job is created through the web form. A setting is tweaked. Another is added a month later by someone else. A plugin option is changed during an incident. After a year, the job works, but nobody can say exactly why, or recreate it if it were lost.

Why it hurts

• Drift: two jobs that should be alike are subtly different.
• No history: nobody knows who changed what, or why.
• No review: changes bypass the checks that code gets.
• No rebuild: if the controller is lost, the configuration is lost with it.

flowchart LR
    subgraph HAND["Configured by hand"]
        H1["Click in web page"] --> H2["Stored only on controller"]
        H2 --> H3["Unknown history, hard to copy or restore"]
    end
    subgraph CODE["Configured as code"]
        C1["Change in files"] --> C2["Reviewed and versioned"]
        C2 --> C3["Applied automatically, easy to rebuild"]
    end

The fix

A secret that is easy to reach for you is easy to reach for others.

What it looks like

• A token pasted into the Jenkinsfile, or into a shell script in the repository.
• A password saved as an ordinary job parameter, visible on the build page.
• A secret printed to the log “just to check that it arrived.”
• One shared credential used everywhere, with wide permissions.
• A secret placed into a command using double quotes, so Jenkins bakes the real value into the command text.

Why it hurts

Repositories keep every version forever, so a secret removed today still lives in history. Logs are readable by many people and often copied into tickets and chat. Shared broad credentials turn one leak into a total compromise. And masking in logs is only a safety net: it hides exact copies of a bound secret, not changed versions or file contents.

A build with no time limit is a build that may never end.

What it looks like

Pipelines have no timeout anywhere. A test waits for a network answer that never comes. A deployment tool asks a question nobody sees. An approval prompt sits unanswered over a long weekend. Nothing turns red. The build simply keeps running.

Why it hurts

• Each hung build holds an executor, a work slot on an agent. A handful of them can freeze a small fleet.
• On cloud machines billed by time, hangs cost real money.
• Nobody is alerted, because nothing has failed.
• Approvals left open for days may be answered later, deploying stale code.

The fix

A machine slot occupied by a build that is doing nothing.

What it looks like

An approval prompt, a long sleep or a wait for an outside event is written inside an agent block. The build holds an executor for the entire wait. Because agents have a fixed number of executors, waiting builds quietly reduce the capacity available to everyone else.

flowchart LR
    subgraph BAD["Executor held while waiting"]
        B1["Agent allocated"] --> B2["Build and test"]
        B2 --> B3["Wait for approval, executor idle"]
        B3 --> B4["Deploy"]
    end
    subgraph GOOD["Executor released while waiting"]
        G1["Agent allocated"] --> G2["Build and test, stash outputs"]
        G2 --> G3["Agent released"]
        G3 --> G4["Wait for approval with no agent"]
        G4 --> G5["New agent allocated"]
        G5 --> G6["Deploy"]
    end

Why it hurts

With ten executors and three builds waiting for approval, a third of your capacity is idle but unavailable. Queues grow while machines do nothing. The problem is invisible on a quiet system and painful on a busy one.

The fix

• Place approvals and long waits in their own stage with no agent, or use the declarative input directive, which pauses before an agent is allocated.
• Before releasing the agent, save any files you need, using stash for small files or an artifact repository for large ones.
• Always give the wait a timeout.
• Allocate a fresh agent afterwards for the steps that follow.

The pipeline script runs on the controller, not on the agent.

What it looks like

Pipelines that parse large files, loop over thousands of items, build long strings and do complex calculations in the pipeline’s own script language. It works on a quiet day. It is slow, uses a lot of memory, and gets worse the more pipelines run at once.

Why it hurts

The commands you run through steps execute on the agent. But the script that decides what happens next runs on the controller, inside a special engine that can pause and resume so that pipelines survive restarts. This engine is slower than ordinary code and uses controller memory. A few pipelines doing heavy work in it can slow every other pipeline.

flowchart LR
    subgraph CT["Controller"]
        SC["Pipeline script logic"]
    end
    subgraph AG["Agent"]
        CM["Commands and tools"]
    end
    SC -->|"short instructions"| CM
    CM -->|"short results"| SC
    HV["Heavy loops and parsing"] -.->|"wrong place"| SC
    HV -.->|"right place"| CM

Related traps

• Objects that cannot be saved. Because pipelines save their progress, some kinds of objects, such as certain iterators or regular-expression matchers, cannot be stored, and holding one across a step causes a “not serializable” error.
• Behavior that differs from ordinary code. Some familiar language features work differently inside the pipeline engine. Test unusual constructs before relying on them.
• Special helper functions. Jenkins lets you mark a function so that it runs as ordinary code, which is faster, but such a function cannot call pipeline steps.

Each shell step is a fresh, short-lived world.

Most pipelines are mostly shell steps. Understanding how they behave prevents a family of confusing bugs.

Long scripts

When a shell step grows beyond a handful of lines, it becomes hard to read and impossible to test outside Jenkins. Move it into a script file in the repository, run it as a single command and test it on a laptop. The pipeline stays short, and the script can be reused and reviewed as normal code.

A green build that hides a problem is worse than a red one.

What it looks like

• Wrapping steps so failures are ignored, then forgetting why.
• Treating yellow (unstable) builds as if they were green.
• Using retries to make failures disappear, without looking at why they happen.
• Alerts that listen only to failures, so aborted and unstable runs pass in silence.
• Reading the color of a build instead of reading the reason.

The results, explained

flowchart TD
    E["A step fails"] --> Q{"Is the failure expected and harmless?"}
    Q -->|"No"| F["Let the build fail and alert"]
    Q -->|"Yes, optional check"| S["Mark unstable, keep going, alert owners"]
    Q -->|"Yes, known flaky"| R["Retry with a limit, count every retry, track the flake"]
    S --> D["Decide: can unstable builds be released?"]
    R --> T["Fix or quarantine the flaky part"]

The fixes

• Use error-catching wrappers deliberately. The step that catches an error and continues can set a chosen result, such as unstable. Use it only for genuinely optional checks, and leave a comment saying why.
• Decide what yellow means. Either block releases from unstable builds, or make yellow trigger a visible alert with an owner.
• Limit retries. Retry only the specific operation known to be flaky, cap the number of attempts and report each retry.
• Alert on unsuccessful. The post section’s unsuccessful condition covers failure, unstable and aborted together.

The end-of-run block is powerful and often misplaced.

The post section runs when a stage or pipeline ends and reacts to the result. It has named conditions such as always, failure, success, unstable, aborted, changed, fixed, regression and cleanup, run in a fixed order with cleanup last. Intermediate users often use it in ways that cause trouble.

Yesterday’s files quietly affect today’s build.

A workspace is the folder on an agent where a build downloads code and writes output. On long-lived agents, workspaces are reused, and whatever a previous build left behind is still there. That is sometimes useful, as a cache. It is also a source of the most confusing failures in Jenkins.

The fixes

• Prefer ephemeral agents, such as containers created for one build, so every build starts clean by design.
• Clean at the end of builds that handled sensitive data, using the cleanup condition of the post section.
• Clean at the start when you need certainty and can afford the re-download.
• Run a fully clean build on a schedule, to prove that nothing depends on leftovers.
• Remove old workspaces automatically, and take agents offline when disk space runs low.

If the ingredients change silently, the recipe changes silently.

What it looks like

The pipeline uses whatever version of a tool, image or library is the newest today. A container image is referred to by the tag latest. A dependency has a floating version range. A shared library is loaded from the moving tip of a branch. A plugin is updated whenever someone clicks the button. Everything works, until the morning when an upstream project publishes a change and your unchanged code suddenly fails.

The two-sided lesson

Pinning makes builds reproducible, but pinned things go stale and miss security fixes. The healthy pattern is pin and update on purpose: keep exact versions, and have a regular, tested routine for moving them forward, ideally automated with a tool that proposes updates as reviewable changes.

Plugins are Jenkins’ strength and its most common source of trouble.

What it looks like

Over the years, every problem is answered with another plugin. Nobody removes old ones. Updates are applied all at once on the production controller, on a Friday, without a copy to test on. When something breaks, nobody knows which plugin caused it.

Why it hurts

• Every plugin adds memory use, start-up time and security surface.
• Plugins depend on each other, and one update can force others.
• Abandoned plugins stop receiving security fixes.
• Untested updates can break jobs, and reversing them is hard.

flowchart TD
    A["Update available"] --> B["Read release notes and security advisories"]
    B --> C["Take a fresh backup"]
    C --> D["Apply on a test controller with copied configuration"]
    D --> E["Run sample and critical pipelines"]
    E --> F{"All good?"}
    F -->|"Yes"| G["Schedule update on production in a quiet window"]
    F -->|"No"| H["Hold back, report problem, plan alternative"]
    G --> I["Verify, keep rollback plan ready"]

Healthy plugin habits

The controller is a single point of failure, and it holds everything.

What it looks like

Either nobody has thought about backups, or a backup job exists but has never been restored. The Jenkins home folder sits on one disk. When the disk fails, or an upgrade goes wrong, the team discovers how much lived only there: job definitions, credentials, build history, keys, plugin settings.

flowchart LR
    S["Schedule regular backup of Jenkins home"] --> E["Encrypt and store off the machine"]
    E --> T["Regularly restore to a test controller"]
    T --> V["Verify jobs, credentials and agents work"]
    V --> R{"Restore works?"}
    R -->|"Yes"| D["Record time taken and update runbook"]
    R -->|"No"| F["Fix the backup and repeat"]
    F --> S

What to protect

The habits

• Back up regularly and store copies away from the controller machine.
• Protect backups carefully. A backup contains the vault and its key; encrypt it and restrict who can read it.
• Practice the restore at least twice a year, and time it. An untested backup is a hope, not a plan.
• Reduce what needs backing up by putting configuration and pipelines in code.

Where a build runs matters as much as what it does.

Everything you keep costs disk, memory and load time.

By default, Jenkins keeps every build of every job forever, along with its logs and archived files. On a busy system this grows without limit, until the controller’s disk fills, pages load slowly and start-up takes many minutes.

Put the options in one place

Because these options are per job, teams forget them on new jobs. The reliable fix is to include them in a shared template or library, so every new pipeline starts with sensible history limits, timestamps and a timeout without anyone remembering.

How builds get started matters as much as what they do.

The hash symbol

Jenkins schedules use a syntax similar to the classic cron format. In it, a special hash symbol in place of a fixed number means “pick a value for this job, spread across the range.” Instead of every nightly job starting at exactly two o’clock, they start at various times across the hour. It is a tiny change that prevents a daily traffic jam.

Reuse is good. Uncontrolled copying and uncontrolled sharing are both problems.

The copy-paste problem

A working Jenkinsfile is copied into fifty repositories. Each copy is then edited slightly. A year later, no two are the same, a security fix must be applied fifty times, and new joiners cannot tell which version is right.

The shared library answer, and its traps

A shared library is a reusable module of pipeline code that many Jenkinsfiles can call. It solves duplication, but brings its own mistakes.

The healthy path

• Treat the library as a product: version it, test it, document it and release it with notes.
• Let projects pin to a released version and upgrade deliberately.
• Keep the functions thin, delegating heavy work to commands on agents.
• Provide a few opinionated standard pipelines rather than infinite options.

When the build recipe lives in the code, whoever writes the code writes the recipe.

What it looks like

A public or shared repository builds every pull request automatically, including those from copies of the repository owned by outsiders. The pipeline has access to credentials and to internal systems. Everything works, and nobody thinks about what a malicious pull request could do.

Why it hurts

A pull request can change the Jenkinsfile. If Jenkins runs the changed file with access to secrets, an outsider can print them, send them elsewhere or use them to reach internal systems. Even well-meaning contributors can accidentally leak secrets through debug output.

flowchart TD
    PR["Pull request opened"] --> W{"From the same repository?"}
    W -->|"Yes"| TR["Trusted: full pipeline"]
    W -->|"No, from a fork"| TP{"Author trusted by policy?"}
    TP -->|"Yes"| TR
    TP -->|"No"| UT["Untrusted: use target branch Jenkinsfile"]
    UT --> NS["No secrets, isolated disposable agent, tests only"]
    TR --> ST["Status reported to pull request"]
    NS --> ST

The habits

Access is granted in a hurry and revoked never.

A red build that means nothing teaches everyone to ignore red.

What it looks like

A flaky test sometimes passes and sometimes fails on the same code. It might depend on timing, on shared data or on an outside service. Teams learn to press “rebuild” until the build is green, and eventually add automatic retries that hide the problem completely.

Why it hurts

• Real failures hide among the noise, because everyone assumes a red build is probably flaky.
• Reruns waste capacity and time, often doubling the cost of a pipeline.
• Trust in the pipeline erodes, and people begin to merge on red.
• Silent retries hide genuine intermittent bugs in the product itself.

Habits that work

• Track which tests fail intermittently, and show the list to the team.
• Give each flaky test an owner and a deadline, or remove it.
• Replace fixed sleeps in tests with waits for a condition, and make tests independent of each other’s data.
• Report every retry in the log and in a metric, so retries never become invisible.

The path to production should be narrow, explicit and recorded.

What it looks like

• A deploy stage without any branch condition, so any branch can deploy.
• Deployment conditions written as “run unless it is a pull request,” so a new kind of branch reaches production by accident.
• Deployment credentials available to every build, including builds of outside pull requests.
• No approval before production, and no record of who released what.
• Approval prompts that wait forever, so stale builds can be approved days later.
• Deployment steps that are not safe to run twice, followed by blind retries.

flowchart LR
    C["Code pushed"] --> Q{"Build type"}
    Q -->|"Pull request"| PR["Build and test only, no secrets"]
    Q -->|"Feature branch"| FB["Build and test only"]
    Q -->|"Main branch"| MB["Build, test, deploy to staging"]
    MB --> AP["Approval by named group with timeout"]
    AP --> PROD["Deploy to production with scoped credentials"]
    PROD --> REC["Record who, what and when"]

The habits

If you cannot see the system, you will learn about problems from your users.

What it looks like

Nobody watches the queue, the disk, the controller’s memory or the number of failed builds. Problems are discovered when somebody complains. Logs are enormous, full of noise and progress bars, so real errors are hard to find, and secrets have more chances to slip in.

Log hygiene

• Use quiet modes and disable progress bars in routine builds.
• Write verbose details to a file, and archive it only when a build fails.
• Turn debug output on only while investigating.
• Keep timestamps on, and print short summaries with links to reports.
• Never print environment dumps that may contain secrets.

A quick review you can do in an afternoon.

flowchart LR
    L1["Level 1: Works: builds run and go green"] --> L2["Level 2: Safe: no builds on controller, secrets in the store, least privilege"]
    L2 --> L3["Level 3: Reliable: timeouts, cleanup, backups tested, pinned versions"]
    L3 --> L4["Level 4: Reproducible: everything as code, agents from images"]
    L4 --> L5["Level 5: Observable and improving: metrics, alerts, regular reviews"]

Instead of remembering twenty rules, build a few habits into the system.

Context decides. A shortcut is only a mistake if you did not choose it.

Not every item in this guide is wrong in every situation. A personal test installation on a laptop may run builds on the controller. A prototype may have hand-made jobs. What turns a shortcut into a mistake is not knowing you took it, or not noticing when the situation changes.

Write down the shortcuts you have chosen and why, with a date to revisit them. A documented shortcut is a decision. An undocumented one is a future incident.

How these mistakes show up in the kinds of organizations that run Jenkins at scale.

Follow one imaginary team through a review of their own Jenkins.

A team of forty developers runs Jenkins that “just works.” After a scare, a colleague named Priya proposes a small review using the self-audit table. It takes an afternoon and finds many “no” answers.

None of these steps needed new products or a large budget. They needed an afternoon of honest review and a few weeks of small, ordered fixes. The team’s Jenkins looked exactly the same from the outside, and behaved very differently on the days that mattered.

When something goes wrong, this table points at the habit behind it.

A calm debugging routine

1. Describe the symptom precisely, including when it started.
2. Ask what changed: code, pipeline, plugin, agent image, upstream version.
3. Check the layer: controller, agent, pipeline code, secrets, delivery or people.
4. Read the log with timestamps to find the exact moment things diverged.
5. Fix the cause and add the habit to the template or the audit, so it cannot return.

Every important word in this guide in one place.

Short answers to the questions that come up most often.

The whole story in a few lines.

Intermediate Jenkins mistakes are quiet shortcuts that work today and cost more tomorrow: builds on the controller, hand-made configuration, secrets in the wrong places, missing deadlines and cleanup, executors held during waits, heavy scripts on the controller, swallowed errors, floating versions, plugin sprawl, untested backups, loose permissions, trusting outside pull requests and deployments from anywhere. They share one cure: make good habits automatic through templates, code, guardrails and regular reviews, and fix the largest blast radius first.