Backup Strategy Basics in Jenkins
Your build server holds years of settings, history, and secrets. Learn what to copy, how often, where to keep it, and how to prove that you can get everything back on the day something goes wrong.
Picture a small bakery with a wall of handwritten recipe cards. Every card was refined over years: a pinch more salt here, ten minutes less in the oven there. One night a pipe bursts, and the cards turn to pulp. The ovens survived, the flour survived, but the knowledge is gone. Rebuilding from memory takes months, and some recipes never taste the same again.
A Jenkins server is that wall of recipe cards. It holds every job definition, every setting, every stored password, every plugin choice, and every build record that the team has built up over years. The machine itself can be replaced in an hour. What sits on its disk cannot. A backup strategy is the photocopy of the cards, kept in another building, checked regularly to be sure the copies are readable.
In this guide you will learn what a Jenkins backup strategy is, what lives inside Jenkins that must be saved, what can safely be skipped, which methods exist, how to decide how often to back up, how to protect the copies, and, most importantly, how to test that a restore really works. We assume you know nothing about Jenkins operations yet, and we explain each new word the first time it appears.
1What Is a Jenkins Backup Strategy?
A backup strategy is a plan that answers four questions: what to copy, how often, where to keep the copies, and how to get everything back.
First, the basics
Jenkins is a free automation server that builds, tests, and delivers software. Its main server is called the controller. The controller stores everything it knows in a folder on disk, called the Jenkins home directory, often written JENKINS_HOME. Almost everything worth saving lives there.
What is a backup?
A backup is a copy of data, stored separately from the original, that you can use to recover if the original is lost or damaged. The word “separately” matters. A copy on the same disk is not a backup, because the same failure destroys both.
What is a strategy?
A strategy is more than a single copy. It is a written, repeatable plan that says which data is protected, how often it is copied, how long copies are kept, where they are stored, who is responsible, how they are protected, and how the team proves that a restore works. Without a plan, backups tend to be forgotten, incomplete, or unreadable exactly when needed.
Think of home insurance. Buying a policy is not enough. You must know what is covered, how much you would receive, how to file a claim, and where the paperwork is. A backup strategy is the whole policy, not just the piece of paper.
Why the word “restore” is more important than “backup”
Nobody wants a backup. People want a restore: the ability to bring Jenkins back to a working state after a problem. A backup that cannot be restored is worth nothing. So a good strategy is judged by one test: how quickly and completely can you recover?
You do not have a backup until you have restored from it at least once. Until then, you have a hopeful copy.
2Why It Matters: What Can Go Wrong
Jenkins servers rarely fail in dramatic ways. They fail in ordinary, boring ways, and those are the ones you must plan for.
The history of a growing server
Most Jenkins servers start small. A developer installs Jenkins on a spare machine to build one project. Over time, more teams arrive, more jobs are created, credentials pile up, and plugins are added. Slowly the server becomes something the whole company depends on. But nobody ever decided that it had become critical, so the protections that a critical system deserves were never added.
Common ways to lose data
Disk failure
The disk holding Jenkins home fails or the virtual machine is deleted. Without a copy elsewhere, everything is gone.
Accidental deletion
Someone deletes a folder of jobs, or a whole job, while tidying up. It happens more often than anyone admits.
Bad upgrade
A plugin or Jenkins upgrade breaks jobs or corrupts settings, and the old version cannot simply be reinstalled.
Wrong configuration
An administrator changes a global setting or security realm and locks everyone out.
Ransomware or intrusion
An attacker encrypts or deletes the server’s data, sometimes including the backups if they are within reach.
Site or region outage
A data centre or cloud region becomes unavailable for hours or days.
The cost of losing Jenkins
When Jenkins disappears, releases stop. Nobody can build, test, or deploy. Rebuilding by hand means recreating hundreds of jobs from memory, re-entering every credential, and rediscovering which plugin versions worked together. Teams that have lived through this describe weeks of lost productivity and a permanent loss of build history that auditors may have wanted to see.
Without a backup strategy
- Recovery means rebuilding from memory.
- Credentials must be re-created and re-distributed.
- Build history and audit evidence are lost.
- Nobody knows how long recovery would take.
With a tested strategy
- Recovery follows a written, rehearsed procedure.
- Configuration and credentials return intact.
- History is kept for as long as you decided.
- Recovery time is known in advance.
A fire extinguisher on the wall is boring right up to the day it is priceless. Nobody schedules a fire, and nobody schedules a disk failure. Backups, like extinguishers, are bought for the day you hope never comes.
3Core Concepts: The Words You Must Know
A handful of ideas explain almost every backup decision.
Recovery Point Objective
Often shortened to RPO. It is the largest amount of data loss you can accept, measured in time. An RPO of one day means that losing up to a day of changes is acceptable.
Recovery Time Objective
Often shortened to RTO. It is the longest time you can afford to be without Jenkins while you recover. An RTO of four hours means service must return within four hours.
Full backup
A complete copy of everything chosen. Simple to restore but large and slow to make.
Incremental backup
A copy of only what changed since the last backup of any kind. Small and fast, but restoring needs the whole chain.
Differential backup
A copy of what changed since the last full backup. Larger than incremental, but restoring needs only two pieces.
Snapshot
A frozen picture of a disk or volume at one moment, created by the storage system very quickly.
Retention
How long each backup is kept before it is deleted to save space.
Offsite copy
A copy kept in a different location, so that one fire, flood, or account compromise cannot destroy every copy.
Vocabulary table
| Word | Meaning | Example |
|---|---|---|
| Restore drill | A planned practice of recovering from a backup into a test environment | Quarterly rebuild of Jenkins in a scratch network |
| Immutable backup | A backup that cannot be changed or deleted for a set time, even by administrators | Object storage with a retention lock |
| Quiet down | A Jenkins state where no new builds start, used before maintenance | Preparing for a consistent copy |
| Runbook | A written, step-by-step recovery procedure | The page an on-call engineer opens at night |
| Checksum | A short fingerprint used to detect that a file was damaged | Verifying that a backup file is intact |
The 3-2-1 rule
A famous rule of thumb says: keep three copies of important data, on two different kinds of storage, with one copy offsite. It is simple, and it protects against the most common causes of total loss. We come back to it in the goals chapter.
RPO and RTO are business questions before they are technical ones. Ask the teams that depend on Jenkins how much loss and downtime they can live with, then design the strategy to meet those numbers.
4Architecture and Components
A Jenkins installation has several parts. Only some of them need backing up.
The players
The controller is the Jenkins brain, and its Jenkins home directory holds the state. Agents are worker machines that run builds and are usually disposable. The source repository holds your code and Jenkinsfiles, and has its own protection. The artifact store keeps built packages. The backup target is the separate place where copies are stored, ideally in more than one location.
flowchart LR
SCM["Source Repository with Jenkinsfiles"] --> CTRL["Jenkins Controller"]
CTRL --> HOME["Jenkins Home Directory"]
CTRL --> AG["Disposable Agents"]
AG --> ART["Artifact Store"]
HOME -->|"Scheduled copy"| B1["Local Backup Storage"]
B1 -->|"Replicate"| B2["Offsite or Cloud Storage"]
B2 --> IMM["Locked or Immutable Copy"]
CFG["Configuration as Code Files"] --> SCM
HOME -.->|"Restore path"| NEW["Replacement Controller"]
B2 -.->|"Restore path"| NEW
Reading the diagram
Source code and Jenkinsfiles live in the source repository, which is protected by its own backups. The controller keeps its state in the Jenkins home directory. A scheduled process copies that state to local backup storage, then replicates it to an offsite location, with one copy locked so it cannot be altered. Configuration files kept as code can rebuild much of the setup. If the controller is lost, a replacement controller is started and the backups fill it with the saved state.
What lives where
| Part | Holds | Needs its own backup? |
|---|---|---|
| Controller home directory | Settings, jobs, credentials, history, plugins | Yes, this is the main target |
| Agents | Temporary workspaces and tools | No, rebuild them from images or code |
| Source repository | Code and Jenkinsfiles | Yes, but through the repository’s own backup |
| Artifact store | Built packages | Yes, through its own backup |
| Vault or secret store | External secrets, if used | Yes, through its own backup |
In a restaurant, the recipe book and the customer reservations are irreplaceable. The pots and pans can be bought again. A backup strategy decides which things go into the fireproof safe and which are left on the shelf.
5The Anatomy of the Jenkins Home Directory
Before you can decide what to back up, you must know what is inside the folder. Think of it as a tour of the recipe cabinet.
Where is it?
The location depends on how Jenkins was installed. On many Linux installations it is under a system folder for Jenkins data. In containers it is usually a mounted volume. Inside Jenkins, the system information page shows the exact path. Find it first, and write it down.
What is inside
| Item | What it holds | Importance |
|---|---|---|
| Main configuration files (top-level XML files) | Global settings, security setup, plugin settings, credential store | Critical |
| jobs folder | One folder per job, each with a configuration file and, usually, build history | Critical (configuration), optional (history) |
| secrets folder | Encryption keys that unlock stored credentials | Critical, and highly sensitive |
| users folder | User accounts, API tokens, preferences | Important |
| nodes folder | Definitions of agents | Important |
| plugins folder | Installed plugin packages and their unpacked files | Important, but can be reinstalled from a list |
| userContent folder | Files that users published through Jenkins, such as logos | Useful |
| fingerprints folder | Records used to track which build produced which file | Optional |
| workspace folder | Temporary copies of code and build output | Skip |
| caches, war, and logs | Regenerated or disposable material | Skip in most cases |
Inside a job folder
Each job has its own folder. The job’s configuration file says what the job does and how it is triggered. The builds folder holds one subfolder for every run, with the log, the results, and any archived files. For pipeline jobs from repositories, the Jenkinsfile lives in the repository, and Jenkins stores only the settings that point to it.
Why size varies so much
The configuration parts are tiny, often a few megabytes even for hundreds of jobs. The heavy parts are build history and archived artifacts, which can reach hundreds of gigabytes on an old, busy server. This difference is central to strategy: you can back up the small, critical parts very frequently, and the large parts less often, or not at all.
Think of a filing cabinet. The top drawer holds the contracts and keys, small and priceless. The bottom drawers are full of old receipts, bulky and rarely opened. You photocopy the top drawer every week, and the bottom drawers once in a while, if at all.
Do not guess what is inside. Look at your own Jenkins home directory, list the biggest folders, and note which ones grow. Surprises are common, such as one job that archives huge files and fills the disk.
6What to Back Up and What to Skip
Backing up everything is simple but wasteful. Backing up too little is dangerous. A sensible middle path protects what is hard to recreate.
Sort by how hard it is to recreate
Configuration
Global settings, job configurations, node definitions, users, and plugin settings. Recreating them by hand takes weeks.
Secrets and credentials
Stored credentials and the keys that decrypt them. Without the keys, the credential file is unreadable.
Build history
Logs and results of past runs, useful for audits and trend analysis. Keep as long as your rules demand.
Plugin list and versions
A record of exactly which plugins and versions were installed, so the same set can be rebuilt.
Archived artifacts
Built packages stored inside Jenkins. Better kept in a dedicated artifact store with its own backup.
Workspaces and caches
Temporary, huge, and regenerated by the next build.
flowchart TD
S["Look at an item in Jenkins home"] --> Q1{"Can it be recreated automatically from code or a list"}
Q1 -->|"Yes, such as plugins or agents"| R["Record the list, skip the files"]
Q1 -->|"No"| Q2{"Is it temporary or regenerated by builds"}
Q2 -->|"Yes, such as workspaces and caches"| SK["Skip it"]
Q2 -->|"No"| Q3{"Is it sensitive, such as keys and credentials"}
Q3 -->|"Yes"| ENC["Back up encrypted with strict access"]
Q3 -->|"No"| BK["Back up on the normal schedule"]
Reduce what you need to save
- Discard old builds. A build discarder setting keeps only the last number of builds, or those from the last number of days. Smaller history means smaller backups.
- Move artifacts out. Store packages in an artifact repository, not in Jenkins.
- Keep configuration as code. Jenkinsfiles and configuration files in a repository are automatically protected by the repository’s own history.
- Clean workspaces. Remove old workspaces on agents and on the controller.
Do not skip by accident
Some teams exclude the builds folder to save space, and later discover that the folder also contained information they needed for audits. Decide on purpose, write the decision down, and get agreement from the people who rely on the history.
When you pack for a house move, you carry the passport and the family photos yourself, and you let the movers handle the furniture. Sorting by irreplaceability is the first step of any sensible backup.
7Secrets and Credentials: The Special Case
The most valuable and most dangerous data in your backup is the set of stored passwords and keys.
How Jenkins stores secrets
Jenkins keeps stored credentials in a configuration file, encrypted. The keys that decrypt them live in the secrets folder in the same home directory. Jenkins reads the keys when it starts and uses them to unlock credentials for builds.
The two-sided problem
If you forget the keys
- You restore the credential file, but it cannot be decrypted.
- Every stored password appears broken.
- Every credential must be re-entered by hand.
If you store keys carelessly
- Anyone who reads the backup can decrypt every credential.
- A copy in an open storage bucket becomes a breach.
- Keys next to the encrypted file defeat the encryption.
How to handle it
Include the secrets folder in the backup
Without it, the credential file cannot be used. Test this in a restore drill.
Encrypt the backup itself
Use strong encryption for the whole backup, with the backup key kept in a separate, well-protected place.
Restrict who can read backups
Only a few named people or systems may open them. Log every access.
Keep them offsite in a protected account
Use a separate storage account or project whose credentials are not stored in Jenkins.
Reduce the secrets you keep
Move important credentials to an external secret manager. Then the backup holds fewer real secrets.
External secret stores
If your Jenkins fetches secrets on demand from a tool such as Vault, the backup of Jenkins contains only a small login, not the real passwords. The secrets are then protected by the backup of that secret store instead. That is a strong design, but remember that the secret store now needs its own tested backup.
Never place an unencrypted backup of the Jenkins home directory in a shared folder, a public bucket, or an email attachment. It contains the keys to your build and deployment systems.
A locked box is useless if the key is taped to its lid. A copy of the box without its key is equally useless. A backup of secrets needs the box, the key, and a safe place for the key, all handled with care.
8The Main Backup Methods at a Glance
There is no single best way. Most solid strategies combine two or three methods.
| Method | How it works | Strength | Weakness |
|---|---|---|---|
| Backup plugin | A Jenkins plugin copies configuration on a schedule and can restore it through the web page | Easy to start, understands Jenkins | Depends on Jenkins running; usually protects configuration more than everything |
| File-level copy | A scheduled script or tool copies the home directory to another place | Simple, flexible, works with any storage | Needs care for consistency and security |
| Disk or volume snapshot | The storage system freezes the disk at one moment | Very fast, captures everything at once | Often stays inside the same platform; needs offsite copies too |
| Configuration as code | Settings, jobs, and plugin lists live in a repository, so Jenkins can be rebuilt from files | Fast rebuilds, reviewed history, no drift | Does not cover build history or secrets by itself |
| Machine image backup | The whole virtual machine is imaged | Restores everything, including the operating system | Heavy, slow, and coarse |
Which combination is common?
A popular and sturdy combination is: configuration as code for fast rebuilds, plus scheduled file-level or snapshot backups of the home directory for history and credentials, plus an offsite, locked copy. Each layer covers a weakness in another.
Code in a repository
Jenkinsfiles, configuration files, and plugin lists. Rebuild the shape of Jenkins in minutes.
Frequent state backup
Small, regular copies of configuration and credentials, so recent changes are safe.
Full periodic backup
A larger copy that includes history, made daily or weekly.
Offsite locked copy
A copy in another place that attackers and accidents cannot alter.
A careful traveller carries a photocopy of the passport in the suitcase, a scan in email, and a photo in the phone. Each method has weaknesses, but together they cover the one bad day.
Whatever methods you choose, make sure the backup process does not depend only on Jenkins itself. A backup job that runs inside Jenkins stops the moment Jenkins fails, which is exactly when you need it.
9Backup Plugins: ThinBackup and Periodic Backup
Two well-known plugins make it easy to start. Understand their strengths and limits before you trust them.
ThinBackup
The ThinBackup plugin creates scheduled backups of Jenkins configuration. It can make full backups and differential backups on separate schedules, keep a chosen number of backup sets, and optionally include build results. You can start a restore from a Jenkins page. It focuses on configuration files, which is why it is called “thin,” and it does not replace a full backup of everything.
Periodic Backup
The Periodic Backup plugin makes scheduled copies of the Jenkins home directory. You choose what to include and exclude through patterns, how to package the copy, such as a compressed archive, and where to store it, for example in a local folder, with options to send it elsewhere. It offers a restore option in the web page and can be used to keep several generations of backups.
Comparison
| Question | ThinBackup | Periodic Backup |
|---|---|---|
| Main focus | Configuration files | Broader home directory copy with include and exclude patterns |
| Schedule types | Full and differential schedules | A single schedule for archives |
| Restore in the UI | Yes | Yes |
| Offsite storage | Copy the backup folder elsewhere yourself | Storage options exist, or copy elsewhere yourself |
| Best for | Quick protection of settings | Scheduled archives of Jenkins home |
Limits of any backup plugin
- Same server: plugins usually write to a folder on the same machine unless you send it away. A copy on the same disk is not a backup.
- Same process: the plugin runs inside Jenkins. If Jenkins will not start, there is no schedule and no restore button.
- Maintenance: plugins vary in how actively they are maintained. Check the plugin’s page and recent releases before relying on it, and do not use deprecated backup plugins.
- Secrets: make sure the secrets folder is included, and that the destination is protected.
Some older configuration-syncing plugins have been deprecated or left unmaintained. Always check a plugin’s current status in the plugin manager before choosing it for something as important as backups.
A backup plugin is like a photocopier that sits in the same room as the documents. It makes copying easy, but if the room floods, both the originals and the copies are lost. You still need to carry a copy to another building.
10File-Level Backups with System Tools
Copying the home directory with ordinary tools is simple, flexible, and independent of Jenkins itself.
How it works
A scheduled task on the server, or on a separate backup machine, copies the Jenkins home directory to a backup location. Common tools include synchronisation utilities that copy only changed files, archive tools that pack folders into a single file, and modern backup programs that compress, encrypt, and remove duplicate data. The scheduler is the operating system’s own, not Jenkins.
The consistency problem
Copying files while Jenkins is running can capture a moment when some files are new and others are old. Most configuration files are written safely, but a busy Jenkins can still produce a mixed copy, especially in the builds folder. There are three ways to handle it:
- Quiet down first. Put Jenkins in a state where no new builds start, wait for running builds to finish, copy, and then resume. This is safe but can delay work.
- Use a snapshot. Freeze the disk instantly, then copy from the frozen picture. This gives a consistent view without pausing builds for long.
- Accept and verify. For configuration-only backups taken often, accept the small risk, and rely on frequent copies and restore drills to catch problems.
What to exclude
Workspaces
Large, temporary, and regenerated.
Caches and unpacked web files
Rebuilt automatically when Jenkins starts.
Old build folders
Decide by retention rules. Big, but sometimes needed for audit.
Everything else
Configuration, secrets, users, nodes, jobs, and the plugin list.
Running the backup from outside
Because the scheduled task is not part of Jenkins, it keeps working when Jenkins is down. A separate backup machine that pulls data from the controller is even better, since a compromised controller cannot easily reach and destroy the backups.
Verify every backup
After each copy, check that it finished without errors, compare file counts or sizes with expectations, and record a checksum. A backup script that fails silently is the classic cause of “we thought we had backups.”
Photocopying a book while someone is still writing in it can give you pages from two different drafts. Asking the writer to pause for a minute, or photographing the desk in one instant, gives you a copy that makes sense.
11Snapshots: Disks, Volumes, and Cloud Storage
Snapshots freeze an entire disk in seconds. They are a favourite of cloud and container setups.
What is a snapshot?
A snapshot is a point-in-time picture of a disk or volume, created by the storage system itself. It records the state of every block at one moment, usually in a few seconds, and uses very little extra space at first because it stores only later changes.
Where you will meet them
- Cloud disks: major cloud providers offer scheduled snapshots of virtual machine disks.
- Kubernetes volumes: when Jenkins runs in a cluster, its home directory sits on a persistent volume, and the cluster’s storage system can snapshot it.
- Advanced file systems and logical volumes: local storage systems often offer snapshots too.
Strengths
Advantages
- Very fast, with little effect on running builds.
- Captures everything, including settings, history, and secrets.
- Consistent view of the disk at one instant.
- Easy to schedule with platform tools.
Limits
- Often lives in the same account and region as the original.
- Restores whole disks, not single files, unless you mount the snapshot.
- A compromised cloud account can delete both disk and snapshots.
- Snapshots hold data that may include secrets, and need protecting.
Make snapshots into real backups
Copy across regions or accounts
Replicate snapshots to another region and, ideally, another account, so a single problem cannot destroy all copies.
Protect deletion
Restrict who can delete snapshots, and use locks where the platform offers them.
Encrypt
Turn on encryption for disks and snapshots, with keys managed separately.
Test mounting
Practise creating a new disk from a snapshot and starting Jenkins from it.
Snapshots plus a file-level backup make a strong pair. Snapshots give fast whole-disk recovery. File-level backups let you pull back one deleted job without restoring the entire server.
A snapshot is like taking a photograph of a room before guests arrive. If something is broken afterwards, you can compare with the photo or arrange the room exactly as it was. But the photo lives in the same house, so keep a second copy at a friend’s place.
12Configuration as Code: Rebuilding Instead of Restoring
The best backup for settings is a file in a repository. If Jenkins can be rebuilt from code, much of the fear of losing it disappears.
The idea
Instead of clicking through web pages to configure Jenkins, you describe the configuration in text files: the security setup, agent definitions, global tool settings, and the list of plugins. A plugin called Jenkins Configuration as Code reads such a file and applies it to a fresh Jenkins. Jobs can be created from code too, through tools such as Job DSL, or discovered automatically from repositories that contain a Jenkinsfile.
Why this counts as a backup
- The files live in a source repository, which has history and its own backups.
- Every change is reviewed, so nobody can quietly alter settings.
- A new Jenkins can be built in minutes, not days.
- The same files can build a test copy for experiments and drills.
What code does not cover
| Item | Covered by code? | What to do |
|---|---|---|
| Global settings and security setup | Yes | Keep in a repository |
| Job definitions | Yes, if defined as code or in Jenkinsfiles | Avoid hand-made jobs, or export them |
| Plugin list | Yes, as a list of names and versions | Keep the list in the repository |
| Stored credentials | No, and they should not be in the repository | Use an external secret store or backups of secrets |
| Build history and logs | No | Back up the builds folders by policy |
| User API tokens and some user data | Mostly no | Back up the users folder |
Cattle, not pets
Engineers often say that servers should be treated like cattle, not pets. A pet is unique and irreplaceable, so you must nurse it back to health. Cattle are identical and replaceable, so a sick one is simply replaced. A Jenkins controller rebuilt from code is cattle. A hand-tuned Jenkins that nobody dares to touch is a pet.
A realistic middle path
Few real installations are entirely code. Most have a mixture: some settings in code, some clicked by hand years ago. Move step by step. Start by exporting the plugin list and the global settings, and then convert the most important jobs. Keep the file-level backups as a safety net until you are confident that you can rebuild.
A restaurant with a printed recipe book can open a new branch in a week. A restaurant where the chef keeps everything in her head cannot open a branch, and cannot survive her illness. Configuration as code is the printed recipe book.
Every time you change Jenkins settings through the web page, ask: “Should this be in code?” If yes, write it down in the repository at once. The habit prevents the slow return of the hand-made pet server.
13Setting Goals: RPO, RTO, and the 3-2-1 Rule
A backup plan without numbers is a wish. Start by deciding how much loss and downtime you can tolerate.
Recovery Point Objective
The RPO is how much recent work you can afford to lose. If your RPO is 24 hours, a daily backup meets it. If your RPO is one hour, you need hourly copies of the important data. Different parts of Jenkins may have different RPOs: configuration may need frequent copies, while old build logs can wait a day.
Recovery Time Objective
The RTO is how long Jenkins may be unavailable. If your RTO is eight hours, a careful manual rebuild may be acceptable. If it is one hour, you need automation, prepared replacement machines, and rehearsed procedures.
The numbers drive the design
| Need | What it usually means |
|---|---|
| RPO of one day, RTO of two days | Nightly file backup, written runbook, manual rebuild |
| RPO of one hour, RTO of half a day | Frequent configuration backups, snapshots, tested restore procedure |
| RPO of minutes, RTO of an hour | Configuration as code, continuous or very frequent snapshots, prepared standby, regular drills |
Ask the users
Talk to the teams that use Jenkins. Ask: “If Jenkins was gone for a day, what would happen? If it lost the last week of changes, what would you have to redo?” Their answers set the goals. Often the acceptable loss is smaller than administrators assume, and the acceptable downtime is shorter.
The 3-2-1 rule
Three copies
The original plus two backups. One backup can fail, and you still have another.
Two kinds of storage
For example, a local disk and cloud object storage. A single technology problem cannot damage both.
One offsite copy
Kept in another location or account, so fire, flood, or a compromised account cannot reach it.
Locked and verified
Modern advice adds one immutable copy and zero errors: a locked copy that cannot be altered, and backups that are checked and restore without errors.
Numbers on paper are worthless until you measure them. Time a real restore in a drill, and compare the result with your RTO. Teams are often shocked to find that a “two-hour” restore takes a whole day.
A hospital knows exactly how many minutes an ambulance may take to arrive, and it plans routes, crews, and spare vehicles around that number. A backup plan built around clear time targets works the same way.
14Scheduling and Retention
Two decisions shape the rhythm of a strategy: how often to copy, and how long to keep each copy.
Match frequency to change
Configuration changes when administrators and teams edit jobs and settings. Build history changes with every build. Choose a schedule for each layer:
| Layer | Typical schedule | Reason |
|---|---|---|
| Configuration and secrets | Every few hours, or after changes | Small, cheap, and expensive to lose |
| Full home directory | Daily, in quiet hours | Larger, changes with every build |
| Whole-disk snapshot | Daily or more often | Fast, gives simple whole-server recovery |
| Offsite copy | Daily, right after the local copy | Protects against site-wide loss |
| Pre-change backup | Before every upgrade or big change | A guaranteed way back |
Retention: how long to keep
Old backups cost storage, but they also protect you from problems that were not noticed immediately. If a job was accidentally deleted three weeks ago and nobody noticed, a two-week retention leaves you with nothing to restore.
A common pattern
Many teams use a “grandfather, father, son” pattern: keep daily copies for a week or two, weekly copies for a month or two, and monthly copies for half a year or more. Old copies thin out, but a long history remains.
Daily
Keep the last seven to fourteen. Good for recent mistakes.
Weekly
Keep four to eight. Good for slow-to-notice problems.
Monthly
Keep six to twelve. Good for audits and long-hidden damage.
Before upgrades
Keep until the upgrade has proven itself, then treat as normal.
Consider rules and law
Some industries require build records and approvals to be kept for years. Others require personal data to be deleted after a set time. Discuss retention with the people responsible for compliance, so that your backup policy does not silently break a rule in either direction.
Automate deletion of expired backups as carefully as creation. A backup disk that fills up silently is a common reason why backups stop.
A newspaper archive keeps every edition of the last month on the shelf, one copy per week for the last year, and one per month forever. Recent history is dense, old history is thin, and nothing important disappears.
15Data Flow and Lifecycle of a Backup
Follow one backup from the moment the schedule fires to the moment the copy is safely stored and verified.
sequenceDiagram
participant SC as Scheduler
participant CT as Jenkins Controller
participant BK as Backup Tool
participant LS as Local Storage
participant OS as Offsite Storage
participant MN as Monitoring
SC->>BK: Start scheduled backup
BK->>CT: Optional quiet down or take snapshot
CT-->>BK: Ready
BK->>CT: Read home directory
BK->>BK: Skip excluded items and compress
BK->>BK: Encrypt the archive
BK->>LS: Write backup file
BK->>OS: Replicate copy offsite
BK->>BK: Verify checksum and size
BK->>CT: Resume normal work
BK->>MN: Report success or failure
MN-->>SC: Alert if missing or failed
The lifecycle in plain steps
Trigger
The operating system scheduler, the platform, or a backup service starts the process at the planned time.
Prepare
Optionally pause new builds or take a snapshot, to make sure the copy is consistent.
Copy
The tool reads the chosen files, skipping excluded items, and packs them.
Protect
The archive is encrypted, and the encryption key is handled separately.
Store and replicate
The archive is written locally and copied offsite, ideally to a locked location.
Verify
The tool checks that the file exists, has a reasonable size, and matches its checksum.
Report
A message goes to monitoring. A missing message is itself an alert.
Expire
Old backups are deleted according to the retention rules.
Why the last two steps matter
Backups fail silently more often than they fail loudly. A full disk, a changed password, or a renamed folder can make a nightly job do nothing for months. Reporting and expecting a report every day, so that silence is treated as failure, is what turns a backup script into a backup system.
Configure the alert as “no success message received in 26 hours,” not only “a failure message was received.” The first catches broken schedulers. The second cannot.
16The Restore Process, Step by Step
A restore is a calm procedure, not a scramble. Write it down before you need it.
The general recovery path
Decide what happened
Was it a lost disk, a deleted job, a bad upgrade, or an attack? The answer decides how much to restore and from which backup.
Stop the damage
If the cause is a compromise, isolate the server first. Restoring onto a machine that an attacker still controls only repeats the problem.
Prepare a clean machine
Start a new server or container with the same Jenkins version and a compatible Java version, using code or images where possible.
Choose the right backup
Pick the newest backup that is known to be good and taken before the problem began. Check its checksum.
Restore the home directory
Decrypt the backup and place the files in the new Jenkins home directory, with the right owner and permissions. Include the secrets folder.
Restore plugins
Install the same plugin versions from your saved list, unless the backup included them.
Start and check
Start Jenkins and read the startup log for errors. Confirm that jobs, users, nodes, and credentials appear.
Verify with real builds
Run a few important jobs, including one that uses a stored credential, to prove that the whole chain works.
Reconnect and announce
Reconnect agents, webhooks, and integrations, and tell users that Jenkins is back, including what recent changes may be missing.
Partial restores
Not every problem needs a full restore. If one job was deleted, you can copy just that job’s folder from a backup into the running Jenkins, and reload the configuration from disk. If a setting was changed wrongly, you can compare and copy back the one configuration file. Partial restores are faster and less risky, but they need a backup format where individual files are reachable.
Common restore hurdles
| Hurdle | Symptom | Prevention |
|---|---|---|
| Missing secrets folder | Credentials cannot be decrypted | Always include and test it |
| Different Jenkins or Java version | Startup errors or migration warnings | Record versions, keep old installers or images |
| Missing or newer plugins | Jobs fail to load, warnings about missing plugins | Keep a pinned plugin list |
| Wrong file ownership | Jenkins cannot read or write its files | Restore with correct permissions |
| Old webhook addresses | No builds are triggered after recovery | Document integrations in the runbook |
An airline pilot follows a checklist even in an emergency, because a calm list beats a clever guess. A restore runbook is the emergency checklist, written by someone who was calm, for someone who may not be.
17Restore Drills: Proving That It Works
Only a real restore proves that a backup is usable. Practise on calm days.
What a drill is
A restore drill is a planned exercise in which you rebuild Jenkins from backups in a separate, disposable environment, and then check that it works. Nothing in production is touched. The goal is to find problems while they are cheap.
How to run one
Choose a backup at random
Do not pick the newest every time. Random selection catches old, corrupted, or missing copies.
Use the runbook, not memory
Follow the written steps exactly. Where the runbook is wrong or unclear, fix it on the spot.
Isolate the test copy
Cut off outside connections, or use a test network, so the restored Jenkins does not trigger real deployments or send real notifications.
Time the recovery
Record how long each step took, and compare the total with your RTO.
Check what came back
Compare job counts, users, nodes, and plugins with the original. Open a few old builds. Run a test job that uses a credential.
Write up and improve
Record what went wrong and fix it: update the runbook, adjust the schedule, or change the tools.
What to check
Does it start?
Jenkins boots without errors, and plugins load.
Are jobs there?
Job count and structure match the original.
Do secrets work?
A job using a stored credential can log in to its target.
Is history intact?
Old builds, logs, and results open correctly.
Can people log in?
Users, permissions, and security settings behave as before.
How long did it take?
Compare with the recovery time objective.
How often
Run a full drill at least twice a year, and after any major change to the backup system, the Jenkins version, or the hosting platform. Run smaller checks, such as restoring a single job, every month. Frequent small drills keep the skills and the documents fresh.
A drill that restores into a network with real webhooks and real deployment credentials can accidentally trigger real builds and real deployments. Always isolate the test environment from production systems.
A fire drill is annoying, and it reveals that the back door is locked and the map on the wall is out of date. Finding that on a Tuesday afternoon is a gift. Finding it during a real fire is a tragedy.
18Upgrades, Plugins, and the Way Back
Many Jenkins disasters are not caused by hardware. They are caused by upgrades that go wrong.
Why upgrades are risky
Jenkins and its plugins update often. A new Jenkins version may need a newer Java, or a plugin may change how it stores settings. Some changes migrate the configuration files in a way that older versions cannot read. That means simply reinstalling the previous version after a bad upgrade may not work, unless you also have the old data.
Before every upgrade
- Take a fresh backup of the home directory, including secrets, and verify it.
- Record the versions: Jenkins core, Java, and every plugin.
- Read the release notes for breaking changes and migration warnings.
- Test on a copy restored from backup, before touching production.
- Keep the old installer or image so you can return to the previous version.
- Pick a quiet time and tell the users.
The plugin list as a backup
A plugin list, with exact versions, is a tiny file that makes a big difference. It lets you rebuild the same set after a restore or on a new machine. Keep it in a repository, and update it whenever plugins change. Pin versions in your container images or setup scripts, so that a rebuild does not silently pick newer, untested plugins.
Rollback plan
| Situation | Rollback |
|---|---|
| A plugin upgrade breaks jobs | Roll back that plugin to its previous version, or restore the plugin folder from backup |
| A core upgrade breaks startup | Restore the previous version and the pre-upgrade backup of the home directory |
| Configuration was migrated in a bad way | Restore the pre-upgrade backup into a clean copy of the old version |
| A change to global security locks people out | Restore the affected configuration file, or use the emergency access procedure in the runbook |
Before renovating a house, you take photographs of every wall and mark where the pipes run. If the plan goes wrong, you know how it was. A pre-upgrade backup and version list are those photographs.
Upgrade in small steps, and never install every available update at once on a Friday afternoon. A small change is easy to undo, and a Monday morning is a good time to notice problems.
19Security of the Backups Themselves
A backup is a second copy of everything valuable. If the original is protected but the copy is not, attackers simply take the copy.
What is at risk
Backups of the Jenkins home directory contain settings that reveal your network layout, user accounts, API tokens, and encrypted credentials with the keys to decrypt them. Anyone who obtains an unprotected backup may be able to impersonate Jenkins and reach your deployment targets.
Protection layers
Encryption at rest
Encrypt backup files, and keep the key separate from the backup and from Jenkins.
Least privilege
Only a few named identities may read backups. The account that writes backups should not be able to read or delete old ones.
Separate credentials
The backup system uses its own credentials, not ones stored in Jenkins. A compromised Jenkins should not reach the backups.
Encrypted transfers
Copy backups over encrypted connections only.
Audit access
Record every read, write, and delete of backups, and review the log.
Secure deletion
Expired backups must really be deleted, on every copy, according to your retention rules.
Pull, not push
A strong design has a separate backup system that pulls data from Jenkins. Then Jenkins holds no credentials that can reach the backups. In a push design, Jenkins holds a credential that can write to the backup location, and an attacker who takes Jenkins may be able to use it to damage the backups.
Test the protection
Ask the awkward questions. Who can read the backups today? Who could delete them? If the Jenkins controller were fully compromised, what could the attacker do to the backups? Fix the answers you do not like.
Safer habits
- Encrypted backups with separately stored keys.
- A separate backup account or system.
- Write-only access for the backup writer.
- Logged, reviewed access.
Risky habits
- Plain archives in a shared folder.
- Jenkins holding admin rights to the backup store.
- One key for everything, stored next to the backups.
- Old backups that nobody remembers or protects.
A bank does not leave its duplicate keys in the same drawer as the originals, and it does not let the night guard hold the only key to the spare-key cabinet. Backup security is about who can touch the copies, not only the originals.
20Ransomware and Immutable Backups
Modern attackers know that backups are the victim’s escape route, so they try to destroy them first.
The attack pattern
An attacker gains access to a server, moves quietly through the network, finds the backup storage, and deletes or encrypts the backups. Only then do they encrypt the live data and demand payment. If the backups are reachable with the same credentials, the victim has no way out.
Immutable backups
An immutable backup cannot be changed or deleted for a set period, even by an administrator. Many cloud object stores and backup systems offer such locks, sometimes called retention or object lock features. A copy protected this way survives even if the attacker owns your accounts, because the lock is enforced by the storage system itself.
Other defences
- Offline or air-gapped copies: a copy that is not connected to the network at all, such as a rotated disk, cannot be reached remotely.
- Separate accounts and separate administrators: nobody who controls Jenkins should control the locked copy.
- Delayed deletion: deleted backups stay recoverable for a few days before they are truly removed.
- Multi-person approval: deleting backups or shortening retention requires two people.
- Alerts on unusual activity: sudden mass deletions or a change to retention settings should notify the security team at once.
Detection matters too
If ransomware slowly damages files, your daily backups may quietly copy the damage. Long retention lets you go back to a clean copy from before the attack. It is a good reason to keep weekly and monthly copies, and not only the last few days.
A restored Jenkins from a clean backup, onto the same vulnerable server, with the same weak password, will be attacked again. After a compromise, fix the cause, rotate every secret that Jenkins could reach, and only then bring the system back.
A jeweller keeps the best pieces in a time-lock safe that even the owner cannot open outside set hours. If a thief forces the owner to open the shop, the safe still holds. An immutable backup is that time-lock safe for your data.
21Performance and Scalability
A backup that slows Jenkins to a crawl, or grows without limit, will soon be switched off.
Where the cost comes from
Backups read many files, compress them, encrypt them, and send them across a network. On a busy controller, that competes with builds for disk and processor time. The larger the build history, the longer and heavier the backup.
Practical remedies
Discard old builds
Set build retention on jobs. The single biggest reduction in backup size usually comes from keeping less history.
Externalise artifacts
Store built packages in an artifact repository with its own backup, instead of inside Jenkins.
Separate small and large
Back up configuration often and history less often. Small, frequent copies are cheap.
Use incremental methods
Copy only what changed. Tools that deduplicate data can make daily copies very small.
Schedule quiet hours
Run heavy backups at night or on weekends, when fewer builds run.
Read from a snapshot
Copy from a frozen snapshot on another machine, so the live controller is barely touched.
Watch growth
Track the size of the home directory and of each backup over time. A sudden jump often means a job started archiving huge files or a discarder setting was removed. Catching that early prevents a full disk that would stop both Jenkins and the backups.
Restore speed matters as much as backup speed
A backup that takes ten minutes to write may take hours to restore, especially from cloud storage over a slow link. Measure the restore, not just the copy. Keep the most recent backup close to where you would restore it, and consider tiering older ones to cheaper, slower storage.
Many controllers
Large organisations run several Jenkins controllers. Give every controller the same policy and standard tooling, with a central view of which ones are protected and when the last successful backup happened. A forgotten small controller is the one that fails without a backup.
Moving house takes far less time if you first throw away what you do not need. Cleaning out old build history before backing up is the same idea: smaller loads move faster and cost less to store.
22High Availability Versus Backups
People often confuse the two. They protect against different problems, and you usually need both.
Two different questions
| Idea | Question it answers | Protects against |
|---|---|---|
| Backup | Can I get my data back? | Deletion, corruption, attack, bad upgrades, disasters |
| High availability | Can service continue while a part fails? | Machine or process failure |
A mirrored disk or a standby server copies your mistakes instantly. If someone deletes a job, the mirror deletes it too. Only a backup, kept separately and from an earlier time, can bring it back.
Jenkins and availability
The open-source Jenkins controller is a single active server by design: one controller owns the home directory at a time. Running two active controllers on the same data is not supported. Because of that, most teams get resilience from a mix of fast rebuilds and backups: a replacement controller can be started quickly from code and a recent backup or snapshot. Some commercial distributions offer additional high-availability features, so check the vendor’s documentation if you need them.
Ways to shorten the outage
Keep the home directory on separate storage
Put it on a volume that can be detached and attached to a new server or container.
Use images and code
A prepared image with Jenkins and plugins installed starts fast, and configuration files apply the settings.
Use a supervisor
A container platform or service manager can restart a failed controller automatically.
Prepare a cold standby
Keep a second environment in another zone or region ready to start from the latest backup when needed.
Practise the switch
Rehearse the failover so that the real one takes minutes, not a day of confusion.
Agents are the easy part
Agents do not hold anything precious. If they are defined by code and images, a new set can be created in minutes. Focus your protection effort on the controller and on the outside systems it depends on.
Do not tell the team “we have a standby server, so we do not need backups.” A standby reproduces every mistake and every attack immediately. Only a separate, older copy can undo them.
A spare tyre lets you keep driving after a puncture. It does not help if you drove into a river. A standby server is the spare tyre, and a backup is the tow truck plus the insurance policy.
23Monitoring, Logging, and Alerting
A backup you do not monitor is a backup you will find broken at the worst possible time.
What to watch
Useful checks
- Freshness: alert if there is no successful backup within the expected period plus a margin.
- Size trend: alert if a backup is suddenly much smaller, which may mean missing folders, or much larger, which may mean runaway growth.
- Integrity: verify checksums, and periodically test that an archive can be opened.
- Offsite copy: confirm that the replica exists and matches the local copy.
- Space: alert when backup storage or the Jenkins disk is nearly full.
- Retention: confirm that old backups are removed on schedule, and that locks are still in place.
- Drill dates: track when the last restore drill happened, and alert when it is overdue.
Logging
Every backup run should write a short log: start time, end time, what was included, size, checksum, and result. Keep those logs somewhere other than the controller. When a restore is needed, the log helps you pick the right backup and prove which one you used.
Alert on silence
The most useful alert is the one that fires when nothing happens. If the backup system dies, it cannot send a failure message. Treat a missing success message as a failure.
Send alerts to a human
An alert that goes to an unread mailbox is not an alert. Send backup alerts to a channel that someone watches, with a named owner and a clear next step in the message.
| Signal | Possible meaning | Action |
|---|---|---|
| No success message for a day | Scheduler broken, machine down, or a script failure | Investigate and run a manual backup |
| Backup much smaller than usual | A folder was skipped or a mount is missing | Check include rules and mounts |
| Backup much larger than usual | A job archives huge files or history was not trimmed | Find the source and trim it |
| Checksum mismatch | Corruption during copy or storage | Retake the backup and check storage health |
| Storage nearly full | Retention not deleting or growth | Fix retention and add space |
A night watchman who never reports back could be fine, or could be asleep. A watchman who must call in every hour tells you something even when nothing happens. A backup monitor should behave like the second watchman.
24Deployment, Cloud, and Infrastructure as Code
Backups work best when the whole Jenkins setup is described in code, so recovery is a rebuild followed by a restore of data.
The two halves of recovery
Recovery has two halves. The first is the shape of Jenkins: the server, the image, the plugins, and the settings. The second is the state: history, credentials, and things that changed since. Code recreates the shape, and backups return the state. Keeping them separate makes each simpler.
flowchart LR
REPO["Repository with Infrastructure and Configuration Code"] --> BUILD["Automated Rebuild Process"]
IMG["Pinned Jenkins Image with Plugin List"] --> BUILD
BUILD --> NEWC["New Controller"]
VOL["Persistent Volume for Jenkins Home"] --> NEWC
SNAP["Scheduled Volume Snapshots"] --> VOL
SNAP --> REP["Replicated Snapshot in Another Region"]
FILE["File Level Encrypted Backups"] --> RESTORE["Restore of State"]
RESTORE --> NEWC
NEWC --> CHECK["Automated Health and Restore Checks"]
CHECK --> MON["Monitoring and Alerts"]
Containers and Kubernetes
When Jenkins runs in a container platform, the home directory sits on a persistent volume. The platform’s snapshot feature can protect it, and a new controller pod can attach a volume created from a snapshot. The image and configuration files describe everything else. Test the process end to end: create a volume from a snapshot in a test cluster and start Jenkins from it.
Virtual machines
On virtual machines, disk snapshots and machine images give quick recovery. Combine them with file-level backups so single files can be recovered without swapping the whole disk.
Automate the rebuild
Write the rebuild as an automated process that anyone on the team can run. Keep it in the repository, test it on a schedule, and make it produce a working, empty Jenkins. Then adding the restored state becomes one more automated step.
Cloud safety
Separate accounts
Store backups in an account that Jenkins administrators cannot log in to.
Second region
Keep at least one copy in another region.
Object lock
Use retention locks so nobody can delete recent backups.
Lifecycle rules
Move old copies to cheaper storage automatically, and delete them on schedule.
Use case: recovering from a lost region
A cloud region has a long outage. The team starts Jenkins in a second region from the same code, attaches a volume created from the replicated snapshot, and reconnects agents. Recent changes made since the last snapshot are lost, which the team already accepted as part of its recovery point objective. Builds resume within the hour.
25Advantages, Disadvantages, and Trade-offs
Backups are essential insurance, and every design has trade-offs of cost, complexity, and speed.
Advantages
- Recovery from deletion, corruption, attacks, and bad upgrades.
- Known recovery time and data loss limits.
- Confidence to upgrade and change things.
- Audit history preserved for as long as needed.
- Faster onboarding of new environments from copies.
- Lower stress during incidents.
Disadvantages
- Storage and network costs that grow over time.
- Extra systems to run, secure, and monitor.
- Backups hold secrets and create another target.
- Restores take real time and skill.
- Untested backups create false confidence.
- Policies need regular review.
Balancing choices
| Choice | Cheaper and simpler | Safer and faster to recover |
|---|---|---|
| Frequency | Daily | Hourly for configuration |
| Copies | One local copy | Local, offsite, and locked copies |
| Retention | Two weeks | Months, with tiered storage |
| Restore testing | Yearly | Monthly small checks, twice-yearly full drills |
| Rebuild | Manual steps | Fully automated from code |
How much is enough?
Match the effort to the value of Jenkins. A hobby server needs a nightly copy to another disk. A company’s release pipeline needs offsite, locked, encrypted copies, monitoring, and rehearsed recovery. There is no universal answer. The right question is: what would an hour, a day, and a week of losing Jenkins cost us?
26Design Patterns and Anti-patterns
A pattern is a solution many teams discovered independently. An anti-pattern is a trap many teams fell into.
Patterns worth copying
Rebuild plus restore
Code recreates the shape of Jenkins, and backups restore the state.
Layered schedule
Frequent small backups for configuration, daily larger ones for history, and pre-change backups before upgrades.
Pull-based backup
A separate backup system reads from Jenkins, so Jenkins holds no credentials to the backups.
Immutable offsite copy
One locked copy in another account and location.
Scheduled restore drills
Random restores into a disposable environment, timed and documented.
Alert on silence
A missing success message counts as a failure.
Anti-patterns to avoid
The Backup on the Same Disk
A backup folder inside the Jenkins home directory, or on the same disk. When the disk fails, everything vanishes together.
Better approach
Store copies on separate storage, and keep one offsite.
The Untested Backup
A backup job that runs every night and has never been restored. Nobody knows whether it works.
Better approach
Schedule restore drills, and treat an untested backup as no backup.
Jenkins Backing Up Jenkins
The only backup schedule is a Jenkins job. When Jenkins breaks, backups stop and the restore button disappears.
Better approach
Run backups from the operating system, the platform, or a separate backup system.
Keys Left Behind
Backing up the credential file but not the secrets folder, or storing both together unencrypted.
Better approach
Back up both, encrypt the backup, and store the backup key separately.
Short Memory
Keeping only three days of backups, so slow-to-notice damage cannot be undone.
Better approach
Use tiered retention with weekly and monthly copies.
The Standby Excuse
Believing that a mirrored disk or standby server replaces backups.
Better approach
Use both. Availability keeps you running. Backups let you go back in time.
27Best Practices and Common Mistakes
These lessons are usually learned the hard way. You can learn them the easy way.
Best practices
- Write down your goals: recovery point and recovery time, agreed with the users.
- Follow 3-2-1, and add one immutable copy.
- Back up the secrets folder with the configuration, and encrypt the result.
- Run backups from outside Jenkins, preferably pulling from a separate system.
- Keep configuration in code and the plugin list with exact versions in a repository.
- Trim build history and move artifacts to a dedicated store.
- Back up before every upgrade, and keep the old version handy.
- Monitor for success and for silence, and alert a real person.
- Test restores regularly, time them, and update the runbook.
- Restrict and log access to backups.
- Review the policy at least once a year.
Common mistakes and their fixes
| Mistake | What happens | Fix |
|---|---|---|
| Backups on the same disk | Disk failure destroys both | Separate storage and an offsite copy |
| Never testing a restore | Backup turns out unusable | Schedule drills |
| Excluding the secrets folder | Credentials cannot be decrypted after restore | Include it and test with a credential job |
| Unencrypted backups in shared storage | Leak of every credential | Encrypt and restrict access |
| No plugin version list | Rebuild picks different plugins and things break | Keep an exact list in a repository |
| Silent backup failures | No good backup for months | Alert on missing success messages |
| Copying while builds write files | Inconsistent backup | Use quiet down or snapshots |
| Too short retention | Slow-to-notice problems cannot be reversed | Tiered retention |
| Restoring onto a compromised server | Attack repeats | Fix the cause and rotate secrets first |
The nightly backup has been “successful” for two years, but a disk mount changed and it has been copying an empty folder ever since. Only a restore drill, or a size alert, would have caught it.
28A Worked Example: Building a Backup Strategy for a Team
Let us walk through a complete setup for a fictional company, step by step, without writing any code.
The situation
Northwind Logistics runs one Jenkins controller on a virtual machine. About sixty engineers depend on it. It has around four hundred jobs, five years of build history, and many stored credentials. The only protection is a weekly copy that a former administrator set up, which writes to a folder on the same disk. Nobody has ever restored from it.
Step by step
Ask the users
Talks with the teams reveal that losing more than a day of configuration changes would be painful, and that Jenkins must return within a working day. The team sets a recovery point of one day for history and four hours for configuration, and a recovery time of eight hours.
Look inside
The team finds that the home directory is 380 gigabytes, of which 300 are old build folders, and that one job archives large test recordings. Only 40 megabytes are configuration.
Trim first
They set build retention on the biggest jobs and move the test recordings to an artifact store. The home directory drops to 90 gigabytes.
Protect the shape
They export the plugin list with exact versions, place the global settings in configuration files, and put both in a repository. New jobs must use Jenkinsfiles.
Set up layered backups
A separate backup server pulls the configuration and secrets every four hours and the full home directory every night. A cloud snapshot of the disk is taken daily.
Protect the copies
Backups are encrypted, and the key is kept in a separate secret store. Copies replicate to a second cloud account in another region with a retention lock. Jenkins holds no credentials to any backup location.
Set retention
Fourteen daily copies, eight weekly, and twelve monthly, plus special copies before each upgrade.
Add monitoring
Alerts fire when no success arrives within 26 hours, when size changes by more than thirty percent, or when storage passes eighty percent.
Write the runbook and run a drill
An engineer who did not write the runbook follows it in a test network. The drill takes eleven hours, longer than the target, and reveals a missing plugin list step and a wrong file owner. The team fixes both.
Repeat and improve
A second drill, after automating the rebuild, takes three hours. The team schedules drills twice a year.
The result
| Question | Before | After |
|---|---|---|
| Where are backups? | Same disk, weekly | Local, offsite, and locked copies |
| Are secrets covered? | Unknown | Included, encrypted, and tested |
| Recovery time | Unknown | Three hours, measured in a drill |
| Backup size | 380 gigabytes | 90 gigabytes, with small frequent configuration copies |
| Would we notice a failed backup? | No | Yes, within 26 hours |
The most valuable steps were the cheap ones: asking users about their needs, trimming history, and running one honest drill. Fancy tools mattered far less than measuring a real restore.
29Starting from Nothing, or from an Old Setup
Many teams discover that their backups are missing or unreliable only after a scare. Here is how to improve calmly.
What you may find
- No backups at all, because “it has always worked.”
- A backup plugin writing to the same disk as Jenkins.
- A script written years ago by someone who left, with no monitoring.
- Snapshots that exist but were never tested.
- Backups that miss the secrets folder or plugins.
A safe improvement path
Take an emergency copy today
Before anything else, make one manual, encrypted copy of the home directory to a separate place. It is not a strategy, but it is far better than nothing.
Document what exists
Record the Jenkins and Java versions, the plugin list, the home directory location, and how the server is hosted.
Try a restore
Restore that emergency copy into a scratch environment. The first attempt will teach you more than any document.
Automate a basic schedule
Set up a regular copy from outside Jenkins, with monitoring that alerts on silence.
Add the offsite and locked copy
Replicate to another location or account with a retention lock.
Reduce the load
Trim build history, and move artifacts to a dedicated store.
Move to code
Step by step, place settings, jobs, and the plugin list into a repository.
Schedule drills
Put the next drill in the calendar, with an owner.
Do not retire the old backup method until the new one has been restored successfully at least once. Two imperfect backups beat one that has not been proven.
The reward
After the improvement, the team knows what is protected, how fast it can recover, and who is responsible. The fear of “what if Jenkins dies?” gives way to a calm, practised answer.
30Governance: Owning the Strategy Over Time
Backup strategy decays if nobody owns it. Systems change, people leave, and old assumptions stop being true.
Assign clear owners
| Area | Owner | Responsibility |
|---|---|---|
| Backup policy | Platform or operations lead | Goals, schedules, retention, yearly review |
| Backup tooling | Operations team | Running, monitoring, and updating the system |
| Security of backups | Security team | Encryption, access, locked copies, audits |
| Restore drills | Rotating engineers | Running drills, updating the runbook |
| Job and team data | Each application team | Retention needs for their build history, keeping definitions as code |
The runbook
The runbook is the written recovery procedure. It should say where backups are, how to decrypt them, how to rebuild the server, how to restore, how to verify, and whom to call. Store a copy where it can be read even if Jenkins and the company network are down, such as in a separate document system or printed for the on-call binder.
Agree on rules
- Recovery goals agreed with the business, and reviewed each year.
- Change rules: new Jenkins servers are added to the backup system before they go live.
- Drill rules: a full drill twice a year, small checks monthly, results recorded and shared.
- Access rules: a short, reviewed list of who can read or delete backups.
- Incident rules: after any real recovery, hold a review and update the plan.
Review regularly
Every year, and after any big change such as a new hosting platform, review the whole strategy. Has the amount of data changed? Have the users’ needs changed? Are the tools still supported? Is anyone still able to run the runbook?
Every server is covered
No controller goes live without backups and monitoring.
Nobody restores alone the first time
Pair people during drills so that knowledge spreads.
Write it down
A runbook that lives only in one head is not a runbook.
Learn from every incident
Real recoveries improve the plan.
A ship carries lifeboats, and every crew member knows which boat is theirs and practises launching it. A lifeboat that nobody has ever launched is a decoration. A backup strategy needs the same rehearsed ownership.
31Real-World and Industry Examples
These scenarios show how organisations handle Jenkins backups. They are illustrative, so you can map them to your own situation.
Scenario: the deleted folder
An administrator, tidying up, deletes a folder that contains forty jobs. Nobody notices for two days. Because the team keeps daily copies and can restore individual job folders, they copy the folder from a backup taken before the deletion, reload configuration from disk, and lose nothing except two days of build history for those jobs.
Scenario: the upgrade that went wrong
A team upgrades Jenkins and several plugins on a Friday. On Monday, half the pipelines fail with strange errors. They had taken a verified backup and kept the old image. Restoring the old version and data takes ninety minutes. Afterwards, they upgrade on a test copy first and change the policy to one small upgrade at a time.
Scenario: ransomware and the locked copy
An attacker gains access to the network and encrypts the Jenkins server and the local backup server. The company has a locked copy in a separate cloud account that the attacker never reached. After cleaning up the intrusion, changing every secret, and rebuilding from code, they restore configuration and history from the locked copy.
Scenario: rebuilding from code
A company whose Jenkins is defined entirely as code loses a virtual machine. An automated process starts a new controller from the repository in fifteen minutes. Build history and stored secrets are then restored from backup in another twenty. The whole event is a routine incident, not a crisis.
Scenario: the backup that was empty
A team’s nightly backup reports success for years. During a drill, the restored Jenkins has no jobs. A disk had been remounted under a different path, and the backup was copying an empty folder. After the drill, they add a size alert and a job count check to every backup.
What these examples share
In every good outcome, the team had copies that were separate, recent, protected, and tested. The bad outcomes were avoided or reduced by small habits: verified backups before upgrades, locked offsite copies, and honest restore drills.
32How Other CI Tools Handle Backups
Every CI system holds state that must be protected. Comparing them shows what is essential and what is special to Jenkins.
| Tool | Backup approach | Notable difference |
|---|---|---|
| Jenkins | Back up the home directory, use snapshots, and keep configuration as code | State is plain files on disk, so any file tool works, but you assemble the strategy yourself |
| TeamCity | Built-in backup feature in the administration area, plus external database backups | Includes a product feature for backups, with data spread across files and a database |
| Self-hosted GitLab | A built-in backup command for its data, plus separate handling of configuration and secrets files | Official tooling exists, and secrets must be saved separately |
| Hosted services such as GitHub Actions | Workflow definitions live in the repository; the vendor operates the platform | Your backup is mostly the repository and any external secrets and artifacts |
Features change, so check each vendor’s current documentation before relying on this table.
What Jenkins does differently
Jenkins offers no single official, complete backup button. Its strength is that state lives in plain files, so you can use snapshots, archives, or any storage tool. Its weakness is that you must design and test the whole strategy yourself.
What is universal
The lessons apply everywhere: know what state you hold, keep configuration as code where you can, protect secrets separately, follow 3-2-1 with a locked copy, monitor for silence, and practise restores.
33Advanced Topics: What to Learn Next
Once the basics feel comfortable, these ideas help larger and more security-focused organisations.
Point-in-time recovery
Some storage systems can keep a continuous trail of changes, so you can restore to any moment, not only to the last scheduled backup. That shrinks the recovery point from hours to minutes. It costs more and needs careful design, but it is valuable for very important servers.
Deduplicating backup tools
Modern backup programs split data into chunks, store each unique chunk once, and encrypt everything. Daily backups of a mostly unchanged folder become tiny, so you can keep long histories cheaply and verify integrity automatically.
Backing up the ecosystem
Jenkins depends on other systems: the source repository, the artifact store, the secret manager, the identity provider, the agents’ images, and the webhook configuration. A complete plan lists all of them and gives each one a tested backup and an owner. A perfect Jenkins backup is not much use if the secret store it depends on is gone.
Many controllers and a central view
Large companies run dozens of controllers. Standardise tooling and policy, publish a dashboard that shows the last successful backup and last successful drill of every controller, and automatically enrol new controllers in the backup system.
Cloud-native and commercial options
Kubernetes-focused backup tools can protect volumes and cluster resources together. Commercial Jenkins distributions add their own backup and high-availability features. Evaluate them against your recovery goals, and test them as you would any other method.
Automated verification
The most mature teams automate restore drills. Every week, a job restores the latest backup into a temporary environment, starts Jenkins, runs a set of checks, and reports the result, then deletes the environment. A backup is then proven every week without human effort.
Point-in-time recovery
Restore to any moment, not only to scheduled backups.
Deduplication
Small, encrypted, verified backups with long histories.
Ecosystem coverage
Protect every system that Jenkins depends on.
Automated drills
Prove restores weekly without manual work.
Add one advanced feature at a time, and only after the basics are tested. A complicated strategy that nobody understands often fails at the moment it is needed.
34Troubleshooting Backups and Restores
When something feels wrong, walk through these questions in order.
Did the backup run at all?
Check the scheduler and the backup log. If there is no record for last night, the scheduler, the machine, or the credentials may have failed.
What does the backup contain?
Look at the size and list the contents. Confirm that the main configuration files, jobs, and secrets folder are present.
Is it intact?
Verify the checksum and try opening the archive. Corruption can appear during copy or storage.
Can you decrypt it?
Confirm that the backup key is available and correct, and that it has not been rotated without updating the runbook.
Do the versions match?
Compare the Jenkins version, Java version, and plugin versions between the backup and the new server.
Are permissions correct?
Check the owner and access rights of the restored files. Wrong ownership causes many startup problems.
What does the startup log say?
Read it from the top. Messages about missing plugins, unreadable files, or failed migrations point to the cause.
Do credentials work?
Run a job that uses a stored credential. If it fails, the secrets folder or its keys are probably missing or mismatched.
| Symptom | Likely cause | First check |
|---|---|---|
| Backup is tiny | Folder missing or mount changed | Include rules and mounts |
| Jenkins will not start after restore | Version mismatch or wrong file ownership | Startup log and permissions |
| Jobs load but credentials fail | Secrets folder missing or mismatched | Presence of the secrets folder |
| Plugins reported missing | Plugin list not restored | Reinstall from the pinned list |
| No builds trigger after recovery | Webhooks point to the old address | Integration settings in the runbook |
| Restore takes far longer than expected | Slow storage or large history | Timing per step and tiering |
Keep a short page beside the runbook that lists the most common restore errors in plain language, with the fix for each one. It helps whoever is on call at three in the morning.
35Glossary of Terms
Every important word from this guide in one place, explained in plain language.
| Term | Plain meaning |
|---|---|
| Agent | A worker machine or container that runs builds for Jenkins. |
| Backup | A copy of data stored separately so it can be recovered after loss or damage. |
| Build discarder | A Jenkins setting that deletes old builds to save space. |
| Checksum | A short fingerprint used to detect whether a file was damaged. |
| Configuration as code | Describing settings in text files so Jenkins can be rebuilt from them. |
| Controller | The central Jenkins server that stores state and schedules builds. |
| Differential backup | A copy of everything changed since the last full backup. |
| Full backup | A complete copy of all chosen data. |
| High availability | Design that keeps a service running when a component fails. |
| Immutable backup | A backup that cannot be changed or deleted for a set time. |
| Incremental backup | A copy of only what changed since the last backup of any kind. |
| Jenkins home directory | The folder where Jenkins keeps its settings, jobs, secrets, and history. |
| Offsite copy | A backup kept in a different location or account. |
| Quiet down | A Jenkins state where no new builds start, used before maintenance. |
| Recovery Point Objective | The most recent data loss you can accept, measured in time. |
| Recovery Time Objective | The longest outage you can accept while recovering. |
| Restore drill | A planned practice of recovering from backup in a test environment. |
| Retention | How long each backup is kept before deletion. |
| Runbook | A written, step-by-step recovery procedure. |
| Secrets folder | The Jenkins folder holding keys that decrypt stored credentials. |
| Snapshot | A point-in-time picture of a disk or volume made by the storage system. |
| 3-2-1 rule | Keep three copies, on two kinds of storage, with one offsite. |
| Workspace | The temporary folder where a build checks out code and produces output. |
36Frequently Asked Questions
Short answers to the questions people ask most.
37Summary and Key Takeaways
A backup strategy turns the fear of losing Jenkins into a practised, predictable procedure.
You began with a bakery whose recipe cards were lost to a burst pipe and finished with a plan that copies the cards, stores them in another building, locks one copy, and rehearses reading them back. Along the way you learned what lives in the Jenkins home directory, why the secrets folder is special, how plugins, file copies, snapshots, and configuration as code fit together, how recovery goals guide the schedule, and how restore drills prove that the plan works. The main rules are simple: know what you hold, keep copies separate and protected, monitor for silence, and always practise the restore.
Key Takeaways
- The Jenkins home directory holds the state. Configuration, jobs, users, and history live there, while workspaces and caches can be skipped.
- Treat secrets specially. Include the secrets folder, encrypt the backup, keep the key elsewhere, and consider moving important credentials to an external secret store.
- Set goals first. Agree on recovery point and recovery time with the users, and let them drive schedules and tools.
- Follow 3-2-1 and add a locked copy. Use separate storage, an offsite copy, and an immutable copy that attackers and accidents cannot destroy.
- Combine methods. Configuration as code rebuilds the shape, snapshots and file backups restore the state, and backups run from outside Jenkins.
- Back up before every upgrade. Keep a pinned plugin list, the old version, and a rollback plan.
- Monitor, test, and own it. Alert on silence, run timed restore drills, keep a runbook, and review the strategy every year.