Blue-Green Deployment in Jenkins
Release new versions without switching the lights off. Learn the basic idea of blue-green deployment, how Jenkins pipelines drive it, and how to switch traffic, verify the result, and roll back in seconds when something goes wrong.
Imagine a restaurant that wants to change its whole menu, its kitchen equipment, and its seating plan. The owner has two options. The first is to close the restaurant on Sunday night, rebuild everything, and hope that it reopens on Monday morning without surprises. The second is to build an identical second restaurant next door, test every dish in it, invite a few friends to try it, and then move the customers over by simply changing the sign on the pavement. If anything is wrong, the sign goes back, and the customers return to the old place, which is still exactly as they left it.
The second option is blue-green deployment. Instead of updating the running application in place, a team keeps two identical environments. One serves real users. The other stands ready. A new version is installed and tested in the idle one, and then traffic is switched over in a single step. The old environment waits untouched for a while as an instant escape route.
In this guide you will learn what blue-green deployment is, how the traffic switch works, why databases make it tricky, how a Jenkins pipeline drives each step, how to verify and roll back, and which mistakes cause the most trouble. This is an introduction, so we focus on the essential ideas and explain each new word the first time it appears. We assume you know nothing about Jenkins or deployment strategies yet.
1What Is Blue-Green Deployment?
Blue-green deployment keeps two identical production environments and moves users from the old one to the new one by switching traffic, not by changing the running system.
First, the basics
A deployment is the act of putting a new version of software where users can reach it. An environment is a complete place where the application runs: servers or containers, network settings, configuration, and connections to other services. Production is the environment that real users use.
Jenkins is a free automation server that builds, tests, and delivers software. A Jenkins pipeline is a written list of steps, stored in a file called a Jenkinsfile. Pipelines are very good at deployments, because deployments are a sequence of careful, repeatable steps.
The two colours
In blue-green deployment, the two environments are simply named blue and green. At any moment, one of them is live, receiving user traffic, and the other is idle. The colours have no meaning except to tell the two apart. After each release, the roles swap: the environment that was idle becomes live, and the old live one becomes the idle standby for the next release.
The steps in one sentence
Deploy the new version to the idle environment, test it there while nobody is using it, switch user traffic to it, watch closely, and keep the old environment ready so you can switch back if needed.
Think of a railway with two parallel tracks and a set of points. A train runs on the first track while workers repair the second one. When the second track is ready and tested, the signaller flips the points, and trains run on the new track. The old track stays in place, so if the repair turns out badly, the points flip back. The points are the traffic switch.
What it gives you
- Little or no downtime: users never see a maintenance page, because the new version is already running before they are moved.
- Testing in a real environment: the idle environment is a production twin, so tests there are realistic.
- Fast rollback: going back is another flip of the switch, not a new deployment.
- Calmer releases: the risky work happens in private, and only the quick switch is visible to users.
Blue-green deployment does not make bad code good. It makes releasing and undoing a release safer and faster. Testing is still essential.
2Why It Exists: The Problem and the History
Releases used to mean downtime and dread. Blue-green deployment grew out of the wish to make releases boring.
The old way: update in place
In the traditional approach, the team stops the application on the production servers, copies new files over the old ones, runs any database changes, and starts the application again. During that time, the service is unavailable or half-updated. If something goes wrong, the team must repair or reverse the changes while users wait, often late at night and under pressure.
The cost of fear
Because releases were risky, teams did them rarely, and each release contained a large pile of changes. Big releases are harder to test and harder to debug, which made them even riskier. It was a vicious circle: fear led to big, rare releases, which caused the very failures that everyone feared.
The idea takes shape
The blue-green idea is generally credited to practitioners of continuous delivery, and it was popularised by Jez Humble and Dave Farley in their writing on the subject, and described clearly by Martin Fowler. The insight was simple: if you cannot make the release itself risk-free, make it reversible, and do the risky part where users cannot see it. Cloud platforms later made a second environment cheap to create, which turned a luxury into an everyday technique.
Updating in place
- Downtime or half-updated states during the release.
- Rollback means redoing work under pressure.
- Testing happens in an environment that differs from production.
- Releases are rare, large, and frightening.
Blue-green deployment
- The new version is ready before users arrive.
- Rollback is a quick traffic switch.
- Testing happens in a production twin.
- Releases become small, frequent, and calm.
A theatre that changes its stage set during the performance, with the audience watching, is asking for embarrassment. A theatre that prepares the next set behind a curtain and reveals it in a moment is doing blue-green deployment. The audience sees only the smooth change.
Real-world motivation
Many cloud platforms now include features that follow the same idea, such as deployment slots that can be swapped, or environment swaps that exchange addresses between two copies. Teams that release many times a day rely on such techniques, because a strategy that needs a night of downtime does not fit a working day full of releases.
3Core Concepts: The Words You Must Know
A handful of ideas explain almost everything about blue-green deployment.
Live environment
The environment that currently serves real users. Also called active or production-facing.
Idle environment
The identical twin that is not serving users. It receives the new version and is tested before the switch.
Router or traffic switch
The component that decides which environment users reach, such as a load balancer, a reverse proxy, a DNS record, or a service definition.
Health check
A small automatic test that asks an environment “are you working?” and expects a clear yes.
Smoke test
A short set of tests that confirms the most important functions work, run before and after the switch.
Rollback
Returning to the previous version. In blue-green, this means switching traffic back to the old environment.
Backward-compatible change
A database change that both the old and the new version of the application can work with at the same time.
Connection draining
Letting requests already in progress finish on the old environment before it stops receiving traffic.
Vocabulary table
| Word | Meaning | Example |
|---|---|---|
| Artifact | The built, versioned package that gets deployed | A container image tagged with a version |
| Parity | Two environments being alike in every way that matters | Same sizes, same settings, same versions of tools |
| Cutover | The moment traffic moves from one environment to the other | Flipping the load balancer target |
| Rollback window | The period during which the old environment is kept ready | One day after the switch |
| Downtime | A period when users cannot use the service | A maintenance page |
| Zero-downtime deployment | A release during which the service stays available | Blue-green, rolling, and canary strategies |
The colours are not permanent labels. Blue is not always production. After every release the roles swap, and a pipeline must always ask “which colour is live now?” rather than assuming.
4Architecture and Components
A blue-green setup has four main parts: the two environments, the router in front of them, the shared services behind them, and the Jenkins pipeline that controls it all.
The players
The router sits in front and sends users to the live environment. The blue and green environments each run a full copy of the application. Behind them, shared services such as the database, message queues, and storage serve both environments. Jenkins builds the artifact, deploys it to the idle environment, runs tests, and asks the router to switch. Monitoring watches both environments and the traffic.
flowchart LR
U["Users"] --> R["Router or Load Balancer"]
R -->|"Live traffic"| B["Blue Environment running version 1"]
R -.->|"No traffic yet"| G["Green Environment running version 2"]
B --> DB["Shared Database"]
G --> DB
J["Jenkins Pipeline"] -->|"1 Deploy new version"| G
J -->|"2 Test the idle environment"| G
J -->|"3 Switch traffic"| R
MON["Monitoring and Alerts"] --> B
MON --> G
MON --> J
Reading the diagram
Users always talk to the router, never directly to an environment. The router currently sends everything to blue, which runs version one. Jenkins deploys version two to green, tests it privately, and only then tells the router to switch. Both environments use the same database, which is why database changes need special care, as we will see. Monitoring watches both environments and the pipeline, so that problems after the switch are noticed within minutes.
Where the decisions live
| Question | Who knows the answer |
|---|---|
| Which colour is live? | The router, which is the single source of truth |
| What version runs in each colour? | The environments and the deployment records |
| Is the new version healthy? | Health checks, smoke tests, and monitoring |
| Who may switch traffic? | Pipeline permissions and approvals |
Picture an airport with two identical terminals and a signboard at the entrance saying which terminal is open. Staff can prepare the closed terminal freely. Passengers only ever follow the signboard. The signboard is the router, and whoever holds the pen for it holds the power of the release.
5The Traffic Switch: Ways to Flip
The switch is the heart of the technique. Different platforms offer different ways to build it, each with its own speed and risks.
Common switching mechanisms
| Mechanism | How it switches | Strength | Watch out for |
|---|---|---|---|
| Load balancer target | The balancer is told to send traffic to a different group of servers | Fast, precise, supports draining and health checks | The balancer itself is a critical piece to protect |
| Reverse proxy configuration | The proxy’s routing rule is changed and reloaded | Simple and flexible | Configuration mistakes and reload behaviour |
| Kubernetes service selector | The service that fronts the application is pointed at pods carrying a different label | Native to the platform, immediate | Both sets of pods must be ready before the switch |
| DNS record change | A name is changed to point to a different address | Works anywhere, needs no special equipment | Caching means some users keep reaching the old side for a while |
| Platform slot or address swap | The platform exchanges the addresses of two environments | Built in, one command | Behaviour varies by platform, so read its documentation |
Why DNS is the trickiest
DNS is the system that turns names into addresses. Answers are cached by many computers for a time called the TTL, or time to live. If you change a DNS record, some users continue to reach the old environment until their cached answer expires. That makes the cutover gradual and hard to reverse cleanly. Teams that use DNS often lower the TTL well before the release, and keep the old environment running long enough for caches to expire.
Properties of a good switch
Takes effect quickly
Seconds, not minutes. A slow switch turns rollback into a slow process too.
All at once, or cleanly gradual
Users should not bounce between old and new in a confusing way.
Easy to undo
The same mechanism must switch back just as quickly.
Reports its state
Anyone can ask “which environment is live?” and get a reliable answer.
Guarded by permissions
Only the pipeline, or named people, may change it.
Drains connections
Requests in progress finish before the old side is closed off.
Changing the number on a shop door is instant for everyone who walks past. Changing the phone book entry is slow, because old phone books remain in people’s drawers for months. A load balancer is the door number. DNS is the phone book.
If you have a choice, prefer a switch that lives inside your own infrastructure, such as a load balancer or a service selector, over one that depends on caches you do not control, such as DNS.
6Data Flow and Lifecycle of a Release
Follow one release from a finished build to a completed switch, and see who talks to whom at every step.
sequenceDiagram
participant J as Jenkins
participant AR as Artifact Store
participant G as Idle Environment Green
participant R as Router
participant B as Live Environment Blue
participant M as Monitoring
J->>AR: Build once and publish versioned artifact
J->>R: Ask which colour is live
R-->>J: Blue is live so Green is idle
J->>G: Deploy artifact to Green
J->>G: Run health checks and smoke tests
G-->>J: All checks pass
J->>J: Wait for human approval without holding an agent
J->>R: Switch traffic to Green with connection draining
R->>G: New requests now go to Green
J->>M: Watch errors and latency for a set period
M-->>J: Metrics healthy
J->>B: Keep Blue ready as rollback target
The lifecycle in plain steps
Build once
Jenkins builds the application and publishes a versioned artifact. This exact artifact is what will run in production.
Find the idle colour
The pipeline asks the router which environment is live, and picks the other one as the target.
Deploy to idle
The new version is installed in the idle environment. Nobody is using it, so nothing can break for users.
Test privately
Health checks and smoke tests run against the idle environment using an address that only the team can reach.
Approve
For important systems, a person confirms that the release should go live.
Switch
The router sends new traffic to the new environment. Requests in progress on the old side are allowed to finish.
Watch
Monitoring compares error rates and response times with normal values. If they worsen, the pipeline or a person rolls back.
Keep, then retire
The old environment stays ready during the rollback window, then is scaled down or reused for the next release.
Where the risk hides
Most of the release happens where users cannot see it. The only step that touches users is the switch, and it is short and reversible. That is the secret of the technique: the dangerous parts are moved into private time, and the visible part is made tiny.
Write each step’s name and result into the build log with a clear marker, such as “Green deployed,” “Smoke tests passed,” “Traffic switched.” At three in the morning, a readable log is worth a great deal.
A hotel changes its breakfast menu by preparing the new dishes in a second kitchen, tasting them, and then moving the serving station in one go. Guests notice only the moment of change, and the old kitchen stays warm in case a new dish turns out to be a disaster.
7The Database Challenge: Expand and Contract
Two environments are easy to duplicate. One shared database is not. This is the hardest part of blue-green deployment.
The problem
Both environments usually use the same database, because copying live data between two databases for each release is slow and risky. But the new version of the application may need a different table layout than the old one. If you change the layout for the new version, the old version, which is still live, may break. And if you roll back after users have written data in the new layout, the old version must still understand it.
The rule: every change must work with both versions
The solution is to make database changes backward compatible. At any moment during a release, both the old and the new application version must work correctly with the database as it stands. This is achieved with a technique often called expand and contract, or the parallel change pattern.
Expand
Add the new structure without removing the old one: a new column, a new table, or a new index. The old version ignores the additions, so it keeps working.
Migrate
Deploy the new application version, which can read and write both the old and new structures. Copy existing data across if needed, in small batches.
Switch
Move traffic to the new version through blue-green deployment. Rollback stays safe, because the old structure is still there.
Contract
Only in a later release, after the rollback window has passed and the old version is retired, remove the old structure that nobody uses.
Examples
| Change wanted | Unsafe way | Safe way |
|---|---|---|
| Rename a column | Rename it in one step | Add the new column, write to both, migrate, later drop the old one |
| Remove a column | Drop it with the release | Stop using it in code first, drop it a release or two later |
| Add a required field | Make it required at once | Add it as optional with a default, enforce it after the old version is gone |
| Change a data type | Alter the column in place | Add a new column of the new type, copy, switch, and retire the old |
The pipeline’s part
Database changes should be applied by a controlled step of the pipeline, before the switch, using a versioned migration tool. The migrations themselves must be reviewed for compatibility. Jenkins cannot decide whether a change is safe, but it can run the migration once, record it, and stop the release if it fails.
A destructive database change, such as dropping a column, makes rollback impossible. If the change is not reversible, the blue-green safety net has a hole. Treat such changes as separate, carefully planned releases.
When a library moves to a new shelving system, it first adds new labels while keeping the old ones, lets visitors use either, and removes the old labels only when nobody looks for them any more. Removing the old labels on the first day would strand every visitor who still uses them.
8State, Sessions, Caches, and Background Jobs
Besides the database, several other kinds of state can cause surprises at cutover.
User sessions
A session remembers who is logged in and what they are doing. If sessions are kept in the memory of one environment, switching traffic logs everybody out, because the new environment knows nothing about them. The fix is to keep sessions in a shared store that both environments can read, or in signed tokens that carry the information themselves.
Caches
A new environment starts with empty caches, so its first requests are slow, and the sudden load can overwhelm the database. This is called a cold start. Reduce it by warming up the new environment before the switch, for example by sending it a sample of realistic requests, and by switching traffic gradually where the platform supports it.
Background jobs and queues
Many applications run background workers that take messages from a queue, or scheduled jobs that run every hour. If both environments run these at once, work may be done twice: two emails sent, two invoices created. Decide which environment runs the workers and schedules, and switch them deliberately with the traffic. Make jobs safe to repeat where possible.
External connections
Other systems may talk to your environment: payment providers sending notifications, partners calling your address, or monitoring tools checking a particular URL. If these point to a fixed environment rather than to the router, they will break or go to the wrong side at cutover. Always give external parties the router’s stable address.
Sessions
Use a shared store or tokens, so users stay logged in through the switch.
Caches
Warm the new side before the switch, and expect a short period of extra load.
Workers and schedules
Run them in one environment at a time, and make them safe to repeat.
Uploaded files
Keep them in shared storage, not on the local disk of one environment.
External callbacks
Point outside systems to the router, never to a colour.
Configuration and secrets
Provide each environment with the correct values, held in a proper store.
When a shop moves to a new building overnight, the shelves, the till, the staff rota, and the phone number all need attention. Moving the stock is the easy part. Remembering the answering machine and the loyalty-card records is where things go wrong.
Stateless applications suit blue-green deployment best. The more state the application keeps inside an environment, the more careful the cutover must be.
9Blue-Green Compared with Other Strategies
Blue-green is one of several ways to release. Choosing well means knowing the alternatives.
| Strategy | How it works | Downtime | Rollback | Extra cost |
|---|---|---|---|---|
| Recreate | Stop the old version, start the new one | Yes, a gap between the two | Redeploy the old version | None |
| Rolling | Replace instances a few at a time | Usually none | Roll forward or roll back gradually | Small |
| Blue-green | Prepare a full twin, then switch all traffic | None or minimal | Switch back, very fast | A full second environment during the release |
| Canary | Send a small share of users to the new version first | None | Send the share back | Small, plus tooling |
| Feature flags | Ship code turned off, enable it for chosen users | None | Turn the flag off | Flag management |
flowchart TD
S["Choose a release strategy"] --> Q1{"Is downtime acceptable"}
Q1 -->|"Yes, for example internal tool at night"| RC["Recreate is simple enough"]
Q1 -->|"No"| Q2{"Can you run a second full environment during release"}
Q2 -->|"No, too costly or not possible"| RL["Rolling update or canary"]
Q2 -->|"Yes"| Q3{"Can database changes be made backward compatible"}
Q3 -->|"No"| FIX["Fix the data approach first or use careful maintenance windows"]
Q3 -->|"Yes"| Q4{"Need to test with a small share of real users first"}
Q4 -->|"Yes"| CAN["Blue-green plus a canary step"]
Q4 -->|"No"| BG["Blue-green deployment"]
How they combine
These strategies are not rivals. Many teams combine them: feature flags for controlling visible changes, blue-green for safe infrastructure switches, and canary steps for gradual exposure. Blue-green answers the question “how do I release safely and undo instantly?” while feature flags answer “who should see this feature, and when?”
Choose blue-green when
- Downtime is not acceptable.
- You want a very fast rollback.
- You can afford a second environment for a while.
- The application is mostly stateless.
Consider something else when
- A second full environment is too expensive.
- Data changes cannot be made compatible.
- You want to test with a small share of users first.
- The system is small and can tolerate short downtime.
There are many ways to repaint a house: close the house and repaint everything (recreate), paint one room at a time while living in the others (rolling), or build an identical house next door and move in on the day it is ready (blue-green). Each has its price and its comfort.
10Jenkins’s Role: The Shape of a Blue-Green Pipeline
Jenkins does not perform blue-green deployment by magic. It orchestrates a sequence of steps that other tools carry out.
What Jenkins is good at
A deployment is a series of careful, ordered steps with decisions between them. Jenkins pipelines excel at exactly that: running steps in order, stopping on failure, pausing for approvals, keeping a log, and using credentials safely. The actual switch is performed by a command or a call to the router, the platform, or a deployment tool, which the pipeline starts.
The stages of a typical pipeline
| Stage | Purpose | Runs on an agent? |
|---|---|---|
| Build and test | Compile, run unit tests, produce the versioned artifact | Yes |
| Publish artifact | Store the artifact in a registry | Yes |
| Detect live colour | Ask the router which environment is live | Yes |
| Deploy to idle | Install the new version in the idle environment | Yes |
| Verify idle | Health checks and smoke tests against the idle environment | Yes |
| Approval | A person confirms the release | No, waiting should not hold an agent |
| Switch traffic | Tell the router to send users to the new environment | Yes |
| Post-switch verification | Watch metrics and run checks against the live address | Yes |
| Finish or roll back | Mark success, or switch back if checks fail | Yes |
| Cleanup | Scale down or retire the old environment after the window | Yes, often a separate scheduled job |
Principles for the pipeline
- Build once. Never rebuild for production. Promote the same artifact that passed the tests.
- Ask, do not assume. Query the router for the live colour every time.
- Automate the boring parts, and keep a human decision where the risk justifies it.
- Make every step safe to repeat, so a failed run can be started again.
- Keep rollback one click away, in the same pipeline or a companion job.
Draw your pipeline on a whiteboard before you write it, with the live and idle environments, the router, and the decision points. If the picture is confusing, the pipeline will be too.
A stage manager does not perform in the play. She calls the cues: lights, curtain, music, and scene change, in the right order, and stops the show if something is wrong. A Jenkins pipeline is the stage manager of a release.
11Build Once, Deploy Many: The Artifact Rule
The package that reaches production must be the very same package that passed the tests. Anything else is a gamble.
What an artifact is
An artifact is the built, versioned result of your code: a container image, a package file, or an archive. It is stored in an artifact registry, a service designed to hold such packages, each with a unique version.
Why you must not rebuild
If the pipeline rebuilds the application for production, the result may differ from the tested build. A dependency may have released a new version, a build tool may behave differently, or a clock-dependent step may produce different output. You would be shipping something that was never tested. The rule is: build once, test that artifact, and promote it unchanged through every environment.
Identity by version
| Practice | Why it helps |
|---|---|
| Give every artifact a unique, immutable version | You always know exactly what is running |
| Never overwrite a version tag | A rollback returns to precisely the code that ran before |
| Record the artifact version in each environment | Anyone can ask what version runs in blue and in green |
| Store a checksum or signature | You can prove that the package was not altered on the way |
| Keep old versions available | Rollback and audits need them |
Configuration stays outside
The artifact must be the same for blue and green and for every stage. Differences between environments, such as addresses and credentials, are provided at run time from configuration and secret stores, not baked into the package. Then one package can run anywhere, and nothing environment-specific is hidden inside it.
One build
The pipeline builds exactly once per change.
One package
The same artifact goes to test, idle, and live environments.
Immutable versions
A version number never changes its content.
Config at run time
Settings come from outside the package.
A publisher prints one book, checks it, and ships that printing to all shops. Reprinting from the manuscript for each shop would produce copies that nobody checked. The artifact is the checked printing.
Using a moving tag such as “latest” for production deployments is a classic trap. Two deployments hours apart can install different code under the same name, and rollback becomes guesswork. Always deploy an exact version.
12Which Colour Is Live? Keeping One Source of Truth
Deploying to the wrong colour is the most dramatic mistake in this technique. The pipeline must always know which side is live.
The danger
If the pipeline deploys the new version to the environment that is currently serving users, the whole benefit is lost, and users may see a broken release directly. It sounds too silly to happen, but it does, especially when colours are stored in a place that can drift out of date.
Where the truth should live
| Place | Verdict |
|---|---|
| The router or platform itself | Best: it is the thing that actually decides where traffic goes |
| A tag or label on the environments, kept in sync with the router by the switch step | Acceptable if updated in the same operation as the switch |
| A parameter that a human types into the Jenkins job | Risky: humans make mistakes and forget |
| A file or variable stored in Jenkins | Risky: it drifts when someone switches manually or a build fails halfway |
| Assumption, such as “blue is always production” | Wrong: roles swap after every release |
Ask first, then act
The first step of the deploy stage should be a query to the router: “which environment are you sending users to?” The pipeline records the answer in the log and in the build description, picks the other colour as the target, and refuses to continue if the answer is unclear or contradictory, for example if traffic appears to go to both.
Guard rails
- Refuse to deploy to the live colour. A hard check that stops the pipeline.
- Confirm the idle colour is really idle, by checking that it receives no user traffic.
- Print the decision clearly, for example “Live is blue. Deploying version 2.4.0 to green.”
- Use locks so two pipelines cannot decide at the same time and collide.
Before painting a road line, a worker checks which lane is closed to traffic, by looking at the cones rather than trusting a note from last week. The router is the row of cones, and the pipeline should look at it every time.
Add the live colour and the version to the build’s display name, for example “release 2.4.0 to green.” The history of builds then reads like a diary of the switch.
13Deploying to the Idle Environment
The deploy step is the quiet, private part of the release. It should be boring, repeatable, and thoroughly logged.
What happens in this stage
- The pipeline fetches the exact artifact version from the registry.
- It applies backward-compatible database migrations, if the release has any.
- It installs or starts the new version in the idle environment.
- It provides configuration and secrets from the proper stores.
- It waits until the environment reports that it is ready.
Use the right tool for the deployment
Jenkins itself should not contain deep deployment logic. It calls a deployment tool that understands your platform: a container platform’s own commands, a package manager for the cluster, an infrastructure tool, or a configuration tool. The tool describes the desired state of the environment, and the pipeline runs it with the right parameters. This keeps the pipeline short and the deployment logic reusable and testable outside Jenkins.
Make it idempotent
An operation is idempotent if running it twice has the same effect as running it once. Deployment steps that describe the desired end state, rather than a list of actions, are naturally idempotent. This matters, because a pipeline may fail halfway and be run again, or a retry may repeat a step. Ask of every step: “What happens if this runs twice?”
Do not touch the live side
Give the deploy step credentials that can change the idle environment only, if the platform allows it. Then even a bug in the pipeline cannot damage the live side. Separate the permission to deploy from the permission to switch.
Start clean
The idle environment may still contain the previous release’s leftovers from two releases ago. Either replace it entirely, or make sure the deployment overwrites everything it should. An idle environment that has drifted from the live one defeats the promise of parity.
| Check | Why it matters |
|---|---|
| Correct artifact version deployed | Prevents deploying an old or wrong build |
| Configuration matches the live twin | Prevents surprises at cutover |
| Migrations applied and recorded | The database matches what the code expects |
| Environment reports ready | Do not test or switch before it is fully started |
| No user traffic reaches it yet | Keeps testing private |
Setting up the second track of a railway, the workers use a checklist for every rail, bolt, and signal, and they only work on the track that is closed. The checklist and the closed track together make the work safe.
14Health Checks and Smoke Tests on the Idle Side
The idle environment is a free testing ground. Use it to prove that the release works before any user sees it.
Health checks
A health check is a tiny automatic question: “are you alive and ready?” The application answers with a simple yes or no, possibly after checking its own connections to the database and other services. Platforms and routers use health checks to decide whether an environment may receive traffic. The pipeline uses them to wait until the new environment has started.
Waiting for readiness
A new environment takes time to start. The pipeline must wait, but not forever. Jenkins offers the waitUntil step, which repeats a check until it succeeds, with growing pauses, and the timeout step, which sets a firm limit. Together they say: “Keep asking whether green is healthy, for up to ten minutes, and fail if it never is.” Retrying such a read-only check is safe, because asking twice changes nothing.
Smoke tests
A smoke test is a short set of checks of the most important functions. The name comes from electronics: switch it on and see whether smoke comes out. Typical smoke tests include logging in, loading the home page, searching, and completing a small, harmless transaction. They run against the idle environment through a private address.
What to test on the idle side
Basic health
The application starts and answers its health check.
Key journeys
Login, main page, and the top few business actions work end to end.
Connections
The application reaches the database, queues, and other services.
Compatibility
The new version works with the current database and current data.
Performance sanity
Response times are in a reasonable range under a light load.
Configuration
Settings and secrets are present and correct for production.
Be careful with real data
The idle environment usually shares the live database. Smoke tests that create records will create them in production data. Use clearly marked test accounts, keep test actions small and harmless, and clean up afterwards. Never run destructive tests, such as deleting data or heavy load tests, against a shared production database.
A green health check does not prove the release is correct. It proves only that the application started. Real confidence comes from smoke tests of important journeys and from careful observation after the switch.
Before opening a new restaurant to guests, the chef cooks a full test meal, the manager checks the till and the lights, and a friend tries the front door. Each check is small, and together they catch most problems before a real customer arrives.
15Approvals Without Wasting an Agent
Many teams want a person to confirm a production switch. Jenkins supports this, but the way you write it matters for cost and capacity.
The input step
The input step pauses a pipeline and asks for a decision. It can limit who may answer by user or group, which lets you insist that only release managers may approve. It can also collect a short message or a choice. The pipeline continues only after somebody approves, or fails if someone declines or the wait times out.
Do not hold an executor while waiting
A pipeline that waits for approval inside an agent block holds an executor the whole time. A release that waits overnight for a manager occupies a build slot all night, and if many releases do this, the build queue grows for everyone else. Place the approval outside any agent block, so the pipeline waits on a lightweight slot that does not consume a normal executor, and take an agent again when the release resumes.
What a good approval shows
- The version and the change summary or a link to it.
- The results of the tests and smoke checks on the idle environment.
- Which colour will become live, and which one is live now.
- A link to the dashboards to watch after the switch.
- A clear time limit, after which the release is cancelled rather than left hanging.
When automation is enough
| Situation | Approval approach |
|---|---|
| High-risk or regulated system | Human approval, named approvers, recorded |
| Frequent low-risk releases with strong automated checks | Automatic switch when all checks pass |
| Business-hours restriction | Automatic checks plus a rule that the switch happens only in agreed hours |
| First releases after a big change | Manual approval until confidence grows |
A theatre director gives the final “go” for the curtain, but does not stand holding the curtain rope for an hour while waiting. She sits down, and stands up when the moment comes. A pipeline should hand back its machine while it waits for the “go.”
Always set a timeout on approvals, for example a few hours. A forgotten approval that stays open for days leaves an idle environment with an old, half-verified version waiting to be switched by accident.
16Switching Traffic Safely
The switch is short, but it deserves the most care of any step.
Before the switch
- Confirm that the idle environment is still healthy right now, not only ten minutes ago.
- Warm up caches and connection pools if the application needs it.
- Check that background workers and schedules are ready to move.
- Make sure the rollback path works: the old environment is healthy and untouched.
- Take a lock so that no other pipeline changes the router at the same time.
Options for the cutover
Instant switch
All new requests go to the new environment immediately. Simple, and the fastest to roll back, but every user is exposed together.
Weighted shift
Traffic moves in steps, such as ten, fifty, then a hundred percent, watching metrics between steps. Combines blue-green with canary ideas.
Internal first
Staff and test users reach the new environment first, using a header or a rule, before everyone else.
Off-peak switch
The switch happens at a quiet hour, so any problem affects fewer users.
Connection draining
Requests already in progress on the old environment should be allowed to finish. The router stops sending new requests to the old side but keeps existing connections open for a short time, often thirty seconds to a few minutes. Without draining, users in the middle of a payment or an upload may see an error at the moment of the switch.
Make the switch itself idempotent
A switch step that says “send traffic to green” is safe to run twice. A switch step that says “flip to the other colour” is not: run twice, it flips back. Always state the desired end state explicitly, so a retry or a re-run cannot accidentally reverse the release.
Verify immediately
Right after the switch, the pipeline runs a few checks against the public address, not the private one: does the home page load, does login work, and are the error rates normal? If anything fails within the first minutes, the pipeline switches back automatically or alerts a person who can.
A “toggle” style switch, which flips whatever is live, is dangerous in automation. If a pipeline step is repeated by a retry or a rerun, the release silently reverses. Always say which colour should be live, never simply “the other one.”
A ferry changing berths lets the passengers who are already walking on the gangway finish boarding before it pulls the ramp. Connection draining is the same courtesy for requests that are already on the way.
17Rolling Back: The Big Payoff
The reason many teams adopt blue-green deployment is the speed and calm of rollback.
What rollback means here
Because the old environment is still running the old version, rollback is simply another traffic switch, in the opposite direction. There is no rebuild, no redeployment, and no waiting for servers to start. It can take seconds. The old version returns to service exactly as it was, with its caches still warm and its configuration untouched.
flowchart TD
A["Pipeline starts release"] --> B["Build once and publish artifact"]
B --> C["Detect live colour and choose idle target"]
C --> D["Deploy to idle and run smoke tests"]
D -->|"Fail"| X["Stop release, live side untouched"]
D -->|"Pass"| E["Approval without holding an agent"]
E -->|"Declined or timed out"| X
E -->|"Approved"| F["Switch traffic to new colour"]
F --> G["Watch metrics and run live checks"]
G -->|"Healthy"| H["Mark success and keep old side for rollback window"]
G -->|"Unhealthy"| R["Switch back to old colour"]
R --> S["Investigate while users are safe"]
H --> T["After window: scale down or reuse old side"]
When to roll back
| Signal | Response |
|---|---|
| Error rate jumps above the agreed threshold | Roll back at once, investigate later |
| Response times worsen noticeably | Roll back or reduce traffic while checking |
| A key business function fails | Roll back |
| Minor cosmetic issue, no functional harm | Do not roll back, fix in the next release |
| Data problems suspected | Roll back traffic and pause writers, then investigate carefully |
Manual and automatic rollback
Some teams roll back automatically when monitoring shows that the new version crosses a clear threshold. Others require a person to decide. Automatic rollback is fast but needs reliable, well-tuned signals, so that a harmless blip does not reverse a good release. Whichever you choose, provide a manual rollback job that anyone on call can run without help.
The limits of rollback
- Data changes: rolling back the application does not undo data that the new version wrote. This is why compatible data changes matter.
- External effects: emails sent and payments made by the new version cannot be unsent.
- Old side must still exist: once the old environment is scaled down or reused, rollback again requires a deployment.
A mountaineer who climbs with a rope fixed to the last safe ledge can step back in seconds if the next ledge crumbles. The old environment is the ledge, the switch is the rope, and rollback is the step back.
Practise the rollback. A rollback job that has never been run is like a fire door that has never been opened. Run it in a test environment regularly, and time it.
18After the Switch: The Rollback Window and Cleanup
The release is not finished at the switch. The old environment is an asset for a limited time, and then a cost.
The rollback window
Keep the old environment running, untouched, for a period that matches your risk. Some problems appear only after hours or days, such as a monthly report that fails, or a slow memory leak. A typical window is between a few hours and a couple of days. Shorter windows save money. Longer ones give more safety.
What to do with the old environment
| Option | Benefit | Cost or risk |
|---|---|---|
| Keep it running at full size | Instant, full-capacity rollback | Double the running cost |
| Scale it down to a minimum | Cheaper, still quick to restore capacity | Rollback needs a brief scale-up |
| Stop it but keep the definition | Very cheap | Rollback needs a restart, and takes minutes |
| Destroy it and recreate on demand | No idle cost, always fresh | Slowest rollback |
Cleanup jobs
A separate scheduled Jenkins job can retire old environments after the window ends: check that no rollback was requested, scale down or remove the old side, and record the action. Keeping cleanup separate from the release keeps the release pipeline short and lets the window be set by policy.
Reuse as the next idle side
The old live environment becomes the idle target of the next release. Before reuse, make sure it is brought back into line with the live one: same infrastructure, same configuration, updated tools. A neglected idle environment slowly drifts from production, and the next release finds surprises there.
Costs to watch
Do not scale down or reuse the old environment the moment the switch succeeds. Give the new version time to prove itself, including the slow, rare events such as end-of-month jobs.
After moving house, you keep the keys to the old flat for a week, in case you left something behind or the new place has a problem. You do not hand the keys back on the first evening.
19Parameters, Locks, and Concurrency in the Release Job
A release pipeline is a sensitive job. Small Jenkins features keep it from being started wrongly or twice.
Parameters for controlled choices
Jenkins parameters are inputs supplied when a run starts. A release job commonly has a few: the artifact version to deploy, and the action to perform, such as deploy and switch, deploy only, switch only, or rollback. Use choices and checkboxes instead of free text where you can, validate every value in the first stage, and default to the safe option. Do not offer a parameter for the colour, because the pipeline should discover it.
Only one release at a time
Two releases running together against the same router would be chaos: two pipelines might both decide green is idle and overwrite each other. Use the disableConcurrentBuilds option so a second run waits or fails, and use a lock, such as one from the Lockable Resources plugin, named after the application. The lock ensures that only one pipeline may change the environments and the router at a time.
Order matters
If two releases are queued, you usually want the newest to win, not the oldest. Milestone steps can cancel older runs when a newer one arrives. But be careful: never cancel a release that has already switched traffic, because it may leave things half-done. Place milestones before the risky steps.
Who may run it
| Action | Who should be allowed |
|---|---|
| Deploy to idle only | Developers and release engineers |
| Switch production traffic | Release managers, or the pipeline after approval |
| Roll back | On-call engineers and release managers |
| Change the job itself | A small group, through reviewed changes to the Jenkinsfile |
Use the right agents
Release jobs hold powerful credentials. Run them on a dedicated pool of agents with their own label, so ordinary builds and untrusted pull request builds can never share those machines. This also protects the release from a busy build queue, because releases have their own capacity.
A control tower lets one aircraft use a runway at a time, and only authorised controllers give the clearance. Locks, single-run rules, and permissions are the release pipeline’s control tower.
Show the lock name and the current holder in the log when a release waits. “Waiting for lock held by build 118” saves many confused messages.
20Standardising with a Shared Library
If every team writes its own blue-green pipeline, you get dozens of styles and dozens of ways to fail. A shared library gives you one good way.
The problem with scattered pipelines
Without a common approach, one team checks the live colour and another assumes it. One drains connections and another does not. Some verify after the switch and some forget. When a security or safety improvement is needed, hundreds of Jenkinsfiles must be edited.
The shared library idea
A Jenkins shared library is reusable pipeline code kept in its own repository and loaded by many pipelines. It can provide a single custom step, for example a step that performs a complete blue-green release. The Jenkinsfile of each application then says only a few things: the application name, the artifact version, the health check address, and how long to watch after the switch.
What a standard release step can do
- Discover the live colour from the router and refuse to touch it.
- Take the application lock and apply the single-run rule.
- Deploy the exact artifact to the idle environment.
- Wait for readiness with a timeout, and run the standard smoke tests.
- Request approval outside any agent, with a standard message.
- Switch traffic with draining, stating the end state explicitly.
- Watch metrics for a period and roll back on defined thresholds.
- Write clear log lines and send standard notifications.
- Record the deployment for auditing.
Design guidelines
| Choice | Guideline |
|---|---|
| Settings | Named options with safe defaults and early validation |
| Hooks | Allow application teams to add their own tests at agreed points |
| Versions | Release the library with semantic versions, and let teams pin them |
| Testing | Test the step against a simulated router and both colours |
| Rollback | Ship a companion step so rollback uses the same trusted code |
A shared step that controls production is a powerful tool and a single point of failure. Protect the library repository with code review, test it heavily, and release changes gradually, so that one mistake cannot break every team’s releases at once.
An airline uses one standard pre-flight checklist for every aircraft, not a personal version for each pilot. The checklist is improved centrally, and every crew benefits. A shared release step is the pre-flight checklist for your deployments.
21Security: Guarding the Switch
Whoever can flip the switch controls what every user sees. Treat that power with the seriousness it deserves.
What is at risk
A release pipeline holds credentials that can deploy software and change routing. An attacker who takes over the pipeline can ship malicious code to production, redirect traffic, or take the service down. A careless person can do similar harm by accident.
Defences
Separate permissions
Permission to deploy to the idle side is different from permission to switch production traffic. Grant each to the right people only.
Proper secret handling
Use Jenkins credentials or an external secret store. Fetch secrets at the moment of use, keep them out of logs, and give each environment its own.
Trusted code only
Release jobs run only reviewed code from protected branches. Pull requests from strangers never reach these credentials.
Dedicated agents
Release builds run on their own clean, restricted agents.
Trusted artifacts
Deploy only artifacts that were built by the pipeline, stored in your registry, and verified by checksum or signature.
Record everything
Log who approved, who switched, which version, and when, in a place that pipeline users cannot edit.
Protect the idle environment
The idle environment is a full copy of production, often with production data behind it. It must not be open to the public just because it is idle. Give it a private test address, restrict it to the team and the pipeline, and apply the same security controls as the live side. An unguarded idle twin is an easy target.
Separation of duties
In regulated settings, the person who writes a change should not be the only person who releases it. Use approvals by a different person, with the approval recorded. Jenkins can restrict who may answer an approval prompt and can write the decision to the log.
Least privilege in the platform
Give the pipeline’s platform identity only the rights it needs: change the router entry for this application, deploy to these two environments, read this registry. It should not be able to delete the database, change other applications, or alter its own permissions.
Safer habits
- Narrow, separate roles for deploy and switch.
- Approvals by named people, recorded.
- Private, protected idle environment.
- Signed or verified artifacts.
Risky habits
- One powerful credential used everywhere.
- Idle environment reachable from the internet.
- Release jobs runnable by anyone.
- Rebuilding for production from unreviewed code.
A railway signal box is locked, and only trained signallers may pull the levers. Nobody argues that this slows things down, because a wrong lever means a crash. The switch of a production system deserves the same locked door.
22Performance and Capacity During a Release
The moment of cutover is one of the most demanding times for a system. Plan for it.
Capacity for both sides
For a period, you run two full environments. The new one must be able to handle the whole production load the instant the switch happens, not a fraction of it. Check that the idle environment is sized like the live one. A smaller idle side that is fine in tests may collapse under real traffic.
Warm-up
New environments have empty caches, cold connection pools, and code that has not yet been compiled by the runtime’s just-in-time optimisers. The first requests are slow. A sudden flood of cold requests can also overwhelm shared resources such as the database. Warm up before the switch by sending representative requests, preloading common data, and starting with a gradual shift when possible.
Pressure on shared services
Both environments share the database and other backends. During the release, migrations, cache filling, and health checks add load. Avoid heavy migrations during peak hours, and run them in small batches. Watch the database’s connection count, since two environments may each hold a full pool of connections.
Scaling rules
| Concern | Practice |
|---|---|
| Sizing of the idle side | Match the live side, or scale up before the switch |
| Autoscaling | Make sure the new side’s scaling rules are active and tested before it takes traffic |
| Connection limits | Count connections from both sides, and raise limits if needed |
| Warm-up | Send realistic requests before the cutover |
| Timing | Prefer quiet hours for the switch on very busy systems |
Pipeline performance
The release pipeline itself should be quick, because the sooner the switch happens, the sooner the team can watch the result. Deploy in parallel where independent, cache dependencies, and keep smoke tests short. A release that takes three hours discourages frequent releases.
A new bridge is tested with heavy trucks before it opens to traffic, because it must carry the full rush hour on day one. Warm-up and capacity checks are the trucks on the new bridge.
Compare the idle and live environments’ key sizes in the pipeline before the switch, such as instance counts and memory limits, and stop the release if they differ from the agreed match.
23High Availability and Reliability
Blue-green deployment improves availability during releases, but it introduces its own single points of failure.
What it improves
- No downtime for deployments, because the live side is never touched.
- Fast recovery from bad releases, through an immediate switch back.
- Less pressure during releases, which reduces human error.
What it does not protect against
- Failures unrelated to releases, such as a broken database or a cloud outage.
- Bad data written by the new version, which rollback does not undo.
- Failure of the switch itself, such as an overloaded or misconfigured router.
The router as a critical component
Everything depends on the router. It must be highly available: run it redundantly, monitor it, and protect its configuration. Changes to it should be as reviewed as code. A mistake in the router’s configuration can send all traffic nowhere.
When the pipeline itself fails
| Failure point | Result | Defence |
|---|---|---|
| Build or test fails | Release never starts, live side untouched | Fix and rerun |
| Deploy to idle fails | Live side untouched | Clean up idle, rerun; steps must be repeatable |
| Jenkins fails after the switch | The release is live but unwatched | Monitoring and alerts independent of Jenkins, manual rollback path |
| Switch step fails halfway | Router state unclear | Explicit end-state switch, and a query to verify the result |
| Rollback fails | The bad version stays live | Practise rollback, keep an alternative path such as a direct router change |
Multiple regions
Larger systems run in several regions or zones. Releasing region by region, one after another, limits the blast radius. If the first region shows trouble, the release stops before it reaches the others. Blue-green within each region gives fast rollback locally.
Never make the release pipeline the only way to change the router. If Jenkins is down when a rollback is needed, an on-call engineer must have a documented, tested alternative.
A hospital with two operating theatres can keep working while one is cleaned and prepared, but it still depends on the corridor between them. The corridor is the router, and it needs a plan of its own.
24Monitoring, Logging, and Metrics
The switch is only safe if you can see what happens after it. Monitoring is the pilot’s window of the release.
What to watch after the switch
Compare blue and green
The best evidence comes from comparison. Tag every metric and log line with the environment colour and the application version. Then you can put the two sides next to each other: error rates on the new side against the old side’s baseline. A jump that appears only on the new colour points straight at the release.
Business signals
Technical health is not enough. Watch the numbers that matter to the business: sign-ups, orders, payments completed, search results clicked. A release that passes every technical check but halves the number of completed orders has still failed.
Delivery metrics
Beyond a single release, track how the whole delivery process performs. Widely used measures, often grouped as the four key metrics of software delivery, are how often you deploy, how long a change takes to reach production, how often releases cause failures, and how quickly you recover. Blue-green deployment should improve all four over time.
Alerts and thresholds
| Signal | Action |
|---|---|
| Error rate above threshold for a few minutes | Automatic or manual rollback |
| Latency well above baseline | Pause, investigate, consider rollback |
| Business metric drops sharply | Investigate at once, consider rollback |
| Health checks failing on the new side | Rollback and stop the release |
| Old side receiving unexpected traffic | Check DNS caches, hard-coded addresses, or a partial switch |
Logs for the release itself
Keep the pipeline’s own log clear and complete: the live colour, the version, checks, approvals, the switch result, and the watch period. Send notifications to the team’s chat at each key moment, so everyone knows what is happening.
Decide the rollback thresholds before the release, not during it. In the middle of an incident, people argue about whether a number is “bad enough.” A number agreed calmly in advance ends the argument.
Pilots watch instruments constantly after take-off, with agreed limits for when to abort. They do not invent the limits in the air. Release monitoring works best with limits chosen on the ground.
25Testing the Whole Process
A release process is software too. Test it, especially the parts that only run on bad days.
What to rehearse
A full release in a test setup
Run the entire pipeline against a test router and two test environments. Confirm that colours swap correctly, twice in a row.
Rollback
Switch, then switch back, and time both. Confirm that the old side serves correctly.
Failing smoke tests
Break a test on purpose and confirm that the pipeline stops and the live side stays untouched.
Half-finished runs
Kill the pipeline midway and confirm that a rerun completes safely and does not flip back.
Database compatibility
Run the old version against the migrated database, and the new version against the old data.
Concurrent releases
Start two releases at once and confirm that the lock lets only one proceed.
Test the compatibility
The single most valuable automated test for blue-green deployment is one that runs the old application version and the new one against the same database at the same time, and checks that both work. If both pass, rollback is safe at the data level.
Game days
Once or twice a year, hold a rehearsal in which you practise an incident: a bad release that must be rolled back, or a router that fails. Time the response. Gaps in alerts, permissions, and documentation appear quickly.
Keep the test setup real enough
A test router and small test environments built from the same infrastructure code as production give realistic results. A completely different setup will hide problems. Reuse the same pipeline code, with different settings.
A ship’s crew practises abandoning ship in calm water, so that they know what to do in a storm. Rollback drills in a test environment serve the same purpose for your release team.
26Environments as Code: Keeping Blue and Green Identical
The promise of blue-green deployment depends on the two environments being twins. Code is what keeps them that way.
The parity problem
If blue and green are set up by hand, they drift apart. One gets a patched operating system, the other a forgotten manual change. The release then behaves differently on each side, and the tests on the idle environment tell you less about what will happen on the live one.
Infrastructure as code
Infrastructure as code means describing servers, networks, and platform settings in text files kept in a repository, and creating them with automated tools. Both colours are created from the same description, so they cannot drift, and any change is reviewed. The Jenkins pipeline itself can run these tools, or a separate pipeline can keep environments up to date.
What must match
| Item | Should match? |
|---|---|
| Machine size, count, and scaling rules | Yes |
| Operating system and runtime versions | Yes |
| Network rules and security groups | Yes, except the private test address |
| Configuration values | Yes, apart from the environment’s own name |
| Secrets | Same purpose, held in the proper store for each |
| Deployed application version | No, this is what differs during a release |
Containers and Kubernetes
On a container platform, blue and green can be two sets of pods in the same cluster, labelled with the colour. A service definition points at one label, and the switch changes the label it selects. The pipeline updates the deployment for the idle colour, waits for its pods to be ready, and then changes the service. Because both sets use the same cluster settings, parity comes almost for free.
Jenkins as code too
The Jenkins controller, its agents, and its jobs should also be described in code, for example with the Jenkins Configuration as Code plugin and Jenkinsfiles in repositories. A release system that cannot be rebuilt from files is a fragile foundation.
Use case: a stale idle twin
A team keeps green idle for months between releases and applies operating system patches only to blue. When they finally deploy to green, the new version fails on an old library. Since then, they rebuild the idle side from code before every release, and the problem has not returned.
Rebuild or refresh the idle environment from code at the start of every release. It costs a few minutes, and it turns “we think they match” into “they match.”
27Advantages, Disadvantages, and Trade-offs
Blue-green deployment is a powerful technique, and it costs something.
Advantages
- Releases without downtime.
- Very fast, simple rollback.
- Testing in a production twin before users arrive.
- Less stress and fewer late-night releases.
- Encourages automation, environment parity, and discipline.
- Makes small, frequent releases practical.
Disadvantages
- Double infrastructure during the release and rollback window.
- Database changes need careful, compatible design.
- State, sessions, and background jobs need attention.
- All users switch at once unless combined with gradual shifts.
- Extra complexity in the pipeline and the router.
- A bad release can still cause harm that rollback cannot undo.
Balancing choices
| Choice | Cheaper and simpler | Safer and smoother |
|---|---|---|
| Idle environment | Create it only during releases | Keep a warm twin always ready |
| Rollback window | A few hours | A few days |
| Cutover | All at once | Gradual shift with checks |
| Approval | Automatic on green checks | Human approval with named approvers |
| Verification | Health check only | Smoke tests plus business metrics |
Is it worth it?
For a small internal tool that nobody uses at night, a simple restart may be enough. For a service that earns money every minute, or that many teams depend on, the cost of a second environment is small compared with the cost of downtime. There is no universal answer. Compare the cost of the extra environment with the cost of an outage and of a slow, frightening rollback.
28Design Patterns and Anti-patterns
A pattern is a solution many teams discovered independently. An anti-pattern is a trap many teams fell into.
Patterns worth copying
Router as the source of truth
The pipeline asks the router which colour is live, every time.
Build once, promote
The same immutable artifact goes through every stage.
Expand and contract
Every database change works with both the old and new versions.
Explicit end-state switch
The switch says which colour should be live, so repeating it is harmless.
Watch, then decide
Agreed thresholds decide rollback, with a manual job always available.
Twins from code
Both environments are created from the same infrastructure code.
Anti-patterns to avoid
The Toggle Switch
A switch step that flips to “the other colour.” A rerun or retry reverses the release without anyone noticing.
Better approach
State the desired colour explicitly, and verify the result by asking the router.
Rebuild for Production
Building a fresh package for the production deployment instead of promoting the tested artifact.
Better approach
Build once, tag immutably, and deploy exactly that artifact everywhere.
The Destructive Migration
Dropping or renaming a column in the same release that stops using it, making rollback impossible.
Better approach
Use expand and contract, and remove old structures in a later release.
Drifting Twins
An idle environment that is never maintained, so it differs from the live one in size, versions, and settings.
Better approach
Create both from code, and refresh the idle side before each release.
Deleting the Safety Net Early
Scaling down or reusing the old environment the moment the switch succeeds, removing the quick rollback path.
Better approach
Keep the old side for a defined rollback window that matches your risk.
The Silent Switch
Switching traffic and walking away, with no verification and no monitoring comparison.
Better approach
Verify immediately and watch agreed metrics for a defined period, with clear rollback thresholds.
29Best Practices and Common Mistakes
These lessons are usually learned the hard way. You can learn them the easy way.
Best practices
- Query the router for the live colour, and refuse to deploy to it.
- Build once and deploy an exact, immutable version everywhere.
- Make every database change backward compatible, and contract in a later release.
- Test the idle side with health checks and smoke tests through a private address.
- Put approvals outside agent blocks, with named approvers and a time limit.
- State the end state of the switch, drain connections, and verify immediately.
- Agree rollback thresholds in advance, and keep a manual rollback job ready.
- Keep the old side for a rollback window, then retire or refresh it.
- Use a lock and single-run rule so only one release changes the router at a time.
- Create both environments from code, and give release jobs their own agents.
- Tag metrics and logs with colour and version, and compare the two sides.
- Rehearse releases and rollbacks regularly.
Common mistakes and their fixes
| Mistake | What happens | Fix |
|---|---|---|
| Deploying to the live colour | Users see the untested release directly | Query the router, add a hard guard |
| Incompatible database change | Old or new version breaks, rollback impossible | Expand and contract |
| DNS switch with a long cache time | Users split between old and new for hours | Prefer a router switch, or lower the TTL in advance |
| Background jobs on both sides | Duplicate emails and invoices | Run workers in one environment, make jobs repeatable |
| Sessions stored in memory | Everyone is logged out at the switch | Shared session store or tokens |
| Cold caches after cutover | Slow responses, database overload | Warm up, shift gradually |
| Callbacks pointing to one colour | Outside systems break at cutover | Give them the router’s address |
| Old side removed too early | No quick rollback when a slow problem appears | Keep a defined rollback window |
| Rollback never rehearsed | It fails when needed most | Practise it and time it |
The release is “successful” and the team relaxes. Three hours later the monthly report job fails on the new version, and the old environment has already been scaled to zero. Match the rollback window to the slowest thing that could go wrong.
30A Worked Example: Introducing Blue-Green to a Web Shop
Let us walk through a complete introduction for a fictional company, step by step, without writing any code.
The situation
Fernhill Books runs an online shop as a web application with a shared database. Releases happen every second Thursday at ten at night, using an in-place update that takes the shop offline for twenty minutes. Twice last year, a release failed and the shop stayed down for over an hour. The team wants to release during the day, without downtime, and to undo mistakes in minutes.
Step by step
Check the fit
The application keeps sessions in a shared store and files in shared storage, so it is mostly stateless. A second environment is affordable. Database changes are the main concern.
Create the twins from code
The team describes the shop’s servers, network rules, and settings in infrastructure code, and creates blue and green from the same description, with a load balancer in front.
Agree the data rules
A written rule says every migration must work with both the old and new application versions. Destructive changes wait for a later “contract” release.
Move workers and schedules
Background workers and the nightly stock job run only in the live environment, and they move with the switch. The jobs are changed so that running twice does no harm.
Build the pipeline
The Jenkinsfile builds once, publishes an immutable image, asks the load balancer for the live colour, deploys to the other, and waits for readiness with a time limit.
Add smoke tests
Ten short checks run against the idle side through a private address: home page, search, login, add to basket, and a test order with a marked test account.
Add approval and switch
A release manager approves, outside any agent. The pipeline then tells the load balancer explicitly to send traffic to the new colour, with a two-minute drain.
Watch and roll back
For thirty minutes, the pipeline compares error rate, latency, and order completion against the old side. Agreed thresholds trigger a rollback. A manual rollback job also exists.
Rehearse
The team runs three full releases and two rollbacks in a test setup, and one game day with a broken release.
Go live and tidy
The first real release goes out at noon on a Tuesday. The old side stays for two days, and then a cleanup job scales it down. Releases become weekly, then more frequent.
The result after three months
| Measure | Before | After |
|---|---|---|
| Downtime per release | About twenty minutes | None |
| Time to undo a bad release | An hour or more | Under two minutes |
| Release frequency | Every two weeks | Several per week |
| Release timing | Ten at night | During working hours |
| Team stress | High | Much lower |
The technology was the smaller part. The biggest gains came from the data rules, the rehearsals, and the habit of watching agreed numbers after every switch.
31Migrating from In-Place Deployments
Most teams do not start with blue-green. They arrive at it after living with the pain of in-place releases. Here is a calm way to move.
What you may find
- A single production environment updated by copying files or running a script by hand.
- A test environment that differs greatly from production.
- Database changes applied manually, sometimes with no record.
- Servers that were changed by hand over the years, nobody quite sure how.
- Release notes and steps that live in one person’s head.
A safe migration path
Automate the plain deployment first
Before adding a second environment, make the existing deployment a repeatable pipeline. Build once, deploy from an artifact, and record versions.
Put the environment in code
Describe the servers and settings as code, so that a second copy can be created reliably.
Make the application stateless enough
Move sessions, uploads, and scheduled work to shared stores or single-environment control.
Adopt compatible data changes
Change the team’s migration habits to expand and contract, starting with the next release.
Introduce a router
Put a load balancer or similar switch in front of the existing environment, so users always reach the application through it.
Create the second environment
Build the twin from code, deploy the current version to it, and test it privately.
Practise the switch on a quiet day
Switch to the twin with no change in version, and switch back. Prove that the mechanics work before a real release depends on them.
Run the first real blue-green release
Pick a small, low-risk release, keep the old side for a generous window, and review what happened.
Do not try to change the deployment method and the release contents at the same time. Adopt blue-green with a boring release, so that any problem clearly belongs to the new process and not to the new features.
The reward
After migration, releases become routine events. The team gains the confidence to release small changes often, and the fear that once surrounded deployment day fades.
32Governance: Owning the Release Process
A release process decays if nobody owns it. Clear roles and written agreements keep it healthy.
Who owns what
| Area | Owner | Responsibility |
|---|---|---|
| Release pipeline and shared library | Platform team | Correctness, safety features, versions, support |
| Router and environments | Infrastructure or operations team | Availability, parity, security, cost |
| Application changes and migrations | Application team | Backward-compatible data changes, smoke tests, readiness checks |
| Approvals for production | Release managers or product owners | Deciding when to go live |
| Rollback decisions | On-call engineer | Acting quickly on agreed thresholds |
Agreements to write down
- The data rule: every migration works with both versions, and destructive changes wait.
- The rollback rule: thresholds, who decides, and how long the window is.
- The approval rule: who may approve which releases, and when automatic switching is allowed.
- The parity rule: both environments come from the same code, and the idle side is refreshed before each release.
- The freeze rule: periods when production switches are not allowed, such as peak sales days.
Regular reviews
After every release that needed a rollback, hold a short, blame-free review: what happened, what signal caught it, how long did it take, and what will change. Once a quarter, review the whole process: window lengths, costs, rehearsal results, and near misses.
Every release is reversible
If a change cannot be undone, it gets extra planning and a separate approval.
Rehearse before relying
New pipelines and new environments are practised before real use.
Write down the thresholds
Rollback numbers are chosen calmly and shared widely.
Learn from every incident
Reviews change the process, not only the documents.
An airline does not leave flight procedures to each captain’s mood. It writes them, trains crews, rehearses emergencies, and reviews every incident. A release process needs the same shared, rehearsed rules.
33Real-World and Industry Examples
These scenarios show how organisations use blue-green deployment. They are illustrative, so you can map them to your own situation.
Scenario: the daytime release
A retail company used to release only at night because of downtime. After adopting blue-green deployment, the team releases at midday, when everyone is at their desks and ready to help. Problems are noticed and fixed faster, and nobody works through the night any more.
Scenario: the two-minute rollback
A release contains a subtle bug in the checkout page. Ten minutes after the switch, the order-completion metric falls. The on-call engineer runs the rollback job, traffic returns to the old side, and orders recover within two minutes. The team fixes the bug calmly the next day.
Scenario: the migration that saved the day
A team renames a database column using expand and contract: add the new column, write to both, switch, and drop the old column three releases later. When a release is rolled back, the old version still finds its data. A team that had renamed the column in one step would have been unable to roll back.
Scenario: the cold cache
A high-traffic site switches instantly and watches its database fall over under the flood of cold requests. Afterwards, the team adds a warm-up step and a gradual traffic shift. The next release passes without a spike, and the database stays calm.
Scenario: the double emails
After a release, customers receive every notification twice. Both environments were running the background workers. The team changes the pipeline so workers run only in the live environment, and makes the notification job safe to repeat. The problem never returns.
What these examples share
In every good outcome, the team treated the release as a process with rules: compatible data, private testing, an explicit switch, watched metrics, and a practised rollback. In the bad outcomes, one of these was missing, usually the data rule or the watching.
34How Other CI/CD Tools Handle This
The blue-green idea is independent of any single tool. Comparing platforms shows what is essential.
| Tool or platform | How blue-green is supported | Notable difference |
|---|---|---|
| Jenkins | Pipelines orchestrate any platform’s switch mechanism | Very flexible, and you assemble the safety features yourself |
| Cloud application platforms | Deployment slots or environment swaps as built-in features | The swap is one operation, with platform-specific behaviour |
| Kubernetes-based delivery tools | Some offer blue-green and canary release strategies as built-in options | The strategy is a setting, and traffic control is integrated |
| Cloud deployment services | Managed blue-green deployments for virtual machines and containers | Traffic shifting and rollback are provided by the service |
| Hosted CI services | Pipelines call the same platform tools, with workflow features for environments and approvals | Similar orchestration role to Jenkins |
Features change quickly, so check each tool’s current documentation before relying on this table.
What Jenkins does differently
Jenkins is a general orchestrator. It does not itself contain a blue-green feature; it runs your steps in order, with approvals, locks, and credentials. Many teams pair it with a specialised delivery tool that manages the traffic shifting and rollback logic, while Jenkins builds, tests, and starts the release.
What is universal
The lessons apply everywhere: build once, keep a twin environment, test privately, make data changes compatible, switch explicitly, watch agreed metrics, keep the old side for a window, and rehearse rollback.
35Advanced Topics: What to Learn Next
Once the basics feel comfortable, these ideas help larger and more demanding teams.
Gradual traffic shifting
Instead of moving all users at once, shift traffic in stages, watching metrics between steps. This blends blue-green with canary ideas: the new side gets five percent, then twenty-five, then everything. If a problem appears early, only a few users are affected.
Feature flags
Feature flags separate deploying code from turning on features. Code goes live with the feature off, and product teams enable it for chosen users when ready. Combined with blue-green, they let you switch infrastructure safely and reveal features gradually, with two independent safety nets.
Automated analysis and rollback
Advanced setups compare the new side’s metrics with the old side’s automatically, using statistical checks, and roll back without human help when the new side is worse. This needs high-quality metrics and careful tuning, but it can make releases both faster and safer.
Shadow traffic
Copy real requests to the idle environment, without letting its answers reach users. The team can then see how the new version behaves under real traffic before switching. It needs care with side effects, such as writes and outgoing messages.
Multi-region releases
Release region by region, with pauses and checks between regions, and blue-green within each. A problem is caught in the first region, and the rest are protected.
Separate databases per colour
Some teams give each colour its own database, with replication or migration between them, to allow incompatible changes. It is powerful and difficult: data written during the rollback window must be reconciled. Consider it only when expand and contract cannot serve you.
Gradual shifts
Move traffic in steps, with checks between them.
Feature flags
Separate deployment from exposure.
Automated analysis
Let metrics decide rollback.
Region by region
Limit the blast radius of any mistake.
Add one advanced feature at a time, and only after the basic process is reliable and rehearsed. A clever release system that nobody understands fails at the worst moment.
36Troubleshooting Blue-Green Releases
When something feels wrong, walk through these questions in order.
Which colour is live, really?
Ask the router directly, not the pipeline log or a note. Many surprises come from a wrong assumption.
Which version runs on each side?
Check the deployed artifact versions in both environments and compare them with what the pipeline reported.
Is the idle side healthy and ready?
Look at its health checks, start-up logs, and connection to the database and other services.
Did the switch really complete?
Check the router’s state, and look for a partial switch, cached DNS answers, or old connections still in use.
Is it a data compatibility problem?
Look for errors mentioning missing columns or unexpected values. Compare the migration with what the old version expects.
Are background jobs behaving?
Check whether workers run on both sides, or on neither, and whether schedules fired twice or not at all.
Does the environment differ from its twin?
Compare sizes, versions, and configuration between the two sides. Drift explains many “it worked on green” puzzles.
Can you simply roll back?
If users are affected and the cause is not obvious, roll back first and investigate afterwards. That is the point of the technique.
| Symptom | Likely cause | First check |
|---|---|---|
| Some users still see the old version | Cached DNS or a hard-coded address | How the switch works, TTL, and callers using colour addresses |
| Everyone logged out at the switch | Sessions stored in the old side’s memory | Where sessions are stored |
| Slow responses right after cutover | Cold caches, small idle environment | Warm-up and sizing |
| Errors about missing data fields | Incompatible migration | Migration history and version compatibility |
| Duplicate emails or invoices | Workers on both environments | Where workers and schedules run |
| Release reversed itself after a rerun | Toggle-style switch step | Switch step definition |
| Pipeline waits forever | Missing timeout on readiness or approval | Timeouts and the lock holder |
Keep a one-page release runbook beside the pipeline: how to find the live colour, how to roll back, whom to call, and the thresholds. On-call engineers should never have to read code to recover.
37Glossary of Terms
Every important word from this guide in one place, explained in plain language.
| Term | Plain meaning |
|---|---|
| Artifact | The built, versioned package that gets deployed. |
| Backward compatible | Working correctly with both the old and the new version at the same time. |
| Blue-green deployment | A release method that keeps two identical environments and switches traffic between them. |
| Canary release | Sending a small share of users to a new version first. |
| Connection draining | Letting requests in progress finish before an environment stops receiving traffic. |
| Cutover | The moment traffic moves from one environment to the other. |
| Downtime | A period when users cannot use the service. |
| Expand and contract | A safe way to change a database: add first, migrate, and remove old structures later. |
| Feature flag | A switch that turns a feature on or off without redeploying code. |
| Health check | A small automatic test that asks an environment whether it is ready. |
| Idempotent | Safe to repeat, because running twice has the same effect as running once. |
| Idle environment | The twin that is not serving users and receives the next release. |
| Infrastructure as code | Describing servers and settings in files so environments can be created reliably. |
| Jenkinsfile | The text file that defines a pipeline, stored beside the source code. |
| Live environment | The environment currently serving real users. |
| Lock | A named resource that only one pipeline may hold at a time. |
| Parity | Two environments being alike in every way that matters. |
| Rolling update | Replacing instances of an application a few at a time. |
| Rollback | Returning to the previous version, here by switching traffic back. |
| Rollback window | The period during which the old environment is kept ready. |
| Router | The component that decides which environment users reach. |
| Smoke test | A short set of checks of the most important functions. |
| TTL | Time to live, how long DNS answers may be cached. |
38Frequently Asked Questions
Short answers to the questions people ask most.
39Summary and Key Takeaways
Blue-green deployment turns a frightening release into a calm, reversible switch.
You began with a restaurant that built an identical twin next door and moved the customers by changing the sign, and finished with a full picture of how a Jenkins pipeline builds once, deploys to the idle environment, tests it privately, asks for approval, switches traffic, watches the result, and can roll back in seconds. Along the way you learned how to choose a traffic switch, why database changes must be compatible with both versions, how sessions, caches, and background jobs affect the cutover, and how locks, permissions, shared libraries, monitoring, and rehearsals keep the process safe. The main rules are simple: ask the router, build once, make data changes compatible, state the switch explicitly, watch agreed numbers, and keep the old side for a window.
Key Takeaways
- Keep two identical environments. One is live, the other idle, and their roles swap after every release.
- Build once, promote the same artifact. Use immutable versions, and keep configuration and secrets outside the package.
- Always ask the router which colour is live. Refuse to deploy to the live side, and state the switch as an explicit end state, never as a toggle.
- Make database changes backward compatible. Expand first, contract in a later release, so rollback stays possible.
- Test the idle side privately, and verify after the switch. Use health checks, smoke tests, and metrics compared between the two colours, with rollback thresholds agreed in advance.
- Do not waste agents or risk collisions. Put approvals outside agent blocks, use locks and single-run rules, and run releases on dedicated agents with narrow permissions.
- Keep the safety net, then tidy up. Hold the old environment for a rollback window, keep twins identical through code, and rehearse releases and rollbacks regularly.