We start with the simplest picture: two versions of the same application running side by side, with a dial that controls how many users see the new one.

Jenkins is a tool that automates building, testing, and delivering software. Deployment is the act of putting a new version of an application onto the servers where real users reach it. A canary deployment is a way of deploying in which the new version, called the canary, runs alongside the current version, called the stable or baseline version. A traffic router sends a small share of user requests to the canary, and the rest to the stable version. Nobody is asked to choose. Users simply land on one or the other.

While the canary serves its small share, the team watches metrics: how many requests fail, how slow they are, and whether business numbers such as orders or sign-ins are healthy. If the canary looks as good as the stable version, the share is increased in steps. If it looks worse, the share is set back to zero and the stable version carries on as if nothing had happened.

flowchart LR
  A["Users send requests"] --> B["Traffic router"]
  B -->|95 percent| C["Stable version"]
  B -->|5 percent| D["Canary version"]
  C --> E["Metrics are collected"]
  D --> E
    

The four ideas in one sentence each

To see why teams bother with this extra care, imagine releasing without it.

The problem: big-bang releases

In a big-bang release, the new version replaces the old one for everybody at the same moment. If it works, all is well. If it hides a bug, every user meets that bug at once. The team learns about the problem from angry messages and alarms, then scrambles to undo the release while the damage grows by the minute.

Why testing cannot catch everything

Tests run in environments that only approximate production. They use small data, simple traffic, and a few machines. Real users behave in ways nobody predicted: unusual browsers, odd inputs, huge accounts, and bursts of traffic. Some bugs only appear under real load, or only with real data. A canary release accepts this truth. It treats production as the final test, but keeps that test small and reversible.

How the idea grew

Large internet companies were among the first to release to a small share of traffic before a full rollout, because with millions of users, even a small bug reaches a great many people. The approach became widely known as software delivery moved towards frequent releases, and as tools for splitting traffic, such as container platforms, service meshes, and cloud load balancers, made it practical for ordinary teams. Today it is often described as part of progressive delivery, which means releasing new features gradually and with measurement.

A handful of terms come up in every canary discussion. Learn them once here.

The four golden signals

A well-known list of things to watch for any service is called the four golden signals. Latency is how long requests take. Traffic is how many requests arrive. Errors is how many requests fail. Saturation is how full the service’s resources are, such as processor, memory, and connections. A canary is judged mainly on these four, plus a few numbers that matter to the business.

A few Jenkins words too

The controller is the main Jenkins server that plans the work. An agent is a worker machine that runs commands. A pipeline is the ordered list of stages in a Jenkinsfile. A stage is one named phase, such as Build or Deploy. A credential is a stored secret, such as the key that lets a pipeline reach a cluster.

A canary is one of several ways to release software. Knowing the others helps you pick the right one.

How they combine

These strategies are not enemies. Many teams use a rolling update inside each step of a canary. Others use feature flags together with canaries: the canary tests that the new build is healthy, and flags control when its new features reach users. A blue-green environment can host a canary too, by sending a small share of traffic to the new side before switching fully. Choose the combination that fits your system, and do not treat the names as rigid boxes.

An important point for beginners: Jenkins does not split traffic by itself. It is the conductor of the orchestra, not one of the instruments.

Jenkins is very good at running steps in order, on a schedule, with approvals and error handling. It is not a traffic router and not a monitoring system. A canary deployment therefore needs three kinds of tools working together.

flowchart TD
  A["Developer merges code"] --> B["Jenkins builds and tests"]
  B --> C["Publish the artifact to the repository"]
  C --> D["Jenkins deploys the canary"]
  D --> E["Traffic layer sends a small share to the canary"]
  E --> F["Monitoring system collects metrics"]
  F --> G["Jenkins reads the results"]
  G --> H{"Is the canary healthy"}
  H -->|Yes| I["Increase the share or promote"]
  H -->|No| J["Roll back"]
    

What Jenkins actually does in a canary

• Builds and tests the change, as in any pipeline.
• Publishes an immutable, versioned artifact to a repository.
• Deploys the canary to the platform, by calling its tools or updating its configuration.
• Sets the traffic share, again by calling the traffic layer’s tools or changing its configuration.
• Waits for the bake time, without holding a build machine idle.
• Reads metrics from the monitoring system, and applies the rules for success.
• Asks a person, when the process needs human judgement.
• Promotes or rolls back, and tells the team what happened.

The traffic layer is what makes a canary possible. Here are the common ways to build it, from simple to sophisticated.

Stickiness: keeping a user on one version

Without care, one user could hit the stable version on one click and the canary on the next, seeing pages that change back and forth, or losing a shopping basket. Sticky routing keeps a user on the same version, usually by remembering a value in a cookie or a header. Stickiness gives a steadier experience and cleaner data, since each user is clearly in one group. The trade-off is that a few unlucky users stay on a bad canary until it is removed, so keep the canary group small.

Choosing who sees the canary

Let us walk through the stages of a typical canary pipeline in Jenkins, from a commit to a fully released version.

One pipeline or several?

Some teams keep everything in one long pipeline that runs for an hour or more. Others split it: one pipeline builds and publishes, and a second one handles the release, started automatically or by a person. The second style helps when the release needs its own permissions, its own approvals, or a different schedule. Whichever you choose, keep the release steps clear, named, and visible on the Jenkins screen so that anyone can see how far the canary has got.

The whole value of a canary depends on one question: how do we know the canary is healthy? This chapter answers it.

Compare, do not just measure

Looking at the canary’s error rate alone can mislead. If the whole website is having a bad afternoon, the canary will show errors too, and it would be wrong to blame the new version. The reliable method is to compare the canary with the stable version at the same time. Both run under the same conditions, so any difference between them is much more likely to come from the change itself.

flowchart TD
  A["Canary is running with a small share"] --> B{"Enough traffic and time to judge"}
  B -->|No| C["Keep waiting or lengthen the bake time"]
  B -->|Yes| D{"Errors and latency close to the stable version"}
  D -->|No| E["Roll back and investigate"]
  D -->|Yes| F{"Saturation and business metrics healthy"}
  F -->|No| E
  F -->|Yes| G["Increase the share to the next step"]
    

What to compare

Enough data to judge

A canary with a tiny share of traffic may see only a few requests a minute. With so little data, one lucky or unlucky minute can look like a trend. Suppose the service handles ten thousand requests a minute and the canary gets one percent. That is one hundred requests a minute, or about a thousand in ten minutes. If the stable version’s error rate is half of one percent, we would expect about five errors in a thousand requests. If the canary shows thirty errors, the difference is large and clear. If it shows seven, it is hard to say. Low-traffic services may need a longer bake time, a larger share, or a different way of testing.

Manual or automatic analysis

• Manual. A person looks at dashboards and decides. Simple to start, and good for learning what matters, but slow and subject to fatigue.
• Scripted. A pipeline step queries the monitoring system, compares numbers with fixed rules, and returns pass or fail. Consistent and quick, but the rules must be chosen well.
• Specialised analysis tools. Some platforms provide automated canary analysis that compares many metrics statistically, and gives a score. Powerful, but needs setup and tuning.

A gate is a point where the pipeline stops and asks, “May we continue?”. Canary pipelines use gates at each traffic step.

The manual gate

Jenkins has an input step that pauses the pipeline and waits for a person to click Proceed or Abort. It can be limited to named people or groups, so that only authorised users can approve. It can carry a message and a few questions, and it should carry a time limit, so that a forgotten approval does not leave a canary running for days.

The automated gate

An automated gate is a stage that reads the metrics, applies the rules, and either continues or fails the pipeline. It has no waiting for a person. Because it is a script, it is repeatable and quick, and it works at any hour. Its weakness is that it can only check what you told it to check.

Choosing between them

A common blend

Many teams automate the early steps and keep a person for the final one. The first steps at one, five, and twenty-five percent pass on numbers alone. Before going from twenty-five to a hundred percent, a person confirms that nothing else, such as an ongoing incident or a sales event, makes this a bad moment to finish.

How big should each step be, and how long should the pipeline wait? Here is a sensible way to think about it.

Choosing step sizes

• Start smaller when risk is higher. A payment service may start at a fraction of a percent.
• Grow faster when risk is lower. A small copy change on a page may go from five to a hundred in one move.
• Watch capacity as the share grows. At fifty percent, the canary carries real load, and any weakness of scale appears.
• Keep the number of steps small. Five or six steps give safety without stretching the release over days.

Choosing bake time

The bake time should be long enough for two things: to collect enough requests to judge, and to reveal problems that take a while to appear. Some faults show at once, such as a crash on start. Others, such as a slow memory leak, take hours. A short bake catches the first kind. For the second kind, plan a longer watch period after full rollout, and keep the old version and its artifact ready for a while.

The promise of a canary is that failure is cheap. That promise holds only if rollback is fast, automatic, and well practised.

sequenceDiagram
  participant J as Jenkins Pipeline
  participant M as Monitoring
  participant T as Traffic Router
  participant C as Canary Version
  participant S as Stable Version
  J->>T: Send five percent of traffic to the canary
  T->>C: Requests for the canary
  M-->>J: Error rate is above the threshold
  J->>T: Set the canary share to zero
  T->>S: All traffic returns to the stable version
  J->>C: Scale the canary down
  J->>J: Mark the build failed and notify the team
    

Rollback is about traffic first

The fastest and safest rollback moves the traffic. Set the canary’s share to zero, and the stable version, which never stopped running, serves everybody again within seconds or minutes. Only after users are safe should the pipeline remove the canary instances and collect their logs for investigation. Never delete the evidence before you have saved it.

What triggers a rollback

• An automated gate fails, for example the error rate is above the limit.
• A person aborts the pipeline at a manual gate, or at any time.
• The pipeline itself fails during a deployment step, and cleanup logic runs.
• An alarm fires from the monitoring system, which can be wired to trigger the rollback pipeline.

Making rollback reliable in Jenkins

During a canary, two versions of your application run at the same time and share the same data. That requires them to get along.

Why compatibility matters

Both versions read and write the same database, call the same services, and often use the same caches and message queues. If the new version changes the shape of data in a way the old version cannot read, then the stable version, which is still serving most users, may break, and a rollback may not fix it, because the data has already changed.

The expand and contract approach

A widely used method for database changes is expand and contract. First, expand: add the new column or structure without removing the old one, so both versions work. Then release the new version through the canary. Once the new version is fully in place and stable, contract: remove the old structure in a later release. This turns one risky change into several safe ones, each of which can be rolled back on its own.

A canary pipeline may run for an hour or more, most of it waiting. Understanding how Jenkins waits keeps that time cheap and safe.

Two layers again

A Jenkins pipeline has a planning layer on the controller, which tracks where the pipeline has got to, and a doing layer on agents, where commands run. Waiting is planning-layer work. If the wait sits outside an agent block, Jenkins uses only a lightweight thread on the controller, and no build machine is tied up. If the wait sits inside an agent block, an executor is held the whole time, doing nothing.

The shape of a good pipeline

1. The pipeline starts with no agent at the top.
2. Build and test stages request an agent, use it, and give it back.
3. Each deployment or traffic-shift stage requests an agent only for the short time the command needs.
4. Bake times and approvals sit in their own stages, with no agent, guarded by timeouts.
5. Metric checks that need a tool on a machine take an agent briefly, and release it.

Long waits and restarts

Jenkins saves the progress of a pipeline as it goes, so that a pipeline can carry on after the controller restarts. This makes a long, mostly idle canary pipeline practical. There are limits: the durability setting of a job trades speed for how completely progress is saved, and a restart in the middle of a command on an agent may still need a rerun. Design steps so that repeating them is safe. Setting a canary share to five percent twice should be the same as setting it once.

A canary pipeline is stronger when it uses the other safety tools Jenkins offers.

Locks and milestones together

The combination described in the lock and milestone guide fits canaries very well. A milestone before the deployment cancels outdated builds waiting to release. A lock around the release, with the newest waiting build first, ensures only one canary per service at a time. A second milestone inside the lock catches any older build that slipped in before a newer one arrived. Together they guarantee that only the latest change is being rolled out, and only one rollout is under way.

A canary trades some extra time and capacity for safety. This chapter explains what it costs, and how canaries themselves can fail.

The cost of caution

A canary release takes longer than a big-bang release. With the example steps in this guide, the traffic steps alone take about fifty minutes. It also uses some extra capacity: the canary and stable versions both run for a while, and the canary needs enough resources to carry its share. Compared with blue-green, which needs a full second copy of the environment, a canary is usually cheaper, since it only adds capacity in proportion to the share.

Scaling to many services

One canary is easy to watch. Fifty services releasing canaries in the same hour can overwhelm your monitoring, your on-call staff, and your platform. Standardise the release pipeline as a shared template, so every service follows the same steps and the same rules. Limit the number of simultaneous canaries where a shared platform is affected, and avoid releasing many dependent services at once, since problems then become hard to trace.

How canaries themselves fail

Protections that help

• Judge on enough data. Require a minimum number of requests before a decision is allowed.
• Watch after promotion. Keep the old version and its artifact ready, and watch closely for a period after reaching a hundred percent.
• Include several kinds of signal. Combine technical metrics with business ones and with log checks.
• Test at scale before the last step. Load-test staging, or use a larger canary share before the final move.
• Timeout everything. A stuck release must fail safely and clean up.
• Make every step safe to repeat, so restarts and retries cannot leave a half-changed system.

A canary pipeline can change what real users see and touch production. That makes it a powerful tool, and a sensitive one.

Habits that protect you

• Separate duties. The person who writes a change should not be the only approver of its release to all users.
• Use approval rights. The manual gate should list who may approve, using groups from your security realm.
• Keep a record. Log who started, approved, promoted, and rolled back each release, with the version and time.
• Protect the traffic layer. Access to change routing weights is as sensitive as access to deploy.
• Protect the rollback path. Make sure the rollback job works even if the normal release pipeline is broken.
• Sign and scan artifacts. Check images for known vulnerabilities before the canary, not after.

When a canary result is confusing, or a release fails, here is how to find out what really happened.

What to watch during a release

• The Jenkins stage view. Shows which traffic step the release has reached, and which stage failed.
• The console log. Shows the commands run, the metrics read, and the decision made at each gate.
• Dashboards comparing canary and stable. Show the golden signals side by side, so a person can see the same picture the pipeline used.
• Platform events. Show whether instances started, passed health checks, or restarted.
• Traffic layer records. Show the real share each version received, which may differ from the intended one.
• Notifications. Tell the team when a release starts, advances, is paused for approval, fails, or completes.

flowchart TD
  A["Canary result is confusing or failed"] --> B{"Did the canary receive real traffic"}
  B -->|No| C["Check routing rules, weights, and session stickiness"]
  B -->|Yes| D{"Are the metrics being read correctly"}
  D -->|No| E["Check queries, labels, time windows, and monitoring delay"]
  D -->|Yes| F{"Is the difference bigger than normal noise"}
  F -->|No| G["Extend the bake time or compare over a longer window"]
  F -->|Yes| H["Treat it as a real regression and roll back"]
    

Common symptoms and causes

Numbers worth tracking over time

Track how many releases went through the canary, how many were rolled back, and how quickly each rollback happened. Track the time from merge to full release. Track incidents that the canary missed, since each one shows a gap in the metrics you watch. A regular review of these numbers helps you tune bake times and rules, and shows whether the canary process is earning its cost.

Patterns are habits that keep releases calm. Anti-patterns are habits that turn a safety net into a false comfort.

Patterns that help

Anti-patterns to avoid

A balanced view of canary deployment, and the habits that keep it working for you.

Best practices in one place

1. Test first. A canary supplements tests and never replaces them.
2. Build once, and canary the same immutable artifact that will be promoted.
3. Start with a tiny share, and grow in a few clear steps.
4. Compare with the stable version at the same time, using the golden signals and one or two business metrics.
5. Require enough data before any decision.
6. Make rollback simple, fast, and practised.
7. Keep changes backward compatible, using expand and contract for data.
8. Wait outside agent blocks, and put timeouts on every wait and gate.
9. Use a lock and milestones so only the newest change rolls out, one at a time.
10. Own the traffic weights in one place, and make steps idempotent.
11. Restrict who can release, and record every release and rollback.
12. Review incidents, and add the signal that would have caught each one.

The mistakes seen most often

Let us follow one imaginary team as it replaces a nervous big-bang release with a calm canary, one step at a time.

The starting point

A team runs the checkout service of an online shop, which handles about ten thousand requests a minute. Releases are big-bang: Jenkins builds, tests, and replaces the old version everywhere in one go. Last quarter, a release contained a bug that made six percent of requests fail. The team noticed after forty-five minutes, from customer complaints, and needed another ten minutes to undo the release. About six hundred requests a minute failed for forty-five minutes, which is around twenty-seven thousand failed requests, some of them paid orders.

What happens the next time a bug slips through

A later release again contains a bug that makes six percent of requests fail. This time it reaches only one percent of traffic, which is one hundred requests a minute, so about six failures a minute. After a few minutes, the comparison shows the canary’s error rate far above the stable version’s, the gate fails, and the pipeline moves all traffic back. At most about sixty requests failed in the whole ten-minute bake, and probably fewer. Nobody had to phone anybody at night.

Here is how canary releases tend to appear in real delivery pipelines. These are illustrative patterns in general terms, not claims about the private setups of any particular company.

Jenkins is one of the most widely used automation servers, and canary releases are a common companion to it in organisations that release often. What tends to be common is a habit of starting small, comparing with the stable version, keeping rollback simple, and letting the platform, not Jenkins, do the traffic shaping. The patterns below show that habit in practice.