Azure Pipelines

Azure Pipelines

A complete, plain-English guide to the service that builds, tests, and delivers your software automatically: how a run really executes, how to keep it fast, secure, and reliable, and how to run it well in production.

Imagine a bakery that sells bread to a whole city. Every morning the bakers could mix the dough by hand, check each loaf with their own eyes, wrap it, and carry it to every shop on foot. It would work for a village. For a city, it would be slow, tiring, and full of mistakes. One tired baker forgets the salt, one loaf is undercooked, one shop gets the wrong delivery. So the bakery builds an assembly line: machines mix the dough in exact amounts, ovens bake on a timer, inspectors check every batch against a checklist, and vans leave for each shop in order, with a record of what went where. The bread is better, the shelves are full on time, and when something goes wrong, the line stops before bad loaves reach customers.

Software teams face exactly the same problem. Writing code is only the first step. The code must be assembled into a working program, tested, packaged, and delivered to the places where people use it, often many times a day. Azure Pipelines is Microsoft’s cloud service for building that assembly line. In this guide you will learn what it is, why it exists, every important part inside it, how one run travels from a code change to a live system, and the mistakes that trip up even experienced engineers. Each new word is explained the first time it appears, so you can begin with no background at all.

1What Azure Pipelines Is

Start with the big picture: what it does, where it lives, and the three letters that define it.

Azure Pipelines is a cloud service that automatically builds, tests, and deploys your code. It is one part of a bigger product family called Azure DevOps Services, which also includes places to store code (Repos), plan work (Boards), manage test cases (Test Plans), and share software packages (Artifacts). Pipelines can be used on its own, and it works with code stored almost anywhere: Azure Repos, GitHub, Bitbucket, or any other Git server. It also works with nearly any language and target: Java, .NET, Python, JavaScript, Go, mobile apps, containers, and more, on Windows, Linux, and macOS, deploying to Azure, other clouds, or your own servers.

Three terms sit at the heart of the idea. Continuous integration, shortened to CI, means that every time someone changes the code, the system automatically builds the program and runs tests, so problems are found within minutes instead of weeks. Continuous delivery means that every change that passes the tests is automatically prepared and made ready to release, so that releasing is a routine button press rather than a nervous event. Continuous deployment, the third idea, goes one step further and releases to users automatically without any button press. Both of the last two are abbreviated CD, which causes endless confusion, so context matters. Together the practice is called CI/CD.

Analogy

CI is the quality inspector at the end of every shift, checking that today’s work fits with everyone else’s. Continuous delivery is the packing department that has every approved box wrapped and ready on the loading dock. Continuous deployment is the van that leaves for the shops by itself the moment the box is ready. Azure Pipelines is the assembly line and the foreman that runs all three.

Any
Language, platform, and cloud
YAML
Pipelines described as code in your repository
Hosted
Microsoft runs the build machines, or you bring your own

A short history

In the early 2000s, teams built software on a single “build server” in a cupboard, often babysat by one person. Microsoft offered build tools in Team Foundation Server, a product installed on a company’s own machines. When the cloud arrived, Microsoft moved these ideas online as Visual Studio Team Services, and in 2018 renamed the family Azure DevOps, with Azure Pipelines as its build and release engine. Around the same time, Microsoft made Pipelines free for open source projects and added support for GitHub, which helped it spread far beyond the Microsoft world. Since then, the biggest shift has been from clicking together pipelines in a visual editor to writing them as text files that live next to the code. Microsoft also owns GitHub Actions, a similar service, and many organizations use one or both. We compare them later.

i
Tip

Prices, free allowances, limits, and agent images change over time. Use the ideas in this guide as a map and confirm exact numbers in Microsoft’s current documentation before you finalize a design.

2The Problem It Solves

Every tool exists because something hurt. Here is the pain that Azure Pipelines removes.

Picture a team of ten developers building a shopping website. Each person changes different parts of the code on their own computer. Without automation, a typical week looks like this. On Monday someone builds the program on their laptop and it works. On Wednesday a colleague builds it on theirs and it fails, because their laptop has a different version of some tool. On Friday the team joins everything together for the first time, and a hundred small conflicts appear at once, a painful event nicknamed integration hell. Releasing to customers happens once a month, late at night, with a long checklist of manual steps, and every release is a risk. When something breaks in production, nobody is quite sure which change caused it.

Automation fixes each of these. A pipeline builds the code the same way every time, on a clean machine, so “it works on my machine” stops being an excuse. It runs tests on every change, so problems are found while the change is still fresh in the author’s mind. It deploys using the same scripted steps to every environment, so releases are boring and repeatable. And it keeps a full record of what was built, tested, approved, and deployed, so every release can be traced back to the exact code that created it.

Pain 1

Works on my machine

Builds on different laptops give different results, so nobody trusts them.

Pain 2

Late, painful integration

Joining everyone’s work only once in a while produces big, tangled conflicts.

Pain 3

Scary manual releases

Long checklists done by hand cause mistakes and keep teams from releasing often.

Pain 4

No traceability

It is unclear what exactly is running in production and who approved it.

Everyday example: the pull request check

A developer proposes a change to the shopping cart. Within minutes, a pipeline checks out the code, builds it on a clean machine, runs two thousand tests, and reports a green tick or a red cross right on the proposal page. A teammate sees a failed test, tells the author, and the bug is fixed before it ever reaches the main code. Nobody had to remember to run anything.

3Core Concepts: The Pipelines Vocabulary

Azure Pipelines has a small set of building blocks. Learn these words and every diagram becomes easy to read.

The central object is the pipeline: the full automated process, from a code change to a result. Each time a pipeline starts, that is a run. A pipeline is organized as a set of boxes inside boxes. The biggest boxes are stages, the major phases such as Build, Test, and Deploy. Inside a stage are jobs, groups of work that run together on one machine. Inside a job are steps, the individual actions, one after another. A step is either a task, a ready-made, packaged action such as “run the .NET build” or “deploy to an Azure web app,” or a script, a few lines of your own commands.

TermPlain meaningBakery comparison
PipelineThe whole automated processThe entire assembly line
RunOne execution of a pipelineOne day’s production
StageA major phase, like Build or DeployMixing hall, baking hall, shipping dock
JobWork done together on one machineOne team in one hall
StepOne action inside a jobOne instruction on the recipe card
TaskA packaged, reusable stepA standard machine bought from a supplier
AgentThe machine that runs a jobThe worker or station doing the job
Agent poolA group of agentsA shift of workers
TriggerThe event that starts a runThe morning bell
ArtifactA file produced by a runA finished, boxed loaf
EnvironmentA named place where you deployA particular shop on the delivery route
Service connectionA saved, secure link to an outside systemA signed delivery pass for the shop’s back door
Analogy

Think of making a school lunch in a big kitchen. The recipe for one lunch is the pipeline. The big phases, prepare, cook, and pack, are stages. Each cook’s area is a job. Every instruction like “slice the bread” is a step. The cooks themselves are the agents, and the bell that wakes the kitchen in the morning is the trigger.

i
Tip

Hold on to the split between definition and execution. The pipeline definition is only text describing what should happen. Nothing runs until a trigger fires, the service queues a run, and an agent picks up the work.

4Architecture and Components

Now let us see how the pieces connect, from your code to the system that runs it.

Azure Pipelines has two big sides. The service side, hosted by Microsoft in the cloud, is the brain. It stores your pipeline definitions, watches for triggers, decides what should run and in what order, keeps the queue of waiting jobs, stores logs and results, and handles permissions and approvals. The agent side is the muscle. An agent is a small program running on a machine, which could be a virtual machine that Microsoft manages for you or a computer of your own. It asks the service, “Do you have any work for me?” When there is a job waiting, it downloads the steps, executes them, and sends back the logs and results.

Two details are worth noting. First, the agent pulls work from the service. The service does not push into your network. For self-hosted agents, this means you only need to allow outbound connections to Azure DevOps, with no inbound holes in your firewall. Second, the service never needs to see your running application. It only coordinates, while the agent does the real work next to your code and your tools.

flowchart LR
    DEV["Developer pushes code"] --> REPO["Code repository"]
    REPO -->|"trigger event"| SVC["Azure Pipelines service"]
    SVC --> Q["Job queue"]
    Q --> POOL["Agent pool"]
    POOL --> A1["Agent 1"]
    POOL --> A2["Agent 2"]
    A1 --> ART["Artifacts and packages"]
    A2 --> ENV["Deployment environments"]
    A1 -.->|"logs and results"| SVC
    A2 -.->|"logs and results"| SVC
    
Figure 1. The service queues the work and agents in a pool pick it up and report back.

Read the diagram from left to right. A developer pushes code to the repository. The repository tells the service that something changed. The service checks the pipeline definition, creates a run, and puts the jobs in a queue. Idle agents in the matching pool pick jobs up, execute their steps, and send results back. Along the way, finished files are saved as artifacts, and deployment jobs reach out to the environments where the software will live.

Brain

Pipelines service

Stores definitions, watches triggers, queues jobs, and keeps logs and history.

Muscle

Agents and pools

Programs on machines that pull jobs and run the steps.

Source

Repositories

Azure Repos, GitHub, Bitbucket, or any Git server holding the code.

Output

Artifacts and packages

Built files and software packages saved for later stages.

Targets

Environments

Named destinations such as test, staging, and production.

Access

Service connections

Secure links to Azure, container registries, and other systems.

5YAML Pipelines and Classic Pipelines

There are two ways to describe a pipeline. One of them is now strongly preferred.

The modern way is a YAML pipeline. YAML is a simple, human-readable text format that uses indentation to show structure, a little like an outline. You write the pipeline as a text file, usually named azure-pipelines.yml, and store it in the same repository as your code. Because it is just a file in the repository, it enjoys all the benefits that code has. It has a full history, so you can see who changed the pipeline and why. It can be reviewed through a pull request like any other change. It lives on branches, so a new idea can be tried on a branch without disturbing the main line. And it travels with the code, so an old version of the code is built by the old version of the pipeline. This idea is called pipeline as code.

The older way is the classic editor, a visual designer where you click to add tasks and fill in forms. Classic pipelines come in two kinds: build pipelines, which compile and test, and release pipelines, which deploy. They are easy to start, but they live inside the service, not next to your code, so they are harder to review, copy, and reproduce. Microsoft encourages new work to use YAML, which now handles both build and deployment through multi-stage pipelines, and classic release pipelines are considered the legacy approach.

FeatureYAML pipelinesClassic pipelines
Where the definition livesIn your repository as a text fileInside the service as forms
Change history and reviewFull, through pull requestsLimited history
Reuse through templatesStrongLimited, with task groups
Build and deploy in one placeYes, with multi-stage pipelinesSeparate build and release pipelines
Learning curveNeeds to learn the formatEasier to start by clicking
Recommended for new workYesMainly for existing pipelines
Analogy

A classic pipeline is a recipe someone keeps in their head, and perhaps written on a sticky note inside a drawer in the kitchen. A YAML pipeline is a recipe printed in the shared cookbook, with a page history showing who changed the amount of salt and when. Anyone can read it, suggest a change, and cook the same dish.

i
Tip

The portal also offers a visual assistant that helps you write the YAML by clicking tasks and inserting the matching text. It is a gentle way to learn the format while keeping your pipeline as code.

6Stages, Jobs, and Steps in Depth

The three nested boxes decide what runs in sequence and what runs side by side.

A stage is a logical phase and the main way to organize a pipeline. By default, stages run one after another, but you can describe dependencies so that some stages run in parallel, or one stage waits for two others. Each stage can have conditions, such as “run only on the main branch,” and can contain approvals and checks, which we meet later.

A job is a collection of steps that run on one agent. Jobs in the same stage run in parallel by default, as long as enough agents are free, and you can add dependencies when order matters. A special job type, the deployment job, targets an environment and keeps a history of what was deployed. Every job starts with a clean slate on its agent, so a later job cannot see files from an earlier one unless you pass them through artifacts. This clean-slate rule surprises many beginners, and it is also what keeps builds repeatable.

A step is the smallest unit: a task or a script. Steps inside a job run strictly in order, and by default a failing step stops the job. A step can be marked to run even after a failure, which is useful for clean-up actions such as shutting down a test database. Jobs can also use a matrix strategy, a feature that repeats the same job for several combinations of settings, for example building on three versions of a language and two operating systems, six copies in all, running in parallel.

flowchart LR
    subgraph B["Stage - Build"]
      B1["Job - compile and unit test"]
      B2["Job - code analysis"]
    end
    subgraph T["Stage - Test"]
      T1["Job - integration tests"]
    end
    subgraph D["Stage - Deploy staging"]
      D1["Deployment job"]
    end
    subgraph P["Stage - Deploy production"]
      P1["Deployment job after approval"]
    end
    B --> T --> D --> P
    
Figure 2. Stages run in order, jobs inside a stage can run side by side, and later stages wait for earlier ones.
1

Stage: Build

Compile the code, run quick tests, and publish the result as an artifact.

2

Stage: Test

Deploy to a temporary test area and run slower integration tests.

3

Stage: Staging

Release to a production-like environment for final checks.

4

Approval

A named person reviews and approves the move to production.

5

Stage: Production

Release to customers using a careful strategy and verify health.

!
Warning

Every job starts on a fresh agent state. If a later job needs the files built by an earlier job, publish them as an artifact and download them again. Relying on leftover files from a previous job is a common reason for pipelines that work on one run and fail on the next.

7Agents and Agent Pools

The agent is where your code really builds. Choosing the right kind of agent affects cost, speed, and security.

An agent pool is a group of agents. A job asks for a pool, and any free agent in that pool can take it. There are several kinds of agents.

Microsoft-hosted agents

These are virtual machines that Microsoft prepares and runs for you. They come with a wide set of common tools preinstalled: compilers, language runtimes, Docker, cloud tools, and more, available on Ubuntu Linux, Windows, and macOS images. Each job gets a brand-new machine that is thrown away when the job ends. This gives you clean, repeatable builds and removes any maintenance. The trade-offs are that you cannot install permanent software on them, they cannot reach servers sitting inside your private network by default, and they start from a cold state each time, so downloads and restores must be repeated unless you use caching.

Self-hosted agents

These are machines that you own and run, whether physical computers, virtual machines in Azure or elsewhere, or containers. You install the agent program and register it in a pool. Because you control them, they can have special tools, powerful hardware, graphics cards, or access to private networks and on-premises systems. They keep their disk between jobs, which can speed up builds by reusing downloaded files. The cost is that you must patch, secure, and scale them yourself.

Scale set agents and managed pools

For large teams, an agent pool backed by a virtual machine scale set automatically adds and removes self-hosted virtual machines as the queue grows and shrinks, and can reset each machine after use. A newer option, Managed DevOps Pools, lets Microsoft handle much of this scaling and maintenance while the machines run inside your own Azure subscription and network. This gives you private network access with less hands-on work.

OptionWho runs the machinesBest forTrade-off
Microsoft-hostedMicrosoftMost builds, clean and simpleNo private network access, cold start each job
Self-hostedYouSpecial tools, private networks, big hardwareYou patch and secure them
Scale set agentsYou, with automatic scalingLarge, busy teams needing private accessMore setup and cost control needed
Managed DevOps PoolsMicrosoft, inside your subscriptionPrivate network access with less upkeepNewer option, check current features
Analogy

Microsoft-hosted agents are like renting a fully equipped, spotless kitchen by the hour. You walk in, cook, and leave, and someone else cleans it. Self-hosted agents are like your own restaurant kitchen: you can install any special oven you want and keep supplies on the shelves, but you also clean it, repair it, and pay for it even when it is idle.

!
Warning

A self-hosted agent that is reused by many jobs can leak files or secrets from one job to the next, and anyone who can change a pipeline can run commands on that machine. Keep untrusted pipelines, such as those from public forks, off your own agents.

8Triggers: What Starts a Run

A pipeline is only a description until something starts it. Triggers are the starters.

A trigger decides when a pipeline runs. Several kinds exist, and one pipeline can use more than one at the same time.

Code

CI trigger

Starts a run when code is pushed to chosen branches or paths. This is the classic continuous integration trigger.

Review

Pull request trigger

Starts a run when someone proposes a change, so tests run before the change is merged into the main code.

Time

Scheduled trigger

Starts a run at set times, such as a full test suite every night, written with a cron-style schedule.

Chain

Pipeline completion trigger

Starts when another pipeline finishes, so a deployment pipeline can follow a build pipeline.

Resource

Container or package trigger

Starts when a new container image or package version appears.

Human

Manual run

Someone presses the Run button, often with chosen settings called runtime parameters.

Triggers can be narrowed with filters. A branch filter says which branches count, such as only the main branch and release branches. A path filter says which folders matter, so that a change to the documentation does not start a full build of the application. Skipping a run for a message such as a tag in the commit is also possible, and a batch option groups several quick pushes into one run, which saves capacity when many developers push at once.

The pull request check deserves special attention. By adding a build validation policy to a branch in your repository, you can require that the pipeline passes before a change may be merged. This turns the pipeline from a helpful tool into a gatekeeper that protects the main branch from broken code.

flowchart LR
    T1["Push to branch"] --> P["Pipeline run"]
    T2["Pull request opened"] --> P
    T3["Nightly schedule"] --> P
    T4["Another pipeline completes"] --> P
    T5["New package or image"] --> P
    T6["Manual run"] --> P
    
Figure 3. Many kinds of event can start the same pipeline.
Analogy

A CI trigger is a doorbell that rings whenever a parcel arrives. A pull request trigger is a security guard who checks the parcel before it is allowed through the gate. A scheduled trigger is the alarm clock that wakes the kitchen at the same time each morning, delivery or not.

i
Tip

Use path filters in large repositories that hold several applications. Otherwise every small change anywhere starts every pipeline, and you spend money and waiting time on builds that could not have changed.

9Internal Working: Life of One Run

What actually happens, step by step, between a developer pressing save and a green tick appearing?

Let us follow one push. A developer sends a change to the repository. The repository notifies Azure Pipelines through a connection called a webhook, which is simply a message sent automatically when something happens. The service finds the pipeline definition for that branch and reads it. If the definition uses templates from other places, the service pulls those in and combines everything in a step called expansion. It then checks the triggers and filters. If the change matches, the service creates a run and breaks it into stages and jobs, working out which can start first.

For each job that is ready, the service looks at what pool and what machine type it asks for, and places the job in a queue. It also checks whether you have enough parallel jobs, the number of jobs your organization is allowed to run at the same moment. If all slots are busy, the job waits. When a slot and an agent are free, an agent receives the job. For Microsoft-hosted agents, a fresh virtual machine is prepared. The agent downloads the steps, checks out the source code from the repository, and runs each step in order, streaming the log lines back to the service as they appear, which is why you can watch a run live in your browser.

When the job ends, the agent reports the result: succeeded, failed, canceled, or partially succeeded. The service records it, saves the logs and any published artifacts, and decides what comes next. If the next stage depends on this one, it now becomes ready. If there is an approval waiting, the service pauses and notifies the approvers. The agent’s machine is deleted if it was Microsoft-hosted. At the end of the last stage, the run is marked complete and the final status is sent back to the repository, where it appears as a tick or cross beside the change.

sequenceDiagram
    participant Dev as Developer
    participant Repo as Repository
    participant Svc as Pipelines service
    participant Ag as Agent
    participant Env as Environment
    Dev->>Repo: Push code
    Repo->>Svc: Webhook notifies change
    Svc->>Svc: Read definition and create run
    Svc->>Ag: Offer queued job to free agent
    Ag->>Repo: Check out source code
    Ag->>Ag: Run steps in order
    Ag-->>Svc: Stream logs and send result
    Svc->>Env: Start deployment job after checks
    Svc-->>Repo: Report status tick or cross
    
Figure 4. Life of one run: the service coordinates and the agent does the work.
Step 1

Event arrives

A webhook tells the service that code changed or a schedule fired.

Step 2

Definition is expanded

Templates and conditions are resolved into a concrete plan of stages and jobs.

Step 3

Jobs wait in a queue

The service waits for a free parallel slot and a matching agent.

Step 4

Agent executes

It fetches code, runs steps, and streams logs back live.

Step 5

Checks and approvals

The service pauses at gates where a human or a rule must agree.

Step 6

Result is reported

Status returns to the repository and records are kept for history.

Analogy

It works like a restaurant ordering system. The waiter, the service, takes the order and writes it on a ticket. The ticket goes to the kitchen queue. A free cook, the agent, picks it up, makes the dish step by step, and calls out progress. The waiter checks with the manager before serving anything special, then brings the dish to the table and records the sale.

!
Warning

When all parallel job slots are busy, runs wait in the queue and may look stuck. Before you suspect a bug, check the queue and your parallel job limit. A shortage of slots is among the most common reasons for slow pipelines.

10Variables, Parameters, and Secrets

The same pipeline must work for many settings. Variables and parameters make that possible without copying the pipeline.

A variable is a named value that a pipeline can use in its steps, such as the name of a server, a version number, or a build configuration. Variables can be defined inside the pipeline file, in the portal, or created while a run is going. The service also provides many built-in variables describing the run itself: the name of the branch, the number of the build, the folder where the source code was placed, and the identity of who started it. Using these avoids typing the same facts in many places.

A variable group is a named collection of variables stored in the shared library of a project, and many pipelines can use the same group. It is the right place for values that are shared by several pipelines, such as the address of a test database. A variable group can be linked to Azure Key Vault, a locked cabinet for secrets in Azure. The pipeline then reads secrets from the vault at the moment a run starts, so the secrets are never copied into the pipeline service for long-term storage.

A runtime parameter is different. It is a value that the person starting the run can choose, such as which environment to deploy to or whether to skip a slow test. Parameters have a type, such as text, number, true or false, and can offer a fixed list of allowed choices, which appear as a form in the portal when someone runs the pipeline. Because parameters are decided when the pipeline is expanded, they can even change the shape of the pipeline, for example by deciding whether a whole stage appears.

KindWhere it livesBest forCare needed
Pipeline variableInside one pipelineSettings used only by that pipelineDo not store secrets in plain text
Variable groupShared library in the projectSettings shared by several pipelinesControl who can use and edit the group
Key Vault linked groupAzure Key VaultSecrets such as passwords and tokensThe pipeline needs permission to read the vault
Runtime parameterChosen when starting a runChoices such as target environmentLimit choices to safe values
Built-in variableProvided by the serviceFacts about the run and the branchSome values can be influenced by users

How secrets are protected

A variable marked as secret is encrypted in the service and its value is masked in logs, shown as stars. Secret variables are deliberately not passed to scripts automatically. You must map a secret into a step explicitly, which makes it harder to leak by accident. Remember, though, that masking is a courtesy, not a guarantee. A script that you allow to run can still transform a secret, such as printing it with spaces between the letters, and defeat the mask. Never run untrusted code in a pipeline that has access to important secrets.

Analogy

Variables are labeled jars in the kitchen, such as “sugar” or “oven temperature,” that every recipe can refer to. A variable group is the shared pantry shelf used by every cook. A secret is the key to the locked cupboard of expensive spices, handed to a cook only for the exact dish that needs it, and taken back right afterward.

!
Warning

Never put passwords, keys, or tokens in the pipeline file or in plain variables. They end up in your repository history, where anyone with read access can find them forever. Use secret variables, Key Vault, or better still, identity-based access with no secret at all.

11Templates and Reuse

The difference between a beginner’s pipeline and a professional one is reuse. Templates let a hundred projects share one good design.

A template is a piece of pipeline definition stored in its own file, which other pipelines can include. Instead of copying twenty lines for building a web application into every project, you write them once as a template and each project refers to it, passing in parameters for the few things that differ. When you improve the template, every project that uses it improves too. Templates come in several sizes: a step template holds a few steps, a job template holds a whole job, a stage template holds a stage, and a variable template holds a set of variables. Templates can include other templates, so a stage template might use job templates that use step templates, like nesting dolls.

Templates can live in the same repository, or in a shared repository maintained by a platform team and referenced by many teams. A shared repository lets an organization publish its approved building blocks, such as “how to build and scan a container image” or “how to deploy to our Kubernetes clusters.” Versions of templates can be pinned using branches or tags, so that a change in the template does not break every pipeline at once.

Templates as guardrails

Templates are also a powerful tool for governance, which means keeping everyone within safe rules. In an extends template, the pipeline is not allowed to define its own structure. Instead it must extend a master template written by the security or platform team. The master decides which steps are mandatory, such as vulnerability scanning and compliance checks, and what teams can customize, such as the application’s build commands. You can then add a required template check to protected resources such as a production environment or a service connection, so that only pipelines extending the approved template may use them. A team cannot quietly remove the scan step, because the resource refuses pipelines that do not follow the template.

flowchart TB
    PT["Platform team approved template"] --> S1["Mandatory scan step"]
    PT --> S2["Mandatory compliance check"]
    PT --> SLOT["Team supplied build steps"]
    APP1["Team A pipeline extends template"] --> PT
    APP2["Team B pipeline extends template"] --> PT
    PT --> PROD["Production environment - requires template"]
    
Figure 5. Teams plug their own build steps into an approved template, and production accepts only pipelines that use it.
Analogy

A step template is a standard recipe card for “make white sauce” that every cook can slot into their dish. An extends template is a franchise operations manual: every branch must follow the same safety and hygiene steps, and head office will refuse to supply stock to a branch that skips them, though each branch may choose its own local specials.

Everyday example: one template, fifty services

A company runs fifty small services. A platform team writes one template that builds, tests, scans, and publishes a container image. Each service has a ten-line pipeline file that calls the template with its own name and settings. When a new security scanner is added to the template, all fifty services get it the next time they run, with no change to any of them.

i
Tip

Keep templates small and focused, name their parameters clearly, and document them with a short example. A template that nobody understands will be copied and edited, and the benefit of sharing is lost.

12Artifacts, Caching, and Packages

Because every job starts clean, you need ways to pass files forward and to avoid repeating slow work.

A pipeline artifact is a file or folder that a run saves so that later jobs and stages can use it, or so that people can download it. For example, the Build stage compiles the application and publishes the result as an artifact named “drop.” Every later stage downloads exactly this same artifact. This is the key to a core rule of good delivery: build once, deploy many. The same tested package that went to the test environment should go on to staging and production, rather than being rebuilt at each step, because a rebuild could differ in tiny ways and make your testing meaningless.

Azure Artifacts is a related service that hosts packages, which are reusable pieces of code that your programs depend on or that you share between teams. It supports common formats such as NuGet, npm, Maven, Python, and a general-purpose universal format. A feed is a private store of packages with its own permissions. Teams publish internal libraries to a feed, and pipelines pull them back when building other projects. A feed can also act as a safe proxy to public registries and keep a copy of the public packages you used, which helps protect you if a public package is removed or tampered with.

Caching saves time. Most builds begin by downloading the same dependencies, such as hundreds of libraries, that did not change since the last run. A cache task stores a folder, such as the downloaded libraries, under a key that is based on the files that describe your dependencies. When the key matches on a later run, the folder is restored in seconds instead of downloaded again. When the dependency list changes, the key changes, and a new cache is saved. This is especially valuable on Microsoft-hosted agents, which start from nothing on every job.

Pass forward

Pipeline artifacts

Publish build output once, download it in later jobs and stages.

Share

Azure Artifacts feeds

Private stores for NuGet, npm, Maven, Python, and universal packages.

Speed

Cache task

Restore unchanged dependencies from a cache keyed by your dependency files.

Containers

Container registries

Push built container images to a registry such as Azure Container Registry.

Trace

Versioning and metadata

Tag every artifact with the build number and the source commit.

Clean-up

Retention policies

Control how long old artifacts and runs are kept to limit storage.

Analogy

Build once, deploy many is like baking a batch of loaves, sealing them in numbered boxes, and sending the very same boxes to every shop. If you baked fresh bread at each shop, no one could be sure the bread the inspector approved was the bread the customers ate. Caching is like keeping flour in the kitchen’s cupboard instead of going to the mill for every loaf.

i
Tip

Record the commit and the build number inside every package, for example in its version. When something goes wrong in production you can find, within seconds, exactly which code is running.

13Testing in Pipelines

A pipeline that builds but does not test only proves the code compiles. Tests prove it works.

Tests come in layers, often drawn as a pyramid. At the wide base are unit tests, tiny and very fast checks of single functions, which run in seconds and can number in the thousands. In the middle are integration tests, which check that parts work together, such as the application talking to a real database. At the narrow top are end-to-end tests, which drive the whole system like a user would, clicking through a browser. Higher tests are slower and more fragile, so a healthy suite has many unit tests and few end-to-end tests. Other checks add safety: static analysis reads code for mistakes and style problems without running it, security scans look for known vulnerabilities in your code and in the libraries you use, and performance tests check speed under load.

Azure Pipelines understands the results of common test tools. A publish test results step reads the output files, and the run page then shows how many tests passed and failed, which tests are new failures, and how long each took. Code coverage reports show what fraction of the code the tests actually exercised. The results flow back to the pull request page, so reviewers see a failing test beside the change that caused it. Over time, the service tracks test trends and can flag flaky tests, which sometimes pass and sometimes fail on the same code without any change.

1

Static checks and unit tests

Run first, in seconds to minutes. Fail fast, so the author gets feedback while still thinking about the change.

2

Security and dependency scans

Look for known weaknesses in code and packages before anything is deployed.

3

Integration tests

Run against a real database or service in a temporary environment.

4

End-to-end and performance tests

Run on a staging environment closest to production.

5

Smoke tests after deployment

A handful of quick checks confirm that the new version is alive and serving.

Analogy

Testing is like checking a new car. Unit tests are checking each bolt and bulb on the bench. Integration tests are making sure the engine, gearbox, and wheels work together. End-to-end tests are a full drive on the road. A smoke test is turning the key in the showroom to hear the engine start before handing over the keys.

!
Warning

Flaky tests destroy trust. If people get used to re-running a red pipeline until it turns green, real failures will be ignored as well. Fix or quarantine flaky tests quickly rather than hiding them behind automatic retries.

14Deployment: Environments and Strategies

Getting the package to users safely is where pipelines earn their keep.

An environment is a named target for deployment, such as Test, Staging, or Production. It is more than a label. An environment keeps a history of every deployment to it, showing which run, which commit, and who approved it. It can hold resources, such as a Kubernetes namespace or a group of virtual machines, so that the pipeline can see what it is deploying onto. And, importantly, an environment is where you attach approvals and checks, the gates that stop a deployment until a rule is satisfied.

The job that deploys to an environment is a deployment job. It supports deployment strategies, different recipes for rolling out a change with less risk.

StrategyHow it worksRiskGood for
Run onceDeploy everywhere in one goAll users affected if it is badSimple apps and test environments
RollingUpdate a few servers at a time while others keep servingMedium, since the old and new run together brieflyGroups of virtual machines
CanarySend a small share of traffic, for example five percent, to the new version first, then grow itLow, a bad change reaches few usersLarge user bases and Kubernetes
Blue-greenRun the new version beside the old one and switch traffic in one step, with an instant way backLow, with quick rollbackWeb apps with deployment slots
flowchart LR
    NEW["New version"] --> C1["Canary - 5 percent of users"]
    C1 --> CHK["Check errors and speed"]
    CHK -->|"healthy"| C2["Grow to 50 percent"]
    C2 --> C3["Roll out to 100 percent"]
    CHK -->|"unhealthy"| RB["Roll back to old version"]
    
Figure 6. A canary rollout grows only while health checks look good, otherwise it goes back.

Deployment strategies provide lifecycle hooks, which are named points where you can place your own steps: before deployment, during it, after traffic is routed, and then separate steps for on success and on failure. For example, a failure hook can restore the previous version and send an alert. For web apps in Azure App Service, deployment slots let you deploy to a hidden twin of the live site, warm it up, test it, and then swap the twin into place, which gives zero downtime and a one-click swap back.

Analogy

A canary rollout is like a restaurant trying a new recipe on one table first and checking the reaction before serving it to the whole room. Blue-green is having a second, fully set-up dining room ready: when it is perfect, you simply open its doors and close the old room, and if guests complain, you open the old room again.

i
Tip

Design database changes so that the old version of the application still works with the new database layout during a rollout. This is often done in small steps, adding before removing, so you can always roll back safely.

15Service Connections and Identity

To deploy into Azure or push to a registry, the pipeline needs permission. Service connections are how that permission is held and controlled.

A service connection is a saved, secure link between your Azure DevOps project and an outside system. Examples include an Azure subscription, a container registry, a Kubernetes cluster, GitHub, or a package server. A pipeline never receives the underlying credential in plain sight. It refers to the connection by name, and the service passes the access to the right task at the right moment. Each connection has its own permissions: which pipelines may use it, and which users may edit it.

For connections to Azure, the older way was a service principal with a secret, which is an identity in Microsoft Entra ID with a password-like value. The weakness is that the secret expires, must be rotated, and could leak. The modern and recommended way is workload identity federation. Here the pipeline proves its identity to Entra ID using a short-lived signed token, through a standard called OpenID Connect, and Entra ID is configured to trust that exact pipeline. Nothing secret is stored, nothing expires silently, and the access the pipeline receives is short-lived. For self-hosted agents running on Azure virtual machines, a managed identity offers a similar secret-free route.

MethodSecret storedLifetime of accessRecommendation
Service principal with secretYesLong, until rotatedAvoid for new work
Service principal with certificateYes, a certificateLong, until it expiresBetter than secrets, but still stored
Workload identity federationNoShort, per runPreferred for Azure
Managed identity on agentNoShort, from the machineGood for self-hosted agents in Azure
Analogy

A service principal with a secret is a key copied to the delivery driver, which stays valid until someone remembers to change the lock. Workload identity federation is a day pass that the security desk prints for each delivery after checking the driver’s company ID. It works for that one visit and then turns to paper.

!
Warning

Give each service connection the smallest permission it needs, and one connection per environment. A single connection with owner rights over the whole subscription, usable by every pipeline, turns one compromised pipeline into a compromise of everything.

16Security and Governance

A pipeline can change your production systems. That power makes it a target, and a place where good rules pay off.

Think of security in layers. The first layer is who may change the pipeline. Because pipeline files live in the repository, branch policies and code review protect them: nobody should be able to push a change to the main pipeline file without review. The second layer is protected resources. Environments, service connections, variable groups, agent pools, and secure files can each be marked as protected and carry checks that every run must pass before it may use them.

Check

Approvals

A named person or group must say yes before a stage may use an environment, for example before production.

Check

Branch control

Only runs from allowed branches, such as main, may deploy to production.

Check

Business hours

Allow deployments only in agreed time windows.

Check

Required template

Only pipelines that extend an approved template may use the resource.

Check

Exclusive lock

Only one run at a time may deploy to an environment, so releases do not collide.

Check

Invoke a function or call a service

Run your own automated check, such as a health query or a ticket status test, before allowing the run to continue.

Threats worth knowing

Code from forks. On public projects, strangers can propose changes from their own copies of your repository. By default, secrets are not given to builds of such changes, and you should keep it that way. Injection. If a pipeline builds a command from text that a user controls, such as a branch name or a pull request title, a clever attacker may sneak extra commands into it. Treat these values as untrusted data, never paste them directly into scripts. Supply chain attacks. A tampered library or build tool can slip into your build. Pin versions, use your private feed as a proxy, scan dependencies, and prefer trusted tasks from verified publishers. Over-privileged pipelines. A pipeline that can do everything gives an attacker everything. Use the setting that limits the job authorization scope, so that a run can only touch resources in its own project.

flowchart TB
    CODE["Code change by developer"] --> PR["Pull request review and branch policy"]
    PR --> RUN["Pipeline run from approved branch"]
    RUN --> CHK["Environment checks - approval, branch, template"]
    CHK --> CONN["Service connection with least privilege"]
    CONN --> PROD["Production deployment"]
    RUN -.-> LOG["Audit log and run history"]
    
Figure 7. Layers of control between a code change and production.
Analogy

Pipeline security is like the safety rules at a bank vault. The guard checks your badge, a second person must sign the form, the vault opens only during working hours, and a log records who went in and when. No single mistake or single stolen key is enough to empty the vault.

i
Tip

Review the audit log regularly. It records changes to permissions, service connections, and policies, and can be sent to a central security system. Many incidents are first spotted as an unexpected permission change.

17Reliability and Failure Handling

Pipelines fail for many reasons, and good ones fail clearly, safely, and recoverably.

Start by understanding the result of a run. A run can succeed, fail, be canceled, or end partially succeeded, which means everything that was required passed, but some step marked as allowed to fail did not. Each step and job can have a timeout, the longest it may run before the service stops it, and it is wise to set a realistic limit. Otherwise a hung job may hold an agent and a parallel slot for hours. Tasks can be given a retry count to try again after a failure, which helps with short problems such as a brief network glitch while downloading a package.

Use conditions to control what runs after trouble. A clean-up step can be set to run always, whether the earlier steps passed or not, such as removing a temporary test environment so it does not keep costing money. A notification step can run only on failure. When a stage fails after a long time, you do not need to start over: you can often rerun failed jobs or rerun from the failed stage, saving the work that already succeeded.

Failure scenarios and recovery

A

A test fails

The run stops at that stage, the result shows which test failed, and the author fixes the code. No bad build moves forward.

B

A download fails briefly

A retry on the task cures it without anyone noticing. Persistent failures show up in the logs.

C

No agent is available

The job waits in the queue. Add more agents, more parallel jobs, or use scale-out pools.

D

A deployment goes wrong

Health checks fail, and the failure hook rolls back to the previous artifact or swaps the slot back.

E

A secret expires

Service connections fail. Use identity federation to avoid expiry, and alert before certificates expire.

F

The service has an outage

Builds cannot start. Keep a documented manual path for an urgent fix, and keep your pipeline definitions in the repository so they can be recreated.

Analogy

A resilient pipeline is like a well-run airline. If a plane has a fault, ground staff catch it before takeoff, a backup plane is available, and a clear plan says who to tell and what to do. A timeout is the rule that a flight which cannot leave in time gets canceled instead of waiting forever on the runway.

!
Warning

Do not use automatic retries to hide real problems. If a step needs three tries to pass every day, it is broken and the retries are only delaying the repair. Review the retry statistics from time to time and fix the causes.

18Monitoring, Logging, and Metrics

A healthy delivery system is one you can see into. Pipelines give you several windows.

The first window is the run page. For every run you can see the stages and jobs as a live diagram, open the log of each step, download all logs, and read the summary of tests, artifacts, and linked work items. When a step fails, the log shows the exact command and error. For deeper detail, you can run with system diagnostics turned on, which prints extra information about what the agent is doing, though it can reveal more than you want, so use it carefully.

The second window is analytics. Pipeline reports show the pass rate, the average duration, and the time spent in each stage and task over weeks and months. A pipeline that slowly grows from ten minutes to forty is a problem you can spot early. You can also see how often tests fail, which tests are flaky, and how long jobs wait in the queue for an agent. The third window is notifications and service hooks. Notifications send email when runs fail or approvals wait. Service hooks send events to chat tools such as Teams or Slack, ticket systems, or your own web address, so that the right people hear about a failed production deployment within moments.

The fourth window concerns agents and pools. You can watch how many agents are online, how busy they are, and how many jobs are waiting. For self-hosted agents, also monitor the machines themselves: disk space, memory, and operating system updates. A full disk on a build machine is a classic cause of mysterious failures.

What to watchWhy it mattersTypical action
Pass rate of the main pipelineA falling rate shows quality or stability troubleInvestigate the top failing tests and steps
Average run durationSlow pipelines reduce how often people merge and deployUse caching, parallelism, and trimming
Time waiting in queueShows a shortage of agents or parallel jobsAdd capacity or reduce wasteful runs
Flaky testsErode trust and waste timeFix or quarantine them
Deployment frequency and failuresCore measures of delivery healthTrack trends and learn from failures
Agent healthDisks, memory, and updates on self-hosted agentsAutomate cleaning and patching
Audit eventsPermission and policy changesSend to security monitoring

Measures that tell the real story

Researchers who study software delivery, in a body of work known as DORA metrics, found four numbers that distinguish strong teams: deployment frequency, how often you release; lead time for changes, how long from a code change to production; change failure rate, what share of releases cause trouble; and time to restore service, how quickly you recover. Your pipeline records the data needed for the first two and helps with the others, so it is a natural place to measure them.

i
Tip

Run history is kept according to retention policies, so old runs and artifacts are eventually deleted. If you need long-term records for audits, export the key facts or keep the artifacts you must retain for the period your rules require.

19Cost and Pricing

Azure Pipelines is priced by how much work can run at the same time, not by how many minutes you use in total.

The key unit is the parallel job. One parallel job lets one job run at a time. If you have two parallel jobs, two jobs can run side by side, and a third must wait in the queue. You buy parallel jobs separately for Microsoft-hosted agents and for self-hosted agents. Public projects usually get generous free capacity, and private projects typically get a small free grant that can be limited in minutes per month, so it is wise to check the current terms for your organization on the official pricing page.

Beyond the free grants, you pay a monthly price for each extra parallel job. Self-hosted parallel jobs are cheaper than Microsoft-hosted ones, because you also provide and pay for the machines yourself. Related services, such as Azure Artifacts storage beyond a free amount and user licenses for Azure DevOps, are priced separately.

Cost driverWhat it meansHow to control it
Parallel jobsHow many jobs may run at onceBuy only what queue times justify
Minutes on hosted agentsFree grants may limit monthly minutes for private projectsCache, trim tests, avoid pointless runs
Self-hosted machinesYou pay Azure or your data center for the virtual machinesUse scale-out pools that shut down when idle
Artifact storagePackages and artifacts beyond the free amountSet retention and clean old versions
UsersLicenses for people beyond the free tierReview who really needs access
Test environmentsCloud resources created during the pipelineAlways tear them down in a final step
Analogy

Parallel jobs are like checkout counters in a shop. You do not pay per customer. You pay for how many counters are open. With one counter, customers queue. Too many counters and most sit idle, wasting wages. The right number is the one where queues are short without empty counters.

i
Tip

Use path filters so that a change to the documentation does not trigger a full build and test cycle. Cutting unnecessary runs is the cheapest optimization there is.

20Speed and Performance

Developers wait for pipelines. A faster pipeline means faster feedback, happier teams, and more frequent releases.

There are five main ways to speed things up. First, run in parallel: split independent work into separate jobs, and use a matrix to run test groups at the same time. Second, cache dependencies and build outputs so that unchanged work is not repeated. Third, do less: use branch and path filters, shallow clones that fetch only recent history, and incremental builds that rebuild only what changed. Fourth, order smartly: put fast checks first so that obvious failures stop the run early. Fifth, choose the right agent: a bigger machine, or a self-hosted agent that keeps its tools and caches warm between runs, can cut minutes from every run.

Before changing anything, measure. The run page shows how long each step took. Often a single step, such as downloading packages or running one slow test suite, takes half of the total time. Fix the biggest slice first. Be careful about parallelism, too: more jobs mean more parallel slots consumed, and splitting a small job adds start-up overhead for each piece.

10 minA commonly cited target for feedback on a pull request
Fast firstCheap checks before expensive ones
MeasureFind the slowest step before tuning
Analogy

Speeding up a pipeline is like speeding up a kitchen. You chop vegetables at the same time as the water boils (parallel), keep the chopped onions ready in a tub (cache), skip dishes nobody ordered (filters), and taste the sauce early so you do not cook a whole meal on a bad base (fail fast).

21Design Patterns and Anti-Patterns

Teams that have run thousands of pipelines keep finding the same good ideas and the same traps.

Patterns that work

  • Pipeline as code. Keep the pipeline file in the repository with the code it builds.
  • Build once, deploy many. Promote the same artifact through all environments.
  • Shared templates. Publish approved building blocks for all teams.
  • Fail fast. Cheap checks first, expensive ones later.
  • Trunk-based flow with short branches. Merge small changes often and let the pipeline guard main.
  • Progressive delivery. Canary, slots, and checks for production.
  • Ephemeral environments. Create a test environment for the run and delete it afterward.
  • Identity-based access. Workload identity federation instead of stored secrets.

Anti-patterns to avoid

  • Click-built classic pipelines. Nobody can review, copy, or recover them easily.
  • Rebuilding per environment. What you tested is not what you ship.
  • The mega-pipeline. One enormous file that nobody dares to touch.
  • Copy and paste across repositories. Fixes never reach all copies.
  • Secrets in the repository. They live forever in history.
  • Ignoring red builds. A broken main branch becomes normal.
  • Manual steps in the middle. Hand-run scripts that only one person knows.
  • One shared, all-powerful service connection. One leak opens every door.
Analogy

Good patterns are the habits of a tidy workshop: every tool has a place, every job has a checklist, and the same sample is measured at every stage. Anti-patterns are the cluttered workshop where nobody knows which saw is sharp and the only person who understands the machine is on holiday.

22Best Practices and Common Mistakes

A short checklist of what experienced teams do, and the slips that cost beginners the most time.

BP-1Do

Define pipelines in YAML, keep them in the repository, and protect the pipeline file with the same review rules as code.

BP-2Do

Require the pipeline to pass as a branch policy before a pull request can be merged, so main is always healthy.

BP-3Do

Use separate stages and environments with approvals and checks on production, and keep service connections limited per environment.

BP-4Do

Pin versions of tools, tasks, templates, and base images so a surprise upgrade does not break your build on a Friday afternoon.

BP-5Do

Set timeouts, clean up temporary resources in an always-run step, and keep retention rules sensible.

Mistakes that cost the most

1

Leaving the default permission too wide

Pipelines that can reach everything are a risk. Restrict scope and authorize resources one by one.

2

Forgetting that each job starts clean

Files made in one job vanish unless published as artifacts. This confuses almost every beginner.

3

Trigger loops

A pipeline that pushes a commit can trigger itself forever. Skip CI on automated commits.

4

Forgetting to tear down test environments

Resources left running after a failed run quietly build up a large bill.

5

Putting everything in one long job

A failure at the end wastes everything before it, and nothing can run in parallel.

6

Treating the pipeline as the only documentation

Write a short readme: what the pipeline does, how to run it, who owns it, and how to recover when it breaks.

23Azure Pipelines Compared with Others

There are several good tools for automation. The right choice depends on where your code lives and what you need.

ToolStrengthsWatch out for
Azure PipelinesWorks with code in Azure Repos, GitHub, and others. Strong multi-stage deployment, environments, approvals, and checks. Hosted agents for Linux, Windows, and macOS. Tight Azure integration.Parallel-job pricing model. Two ways to define pipelines (YAML and classic) can confuse newcomers.
GitHub ActionsBuilt into GitHub, a large marketplace of reusable actions, simple workflow files, and strong community.Enterprise-style release gates and governance need more design. Check current feature differences.
JenkinsFree and open source, enormous plugin collection, total control over your own servers.You maintain the servers, upgrades, plugins, and security yourself.
GitLab CI/CDPart of an all-in-one GitLab platform with built-in registry and security scans.Best when your code is already in GitLab.
Cloud-native build servicesDeep integration with their own cloud and low setup work.Tied to a single cloud provider.

Many organizations use more than one. A common pattern is to keep source code on GitHub and use Azure Pipelines for build and deployment, because it connects well to GitHub while adding environments and approvals. Microsoft invests heavily in GitHub Actions too, so it is sensible to check current guidance on which service best fits new projects, and to avoid moving pipelines just for fashion. Moving has a real cost, and a working, well-designed pipeline is worth keeping.

Analogy

Choosing a CI/CD tool is like choosing a delivery service. One is built into the shop you already use, another is a flexible do-it-yourself van fleet you must maintain, and another is a large logistics company with strict scheduling and inspection points. None is best in all cases. The best one matches your cargo, your route, and your rules.

24Real-World Examples

Three stories show how the pieces fit together in daily practice.

Example 1: A small web shop with a single pipeline

A small team builds an online shop on Azure App Service. The pipeline triggers on changes to main. It restores packages from a cache, builds the code, runs unit tests, publishes the package as an artifact, deploys to a staging slot, runs a few smoke tests, and then swaps the slot into production after one manager approves. The whole process takes around fifteen minutes, and the team deploys several times a day. If the smoke tests fail, the slot swap never happens, so customers never see the problem.

Example 2: A bank-style platform with strong governance

A large financial company has hundreds of services. A platform team owns an extends template that forces vulnerability scanning, license checks, and signing of container images. Production environments require the approved template, a change ticket check, and approval from two people. Workload identity federation removes stored secrets. Self-hosted agents run inside the company network to reach private systems, and are rebuilt from a clean image weekly. Auditors can see, for any production change, the commit, the run, the approvers, and the test results.

Example 3: A mobile and cloud product with many platforms

A company ships an app for Android and iOS plus a cloud back end. A matrix runs the same test suite on several operating system versions at once. Microsoft-hosted macOS agents build the iPhone app, while Linux agents build the back end. Pull requests run quick checks in minutes, while a nightly scheduled pipeline runs the long performance and end-to-end suites. Release builds go to testers first through a distribution service and only then to the stores.

i
Tip

Notice the shared shape in every story: small changes, automatic checks, one artifact moving forward, and human approval only where risk is high. That shape matters more than any specific tool.

25When to Use Azure Pipelines, and When Not To

A good architect knows both where a tool shines and where it does not.

Advantages

  • Works with many source hosts and many languages and platforms.
  • Hosted agents for Linux, Windows, and macOS need no maintenance.
  • Strong multi-stage releases with environments, approvals, and checks.
  • Pipelines as code, templates, and extends for governance at scale.
  • Good Azure integration and identity federation.
  • Free tiers for getting started and for public projects.

Disadvantages and trade-offs

  • Concurrency-based pricing can surprise teams that need a lot of parallel work.
  • YAML has a learning curve, and complex templates become hard to read.
  • Hosted agents start clean, so caching must be designed carefully.
  • Self-hosted agents need patching, monitoring, and security hardening.
  • Classic and YAML experiences coexist, which splits documentation and habits.
  • A powerful pipeline is a powerful target and needs careful security.

A simple decision guide

Good fit

Azure-heavy delivery

You deploy mostly to Azure and want approvals, environments, and identity integration.

Good fit

Mixed platforms

You build for Windows, Linux, and macOS from one place.

Good fit

Governance needs

You need central rules and audit trails across many teams.

Think twice

Everything lives in another platform

If your code, issues, and security tooling all sit in a different all-in-one platform, its built-in pipelines may be simpler.

Think twice

Tiny hobby project

A very small project may be well served by a simpler built-in workflow.

Think twice

Ultra-custom build farm

If you need deep control over every machine, a self-managed system may suit you better, at the price of upkeep.

26Frequently Asked Questions

Q1Is Azure Pipelines the same as Azure DevOps?

No. Azure DevOps is a suite of services, and Azure Pipelines is one of them. The others cover boards for planning work, repositories for code, test plans, and artifact feeds. You can use Pipelines alone, with code stored elsewhere.

Q2What is the difference between CI and CD?

Continuous integration builds and tests every change automatically. Continuous delivery keeps every tested change ready to release at the push of a button. Continuous deployment goes one step further and releases automatically with no button. Most teams use deployment automation with a human approval for production.

Q3Should I use YAML or classic pipelines?

Use YAML for new work. It lives with your code, can be reviewed, and supports templates. Classic pipelines still work, but YAML is where most improvements have been focused.

Q4What is an agent?

An agent is the computer that actually performs the work of a job. It asks the service for work, runs the steps, and reports results. It can be hosted by Microsoft or managed by you.

Q5Why is my job waiting in the queue?

Usually because all parallel jobs are in use or no agent in the pool matches what the job needs. Check the queue reason on the run page, then add parallel jobs, add agents, or relax over-strict requirements.

Q6Why are my files gone in the next job?

Each job normally starts on a fresh, clean agent. Publish files as artifacts at the end of one job and download them in the next.

Q7How do I keep secrets safe?

Prefer no secret at all, using workload identity federation or a managed identity. Where secrets are unavoidable, keep them in Key Vault or secret variables, pass them only to the steps that need them, and never run untrusted code beside them.

Q8Can I deploy without downtime?

Yes. Use deployment slots, blue-green, rolling, or canary strategies, combined with health checks and a quick rollback path. Database changes need extra care so that old and new application versions can both work during the change.

Q9How do I make a pipeline faster?

Measure first, then use caching, parallel jobs, filters, shallow clones, and better ordering. Often one slow step is most of the problem.

Q10Can one pipeline deploy to many environments?

Yes. Define a stage per environment, each with a deployment job that targets an environment. Add approvals and checks to the environments that need them, and promote the same artifact through all stages.

Q11What happens when a run fails halfway?

Later stages that depend on it do not run unless you ask them to. You can read the logs, fix the cause, and rerun failed jobs or the failed stage instead of starting over.

Q12Is it free?

There are free grants, and public projects usually get more. Heavier use needs paid parallel jobs. Terms change, so check the official pricing page for current numbers.

27Summary and Key Takeaways

Azure Pipelines is an automated assembly line for software. A change goes in, and a tested, traceable, deployable package comes out.

We started with the bakery idea: the same careful steps, run by machines, in the same order every time. We then met the hierarchy of pipeline, stage, job, and step; the agents that do the work; the triggers that start it; and the life of a single run from webhook to final status. We looked at variables, templates, artifacts, caching, testing, and deployment strategies. We saw how service connections, identity federation, checks, and approvals keep power under control, and how monitoring, retries, and clean-up keep the system dependable. Finally we compared costs, patterns, and alternatives, so that you can decide when this tool fits.

Build once, test early, deploy the same package everywhere, and let people decide only where the risk is high.

Key Takeaways

  • Azure Pipelines automates build, test, and deployment, and works with code from many sources.
  • Use YAML pipelines in your repository, organized as stages, jobs, and steps, with shared templates for reuse and governance.
  • Agents do the work, and each job starts clean, so pass files forward as artifacts and speed up with caching.
  • Build once and deploy the same artifact through environments, using approvals, checks, and safe rollout strategies.
  • Prefer workload identity federation and least-privilege service connections over stored secrets.
  • Watch pass rate, duration, queue time, and flaky tests, and fix causes rather than hiding them with retries.
  • Pricing is based on parallel jobs, so measure queues before buying more capacity.