Azure Pipelines is a cloud service that automatically builds, tests and deploys your code every time something changes.

The problem it solves

Before automation, releasing software was a manual ceremony. Someone compiled the code on their own laptop, copied files to a server, and followed a checklist that lived in a document nobody updated. Two people following the same checklist could produce different results, and a single forgotten step could take a website down. These failures were not caused by careless people; they were caused by a process that depended on human memory.

Azure Pipelines replaces that fragile ceremony with a repeatable machine. The steps are written down once, stored with the code, and executed identically on every run. When something fails, the system shows exactly which step broke and why, which turns a long investigation into a quick fix.

What it can build and where it can deploy

Azure Pipelines is not limited to one language or one cloud. It can build applications written in .NET, Java, JavaScript, Python, Go, C++, mobile languages and more. It can run on Linux, Windows and macOS machines, and it can deploy to Azure, other public clouds, on-premises servers, container platforms and app stores. This flexibility is why many companies that do not use Azure for hosting still use Azure Pipelines for automation.

Where it sits in Azure DevOps

Azure Pipelines is one of five services in Azure DevOps, alongside Boards, Repos, Test Plans and Artifacts. It can also be used completely on its own. For example, a team can keep its source code in GitHub and still use Azure Pipelines to build and release it, connecting the two through an installed application.

Three similar-sounding terms describe three levels of automation, and it helps to keep them apart.

Continuous integration

Continuous integration, or CI, means that every change pushed by a developer is automatically combined with the latest shared code, built and tested. The goal is to discover conflicts and bugs within minutes. Without CI, developers work alone for weeks, and then a painful “integration week” reveals how much their changes clash.

Continuous delivery

Continuous delivery, often shortened to CD, goes one step further. After CI succeeds, the system automatically prepares the build for release and moves it through test and staging environments. The software is always in a state where it could be released, but a human still presses the final button for production.

Continuous deployment

Continuous deployment removes even that final button. If every automated check passes, the change goes to production on its own. This requires very strong tests and monitoring, so many organizations choose continuous delivery instead and keep a human approval at the last step.

A pipeline is built from a small set of nested parts, and learning their names makes everything else easier.

flowchart TD
    P["Pipeline definition"] --> TR["Trigger starts a run"]
    TR --> S1["Stage: Build"]
    TR --> S2["Stage: Test"]
    S1 --> J1["Job A on Agent 1"]
    S1 --> J2["Job B on Agent 2"]
    J1 --> ST1["Step: restore packages"]
    J1 --> ST2["Step: compile"]
    J2 --> ST3["Step: run scripts"]
        

Tasks versus scripts

A step can be a script, where you write the commands yourself, or a task, which is a packaged action maintained by Microsoft or the community. Tasks hide complexity. A single task can sign in to a cloud account, publish test results or build a container image. Scripts give you total freedom. Good pipelines mix both, using tasks for common jobs and scripts where something custom is needed.

There are two ways to define a pipeline, and one of them is clearly the future.

The two styles

Classic pipelines are created through a graphical editor. You drag tasks into a list, click to configure them, and the definition is saved inside the service. YAML pipelines are described in a text file that lives in your repository. YAML is a simple, human-readable format that uses indentation to show structure, and it has become the standard way to describe automation across the industry.

Pipeline as code

The idea behind YAML pipelines is often called pipeline as code. Because the definition is a file, you get every benefit developers already enjoy: history showing who changed a step and why, the ability to undo a bad change, and review before changes take effect. A broken pipeline change can be reverted just like a broken code change.

The multi-stage advantage

One big reason to prefer YAML is that a single file can describe the whole journey, from build through test to production. In the older model, build and release were separate pieces with separate screens. With multi-stage YAML, the complete flow is visible in one place, and the deployment history, approvals and environments are all connected to the same run.

A trigger is the doorbell of a pipeline, and choosing the right ones keeps feedback fast and costs low.

Filters keep triggers sharp

A trigger without filters fires on everything, which is wasteful. Branch filters limit runs to chosen branches, such as the main branch and release branches. Path filters limit runs to changes in certain folders, which is very useful in a repository that holds several applications. If only the documentation folder changed, there is no reason to rebuild the entire product.

Batching and cancelling

On a busy repository, many pushes can arrive while a build is still running. Batching tells the system to wait for the current run to finish and then start one run that includes all the new changes, instead of queuing a separate run for every push. For pull requests, newer pushes usually make older runs pointless, so cancelling superseded runs saves agent time.

flowchart LR
    DEV["Developer pushes code"] --> BF{"Branch and path filters match?"}
    BF -->|Yes| RUN["Pipeline run starts"]
    BF -->|No| SKIP["No run, no cost"]
    CRON["Schedule fires"] --> RUN
    OTHER["Another pipeline finishes"] --> RUN
        

The pipeline is the plan, and the agent is the worker who carries it out.

What an agent does

An agent is a computer, physical or virtual, with agent software installed. When a job is ready, the service finds a suitable free agent, sends it the instructions, and the agent downloads the code, runs each step and streams the logs back. When the job finishes, the agent reports success or failure and becomes available for the next job.

Pools

Agents are grouped into pools. A job requests a pool, and any available agent in that pool can take it. Pools make it easy to add capacity, because adding another agent to a pool automatically shortens the queue for everyone using it. Pools also let you separate machines by purpose, such as one pool for ordinary builds and another for deployments that need access to a secure network.

Hosted and self-hosted

Managed pools and demands

A middle path exists for teams that want control without the chore of maintenance. Managed DevOps Pools let Microsoft operate the machines inside your own Azure subscription, so you choose machine sizes and network placement while the service handles creation, scaling and cleanup. Separately, demands let a job say what it needs, such as a certain tool or operating system. The service then only assigns the job to agents that match.

Parallel jobs and queues

The number of jobs that may run at the same time is called parallelism. Each organization receives a small free allowance for private projects, and more can be purchased. If five jobs are ready and only two slots exist, three wait in the queue. Many teams think their build is slow when in truth it is waiting for a free slot.

Jobs are where the real orchestration happens, and a few concepts give you great control over them.

Types of jobs

Dependencies

By default, jobs in a stage run at the same time, and stages run one after another. You can change this by stating dependencies. A deployment job can wait for a build job, and two independent test jobs can run side by side. Describing true dependencies, and no more than necessary, is the single best way to shorten total run time.

flowchart LR
    B["Job: Build"] --> T1["Job: Unit tests"]
    B --> T2["Job: Integration tests"]
    B --> L["Job: Security scan"]
    T1 --> D["Job: Deploy"]
    T2 --> D
    L --> D
        

Conditions

A condition decides whether a step, job or stage should run. By default, something runs only if everything before it succeeded. You can override this. A cleanup step can be told to run always, even after a failure. A notification step can run only on failure. A deployment can run only when the source branch is the main branch. Conditions let one pipeline definition behave sensibly in many different situations.

Matrix and parallel strategies

Sometimes you need to run the same job many times with small differences, such as testing on three operating systems and two language versions. A matrix strategy expands one job definition into many copies, one for each combination, and runs them in parallel. A separate parallel strategy splits a large workload, such as a long test suite, across several copies so that each handles a slice.

Real pipelines must adapt to different branches, environments and situations, and variables and parameters make that possible.

Variables

A variable is a named value that steps can read. Some variables are defined by you, such as the name of a server or a feature switch. Others are predefined by the system, such as the name of the branch being built, the build number or the folder where code was downloaded. Variables can be defined at the pipeline, stage or job level, and a narrower definition overrides a wider one.

Variable groups and libraries

When many pipelines need the same values, you can place them in a variable group in the shared library. Change the value once, and every pipeline using the group sees the update. Variable groups can also be linked to a secure vault, so sensitive values are fetched on demand rather than copied around.

Parameters

Parameters look similar to variables but behave differently. They are inputs chosen when a run is queued, and they can be typed, such as a text box, a true or false switch or a dropdown list. Because they are processed before the run begins, they can change the structure of the pipeline itself, for example by adding or removing a whole stage.

Secrets in variables

Any variable can be marked secret. The service then hides its value in logs and prevents it from being read back through the interface. A secret is not automatically available to every step, however; it must be passed explicitly to the steps that need it. This design reduces the chance of a stray script printing a password into a log.

Templates are the main tool for keeping dozens of pipelines consistent without maintaining dozens of copies.

What a template is

A template is a reusable piece of pipeline definition that other pipelines can include. It can hold steps, jobs, stages or even variables. The pipeline that uses it passes in parameters to customize the behavior. Instead of copying twenty lines of build steps into every project, each project refers to the template and supplies a few inputs.

Include templates and extends templates

There are two ways to use templates. With the include approach, a pipeline pulls in pieces wherever it wants and remains in full control of its own structure. With the extends approach, the pipeline starts from a complete template that defines the overall shape, and the author only fills in allowed parts. Extends is the stronger tool for governance, because the template can insist on certain stages, such as a security scan, that individual teams cannot remove.

flowchart TD
    CT["Central template repository"] --> SEC["Mandatory security scan stage"]
    CT --> BLD["Standard build stage"]
    CT --> DEP["Standard deploy stage with approvals"]
    TA["Team A pipeline extends template"] --> CT
    TB["Team B pipeline extends template"] --> CT
    TC["Team C pipeline extends template"] --> CT
        

Versioning templates

Templates stored in another repository can be referenced at a specific branch or tag. This gives you a choice. Pointing at a moving branch means everyone receives improvements immediately but may also receive surprises. Pointing at a fixed tag gives stability, and teams upgrade deliberately. Many platform teams publish tagged versions and announce changes the same way library authors do.

Pipelines often need more than the single repository that holds the pipeline file.

What resources are

A resource is anything outside the pipeline that it depends on or consumes. Declaring resources explicitly lets the system track them, trigger runs when they change, and show exactly which versions were used in each run. This also helps with security, because access to each resource can be authorized once and reviewed later.

Working across repositories

Large products are often split across several repositories, such as a front end, a back end and a shared library. A pipeline can check out all of them in a single run and build them together. This avoids the awkward practice of copying code between repositories just to make a build work.

Controlling repository access

By default, a pipeline may only access repositories in its own project, and the identity running the job is limited in scope. Administrators can tighten this further so that pipelines must be explicitly authorized before they can check out an additional repository. This prevents a newly written pipeline from quietly reading source code it was never meant to see.

Every agent starts clean, so anything you want to keep or reuse must be saved deliberately.

Why artifacts exist

Different stages and jobs often run on different machines. The agent that compiled your application is gone by the time the deployment stage starts. A pipeline artifact solves this by uploading the build output to the service, from where later jobs download it. This also supports one of the most important delivery principles: build once, then deploy that same output everywhere.

Three different things with similar names

Caching dependencies

Most builds start by downloading the same libraries again and again. A cache step saves a folder of downloaded dependencies at the end of a run and restores it at the start of the next. The cache is identified by a key, usually based on a fingerprint of the file that lists your dependencies. When that file changes, the key changes, and a fresh cache is built automatically.

Retention

Runs and their artifacts consume storage, so the service deletes old ones according to retention rules. Runs that matter, such as a release that went to production, can be retained longer. Setting thoughtful retention keeps costs down while preserving the evidence you may need later for audits or investigations.

A pipeline without tests is only an automatic way to ship mistakes faster.

The testing pyramid

Healthy teams arrange tests in a pyramid. At the wide base are many fast unit tests that check small pieces of code in isolation. In the middle are fewer integration tests that check how pieces work together. At the narrow top are a small number of slow end-to-end tests that behave like a real user. Fast tests run on every change, and slow tests run later or less often.

flowchart TD
    E["End-to-end tests: few, slow, run before release"] --> I["Integration tests: some, medium speed"]
    I --> U["Unit tests: many, very fast, run on every change"]
        

Publishing results

Pipelines can collect test results and code coverage reports and publish them to the run summary. Developers then see which tests failed, how long they took and how results trend over time. Pipelines can also detect flaky tests, which pass and fail randomly, and flag them so the team can fix them before they destroy trust in the build.

Speeding up test runs

Long test suites are a common source of slow pipelines. The usual remedies are to split the suite across several parallel jobs, to run only tests related to the changed code where possible, and to move rarely failing slow tests to a nightly schedule. Each remedy trades a little complexity for a lot of saved waiting time.

Quality gates

A quality gate turns a measurement into a rule. For example, a build can fail if any test fails, if coverage falls below an agreed level, or if a scan finds a serious vulnerability. Good gates are few and meaningful. A pipeline with too many strict rules trains people to look for ways around them.

Deployment is the riskiest part of the journey, so Azure Pipelines gives it dedicated tools.

The deployment job

A deployment job is a specialized job that targets an environment. Unlike a regular job, it automatically downloads artifacts from the build, records which version was deployed to which environment and when, and gives you the full history on the environment page. It also unlocks the built-in deployment strategies.

Lifecycle hooks

A deployment follows a lifecycle with named phases, and you can attach steps to each one. A pre-deploy phase prepares the target. The deploy phase installs the new version. A route-traffic phase switches users toward it. A post-route phase verifies behavior. Finally, on-success and on-failure phases handle cleanup or rollback. This structure gives every deployment the same predictable shape.

Built-in strategies

Blue-green deployment is not a separate named strategy in the pipeline, but you can achieve it by deploying to an idle copy of your environment, such as a staging slot on a web hosting service, testing it, and then swapping it into production. Rollback then becomes a simple swap back.

Environments are the places you deploy to, and checks are the guards standing at their doors.

Environments

An environment is a named target such as Development, Staging or Production. It shows the deployment history, links back to the runs and work items that reached it, and can be connected to real resources such as a Kubernetes namespace or a group of virtual machines. The page becomes a useful answer to the question “what is running there right now?”

Approvals and checks

Checks are conditions that must pass before a deployment may begin. The pipeline pauses and waits, and only continues when all checks succeed. Because the checks belong to the environment and not to the pipeline file, a developer cannot remove them by editing the pipeline.

flowchart LR
    A["Artifact from build"] --> DEV["Deploy to Development"]
    DEV --> TST["Deploy to Test"]
    TST --> C1{"Approval and branch check"}
    C1 -->|Pass| STG["Deploy to Staging"]
    STG --> C2{"Approval, hours and health check"}
    C2 -->|Pass| PRD["Deploy to Production"]
    C1 -->|Fail| STOP["Run stops safely"]
    C2 -->|Fail| STOP
        

Pipelines must prove who they are to other systems, and doing that safely is a core skill.

Service connections

A service connection is a stored and controlled link to an outside system such as a cloud subscription, a container registry, a Kubernetes cluster or a code host. Pipelines refer to it by name, and never hold the credentials directly. Administrators decide which pipelines may use each connection, and can require approval for its first use by a new pipeline.

Workload identity federation

The older way to connect to Azure used a secret that was stored and had to be rotated before it expired. The modern recommendation is workload identity federation. Here, the identity service trusts the pipeline’s own identity, and each run receives a short-lived token that works only for that run. There is no password to leak and nothing to renew, which removes both a security risk and a common cause of surprise failures.

Least privilege

Each service connection should be able to do only what its pipeline needs. A connection used to deploy to a test environment should not have permission to touch production resources. Splitting connections by environment limits the damage if a pipeline is misused or a mistake is made.

Because a pipeline can change production, attackers see it as a very attractive door.

Why pipelines are targets

If someone can alter a pipeline, they may be able to insert harmful code into every product it builds, steal secrets from its environment, or push changes to production without review. This kind of attack on the build process is called a supply chain attack, and it has affected well-known organizations. Defending the pipeline is therefore as important as defending the application.

Layers of defense

Pull requests from outside contributors

When a pipeline builds code from forks or from unknown contributors, extra care is needed. A malicious pull request could contain a script designed to print or send secrets. For this reason, secrets are normally withheld from builds triggered by forks, and many organizations require a maintainer to review and approve such runs before they begin.

Modern delivery often means packaging software in containers and creating the infrastructure itself through automation.

Building and publishing container images

A container image is a sealed package holding an application and everything it needs to run. Pipelines commonly build the image, scan it for known weaknesses, tag it with a version and push it to a registry. Because the image is the same everywhere, the application behaves the same in test and production, which removes the classic excuse that it worked on my machine.

Deploying to Kubernetes

Kubernetes is a system that runs and manages containers across many machines. Pipelines can deploy to it through environments linked to a cluster namespace, applying configuration files, and then watching the rollout. Teams that prefer a different style adopt GitOps, where the pipeline only updates a configuration repository and a tool inside the cluster pulls the change and applies it.

Infrastructure as code

Infrastructure as code means describing servers, networks and databases in files instead of creating them by clicking through consoles. Pipelines can run these files, first showing a preview of planned changes and then applying them after approval. This makes environments reproducible, reviewable and easy to rebuild.

A fast pipeline gets used and trusted, while a slow one gets bypassed.

Where time goes

Time in a pipeline is spent in four places: waiting in a queue, preparing the agent and downloading code, restoring dependencies, and doing the real work of building and testing. The first step in tuning is to measure which of these dominates. Guessing leads to optimizing the wrong thing.

Controlling cost

Speed and cost are closely connected. Fewer unnecessary runs, shorter jobs and cleaned-up artifacts all reduce spending. Scheduled heavy tests can run at quiet times, and self-hosted agents can be scaled down when idle. Review pipeline analytics regularly to spot pipelines that run constantly yet deliver little value.

Pipelines fail sometimes, and the skill is in finding the cause quickly.

Reading a run

Every run has a summary page showing its stages, jobs and steps with their status and duration. Selecting a step shows its full log. A good habit is to read from the first error, not the last, because later errors are often just consequences. The summary also lists artifacts, test results, linked commits and work items, which together tell you what the run contained.

Diagnostic tools

When the default logs are not enough, you can enable verbose logging for a run to see far more detail. For self-hosted agents, the agent’s own log files reveal problems with registration, permissions and tool installation. For intermittent problems, comparing a failing run with the last successful run, including the exact commits and tool versions, usually exposes what changed.

Common causes of failure

Notifications and analytics

Notifications tell the right people when something breaks, through email or chat tools connected by service hooks. Pipeline analytics show pass rates, durations and failure trends over time, helping you find the slowest or most fragile pipelines. These trends turn vague feelings about quality into evidence the team can act on.

Teams that run pipelines well repeat a handful of habits and avoid a handful of traps.

Patterns worth adopting

Traps to avoid

A short checklist, followed by three examples of the ideas working together.

Real-world scenarios

sequenceDiagram
    participant Dev as Developer
    participant Repo as Repository
    participant Svc as Pipelines service
    participant Ag as Agent
    participant Env as Environment
    Dev->>Repo: Push change or open pull request
    Repo->>Svc: Trigger matches filters
    Svc->>Ag: Assign build job
    Ag-->>Svc: Logs, test results, artifact
    Svc->>Env: Start deployment job after checks pass
    Env-->>Svc: Health confirmed
    Svc-->>Dev: Run summary and notification
        

No. Pipelines can work with Azure Repos, GitHub, Bitbucket and other Git sources. Many teams use it purely as their automation engine while keeping code elsewhere.

No. It can deploy to other clouds, on-premises servers, container platforms and app stores. Azure is simply one of many supported targets.

New pipelines should normally be YAML. It supports pipeline as code, templates and multi-stage delivery, and it is where new capabilities are focused.

A stage is a major phase of the pipeline, such as build or deploy. A job is a group of steps that runs on a single agent inside a stage. A stage can contain several jobs that run in parallel.

Typical reasons are that all parallel jobs are in use, no agent matches the job’s demands, or the run is waiting for an approval, a check or permission to use a resource. The run page names the reason.

Parameters are inputs chosen when a run is queued, and they are processed before the run begins, so they can change the pipeline’s structure. Variables are values read during execution and cannot reshape the pipeline in the same way.

Choose them when jobs must reach private networks, need special tools or hardware, or run so frequently that owning the machines is cheaper. Otherwise, Microsoft-hosted agents are simpler to operate.

Use extends templates together with the required template check on environments. The environment then refuses to deploy any pipeline that does not extend the approved template.

No. A cache speeds up repeated downloads and may disappear at any time. An artifact is a deliberate output of a run that later stages and people rely on.

Use a service connection based on workload identity federation. It gives each run a short-lived token, so there is no stored secret to rotate or leak.

Yes. Declare the extra repositories as resources and check them out during the run. Access to those repositories can be controlled so only authorized pipelines may use them.

Azure Pipelines turns the risky, manual journey from code to customer into an automatic and repeatable process. A pipeline is built from stages, jobs and steps, started by triggers, executed by agents and connected by artifacts. YAML definitions, templates and extends keep many pipelines consistent, while environments, approvals and checks add control exactly where risk lives. Secure authentication, least privilege and careful treatment of pipelines as sensitive code protect the supply chain. With caching, parallel jobs and thoughtful triggers, pipelines stay fast enough that people trust and use them, and that trust is what makes frequent, safe releases possible.