Before any tool, feel the pain that pipelines remove.

Without a pipeline, releasing software is a ceremony. Someone builds the program on their own laptop. Someone else logs in to a server and copies files. A third person remembers a special command that only works on Tuesdays. Every step depends on human memory, and humans forget. The result is that releases are rare, scary and slow, so changes pile up, and the bigger the pile, the more likely something inside it is broken.

A pipeline turns that ceremony into a boring routine. The steps are written down as code. They run the same way each time, on a clean machine, with nobody typing secret commands. When something fails, it fails early and loudly, on a small change that is easy to understand, instead of quietly in front of customers.

Three terms people mix up

• Continuous Integration (CI). Every time a developer shares a change, the system automatically builds the software and runs tests. The goal is to learn within minutes whether the change breaks anything.
• Continuous Delivery (CD). Every change that passes the tests is automatically prepared and made ready to release. A human still presses the final button.
• Continuous Deployment. The same as delivery, except the final button is pressed by the system. Every passing change goes live on its own.

flowchart LR
  A["Developer writes code"] --> B["Change shared in Git"]
  B --> C["CI builds and tests"]
  C --> D["Package stored in a registry"]
  D --> E["CD updates the cluster"]
  E --> F["Users get the new version"]
  F --> G["Monitoring gives feedback"]
  G --> A
    

A short dictionary. Once these words feel friendly, every later chapter is easy.

Two ideas worth tattooing on your brain

The first is immutability. A container image should never be changed after it is built. If something needs to change, you build a new image with a new label. The image that passed testing is then the exact same image that goes to production, not a lookalike built later.

The second is declarative configuration. In Kubernetes you do not tell the cluster to “start three servers, then connect them”. You describe the end state you want, “three copies of this image, reachable on this address”, and Kubernetes keeps working until reality matches the description. A pipeline’s job is therefore not to run a long list of commands inside the cluster. Its job is to change the description, and let Kubernetes do the rest.

You do not need to master Kubernetes to build a pipeline, but you must know which objects the pipeline will change.

The Deployment is the star

When a pipeline “deploys” an application, it usually just changes one line in a Deployment: the name of the image. Kubernetes notices the difference between the old description and the new one, creates a new ReplicaSet for the new version, starts new pods, waits until they are healthy, and stops the old ones step by step. Your users keep getting answers during the whole process. That gentle swap is called a rolling update, and we study it closely in a later chapter.

The cluster has a front door

Anyone or anything that changes the cluster talks to the Kubernetes API server. A person uses a command-line tool, a pipeline uses a service account, and a GitOps agent uses its own identity. All of them go through the same checks: who are you, and are you allowed to do this? That is why permissions matter so much, a topic we return to in the security chapters.

Every pipeline, however fancy, is built from the same handful of building blocks.

The picture below shows a typical modern setup. A developer pushes code to a Git host. A CI system notices, builds and tests the code, and stores a new image in a registry. A second repository holds the cluster descriptions. The pipeline, or a tool living inside the cluster, updates those descriptions. The cluster then pulls the new image and rolls it out.

flowchart LR
  DEV["Developer"] -->|"push code"| GIT["App Git repository"]
  GIT -->|"webhook"| CI["CI system and runners"]
  CI -->|"test and build"| REG["Container registry"]
  CI -->|"update image tag"| CFG["Config Git repository"]
  CFG -->|"watched by"| CD["CD controller"]
  CD -->|"apply desired state"| API["Kubernetes API server"]
  API --> NODES["Worker nodes run pods"]
  NODES -->|"pull image"| REG
  NODES -->|"metrics and logs"| OBS["Monitoring and alerts"]
    

The building blocks

Why two repositories?

Many teams keep application code in one repository and deployment descriptions in another. The reason is simple. Code changes constantly and is reviewed by developers. Deployment descriptions change when releasing, scaling or fixing configuration, and may need approval from operations or security staff. Separate repositories keep these conversations apart, and make it easier to give each group the right permissions. Smaller teams often keep both together at first, and that works too.

The first half of the pipeline answers one question: is this change good enough to package?

flowchart TD
  A["Commit arrives"] --> B["Check out the code"]
  B --> C["Static checks and lint"]
  C --> D["Unit tests"]
  D --> E["Build container image"]
  E --> F["Scan image and dependencies"]
  F --> G{"All checks passed?"}
  G -->|"No"| H["Fail fast and notify the author"]
  G -->|"Yes"| I["Push image to registry"]
  I --> J["Sign image and record details"]
    

The golden rule: fail fast

Order the steps from cheapest to most expensive. A formatting error found in ten seconds is far better than one found after a twenty-minute test run. Run independent steps side by side, for instance unit tests and a security scan, to shorten the total time. Developers lose focus when feedback takes more than about ten minutes, so speed is a feature, not a luxury.

Make runs repeatable

• Pin versions. Fix the versions of build tools and libraries, so tomorrow’s run matches today’s.
• Use clean workers. A fresh container for each run prevents leftovers from an earlier run changing the result.
• Cache wisely. Reuse downloaded libraries and image layers between runs to save time, but never let a cache hide a real problem.
• Treat flaky tests as bugs. A test that sometimes fails for no reason teaches people to ignore red results.

The image is the package your whole pipeline carries. A careless package causes trouble at every later stop.

An image is built in layers, each layer being the result of one instruction in a recipe file. Layers are cached, so unchanged layers are reused and only changed ones are rebuilt. Understanding this makes builds both faster and smaller.

Habits of good images

Tagging: how to name releases

A tag is a label on an image, such as a version number. Tags can be moved to a different image, which is dangerous. The famous tag called latest is the worst offender, because it points to something different over time and tells you nothing about what is running. Better choices are listed here.

Images are your product. Treat the warehouse that holds them with respect.

A registry stores images and serves them to the nodes that need to run them. Every time a pod starts on a node that does not have the image, the node pulls it from the registry. That means the registry sits directly in the path of every deployment, every scale-up and every recovery after a failure. If it is down or slow, your pods cannot start.

Choosing a registry

• Cloud registries such as Amazon ECR, Google Artifact Registry and Azure Container Registry sit close to your cluster, integrate with cloud identity, and usually offer built-in scanning.
• Platform registries bundled with Git hosts, such as those from GitHub and GitLab, keep code and images together under one login.
• Self-hosted registries such as Harbor give you full control, replication, and policy features, at the cost of running them yourself.
• Public registries such as Docker Hub are convenient for base images, but apply limits to anonymous pulls and may change terms. Many teams mirror the images they depend on into their own registry.

Housekeeping that saves money and pain

• Retention rules. Delete untagged and very old images automatically, but keep what is currently deployed and recent rollback targets.
• Immutable tags. Many registries can refuse to overwrite an existing tag, which protects the rule that a tag means one image forever.
• Close to the cluster. Pull from a registry in the same region as your nodes. It is faster and avoids data transfer charges.
• Pull credentials. Nodes or pods need permission to pull from private registries. Prefer cloud identity over long-lived passwords stored in Secrets.
• Promote, do not rebuild. When software moves from staging to production, copy or re-label the exact same image. Never rebuild it, because a rebuild can differ in subtle ways.

Plain manifests work, but real teams need a way to reuse them across environments.

A manifest for one application in one environment is easy. The trouble starts when you have forty applications and three environments. Most of each file is identical, and only a few values differ: the number of copies, the memory size, the web address. Copying and editing files by hand leads to drift, where production slowly stops matching staging. Two popular tools solve this in different ways.

Helm: charts and values

Helm packages Kubernetes manifests into a chart, a folder of templates with blanks in them. You fill the blanks using a values file. Installing a chart produces a release, and Helm remembers every revision of that release, which makes rollback a single command. Helm is also the standard way to install third-party software, because thousands of ready-made charts exist.

Kustomize: base and overlays

Kustomize takes the opposite approach. There are no templates. You keep a plain, valid base set of manifests, and describe small overlays that patch it for each environment, such as “in production, use five replicas”. Kustomize is built into the standard Kubernetes command-line tool, so no extra software is needed.

Many teams combine the two: install third-party software with Helm, and manage their own applications with Kustomize, or use Helm for both and keep environment values in separate files. There is no single winner. Choose the simplest approach that stops you copying files by hand.

The biggest design choice in CD is who touches the cluster: the pipeline, or the cluster itself.

Push-based delivery

In the classic model, the CI system finishes its work and then runs a final step that connects to the cluster and applies the new manifests. The pipeline holds a credential for the cluster. It is simple and easy to follow, and it works with any CI tool.

Pull-based delivery

In the pull model, a small controller runs inside the cluster. It watches a Git repository holding the desired state. When the repository changes, the controller pulls the change and applies it. The pipeline never needs a credential to the cluster at all. It only needs permission to write to Git.

flowchart LR
  subgraph PUSH["Push model"]
    P1["CI pipeline"] -->|"holds cluster credentials"| P2["Kubernetes API"]
  end
  subgraph PULL["Pull model"]
    Q1["CI pipeline"] -->|"commits new image tag"| Q2["Git config repository"]
    Q3["Controller inside cluster"] -->|"watches and pulls"| Q2
    Q3 -->|"applies changes"| Q4["Kubernetes API"]
  end
    

Push is easier to start with. Pull is easier to secure and to scale. Smaller teams on a single cluster often begin with push and move to pull as they grow, while larger organisations choose pull from the start.

GitOps is the pull model with a clear philosophy attached.

The word GitOps was popularised around 2017 by the team at Weaveworks. The idea is that Git is the single source of truth for what should run. You never change the cluster by hand. You change Git, and software makes the cluster agree. The four ideas behind it are that the system is described declaratively, the description is versioned and immutable in Git, approved changes are applied automatically, and software agents continuously check that reality matches the description.

flowchart TD
  A["Desired state in Git"] --> B["Controller reads Git"]
  B --> C["Controller reads live cluster state"]
  C --> D{"Do they match?"}
  D -->|"Yes"| E["Report Synced and healthy"]
  D -->|"No"| F["Apply the difference"]
  F --> C
  E --> G["Wait for next change or interval"]
  G --> B
    

The two leading tools

Two open-source tools dominate this space, and both are graduated projects of the Cloud Native Computing Foundation. Argo CD offers a friendly web interface showing every application, its sync status and its health, and models each deployed app as an Application object. Flux is a set of small controllers, configured through objects such as a Git source and a Kustomization, with a strong command-line and automation feel. Either one gives you drift detection, automatic sync, pruning of removed resources, and a record of every change.

Drift, healing and pruning

• Drift means the live cluster differs from Git, usually because someone ran a manual command.
• Self-healing means the controller reverts such manual changes back to what Git says.
• Pruning means the controller deletes objects from the cluster when you remove them from Git.

Laying out the repository

A common layout has a folder per environment, each pointing at shared bases. Promoting a release then means a pull request that changes the image tag in the staging folder, followed by another that changes it in production. Each pull request is reviewed, tested and recorded. Larger organisations use an “app of apps” pattern, where one top-level application describes all the others, so a whole cluster can be recreated from one starting point.

The image tag question

Who updates the image tag in Git after CI builds a new image? Either the CI pipeline commits the change itself, or a helper tool watches the registry and commits for you. Both Argo CD and Flux have image automation add-ons for this. The key is that the change is always a Git commit, so it is visible and reversible.

Understanding the default deployment strategy explains most rollout surprises.

When the image in a Deployment changes, Kubernetes does not stop everything and restart. It performs a rolling update. Behind the scenes, the Deployment controller creates a new ReplicaSet for the new version and gradually moves pods from the old ReplicaSet to the new one.

flowchart TD
  A["Deployment updated with new image"] --> B["New ReplicaSet created"]
  B --> C["Start a few new pods, within maxSurge"]
  C --> D{"New pods Ready?"}
  D -->|"No, waiting"| D
  D -->|"Yes"| E["Stop some old pods, within maxUnavailable"]
  E --> F{"All pods on new version?"}
  F -->|"No"| C
  F -->|"Yes"| G["Old ReplicaSet kept at zero for rollback"]
    

The two dials

Two settings control the pace. maxSurge says how many extra pods may exist above the desired count during the update. maxUnavailable says how many pods may be missing. Both default to twenty-five percent. With ten replicas, that means up to three extra pods may start (rounded up) while up to two may be unavailable (rounded down). Setting maxUnavailable to zero guarantees full capacity throughout, at the price of needing spare room for the surge pods.

Readiness is the traffic light

A new pod only counts as ready, and only receives user traffic, once its readiness check passes. The rollout moves on only when new pods are ready. If the new version is broken and never becomes ready, the rollout simply stalls, and the old pods keep serving users. That is a wonderful safety feature, as long as you actually define a good readiness check.

Handy facts

• A Deployment keeps a history of old ReplicaSets, ten by default, which is what makes quick rollback possible.
• If a rollout makes no progress for ten minutes by default, the Deployment is marked as failed to progress. The controller does not roll back by itself, so your pipeline should notice and act.
• An optional setting makes Kubernetes wait a minimum number of seconds after a pod is ready before counting it as available, which filters out pods that crash soon after starting.

Different releases deserve different levels of caution.

Blue-green, in short

You keep two identical environments, called blue and green. Users are on blue. You deploy the new version to green, test it quietly, and then flip the Service or the ingress to point at green. If anything goes wrong, you flip back in seconds. The old environment stays around until you are confident.

Canary, in short

The name comes from canaries once used in coal mines as an early warning. You send a tiny share of users, say five percent, to the new version. A system watches error rates and response times. If they stay healthy, the share grows to twenty, fifty and finally one hundred percent. If they worsen, the system pulls traffic back automatically.

flowchart LR
  U["Users"] --> R["Traffic router"]
  R -->|"95 percent"| S["Stable version"]
  R -->|"5 percent"| C["Canary version"]
  C --> M["Metrics analysis"]
  M -->|"healthy"| P["Increase canary share"]
  M -->|"unhealthy"| B["Roll back to stable"]
    

Tools that help

Kubernetes alone offers rolling and recreate. For canary and blue-green, teams add a controller. Argo Rollouts provides a Rollout object that replaces the Deployment and supports both strategies with automated analysis. Flagger automates canary releases on top of existing Deployments. Traffic shifting itself is done by a service mesh, an ingress controller or the Gateway API, which can send a chosen percentage of requests to each version.

Kubernetes can only protect users if your application tells it the truth about its health.

Three kinds of probes let a container report its state. Each answers a different question, and mixing them up is one of the most common causes of bad releases.

A good readiness check

The best readiness check confirms that the application can really do its job: its own startup is complete and its must-have dependencies, such as a database connection, are reachable. A check that always says yes is worse than none, because it lets broken pods receive traffic. On the other hand, a check that depends on every optional service can make the whole fleet go unready when one minor system hiccups, so keep it focused.

Shutting down gracefully

When Kubernetes stops a pod, it sends a polite stop signal, waits for a grace period of thirty seconds by default, and then forcibly ends the container. During this time the pod is also being removed from the Service, but these two things happen in parallel, so a few requests may still arrive after the signal. A well-behaved application finishes its current requests, stops accepting new ones, and exits. A short pause in a preStop hook helps traffic move away first.

Protecting capacity

• Run at least two replicas of anything important, so a single pod restart is never an outage.
• Set a PodDisruptionBudget so that maintenance and node drains cannot take down too many pods together.
• Set resource requests, so pods land on nodes that really have room.
• Spread replicas across nodes and zones, so one failure does not remove them all.

Software should earn its way to production through a series of safer places.

Most teams have at least three environments. Development is a playground where things break often. Staging is a rehearsal space that looks as much like production as budget allows. Production is where real users live. A change moves forward only after it proves itself in the previous stage. This movement is called promotion.

One cluster or many?

A common compromise is one cluster for all non-production environments, split by namespace, and a separate cluster for production, sometimes one per region.

Promotion rules

Preview environments

A powerful trick is to create a short-lived environment for every pull request, usually a namespace with its own copy of the application and a unique web address. Reviewers can click and try the change before it is merged. When the pull request closes, the environment is deleted automatically. This catches problems that code review alone cannot, though it needs care to keep costs under control.

Separating deploy from release

Feature flags let you ship code to production switched off, then turn the feature on for chosen users later. This separates the technical act of deploying from the business decision of releasing. If a feature misbehaves, you switch it off in seconds without redeploying anything.

There are many tools. They fall into a few families, and you will usually combine one from each.

A short family history

Jenkins grew out of an earlier project called Hudson and became the default CI server of the 2010s, thanks to its enormous plugin ecosystem. Hosted services then made CI almost effortless for small teams. Tekton emerged from the Knative project as a Kubernetes-native way to define pipelines, and is now part of the Continuous Delivery Foundation. Netflix open-sourced Spinnaker in 2015 to manage its own multi-cloud releases, and Google contributed heavily. Argo began at Applatix, which Intuit later acquired, and Flux was created at Weaveworks. Both Argo and Flux are now graduated Cloud Native Computing Foundation projects.

How to choose

• Start where your code lives. If your repositories are on GitHub or GitLab, their built-in CI is the shortest path.
• Prefer fewer moving parts. Each tool is something you must upgrade, secure and understand during an outage.
• Separate CI from CD. Using one tool for building and another for deploying lets you replace either one without rebuilding everything.
• Check the community. Active projects with good documentation and many users are far easier to live with.

Kubernetes is not only where software runs. It is also a fine place to run the builders.

Instead of keeping a fleet of build servers that sit idle most of the day, many teams run each pipeline job as a short-lived pod. A job starts, does its work in a clean container, and disappears. The cluster provides the capacity, and its autoscaler adds machines when many jobs arrive and removes them when the queue is empty.

How the popular tools do it

• GitHub Actions can use runners managed by the Actions Runner Controller, which creates runner pods on demand and scales them with the job queue.
• GitLab CI has a Kubernetes executor that starts a pod for every job.
• Jenkins has a Kubernetes plugin that launches agents as pods from templates.
• Tekton and Argo Workflows are built for this model from the start.

The image-building puzzle

Building container images inside a container is tricky, because a normal image build needs a container engine and extra privileges. Two unsafe shortcuts are common and should be avoided. Mounting the host’s container engine socket into a job gives that job control over the entire node. Running a privileged “docker in docker” container gives it nearly the same power. A compromised job could then escape to the machine.

Safer options build images without a privileged daemon. Rootless BuildKit, Buildah and similar tools can create images inside ordinary containers. Kaniko was widely used for this purpose, but check its current maintenance status before choosing it for a new project, as its stewardship has changed. Whichever you pick, run builds on dedicated, isolated nodes when possible.

Make it fast and fair

• Resource requests and limits on runner pods stop one heavy job from starving others.
• A shared cache for layers and downloaded libraries cuts build time dramatically.
• Dedicated node groups with taints keep noisy build pods away from production workloads.
• Autoscaling lets the build fleet grow at lunchtime and shrink at night. The Cluster Autoscaler, or a similar tool, handles the nodes while a controller handles the runner pods.
• Timeouts make sure a stuck job does not hold expensive capacity for hours.

The same image runs everywhere. What changes is its settings, and some settings are secret.

Separate config from code

An application should read its settings, such as database addresses and feature switches, from the environment instead of having them baked into the image. In Kubernetes, ordinary settings live in a ConfigMap and sensitive ones in a Secret. Both are handed to pods as environment variables or as files. This is what lets one image serve development, staging and production unchanged.

A Secret is not very secret

A surprising fact for newcomers is that a Kubernetes Secret is only encoded, not encrypted, by default. Anyone who can read it can decode it instantly. Encryption at rest can be switched on in the cluster, and access can be limited with permissions, but the real danger comes when secrets are put into Git in plain form. Once a password is in Git history, it is effectively public to everyone with repository access, forever.

Safe ways to deliver secrets

Pipeline credentials

The pipeline itself needs permissions to push images, and sometimes to reach a cluster. The old way is to create a long-lived key and paste it into the CI system as a secret. If that key leaks, an attacker has lasting access. The modern way uses OpenID Connect federation. The CI system proves its identity to the cloud with a short-lived signed token, and the cloud hands back temporary credentials valid for minutes. No permanent key exists to steal. GitHub Actions, GitLab CI and others support this approach.

Attackers increasingly aim at the pipeline itself, because it has the keys to production.

A supply chain attack does not break into your application directly. It poisons something your application trusts: a library, a base image, a build tool or the pipeline. Because modern software is assembled from hundreds of outside pieces, defending the chain is as important as defending the final product. The pipeline is the natural place to put the checks.

flowchart LR
  A["Code review and branch protection"] --> B["Secret and code scanning"]
  B --> C["Dependency and image scanning"]
  C --> D["Build in clean isolated runner"]
  D --> E["Generate bill of materials"]
  E --> F["Sign the image"]
  F --> G["Admission control verifies signature"]
  G --> H["Pod runs under restricted settings"]
    

The gates, one by one

• Branch protection and review. No one pushes straight to the main branch. At least one other person approves each change.
• Scanning. Tools such as Trivy and Grype look for known vulnerabilities in libraries and operating system packages. Static analysis looks for risky patterns in your own code.
• Bill of materials. A list of everything inside the image, in formats such as SPDX or CycloneDX. When a new vulnerability is announced, you can search the lists instead of guessing.
• Signing. Tools in the Sigstore family, such as Cosign, attach a signature to the image proving which pipeline built it.
• Admission control. Policy engines such as Kyverno and OPA Gatekeeper, or built-in validating policies, sit at the cluster door and refuse anything that breaks the rules: unsigned images, images from unknown registries, containers running as root.
• Pod security standards. Kubernetes defines privileged, baseline and restricted profiles. Aim for restricted wherever possible.

Permissions for the deployer

A push-based pipeline should use a service account limited to the namespaces and object types it truly needs. Cluster-wide administrator rights for a pipeline are an invitation to disaster. In the pull model, the in-cluster controller needs broad rights, so protect the Git repository holding its instructions with strict reviews, because whoever can change that repository can change the cluster.

An industry framework called SLSA, pronounced “salsa”, describes levels of supply chain maturity, from basic build records to tamper-resistant, verifiable builds. It gives teams a roadmap instead of an all-or-nothing goal.

Different tests answer different questions, and each belongs at a different point.

The shape of a healthy test suite

Aim for many fast unit tests, fewer integration tests and a small number of end-to-end tests. This is often drawn as a pyramid. Teams that invert it, relying mostly on slow browser tests, wait ages for feedback and suffer from flaky results.

Tests that are special to Kubernetes

• Schema validation. Tools such as kubeconform check each manifest against the official Kubernetes schemas.
• Policy tests. Tools like Conftest or the policy engine’s own command-line check your files against company rules before merge.
• Dry runs. Asking the real API server to validate a change without applying it catches problems the schema alone cannot.
• Chart checks. Linting Helm charts and rendering them for every environment shows exactly what will be applied.
• Temporary clusters. Lightweight local clusters, started inside the CI job, let you test the whole deployment end to end for the cost of a minute or two.

Post-deploy verification

Tests before release prove that the software could work. Checks after release prove that it does. A short smoke test hitting the live health endpoints and one or two key user flows is the cheapest insurance you can buy. Combined with dashboards watching error rates and delays, it tells you in minutes whether to keep going or roll back.

Code can be swapped in seconds. Data cannot. This is where many pipelines get into trouble.

When a new version needs a changed database layout, such as a new column, the database must change too. During a rolling update, old and new pods run side by side, talking to the same database. So the database must work with both versions at the same moment. A change that suits only the new code will break the old pods still serving users.

Expand, migrate, contract

The safe pattern splits one risky change into several safe ones.

Where do migrations run?

• A Kubernetes Job that runs once before the new pods roll out. It is explicit and easy to observe.
• Hooks in Helm or Argo CD, such as a pre-upgrade or pre-sync step, which run a migration job at the right moment in the release.
• Init containers that run inside each new pod before the main container. Simple, but if many pods start together they may all try to migrate at once, so migrations must be safe to repeat and protected by locks.

Think about rollback early

Rolling back code is easy. Rolling back data is often impossible, because new data may already have been written. That is why backwards-compatible changes matter so much: if the old code can still work with the new database layout, then rollback of code remains a safe option. Take a backup or snapshot before risky migrations, and rehearse restoring it.

Every release plan needs a way out. Decide it before you need it.

Even excellent pipelines ship bad releases sometimes. What separates good teams from struggling ones is how quickly they notice and recover. The goal is a rollback so simple and well practised that people do it without fear.

flowchart TD
  A["Release finishes"] --> B{"Are health signals normal?"}
  B -->|"Yes"| C["Keep new version"]
  B -->|"No"| D{"Is the fix tiny and obvious?"}
  D -->|"No"| E["Roll back to last good version"]
  D -->|"Yes"| F["Roll forward with a quick fix"]
  E --> G["Confirm recovery on dashboards"]
  F --> G
  G --> H["Write a short review and improve the pipeline"]
    

Ways to roll back

Roll back or roll forward?

Rolling back is usually safest and fastest when a recent change caused the problem and the old version is known to be good. Rolling forward, meaning shipping a quick fix, makes sense when the fix is tiny and obvious, or when rollback is blocked, for example by an incompatible data change. The rule of thumb is to restore service first and investigate afterwards.

Make recovery routine

• Keep the last few good versions available. Retention rules must never delete the image you might roll back to.
• Rehearse. Run rollback drills in staging so the steps are familiar.
• Automate detection. Alerts tied to releases tell you quickly that a deploy coincided with trouble.
• Hold blameless reviews. After incidents, ask which gate could have caught it, and add that gate.

You cannot improve a pipeline you cannot measure, and you cannot trust a release you cannot observe.

Watch the release, not just the servers

After each deployment, the questions are simple. Are error rates normal? Are response times normal? Are pods restarting? Is traffic reaching the new version? To answer them quickly, mark every release on your dashboards with a vertical line or annotation. When a graph bends, you can see at a glance whether it bent right after a deploy. Many teams also tag logs and metrics with the version number, so they can compare old and new side by side.

Four numbers that describe delivery health

A research programme known as DORA, short for DevOps Research and Assessment, studied thousands of teams and found four measurements that predict how well a software organisation performs. They are now widely used.

A fourth number, the time taken to recover from a failed release, completes the set. Notice the balance: two numbers measure speed, and two measure stability. Teams that do well tend to be good at both, because small, frequent, automated releases are safer than rare, huge ones.

What to measure in the pipeline itself

Notifications that people read

Send build and deploy results to the places where the team already works, such as chat channels, but keep the noise low. A message for every successful step trains people to mute the channel. A clear message when something fails, or when a production release starts and ends, is worth its weight in gold. Include a link to the exact run and the commit author, so the right person can act at once.

Gitops-specific signals

Tools like Argo CD and Flux expose metrics and events for sync status, drift and health. Alert on applications stuck out of sync or degraded for longer than a few minutes. A controller that silently stops syncing is as dangerous as a failed pipeline, because nobody notices that new releases are no longer arriving.

Automated delivery to Kubernetes is powerful, and it is not free.

The big trade-offs

There is no universally correct row. A start-up with five engineers and one cluster should usually choose the simple side of each. A bank with strict audit rules will land on the controlled side of several. The art is to revisit the choices as the organisation grows, rather than copying what a giant company does.

Habits that make delivery smooth, and habits that quietly cause pain.

Patterns that work

Anti-patterns to avoid

A short checklist, followed by a troubleshooting guide for the failures you will definitely meet.

Best practices

• Keep the main branch green. Fix or revert broken changes immediately.
• Keep pipelines fast. Aim for feedback on a change within about ten minutes.
• Make everything repeatable. Pin versions of tools, base images and libraries.
• Use immutable, traceable tags. Every running pod should map to a commit.
• Define good probes and resource requests for every workload.
• Protect production with required reviews, narrow permissions and policy checks.
• Practise rollbacks regularly, so they are routine.
• Document the pipeline so a new teammate can follow the path from commit to cluster in an hour.

Common mistakes

• Tolerating flaky tests until nobody trusts the pipeline.
• Keeping secrets in repositories or in plain environment settings of the CI tool.
• Forgetting resource limits on build pods, so builds crowd out real workloads.
• Allowing different people to deploy the same service by different routes.
• Skipping non-production environments to “save money”, then paying more during outages.
• Never cleaning up old images, preview environments and finished jobs.

Troubleshooting common deployment failures

How well-known organisations shaped the tools and ideas we use today.

If you remember only a handful of things, remember these.