AWS CodePipeline: The Execution Engine Behind Reliable Continuous Delivery

AWS CodePipeline: The Execution Engine Behind Reliable Continuous Delivery

A deep, engineer-grade walkthrough of how CodePipeline actually orchestrates releases — execution state internals, artifact propagation, superseding logic, cross-account deployment, and the design decisions behind pipelines that stay predictable under real production pressure.

Most engineers meet CodePipeline as a drag-and-drop diagram: source, build, deploy, done. That’s enough for a side project. It is not enough when three commits land within ninety seconds of each other and you need to know exactly which execution actually reached production, when a cross-account deployment needs a KMS key policy that took two attempts to get right, or when a single stuck manual approval is quietly blocking every subsequent release. This tutorial skips the “what is CI/CD” basics entirely. Instead it goes straight into the machinery experienced release engineers actually argue about: how executions get superseded mid-flight, how artifacts physically move between actions, how cross-account pipelines share IAM trust and KMS access, and how to design pipelines that fail loudly and safely instead of silently shipping the wrong artifact. If you already know that a pipeline has stages and actions and that CodeBuild fits somewhere in the middle, you are exactly the reader this was written for.

1Advanced Core Concepts I — Pipeline Structure Internals

Before touching cross-account deployment or execution semantics, you need a precise model of what a pipeline actually is beneath its visual diagram.

Stages, Actions, and the Artifact Store

A pipeline is a declarative structure of ordered stages, each containing one or more actions, categorized into six provider types: Source, Build, Test, Deploy, Approval, and Invoke. Every action that produces output declares an output artifact, and every action that consumes upstream output declares a matching input artifact — but these artifacts are not passed in memory between services. CodePipeline physically writes each output artifact as a zipped object into a dedicated S3 bucket (the pipeline’s artifact store), and the next action’s execution environment downloads and unzips that object before running. Understanding this is essential: artifact size, S3 permissions, and encryption settings on that bucket are not implementation details, they are load-bearing parts of every pipeline’s correctness.

runOrder: Sequencing Within a Stage

Actions within a single stage don’t have to run one after another — the `runOrder` property lets you group actions into numbered sub-sequences, where all actions sharing a runOrder value execute in parallel, and the stage only advances once every action at the current runOrder completes successfully. This is how a single “Test” stage can run five independent test suites in parallel at runOrder 1, then a single aggregation action at runOrder 2 that depends on all five having finished.

Pipeline Type V2 and Variable Namespaces

Newer V2-type pipelines introduce variables: any action can export named output variables into a namespace, which downstream actions in later stages can reference directly, without smuggling data through artifact files. This is what makes patterns like passing a CodeBuild-generated image tag directly into a downstream CloudFormation deploy action clean, instead of relying on a shared artifact file convention.

Analogy

Think of the artifact store as a relay race baton that’s physically mailed between runners instead of handed off directly — every runner (action) must wait for the baton (S3 object) to arrive before starting, which is why S3 permissions and bucket policies on that “mailroom” matter as much as any action’s own configuration.

Structure

Stages & Actions

Ordered stages containing typed actions: Source, Build, Test, Deploy, Approval, Invoke.

Storage

Artifact Store (S3)

Every artifact is a real zipped S3 object, not an in-memory handoff.

Sequencing

runOrder

Groups actions into parallel sub-sequences within one stage.

V2 Feature

Variable Namespaces

Pass named values directly between actions without an artifact file convention.

2Advanced Core Concepts II — Advanced Action Providers & Custom Actions

The default action library covers most cases. Production pipelines that handle real complexity lean on the less obvious providers.

Lambda Actions as an Escape Hatch

When no built-in action provider fits — a custom validation step, an internal API call, a bespoke notification format — a Lambda invoke action lets a pipeline call arbitrary code, and critically, that Lambda function must explicitly call `PutJobSuccessResult` or `PutJobFailureResult` back to CodePipeline, because the action’s completion state is not inferred from the Lambda invocation’s own return value. Forgetting this callback is one of the most common causes of a pipeline stage that hangs indefinitely.

CloudFormation Actions and Change Sets

The CloudFormation deploy action supports a two-phase mode: creating a change set in one action, then executing it in a separate, often manually-approved, subsequent action. This split is what enables a genuinely reviewable infrastructure change — a human or automated policy can inspect exactly what the change set will modify before the execute-change-set action actually applies it, rather than deploying blind.

Cross-Account and Cross-Region Actions

An action can target resources in a different AWS account or Region than the pipeline itself, which requires two independent trust relationships: an IAM role in the target account that the pipeline’s role can assume, and — for cross-account artifact access specifically — a KMS key policy granting the target account’s role decrypt permission on the artifact bucket’s encryption key. Missing either half of this pair produces access-denied errors that look identical from the pipeline console but have completely different root causes.

Escape Hatch

Lambda Invoke Action

Custom logic; must explicitly signal success/failure back to CodePipeline.

Infra as Code

CloudFormation Change Set

Two-phase create/execute split enables reviewable infrastructure changes.

Multi-Account

Cross-Account Actions

Require both an assumable IAM role and KMS key policy grants in the target account.

3Internal Working

CodePipeline is, internally, a state machine tracking execution progress across stages, with EventBridge as its nervous system.

Every action provider integration is backed by a specific AWS service performing the actual work — CodeBuild runs builds, CodeDeploy performs deployments, Lambda executes custom code — while CodePipeline’s own control plane tracks only the state of each action: not-started, in-progress, succeeded, or failed, along with the execution ID it belongs to. Source stage changes are detected either via CloudWatch Events (EventBridge) rules triggered by a webhook from the source provider (GitHub, CodeCommit) or via periodic polling, depending on configuration — EventBridge-based detection is dramatically faster and is the default for modern source integrations.

flowchart TB
    Source["Source Provider
(GitHub/CodeCommit/S3)"] -->|"Change Event"| EB["EventBridge Rule"] EB -->|"Start Execution"| CP["CodePipeline
Execution State Machine"] CP --> S1["Source Stage
Action"] S1 -->|"Artifact to S3"| Store["Artifact Store (S3)"] Store --> S2["Build Stage
(CodeBuild)"] S2 -->|"Artifact to S3"| Store Store --> S3["Deploy Stage
(CodeDeploy/CFN/ECS)"] CP -->|"State Changes"| Notify["EventBridge Notifications
+ CloudTrail"]

Fig 1. CodePipeline’s execution control plane, artifact store, and event integration

Notice that the control plane never directly runs your build or deploy logic — it delegates to the underlying service and simply tracks the resulting job status. This is why a “stuck” pipeline stage is almost always a stuck underlying job (a CodeBuild project waiting on a resource, a Lambda that never called back), not a CodePipeline malfunction itself.

4Data Flow & Lifecycle

The most consequential internal behavior in CodePipeline is what happens when a new source change arrives while a previous execution is still running.

flowchart LR
    A["Execution A starts
(Commit 1)"] --> B["Reaches Deploy Stage"] C["Commit 2 pushed
while A still running"] --> D["Execution B starts"] D --> E{"Has Execution A
passed this stage?"} E -->|"No, still in earlier stage"| F["Execution A SUPERSEDED
at that stage"] E -->|"Yes, already past it"| G["Execution A continues
unaffected in later stage"] F --> H["Execution B proceeds
through remaining stages"]

Fig 2. Superseding logic when overlapping executions compete for the same stage

By default, CodePipeline uses superseding execution mode: if a newer execution reaches a stage that an older, still-running execution hasn’t yet passed, the older execution is marked superseded at that stage and does not proceed further — only the newest execution’s changes continue toward production. This prevents an older commit from “winning the race” and deploying after a newer commit, but it also means an execution can silently stop partway through, which is a common source of confusion when engineers expect every execution to always run to completion. V2 pipelines also support QUEUED and PARALLEL execution modes for teams that need every commit to fully execute, or need strict one-at-a-time ordering instead.

5Advantages, Disadvantages & Trade-offs

Advantages

  • Deep, native IAM and KMS integration makes cross-account and cross-region deployment a first-class, well-supported pattern.
  • Superseding execution mode naturally prevents old commits from deploying after newer ones without custom locking logic.
  • Broad action-provider ecosystem (CodeBuild, CodeDeploy, CloudFormation, Lambda, third-party integrations) reduces custom glue code.
  • Fully managed control plane with no orchestration infrastructure for a team to operate or patch.

Disadvantages

  • Artifact-store-based data passing (rather than direct in-memory context) adds latency and S3 permission complexity compared to some competing CI/CD tools.
  • Superseding behavior, while usually desirable, surprises teams that assume every triggered execution always completes.
  • Lambda action callback requirements are an easy, silent failure mode for engineers unfamiliar with the PutJobSuccessResult/PutJobFailureResult contract.
  • Visual pipeline complexity grows quickly once cross-account, parallel, and conditional patterns are combined, without a first-class local testing story.

6Performance & Scalability

Scaling pipelines isn’t about raw throughput — it’s about designing execution and stage structure so that concurrency doesn’t become confusion.

Parallelism via runOrder and Parallel Stages

Beyond `runOrder` within a stage, entire deployment targets can be fanned out in parallel — deploying to five Regions or environments simultaneously as separate parallel actions within one stage — dramatically reducing total release time compared to a strictly sequential deploy-one-then-the-next structure, at the cost of needing every target’s rollback story to be equally solid, since failures can now arrive from multiple directions at once.

Execution Modes at Scale

High-commit-velocity teams (many merges per hour) benefit most from the default SUPERSEDED mode, since it naturally collapses a burst of commits into a single execution reaching production, rather than queuing dozens of redundant deployments. Teams with strict auditability requirements — where every single commit must be independently verified in production — instead choose QUEUED mode deliberately, accepting the throughput cost for complete execution history.

Artifact Size and Stage Latency

Because every artifact handoff is a real S3 upload/download cycle, unnecessarily large build artifacts (full dependency trees instead of just build output, verbose logs bundled into the artifact) directly inflate stage transition latency across the entire pipeline — trimming artifact contents to only what downstream stages actually need is a frequently overlooked performance lever.

6
ACTION PROVIDER
CATEGORIES
2
CROSS-ACCOUNT TRUST
RELATIONSHIPS REQUIRED
3
EXECUTION MODES
(SUPERSEDED/QUEUED/PARALLEL)
“A pipeline doesn’t fail because it’s slow. It fails because someone assumed an execution that reached the deploy stage would always be the one that finishes there.”

7High Availability & Reliability

Reliability in CodePipeline is largely about designing for graceful, visible failure rather than silent success. Manual approval actions act as deliberate reliability gates before high-risk stages, but a common oversight is leaving an approval pending indefinitely with no timeout or notification, which quietly blocks every subsequent commit behind it under superseding logic misunderstanding. Retry configuration on individual actions lets transient failures (a flaky test runner, a momentary service throttle) self-heal without a human re-triggering the entire execution.

For deployment-stage reliability specifically, pairing CodePipeline with CodeDeploy’s built-in rollback-on-alarm capability, or with CloudFormation’s automatic rollback on stack failure, means a bad deployment can be reverted automatically based on real health signals, rather than requiring someone to notice a production incident and manually roll back through the pipeline console.

!
Reliability Trap

Teams frequently discover, mid-incident, that a manual approval action left pending for days has been silently superseding every newer execution’s ability to complete — configure notifications and expiration on approval actions specifically because they are a common invisible bottleneck.

8Security

Every pipeline runs under a dedicated service role that must be scoped precisely to the actions it performs — a common security anti-pattern is granting this role broad administrative permissions “to make the pipeline work” instead of scoping it to exactly the source, build, and deploy resources it needs. Individual actions can also assume separate, more narrowly scoped roles, which is the recommended pattern for pipelines that deploy to multiple environments with different sensitivity levels, so a bug in a dev-deploy action’s permissions can’t accidentally touch production resources.

For cross-account pipelines, the artifact bucket’s KMS key policy must explicitly grant decrypt permissions to every target account’s deployment role — this is the single most common cross-account setup failure, because IAM role trust alone is not sufficient; the encryption key itself needs its own explicit cross-account grant. Source webhooks (from GitHub, for example) should be configured with signature validation enabled, so the pipeline doesn’t trigger on spoofed webhook payloads from outside the trusted source repository.

DECISION · SEC-CP-01Common Pattern
Context

A pipeline in a central “tooling” account must deploy CloudFormation stacks into three separate application accounts.

Decision

Create a distinct cross-account deploy role per target account, each trusted by the pipeline’s service role, and grant each target account’s role explicit decrypt permission on the artifact bucket’s KMS key.

Consequence

More IAM and KMS policy objects to maintain, but a compromise or misconfiguration in one target account’s role cannot affect the others, and access can be revoked per account independently.

9Monitoring, Logging & Metrics

CodePipeline emits execution and action state-change events to EventBridge, which is the recommended integration point for real-time notifications — routing failed-action events to SNS or a chat-ops channel gives teams immediate visibility rather than requiring someone to check the console. Every execution’s full stage-by-stage history, including which execution superseded which, is retained and queryable, which is essential when investigating “why didn’t commit X reach production” incidents. All API-level actions against the pipeline (including manual approvals, retries, and configuration changes) are recorded in CloudTrail, providing the audit trail regulated environments require for release governance.

SignalWhat It SignalsAction Threshold
EventBridge action failure eventsReal-time stage/action failuresAny failure on a deploy action → immediate alert
Execution superseded rateCommit velocity vs. pipeline throughputConsistently high → consider parallel/queued mode
Pending manual approval agePotential release bottleneckIdle beyond SLA → notify approver, escalate
CloudTrail approval/retry eventsHuman intervention audit trailUnexpected approver → review access controls

10Deployment & Cloud

CodePipeline’s deployment shape is defined almost entirely by account topology and target compute platform.

In a hub-and-spoke multi-account organization, a single pipeline lives in a central tooling or CI/CD account and deploys outward into separate dev, staging, and production accounts using the cross-account role and KMS pattern from Chapter 8 — this isolates blast radius per environment while keeping release logic centralized and auditable in one place. For container workloads, the deploy stage typically targets ECS or EKS through CodeDeploy’s blue/green deployment controller, shifting traffic gradually and rolling back automatically on CloudWatch alarm breach. For infrastructure-as-code pipelines, the deploy stage is a CloudFormation or CDK-synthesized change-set execution rather than an application deployment at all, making CodePipeline as much an infrastructure release tool as an application one.

Hub-and-Spoke Multi-Account Deployment

Central tooling-account pipeline deploying into isolated per-environment accounts via cross-account roles and KMS grants.

Container Blue/Green via CodeDeploy

ECS/EKS traffic-shifting deployments with automatic alarm-based rollback, minimizing blast radius of a bad release.

Infrastructure-as-Code Release Pipeline

Change-set create/execute pattern turning CodePipeline into a governed infrastructure change-management tool.

11Design Patterns & Anti-patterns

The dominant modern pattern is pipeline-as-code: defining the pipeline itself via CloudFormation or CDK rather than hand-configuring it in the console, so that pipeline changes go through the same review process as application code, and new pipelines for new services can be generated consistently from a shared construct or template rather than recreated by hand each time.

ANTI-PATTERN · AP-CP-01Avoid
Pattern

One monolithic pipeline handling build and deployment for many unrelated microservices, added incrementally as new stages over time.

Why it fails

An unrelated service’s slow test suite or flaky deploy blocks releases for every other service sharing the pipeline, and IAM permissions for the shared service role expand to cover the union of every service’s needs, weakening least-privilege posture.

Better alternative

One pipeline per deployable service, generated consistently from a shared pipeline-as-code template, so failures and permissions stay isolated per service while pipeline structure stays standardized.

A second anti-pattern is treating a manual approval action as the only safety net, with no automated rollback mechanism behind it — a human approving a deployment says nothing about whether that deployment will actually behave correctly once live; automated, alarm-based rollback (Chapter 7) catches failure modes no manual review would ever anticipate.

12Best Practices & Common Mistakes

Do: define pipelines as code

Use CloudFormation or CDK so pipeline changes are reviewed and reproducible, not hand-edited in the console.

Don’t: grant broad admin permissions to the service role

Scope pipeline and action roles precisely to the resources they actually touch.

Do: split roles per environment

Use distinct, narrowly scoped deploy roles per target account or environment rather than one shared role.

Don’t: forget the Lambda action callback

Every custom Lambda action must explicitly call PutJobSuccessResult or PutJobFailureResult, or the stage hangs.

Do: pair manual approvals with alerting

Notify approvers immediately and set expectations for approval SLA to avoid silent release bottlenecks.

Don’t: assume every execution completes

Design monitoring around superseded executions being normal, expected behavior, not a failure state.

13Real-World & Industry Examples

Enterprise Multi-Account Release Governance

Large regulated organizations commonly centralize CodePipeline in a dedicated tooling account, using cross-account roles and CloudTrail-backed audit trails to satisfy release-governance and change-control requirements across dozens of application accounts.

Container Platform Teams — Blue/Green at Scale

Organizations running large ECS or EKS fleets pair CodePipeline with CodeDeploy’s traffic-shifting blue/green controller to minimize deployment risk across many services deploying multiple times a day.

Infrastructure Teams — GitOps for CloudFormation/CDK

Platform engineering teams use CodePipeline purely for infrastructure change management, treating every CloudFormation change set as a reviewable, auditable release rather than an ad-hoc console edit.

14Frequently Asked Questions

Q1Why did my execution stop partway through the pipeline?
It was likely superseded by a newer execution that reached the same stage first — this is default, expected behavior, not a failure, unless you’ve explicitly configured QUEUED or PARALLEL mode.
Q2Why does my cross-account deploy action fail with access denied even though the IAM role trust looks correct?
IAM role trust and KMS key access are two separate grants. The target account’s role also needs explicit decrypt permission on the artifact bucket’s KMS key, not just permission to assume the deployment role.
Q3Why is my custom Lambda action stuck “In Progress” forever?
The Lambda function must explicitly call PutJobSuccessResult or PutJobFailureResult back to CodePipeline; a normal Lambda return value alone does not signal completion to the pipeline.
Q4What’s the real difference between runOrder and separate stages?
runOrder groups actions for parallel execution within a single stage, all sharing the same stage-level transition conditions; separate stages are fully independent checkpoints with their own approval and transition logic.
Q5Should I use QUEUED execution mode instead of the default?
Only if every individual commit must independently and fully execute for audit or verification reasons — otherwise, default SUPERSEDED mode better matches high commit-velocity teams by naturally collapsing rapid commit bursts.
Q6Is a manual approval action a sufficient deployment safety mechanism on its own?
No. It verifies intent to deploy, not runtime correctness after deployment; pairing it with automated, alarm-based rollback covers failure modes a human reviewer can’t anticipate in advance.
Q7Should one pipeline handle multiple microservices?
Generally no — a shared pipeline couples unrelated services’ release velocity and widens the shared service role’s permission scope; one pipeline per deployable service, generated from a shared template, isolates both failures and permissions.
Q8How large can an artifact be before it becomes a performance problem?
There’s no universal number, but since every artifact is a real S3 upload/download between stages, bundling unnecessary files (full dependency trees, verbose logs) measurably slows every downstream stage transition — keep artifacts limited to what later stages actually need.

15Summary and Key Takeaways

Key Takeaways

  • Artifacts are real S3 objects, not in-memory handoffs — bucket permissions, encryption, and artifact size directly affect every pipeline’s correctness and speed.
  • Superseding is default, expected behavior — an execution stopping partway through because a newer commit arrived is not a failure state.
  • Cross-account deployment needs two separate grants — an assumable IAM role and explicit KMS key decrypt permission — missing either produces identical-looking access errors.
  • Custom Lambda actions must explicitly signal completion via PutJobSuccessResult/PutJobFailureResult, or the stage hangs indefinitely.
  • Manual approvals are an intent gate, not a correctness guarantee — pair them with automated, alarm-based rollback for real deployment safety.
  • Pipeline-as-code and one-pipeline-per-service keep release velocity and IAM scope isolated as an organization scales to many deployable services.
  • Choose execution mode deliberately — SUPERSEDED for high-velocity teams, QUEUED or PARALLEL when every commit must independently reach a verifiable outcome.