The Saga Pattern
How independent services keep a long, multi‑step business promise together — and how they gracefully undo it when one chapter of the story goes wrong. This chronicle builds the pattern from first principles all the way to production, no prior knowledge assumed.
Introduction & History
Imagine you are planning a wedding. You need to book a hall, hire a caterer, arrange flowers, and book a band. Each of these is handled by a different vendor. There is no single “master switch” that books all four at once. You call the hall, then the caterer, then the florist, then the band — one after another.
Now imagine the band cancels at the last moment. You cannot just shrug. You have to go back and undo some of your earlier bookings, or find a replacement, because a wedding with no band but a booked hall, food, and flowers is a broken plan. You might cancel the flowers, downgrade the catering, or simply find a backup band. Whatever you do, you take a series of “undo” actions to bring the overall plan back to a sensible, complete state.
This is exactly the situation that modern software finds itself in in a world of microservices. A microservice is a small, independently deployable program that does one job well — for example, an “Order Service” that only knows how to create orders, or a “Payment Service” that only knows how to charge cards. When a business operation like “place an order” needs several of these small services to cooperate, we run into the same wedding‑planning problem: no single vendor (service) can guarantee the whole plan by itself.
The Saga pattern is the software industry’s answer to this problem. It is a design pattern for managing data consistency across multiple services in a distributed system, by breaking a long business transaction into a sequence of smaller local transactions, each with a matching “undo” step called a compensating transaction.
1.1 A Short History
The word “saga” was first used in this technical sense in a 1987 database research paper by Hector Garcia‑Molina and Kenneth Salem, titled “Sagas.” At that time, the problem being solved was different from today’s microservices world — it was about very long‑running database transactions (for example, a transaction that takes hours to process a large batch of data). Holding a single database lock for that long was expensive and impractical, so the authors proposed splitting the long transaction into a chain of smaller transactions, each committed independently, with compensating transactions ready to reverse any of them if a later step failed.
Fast‑forward roughly three decades: the rise of microservice architecture at companies like Netflix, Amazon, and Uber created a strikingly similar problem, but for a new reason. It was no longer about one long transaction on one database — it was about one business process spanning many independent databases owned by many independent services. The old idea from 1987 turned out to be exactly the right shape of solution, and the Saga pattern was rediscovered and popularised for distributed systems, especially after Chris Richardson documented it thoroughly as part of his well‑known microservices patterns catalog.
Think of a saga as a “story with chapters.” Each chapter (a local transaction) must finish before the next chapter starts. If a later chapter fails, the story does not just stop awkwardly — the storyteller goes back and rewrites the earlier chapters (compensating transactions) so the story still makes sense as a whole, even though the ending is different from what was originally planned.
Today, the Saga pattern is considered a foundational technique in distributed systems design. It shows up constantly in system design interviews, in real production architectures at e‑commerce companies, ride‑sharing platforms, banking systems, and travel booking sites — anywhere a single user action must trigger coordinated work across several independently‑owned services.
The Problem & Motivation
To understand why the Saga pattern exists, we first need to understand what life was like before it, and why the old way stopped working.
2.1 The Old World: One Big Database, One Big Transaction
In a traditional application with a single database, if you need to do several related things — say, “create an order” and “reduce stock” — you wrap them in a single database transaction. A transaction is a group of operations that the database guarantees will either all succeed together, or all fail together, with nothing “half‑done” left behind. This all‑or‑nothing guarantee is described by four properties, remembered by the acronym ACID:
| Letter | Property | Meaning in plain English |
|---|---|---|
| A | Atomicity | Either every step in the transaction happens, or none of them do. No partial results. |
| C | Consistency | The database moves from one valid state to another valid state. Rules (like “stock cannot go negative”) are never broken. |
| I | Isolation | Transactions running at the same time don’t interfere with each other or see each other’s half‑finished work. |
| D | Durability | Once a transaction is confirmed (“committed”), it survives crashes, power failures, and restarts. |
This works beautifully when everything lives inside one database. But it completely falls apart the moment “create an order” and “reduce stock” live in two different databases owned by two different services, which is exactly what happens in a microservices architecture. There is no longer one database that can promise atomicity across both operations.
2.2 Why Not Just Use a Distributed Transaction?
Computer scientists did try to solve this with a protocol called Two‑Phase Commit (2PC). In 2PC, a coordinator asks every participating database, “Are you ready to commit?” (Phase 1 — the “prepare” phase). Only if every single participant says yes does the coordinator tell them all to actually commit (Phase 2 — the “commit” phase).
Two‑Phase Commit is like a group of friends agreeing to jump into a pool at the exact same moment. Everyone stands at the edge and says “ready?” One by one they confirm “yes, ready.” Only when every single friend has confirmed does someone shout “jump!” and everyone jumps together. But if even one friend is stuck tying their shoelace, everyone else has to keep standing at the edge, waiting — nobody can move until that one friend is ready or clearly says no.
That “everyone has to wait” behaviour is the fatal flaw of 2PC in modern distributed systems. It requires all participants to hold locks on their data and stay blocked until the coordinator gives the final word. If the coordinator crashes, or the network between services is slow (which is common across data centres, cloud regions, or third‑party services), every participant is stuck holding locks, unable to serve other requests. This is called blocking, and it is a scalability killer. It also does not work at all across different technologies — for example, you often cannot run 2PC between a payment gateway’s API and your own database, because the payment gateway does not expose a “prepare” step.
In cloud‑scale systems with thousands of requests per second, holding database locks while waiting on a network round‑trip to another service is unacceptable. A single slow or failing service could freeze large parts of the whole system. This is precisely the motivation for the Saga pattern: trade strict, instant consistency for availability and responsiveness, while still guaranteeing that the system eventually reaches a correct, consistent state.
2.3 The Real‑World Scenario That Motivates Sagas
Consider a typical e‑commerce checkout. A single click of “Place Order” needs to:
- Create an order record (Order Service)
- Reserve stock for the items (Inventory Service)
- Charge the customer’s card (Payment Service)
- Schedule a delivery (Shipping Service)
Each of these services almost always owns its own private database — this is a core microservices rule called database‑per‑service. That means no single database transaction can span all four steps. If the payment fails after stock has already been reserved, someone has to release that reserved stock. If shipping cannot be scheduled after payment succeeds, someone has to refund the payment. The Saga pattern is the formal, disciplined way of handling exactly this chain of “what if the next step fails” scenarios.
Core Concepts
Before going further, let’s build a clear, precise vocabulary. Every term below will be used repeatedly for the rest of this chronicle, so take your time here.
3.1 Saga
WhatA sequence of local transactions, where each local transaction updates data within a single service and then publishes a message or event to trigger the next local transaction in the sequence.
WhyBecause no single ACID transaction can span multiple independently‑owned databases, a saga replaces one big transaction with a chain of small, independent ones, tied together by messages and a rollback plan.
WhereOrder processing, travel booking, banking transfers, ride‑sharing trip lifecycle, subscription billing — anywhere a business process touches multiple services.
AnalogyA relay race. Each runner (local transaction) only runs their own leg. They cannot run the next leg themselves — they hand off a baton (an event/message) to the next runner. The race (the saga) is only “won” when every runner has finished their leg in order.
3.2 Local Transaction
WhatA normal ACID transaction that happens entirely within one service’s own database.
WhyBecause that is the largest unit of guaranteed atomicity available in a distributed system — a single service’s own database.
ExampleThe Inventory Service running UPDATE stock SET reserved = reserved + 1 WHERE product_id = 42 inside its own transaction.
3.3 Compensating Transaction
WhatA second, separate local transaction whose entire job is to semantically undo the effect of an earlier local transaction, because the saga can no longer continue forward.
WhyDistributed systems cannot use database rollback across services (there is no shared transaction), so instead of “rolling back,” a saga “rolls forward” with a deliberate undo action.
Simple analogyIf you already mailed a letter and then realise you made a mistake, you cannot “unsend” it the way you would delete an unsent draft. Instead, you send a follow‑up letter that corrects or cancels the first one. That follow‑up letter is the compensating transaction.
ExampleIf Payment fails after Inventory already reserved stock, Inventory Service runs a compensating transaction: UPDATE stock SET reserved = reserved – 1 WHERE product_id = 42.
A compensating transaction is not a database rollback. Database rollback undoes changes that were never actually committed. A compensating transaction undoes changes that were already committed and possibly already visible to other parts of the system. This is a crucial, often interview‑tested distinction.
3.4 Choreography
WhatA style of saga coordination where there is no central “boss” service. Each service listens for events, does its own local transaction, and publishes a new event. The next service reacts to that event, and so on — like a chain reaction.
WhyIt keeps services fully decoupled — no service needs to know about any other service directly, only about event types.
AnalogyA choreography dance troupe. There is no single director shouting instructions during the performance — each dancer knows their own cues and reacts to the dancer before them.
3.5 Orchestration
WhatA style of saga coordination where a central component called an orchestrator explicitly tells each service what to do next, and explicitly tells services to compensate if something fails.
WhyAs the number of steps grows, “everyone reacts to everyone else’s events” (choreography) becomes hard to follow and debug. A single orchestrator makes the entire business process visible in one place.
AnalogyAn orchestra conductor. The conductor does not play any instrument, but tells each musician exactly when to start and stop, and can immediately signal “stop, go back” if something goes wrong.
3.6 Eventual Consistency
WhatA guarantee that the system will become consistent at some point in the near future, rather than instantly. During the in‑between period, different services might briefly show different, temporarily‑mismatched views of the data.
WhyBecause sagas execute step‑by‑step over time (sometimes seconds, sometimes minutes), there is inevitably a window where, say, an order shows as “pending” while payment has not confirmed yet.
AnalogyWhen you transfer money between two of your own bank accounts at different banks, there is often a short delay before the receiving bank shows the new balance. Nobody panics about this small delay because everyone trusts it will resolve correctly very soon.
3.7 Idempotency
WhatA property of an operation meaning that performing it multiple times has the exact same effect as performing it once.
WhyIn distributed systems, messages can be delivered more than once (due to retries after network failures). If “charge the customer $50” ran twice by accident, the customer would be charged $100. Idempotency prevents this.
ExampleInstead of “add $50 to total charged,” design the operation as “ensure total charged for order #123 equals $50” — running it five times still results in exactly $50 being charged.
3.8 Semantic Lock
WhatSince a saga cannot hold a real database lock across services, it instead marks a record with a status flag (like PENDING) to warn other operations that this record is “in progress” and not yet finalised.
AnalogyPutting a “reserved” sign on a restaurant table. It is not a physical lock — anyone could still sit there — but everyone agrees to respect the sign.
3.9 Sequential (Linear) vs Parallel Sagas
Not every saga is a simple straight line. Sometimes steps can happen at the same time because they don’t depend on each other. For example, “send confirmation email” and “notify the warehouse” might both be allowed to run in parallel once payment succeeds, since neither depends on the other’s result. A saga can therefore be modelled as a directed graph of steps, not just a simple list — though the simple linear case is the easiest starting point and covers most real systems.
Architecture & Components
Let’s look at the two architectural styles in detail, since almost every real implementation is one of these two (or, sometimes, a mix).
4.1 Style 1 — Choreography‑Based Saga
In choreography, services communicate purely through events published on a message broker (like Kafka, RabbitMQ, or AWS SNS/SQS). Each service:
- Subscribes to the events it cares about
- Performs its own local transaction when a relevant event arrives
- Publishes a new event describing what it just did (success or failure)
Fig 4.1 — Choreography saga: every service reacts only to events, no central controller exists.
Notice something important: Order Service never talks directly to Payment Service. It only publishes an event and trusts that whoever is interested will pick it up. This is what makes choreography so loosely coupled — you could add a brand‑new service (say, a Fraud Check Service) that listens to OrderCreated without changing any existing service’s code at all.
Think of a classroom “pass it on” game. The teacher says one instruction to the first student (“stand up”), and that student, once done, whispers the next instruction to the next student, and so on. No student needs to know the entire chain of instructions — only their own step and who to tell next.
4.2 Style 2 — Orchestration‑Based Saga
In orchestration, one dedicated component — the Saga Orchestrator — owns the entire recipe of steps. It calls each service directly (often through a command message or a synchronous API call), waits for the result, and decides what happens next, including which compensations to trigger on failure.
Fig 4.2 — Orchestration saga: the orchestrator directs every step and every compensation.
Key Components of an Orchestration‑Based Saga
| Component | Responsibility |
|---|---|
| Saga Orchestrator | Holds the saga’s step definitions, current state, and decides the next action or compensation. |
| Saga Log / State Store | Persists which step the saga is currently on, so it can resume after a crash. |
| Participant Services | Order, Payment, Inventory, Shipping — each exposes an action endpoint and a compensating endpoint. |
| Message Broker / Command Channel | Delivers commands from orchestrator to participants, and results back. |
| Timeout / Retry Manager | Detects when a participant did not respond in time and triggers retry or failure handling. |
Key Components of a Choreography‑Based Saga
| Component | Responsibility |
|---|---|
| Event Producers | Each service publishes events describing what happened after its local transaction. |
| Event Consumers | Each service subscribes to the specific events relevant to its own next action. |
| Message Broker | Routes events reliably between services (e.g., Kafka topics, RabbitMQ exchanges). |
| Outbox Table (per service) | Guarantees that “update my database” and “publish my event” happen together reliably (explained in Chapter 5). |
A simple rule of thumb used widely in the industry: use choreography for 2–4 simple steps, use orchestration for 4+ steps or when the business logic between steps is complex. As soon as you find yourself struggling to draw a clear diagram of “who reacts to what,” that is a strong signal to switch to orchestration.
Internal Working
Let’s zoom into the machinery that makes a saga actually reliable, not just a nice diagram. Three concrete mechanisms are essential: the Saga Execution Coordinator (SEC), the Transactional Outbox, and the Saga State Machine.
5.1 The Saga State Machine
Every saga, whether choreographed or orchestrated, behaves like a state machine — it moves from one well‑defined state to another based on events.
Fig 5.1 — The universal saga state machine. Every saga implementation is a variation of this shape.
Why does this matter so much? Because the state must be saved (persisted) after every single transition, not just kept in memory. If the orchestrator process crashes right after Payment succeeds but before it tells Inventory, we need a durable record saying “this saga was at step 2 of 4” so that when a new orchestrator instance starts up, it can resume exactly where things left off — instead of restarting from scratch or losing track entirely.
5.2 The Transactional Outbox Pattern
Here is a subtle but critical problem: when a service finishes its local transaction (e.g., “mark order as CREATED”), it also needs to publish an event (“OrderCreated”). But what if the database commit succeeds and then the service crashes before it manages to publish the event? The rest of the saga would simply never continue — a silent, invisible failure.
The Transactional Outbox pattern solves this. Instead of publishing the event directly to the message broker, the service writes the event into a special “outbox” table in the very same local database transaction that updates the business data. Since both writes happen in one atomic local transaction, they either both happen or neither does — no more silent gaps. A separate background process (often called a “message relay” or using a tool like Debezium for Change Data Capture) reads the outbox table and reliably publishes those events to the broker.
Fig 5.2 — Transactional Outbox: the database write and the event write happen atomically, so no event is ever silently lost.
It is like writing a memo to yourself (“mail this letter tomorrow”) and stapling it to a document in the same folder, rather than trying to physically drop a letter in the mailbox on your way out the door. Even if you get distracted and forget to visit the mailbox today, the stapled memo guarantees the letter will still get sent later, because someone reviews the folder regularly.
5.3 Retry, Timeout, and Failure Detection
A saga step is not simply “succeeded” or “failed” — there is a very important third possibility: unknown, caused by a network timeout. Did the Payment Service actually charge the card, and the response just got lost on the way back? Or did it never receive the request at all?
Production sagas handle this with:
- Idempotency keys — every command carries a unique ID, so retrying a “charge card” command with the same ID does not charge twice.
- Timeouts with exponential backoff — if no response arrives within X seconds, retry after a growing delay, rather than hammering a struggling service.
- Circuit breakers — after repeated failures, temporarily stop calling a failing service and immediately trigger compensation instead of waiting forever.
Compensation is not “delete the row.” It is a deliberate business action. Cancelling a payment might mean issuing a refund (if the money already left the account) rather than literally deleting the payment record, because the payment record itself may be needed for accounting and audit purposes. This is why compensating transactions are written by hand by developers who understand the business domain — they cannot be auto‑generated.
Data Flow & Lifecycle
Let’s trace one complete order, end to end, through a full success path and then a full failure‑with‑compensation path, using an orchestrated saga. This is the kind of walkthrough interviewers love to hear explained clearly.
6.1 Happy Path Lifecycle
- Trigger: Customer clicks “Place Order.” The Order API receives the request and starts a new saga instance with a unique sagaId.
- Step 1 — Create Order: Order Service inserts an order row with status PENDING. This is committed locally. The orchestrator’s saga log is updated to say “step 1 done.”
- Step 2 — Reserve Inventory: Orchestrator sends a command to Inventory Service. Inventory reserves stock, commits locally, replies success. Saga log updated to “step 2 done.”
- Step 3 — Charge Payment: Orchestrator sends a command to Payment Service. Payment charges the card, commits locally, replies success. Saga log updated to “step 3 done.”
- Step 4 — Schedule Shipping: Orchestrator sends a command to Shipping Service. Shipping books a delivery slot, commits locally, replies success.
- Completion: Orchestrator marks the saga COMPLETED and tells Order Service to flip the order status from PENDING to CONFIRMED.
6.2 Failure Path Lifecycle (with Compensation)
Now suppose Step 3 (Charge Payment) fails because the card is declined.
- Payment Service replies with a failure result: “card declined.”
- Orchestrator marks the saga state as COMPENSATING.
- Orchestrator invokes compensations in reverse order for every step that already succeeded:
- Compensate Step 2: Inventory Service releases the reserved stock.
- Compensate Step 1: Order Service marks the order as CANCELLED (not deleted — kept for history and analytics).
- Orchestrator marks the saga FAILED and notifies the customer that the order could not be completed.
Fig 6.1 — Full lifecycle showing the forward path and the reverse compensation path triggered by a payment failure.
Notice the compensations run in strict reverse order of the completed steps — this mirrors how you would undo stacked plans in real life: last decision made, first decision undone. This is very similar in spirit to a stack data structure (Last‑In‑First‑Out), and many saga orchestrator implementations literally use a stack internally to track which compensations still need to run.
6.3 Java Example — A Minimal Saga Orchestrator
Below is a simplified, illustrative orchestrator written in Java. It is intentionally minimal to show the core idea clearly — production systems would add persistence, retries, and messaging infrastructure around this shape.
public class OrderSagaOrchestrator {
private final OrderServiceClient orderService;
private final InventoryServiceClient inventoryService;
private final PaymentServiceClient paymentService;
private final ShippingServiceClient shippingService;
private final SagaLogRepository sagaLog;
public SagaResult executeOrderSaga(OrderRequest request) {
String sagaId = sagaLog.startNewSaga(request);
List<Runnable> compensations = new ArrayList<>();
try {
orderService.createOrder(sagaId, request);
compensations.add(() -> orderService.cancelOrder(sagaId));
sagaLog.recordStep(sagaId, "ORDER_CREATED");
inventoryService.reserveStock(sagaId, request.getItems());
compensations.add(() -> inventoryService.releaseStock(sagaId, request.getItems()));
sagaLog.recordStep(sagaId, "STOCK_RESERVED");
paymentService.chargeCard(sagaId, request.getPaymentInfo());
compensations.add(() -> paymentService.refund(sagaId));
sagaLog.recordStep(sagaId, "PAYMENT_CHARGED");
shippingService.scheduleDelivery(sagaId, request.getAddress());
sagaLog.recordStep(sagaId, "SHIPPING_SCHEDULED");
sagaLog.markCompleted(sagaId);
return SagaResult.success(sagaId);
} catch (SagaStepException ex) {
// Undo everything that already succeeded, in reverse order
Collections.reverse(compensations);
for (Runnable compensate : compensations) {
compensate.run();
}
sagaLog.markFailed(sagaId, ex.getMessage());
return SagaResult.failure(sagaId, ex.getMessage());
}
}
}A minimal illustrative orchestrator. Each service call is guarded with a matching compensation pushed onto a list, which is reversed and executed on failure.
In real production code, the compensation list above (kept in memory) must instead be reconstructed from the persisted sagaLog after any crash — never trust in‑memory state alone, since the orchestrator process itself might restart mid‑saga.
Advantages, Disadvantages & Trade‑offs
7.1 Advantages
| Advantage | Explanation |
|---|---|
| No distributed locks | Each local transaction commits immediately; nothing sits blocked waiting on a network call to another service. |
| Service autonomy | Each service keeps its own database and technology choice; no shared transaction manager needed. |
| High availability | A slow or temporarily down service does not freeze the whole system — the saga can retry or compensate instead of blocking forever. |
| Natural fit for microservices | Matches the “database‑per‑service” principle that most microservice architectures already follow. |
| Clear audit trail | The saga log itself becomes a readable history of exactly what happened to a business process, useful for debugging and compliance. |
7.2 Disadvantages
| Disadvantage | Explanation |
|---|---|
| No true isolation | Other requests can read intermediate, “in‑progress” data before the saga finishes (e.g., an order showing PENDING for a few seconds). |
| Complex compensations | Some actions are hard or impossible to perfectly undo (e.g., an email already sent, a physical package already shipped). |
| Debugging difficulty | A business process is now spread across several services’ logs, making it harder to trace than a single stack trace. |
| Eventual, not instant, consistency | Users or downstream systems must tolerate a short window where data looks inconsistent across services. |
| More code to write and test | Every step needs a matching, carefully‑tested compensating action — doubling the logic that must be correct. |
What do you do when a compensating action for “send confirmation SMS” does not really exist? You cannot un‑send a text message. In practice, teams handle this by sending a clearly‑labelled follow‑up message (“Sorry, your order #123 has been cancelled”) rather than pretending the first message never happened. This is called a semantic compensation — you don’t erase history, you add a correcting fact to it.
7.3 When To Use a Saga (and When Not To)
- Use a saga when a business process spans multiple services/databases and strict, instant consistency is not legally or practically required.
- Avoid a saga when all the data actually lives in one database — just use a normal local ACID transaction; a saga would only add needless complexity.
- Avoid a saga when the domain truly requires strict atomic consistency across services (e.g., certain financial settlement operations) — in those rare cases, a well‑tuned distributed transaction protocol or a redesign to keep the data together may be more appropriate.
Performance & Scalability
Because sagas avoid holding cross‑service locks, they scale far better under high load than two‑phase commit. Each service can process its own local transaction as fast as its own database allows, completely independent of how slow another service might be at that moment.
8.1 Throughput Considerations
- Choreography scales horizontally very well because message brokers like Kafka are built for massive parallel throughput, and services only need to keep up with their own topic partitions.
- Orchestration adds a bit of overhead because every step passes through the orchestrator, which must persist state changes — but this is usually a small, worthwhile cost for the visibility it buys you.
- Batching and parallel steps can significantly cut total saga latency — e.g., “send confirmation email” and “notify warehouse” can run in parallel since neither depends on the other’s outcome.
8.2 Latency vs Consistency Trade‑off
A saga’s total completion time is the sum of its sequential steps (plus network round trips). For a saga with 5 steps averaging 150ms each, that is roughly 750ms minimum before the whole business process is confirmed — much slower than a single local transaction, but still fast enough for most user‑facing operations if designed carefully with parallel steps and asynchronous notifications (“Your order is confirmed, we’ll email you the shipping details shortly”).
Ride‑sharing apps like Uber use saga‑like flows for the trip lifecycle: request ride → match driver → start trip → end trip → charge fare → pay driver. Each of these steps is independently scaled — the “match driver” service can be scaled up during rush hour without needing the “charge fare” service to scale at the same rate, since they are decoupled by events.
8.3 Scaling the Saga Log / State Store
In high‑throughput orchestrated systems, the saga state store itself can become a bottleneck if it is a single relational database receiving every step update. Common solutions include:
- Sharding the saga log by sagaId hash across multiple database partitions.
- Using a distributed key‑value store (like DynamoDB or Cassandra) purpose‑built for high write throughput with simple lookups.
- Writing saga state changes as an append‑only event log (event sourcing), which scales writes extremely well since nothing is ever updated in place, only appended.
High Availability & Reliability
A saga’s biggest promise is that partial failure never leaves the system in a broken, inconsistent state forever — it will always either complete or fully compensate. Delivering on that promise reliably requires several supporting techniques.
9.1 Guaranteed Message Delivery
Message brokers used for sagas (Kafka, RabbitMQ, AWS SQS/SNS) provide at‑least‑once delivery — meaning a message might be delivered more than once, but it will never simply vanish. Combined with idempotent consumers (Chapter 3), this gives an effective “exactly‑once processing” guarantee from the business logic’s point of view, even though the network layer only promises “at least once.”
9.2 Orchestrator Failover
If the orchestrator process crashes mid‑saga, a replacement instance must be able to pick up exactly where the crashed one left off. This works because:
- The saga’s current step is persisted durably in the saga log after every transition (not just held in memory).
- On startup, any orchestrator instance scans for sagas stuck in an incomplete state and resumes them from their last recorded step.
- Multiple orchestrator instances typically run behind a leader‑election or partitioned‑ownership scheme, so exactly one instance is responsible for driving forward any given saga at a time, preventing duplicate, conflicting actions.
Fig 9.1 — Because saga progress is persisted, any healthy instance can resume a saga after the original orchestrator crashes.
9.3 Handling the “Stuck Saga” Problem
Sometimes a saga gets stuck waiting on a participant that never responds (not failed, not succeeded — just silent). Production systems guard against this with:
- Timeout watchdogs — a background job scans for sagas that have been “in progress” longer than an expected threshold and forces a retry or compensation decision.
- Dead‑letter queues — messages that repeatedly fail processing are routed aside for manual or automated investigation instead of blocking the queue forever.
- Manual intervention dashboards — for rare, truly ambiguous cases, an operator can review a stuck saga and manually decide to force‑complete or force‑compensate it.
The golden rule of saga reliability: never lose track of where a saga is. If you can always answer “which step is this saga on, and did that step’s local transaction commit?”, you can always recover correctly, no matter what crashes.
Security
Sagas introduce their own specific security considerations because business‑critical actions (charging money, releasing goods) now flow across network boundaries as messages and events.
10.1 Authenticating Service‑to‑Service Calls
Every command sent by an orchestrator, or every event published in choreography, must be authenticated so a malicious or buggy service cannot forge a “PaymentCompleted” event. Common approaches:
- Mutual TLS (mTLS) between services, often provided automatically by a service mesh like Istio or Linkerd.
- Signed tokens (e.g., short‑lived JWTs) attached to each command or event, verified by the receiving service before acting on it.
10.2 Protecting the Saga Log
The saga log often contains sensitive business context (order details, partial payment references). It should be encrypted at rest, access‑controlled, and never log raw sensitive fields like full card numbers — only tokens or masked references (e.g., **** **** **** 4242), following standards such as PCI‑DSS for payment data.
10.3 Preventing Replay and Duplicate‑Action Attacks
Because messages can be retried, an attacker who intercepts and re‑sends an old “ChargeCard” command could attempt to trigger a duplicate charge. Idempotency keys (Chapter 3) close this gap: the Payment Service checks “have I already processed this exact idempotency key?” before charging again, regardless of how many times the message physically arrives.
10.4 Compensation as an Attack Surface
Compensating transactions are powerful — a refund action, if triggered maliciously or by a bug, moves real money. Compensation endpoints deserve the same authorisation scrutiny as the original action, not less. A common mistake is to treat “undo” endpoints as low‑risk internal plumbing and skip proper authorisation checks on them — this is exactly the kind of gap attackers look for.
Every saga step and every compensation must independently verify: “Is this request authenticated, authorised, and have I already processed this exact idempotency key?” Never assume a message is safe just because it arrived through an internal message broker — internal networks can still be compromised.
Monitoring, Logging & Metrics
Because a saga spans multiple services, ordinary single‑service monitoring is not enough. You need visibility into the whole story, not just individual chapters.
11.1 Distributed Tracing
Every saga instance should carry a single sagaId (and often a broader correlationId or traceId) that is attached to every log line, event, and command throughout its entire journey. Tools like OpenTelemetry, Jaeger, or Zipkin can then reconstruct the full timeline of a single saga across every service it touched — turning scattered logs into one connected trace, much like following a single thread through a tapestry.
Fig 11.1 — A shared sagaId lets tracing tools stitch together logs from many services into one readable timeline.
11.2 Key Metrics To Track
| Metric | Why it matters |
|---|---|
| Saga completion rate | Percentage of sagas that reach COMPLETED vs FAILED — a sudden drop signals a systemic issue. |
| Saga duration (p50/p95/p99) | How long sagas typically take; spikes hint at a slow downstream participant. |
| Compensation rate | How often compensations trigger — a rising trend often points to a flaky or overloaded service. |
| Stuck saga count | Sagas that have exceeded their expected time‑in‑step, indicating a possible lost message or dead participant. |
| Dead‑letter queue size | Messages that failed processing repeatedly and need investigation. |
11.3 Alerting Strategy
Good production systems alert on business‑relevant thresholds, not just infrastructure metrics — for example, “compensation rate for checkout saga exceeded 5% in the last 10 minutes” is far more actionable to an on‑call engineer than a generic CPU alert, because it points directly at a customer‑facing problem.
Deployment & Cloud
Sagas are cloud‑native by nature — they assume independently deployable services communicating over a network, which maps directly onto container orchestration and managed messaging services.
12.1 Typical Cloud Building Blocks
- Container orchestration — Kubernetes typically hosts each participant service and the orchestrator as independently scalable deployments.
- Managed message brokers — Kafka (or a managed equivalent like Amazon MSK / Confluent Cloud), RabbitMQ, or cloud‑native options like AWS SNS/SQS, Azure Service Bus, or Google Pub/Sub.
- Managed workflow engines — many teams now use managed orchestration engines (like AWS Step Functions, Temporal, or Camunda) instead of hand‑rolling an orchestrator, since these tools already provide durable state, retries, and visual step tracking out of the box.
Older systems often hand‑wrote their own orchestrator with a custom database table for saga state. The modern best practice for new systems is to lean on a purpose‑built durable workflow engine such as Temporal.io, AWS Step Functions, or Camunda, which already solve state persistence, retries, timeouts, and visualisation — letting engineers focus purely on writing the business steps and compensations rather than reinventing orchestration infrastructure.
12.2 Blue‑Green and Canary Deployments With Sagas
Because a saga can be “in flight” for seconds or minutes, deploying a new version of a participant service mid‑saga needs care. Best practice: keep saga command/event schemas backward compatible across versions (never remove a field a previous version might still send), and let in‑flight sagas finish on the old version’s logic while new sagas pick up the new version — a technique often called “version‑aware routing.”
12.3 Disaster Recovery
Because the saga log is the single most important piece of recovery information, it needs the strongest backup and multi‑region replication guarantees in the whole system — losing the saga log means losing the ability to know whether in‑flight business transactions were completed, compensated, or abandoned halfway.
Data Storage & State Management
13.1 Database‑Per‑Service
Sagas assume each participant owns its own database, invisible to other services. This is what makes services independently deployable and scalable, but it is also exactly what creates the need for sagas in the first place — a nice, symmetric relationship between the microservices principle and the pattern that supports it.
13.2 The Saga State Store
For orchestration, the state store needs to answer, quickly and reliably: “For sagaId X, what step are we on, and what compensations are pending?” This is typically a simple table:
CREATE TABLE saga_instance (
saga_id VARCHAR(36) PRIMARY KEY,
saga_type VARCHAR(50) NOT NULL,
current_step VARCHAR(50) NOT NULL,
status VARCHAR(20) NOT NULL, -- STARTED, COMPENSATING, COMPLETED, FAILED
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);A simple saga_instance table — the single source of truth for “where is this saga right now?”
13.3 Event Sourcing as an Alternative Storage Model
Instead of overwriting the current_step column, some teams store every state transition as an immutable, appended event (OrderCreated, StockReserved, PaymentFailed, StockReleased…). The current state is then simply “replay all events for this sagaId in order.” This approach, called event sourcing, gives a perfect audit trail for free and scales writes extremely well, at the cost of slightly more complex reads (you must replay, or maintain a separate cached “current state” projection).
13.4 Caching Considerations
Because saga steps often need to read reference data (like current stock levels or a customer’s loyalty tier), a cache (like Redis) can speed things up — but be careful: a stale cache read during a saga step could cause an incorrect business decision (e.g., approving an order based on outdated stock counts). A common safe pattern is to cache read‑only reference data aggressively, but always read the specific record being modified (like the actual stock row for this order) directly from the source‑of‑truth database.
13.5 Data Partitioning
At very high saga volumes, saga instance tables are often partitioned by sagaId hash or by date, so that no single database node becomes a bottleneck as saga volume grows, and so that older completed sagas can be archived out efficiently once they are no longer actively needed for real‑time lookups.
APIs & Microservices
Sagas are, at their heart, a microservices communication pattern, so let’s look at exactly how the API surface of each participant is typically shaped.
14.1 Each Participant Needs Two Endpoints
For every forward action a service can perform in a saga, it should also expose a matching compensating action:
// Payment Service — REST-style API for saga participation
POST /payments/charge
Request:
{
"sagaId": "abc123",
"orderId": "ord-789",
"amount": 49.99,
"idempotencyKey": "abc123-charge"
}
Response (success):
{
"status": "CHARGED",
"transactionId": "txn-555"
}
POST /payments/refund
Request:
{
"sagaId": "abc123",
"transactionId": "txn-555",
"idempotencyKey": "abc123-refund"
}
Response:
{
"status": "REFUNDED"
}Notice both endpoints accept an idempotencyKey — calling either endpoint twice with the same key has no extra effect.
14.2 Synchronous vs Asynchronous Communication
| Style | Description | Typical Use |
|---|---|---|
| Synchronous (REST/gRPC) | Orchestrator calls a participant and waits directly for the HTTP/gRPC response. | Orchestration sagas, simpler to reason about, but couples availability to the participant being reachable right now. |
| Asynchronous (events over a broker) | Caller publishes a message and moves on; the response (if any) arrives later as another message. | Choreography sagas, and orchestration sagas that want maximum resilience to temporary participant downtime. |
14.3 API Gateway’s Role
The customer‑facing API Gateway typically triggers the saga (e.g., POST /orders) and immediately returns a fast, simple response like “Order received, processing” along with a sagaId the client can poll or subscribe to for status updates — since waiting synchronously for a multi‑second saga to fully finish would create a poor, sluggish user experience.
Many e‑commerce checkout APIs immediately return HTTP 202 Accepted with an order status URL, rather than making the customer’s browser wait several seconds for every saga step to finish. The frontend then polls that URL, or receives a WebSocket/push notification, once the saga reaches COMPLETED or FAILED.
Design Patterns & Anti‑patterns
15.1 Related and Supporting Patterns
| Pattern | Relationship to Saga |
|---|---|
| Transactional Outbox | Guarantees a service’s database write and its event publication happen atomically (Chapter 5). |
| Event Sourcing | An alternative way to persist saga (and service) state as an append‑only log of events. |
| CQRS (Command Query Responsibility Segregation) | Often paired with sagas so that “write” commands driving the saga are separated from “read” views used to show status to users. |
| Circuit Breaker | Protects the saga from repeatedly calling a participant that is clearly down, failing fast into compensation instead. |
| Retry with Backoff | Handles transient failures within a single saga step before giving up and triggering compensation. |
| Process Manager | A more general pattern (from enterprise integration) that orchestration‑based sagas are a specific application of. |
15.2 Common Anti‑patterns
Building a saga where every step makes a synchronous call and waits, chaining five or six services together in one blocking sequence, effectively recreates all the fragility of a single big transaction — except now spread across a slow, unreliable network, with none of ACID’s actual guarantees. If one service in the chain is slow, the whole chain is slow, and a lot of the saga’s promised resilience is lost.
Writing saga step logic that assumes “this command will only ever be received once” is one of the most common production bugs. Message brokers retry. Networks glitch. Without idempotency keys, a retried “charge card” command can double‑charge a real customer.
A compensating transaction can itself fail or be retried. If “release stock” is not idempotent either, retried compensations can over‑release stock, causing overselling later. Every compensation needs the same idempotency discipline as the forward action.
A saga with no timeout can sit “in progress” forever if a participant never responds, silently locking business data (e.g., reserved stock never released) with nobody aware anything is wrong. Always define an expected maximum duration per step, with automated escalation.
As a choreographed saga grows past 4–5 steps, event chains become genuinely hard to trace (“which of these six services published this event, and who exactly reacts to it?”). This is the strongest signal to migrate to orchestration, where the entire flow lives in one readable place.
Best Practices & Common Mistakes
16.1 Best Practices Checklist
- Design compensations first. Before writing the “do” logic, ask “how would I undo this?” If you cannot answer that clearly, redesign the step.
- Make every step and every compensation idempotent. Assume every message will be delivered more than once, always.
- Persist saga state after every transition, never rely on in‑memory state surviving a crash.
- Use semantic locks (status flags) like PENDING to warn other parts of the system that a record is mid‑saga.
- Set explicit timeouts per step, with automated retry or escalation to compensation.
- Attach a single sagaId/correlationId to every log line and message for end‑to‑end traceability.
- Prefer orchestration once you pass roughly 4 steps, for readability and centralised control.
- Keep event/command schemas backward compatible so in‑flight sagas survive rolling deployments.
- Test failure paths as rigorously as success paths. Simulate every possible step failing, not just the “happy path.”
- Monitor compensation rate as a first‑class business metric, not an afterthought.
16.2 Common Mistakes Beginners Make
- Treating a saga like a distributed transaction with real isolation. It is not — other requests can see intermediate states. Design your UI and downstream consumers to tolerate this.
- Skipping compensation logic for “simple” steps. Even a step that seems minor (like updating a loyalty points balance) needs a real undo path if it might need reversing later.
- Hard‑coding step order without a persisted plan. If the code that decides “what is next” only lives in memory, a crash loses the saga’s progress entirely.
- Ignoring partial compensation failure. What happens if the compensation itself fails? Production systems need a retry‑and‑alert strategy for this exact scenario, not silent failure.
- Confusing saga status with HTTP status. Returning HTTP 200 immediately does not mean the whole business process succeeded — it usually just means “the saga started,” which must be communicated clearly to API consumers.
Real‑World & Industry Examples
The Saga pattern is not just theory — it powers business processes at some of the largest companies in the world. Here are a few concrete examples of how sagas manifest in production, each one echoing a chapter of the story we have been telling.
Amazon — Order Fulfilment
Amazon’s order pipeline coordinates inventory reservation, payment authorisation, and warehouse fulfilment across many independently owned systems. When a payment authorisation fails after inventory has been reserved, the reservation must be released — a textbook saga compensation, operating at massive global scale across many regional systems.
Uber — Trip Lifecycle
A single Uber trip touches many independent services: rider request, driver matching, real‑time location tracking, trip completion, fare calculation, and driver payout. If fare calculation fails after a trip is marked complete, compensations must correct the driver’s payout record and possibly issue a rider adjustment — coordinated as a saga‑like chain of steps and corrections.
Netflix — Subscription & Billing
Netflix’s account, billing, and content‑access systems are separate services. A subscription upgrade might involve charging a new plan rate and separately unlocking new content tiers. If billing fails after content access was already granted, a compensating action must revoke that early access grant until billing succeeds.
Airlines & Travel Booking
Booking a trip often means reserving a flight seat, a hotel room, and a rental car — frequently across three completely different companies’ systems. This is one of the clearest real‑world sagas: if the hotel booking fails after the flight is confirmed, the flight reservation may need to be released (compensated) or the customer offered an alternative, exactly mirroring the wedding‑planning analogy from Chapter 1.
Banking — Funds Transfer Between Institutions
Transferring money between two different banks cannot use a single ACID transaction, because no shared database exists between two banks. Instead, systems like ACH and SWIFT use saga‑like, multi‑step settlement processes with well‑defined reversal (“compensating”) procedures if a later step in the transfer chain fails.
Food Delivery — Order to Doorstep
An order on a food‑delivery platform accepts payment, places an order with a restaurant, dispatches a courier, tracks the delivery, and finally settles pay for both restaurant and courier. If the restaurant rejects the order after payment is captured, the platform compensates by refunding the customer — a saga in miniature that plays out millions of times a day.
FAQ, Summary & Key Takeaways
18.1 Frequently Asked Questions
Is a saga the same thing as a distributed transaction?
No. A distributed transaction (like Two‑Phase Commit) tries to give all‑or‑nothing atomicity across services in real time, holding locks until everyone agrees. A saga instead lets each local transaction commit independently and immediately, and relies on compensating transactions afterward if something later fails. Sagas trade strict, instant consistency for availability and much better scalability.
Can a saga guarantee that data is never inconsistent, even briefly?
No, and this is by design. A saga guarantees eventual consistency — the system will reach a correct final state, but there may be a short window during execution where different services show different, temporarily‑mismatched views of the data.
Should I always use orchestration instead of choreography?
Not always. Choreography is simpler and more decoupled for short sagas (2–4 steps). Orchestration becomes clearly worth its extra centralisation once the number of steps or the branching complexity grows, because it keeps the entire business process visible and manageable in one place.
What happens if a compensating transaction itself fails?
This must be handled explicitly — usually with automatic retries (since compensations should be idempotent) and, if retries are exhausted, an alert to a human operator or a dead‑letter queue for manual resolution. A saga design is not complete until this exact scenario has a defined answer.
Do all saga steps need a compensating transaction?
Almost all do, but a step that is the very last one in the saga (nothing can fail after it) technically does not need a compensation, since there is nothing left to protect against. Similarly, some naturally “pivot” steps (once passed, the saga is guaranteed to complete, such as after an irreversible external confirmation) do not need further compensations beyond that point.
Is the Saga pattern only for microservices?
It was actually invented before microservices existed, originally for very long‑running single‑database transactions. But its most common modern use, by far, is coordinating business processes across independently‑owned microservices, which is the focus of this chronicle.
How is a saga different from a plain workflow engine?
A workflow engine (like BPM tools) is a general way to model business processes as steps. A saga is a specific discipline that a workflow engine can implement — one where every step has a matching compensating step and the system aims at eventual consistency across independent databases. Many modern workflow engines like Temporal or AWS Step Functions are, in practice, excellent tools for building sagas.
Can a saga span more than one team?
Yes — and this is very common in large organisations, since each participant service is often owned by a different team. The important thing is that all teams agree on the shared event/command schemas and idempotency conventions, so their services can cooperate across the saga without stepping on each other.
18.2 Summary
The Saga pattern breaks one long business transaction into a chain of small, local transactions, each owned entirely by one service, with a matching compensating transaction ready to undo it if a later step in the chain fails. It replaces the strict, blocking guarantees of distributed transactions like Two‑Phase Commit with a more scalable, available design built on eventual consistency. There are two coordination styles — choreography (services react to each other’s events with no central controller) and orchestration (one central component drives every step and every compensation) — and the right choice depends mainly on how many steps and how much branching logic the business process involves.
Making sagas reliable in production depends on a handful of disciplined techniques: idempotent operations so retries never cause double effects, a durable saga state store so progress is never lost to a crash, the transactional outbox pattern so a service’s database write and its event are never silently out of sync, distributed tracing tied together by a single sagaId, and clear timeout and escalation rules so no saga is ever silently stuck forever.
18.3 Key Takeaways
- 01Sagas exist because no single ACID transaction can span multiple independently‑owned microservice databases.
- 02A saga is a sequence of local transactions, each with a compensating transaction to undo it if needed.
- 03Choreography = event‑driven, no central controller. Orchestration = one central coordinator drives every step.
- 04Sagas provide eventual consistency, not the instant, strict consistency of a single database transaction.
- 05Idempotency is non‑negotiable — every step and every compensation must safely handle being triggered more than once.
- 06Persisted saga state, distributed tracing, and clear timeout policies are what separate a fragile saga implementation from a production‑grade one.
- 07Real systems at Amazon, Uber, Netflix, airlines, and banks all rely on saga‑like patterns to coordinate business processes across independently‑owned systems.
“A saga does not promise that nothing goes wrong. It promises that when something does, the story never ends in the middle of a chapter.”