Microservices & Distributed Transactions — Why 2PC Broke and Sagas Took Over
A complete, first‑principles walk through why ACID transactions do not survive the jump from one big database to many small ones — and exactly how modern microservices stay consistent using Sagas, the Outbox pattern, idempotency, and eventual consistency instead.
Introduction & History
Imagine you have one big toy box. Everything you own is in that one box. If you want to move a toy from one corner of the box to another, it’s easy — you just reach in with one hand and move it. Nothing can go wrong halfway, because it’s all one box, one action, one moment.
Now imagine you have ten separate toy boxes, kept in ten different rooms, sometimes in ten different houses, owned by ten different friends. If you want to move a toy from box 3 to box 7, you now need to talk to two friends, coordinate the timing, and hope nobody closes their box at the wrong moment. This is much harder. This exact difference — “one box” versus “ten boxes in ten houses” — is the entire reason this tutorial exists.
In software terms, the “one big toy box” is a monolithic application: one program, one database, running mostly on one type of infrastructure. The “ten boxes in ten houses” are microservices: many small, independent programs, each with its own database, often running on different machines, sometimes in different data centers or even different countries.
1.1 A short history
In the 1970s and 1980s, most business software was monolithic. A bank’s software, an airline’s booking system, a retailer’s inventory system — each was usually one large application talking to one central database. When you needed to update multiple things at once (like “subtract money from account A and add money to account B”), the database itself guaranteed that either both changes happened, or neither did. This guarantee is called a transaction.
As these single databases became bottlenecks for very large systems, engineers started spreading data across multiple databases or multiple machines — this is called a distributed system. To keep the same “all or nothing” guarantee across multiple machines, researchers in the late 1970s (notably at IBM and in the distributed database research community) invented a protocol called Two-Phase Commit (2PC). It let multiple databases agree together on whether a transaction should succeed or fail, as if they were one big database.
2PC worked reasonably well when:
- All the databases were on the same reliable local network.
- The number of participants was small.
- The system was owned and operated by a single team or company.
Then, starting around 2011–2014, companies like Netflix, Amazon, and later Uber and Airbnb popularized microservices architecture — breaking one giant application into dozens or hundreds of small, independently deployed services, each owning its own database. This solved many problems (scaling teams, scaling traffic, faster deployments) but it reopened the old distributed transaction problem — except now at a much bigger, messier, internet scale, where 2PC’s old assumptions no longer hold true.
This tutorial explains, from first principles, exactly why traditional distributed transactions (like 2PC) do not fit well into a microservices world, and what engineers use instead today.
Diagram 1 — How the industry moved from single databases to microservices, and why new transaction techniques had to be invented along the way.
Problem & Motivation
Let’s build the problem up slowly, using a simple, real example that we will reuse throughout this entire tutorial: an online shopping checkout.
When you buy a pair of shoes on an e-commerce site, several things must happen together:
- Order Service creates a new order record.
- Payment Service charges your credit card.
- Inventory Service reduces the shoe stock count by 1.
- Shipping Service schedules a delivery.
If any one of these steps fails — say, the payment fails — you don’t want an order created with no payment, or stock reduced with no order behind it. Either all four things happen, or none of them happen. That’s the guarantee we want.
2.1 Why this was easy in a monolith
In a monolithic system, all four of those “services” are really just modules inside one application, and all four pieces of data (orders, payments, inventory, shipping) live in tables inside one single database. The database engine itself (like PostgreSQL, MySQL, or Oracle) has built-in support for wrapping multiple changes into one transaction:
SQL — one transaction covers all four tables in a monolithic app
BEGIN TRANSACTION;
INSERT INTO orders (id, customer_id, status) VALUES (101, 55, 'CREATED');
UPDATE payments SET status = 'CHARGED' WHERE order_id = 101;
UPDATE inventory SET stock = stock - 1 WHERE product_id = 77;
INSERT INTO shipments (order_id, status) VALUES (101, 'SCHEDULED');
COMMIT;
If anything fails before COMMIT, the database automatically undoes (rolls back) everything. This “all or nothing” behavior is guaranteed by the database engine itself, for free, because everything is in one place, managed by one piece of software, on one machine (or one tightly-coupled cluster).
2.2 Why this becomes hard with microservices
Now split that single database into four separate databases, each privately owned by its own microservice:
Diagram 2 — In a monolith, one transaction covers everything inside one database. In microservices, the same business action is spread across four separate, independently-owned databases.
Now, a single BEGIN TRANSACTION ... COMMIT statement is impossible, because no single database engine can see or control the other three databases. You cannot write one SQL transaction that spans PostgreSQL (Order DB), a different PostgreSQL instance (Payment DB), MongoDB (Inventory DB), and MySQL (Shipping DB) at once — they are different processes, on different machines, possibly different vendors entirely.
This is the exact motivation for this whole topic: microservices architecture intentionally breaks apart the single source of truth that traditional transactions depend on, in exchange for independent scaling, independent deployment, and team autonomy. The rest of this tutorial explains precisely why old solutions (like 2PC) don’t work well here, and what modern solutions look like instead.
Core Concepts
Before going further, let’s carefully define every important term. Nothing is assumed.
WhatACID is an acronym describing four guarantees a database transaction provides: Atomicity (all steps happen or none do), Consistency (data always moves from one valid state to another valid state), Isolation (transactions don’t interfere with each other, even if run at the same time), and Durability (once committed, changes survive crashes, power loss, etc.).
WhyWithout these guarantees, concurrent updates to shared data could corrupt it — e.g., two people withdrawing money from the same account at the same instant could both succeed even if there wasn’t enough money.
WhereEvery traditional relational database (PostgreSQL, MySQL, Oracle, SQL Server) and even some NoSQL databases within a single document or partition.
AnalogyACID is like a strict referee in a card game. Either the whole move is played correctly, following all the rules, or the referee cancels the entire move and puts every card back exactly where it was. No half-finished moves are ever allowed to stay on the table.
WhatA transaction whose steps span more than one physical database, service, or machine, but which still needs to behave with the same “all or nothing” guarantee as a normal transaction.
WhyOnce data is spread across multiple systems (for scaling, ownership, or technology reasons), you still sometimes need multiple systems to agree together on a single outcome.
WhereBanking transfers between two different banks, travel booking systems combining flights + hotels + car rentals from different providers, and — our focus here — microservices.
WhatA classic protocol that coordinates multiple databases (called “participants”) through a central “coordinator” so they all commit or all abort together.
WhyIt was the first widely adopted way to get atomicity across multiple independent databases.
WhereOlder enterprise systems, some traditional message queues (JMS/XA transactions), some legacy banking middleware.
AnalogyImagine a wedding planner (the coordinator) calling the caterer, the florist, and the band (the participants) and asking each: “Are you 100% ready to confirm for Saturday?” Only after all three say “yes, I’m ready and locked in” does the planner call back and say “Great, everyone confirm now.” If even one says “no,” the planner tells everyone to cancel. The problem: while everyone is “locked in and waiting,” none of them can take any other booking for that day — even if the final confirmation call takes hours to arrive.
WhatAn architectural style where an application is built as a collection of small, independently deployable services, each responsible for one specific business capability (e.g., “Payments,” “Inventory”), each usually owning its own private database.
WhyTo let large engineering organizations build, deploy, and scale different parts of a system independently, without one team’s mistake or one part’s traffic spike affecting everyone else.
WhereNetflix, Amazon, Uber, Airbnb, Spotify, and most large modern tech companies.
WhatThe rule that each microservice owns its own database exclusively, and no other service is allowed to read or write that database directly — they can only ask through the service’s API.
WhyIf every service could reach into every other service’s database directly, you would lose independence — you couldn’t change a database’s internal table structure without breaking five other services. This is the single most important rule causing our whole topic’s problem.
AnalogyEach friend keeps their toy box in their own locked room, and only they hold the key. If you want a toy moved, you have to ask them nicely (through their “front door,” i.e., an API) — you can’t just walk into their room and grab it yourself.
WhatA weaker consistency model where, after an update, all parts of the system will eventually reflect that update — but maybe not immediately.
WhyWaiting for every single system to be perfectly in sync at every instant is often too slow or too fragile across a network. Accepting a short delay makes systems faster and more resilient.
AnalogyIf you text five friends “party moved to 6pm,” they don’t all read it in the same millisecond. For a few minutes, some friends know the new time and some don’t. Eventually, everyone reads the text and they all agree. That short window of disagreement is “eventual consistency.”
WhatA sequence of local transactions, where each step has a matching “undo” step (called a compensating transaction), used instead of one giant distributed transaction.
WhyIt replaces the need for locking everyone together (like 2PC does) by instead saying: “do your part now, and if something later fails, we’ll individually undo each part that already happened.”
Architecture & Components
To understand why 2PC struggles in microservices, we need to look closely at its architecture.
4.1 Components of Two-Phase Commit
| Component | Role |
|---|---|
| Transaction Coordinator | The central authority that starts the transaction, asks all participants to “prepare,” collects their votes, and tells everyone the final decision (commit or abort). |
| Participants (Resource Managers) | The individual databases/services taking part. Each must be able to “prepare” (lock resources and be ready to commit) and later either commit or roll back on command. |
| Transaction Log | A durable record the coordinator keeps so that if it crashes, it can recover and remember what decision was in progress. |
| XA Protocol | An open industry standard (from The Open Group) that lets different vendors’ databases and message queues speak the same “prepare/commit/rollback” language to a coordinator. |
Diagram 3 — The 2PC architecture: one coordinator, many participants, two rounds of communication.
4.2 Why this architecture clashes with microservices
Microservices architecture is built around these principles:
- Service autonomy — each service should be deployable and operable independently.
- No shared database — each service privately owns its data.
- Loose coupling — services should not need to know internal details of other services.
- Polyglot persistence — different services often use different database technologies (SQL, NoSQL, key-value, etc.) suited to their needs.
- Independent scaling and failure domains — one service being slow or down should not lock up the whole system.
2PC’s architecture directly conflicts with almost every one of these principles:
| Microservices Principle | What 2PC Requires | Conflict |
|---|---|---|
| Loose coupling | A central coordinator that knows about and controls every participant | Creates a new, powerful, tightly-coupled central component that all services depend on |
| Independent scaling | Every participant must lock its data and wait during the “prepare” phase | A slow participant blocks and holds locks in every other participant, hurting throughput of the whole group |
| Polyglot persistence | All participants must support the same 2PC/XA protocol | Many modern NoSQL databases (MongoDB, Cassandra, DynamoDB) and message brokers (Kafka) do not support XA at all |
| Independent failure domains | If the coordinator crashes mid-protocol, participants can be stuck holding locks indefinitely (“blocking problem”) | One coordinator crash can freeze many unrelated services |
| Fast, independent deployment | Long-lived locks during network calls across services | Network latency between services (much higher than within one database) makes locks held far longer, killing throughput |
Internal Working of Two-Phase Commit (Step by Step)
Let’s walk through 2PC in complete detail, using our checkout example (Order, Payment, Inventory).
5.1 Phase 1: Voting / Prepare Phase
- The coordinator sends a PREPARE message to every participant (Order DB, Payment DB, Inventory DB).
- Each participant does all the work needed to be ready to commit — it writes the change to a temporary log, acquires locks on the affected rows, but does not make the change visible yet.
- Each participant replies with a vote: YES (“I can commit”) or NO (“I cannot commit, e.g. insufficient stock”).
5.2 Phase 2: Commit / Abort Phase
- If all participants voted YES, the coordinator sends COMMIT to everyone. Each participant makes its change permanent and releases its locks.
- If any participant voted NO (or timed out), the coordinator sends ABORT to everyone. Each participant undoes its temporary change and releases its locks.
Diagram 4 — The happy path of 2PC: all participants vote YES, and everything commits together.
5.3 The critical weakness: the “blocking problem”
Now consider what happens if the coordinator crashes after sending PREPARE but before sending the final COMMIT/ABORT:
Diagram 5 — The blocking problem. Every participant is stuck holding locks — unable to serve other requests on those rows — until the coordinator recovers.
In a single data center with one coordinator and databases from the same vendor, this crash window is small and rare. But in microservices, the “coordinator” would need to talk across many independently deployed, independently scaled, independently failing services — over an unreliable network, with each service possibly being restarted, redeployed, or auto-scaled at any moment (which is completely normal in microservices!). This dramatically increases how often such a “stuck, locked, waiting” scenario would happen — and it can freeze parts of totally unrelated features, since locked rows can’t be touched by any other operation.
Java example: what a “prepare” step looks like (simplified, conceptual)
Java — XA-style resource with prepare / commit / rollback
// Simplified illustration of an XA-style resource in Java (JTA / XAResource concept)
public class InventoryXAResource implements javax.transaction.xa.XAResource {
public int prepare(Xid xid) throws XAException {
// Lock the inventory row, write to an undo-log, but do NOT commit yet
inventoryDao.lockRowForTransaction(xid, productId);
boolean hasStock = inventoryDao.checkStock(productId, quantity);
if (!hasStock) {
return XAResource.XA_RDONLY; // vote to rollback
}
return XAResource.XA_OK; // vote YES, ready to commit
}
public void commit(Xid xid, boolean onePhase) throws XAException {
inventoryDao.applyReservedChange(xid); // make it permanent
inventoryDao.releaseLock(xid);
}
public void rollback(Xid xid) throws XAException {
inventoryDao.discardReservedChange(xid); // undo
inventoryDao.releaseLock(xid);
}
}
Notice how prepare() locks a row and waits. If the coordinator never calls commit() or rollback() (because it crashed), that lock stays held — this is the blocking problem made concrete in code.
Data Flow & Lifecycle in Microservices (Without 2PC)
Since 2PC doesn’t fit well, let’s preview (in detail later, section 16) what a modern data flow looks like using the Saga pattern, so you can compare lifecycles directly.
Diagram 6 — A Saga’s lifecycle: each service commits its OWN local transaction immediately and independently, then triggers the next step. No cross-service locks are ever held.
The critical difference from 2PC: at every single step, each local database transaction is committed immediately and permanently using ordinary ACID guarantees within that one service’s own database. There is no waiting, no cross-service locking, and no single coordinator holding everyone hostage. If a later step fails, we don’t roll back a lock — we run a new compensating transaction (e.g., “refund payment,” “restore stock”) to logically undo the earlier committed step.
Diagram 7 — Failure lifecycle in a Saga. Instead of a lock rollback, we run a compensating transaction (refund) to logically undo what was already committed.
Advantages, Disadvantages & Trade-offs
7.1 Two-Phase Commit (2PC) — Trade-offs
Advantages
Strong consistency — data is never in a “partially done” state visible to others.
Simple mental model: “all or nothing,” just like a local transaction.
No need to write compensating (undo) logic.
Well-understood, decades of tooling in traditional enterprise middleware.
Disadvantages
Blocking problem if the coordinator crashes.
Requires all participants to support XA/2PC — most modern NoSQL stores and message brokers (Kafka, MongoDB, DynamoDB) do not.
Holds locks for the entire round trip across the network — very slow at scale.
Single coordinator becomes a central point of failure and a scaling bottleneck.
Tightly couples independently-deployed services together at the transaction level, defeating the purpose of microservices.
7.2 Saga Pattern (the modern alternative) — Trade-offs
Advantages
No cross-service locks — each service commits independently and immediately.
Works with any database technology per service (polyglot persistence).
Services remain independently deployable and scalable.
Resilient to individual service slowness — doesn’t freeze unrelated services.
Disadvantages
Data is only eventually consistent — there’s a window where things look “in progress.”
Requires writing compensating transactions (extra development effort).
Harder to reason about and debug (more moving parts, need good tracing).
Compensations aren’t always perfectly possible (e.g., you can’t “un-send” an email; you can only send an apology).
2PC trades availability and scalability for strong consistency. Sagas trade strong, immediate consistency for availability, scalability, and service independence. Microservices architecture almost always prioritizes the second set of properties, which is exactly why the industry moved toward Sagas and event-driven patterns.
CAP Theorem & Consistency Models
8.1 What is the CAP theorem?
A rule, proven formally by computer scientist Eric Brewer and later formalized by Gilbert and Lynch, stating that a distributed system can only guarantee two of these three properties at the same time, whenever a network problem (a “partition”) occurs:
- Consistency (C): every node sees the same, most up-to-date data at the same time.
- Availability (A): every request gets a (non-error) response, even if it’s not the very latest data.
- Partition Tolerance (P): the system keeps working even if network messages between nodes are lost or delayed.
Imagine two security guards at two entrances of a concert, each keeping a paper list of who has already entered (to stop the same ticket being used twice). If the phone line between them goes down (a “partition”), each guard has two choices: (1) stop letting anyone in until the phone line is fixed (sacrificing Availability, to protect Consistency), or (2) keep letting people in using their own local list, risking that the same ticket might get used twice at both doors (sacrificing Consistency, to protect Availability). They cannot have perfect, instant agreement and keep serving everyone and survive the phone line being down, all three at once.
8.2 Why this matters for our topic
A network partition is not some rare, exotic event in a microservices world — it happens constantly and in small forms: a slow response, a timeout, a dropped packet, a service being redeployed for 10 seconds, an auto-scaler removing an instance. Since network partitions are a “when,” not an “if,” in microservices, and since most business systems need to stay Available (a shopping site can’t just stop responding), most microservices architectures deliberately choose AP (Availability + Partition Tolerance) over strict CP (Consistency + Partition Tolerance) for cross-service operations — accepting eventual consistency instead of the strict, instant consistency that 2PC tries to guarantee.
Diagram 8 — The CAP theorem forces a choice during network partitions. Most consumer-facing microservices choose Availability (AP) over strict Consistency (CP).
8.3 Consistency Models Explained Simply
| Model | Meaning | Simple Analogy |
|---|---|---|
| Strong Consistency | Every read always returns the latest write, instantly, everywhere | Everyone in the room hears an announcement at the exact same millisecond |
| Eventual Consistency | Reads may be stale briefly, but will catch up over time | A group text message that takes a few seconds to reach everyone |
| Causal Consistency | Related events are seen in the correct order, but unrelated events can be seen in any order | You always see a question before its reply, but not necessarily in the same order as an unrelated chat thread |
Performance & Scalability
Let’s quantify why 2PC hurts performance so much in a microservices environment, using simple, honest estimates (not exact numbers from any specific vendor, just illustrative reasoning).
9.1 Latency math
Inside a single database, a lock might be held for microseconds to low milliseconds — the time it takes the CPU to write to disk or memory. But across microservices over a network:
- A network round trip between two services might take 5–50 milliseconds (more with retries, TLS handshakes, or cross-region calls).
- 2PC needs two full network round trips to every participant (PREPARE, then COMMIT/ABORT).
- Locks are held across all of that time, on every participant, for the entire two-phase exchange.
If 3 services are involved and each round trip takes even just 20ms, that’s roughly 40ms minimum of held locks per transaction (often much more with retries or a slow participant). Now imagine 5,000 checkouts happening per second on a busy sale day. Every one of those has to queue up and wait if it touches an inventory row another transaction has locked. This creates a severe throughput ceiling — the system can no longer scale just by adding more servers, because the bottleneck is the length of time locks are held, not the number of machines.
9.2 Why Sagas scale better
In a Saga, each service commits its own change immediately (no waiting on others), so locks are only held for microseconds to milliseconds — same as a normal local database transaction. Coordination between services happens after the data is already safely committed, usually via asynchronous events, so slow downstream services don’t block upstream ones from finishing their own local work quickly.
Diagram 9 — In 2PC, locks span the entire multi-step network conversation, which is the main scalability killer.
9.3 Horizontal scaling and sharding
What it is: Splitting a service’s own database into multiple smaller pieces (shards), each holding a portion of the data (e.g., by customer ID range), so the service can handle more load by adding more database nodes.
Why it matters here: 2PC becomes even harder once each participant is itself sharded, because now the coordinator may need to talk to many more shard-level participants, multiplying the blocking problem.
High Availability & Reliability
10.1 Single point of failure
The 2PC coordinator, by design, is a single, central authority. If it goes down while transactions are “in-flight” (prepared but not committed), every participant with locked rows is stuck until it comes back online and finishes the protocol (this is called transaction recovery). In a microservices world with hundreds of services and thousands of transactions per second, having any single component whose crash can freeze rows across many services is a serious reliability risk — it violates the “independent failure domain” principle that makes microservices valuable in the first place.
10.2 How Sagas improve availability
Because each step commits independently, a failure in one service (say, Shipping Service is down) doesn’t prevent Order Service and Payment Service from completing their own local work. The system can retry the failed step later, or trigger compensations, without freezing the healthy parts of the system.
Diagram 10 — A Saga tolerates a temporarily down service without freezing the rest of the system, unlike 2PC’s blocking behavior.
10.3 Idempotency — a critical reliability concept
What it is: An operation is “idempotent” if performing it multiple times has the exact same effect as performing it once.
Why it exists: In distributed systems, messages can be retried (due to timeouts, network blips) and might arrive more than once. If “charge the card $50” isn’t idempotent, a retried message could charge the customer twice.
Pressing an elevator call button five times doesn’t call five elevators — the button remembers “already requested” and ignores repeats. That’s idempotency.
Java example: idempotent payment processing using an idempotency key
Java — idempotent charge via a unique key
@Service
public class PaymentService {
private final PaymentRepository paymentRepo;
public PaymentResult chargeCard(String idempotencyKey, String cardToken, BigDecimal amount) {
// Check if we've already processed this exact request before
Optional<Payment> existing = paymentRepo.findByIdempotencyKey(idempotencyKey);
if (existing.isPresent()) {
return PaymentResult.from(existing.get()); // return the SAME result, don't charge again
}
Payment payment = new Payment(idempotencyKey, cardToken, amount, "PROCESSING");
paymentRepo.save(payment); // committed immediately, local ACID transaction
boolean success = paymentGateway.charge(cardToken, amount);
payment.setStatus(success ? "SUCCESS" : "FAILED");
paymentRepo.save(payment);
return PaymentResult.from(payment);
}
}
This pattern lets any retried “charge the card” message be safely re-processed without double-charging — an essential building block for reliable Sagas and event-driven microservices.
Security
Security concerns shift significantly once you move away from centralized 2PC transactions.
11.1 Authentication and authorization between services
In a monolith, one function calling another is just a local method call — no network, no security boundary. In microservices, every step of a Saga (Order → Payment → Inventory) crosses a network boundary, so each call must be authenticated (proving “who is calling”) and authorized (proving “are they allowed to do this”). Common approaches include mutual TLS (mTLS) between services and short-lived JWT (JSON Web Tokens) passed along with requests.
11.2 Data exposure across the network
Because Sagas often use events published to a message broker (like Kafka or RabbitMQ), sensitive data (like partial card details or personal information) that used to stay inside one database now travels across the network and often sits in a durable event log. This must be encrypted in transit (TLS) and, where needed, at rest, and access to topics/queues must be tightly scoped.
11.3 Compensating transactions and audit trails
Because there is no single ACID transaction log to rely on, and because refunds, cancellations, and retries all leave separate trails across services, systems need strong audit logging (recording exactly what happened, when, and why) to support fraud investigations, compliance, and dispute resolution — this is more complex than in a monolith where one transaction log tells the whole story.
Trusting an internal event blindly just because it “came from another internal service.” In a mature microservices setup, every service call — even internal ones — should be authenticated and validated, following a “zero trust” model, because a compromised internal service could otherwise forge fraudulent Saga events (e.g., a fake “payment succeeded” event).
Monitoring, Logging & Metrics
This is one of the biggest practical challenges microservices introduce when they replace one big transaction with many small, independent steps: how do you know if a whole business operation actually succeeded, when its steps are scattered across many services?
12.1 Distributed Tracing
What it is: A technique where every request is tagged with a unique trace ID that follows it across every service it touches, so you can reconstruct the full journey later.
Why it exists: Without it, if “Order #101” fails somewhere, you’d have to manually search through logs of four different services to find out what happened — extremely slow.
Where it’s used: Tools like OpenTelemetry, Jaeger, and Zipkin are industry standards for this today.
It’s like putting a tracking sticker on a package the moment it leaves the warehouse. As it passes through the delivery truck, the sorting center, and the final courier, everyone scans the same sticker — so you can look up one tracking number and see its entire journey, even though many different people and vehicles were involved.
Diagram 11 — A single trace ID stitches together logs from every service touched by one business operation.
12.2 Saga State Tracking
Since a Saga has no single database transaction to check, teams build a Saga state table or state machine (often inside an “Order” or “Saga Orchestrator” service) recording exactly which step each business operation has reached: STARTED → PAYMENT_DONE → INVENTORY_RESERVED → SHIPPED → COMPLETED, or a failure branch like PAYMENT_DONE → INVENTORY_FAILED → COMPENSATING → CANCELLED. Dashboards then show how many Sagas are stuck in each state, how long they take, and alert if any Saga is stuck too long — a sign something needs manual attention or an automatic retry.
12.3 Key Metrics to Track
Percentage of Sagas reaching a successful final state.
How long, typically and in the worst cases, does a full business operation take end-to-end?
How often do we need to roll back via compensating transactions? A rising trend can signal an upstream problem (e.g., inventory running out often).
How many events failed processing repeatedly and need human review?
Deployment & Cloud
13.1 Independent deployability
One of the biggest promises of microservices is that Team A can deploy the Payment Service ten times a day without ever coordinating with Team B who owns Inventory Service. A 2PC-style architecture directly undermines this: if all services must implement the exact same XA/2PC protocol version, and the central coordinator must know about every participant’s schema and behavior, then deploying changes safely requires much tighter coordination — the opposite of the microservices goal.
13.2 Cloud-native message brokers as the backbone
Modern cloud deployments (AWS, GCP, Azure, or self-hosted Kubernetes clusters) usually rely on managed, durable message brokers — like Apache Kafka, AWS SQS/SNS, Google Pub/Sub, or RabbitMQ — as the backbone connecting Saga steps together. These brokers guarantee that once an event is published, it will be delivered (at least once) even if the receiving service is temporarily down, which is essential for building reliable Sagas without a central blocking coordinator.
Diagram 12 — A cloud-native, event-driven backbone. Services never call each other’s databases directly — they publish and consume durable events through a broker.
13.3 Containers and orchestration
Microservices are typically packaged as containers (Docker) and run under an orchestrator (Kubernetes). Orchestrators routinely restart, reschedule, and scale instances up and down automatically — meaning any given service instance can disappear and reappear at any time. This “constant churn” is completely normal and healthy for microservices but would be dangerous for a 2PC coordinator, since a coordinator instance disappearing mid-transaction is exactly the scenario that causes the blocking problem.
Databases, Caching & Load Balancing
14.1 Database-per-service, revisited
We introduced this pattern in Section 3, but let’s go deeper. Each microservice choosing its own best-fit database technology (called polyglot persistence) is a major benefit of microservices — for example:
| Service | Likely Database Choice | Why |
|---|---|---|
| Order Service | PostgreSQL (relational) | Orders have clear structured relationships (customer, items, totals) needing strong local ACID guarantees |
| Inventory Service | Redis or a fast key-value store, or a relational DB with strict row locking | Needs extremely fast read/write for stock counts under high concurrency |
| Shipping Service | MongoDB (document store) | Shipment data (varying fields per carrier) fits flexible document schemas well |
| Search/Catalog Service | Elasticsearch | Optimized for fast, flexible text search across products |
2PC/XA protocols were designed primarily for relational databases. Many of these modern data stores (MongoDB in most configurations, Redis, Elasticsearch, Cassandra, DynamoDB) either don’t support XA transactions at all, or only support them in limited, non-standard ways. This alone is often the single biggest practical reason 2PC “cannot easily” be used across a modern microservices stack — the technology to do it consistently across all these different stores simply doesn’t broadly exist.
14.2 Caching considerations
Caches (like Redis or an in-memory cache) are commonly placed in front of databases to speed up reads. In an eventually-consistent, Saga-based world, caches must be invalidated or updated carefully as events flow through the system, or users might briefly see stale data (e.g., “In Stock” for a split second after the last item was actually just sold). Common techniques include cache invalidation via the same event stream used for Sagas, and short cache TTLs (time-to-live) for fast-changing data like stock counts.
14.3 Load balancing across service instances
Each microservice typically runs multiple instances behind a load balancer for scalability and fault tolerance. Because each instance is stateless (it doesn’t hold any in-memory transaction locks tied to other services, unlike a 2PC participant which effectively becomes “stateful” while a lock is held), any instance can pick up any request. This statelessness is another reason Sagas fit microservices’ horizontal scaling model much better than 2PC.
APIs & Microservice Communication
15.1 Synchronous vs. Asynchronous communication
| Style | How it works | Trade-off |
|---|---|---|
| Synchronous (REST/gRPC) | Service A calls Service B directly and waits for a response before continuing | Simple to reason about, but couples the availability of A to B’s uptime and speed; risky to use for long Saga chains |
| Asynchronous (Events via Kafka/RabbitMQ) | Service A publishes an event and moves on; Service B consumes it whenever it’s ready | More resilient and scalable, ideal for Sagas, but adds eventual consistency and more operational complexity (message brokers, retries, ordering) |
15.2 API Gateway
What it is: A single entry point that routes external client requests to the correct internal microservice, often handling authentication, rate limiting, and request routing.
Why it exists: Clients (mobile apps, browsers) shouldn’t need to know about dozens of internal service addresses; the gateway simplifies this into one front door.
15.3 Java example: publishing a domain event after a local commit
Java — the naive “commit then publish” pattern (has a subtle risk)
@Service
public class OrderService {
private final OrderRepository orderRepo;
private final EventPublisher eventPublisher; // wraps Kafka/RabbitMQ producer
@Transactional
public Order createOrder(OrderRequest request) {
Order order = new Order(request.getCustomerId(), request.getItems(), "PENDING");
orderRepo.save(order); // local ACID commit, ONLY this service's DB is involved
// After commit, publish an event so other services can react
eventPublisher.publish(new OrderCreatedEvent(order.getId(), order.getItems()));
return order;
}
}
Notice: the database commit and the event publish are two separate actions. This gap is exactly the problem the Outbox Pattern (Section 16) solves, to avoid a case where the database commits but the event fails to publish (or vice versa).
Design Patterns & Anti-patterns
16.1 Saga Pattern — Choreography Style
What it is: Each service listens for events from other services and decides on its own what to do next — there’s no central controller.
Diagram 13 — Choreography: services react to each other’s events independently, with no central brain.
Each service subscribes to a Kafka topic relevant to it. Payment Service subscribes to “order-events” and reacts only to OrderCreated. Inventory Service subscribes to “payment-events” and reacts only to PaymentCharged. No service needs to know the full end-to-end flow — each just knows its own trigger and its own next event to emit.
Pros: simple for a small number of steps, no single point of control/failure.
Cons: as the number of steps grows, it becomes hard to see or reason about the overall flow (“who’s listening to what?” spreads across many codebases) — this is sometimes called the “choreography sprawl” problem.
16.2 Saga Pattern — Orchestration Style
What it is: A dedicated “orchestrator” service (or component) explicitly tells each participant what to do next, step by step, and tracks the overall Saga’s state.
Diagram 14 — Orchestration: one component directs each step and knows the full flow, making it easier to monitor and reason about.
Java example: a simple Saga orchestrator
Java — orchestrator with compensation on inventory failure
@Service
public class OrderSagaOrchestrator {
private final PaymentClient paymentClient;
private final InventoryClient inventoryClient;
private final ShippingClient shippingClient;
private final SagaStateRepository sagaStateRepo;
public void handleOrderCreated(OrderCreatedEvent event) {
SagaState state = new SagaState(event.getOrderId(), "STARTED");
sagaStateRepo.save(state);
try {
paymentClient.chargeCard(event.getOrderId(), event.getAmount());
state.advanceTo("PAYMENT_DONE");
sagaStateRepo.save(state);
inventoryClient.reserveStock(event.getOrderId(), event.getItems());
state.advanceTo("INVENTORY_RESERVED");
sagaStateRepo.save(state);
shippingClient.scheduleShipment(event.getOrderId());
state.advanceTo("COMPLETED");
sagaStateRepo.save(state);
} catch (PaymentFailedException e) {
state.advanceTo("FAILED_AT_PAYMENT");
sagaStateRepo.save(state);
// no compensation needed yet, nothing else succeeded
} catch (InventoryUnavailableException e) {
state.advanceTo("COMPENSATING");
paymentClient.refund(event.getOrderId()); // compensating transaction
state.advanceTo("CANCELLED");
sagaStateRepo.save(state);
}
}
}
Pros: the whole business flow lives in one clear, testable place; easier monitoring, easier to add retries/timeouts centrally.
Cons: the orchestrator itself must be built to be highly available (though, importantly, it does not hold cross-service database locks the way a 2PC coordinator does — it just tracks state and calls services one at a time, so a crash doesn’t freeze other services’ data).
16.3 Outbox Pattern
What it is: A pattern that solves the “dual write problem” — the risk of committing a database change but then failing to publish the matching event (or the reverse), which would leave the system in an inconsistent state.
How it works: Instead of writing to the database and separately publishing to a message broker, the service writes its business data change and a row representing the “event to publish” into the same local database transaction (in an “outbox” table). A separate background process then reads new outbox rows and reliably publishes them to the broker, marking them as sent.
Diagram 15 — The Outbox Pattern. Both the business data and the “event to send” are committed together in one local ACID transaction, guaranteeing they either both happen or neither does — then a separate relay reliably delivers the event.
Java example: Outbox pattern in one local transaction
Java — business row + outbox row committed together
@Service
public class OrderService {
@Transactional
public Order createOrder(OrderRequest request) {
Order order = new Order(request.getCustomerId(), request.getItems(), "PENDING");
orderRepo.save(order);
// Same local transaction, same database, same commit as the order itself
OutboxEvent event = new OutboxEvent(
"OrderCreated",
toJson(new OrderCreatedPayload(order.getId(), order.getItems()))
);
outboxRepo.save(event);
return order; // both rows commit together, or neither does
}
}
This guarantees that we never end up with an order saved but no event sent (which would leave downstream services unaware forever), nor an event sent about an order that was never actually saved.
16.4 Anti-pattern: “Distributed Monolith via Fake 2PC”
A common and costly mistake: teams try to bolt a 2PC-like coordinator onto microservices anyway (sometimes using an XA-compliant message queue or a custom-built coordinator), tightly coupling all services’ transaction logic together. This recreates the worst of both worlds — the operational complexity of microservices (network calls, independent deployments, partial failures) plus the tight coupling and blocking risk of a monolith’s transaction model, without gaining the benefits of either. This is often called a “distributed monolith”.
16.5 Anti-pattern: Shared Database Between Services
Some teams, to “make transactions easy again,” let two services share the same database so they can use a normal local ACID transaction across both. This defeats the entire purpose of microservices — any schema change now risks breaking a completely different team’s service, and the services can no longer be deployed, scaled, or evolved independently. This is a very common early-career mistake and is a major warning sign in system design interviews.
16.6 CQRS (Command Query Responsibility Segregation) — related pattern
What it is: A pattern where the “write” model (commands that change data) and the “read” model (queries that fetch data) are separated, often into different data stores, kept in sync via events.
Why it’s relevant here: In a Saga-based system, a service might need to display a combined view of data owned by multiple other services (e.g., “show me the order with its shipping status”). Instead of querying five services live (slow, fragile), a read-optimized view is built up ahead of time by consuming the same events used for Sagas, and stored locally for fast reads.
Best Practices & Common Mistakes
17.1 Best Practices
- DesignDesign compensating transactions up front — for every “do” step, explicitly design its “undo” step before writing code (e.g., “charge card” ↔ “refund card”).
- IdempotentMake every step idempotent — assume every message might be delivered more than once, and design handlers to safely ignore duplicates.
- OutboxUse the Outbox Pattern for any service that both writes to its own database and publishes events — never do these as two separate, uncoordinated actions.
- TracingInvest early in distributed tracing — retrofitting tracing after a production incident is much harder than building it in from day one.
- Short SagasKeep Sagas short and shallow — the more steps in a Saga, the more compensating logic and the more failure states you must handle; break very long business processes into smaller, well-bounded Sagas where possible.
- VersionedVersion your events — as services evolve independently, event schemas will change; use versioned event formats and backward-compatible changes so older consumers don’t break.
- TimeoutsSet clear timeouts and retry policies with exponential backoff for every network call between services.
17.2 Common Mistakes
- Assuming eventual consistency means “sometimes wrong forever” — it should always converge to a correct final state; if it doesn’t, that’s a bug, not an accepted trade-off.
- Forgetting that some actions are not truly compensable — e.g., you can’t take back an email or SMS already sent; compensations for these usually mean sending a follow-up correction, not a true “undo.”
- Not monitoring “stuck” Sagas — without dashboards and alerts, a Saga that got stuck halfway can silently sit unresolved for days.
- Treating message brokers as guaranteed-once delivery, when most (like Kafka by default) guarantee at-least-once delivery — this is exactly why idempotency matters so much.
- Building an orchestrator that becomes a “God service” doing too much business logic itself, instead of just sequencing calls to the real domain services.
Real-World & Industry Examples
Netflix, one of the earliest large-scale adopters of microservices, famously avoids distributed transactions across services. Instead, they rely heavily on asynchronous, event-driven communication and built internal tools for resilience (like the well-known Hystrix circuit breaker library, and later resilience patterns embedded in their service mesh). Their guiding principle has long been designing each service to tolerate the failure or slowness of others gracefully, rather than trying to coordinate everything into one atomic operation.
Amazon’s order and fulfillment systems are composed of many independent services (ordering, payments, fulfillment, shipping, returns). Publicly discussed patterns from Amazon architects describe using asynchronous workflows and eventual consistency rather than distributed ACID transactions, along with strong use of idempotent APIs so that retries (common at Amazon’s scale) never cause duplicate charges or duplicate shipments.
Uber’s ride-matching, pricing, and payment systems span many services and had to solve exactly this problem as they scaled past a monolithic core. Uber has published engineering blog posts describing their move toward Saga-like workflows and internal workflow orchestration engines (such as their open-sourced Cadence, and its successor concepts) to reliably manage long-running, multi-step business processes like a full trip lifecycle (request → match → ride → payment → receipt) without relying on distributed 2PC transactions.
Interestingly, some core banking ledger systems still use strong, tightly-coupled transactional guarantees for the most critical “money-moving” operations, because regulatory and correctness requirements there prioritize strict consistency very heavily. However, even many modern fintech companies now build their “core ledger” as a strongly consistent internal system, while everything around it (notifications, user-facing dashboards, fraud checks, customer support tools) uses eventual consistency and Saga-like patterns — illustrating that the right approach often depends on which specific piece of the system you’re building.
Frequently Asked Questions
Q1: Is 2PC completely banned or unusable in microservices?
Not literally banned, but almost always avoided for the reasons covered in this tutorial: it doesn’t scale well, doesn’t tolerate common failure modes, and isn’t supported by many modern data stores. It’s occasionally still used within a small, tightly controlled set of services that share the same reliable infrastructure and truly need strict atomicity, but it is not the default or recommended approach for general cross-service consistency.
Q2: If Sagas don’t give strong consistency, isn’t that risky for something like payments?
Sagas give strong consistency within each individual step (each local transaction is fully ACID) and eventual, guaranteed-to-converge consistency across the whole flow, backed by reliable retries, idempotency, and compensations. This is different from “no guarantees” — it’s a different, carefully engineered kind of guarantee, and it’s what nearly every major payment and e-commerce platform actually uses in production today.
Q3: How do you handle a step that truly cannot be undone (like sending an email)?
You design around it: either move non-undoable side effects to the very last step of the Saga (after everything else has succeeded and is unlikely to fail), or treat the “undo” as a corrective follow-up action (e.g., sending a second “please disregard the previous email” message) rather than a true rollback.
Q4: What’s the difference between a Saga and just “wrapping calls in try/catch”?
A simple try/catch only handles failures in the current process’s memory and doesn’t survive a crash, a restart, or a slow/down downstream service. A Saga is explicitly designed to be durable (its state is persisted, not just held in memory) and resumable — if the orchestrator crashes and restarts, it can look up the Saga’s last known state and continue exactly where it left off.
Q5: Do all microservices need Sagas?
No — only operations that genuinely span multiple services’ data. Many operations are naturally confined to just one service’s own database and can use a completely normal, simple local ACID transaction with no Saga needed at all.
Summary & Key Takeaways
Microservices intentionally give each service its own private database to gain independent deployment, independent scaling, and freedom to choose the best-fit data technology per service. This directly breaks the single-database assumption that traditional ACID transactions and protocols like Two-Phase Commit rely on. While 2PC can technically coordinate multiple databases, its central coordinator, cross-service locking, blocking failure mode, and lack of support in many modern data stores make it a poor fit for the loosely-coupled, independently-scaling, failure-tolerant world microservices are built for.
Instead, the industry has converged on patterns like the Saga pattern (choreography or orchestration), the Outbox pattern for reliable event publishing, idempotent operation design, and eventual consistency backed by strong monitoring and distributed tracing — trading strict, instant, all-or-nothing guarantees for resilience, scalability, and true service independence.
Key Takeaways
- 01ACID transactions guarantee “all or nothing” within one database; microservices split data across many databases, breaking that single-database assumption.
- 02Two-Phase Commit (2PC) can coordinate multiple databases, but requires a central coordinator and holds locks across a network round trip — leading to the notorious blocking problem if the coordinator fails.
- 032PC conflicts with core microservices principles: loose coupling, independent scaling, polyglot persistence, and independent failure domains.
- 04Many modern databases (MongoDB, Redis, Cassandra, DynamoDB) and brokers (Kafka) don’t support XA/2PC well or at all.
- 05The CAP theorem shows that during network partitions (common in distributed systems), you must choose between strict Consistency and Availability — most microservices choose Availability.
- 06The Saga pattern replaces one big distributed transaction with a chain of local transactions, each with a compensating “undo” step for failures.
- 07The Outbox pattern solves the “dual write” problem of keeping a database change and its matching event announcement perfectly in sync.
- 08Idempotency and distributed tracing are essential supporting techniques for building reliable, debuggable Saga-based systems.
- 09Real companies like Netflix, Amazon, and Uber all moved away from cross-service distributed transactions toward asynchronous, event-driven, Saga-like workflows as they scaled.
- 10The underlying trade-off to remember: 2PC buys strong consistency at the cost of availability and scalability; Sagas buy availability and scalability at the cost of instant consistency. Microservices architecture almost always needs the latter.
“2PC asks the network to be perfect. Sagas assume it never will be — and design for that reality.”