01 · Introduction: What Is the Outbox Pattern?

Imagine you run a small toy shop. Every time you sell a toy, you do two things: you write the sale down in your ledger, and you shout across the street to your friend at the delivery shop so they know to pick up the toy and take it to the customer. Most days this works fine. But one day, right after you write the sale in your ledger, you get distracted by a customer and forget to shout across the street. The sale is recorded. The toy never gets delivered. The customer waits and waits, and nobody even knows something went wrong.

This tiny, everyday mistake is exactly the problem the outbox pattern was invented to solve inside computer systems — except instead of toys, it is orders, payments, sign-ups, and shipments, and instead of shouting across a street, it is one computer program sending a message to another one over a network.

In plain terms, the outbox pattern is a way of guaranteeing that when a service saves something important to its database and needs to tell other services about it, both things happen together — reliably, every single time — even if the network fails, the message broker is down, or the server crashes at the worst possible moment.

It does this with a surprisingly simple trick: instead of trying to write to the database and send a message to two different systems at the same time (which is risky), the service writes both the business data and a record of the message into the same database, in the same transaction. Because a single database transaction is atomic — meaning it either fully succeeds or fully fails, with no in-between — this guarantees that the sale and the “please tell someone” note are never separated. A second background process then reads that note and delivers the message, retrying patiently until it succeeds.

This pattern is one of the most widely used building blocks in modern microservices architecture — systems built from many small, independent services that talk to each other, usually through message brokers like Apache Kafka, RabbitMQ, Amazon SQS, or Azure Service Bus. Wherever a service needs to update its own data and reliably announce that update to the rest of the system, the outbox pattern tends to show up.

02 · A Short History: Where Did This Pattern Come From?

The idea behind the outbox pattern is not brand new. Long before “microservices” was a popular word, banking systems and telecom billing systems faced the exact same headache: how do you update an account balance and reliably tell a downstream system about it, without ever losing the message or duplicating it?

In the 1990s and early 2000s, the standard answer was a technology called distributed transactions, often implemented through a protocol called XA and a coordination technique called the two-phase commit (2PC). The idea was to make the database write and the message send part of one giant transaction that spanned two completely different systems. It technically worked, but it was slow, fragile, and it required every piece of infrastructure involved — the database, the message queue, the application server — to support the same complicated coordination protocol. Many modern databases and brokers, including most of the tools teams use today such as Kafka, PostgreSQL in its common configuration, and MongoDB, either don’t support XA well or don’t support it at all.

As systems moved toward service-oriented architecture in the mid-2000s and then toward fine-grained microservices in the 2010s, this problem became far more common and far more painful. A single monolithic application talking to one database rarely needed distributed transactions. But once a system was broken into dozens of independent services, each with its own database, “update my data and notify everyone” became a daily occurrence, not a rare edge case.

The name “outbox” itself borrows from the humble idea of a physical office outbox — a tray on your desk where you place letters that are ready to be mailed. You don’t personally walk each letter to the post office the instant you finish writing it. You place it in the outbox, and the mail clerk collects everything in the tray on their normal rounds. If the mail clerk is late one day, your letters are not lost — they are still sitting safely in the tray.

The pattern was popularized in its modern microservices form through the writing of software architect Chris Richardson, particularly in his book Microservices Patterns and on his companion site microservices.io, where it was formally catalogued as the Transactional Outbox pattern alongside related patterns like Saga and CQRS. Around the same time, the open-source Debezium project — a change-data-capture (CDC) tool originally built at Red Hat — added first-class support for reading outbox tables and automatically routing their contents to Kafka topics, which turned the pattern from a manual, hand-rolled trick into something teams could adopt with well-tested, off-the-shelf tooling.

Key milestones

1. 1990s — Distributed Transactions & 2PC. Banking and telecom systems used two-phase commit and XA transactions to keep databases and message queues in sync — powerful, but slow and operationally heavy.
2. Mid-2000s — Service-Oriented Architecture. As systems split into services, the “update data and notify others” problem became more frequent, and lighter-weight patterns started to emerge in enterprise integration circles.
3. 2010s — Microservices Go Mainstream. Chris Richardson catalogues the Transactional Outbox pattern formally as part of the broader microservices patterns language, alongside Saga and CQRS.
4. 2018-2019 — Debezium Adds Outbox Support. The Debezium project ships an “Outbox Event Router” transform, letting teams stream outbox rows straight into Kafka using change data capture, without hand-writing a polling worker.
5. Today — A Default Building Block. The outbox pattern is now considered a standard, almost assumed, piece of infrastructure in any serious event-driven microservices system — as ordinary as using a load balancer.

03 · The Problem & Motivation: The Dual-Write Problem

To understand why the outbox pattern exists, you first need to understand the exact problem it fixes. That problem has a name: the dual-write problem.

Picture a simple online store. A customer places an order. The order service needs to do two separate things:

1. Save the new order into its own database (so the order history shows up correctly).
2. Publish an OrderCreated event to a message broker, so that the inventory service can reduce stock and the shipping service can start preparing a delivery.

The tempting, “obvious” way to write this code looks something like: save the order to the database, then send the message to the broker. That looks completely reasonable. The trouble is that these are two entirely separate systems — a database and a message broker — and there is no way to guarantee that both operations succeed or fail together. Between step one and step two, almost anything can go wrong.

Each of these scenarios is individually rare. But across millions of orders per day, “rare” adds up quickly to a steady trickle of silently lost or duplicated events — the worst kind of bug, because nothing crashes, no error is logged, and the system looks perfectly healthy while quietly losing data.

This is called a dual write because the application is writing to two independent, unrelated systems and treating that as if it were one atomic operation, when in reality there is no shared guarantee tying the two writes together.

04 · Why the “Easy” Fixes Don’t Really Work

When engineers first run into the dual-write problem, they usually try one of a few quick fixes. It helps to understand why each one falls short, because it explains exactly what the outbox pattern is designed to avoid.

None of these workarounds address the root cause: two independent systems, no shared transaction, no way to guarantee both succeed together. The outbox pattern solves this not by making two systems agree, but by removing the need for them to agree at all — it turns a hard distributed-systems problem into an easy local-database problem.

05 · Core Concepts & Terminology

Before going further, let’s build a shared vocabulary. Each of these terms will come up again and again through the rest of this guide.

06 · Architecture & Components

The outbox pattern has surprisingly few moving parts. Once you can name each piece, the whole picture becomes much easier to hold in your head.

Let’s name each piece in the picture above:

07 · Internal Working: How It Actually Happens

Let’s walk through exactly what happens inside the order service, step by step, when a customer places an order.

First, the service opens a database transaction. Inside that single transaction, it performs two inserts: one into the orders table with the new order’s details, and one into the outbox table describing an OrderCreated event, including a JSON payload with the order ID, customer ID, and item list. Both statements are sent to the database as part of the same transaction block.

Here’s a minimal Java example using Spring’s @Transactional annotation, which tells the framework to wrap everything inside the method in a single database transaction:

@Transactional
public void placeOrder(OrderRequest request) {

    // 1. Save the business entity
    Order order = new Order(request);
    orderRepository.save(order);

    // 2. Build the event payload
    String payload = objectMapper.writeValueAsString(
        new OrderCreatedEvent(order.getId(), order.getItems())
    );

    // 3. Save the outbox row - same transaction, same commit
    OutboxMessage message = new OutboxMessage(
        "Order",             // aggregateType
        order.getId(),       // aggregateId
        "OrderCreated",      // eventType
        payload
    );
    outboxRepository.save(message);

    // Both rows commit together, or neither does.
}

Notice what is not in this method: there is no call to Kafka, no call to RabbitMQ, no network call to any other service at all. The method’s entire job is to write two rows to one database and let the database’s transaction guarantee do the heavy lifting. If the transaction commits, both rows exist. If anything fails — a validation error, a constraint violation, a crash — the entire transaction rolls back and neither row exists. There is no possible in-between state.

A separate, independent process — the message relay — is responsible for actually getting that outbox row to the broker. It might poll the table every few hundred milliseconds looking for unprocessed rows, or it might use change data capture to react the instant a new row is written to the database’s transaction log. Either way, this relay process runs continuously and retries automatically if the broker is temporarily unavailable, because unlike the original request, it isn’t blocking a customer waiting for a response — it can take its time and be patient.

It helps to notice what this design deliberately gives up in exchange for that safety. The order service no longer knows, at the moment it responds to the customer, whether the OrderCreated event has actually reached Kafka, RabbitMQ, or any downstream system. It only knows that the event is durably recorded and guaranteed to be delivered eventually. For most businesses this trade is an easy one to accept — a few hundred milliseconds of extra delivery latency is a small price for never silently losing an order confirmation. But it does mean the pattern is a poor fit for situations where a caller genuinely needs to know, synchronously, that a downstream system has already reacted before the response returns — for that narrow class of problem, a direct synchronous API call with its own retry and timeout handling is usually the more honest tool.

08 · Data Flow & Lifecycle

Every outbox event moves through the same predictable lifecycle, from the moment it’s created to the moment it’s safely consumed elsewhere.

1. Request arrives. A customer action (placing an order, updating a profile, cancelling a subscription) triggers a service method.
2. Transaction begins. The service opens a database transaction covering both the business write and the outbox write.
3. Two inserts, one commit. The business row and the outbox row are inserted. The transaction commits — or, if anything fails, both are rolled back together.
4. Response returns to the caller. The original HTTP request or API call returns success immediately after the commit — it does not wait for the message to actually reach Kafka or RabbitMQ.
5. Relay picks up the row. The polling worker or CDC connector detects the new outbox row, moments after the commit.
6. Message published to the broker. The relay sends the event to the appropriate topic or queue, typically keyed by aggregate ID for ordering.
7. Row marked processed (or deleted). Once the broker confirms receipt, the outbox row is deleted or flagged as sent, so it isn’t re-sent by a future poll.
8. Downstream consumers react. Other services consume the event from the broker and update their own local state — idempotently, in case the same event arrives more than once.

Notice step 4: the caller gets a response the instant the local database transaction commits. This is one of the most valuable side effects of the pattern — the customer-facing request stays fast, because it never waits on the message broker, which might be slow, overloaded, or briefly unreachable. Delivery is decoupled from the request/response cycle entirely.

09 · Two Ways to Build the Relay: Polling vs. CDC

There are two common strategies for the message relay, and understanding the difference matters when deciding how to implement the pattern in your own system.

Polling is easy to reason about and gets a team most of the reliability benefit with almost no new infrastructure — but it adds a small delay (the polling interval) and adds repeated read load to the outbox table as it grows. CDC removes both of those downsides by reading the transaction log instead of the table itself, but it requires you to run and monitor a connector, and it can be a heavier operational lift for a small team.

Debezium in particular ships a built-in Outbox Event Router, a Kafka Connect transform purpose-built for this exact table shape. It reads new outbox rows, extracts the aggregate_type to decide which Kafka topic the event belongs to, and uses aggregate_id as the Kafka message key — which, combined with Kafka’s per-partition ordering guarantee, ensures all events about the same order (or user, or account) arrive at consumers in the order they were created.

10 · Advantages, Disadvantages & Trade-offs

No pattern is free. The outbox pattern trades a small amount of added complexity for a large amount of reliability — and it’s worth being honest about both sides.

The single most important trade-off to internalize is the shift from “exactly-once” thinking to “at-least-once” thinking. The outbox pattern cannot promise that a downstream consumer will see an event exactly one time — a relay retry, a broker rebalance, or a consumer restart can all cause the same event to be delivered twice. What it can promise is that the event will never be lost. Turning “at least once” into something that behaves like “exactly once” from the consumer’s point of view is the job of idempotent consumer design, which we’ll cover in the best practices section.

11 · Performance & Scalability

Because the outbox pattern adds one extra row insert to an existing transaction, its direct performance cost is small — inserting a row is fast, and it happens inside a transaction the service was already opening for the business write. The real performance conversation is about the outbox table itself and the relay process, not the write path.

As traffic grows, a few specific pressure points show up:

On the consumer side, scalability usually comes from the message broker itself rather than the outbox pattern. Kafka, for example, scales horizontally by adding partitions and consumer instances within a consumer group, letting event processing scale independently of how many rows are being written to any single service’s outbox table. Because the aggregate ID is typically used as the partition key, load spreads evenly across partitions as long as aggregate IDs are well distributed — a skewed key (like always using a fixed “GLOBAL” aggregate) would concentrate all events onto a single partition and become a bottleneck.

12 · High Availability, Reliability & Failure Recovery

The entire point of the outbox pattern is reliability, so it’s worth walking through exactly how it survives the failure scenarios that broke the naive approach earlier in this guide.

This is why the pattern is often described as trading a hard distributed-systems problem (keeping two independent systems consistent) for an easy local problem (keeping one database consistent with itself) plus a background retry loop. Databases are extremely good at the first job. Retry loops are a well-understood, easy-to-test piece of software for the second.

For high availability, teams typically run more than one instance of the relay process for redundancy, using a distributed lock (or the natural partitioning of a Kafka Connect cluster, when using Debezium) to make sure only one instance is actively processing a given partition of the outbox table at a time — this avoids duplicate publishing from multiple relays racing each other, while still surviving the loss of any single relay instance.

13 · Security Considerations

The outbox pattern doesn’t introduce a fundamentally new security surface, but it does touch a few areas worth treating carefully, since event payloads often travel far beyond the service that created them.

14 · Monitoring, Logging & Metrics

An outbox implementation that nobody is watching is only reliable until the day it silently isn’t. A handful of metrics catch almost every real-world failure mode early.

Good logging practice includes tagging every log line related to an event with its outbox row ID and aggregate ID, so a single event’s journey — from creation, to relay pickup, to broker publish, to consumer processing — can be traced end-to-end across services. Many teams pair this with distributed tracing tools (like OpenTelemetry) that propagate a trace ID through the event payload itself, so a single trace shows the complete path an order took through five or six different services.

15 · Deployment & Cloud Options

Every major cloud provider offers building blocks that fit naturally into an outbox implementation, so teams rarely need to build the infrastructure from scratch.

In a typical Kubernetes deployment, the relay runs as its own lightweight deployment or, when using Debezium, as a Kafka Connect worker pod, entirely separate from the business service’s own pods. This separation matters: the relay can be scaled, restarted, or redeployed independently of the service that writes to the outbox table, and a bug in relay logic can never take down the customer-facing service, since the two only interact through the database.

For teams on managed cloud data platforms, some databases now offer built-in “outbox-friendly” features — for instance, PostgreSQL’s logical replication (the foundation CDC tools rely on) is available out of the box on most managed Postgres offerings, meaning teams can adopt CDC-based relays without operating their own database infrastructure at all.

16 · Databases, Message Brokers & Related Infrastructure

The outbox pattern is deliberately unopinionated about which database and broker you use — its only real requirement is that the database supports transactions, which nearly every relational database does.

Here’s a representative outbox table schema, the same shape used across most implementations regardless of the underlying database engine:

CREATE TABLE outbox_events (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type  VARCHAR(255) NOT NULL,
    aggregate_id    VARCHAR(255) NOT NULL,
    event_type      VARCHAR(255) NOT NULL,
    payload         JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at    TIMESTAMPTZ
);

-- Index that keeps polling queries fast
CREATE INDEX idx_outbox_unprocessed
    ON outbox_events (created_at)
    WHERE processed_at IS NULL;

Beyond the database and broker themselves, caching and load balancing sit around, not inside, the pattern. A load balancer distributes incoming API requests across multiple instances of the business service, none of which need any special coordination with each other — each instance can independently write to the shared database, since the database’s transactional guarantee is what keeps everything consistent, not the application instance handling the request. Caching, similarly, applies to how consumers serve reads after processing an event (for example, updating a Redis cache once an OrderCreated event is consumed) rather than to the outbox mechanism itself.

One nuance worth knowing: NoSQL and document databases can also support the pattern, as long as they offer transactional writes across two documents or collections in the same database. MongoDB, for instance, has supported multi-document ACID transactions since version 4.0, making an outbox collection alongside a business collection perfectly workable, with tools like MongoDB’s own Kafka connector able to tail its change streams in a CDC-like fashion.

17 · APIs, Microservices & Where the Pattern Fits

In a microservices architecture, each service is meant to own its own data exclusively — no other service is allowed to reach into its database directly. The only sanctioned ways for services to interact are through well-defined APIs (synchronous, like REST or gRPC) or through events (asynchronous, via a message broker). The outbox pattern is specifically about making that second path — the event path — reliable.

This matters because synchronous APIs and asynchronous events solve different problems. A REST call is good when you need an immediate answer: “Is this item in stock right now?” An event is good when you need to announce that something happened and let anyone interested react in their own time: “This order was just placed.” The outbox pattern only concerns itself with the second category — it has nothing to do with synchronous request/response calls, which have their own, separate reliability techniques like retries, circuit breakers, and timeouts.

It’s worth being precise about scope: the outbox pattern guarantees that an event leaves the originating service reliably. It says nothing about what happens after that — whether the broker delivers it, whether the consumer processes it correctly, or whether a downstream failure needs to be compensated for. Those concerns belong to the broker’s own delivery guarantees and to patterns like the Saga pattern, which the outbox pattern is frequently paired with, not a replacement for.

18 · Related Patterns & Anti-Patterns

The outbox pattern rarely stands alone. Understanding its neighbors — and a few common ways teams misuse it — rounds out the full picture.

Companion patterns

Anti-patterns to avoid

19 · Best Practices & Common Mistakes

Teams who’ve run this pattern in production for years tend to converge on the same handful of habits. Here’s the short list worth internalizing.

A short but important note on schema evolution: because events published through the outbox often outlive the code that produced them — old messages might still be sitting in a Kafka topic days later — additive, backward-compatible changes (adding a new optional field) are safe, while removing or renaming fields consumers already depend on is not. Many teams enforce this with a schema registry that rejects incompatible changes before they ever reach production.

20 · Real-World & Industry Examples

The outbox pattern isn’t a theoretical exercise — it’s quietly running behind some of the largest software systems in the world.

The common thread across every one of these examples is scale and consequence: once a company operates enough independent services, and once losing an event has a real cost — a missing delivery, a stuck payment, an inventory mismatch — the outbox pattern stops being an optional refinement and becomes basic infrastructure, in the same category as a load balancer or a health check.

It’s also worth noticing what these companies have in common beyond their size: none of them treat the outbox pattern as a one-time project that gets finished and forgotten. It becomes part of the standard toolkit that new services are built with from day one, the same way a team might automatically add health-check endpoints or structured logging to every new service without debating it each time. Platform teams at these companies often build shared libraries that give any new service an outbox table, a relay, and monitoring dashboards “for free,” simply by including the library — a strong signal that the pattern has moved from a clever trick into foundational plumbing, much like TCP retransmission or DNS caching: invisible when it works, and only noticed by name when someone goes looking for how reliability is actually achieved under the hood.

21 · Frequently Asked Questions

22 · Summary & Key Takeaways

The outbox pattern takes a genuinely hard distributed-systems problem — keeping a database and a message broker consistent — and turns it into an easy local-database problem plus a patient retry loop. It does this without distributed transactions, without exotic infrastructure, and without asking any two independent systems to magically agree with each other.

Remember this

• The dual-write problem happens because a database write and a message publish are two separate operations with no shared guarantee tying them together.
• The outbox pattern fixes this by writing the business data and the event into the same database, in the same transaction — they succeed or fail together, always.
• A separate relay (polling, or CDC via tools like Debezium) reads the outbox table and delivers events to the broker, retrying patiently until it succeeds.
• The delivery guarantee is at-least-once, not exactly-once — so downstream consumers must be built to safely handle duplicate events.
• Polling relays are simple and get most teams most of the benefit; CDC-based relays add real-time delivery and remove database query load, at the cost of extra infrastructure.
• The pattern pairs naturally with idempotent consumers, the Saga pattern, and CQRS read models, and is standard infrastructure at companies like Uber and Shopify.

If there’s one idea worth carrying away from this entire guide, it’s this: reliability in distributed systems rarely comes from clever, heroic code trying to force two unrelated systems to agree in the moment. It comes from patiently borrowing a guarantee one system already provides — the database’s transaction — and building everything else around retrying until success, since retrying is a problem software is very good at solving.