Service Choreography

Service Choreography

What is Service Choreography ?

How independent services learn to move together — without anyone standing in the middle telling them what to do next.

01

Introduction & History

Two very different ways to make many pieces of software cooperate — and why one of them, borrowed from dance, keeps winning at internet scale.

Picture a school play. There are two ways to run it backstage.

Way one: a director stands in the wings holding the script. She points at each actor and says “you, walk in now,” “you, say your line now,” “you, cut the lights now.” Every actor waits for her signal. Nobody moves unless she tells them to.

Way two: there is no director backstage at all. Every actor has memorised the whole play. When actor A finishes a line, actor B knows — just from hearing that line — that it’s their turn. Actor C sees actor B walk on stage and knows that’s their cue to start playing music. Nobody is in charge. Everyone reacts to what just happened around them.

The first approach is called orchestration. The second is called service choreography. This tutorial is about the second one — how independent pieces of software work together, in sync, without a boss issuing instructions at every step.

1.1 Where did this idea come from?

Service choreography is not brand new — the vocabulary comes straight from real-world dance and theatre, applied to software design starting in the early-to-mid 2000s, when companies began breaking large “monolithic” applications (one giant program that does everything) into smaller, independent programs called services.

The term became formalised around the same era as Service-Oriented Architecture (SOA), in specifications like WS-CDL (Web Services Choreography Description Language) in the mid-2000s. It later became far more practical and popular with the rise of microservices (roughly 2011–2015 onward), when companies like Netflix, Amazon, and Uber needed hundreds — sometimes thousands — of small services to cooperate without creating a single fragile “brain” that everything depended on.

Analogy. Think of an orchestra with a conductor (orchestration) versus a flock of birds flying in formation (choreography). The flock has no leader bird issuing orders. Each bird simply watches its neighbours and reacts. Yet the whole flock moves beautifully together. That is choreography.

1.2 Why does this matter today?

Modern applications — think of a food delivery app, an online store, or a ride-sharing app — are built from dozens or hundreds of small services: an Order Service, a Payment Service, an Inventory Service, a Notification Service, a Delivery Service, and so on. These services need to cooperate to complete one business goal (like “deliver this pizza”), but keeping them cooperating without a fragile central controller is a real engineering challenge. Service choreography is one of the two major answers to that challenge.

Pro tip. When you hear an engineer say “we’re event-driven,” ask specifically whether their multi-step business processes are coordinated implicitly through events (choreography) or explicitly through a workflow engine (orchestration). The answer often reveals more about their team culture than about their technology stack.
02

Problem & Motivation

Let’s build the problem up slowly, the way you’d explain it to somebody who has never written a line of software.

2.1 Step 1 — one giant program (the monolith)

In the beginning, most applications were built as one big program — a monolith. All the logic for orders, payments, inventory, and shipping lived inside the same codebase, running as a single unit.

Analogy. A monolith is like one person doing every job in a restaurant alone — cooking, serving, cleaning, and running the till. It works fine when the restaurant is small. The moment it gets busy, that one person becomes the bottleneck, and if they get sick the whole restaurant closes.

2.2 Step 2 — breaking it apart (microservices)

To fix this, engineers split the monolith into small, independent services — one team owns Orders, another owns Payments, another owns Inventory. Each service has its own codebase, its own database, and can be deployed independently. This is called a microservices architecture.

But now a new problem appears: how do these separate services talk to each other to finish one task?

Concrete example. When a customer places an order on an e-commerce site, at least four services must cooperate:
  1. Order Service — record the order.
  2. Payment Service — charge the customer.
  3. Inventory Service — reduce stock.
  4. Notification Service — send a confirmation email.

Somebody or something has to make sure all four steps happen, in the right order, even if one of them fails halfway.

2.3 Step 3 — two answers: orchestration and choreography

There are exactly two broad strategies to solve this “who talks to whom, and in what order” problem:

  • Orchestration — a central controller (an “orchestrator”) calls each service directly, one after another, like a recipe being followed step by step.
  • Choreography — there is no central controller. Each service announces “something happened” (an event), and other services listen for events they care about and react on their own.

This tutorial focuses on choreography — but we will compare it to orchestration constantly, because understanding one deeply requires understanding the other.

Watch out. A common beginner mistake is thinking one approach is always “better.” They solve the same problem with different trade-offs — much like walking versus driving. Walking is simpler and needs no fuel; driving is faster over long distances. Neither is universally superior.
03

Core Concepts

Before going further, let’s define every important word — clearly, one at a time.

service event broker pub / sub autonomy decoupling saga idempotency event sourcing eventual consistency command vs event topic vs queue choreography vs pub/sub

3.1 Service

What it is. A small, independent piece of software that does one job well (for example, handling payments) and exposes that capability to others.

Why it exists. So teams can build, test, and deploy pieces of a big application separately, without stepping on each other’s toes.

Simple analogy. A specialist worker in a factory — one person only tightens bolts, another only paints, another only checks quality.

A “Payment Service” only knows how to charge cards and refund money. It knows nothing about shipping addresses.

3.2 Event

What it is. A fact that “something happened” — stated in the past tense. For example: OrderPlaced, PaymentCompleted, StockReserved.

Why it exists. Events let a service announce facts without needing to know who is listening, or caring what they will do about it.

Simple analogy. A school bell ringing. The bell does not command anyone. It just announces “class is over.” Students, teachers, and cafeteria staff each react in their own way — nobody is told individually.

When a customer’s payment succeeds, the Payment Service publishes a PaymentCompleted event. It does not know or care which services will react to it.

3.3 Event Bus / Message Broker

What it is. A piece of infrastructure that carries events from the service that published them to every service that wants to receive them. Popular real tools: Apache Kafka, RabbitMQ, AWS EventBridge, Google Pub/Sub, Azure Service Bus.

Why it exists. Without it, every service would need to know the network address of every other service — a tangled mess as the system grows.

Simple analogy. A postal system. You drop a letter in a mailbox (publish); you don’t personally deliver it to the reader’s door. The postal system (broker) figures out delivery.

3.4 Publish / Subscribe (Pub/Sub)

What it is. A messaging pattern where publishers send events to named “topics” or “channels,” and subscribers listen to the topics they care about.

Why it exists. It decouples the sender from the receiver — the sender never needs a list of receivers.

Simple analogy. A YouTube channel. The creator (publisher) uploads a video to their channel. They don’t email every subscriber directly — YouTube (the broker) notifies whoever subscribed.

3.5 Autonomy

What it is. Each service decides, on its own, how to react to an event — no external party forces its behaviour.

Why it exists. Autonomy is what allows teams to change their service’s internal logic without asking permission from other teams.

Simple analogy. When the school bell rings, the cafeteria staff decides on their own how to prepare lunch. The bell does not dictate the recipe.

3.6 Decoupling

What it is. Reducing the direct dependencies between two pieces of software, so a change in one does not force a change in the other.

Why it exists. Tightly coupled systems become fragile — one broken piece can crash everything connected to it.

Simple analogy. Cars on a highway are not welded together. Each drives independently, following shared rules (traffic lights, lanes) rather than being physically tied to the car in front.

3.7 Saga Pattern

What it is. A way to manage a business transaction that spans multiple services, using a sequence of local transactions, each triggered by the completion of the previous one, with defined “compensating” actions if something fails.

Why it exists. In a single database, you can use a fast, all-or-nothing (ACID) transaction. Across separate services with separate databases, you cannot — so Sagas simulate the same all-or-nothing feeling using events and rollback logic.

Simple analogy. Planning a trip with flights, hotels, and a rental car booked separately. If the hotel booking fails after you have already booked the flight, you cancel the flight (a compensating action) rather than being stuck.

3.8 Idempotency

What it is. A property where performing the same operation multiple times has the same effect as doing it once.

Why it exists. In distributed systems, messages can be delivered more than once (duplicate delivery). If “charge $50” isn’t idempotent, a duplicate event could charge the customer twice.

Simple analogy. Pressing an elevator button five times does not call the elevator five times. One request is enough — pressing again changes nothing.

3.9 Event Sourcing

What it is. Storing every change to data as a sequence of events, instead of just storing the current state. The current state is rebuilt by replaying events.

Why it exists. It gives a complete history/audit trail and lets you rebuild past states, which is very valuable in choreography-heavy systems that already think in “events.”

Simple analogy. A bank statement. Instead of only showing your current balance, it lists every deposit and withdrawal — you can always recompute the balance from the full history.

3.10 Eventual Consistency

What it is. A guarantee that, given enough time (usually milliseconds to seconds) and no new updates, all parts of a distributed system will reflect the same, correct data — even though at any single instant they might briefly disagree.

Why it exists. Choreographed systems update different services at different, independent moments. There is a small window of “some services know, some don’t yet” before everything settles.

Simple analogy. Gossip in a small town. When something happens, not everyone learns about it in the same second — but within a day, everyone knows. The information “eventually” spreads to everyone.

3.11 Command vs Event

What it is. A command is a request telling something to happen in the future (“ChargeCustomer”). An event is a statement that something has already happened in the past (“CustomerCharged”). Choreography is built almost entirely on events, not commands, because commands imply the sender knows exactly who should act — which reintroduces the coupling choreography tries to avoid.

Why it exists. Naming events in the past tense keeps every service free to interpret the fact however it wants, instead of being told exactly what to do.

A traffic light turning red is an event (“LightTurnedRed”), not a command. It does not order any specific car to stop — each driver independently decides to stop because of the shared rule they already know.

3.12 Topic vs Queue

What it is. A topic (used in publish/subscribe) delivers a copy of each event to every subscriber. A queue delivers each individual message to only one consumer, even if many consumers are listening — used for spreading out work rather than broadcasting facts.

Why it exists. Choreography typically needs topics, because many different services (Payment, Inventory, Analytics) all need to know the same fact independently. Queues are more useful for distributing a large pile of identical jobs among worker instances of the same service.

Analogy. A topic is like a radio broadcast — everyone with a radio tuned to that station hears the same announcement. A queue is like a single ticket dispenser at a deli counter — each ticket (task) goes to exactly one customer (worker), never duplicated.

3.13 Choreography vs Simple Pub/Sub

It is worth clarifying a subtle point: not every pub/sub system is “choreography.” Choreography specifically refers to using events to coordinate a multi-step business process across several services, where the completion of one step is what triggers the next. Simple pub/sub notifications (like “send an analytics event whenever anyone clicks a button”) are event-driven, but they aren’t really choreographing a business transaction unless later steps depend on earlier ones completing successfully.

04

Orchestration vs Choreography

This is the single most important comparison to internalise — and one of the most common interview and design-review topics in the industry.

Orchestration vs Choreography CENTRAL CONTROL Orchestrator Order Payment Inventory Notification one component decides the flow PEER TO PEER VIA EVENTS Event Bus Kafka · Rabbit Order Payment Inventory Notification OrderPlaced each service decides its own reaction
Fig 1 · Left: an orchestrator directly calls every service in order. Right: services only talk to the event bus; they never call each other directly.

4.1 How to read this diagram

On the left, the Orchestrator is like the director in the play analogy — it knows the entire recipe and calls each service directly, waiting for a response before calling the next. On the right, there is no single box that “knows everything.” The Order Service simply shouts “OrderPlaced!” into the Event Bus and moves on. The Payment Service happens to be listening for that shout, reacts, and then shouts its own event. This chain continues — like a row of dominoes, each one falling because the previous one fell, not because someone pushed each domino individually.

4.2 Side-by-side comparison

AspectOrchestrationChoreography
ControlCentralised (one component decides the flow)Distributed (each service decides its own reaction)
CouplingServices coupled to the orchestratorServices coupled only to event formats/topics
Visibility of overall flowEasy — read the orchestrator’s code/workflowHarder — flow is scattered across many services
Failure handlingOrchestrator manages retries/rollback centrallyEach service manages its own compensations
Adding a new stepMust modify the orchestratorJust add a new listener — no existing service changes
Single point of failure riskHigher (orchestrator down ⇒ flow stuck)Lower (no central brain, but debugging is harder)
Best forComplex, long-running, tightly governed business workflowsLoosely coupled, highly scalable, reactive systems
Common toolsCamunda, AWS Step Functions, Temporal, Netflix ConductorKafka, RabbitMQ, EventBridge, Pub/Sub

Analogy. Orchestration is a recipe card in the hands of a head chef, telling each cook exactly when to chop, fry, and plate. Choreography is a busy professional kitchen where each cook simply watches the plate in front of them — when they see chopped vegetables arrive, they know it’s their turn to fry, without the head chef saying a word.

Watch out. A very common interview trap: people say “choreography has no coordination.” That’s wrong — coordination still exists, it’s just implicit, spread across event contracts, instead of explicit in one orchestrator’s code.
05

Architecture & Components

Let’s zoom into what a real choreography-based system is built from.

Choreography Architecture — Three Topics, Many Subscribers Order Service producer Topic: orders event bus publish Payment Service Inventory Service Analytics Service Topic: payments event bus PaymentCompleted Notification Topic: inventory event bus StockReserved Shipping Service Every service publishes to its own topic. Any number of others can subscribe — no direct API calls between services.
Fig 2 · Each service publishes to its own topic. Any number of other services can subscribe. No service ever calls another service’s API directly for this flow.

5.1 The main building blocks

  • Producer service — publishes events when something happens inside it (for example, the Order Service publishing OrderPlaced).
  • Consumer service — subscribes to one or more event types and reacts (for example, the Payment Service consuming OrderPlaced).
  • Event broker / message bus — the “postal system” (Kafka, RabbitMQ, and so on) that reliably delivers events.
  • Event schema / contract — an agreed-upon structure for each event (fields, types), so producers and consumers understand each other. Often defined with tools like Avro, Protobuf, or JSON Schema.
  • Dead-letter queue (DLQ) — a special holding area for events that failed processing repeatedly, so they don’t get lost or block the main flow.
  • Schema registry — a central catalog that stores and validates event schema versions, preventing producers and consumers from silently breaking each other.

A real event contract for OrderPlaced might look like this in JSON:

{
  "eventType": "OrderPlaced",
  "eventId": "b23f-...-91a2",
  "occurredAt": "2026-07-16T10:15:00Z",
  "orderId": "ORD-88213",
  "customerId": "CUST-4471",
  "items": [
    { "sku": "PIZZA-001", "qty": 2 }
  ],
  "totalAmount": 24.50
}

Every consumer that cares about orders reads this exact shape. If the Order Service changes a field name without warning, every consumer could break — which is why schema contracts matter so much.

06

Internal Working

Let’s trace, step by step, what actually happens inside the machines when an event is published and consumed.

Internal Working — Publish and Deliver Order Service Event Broker Payment Service Inventory Service publish(OrderPlaced, orderId=88213) ack (event stored durably) deliver(OrderPlaced) charge customer card publish(PaymentCompleted) deliver(PaymentCompleted) reserve stock publish(StockReserved) Arrows into the broker are publishes; arrows out are deliveries. No service calls another service’s endpoint directly.
Fig 3 · Every arrow into the broker is a “publish,” and every arrow out of the broker is a “deliver.” No service ever calls another service’s endpoint directly.

6.1 Step-by-step explanation

  1. Save locally

    The Order Service finishes creating an order in its own database.

  2. Publish the event durably

    It publishes an OrderPlaced event to the broker. The broker durably stores this event (writes it to disk, often replicated across machines) before confirming — critical, otherwise events could be lost if the broker crashes.

  3. Deliver to subscribers

    The broker then pushes (or the consumer pulls) this event to every subscriber of the orders topic — here, the Payment Service.

  4. Process independently

    The Payment Service processes the event: it charges the card using its own internal logic, entirely unaware of what Inventory or Notifications will do.

  5. Publish the next fact

    Once done, Payment Service publishes its own event, PaymentCompleted.

  6. Chain continues

    The Inventory Service, subscribed to payment events, receives it and reserves stock — again, without knowing who’s listening downstream.

Analogy. This is exactly like a relay race. Each runner (service) only cares about receiving the baton (event) and running their own leg well. They don’t need a race director yelling instructions mid-race — the act of receiving the baton is itself the instruction to start running.

6.2 Java example: a simple event producer

Below is a simplified, illustrative example (not tied to one specific broker’s exact API) showing how a Java service might publish an event after saving an order.

OrderService.java
public class OrderService {

    private final EventPublisher eventPublisher;
    private final OrderRepository orderRepository;

    public OrderService(EventPublisher eventPublisher, OrderRepository orderRepository) {
        this.eventPublisher = eventPublisher;
        this.orderRepository = orderRepository;
    }

    public void placeOrder(OrderRequest request) {
        // 1. Save order locally — this is our own service's transaction
        Order order = orderRepository.save(new Order(request));

        // 2. Build the event payload — the "fact" we are announcing
        OrderPlacedEvent event = new OrderPlacedEvent(
            order.getId(), order.getCustomerId(), order.getItems(), order.getTotal()
        );

        // 3. Publish to the broker — we do NOT call PaymentService directly
        eventPublisher.publish("orders.OrderPlaced", event);
    }
}

6.3 Java example: a simple event consumer

PaymentEventListener.java
public class PaymentEventListener {

    private final PaymentGateway paymentGateway;
    private final EventPublisher eventPublisher;
    private final ProcessedEventStore processedEventStore; // for idempotency

    @EventListener(topic = "orders.OrderPlaced")
    public void onOrderPlaced(OrderPlacedEvent event) {

        // Guard against duplicate delivery (idempotency check)
        if (processedEventStore.alreadyHandled(event.getEventId())) {
            return; // already processed, do nothing
        }

        boolean success = paymentGateway.charge(event.getCustomerId(), event.getTotal());

        if (success) {
            eventPublisher.publish("payments.PaymentCompleted",
                new PaymentCompletedEvent(event.getOrderId()));
        } else {
            eventPublisher.publish("payments.PaymentFailed",
                new PaymentFailedEvent(event.getOrderId(), "card_declined"));
        }

        processedEventStore.markHandled(event.getEventId());
    }
}

Notice two important defensive habits in the consumer code: checking whether the event was already handled (idempotency), and publishing a failure event rather than silently swallowing errors. Both are essential in real choreography systems, because network hiccups and broker retries can deliver the same event more than once.

07

Data Flow & Lifecycle (Saga Example)

Now let’s follow a complete business transaction from start to finish — including what happens when something fails halfway through.

This uses the Choreography-based Saga pattern, one of the most asked-about patterns in system design interviews.

Choreography-Based Saga — Failure and Compensation Order Service Payment Service Inventory Service Shipping Service OrderPlaced PaymentCompleted StockReserved stock actually runs out here ShippingFailed StockReservationFailed · compensate PaymentRefunded · compensate Order → CANCELLED Compensating events flow backwards through the same peer-to-peer chain — no orchestrator required to unwind.
Fig 4 · When a later step fails, each earlier service listens for a “failure” event and runs its own compensating action — undoing its part, one step backward, like a chain reaction in reverse.

7.1 Why compensation instead of “rollback”?

In a single database, if something fails, you just run ROLLBACK and everything reverts instantly. But here, the Payment Service already charged the customer’s card in a separate system — you cannot “un-charge” it with a database command. Instead, you must explicitly issue a refund — a compensating action that undoes the business effect, not a magic technical undo.

Analogy. Imagine booking a multi-city trip: flight, hotel, and car. If the car rental fails after you’ve paid for the flight and hotel, you don’t get a magic “undo” button — you actively cancel the flight and hotel, which might involve cancellation fees or refund requests. That active, deliberate undoing is a compensating transaction.

7.2 Lifecycle stages of an event

  1. Creation

    A service decides a fact is worth announcing.

  2. Serialization

    The event object is converted into bytes (JSON, Avro, Protobuf) for transport.

  3. Publish

    Sent to the broker, which durably persists it.

  4. Storage/Retention

    The broker keeps the event for a configured period (Kafka, for example, can retain events for days or indefinitely).

  5. Delivery

    Pushed or pulled to each subscribed consumer.

  6. Processing

    The consumer executes its own local logic.

  7. Acknowledgement

    The consumer tells the broker “I’ve handled this,” so it isn’t redelivered unnecessarily.

  8. Dead-lettering (if needed)

    If processing keeps failing, the event is moved to a DLQ for manual or automated investigation.

08

Design Patterns

A small vocabulary of reusable shapes shows up over and over in real choreography systems. Learn these once, spot them everywhere.

8.1 Choreography-based Saga

Already explained above — a chain of events and compensations with no central coordinator.

8.2 Event Sourcing

Storing state as an ordered log of events rather than as a single row of “current values.” Very natural fit for choreography since the system already thinks in events.

8.3 CQRS (Command Query Responsibility Segregation)

What it is. Splitting the model used to write data (commands) from the model used to read data (queries), often paired with event sourcing and choreography — write-side services emit events, and separate read-side services build fast, denormalised “views” by consuming those events.

Analogy. A restaurant kitchen (writes: cooking orders) is organised totally differently from the dining floor menu display (reads: showing what’s available). They’re optimised for different jobs even though they describe the same food.

8.4 Event-Carried State Transfer

What it is. Events that carry enough data so the receiving service doesn’t need to call back to ask for more details.

Instead of an OrderPlaced event containing only an ID (forcing Payment Service to call back “give me order details”), it includes the customer ID, amount, and items directly — reducing chatty network calls.

8.5 Competing Consumers

What it is. Multiple instances of the same service subscribe to the same topic/queue, but each event is only processed by one of them — spreading load across instances.

Analogy. Several cashiers (consumer instances) share one line of customers (a queue). Each customer (event) is served by exactly one cashier, and cashiers work in parallel to go faster.

8.6 Outbox Pattern

What it is. A technique to guarantee that a database write and an event publish happen together reliably. The service writes the event into an “outbox” table in the same local database transaction as its business data, and a separate background process reads that table and publishes the events to the broker.

Why it exists. Without it, you risk saving data but crashing before publishing the event (or vice versa) — a classic “dual write” problem.

Outbox — two writes, one local transaction
public void placeOrder(OrderRequest request) {
    // Both writes happen in ONE local database transaction
    orderRepository.save(order);
    outboxRepository.save(new OutboxMessage("OrderPlaced", order.toEventPayload()));
    // A separate poller reads outbox rows and publishes them to Kafka reliably
}
09

Anti-Patterns

The failure modes are surprisingly consistent across companies. Here are the ones you will meet.

The “Hidden Orchestrator” anti-pattern. Sometimes one service quietly ends up reacting to almost every event and making most of the decisions — it has become an orchestrator in disguise, but without the visibility and tooling a real orchestrator gives you. If you find one service is a “listener of everything,” ask whether you actually need explicit orchestration.
Event chain too long / “Domino Hell.” When Event A triggers Event B, which triggers Event C, which triggers Event D… it becomes almost impossible to trace what happens when a customer places one order. Beyond 4–5 hops, teams typically lose track of the full business flow without strong tracing tools.
Shared mutable event schemas without versioning. If Team A changes an event’s fields without a versioning strategy, every downstream consumer can silently start failing — sometimes without any errors, just wrong data.
Ignoring idempotency. Assuming “the broker will only deliver each message once” is a very common and dangerous mistake — nearly all real brokers offer “at-least-once” delivery by default, meaning duplicates ARE possible and must be handled by the consumer.
No dead-letter strategy. If a consumer keeps crashing on a “poison” event (a malformed or unexpected message) and there is no DLQ, it can block the entire queue behind it, or loop forever, wasting compute and delaying every other event.
10

Advantages, Disadvantages & Trade-offs

Choreography is a superb fit for many systems — and a terrible fit for others. Know what you are trading.

Advantages

  • Loose coupling — services don’t need to know about each other’s location or API.
  • Easy to add new consumers without touching existing services.
  • No single “orchestrator” bottleneck or single point of failure for the flow logic.
  • Scales naturally — each service scales independently based on its own load.
  • Encourages a more resilient, “publish and move on” style of coding.

Disadvantages

  • Hard to see the “big picture” business flow — it’s scattered across many services’ code.
  • Debugging failures requires distributed tracing across many hops.
  • Risk of hidden/implicit dependencies between services via shared event contracts.
  • Testing end-to-end flows is more complex than testing one orchestrator.
  • Compensating logic (undo actions) must be built and maintained by every participating service.
Trade-off DimensionWhat you gainWhat you give up
CouplingServices evolve independentlyOverall flow visibility
ScalabilityIndependent scaling per serviceConsistent global ordering guarantees (harder)
Speed of adding featuresNew consumers plug in easilyRisk of uncontrolled growth (“event sprawl”)
Failure isolationOne slow consumer doesn’t block producersDebugging where a chain broke is harder
11

Performance & Scalability

Choreography systems typically scale beautifully — but only if you understand how the broker distributes work.

Choreography systems typically scale extremely well, because each service can add more instances independently, and the event broker (like Kafka) is specifically engineered to handle very high throughput.

11.1 Partitioning

What it is. Splitting a topic into multiple independent partitions so multiple consumer instances can process events in parallel, while events with the same key (for example, same order ID) still go to the same partition to preserve order for that key.

Analogy. Imagine one long queue at a bank split into eight separate teller lines. Customers with the same account number always go to the same teller (so their transactions stay in order), but different accounts can be served simultaneously by different tellers.

11.2 Throughput vs Latency

Throughput is how many events per second the system can process. Latency is how long it takes one single event to be fully processed end-to-end. Choreography systems often optimise heavily for throughput (batching events, asynchronous processing), which can slightly increase per-event latency compared to a direct synchronous call — a classic trade-off.

1M+events/sec per Kafka broker at peak with batching & compression
10–50mstypical end-to-end pub → consume latency at low volume
0central bottleneck services — each scales alone

11.3 Backpressure

What it is. A mechanism to slow down producers (or buffer events) when consumers can’t keep up, preventing the system from being overwhelmed.

Analogy. If a water hose sprays a bucket faster than the bucket can drain, the bucket overflows. Backpressure is like a valve that senses the bucket filling up and reduces the water flow automatically.

11.4 Batching and Compression

Real brokers like Kafka batch many small events together and compress them before sending over the network — dramatically improving throughput at the cost of a few milliseconds of added latency per batch.

12

High Availability & Reliability

Delivery guarantees, replication, and safety valves — the ingredients that turn choreography from a nice idea into a production system.

12.1 Replication

What it is. Keeping multiple copies of the same event data across different machines (brokers), so if one machine dies, another copy is still available.

Analogy. Keeping copies of an important document in three different filing cabinets in different buildings. If one building burns down, you still have the document.

12.2 At-least-once vs Exactly-once vs At-most-once delivery

GuaranteeMeaningRisk
At-most-onceEvent delivered zero or one timePossible message loss
At-least-onceEvent delivered one or more timesPossible duplicates — needs idempotency
Exactly-onceEvent delivered and processed exactly once, no loss or duplicationHardest and most expensive to guarantee; usually achieved via idempotent processing + transactional writes rather than true “magic” exactly-once delivery

12.3 Retry with backoff

When a consumer fails to process an event (say, a downstream payment API is briefly down), it shouldn’t retry instantly and repeatedly. Instead, it waits progressively longer between attempts — this is called exponential backoff — to avoid hammering an already struggling downstream system.

12.4 Circuit Breaker

What it is. A safety mechanism that “trips” and stops calling a failing dependency for a while, instead of retrying endlessly and making things worse.

Analogy. An electrical circuit breaker in your house trips and cuts power when there’s a dangerous surge — protecting your appliances from damage, rather than letting the surge keep flowing.

12.5 Disaster Recovery

Production choreography systems typically replicate their event broker across multiple data centres or cloud regions, and maintain event retention long enough that consumers can “replay” missed events after an outage, catching up to the current state.

13

Advanced Concepts

The theoretical bedrock that shows up under every serious distributed-systems interview: CAP, consensus, partitioning, replay, concurrency.

13.1 CAP Theorem

What it is. A rule stating that in a distributed system, when a network partition (a communication breakdown between machines) happens, you must choose between Consistency (every reader sees the same, latest data) and Availability (every request gets a response, even if it might be slightly outdated).

Analogy. Imagine two friends in different cities trying to keep identical shopping lists. If their phones lose signal (a partition) and one crosses off “milk,” should the other friend’s list immediately show it crossed off too (consistency, but maybe unavailable if there’s no signal) or should it just keep showing the old list until reconnected (available, but temporarily inconsistent)?

Most choreography-based systems lean toward Availability + Eventual Consistency (AP) — services keep working locally and reconcile once events catch up, rather than freezing until every service agrees.

13.2 Consensus

What it is. A way for multiple machines to agree on one single value or decision, even if some machines fail or messages are delayed. Algorithms like Raft and Paxos solve this — and they’re what keeps a Kafka broker cluster (via a controller/coordination layer) agreeing on which broker is the leader for each partition.

13.3 Partitioning & Sharding

Already touched on above — splitting data or event streams by a key so work can be parallelised across machines, while keeping related events (same key) in order.

13.4 Failure Recovery & Replay

Because brokers like Kafka retain events for a configurable time, a consumer that crashes for an hour can simply “replay” from where it left off (its last committed offset) once it’s back up — catching up automatically without any events being lost. This “replayability” is one of choreography’s biggest reliability superpowers compared to older messaging systems that deleted messages the instant they were read.

13.5 Concurrency Considerations

When multiple instances of the same consumer service process events in parallel, you must guard against race conditions — for example, two instances trying to reserve the last item of stock at the same time. Techniques include optimistic locking (checking a version number before saving) and using a partition key (e.g., product ID) so all events about the same product always land on the same consumer instance, avoiding parallel conflicts entirely.

14

Security

Events live longer, travel wider, and replay more often than a regular API call. That makes security thinking sharper, not softer.

  • Authentication between services and the broker — using mutual TLS (mTLS) or SASL credentials so only trusted services can publish or subscribe to topics.
  • Authorization / ACLs — restricting which services are allowed to publish to or consume from specific topics (for example, only Payment Service can publish to the payments topic).
  • Encryption in transit — TLS between services and broker, and between broker nodes.
  • Encryption at rest — encrypting event data stored on broker disks, important since brokers often retain events for days.
  • Sensitive data minimisation — avoid putting things like full credit card numbers or passwords inside events; use tokens/references instead, since events may be replayed, logged, or retained much longer than a typical API call.
  • Schema validation as a security boundary — rejecting malformed or unexpected event payloads at the edge prevents injection-style attacks from propagating deep into your system.
  • Audit trails — since events are naturally an append-only log of “what happened,” they double as a strong audit trail for compliance (useful in finance, healthcare).
Never publish raw secrets (passwords, full card numbers, tokens) inside an event payload. Events often live far longer than a typical request/response call, get replicated to multiple consumers, and can be replayed — dramatically increasing the exposure window if leaked.
15

Monitoring, Logging & Tracing

Because the flow is scattered across many services, observability isn’t a nice-to-have — it’s the difference between diagnosable and terrifying.

Because the “flow” in choreography is scattered across many services, observability tooling becomes absolutely essential — arguably more important here than in orchestration.

15.1 Distributed Tracing

What it is. Attaching a shared “trace ID” (sometimes called a correlation ID) to an event when it’s first created, and passing that same ID along through every event in the chain, so you can later reconstruct the entire journey of one business transaction across all the services it touched.

Analogy. A tracking number on a package. As it moves through different trucks, warehouses, and delivery vans (different services), the same tracking number lets you see its full journey in one place, even though many different companies (services) handled it.

Distributed Tracing — One traceId Across All Events OrderPlaced traceId=abc123 PaymentCompleted traceId=abc123 StockReserved traceId=abc123 ShipmentCreated traceId=abc123 Tracing Dashboard shows full abc123 journey (Jaeger / Zipkin / APM)
Fig 5 · The same traceId is carried through every event, letting monitoring tools like Jaeger, Zipkin, or a cloud APM stitch together the full story of one order.

15.2 Key metrics to monitor

  • Consumer lag — how far behind a consumer is from the latest event published (a growing lag means trouble).
  • Event throughput — events published/consumed per second, per topic.
  • Processing latency — time between an event being published and being fully processed.
  • Dead-letter queue size — a growing DLQ signals a systemic processing problem.
  • Error/retry rate — spikes often indicate a downstream dependency issue.

15.3 Logging best practices

Always log the event type, event ID, and trace ID together with any error — this is the minimum needed to later reconstruct “what went wrong, and where in the chain.”

Pro tip. Standardise on a single trace-header convention (for example, W3C traceparent) across every service on day one. Retrofitting tracing later is one of the most painful and interruptive migrations a platform team can undertake.
16

Deployment & Cloud

Containers, managed brokers, and serverless consumers — how choreography is actually run in production today.

In modern cloud environments, choreography is typically deployed using:

  • Containers (Docker) + Kubernetes — each service (and often the broker itself) runs in containers, orchestrated (in the deployment sense, not the messaging sense!) by Kubernetes for scaling and self-healing.
  • Managed event brokers — AWS EventBridge/MSK (managed Kafka), Google Cloud Pub/Sub, Azure Service Bus/Event Hubs — so teams don’t have to operate the broker cluster themselves.
  • Serverless consumers — AWS Lambda or Google Cloud Functions can be triggered directly by events, scaling automatically to zero when idle and up instantly when volume spikes.
  • CI/CD pipelines — each service is deployed independently, often multiple times a day, since choreography’s loose coupling means one team’s deployment rarely requires coordinating with every other team.
  • Blue-green / canary deployments — safely rolling out a new version of a consumer by initially routing only a small percentage of events to it, watching for errors before shifting all traffic.

16.1 Cost Optimisation

Running a message broker and many independently deployed services isn’t free, so production teams actively manage cost in a few common ways:

  • Right-sizing partitions and retention — keeping only as much event history as you actually need for replay and audit purposes; excessive retention on a high-throughput topic can quietly become one of the largest storage costs in a system.
  • Autoscaling consumers based on lag — instead of running a fixed, always-on fleet sized for peak traffic, consumer instance counts scale up when consumer lag grows and scale back down when the queue is caught up.
  • Compression — compressing event batches (e.g., using Snappy or LZ4 with Kafka) meaningfully reduces both network and storage costs at high volume.
  • Choosing managed brokers wisely — for smaller workloads, a managed serverless broker (pay-per-event) is often cheaper than operating a self-managed cluster sized for constant availability; at very high, sustained volume, the economics can flip in the other direction.
17

Databases, Caching & Load Balancing

A shared database is the single fastest way to destroy a choreography system. Here’s how the data side is meant to look.

17.1 Database-per-service

In choreography-based microservices, each service typically owns its own private database — no other service is allowed to read or write to it directly. This is essential: if two services shared a database, they’d become tightly coupled again, defeating the whole purpose of choreography.

Analogy. Each department in a company keeps its own filing cabinet, and other departments must ask (or, in this case, listen for an announcement) rather than rummaging through someone else’s files directly.

17.2 Caching read models

Following the CQRS pattern mentioned earlier, a service can build a fast, cached “read view” by consuming events from other services — for example, a Product Catalog Service might cache “current stock levels” locally by listening to StockReserved and StockReplenished events, avoiding a live call to the Inventory Service for every page view.

17.3 Load balancing consumers

As described in the “competing consumers” pattern, multiple instances of the same consumer service are typically placed behind the broker’s own partition-assignment mechanism (not a traditional HTTP load balancer) — the broker itself decides which instance gets which partition’s events, rebalancing automatically if an instance crashes or scales up.

18

APIs & Microservices

Pure choreography is rare; pure orchestration is rare. Real systems mix synchronous APIs with asynchronous events on purpose.

Choreography doesn’t mean services never use regular request/response APIs at all — most real systems use a healthy mix:

  • Synchronous APIs (REST/gRPC) — used for immediate, user-facing queries where the caller genuinely needs a fast answer right now (for example, “show me this product’s price”).
  • Asynchronous events (choreography) — used for background business processes that don’t need an instant answer, and that benefit from loose coupling (for example, “process this order’s payment eventually, reliably”).
A typical e-commerce checkout page might use a synchronous API call to validate a coupon code instantly (the user is waiting), but use choreographed events for the actual order fulfilment process happening in the background after checkout completes.

This mixed style is sometimes called a hybrid architecture, and it is extremely common in real production systems — pure, 100% choreography is rare; pure orchestration is also rare. Most large systems use orchestration for a few well-defined, critical workflows (like refund approval) and choreography for the broader, high-volume, loosely coupled event flows.

19

Best Practices & Common Mistakes

The habits that separate a team calmly running choreography in production from one that quietly wishes they’d used an orchestrator.

19.1 Best Practices

  1. Version every event schema from day one

    Use names like OrderPlacedV1, OrderPlacedV2 so changes never silently break consumers.

  2. Always design consumers to be idempotent

    Assume duplicate delivery will happen — because on any real broker, sooner or later, it will.

  3. Attach a trace/correlation ID to every event

    This is the observability foundation on which everything else stands or falls.

  4. Keep events as immutable facts stated in the past tense

    OrderPlaced, not PlaceOrder. Names matter more than most engineers appreciate.

  5. Document which services publish and consume which events

    A simple “event catalog” saves enormous confusion later.

  6. Use the Outbox pattern

    To avoid the dual-write problem between your database and your broker.

  7. Set up a dead-letter queue and actively monitor it

    Don’t let poison messages silently vanish or endlessly retry.

  8. Keep event payloads reasonably self-contained

    Event-carried state transfer reduces chatty follow-up calls.

  9. Invest early in contract testing between producers and consumers

    So a schema change is caught in a build pipeline rather than discovered in production weeks later.

  10. Build an event catalog dashboard or wiki page

    List every event type, its owning team, its schema, and who consumes it — this single artifact often saves more debugging time than any other practice on this list.

  11. Treat compensating actions as first-class features

    Give them their own tests and design reviews, not an afterthought bolted on after the “happy path” is done.

19.2 Common Mistakes

MistakeWhy it hurts
Assuming events always arrive exactly once and in orderThey usually don’t, without extra design effort.
Letting one service become a “hidden orchestrator”You silently give up the benefits of choreography without gaining the benefits of orchestration.
Forgetting to design compensating actionsThe first real failure becomes a customer-facing incident.
Sharing a database between two services “just this once”Quietly destroys the benefits of choreography.
Not planning for schema evolutionBreaking changes cascade across teams.
No end-to-end tracingProduction incidents become extremely painful to diagnose.
20

Real-World Examples

Six well-known systems where choreography carries real load every day.

Streaming

Netflix

Uses event-driven, choreography-style architecture heavily for things like viewing-history updates, recommendation pipeline triggers, and content processing pipelines — different microservices react independently to streaming and content events without a single master controller for every workflow. Netflix also built and open-sourced Conductor, an orchestration engine, showing how they use orchestration for some workflows and choreography for others — a hybrid approach.

E-commerce / Cloud

Amazon & AWS

Amazon pioneered much of the “services communicate through well-defined interfaces, never through shared databases” philosophy internally, and popularised event-driven patterns broadly through AWS services like EventBridge, SNS, and SQS — which are essentially productised tools for building choreography-style systems at massive scale.

Ride-sharing

Uber

Relies on event streaming (built on top of Kafka) extensively for things like trip-state changes, driver location updates, and pricing signals — many services independently react to a rider requesting a trip, without one giant central “trip orchestrator” managing every downstream effect.

Finance

Banking & Payment Systems

Widely use choreography-based Sagas for multi-step transactions (like an international money transfer that must debit one account, credit another, and notify both parties), because compensating transactions are a natural fit for financial “undo” operations like reversals and refunds.

Travel

Airlines & Travel Booking Platforms

Often choreograph the multi-vendor booking problem: a flight, a hotel, and a rental car are booked from entirely separate external providers. Each provider’s confirmation (or failure) is treated as an event, and the platform’s own services react independently — issuing refunds or cancellations as compensating actions if one leg of the trip cannot be completed.

Payments / Fraud

PayPal & Payment Processors

Use event-driven, choreography-style pipelines internally for fraud detection — a single incoming transaction event is independently observed by multiple specialised services (velocity checks, device fingerprinting, risk scoring), each reacting on its own, rather than one central “fraud orchestrator” calling each check in sequence and adding latency to every transaction.

The common thread. Every one of these companies treats events as durable, replayable facts — not as fire-and-forget notifications. That single mindset shift is what unlocks choreography at scale.
21

Frequently Asked Questions

The questions that show up in interviews, in design reviews, and on the third day of every new microservices project.

Is choreography the same as “event-driven architecture”?

They’re closely related but not identical. Event-driven architecture is the broader style of building systems around events. Choreography is specifically about how services coordinate a multi-step business process using events, without a central coordinator — it’s one common application of event-driven architecture.

Can a system use both orchestration and choreography together?

Yes, and most large real-world systems do exactly this. A well-defined, critical, or highly regulated workflow (like a loan approval process) might use orchestration for clarity and auditability, while high-volume, loosely coupled background flows use choreography.

Does choreography mean no coordination at all?

No. Coordination still exists — it’s just implicit, expressed through shared event contracts and each service’s own reaction logic, rather than explicit, expressed in one central controller’s code.

How do you debug a choreographed system when something goes wrong?

Primarily through distributed tracing (a shared trace ID across all related events), centralised logging, and dashboards showing consumer lag and dead-letter queue sizes. Without these tools in place, debugging becomes extremely painful.

Is choreography always better for microservices?

No single approach is universally “better.” Choreography shines when you want loose coupling, independent scaling, and high flexibility to add new consumers. Orchestration shines when you need a very clear, auditable, centrally-controlled business workflow with fewer, well-known participants.

What happens if the event broker itself goes down?

This is why brokers are run as a replicated cluster (multiple machines), not a single server — if one broker node fails, others take over, and no events are lost as long as replication is correctly configured.

How do you test a choreographed system?

Testing happens at several levels. Unit tests cover a single service’s reaction to a single event, mocking the broker. Contract tests verify that a producer’s event schema matches what consumers expect, often using a shared schema registry as the source of truth. Integration tests run a real (or lightweight local) broker alongside a handful of services to verify a short chain of events behaves correctly. Full end-to-end tests across every service in a long chain are expensive to maintain, so most teams rely more heavily on contract tests plus strong production observability instead.

How do you keep events in the right order?

Most brokers (like Kafka) only guarantee ordering within a single partition, not across an entire topic. To keep related events in order — for example, all events about the same order — you route them using a consistent partition key (like the order ID), so they always land on the same partition and are processed in the sequence they were published.

What is the difference between choreography and a workflow engine like Temporal or Camunda?

Workflow engines are tools for building orchestration — they let you write the entire multi-step process as one piece of code or one diagram, and the engine executes it centrally, tracking state and handling retries for you. Choreography deliberately avoids this central piece; the “workflow” only exists implicitly, as the sum of every service’s individual event reactions.

Is choreography suitable for small applications or startups?

Often, no — at least not on day one. The overhead of running a message broker, designing event schemas, and handling eventual consistency is usually not worth it for a small team with two or three services and simple, synchronous needs. Choreography tends to pay off once a system grows to enough independent teams and services that the coupling cost of direct calls or a central orchestrator becomes painful.

22

Summary & Key Takeaways

Everything above, distilled to the sentences worth remembering.

Service choreography is a distributed cooperation style: many independent services complete one business process together, coordinating not through commands from a boss but through events they publish and consume. The result is loose coupling and near-linear scaling, at the cost of harder end-to-end visibility.

  1. Service choreography is a way for independent services to cooperate on a shared business process by reacting to events, without a central controller directing every step.
  2. It contrasts with orchestration, where one component explicitly calls each service in sequence.
  3. Core building blocks: services, events, an event broker/bus, publish/subscribe messaging, and agreed-upon event schemas.
  4. The Saga pattern (with compensating actions) is the standard way to handle multi-step transactions and failures in a choreographed system.
  5. Idempotency, schema versioning, distributed tracing, and dead-letter queues are not optional extras — they are essential production requirements.
  6. Real companies (Netflix, Amazon, Uber, and many banks) use choreography extensively, usually blended with orchestration for specific critical workflows.
  7. Choreography trades easy visibility of the “whole picture” for looser coupling, independent scalability, and resilience against any single point of control failing.
  8. Events are past-tense facts, not commands — and this naming discipline is what keeps services autonomous.
  9. Topics broadcast; queues distribute. Choreography almost always wants topics.
  10. The single biggest reason choreography systems fail in production is missing observability, not missing features.
If you remember only one idea. Remember the flock of birds. No single bird leads by giving orders — each bird just reacts to its neighbours — yet together they form one graceful, coordinated shape in the sky. That is the essence of service choreography.
“Events aren’t messages that ask for something — they are announcements of what has already become true.”
#service-choreography #event-driven #microservices #saga-pattern #kafka #pub-sub #eventual-consistency #idempotency #outbox-pattern #distributed-tracing #system-design #interview-prep