CQRS with Kafka

CQRS with Kafka

Why do some systems keep one model for writing data and a completely different model for reading it? Learn what CQRS means, how Kafka acts as the road between the two sides, and when this pattern helps or hurts.

Think about a busy restaurant. There are two very different jobs. One job is taking orders and cooking meals: it needs care, rules, and correct steps. The other job is telling guests what is on the menu and whether their table is ready: it needs to be quick and friendly, and it must answer thousands of small questions. In a small cafe, one person does both jobs. In a big restaurant, these jobs are split. Waiters and the kitchen handle orders, while a host desk and a big menu board handle questions.

Software systems face the same split. Some requests change things: place an order, transfer money, cancel a booking. Other requests only look at things: show my orders, list today’s flights, display my balance. The pattern that deliberately separates these two kinds of work is called CQRS. When you add Kafka, the messages that flow between the two sides become a reliable, replayable record of everything that happened.

This guide explains CQRS from the very beginning, shows how Kafka fits into it, walks through the data flow step by step, and warns you about the traps. No code is needed; we use pictures and everyday comparisons instead.

i
Quick answer

CQRS stands for Command Query Responsibility Segregation. It means using one model to handle commands (requests that change data) and a separate model to handle queries (requests that read data). With Kafka, the command side publishes events to Kafka topics, and one or more read-side programs consume those events to build fast, ready-to-use views of the data.

The Problem That CQRS Tries to Solve

One model doing two very different jobs eventually gets tired.

A single model for everything

Most applications begin with one database and one set of tables. The same tables are used to save data and to show data. This is simple and works well for a long time. But as the system grows, the two jobs start to pull in opposite directions.

  • Writing wants safety. Data must be checked, kept consistent, and stored in a tidy shape without repetition.
  • Reading wants speed. Screens and reports want data already combined, sorted, and shaped exactly as displayed, often with repetition to avoid slow joins.
  • The volumes differ. In many systems reads are ten, a hundred, or a thousand times more frequent than writes.
  • The teams differ. Business rules change one way, while screens and reports change another.
Everyday analogy

Imagine a single notebook used both as the official accounts ledger and as the price list that customers browse. Accountants want neat, checked entries. Customers want a big, readable board. Forcing both into one notebook makes the accountants slow and the customers frustrated.

The pain points

Pain

Slow screens

Complex joins across many tables make popular pages slow.

Pain

Locked-up writes

Heavy reports compete with orders for the same database resources.

Pain

Tangled code

One class tries to validate, save, and format for display all at once.

Pain

Hard to scale

You cannot scale reads without also scaling writes.

The Vocabulary You Need First

Learn these words once, and the rest of the guide becomes easy.

TermPlain meaningEveryday picture
CommandA request to change something, such as “place this order”A customer telling the waiter what they want
QueryA request to look at something, such as “show my orders”A guest asking “is my table ready?”
EventA fact that already happened, such as “order was placed”A receipt
Write modelThe part of the system that checks commands and records the resultThe kitchen
Read modelA ready-to-read view built for answering queriesThe menu board
ProjectionThe process of turning events into a read modelThe person who updates the board when dishes sell out
Eventual consistencyThe read side catches up a short moment after the write sideThe board updates a minute after the kitchen runs out
Event sourcingStoring every event as the source of truth, instead of only the latest stateKeeping every bank transaction instead of only the balance
TopicA named, ordered log in KafkaA numbered notebook
PartitionOne ordered lane inside a topicOne checkout lane
Consumer groupA team of programs sharing the reading of a topicA team of clerks sharing a pile of mail
IdempotentDoing it twice has the same effect as oncePressing a lift button twice
i
Remember

A command is a request that can be refused. An event is a fact that cannot be refused, because it already happened. This difference between “please do this” and “this was done” appears again and again in CQRS.

CQRS in Plain Words

The idea is small. The consequences are big.

The core idea

CQRS says: do not use the same model to change data and to read data. Create one path for commands and one path for queries. Each path can have its own code, its own data shape, its own database, and even its own servers.

The older idea behind it

CQRS grew from a simple rule about methods in programs: a method should either do something (a command) or answer something (a query), but never both. A method that returns a value should not also secretly change things. CQRS takes this rule and applies it to the whole system, not just to single methods.

flowchart LR
    U["User"] -->|"command: change something"| W["Write side"]
    U -->|"query: look at something"| R["Read side"]
    W --> WD[("Write store")]
    R --> RD[("Read store")]
    
FIG 1 — Two paths for two kinds of requests

In the picture, the user sends two kinds of requests. Commands go to the write side, which has its own store. Queries go to the read side, which has its own store. What is still missing is how the read store learns about changes made on the write side. That is where Kafka enters in the next chapters.

Everyday analogy

A library has a returns desk and an information desk. If you want to return or borrow a book, you go to the returns desk, where staff follow strict steps. If you only want to know whether a book is available, you ask the information desk or check the screen, which is quick and needs no strict steps. The two desks are different jobs, run by different people, looking at different tools.

CQRS is not a database, a product, or a framework. It is a decision: reads and writes are different jobs, so let them have different designs.

Why Kafka Fits CQRS So Well

Once you split the system in two, you need a dependable road between the halves.

What Kafka gives you

Kafka is a system that stores messages in ordered logs, called topics, and lets many programs read them independently. Those features solve the exact problems CQRS creates.

CQRS needsKafka provides
A reliable way to tell the read side about changesDurable topics that keep messages safely on disk
Changes delivered in the right orderOrder within each partition
Several read models from one set of changesMany independent consumer groups reading the same topic
Slow readers that must not slow the writersDecoupling: producers and consumers never wait for each other
Ability to rebuild a read model from scratchRetention and replay: read the log again from the start
Scaling with loadPartitions and consumer groups spread the work
Everyday analogy

Think of a town’s public notice board and its archive. Every announcement is posted in order and kept. Anyone can come and read new notices, or go back to read all past notices. A newspaper, a radio station, and a school each read the same board for their own purposes, without disturbing each other. Kafka is that notice board, kept safe and organized.

The spine of the system

In a CQRS system built with Kafka, the topics carrying events form the spine. The write side puts facts on the spine. Every read side hangs off the spine and takes what it needs. If a read side breaks, the spine still holds every fact, and the read side can be repaired by reading the spine again.

Durable
Events are kept on disk
Ordered
Sequence kept per partition
Replayable
Read models can be rebuilt

The Big Picture: Architecture and Components

A tour of all the moving parts, from a user’s click to a screen update.

flowchart LR
    C["Client"] -->|"command"| CH["Command handler"]
    CH --> WDB[("Write database")]
    CH -->|"publishes event"| K[["Kafka topic"]]
    K --> P1["Projection A"]
    K --> P2["Projection B"]
    P1 --> RDB1[("Read store A")]
    P2 --> RDB2[("Read store B")]
    C -->|"query"| QH["Query handler"]
    QH --> RDB1
    QH --> RDB2
    
FIG 2 — The complete CQRS architecture with Kafka

Meet the components

Write side

Command handler

Receives commands, checks the rules, and decides whether to accept or refuse.

Write side

Write database

The system’s source of truth, shaped for correctness.

Bridge

Kafka topics

Carry the events that describe what changed.

Read side

Projections

Consumers that turn events into ready-to-read views.

Read side

Read stores

Databases or caches shaped for particular screens or reports.

Read side

Query handler

Answers questions by looking in the right read store, and never changes anything.

Everyday analogy

A newsroom. Reporters (command handlers) gather facts and check them. The published stories flow through a distribution desk (Kafka). Separate teams turn them into the morning paper, the website, and the radio bulletin (projections and read stores). Readers never phone the reporter; they pick up the paper.

The Command Side in Detail

Where the rules live and where decisions are made.

What a command looks like

A command names an intention: “Place order”, “Change delivery address”, “Cancel booking”. It carries the data needed, such as the customer, the items, and a unique identifier. It is written in the imperative, as an instruction, and it may be rejected.

What the handler does

  1. Receive the command from the client.
  2. Check that the caller is allowed and the data is valid.
  3. Apply the business rules. For example, an order cannot be placed if stock is zero, and money cannot be withdrawn beyond the limit.
  4. Save the result to the write database.
  5. Publish one or more events to Kafka, saying what has happened.
  6. Reply with success or a clear reason for refusal.
flowchart TB
    IN["Command arrives"] --> V{"Allowed and valid?"}
    V -->|"No"| REJ["Refuse with a reason"]
    V -->|"Yes"| RULE{"Business rules pass?"}
    RULE -->|"No"| REJ
    RULE -->|"Yes"| SAVE["Save to write database"]
    SAVE --> PUB["Publish event to Kafka"]
    PUB --> OK["Reply: accepted"]
    
FIG 3 — The decisions inside a command handler

Why the write model can be simple and strict

Because the write side no longer has to serve screens, it can be shaped purely around rules. It can use a tidy structure with no repetition, strong checks, and small transactions. It rarely needs to be fast at reading, so it does not need many indexes or complicated joins.

!
Watch out

A command handler should not depend on the read side to make decisions. If it needs to look something up to check a rule, it should look in the write model, which is always up to date. The read side may be a moment behind.

The Read Side in Detail

Where speed and convenience are the only goals.

Read models are built for the question

A read model is shaped for the way the answer will be shown. If a page needs “customer name, latest three orders, and total spent”, the read model stores exactly that, already combined. Answering the query becomes a single quick lookup, with no joining at question time.

Projections do the building

A projection is a program that reads events from Kafka and updates the read model. When it sees “order placed”, it adds the order to the customer’s list and raises the total. When it sees “order cancelled”, it removes the order and lowers the total. The projection is like a clerk who keeps the summary sheet up to date every time a new receipt arrives.

flowchart LR
    E1["Order placed"] --> PR["Projection"]
    E2["Order paid"] --> PR
    E3["Order cancelled"] --> PR
    PR --> V[("Customer summary view")]
    Q["Query: show customer summary"] --> V
    
FIG 4 — Events shaped into a ready-to-read view

Freedom of choice

Each read model can live in whatever store suits it best.

  • A search screen may use a search engine.
  • A profile page may use a fast key-value store.
  • A monthly report may use an analytical database.
  • A dashboard may use an in-memory cache.

This is sometimes called polyglot persistence, which simply means using several kinds of storage, each for what it does best.

Everyday analogy

A shop keeps its prices on a printed catalog, a shelf label, and an online page. All three come from the same master price list, but each is laid out for its own audience. Nobody reads the master list to shop.

CQRS and Event Sourcing: Cousins, Not Twins

These two ideas are often mentioned together, and often confused.

What event sourcing is

Normally a database stores the latest state: the account has 500 rupees. With event sourcing, the system stores every event that ever happened: deposited 1000, withdrew 300, withdrew 200. The current state is worked out by adding the events. The events are the true record; the state is just a summary.

How they relate

  • CQRS without event sourcing is common. The write side keeps ordinary state in a normal database, and still publishes events so read models can be built.
  • Event sourcing without CQRS is possible but awkward, since reading current state from raw events on every query is slow. Read models usually appear anyway.
  • Both together is a natural pair: the event log is the write side’s truth, and projections build the read side.
QuestionCQRSEvent sourcing
What does it decide?Separate models for writing and readingStore events instead of only the latest state
Needs the other?NoNo, but they pair well
Main benefitIndependent scaling and shaping of readsComplete history and the power to replay
Main costTwo models and eventual consistencyComplexity of thinking in events
Everyday analogy

Event sourcing is keeping every bank statement line. CQRS is deciding to have one desk for depositing money and a separate screen for viewing balances. You can do the second without the first, but having every statement line makes the balance screen easy to rebuild.

i
Tip

Do not adopt event sourcing just because you adopt CQRS. Start with CQRS and ordinary state on the write side. Add event sourcing only if you truly need the complete history.

Eventual Consistency: The Price You Pay

The read side is always a little behind. Learn to live with it.

What it means

When a command succeeds, the write database is updated right away. But the read model is updated only after the event travels through Kafka and the projection processes it. In between, a query may return the old answer. Given a short time, the read model catches up. This is called eventual consistency.

1

Command accepted

The user changes their address. The write database is updated.

2

Event published

An “address changed” event is placed in a Kafka topic.

3

A short gap

If the user reloads their profile now, the read model may still show the old address.

4

Projection catches up

The projection consumes the event and updates the read store. From now on, queries show the new address.

Everyday analogy

You change your phone number with your bank. The clerk updates the master file immediately, but the number printed on tomorrow’s statement may be old because the statement was prepared overnight. Nothing is wrong. The statement is simply a little behind the master file.

How big is the gap?

In a healthy system, the gap is often tiny, from a few milliseconds to a few seconds. It grows when projections are slow, when Kafka consumers are lagging, or after an outage. You should measure it, watch it, and decide what is acceptable for each screen.

!
Watch out

If any part of your business truly cannot tolerate even a short delay between a change and its visibility, that part may need to read from the write side, or may not suit CQRS at all.

The Life of One Command, End to End

Follow a single request from the click to the updated screen.

sequenceDiagram
    participant U as User
    participant CH as Command handler
    participant DB as Write database
    participant K as Kafka
    participant P as Projection
    participant RS as Read store
    participant QH as Query handler
    U->>CH: Place order
    CH->>DB: Check rules and save
    CH->>K: Publish "order placed"
    CH-->>U: Accepted
    K->>P: Deliver event
    P->>RS: Update customer view
    U->>QH: Show my orders
    QH->>RS: Look up
    RS-->>QH: Ready answer
    QH-->>U: Orders list
    
FIG 5 — The full journey of a command and the query that follows

Reading the diagram slowly

  1. The user clicks “place order”. The command handler receives it.
  2. The handler checks the rules against the write database and saves the order.
  3. It publishes an “order placed” event to Kafka.
  4. It answers the user: accepted. The user does not wait for the read side.
  5. Kafka delivers the event to the projection, which updates the read store.
  6. Later, the user asks to see their orders. The query handler looks in the read store and returns a ready answer.

Notice that the query never touches the write database, and the command never touches the read store. The only link between them is the event in Kafka.

Everyday analogy

You post a form at the post office counter (command). The clerk checks it, stamps it, and files a copy (write database). A note goes into the internal mail (Kafka) to the records department, who update the public register (read store) a little later. When you check the register (query), you see the update once the internal mail has done its rounds.

Designing Kafka Topics for CQRS

The shape of your topics decides how easy everything else will be.

Commands as messages, or only events?

There are two common styles. In the first, commands travel over HTTP directly to the command handler, and only events are placed in Kafka. In the second, commands themselves are also put into a Kafka topic, and command handlers consume them. The second gives more buffering and lets the system absorb spikes, but it makes the reply to the user harder, since the user does not get an immediate answer. Many teams start with the first style for its simplicity.

One topic per kind of thing

A useful rule is to have one topic for each kind of business entity or “aggregate”, such as orders, customers, or payments. All events about orders go to the “orders” topic, so their order is preserved and consumers can follow the whole story of an order in one place.

ChoiceGood sideCareful about
One topic per entity typeOrder of an entity’s events is preservedTopics can grow big; choose partitions wisely
One topic per event typeConsumers subscribe only to what they needOrder between event types is lost
One giant topic for everythingSimple to startEvery consumer reads everything; hard to secure and to scale

What an event should carry

  • A unique event id, so duplicates can be recognized.
  • The entity id, so the event can be routed and ordered.
  • The event type and a version number for the message format.
  • A timestamp for when it happened.
  • The facts, written in past tense: “order placed”, not “place order”.
  • Enough data for consumers to act without calling back the write side.
Everyday analogy

A good receipt says who bought, what, when, and for how much, and has a unique number. Someone in another building can file it without phoning the shop. An event should be like that receipt: complete on its own.

i
Tip

Name events after business facts, not technical actions. “Order shipped” tells a story. “Row updated” tells nothing.

Keys, Partitions, and Ordering

Kafka keeps order only inside a partition, so choose the key carefully.

Why order matters

Suppose an order is placed, then paid, then cancelled. If a projection sees them as cancelled, placed, paid, it ends up with the wrong picture. Order matters, at least for events about the same thing.

How Kafka helps

Each message can have a key. Kafka uses the key to choose a partition: the same key always goes to the same partition. Inside a partition, messages are stored and read in the order they were written. If you use the order id as the key, all events for one order land in one partition, in the right order.

flowchart LR
    E1["Order 7: placed"] --> P0["Partition 0"]
    E2["Order 7: paid"] --> P0
    E3["Order 9: placed"] --> P1["Partition 1"]
    E4["Order 7: shipped"] --> P0
    E5["Order 9: cancelled"] --> P1
    
FIG 6 — The same key always lands in the same partition

Choosing the key

  • Use the id of the entity whose order you care about, such as the order id, account id, or customer id.
  • Avoid keys that put most traffic on one value, since one partition will become overloaded, like a supermarket with one busy lane and many empty ones.
  • Choose the partition count with future growth in mind. Adding partitions later changes which partition a key maps to, which can disturb ordering for existing data.
Everyday analogy

A post office sorts letters into bins by postcode. All letters for the same address go into the same bin and stay in the order they arrived. Different bins are handled by different clerks in parallel.

!
Watch out

Kafka does not guarantee order between different partitions. If your projection needs order across different entities, you must design for it, or accept that it cannot be assumed.

Building Projections: The Main Ways

Several tools can turn events into read models. Each suits different needs.

Way 1

A plain consumer application

A small service reads the topic and writes to a database. Easy to understand and fully flexible.

Way 2

Kafka Streams

A library for building processing applications that can join, group, and count events, and keep local state.

Way 3

Kafka Connect sinks

Ready-made connectors that copy events into stores such as search engines or databases, with little or no custom code.

Way 4

Stream SQL engines

Tools that let you describe views in a query language and keep them updated continuously.

WayBest whenTrade-off
Plain consumerThe logic is custom or simpleYou write and maintain everything yourself
Kafka StreamsYou need joins, counts, or windows over eventsMore concepts to learn; state must be managed
Connect sinkThe view is a straight copy into a storeLimited transformation
Stream SQLAnalysts or many small views need quick definitionsAnother platform to run
flowchart LR
    T[["Orders topic"]] --> A["Plain consumer"]
    T --> B["Kafka Streams app"]
    T --> C["Connect sink"]
    A --> DB1[("Relational view")]
    B --> DB2[("Aggregated view")]
    C --> SE[("Search index")]
    
FIG 7 — Three different projections reading one topic
Everyday analogy

The same daily newspaper can be cut up by a librarian for the archive, summarized by an analyst for a briefing, and copied by a photographer for a scrapbook. Same source, three methods, three results.

KStreams and KTables as Read Models

Kafka has a built-in way of thinking about “latest state per key”.

A stream and a table

In Kafka Streams, a stream is a never-ending sequence of events, and a table is the latest value for each key. If the events are “customer 5 moved to Pune” and later “customer 5 moved to Delhi”, the stream shows both, and the table shows only Delhi. A table is a natural read model: it always answers “what is the current state of this key?”.

Interactive queries

Kafka Streams can store that table in local state on the processing application, and let other programs ask it questions directly. This means an application can be both the projection and the query service, with no separate database. It is elegant for simple lookups, but it adds care around moving state when instances restart or scale.

Everyday analogy

A stream is a diary of everything that happened. A table is the address book, where each person has only their current address. The address book is built from the diary but is far quicker to look things up in.

Strengths

  • No extra database for simple views
  • State is backed up in Kafka itself
  • Scales with partitions
  • Natural fit for latest-state-per-key

Limits

  • Complex ad hoc queries are hard
  • State restore after restarts takes time
  • Data lives with the application, so access needs planning
  • Not ideal for large search or analytics views

The Outbox Pattern: Saving and Publishing Safely

A subtle trap sits between the write database and Kafka.

The trap

The command handler must do two things: save to the write database and publish an event to Kafka. These are two different systems. What if the save works, but the publish fails? The database says the order exists, but the read side never hears about it. What if the publish works and the save fails? The read side shows an order that does not exist. This is called the dual write problem.

The outbox solution

Instead of writing to two systems, the handler writes to one. In the same database transaction that saves the order, it also saves the event to a special table, the outbox. Either both are saved or neither. A separate process then reads new rows from the outbox and publishes them to Kafka. Some teams use a change-capture tool that watches the database log and sends new outbox rows to Kafka automatically.

flowchart LR
    CH["Command handler"] -->|"one transaction"| DB[("Write database")]
    DB --- OB["Orders table and outbox table"]
    OB --> RELAY["Relay or change capture"]
    RELAY --> K[["Kafka topic"]]
    
FIG 8 — The outbox: one save, then a safe hand-off to Kafka
Everyday analogy

When you write a letter, you do not run to the post box in the middle of writing your diary entry. You put the letter in your own outbox tray at the same moment you finish the entry. Later, the mail collector takes everything from the tray. Even if the collector is late, the letter is not forgotten.

i
Tip

The outbox delivers events at least once: after a crash, the relay may send an event twice. That is fine, if consumers are idempotent, which the next chapter covers.

Duplicates, Idempotency, and Exactly Once

Messages can be delivered more than once. Projections must cope.

Why duplicates happen

  • A consumer processes an event, then crashes before recording its progress. After restart, it reads the same event again.
  • A producer retries after a timeout and the first attempt had in fact succeeded.
  • The outbox relay sends a row twice after a restart.

The remedy: idempotent projections

An operation is idempotent if doing it twice gives the same result as doing it once. Projections can be made idempotent in several ways.

  • Set values, do not add to them. “Set the order status to paid” is safe to repeat. “Add one to the paid count” is not.
  • Remember processed event ids. Keep a small record of ids already handled, and skip repeats.
  • Use version numbers. Apply an event only if its version is newer than what the view already holds.

What about “exactly once”?

Kafka offers features that can make reading, processing, and writing back into Kafka behave as if each message were handled exactly once, within Kafka’s own world. But once you write to an outside database, the guarantee is only as good as the database write. In practice, the safest habit is at-least-once delivery plus idempotent processing.

Everyday analogy

A shop worker who writes “price of milk is 60” on the shelf label does no harm if told twice. A worker who is told “raise the price by 5” twice makes a mistake. Setting a value is safe to repeat; adding to it is not.

Replay: Rebuilding a Read Model From Scratch

One of the greatest gifts of putting Kafka in the middle.

Why replay is powerful

Read models are disposable. If one is damaged, or if you invent a new view, you can throw the old one away and rebuild it by reading the topic again from the beginning. A new team can create a brand new view of last year’s data without asking anybody to export anything.

1

Create the new projection

Give it a new consumer group name so it starts from scratch.

2

Start at the beginning

Tell it to read from the earliest available message.

3

Catch up

It processes history quickly, then keeps up with new events.

4

Switch over

Once caught up, point the query side to the new store and retire the old one.

The limits of replay

  • Retention. Replay works only as far back as Kafka keeps data. For full rebuilds, use long or unlimited retention on the event topics, or compaction where only the latest event per key matters.
  • Time. Replaying billions of events takes a while. Plan for it, and consider snapshots.
  • Side effects. A projection that also sends emails would send them all again. Keep projections free from side effects, or guard them.
Everyday analogy

If a class summary sheet gets spilled on, a teacher can produce a new one by going through the original marks register again. The register is the record; the summary sheet is just a convenience.

Evolving Events Over Time

Events are kept for years, so their format must be able to grow without breaking readers.

The challenge

Today’s “order placed” event has five fields. Next year, the business wants a sixth. Old events in the topic have five, new ones have six. Every consumer, old and new, must be able to read both.

Habits that help

  • Use a schema. Describe each event’s shape formally, and store schemas in a schema registry so producers and consumers agree.
  • Add, do not remove. New fields are usually safe if they are optional or have defaults. Removing or renaming fields breaks readers.
  • Set compatibility rules. The registry can refuse a new schema version that would break existing consumers.
  • Version big changes. If a change is truly incompatible, create a new event type or topic and migrate slowly.
  • Be liberal in what you read. Projections should ignore fields they do not know.
Everyday analogy

A school report card changes design over the years. Old cards do not get reprinted. The people who read them, such as universities, must understand both old and new layouts. A schema is the agreed guide to reading each layout.

!
Watch out

Events are a public promise. Once other teams depend on them, changing them carelessly breaks other people’s systems. Treat event formats like an interface you must keep stable.

The “I Just Saved It, Where Is It?” Problem

The most visible symptom of eventual consistency, and how to soften it.

The situation

A user adds an item to their list and is sent straight to the list page. The page is served from the read model, which has not yet received the event. The item is missing. The user thinks the save failed, and clicks again, creating a duplicate.

Ways to handle it

Fix 1

Show the change locally

The screen adds the item itself right away, as a “pending” entry, and replaces it when the real data arrives.

Fix 2

Return the new state

The command reply includes the new data so the page needs no immediate query.

Fix 3

Wait for a version

The reply carries a version number, and the query waits briefly until the read model reaches it.

Fix 4

Read your own write

For a short time after a change, queries from that user go to the write side.

sequenceDiagram
    participant U as User screen
    participant W as Write side
    participant R as Read side
    U->>W: Add item
    W-->>U: Accepted, version 42
    U->>R: Show list, wait for version 42
    Note over R: Projection catches up
    R-->>U: List including new item
    
FIG 9 — Waiting for a version so users see their own change
Everyday analogy

You hand in your form and the clerk says, “Your reference number is 42. Come to the notice board in a minute and look for number 42.” You know exactly what to expect, so you are not worried when it is not yet listed.

Many Read Models From One Source

The quiet superpower of CQRS with Kafka.

Because Kafka lets many consumer groups read the same topic independently, you can attach as many read models as you like. Each one is shaped for one audience, and none affects the others. Adding a new view means adding a new consumer group, and never changing the write side.

flowchart LR
    W["Write side"] --> T[["Events topic"]]
    T --> V1["Customer profile view"]
    T --> V2["Search index"]
    T --> V3["Analytics warehouse"]
    T --> V4["Notification service"]
    T --> V5["Fraud checks"]
    
FIG 10 — One event stream, many independent consumers
Read modelQuestion it answersFitting store
Profile pageWho is this customer and what did they order?Key-value or document store
Product searchWhich items match these words?Search engine
Monthly reportHow much did we sell by region?Analytical database
Live dashboardWhat is happening right now?In-memory store
NotificationsWho must be told about this change?A small service with a queue
Everyday analogy

One cricket match, many audiences. The scoreboard shows runs. The commentator tells the story. The statistician updates career records. The betting shop updates odds. Each takes the same ball-by-ball facts and makes something different, and none of them bother the players.

Long-Running Processes and Sagas

What happens when one business action needs several services to agree.

The situation

Placing an order might need the stock service to reserve items, the payment service to charge the card, and the shipping service to book a courier. No single database transaction can cover all three services. If payment fails after stock was reserved, the reserved stock must be released.

The saga

A saga is a sequence of steps, where each step is a local action followed by an event, and where each step has a matching compensation, an action that undoes it if a later step fails. Kafka carries the events between the steps.

flowchart LR
    O["Order placed"] --> S["Stock reserved"]
    S --> P["Payment taken"]
    P --> H["Shipment booked"]
    P -->|"payment fails"| C["Release stock"]
    C --> X["Order cancelled"]
    
FIG 11 — A saga with a compensating step

Two styles

  • Choreography. Each service listens for events and reacts. There is no central boss. It is simple for small flows, but the whole story becomes hard to see when there are many services.
  • Orchestration. A central coordinator, sometimes called a process manager, tells each service what to do and listens for the results. It is easier to follow and change, at the price of one more component.
Everyday analogy

Booking a holiday involves flights, hotel, and a taxi. If the hotel cannot confirm, you cancel the flight you already booked. That cancellation is the compensation. A travel agent who runs the whole booking is an orchestrator; friends who each book a part and tell each other what they did are choreography.

When to Use CQRS, and When Not To

The pattern is powerful, and power has a cost. Choose it with open eyes.

Good signs

  • Reads far outnumber writes, or vice versa
  • Screens need data combined from many sources
  • Different audiences need very different views
  • Business rules on writing are complex
  • Several teams want to build on the same data
  • You need history, audit, or replay

Warning signs

  • The application is mostly simple forms and lists
  • The team is small and new to messaging
  • Users cannot tolerate any delay between write and read
  • There is no plan to operate Kafka well
  • The main reason is “it sounds modern”
flowchart TB
    S["Considering CQRS"] --> Q1{"Are reads and writes very different in shape or volume?"}
    Q1 -->|"No"| N1["A single model is probably enough"]
    Q1 -->|"Yes"| Q2{"Can users accept a short delay?"}
    Q2 -->|"No"| N2["Use CQRS only where safe, read from write side elsewhere"]
    Q2 -->|"Yes"| Q3{"Can your team run Kafka and projections?"}
    Q3 -->|"No"| N3["Start smaller or use a managed service"]
    Q3 -->|"Yes"| Y["CQRS with Kafka is a good fit"]
    
FIG 12 — A simple decision path
Apply CQRS to the part of the system that hurts, and not to the whole company at once.
Everyday analogy

A small tea stall does not need separate order takers, kitchens, and menu boards. A large hotel does. The right answer depends on how busy and how complicated the place is.

Advantages, Disadvantages, and Trade-offs

A fair balance sheet.

Advantages

  • Read and write sides scale separately
  • Read models shaped exactly for their screens
  • Fast queries with no heavy joins
  • Heavy reports no longer block orders
  • Clear separation of code and responsibility
  • New views added without touching the write side
  • Rebuild and replay from the event log
  • Natural audit trail

Disadvantages

  • Eventual consistency confuses users and developers
  • More moving parts to build and operate
  • Duplicate handling and ordering need care
  • Event formats must be managed for years
  • Harder to debug across several components
  • Overkill for simple applications

The main trade-offs

  • Consistency versus speed. You give up instant agreement between the sides to get fast, scalable reads.
  • Simplicity versus flexibility. One model is easy to understand. Two models offer freedom, but ask for more discipline.
  • Storage versus speed. Read models repeat data on purpose, using extra storage to save time at question time.
  • Team independence versus coordination. Teams can build their own views, yet must agree on event contracts.
Everyday analogy

Preparing meals in advance for a busy week saves time every evening, but costs a lot of effort on Sunday and takes more fridge space. CQRS pays effort and storage up front to save time on every read.

Security in a CQRS System

More places hold the data, so more places need guarding.

Guard the commands

Commands change things, so they need strong authentication, proving who the caller is, and authorization, checking whether that caller may do this action. The command handler is the right place to enforce both, before any rule is applied.

Guard the topics

  • Encrypt the traffic between applications and Kafka.
  • Use access rules so only the command side may write to event topics, and each projection may read only the topics it needs.
  • Give every application its own identity, so you can tell who did what.

Guard the read models

Every read store holds a copy of some data. Apply the same care to each: who may query, which fields may be shown, and how long data may be kept. A screen for support staff may hide card numbers that the accounting view needs.

Sensitive data in events

Events stay in Kafka for a long time, maybe forever. Personal data placed in an event may be difficult to remove later. Options include keeping personal details out of events and storing only an id, encrypting sensitive fields, or planning a way to erase them, in line with the privacy laws that apply to you.

!
Watch out

An immutable log and a “right to be forgotten” pull in opposite directions. Decide how you will handle deletion requests before the first event is written, and not after a regulator asks.

Everyday analogy

A company with a strong front door but many unlocked side rooms is not secure. Each read model is a room holding copies of valuables. Each needs its own lock.

Monitoring, Logging, and Metrics

Because the system is spread out, you need to see across it.

Freshness

Projection lag

How many events, or how many seconds, each read model is behind. This is the number that describes your eventual consistency in real life.

Flow

Event rate

How many events per second flow through each topic.

Speed

Command latency

How long commands take from arrival to reply.

Speed

Query latency

How long queries take. Slow queries often signal a poorly shaped read model.

Health

Error and retry counts

Failed events, refused commands, and messages sent to a dead letter topic.

Health

Outbox backlog

Rows waiting to be published. Growth means events are stuck.

Tracing across the gap

When a user says “my change did not show up”, you must follow it from the command, through the outbox and Kafka, to the projection and the read store. Give each command a correlation id that is copied into every event and every log line. Searching for that one id then shows the whole journey.

flowchart LR
    CMD["Command, id abc-123"] --> EVT["Event, id abc-123"]
    EVT --> PRJ["Projection log, id abc-123"]
    PRJ --> RDM["Read store update, id abc-123"]
    
FIG 13 — One correlation id links the whole journey
i
Tip

Alert on projection lag above what each screen can tolerate. It is the earliest and clearest sign that the read side is drifting from the write side.

Performance and Scaling

Scale each side according to its own load.

Scaling the write side

The write side scales by adding command handler instances and, where needed, by spreading the write database. Partitioning by entity keeps related work together and avoids conflicts.

Scaling the read side

Reads are usually the bigger load. You can add more query handler instances, add replicas to the read store, add caches, or create extra read models tuned to heavy screens. None of this touches the write side.

Scaling the bridge

Kafka scales by partitions. More partitions allow more consumers in a group to work in parallel. But a consumer group can never have more active members than partitions, so choose enough partitions for your future peak.

PressureWhat to scaleHow
Many commandsCommand handlers, write databaseMore instances, better partitioning of data
Many queriesQuery handlers, read storesMore instances, replicas, caches
Slow projectionsConsumer groupMore members, up to the partition count
Huge event rateKafka topics and brokersMore partitions and brokers
Everyday analogy

A stadium with many entrances. When the crowd grows, you open more gates. But the gates can only be as many as the number of walls you built. Partitions are the walls.

Handling Failures Gracefully

Things will go wrong. Decide beforehand what should happen.

When a projection cannot process an event

Some events are bad: malformed, or referring to something unknown. If a projection keeps failing on the same event, it blocks the partition, and everything behind it waits. Common choices are:

  • Retry a few times with pauses, in case the trouble is temporary.
  • Move the bad event to a dead letter topic so processing continues, and inspect it later.
  • Stop and alert for events that must never be skipped, because skipping would make the view wrong.

When Kafka is unavailable

With an outbox, the command side keeps working: events pile up in the outbox and are published when Kafka returns. Queries keep working too, since the read stores are separate, though they get gradually staler.

When a read store is lost

Rebuild it by replaying the topic. This is the reason to keep event topics long enough.

flowchart TB
    EV["Event arrives"] --> TRY{"Processed OK?"}
    TRY -->|"Yes"| DONE["Commit progress"]
    TRY -->|"No"| RETRY{"Retries left?"}
    RETRY -->|"Yes"| WAIT["Wait and try again"]
    WAIT --> TRY
    RETRY -->|"No"| DLQ["Send to dead letter topic and alert"]
    DLQ --> DONE
    
FIG 14 — A simple failure path for a projection
Everyday analogy

A mail sorter meets a letter with an unreadable address. Rather than stopping the whole line, they put it in a “problem” tray for a supervisor and carry on. The tray is the dead letter topic.

Testing CQRS Systems

Splitting the system also splits the way you test it.

1

Test commands by their events

Given some history, when this command arrives, then these events must result. It reads like a story and is easy to check.

2

Test projections by their views

Given these events, the read model must look like this. Include duplicates and out-of-order cases.

3

Test contracts

Check that event formats remain readable by every consumer as they evolve.

4

Test the whole path

Send a command and wait until the read model reflects it, measuring how long it takes.

5

Test rebuilds

Regularly delete a read model and rebuild it from the topic, to prove that replay works.

Everyday analogy

Testing a recipe means checking both that the cooking steps produce the right dish and that the dish looks right on the menu photo. You test the kitchen and the display separately, then together.

Deploying and Operating the System

A few habits keep day-to-day life calm.

  • Deploy sides independently. The great benefit of the split is that you can release a new projection without touching the command side.
  • Deploy new consumers before new producers. When a new event type is introduced, make sure readers can handle it first.
  • Use a new consumer group for a changed projection, let it catch up in parallel, then switch queries to it.
  • Keep infrastructure as code, including topics, partitions, retention, and access rules.
  • Plan capacity for catch-up. After an outage, projections work harder than normal to reach the present.
  • Document event contracts in one place that every team can read.
flowchart LR
    OLD["Old projection, live"] --> Q["Queries"]
    NEW["New projection, catching up"] -.-> Q
    NEW -->|"caught up: switch"| Q
    OLD -->|"then retire"| END["Retired"]
    
FIG 15 — Replacing a projection without downtime

Moving From a Classic Application to CQRS

Few teams start from an empty page. Most move step by step.

1

Find the pain

Pick one slow or tangled part, for example a heavy report or a busy search page.

2

Publish events

Make the existing application publish events for its changes, using an outbox or change capture on the database.

3

Build one read model

Create one projection and one new query path for that painful screen.

4

Compare and switch

Run old and new side by side, compare the answers, then move traffic across.

5

Repeat where it pays

Move other screens only if there is a clear gain.

Everyday analogy

Renovating a house you still live in. You do the kitchen first, while the rest stays as is. You do not knock the whole house down and ask the family to camp outside for a year.

i
Tip

This gradual approach is sometimes called the strangler style: the new parts slowly take over, and the old parts shrink until they can be removed.

Kafka Features That Matter Most to CQRS

A short tour of the settings and abilities you will lean on.

Kafka featurePlain meaningWhy CQRS cares
RetentionHow long messages are keptDecides how far back you can rebuild a read model
Log compactionKeep only the latest message for each keyGood for “current state” topics, saving space while keeping the newest fact per entity
Replication factorNumber of copies of each partition across brokersProtects the event log against machine loss
AcknowledgementsHow many copies must confirm a write before it countsStronger settings reduce the chance of losing an event
Consumer groupsTeams of readers sharing a topicGive each projection its own independent progress
Committed offsetsA saved bookmark of how far a group has readLets a projection restart from where it stopped
TransactionsWrite several messages, and progress, as one unit inside KafkaHelps processing that reads and writes Kafka topics avoid partial results
Schema registryA shared library of message formatsKeeps events readable as they evolve

Compaction versus long retention

Two strategies exist for keeping enough history to rebuild read models. With long retention, every event is kept for a long period, or forever, so the full story can be replayed. With compaction, Kafka keeps only the latest message per key, which is enough for views that care about current state, but loses the history between. Choose long retention for event streams that tell stories, and compaction for topics that carry “latest state” records.

Everyday analogy

A filing cabinet can keep every letter ever received, which is long retention. Or it can keep just the most recent letter from each person, which is compaction. The first tells the whole history. The second gives the current position with far less paper.

i
Tip

Write acknowledgement and replication settings for event topics with care. Losing an event on the spine means every read model quietly misses it.

Design Patterns That Work

Reusable ideas that keep a CQRS system healthy.

Pattern

Transactional outbox

Save state and event together, and publish afterwards, to avoid the dual write problem.

Pattern

Idempotent projection

Design views so that repeated events change nothing.

Pattern

Aggregate-keyed events

Use the entity id as the key so its events stay in order.

Pattern

Materialized view per screen

Shape a read model to match exactly one screen or report.

Pattern

Saga

Coordinate steps across services, with compensations for failure.

Pattern

Dead letter topic

Park bad events so good ones keep flowing.

Pattern

Correlation id

Carry one identifier through commands, events, and logs.

Pattern

Parallel rebuild

Build a changed view beside the old one and switch when ready.

Anti-Patterns to Avoid

Mistakes that turn a neat idea into a messy system.

ANTI-PATTERN 01Avoid
What people do

They write to the database and then publish to Kafka as two separate steps.

Why it hurts

A crash between the steps leaves the two sides disagreeing, and the difference is hard to notice.

Better approach

Use a transactional outbox or database change capture.

ANTI-PATTERN 02Avoid
What people do

They let commands check business rules by querying the read side.

Why it hurts

The read side may be behind, so rules are decided on stale facts.

Better approach

Validate against the write model, which is always current.

ANTI-PATTERN 03Avoid
What people do

They publish thin events such as “order changed” with no details.

Why it hurts

Every consumer must call the write side to learn what changed, which recreates tight coupling and load.

Better approach

Publish meaningful, self-contained events named after business facts.

ANTI-PATTERN 04Avoid
What people do

They ignore eventual consistency in the user interface.

Why it hurts

Users save something, see nothing, and try again, causing duplicates and support calls.

Better approach

Return the new state, show pending items, or wait for a version.

ANTI-PATTERN 05Avoid
What people do

They apply CQRS to every part of a simple application.

Why it hurts

The team pays for extra complexity where nothing was broken.

Better approach

Use CQRS only where reads and writes truly pull apart.

ANTI-PATTERN 06Avoid
What people do

They change event formats freely, renaming and removing fields.

Why it hurts

Old events and old consumers break, sometimes silently.

Better approach

Use a schema registry with compatibility rules, and add fields instead of changing them.

ANTI-PATTERN 07Avoid
What people do

They let projections send emails or call outside systems as they process.

Why it hurts

A replay or a duplicate sends the same email again and again.

Better approach

Keep projections pure, and put side effects in separate, guarded consumers.

Best Practices and Common Mistakes

A checklist gathered from many projects.

Best practices

  • Start with one painful area, not the whole system
  • Name events after business facts, in the past tense
  • Use the entity id as the message key
  • Use an outbox for safe publishing
  • Make projections idempotent
  • Keep a schema registry with compatibility rules
  • Monitor projection lag per read model
  • Rebuild a read model regularly to prove it works

Common mistakes

  • Assuming the read side is always current
  • Choosing too few partitions at the start
  • Putting secrets and personal data into events
  • Forgetting to plan for duplicates
  • Using long-lived events without a format policy
  • Letting each team invent its own event style
  • Skipping correlation ids and then being unable to trace problems
  • Adding event sourcing without a real need
Everyday analogy

A good kitchen has labeled containers, a clear order of work, and a habit of tasting before serving. None of these is fancy. Together they stop small mistakes from reaching the table.

A Complete Walkthrough: An Online Shop

A made-up example, told in plain steps.

The situation

An online shop has slow product pages and a checkout that sometimes stalls while big sales reports run. The team decides to apply CQRS to orders and products, using Kafka as the bridge.

1

Separate the write model

The order service keeps a tidy relational database that enforces rules such as stock limits and valid payment.

2

Add an outbox

Each accepted command saves the order and an “order placed” event in one transaction. A relay publishes the event to an orders topic, keyed by order id.

3

Create read models

One projection builds a customer order history in a fast document store. Another feeds a search index for products. A third fills an analytical warehouse for reports.

4

Handle the gap

After checkout, the confirmation page uses the data returned by the command, and does not query the order history immediately.

5

Monitor

A dashboard shows projection lag for each view. An alert fires when order history is more than thirty seconds behind.

6

Survive a problem

A bad deployment corrupts the search index. The team deletes it, starts a new consumer group from the beginning of the topic, and rebuilds the index while the shop keeps running.

flowchart LR
    CU["Customer"] -->|"place order"| OS["Order service"]
    OS --> ODB[("Orders database")]
    OS --> OBX["Outbox"]
    OBX --> KT[["Orders topic"]]
    KT --> H["History projection"]
    KT --> S["Search projection"]
    KT --> A["Analytics projection"]
    H --> HS[("Document store")]
    S --> SI[("Search index")]
    A --> WH[("Warehouse")]
    CU -->|"view history"| HS
    CU -->|"search products"| SI
    
FIG 16 — The online shop with CQRS and Kafka

The result: checkout is no longer disturbed by reports, product pages are fast, and new views can be added by attaching another consumer. The cost: the team now watches lag, manages event formats, and designs screens with eventual consistency in mind. It is a fair trade for a shop of this size, and a poor trade for a tiny one.

Myths and Misunderstandings

Ideas that lead teams into trouble.

Myth

“CQRS requires event sourcing.”

No. They are independent ideas that happen to work well together.

Myth

“CQRS needs two databases.”

Not always. The models can share a database at first. Separate stores come when the need appears.

Myth

“Kafka is required for CQRS.”

No. Other messaging tools or even database features can carry changes. Kafka is popular because of its durability, order, and replay.

Myth

“CQRS makes everything faster.”

It makes reads faster and scales sides separately, but adds delay between write and read, and more work to run.

Myth

“Eventual consistency means data may be wrong.”

It means data may be briefly old, and then becomes right.

Myth

“Every system should use CQRS.”

Simple systems are better served by a simple model.

Real-World Patterns

Generic pictures of how different kinds of businesses use these ideas, without naming particular companies.

Banking: balances and statements

Transfers are commands checked by strict rules on the write side. Events feed a fast balance view for mobile apps, a statement generator, and a fraud detection stream, each as a separate read model.

Ride hailing: live and historical views

Trip events flow into a live map view for drivers and riders, a billing view, and a warehouse for analysis. The live map tolerates only a few seconds of delay, while reports tolerate hours.

Logistics: tracking parcels

Scanning a parcel is a command. Scan events feed a tracking page, a delivery-time estimator, and an alerting service for delays.

Retail inventory: many stores, one truth

Stock changes are events. Store screens, online product pages, and supplier reorder systems each read their own view of the same events.

Aggregates: The Unit of Consistency on the Write Side

A small idea that explains how the write side stays correct.

What an aggregate is

An aggregate is a group of related things that must always be consistent together, treated as one unit. An order and its order lines form one aggregate: the total must always match the lines. A command changes one aggregate at a time, and the aggregate checks its own rules before agreeing.

Why it matters for Kafka

  • The aggregate id is the natural message key. Events of one aggregate land in one partition, in order.
  • One command, one aggregate. Changing two aggregates in one transaction couples them. Use events and sagas to coordinate instead.
  • Small aggregates scale better. Big ones cause many commands to compete for the same data.
flowchart LR
    CMD["Command: add item to order 7"] --> AG["Order 7 aggregate"]
    AG --> CHK{"Rules pass?"}
    CHK -->|"Yes"| EV["Event: item added to order 7"]
    CHK -->|"No"| ER["Refuse"]
    EV --> KP["Partition chosen by key: order 7"]
    
FIG 17 — One command changes one aggregate and produces events keyed by it
Everyday analogy

A shopping basket. Whatever you add or remove, the basket must always show the right total for the items inside. The basket is the aggregate. Two different customers’ baskets have nothing to do with each other, so both can be changed at the same time, in different lanes.

Handling two commands at once

What if two commands try to change the same aggregate at the same moment? A common answer is optimistic concurrency: each aggregate carries a version number. A command says “I am changing version 5”. If the aggregate is already at version 6, the command is refused and can be retried with fresh data. This avoids locking and keeps writes fast.

i
Tip

The same version number can travel in events. Projections and screens can then compare versions to know whether a read model is up to date.

Three Levels of CQRS: Start Small

You do not have to jump to the biggest version. There is a ladder.

1

Level one: separate code paths

Commands and queries use different classes and services, but the same database. There is no eventual consistency yet. It already makes the code cleaner.

2

Level two: separate read store

Queries use their own database, fed by change events. Eventual consistency arrives, and so does a big gain in read speed.

3

Level three: event-driven with many read models

Kafka carries events, many projections build many views, and replay is routine. Maximum flexibility, with maximum operational work.

LevelWhat you gainWhat you pay
OneClean code, easier reasoningLittle
TwoFast, scalable readsEventual consistency, one synchronization mechanism
ThreeMany views, replay, team independenceKafka operations, event contracts, projection management
Everyday analogy

Learning to swim: first you practice in the shallow end, then the deeper end, then the open sea. Each step adds power and adds risk. You climb only as high as you need.

The best CQRS system is the smallest one that removes your actual pain.

Consistency Promises You Can Make to Users

Turn “eventually” into something concrete.

Write down a freshness target

For every read model, agree on a number: “This view is normally less than two seconds behind, and never more than thirty.” That statement can be measured, alerted on, and understood by product owners. Without it, “eventual” can mean anything.

Different screens, different needs

ScreenAcceptable delayApproach
Confirmation after checkoutNoneShow data returned by the command
Order historyA few secondsRead model with version wait if needed
Product searchSeconds to minutesSearch index projection
Daily sales reportHoursBatch-friendly warehouse projection
Available stock before purchaseVery small or noneCheck against the write model at the moment of ordering
flowchart TB
    Q["How fresh must this screen be?"] --> N{"Must be exact right now?"}
    N -->|"Yes"| W["Read from the write side, or use the command reply"]
    N -->|"A few seconds is fine"| V["Read model plus version wait"]
    N -->|"Minutes or hours are fine"| B["Ordinary read model or warehouse"]
    
FIG 18 — Choosing freshness for each screen
Everyday analogy

A train station has a live departure board, a printed timetable, and a yearly booklet. Each is accurate enough for its purpose. You would not use the yearly booklet to catch the train leaving in five minutes.

!
Watch out

The last row deserves attention. Critical checks such as available stock or account balance limits belong to the write side, which is always current, and not to a read model, which may be behind.

CQRS Across Many Services

In larger companies, the two sides may belong to different teams.

Each service owns its writes

In a system of many small services, each service is the only writer of its own data. The orders service alone changes orders. The customers service alone changes customers. Other services that need this data do not reach into its database. Instead they listen to its events on Kafka and keep their own read models.

flowchart LR
    OS["Orders service"] -->|"order events"| K[["Kafka"]]
    CS["Customers service"] -->|"customer events"| K
    K --> SH["Shipping service: own view of orders and customers"]
    K --> BI["Billing service: own view of orders and customers"]
    K --> RC["Recommendations: own view of orders"]
    
FIG 19 — Services sharing facts through events and keeping their own views

Why this is good

  • No shared database. Each team can change its storage freely.
  • Less live coupling. A service can work even when another is down, using its last known view.
  • Clear ownership. Only the owner may write; everybody else may only read.

What to watch

  • Copies of data can drift. Monitor lag in every consuming service.
  • Event formats become contracts between teams, so agree who owns them and how they change.
  • Do not build a maze where every service depends on everyone’s events. Keep the picture readable.
Everyday analogy

Each department in a company keeps its own records and announces changes through a notice board. Other departments copy what they need into their own files. Nobody walks into another department’s cabinet.

Real-world picture: a delivery company

The orders team publishes order events. The routing team, the billing team, and the customer-app team each build their own views from them. When the customer-app team wants a new “estimated arrival” screen, they attach a new consumer group and build a new view, without asking the orders team to change anything.

Cost and Team Considerations

Technology choices are also choices about people and money.

The running cost

  • Infrastructure: Kafka brokers, extra databases for read models, and the machines running projections.
  • Storage: Read models repeat data on purpose, and event topics may be retained for a long time.
  • Operations: Someone must watch lag, handle failed events, and manage upgrades.

The skills needed

Teams need to understand messaging, ordering, idempotency, and schema evolution. These are learnable, but they take time, and mistakes in the first months are normal. Plan for learning, and start with a small, low-risk area.

Ways to lower the burden

  • Use a managed Kafka service, so brokers are someone else’s daily concern.
  • Use Kafka Connect for simple copies, rather than writing custom code.
  • Provide shared libraries, templates, and standards for events, so each team does not start from zero.
  • Write a short guide of “how we do CQRS here” and keep it alive.
Everyday analogy

A bigger restaurant earns more, but needs more staff, more training, and more rules. It is worth it when the crowds justify it. If only a few guests come, the extra staff simply stand idle.

i
Tip

Before committing, write down the specific pain that CQRS will remove, and how you will measure that it did. If you cannot, wait.

A Troubleshooting Guide: Symptoms, Causes, and Fixes

When something looks wrong, start from what you can see.

What you seeLikely causeWhat to do
Users say their change never appearedProjection lag is high, or the event never left the outboxSearch by correlation id, check outbox backlog and consumer lag
Totals in a view are too highDuplicate events counted twiceMake the projection idempotent, then rebuild the view
A view shows an order as paid before it is placedEvents for one entity arrived out of order, often from a wrong or missing keyCheck keys and partitions, apply version checks
One projection stops while others continueA bad event blocks one partition for that groupInspect the failing event, retry, or send it to a dead letter topic
New consumers cannot read old eventsRetention has expired the old dataIncrease retention for event topics, or restore from a snapshot
Consumers crash after a new event versionAn incompatible schema change was deployedRoll back the producer, and enforce compatibility rules
Queries are slow even on the read sideThe read model does not match the questionReshape the view or add an index, and do not add joins at query time
Commands time outThe write database is overloaded, or publishing blocks the handlerUse an outbox, scale the write side, and shorten transactions
flowchart TB
    P["User reports missing change"] --> A{"Did the command succeed?"}
    A -->|"No"| A1["Look at command handler logs and rule failures"]
    A -->|"Yes"| B{"Is the event in Kafka?"}
    B -->|"No"| B1["Check the outbox and relay"]
    B -->|"Yes"| C{"Has the projection consumed it?"}
    C -->|"No"| C1["Check consumer lag and errors"]
    C -->|"Yes"| D["Check the read store and the query handler"]
    
FIG 20 — Following a missing change through the pipeline
Everyday analogy

When a parcel does not arrive, you check each step in order: was it posted, did it leave the sorting center, is it on the delivery van, was it left with a neighbor? Debugging a CQRS system is the same: follow the change from step to step until you find where it stopped.

A Short Checklist Before Going Live

Confirm these before real users depend on the system.

1

Business rules are on the write side

No command depends on a read model for a critical decision.

2

Publishing is safe

An outbox or change capture prevents lost or phantom events.

3

Keys and partitions are right

Entity ids are used as keys, and the partition count leaves room to grow.

4

Projections are idempotent

Duplicates and replays have been tested.

5

Schemas are registered and protected

Compatibility rules are switched on.

6

Screens handle the gap

Every screen has a plan for the delay between write and read.

7

Monitoring and alerts exist

Projection lag, error counts, dead letters, and outbox backlog are visible.

8

Rebuild has been practiced

A read model was deleted and rebuilt successfully, with the time noted.

9

Security is in place

Commands are authorized, topics have access rules, and sensitive data is handled with care.

i
Tip

Keep this list where the whole team can see it, and review it after every incident. Each surprise in production is a new line for the list.

Everyday analogy

Pilots read a checklist before every flight, even after thousands of flights. It is not because they forget how to fly, but because the checklist catches the small things that human memory misses.

Frequently Asked Questions

Q1What does CQRS stand for?
Command Query Responsibility Segregation. It means separating the model that handles requests to change data from the model that handles requests to read data.
Q2What is the role of Kafka in CQRS?
Kafka carries the events from the write side to the read side. It stores them durably, keeps them in order per partition, lets many consumers read independently, and allows replay to rebuild read models.
Q3Do I need event sourcing to use CQRS?
No. You can keep ordinary state in the write database and still publish events for read models. Event sourcing is an optional extra.
Q4What is a command and what is an event?
A command is a request to do something and may be refused. An event is a fact that already happened and cannot be refused.
Q5What is a projection?
A program that consumes events and updates a read model, keeping a ready-to-read view up to date.
Q6What is eventual consistency?
The read side becomes correct shortly after the write side changes, but may briefly show old data in between.
Q7How do I stop users seeing old data after saving?
Return the new state in the command reply, show a pending entry on screen, wait until the read model reaches a given version, or briefly read from the write side for that user.
Q8What is the dual write problem?
Saving to a database and publishing to Kafka as two separate actions can leave them disagreeing if one fails. The outbox pattern fixes it by saving both in one transaction.
Q9How do I keep events in order?
Use the entity id as the message key, so that all events for that entity go to the same partition. Kafka preserves order inside a partition.
Q10Can duplicate events occur?
Yes, after retries or crashes. Make projections idempotent by setting values, tracking processed ids, or using version numbers.
Q11Can I rebuild a read model?
Yes. Start a new consumer group from the earliest message and let it process the history. This works as far back as Kafka retention allows.
Q12How should I choose topics?
A common approach is one topic per kind of entity, such as orders or customers, so that events about the same entity stay together and in order.
Q13Which tools build projections?
Plain consumer applications, Kafka Streams, Kafka Connect sinks, and stream SQL engines, depending on how custom the logic is.
Q14How do I change an event format safely?
Use a schema registry with compatibility rules, add optional fields with defaults, avoid removing or renaming fields, and version truly incompatible changes.
Q15What is a saga?
A sequence of local steps across services, connected by events, where each step has a compensating action to undo it if a later step fails.
Q16What should I monitor?
Projection lag for every read model, event rates, command and query latency, error counts, dead letter volume, and outbox backlog.
Q17What if a projection keeps failing on one event?
Retry a few times, then move the event to a dead letter topic and alert, unless the event must never be skipped, in which case stop and fix the cause.
Q18How does CQRS affect privacy?
Data is copied into several places and events live for a long time. Keep personal data out of events where possible, encrypt sensitive fields, and plan for deletion requests early.
Q19Can CQRS be used with only one database?
Yes. Even separating the code paths for commands and queries, with the same database underneath, counts as a first, gentle form of the pattern.
Q20Is CQRS suitable for small applications?
Usually not. Simple forms and lists are better served by a single model. Consider CQRS when reads and writes truly differ in volume or shape.
Q21How many partitions should I choose?
Enough for your expected future peak, since consumer groups cannot use more members than partitions, and adding partitions later can change which partition a key maps to.
Q22Should commands go through Kafka too?
It is possible and gives buffering, but complicates giving the user an immediate answer. Many teams send commands directly to the handler and use Kafka for events.
Q23How do I test a projection?
Feed it a known list of events, including duplicates and out-of-order cases, and check that the resulting read model matches what you expect.
Q24What is the best first step?
Pick one painful screen or report, publish events for its data with an outbox, build one projection, and compare it with the old answer before switching.
Q25What is an aggregate?
A group of related things that must be consistent together, treated as one unit by the write side. A command changes one aggregate, and its id is the natural message key.
Q26What if two commands change the same thing at once?
Use optimistic concurrency with version numbers. If the version has moved on, the later command is refused and can be retried with fresh data.
Q27Which checks must never use a read model?
Critical rules, such as available stock or balance limits, should be checked on the write side at the moment of the command, because read models may be a little behind.
Q28Can each microservice keep its own read model?
Yes, and this is common. Each service is the only writer of its own data, publishes events, and others build their own views from those events.
Q29How can I reduce the operating effort?
Use a managed Kafka service, use Connect for simple copies, share standards and templates for events, and start with one small area.
Q30How do I find out where a change got stuck?
Follow it by correlation id: first the command result, then the outbox, then the Kafka topic, then the projection’s lag and errors, and finally the read store.
Q31Why do totals sometimes come out too high?
Usually duplicate events were added twice. Make the projection set values or track processed event ids, and rebuild the view from the topic.
Q32Is CQRS the same as having a read replica of a database?
No. A replica is an exact copy of the same shape. In CQRS, the read model is deliberately a different shape, built for the questions it answers, and often lives in a different kind of store.
Q33Should read models store more data than the screen needs?
Usually no. A read model is most useful when it holds exactly what one screen or report needs, already combined and sorted. Extra fields cost storage and make the view harder to maintain. If another audience needs different fields, give it its own view.
Q34How long should event topics be kept?
Keep them at least as long as the longest time you might need to rebuild a read model or investigate a problem. For event streams that act as the source of truth, many teams keep them for a very long time, or forever, and use compaction only for latest-state topics.
Q35Who should own the event definitions?
The team that owns the write side should own the events it publishes, document them clearly, and change them only through agreed compatibility rules. Consumers may ask for changes, but must not depend on undocumented details.

Summary and Key Takeaways

Everything in one place.

CQRS splits a system into a side that changes data and a side that reads data, because these two jobs have different needs. Kafka is the natural bridge: the write side publishes events, and any number of read-side projections consume them to build views shaped for their screens. The gains are independent scaling, fast queries, freedom of storage, and the power to replay. The costs are eventual consistency, more parts to run, and discipline about events. Careful keys, an outbox, idempotent projections, schemas, monitoring of lag, and a gradual adoption plan keep the costs under control.

Key Takeaways

  • CQRS separates commands from queries, giving each its own model, code, and storage.
  • Kafka is the spine: durable, ordered per partition, readable by many groups, and replayable.
  • Events are facts in the past tense, and should be self-contained and named after business meaning.
  • Accept eventual consistency and design for it, with returned state, pending items, or version waits.
  • Use an outbox, message keys, and idempotent projections to stay safe against crashes, duplicates, and disorder.
  • Watch projection lag and event formats. They are the health and the contract of the whole system.
  • Adopt it gradually, only where reads and writes truly pull apart.