Most applications store only the latest value of things. That is simple and useful, but it quietly throws away a lot of valuable information.

How most systems store data today

In a typical application, a database has a table for customers, a table for orders, a table for products. Each row shows the current state. When a customer changes their address, the old address is replaced by the new one. When an order moves from “packed” to “shipped”, the word in the status column is overwritten. This style is often called CRUD, which stands for create, read, update and delete. It is easy to understand, and for many applications it is exactly the right choice.

What gets lost

The trouble is that every update destroys the previous value. After the change, the system can no longer answer questions such as these:

• What was the customer’s address last month, when that parcel was sent?
• How many times did this order change its delivery date before it finally shipped?
• In which order did the changes happen, and who made each one?
• What would the numbers look like if we had applied a different discount rule from the start?

Teams often try to patch this by adding audit tables, history columns or logging files. These help, but they are extra pieces added on the side, and they can drift away from the real data. If the audit table says one thing and the main table says another, which one is right?

flowchart LR
    A[Address is Pune] --> B[Customer moves]
    B --> C[Row overwritten with Mumbai]
    C --> D[Pune is lost forever]
        

Event Sourcing flips the usual design. Instead of storing the current state and forgetting how it got there, you store the changes and calculate the state.

Event Sourcing is a way of building software in which every change to the system is recorded as an event, a small, permanent record of something that has happened. These events are stored in the order they happened and are never edited or deleted. The current state of anything, such as an account balance or the status of an order, is not stored as the main truth. It is worked out by reading the events and applying them one after another.

The word “sourcing” here means that events are the source of truth. If you lose every table, every cache and every screen of the application, but you still have the events, you can rebuild everything. If you keep all the tables but lose the events, you have lost the real history.

A tiny example, in words

Imagine a shopping basket. With the traditional style, the database has one row saying the basket contains three items. With Event Sourcing, the record looks like this list:

1. Basket was created.
2. A book was added.
3. A pen was added.
4. A notebook was added.
5. The pen was removed.
6. A pencil was added.

To know what is in the basket now, you read the list from the top and keep track: book, notebook, pencil. Three items. The same three items as before, but now you also know the whole story, including the pen that was added and then taken away.

flowchart LR
    E1[Basket created] --> E2[Book added] --> E3[Pen added] --> E4[Notebook added] --> E5[Pen removed] --> E6[Pencil added]
    E6 --> S[Current state: book, notebook, pencil]
        

A handful of words appear again and again in Event Sourcing. Learn them once and the rest of the guide becomes easy to follow.

People often mix up these three ideas. Keeping them apart makes designs much cleaner.

Commands ask, events tell

A command is written in the present or future, like a request: “Place this order”, “Withdraw 500 rupees”. It expresses an intention. The system may say no, because the stock has run out or the balance is too low. An event is written in the past tense: “Order was placed”, “500 rupees were withdrawn”. It states something that is now a fact. Nobody can say no to a fact.

Events versus general messages

Kafka carries many kinds of messages: notifications, requests, sensor readings, log lines. Not every message is a business event in the Event Sourcing sense. In Event Sourcing, the events are the complete and authoritative history of your business objects. A message saying “please refresh your cache” is just a signal. A message saying “the customer changed their delivery address” is a business event that belongs in the permanent record.

You do not need to be an expert, but a few Kafka ideas matter a lot for this topic. Here they are in plain words.

The log at the heart of Kafka

Kafka stores records in topics. Each topic is divided into partitions, and each partition is an append-only log: new records are added only at the end, and each record receives a position number called an offset. Existing records are not edited. Producers write to the end, and consumers read from any position they like, and each consumer keeps track of its own position.

Three features that matter here

flowchart LR
    P[Producer] -->|same key, same partition| O0
    subgraph LOG[One partition, append only]
        O0[Offset 0] --> O1[Offset 1] --> O2[Offset 2] --> O3[Offset 3]
    end
    O3 --> C1[Reader A at offset 3]
    O1 --> C2[Reader B at offset 1]
        

Event Sourcing needs a place that stores ordered facts durably and lets many readers replay them. That description sounds a lot like Kafka.

The match, point by point

Events flowing out to everyone who cares

Another great benefit is that Kafka is not only a store but also a delivery system. When a new event is appended, all the interested programs can pick it up almost immediately: one program updates a search screen, another sends an email, another updates a report. The producer does not need to know who they are. This is called loose coupling, and it makes it easy to add new features without touching the original service.

Let us follow one action, from the moment a user clicks a button to the moment a screen shows the new result.

sequenceDiagram
    participant U as User
    participant S as Service
    participant K as Kafka topic
    participant V as Balance view
    U->>S: Command withdraw 500
    S->>K: Read past events for the account
    S->>S: Check rules
    S->>K: Append event MoneyWithdrawn
    K-->>V: New event delivered
    V-->>U: Updated balance shown
        

The quality of an event-sourced system depends mostly on the quality of its events. Well-shaped events stay useful for years; poorly shaped events cause pain for years.

Name events after business facts

A good event name uses the language of the business, in the past tense: “OrderPlaced”, “PaymentReceived”, “ParcelDelivered”, “AddressChanged”. Avoid vague names such as “OrderUpdated” or “DataChanged”. A vague name hides the reason for the change. “OrderUpdated” could mean the address changed, the item changed or the order was cancelled. Separate, clear events keep the meaning visible.

Capture the reason, not only the values

Compare two ways of recording the same change. One says only “Price is now 120”. The other says “Price was reduced from 150 to 120 because of the festival discount, by the pricing team”. The second is far more useful for explanation, audit and later analysis. Capture what changed, who or what caused it, when it happened, and any reason that the business would care about later.

Include what readers need

Each event should carry enough information that a reader can understand it without calling back to another system, especially because the other system may have changed since. Typical parts of an event are:

• The identity of the entity, such as the order number.
• The event type and the version of the event format.
• The time it happened.
• The business details, such as items, amounts and addresses.
• Information about the cause, such as the user or service responsible.

Keep events small and focused

One event should describe one thing that happened. Avoid huge events that carry the full picture of the entity and everything around it. Very large events cost storage and network time, and they blur the meaning. On the other hand, avoid splitting one business fact into many tiny events that only make sense together.

Where you put the events, and which key you give them, decides whether order is safe and whether the system can grow.

The key is the entity identity

The single most important rule is this: use the identity of the entity as the message key. If the entity is an account, the key is the account number. If it is an order, the key is the order number. Kafka then sends all events with the same key to the same partition, in the order they were written. That gives you what you need: for any one account, the events are strictly ordered. Between two different accounts, order does not matter, so they can live in different partitions and be processed in parallel.

One topic per entity type

A common and simple design is one topic for each kind of entity: an “orders” topic, an “accounts” topic, a “baskets” topic. Inside, the events of many orders are mixed together, but each order stays in its own partition because of the key. Some teams instead create one topic per event type, but that makes it harder to keep the order of different kinds of events for the same entity. Keeping all events of one entity type in one topic protects the story of each entity.

flowchart TB
    subgraph T[Orders topic]
        direction LR
        P0[Partition 0: orders 11, 14, 19]
        P1[Partition 1: orders 12, 15, 17]
        P2[Partition 2: orders 13, 16, 18]
    end
    A[Events with key order 11] --> P0
    B[Events with key order 12] --> P1
    C[Events with key order 13] --> P2
        

Choosing the number of partitions

More partitions allow more parallel readers, but the count is awkward to change later, because changing it can change which partition a key belongs to. That would break the order guarantee for existing keys during the change. It is wise to plan for future growth at the start, choosing a comfortable number rather than the smallest possible one.

This is the heart of the idea: state is a calculation over events, and the calculation can be repeated whenever you wish.

Fold the events into a state

To rebuild the state of one entity, start with an empty state and apply each event, one by one, in order. “Account opened” creates the account with zero balance. “Money deposited” adds to the balance. “Money withdrawn” subtracts. When you reach the last event, you have the current state. Programmers call this folding or reducing the events, but the idea is just running total arithmetic.

Rebuild for the whole system

The same idea works for the whole system. A brand new service can start reading the topic from the very first offset, apply every event to its own storage, and in a short time it is up to date, holding exactly the picture it needs. The old services are not affected at all.

flowchart LR
    K[(Orders topic: full history)] --> R1[Reader: current order status]
    K --> R2[Reader: daily sales report]
    K --> R3[Reader: search index]
    K --> R4[New reader added next year]
        

Time travel

Because events carry time, you can stop the replay at any moment and see the state as it was then. What was the balance last Tuesday at noon? What did the basket contain when the customer left the site? Just apply events up to that point. Traditional storage can answer these only if someone thought to keep history in advance.

Fixing mistakes

Suppose you discover a bug in the code that calculates a view, so the view has been wrong for two weeks. In a traditional system you might need to repair rows by hand, guessing what the right values should be. In an event-sourced system you fix the code, delete the wrong view, and replay the events. The corrected code produces the correct result from the same facts.

Replaying ten events is instant. Replaying ten million is not. Snapshots keep things quick as history grows.

The problem of long streams

An account that has existed for twenty years may have thousands of events. Reading all of them every time a customer withdraws money would be slow and wasteful. In practice most entities do not have such long streams, but the ones that do need a plan.

What a snapshot is

A snapshot is a saved copy of an entity’s state at a certain point, together with a note of which event it reached. To get the current state, you load the latest snapshot and then apply only the events that came after it. A snapshot is only a shortcut. It is never the truth. If it is lost or found wrong, you can throw it away and rebuild it from the events.

flowchart LR
    E1[Events 1 to 9000] --> SN[Snapshot at event 9000]
    SN --> E2[Events 9001 to 9040]
    E2 --> ST[Current state]
        

Where to store snapshots

Snapshots can live in a separate topic, in a small database or in a cache. A popular Kafka-friendly choice is a separate topic in which the key is the entity identity and only the most recent value is kept, a feature called log compaction. With compaction, Kafka eventually removes older records that have a newer record with the same key, keeping at least the latest one. That is a perfect fit for “the latest snapshot per entity”.

A record of facts is only trustworthy if events are written once, in the right order, and only when the rules allow.

The two-writers problem

Suppose two people try to withdraw the last 500 rupees of an account at almost the same moment. Both programs read the events and see a balance of 500. Both decide the withdrawal is fine. Both write an event. The account is now overdrawn, though each program followed the rules as it saw them. This is called a concurrency conflict.

Ways to avoid it

Writing exactly once

Network problems can cause a producer to retry a write and accidentally store the same event twice. Kafka offers an idempotent producer setting that makes retries safe by detecting duplicates within a partition. When a single action must write events to more than one topic together, or read and write as one unit, Kafka also offers transactions, which make a group of writes succeed or fail as a whole. Readers can be told to see only committed data. Together, these give strong protection against duplicates and half-finished actions.

Event Sourcing is often used together with another pattern, CQRS, and the two fit like a lock and key.

What CQRS means

CQRS stands for Command Query Responsibility Segregation. In simple words, it means that the part of the system that changes things and the part that reads things are separated and shaped differently. The writing side receives commands, checks rules and produces events. The reading side holds views that are shaped for fast and simple questions.

Why they go together

With Event Sourcing, the event log is excellent for recording history but awkward for questions like “show me all orders from customers in Delhi this week”. Reading side views, built by replaying events into a search index or a reporting table, answer such questions quickly. Each view is shaped for its audience: a screen, a report, a search box. You can have as many as you like.

flowchart LR
    U1[User action] --> CMD[Write side: check rules]
    CMD --> LOG[(Event topic in Kafka)]
    LOG --> V1[Read view: order screen]
    LOG --> V2[Read view: search index]
    LOG --> V3[Read view: reports]
    U2[User question] --> V1
    U2 --> V2
        

The price: eventual consistency

Read views are updated a short moment after the event is written. For a brief time, a user might change something and then see the old value on screen. This is called eventual consistency. It is usually a matter of milliseconds to seconds, but the design must expect it. Common solutions include showing the user a message such as “your change is being processed”, or having the interface remember the change it just made until the view catches up.

You do not have to build all the reading machinery by hand. Some tools around Kafka are made for turning events into views.

Turning a stream into a table

A helpful idea in the Kafka world is the stream-table duality. A stream is a list of changes. A table is the latest value per key. They are two views of the same thing: if you apply all the changes in a stream, you get a table, and if you record every change to a table, you get a stream. Event Sourcing uses exactly this idea. The event topic is the stream, and each view is a table you get by applying the stream.

Helpful tools

Events last forever, but your understanding of the business does not stand still. Handling change is one of the trickiest parts of the whole approach.

Why change is hard here

In a traditional database, you can alter a table and update all rows to the new format. With Event Sourcing, old events are never rewritten. So the system will hold events written under old rules, side by side with new ones, and your code must understand them all. A five-year-old event may still be read today.

Schemas and versions

The answer starts with a schema, a written description of what fields an event has. A schema registry stores these descriptions and checks that new versions stay compatible with old ones. Two kinds of compatibility matter: new readers should be able to read old events, and, where possible, old readers should be able to read new events. Adding an optional field with a sensible default is usually safe. Removing a field, renaming one or changing its meaning is dangerous.

Ways to handle bigger changes

Several patterns use events and Kafka but are not the same thing. Mixing them up leads to confusing designs.

Event-driven is not the same as event-sourced

Many systems send events between services, yet each service still keeps its own current-state database as the truth. That is event-driven design. It is very common and very useful. It becomes Event Sourcing only when the events are the truth and the state is derived.

Change data capture and the outbox

Change data capture watches a database and turns each row change into a Kafka record. The events describe database changes, such as “row 15 column status changed”, which are technical rather than business-level. The outbox pattern writes a business event into a special table in the same database transaction as the change, and a separate process publishes it to Kafka, which avoids losing events between the database and Kafka. Both are helpful when you cannot move fully to Event Sourcing, and they are common stepping stones.

flowchart LR
    subgraph ES[Event Sourcing]
        A1[Events are the truth] --> A2[State is derived]
    end
    subgraph ED[Event-driven design]
        B1[Database is the truth] --> B2[Events tell others]
    end
        

Event Sourcing is powerful, but it is not free. A fair look at both sides helps you decide where it belongs.

When it is a good fit

Event Sourcing shines where history itself has value: banking and payments, order and delivery tracking, inventory movements, insurance claims, booking systems, and any area with strong audit needs or complex rules that change over time.

When it is not

For simple data where nobody cares about history, such as a list of user preferences or a small admin table, plain storage is simpler and cheaper. Applying Event Sourcing everywhere is a classic case of using a heavy tool for a light job.

An unchangeable history sounds wonderful until a customer asks you to delete their personal data. This chapter explains the tension and the common ways to handle it.

Keeping events for as long as needed

By default, Kafka topics delete old records after a time or size limit. For an event source of truth, you normally set retention to keep records for a very long time or forever, and you plan storage for that growth. Kafka can also move older data to cheaper storage in some setups, which keeps long histories affordable. Please check what your version and platform support.

The privacy problem

Laws in many places give people the right to have their personal data erased. But events are meant to be permanent. If a name or phone number sits inside thousands of old events, you cannot simply delete a row. Several practical approaches exist:

flowchart LR
    EV[Event: order placed, customer id 77] --> ID[Customer store: name and address]
    EV --> KEEP[Event stays forever]
    ID --> DEL[Delete on request]
        

Some things become easier with events, and a few become different. Knowing both saves time.

Testing becomes a story

A neat way to test an event-sourced rule is to describe it as a story: “Given these past events, when this command arrives, then these new events should be produced” or “then the command must be refused”. Because events are plain facts, such tests are easy to read, even for non-programmers, and they double as documentation of business rules.

Debugging by replay

When something goes wrong in production, you can copy the events of the affected entity into a test environment and replay them slowly, watching how the state changes step by step. This is very powerful: the bug happens again on your screen, with real data, without touching the live system.

Careful with side effects

The one danger of replay is that a reader may do things in the outside world, such as sending emails or charging cards. If you replay events into that reader, those actions could happen again. Keep the parts that build views separate from the parts that act on the world, and make sure the acting parts remember what they have already done, or are switched off during a replay.

Since events are your source of truth, the health of the topics matters more than in a typical messaging setup.

What to watch

• Replication health: under-replicated partitions are a risk to your only copy of the truth. Alert early.
• Consumer lag: how far each view is behind the newest event. Growing lag means stale screens for users.
• Retention settings: confirm that event topics are not accidentally set to delete old data.
• Topic growth: track size and plan storage and partition needs ahead of time.
• Schema compatibility failures: a rejected schema change is a warning that a release would have broken readers.
• Replay duration: measure how long it takes to rebuild a view, so you know your recovery time before you need it.

Protecting the source of truth

Use a sensible replication factor, spread replicas across failure zones, restrict who can delete topics or change retention, and consider a backup or a linked second cluster for disaster recovery. Treat the event topics like the crown jewels: guarded, monitored and rehearsed.

Ideas become clearer when we see where they are used. Here are a few everyday examples.

A short list of habits that keep event-sourced systems healthy, followed by mistakes seen again and again.

Good habits

• Start small: apply Event Sourcing to one area where history clearly matters.
• Name events in the language of the business, in the past tense.
• Always use the entity identity as the key.
• Version every event and use a schema registry with compatibility checks.
• Keep personal data out of events, or encrypt it per person.
• Keep event topics on long retention, and never compact them.
• Build views as disposable: be ready to delete and rebuild.
• Practise a full replay before you need one.

Short answers to the questions beginners ask most often.

It is a way of storing every change as a permanent event and calculating the current state from those events, instead of storing only the latest value.

Kafka keeps ordered, append-only logs, retains data for as long as you choose, allows many independent readers, and can replay from any point. Those are exactly the things Event Sourcing needs.

No. It gives strong building blocks, but you still design events, rules, version checks, snapshots and views. Some teams use dedicated event stores, and others use Kafka with a small layer of their own logic.

The identity of the entity, for example the account or order number. This keeps all events of one entity in one partition, in order.

Not in the normal way. Mistakes are fixed by adding a correcting event. Deleting personal data is handled by design choices such as keeping it out of events or encrypting it with a per-person key.

A snapshot is a saved state at a point in time that shortens replays. It is optional, and it is only a shortcut. The events remain the truth.

Not if you want the full history, because compaction removes older records with the same key. It suits snapshot or latest-value topics.

Route commands for an entity to a single handler at a time, and check that the entity has not changed since it was read, retrying if it has. Kafka partitioning helps, but the version check needs to be part of your design.

Views are updated a short time after the event is written, so a screen may briefly show old data. Design the user interface to expect this. It is a normal trade-off and usually lasts a very short time.

No. In event-driven design services send events to each other but keep their own current-state databases as the truth. In Event Sourcing the events themselves are the truth.

Prefer adding optional fields or new event types, use a schema registry with compatibility rules, and translate old events to the newest shape when reading them. For huge changes, copy the stream to a new topic.

No. Use it where history, audit or complex changing rules give real value, and use ordinary storage elsewhere.

A short recap of the whole journey.

Event Sourcing keeps every change as a permanent, ordered fact, and calculates the current state from those facts, like a bank statement from which the balance is derived. Kafka fits this well: its partitions are ordered append-only logs, records can be kept as long as needed, many readers can follow independently, and any reader can replay from the start. Good designs name events after business facts, use the entity identity as the key, carry reasons and causes, and version their formats. State is rebuilt by folding events, snapshots shorten long replays, and views are disposable tables built for particular purposes, often in a CQRS style with eventual consistency. Writers must be protected against conflicts and duplicates through single-writer handling, version checks, idempotent producers and transactions. Privacy needs early planning through identifiers, per-person encryption or careful rewrites. Used where history really matters and watched with proper monitoring, Event Sourcing gives audit, insight and freedom to grow. Used everywhere without need, it adds cost for little gain.