The phrase sounds formal, but the idea is simple: when two copies of the same fact disagree, which one do you believe?

The plain meaning

A system of record is the authoritative source for a piece of data. It is the place where the data is created or first captured, and the place that is considered correct when copies elsewhere differ. Other systems may hold copies for speed or convenience, but they are treated as secondary and can be refreshed or rebuilt from the record.

What a system of record must provide

The usual candidates

Traditionally, relational databases such as PostgreSQL, MySQL or Oracle serve as systems of record, because they offer strong transactions and rich queries. Document stores, key-value stores and even mainframes fill the role in other organizations. The question of this guide is whether a distributed event log, Kafka, can share or take over that role for some kinds of data.

Why the question matters more with streaming

In older architectures, each application owned one database and other systems received nightly file copies. The question of which copy wins rarely came up, because the copies were clearly labelled as stale. Real-time streaming changes this. When many systems receive updates within milliseconds, each of them starts to look like a live source, and disagreements become visible and confusing. Naming one authoritative place, and making every other copy visibly derived from it, is what keeps a fast, connected organization from arguing with itself about facts.

It also affects daily work. Support staff need to know which screen to trust when a customer disputes a charge. Analysts need to know which table to reconcile against. Auditors need to know where to look for evidence. A clearly declared record answers all of these questions with one sentence, and that clarity is often worth more than any technical feature.

To see why Kafka can play this role, you need to understand the one data structure at its core: the append-only log.

What a log is

A log is a list of records added one after another at the end, each with a position number called an offset. Records are never changed in place. New information is simply added. Anyone can read from any position and get exactly the same records in exactly the same order, again and again.

flowchart LR
    A["Offset 0: account opened"] --> B["Offset 1: deposit 500"]
    B --> C["Offset 2: withdrawal 120"]
    C --> D["Offset 3: deposit 40"]
    D --> E["Offset 4: next event will be added here"]
    

Why a log suits a record

• Order is explicit: the offset tells everyone exactly what came before and after, so there is no argument about sequence within a partition.
• Nothing is overwritten: the history of what happened is kept by default, which is what auditors and analysts want.
• Readers are independent: each reader keeps its own position, so a new system can start from the beginning without disturbing anyone.
• Replication is natural: copying a log to other machines is simply copying the same records in the same order.
• Rebuilding is possible: any view of the data can be recreated by replaying the log.

How Kafka organizes logs

Kafka groups records into topics. Each topic is split into partitions, and each partition is its own log stored on a broker and copied to other brokers. Producers append records, and consumers read them. Records with the same key go to the same partition, so all the events about one customer or one order stay in order together.

Reading without consuming

A detail that surprises newcomers is that reading a record in Kafka does not remove it. In a traditional queue, a message disappears once someone takes it. In Kafka, the log stays put and each reader simply remembers its own position. Ten different teams can read the same events at their own pace, and a reader that is down for a day resumes where it stopped. This property is what lets one record serve an entire company, since adding a new reader costs the existing ones nothing and requires no change to the writers.

Engineers have argued about this for years. The honest answer is that it depends on what you ask the record to do.

The case for

Kafka replicates data across machines, keeps it on disk, can retain it as long as you choose, gives strict ordering per key, and supports transactions across partitions. A well-known 2017 article by Jay Kreps, one of Kafka’s creators, titled along the lines of “It’s okay to store data in Apache Kafka”, argued that these properties make it a sound place for durable data. Many organizations now do so for event streams that describe their business.

The case against

Kafka is not a general-purpose database. It has no built-in way to look up one record by any field, no joins written as ordinary queries, no secondary indexes and no row-level update. If you ask “what is the current address of customer 42?”, Kafka alone cannot answer quickly. You must build or use a view for that, which is why Kafka is usually paired with other systems.

The balanced view

The most useful way to think about it is this: Kafka can be the system of record for events, the durable stream of things that happened, while other databases serve as fast, queryable views of the current state. In this design, the log is the truth and the databases are caches you can rebuild. Some teams go the other way, keeping a database as the record and using Kafka as a pipe. Both are valid, and this guide helps you know which fits.

A middle path many teams take

In practice, few organizations go all the way in either direction. A common compromise treats Kafka as the record for facts that are naturally events, such as clicks, orders placed, payments received and devices reporting, while leaving slowly changing master data, such as the list of legal entities, in a database with strong constraints. The two are linked by change-data-capture so the log still carries every change. This gives most of the benefits of an event record without forcing awkward designs onto data that is really a set of rows.

Whatever you choose, be explicit and consistent. The worst outcome is ambiguity, where two systems each believe they are the authority for the same data, because then every disagreement becomes an argument with no way to settle it.

The biggest mental shift in this whole topic is moving from storing what things are to storing what happened.

Two ways to record the same facts

A state record says what is true now: “Account 42 has a balance of 420.” An event record says what happened: “Account 42 received a deposit of 500.” “Account 42 had 120 withdrawn.” “Account 42 received a deposit of 40.” From the events you can always compute the state. From the state alone, you cannot recover the events.

flowchart LR
    E["Events in Kafka: deposit 500, withdraw 120, deposit 40"] --> V["Replay and add up"]
    V --> S["State: balance 420"]
    S -. "cannot rebuild" .-> E
    

Event sourcing

Event sourcing is the design style in which an application stores every change as an event, and treats the current state as something computed from those events. It is a natural companion to Kafka as a system of record. The events are appended to a topic, and services build their own current-state views by reading the topic.

What you gain

What you must accept

• Mistakes are corrected by adding events: you do not edit history. A wrong deposit is fixed with a correcting entry, just like in accounting.
• Views are eventually consistent: a derived database may lag a little behind the log.
• Event design is important: events must be named and shaped carefully because they live for years.
• Reading current state needs a view: you plan for it from the start.

Facts, commands and notifications

Not everything sent through Kafka is a fact worth keeping forever. A command asks someone to do something, such as “please ship this order”, and may be refused. A notification tells others that something happened, often with minimal detail. A fact, or domain event, records what did happen with all the information needed to understand it later. A record topic should hold facts. Mixing commands and facts in one topic makes history confusing, because a reader cannot tell whether an entry describes something that happened or something that was merely requested. Keep them in separate topics with clear names.

A system of record must not lose what it has been told. Kafka gives you the tools, but they must be set deliberately.

Replication: many copies

Each partition has several copies on different brokers, called replicas. One is the leader, which handles reads and writes, and the others are followers, which copy it. The number of copies is the replication factor. A replication factor of three is the common choice for important data, so two machines can fail without losing anything.

sequenceDiagram
    participant P as Producer
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    P->>L: write event
    L->>L: append to log
    F1->>L: fetch and copy
    F2->>L: fetch and copy
    L-->>P: acknowledged after enough copies exist
    

The safe settings together

Why “unclean” elections are dangerous

If the leader fails and every up-to-date follower is also gone, Kafka can either wait for one of them to return, or elect an out-of-date replica as the new leader. The second choice keeps the partition available, but it discards events that were acknowledged but not yet copied. For a system of record, availability is not worth silently losing facts, so the setting that allows it should be off.

The disk and the flush

Kafka writes to the operating system’s memory first and flushes to disk in the background, relying on replication for safety instead of flushing every message. For most systems of record, three replicas in different zones are a stronger guarantee than forcing every write to disk on one machine. Some regulated environments add flush settings for extra assurance, at a cost in speed.

A worked story of a failure that does not lose data

Suppose a topic has three replicas on brokers one, two and three, with a minimum of two in sync and acknowledgement of all. A producer writes an event, and brokers one and two confirm, so the producer is told success. A moment later, broker one, the leader, loses power. Broker two holds the event, so it becomes the new leader, and the event is still there. Broker three catches up and joins as a follower. Nothing acknowledged was lost, and nobody had to do anything. If broker two had also failed before broker three caught up, the partition would stop accepting writes rather than accept them unsafely, and it would resume when a replica returned.

This behaviour is the reason the settings work as a set. Acknowledgement of all without a minimum of two could accept writes with only the leader alive. A minimum of two without acknowledgement of all would not check at all. Only together do they form a promise.

By default, Kafka forgets old data after a while. A system of record must be told to remember, and it must be able to afford to.

The default is to expire

Most topics are created with a retention period, commonly seven days, after which old segments of the log are deleted. This suits data used as a temporary pipe. For a system of record, the retention must be long enough to match the business need, or infinite.

The cost of forever

Keeping every event forever means storage grows continuously. Brokers with large local disks become expensive, and recovering a failed broker takes longer because more data must be copied. Two developments make this practical.

Retention and the consumer who comes back late

Retention also decides how long a slow or stopped consumer can be away. If a consumer is offline for longer than the retention period, the events it missed are gone, and it can no longer catch up by reading. For a record with infinite retention this cannot happen, which is another practical benefit. For other topics, alert when a consumer’s lag approaches the retention limit, since crossing it turns a delay into permanent loss for that reader.

Some records matter only in their latest form. Compaction lets Kafka keep the newest value for each key forever without keeping every past value.

How compaction works

In a compacted topic, Kafka runs a background cleaner that looks for keys that appear more than once. It keeps the most recent record for each key and discards older ones. The result is a topic that always holds at least the latest value for every key, which is effectively a table of current state stored as a log.

flowchart LR
    subgraph Before
        A1["cust-1: address A"] --> A2["cust-2: address B"] --> A3["cust-1: address C"] --> A4["cust-2: address D"]
    end
    subgraph After
        B1["cust-1: address C"] --> B2["cust-2: address D"]
    end
    Before --> After
    

Tombstones: deleting a key

To delete a key from a compacted topic, you write a record with that key and an empty value, called a tombstone. Compaction eventually removes the earlier records for that key, and after a configurable delay, the tombstone itself is removed. This is how “delete” works in a log that never edits in place.

When to compact and when to keep everything

Many systems use both patterns together: a full-history topic of events as the true record, and a compacted topic derived from it holding the current state of each entity, ready for fast rebuilds of views.

A record that lasts for years must still be understood by programs written years later. That requires disciplined data formats.

The problem of time

Kafka stores bytes and does not care what they mean. Six months from now, an event written today will be read by a program that did not exist yet. If the meaning of a field has changed, or the field has been renamed, that program will misread the data or crash. Since a system of record never throws history away, the old and new shapes of data live side by side in the same topic.

Schemas and the registry

A schema describes the fields of an event: names, types and which are required. Formats such as Avro, Protocol Buffers and JSON Schema let you define a schema and encode data compactly. A schema registry is a service that stores every version of every schema and gives each a numeric identifier. Each message carries only that small identifier, and readers look up the full schema when they need it. Confluent’s Schema Registry is the best known, and other compatible registries exist.

Compatibility rules

The registry can refuse a new schema version if it would break readers. This is one of the most valuable protections for a long-lived record.

When a breaking change is unavoidable

Sometimes the model of the business changes so much that compatibility cannot be kept. The standard answer is to create a new topic with the new schema, and run a job that translates old events into the new form, or to keep both and let readers handle the version by name. Plan such migrations carefully, since the old topic remains the historic record.

Choosing a format

Avro is popular with Kafka because its schemas travel well and support clear evolution rules. Protocol Buffers are common in organizations that already use them for service calls, and they also handle evolution well through numbered fields. JSON Schema is friendlier to people and tools that already exchange JSON, though it is larger on the wire. Whichever you choose, the important thing is that a schema exists, is stored centrally and is checked before data is written. Pick one house style so that readers across the company do not need to handle several.

A record is only useful if you can trust the sequence of facts. Kafka’s guarantee is precise, and you need to know exactly where it stops.

What Kafka guarantees

• Order within a partition: records appear in the order they were written, and every reader sees the same order.
• Durability after acknowledgement: with safe settings, an acknowledged record survives broker failures.
• Consistent reads: consumers read only data that has been copied to all in-sync replicas.

What Kafka does not guarantee

• Order across partitions: two events with different keys may be read in either order.
• Global uniqueness rules: Kafka does not check that “no two users have the same email” at write time.
• Read-your-own-write in views: a database built from the log may not yet show an event just written.

Choosing keys wisely

The key decides the partition, and therefore what stays in order. For a system of record, the key should be the identity of the entity whose history matters: the account number, the order identifier, the customer identifier. All events about that entity go to one partition and remain in order. A poorly chosen key, such as a random value, spreads related events across partitions and destroys their order.

Enforcing rules without a central lock

Business rules such as “an account must never go below zero” are harder. Since Kafka only appends, the rule must be enforced by the process that writes the event. A common design is to have a single service own each entity, process commands for that entity one at a time, check the rule against its own view of the state, and only then append the event. Because all commands for one key land in one partition and are handled by one consumer at a time, the checks cannot race with each other.

flowchart LR
    C["Command: withdraw 100 from account 7"] --> S["Owning service reads its state for account 7"]
    S --> D{"Rule satisfied?"}
    D -- "yes" --> E["Append event: MoneyWithdrawn"]
    D -- "no" --> R["Append event: WithdrawalRejected"]
    

Idempotent handling of the same command

Because commands may be retried, the owning service should recognize a command it has already handled. A simple technique is to include a unique command identifier and remember the identifiers recently applied to each entity. If the same command arrives again, the service replies with the earlier outcome instead of appending a second event. This mirrors what payment providers do with their own request identifiers, and it turns retries from a risk into a routine event.

A record that contains duplicates or half-finished updates cannot be trusted. Kafka offers two features that protect against both.

Idempotent producers: no duplicates from retries

When a network hiccup hides an acknowledgement, the producer resends. Without protection, the same event could be stored twice. An idempotent producer labels each batch with a producer identity and a sequence number, so the broker recognizes and ignores repeats. It is enabled by default in recent client versions and should stay on for any system of record.

Transactions: all or nothing across partitions

Sometimes several events must be recorded together. A money transfer produces “debited from A” and “credited to B”, and possibly a third event on a different topic. A transaction lets a producer write to several partitions and topics as one atomic unit. Consumers configured to read only committed data either see all the events or none of them.

Consume, process, produce

Stream processing applications often read events, transform them and write results. Kafka transactions can bundle the writing of results with the saving of the read position, so that after a crash and restart, the application neither loses events nor produces results twice. This is the basis of Kafka Streams’ exactly-once processing mode.

The limits of the guarantee

Exactly-once in Kafka covers the flow from Kafka to Kafka. It cannot make an outside system, such as an email service or a payment gateway, do something exactly once. For those side effects, you still need unique identifiers and duplicate checks in the receiving system. A system of record protects the record itself, and the outside world must be handled with care.

The log records; views answer. This chapter shows how current state, search and reports are derived from the record.

The command and query split

An architecture style known as CQRS, short for command query responsibility segregation, separates the side that accepts changes from the side that answers questions. Changes are written as events to Kafka. One or more consumers read those events and build separate read-optimized stores: a relational database for lookups, a search index for text queries, a cache for speed, a warehouse for analysis.

flowchart TB
    W["Applications write events"] --> K["Kafka topics: the system of record"]
    K --> V1["Consumer builds a relational view"]
    K --> V2["Consumer builds a search index"]
    K --> V3["Consumer builds a cache"]
    K --> V4["Consumer feeds the data warehouse"]
    V1 --> Q1["Queries by account and customer"]
    V2 --> Q2["Full-text search"]
    V3 --> Q3["Fast lookups"]
    V4 --> Q4["Reports and analytics"]
    

Tools for building views

Interactive queries

Kafka Streams applications keep state in local stores, and they can expose it so other services can ask questions directly, such as “what is the current count for this customer?”. This works well for many uses but ties the query path to the running application, so many teams still prefer a separate database for heavy querying.

Eventual consistency

A view lags behind the log by a small amount, often milliseconds to seconds. After a customer changes an address, the search index might show the old one for a moment. Design your screens and processes to handle this. A common trick is to return the new value in the response to the change, so the user sees it immediately even before the view updates.

Choosing between views

Different questions call for different stores. If people ask “show me this customer’s orders”, a relational database with an index on the customer is ideal. If they ask “find products matching these words”, a search engine is the better fit. If a website needs a value in a millisecond, a cache or key-value store serves it. If analysts want to combine years of data, a warehouse or data lake is right. The record does not have to answer any of these itself, which is exactly why it can stay simple, stable and trustworthy while the views change as needs change.

Because the log keeps history, you can rebuild any view, fix any bug and answer any new question by reading it again.

What replay means

Replay means creating a new consumer, or resetting an existing one, and reading from an earlier position, even from the beginning. Because each consumer group keeps its own position, this affects nobody else. The new reader simply goes through the history and builds whatever it needs.

Common uses

Making replay practical

• Make consumers idempotent: replays mean the same event may be processed more than once, so repeated handling must be harmless.
• Watch side effects: replaying “send email” events would email customers again. Separate view-building logic from actions that affect the outside world.
• Plan for time: replaying years of data takes hours or days. Snapshots, compacted topics and parallel consumption shorten it.
• Keep event time: store the time the event happened inside the event, not only the time Kafka received it.

Replay across versions of your own code

A subtle point is that replaying history with newer code may produce different results from what the older code produced at the time. This is often the whole point, such as fixing a bug, but sometimes you need the historic result exactly as it was, for example when reproducing a past regulatory report. In that case, keep the version of the code, or the saved output, alongside the date. Treating the results of important calculations as events in their own right, written to the log at the time, gives you a faithful record of what the organization concluded then, separate from what it would conclude now.

Replication inside one cluster protects against machine failures. A true record also needs a plan for the loss of a whole site, and for human error.

Replication is not backup

Replicas copy everything, including mistakes. If someone deletes a topic or writes a wrong batch of events, all replicas obey. Replication protects against hardware trouble, while backup and recovery protect against disasters and mistakes. A system of record needs both.

Two levels of failure

Cross-cluster replication

flowchart LR
    subgraph Region A
        KA["Primary Kafka cluster"]
    end
    subgraph Region B
        KB["Standby Kafka cluster"]
    end
    KA -- "continuous copy" --> KB
    KA --> AppA["Applications in Region A"]
    KB -. "used after failover" .-> AppB["Applications in Region B"]
    

Two numbers to agree on

Business owners should decide two targets. The recovery point objective is how much recent data you can afford to lose in a disaster. The recovery time objective is how long you can be without the system. Asynchronous copying between regions gives a small nonzero recovery point but is fast and cheap. Synchronous stretch designs can reach a recovery point of zero, at a cost in write latency.

Thinking about the size of the gap

With asynchronous copying, there is always a small window between a write in the primary region and its arrival in the standby. In normal conditions this may be a fraction of a second, but during heavy load or network trouble it can grow. Monitoring that gap continuously tells you your real, current recovery point. Some organizations also keep a periodic, immutable archive of all events in object storage with locks against deletion, as a last line of defence against catastrophic mistakes or malicious action.

An unchangeable, permanent log collides with laws that say personal data must be deleted on request. This is the most delicate part of the whole subject.

The tension

Privacy laws such as the European GDPR give people the right to have their personal data erased under certain conditions. A log that keeps everything forever and never edits records seems to make this impossible. Yet organizations do meet these rules with Kafka, using several techniques. This is not legal advice, and your legal team should decide what your obligations are.

Techniques

Designing for privacy from the start

• Classify fields: mark which fields are personal, so tools and reviews can treat them specially.
• Minimize: put only what is needed into events. A fact you never stored cannot leak or need deletion.
• Encrypt selected fields: field-level encryption protects personal data even from people with broad read access to a topic.
• Track where copies go: deletion must also reach views, search indexes and warehouses. The record helps by letting you see every consumer.
• Document your approach: auditors ask how deletion requests are honoured, and a clear design is your answer.

A system of record holds the most valuable and sensitive data an organization has. It must be protected accordingly.

The layers of protection

Governance: who owns the record?

Technology alone does not make a good record. You also need clear ownership. Every record topic should have a named owning team, a documented schema, a stated retention and deletion policy, and a description of who may read it and why. Data catalogs and lineage tools help by showing where each topic comes from and which systems consume it. Without this, “the record” becomes a pile of topics that nobody dares to touch.

Separating duties

A useful principle from finance applies well here: the people who can change the system should not be the only people who can approve or audit the changes. Give operators the ability to run the platform without granting them the ability to read every sensitive payload. Give application teams write access to their own topics only. Let a separate group review access requests. Small steps like these limit the harm from mistakes and from stolen credentials alike.

If Kafka is the record, then knowing its health, at every moment, is not optional. Small problems in a record become large problems years later.

The signals that matter most

Beyond metrics: checking correctness

Health metrics tell you the platform is running. They do not tell you the data is right. Additional checks are valuable for a record: count events written at the source and events read at the destination and compare, run periodic reconciliation between views and the log, and validate a sample of events against their schemas and business rules. A steady reconciliation job that says “the view and the record agree” gives confidence that no quiet corruption exists.

Runbooks and drills

Each alert on a record should link to a short page saying what it means, how serious it is and what to try first. For a record, add the questions that matter to the business: is any data lost, is any data unavailable, and who must be told. Practise these runbooks in drills where a broker is stopped or a zone is disconnected on purpose in a test environment. The first time an engineer follows a runbook should never be during a real incident at night.

A record grows every day. Planning for its size, its speed and its recovery time is part of the design.

Storage planning

With long or infinite retention, storage needs equal the event rate times the event size times the retention time, times the replication factor for local disks. Compression and tiered storage lower this considerably. Estimate growth for several years, and remember that recovery and rebuild times increase along with the amount of stored data.

Recovery time

When a broker fails and is replaced, the new one must copy all the data of its partitions from other brokers. With many terabytes on each broker, this can take hours, during which the cluster has less redundancy. Tiered storage helps, since only the recent local part must be copied, and older data stays in object storage.

Partition planning

Partitions cannot be reduced later, and increasing them changes key placement. For a record where order per key matters, choose the partition count generously up front, based on your expected peak load and the number of parallel consumers you may need in the future, while avoiding extreme numbers that strain the cluster.

Replay speed

Rebuilding a view means reading a large history quickly. Reads of old data come from disk, or from object storage under tiered storage, and can compete with live traffic for resources. Run large rebuilds with quotas or on a separate consumer cluster, so they do not slow down live writers, and read partitions in parallel using enough consumer instances.

Keeping the cost visible

Because a record never shrinks, its cost belongs on a dashboard that leaders can see. Track total stored data, growth per month, the share on fast disks versus object storage, and the cost per topic. Topics that grow unexpectedly, perhaps because a producer began sending duplicate or oversized events, then show up early. Reviewing these numbers each quarter, together with the teams that own the topics, keeps the record affordable and keeps surprises away.

Teams that have run a Kafka-based record for years repeat certain designs and avoid certain mistakes.

Patterns worth copying

Anti-patterns to avoid

A compact list of habits that keep a Kafka-based record trustworthy, and traps that catch many teams.

Mistakes teams meet first

A decision checklist before you commit

Public talks and articles from well-known organizations show the idea in practice. Details may have changed since they were published.

Short answers to questions that newcomers and experienced engineers ask often.

If you keep only one page of this guide, keep this one.

A system of record is the place whose version of the truth wins. Kafka’s replicated, ordered, append-only log can play that part for event data, provided you configure it for durability, keep data as long as needed, control the shape of events with schemas, and plan for recovery, privacy and security. Databases, search indexes and caches then become views that can be rebuilt from the log. The reward is flexibility, history and replay. The price is discipline.