Before meeting the solution, feel the problem that makes it necessary.

Stateless versus stateful processing

Some stream processing is stateless: each record is handled entirely on its own. Converting a temperature from Celsius to Fahrenheit, or dropping records that fail a filter, needs no memory of past records. If the application crashes and restarts, it simply carries on with the next record, and nothing is lost.

Much more useful processing is stateful. Counting page views per user, keeping the latest balance per account, joining an order with the customer’s profile, or computing an average over the last five minutes all require the application to remember something about the past. That memory is called state, and it has to be stored somewhere.

Why local storage

Reaching across the network to a remote database for every single record would be slow, and it would make the processing speed depend on that database. So stream processing libraries keep state next to the code, in an embedded store on the local disk, or in memory. A lookup then takes microseconds. This design is fast and scales well, because each application instance owns the state for only its share of the data.

The danger of local storage

Local state has a fatal weakness: it lives and dies with the machine. If the machine fails, the disk may be lost. If the cluster manager moves the work to another machine, the state stays behind. If the application is scaled out, some state must move to the new instance. Without a plan, any of these events means the counts restart at zero or the lookup table is empty, and the results downstream are silently wrong.

A changelog is not a copy of the state. It is the story of how the state came to be.

State as the sum of its history

Consider a table that holds the number of visits per user. At any moment, the table shows only the current numbers: Asha has 7, Bilal has 3. But there is another way to describe the same table: as a list of events that produced it. “Asha goes to 1, Bilal goes to 1, Asha goes to 2, Asha goes to 3, Bilal goes to 2”, and so on. If you replay that list from the start, you arrive at the current table. The list is the changelog.

flowchart LR
    A["Event: Asha visits"] --> T1["Table: Asha 1"]
    T1 --> C1["Changelog record: Asha = 1"]
    B["Event: Bilal visits"] --> T2["Table: Asha 1, Bilal 1"]
    T2 --> C2["Changelog record: Bilal = 1"]
    D["Event: Asha visits"] --> T3["Table: Asha 2, Bilal 1"]
    T3 --> C3["Changelog record: Asha = 2"]
    C1 --> LOG["Changelog topic: Asha=1, Bilal=1, Asha=2"]
    C2 --> LOG
    C3 --> LOG
        

The stream-table duality

This idea is often called the duality of streams and tables. A stream of changes can be turned into a table by keeping the latest value for each key. A table can be turned back into a stream of changes by emitting a record every time a row changes. They are two views of the same information, and each can be rebuilt from the other. A changelog topic is simply the stream view of a state store, stored durably in Kafka.

What each record contains

A changelog record carries the key of the row that changed and the new value for that key. It does not say “add one”. It says “the value for Asha is now 2”. This choice is important, because it makes each record self-contained. To find the current value of a key, you only need the latest record for it, and older records for the same key are redundant. That property is what makes compaction, explained later, possible.

Each term below is defined once here and then used consistently for the rest of the tutorial.

The design combines the speed of local storage with the safety of a replicated log.

Two copies, two jobs

Every stateful task keeps its state in two places. The local state store is the working copy: it answers reads and writes in microseconds and lives on the instance’s disk or in memory. The changelog topic is the durable backup: it lives in the Kafka cluster, replicated across brokers, and survives the loss of any instance. Each time the task updates the local store, it also appends the update to the changelog.

flowchart TD
    subgraph Instance["Application instance"]
        IN["Input record"] --> PR["Processing logic"]
        PR --> ST["Local state store: fast reads and writes"]
        ST --> CL["Changelog writer"]
    end
    CL --> TOP["Changelog topic in Kafka: replicated, compacted"]
    TOP --> RES["After failure: another instance restores the store from the topic"]
    RES --> ST2["Rebuilt local store on new instance"]
        

Why this arrangement is smart

Several parts of the stream library and of Kafka cooperate to keep the local store and the changelog in step.

flowchart LR
    subgraph Inst["Stream application instance"]
        TASK["Stream task for partition 2"]
        STORE["State store: embedded database on disk or in memory"]
        WRAP["Logging layer around the store"]
        COL["Record collector: producer"]
        RC["Restore consumer"]
        CHK["Checkpoint file"]
    end
    subgraph Cluster["Kafka cluster"]
        CLT["Changelog topic, partition 2"]
        OUT["Output topics"]
        OFF["Offsets topic"]
    end
    TASK --> WRAP
    WRAP --> STORE
    WRAP --> COL
    COL --> CLT
    COL --> OUT
    COL --> OFF
    CLT --> RC
    RC --> STORE
    RC --> CHK
        

The roles, one by one

Every changelog topic follows the same conventions for its name, partitions, and settings.

The name

A changelog topic is created automatically by the stream library, and its name follows a fixed pattern: the application ID, then the name of the state store, then the word changelog, all joined by hyphens. For an application called page-views-app with a store named views-per-user, the topic becomes page-views-app-views-per-user-changelog. Because the application ID is part of the name, two applications never share changelog topics by accident, unless someone gives them the same ID.

The partitions

The changelog has exactly as many partitions as the input the store is built from. If a task processes partition 2 of the input, its store’s changelog records go to partition 2 of the changelog. This alignment is deliberate. It means one task owns one input partition and one changelog partition per store, and moving a task means moving a matched set of partitions. The stream library also checks these counts at startup, and refuses to run if the changelog has a different number of partitions than expected.

flowchart LR
    subgraph In["Input topic: 3 partitions"]
        I0["Partition 0"]
        I1["Partition 1"]
        I2["Partition 2"]
    end
    subgraph Tasks["Three tasks"]
        T0["Task 0"]
        T1["Task 1"]
        T2["Task 2"]
    end
    subgraph CL["Changelog topic: 3 partitions"]
        C0["Partition 0"]
        C1["Partition 1"]
        C2["Partition 2"]
    end
    I0 --> T0
    I1 --> T1
    I2 --> T2
    T0 --> C0
    T1 --> C1
    T2 --> C2
        

The settings

What a record looks like

The key of each changelog record is the serialized key of the store row. The value is the serialized value, which in many store types is stored together with a timestamp so that time-aware operations remain correct after restoration. The timestamp of the changelog record itself is normally taken from the input record that caused the update, so the changelog remains a faithful timeline of the input. A record with an empty value is a tombstone, which signals a deletion.

Follow one input record as it changes the state and leaves a footprint in the changelog.

sequenceDiagram
    participant In as Input partition
    participant T as Task
    participant S as Local store
    participant P as Record collector (producer)
    participant C as Changelog partition
    In->>T: Page view for Asha
    T->>S: Get Asha
    S-->>T: 6
    T->>S: Put Asha = 7
    S->>P: Forward change (Asha, 7)
    P->>C: Append record (Asha, 7)
    C-->>P: Acknowledged after replication
    T->>T: Commit interval reached
    T->>S: Flush store
    T->>P: Flush producer, commit input offsets
        

Batching and the commit interval

The library does not wait for a changelog acknowledgement after every single update. Records are batched by the producer and sent in the background, and acknowledgements are awaited when the task commits. The commit interval therefore controls how much work is in flight at any time. A longer interval allows bigger batches and higher throughput, and a shorter interval gives smaller batches and tighter durability.

Caching before the changelog

Stateful operations such as aggregations often place a small in-memory cache in front of the store. If the same key is updated many times between commits, only the last value needs to reach the store and the changelog. This reduces write volume considerably for hot keys, and it also reduces the number of output records emitted downstream. The cache is flushed at each commit, so the changelog always catches up at commit boundaries.

Without compaction, a changelog would grow forever. With it, the log stays roughly as large as the live state.

The growth problem

If Asha’s counter is updated a million times, the changelog holds a million records for the key Asha. Yet only the last one matters for restoring the state. Keeping all million would waste disk space and make restoration slow, since a restoring instance would have to read every one of them. A plain time-based retention would not help either, because deleting old records could delete the only record for a key that never changes.

What compaction does

Log compaction is a broker background process that scans closed log segments and, for each key, keeps only the most recent record, removing older ones. The log’s offsets are preserved for the records that remain, so consumers still see a consistent order, but many offsets simply no longer exist. The end result is a log that holds, at minimum, the latest value for every key that has not been deleted.

flowchart TD
    subgraph Before["Before compaction"]
        B1["offset 0: Asha = 1"]
        B2["offset 1: Bilal = 1"]
        B3["offset 2: Asha = 2"]
        B4["offset 3: Chen = 1"]
        B5["offset 4: Asha = 3"]
        B6["offset 5: Bilal = 2"]
    end
    Before --> CMP["Log cleaner keeps latest record per key"]
    CMP --> After
    subgraph After["After compaction"]
        A1["offset 3: Chen = 1"]
        A2["offset 4: Asha = 3"]
        A3["offset 5: Bilal = 2"]
    end
        

What compaction guarantees and what it does not

The tuning knobs

State is not only created and updated. Sometimes keys must disappear, and the changelog has to say so.

What a tombstone is

When the application deletes a key from the store, the logging layer appends a record with that key and an empty value to the changelog. This is a tombstone. It tells any reader replaying the changelog that the key is gone. During restoration, the library sees the tombstone and removes the key from the rebuilt store, exactly as the original deletion did.

The life of a tombstone

Deletions and unbounded growth

A store whose keys are never deleted grows without limit, and so does its changelog. An application that tracks one entry per user session, for example, accumulates entries for every session ever seen unless something removes them. Good designs remove entries when they are no longer needed, by emitting tombstones or by using windows that expire on their own. Without this discipline, the changelog and restoration time grow steadily for months until they become a problem.

Time-based state has a natural expiry, and the changelog must know about it.

Windows add time to the key

A windowed aggregation, such as “page views per user per five minutes”, stores one row for each user and each window. The store’s key is really the pair of user and window start time. The changelog records these composite keys, so every window of every user is a separate key with its own latest value.

Why compaction alone is not enough

Windows end. After a window closes and its allowed lateness passes, its row will never change again, and nobody needs it after the retention period. But compaction only removes superseded records, not finished ones. If the changelog used compaction alone, closed windows would stay forever, and the log would grow with every new window.

flowchart TD
    A["Window row updated as events arrive"] --> B["Window closes, late arrivals still allowed for a grace period"]
    B --> C["Window becomes final"]
    C --> D{"Older than window retention plus margin?"}
    D -->|"No"| E["Row stays in store and changelog"]
    D -->|"Yes"| F["Local store drops the row"]
    D -->|"Yes"| G["Changelog segments older than the retention are deleted by the broker"]
        

Compact and delete together

For windowed stores, the changelog uses both cleanup policies at once: compaction to remove superseded values, and time-based deletion to remove old windows. The retention time is set to the window retention of the store plus an extra margin, by default about a day, so that the changelog never deletes data the local store might still legitimately need. This is a careful balance: too short, and a restore could lose windows that still matter, and too long, and the log holds pointless history.

This is the moment the changelog earns its keep. Follow it from the failure to the first processed record.

When restoration happens

The restoration steps

sequenceDiagram
    participant App as Instance
    participant CP as Checkpoint file
    participant B as Broker
    participant S as Local store
    App->>CP: Read last restored offset
    CP-->>App: Offset 4,200,000 or none
    App->>B: Get end offset of changelog partition
    B-->>App: End offset 4,350,000
    loop Until end offset reached
        App->>B: Fetch next batch from changelog
        B-->>App: Records
        App->>S: Write keys and values, delete for tombstones
    end
    App->>CP: Save new checkpoint
    Note over App: Task starts processing input
        

Reading a whole changelog on every restart would be wasteful. Checkpoints let a restart resume where it left off.

What the checkpoint file records

When an instance shuts down cleanly, it writes a small checkpoint file into the state directory. For each store, the file records the changelog offset up to which the local data is known to be complete. On the next start, if the local store is still on disk and the checkpoint is present, the restoration begins at that offset rather than at the start of the log. Only the records written after the checkpoint need to be replayed, which is often a tiny fraction of the total.

flowchart TD
    A["Instance starts task"] --> B{"Local store and checkpoint present?"}
    B -->|"Yes"| C["Restore from checkpoint offset to end of changelog"]
    B -->|"No local store"| D["Restore from the beginning of the changelog"]
    B -->|"Store present, checkpoint missing or untrusted"| E["Discard local store, restore from the beginning"]
    C --> F["Small replay, fast start"]
    D --> G["Full replay, slowest start"]
    E --> G
        

Why the state directory matters

The location of the local state is set by a configuration value for the state directory. If that directory is on storage that survives restarts, such as a persistent volume, restarts are fast. If it is on temporary storage that disappears with the container, every restart becomes a full restoration. On container platforms, this single decision often decides whether deployments feel instant or take a long time.

Fast bulk loading

When restoring a large amount of data into the embedded database, the library switches the store into a mode optimized for bulk writes, for example by postponing background compaction inside the database until the restore ends. This makes restoration considerably faster than normal writes, since the data arrives in order and needs no reads to check for existing values. After the restore finishes, the store returns to its normal mode.

Restoration takes time. Standby replicas remove most of that time by keeping a second copy always ready.

The idea

A standby replica is a copy of a task’s state stores that lives on a different instance, and keeps itself up to date by continuously reading the changelog. It does not process input or produce output. It only maintains the store. If the instance holding the active task fails, the group moves the task to the instance that holds its standby, which already has almost all the data. The remaining restoration is a few seconds of catch-up instead of minutes.

flowchart LR
    subgraph A["Instance A"]
        AT["Active task 0: processes input, writes changelog"]
    end
    subgraph B["Instance B"]
        SB["Standby task 0: reads changelog, keeps copy warm"]
    end
    AT -->|"appends updates"| CL["Changelog partition 0"]
    CL -->|"continuous reads"| SB
    AT -.->|"instance A fails"| SB
    SB -->|"promoted to active with almost no restore"| NEW["Active task 0 on instance B"]
        

The cost and the benefit

Warm-up replicas during moves

A similar mechanism helps when a task must move, for example onto a newly added instance. Rather than stopping the task and restoring from scratch on the destination, the library creates a temporary warm-up replica on the destination, lets it catch up in the background while the old owner keeps processing, and only then completes the move. A periodic check, called a probing rebalance, decides when the destination has caught up close enough to the end of the changelog. The gap it must close is governed by an acceptable recovery lag setting, and the number of simultaneous warm-ups is limited so that the changelog and the network are not flooded.

State updates must follow the same all-or-nothing rules as output records, otherwise recovery would double count.

The problem without atomicity

Suppose a task updates a counter, writes the new value to the changelog, and crashes before committing its input offsets. On restart, it reads the same input record again and increments the counter a second time. The state now reflects the record twice. In a counting application, this is a permanent error, because nothing will ever subtract the extra one.

Changelog writes inside the transaction

In exactly-once mode, the changelog writes, the output records, and the input offset commits are all part of a single transaction. If the transaction commits, the state change, the output, and the progress become visible together. If it aborts, the changelog records are hidden by an abort marker, the offsets do not move, and the input is read again. Restoration reads the changelog in committed-only mode, so aborted updates never reach a rebuilt store.

sequenceDiagram
    participant T as Task
    participant S as Local store
    participant P as Transactional producer
    participant CL as Changelog partition
    participant O as Output partition
    T->>P: Begin transaction
    T->>S: Update counter to 7
    S->>P: Changelog record (Asha, 7)
    P->>CL: Append, flagged transactional
    T->>P: Output record for downstream
    P->>O: Append, flagged transactional
    T->>P: Add input offsets, commit
    P->>CL: Commit marker
    P->>O: Commit marker
    Note over CL,O: State, output, and progress become visible together
        

Why local state cannot always be trusted after a crash

In exactly-once mode, the local store may contain updates from a transaction that never committed, because the store was written before the commit. If the instance crashes, its local disk holds state that includes those uncommitted changes, and the changelog does not. To stay correct, the library takes a cautious approach: after an unclean shutdown, it treats the local store as untrustworthy, discards it, and restores from the committed changelog. A clean shutdown writes a checkpoint and the local copy can be reused. This is why crashes cost more in exactly-once mode than in at-least-once mode, and why standby replicas are valuable there.

Sometimes the changelog would only copy data that already exists in another topic. The library can skip the duplication.

Tables built straight from a topic

Suppose a table is defined directly on a topic that already holds the latest value per key, for example a compacted customer profile topic. The store’s changelog would contain exactly the same data as the source topic, doubling storage and write cost for no benefit. With a suitable optimization setting, the library skips the separate changelog and, when it needs to restore the store, reads the source topic itself.

flowchart TD
    A["Table defined directly on a compacted source topic"] --> B{"Source topic reuse optimization enabled?"}
    B -->|"No"| C["Separate changelog topic created, data duplicated"]
    B -->|"Yes"| D["No separate changelog, source topic serves as the changelog"]
    D --> E["Restoration reads the source topic"]
    C --> F["Restoration reads the changelog topic"]
        

Global tables

A global table is fully replicated to every instance, so every instance can look up any key. It has no changelog of its own. Each instance restores it directly from the source topic, reading all partitions, and then keeps it current by continuing to read them. The trade-off is that every instance must hold the entire table, which is fine for small reference data and costly for large data.

Once you recognize the pattern of a compacted log of state changes, you will see it in several other places.

Why the pattern keeps recurring

Log-based state has attractive properties for a distributed system: writes are simple appends, replication is already solved by Kafka, recovery is reading from the start, and any number of readers can build their own view. The changelog topics of stream applications are simply the same design applied to each application’s private state.

A short list of settings controls how changelogs are created, written, and restored.

Per-store control

The store definition lets a developer give the store a stable name, override changelog settings for that store, or turn logging off entirely. Turning logging off removes the changelog and with it the fault tolerance of that store: after a failure the store starts empty. This is acceptable only for caches that can be rebuilt cheaply from other sources, or for stores whose loss is truly harmless.

Broker-side settings that matter

The most damaging changelog problems come from innocent-looking code changes, not from crashes.

How names are generated

If a developer does not name a store or an operation explicitly, the library generates a name from the operation type and a running number, based on the position of the operation in the topology. A changelog name might therefore contain a number such as 0000000005. The number depends on how many operations came before it in the definition of the topology.

The trap

Now imagine a developer adds a new filter early in the topology. Every later operation shifts by one position, and the generated names change. On the next deployment, the application looks for changelog topics with the new names, finds none, and creates empty ones. The old changelogs, holding all the accumulated state, are now orphaned. The application starts with empty state, and every count starts again from zero.

flowchart TD
    A["Version 1: filter, then aggregate"] --> B["Aggregate store auto-named with number 3, changelog with number 3"]
    C["Version 2: new map added at the start"] --> D["Aggregate store now auto-named with number 4"]
    D --> E["Application looks for changelog number 4, finds none"]
    E --> F["Creates a new empty changelog, ignores the old one"]
    F --> G["State appears reset, counts restart from zero"]
        

Other changes that need care

Changelog size is predictable once you know which numbers drive it.

The main drivers

After compaction, a plain changelog holds roughly one record per live key. Its size is therefore about the number of live keys multiplied by the average row size. On top of that sits an uncompacted tail: records written since the last cleaning, which grows with the update rate and the time between cleanings. Replication multiplies the total on disk across the cluster. Windowed stores add another factor: the number of windows kept.

Spreading across tasks

The total state is divided among tasks, one per input partition. A store of five gigabytes across twenty partitions gives each task about 250 megabytes to restore, and tasks restore in parallel on different instances. More partitions therefore mean smaller pieces and faster parallel recovery, up to the point where per-partition overhead outweighs the benefit. When planning partition counts for a stateful application, think about recovery time as well as processing throughput.

What to watch as it grows

Follow one small application through updates, compaction, a crash, and a restore.

The setting

A counting application tracks page views per user. Its input has three partitions, so it runs three tasks, each with one store and a matching changelog partition. Focus on task 1 and its changelog partition. The store uses the key user name and the value view count.

Phase one: updates

Phase two: compaction

The broker later compacts the closed segments of this partition. Offsets 0 and 2 are superseded by offset 4 for Asha, and offset 1 is superseded by the tombstone at offset 5 for Bilal. After compaction, the partition holds offset 3 for Chen, offset 4 for Asha, and offset 5, the tombstone for Bilal. Six records became three. Later, once the tombstone retention passes, offset 5 also disappears, leaving only the two live keys.

Phase three: the crash and restore

The instance running task 1 dies and its disk is lost. A different instance is assigned task 1. It has no checkpoint, so it restores from the start of the compacted changelog. It reads Chen = 1, applies it, reads Asha = 3, applies it, and reads the tombstone for Bilal, which removes Bilal from the store, though Bilal was never there in this fresh store. It reaches the end offset, and the store now shows Asha 3 and Chen 1, exactly matching the state before the crash.

sequenceDiagram
    participant Old as Old instance
    participant CL as Changelog partition 1
    participant New as New instance
    Old->>CL: Asha=1, Bilal=1, Asha=2, Chen=1, Asha=3, Bilal=tombstone
    Note over CL: Compaction keeps Chen=1, Asha=3, Bilal=tombstone
    Note over Old: Crash, disk lost
    New->>CL: Read from start
    CL-->>New: Chen=1
    CL-->>New: Asha=3
    CL-->>New: Bilal=tombstone
    Note over New: Store shows Asha 3 and Chen 1, matches the old state
    New->>New: Resume processing input
        

What if compaction had not run yet?

The result would be identical, only slower. The new instance would read all six records, applying Asha = 1, then 2, then 3, and so on, and it would end in the same final state. Compaction changes how much work restoration takes, never what state it produces. This is the property that makes the changelog design robust: compaction can run late, run early, or run partially, and correctness is unaffected.

Every choice around changelogs trades speed in normal operation against speed in recovery.

Costs in normal operation

Levers to shorten recovery

flowchart LR
    A["Large store, no standby, temporary disk"] --> B["Full restore after each failure"]
    C["Large store, persistent disk"] --> D["Short replay from checkpoint after clean restart"]
    E["Large store, standby replica"] --> F["Seconds of catch-up after failure"]
    G["Small store, compacted changelog"] --> H["Restore in seconds even from scratch"]
        

Changelog topics contain the complete state of your application, including anything sensitive inside it.

Access control

A changelog topic contains every key and value stored in the state store, which may include personal data, account balances, or business secrets. Anyone who can read the topic can read the entire state. Access should therefore be limited to the application’s own identity. A convenient approach is to grant permissions by a prefix of the application ID, so that the application can create, read, write, and describe all internal topics whose names begin with its ID, and nobody else can.

Deletion and the right to erasure

Privacy rules often require that data about a person can be deleted on request. In a changelog, deletion means emitting a tombstone for the key, and waiting for compaction and the tombstone retention period to remove the data physically. Until then, older values may still be present in the log. Data may also linger in the local stores of standby replicas, in aborted transactional records, in backups, and in downstream copies. A privacy-aware design documents this timeline, sets compaction and retention to meet the required deadlines, and, for very sensitive fields, encrypts values with per-person keys that can be destroyed to make the data unreadable at once.

A healthy changelog pipeline is quiet. These signals show when something drifts.

Hooks and tools

flowchart TD
    A["Alert: input lag rising on one partition"] --> B{"Is the task restoring?"}
    B -->|"Yes"| C{"Is restore consumer lag falling?"}
    C -->|"Yes"| D["Wait, and consider standbys or smaller state"]
    C -->|"No"| E["Check changelog broker health, network, and restore consumer settings"]
    B -->|"No"| F{"Is the store disk full or slow?"}
    F -->|"Yes"| G["Free space, tune the embedded database, or add capacity"]
    F -->|"No"| H["Look at processing logic and downstream backpressure"]
        

Walk through what happens for each common failure, and what you should do.

flowchart TD
    A["Task needs its state"] --> B{"Local store trusted and checkpoint valid?"}
    B -->|"Yes"| C["Replay only the tail of the changelog"]
    B -->|"No"| D{"Changelog intact and readable?"}
    D -->|"Yes"| E["Full restore from changelog"]
    D -->|"No"| F["Reset application, replay input topics to rebuild state"]
    C --> G["Resume processing"]
    E --> G
    F --> G
        

Some designs around changelogs repeatedly work well. Others repeatedly fail, usually at three in the morning during a rebalance.

Patterns worth copying

Anti-patterns to avoid

Most changelog trouble comes from a small set of repeated mistakes. Knowing them ahead of time is cheaper than discovering them in an incident.

A working checklist

Common mistakes and their fixes

flowchart LR
    A["Design: keys, names, value size"] --> B["Configure: replicas, standbys"]
    B --> C["Provision: disks, brokers"]
    C --> D["Secure: ACLs, erasure plan"]
    D --> E["Observe: lag, restore, disk"]
    E --> F["Rehearse: kill and time recovery"]
    F --> A
        

The loop matters because none of these steps is a one-time task. Data volumes grow, topologies change, and infrastructure is replaced. Each change can quietly undo an earlier assumption, so the cycle of design, configuration, observation and rehearsal keeps repeating for as long as the application lives.

The pattern of a fast local store backed by a compacted log appears wherever a stream of events must be turned into current facts.

What the examples share

In every case there is a stream of small updates, a need for very fast lookups of the latest answer, and a requirement that a machine failure must not change that answer. The combination of local storage for speed and a compacted log for durability fits each of these needs, which is why the same design appears across industries.

Short, direct answers to the questions that come up most often when people first meet changelog topics.

Usually not. The stream processing library creates them automatically when the application starts, provided its identity has permission to create topics. Some locked-down clusters require an administrator to create them ahead of time using the exact expected names.

Because it stores current state, not history. A value written a year ago is still the current value if nothing changed it. Deleting by age would erase valid state, so compaction keeps the latest record per key indefinitely.

Equivalent, not identical. The changelog holds the history of changes, of which compaction keeps the newest per key. Replaying it produces the store’s contents, but the store itself is a separate, indexed local structure.

Technically yes, but it is discouraged. The internal format and naming are implementation details that may change. If others need the data, publish a purpose-built output topic.

By default, yes, except stores for global tables and stores where logging was explicitly disabled. Some tables also reuse their source topic as the changelog when a specific optimization is enabled.

The application loses its backup. While it runs, the local stores still work, but the next restart or rebalance to a new instance finds nothing to restore. Rebuilding needs an application reset and a replay of the input topics.

It depends on the size of the changelog and the restore speed. Divide the data size by the sustained read and write rate of the instance. Measuring in a test is far more reliable than estimating.

Processing on a task would otherwise use incomplete state and produce wrong results. The library waits until the store has caught up with the changelog before resuming that task.

Extra disk, network and processing for the copy of the store it maintains. In return, failover starts from nearly current state, cutting recovery from minutes or hours to seconds.

They can be used for reading stale data through interactive queries when enabled. They are not the primary, so their data may lag slightly behind.

Not safely for a running stateful application. Keys would map to different partitions, so existing state would sit in the wrong place. The safe route is a new application identity and a rebuild from the source.

Write a tombstone for that key so the store and the changelog drop it. Remember that the tombstone itself is retained for a period before compaction removes it, and that copies in other topics need their own handling.

Compaction runs in the background, not instantly. Until it catches up, old versions of keys remain in the log. Tuning the cleaning thresholds trades storage against background work.

Yes. Changelog writes join the transaction, so an aborted attempt leaves no visible changes. After an unclean crash, the local store may contain uncommitted changes, so the library discards it and restores from the changelog.

A standby is a long-lived backup copy kept ready for failover. A warm-up replica is a temporary copy built on a new or different instance so that a task can move there without a long stall, after which it becomes the active copy.

Yes, in effect. The changelog names are built from the application identity, so a new identity looks for new, empty topics. Keep the identity stable for the life of the state.

Check that it exists with the expected partition count and replication, that its consumer lag from the application is near zero during normal running, and that its size stays proportional to the number of keys.

It holds whatever the store holds, so it needs the same protection as any topic carrying that data: access control, encryption in transit, encryption at rest where required, and a plan for erasure.

Reading a log to answer a lookup means scanning it, which is slow. The local store provides an index for instant reads. The changelog is the durable record, and the store is the fast view of it.

Run the application with realistic data, stop an instance abruptly, delete its local directory, and time how long until processing resumes and outputs match a known correct result. Repeat with a broker failure.

A stateful stream application needs memory, and memory kept on one machine dies with that machine. Kafka’s answer is the changelog: every change to a local store is also written to a compacted, replicated topic, so the store is a fast view of a durable log. Compaction keeps the latest value per key and tombstones record deletions. Windowed stores add time-based retention. When a task moves or a machine returns, restoration replays the changelog, shortened by checkpoints and by standby and warm-up replicas. Exactly-once processing folds changelog writes into transactions, and stable names, sound replication, restricted access and careful monitoring keep the whole arrangement trustworthy over years of change.