Before we meet the state store, we need to understand the problem it solves: events arrive one at a time, but useful answers often depend on many events.

A quick Kafka refresher

Apache Kafka stores streams of events in topics. An event is a small record of something that happened, such as a payment, a page view or a temperature reading. Programs that write events are producers, and programs that read them are consumers. Kafka Streams is a library that lets you write programs that read events from topics, process them, and write new events to other topics. It runs inside your own application, not as a separate server.

Stateless versus stateful processing

Some processing needs no memory. Converting a temperature from Celsius to Fahrenheit, dropping events that are missing a field, or renaming a column looks at one event at a time and forgets it immediately. This is called stateless processing. Each event can be handled alone, so the program needs nothing beyond the event itself.

Other processing needs memory. Counting how many orders each customer placed, adding up the money spent per day, joining a stream of clicks with a table of user profiles, or detecting three failed logins within five minutes all require the program to remember what came before. This is called stateful processing, and the remembered information is called state.

flowchart LR
  subgraph SL["Stateless: each event handled alone"]
    A1["Event in"] --> A2["Transform or filter"] --> A3["Event out"]
  end
  subgraph SF["Stateful: needs memory of earlier events"]
    B1["Event in"] --> B2["Read remembered value"]
    B2 --> B3["Update value"]
    B3 --> B4["Event out"]
    B3 --> B5[("State")]
    B5 --> B2
  end
        

A state store is the place where a Kafka Streams application keeps its remembered information.

A state store is an embedded, local storage area that a Kafka Streams application uses to hold the state needed by stateful operations. It behaves like a small key-value database sitting right next to your processing code. Your application can quickly read a value by its key, write a new value, delete it, or scan a range of keys. Because it lives on the same machine as the code, reads and writes are fast and do not need a network trip to a separate database.

Three properties that define it

Why not just use an external database?

You could store state in an outside database, and some applications do. But every event would then need a network call to read and another to write, which adds delay and creates a dependency on a separate system that must be scaled and kept available. State stores avoid this by keeping data local and using Kafka, which you already run, as the durable backup. The trade-off is that the state is not easily shared with arbitrary other programs, which is the reason a feature called interactive queries exists, covered later.

To understand where the data actually sits, we need three words: task, thread and instance.

Tasks: the unit of work

When a Kafka Streams application starts, it reads the topology, which is the graph of processing steps you defined, and creates tasks. A task is the smallest unit of work and is tied to one partition number of the input topics. If your input topic has six partitions, the application creates six tasks. Each task processes only the events from its own partition, in order.

One store shard per task

Each task that has a stateful step gets its own private piece of the state store. So a store is not one big database. It is a collection of small shards, one per task, and each shard holds only the keys that appear in that task’s partition. Because the same key always goes to the same partition, the same key always lands in the same shard, which keeps the logic consistent.

Threads and instances

Tasks are run by stream threads, and one application can run several threads. You can also run several copies of the same application, each called an instance, perhaps on different machines. Kafka spreads the tasks across all threads and instances. Every task, along with its store shard, runs in exactly one place at a time.

flowchart TB
  subgraph Topic["Input topic with 4 partitions"]
    P0["P0"]
    P1["P1"]
    P2["P2"]
    P3["P3"]
  end
  subgraph I1["Instance 1"]
    T0["Task 0"] --- S0[("Store shard 0")]
    T1["Task 1"] --- S1[("Store shard 1")]
  end
  subgraph I2["Instance 2"]
    T2["Task 2"] --- S2[("Store shard 2")]
    T3["Task 3"] --- S3[("Store shard 3")]
  end
  P0 --> T0
  P1 --> T1
  P2 --> T2
  P3 --> T3
        

Different questions need different shapes of storage, so Kafka Streams offers several kinds of stores.

Key-value stores in depth

A key-value store is like a dictionary. You look up a key and get its value. When you count clicks per user, the key is the user ID and the value is the count. Each new click reads the old count, adds one and writes the new count. This store shape also supports listing all keys in a range, which is useful for reports.

Window stores in depth

A window is a slice of time. A window store records a separate value for each key and each time slice. Counting logins per user per five minutes creates many small entries, one per user per five-minute window. Old windows are removed after a retention period so the store does not grow forever.

Session stores in depth

A session is a period of activity followed by a gap of silence. Unlike fixed windows, sessions have no fixed length. They open when an event arrives and close after the quiet gap passes. If two sessions get joined by a late event, the store merges them into one.

A store can keep its data on disk or purely in memory, and the choice affects speed, size and recovery.

The persistent store and RocksDB

By default, Kafka Streams uses a persistent store built on RocksDB, an embedded database engine designed for fast writes on modern disks. RocksDB keeps recent changes in memory and organises older data in sorted files on disk. It lets a store hold far more data than the machine’s memory could fit, because only the hot parts need to stay in memory.

Because the data is on disk, a persistent store survives an application restart on the same machine. When the application comes back, it can reuse the files that are already there and only fetch the few changes it missed, instead of rebuilding everything from Kafka.

The in-memory store

An in-memory store keeps everything in the application’s memory. It is very fast and has no disk files to manage, but its size is limited by memory, and everything is lost when the process stops. On restart, the whole store must be rebuilt by replaying its backup topic from Kafka. This can take a long time for large stores.

A local store alone would be lost with the machine. The changelog topic is what makes state stores fault tolerant.

What a changelog topic is

For every store shard, Kafka Streams automatically creates and writes to a special Kafka topic called a changelog topic. Each time a value in the store changes, the new value is also sent to this topic. The topic therefore holds a complete record of the store’s history, and replaying it reproduces the store exactly.

Compaction keeps it small

Changelog topics for key-value stores use log compaction. Compaction keeps only the latest value for each key and discards older values in the background. That means the topic does not grow with every update. It grows only with the number of distinct keys. For window and session stores, the changelog also has a retention limit so old windows eventually disappear from the topic as well.

flowchart LR
  IN["Input topic"] --> T["Stream task"]
  T --> ST[("Local state store")]
  T --> CL["Changelog topic in Kafka"]
  T --> OUT["Output topic"]
  CL -. "rebuild after failure" .-> ST
        

Naming and ownership

Changelog topic names are built from the application ID, the store name and a suffix. This is why choosing explicit and stable names for stores is important. If a store gets an automatically generated name and the topology later changes, the generated name may shift and the application will look for a changelog that no longer matches its state.

Follow a single event from arrival to output to see exactly how a store is used.

The record cache

Between the operator and the store sits an in-memory cache. If the same key is updated many times in quick succession, the cache holds only the newest value and passes it on when the cache is flushed. This reduces the number of writes to the store and the number of updates sent downstream. The trade-off is that downstream consumers see fewer, slightly delayed updates. The cache is flushed at commit time or when it becomes full.

The commit interval

The commit interval controls how often progress is saved. A shorter interval means less work is repeated after a crash but more overhead. A longer interval gives better batching and fewer downstream updates but more repeated work after failures. Under exactly-once processing, the default interval is shorter, because commits also finalise transactions.

The true test of a state store is what happens when the machine holding it disappears.

The restore process

When a task starts on a machine and its store shard is missing or out of date, Kafka Streams restores it before processing new events. A special restore consumer reads the changelog topic from the start, or from the last known position if some files exist, and replays each entry into the local store. Only when the store has caught up does the task begin normal processing. This guarantees the task never works with stale or incomplete state.

sequenceDiagram
  participant A as Instance A
  participant K as Kafka changelog
  participant B as Instance B
  A->>K: Writes every change to the changelog
  Note over A: Instance A crashes
  K-->>B: Task is reassigned to Instance B
  B->>K: Restore consumer reads changelog
  K-->>B: Replays changes into local store
  B->>B: Store caught up, resume processing
        

Checkpoints speed up restarts

For persistent stores, Kafka Streams writes small checkpoint files that record how far each store has been synchronised with its changelog. After a clean restart on the same machine, the application reads the checkpoint and only replays the changes after that point. Without a valid checkpoint, for instance after an unclean crash under exactly-once, it may discard the local files and restore from scratch to guarantee correctness.

How long does restoration take?

Restore time depends on the size of the store and the speed of the network and disk. A store with a few megabytes restores in seconds. A store with hundreds of gigabytes can take a long time, during which those partitions are not processed and their output is delayed. This is the single biggest operational risk for large stateful applications, and the next chapter shows how standby replicas reduce it.

Standby replicas keep spare copies of stores ready so a failover is quick.

What a standby replica is

A standby replica is a copy of a task’s store that is kept up to date on a different instance, without processing any input. It only reads the changelog topic and applies the changes to its own local files. If the instance running the active task fails, Kafka Streams can promote the instance holding the standby to active, and processing resumes almost immediately because the store is already nearly current.

flowchart LR
  subgraph I1["Instance 1"]
    A["Active task 0"] --- SA[("Active store 0")]
  end
  subgraph I2["Instance 2"]
    SB["Standby task 0"] --- SS[("Standby store 0")]
  end
  A --> CL["Changelog topic"]
  CL --> SB
        

Warm-up replicas during scaling

When you add a new instance, Kafka Streams does not immediately move active tasks onto it. That would force the new instance to restore first while those partitions stall. Instead, it creates warm-up replicas of the tasks that should move. The new instance builds the store in the background while the old owner keeps processing. When the warm-up copy is close enough to the changelog end, the active task is moved over with only a short pause.

Because state is tied to tasks, scaling a stateful application needs more care than scaling a stateless one.

What moves during a rebalance

When instances join or leave, the group of instances runs a rebalance, which is the process of redistributing tasks. In a stateless application, a moved task simply starts reading from its new home. In a stateful application, the task also needs its store. If the new owner has no copy, it must restore from the changelog. If it already holds a standby, the move is quick.

Sticky assignment

Kafka Streams uses assignment logic that tries to keep tasks on the instances that already hold their state. It prefers instances that have an active copy, then those that have a standby, and only then instances with nothing. This reduces unnecessary restores and makes rebalances cheaper.

The ceiling on parallelism

The number of tasks equals the number of partitions in the input topics, so you can never run more useful workers than you have partitions. Adding instances beyond that gives no extra processing power. This is why the partition count of input topics for a stateful application should be planned with growth in mind, as changing it later would also change how keys map to store shards.

Repartitioning

Sometimes a stateful step needs data grouped by a different key than the one it arrives with. Kafka Streams then automatically inserts a repartition topic, which is an internal topic used to reshuffle events by the new key. The store downstream is then built from the reshuffled data. Repartition topics add storage and latency, so avoid changing keys more often than needed.

A state store is not only for internal use. Applications can expose it so other programs can look up current values directly.

What interactive queries are

Interactive queries let you read the contents of a state store while the streaming application is running. Imagine a dashboard that shows the current total spent by a customer. Instead of copying the result into an external database, the streaming application answers the question straight from its own store. This gives very fresh answers and avoids running and syncing another database.

The challenge of distribution

Because the state is split into shards across instances, any single instance only holds part of the data. A request for a customer may arrive at an instance that does not hold that customer’s shard. Kafka Streams provides metadata that tells which instance owns which key. Your application can then forward the request to the right instance, usually through a small web service the team builds and exposes.

flowchart TD
  C["Client asks for customer 42"] --> I1["Instance 1 web layer"]
  I1 --> D{"Who owns customer 42?"}
  D -- "Local shard" --> L["Read local store and answer"]
  D -- "Remote shard" --> I2["Forward request to Instance 2"]
  I2 --> R["Read its store and answer"]
        

Consistency and availability

During a rebalance or restoration, a store may temporarily be unavailable or slightly behind. Queries can be allowed against standby copies to remain available, at the price of potentially slightly older answers. Applications that use interactive queries should be designed to handle these short periods, for example by retrying or accepting slightly stale data.

Some data is small and needed everywhere. A global store copies it to every instance.

How a global store differs

A normal store is partitioned. Each instance holds only its shard. A global store is the opposite: every instance holds a complete copy of the entire topic, regardless of partitions. It is loaded from the source topic before processing starts and is kept updated in the background by a dedicated thread. Because the source topic itself already holds the data, no separate changelog topic is created.

Why global stores are useful

Joining a stream to a small reference table, such as country codes or product catalogue entries, normally requires both sides to have matching partitions and keys. With a global store, any event can look up any key locally, no matter which partition it came from. It also allows lookups by a field other than the key, using a mapping function.

You rarely create a store by hand. Most stores appear automatically when you use stateful operations.

Naming stores explicitly

Because these stores are created for you, it is easy to leave them with generated names. Give each stateful operation a clear, stable name. Named stores keep the changelog topic names stable across code changes, make monitoring easier to read, and allow you to query stores by name.

For special needs, you can define your own processing steps and attach stores to them directly.

What the Processor API adds

The high-level operations cover many cases, but sometimes you need finer control, such as custom timers, unusual logic or manual updates of the store. The lower-level Processor API lets you write your own processing nodes, attach one or more state stores to them, and read and write those stores in any way you like.

Punctuation and scheduled work

A processor can schedule a periodic action, called a punctuation, based either on the passage of event time or on wall-clock time. A common use is periodically scanning a store to emit summaries or to clean out old entries that are no longer needed, such as sessions that have expired by your own rules.

Plugging in other storage engines

Kafka Streams lets you supply a different implementation of a store, as long as it follows the store interface. Teams occasionally do this to use a specialised engine. Most applications are better served by the built-in stores because they come with changelog handling, restoration and monitoring already integrated.

Correct state is only possible if each event affects the store exactly once, even after failures.

The duplicate problem

Imagine the application updates a count, writes to the changelog and output topic, and then crashes before saving its reading position. On restart it would read the same event again and add to the count a second time. For counts, sums and balances, this is wrong.

How exactly-once helps

With exactly-once processing enabled, Kafka Streams groups the store updates in the changelog, the output events and the input position into one transaction. Either everything in a commit becomes visible together, or nothing does. After a crash, the application resumes from the last committed position, and any uncommitted changelog writes are ignored by readers that only see committed data.

Effect on local files

With exactly-once, the local store may contain changes that were not yet committed when a crash occurred. To avoid this hidden inconsistency, Kafka Streams may discard the local files after an unclean shutdown and rebuild them from the committed changelog. This is why clean shutdowns matter and why restore time affects the recovery of exactly-once applications.

Stores use disk, memory and network. Planning these resources prevents nasty surprises.

Estimating store size

Estimate the number of distinct keys, multiply by the average size of key plus value, and add overhead for the storage engine. For window stores, multiply again by the number of windows kept within the retention period. Then multiply by the number of copies, counting the active store and each standby replica, to find total disk needs across the cluster.

Disk

Persistent stores need a local directory with enough space and good speed. Solid-state disks give much better performance for busy stores than slow spinning disks. Keep the state directory on stable storage that survives restarts if you want quick recovery.

Memory

RocksDB uses memory outside the normal application memory area for its write buffers and caches. If unbounded, its total use grows with the number of store shards on the machine, and the process can be killed by the operating system for using too much memory. Set a bounded total memory budget for all stores in the process, and leave room for the operating system’s file cache.

Cache size

The record cache is shared among the threads of an instance. A larger cache merges more updates, reducing downstream traffic and store writes. A smaller cache gives more up-to-date output. Tune it to the balance you need between freshness and efficiency.

Retention of windowed data

Windows, sessions and versions are kept only for the retention you configure. Long retention means bigger stores and bigger changelogs. Set retention to the longest time a late event or query could realistically need, and no longer.

State stores hold real business data, so they need the same care as any database.

Healthy state stores show themselves through a few key signals.

What to watch

Track the time each application spends restoring state, the lag of restore consumers behind their changelog topics, and the lag of standby copies. Watch store read and write latencies and rates, the size of stores on disk, and the memory used by the storage engine. Also monitor stream thread state, rebalance frequency and duration, and consumer lag on input topics.

Alerts worth having

Set alerts for restores that run longer than your tolerated recovery time, for stores nearing disk limits, and for unusually long pauses in a task’s processing. These give early warning of problems that otherwise appear only as delayed outputs.

Approaches that keep state healthy, and the traps that quietly cause pain.

Patterns worth adopting

Anti-patterns to avoid

A short path from your requirement to the store that fits.

flowchart TD
  S["What do you need to remember?"] --> Q1{"Small reference data needed everywhere?"}
  Q1 -- "Yes" --> G["Global store"]
  Q1 -- "No" --> Q2{"Depends on time?"}
  Q2 -- "No" --> KV["Key-value store"]
  Q2 -- "Yes" --> Q3{"Fixed time slices?"}
  Q3 -- "Yes" --> W["Window store"]
  Q3 -- "No, activity bursts" --> SS["Session store"]
  KV --> Q4{"Large or fast restart needed?"}
  W --> Q4
  SS --> Q4
  Q4 -- "Yes" --> P["Persistent store with standbys"]
  Q4 -- "No, small and hot" --> M["In-memory store"]
        

A quick reference to keep beside you when designing stateful applications.

Five common situations show how the ideas fit together.

No. A topic is a durable, ordered log stored on Kafka brokers. A state store is a local key-value database next to your application. They are linked because every change to a store is also written to a topic, called the changelog.

No. The default persistent store uses an embedded engine that ships with Kafka Streams. There is nothing extra to install or run.

Another instance takes over the task and rebuilds the store from the changelog topic before processing continues. If a standby replica exists, the takeover is much faster.

Not directly. Access is through interactive queries, where your own application exposes an endpoint that reads the store, forwarding requests to the instance that owns the requested key.

The record cache merges rapid updates to the same key and emits only the latest. The final values are unaffected. Only intermediate steps are combined.

One logical store for each stateful operation, and each is split into one shard per task. An application with three stateful steps and eight partitions has three logical stores and twenty-four shards.

Not safely for a stateful application. State is tied to the key-to-partition mapping. Create a new topic and a new application version, and rebuild the state.

When the data is small, needed by every instance, and you want to avoid matching partitions or repartitioning. Avoid it for large or rapidly changing data.

For key-value stores, yes, keeping the latest value per key. For windowed stores, the changelog also has a retention limit so expired windows disappear over time.

A standby is a long-term backup copy kept for fast failover. A warm-up replica is a temporary copy built on a new instance so an active task can move there smoothly, after which it becomes the active store.

Only when duplicate effects would be harmful, such as money or stock. It adds overhead and can lengthen recovery, so approximate or repeat-safe workloads can use the simpler guarantee.

Most likely it is restoring large stores from the changelog because local files were lost or invalidated. Use durable volumes, standby replicas and gradual rollouts to reduce this.

A state store is the local, embedded memory of a Kafka Streams application. It lets stateful steps such as counting, joining, windowing and deduplicating read and write remembered values at local speed. The state is split into shards, one per task, and every change is mirrored to a compacted changelog topic in Kafka, which is what makes the design fault tolerant. When a machine fails, another instance rebuilds the store from the changelog, and standby and warm-up replicas shorten the wait.

Good state store design means choosing the right shape and storage, keeping values small, naming stores explicitly, planning expiry and sizes, securing every place the data lives, and watching restore time, memory and growth. If you remember one sentence, let it be this: the local store makes your application fast, and the changelog makes it safe.