Before any vocabulary, we must feel the problem these features solve.

Stream processing that remembers

Some stream processing is simple. Read a message, change it a little, write it out, and forget it. If an instance running this kind of logic dies, another instance simply carries on. Nothing needs to be rebuilt.

Many useful applications are not like that. They count events, sum values over time windows, join two streams together or remember the latest value for every key. To do this they must remember things between messages. That memory is called state, and it is kept in a local store on the machine that runs the work. If that machine is lost, the memory is lost with it, and it must be reconstructed before work can continue.

Two questions every Streams application must answer

A quick tour of the library gives us the vocabulary needed for everything that follows.

A library, not a cluster

Kafka Streams is a Java library that you embed inside your own application. There is no separate processing cluster to install. You start several copies of your application, and they cooperate through Kafka itself. Adding capacity means starting more copies. Removing capacity means stopping some. The copies find each other and share the work automatically.

The building blocks

Built on the consumer group

Under the surface, all instances with the same application identifier form a single consumer group. That means they use the same group membership machinery that ordinary Kafka consumers use. When an instance joins or leaves, the group rebalances, and the partitions of the input topics are redistributed. Kafka Streams adds its own logic on top to decide how tasks, and therefore state, are distributed.

A task is the atom of a Streams application. Understanding it makes assignment obvious.

Why partitions are the natural unit

A Kafka topic is split into partitions, and within a partition the order of messages is fixed. To keep results correct, all messages for one key must be processed by the same worker, in order. Partitions are the units that already guarantee this. So Kafka Streams divides its work by partition.

What a task is

For each partition of the input, and for each independent part of your topology, the library creates one task. A task reads records from its partition or partitions, runs them through its portion of the processing steps, updates its own local state stores and writes results. A task never shares its state with any other task. Two tasks can therefore run on different machines with no coordination between them, which is what makes Streams scale.

Task identity

Each task has a two-part identity written like a pair of numbers: the first identifies which independent part of the topology it belongs to, and the second is the partition number. A task named with the pair one and three, for example, handles partition three of the second independent part. This identity is stable across restarts, and it is how the library, its state directories and its changelog topics all refer to the same piece of work.

Not every processing graph produces the same number of tasks. The shape of the graph matters.

Independent parts of a topology

A topology may look like one connected picture, but the library splits it into smaller pieces called sub-topologies. A boundary between sub-topologies appears wherever data has to be moved through an intermediate Kafka topic, for example when you re-group records by a new key. Each sub-topology can be processed independently, because the topic in between decouples them.

flowchart LR
    subgraph S0["Sub-topology 0"]
        A["Read orders"] --> B["Filter and re-key"]
    end
    B -->|"repartition topic"| C
    subgraph S1["Sub-topology 1"]
        C["Read repartitioned data"] --> D["Aggregate by customer"] --> E["Write totals"]
    end
        

Counting tasks

The number of tasks is the sum, over all sub-topologies, of the number of partitions each one reads. A sub-topology that reads from a topic with eight partitions produces eight tasks. If a second sub-topology reads from a repartition topic with eight partitions, it produces another eight tasks. The total is sixteen.

Co-partitioning

When a sub-topology joins two input topics, both must have the same number of partitions and use the same key-to-partition method. Only then does key seven in one topic land in the same partition number as key seven in the other. A single task then reads the matching partition from both topics and can join them locally. The library keeps such partitions together in one task for exactly this reason.

Tasks are placed onto threads, and threads live in instances. Two levels of placement are involved.

Two levels of placement

First the library decides which instance owns a task. Then, inside that instance, the task is given to one of its stream threads. Each thread runs its tasks one after another in a loop, reading a bit from each, so a thread with several tasks shares its time among them. A task always runs on exactly one thread at a time.

flowchart TB
    subgraph I1["Instance A"]
        T1["Thread 1: tasks 0_0 and 0_1"]
        T2["Thread 2: tasks 0_2 and 0_3"]
    end
    subgraph I2["Instance B"]
        T3["Thread 1: tasks 1_0 and 1_1"]
        T4["Thread 2: tasks 1_2 and 1_3"]
    end
        

Balancing goals

State is what makes assignment matter. Here is how it is stored and protected.

The local store

Each stateful task keeps its data in a local store. By default this is an embedded key-value database on local disk, with an in-memory cache in front of it. Reads and writes are extremely fast, because they never leave the machine. Each task has its own private store, stored in a directory named after the task, under the application’s state directory.

The changelog topic

Local disks fail, so the library also records every change to the store in a special Kafka topic, called a changelog topic. Each store gets one changelog topic, with the same number of partitions as the task count of its sub-topology, so each task has its own changelog partition. Changelog topics are replicated by Kafka and use log compaction, which keeps only the latest value for each key, so they do not grow forever.

flowchart LR
    IN["Input partition"] --> TK["Task"]
    TK --> ST["Local state store on disk"]
    TK -->|"every update"| CL["Changelog topic partition in Kafka"]
    TK --> OUT["Output topic"]
    CL -. "rebuild after loss" .-> ST
        

Restoring state

When a task starts on a machine that does not have its store, the library restores it by reading the task’s changelog partition from the beginning and applying every record to a fresh local store. Only when this restoration finishes can the task begin processing new input. The bigger the store, the longer the restoration takes.

Whenever group membership changes, work must be redistributed. A special component makes the decision.

What triggers a rebalance

The group leader computes the plan

During a rebalance, one member of the group is chosen as the leader. Every member sends the leader information about itself. In Kafka Streams, this information includes which tasks the member currently holds, how far behind its stores are, and which tasks it holds as standby copies. The leader runs the task assignor, a piece of logic that produces the new plan, and sends each member its share.

A good assignor balances several competing goals at once. Knowing them explains its behavior.

The central tension

The most important tension is between balance and stickiness. Moving a task to fix imbalance means restoring its state at a new location, which takes time. Leaving it where it is avoids restoration but may leave one instance busier than another. A good assignor keeps tasks sticky by default and moves them only when the benefit is worth the cost, and where possible moves them gradually.

Stickiness is the simplest defense against slow recovery. It also has limits.

What sticky means

A sticky assignor tries to give each task to the same member that ran it last time. When a new member joins, only enough tasks are moved to it to balance the group, and the rest stay put. When a member leaves, only its tasks move, and they move to the members that already hold the most useful state for them.

The problem with pure stickiness

A purely sticky approach protects existing state but handles growth poorly. Suppose an application runs on three instances and you add a fourth to relieve load. The new instance has no state at all. If the assignor hands it tasks immediately, it must restore each store from scratch before it can process anything, and during that time the tasks are not making progress anywhere. The system got worse, not better, right after scaling out.

A standby replica is the direct answer to the cost of cold starts.

The idea

A standby replica is a copy of a task’s state store maintained on a different instance from the active task. The standby does not process input or write any output. It only keeps its local store up to date by reading the same changelog topic that the active task writes. If the active task’s instance fails, the standby’s instance can take over almost immediately, because its store is already nearly complete.

Active tasks and standby tasks

flowchart LR
    IN["Input partition"] --> AT["Active task on Instance A"]
    AT --> S1["Store on Instance A"]
    AT -->|"writes"| CL["Changelog partition"]
    CL -->|"reads only"| SB["Standby task on Instance B"]
    SB --> S2["Warm copy of store on Instance B"]
    AT --> OUT["Output topic"]
        

How many standbys

The number of standby copies per task is a single setting. The default is zero, meaning no standbys, and any positive number creates that many additional copies of every stateful task, each on a different instance. One standby protects against a single instance failure. Two protect against two overlapping failures, and so on. Each copy costs disk space and network and processing to keep up to date.

A standby task is simple by design, which is exactly why it is cheap to keep.

What a standby does all day

A standby task runs the same restoration logic used at startup, but it never stops. It continuously reads new records from the changelog partition and applies each update to its local store. It does not run the processing steps of the topology, so it does no business computation and calls no external services. Its whole job is to copy.

Standbys are read-only followers

A standby never writes to the changelog and never writes output. It is a follower in the same sense that a Kafka replica is a follower: it only consumes and applies. This keeps the design free of write conflicts and means a standby can be added or removed without affecting the correctness of the active task.

The payoff of standby copies is easiest to see by comparing two failures side by side.

Failure without a standby

Failure with a standby

sequenceDiagram
    participant A as Instance A active task
    participant B as Instance B standby
    participant G as Group leader
    participant K as Kafka changelog
    A->>K: Write state updates
    B->>K: Read updates and apply
    A->>A: Crash
    G->>G: Detect failure, run rebalance
    G-->>B: Promote to active for the task
    B->>K: Read remaining few updates
    B->>B: Resume processing input
        

A standby is only useful if it is not lost in the same accident as the active task.

The basic rules

Instances versus threads

The separation applies at the level of instances, not threads. Two threads in the same instance share the same machine and the same state directory. A standby on a different thread of the same instance would offer no protection against machine loss, so the assignor places it on a different instance.

flowchart TB
    subgraph IA["Instance A"]
        A1["Active task 0_0"]
        S2["Standby task 0_1"]
    end
    subgraph IB["Instance B"]
        A2["Active task 0_1"]
        S3["Standby task 0_2"]
    end
    subgraph IC["Instance C"]
        A3["Active task 0_2"]
        S1["Standby task 0_0"]
    end
        

Insurance is never free. Understanding the bill helps you choose the right amount.

A sensible way to decide

A standby helps after a failure. Warm-up replicas help during a planned change, like scaling out.

The insight

When a new instance joins, the goal is to give it work. But giving it tasks whose state it does not have would stall those tasks during restoration. A better approach is to leave each task where it is, and separately ask the new instance to build a warm copy in the background. Once the copy has caught up, the task can move with hardly any interruption. This is the idea behind warm-up replicas, introduced with the high-availability task assignor.

sequenceDiagram
    participant O as Old owner
    participant N as New instance
    participant G as Group leader
    N->>G: Join group
    G-->>O: Keep task active for now
    G-->>N: Build warm-up copy of task
    N->>N: Restore in background
    G->>G: Probing rebalance, check lag
    G-->>N: Copy caught up, become active
    G-->>O: Release task, optionally keep standby
        

The assignor needs a measurable notion of “caught up”. That notion is called lag.

What lag means for a task

The lag of a task’s store on some instance is the number of changelog records that have been written but not yet applied to that instance’s copy. A lag of zero means the copy is perfectly current. A lag of a million means the copy is a million updates behind. Every instance reports the lag of each store it holds, so the group leader can compare candidates for each task.

The acceptable recovery lag

The library has a setting that defines how far behind a copy may be while still counting as caught up enough to become active. If a copy’s lag is within this threshold, it is considered ready, since catching up the small remainder takes only moments. If it is above the threshold, the copy is still considered too cold to take over, and the assignor prefers to keep the task where it is while the copy continues to catch up.

Trade-offs when tuning

Assignment is only half the story. How the handover happens decides how much of the application pauses.

The old way: stop the world

In the original rebalance protocol, every member gave up all of its partitions at the start of each rebalance, waited for the new plan and then resumed. For the whole duration, no member processed anything, even the tasks that would end up exactly where they were. A rebalance was therefore a global pause.

The cooperative way: move only what must move

Cooperative rebalancing changes this. Members keep processing the tasks that are not changing owners, and only the tasks that must move are paused and handed over. The rebalance can take two rounds: first, tasks that need to leave are released, then they are given to their new owners in the following round. Everything else keeps running throughout.

sequenceDiagram
    participant A as Instance A
    participant B as Instance B
    participant G as Group leader
    Note over A,B: Both are processing normally
    G-->>A: Release task X only
    A->>A: Pause and hand over task X
    G-->>B: Second round: receive task X
    B->>B: Restore or promote and start task X
    Note over A,B: All other tasks kept running throughout
        

Combined with warm-up replicas

Cooperative rebalancing and warm-up replicas work together. Warm-up copies prepare state in advance, and cooperative rebalancing then moves only the ready tasks, one wave at a time, without pausing the rest of the application. The combined effect is that scaling out and rolling restarts become mostly invisible to downstream consumers.

Newer group protocols

Recent Kafka releases have introduced a newer generation of group protocols in which more of the coordination is moved to the broker side, and there has been work on a dedicated protocol for Kafka Streams applications that lets the broker track and assign tasks. The details depend on the release and feature status, so consult the release notes of the version you run. The underlying concepts in this tutorial, tasks, active and standby roles, lag and warm-up, stay the same.

The cheapest rebalance is the one that never happens.

The cost of routine restarts

Every time an instance restarts, for example during a rolling deployment, the group sees it leave and rejoin, and by default a rebalance occurs each time. In a group of many instances, a rolling restart can trigger a long series of rebalances, each moving tasks around and forcing restorations, only for tasks to end up back where they started.

Static membership

Static membership lets each instance carry a permanent identifier. When an instance restarts and comes back with the same identifier within the session timeout, the group recognizes it as the same member returning and simply gives it back its previous tasks, without a rebalance. Its local state is still on disk, so it only needs a short catch-up from the changelog.

In the cloud, machines belong to zones, and zones fail together. Assignment can be taught to respect that.

Why location matters

If an active task and its standby happen to sit in the same availability zone, a zone outage removes both. Also, reading input partitions from a broker in a different zone costs network fees and adds latency. Modern assignors can take location into account.

flowchart LR
    subgraph ZA["Zone A"]
        A1["Instance 1: active 0_0, standby 0_1"]
    end
    subgraph ZB["Zone B"]
        A2["Instance 2: active 0_1, standby 0_2"]
    end
    subgraph ZC["Zone C"]
        A3["Instance 3: active 0_2, standby 0_0"]
    end
        

Let us watch a real change unfold, step by step, using everything we have learned.

An application runs on three instances named A, B and C. It has twelve stateful tasks, four on each instance. It is configured with one standby per task. Traffic has grown, so the team adds a fourth instance, D.

Scaling in

Removing an instance follows the reverse pattern. If instance D is shut down gracefully, its tasks are redistributed. Where standbys exist, the instances holding those standbys become the natural new owners, and recovery is quick. If standbys are absent, the new owners must restore from the changelog and processing for those tasks pauses during the restore. Where possible, plan scale-in with warm copies already present.

A design is proven on its worst day. Here is what happens in the situations operators fear most.

State stores are not only for processing. Applications can also query them, and standbys can help with that too.

Queryable state

Kafka Streams lets an application expose its state stores to outside callers, for example a web service answering “what is the current total for customer forty-two”. Because state is spread across instances, a query must be sent to the instance that holds the requested key. The library can tell you which instance is the active owner of a key, so a service can route each request correctly.

Standbys as read replicas

During a failure or rebalance, the active copy of a store may be temporarily unavailable, so queries against it would fail. The library allows queries against standby copies, and against stores that are still restoring, if the caller accepts that the data may be slightly stale. This trades freshness for availability and can keep an interactive service responding through disruptions.

A plan that ignores capacity works in tests and fails in production. Here is how to think about the numbers.

Disk

Each instance stores the local state of every active task and every standby it holds. Total disk across the application is roughly the size of all stores multiplied by one plus the number of standbys. Each instance’s share is that total divided by the instance count, with headroom for uneven task sizes and for the temporary extra copies created during migration. Because the embedded database also needs space for its own maintenance work, leave a comfortable margin rather than filling disks.

Memory

Each store has caches and buffers. Standby stores have them too, even though they process no input. Multiply the per-store memory by the number of stores an instance holds, active and standby together. Running out of memory on an instance can cause it to fail, triggering the very rebalances you wanted to avoid.

Network

Every change to state is written to the changelog once and read once by each standby. During restoration, an instance reads a changelog end to end. A mass restoration, such as many instances restarting with empty disks, can saturate network links and broker capacity. Plan headroom for the worst restoration case, not only for steady state.

You do not need every setting memorized, but you should know what each family controls.

Application shape

Assignment and standby

Group membership

The default assignment logic suits most applications, but the library offers ways to adapt it.

Choosing among built-in assignors

The library has shipped several assignment algorithms over the years. An older sticky approach focuses on keeping tasks in place. The high-availability approach adds warm-up replicas and lag awareness. Recent versions let you choose the assignor by configuration, and a fallback option exists for older behavior. For nearly all users, the modern default is the right choice.

Custom assignors

Recent releases have opened an extension point that lets you plug in your own assignor logic. It receives descriptions of the instances, tasks and their state lag, and returns a plan. This is meant for rare, specialized needs, such as placing certain tasks on instances with special hardware, or honoring business rules about data locality.

How you deploy the application decides how much you need standbys, and how well they work.

Ephemeral disks

In many container deployments, an instance’s disk vanishes when the container is replaced. Every restart is then a cold start that needs a full restore from the changelog. For applications with meaningful state, this makes standbys much more valuable, because a standby lives on a different instance and survives the replacement.

Persistent volumes

Alternatively, each instance can be given a persistent volume that follows it across restarts. A restarted instance finds its old state on disk and only needs a short catch-up. Combined with static membership, this makes rolling restarts smooth even with no standbys. Persistent volumes protect against restarts but not against loss of the volume or the machine.

State copies, internal topics and group membership all carry sensitive information that deserves protection.

Healthy assignment and standby behavior is quiet. Good monitoring lets you notice trouble early.

Logs worth reading

Application logs record assignment decisions, restoration starts and finishes, task creations and closures, and probing rebalances. During an incident, collect the logs of every instance and line them up by time. The story of a rebalance is spread across all members, and only the combined picture shows why a task moved or stalled.

You can look at what the application decided rather than guess.

Practical uses

Every recovery feature buys resilience at a price. A clear view of both sides guides good decisions.

Choosing a protection level

These habits keep Streams applications calm and predictable through failures and scaling.

A checklist before going live

Each of these looks harmless until a failure or scaling event makes it expensive.

A few persistent misunderstandings cause a surprising number of production problems.

Standby replicas and careful assignment appear wherever stateful streaming meets strict availability.

A task is the smallest unit of work. It is created for each input partition of each independent part of the topology, and it processes that partition’s records and owns its own local state.

Add up, over all sub-topologies, the number of partitions each one reads. Printing the topology description makes this easy to count before you deploy.

An instance is one running copy of the application, usually one process. A thread is a worker inside an instance. Tasks are assigned to instances and then to threads.

A copy of a task’s state store kept up to date on a different instance by reading the changelog, ready to take over if the active task’s instance fails.

They consume some disk, network and processor time for copying, but they do not process input. The effect on capacity depends on how heavily your application updates state.

Yes. Without a setting, no standby copies are kept, and every task move requires restoring state unless the local disk still has it.

No. The assignor always places standbys on different instances, since a shared machine would defeat the purpose.

A temporary standby created on an instance that will soon take over an active task. It restores in the background, and the task moves only after the copy has caught up enough.

The library runs probing rebalances to check the progress of warm-up copies and to move tasks in stages. They should stop once the migration completes.

A threshold on how many changelog records a copy may still be missing while still being considered ready to become active.

An instance that restarts quickly with the same permanent identifier gets its old tasks back without triggering a rebalance, keeping its local state useful.

Persistent volumes help with restarts, but not with loss of the volume or machine. Standbys protect against those. Many teams use both for large-state applications.

Yes, if you allow stale reads. Results may lag behind the active copy, so this suits use cases where availability matters more than perfect freshness.

Doing so with a stateful application breaks the mapping between keys and state. Plan partition counts in advance and treat later changes as a migration.

No. They have no local state to restore, so they can be moved anywhere at no cost.

In the changelog topic in Kafka, which is replicated by the brokers. Local stores and standbys are speed optimizations built on top of it.

Compare the time spent detecting the failure, the time spent rebalancing and the time spent restoring state. Standbys only shrink the last of these, so the other two may need different fixes.

A Kafka Streams application turns its topology into many small tasks, one for each input partition of each independent part, and hands them out across instances and threads. Stateful tasks keep local stores backed by changelog topics, and moving such a task to a machine without its state forces a slow restoration. Sticky assignment keeps tasks where their state already lives, standby replicas keep warm copies elsewhere so a failure costs only a short catch-up, and warm-up replicas prepare new instances in the background so scale-out does not stall. Cooperative rebalancing, static membership and zone-aware placement complete the picture. Understanding these pieces lets you choose protection levels sensibly, plan capacity honestly and operate stateful streaming with confidence.