A short refresher, so that the rest of the guide stands on firm ground.

Apache Kafka moves streams of events between programs. A program that writes events is a producer, and a program that reads them is a consumer. The servers in the middle are brokers. Events are organised into named streams called topics, and each topic is split into partitions: ordered lists of messages that can live on different brokers so the work can be shared.

Each message has a key and a value. The key decides the partition: messages with the same key always go to the same partition, and inside a partition the order is preserved. This detail is the foundation of everything in this guide. It means that all the events for customer 42 sit together, in order, in one partition.

A Kafka topic is a log. It is wonderful for recording what happened, and awkward for asking what is true now.

Suppose a topic holds every order ever placed. To learn the total for customer 42, you would have to read all of that customer’s orders from the start and add them up. There is no way to say “give me the latest value for key 42” and get an instant reply. A topic has no index on keys, and it is not designed for lookups. Reading it again and again for each question would be slow and wasteful.

The usual fix, and its cost

Teams typically add a stream-processing job that keeps running totals, and then write those totals into an external database such as a relational or key-value store. Applications query that database. This is a proven design, but it has costs: one more system to operate, one more copy of the data, one more place where the copy can be wrong or late, and one more dependency in the request path.

Interactive Queries are a feature of Kafka Streams, so we need a quick tour of that library first.

Kafka Streams is a library, not a separate cluster. You add it to an ordinary application, describe how events should be transformed, and run as many copies of the application as you like. Each copy is called an instance. Together, the instances share the work automatically, using the same group mechanism as consumers.

The description of the processing is called a topology: a graph of steps such as reading from a topic, filtering, grouping by key, counting, joining, and writing to another topic. Two kinds of data structures appear in it. A stream (KStream) is an endless sequence of events. A table (KTable) is the latest value per key, like a spreadsheet that keeps updating. Aggregations, such as a running total per customer, turn a stream into a table.

Tasks: the unit of parallel work

Kafka Streams divides the work into tasks. Each task processes one partition (or one aligned set of partitions) of the input topics. If the input topic has 12 partitions, the application has 12 tasks, and these tasks are spread across the running instances. With three instances, each typically runs four tasks. Add a fourth instance and Kafka Streams moves some tasks to it.

To keep running totals, a task needs somewhere to remember them. That place is called a state store.

A state store is a small database that lives inside the application instance, next to the task that uses it. By default it is backed by RocksDB, a fast embedded key-value store that keeps recent data in memory and the rest on local disk. Reads and writes are local and quick, with no network call, which is why stateful stream processing can be so fast.

Local disks fail, and instances get replaced. To survive this, every write to a state store is also recorded in a special Kafka topic called a changelog topic. If an instance is lost, its replacement reads the changelog and rebuilds the store. The changelog is the backup, and Kafka’s replication keeps that backup safe.

flowchart LR
  IN["Input topic partition"] --> TASK["Stream task"]
  TASK --> STORE["Local state store
RocksDB"]
  STORE -->|"every update"| CL["Changelog topic in Kafka"]
  CL -->|"used to rebuild after a failure"| STORE
  TASK --> OUT["Output topic"]

Restoring takes time

Rebuilding a store means replaying its changelog. For a small store this is seconds. For a store with hundreds of millions of keys, it can take many minutes or even hours. During that time the store is not ready to answer questions. This matters greatly for Interactive Queries, since a query cannot be answered from a store that is still being rebuilt.

Interactive Queries let code outside the stream-processing flow read the state stores that the flow maintains.

Interactive Queries, sometimes called queryable state, are the part of the Kafka Streams library that gives your own code read-only access to the state stores of a running application. Your code might be a web service handler, a command-line tool or a dashboard backend. Instead of asking a database for customer 42’s total, it asks the Streams application, which reads it from its local store.

The two halves of an answer

Because no single instance holds all the state, answering a question takes two steps. If the key you want is in this instance’s own stores, you read it locally. If it is held by a different instance, you must find that instance and ask it. Kafka Streams helps with the first half completely and with the discovery part of the second. The actual conversation between instances is yours to build.

Here is the whole system in one picture, followed by a walk through each part.

flowchart TD
  CL["Client, for example a web page"] --> LB["Load balancer"]
  LB --> I1["Instance A
holds partitions 0 to 3"]
  LB --> I2["Instance B
holds partitions 4 to 7"]
  LB --> I3["Instance C
holds partitions 8 to 11"]
  I1 |"forward when the key lives elsewhere"| I2
  I2 |"forward when the key lives elsewhere"| I3
  I1 |"forward when the key lives elsewhere"| I3
  K["Kafka cluster
input topics and changelog topics"] --- I1
  K --- I2
  K --- I3

Three instances share twelve partitions, four each. Customer 42’s key hashes to partition 5, which currently belongs to Instance B. A request for that customer can land on any instance, because a load balancer spreads requests. If it lands on Instance B, the answer comes straight from B’s local store. If it lands on Instance A, A must work out that partition 5 belongs to B and pass the request along.

Not every store can be queried. Only stores with an explicit name are visible from outside the topology.

When a topology performs an aggregation, Kafka Streams creates a state store behind the scenes. By default that store has a generated internal name, and there is no reasonable way for outside code to refer to it. To make a store queryable, you give it a name when you define the aggregation, using a setting called Materialized. Naming it is the act of publishing it to the outside world.

What naming gives you

Which parts of a topology create queryable state

Different stores answer different questions. The type you choose decides which questions are cheap.

Point lookups versus scans

The cheapest question is a point lookup: give me the value of exactly this key. It touches one partition on one instance and returns quickly. A range scan asks for all keys between two values, and because stores keep keys in sorted order, this is efficient inside one store. But if the keys of interest are spread over several partitions, the scan must be sent to every instance that holds one of them and the results merged. A scan of everything touches every instance.

Let us begin with the simplest case: the key is held by the instance you are talking to.

To read a store, your code asks the running Streams object for it, passing the store’s name and the kind of store it is. The result is a read-only view with methods such as get for one key, range for a span of keys, and all for everything. The view is live: it reflects the current state of the store, which keeps changing as events are processed.

Rules for scans

Methods that return many entries, such as range and all, hand back an iterator. Behind that iterator sits a handle into RocksDB that holds memory and file resources. You must close the iterator when you finish. Forgetting to do so leaks resources, and in a busy service can slowly exhaust memory or make the store misbehave. Treat every iterator like an open file: use it, then close it, even when an error occurs.

The library knows where every partition lives. Your code only has to ask, and to tell the library how to be reached.

Every Kafka Streams instance is a member of a group, and the group knows which instance holds which tasks. To make that knowledge useful for queries, each instance can announce an address at which it accepts outside requests. This is done with a setting called application.server, whose value is a host and port such as “orders-2.internal:8080”. The setting is only a label: Kafka Streams does not open that port or listen on it. It simply shares the text with the other members, so they can read it from the group metadata.

Asking who owns a key

Given a store name, a key and a way to turn the key into bytes (a serializer), the library can compute the partition and answer with a KeyQueryMetadata object. It holds three useful things: the address of the instance with the active copy, the addresses of instances with standby copies, and the partition number. Your code compares the active address with its own. If they match, it reads locally. If not, it forwards the request.

Two ways to get it wrong

Kafka Streams stops at discovery. The wiring between instances is an ordinary web service that you design.

Each instance must run a small server, usually a REST or gRPC service, at the address it advertised. This service has two jobs. The first is to handle requests from clients and answer them, by reading locally or by calling another instance. The second is to handle requests from other instances that were forwarded to it and must not be forwarded again.

Why the two endpoints matter

Imagine ownership changes in the middle of a request. Instance A forwards to B, but B has just given the partition away to C. If B also forwarded, we could end up in a chain, or even a loop between two instances that each believe the other is the owner. The safe rule is that a forwarded request is answered locally or refused with an error, and the caller decides whether to retry. Keep the forwarding depth to exactly one hop.

Choosing the protocol

Let us follow a single question all the way through, including the case where it lands on the wrong instance.

sequenceDiagram
  participant C as Client
  participant A as Instance A
  participant B as Instance B
  C->>A: Get total for customer 42
  A->>A: Ask library for owner of key 42
  A-->>A: Owner is Instance B, partition 5
  A->>B: Internal request for key 42
  B->>B: Read local store
  B-->>A: Total is 1250
  A-->>C: Total is 1250

With three instances and a random balancer, about one request in three lands on the owner, and two in three need a hop. With ten instances, nine in ten need one. This is why forwarding must be efficient, and why some teams later switch to client-side routing or a partition-aware load balancer.

A Streams instance moves through named states. Queries are only safe in some of them.

The KafkaStreams object has a state that you can read and observe. Understanding the states tells you when queries will succeed and when they will fail.

stateDiagram-v2
  [*] --> CREATED
  CREATED --> REBALANCING: start
  REBALANCING --> RUNNING: assignment done and stores restored
  RUNNING --> REBALANCING: membership change
  RUNNING --> PENDING_SHUTDOWN: close
  REBALANCING --> PENDING_SHUTDOWN: close
  RUNNING --> PENDING_ERROR: fatal error
  PENDING_ERROR --> ERROR
  PENDING_SHUTDOWN --> NOT_RUNNING

The errors you will meet

Instances come and go. Each change shuffles who owns what, and queries must survive it.

A rebalance reassigns tasks among instances. It happens when an instance joins, leaves, crashes or is unresponsive for too long, or when the input topics gain partitions. When a task moves, its state store moves with it, in the sense that the new owner must have a copy. It either restores that copy from the changelog, or, if it kept a standby replica, catches up from a nearly-ready copy.

flowchart TD
  Q["Query arrives"] --> R{"Instance state is RUNNING?"}
  R -- "No" --> N["Return not ready, client retries later"]
  R -- "Yes" --> D["Discover owner for key"]
  D --> L{"Owner is this instance?"}
  L -- "Yes" --> S["Read local store"]
  L -- "No" --> F["Forward once to owner"]
  S --> E{"Store migrated or unavailable?"}
  F --> E
  E -- "Yes" --> W["Wait briefly, discover again, retry a few times"]
  W --> D
  E -- "No" --> A["Return answer"]

What a good retry policy looks like

Sometimes a slightly old answer now is better than no answer. Kafka Streams lets you choose, and measure how old.

By default, a query against a store that is being restored or has just moved fails. That protects you from reading incomplete data, but it makes the service unavailable during every rebalance. Since version 2.5, an option lets queries read from stores that are not yet fully up to date, which is called allowing stale stores. This can include standby copies that live on other instances.

Standby replicas

A standby replica is a hot spare of a task’s store. It does not process input or produce output, but it keeps reading the changelog and applying updates. You choose how many with the setting num.standby.replicas. If the active instance dies, a standby is promoted quickly, since most of its data is already in place.

Measuring how far behind

Lag here means the number of changelog records a local store copy has not yet applied. The library can report the lag of every local store partition, active or standby. A caller can then say: “I will accept an answer from a standby, provided it is within 1,000 records of the latest.” Without measuring lag, “stale” is a guess.

flowchart TD
  Q["Query for key"] --> A{"Active host reachable and ready?"}
  A -- "Yes" --> RA["Read from active: freshest answer"]
  A -- "No" --> ST{"Standby exists and lag within limit?"}
  ST -- "Yes" --> RS["Read from standby: possibly slightly stale"]
  ST -- "No" --> ER["Return temporarily unavailable"]

A query answers from the store as it is at that moment, which is not always the same as what has been safely committed.

Stream processing is continuous. A task reads a record, updates its store, and writes to the changelog and output topics, and commits its progress every so often. A query reads the store at some instant, and that instant may fall between updates and commits. So a query can see values that reflect processing that is not yet committed. If the task then fails and restarts from its last commit, those updates are reprocessed, and the store may briefly move backwards before moving forward again.

The original query interface is small and fixed. A newer generation adds richer requests and more information in each reply.

The first interface, often called IQv1, gives you read-only store views with a fixed set of methods. It works but has limits: you cannot ask for how fresh the answer is, you cannot express new kinds of queries without changing the library, and custom stores are hard to support. A newer interface, called IQv2, appeared in the 3.x releases as a redesign. The idea is that a query is now an object you build and send, and stores that understand that kind of query answer it.

Why position matters

Suppose a user submits an order and immediately views their totals. With plain queries, the total might not yet include the order. If the write operation gives back the position of the event in the topic, the read can carry that position and ask the store to answer only once it has caught up. That is far better than guessing with delays.

Sometimes it is easier to give every instance all the data than to work out who owns which part.

A GlobalKTable is a table built from a topic in which every instance holds every partition, not just its own share. Because each instance keeps a complete copy, a lookup of any key can always be answered locally. No discovery and no forwarding are needed.

When it is a good fit

Reading many keys at once is where local queries turn into cluster-wide work.

A scan reads a run of entries in key order. Inside one store partition this is fast. But your question may cover keys living on many partitions, and therefore many instances. The way to handle this is called fan-out and merge: ask each instance that hosts part of the store for its share, then combine the answers.

When results are grouped by time, a key has many values, one per window. Queries must say which times they want.

A window is a slice of time, such as “10:00 to 10:05”. A windowed aggregation keeps one result per key per window, for example the number of page views per user per five minutes. A session is different: it is a burst of activity for a key that ends after a gap of silence, so sessions vary in length. Both store several values for one key, so a lookup needs a key and a time range.

Practical rules

Real problems teach best. Each story below is typical of what teams report, told simply.

Local reads are very fast. Almost every slowdown comes from something around the read.

Ways to add capacity

Neither is always right. The choice depends on what you ask, how fresh it must be, and what you are willing to operate.

A middle path

Many organisations do both. The stream job powers a fast, narrow lookup endpoint for a user-facing feature, and also writes its results to a database for reporting. The lookup service is the sharp tool for one job, and the database is the wide tool for everything else. Related tools, such as ksqlDB’s pull queries, are built on the same underlying idea, hiding the discovery and forwarding work behind a query language.

The moment you add an endpoint, you own a new attack surface. Kafka’s own protections do not cover it.

Kafka can require clients to prove who they are, encrypt traffic, and limit which topics each user may read or write. None of that applies to the web endpoint you built for queries. Anyone who can reach that endpoint may read whatever it exposes, regardless of who is allowed to read the underlying topics.

You cannot fix what you cannot see. Watch the stream job and the query service together.

Application-level signals to add

Interactive Queries are used where a fast answer about “the current state of X” matters.

A few shapes recur in good designs, and a few mistakes recur in bad ones.

Patterns that work

flowchart LR
  W["Write side: services publish events"] --> T["Kafka topics"]
  T --> S["Kafka Streams job builds views"]
  S --> V["Named state stores"]
  V --> Q["Query service: discover, read, forward"]
  Q --> U["Users and other services"]
  S --> DB["Optional database for reports"]

Anti-patterns to avoid

Almost every query problem traces back to how keys map to partitions. It is worth understanding the mapping precisely.

When a producer writes a message with a key, a partitioner turns the key into a number and picks a partition from it. By default, the key is converted to bytes, run through a hash function, and reduced to a partition number by taking the remainder after dividing by the partition count. The same key always gives the same partition, as long as the partition count and the rule stay the same. Kafka Streams uses the same rule when it re-keys data before an aggregation, which is why discovery can compute the owner of a key without asking anyone.

Re-keying and why it changes the answer

Sometimes the topology changes the key before aggregating, for example grouping orders by product instead of by customer. Kafka Streams then writes the re-keyed events to an internal repartition topic. The store you query is keyed by the new key, so discovery must be given the new key, not the original one. Asking for the total of customer 42 in a store that is keyed by product will find nothing.

Restoration is the quiet period when a store rebuilds itself. Knowing what happens explains most availability gaps.

When an instance receives a task whose store is not present, or only partly present, it must catch up before processing. It first checks the local disk for a checkpoint file that records how far the local copy had reached in the changelog. If one exists, it reads the changelog from that position onward. If none exists, it reads from the very start. While this happens, the task does not process input records and the store cannot be queried normally.

flowchart TD
  T["Task assigned to instance"] --> C{"Local store and checkpoint exist?"}
  C -- "Yes" --> P["Read changelog from checkpoint position"]
  C -- "No" --> Z["Read changelog from the beginning"]
  P --> U["Apply records to the store"]
  Z --> U
  U --> D{"Caught up to the end of the changelog?"}
  D -- "No" --> U
  D -- "Yes" --> R["Task starts processing and store becomes queryable"]

What makes it faster

A queryable store is a database that shares a machine with a stream processor. Give it room to breathe.

RocksDB keeps recent writes in memory tables and organises older data in sorted files on disk. It uses a block cache, an in-memory area holding recently read pieces of files, so repeated reads avoid the disk. It also runs background compaction, tidying files to save space and keep reads quick. All of this uses memory, disk and CPU alongside your processing threads and query threads.

A simple sizing walk

Stores live for a long time. Values written today must still be readable next year.

The values in a store are stored as bytes, produced by a serializer. If you change the shape of a value, for example by adding a field, the old bytes still sit in the store and the changelog. New code must be able to read them. Query responses also leak the shape of the value to clients, so a change to the value affects your API as well.

Upgrading the application

Because queries share resources with stream processing, protecting the store from unnecessary reads pays off twice.

Most real traffic is uneven: a small set of keys gets most of the requests. A short-lived cache in the query service, holding recent answers for a few seconds, absorbs repeated requests without touching the store or the network. The price is a small extra delay in freshness, so choose the lifetime to match what users can accept.

A quick reference for every important term used in this guide.

Interactive Queries are one way to serve stream results. Here is how the neighbouring options differ, so you can choose calmly.

Three other approaches often appear in the same conversation. Each solves the “how do I read the answer” problem in a different place, and each moves the cost somewhere else.

A short way to choose

A checklist of habits that keep a query layer healthy, and the slips that most often hurt.

A short pre-launch checklist

Some ideas sound reasonable but are wrong. Clearing them up prevents expensive surprises.

Stateful services are harder to run than stateless ones. A few platform choices make a big difference.

Tests should cover the topology, the store reads, and the ugly moments when ownership changes.

Let us design the customer order-total feature from the start of this guide, end to end.

What we sized

Suppose each of the 12 tasks holds about one million customers, with a few hundred bytes per entry. The active data is roughly a few gigabytes per task in total across the cluster, and the standby adds a second copy. Each instance holds four active and four standby stores, so plan disk for eight stores’ worth, plus room for compaction. A restart with persistent disks replays only the recent changes, while a lost disk replays the entire changelog for that task, which is why standbys are worth their extra space.

The whole guide in a few lines.

Kafka topics store events, and they cannot be queried for current answers. Kafka Streams keeps running results in local state stores, backed up by changelog topics. Interactive Queries let your own code read those stores, read-only, so that a stream job can answer questions without a separate database. Because state is divided among instances, every request needs to find the owner of its key, read locally or forward once, and cope with instances that are starting, restoring or changing owners. You provide the network layer, the security, the retry behaviour and the monitoring. Used for narrow key lookups with realistic expectations of freshness, the approach removes moving parts. Used as a general database, it disappoints.