Before comparing two tools, we need to know the job they were built for.

What is Kafka, in one minute?

Apache Kafka is a system that stores and moves messages between programs. A message is called a record. Records are saved in a named list called a topic. Think of a topic as a very long notebook where you can only add new lines at the bottom and never erase or edit old lines.

A big topic is split into smaller pieces called partitions, so many machines can share the work. Every record inside a partition gets a number called an offset, which is simply its position in line: first, second, third, and so on. Each record also carries an optional key (like a customer ID), a value (the actual content), and a timestamp.

The key is very important. Kafka uses the key to decide which partition a record lands in. All records with the same key go to the same partition, so their order is preserved. Remember this, because the whole KTable idea leans on it.

Why plain topics are not enough

Storing records is only half the story. Companies also need to react to records as they arrive: count them, filter them, combine them with other data, and raise alarms. You could write this yourself using a plain Kafka consumer and producer, but you would quickly need to build many hard things by hand: remembering counts between messages, surviving crashes without losing those counts, splitting work across machines, and handling messages that arrive late.

Kafka Streams is a library that gives you all of this. It is not a separate server you install. You add it to your normal Java or Kotlin application, and your application becomes a stream processor. Run more copies of the application and the work is shared automatically. Kafka Streams arrived with Kafka version 0.10 in 2016, and it has grown a lot since then.

Two mental models for the same data

Kafka Streams gives you two main ways to look at a topic. It calls them abstractions, which just means “ways of thinking that come with built-in tools”.

The most important sentence in this whole article comes first.

A KStream and a KTable are not different kinds of topics. They are two different interpretations of records. The topic is just a notebook of records. Your code decides whether to read that notebook as a diary or as a scoreboard.

Let us use a simple example. A topic called loyalty-points holds these records, in this order, all using the customer name as the key:

• Asha, 10 points
• Ravi, 5 points
• Asha, 25 points
• Ravi, 0 points
• Asha, 40 points

Read as a KStream, this is five separate events. Nothing is forgotten. If you asked “how many records are there?” the answer is five. If you asked for the total of all values, you would add up 10 + 5 + 25 + 0 + 40.

Read as a KTable, this is only two rows. Asha has 40 points. Ravi has 0 points. The earlier values were replaced by later ones. If you asked “how many rows are there?” the answer is two.

flowchart LR
    T["Kafka Topic: loyalty-points (5 records)"]
    T --> KS["KStream view: all 5 events kept"]
    T --> KT["KTable view: latest value per key"]
    KS --> R1["Asha 10, Ravi 5, Asha 25, Ravi 0, Asha 40"]
    KT --> R2["Asha = 40, Ravi = 0"]
    

The diagram shows one topic feeding two views. On the left path, the KStream keeps every event, so the result is a list of five facts. On the right path, the KTable keeps only the newest value for each key, so the result is two rows. The topic did not change at all. Only your reading changed.

The words “insert” and “upsert”

Database people use two small words that explain everything. An insert adds a brand-new row every time. An upsert is a mix of “update” and “insert”: if the key already exists, it updates that row; if the key is new, it adds a row. KStream behaves like insert. KTable behaves like upsert.

A KStream is the simplest of the two, so we start here.

What it is

A KStream is a sequence of records that never ends, where each record is an independent fact. It is often called a record stream. The word “unbounded” describes it well: there is no last record, because new ones keep arriving as long as the world keeps producing events.

Two records with the same key do not affect each other. If a customer clicks “Buy” three times, that is three events, not one event that changed three times. A KStream respects that.

What you can do with it

Most KStream tools are stateless. That means each record can be handled on its own, without remembering anything about earlier records. This makes them cheap and fast.

When a KStream is the right choice

Use a KStream whenever every event matters on its own. Payments, ride requests, page views, temperature readings, and log lines are perfect examples. If losing the history would hurt the business, you want a stream.

A KTable looks small on the surface but does a lot of hidden work.

What it is

A KTable is a view of a topic where each key has exactly one current value. When a new record arrives with a key you have seen before, the old value is replaced. When a record arrives with a brand-new key, a new row appears. Because of this, a KTable is called a changelog stream: it is a stream of changes that, when applied one after another, build a table.

Deleting a row: the tombstone

How do you remove a row from a KTable? You send a record with the same key and an empty (null) value. This special record is called a tombstone, like a small grave marker that says “this key is now gone”. The KTable removes the row when it sees it.

Why a KTable needs memory

To know the “latest value for each key”, the application must remember every key it has seen and its current value. Kafka Streams keeps this in a local database called a state store. By default this store is RocksDB, a fast key-value database that lives on the application’s own disk, with hot data cached in memory.

To protect against crashes, every change to the state store is also written to a special Kafka topic called a changelog topic. If the machine dies, another machine can replay that topic and rebuild the exact same table. We explore this in detail later.

Compacted topics: the KTable’s best friend

Normally Kafka deletes old records after a set time. That would be a disaster for a table, because a key that never changes would slowly disappear. To solve this, Kafka offers log compaction. With compaction turned on for a topic, Kafka keeps at least the latest record for every key and cleans away older ones. The topic turns into a compact snapshot of the current state, plus the recent history.

This is why tables and compaction go together so naturally. A compacted topic is basically a table stored on disk in a Kafka-friendly way.

When a KTable is the right choice

Use a KTable when you care about the current state of something: a customer’s profile, a product’s price, a driver’s status, an account’s balance, a device’s configuration. You will also use KTables as the result of counting or summing records by key.

The title says KStream versus KTable, but you will meet a cousin sooner or later.

A normal KTable is split by partition. If a topic has 12 partitions and you run 3 copies of your application, each copy holds roughly a third of the table. This is great for size and speed, but it has a catch: to combine a stream with such a table, both must be split the same way.

A GlobalKTable is different. Every copy of your application loads all partitions of the topic. So every copy has the complete table on its own disk. This gives you two superpowers. First, you can join a stream to it using any field, not just the key. Second, you never worry about matching partition counts.

The rule of thumb is simple. Use a GlobalKTable for small, slow-changing reference data such as country codes, currency rates, or product categories. Use a regular KTable for large, fast-changing data such as customers or orders.

This is the idea that makes Kafka Streams feel elegant instead of confusing.

Streams and tables are not rivals. Each can be turned into the other, without losing information. This two-way relationship is called stream-table duality.

From stream to table: aggregation

Take a KStream of events and aggregate it. Aggregating means squeezing many records into a summary, such as counting orders per customer. The result is a KTable, because each customer key now has one current count. Every new event updates that count.

From table to stream: the changelog

Take a KTable and watch every change it goes through. Each change, such as “Asha now has 40 points”, is itself an event. Convert the table into a stream of those changes and you get a KStream again. Kafka Streams offers a simple step for this, and the result is the full history of how the table evolved.

flowchart LR
    S1["KStream: order events"] -->|"group by key then count"| T1["KTable: order count per customer"]
    T1 -->|"convert to stream of changes"| S2["KStream: count updates"]
    S2 -->|"aggregate again"| T2["KTable: rebuilt latest counts"]
    

Read the diagram from left to right. Order events flow in as a stream. Grouping and counting turns them into a table of counts. Watching that table’s changes gives a new stream of count updates. Applying those updates rebuilds the table. Nothing is lost at any step.

Why this matters in practice

Duality explains why Kafka can store a table in a topic (the changelog), why a database can be copied into Kafka (change data capture turns table changes into a stream), and why you can rebuild any table by replaying its stream. It also means you rarely need to worry about picking the “wrong” one forever. You can always convert.

Let us open the box and see the parts that make streams and tables work.

The topology: your recipe

A Kafka Streams program is described as a topology, which is a graph of steps. Data enters at source nodes (reading topics), passes through processor nodes (filter, map, aggregate, join), and leaves at sink nodes (writing topics). Think of a factory conveyor belt with stations along the way.

flowchart TD
    subgraph Kafka["Kafka Cluster Brokers"]
        IN["Input topics"]
        REP["Internal repartition topics"]
        CL["Internal changelog topics"]
        OUT["Output topics"]
    end
    subgraph App["Streams Application Instance"]
        SRC["Source nodes"]
        PROC["Processor nodes"]
        SS[("State store RocksDB")]
        SNK["Sink nodes"]
    end
    IN --> SRC
    SRC --> PROC
    PROC  SS
    PROC --> REP
    REP --> SRC
    SS --> CL
    PROC --> SNK
    SNK --> OUT
    

The diagram shows the two worlds. On the top is the Kafka cluster with four kinds of topics. At the bottom is your application. Records come from input topics into source nodes, flow through processors, and reach sink nodes that write to output topics. Stateful processors, such as a KTable, talk to the state store, and each store change is copied to a changelog topic. When a key changes, records take a detour through a repartition topic and re-enter the source.

Key building blocks

How KStream and KTable fit in

A KStream node is mostly plain plumbing. Records go in, and records come out. A KTable node contains a state store. So the moment you use a KTable, or aggregate a KStream into one, your application becomes stateful, and everything about state (disk, backups, restarts) starts to matter.

Follow one record from the moment it lands in a topic until its result appears.

The record cache: why a KTable seems calm

Suppose a customer’s count goes from 1 to 2 to 3 to 4 within a fraction of a second. Sending four separate updates downstream would be wasteful. Kafka Streams keeps a small record cache in front of each state store. Updates to the same key overwrite each other in the cache, and only the latest is sent onward when the cache is flushed, which happens on each commit or when memory fills up.

The result is that a KTable usually emits fewer updates than the number of input records. This surprises many beginners who expect one output per input. That is a feature: a table describes the latest state, so it does not need to announce every tiny step. If you really want every step, you can shrink or disable the cache, at the cost of more traffic.

Processing guarantees

Kafka Streams offers two promises about what happens if something fails mid-flight. At-least-once means no record is lost, but a record might be processed twice after a crash. Exactly-once means the combined effect of reading, updating state, and writing output happens once, even after a crash. Exactly-once works by wrapping the steps in a Kafka transaction. It costs a little speed but is very valuable for money-related tables like balances.

A table lives on a machine’s disk. What happens when the machine disappears?

The safety net

Every state store has a twin: its changelog topic in Kafka. Because Kafka itself is replicated across several brokers, the changelog is safe even if an application machine dies. The local disk is treated as a fast cache of the real truth, which lives in Kafka.

flowchart LR
    A["Instance A owns partition 3"] --> SA[("Local store A")]
    SA --> CL["Changelog topic partition 3"]
    A -. "crash" .-> X["Instance A lost"]
    CL --> B["Instance B takes partition 3"]
    B --> SB[("Rebuilt store B")]
    STB["Standby replica on Instance C"] -. "keeps warm copy" .-> CL
    STB --> B
    

In the picture, Instance A works on partition 3 and backs up its store into the changelog. When A crashes, Kafka gives partition 3 to another instance. That instance replays the changelog to rebuild the table. If a standby replica was already keeping a warm copy on another machine, recovery is nearly instant because there is very little left to replay.

Failure scenarios and recovery

Rebalancing

When an instance joins or leaves, Kafka moves tasks between the remaining instances. This is a rebalance. During a rebalance, moved tasks must restore their state before they can work again. Modern Kafka uses cooperative rebalancing, which moves only the tasks that really need to move, so the rest keep working. Stateless KStream steps rebalance almost instantly, while big KTables take longer.

Real questions usually need more than one dataset, and joins are how KStream and KTable finally work together.

The classic example: enriching events

An order event says “customer 42 bought item 7”. That is not very useful for a report. You want the customer’s name and city. The customer data lives in a KTable. Joining the order KStream with the customer KTable adds those fields to each order as it flows past. This is called enrichment, and it is the most common use of the pair.

flowchart LR
    O["Orders KStream"] --> J["Stream-Table Join"]
    C["Customers KTable"] --> J
    J --> E["Enriched orders KStream"]
    E --> OUT["Output topic"]
    

Every time an order arrives, the join looks up the customer’s current row in the table and stitches them together. The important detail is that only the stream side triggers output. If a customer moves to a new city, no new output appears until that customer places another order.

The join menu

The co-partitioning rule

For the regular joins above, both inputs must be co-partitioned. That means two things: both topics must have the same number of partitions, and records with the same key must have been placed using the same partitioning method. Why? Because the join happens locally inside one task. Customer 42’s orders and Customer 42’s profile must land in the same partition number so the same task sees both.

Stream-to-stream joins need a window

Two streams never end, so a stream-to-stream join cannot keep all records forever. You give it a time window, such as five minutes. An ad impression and an ad click join only if the click appears within five minutes of the impression. Records outside the window are forgotten to save space.

Endless streams need boundaries, or answers like “how many?” would never finish.

If you count every click since the beginning of time, the number only goes up. A more useful question is, “How many clicks happened in each five-minute slice?” A window cuts an endless stream into slices of time. Counting inside windows produces a special KTable where the row key is both the original key and the window.

Which clock do we use?

There are two clocks. Event time is when something actually happened, such as the moment a phone tapped “Buy”. Processing time is when your application got around to handling it. On a bad mobile network these can differ by minutes. Kafka Streams mostly uses event time, taken from the record’s timestamp, because that gives the answer people actually want.

Kinds of windows

Late data and the grace period

Sometimes an event arrives after its window has already moved on. Kafka Streams lets you set a grace period, which is extra time to accept late events into a window. Events arriving after the grace period are dropped. A longer grace gives more accurate counts but requires keeping windows open, and therefore state, for longer.

Time and joins

When a stream joins a table, which version of the table row should it see: the version at the time the event happened, or the newest version at the time the event is processed? By default, it uses whatever the table holds right now. Newer Kafka versions added versioned state stores, which keep short history per key so a join can pick the row that was valid at the event’s own timestamp. This helps when events arrive late or out of order.

In Kafka Streams the key is not decoration. It decides where data lives.

Why keys drive everything

Remember that Kafka sends all records with the same key to the same partition. A KTable depends on this. To keep “the latest value per key”, one task must see every update for that key, in order. Because same-key records share a partition, and one task owns each partition, that guarantee comes for free.

The moment the key changes

Now imagine orders arrive keyed by order ID, but you want to count orders per customer. You must re-key the stream by customer ID. After re-keying, records for one customer are scattered across many partitions. Kafka Streams fixes this automatically by writing the records into a repartition topic using the new key and reading them back. This adds one more hop, more network traffic, and more disk.

Key skew: when one key is too popular

If one key receives a huge share of traffic, such as a single celebrity account, that partition and its task become a bottleneck while other tasks sit idle. This is called skew or a hot key. Streams and tables both suffer, but tables suffer more because the state for that key grows in one place. Ideas to soften it include splitting the key into smaller pieces, pre-aggregating before the main step, or choosing a more even key.

The lifecycle of a row

A row in a KTable is born when a new key first appears. It is updated each time a fresh record arrives. It dies when a tombstone arrives. Compaction eventually removes old versions and, after a delay, the tombstone itself. Understanding this small life story explains almost every strange table behavior you may meet.

Neither view is better. Each is better at something, and each has a price.

KStream

KTable

Trade-offs at a glance

A quiet but important trade-off is consistency. A KTable in your application is a local copy built from Kafka, so it reflects the world with a small delay. Two application instances holding different partitions are each correct for their own keys, but a full-table view across instances is eventually up to date, not instant. For most business uses this is perfectly fine. For strict banking-style checks, teams often combine tables with exactly-once processing and careful design.

When you are staring at a new problem, walk through these questions in order.

Matching real problems to the right tool

Tables quietly keep copies of data in more places than beginners expect.

Protecting the wire and the doors

Internal topics need permissions too

A Streams application creates its own internal topics for changelogs and repartitioning. The names begin with the application ID. Your access rules must allow the application to create, read, and write topics with that prefix. A common beginner mistake is granting access only to input and output topics, and then wondering why the app fails at startup.

Personal data in tables

A KTable that holds customer profiles stores personal information in three places: the source topic, the local state store, and the changelog topic. If a customer asks to be forgotten, you must remove them from all of these. Sending a tombstone for that key removes the row from the table, and compaction then removes older versions from the topics after a configured delay. Plan this delay and test it, because privacy laws expect deletion to actually happen.

Interactive queries and exposure

Kafka Streams lets you query state stores directly from your application, which is handy for showing a live table through an API. Treat that API like any other public service: put it behind an API gateway or authentication layer, and never expose raw state store access to the open internet.

Kafka Streams grows by adding helpers, but only if you understand what limits the growth.

Partitions set the ceiling

Remember that one task handles one input partition. If your input topic has 12 partitions, you can have at most 12 tasks working at the same time. Running 20 copies of the application will not make it faster than 12 busy workers. The other eight copies simply wait as spares. So the first scaling decision is really made when the topic is created: how many partitions will I need at my busiest?

Scaling out and scaling up

What slows a KTable down

A KStream filter is almost free: look at the record, decide, move on. A KTable step is heavier, because each record triggers a read from the state store, a write back, and a queued backup to the changelog. Three things make it heavier still.

• Large rows. Storing big values for every key makes each write slower and the store bigger.
• Very many keys. When the table is too large for memory, lookups go to disk more often.
• Hot keys. One popular key forces one task to do more work than its neighbors.

Throughput and latency

Two words describe speed. Throughput is how many records per second you can process. Latency is how long one record takes from start to finish. They pull against each other. Batching and caching raise throughput but can add a little delay. Tiny commit intervals lower latency but create more small writes. Choose based on what the business needs: a fraud check wants low latency, while a nightly-style analytics count wants high throughput.

Why local state is fast

A big reason Kafka Streams performs well is that table lookups happen on the same machine, on local disk or in memory. There is no network call to a remote database for each record. Compare that with a design where every event triggers a database query over the network: at hundreds of thousands of events per second, the database becomes the bottleneck. Keeping the table right next to the processing code removes that bottleneck.

Compaction sounds boring, yet without it tables would not survive in Kafka.

Two ways a topic can forget

A normal topic forgets by time or size: records older than, say, seven days are deleted, whatever their keys. A compacted topic forgets by replacement: when a newer record exists for a key, older records for that key may be removed. The topic still keeps the newest record per key, indefinitely.

How the cleaner works

Inside each partition, Kafka splits the log into pieces called segments. The newest segment, where records are being written, is left alone. A background worker called the log cleaner looks at older segments, builds a small map of “latest offset for every key”, and writes new, smaller segments that keep only the latest record per key. Old segments are then swapped out.

flowchart LR
    A["Before cleaning: Asha 10, Ravi 5, Asha 25, Ravi 0, Asha 40"] --> C["Log cleaner keeps latest per key"]
    C --> B["After cleaning: Ravi 0, Asha 40"]
    B --> D["Later delete marker removes Ravi if value is empty"]
    

Before cleaning, the partition holds five records. After cleaning, only the newest record for each key remains. If the newest record for a key is a tombstone, it is kept for a while so that slow readers can learn about the deletion, and then it is removed too.

Things compaction does not promise

• Not instant. Cleaning runs in the background, so for a while several records per key may still exist.
• Order is kept. Records that survive keep their original offsets and relative order.
• The active segment is untouched. The most recent writes stay uncompacted until the segment closes.
• Readers can still see old records if they read before the cleaner reaches them, which is why table logic must be safe with duplicates for the same key.

A stateful application is easy to write and easy to deploy badly.

Where the application runs

Because Kafka Streams is a library, your application is an ordinary program. It runs in containers, on virtual machines, or on a container platform such as Kubernetes. The brokers can be your own or a managed Kafka service from a cloud provider. The two sides only need a network path and the right permissions.

Stateless is easy, stateful needs care

A pure KStream application keeps nothing important on disk. If a container disappears, another starts and continues from the last committed offset. Rolling updates are simple. A KTable application is different, because its local disk holds a copy of the table. If the disk disappears with the container, the new container must rebuild the table from the changelog before it can work, which can take a long time for a large table.

Safe rollout steps

Managing topology changes

If you change the shape of your steps in a way that alters internal topic names or state layout, old state may no longer match. For small changes, careful naming avoids trouble. For large changes, teams run the new version side by side with a new application ID, let it build its own tables from the source topics, and switch consumers over when it is ready. This is safer than editing a running application in place.

Let us build a pipeline in words, step by step, using every idea from this article.

Our store, “Brightcart”, wants a live view of revenue per city, per minute, with each order showing the customer’s name and the correct tax. Three inputs exist: a topic of orders, a topic of customers, and a small topic of tax rates by country.

flowchart TD
    OT["Orders topic"] --> OS["Orders KStream"]
    CT["Customers topic compacted"] --> CTB["Customers KTable"]
    TT["Tax rates topic compacted"] --> TG["Tax GlobalKTable"]
    OS --> J1["Join with customers"]
    CTB --> J1
    J1 --> J2["Join with tax rates"]
    TG --> J2
    J2 --> RK["Re-key by city"]
    RK --> WA["Windowed count and sum per minute"]
    WA --> RT["Revenue KTable per city per minute"]
    RT --> OUT["Output topic for dashboards"]
    

What could go wrong, and how the design answers it

Notice how naturally each piece fits its job. Events are streams, current facts are tables, small lookups are global tables, and counting turns a stream into a table again. That is the duality idea at work in a real design.

Some wrong ideas keep travelling from team to team. Here is the correction for each.

A KTable is not only a step in a pipeline. It can also answer questions from the outside.

The idea

Normally, a Streams application reads records, processes them, and writes results to an output topic. Other programs then read that topic. But sometimes a website or a mobile app simply wants to ask, “What is the current loyalty balance for Asha?” Writing a whole new database just to answer that can feel heavy. Interactive queries let your application answer straight from its own state store, which is already holding the latest value for every key it owns.

The catch: each instance holds only a slice

A KTable is split across instances by partition. So Asha’s row might live on instance A, while Ravi’s lives on instance C. If a request for Asha arrives at instance C, that instance does not have the answer. Kafka Streams gives you the tools to find out which instance owns a key, and your application then forwards the request to that instance over a normal web call.

flowchart LR
    U["Mobile app user"] --> G["API gateway"]
    G --> LB["Load balancer"]
    LB --> I1["Instance A holds keys for partitions 0 and 1"]
    LB --> I2["Instance B holds keys for partitions 2 and 3"]
    I1 -- "forwards if key owned elsewhere" --> I2
    I1 --> S1[("Store A")]
    I2 --> S2[("Store B")]
    

The user’s request first passes the API gateway, which checks identity, and then the load balancer, which picks any instance. If that instance does not own the key, it forwards the request to the instance that does. The owning instance reads its local store and replies. This hop is quick because it stays inside your own network.

Things to remember

A stream never sleeps, so you need dashboards that watch it while you do.

A batch job either finishes or fails, and you notice. A streaming application can quietly fall behind for hours while still looking “up”. Good monitoring answers three questions: Is it alive? Is it keeping up? Is it healthy inside?

flowchart LR
    APP["Streams application metrics"] --> COL["Metrics collector"]
    BRK["Broker and topic metrics"] --> COL
    LOG["Application logs"] --> LS["Log platform"]
    COL --> DASH["Dashboards"]
    COL --> AL["Alert rules"]
    LS --> DASH
    AL --> ONC["On-call engineer"]
    

Application metrics and broker metrics flow into a collector. Logs flow to a log platform. Dashboards show trends, and alert rules wake up the on-call engineer when a number crosses a line.

The numbers worth watching

Logging with care

Log events that help humans understand behavior: rebalances, restores, errors, and dropped records. Do not log full record contents, because that spreads personal data into log files. Use a stable identifier instead, such as an offset and partition, which lets an engineer find the exact record later.

Dead-letter handling

Some records will be broken: unreadable, missing fields, or wrongly formatted. If the application crashes on every bad record, it will restart and crash again forever. A better plan is to catch such records, send them to a separate dead-letter topic, and keep going. A person can inspect that topic later and decide what to fix.

Teams keep discovering the same good habits and the same bad ones.

Patterns that work

Anti-patterns to avoid

Small habits that save big headaches in production.

Habits that pay off

Common mistakes

Seeing the ideas at work in familiar products makes them stick.

KStream and KTable are two lenses on the same records. A KStream sees every record as a new event, like a diary that only grows. A KTable sees every record as the latest value for its key, like a scoreboard that always shows the present. GlobalKTable is the fully copied cousin, ideal for small lookup data. The two main ideas convert into each other through aggregation and change streams, which is why Kafka Streams feels so consistent once you see the pattern.