Stateful vs Stateless Operations in Kafka
Some Kafka programs look at each message once and forget it. Others remember what came before. This guide shows the difference, why it matters for speed, cost and safety, and how to choose well in real production systems.
Picture two workers standing beside a busy conveyor belt. The first worker glances at every parcel, stamps it “checked” and lets it go. She never remembers a single parcel after it leaves her hands. The second worker keeps a notebook. Every time a parcel passes, he adds a tally mark for its destination city, and at the end of the hour he reports how many parcels went to each city.
Both workers are useful. Yet if the factory suddenly doubles its belts, the first worker is trivial to duplicate, while the second worker’s notebook creates a real headache. Who holds the notebook? What if he falls ill? How do two workers share one tally? This little story is the whole idea behind stateless and stateful operations in Apache Kafka.
Kafka is a system that moves and stores streams of events, such as clicks, payments, sensor readings and ride requests. Programs that process those events fall into the two camps above. Knowing which camp a piece of logic belongs to helps you predict its memory use, its recovery time, its scaling limits and its failure behaviour before you ever deploy it. Let us build that understanding one small step at a time.
1The Big Picture: What Does “State” Mean?
Before Kafka enters the story, we need a plain-English meaning for the word “state”. Once that is clear, everything else becomes a matter of details.
State is anything a program remembers between one event and the next. If the answer to “what should I do with this event?” depends only on the event itself, the program has no state. If the answer also depends on earlier events, the program must store something about them, and that stored something is its state.
A simple test you can use anywhere
Ask one question about any operation: “Can I finish my job using only the message in my hand?” Converting a temperature from Celsius to Fahrenheit passes this test, because the number in the message is all you need. Counting how many temperature readings arrived today fails the test, because a single reading cannot tell you the total. That second job needs memory.
Think of a cashier at a shop. Scanning an item and showing its price is stateless: the item alone decides the price. Keeping a running bill for the customer, applying a “buy two, get one free” offer and printing a total is stateful: the cashier has to remember every earlier item in the basket.
Why Kafka people care so much
Kafka is built for scale. A single cluster at a large company can carry millions of events every second, spread over hundreds of servers. At that size, “where does the memory live?” stops being a small question. Every byte of state has to be stored somewhere, protected against crashes, and moved around when servers join or leave.
Stateless logic sidesteps all of that. Stateful logic embraces it and pays a price in exchange for far richer answers, such as totals, averages, sessions, alerts and joined views of data.
Where the word applies in Kafka
People use these two words in several places. You will hear them about Kafka Streams (a Java library for processing events), about ksqlDB (a database-style way to process events with SQL-like statements), about Kafka Connect transformations (small edits applied to records while they move between systems) and about plain consumer applications. The ideas are identical everywhere, though Kafka Streams offers the richest set of both kinds, so most of this guide uses it as the main example.
When a teammate says “this job is stateless”, they usually mean the logic keeps no memory. The Kafka brokers underneath still keep plenty of their own data. We will look at that difference in chapter 4.
A short history: from nightly batches to living streams
For many years, companies processed data in batches. Events piled up during the day, and a big job ran overnight to produce reports. Every batch job was naturally stateful in a simple way: it could see the whole day’s data at once, so it could count, sort and join freely. The price was waiting until the next morning for any answer.
Kafka was created at LinkedIn around 2010 and released as open source in 2011, so that events could be moved the moment they happened. At first, Kafka only carried the events, and other tools did the calculating. The Kafka Streams library arrived with Kafka 0.10 in 2016 and brought calculations, and the state they need, into a simple library that any application can embed.
The important shift is that a streaming program never sees “the whole day at once”. It sees one event at a time, forever. That is exactly why the question “does this step need to remember?” became so central. A batch job can answer it by looking at all the data. A streaming job must decide what to keep, where to keep it and for how long.
A batch job is like reading tomorrow’s newspaper, which summarizes yesterday. A stream is like a live news ticker at the bottom of the television screen. The ticker shows what is happening right now, but if you want the day’s total, you have to keep a tally yourself while the headlines scroll past.
2Stateless Operations: Look, Act, Forget
A stateless operation treats every event as a brand-new visitor. It never asks what came before and never worries about what comes next.
In a stateless operation, the output for a given input is always the same, no matter when the input arrives or how many events came earlier. Computer scientists call this a pure transformation. Feed the same message in on Monday or on Friday, and the same result comes out.
The common stateless jobs
Keep or drop
Look at one event and decide whether it stays in the stream. Example: keep only payments above a certain amount, or drop test traffic.
Change the shape
Turn each event into a modified event. Example: convert prices from cents to dollars, or mask a card number.
One in, many out
Split one event into several. Example: break a sentence into separate words, or an order into its line items.
Route by rule
Send each event down one of several paths. Example: domestic orders to one topic and international orders to another.
Pick a new label
Choose which field becomes the event’s key. The step itself is stateless, although it can trigger extra work later (see chapter 12).
Look without touching
Run a side action such as logging or counting a metric, and pass the event along unchanged.
Why engineers love stateless steps
Strengths
- Any copy of the program can handle any event, so adding more copies is easy.
- A crash costs almost nothing: restart and carry on from the last committed position.
- Memory use stays small and predictable.
- Easy to test, because input and output are all you need to check.
Limits
- Cannot compute totals, averages, rankings or counts.
- Cannot spot patterns across several events, such as three failed logins in a row.
- Cannot combine two streams, because that needs memory of the other side.
- Cannot remove duplicates without remembering what it has already seen.
Airport security scans each bag as it rolls through the X-ray machine. The officer decides “pass” or “inspect” based on that bag alone. She does not need to remember the previous bag, and if she takes a break, a colleague can step in without any handover. That is exactly how a stateless step behaves.
Picture the flow
The diagram below shows a typical stateless pipeline. Events enter from an input topic (a named, durable channel in Kafka), pass through a filter and a map, and land in an output topic. Nothing is stored in the middle.
flowchart LR
P["Producer apps"] --> IT["Input topic: raw-orders"]
IT --> F["Filter step: drop test orders"]
F --> M["Map step: mask card number"]
M --> B["Branch step: domestic or international"]
B --> OT1["Output topic: domestic-orders"]
B --> OT2["Output topic: international-orders"]
A real-world example
Netflix and event routing
Netflix has publicly described its Keystone data pipeline, which moves trillions of events per day from streaming devices and services into downstream systems. A large part of that job is routing and light filtering: picking the right events for the right destination. Work like this needs no memory of earlier events, which is a big reason it can be spread over thousands of machines so easily.
“Stateless” does not mean “trivial”. A stateless step can hold heavy business rules, call lookup tables that never change while the app runs, and still qualify. The only rule is that it keeps no memory that depends on earlier events.
Stateless outside Kafka Streams
The same idea shows up in other Kafka tools. In ksqlDB, a query that only selects certain columns or applies a WHERE condition is stateless, because it inspects each row alone. In Kafka Connect, a single message transform (a small edit applied to each record while it travels between Kafka and another system) is stateless too. Typical examples are renaming a field, dropping a column, masking an email address or sending records to a topic chosen by a rule.
Even a hand-written consumer application is stateless when it reads a message, validates it, writes a row to a database and moves on. It does not keep notes about earlier messages; the database does the remembering, and the application does not depend on it for its own logic.
Order and parallelism
Kafka guarantees the order of events inside one partition, but not across partitions. Stateless steps do not care. Since no step compares an event with another, the fact that partition three runs slightly ahead of partition seven changes nothing about the results. That freedom is what makes stateless processing so easy to parallelize.
Stateful steps are different. A count per customer is safe because all events for one customer sit in one partition and arrive in order. Break that arrangement, and results depend on luck. This is the first hint of why keys matter so much later in this guide.
When in doubt about whether a step is stateless, ask what would happen if two copies of your program each handled half of the events at random. If the results stay correct, the step is stateless.
3Stateful Operations: Remember, Then Decide
A stateful operation keeps a notebook. Each new event is read together with the notes from earlier events, and the notes are updated afterwards.
The notebook has a formal name in Kafka: the state store. It is a small database that lives next to your processing code. When an event arrives, your code reads the relevant note, combines it with the event, writes the new note back, and possibly sends a result downstream.
The common stateful jobs
How many so far?
Keep a counter per key. Example: page views per article, or failed logins per user.
Roll up values
Combine many events into one running result. Example: total spend per customer, or the average delivery time per city.
Remember a time slice
Group events into slots of time such as one minute or one hour. Example: orders per minute.
Match two sources
Combine events from two streams, or a stream and a table. Example: attach customer details to every order.
Never twice
Remember which event identifiers were already handled and skip repeats.
Latest value per key
Keep only the newest fact about each thing. Example: current address of each customer.
Imagine a turnstile at a stadium with a display that shows how many fans have entered. Each push adds one to a number that the turnstile remembers. If the power fails and the display resets to zero, the count is wrong for the rest of the match. Kafka’s state stores and their backups exist to prevent exactly this kind of disaster.
What the price looks like
Once memory enters the picture, four new worries appear at once.
Where does the notebook live?
On the local disk of the machine running your code, and often partly in memory for speed.
What if the machine dies?
The local disk may be lost. The notes must be copied somewhere safe so a replacement machine can rebuild them.
What if the work moves?
When machines are added or removed, some keys must move to different machines, and their notes must travel with them.
What if the notebook grows too big?
Millions of keys mean gigabytes of notes. Disk space, memory and rebuild time all grow with them.
Picture the flow
Notice the extra boxes in the next diagram. The state store sits beside the processing step, and every change to it is copied into a special Kafka topic called a changelog topic. That topic is the safety copy.
flowchart LR
IT["Input topic: page-views"] --> G["Group by article id"]
G --> C["Count step"]
C S[("Local state store: counts per article")]
S -.-> CL["Changelog topic in Kafka: backup of every change"]
C --> OT["Output topic: views-per-article"]
A real-world example
Pinterest and advertising budgets
Pinterest has publicly described using Kafka Streams to keep track of how much advertisers have spent, so that campaigns stay near their daily budgets. Tracking spend needs a running total per advertiser, and that total cannot be recomputed from a single event. It is a textbook stateful job.
Adding a single “count” or “join” to an otherwise stateless pipeline turns the whole application into a stateful one from an operations point of view. Plan disk space, recovery time and monitoring accordingly.
Local memory versus a remote database
A fair question is: why does Kafka Streams keep its notebook on the local disk? The program could store counts in an outside database instead, and many older designs do exactly that. Each approach has a different feel.
Local state store
- Reads and writes take microseconds because no network call is needed.
- The store stays in step with the input position, since both are saved together.
- Each task owns its slice of keys, so there is no fight over shared rows.
- Recovery uses the Kafka changelog, which you already run.
Remote database
- Every event pays a network delay, often several milliseconds.
- The database can become a bottleneck when thousands of tasks hit it at once.
- After a crash, the database may already contain a change whose event will be replayed.
- You must run, secure and scale one more system.
A local store is like a small notebook in your pocket: you jot a tally in a second. A remote database is like phoning head office to record every tally: it works, but the phone line is busy, and every call takes time. Kafka’s design keeps the notebook in your pocket and mails photocopies to head office in the background.
Remote databases still have their place. When many different applications must read and update the same records, or when results must be queried in complex ways, a proper database is the better home. The rule of thumb is simple: keep working memory local, and publish finished answers to wherever they will be read.
4Where State Hides Inside Kafka Itself
Even a perfectly stateless application depends on plenty of remembered information inside Kafka. Knowing where it lives clears up a lot of confusion.
Kafka’s core idea is the log: an ordered, append-only list of events kept on disk. A topic is a named log, split into partitions, which are smaller logs that can sit on different servers. The brokers (Kafka’s servers) store these logs for days, weeks or even forever. In a sense, the whole cluster is one large piece of remembered state.
The small memories that make everything work
Offsets
Each consumer group records how far it has read in every partition. The record is saved in an internal Kafka topic, so a restarted program knows where to resume.
Idempotence numbers
With idempotence switched on, the broker remembers a producer’s identity and a running sequence number, so accidental duplicate sends are ignored.
Transaction log
A coordinator keeps an internal topic recording which transactions are open, committed or aborted.
Membership
A group coordinator tracks which consumers belong to a group and which partitions each one owns.
A postal service is a good comparison. A postman delivering letters is stateless: he reads the address on each envelope and delivers it. Behind him, the sorting office keeps ledgers of parcels, routes and delivery receipts. The postman’s job is stateless, yet the postal system as a whole is full of stored records.
Logs and tables: two views of the same data
Kafka gives you a neat trick. A topic with log compaction switched on keeps only the newest event for every key, and quietly removes older ones. Such a topic behaves like a table of latest values, for example “the current email address of each user”. The stream of changes and the table of current values are two views of the same data. This idea is called stream-table duality, and it is the reason Kafka Streams can rebuild a table by replaying a topic.
So what do we call the application?
The common way to speak about it is simple. If the processing code itself keeps notes that depend on earlier events, the application is stateful. If it only reads from and writes to Kafka without keeping such notes, it is stateless, and Kafka’s own internal bookkeeping is not counted against it. Keep this distinction in mind, because it explains why a “stateless” service can still be sensitive to broker outages, while it stays cheap to move around.
When you design a system, list the notes each component keeps. If the list is empty, you have a stateless component. If it is long, treat that component with extra care from day one.
5Meet Kafka Streams: Where the Two Worlds Meet
To see stateless and stateful work side by side, we need a place where both live in one program. That place is Kafka Streams.
Kafka Streams is a library, not a separate server. You add it to your own Java or Kotlin application, describe what should happen to your events, and run the application like any other program. Nothing extra has to be installed on the Kafka cluster. You can start ten copies of the same program, and they team up automatically.
Topology: the recipe for your data
The list of steps your events pass through is called a topology. Think of it as a recipe card: read from this topic, filter, map, count, then write to that topic. Kafka Streams reads the recipe and figures out how to run it in parallel.
Tasks, threads and instances
Kafka Streams cuts your recipe into pieces of work called tasks. A task handles exactly one partition of the input. If your input topic has twelve partitions, your application has twelve tasks for that part of the recipe, and never more. Tasks are the smallest units that can be moved around.
Tasks are run by threads (workers inside one program), and programs are called instances. All instances that share the same application.id form a single team. Kafka splits the tasks among the team members fairly. If one instance disappears, its tasks are handed to the others.
Imagine a call centre with twelve phone lines. Each line is one partition, and each agent handling a line is one task. Agents sit at desks (threads) inside offices (instances). You can move an agent from one office to another, but you can never put two agents on the same phone line, and extra desks beyond twelve stay empty.
The three data shapes
Every event matters
An endless list of facts, each one independent. Ten page-view events mean ten page views.
Latest value per key
A table where a new event for a key replaces the old one. It is backed by a state store, so it is stateful by nature.
Full copy everywhere
Like a KTable, but every instance stores all partitions. Great for small lookup data such as country codes.
The dividing line
In this library the dividing line is easy to spot. Operations that only look at the current record, such as filtering or mapping, need no store. Operations that group, count, combine, window or join need a store. Kafka Streams creates and manages the store for you, but it still costs disk, memory and recovery time.
flowchart TB
subgraph Team["Application team: same application id"]
subgraph I1["Instance A"]
T1["Task 0: partition 0"]
T2["Task 1: partition 1"]
end
subgraph I2["Instance B"]
T3["Task 2: partition 2"]
T4["Task 3: partition 3"]
end
end
IN["Input topic with 4 partitions"] --> T1
IN --> T2
IN --> T3
IN --> T4
Choosing your tool: library, SQL or plain consumer
Kafka Streams is not the only way to process events. Three common choices exist, and each suits a different situation.
Maximum control
You read messages and write all logic yourself. Best for simple stateless jobs, such as validating messages and saving them. Any state you need, you must build and protect yourself.
Library with built-in state
You get stores, changelog backups, windows, joins and recovery for free. Best when you are comfortable with Java or Kotlin and want to embed processing in an ordinary service.
SQL-style statements
You describe filters, aggregations and joins in a familiar query language. Best for teams who prefer queries over code, with the same state ideas underneath.
Cooking dinner can mean chopping and cooking everything yourself (a plain consumer), using a kitchen with good appliances and a recipe book (Kafka Streams), or ordering from a menu and letting the kitchen decide how to prepare it (ksqlDB). Each way reaches dinner; they differ in how much you control and how much you carry.
6Stateless Operators Under the Microscope
Stateless operators look simple, yet a few details decide whether your pipeline stays cheap or quietly becomes expensive.
One record in, zero or more records out
Inside a task, records travel from step to step one at a time, straight through memory. There is no waiting room and no disk write between stateless steps. That is why these steps are fast: the cost of a filter or a map is roughly the cost of running a tiny function.
The key matters: map versus mapValues
Every Kafka record carries a key and a value. The key decides which partition the record belongs to, and therefore which task handles it. Two related operations behave very differently because of this.
| Operation | What it can change | Effect on partitioning |
|---|---|---|
| mapValues | Only the value | Key stays the same, so data stays where it is |
| map | Key and value | Kafka must assume the key changed, so later grouping may trigger a data reshuffle |
| filter | Nothing, only keeps or drops | Key stays the same |
| selectKey | Only the key | Marks the stream for repartitioning before the next stateful step |
If you only need to edit the value, use the value-only version of an operation. It tells Kafka Streams that the key is untouched, which avoids an unnecessary reshuffle later.
Branching and merging
Branching splits one stream into several based on rules, and merging joins several streams back into one. Both are stateless because they never compare records with one another. A merge simply interleaves whatever arrives, in the order it arrives. This makes it a cheap way to combine, for example, orders from a website topic and a mobile-app topic into one stream.
Side effects and the honesty rule
Some stateless steps run side actions such as writing a log line, updating a metric or calling another service. The stream processing guarantees of Kafka do not extend to the outside world. If a task crashes and re-reads a few records, your side action may run twice. Keep such actions harmless when repeated, a property called idempotence.
Suppose a school office stamps every homework sheet with the word “received”. Stamping twice does no damage, because the sheet still says “received”. Now suppose the office also sends a parent a text message per sheet. If the sheet is processed twice, the parent gets two texts. Stamps are idempotent; texts are not.
What people do
They hide a private counter or a small list inside a stateless step, such as a variable in the program that grows as events arrive.
Why it hurts
That variable is invisible to Kafka. It is not backed up, not moved when tasks move, and lost on every restart. Results silently become wrong after a crash or a rebalance, and nobody notices until a report looks odd.
What to do instead
If you need memory, use a proper state store so Kafka can protect and move it.
7Stateful Operators: Grouping, Counting and Aggregating
Aggregation is the classic stateful job: many events go in, one running answer per key comes out.
Step one: group by a key
Before you can count or add anything, events must be grouped so that all events for the same thing meet in the same place. Grouping by the existing key is cheap. Grouping by a brand-new key forces Kafka Streams to move every record to the partition owned by its new key, which we will study in chapter 12.
Step two: choose how to combine
Add one each time
The store holds a number per key. Each new event increases it by one.
Merge same-type values
Combine two values of the same type into one, such as adding two amounts or keeping the larger one.
Build a custom result
Start with an empty result and fold every new event into it. The result type can differ from the event type, for example a list of last five purchases.
The result is a table, not a stream
An aggregation produces a KTable: for each key, the latest running answer. Every time a new event changes an answer, a fresh version of that answer flows downstream. Downstream readers see a stream of updates, such as “user 42 has 7 orders”, then “user 42 has 8 orders”.
Think of a school scoreboard in a tournament. Each team has a row with its current points. When a goal is scored, the row for that team changes, and the change is announced over the loudspeaker. The board is the state; the announcements are the updates that flow onward.
Caching: fewer noisy updates
If a popular key changes a thousand times a second, sending a thousand updates downstream would be wasteful. Kafka Streams keeps a small in-memory record cache in front of each store and merges rapid updates to the same key, forwarding only the latest one when the cache is flushed. The flush happens when the cache fills or when the commit interval passes. By default the cache is about ten megabytes per instance, shared across its threads, and it can be tuned or turned off.
The trade-off is timing. A bigger cache reduces traffic and cost, but downstream systems may see results a little later.
Benefits of aggregation
- Answers questions no single event can answer.
- Results are continuously updated instead of computed nightly.
- Downstream systems get small summaries rather than raw firehoses.
Costs of aggregation
- Needs a store per group of keys, which grows with the number of distinct keys.
- A “hot key” with huge traffic can overload the single task that owns it.
- Changing the aggregation logic later may require rebuilding the store from scratch.
LinkedIn and activity counts
Kafka was created at LinkedIn to carry activity events such as page views and searches, and the company has said its clusters now move trillions of messages per day. Turning that torrent into useful numbers, such as views per profile or searches per keyword, is a grouping and counting problem, and counting is the simplest form of state.
Count the number of distinct keys, not just the number of events. One hundred million events spread over ten keys need a tiny store. Ten million events with ten million different keys need a huge one.
Looking inside the notebook: interactive queries
A state store is not just an internal detail. Kafka Streams can let other programs read the store directly, a feature called interactive queries. A web service could ask, “how many orders has customer 42 placed today?” and get an answer straight from the running application, without a separate database.
There is a catch. Each instance holds only the keys for the tasks it owns, so the right instance must be found first. Kafka Streams can tell you which instance owns a given key, and your service then forwards the question to that instance. During a restore or a rebalance, a store may be temporarily unavailable, so callers should be ready to retry.
Imagine a large library where each librarian looks after certain shelves. To find a book, you first ask the front desk which librarian owns that shelf, then you go to that librarian. If a librarian is on a break while shelves are handed over, you wait a moment and ask again.
Answers from interactive queries are only as fresh as the last processed event. Treat them as “very recent” rather than “perfectly current”, especially right after a failure.
8Windowing: Remembering Only a Slice of Time
Real questions are often about a period, such as “how many orders in the last five minutes?” Windowing lets state cover time slices instead of all of history.
A window is a bucket of time. Events whose timestamps fall inside the bucket are grouped together, and the running answer is kept separately for each key and each bucket. Because old buckets eventually close, the state stays bounded instead of growing forever.
The four kinds of windows
| Window type | How it behaves | Typical use |
|---|---|---|
| Tumbling | Fixed size, no overlap, each event lands in exactly one window | Orders per minute |
| Hopping | Fixed size, but new windows start more often than old ones end, so events land in several windows | Moving average updated every minute over ten minutes |
| Sliding | Windows defined by the time gap between events, evaluated as records arrive | Detect two events within thirty seconds of each other |
| Session | Grows while activity continues and closes after a quiet gap | A user’s visit to a website |
A tumbling window is like a bus that leaves every hour on the hour: you belong to exactly one bus. A hopping window is like buses that leave every ten minutes but each one holds an hour of passengers, so a passenger can appear on several buses. A session window is like a conversation between friends: it lasts as long as people keep talking and ends after a long silence.
Event time and late arrivals
Windows use event time: the time recorded in the event, not the time it reached your program. Networks are messy, so an event stamped 10:04 may arrive at 10:07. Kafka Streams tracks stream time, the largest timestamp it has seen so far. A window stays open for a little while after its scheduled end, called the grace period. Events arriving within the grace period are still counted. Events arriving after it are dropped, because the window has closed.
Getting only the final answer
By default, a windowed count emits an updated result every time it changes. Sometimes you want just one answer when the window ends, for example to send one alert per minute rather than sixty. A “suppress” step holds results back until the window is closed. The price is memory and a delay: results appear only after the window end plus grace.
flowchart LR
E["Events with timestamps"] --> W1["Window 10:00 to 10:01"]
E --> W2["Window 10:01 to 10:02"]
E --> W3["Window 10:02 to 10:03"]
W1 --> S1[("Count per key for 10:00 window")]
W2 --> S2[("Count per key for 10:01 window")]
W3 --> S3[("Count per key for 10:02 window")]
S1 --> R["Results topic"]
S2 --> R
S3 --> R
Ride-hailing demand per area
Ride-hailing companies such as Uber have publicly discussed calculating supply and demand in small areas in near real time to guide pricing and driver positioning. That calls for counts per neighbourhood over short windows, exactly the pattern shown above. Old windows are discarded as time moves on, so memory does not grow without limit.
Choose the grace period by asking, “How late do my slowest events really arrive?” Measure it, do not guess. A very long grace period keeps windows open longer and increases memory use, while a very short one drops legitimate late data.
Three clocks: which “time” does a window use?
Every event can be tied to three different moments, and mixing them up is a classic source of confusing results.
| Clock | Meaning | Strength | Weakness |
|---|---|---|---|
| Event time | When the thing really happened, as stamped by the source | Reflects reality, even when events arrive late | Needs a grace period and can arrive out of order |
| Ingestion time | When the broker stored the event | Always increasing within a partition | Hides delays before the event reached Kafka |
| Processing time | When your program handled the event | Simple to reason about | Changes if you replay old data, so results differ from run to run |
A letter has three dates. The date written at the top is event time. The postmark is ingestion time. The day you finally open the envelope is processing time. If you want to know when your friend had her birthday party, only the date written on the letter tells you the truth.
Kafka Streams uses event time by default, taken from the timestamp stored in each record, and it allows a custom rule for reading timestamps from inside the event body. Event time has a lovely side effect: if you replay old data to rebuild a store, windows fall in the same places as they did the first time, so the results are repeatable.
9Joins: When Two Sources Must Meet
A join combines information from two sources. Since the two halves rarely arrive at the same instant, the program has to remember one side while waiting for the other.
Why joins are always stateful
Suppose orders arrive on one topic and customer details on another. When an order arrives, the program must find the matching customer. It can only do that if it has stored the customer details earlier. That stored data is the state. Even the simplest join therefore needs a store.
The main join families
Both sides are events
Matches events from two streams that happen within a time window of each other. Example: an ad impression and the click that followed within ten minutes. Both sides are stored for the window duration.
Enrich each event
Each incoming event looks up the latest value in a table. Example: add the customer’s city to every order. Only the table side is stored.
Keep two views aligned
Produces a table that updates whenever either side changes. A foreign-key variant lets you join on a field inside the value rather than on the key.
Small lookup data
Every instance holds a full copy of the table, so no reshuffle is needed. Best for small, slowly changing data.
A stream-table join is like a waiter with a seating chart. Each order that arrives is matched against the chart to find the table number. A stream-stream join is like two friends looking for each other in a busy market: each must remember where they last saw the other, but only for a limited time before giving up and going home.
The co-partitioning rule
For most joins, records that must meet have to sit in partitions handled by the same task. That requires two things. The two topics must have the same number of partitions, and both must be split using the same key logic, so that order 500 for customer 42 and the profile of customer 42 end up together. This requirement is called co-partitioning. If it is broken, matches are silently missed, which is one of the most confusing bugs beginners meet.
flowchart LR
O["Orders topic: key is customer id"] --> J["Join step"]
C["Customers topic: key is customer id"] --> ST[("State store: latest customer details")]
ST --> J
J --> EO["Enriched orders topic"]
ST -.-> CL["Changelog or source topic for recovery"]
Good to know
- Stream-table joins are cheap because only one side is stored.
- Global tables avoid reshuffles for small reference data.
- Joins let you avoid calling a database once per event, which removes a slow network hop.
Things that bite
- Stream-stream joins store both sides for the whole window, so memory can balloon.
- Mismatched partition counts break correctness.
- A late customer update does not change orders already enriched and sent.
The order of arrival matters in stream-table joins. If an order arrives before the customer record does, the lookup finds nothing. Plan for this by loading reference data first, or by choosing a join type that handles missing matches.
Custom stateful logic when the built-in operators are not enough
Sometimes the ready-made operators do not fit. You may want to detect three failed logins within two minutes, or hold a record until a partner event arrives. Kafka Streams offers a lower-level programming interface where you read and write your own state stores directly, and you can schedule a small function to run at regular intervals. That function can clean out old entries or emit summaries, even when no new events arrive.
This freedom comes with responsibility. You decide what to store, how big entries may grow and when to delete them. The library still protects the store through the changelog, but it cannot stop you from storing too much. Custom logic is the place where unbounded growth, forgotten clean-up and accidental duplicates most often appear, so keep such code small, well tested and documented.
Spotting repeated failures
A payments team wants to flag a customer whose card fails three times within five minutes. A custom step keeps a short list of recent failure times per customer, adds each new failure, removes entries older than five minutes and raises an alert when the list reaches three. A scheduled clean-up removes customers with empty lists, so the store never fills with people who stopped having problems.
10Inside a State Store: Disk, Cache and Changelog
The state store is the notebook of a stateful application. Understanding its three layers explains most of the speed, cost and recovery behaviour you will see in production.
Layer one: the fast local database
By default, Kafka Streams keeps each store in RocksDB, an embedded database created for fast writes on modern disks. “Embedded” means it runs inside your application process rather than as a separate server, so reading a note takes microseconds, not the milliseconds a network call would need. The files live in a folder on the local disk called the state directory.
You can also choose an in-memory store. It is faster and needs no disk, but its size is limited by available memory, and everything must be rebuilt after each restart.
Layer two: the record cache
In front of the database sits the small record cache from chapter 7. Rapid changes to the same key are merged here before being written and forwarded. This cuts disk writes and downstream traffic.
Layer three: the changelog topic
Every change to a store is also sent to a hidden Kafka topic called a changelog topic. This topic uses log compaction, so it keeps the newest value per key and forgets older ones, which keeps it from growing without limit. It is the durable backup of the store. Because Kafka itself replicates topics across several brokers, the backup survives the loss of any one machine.
Imagine a shopkeeper who writes sales on a small whiteboard by the till (the cache), copies the running totals into a ledger on the desk (the local database), and mails a photocopy of every ledger change to head office (the changelog). If the shop burns down, head office can send the photocopies to a new shop, and the ledger is rebuilt page by page.
sequenceDiagram
participant IN as Input topic
participant T as Stream task
participant C as Record cache
participant R as RocksDB store
participant CL as Changelog topic
IN->>T: New event for key A
T->>C: Read and update value for A
C->>R: Flush on commit or when full
R->>CL: Send change for A to Kafka
T->>IN: Commit input position
Store shapes
Plain lookup
One current value per key. Used by counts, reduces and tables.
Value per key per time slot
Old slots are removed after a retention period, which keeps size bounded.
Value per burst of activity
Sessions can merge when late events join two neighbouring bursts.
The memory surprise
RocksDB keeps part of its data in memory outside the Java heap, in write buffers and a block cache. A team can size the Java heap perfectly and still see the container killed for using too much total memory. Stateful applications therefore need memory limits planned for the heap and the database together, with many instances of RocksDB (one per store per task) sharing that budget.
The New York Times and the replayable log
The New York Times has described keeping every published piece of content in a Kafka log that acts as its source of truth. New services can build their own view of the archive by reading the log from the start. That is the same idea as a changelog topic at a larger scale: state is disposable, because it can always be rebuilt from the log.
Treat local state as a cache of the changelog, not as the master copy. That mindset makes every failure scenario easier to reason about: the master is in Kafka.
Compaction, deletion markers and retention
Changelog topics rely on log compaction. Kafka periodically scans the log and keeps only the newest record for each key. If your counter for customer 42 was updated a thousand times, only the latest value has to be kept for recovery, so restoring reads a much shorter history than the number of events ever processed.
Removing a key works through a tombstone: a record with a key and an empty value. When a store entry is deleted, Kafka Streams writes a tombstone to the changelog. Compaction later removes both the tombstone and the older values, after a configurable waiting time. Until that clean-up finishes, the deleted key still occupies some space.
Window stores use a slightly different rule. Their changelogs combine compaction with time-based deletion, so entries for windows older than the retention period eventually disappear. This is what keeps windowed state bounded and why choosing retention wisely matters.
Think of a sign-in sheet for a club where members cross out their old address and write a new one beside their name. Compaction is the clerk who rewrites the sheet each week, keeping only the newest address per member. A tombstone is a member writing “moved away”, after which the clerk eventually removes that whole line.
Because compaction keeps only the newest value per key, a changelog topic is usually much smaller than the raw input, which is one reason restores are practical even for large tables.
11Failures and Recovery: Where the Two Worlds Really Differ
On a calm day, stateless and stateful apps look similar. On a bad day, they behave very differently.
What happens when a stateless instance dies
Kafka notices the missing instance and hands its partitions to the survivors. Each survivor continues from the last saved reading position. There is nothing to rebuild, so the pause is typically a few seconds while the group reorganizes. This is why stateless services fit neatly with automatic scaling, where machines come and go all day.
What happens when a stateful instance dies
The survivors receive the tasks, but they do not have the notebooks. Each new owner must read the changelog topic from the start and replay every change into a fresh local store. This is called state restoration. For a store of a few megabytes, it takes seconds. For a store of hundreds of gigabytes, it can take many minutes or even hours, and during that time those keys produce no output.
An instance disappears
A crash, a deployment or a network problem removes it from the group.
Tasks are reassigned
The group coordinator redistributes partitions and their tasks among the remaining members.
New owners restore state
Each new owner replays the changelog into a local store. Processing for that task waits until this finishes.
Processing resumes
The task continues from its saved input position, and results flow again.
Ways to shorten the pause
A ready spare
Setting num.standby.replicas above zero asks another instance to keep a live copy of the store by following the changelog. If the main owner fails, the standby takes over almost immediately. The cost is extra disk, network and processing.
Prepare before moving
When tasks must move to a new machine, the old owner keeps working while the new one catches up. Only when it is nearly caught up does ownership switch, which avoids long silent gaps during scaling.
Quick restarts
Giving each instance a permanent identity lets a quickly restarted instance reclaim its old tasks without triggering a full reshuffle.
Keep the notebook
On container platforms, mounting the state directory on durable storage lets a restarted instance reuse its files and replay only the newest changes.
A relay-race team can keep a spare runner who jogs alongside the real runner for the whole lap. If the runner trips, the spare grabs the baton immediately. Without a spare, someone must be called from the bench, put on shoes and warm up, and the race stands still meanwhile.
flowchart LR
CL["Changelog topic in Kafka"] --> A["Instance A: active owner of Task 3"]
CL --> B["Instance B: standby copy of Task 3"]
A -. "crash" .-> X["Instance A lost"]
B --> N["Instance B becomes active owner"]
N --> OUT["Processing resumes quickly"]
With standbys
- Failover takes seconds instead of minutes.
- Rolling deployments hurt much less.
- Great for large stores and strict availability goals.
Without standbys
- Lower cost and simpler operations.
- Every failure means a full restore for the affected tasks.
- Fine for small stores and forgiving business needs.
Rolling restarts of a stateful application without any of these protections can cause a chain reaction: each restarted instance triggers a reshuffle, each reshuffle triggers restores, and the whole application spends its time rebuilding notebooks instead of doing work.
12Repartitioning: The Hidden Cost Between Steps
A stateless step called “select a new key” looks free. Its real cost appears one step later, when a stateful operation needs matching keys to sit together.
Why records must move
State is stored per task, and each task owns certain partitions. A count for “customer 42” must be updated by the one task that owns customer 42. If you change the key from “customer” to “product”, the records are still sitting in partitions chosen by the old key. Counting per product would then produce several partial counts in different tasks, all wrong.
To fix this, Kafka Streams writes the records to an internal repartition topic, keyed by the new key, and then reads them back. Now all records for one product sit in the same partition and the same task.
Imagine a post office that sorts letters into pigeonholes by street name. Tomorrow a manager asks for a count per postcode. Since letters are filed by street, letters for one postcode are scattered across many pigeonholes. Someone must re-sort every letter by postcode before anyone can count. That re-sorting is a repartition.
flowchart LR
IN["Input topic: key is customer id"] --> SK["Select new key: product id"]
SK --> RT["Repartition topic in Kafka: key is product id"]
RT --> CT["Count per product"]
CT --> ST[("State store: count per product")]
CT --> OUT["Output topic"]
What it costs
How to keep it under control
- Choose keys at the source. If most downstream work groups by product, publish events keyed by product in the first place.
- Prefer value-only edits. They keep the key, and therefore the layout, intact.
- Filter before reshuffling. Dropping unwanted events with a stateless filter first means fewer records travel through the repartition topic.
- Name your operators. Give stores and repartition steps explicit names. Automatic names depend on the position of a step in the recipe, so inserting a new step can rename internal topics and orphan old state.
What people do
They change the key several times in one pipeline, grouping first by one field, then another, then a third, each with its own count.
Why it hurts
Every change of key means a fresh repartition topic and another full copy of the data moving through Kafka. Costs and delays add up quickly.
What to do instead
Split the work into separate applications or output topics, each keyed the right way from the start, or redesign the events so a single key serves most needs.
Changing data formats and stored state
State stores hold data in a specific format. If you later change the shape of a stored value, for example by adding a field or renaming one, older entries in the store and the changelog still have the old shape. A new version of your application must be able to read both shapes, or the store must be rebuilt.
The safest habit is to evolve formats in a backward-compatible way: add optional fields, avoid removing or renaming fields, and use a schema registry (a service that stores and checks the agreed data formats) to reject incompatible changes before they reach production. When a breaking change is unavoidable, treat it as a migration: run a new application version with a new application identifier so it builds fresh state from the input topic, compare its results with the old version, and then switch consumers over.
Stateless steps forgive format changes far more easily, because they hold nothing old. Stateful steps carry the past with them, so their formats deserve the same care as a database schema.
13Delivery Guarantees: Why Duplicates Hurt State Most
Crashes can cause the same event to be processed twice. For stateless steps, this is a nuisance. For stateful steps, it can quietly corrupt your numbers.
The two guarantee levels
With at-least-once processing, Kafka Streams promises never to lose an event, but it may repeat some after a crash. With exactly-once processing, switched on through the processing.guarantee setting, each event affects the results only once, even across crashes.
How a crash creates double counting
Consider a counting step at the moment of a crash. The program has updated the count in its store and sent the change to the changelog, but has not yet saved its reading position. On restart, it reads from the old position, sees the same events again, and adds them a second time. The count is now too high, and nobody is warned.
Event arrives
The task adds one to the count for the key and writes to the store and changelog.
Crash before saving position
The reading position has not been committed yet, so Kafka still thinks the event is unread.
Restart and re-read
The same event is read again and counted a second time under at-least-once.
Exactly-once prevents it
State changes, output records and the reading position are committed in one Kafka transaction. If the crash happens before the commit, all of it is discarded together.
Think of a bank transfer. Taking money from one account and adding it to another must happen together or not at all. If the system dies between the two steps, one customer loses money that never arrives. A transaction glues the two steps into one unbreakable action, and exactly-once processing does the same for state, output and position.
The trade-offs
Exactly-once
- Correct counts, sums and joins even after failures.
- Important for money, inventory and compliance.
- Downstream readers can be set to see only committed data.
At-least-once
- Lower latency and less overhead.
- Fine when repeats are harmless, such as approximate dashboards.
- Stateful results may drift upward after crashes.
With exactly-once turned on, Kafka Streams commits more often, by default every hundred milliseconds instead of every thirty seconds, so that transactions stay small. That gives fresher results but adds broker work, so measure the effect on your own cluster.
Stateless pipelines also benefit from exactly-once, because it prevents duplicate output records. If your stateless step feeds a payment system, duplicates are just as harmful as double counts.
14Stateful vs Stateless: The Side-by-Side View
Now that each piece is clear, here is the whole picture on one page.
| Question | Stateless | Stateful |
|---|---|---|
| Remembers earlier events? | No | Yes, in a state store |
| Typical operations | Filter, map, flatMap, branch, peek | Count, aggregate, window, join, deduplicate |
| Memory and disk needs | Small and steady | Grows with keys and windows |
| Recovery after a crash | Seconds | Seconds to hours without standbys |
| Scaling out | Add instances, up to the partition count | Same limit, plus state must move with tasks |
| Effect of duplicates | Repeated output | Corrupted totals unless exactly-once is on |
| Extra Kafka topics created | None | Changelog topics, and repartition topics when keys change |
| Testing effort | Low | Higher: needs time, order and failure cases |
| Upgrade risk | Low | Changing logic may need a store rebuild |
Deciding which one you need
Write the question in plain words
“Show me orders above one hundred dollars” is stateless. “Show me total spend per customer this week” is stateful.
Ask if the answer needs history
If the current event alone is enough, stay stateless and keep the system simple.
Push state to the smallest place
Do stateless filtering and trimming first, so the stateful step sees fewer, smaller records.
Plan for the notebook
Estimate distinct keys, record size and retention, and decide on standbys and delivery guarantees before launch.
The money view
Technical differences turn into budget differences. A stateless application costs mainly processor time and network traffic. A stateful application pays for those too, and then adds several more line items.
| Cost item | Stateless | Stateful |
|---|---|---|
| Compute | Low and predictable | Higher, because of database work and cache handling |
| Local disk | Almost none | Sized to the largest store, plus room for restores |
| Broker storage | Only input and output topics | Also changelog and repartition topics, multiplied by the replication factor |
| Standby copies | Not applicable | Extra disk, processor and network for each standby |
| Network | Input and output traffic | Adds repartition traffic and restore traffic |
| People time | Little operational care | Capacity planning, tuning, restore drills and careful upgrades |
None of this means state is too expensive. It means the cost should buy something. A count that saves a nightly batch job, or a join that removes thousands of database lookups per second, easily pays for its notebook. A stateful step added “just in case” does not.
Because Kafka replicates every topic, a changelog with a replication factor of three stores three copies of your notebook’s history across the brokers, before counting any standby replicas on application machines.
15Scaling and Performance
Both kinds of operations scale by adding instances. State makes the scaling decisions heavier and harder to undo.
The partition ceiling
The number of partitions in the input topic sets the maximum number of active tasks, as we saw in chapter 5. Twelve partitions can keep twelve tasks busy, and a thirteenth instance would sit idle. For stateless work this ceiling is easy to raise: create a topic with more partitions and restart.
For stateful work, raising it later is painful. The rule that turns a key into a partition number depends on the partition count. When the count changes, the same key can land in a different partition, and the notebook that belongs to that key is now in the wrong place. Old state, changelog topics and co-partitioned joins all become inconsistent. Teams therefore pick a generous partition count at the start, often with room for several years of growth.
A library files books on shelves by the first letter of the author’s surname, using twenty-six shelves. If the librarian suddenly switches to fifty shelves, every book’s home changes, and all the catalog cards pointing to shelf locations are wrong. Choosing the number of shelves carefully at the start avoids a giant re-shelving project.
Levers you can pull
More workers per instance
The num.stream.threads setting runs several tasks in parallel inside one program, which helps when machines have spare processor cores.
Merge repeated updates
A bigger record cache reduces disk writes and downstream chatter for busy keys, at the cost of slightly delayed output.
How often to save progress
Longer intervals mean less overhead but more repeated work after a crash. Shorter intervals mean the opposite.
Smaller values
Compact formats and trimmed fields shrink every store entry, every changelog record and every network hop.
Expire old data
Windowed stores drop old slots automatically. Set retention no longer than the business really needs.
RocksDB settings
Block cache size, write buffers and compaction settings can be tuned to match your read and write mix.
Hot keys
A hot key is a key that receives far more events than others, such as one very popular product on a sale day. All of its events go to one partition and one task, so that task becomes the bottleneck no matter how many instances you add. Common fixes include splitting the key into several sub-keys, counting partially and then combining the partial results, or pre-aggregating before the events reach the hot task.
Estimating state size before launch
A rough estimate saves painful surprises. Multiply the expected number of distinct keys by the average size of a stored entry, then multiply again by the number of stores in the topology. For windowed stores, also multiply by the number of open windows. Add the changelog copy on the brokers, and add standby copies if you use them. Compare the result with the disk on each machine, and ask how long a full restore would take at the speed your network and brokers allow.
Load-test with realistic key counts, not just realistic event rates. A test with ten distinct keys will look perfect and tell you nothing about a production workload with ten million.
16Monitoring, Logging and Security
Stateless applications are watched mostly for speed and errors. Stateful applications need extra eyes on storage, recovery and data protection.
What to watch for both kinds
- Consumer lag: how far behind the newest event the application is. Rising lag means it cannot keep up.
- Process rate and latency: how many records per second are handled and how long each takes.
- Error and dropped-record counts: records that failed or were skipped. Late events dropped by a closed window show up here, and a sudden rise is a warning.
- Rebalance frequency: frequent reshuffles suggest unstable instances or badly chosen timeouts.
Extra watch items for stateful applications
| Signal | What it tells you |
|---|---|
| Restore progress and duration | How long a failover or restart will really take |
| Local disk usage of the state directory | Whether stores are growing faster than planned |
| Database read and write rates and cache hit ratio | Whether tuning or more memory is needed |
| Changelog topic size and lag | Whether the safety copy is keeping up |
| Standby replica lag | Whether a standby could really take over quickly |
| Container memory versus heap use | Whether off-heap database memory is about to trigger a kill |
Monitoring a stateless app is like watching the speedometer of a car. Monitoring a stateful app also means watching the fuel gauge, the oil light and the boot: the load it carries grows and can weigh it down, so you check what is inside as well as how fast it moves.
Logging that helps in a crisis
Log the start and end of each state restoration, the reason for each rebalance, and the number of records dropped for lateness. When an incident happens at night, these three lines usually explain whether the application is sick or simply busy rebuilding its notebooks.
Security for remembered data
State turns your application into a data store, so it inherits data-store responsibilities.
Local files are readable
State directories hold the notes in plain form on local disk. Use disk encryption and strict file permissions, especially on shared machines.
Internal topics matter too
Changelog and repartition topics contain copies of your data. Give them the same access rules and encryption as the original topics.
Encrypt connections
Use encrypted connections and proper sign-in between your application and the brokers, so events cannot be read or changed on the way.
Store less
Remove or mask sensitive fields with a stateless step before the stateful step, so the notebook never contains data it should not hold.
Deleting a person’s data on request is harder for stateful applications. The information may sit in the input topic, in the store, in the changelog and in output topics. Plan how a deletion marker will travel through each of them.
A first-response checklist for stateful incidents
When an alert fires for a stateful application, a calm, fixed order of questions saves time. Work through these in turn.
Is lag growing?
If the application falls further behind every minute, it cannot keep up. Look at the input rate and the process rate together.
Is a rebalance happening?
Frequent or long rebalances stop processing. Check whether an instance is flapping or a timeout is too tight.
Is a restore running?
Tasks that are restoring produce no output. Check progress rather than restarting, because a restart can make it start again.
Is the disk nearly full?
Growing stores or a rebuild can fill the state directory. Free space or add capacity before anything else fails.
Is memory near its limit?
Container kills that repeat every few hours often point to database memory that was never budgeted.
Resist the urge to delete state “to fix things” during an incident. Deleting stores forces full restores, which may take much longer than the original problem and can turn a small slowdown into an outage.
17Design Patterns and Anti-Patterns
A handful of repeatable designs solve most everyday problems. A few common shortcuts cause most of the outages.
Patterns that work
Filter early, aggregate late
Drop unwanted events and trim fields with stateless steps first. The stateful step then handles less data and stores smaller entries.
Enrich with a table
Keep reference data in a table topic and join events against it, instead of calling a database for every event.
Dead-letter topic
Send events that cannot be processed to a separate topic for inspection, instead of stopping the whole pipeline. This is a stateless routing step.
Windowed deduplication
Remember event identifiers for a limited time and skip repeats. The window keeps the store bounded.
Pre-aggregate then combine
Count in small groups first, then combine those small counts. This spreads the load of hot keys.
Separate concerns
Run heavy stateful jobs in their own application, so their restores and restarts do not disturb simple stateless routing.
Anti-patterns to avoid
What people do
They aggregate by a key that never repeats or never expires, such as a unique request identifier, and keep every entry forever.
Why it hurts
The store grows without limit. Disk fills, restores take longer each month, and eventually the application cannot recover at all.
What to do instead
Use windows or session windows with sensible retention, or write explicit clean-up logic that removes entries once they are no longer needed.
What people do
They call an outside database or web service once per event inside a “stateless” step.
Why it hurts
Each call adds network delay, and the slowest call sets the speed of the whole pipeline. If the outside service is down, the stream stalls. Repeated calls after a crash also multiply side effects.
What to do instead
Bring the reference data into Kafka as a table and join against it locally, accepting the small cost of a store in return for speed and resilience.
What people do
They change the number of partitions of a live stateful topic or rename operators freely between releases.
Why it hurts
Keys move to different tasks, internal topics are renamed, and old state becomes orphaned. Results silently diverge from reality.
What to do instead
Plan partition counts early, name stores and steps explicitly, and treat topology changes that touch state as migrations with a proper rebuild plan.
18Best Practices and Common Mistakes
The lessons below come up again and again in real deployments.
Do
- Start with the simplest stateless design that answers the question.
- Estimate distinct keys, entry size and retention before you build.
- Turn on standby replicas for large stores with strict availability targets.
- Use exactly-once when a wrong number has real cost.
- Name every store and repartition step explicitly.
- Rehearse a full restore in a test environment and record how long it takes.
- Test with late, out-of-order and duplicate events, not just tidy ones.
Do not
- Hide counters or lists in ordinary program variables.
- Change keys casually or many times in one pipeline.
- Forget that stream-stream joins store both sides.
- Set the grace period without measuring real lateness.
- Give the container only heap-sized memory for a database-backed store.
- Restart all instances of a stateful app at the same moment.
- Store sensitive fields in state that you do not need.
The mistakes beginners meet first
Mismatched partitions in a join
Two topics with different partition counts quietly lose matches. Always verify counts and key rules before joining.
Surprise at restart time
A store that felt small in testing takes an hour to restore in production. Measure early.
Missing results in windows
Results seem to vanish because late events were dropped after the grace period. Watch the dropped-record metric.
Killed containers
Off-heap database memory pushes the container over its limit. Plan the total memory, not only the heap.
Wrong totals after a crash
At-least-once processing double counts. Switch to exactly-once or make the logic tolerant of repeats.
Testing stateful logic with confidence
Stateless logic can be tested like any function. Stateful logic needs a little more imagination, because the correct answer depends on history and on time. Kafka Streams ships with a test driver that runs your whole recipe inside a normal test, with no cluster required. You feed in events with timestamps that you choose, and you read the results back.
Because you control the timestamps, you can prove that a window closes when it should, that a late event inside the grace period is counted, and that one arriving after it is dropped. You do not have to wait real minutes for the clock to move.
Out-of-order events
Send events in a shuffled order and check the totals still match.
Repeated events
Send the same event twice and confirm your logic does what you intend.
Window edges
Test events exactly at, just before and just after a window boundary and its grace period.
Rebuild from scratch
In a staging cluster, wipe the local state and verify the rebuilt values equal the live ones.
Testing a stateful pipeline is like rehearsing a play. A single line can be checked alone, but the scene only makes sense when the actors say their lines in order, on time and after a missed cue. A good rehearsal deliberately makes things go wrong, so the opening night does not.
19Real-World Stories in One Place
Large companies have publicly shared how they use Kafka, and their stories map neatly onto the two families of operations.
Rabobank and customer alerts
Rabobank, a large Dutch bank, has publicly described building real-time customer alerts on Kafka and Kafka Streams. Sending a useful alert often means comparing the current event with recent history, such as checking whether a payment is unusual for that customer. That comparison needs stored context, so it is stateful. The first step, deciding which events are eligible for alerts, is stateless.
Netflix and event delivery
Netflix’s Keystone pipeline routes very large numbers of events from devices and services to analytics and storage systems. Its main work is routing and light processing, and these tasks depend on no earlier events. That keeps the fleet elastic and cheap to recover.
LinkedIn and activity counts
At LinkedIn, where Kafka was created, activity streams carry views, searches and interactions in enormous volumes. Filtering and routing happen statelessly. Turning the streams into counts and rankings per member or per topic is stateful, and the company must protect and size those stores carefully.
Pinterest and spend tracking
Keeping advertiser spend close to budget requires a running total per advertiser. That total is a classic aggregation, kept in state stores and protected by changelog topics.
The New York Times and the log as truth
By keeping published content in a Kafka log that can be replayed, the newspaper lets new services build their own stored views on demand. The stored view is state, yet it is disposable, because the log can recreate it whenever needed.
20Frequently Asked Questions
Short answers to the questions beginners ask most often.
21Summary and Key Takeaways
If you remember only a page of this guide, remember this page.
Stateless operations look at one event and forget it. Stateful operations remember earlier events in a state store so they can count, combine, window and join. Everything else in this guide follows from that single difference: cost, recovery, scaling limits and the care you must take.
Key Takeaways
- State is memory between events. If a result depends only on the current event, the step is stateless. If it needs earlier events, it is stateful.
- Stateless steps are cheap and flexible. They scale by adding instances, recover in seconds and are easy to test, but they cannot count, join or deduplicate.
- Stateful steps are powerful and demanding. They need state stores, changelog topics, disk and memory, and they take longer to recover.
- Repartitioning is the hidden cost. Changing the key before a stateful step adds an internal topic and an extra network trip, so choose keys early.
- Plan for failure. Use standby replicas, static membership and persistent volumes to shorten recovery, and rehearse a full restore before you need it.
- Use exactly-once when numbers matter. At-least-once can double count after a crash, and stateful results suffer most.
- Keep state small and bounded. Filter early, aggregate late, use windows with sensible retention, and never let a store grow without limits.