A quick, friendly refresher on the pieces that every join is built from.

Apache Kafka is a platform for moving and storing streams of events. An event is a small record of something that happened, like “customer 42 placed an order” or “device 9 reported a temperature”. Kafka keeps these events in an ordered log so that many applications can read them, now or later.

Why keys and partitions matter for joins

Everything you will read in this guide depends on a simple fact: Kafka decides where an event lives using its key. If two topics both use the customer number as the key, then customer 42 will always be in a predictable partition of each topic. That predictability is exactly what makes joining possible without asking a database over the network.

Two ways to process Kafka data

You can process Kafka data with a plain consumer that you write yourself, or with a stream processing library such as Kafka Streams, or with a SQL-style engine such as ksqlDB. Joins are a core feature of the latter two, and in this tutorial we will mostly use the vocabulary of Kafka Streams, because ksqlDB is built on top of it and follows the same rules.

Before joining a stream and a table, you must be clear about what each one means.

What is a stream?

A stream is an unending sequence of independent facts. Every new record is a new event, and nothing overwrites anything. If customer 42 places three orders, the stream contains three separate records. A stream answers the question “what happened, and in what order?”

What is a table?

A table holds the latest known value for each key. If customer 42 changes their address twice, the table shows only the newest address, because each new record for a key replaces the previous one. A table answers the question “what is the current state?” In Kafka Streams the table abstraction is called a KTable.

The stream-table duality

Here is a beautiful idea at the heart of Kafka Streams. A stream and a table are two sides of the same coin. If you replay every update ever made to a table, you get a stream, which is often called a changelog. If you apply a stream of updates one at a time, always keeping the latest value per key, you get a table. A bank statement is a stream of deposits and withdrawals, while your current balance is the table you get by adding them all up.

flowchart LR
  S[Stream of updates for each key] -->|keep the latest value per key| T[Table with current state]
  T -->|emit every change as a record| S
        

The idea in one sentence, followed by the details.

A stream-table join takes each incoming event from a stream and looks up the current value for the same key in a table. It then combines the two into a new, enriched event. The output is itself a stream.

A worked story

Suppose an online shop publishes an “orders” stream where each event contains an order number, a customer number as the key, and a total amount. It also maintains a “customers” table where the customer number is the key and the value holds a name, a city, and a loyalty level. The analytics team wants each order to arrive with the customer’s city and loyalty level attached so that they can compute regional sales and decide on discounts.

Without a join, every order would have to be sent on its own to a downstream system, which would then look up the customer in a database for every single order. With a stream-table join, the lookup happens inside the stream processing application, directly against a local copy of the customer table. The enriched order flows out into a new topic, ready for anyone to use.

The important asymmetry

A stream-table join is asymmetric. Only records arriving on the stream side produce output. If the table is updated, for example a customer moves to a new city, no output is produced at that moment. The new city will simply be used the next time an order arrives for that customer. This is a deliberate design choice, and it is the biggest thing that separates a stream-table join from other join types.

The trick that makes joins fast is that the table is stored locally, right next to the processing code.

sequenceDiagram
  participant TT as Table Topic
  participant SS as Local State Store
  participant ST as Stream Topic
  participant APP as Streams Task
  participant OUT as Output Topic
  TT->>APP: Customer update for key 42
  APP->>SS: Store latest value for key 42
  ST->>APP: Order event for key 42
  APP->>SS: Look up key 42
  SS-->>APP: Latest customer value
  APP->>APP: Merge order and customer with the joiner
  APP->>OUT: Write enriched order for key 42
        

Tasks and partitions

Kafka Streams divides its work into tasks. Each task handles one partition from each of the joined topics, so task number 3 reads partition 3 of the stream topic and partition 3 of the table topic. Each task keeps only its own slice of the table in its local state store. When you run several instances of your application, Kafka Streams spreads the tasks across them, so the table is effectively split across all the instances and the join is done in parallel.

Fault tolerance through changelog topics

Local state stores could be lost if a machine dies. To protect against that, Kafka Streams backs up state stores into internal Kafka topics called changelog topics. If an instance fails, another instance can rebuild the lost store by replaying the changelog. When the table is created directly from a source topic, Kafka Streams can even reuse that source topic as the recovery log instead of creating a second copy, which saves storage.

What happens when the lookup finds nothing? That depends on the join type.

Notice what is missing: there is no right join and no full outer join between a stream and a table. This follows from the asymmetry we discussed. Since table updates never trigger output, there is no moment at which a “right-side only” result could be created.

Choosing between inner and left

Use an inner join when an event without context is useless or invalid, such as a payment with no known account. Use a left join when you must never lose an event, even if enrichment is unavailable, such as an order that should still be recorded and routed for later repair. Many teams prefer left joins for critical business data, together with a check on the output that sends events lacking context to a separate topic for inspection.

A stream-table join only works when the two topics line up perfectly in how they are partitioned.

Recall that each task reads one partition of the stream topic and the matching partition of the table topic. For this to make sense, every event for customer 42 in the stream must sit in the same partition number as customer 42’s record in the table. If they land in different partitions, the task handling the stream event will look in its local store, not find customer 42, and produce a wrong result.

flowchart TD
  A[Stream topic partition 0] --> T0[Task 0]
  B[Table topic partition 0] --> T0
  C[Stream topic partition 1] --> T1[Task 1]
  D[Table topic partition 1] --> T1
  E[Stream topic partition 2] --> T2[Task 2]
  F[Table topic partition 2] --> T2
  T0 --> O[Output topic]
  T1 --> O
  T2 --> O
        

What if the key is not the join key?

Very often the stream is keyed by something else, such as the order number, while the table is keyed by customer number. To join, you must first change the key of the stream to the customer number. In Kafka Streams, changing the key marks the stream as needing repartitioning. The library then automatically writes the records into an internal repartition topic keyed by the new key, and reads them back so that they are properly co-partitioned with the table.

Accidents that break co-partitioning

When co-partitioning is inconvenient, a different kind of table can remove the requirement.

A GlobalKTable is a table that is fully copied to every instance of your application. Instead of each task holding only its slice, every instance holds the entire table, read from all partitions of the table topic. Because every instance has every key, the stream’s partitioning no longer matters.

What you gain

With a GlobalKTable there is no co-partitioning requirement. The two topics may have different partition counts. You can also join on a value inside the stream event rather than on its key, by supplying a small function that extracts the lookup key from each event. That means the order stream can stay keyed by order number, and you can still look up customers using the customer number found inside the order. No repartition topic is needed.

The trickiest part of stream-table joins is not the lookup itself. It is deciding which version of the table the event should see.

The core problem: two streams of time

The stream and the table arrive from two different topics, and Kafka gives no guarantee about the exact order in which records from different topics arrive. Imagine a customer signs up and immediately places an order. The customer record and the order event are published within milliseconds of each other. If the application happens to read the order first, the table has no customer yet, and the lookup finds nothing. With an inner join the order is dropped, and with a left join it is emitted without context.

sequenceDiagram
  participant P1 as Customer Service
  participant P2 as Order Service
  participant APP as Join Application
  P1->>APP: New customer record, arrives late
  P2->>APP: Order for that customer, arrives first
  APP->>APP: Lookup finds no customer
  APP-->>P2: Order dropped or emitted without context
  P1->>APP: Customer record finally arrives
  APP->>APP: Table now contains the customer, too late for the order
        

How Kafka Streams helps: timestamp-based ordering

Every Kafka record carries a timestamp. When a task reads from several co-partitioned inputs, Kafka Streams tries to process records in timestamp order across them, choosing the record with the smallest timestamp next. This is a best-effort attempt to keep the stream and the table roughly in step. So if the customer record has an earlier timestamp than the order, the customer will usually be applied to the table first, even if the order happened to be fetched sooner.

There is a configuration that controls how long a task waits for data when one of its input partitions is temporarily empty, to give slower partitions a chance to deliver an earlier-timestamped record. Setting a small wait can improve ordering accuracy at the cost of some latency, and the trade-off should be tested with your real traffic.

Versioned tables: asking “what was true then?”

A traditional KTable remembers only the latest value. If an order from ten minutes ago is processed late, for example after a restart or during a backlog, it will be joined with whatever the customer profile looks like right now, not what it looked like ten minutes ago. For many uses that is fine. For others, such as pricing or compliance, it produces subtly wrong answers.

Recent Kafka versions introduced versioned state stores, which keep several historical values per key along with their timestamps. When a table is stored this way, a stream-table join can look up the version that was valid at the stream event’s timestamp, instead of just the newest one. Newer releases also added an optional grace period on the stream side, which lets slightly out-of-order stream records still find the right historical table version. The exact feature availability depends on your Kafka version, so check the release notes before relying on it.

Practical defences against race conditions

Kafka offers several join families. Choosing the right one starts with understanding the differences.

When to prefer each

Stream-table joins are among the most common patterns in event-driven systems.

In all of these cases the pattern is identical: a fast, high-volume stream of thin events meets a slower, lower-volume table of rich context. The join brings them together right where the data is flowing, avoiding a database call for every event.

A repeatable checklist keeps you out of the common traps.

The local table is what makes joins fast, and also what you must look after.

Where the table lives

By default the table lives in an embedded key-value database on the local disk of each application instance, with a memory cache in front. Its size is roughly the total size of the table’s latest values, divided among the instances. A table with fifty million customers of a few hundred bytes each already needs many gigabytes across the group, so disk capacity matters.

What happens during a restart

If an instance restarts and its local files are intact, it only needs to catch up on records it missed. If the files are gone, as happens with containers that use temporary storage, the instance must rebuild the state by replaying the changelog or the source topic from the beginning. For large tables this can take minutes or even hours, during which the affected tasks cannot process events.

Ways to shorten recovery

Joins scale well when partitions are healthy, and struggle when they are not.

Scaling out

The number of partitions sets the ceiling for parallelism: with twelve partitions there are twelve tasks, and at most twelve instances can be usefully busy. Adding instances beyond that gains nothing. Because the table is split by partition too, adding instances also spreads the table storage, which is a real advantage over a GlobalKTable.

Watch for uneven keys

If a handful of keys produce most of the events, the tasks that own those keys become hot while others sit idle. Stream-table joins do not solve this. In fact, because a join requires all events for a key to go to the same task, splitting a hot key across partitions would break the lookup. Address unevenness upstream, in how the data is produced, or accept it and size the busiest task accordingly.

Speeding up the join itself

A join is only useful if you can trust what comes out of it.

Tombstones: how deletes work

In a table, a record with a key and a null value is called a tombstone, and it means “remove this key”. When such a record arrives on the table topic, the key is deleted from the state store. Later stream events for that key will find nothing, so an inner join drops them and a left join emits them without context. This is exactly what you want when a customer account has genuinely been closed, and exactly what you do not want if the tombstone was sent by mistake.

Records that are ignored

Stream records with a null key cannot be looked up, so they are dropped from the join. Records with a null value on the stream side are also dropped. These are counted in the application’s dropped-record metrics but generate no error. If your data sources sometimes emit missing keys, fix them upstream.

Exactly-once processing

Kafka Streams can be configured for exactly-once processing. In this mode, reading the input, updating the state store, and writing the output are treated as one atomic unit, so a crash and retry does not produce duplicate enriched events. This works within the Kafka world. If your downstream system is outside Kafka, you still need to make its writes safe against duplicates.

Determinism and reprocessing

Suppose you fix a bug and replay a week of the stream through the join. With a regular table, the replay is joined with whatever the table contains as the replay runs, which may be a mix of old and new values depending on how the table topic is re-read. This is one of the strongest reasons to use versioned tables or to design replays deliberately, for instance by replaying the table topic and the stream topic together from the same starting point.

A join that quietly drops data or leaks personal details is worse than no join at all.

What to watch

Security considerations

The join application needs permission to read the stream and table topics, and to create and use its internal topics, such as repartition and changelog topics, which are named after the application identifier. Grant access carefully, and follow the principle of least privilege.

Table data often contains personal information, and the local state store holds a copy of it on disk. Protect the disks with encryption at rest, restrict machine access, and apply the same retention and deletion rules as for the source. If a customer asks for their data to be erased, remember that the tombstone must reach the table topic and its internal copies, and that enriched results already written downstream are separate copies with their own retention.

A few habits keep joins healthy, and a few habits cause long nights.

Helpful patterns

Anti-patterns to avoid

Stream-table joins are one way to enrich events. It helps to know the others.

ksqlDB in a nutshell

ksqlDB lets you express the same join using SQL-style statements over streams and tables, without writing application code. It follows the same rules: the table must be joined on its primary key, the two sides must be co-partitioned, and it can automatically re-key the stream when you join on a non-key column. If your team prefers SQL and the logic is simple, it is an excellent way to get started, while Kafka Streams offers more control for complex logic.

Let us follow a realistic incident from first symptom to lasting fix.

A food-delivery company enriches “delivery requests” with a “restaurants” table so that each request carries the restaurant’s preparation time and address. After a routine release, the operations team notices that the number of enriched requests is only about seventy percent of the number of raw requests.

The lesson is that most stream-table join incidents are not about the join code. They are about keys, partitions, and timing, which is why the first debugging steps are always to compare volumes and examine the keys on both sides.

It is an operation that takes each event arriving on a stream, looks up the latest value for the same key in a table, and produces a combined result as a new stream. Only stream events trigger output.

No. A table update only changes the stored context. The new value is used the next time a stream event arrives for that key. If you need output on table changes too, consider a table-table join.

Each task reads one partition of each topic and keeps only that partition’s slice of the table locally. The stream event and its matching table record must be in the same partition number, or the local lookup will not find the entry.

You can repartition one of them through an intermediate topic so both have the same count, migrate to a topic with the right count, or use a GlobalKTable, which does not require co-partitioning but stores the whole table on every instance.

With a regular table, first re-key the stream so that the desired field becomes its key, which creates a repartition topic. With a GlobalKTable, you can supply a function that extracts the lookup key from each event. For joining on a non-key field of the table itself, a foreign-key table-table join is available.

An inner join emits output only when a match exists in the table. A left join emits output for every stream event, using an empty table part when no match exists.

Likely causes are an inner join with missing table entries, a race where the event arrived before its table record, mismatched key formats, null keys, or broken co-partitioning. Compare input and output counts, check dropped-record metrics, and inspect the keys.

A stream-table join produces a stream. Each output is a new event. Table-table joins, by contrast, produce a table.

A tombstone record on the table topic removes the key from the state store. After that, an inner join drops events for that key, and a left join emits them without context.

If local state is intact, it resumes quickly. If not, it rebuilds the table by replaying the changelog or source topic, which can take a while for large tables. Persistent storage and standby replicas reduce this delay.

Not necessarily. With a regular table, each replayed event is joined with the table’s state at processing time. For time-accurate results, replay the table history together with the stream, or use versioned tables where your Kafka version supports them.

Yes, in most cases. Compaction keeps the latest value per key indefinitely, so a rebuilt table is complete. Time-based deletion can erase keys that have not changed recently.

Rarely. Per-event calls add latency, create a dependency that can stall the pipeline, and put heavy load on the database. Streaming the database’s changes into Kafka and joining locally is usually faster and more resilient.

Yes. ksqlDB supports joining a stream with a table using SQL-style statements, following the same rules for keys and co-partitioning, and it can automatically repartition the stream when needed.

The keys. Make sure both sides use the same key with the same format and partitioning, and monitor the match ratio. Most problems trace back to keys that do not line up.

A stream-table join lets a Kafka application give every event the context it needs, using a local, always-updated copy of a table instead of a slow network call. The stream drives the output, the table provides the latest state, and the result is a new, enriched stream. It is powerful because it is fast, scalable, and fault tolerant, and it is delicate because it depends on keys, partitions, and timing being aligned.

If you remember only a handful of ideas, remember these: only the stream side triggers output, both sides must be co-partitioned unless you use a global table, timing races can cause missed matches, and the table topic should be compacted and monitored. Build with a left join and a repair path where events matter, keep an eye on match ratios, and treat topic keys and partition counts as a shared contract between teams.