Before defining a topic precisely, it helps to see the mess it prevents.

A flood of different kinds of data

A single company might use Kafka to move many different kinds of information at once: customer orders, payment confirmations, delivery updates, website clicks, and system error alerts, all flowing through the same Kafka cluster at the same time. If all of this data landed in one shared, unlabelled pile, any program trying to read it would have to sift through everything just to find the handful of events it actually cared about.

This is not just inconvenient, it is also unsafe. A program that only needs to read payment confirmations should not need to see customer support messages, and giving it access to everything would break the principle of giving each program only the access it truly needs.

The answer: named categories

Kafka solves this by letting data be organised into named categories from the very start. Whoever sends data decides which category it belongs to, and whoever wants to read data simply asks for the category they care about. This category is called a topic, and it is one of the very first ideas anyone learning Kafka needs to understand.

A precise definition, then the plain-English version.

A short definition

A topic in Kafka is a named stream of related events, similar to a category, a folder, or a channel, into which producers write data and from which consumers read it. A producer is any program that sends data into Kafka, and a consumer is any program that reads data out of it. Every single piece of data written to Kafka must belong to exactly one topic, chosen by whoever writes it.

In everyday terms, a topic is simply a label with a name, such as “orders” or “payments”, that groups together everything related to that subject. Anything published under the “orders” topic is treated as part of one continuous stream of order-related events, kept separate from everything happening in other topics.

Topics are not files, and not folders

It is tempting to think of a topic as a file that gets appended to, and that mental picture is a useful starting point, but a topic behaves a little differently. A topic can be split internally, spread across multiple machines, and read by many independent programs at the same time, none of which is normally true of an ordinary file. We will explore that internal structure shortly.

Understanding the contents of a topic makes everything else about Kafka easier to follow.

A topic is a sequence, not a single blob

A topic does not store one giant lump of data. It stores an ordered sequence of individual events, one after another, in the order they were written. New events are always added to the end of this sequence, never inserted in the middle and never edited once written. This append-only behaviour is a defining feature of how Kafka stores data.

A topic looks simple from the outside, but it hides a clever structure inside.

Splitting a topic into pieces

A single topic can grow far too large and busy for one machine to handle comfortably. To solve this, Kafka splits every topic into smaller pieces called partitions. Each partition is its own ordered, append-only sequence, and a topic is really just a named collection of one or more of these partitions working together.

Partitions can live on different machines, called brokers, which lets a topic’s workload be spread across an entire cluster instead of resting on a single computer. A topic named “orders” might have three partitions, each holding roughly a third of all order events, with all three working in parallel.

flowchart LR
  T
  T --> P0[Partition 0]
  T --> P1[Partition 1]
  T --> P2[Partition 2]
        

Order is guaranteed only within a partition

This is one of the most important facts about topics for a beginner to learn early. Kafka guarantees that events stay in the exact order they were written, but only inside a single partition. There is no guarantee about the order of events across different partitions of the same topic. If two events absolutely must stay in order relative to each other, they need to share the same key, since Kafka uses the key to decide which partition an event goes to, ensuring they land in the same partition together.

Every event inside a partition needs a way to be located precisely.

A running position number

Every event within a partition is given a number called an offset, starting at zero and increasing by one for each new event. This offset never changes for a given event and is never reused, even if older events are eventually removed. It works much like a page number in a book, letting anyone say precisely “start reading from position 500” or “I have already read up to position 750”.

Why offsets matter so much

Offsets are what let a consumer remember exactly where it stopped reading. If a consumer goes offline for an hour and then comes back, it can simply resume from the offset it last processed, rather than starting over or missing data. Offsets also let a consumer intentionally rewind and reread older events, which is extremely useful for fixing mistakes, testing new features against real historical data, or building an entirely new system from a topic’s existing history.

flowchart LR
  O0["Offset 0"] --> O1["Offset 1"] --> O2["Offset 2"] --> O3["Offset 3"] --> O4["Offset 4"]
        

A closer look at how data actually enters a topic.

Choosing where each event goes

When a producer sends an event, it must specify which topic the event belongs to. If the topic has more than one partition, the producer, or Kafka acting on its behalf, must also decide which specific partition receives that event. If the event has a key, Kafka runs the key through a formula so the same key always lands in the same partition, keeping related events together and in order. If there is no key, Kafka spreads events across the available partitions in a balanced way.

A producer does not need to know its readers

A producer’s job ends the moment its event is safely written to a topic. It does not need to know who will read that event, how many readers there are, or when they will read it. This separation between writing and reading is one of the reasons Kafka-based systems can grow and change so easily over time.

How programs actually pull events back out of a topic.

Subscribing to a topic

A consumer reads events by subscribing to one or more topics it cares about. Once subscribed, it can request events starting from a particular offset, often simply continuing from wherever it last left off. Multiple, completely unrelated consumers can read from the exact same topic independently, each keeping track of its own position, without disturbing one another in any way.

Consumer groups and shared work

Consumers are often organised into consumer groups, where the partitions of a topic are divided among the group’s members so the reading work is shared. If a topic has six partitions and a group has three consumers, each consumer typically handles two partitions. Different consumer groups are independent of each other, and each group receives its own complete view of the topic, at its own pace.

flowchart TD
  T[Topic: orders]
  T --> C1[Payment Service]
  T --> C2[Warehouse System]
  T --> C3[Analytics Pipeline]
        

A topic does not hold onto everything forever by default, and that is a choice you control.

Time-based and size-based retention

Every topic has a retention setting that decides how long events are kept before being removed. This is commonly measured in time, such as keeping events for seven days, or in total size, such as keeping up to a certain number of gigabytes per partition, after which the oldest events are discarded to make room for new ones. Teams choose retention based on how long downstream systems might realistically need to catch up or replay data.

Log compaction: keeping only the latest per key

Some topics do not need their full history at all, only the most recent value for each distinct key. For these cases, Kafka offers a special retention mode called log compaction, where older events are removed once a newer event with the same key has been written, keeping only the latest state for every key. This is especially useful for topics that represent the current state of something, such as the latest address on file for each customer.

A topic’s partitions are far too important to trust to a single machine.

Copies across multiple brokers

Each partition of a topic can be copied onto several brokers, a setting called the replication factor. A replication factor of three, a common production choice, means every event is stored on three different machines. If one broker fails, the data remains safe and available on the other copies.

Leaders and followers

For every partition, one copy is designated the leader, and the rest are followers that continuously copy whatever the leader receives. All reads and writes for that partition go through the leader. If the leader’s broker fails, one of the up-to-date followers is promoted to become the new leader, and producers and consumers automatically discover and adjust to the change, usually within moments.

One of the most important decisions made when a topic is created.

More partitions, more parallel work

The number of partitions a topic has directly limits how much of its work can happen in parallel. More partitions generally allow more consumers within a single group to work simultaneously, and can allow more producers to write at once without competing for the same resources. A topic with only one partition can never be read in parallel by more than one active consumer within a group, no matter how many consumers that group has.

The cost of too many partitions

More partitions are not automatically better. Each partition adds a small amount of overhead to every broker that hosts it, and having far more partitions than a cluster actually needs can slow down recovery after a failure and add unnecessary strain to the brokers managing them. Choosing the right number of partitions is a balance between expected parallel workload and practical overhead.

A topic is not just a name; it comes with settings that shape its behaviour.

Most of these settings can be adjusted after a topic is created, letting teams start with sensible defaults and refine them as real usage patterns become clearer.

A small amount of discipline here saves a lot of confusion later.

Why naming conventions matter

As a company adopts Kafka more widely, the number of topics tends to grow quickly, often into the hundreds. Without a clear naming convention, it becomes hard to tell what a topic contains just by looking at its name, who owns it, or which team is responsible for it. Many organisations settle on a structured naming pattern, such as combining the domain, the kind of event, and a version number.

One topic per distinct kind of event

A well-organised Kafka deployment generally favours many small, focused topics over a few large, mixed ones. A topic that mixes several unrelated kinds of events forces every consumer to filter through irrelevant data and makes access control much harder to manage cleanly.

Seeing a few concrete examples makes the concept click.

Lessons that save teams from painful surprises later on.

Best practices

Mistakes beginners often make

Not quite. A traditional queue usually removes a message once it has been read, and typically delivers each message to only one reader. A Kafka topic keeps its events for a chosen retention period and can be read independently by many different consumers.

Yes. A topic can have as few as one partition, which is simple and guarantees strict overall ordering, but it also limits how much parallel reading and writing that topic can support.

A single cluster can comfortably host many thousands of topics, though the practical limit depends on the cluster’s size, hardware, and total partition count across all topics combined.

Yes, partitions can be increased later, but this can affect how keys map to partitions, so it is generally safer to plan a reasonable partition count from the start rather than relying on frequent changes.

No. Reading an event from a topic does not remove it. The event remains available to be read again, by the same consumer or a different one, until it is eventually removed according to the topic’s retention rules.

A topic is the named category that consumers subscribe to and producers write to. A partition is one of the internal, ordered pieces that a topic is physically split into, allowing the topic’s data to be spread across a cluster.

Kafka itself does not enforce a structure on the data inside a topic. It is up to the producers and consumers, often guided by an agreed schema or format, to decide and maintain a consistent structure for that topic’s events.

A topic is the fundamental way Kafka organises the flow of data passing through it, giving every kind of event a clear, named home. Internally, a topic is built from one or more partitions, each an ordered, append-only sequence of events identified by increasing offsets. Producers write events into a topic, often guided by a key that decides the partition, while consumers subscribe to read those events independently, at their own pace, without removing them. Settings like replication, retention, and cleanup policy shape how safely and how long a topic’s data is kept, and thoughtful choices around partition count and naming pay off enormously as a Kafka deployment grows. Understanding topics deeply is the single most important step toward understanding Kafka as a whole.