What Are Partitions in Kafka?
A beginner-friendly guide to the slices that make Kafka fast, scalable, and ordered, and to the choices you must make about them before your first event is ever written.
Imagine a supermarket that opens with a single checkout counter. On a quiet Tuesday morning, it works fine. On a busy Saturday evening, the queue stretches out of the door and customers walk away. The owner solves the problem by opening twelve counters, and each cashier now serves their own queue at the same time. The shop can suddenly handle a crowd twelve times larger. Kafka faces exactly this challenge every day, with millions of events instead of customers, and its answer is the same: open many counters. In Kafka, those counters are called partitions. This guide explains what they are, how they work, and how to choose them wisely.
1The Problem Partitions Solve
To understand partitions, first picture what would happen without them.
One giant line of events
Apache Kafka stores streams of events, such as orders, payments, clicks, or sensor readings. Events are grouped into named categories called topics. Imagine a topic called “orders” that stored every order in one single, long line on one single computer. Every producer, which is an application that writes events, would have to push into that one line. Every consumer, which is an application that reads events, would have to read from that one line.
This design breaks down quickly. One computer has only so much disk space, so a huge topic might not fit. One computer has only so much network speed and processing power, so it could serve only a limited number of writers and readers. And if that one computer failed, the whole topic would vanish. A single line is a single point of limit and a single point of failure.
Think of a school with a single teacher for a thousand students. The teacher can only speak to so many students, can only mark so many papers, and if she falls ill, school is cancelled. A school with fifty classrooms and fifty teachers teaches far more students, and one absence does not close the school. Partitions are the classrooms of Kafka.
The Kafka answer
Kafka splits each topic into several smaller logs called partitions. Each partition can live on a different server, be written to by different producers, and be read by different consumers, all at the same time. The topic is still one logical thing to the applications, but underneath it is many parallel lines working together.
Applications talk about a topic by its name. Partitions are the hidden machinery that makes that topic big, fast, and reliable. Yet a good understanding of this machinery is what separates a smooth Kafka system from a troubled one.
2What Is a Partition?
A partition is one ordered, append-only sequence of events that forms a slice of a topic.
The simple definition
A partition is a single log, which is just a file-based list, where new events are always added at the end and never inserted in the middle. Once an event is written, it stays where it is and is not changed. A topic is made of one or more such logs. A topic with three partitions is three separate logs, numbered zero, one, and two.
Three key ideas
Ordered
Inside one partition, events keep the exact order in which they were written.
Append-only
New events go to the end. Old events are not edited. This makes writing very fast.
Independent
Each partition is its own log. Different partitions can be handled by different servers at the same time.
A partition is like a diary in which you only write on the next empty line. You never squeeze a sentence between two old ones and never erase yesterday’s entry. A topic with several partitions is like a family where each member keeps a separate diary, and all diaries together tell the family’s story.
flowchart LR
T["Topic: orders"] --> P0["Partition 0"]
T --> P1["Partition 1"]
T --> P2["Partition 2"]
P0 --> E0["Events 0, 1, 2, 3 ..."]
P1 --> E1["Events 0, 1, 2, 3 ..."]
P2 --> E2["Events 0, 1, 2, 3 ..."]
3Offsets: The Address of Every Event
Inside a partition, every event has a number that tells you exactly where it sits.
What is an offset?
An offset is a position number given to each event in a partition, starting from zero and growing by one each time a new event arrives. The first event is at offset zero, the next at offset one, and so on. Offsets never repeat inside a partition, and they never change.
The full address of an event
Because each partition has its own numbering, an offset alone is not enough to locate an event. The full address of an event is three things together: the topic name, the partition number, and the offset. For example, “orders, partition 2, offset 4051” points to exactly one event in the entire cluster.
Think of finding a book in a big library. You need the section, the shelf number, and the position on the shelf. Topic, partition, and offset are the section, shelf, and position of an event.
Bookmarks for consumers
Consumers use offsets as bookmarks. A consumer reads events in order and remembers the offset of the last event it finished. When it restarts, it continues from the bookmark. It can also move the bookmark backwards to read old events again, as long as those events have not yet been deleted by the retention rules.
Assuming that offset numbers are shared across the whole topic. Offsets are separate for every partition, so offset 10 in partition 0 and offset 10 in partition 1 are two completely different events.
4How Events Choose Their Partition
When a producer sends an event, someone must decide which partition receives it. That decision is made by the partitioner.
Three ways to choose
By key
If the event has a key, such as a customer number, the producer runs the key through a fixed calculation that always gives the same partition for the same key. All events for one customer therefore land in one partition, in order.
Without a key
If there is no key, the producer spreads events across partitions to keep them balanced. Modern clients place a batch of events on one partition and then move to another, which keeps batches large and efficient.
By explicit choice
The application can name the exact partition itself, or supply its own custom rule. This is rare and needs careful thought.
A postal sorting office sends letters by postcode. All letters for the same postcode always go to the same delivery van. That is key-based partitioning. Letters with no clear address are shared out evenly among vans to keep everyone equally busy.
flowchart TD
EV["New event from producer"] --> Q{"Does it have a key?"}
Q -->|"Yes"| H["Calculate from the key"]
H --> SP["Same key always goes to the same partition"]
Q -->|"No"| RR["Spread across partitions to balance load"]
Why the key matters
The key gives you a promise: everything with the same key stays together and stays in order. If a customer creates an account, changes the address, and closes the account, these three events must be handled in that order. Using the customer number as the key keeps them in one partition where their order is protected.
Pick a key that matches the thing whose events must stay in order, such as an order number, an account, or a device. A key that is too broad or too narrow leads to problems described later in this guide.
5Ordering Guarantees and Their Limits
Kafka makes a strong promise about order, but the promise has a boundary you must respect.
What Kafka promises
Inside one partition, events are stored in the order they were written and are delivered to consumers in that same order. If a producer sends event A and then event B to the same partition, every consumer will see A before B.
What Kafka does not promise
Across different partitions, there is no guaranteed order. If event A goes to partition 0 and event B goes to partition 1, a consumer may read B before A, because the two partitions are read independently and at different speeds. There is no single global clock that lines all partitions up.
In a supermarket with several counters, each queue moves in order. The customer at the front of queue three is served before the second customer in queue three. But you cannot say whether a customer in queue one is served before a customer in queue five, because those queues move independently.
Designing around the boundary
If your system needs strict order for one customer, one account, or one device, use that identity as the key so all its events share a partition. If your system needs strict order across everything, you would need a single partition, which gives up the speed that partitions bring. Most real designs accept ordering per key, which is usually all that the business needs.
| Requirement | Design Choice | Result |
|---|---|---|
| Order per customer | Use the customer number as key | Ordered for each customer, parallel across customers |
| Order for the whole topic | Use one partition | Perfect order, but limited speed and no parallel reading |
| No order needed | Send without a key | Best balance and simplest scaling |
Expecting global order from a topic with many partitions. Teams sometimes discover this only after events for the same customer were sent without a key and ended up scrambled across partitions.
6Partitions and Scalability
Partitions are the main reason Kafka can grow from a laptop demo to a system handling enormous traffic.
Spreading across servers
A Kafka cluster is a group of servers called brokers. When you create a topic with twelve partitions in a cluster of three brokers, Kafka spreads the partitions across the brokers, so each broker leads about four of them. Each broker then handles only its share of writes and reads. Need more capacity? Add more brokers and move some partitions onto them.
Parallel writing
Many producers can write to different partitions at the same time, and even to the same partition safely. Because each partition is handled by its own leader broker, the load of writing is divided.
Parallel reading
Consumers that share a group name divide the partitions among themselves. If a topic has six partitions and the group has three consumers, each reads two partitions. If the reading falls behind, you add more consumers, up to the number of partitions.
flowchart LR
subgraph Topic["Topic with 6 Partitions"]
P0["P0"]
P1["P1"]
P2["P2"]
P3["P3"]
P4["P4"]
P5["P5"]
end
P0 --> CA["Consumer A"]
P1 --> CA
P2 --> CB["Consumer B"]
P3 --> CB
P4 --> CC["Consumer C"]
P5 --> CC
A pile of exam papers is divided into six bundles. Three teachers each take two bundles and mark them together. If the deadline is tight, hire three more teachers, and each takes one bundle. But hiring a seventh teacher does not help, since there is no seventh bundle.
The partition count sets the ceiling for parallel reading inside one group. It is the single most important number for deciding how far a consumer group can scale.
7Replication: Every Partition Has Backups
Partitions are also the unit of safety. Each partition is copied to several brokers.
Leaders and followers
Each partition has several copies, called replicas, stored on different brokers. One copy is the leader, which receives all writes and serves reads. The other copies are followers, which continuously fetch new events from the leader to stay up to date. If the leader’s broker fails, one of the followers that is fully caught up becomes the new leader.
The replication factor
The replication factor is the total number of copies of each partition. A factor of three is common in production, meaning one leader and two followers. Kafka never puts two copies of the same partition on the same broker, because a single failure would then remove both.
A company keeps its most important contracts in three separate places. If a fire destroys one office, the other two still have the papers. Each partition is a contract with its own three copies.
flowchart TB
subgraph B1["Broker 1"]
A["P0 Leader"]
B["P1 Follower"]
end
subgraph B2["Broker 2"]
C["P1 Leader"]
D["P0 Follower"]
end
subgraph B3["Broker 3"]
E["P0 Follower"]
F["P1 Follower"]
end
A -.-> D
A -.-> E
C -.-> B
C -.-> F
The in-sync replica set
The followers that are fully caught up with the leader form the in-sync replica set. Only members of this set can be promoted to leader. This guarantees that the new leader has every event that was confirmed to the producer. A follower that has fallen behind is removed from the set until it catches up again.
Kafka tries to spread leaders evenly, so that no single broker leads all the busy partitions. When a failed broker returns, leadership can be handed back to it over time.
8Inside a Partition: Segments on Disk
Let us peek at how a partition is stored on a broker’s disk.
A log made of segments
A partition is not stored as one gigantic file. It is split into smaller files called segments. At any moment, only the newest segment, called the active segment, receives new events. When it grows large enough or old enough, Kafka closes it and starts a new one. The older, closed segments are never modified.
Why segments help
Segments make cleaning up old data easy. When events become older than the retention period, Kafka simply deletes whole old segment files instead of removing individual events. Deleting a file is fast and cheap. Segments also come with small index files that help Kafka jump quickly to a given offset or time, so reading from the middle of a partition does not require scanning from the start.
Consider a company that files its papers in monthly boxes. Only the current month’s box is open. When a box is full or the month ends, it is sealed and a new box begins. After seven years, the oldest boxes are thrown away in one go. Segments are those boxes.
Events are appended
New events are written at the end of the active segment, a very fast sequential disk operation.
The segment rolls
When the segment reaches a size or age limit, it is closed and a new active segment begins.
Old segments age
Closed segments stay on disk so consumers can still read or replay them.
Retention removes them
Once a whole segment is older than the retention time, or the partition exceeds its size limit, the segment file is deleted.
9Retention and Compaction
Kafka does not keep data forever unless you tell it to. Two policies decide what stays.
Delete by time or size
The common policy is deletion. You decide how long events are kept, for example seven days, or how large a partition may grow. Older segments beyond the limit are removed. This policy suits streams of events where history has a natural expiry, such as click logs.
Compaction by key
The other policy is log compaction. Here, Kafka keeps at least the latest event for every key and may remove older events with the same key. This is useful when a topic represents the latest known state of things, such as the current address of each customer. Even after a long time, a reader can rebuild the full current picture by reading the compacted topic.
| Policy | What Is Kept | Good For |
|---|---|---|
| Delete by time | Everything from the last chosen period | Logs, clicks, and events with short useful lives |
| Delete by size | Newest data up to a size limit per partition | Protecting disk space |
| Compaction | At least the latest event for each key | Current state, such as profiles or settings |
Delete-by-time is like a notice board where notices older than a week are taken down. Compaction is like a contact list where each person’s old phone numbers are crossed out, but the newest number always remains.
Retention works per partition. Total storage for a topic is roughly the volume of data times the retention period times the replication factor, spread over all partitions.
10How Many Partitions Should You Choose?
This is the question every Kafka beginner asks, and there is no magic number. There are, however, sensible ways to think about it.
Questions that guide the choice
How fast must we write?
Estimate the target throughput. If one partition handles a certain speed, divide the target by that speed to see how many partitions are needed.
How fast must we read?
If each consumer can process a certain amount per second, the partitions must allow enough consumers to reach the target.
How much will we grow?
Leave room for future growth, because changing the count later has side effects.
Do keys need order?
Remember that the mapping of keys to partitions depends on the number of partitions.
How big is the cluster?
The partitions should spread well across brokers, so the count is often a multiple of the broker count.
A simple way of thinking in words
Suppose you expect one hundred megabytes of new events per second. Testing shows that one partition comfortably takes ten megabytes per second on your hardware. You would need at least ten partitions to write at that speed. Suppose each consumer instance can process only five megabytes per second. To read one hundred megabytes per second, you would need twenty consumers, so you need at least twenty partitions. Take the larger number, then add some headroom for growth and busy days.
Choosing one partition “for now” on a topic that will surely grow. The limit on reading in parallel arrives quietly, and by then many applications depend on the topic.
Too many is also a problem
Bigger is not always better. Every partition uses file handles and memory on the brokers, needs its own leader election after failures, and adds work for the controller that manages the cluster. Recovery after a broker failure takes longer when a broker leads thousands of partitions. Producers also keep a separate batch buffer per partition, so more partitions can mean smaller batches and higher delay. Modern Kafka versions handle far more partitions than older ones, but the cost never disappears completely.
Advantages of More Partitions
- Higher possible write throughput
- More consumers can read in parallel
- Load spreads more evenly across brokers
- More room to grow without changes
Disadvantages of More Partitions
- More files and memory used on each broker
- Longer recovery time after a broker failure
- Smaller producer batches and slightly higher delay
- More work for cluster management and monitoring
- Order is guaranteed only inside each smaller slice
11Changing the Number of Partitions Later
Plans change. Here is what you can and cannot do after a topic exists.
You can add, but never remove
Kafka lets you increase the number of partitions of an existing topic. It does not let you decrease it. If you created far too many partitions, the usual remedy is to create a new topic with the right count and move the data across, which needs planning.
Adding partitions changes the key map
The rule that maps a key to a partition depends on the number of partitions. After you add more, the same key may now point to a different partition than before. New events for a customer may land in a new partition while their older events remain in the old one. For a while, order per key can appear broken across the boundary, because a consumer may read the new partition faster than the old one.
A post office assigns delivery vans by postcode. If it suddenly adds two more vans and redraws the routes, your street may now belong to a different van. Old letters already in the previous van still need delivering, so for a short time two vans serve your street, and the order of delivery can get muddled.
Problem
Adding partitions to a keyed topic in the middle of busy hours, while consumers depend on strict order per key.
Why It’s Harmful
Keys now map to new partitions, so events for the same key can be read out of order or processed by two consumers in the same window.
Correct Approach
Plan the count generously at creation. If you must grow, do it in a quiet period, or create a new topic with the larger count and migrate carefully.
12Hot Partitions and Uneven Load
Partitions only help if the load is shared fairly. Sometimes one partition gets far more work than the rest.
What is a hot partition?
A hot partition is one that receives a disproportionate share of the events. Because a single partition is handled by one leader broker and read by one consumer in a group, a hot partition becomes a bottleneck. Its consumer falls behind while the others sit idle, and its broker works harder than its neighbours.
Common causes
A very popular key
If one customer, such as a giant online store using your platform, generates most of the traffic, and the customer number is the key, all its events land in one partition.
A poor key choice
Using something with very few distinct values, such as a country code or a true/false flag, means only a few partitions ever get used.
A missing or constant key
Accidentally sending every event with the same key, for example an empty placeholder, sends everything to one partition.
Ways to fix it
Choose a key with many evenly distributed values. For a truly dominant customer, add a small extra part to the key, such as a number from one to ten, to spread that customer’s events across several partitions, accepting that order is now kept only within each sub-group. In some cases, a custom partitioner that treats the biggest keys specially is the right answer. Watching per-partition traffic and lag makes the problem visible early.
flowchart LR
subgraph Balanced["Balanced keys"]
B0["P0 - 25 percent"]
B1["P1 - 25 percent"]
B2["P2 - 25 percent"]
B3["P3 - 25 percent"]
end
subgraph Skewed["Skewed keys"]
S0["P0 - 80 percent"]
S1["P1 - 10 percent"]
S2["P2 - 5 percent"]
S3["P3 - 5 percent"]
end
Measuring only the average load of the topic. Averages hide skew. Always look at each partition separately, especially the busiest one.
13Partitions and Consumer Groups Together
The partition is the unit of assignment between Kafka and its readers. Understanding this link explains many surprising behaviours.
One partition, one reader per group
Within a consumer group, each partition is assigned to exactly one member at a time. This is what keeps order safe: only one reader handles a given partition, so events are processed in sequence. Different groups can each read every partition independently.
When the team changes
When a consumer joins, leaves, or crashes, Kafka redistributes the partitions among the remaining members. This redistribution is called a rebalance. During it, partitions move between consumers, and the new owner continues from the last saved bookmark. Frequent rebalances slow down processing, so keeping consumers stable and processing quick is important.
| Situation | Partitions | Consumers in Group | What Happens |
|---|---|---|---|
| Fewer consumers than partitions | 6 | 3 | Each consumer reads two partitions |
| Equal numbers | 6 | 6 | Each consumer reads one partition |
| More consumers than partitions | 6 | 8 | Two consumers sit idle without work |
| One consumer fails | 6 | 6 down to 5 | Its partition goes to another member after a rebalance |
A hospital assigns each patient ward to one nurse per shift. That way, two nurses never give conflicting treatments in the same ward. If a nurse goes home sick, the wards are reassigned quickly, and the replacement reads the patient notes to continue where the previous nurse stopped.
14Real-World Examples
Partitions appear in every Kafka system. Here are common ways teams use them.
E-commerce orders
An online shop uses the customer number as the key. All events for one customer stay in order, while thousands of customers are processed in parallel across many partitions.
Ride-hailing locations
Each driver’s device is the key. Location updates from one driver stay in sequence, so the map never shows a driver jumping backwards, while millions of drivers spread across partitions.
Banking transactions
The account number is the key. Deposits and withdrawals for one account are handled in order, which is essential for correct balances.
Website click analysis
Order across users does not matter, so clicks are sent without a key and spread evenly. Many analytics consumers share the load.
Device telemetry
Factories send sensor readings keyed by machine identity. Each machine’s readings remain in time order for fault detection.
15Monitoring Partitions
A few simple signals reveal the health of your partitions.
What to watch
Watch consumer lag per partition, not only in total, since one lagging partition can hide inside a healthy-looking average. Watch the bytes written per partition to spot hot ones. Watch the number of under-replicated partitions, which are missing one or more copies, and offline partitions, which have no leader at all and are unavailable. Also watch disk usage, because partitions grow until retention removes old segments.
Alert on lag that keeps growing for several minutes, not on brief spikes. A short spike is normal, but a steady climb means consumers cannot keep up.
16Security and Access Around Partitions
Access rules in Kafka are set on topics, so every partition of a topic shares the same protection.
Who may touch the events
Kafka can require applications to prove their identity and can restrict which identities may write to or read from each topic. Because partitions belong to topics, allowing an application to read a topic gives it access to all of the topic’s partitions. If two kinds of data need different protection, place them in different topics rather than hoping to separate them by partition.
Protecting data in motion and at rest
Connections between clients and brokers, and between the brokers themselves, can be encrypted. Disks holding partition segments can be encrypted by the operating system or storage layer. Good practice also limits who can change partition counts and other topic settings, since a careless change can disturb ordering and load.
Access to a filing cabinet is controlled per cabinet, not per drawer. If some papers are more secret than others, put them in a separate cabinet with its own lock.
17Best Practices and More Anti-Patterns
A few habits keep partition design healthy for years.
Problem
Creating a separate topic per customer or per device, resulting in thousands or millions of tiny topics.
Why It’s Harmful
Each topic brings its own partitions, files, and management overhead. The cluster is overloaded by sheer count.
Correct Approach
Use one topic and let the customer or device identity be the key, so partitions spread the customers naturally.
Problem
Assuming that adding more consumers always speeds up reading.
Why It’s Harmful
Consumers beyond the number of partitions do nothing. Money is spent on idle machines while lag remains.
Correct Approach
Check the partition count first. Raise it in a planned way if more parallelism is genuinely required.
Problem
Relying on order across partitions, for example expecting an “order created” event to always be read before an “order paid” event when they use different keys.
Why It’s Harmful
Different partitions are read independently. The paid event may reach a consumer first, causing confusing errors.
Correct Approach
Use the same key, such as the order number, for all related events, so they share a partition and stay in order.
Best practices at a glance
Choose keys deliberately
Use an identity that groups events needing order and has many evenly spread values.
Plan the count with headroom
Size for future traffic so you rarely need to change it.
Use three replicas for important data
Combine replication with safe producer settings to protect events.
Monitor each partition
Track lag and traffic per partition to catch skew early.
Keep related events together
Send all events of one business object with the same key.
Avoid needless sprawl
Do not create more topics or partitions than you can monitor and manage.
18The Journey of One Event Through a Partition
Let us follow one order event from the moment it is created until a consumer reads it.
The producer builds the event
The order service prepares an event with the customer number as key and the order details as value.
The partitioner picks partition 2
The key is turned into a partition number. Every event for this customer will use partition 2.
The leader appends the event
The broker leading partition 2 adds the event to the end of the active segment and assigns the next offset.
Followers copy it
The follower replicas fetch the event and store their own copies, and the producer receives its confirmation.
A consumer in the group reads it
The one consumer that owns partition 2 fetches the event in order and processes it.
The bookmark moves forward
The consumer commits the new offset. Other groups still see the event and read it on their own schedule.
sequenceDiagram
participant P as Producer
participant L as Leader of Partition 2
participant F as Follower Replicas
participant C as Consumer Owning Partition 2
P->>L: Send event with customer key
L->>L: Append at next offset
F->>L: Fetch new event
L-->>F: Return event
L-->>P: Confirm write
C->>L: Fetch from current offset
L-->>C: Deliver event in order
19Frequently Asked Questions
A partition is one ordered, append-only log that forms a slice of a topic, and it is the basic unit of storage, parallelism, and replication in Kafka.
As few as one and as many as your cluster can sensibly manage. Practical limits come from broker memory, files, recovery time, and how many partitions the whole cluster carries.
No. Kafka only allows increasing. To shrink, create a new topic with fewer partitions and move the data and applications over.
No. Order is guaranteed only inside a single partition. Use a key so that related events share a partition.
The producer spreads them across partitions to balance the load. There is no order guarantee between those events.
Most likely you have more consumers in the group than partitions. Each partition can be read by only one member of a group at a time, so the extra members have nothing to read.
No. Kafka spreads them over the brokers, and each partition also has replicas on other brokers for safety.
Not always. More partitions raise the possible parallelism but also add overhead, longer recovery times, and smaller batches. Choose a count based on measured needs plus reasonable headroom.
A partition is a slice of a topic. A replica is one copy of that partition stored on a broker. Each partition normally has several replicas, one of which is the leader.
20Summary and Key Takeaways
We began with a supermarket that opened many counters to serve a crowd. Kafka partitions are those counters. A topic is split into partitions, each an ordered, append-only log with its own offsets, stored in segments on a broker’s disk. Producers pick a partition by key or by balance, brokers lead and copy partitions for safety, and consumers in a group divide the partitions so that each is read by one member at a time. This design delivers scale, speed, and order per key, at the price of no global order, a ceiling on parallel readers, and the need to choose the partition count and keys with care. Get those two choices right, and Kafka rewards you with smooth, predictable growth.
Key Takeaways
- A partition is an ordered, append-only log — every topic is made of one or more, each with its own numbering of offsets.
- The key controls placement — events with the same key go to the same partition, keeping their order, while different keys spread the load.
- Order exists only inside a partition — there is no global order across a topic, so design keys to group events that must stay in sequence.
- Partitions set the ceiling for parallelism — a consumer group cannot usefully have more active members than partitions.
- Replication protects each partition — a leader and followers on different brokers keep data safe, with only in-sync followers eligible to take over.
- Choose the count with care — you can increase but never decrease it, and increasing changes the key-to-partition map; too many partitions add overhead.
- Watch for skew — hot partitions from poor keys create bottlenecks, so monitor lag and traffic for each partition, not just averages.