Before any technical words, let us see why anyone needs to repartition at all.

When you first create a Kafka topic, you must choose how many partitions it has. That choice is made on day one, when you know very little about the future. Months later, traffic may have grown ten times. Or one broker may be running out of disk while others are half empty. Or a stream processing job may need messages grouped by a different field than the one used when they were written. Each situation calls for a different kind of repartitioning.

Why it is risky if done carelessly

Repartitioning touches the parts of Kafka that people rely on most: the order of messages for a key, the safety of copies, and the steady flow of data to consumers. A careless change can break ordering, overload the network, leave copies unsafe, or make a stream processing job give wrong answers. A careful change, on the other hand, is routine. The difference is knowing what happens inside.

A short tour of the words we will use throughout.

Producers write messages. Consumers read them. A message may carry a key, such as a customer number, and a value, which is the content. The key is the label that decides where the message goes.

The brain of the cluster

One special part of the cluster, the controller, keeps the master list of which partitions exist and which brokers hold them. In modern Kafka, this job belongs to the KRaft controller group, which stores every change in an ordered metadata log. Whenever you change the number of partitions or move replicas, you are really asking the controller to update that master list, and the brokers then act on it.

A look inside one partition, so that moving or adding partitions makes sense.

A partition is not an abstract idea. On a broker, it is a real folder on disk. Inside that folder are files called segments. Each segment holds a range of messages in order. New messages are always added to the newest segment, the “active” one. When a segment gets big enough or old enough, Kafka closes it and starts a new one. Beside each segment are small index files that let Kafka jump quickly to a given offset or time.

Why this matters for repartitioning

Because a partition is just files, moving a partition means copying its files to another broker. That is why moving a large partition takes time and network. It also explains a key fact: Kafka cannot easily split or merge partitions. Each partition’s offsets are its own private numbering, and messages inside are already in a fixed order. Combining two partitions would require inventing a new order, which Kafka does not do.

The sorting rule inside the producer, and why it is the source of most repartitioning surprises.

When a producer sends a message with a key, it runs the key through a fixed mathematical function called a hash, which turns any text into a big number. Kafka’s standard hash is called murmur2. The producer then divides that number by the count of partitions and uses the remainder to choose the partition. This means the same key always goes to the same partition, as long as the partition count stays the same.

What if there is no key?

Messages without a key have no reason to stay together, so the producer spreads them out. Modern producers use a “sticky” approach: they fill one batch for one partition, then switch to another. This creates larger batches, which are more efficient, while still balancing load over time. Because there is no key, repartitioning does not break any ordering promise for these messages, since none was made.

Custom partitioners

An application can supply its own rule for choosing partitions, for example to send big customers to dedicated partitions. That rule may depend on the partition count too, so anything you learn in this guide about counts and keys applies to your custom rule as well.

Partitions exist to give you two things, and repartitioning can disturb both.

Notice the tension. More partitions allow more parallel consumers, but each key lives in only one partition. Ordering is preserved per key, never across the whole topic. When you change the number of partitions, you may improve parallelism, but you also change which partition each key maps to, which can break ordering for a short time.

The ceiling on consumers

Within one consumer group, a partition is read by exactly one consumer at a time. So a topic with six partitions can keep at most six consumers busy in one group. A seventh consumer would sit idle. This is the main reason teams add partitions: they need more workers than partitions allow.

A side-by-side map before we go deep into each one.

The best repartitioning is the one you never need. Choose the count well from the start.

A simple sizing method

The costs of too many partitions

The first job, changing the count, seen from the moment you ask to the moment clients notice.

flowchart TD
  A["Operator asks to raise partition count from 6 to 12"] --> B["Request reaches the active controller"]
  B --> C{"New count larger than current count?"}
  C -->|"No"| X["Request rejected - partitions cannot shrink"]
  C -->|"Yes"| D["Controller picks brokers for the 6 new partitions - rack aware"]
  D --> E["Controller writes new partition records to the metadata log"]
  E --> F["Majority of controllers commit the records"]
  F --> G["Brokers fetch the records and create empty partition folders"]
  G --> H["Leaders are chosen and followers start copying - nothing to copy yet"]
  H --> I["Producers refresh metadata and discover 12 partitions"]
  I --> J["Consumer group notices the change and rebalances"]
  J --> K["New keys spread over 12 partitions - old data stays in the first 6"]
    

A closer look at what the cluster’s brain does behind the scenes.

The request to add partitions is called CreatePartitions. Any broker can receive it, but only the active controller can act on it. In a cluster that uses KRaft, the active controller is the elected leader of the controller group. It is the only node allowed to add records to the metadata log, which is the single ordered list of every cluster change.

Manual placement

Administrators may supply a custom placement for the new partitions, for example to put them on newer brokers. The controller checks that the placement is valid: the right number of replicas, existing brokers, and no repeats. If it is valid, it simply records it. If not, it rejects the whole request, and nothing changes. This all-or-nothing behavior keeps the cluster consistent.

Clients learn about new partitions slowly and in a specific order. This gap creates most of the surprises.

Producers

A producer keeps a cached copy of the topic’s layout and refreshes it on a timer, by default every few minutes, or sooner if something goes wrong. Until it refreshes, it still uses the old partition count, so its keys still map to the old partitions. Different producers may refresh at different moments. During that window, two producers can send the same key to different partitions.

Consumers

A consumer group notices new partitions through its own metadata refresh. The group leader then asks the group coordinator to run a rebalance, which is a reshuffling of who reads which partition. During the rebalance, consumers pause briefly. Afterwards, the new partitions are assigned to members of the group.

Let us watch six keys change partitions when the count goes from three to four.

To keep the numbers small, imagine that our hash function gave these six keys the numbers 12, 19, 25, 30, 41, and 58. The partition is the remainder after dividing by the partition count.

Four of the six keys now go somewhere new. Consider K2: all its old messages (say, a customer’s earlier orders) sit in partition 1, but from now on its new messages arrive in partition 3. A consumer reading partition 3 sees K2’s newest orders but none of the earlier ones. Worse, a consumer reading partition 1 may still be processing old K2 messages while the new ones are already being handled in partition 3. The order across the two partitions is not defined.

There is a neat mathematical habit that makes repartitioning easier to reason about.

When you double the partition count, something special happens with the remainder rule. A key that was in partition p can only end up in partition p (staying) or in partition p plus the old count (moving to a brand new partition). It never jumps to a different old partition. Let us see it with the same six keys, going from four to eight partitions.

On average, half of the keys stay and half move, and every moved key goes to exactly one predictable new partition. This is like splitting each old partition into two: the old one keeps some keys and the new one takes the rest. It does not fix the ordering problem, but it makes the problem easier to plan around, for example by knowing exactly which old partitions feed which new ones.

Plan growth in doublings

If you expect to grow, choose a starting count that is easy to double, such as 6, 12, 24, or 48. Then when growth comes, you double and enjoy this predictable pattern. Also, numbers with many factors let consumer groups share partitions evenly among different numbers of consumers.

Kafka only allows growth. Here is why shrinking is not offered.

What to do instead

If you truly need fewer partitions, the standard answer is to create a new topic with the count you want and move your data and your applications over to it. The next two chapters show how. It is more work than a single command, which is another reason to choose your count carefully at the start.

The safest way to change a partition count while keeping every key together.

sequenceDiagram
  participant PR as Producers
  participant OT as Old Topic - 6 partitions
  participant CP as Copier Job
  participant NT as New Topic - 12 partitions
  participant CO as Consumers
  Note over OT,NT: New topic created with the desired partition count
  PR->>OT: Keep writing as usual
  CP->>OT: Read all messages in order per partition
  CP->>NT: Write each message using the same key so keys spread over 12 partitions
  Note over CP: Copier catches up to the end of the old topic
  Note over PR,CO: Cutover window begins
  PR->>NT: Producers switch to the new topic
  CO->>NT: Consumers switch to the new topic after finishing the old one
  Note over OT: Old topic retired after a safety period
    

Three ways to switch from the old topic to the new one without breaking key order.

The drain and switch method in detail

Topics that keep only the latest value per key are especially sensitive to partition changes.

A compacted topic is a topic where Kafka eventually keeps only the newest message for each key. It works like a table: one row per key. Such topics are used for reference data, settings, and state that programs rebuild by reading from the start.

Compaction works inside each partition separately. If the same key lives in two partitions after you add partitions, each partition keeps its own newest value for that key. Now there are two “latest” values for one key, one old and one new. A program rebuilding the table by reading all partitions will see both, and which one it keeps depends on which partition it reads last. That can silently produce wrong data.

A subtle trap: consumers may skip messages that arrive in a new partition before they notice it.

Consumers save their progress as offsets, one per partition. A brand new partition has no saved offset. When a consumer is assigned it for the first time, it must decide where to begin. That decision is controlled by a setting with two common values: start from the earliest message, or start from the latest message. The default is to start from the latest.

How to avoid losing messages

The second job: keeping the same partitions but changing which brokers hold their copies.

Over time, brokers become unevenly loaded. A few brokers may hold the busiest partitions, while others sit idle. You may add new machines that are empty, or you may need to retire old machines that must be emptied first. Or a broker’s disk is filling up faster than the others. In all these cases, the answer is the same: move some partition copies from one broker to another. This is called replica reassignment, or in daily speech, “reassigning partitions.”

The trick is to add first, wait, and only then remove.

flowchart TB
  OP["Operator submits a plan - move partition 3 from brokers 1,2,3 to brokers 2,3,4"]
  CT["Active controller records: adding replica on broker 4, removing replica on broker 1"]
  ML[("Metadata log")]
  subgraph BRK["Brokers"]
    B1["Broker 1 - old replica - LEADER"]
    B2["Broker 2 - follower - stays"]
    B3["Broker 3 - follower - stays"]
    B4["Broker 4 - NEW replica"]
  end
  OP --> CT
  CT --> ML
  ML -.->|"brokers read the plan"| BRK
  B4 -->|"1 fetches all data from the leader"| B1
  B2 -->|"copies as usual"| B1
  B3 -->|"copies as usual"| B1
  B4 -->|"2 joins the in-sync list when caught up"| CT
  CT -->|"3 chooses a new leader if the old one leaves"| B2
  CT -->|"4 removes the old replica and deletes its files"| B1
    

Because reassignments take a long time, their state must survive failures.

A big reassignment can run for hours or days. During that time, brokers may restart, and even the active controller may change. So the plan and its progress cannot live only in someone’s memory. In modern Kafka, the state is stored in the cluster’s metadata log: each partition record carries the current replica list plus the “adding” and “removing” lists. Anyone who reads the log sees exactly what is in progress.

stateDiagram-v2
  [*] --> Stable
  Stable --> Reassigning : plan accepted - adding and removing lists set
  Reassigning --> Reassigning : new replicas still catching up
  Reassigning --> Finishing : all adding replicas are in sync
  Finishing --> Stable : removing replicas deleted and lists cleared
  Reassigning --> Stable : reassignment cancelled - adding replicas dropped
    

There is no special copy tool. A new replica simply behaves like a very late follower.

Kafka reuses the normal replication machinery. The new replica sends fetch requests to the partition leader, starting at the earliest offset still on disk. The leader answers by reading its segments and sending batches of messages. The new replica writes them to its own disk and asks for more. Once it reaches the end and keeps up with new writes, it is considered caught up.

What the leader has to do

A speed limit protects everyday traffic while the move is running.

Kafka lets you set a maximum data rate for the replication that belongs to a reassignment. This limit is applied to the leaders that send the data and to the new replicas that receive it. Only the moving replicas are limited, so normal replication of other partitions continues at full speed.

Too tight a limit

If the throttle is lower than the rate at which new data arrives in a partition, the new replica can never catch up and the move never finishes. Compare the limit with the write rate of the partitions you are moving.

A checklist that turns a risky operation into a routine.

Moving copies does not automatically balance leadership. A second look is needed.

The first broker in a partition’s replica list is called the preferred leader. Kafka tries to spread preferred leaders evenly across brokers, because the leader does more work than followers: it handles all reads and writes. When you reassign replicas, the order of the list matters. A list that puts the same broker first for many partitions makes that broker the preferred leader of them all.

Order your replica lists on purpose

When writing a plan by hand, do not just list brokers in numeric order for every partition. Rotate the first broker so that each broker leads about the same number of partitions. Tools that generate proposals usually do this for you.

Sometimes the problem is one full disk, not one full machine.

Many brokers have several data disks. A partition lives on one of them. If one disk fills up while others have space, you can ask the broker to move that partition to another disk on the same machine. This is called moving between log directories.

Doing all this by hand does not scale, so large operators use automatic balancers.

A balancer is a program that watches how busy each broker is, decides which partitions to move where to even out the load, and carries out the moves gently with a throttle. The best-known open-source one is Cruise Control, created at LinkedIn. Commercial Kafka platforms offer similar built-in features, and operators for Kubernetes such as Strimzi integrate Cruise Control.

Inside stream processing, repartitioning means something different, and happens automatically.

Kafka Streams is a library for building programs that read from topics, transform the data, and write results. Many operations, such as counting per customer or joining two streams, only work correctly if all records with the same key are processed by the same worker. If your data is keyed by order number, but you want to count by customer, the records for one customer are scattered across many partitions. Streams solves this by writing all records again into a special internal topic, using the customer as the new key. This is the repartition topic.

flowchart LR
  IN["Input topic - keyed by order number"] --> S1["Sub-topology 1 - read and filter"]
  S1 --> SK["selectKey - customer becomes the new key"]
  SK --> RT[("Internal repartition topic - keyed by customer")]
  RT --> S2["Sub-topology 2 - count per customer"]
  S2 --> ST[("Local state store")]
  S2 --> CL[("Changelog topic - backs up the state")]
  S2 --> OUT["Output topic - counts per customer"]
    

Joins are the strictest users of partition rules.

To join two streams by key, Streams needs matching records from both sides to be processed together. This only works if the two topics are co-partitioned: they have the same number of partitions and use the same rule to assign keys to partitions. Then partition 3 of the first topic and partition 3 of the second topic hold the same keys, and one task can handle both.

Ways to satisfy co-partitioning

Small habits that prevent big headaches in stream processing jobs.

A move is only good if the new layout is also safe.

Every broker can be told which rack or availability zone it sits in. A rack is a physical shelf of machines that shares power and network switches. A zone is a similar idea in the cloud: a separate building or section of a data center. If all three copies of a partition sit in one rack, a single power failure removes the whole partition. So Kafka tries to spread copies across different racks when it creates topics.

Checking a plan before running it

Numbers make the rule of thumb concrete.

Imagine a topic that must handle 300 megabytes per second of incoming data in a year. A test shows that one partition comfortably takes about 15 megabytes per second on the producer side, and one consumer instance can process about 10 megabytes per second. Which count do we need?

With 48 partitions, the readers can run as 6, 8, 12, 16, 24, or 48 instances with every instance getting an equal share. A count like 47 would give uneven shares for almost any group size. That is the reason for choosing a friendly number.

Checking the cost side

With a replication factor of 3, 48 partitions means 144 copies for that topic across the cluster. If there are 6 brokers, that is 24 copies per broker for this one topic. If the whole cluster hosts hundreds of topics, the total per broker can grow into thousands, which is fine on modern versions but still worth watching. Always add up all topics, not one at a time.

Failures are normal. What matters is knowing what each one does to your data.

The one dangerous combination

The riskiest time to move partitions is when the cluster is already unhealthy: some brokers down, some partitions with too few in-sync copies. Extra copy traffic can push the weak part of the cluster over the edge. This is why every checklist starts with “check the cluster health first.”

Each of the three jobs spends different resources.

How long will a move take?

A rough estimate is the amount of data to move divided by the throttle rate. If you must move 2,000 gigabytes and the throttle allows about 100 megabytes per second, that is around 20,000 seconds, or a bit more than five and a half hours. Real moves take longer because the throttle is shared by all moving partitions, new data keeps arriving, and small partitions still need time to set up.

Every repartitioning choice buys something and costs something.

These are powerful administrative actions, and they should be protected like keys to the building.

Changing the shape of a cluster can cause outages, data exposure, or extra bills. So Kafka lets you control who may do each action, through access rules called ACLs (access control lists), together with authentication that proves who someone is.

What to watch before, during, and after any repartitioning work.

Exact metric names vary by version and by the monitoring tool you use, so treat the table as a guide to the ideas. Three questions guide everything: Is the cluster healthy? Is the move progressing? Are users affected?

Logs worth reading

Habits that make repartitioning boring, which is exactly what you want.

Four popular mistakes and the better ways to work.

From symptom to likely cause to first check.

The same ideas appear in many well-known setups.

Everything you learned, told as one small story.

ShopStream is an online store. Its “orders” topic began life with 6 partitions, keyed by customer number, and 3 brokers. Three years later, orders grew ten times, and the team faces three problems in one quarter.

A quick reference for the words used in this guide.

Quick answers to the questions beginners ask most often.

The whole story in a few short lines.