Before we learn the solution, we need to feel the pain it removes. Tiered storage exists because of one uncomfortable fact about how Kafka was built.

Storage and compute were glued together

A Kafka broker is a server that receives messages from producers, stores them, and hands them to consumers. In classic Kafka, each broker stores its share of the data on its own attached disks. The same machine that does the work of handling network traffic also holds all the bytes. Engineers call this a shared-nothing design, because no broker depends on a central storage system.

This design was a great choice in the early years. Local disks are fast, and Kafka reads and writes them in a straight line, which is the fastest way a disk can be used. But it has a hidden cost. When you want to keep more data, you must add more disk. And the only way to add more disk is to add more brokers, or buy bigger disks for every broker.

Why this hurts in real life

Suppose a company ingests 2 terabytes of events per day and wants to keep 30 days of history so that new applications can replay old data. That is 60 terabytes of unique data. Kafka normally keeps three copies of every byte for safety, so the cluster needs about 180 terabytes of disk. To hold that much, the team may run dozens of brokers whose CPUs are barely busy. They are paying for compute they do not need, just to rent disk space.

Three separate pains

Tiered storage is built on top of how Kafka already stores data. If these building blocks are clear, everything that follows will feel natural.

Topics, partitions and offsets

A topic is a named stream of messages, like a channel called “orders” or “payments”. Because one machine cannot handle a busy topic alone, Kafka splits each topic into partitions. Each partition is an ordered list of messages that only ever grows at the end. Every message in a partition gets a number called its offset: 0, 1, 2, 3 and so on. A consumer remembers its offset the way you remember the page number in a book.

The log is a set of segment files

On disk, a partition is stored as a log. But the log is not one giant file. Kafka chops it into pieces called segments. At any moment only the newest segment, called the active segment, accepts new writes. When it gets too big or too old, Kafka rolls it: the active segment is closed and never changes again, and a fresh active segment is started. Closed segments are called inactive or rolled segments.

What lives inside a segment

Each segment is really a small family of files that share a name based on the first offset they hold.

This list matters later, because tiered storage has to move all of these pieces, not just the messages. A segment without its indexes would be a book without a table of contents: still readable, but painfully slow to search.

Retention: how old data is removed

Kafka is not a database that keeps everything forever. Each topic has a retention rule, such as “keep seven days” or “keep 500 gigabytes per partition”. A background task deletes segments that break the rule. One detail trips up many newcomers: Kafka deletes whole segments, never single messages. A segment is only eligible once its newest message is older than the limit.

Replication: the reason for three copies

Each partition has one leader broker that handles all reads and writes, and one or more followers that copy the leader’s log. If the leader dies, a follower that is fully caught up takes over. The group of caught-up replicas is called the in-sync replica set, or ISR. This is what makes Kafka safe, and it is also what triples the disk you need.

Why Kafka reads are so fast

Kafka leans on the operating system’s page cache, a part of memory the OS uses to hold recently touched file data. A consumer that is close to the tail of the log usually reads straight from memory, never touching the disk. Keep this in mind. The difference between reading fresh data and reading old data is a central theme of tiered storage.

Why segments make tiering natural

Kafka’s habit of writing in straight lines and closing files when they fill up turns out to be the perfect starting point for tiering. A closed segment is a finished, unchangeable package. It can be copied without worrying that someone will edit it halfway through. It can be checked by comparing sizes. It can be read in slices because its index tells you exactly where each offset begins. And it can be deleted as a single unit. If Kafka rewrote old data in place, like a database updating a row, moving that data to another system would be far harder. The append-only log is what makes the whole idea possible.

There is one more useful fact. Each segment has a name based on the first offset it contains, so segments are already in a clear order. Kafka can tell which segment holds any offset just by comparing numbers. This same ordering is used later to name and find segments in the remote tier.

The idea is simple to say and subtle to build: split each partition’s history across two kinds of storage and hide the split from applications.

Two tiers, one log

With tiered storage, a partition’s log has two homes. The local tier is the broker’s own disks, fast and expensive per gigabyte. The remote tier is an external storage system, slower and far cheaper per gigabyte, usually object storage such as Amazon S3, Google Cloud Storage or Azure Blob Storage. Recent segments are kept in the local tier. Older segments are copied to the remote tier and, after a while, deleted from the local disks.

The important point is that it is still one logical log. Offset 5 is offset 5 whether it sits on a local disk or in a bucket. A consumer asks for an offset and Kafka finds it, wherever it lives.

What does not change

A short history

The idea of separating hot and cold data is old in databases and file systems. In Kafka’s world, several vendors built their own private versions first. Confluent shipped tiered storage in its commercial platform and cloud service, and Amazon offered tiered storage on Amazon MSK for provisioned clusters. Other streaming systems took different routes: Apache Pulsar offloads old segments to object storage, and Redpanda has its own cloud storage feature built into the broker.

The open source community wanted a standard version. The design was written up as KIP-405 (a KIP is a Kafka Improvement Proposal). Engineers from several companies, including Uber and Confluent, refined it over several years. It reached early access in Apache Kafka 3.6 in late 2023, and was declared production ready in Apache Kafka 3.9 in late 2024.

Four moving parts make tiering work. Learn their names now and the rest of this guide will read like a story.

The four building blocks

flowchart TB
    PR["Producers"] --> LL
    CO["Consumers"] --> LL
    subgraph LB["Leader Broker"]
        LL["Local Log: active and recent segments"]
        RLM["RemoteLogManager"]
        RSM["RemoteStorageManager plugin"]
        RLMM["RemoteLogMetadataManager plugin"]
        LL --> RLM
        RLM --> RSM
        RLM --> RLMM
    end
    LL --> FB["Follower Brokers: copy the local tier only"]
    RSM --> OBJ["Object Storage: S3, GCS, Azure Blob or HDFS"]
    RLMM --> MT["Internal topic: __remote_log_metadata"]

Reading the diagram

Producers and consumers connect to the leader broker exactly as they always did. Inside the broker, the local log is the familiar set of segment files. Next to it sits the RemoteLogManager, a new piece of broker code that owns everything related to the remote tier. It does not know how to speak to Amazon S3 or Google Cloud Storage. Instead, it calls the RemoteStorageManager plugin, which does. Swapping storage systems means swapping the plugin, not rewriting Kafka.

The RemoteLogMetadataManager plugin answers a different question: “For partition 7 of the orders topic, which segments exist in remote storage, and what offsets does each cover?” The default implementation keeps this catalogue in an internal Kafka topic, so Kafka uses itself to remember where its own history went.

Why two plugins instead of one

Storing bytes and remembering where the bytes went are separate problems. Object stores are excellent at holding large files but poor at fast, consistent lookups across thousands of small records. Kafka topics are excellent at fast, ordered, replicated records. So the design lets each system do what it is best at. The bulk data goes to the bucket. The small catalogue stays in Kafka.

Who does the work

Only the leader of a partition copies its segments to the remote tier. Followers do not upload, which prevents three brokers from uploading the same segment three times. If leadership moves, the new leader picks up where the old one stopped, guided by the catalogue.

The RemoteStorageManager plugin is where Kafka meets the outside world. Understanding its job explains many of the behaviours you will see in production.

The four jobs

Kafka asks the plugin to do only four things. Everything else, such as deciding when to copy and what to keep, stays inside the broker. This small contract is what makes the design so portable.

Good plugins handle the boring details well

Why the key layout matters

Object stores spread data across many machines by looking at the beginning of each object’s name. If every name starts the same way, requests can pile up on the same machines. Well-designed plugins add variety near the start of the key, so traffic spreads out. This is invisible when you are small and painful when you are large, so ask about it when choosing a plugin.

Copying happens quietly in the background. Producers and consumers never wait for it, and it never touches the active segment.

The copy loop

Every broker runs a pool of background threads that belong to the RemoteLogManager. For each partition where the broker is the leader, one task wakes up on a regular schedule (every 30 seconds by default) and asks a simple question: “Are there closed segments that have not been uploaded yet?” If the answer is yes, it starts uploading them, oldest first.

There is one safety rule. A segment is only uploaded when all of its messages are fully settled. That means the messages are replicated to the in-sync followers, and any transactions inside them have finished, either committed or aborted. Kafka tracks this using a marker called the last stable offset. Nothing beyond that marker is allowed to leave the broker. This stops half-finished transactions from being frozen into cold storage.

sequenceDiagram
    participant L as Leader Broker RemoteLogManager
    participant M as Metadata Manager
    participant S as Storage Plugin
    participant O as Object Storage
    L->>L: Find closed segments not yet copied
    L->>M: Record segment as COPY_SEGMENT_STARTED
    L->>S: Hand over segment, indexes and snapshots
    S->>O: Upload the files
    O-->>S: Upload acknowledged
    S-->>L: Copy complete
    L->>M: Record segment as COPY_SEGMENT_FINISHED
    Note over L: Local copy stays until local retention expires

Walking through the steps

What if the upload fails?

Networks fail and buckets have bad days. If an upload does not complete, the segment is simply tried again on the next cycle. The catalogue still says “started”, not “finished”, so no reader is ever pointed at a half-uploaded segment. Leftover partial files are cleaned up later. This is the same idea a careful shipping company uses: the package is not marked as delivered until the receiver signs for it.

Why segment size matters so much

Only closed segments are copied, and a segment closes when it reaches its size limit (1 gibibyte by default) or its age limit (7 days by default), whichever comes first. This has a surprising effect on quiet topics. A topic that receives only a trickle of data may take days to fill a segment, so its data stays on local disks longer than you might expect.

The cost of copying

Uploading is not free. Every tiered byte must be read from the broker’s disk (or page cache) and pushed over the network to the storage system. On a busy cluster this adds steady background traffic on top of normal replication. Recent Kafka versions include throttles that cap how fast a broker may copy and how fast it may read back, so tiering cannot starve real-time traffic. Plan for this extra network load when you size your brokers.

A segment passes through clear stages from birth to deletion. Knowing them helps you read metrics, predict disk usage and debug problems.

stateDiagram-v2
    [*] --> Active
    Active --> Rolled: segment rolls
    Rolled --> Copying: leader starts upload
    Copying --> Rolled: upload fails, retry later
    Copying --> Tiered: upload finished
    Tiered --> RemoteOnly: local retention expires
    Tiered --> Deleting: total retention expires
    RemoteOnly --> Deleting: total retention expires
    Deleting --> [*]: files removed from remote storage

Stage by stage

Two start markers you should know

Kafka tracks two numbers for each partition. The log start offset is the very first offset that still exists anywhere, local or remote. It moves forward only when total retention deletes old data. The local log start offset is the first offset still on the broker’s own disks. It moves forward whenever local retention removes a segment that has already been tiered. The gap between the two numbers is exactly the amount of history that lives only in remote storage.

The safety guarantee behind local deletion

Kafka will never delete a local segment that has not been confirmed as copied. This single rule protects you from data loss. It also has a flip side that we will meet again in the mistakes chapter: if copying stalls, local disks cannot be cleaned, and they slowly fill up.

Metadata states

The catalogue records four states for every remote segment: copy started, copy finished, delete started and delete finished. The two “delete” states mirror the two “copy” states for the same reason. If a broker crashes halfway through cleanup, the catalogue shows that deletion was in progress, and a later run can safely finish the job.

A consumer sends the same fetch request as always. The broker quietly decides whether to answer from the counter or from the storeroom.

flowchart TD
    A["Consumer sends a fetch request with an offset"] --> B{"Is the offset in the local log?"}
    B -->|"Yes"| C["Read from page cache or local disk"]
    B -->|"No, it is older than the local log start"| D["Ask the metadata manager which remote segment holds it"]
    D --> E["Remote reader thread fetches the needed bytes through the storage plugin"]
    E --> F["Broker returns the data to the consumer"]
    C --> F

The local path

If the requested offset is at or after the local log start offset, nothing is different from classic Kafka. The broker looks up the position in the offset index, reads the bytes and returns them. Real-time consumers, which stay close to the tail of the log, live here. They see the same speed as before.

The remote path

If the offset is older than anything on local disk, the broker asks the metadata manager: “Which remote segment covers this offset?” It then asks the storage plugin for just the part it needs. Object stores allow reading a byte range from inside a large object, so the broker does not download a whole 1 gibibyte segment to serve a small fetch. It first reads the small index files to find the right position, then reads the exact range of messages.

Crucially, this work does not happen on the broker’s main network threads. Slow remote reads are handed to a separate pool of remote reader threads, and the fetch request waits in a holding area until the data arrives. This keeps a slow bucket from freezing the broker’s normal traffic.

Speed expectations

Local reads from memory take well under a millisecond. Remote reads must cross a network to an object store, and the first byte typically arrives after tens of milliseconds. That is fine for a job replaying last month’s data, but it would be a poor fit for a service that needs an answer within five milliseconds. Once the first bytes arrive, however, remote throughput can be very high, especially when many partitions are read in parallel, because object stores scale out well.

A hidden benefit: protecting the page cache

In classic Kafka, when a consumer replays old data, the broker must read old segments from disk. Those reads pull cold data into the page cache and push out the hot data that real-time consumers need. The result is a “noisy neighbour” problem: one big replay job can slow everyone else. With tiered storage, old data is served through the remote path instead of the local disk, which reduces this interference.

Indexes are cached

Reading the same small index files from the bucket again and again would be wasteful. The broker keeps a local cache of recently used remote index files, with a configurable total size. This is a small cache, but it makes repeated reads from the same remote segment far faster.

Let us follow a single order event through its entire life in a tiered cluster. This ties every earlier chapter together.

Imagine an online shop sends an event called “order 84213 placed” to a topic named orders. The topic keeps data for 30 days in total and one day on the brokers. Here is what happens.

What this story shows

The event was written once, replicated locally for a short time, copied once into the bucket, and deleted twice: once from the brokers and once from the bucket. Its offset never changed. The application that replayed it on day 10 used the same client library as the shipping service on second 3. The complexity lives inside the broker, which is exactly where it should live.

With two tiers you need two clocks. Getting these settings right is the difference between a smooth rollout and a full disk at 3 AM.

Two kinds of retention

The familiar retention.ms and retention.bytes settings now describe the total lifetime of the data, across both tiers. Two new settings, local.retention.ms and local.retention.bytes, describe how long data stays on the broker’s own disks. By default the local settings follow the total settings, which means no data would leave the broker early. You must lower the local values to see any benefit.

A timeline for one segment

Infinite retention

Setting total retention to unlimited turns Kafka into a long-term record of everything that ever happened. This is powerful for event sourcing, audit trails and replaying history into new systems. Before tiering, it was too costly for most teams. With object storage, keeping years of data can cost less than a single extra broker.

Rules that must hold

• Local retention should never be larger than total retention. It would make no sense to keep data locally that is already deleted overall.
• Tiering is switched on per topic. Enabling it on a topic that already has data means existing closed segments are copied over time, not instantly.
• Deleting a topic also removes its data from remote storage, so treat topic deletion with the same care as before.

Choosing the local window

A helpful rule of thumb is to keep on local disks whatever your normal, latency-sensitive consumers read. If all your real-time consumers stay within an hour of the tail, a local window of several hours to a day gives comfortable room for consumer restarts and small outages. Anything older is a replay, and replays can tolerate the remote path.

This is where tiered storage quietly pays off the most. Operations that used to take a day can shrink to an hour.

Followers copy only the local tier

The remote tier already stores each segment once, inside a storage system that keeps its own extra copies for safety. So Kafka does not ask three brokers to each hold a full local copy of the old history. Followers replicate only what is on the leader’s local disks. The leader’s local log start offset tells a follower where to begin.

Think about what happens when a broker dies and a replacement joins. In classic Kafka, the newcomer must copy every partition it now owns, from the very first offset to the last. With tiering, it starts from the leader’s local log start offset and copies only the recent window. The older history stays in the bucket where it always was. The newcomer also downloads a few small pieces of state, such as the leader epoch history and producer snapshots, so it can behave correctly.

Leader elections

If the leader fails, a caught-up follower is elected as the new leader, just as before. The new leader does not have to rebuild anything for the remote tier. It reads the catalogue, sees which segments are already uploaded, and continues copying from the next one. The catalogue records the leader epoch for every remote segment. A leader epoch is a counter that goes up each time a new leader is chosen, and it acts like a version number for the log’s history. Using it, Kafka can tell which remote segments belong to the current history and which came from an older history that was cut off by a leader change.

Failure scenarios

Rebalancing and scaling out

Adding a broker to spread the load used to be a painful, hours-long event. Now the amount of data that must move is small, so scale-out becomes a routine task. Likewise, replacing a failed disk or an ageing machine no longer feels like an emergency. This is often the strongest argument for tiering on busy production clusters, even for teams that do not care about long retention.

A storage feature is only trustworthy if you know exactly which promises it keeps. Here are the promises that survive tiering, and the ones that change shape.

Promises that stay the same

How durability changes

Before tiering, durability of a message came from three broker disks holding copies. With tiering, old data gains a second kind of durability. Once a segment is uploaded, its safety rests on the storage system, which is designed for extreme durability. Amazon S3, for example, is designed for eleven nines of durability, meaning the chance of losing a given object in a year is astronomically small. This is often better than three local disks.

But there is a subtle point. Durability now depends on two different systems, and their protections are configured differently. The brokers protect recent data with replication. The bucket protects old data with its own internal copies. Neither protects you from someone deleting things by mistake. That is why access control and deletion safeguards for the bucket are part of your durability plan, not just your security plan.

How availability changes

The cluster is now available for old data only if both Kafka and the storage system are available. It is available for recent data even when the storage system is down. In other words, the tiering feature adds a dependency for one kind of read, but not for the core work of accepting and serving fresh messages. This partial independence is a deliberate design choice: the bucket being slow or unreachable should hurt replay jobs, not the live business.

What happens on unclean leader elections

Sometimes every up-to-date replica is lost and Kafka is configured to allow a behind replica to become leader anyway. This is called an unclean leader election, and it can drop the newest messages. With tiering, the leader epoch history stored with remote segments helps Kafka keep the two tiers consistent. Segments written under a history that got cut off are not served as if they belonged to the new history. The rules for what is lost do not become worse, but the extra bookkeeping is one more reason the catalogue must be kept healthy.

Consistency between tiers

A reader who asks for offset 100 gets the same message whether it comes from local disk or remote storage, because both copies were made from the same closed, unchangeable segment. The copy is only trusted after the catalogue records it as finished. There is no time when two readers can see two different versions of the same offset.

The catalogue is the brain of tiered storage. If the bucket holds the books, the catalogue knows which shelf each book is on.

What the catalogue must answer

The default: a Kafka topic

Kafka ships with a catalogue implementation that stores its records in an internal topic named __remote_log_metadata. Each event, such as “copy started” or “delete finished”, is one message in that topic. Every broker reads the topic and builds an in-memory picture of the segments it cares about. Because the topic is an ordinary replicated Kafka topic, it inherits Kafka’s durability and ordering for free.

The internal topic has its own number of partitions and its own replication factor, both configurable. Different user partitions map to different metadata partitions, which spreads the catalogue’s load. The metadata topic itself is not tiered, and its records are kept indefinitely so the catalogue never forgets a segment that still exists.

What happens at broker start-up

When a broker restarts, it must read the metadata topic to rebuild its picture. For clusters with an enormous number of tiered segments, this takes time, and remote reads for a partition wait until the broker has caught up on that partition’s metadata. This is one more reason not to create absurd numbers of tiny segments.

Consistency and object stores

Older object stores were only “eventually consistent”, meaning a freshly uploaded file might not be visible right away. This used to make designs like this risky. Modern services such as Amazon S3 now give strong read-after-write consistency, so once an upload is confirmed, it can be read. Even so, Kafka does not rely on this alone. Reads are only pointed at segments the catalogue marks as finished, which adds a second layer of safety.

Custom catalogues

The metadata manager is also a plugin. A company could replace the default with a catalogue stored in a database or in the object store itself. Most teams keep the default because it needs no extra infrastructure.

Kafka is more than a log. Transactions, consumer groups, stream processing and mirroring all touch stored data, so it helps to know how each behaves when history is split across two tiers.

Transactions and exactly-once processing

A transaction lets a producer write to several partitions and have them all become visible together, or not at all. Consumers configured to read only committed data must skip messages from transactions that were aborted. To do that, Kafka keeps a small transaction index next to each segment. Tiered storage uploads this index along with the segment, so a consumer reading old data through the remote path still skips aborted messages correctly. The rule that only stable data may be uploaded exists for exactly this reason: a transaction still in progress could otherwise be frozen in an unknown state.

Idempotent producers

An idempotent producer is one that Kafka protects from writing duplicates, even when it retries after a network error. The broker remembers the latest sequence numbers per producer in a producer snapshot file. As you saw earlier, that snapshot is part of the family of files uploaded with each segment. When a new broker takes over, it restores the snapshot so duplicate protection keeps working.

Consumer groups and committed offsets

Consumer groups remember their progress by saving offsets into an internal topic. That internal topic is small and compacted, and it is not tiered. Your committed offsets therefore work exactly as before. If a group has been down for two weeks and resumes, its saved offset simply points into the remote tier, and the group catches up through the remote read path without any special action.

Looking up messages by time

Applications often ask, “Give me the first message after 3 PM last Tuesday.” Kafka answers this using the time index. With tiering, the time index of remote segments is stored in the bucket, so these lookups work across both tiers. The lookup may take a little longer when it has to fetch a small index file from remote storage, but the result is the same.

Kafka Streams, Kafka Connect and mirroring

Topics that cannot be tiered

Two kinds of topics stay on broker disks. Compacted topics are excluded because compaction rewrites segments. Kafka’s own internal topics, such as those that store consumer offsets and transaction state, are excluded too, because they are compacted and small. This is a good design: the data that needs constant rewriting stays close to the brokers, while the large, append-only event streams are the ones that move.

Enabling tiering is a small number of switches, but the order and the values matter. Here is the path in plain steps.

The settings that matter most

Sizing segments sensibly

Segment size is the most underrated knob. Too large and quiet topics take ages to tier and local disks stay full. Too small and you create huge numbers of tiny objects, which slows the catalogue, multiplies storage requests and increases cost. Many teams settle on segments of a few hundred megabytes to one gibibyte for busy topics, and use an age limit to force rolls on quiet ones.

Turning it off

Before Kafka 3.9, tiering could not be safely switched off once enabled. From 3.9 onward it can be disabled on a topic, and you choose whether the remote data is deleted or left in place. Because this is a big decision, treat enabling tiering as a commitment and test the exit route in a staging environment.

Tiered storage is not a free lunch. It trades a little speed for a lot of savings, and it moves some bottlenecks to new places.

Different readers, different experiences

Where the new bottlenecks appear

Scalability in two directions

Before tiering, if you wanted twice the history you needed roughly twice the disks. Now history grows in the bucket, which has effectively no ceiling, while the number of brokers depends on throughput: how many megabytes per second you write and read. This separation of storage from compute is the deepest benefit of the feature. It lets you size each resource for what it actually does.

Keeping replays from hurting real-time traffic

A well-run cluster puts limits around the remote path. Throttle how fast brokers copy and how fast they serve remote reads, so that a giant replay cannot eat all the network. Client quotas add another layer: they can cap how much bandwidth any one application may use. The goal is to make sure a batch job stays a polite guest.

Tuning consumers that read remote data

Because the first byte from object storage takes longer to arrive, consumers that replay old data should ask for more each time they fetch. Every fetch request pays a fixed waiting cost, so a small fetch wastes most of its time waiting. Raising the maximum bytes per fetch and per partition lets each round trip carry a much larger batch, which spreads that waiting cost across more messages.

Parallelism helps too. A topic with many partitions can be read by many consumer instances in the same group at once, up to one instance per partition. Since each instance issues its own remote reads, throughput scales almost in a straight line until you reach a limit in the network, the reader thread pool or the storage service. Finally, review client timeouts. A replay job that gives up after a few hundred milliseconds will fail needlessly, so give remote fetches a generous allowance.

Three resources need sizing: broker disks, network, and remote storage. Let us reuse the earlier scenario of 2 terabytes per day and work through each.

Step 1: turn daily volume into a rate

Two terabytes per day is about 23 megabytes per second on average (2,000,000 megabytes divided by 86,400 seconds). Traffic is rarely flat. Real systems see peaks of three to five times the average, so design for that peak, not the average.

Step 2: size the local disks

Local disk equals daily volume, times days kept locally, times the replication factor, plus headroom. With one day local and three copies, that is 6 terabytes. Keeping about 30 percent free for growth, uneven partitions and recovery brings it to roughly 8 terabytes across the whole cluster. Compare that with about 180 terabytes without tiering.

Step 3: size the network

Each byte crosses the network several times. Suppose three separate consumer groups read every message.

Tiering adds one more multiple of the ingest rate for uploads. Replays of old data add remote reads on top, and those can be large if many jobs run together. Spread the total across your brokers and check each broker’s network card with a comfortable margin.

Step 4: size the remote storage and the catalogue

Remote storage equals daily volume times total retention, which is 60 terabytes here, before any compression. If segments are up to 1 gibibyte, you will have at least sixty thousand segments to track. In practice more, because many partitions close segments earlier than the size limit. Keep an eye on the catalogue’s memory use as this number grows.

Step 5: size the thread pools

Copy and cleanup threads should be enough for uploads to keep ahead of closing segments. Remote reader threads should match how many replays you expect at once. Start with the defaults, watch the queue behaviour, and adjust.

Numbers make the idea concrete. Here is a simple, honest calculation you can adapt to your own cluster.

The scenario

A company ingests 2 terabytes of new data per day into Kafka and wants to keep 30 days of history. Replication factor is 3. To keep things simple we assume block storage on the brokers costs about $0.08 per gigabyte per month and object storage costs about $0.023 per gigabyte per month. These are rounded, illustrative list prices. Real prices vary by provider, region and volume, and they change over time.

Without tiered storage

Thirty days at 2 terabytes per day is 60 terabytes of unique data. Three copies mean 180 terabytes of broker disk, or 180,000 gigabytes. At $0.08 that is about $14,400 per month for disks alone. And this ignores the free space real clusters must keep, which pushes the number higher.

With tiered storage

Keep one day on the brokers: 2 terabytes times 3 copies is 6 terabytes, or 6,000 gigabytes, costing about $480. The remote tier holds all 30 days once, which is 60 terabytes, or 60,000 gigabytes, costing about $1,380. The total is roughly $1,860 per month.

Costs that appear on the other side

• Request charges. Uploads and downloads are billed per request. With segments around 1 gibibyte, a few thousand uploads a day is tiny. Heavy replays that make millions of small reads are where request charges grow.
• Data transfer. Traffic inside a region to object storage is usually free or cheap, but crossing regions or leaving the cloud is not. Keep the bucket next to the brokers, and use private network endpoints so traffic does not pass through expensive gateways.
• Extra network load on brokers. You may need slightly stronger network capacity, but this is small compared with the disk you no longer need.
• Fewer brokers. The bigger saving often comes from shrinking the fleet. If brokers were added only for disk, many can go away, saving compute costs as well.

Moving data out of the brokers moves your security responsibilities too. The bucket is now part of your Kafka system.

Security: protect the bucket like a broker

Kafka access control lists protect topics on the brokers. They do not protect files in a bucket. Anyone who can read the objects can read the messages inside, because segments hold messages in Kafka’s ordinary format. So the bucket must be locked down as tightly as the brokers themselves.

Choosing the network path to the bucket

The route that bytes take to reach the bucket affects both safety and cost. If brokers reach the storage service over the public internet, traffic leaves your private network and may be exposed to more risk. If it travels through a shared gateway that charges by the gigabyte, a busy tiering setup can create a surprising bill. Most cloud providers offer a private endpoint, which connects your network to the storage service directly, keeps traffic off the public internet and avoids the gateway charges. Using it is usually the right default. Also confirm that brokers in every availability zone can reach the endpoint, or a zone failure could take tiering down with it.

What to monitor

Kafka exposes new broker metrics for the remote tier. Names differ across versions, but they cover the same ideas: bytes copied per second, bytes fetched from remote per second, request rates, error counts for copy, fetch and delete, and the total size of the remote log. Recent versions also report how far behind the copy process is.

Logging

Broker logs record what the RemoteLogManager and the storage plugin are doing. Upload failures, permission errors and timeouts appear here first. Make sure these logs are shipped to your central logging system and that you have alerts on repeated errors.

Operating routines

• Test restoring a broker from scratch while tiering is on, and time it.
• Practise a bucket outage in staging so the team knows what a full local disk looks like.
• Upgrade brokers and plugins together, and read release notes for tiered storage changes.
• Do not add bucket lifecycle rules that delete or archive Kafka’s objects. Kafka manages deletion itself, and outside interference breaks the catalogue’s promises.

When something looks wrong, start with the symptom. This table covers the problems teams meet most often.

A calm way to investigate

Every useful feature has edges. Knowing them in advance saves painful surprises.

Built-in limitations

Compaction rewrites old segments to remove outdated values. That does not fit with immutable, already-uploaded segments, which is why compacted topics are excluded. Also remember that early releases had gaps, so check the notes for your version before relying on a feature.

Common mistakes

A short checklist before go-live

• Have you confirmed the plugin supports your exact Kafka version?
• Have you chosen local retention using real consumer behaviour?
• Are throttles and quotas set?
• Do alerts exist for copy lag, errors and local disk?
• Have you rehearsed a bucket outage?

Where does tiering pay off in practice? These patterns show up again and again in production.

The vendor landscape

Questions to ask a managed service provider

Many teams will not run the plugin themselves but will buy tiered storage as part of a managed Kafka service. That is a sensible choice, but the details still matter. Before you sign, ask a few plain questions and expect plain answers.

Deciding whether you need it

Tiering is one answer to the long-retention problem. Here is how it stacks up against the alternatives teams often consider.

Tiering and the data lake are friends

Teams sometimes think they must choose between tiered storage and a data lake. They solve different problems. Tiering keeps history inside Kafka for replay by streaming applications. A data lake stores data in analytics-friendly formats for queries, dashboards and reports. Many mature platforms run both: Kafka with tiering for the event log, and a connector that lands selected topics into the lake.

Looking ahead

The Kafka community is discussing designs that go further than tiering, such as writing data straight to object storage without local replication for some topics. These ideas trade latency for even lower cost and simpler operations. They are proposals and experiments today. Choose the tools that are stable in your Kafka version, and revisit the field as it matures.

A fair look at both sides, so you can decide with clear eyes.

Classic Kafka against tiered Kafka

A compact playbook drawn from everything above.

Design patterns that work well

The safest way to adopt tiering is slowly, with a way back at every step. This plan works for both small and large clusters.

Phase 1: prepare

Upgrade to a Kafka version where tiered storage is production ready, or use a managed service that offers it. Choose and test a plugin in a staging cluster. Create the bucket, the role, the encryption keys and the network path. Write down who owns the bucket, because ownership gaps cause the worst incidents.

Phase 2: pilot on one topic

Choose a large but low-risk topic. Enable tiering, but leave local retention equal to total retention at first. This means nothing is deleted locally yet, so if anything goes wrong, all data is still on the brokers. Wait until the copy lag metric falls to near zero and the remote log size matches your expectation.

Phase 3: verify reads

Before removing any local data, prove that remote reads work. Use a test consumer to read from the beginning of a tiered segment range and compare counts and sample content with the original. Test with restarts and with a simulated leader change.

Phase 4: shrink the local window gradually

Now lower local retention in steps, for example from 30 days to 14, then 7, then 3, then 1. After each change, wait long enough to see disk usage fall, and check latency for your real-time consumers. Stop as soon as a consumer complains or metrics show pressure.

Phase 5: expand and operate

Repeat for other large topics. Add alerts, dashboards and run books. Practise the failure scenarios. Review costs each month against your prediction.

A quick reference for every term used in this guide, in the simplest words we can find.