Before learning how to isolate tenants, let us understand why anyone takes on the trouble of sharing.

Apache Kafka is a system that moves streams of events between programs. An event might be a payment, a click, a sensor reading, or a status change. Kafka stores these events in named channels, keeps them safe by copying them across machines, and lets many programs read them at their own pace. It began at LinkedIn and became open source in 2011.

Running Kafka well is real work. Someone must size the machines, upgrade versions, watch disks, tune settings, and answer the phone when something breaks. A company with thirty teams could give each team its own cluster. But that means thirty upgrade schedules, thirty monitoring setups, and thirty sets of idle capacity that nobody can lend to a busier neighbor.

The price of sharing

Sharing has a cost, and the cost is risk. On a shared cluster, one tenant can hurt another in many ways, sometimes by accident and sometimes on purpose. A buggy program can flood the network. A forgotten test can fill the disks. A curious engineer can read data that belongs to someone else. A single mistaken command can delete a topic that another team depends on.

Multi-tenant isolation is the set of walls, meters, and rules that keep those risks small. The goal is simple to state: each tenant should behave as if it had its own cluster, while the company pays for one.

The word “tenant” means different things in different companies. Agree on a meaning first, because every rule you write will hang on it.

A tenant is any group of users and programs that you want to treat as one unit for isolation, billing, and responsibility. That is deliberately loose. Here are the most common meanings.

Kafka pieces you need to know

To talk about isolation, we need a short list of Kafka parts. Each is a place where tenants can collide.

There is no single way to be multi-tenant. Most companies end up using a mix of three basic shapes.

flowchart LR
  subgraph M1["Model 1: Cluster per tenant"]
    TA1["Tenant A"] --> CA["Cluster A"]
    TB1["Tenant B"] --> CB["Cluster B"]
    TC1["Tenant C"] --> CC["Cluster C"]
  end
  subgraph M2["Model 2: Shared cluster"]
    TA2["Tenant A"] --> SH["Shared Cluster"]
    TB2["Tenant B"] --> SH
    TC2["Tenant C"] --> SH
  end
  subgraph M3["Model 3: Hybrid tiers"]
    TA3["Tenant A critical"] --> DC["Dedicated Cluster"]
    TB3["Tenant B"] --> SC["Shared Cluster"]
    TC3["Tenant C"] --> SC
  end
    

Model 1: a cluster per tenant

Every tenant gets its own Kafka cluster. Isolation is as strong as it can be, because nothing is shared except the cloud or data center underneath. A failure, a bad deployment, or a runaway program stays inside one cluster. The downside is cost and work. Each cluster needs its own machines, upgrades, and monitoring, and most of them run mostly idle.

Model 2: one shared cluster

All tenants live together. This is cheapest and easiest to manage, and it makes sharing data between teams easy. But every safeguard must be built inside Kafka itself, using naming rules, access lists, and quotas. One serious failure affects everyone at once.

Model 3: hybrid

Most tenants share a cluster. A few special ones, perhaps those with strict legal needs or huge traffic, get their own. This balances cost and risk and is what many mature platforms settle on.

Isolation is not one thing. It is a stack of separate protections, and a gap in any layer can let trouble through.

flowchart TD
  T["Tenant applications"] --> L1["1 Identity: who are you"]
  L1 --> L2["2 Access: what may you touch"]
  L2 --> L3["3 Naming: where do your things live"]
  L3 --> L4["4 Traffic limits: how much may you send"]
  L4 --> L5["5 Storage limits: how much may you keep"]
  L5 --> L6["6 Placement: which machines hold your data"]
  L6 --> L7["7 Visibility: who can see your data and metrics"]
  L7 --> B["Shared brokers, disks, network, and controller"]
    

The next chapters take these layers one at a time. As you read, keep asking the same question about each: if this layer fails, what is the worst a neighbor could do?

You cannot limit or protect what you cannot name. Identity comes first, and it must be done carefully.

When a program connects to a broker, Kafka can require it to log in. After login, Kafka labels the connection with a principal, a name such as User:payments-ledger. Every rule that follows, whether an access rule or a quota, attaches to that name. So the quality of your isolation can never be better than the quality of your identities.

How Kafka logs clients in

Kafka supports several login methods. Mutual TLS uses certificates. SCRAM uses salted password proofs stored in the cluster. OAuth uses short-lived signed tokens from an identity provider. Kerberos uses tickets. All of them should run over an encrypted connection, so nobody on the network can copy credentials. Which one you pick matters less than using one, and using it for every tenant.

Rules for tenant identities

The anonymous connection trap

If a listener allows connections without login, every client on it arrives with the same anonymous name. Quotas and access rules then treat all anonymous clients as one giant tenant. Make sure every client-facing listener requires login. Also make sure the setting that lets everyone in when no rule exists is turned off, so that missing rules mean no access rather than open access.

Client IDs are labels, not proof

Kafka clients can also announce a client ID, a free-text label. It is handy for metrics and for quotas, but clients choose it themselves, so one tenant could claim another’s label. Use the authenticated principal for anything security-related, and use the client ID only as an extra tag that helps you tell apart the several programs one tenant runs.

Kafka has no built-in folders for tenants. A good naming pattern acts as a set of folders, and it makes almost every other rule simple.

In Kafka, all topic names live in one flat list for the whole cluster. There are no real sub-areas. So multi-tenant platforms invent them by agreeing on a naming convention: every topic name starts with a fixed prefix that identifies its owner.

A pattern that works

A common recipe puts the tenant first, then the domain or application, then the data name, and sometimes a version or environment. For example, a topic name might read like tenant, then team, then dataset, then version, separated by dots. Something like a retail tenant’s fraud team writing a stream of scored transactions could become retail.fraud.scored-transactions.v1.

Names beyond topics

Topics are not the only things that need a prefix. Consumer group names, transactional identifiers, and schema subjects are all shared lists too. If one tenant names its group “billing” and another does the same, they could accidentally join each other’s group and split the work between unrelated programs, or read offsets that belong to someone else. Apply the same tenant prefix to groups, transaction IDs, and connector names.

Reserved and shared names

Keep a small set of reserved prefixes for platform-owned topics, such as shared reference data or company-wide event streams. Tenants get read-only access to those. Reserve names for internal Kafka topics too, and make sure no tenant can create or write them.

Once Kafka knows who is asking, access rules decide what they may do. This is the wall between your tenants’ data.

An ACL, short for access control list, is a rule such as “this principal may write to this topic”. Kafka checks these rules on every request. If no rule allows the request, Kafka refuses it. Getting these rules right, and keeping them right, is the heart of data isolation.

The parts of a rule

Prefixed rules: the tenant’s superpower

Kafka lets a rule match a resource name in three ways: an exact name, a wildcard that matches everything, or a prefix that matches every name starting with given text. The prefix style, added in Kafka 2.0, is perfect for tenants. One rule can say “the retail tenant’s service may read and write every topic starting with retail.“. New topics that follow the convention are covered automatically, with no new rules to write.

Deny by default

Kafka has a setting that decides what happens when no rule mentions a resource. Leave it at its strict default, so that no rule means no access. If it is flipped to permissive, every missing rule quietly becomes an open door. Check this setting first on any cluster you inherit.

Which resources need protecting?

Hidden topics

Kafka only reveals a topic in listings to principals allowed to describe it. So when describe rights follow the prefix rule, each tenant sees only its own topics. Nobody can browse a neighbor’s list of topic names, which matters because names alone can leak business secrets, like the codename of an unreleased product.

Deny rules and super users

A deny rule beats an allow rule when both match. That makes deny rules useful as emergency brakes, for example to cut off one identity quickly. Separately, Kafka lets you name super users who skip all checks. Keep this list tiny, give each person their own named identity instead of a shared one, and never place a tenant application on it.

A shared cluster has more ways to go wrong than a private one. Walking through the threats one by one shows where each control earns its place.

Security people start with a threat model, which is simply a list of who might cause harm and how. For a multi-tenant Kafka cluster, the interesting threats are not only outside attackers. They include honest mistakes by tenants and curious insiders. Here is a practical list.

Separate listeners by audience

Give different audiences different doors. Tenants connect to a client listener. Brokers talk to each other on a separate internal listener with its own login method. Administrative tools use a third listener reachable only from a management network. Even if someone finds a weakness in the tenant-facing door, it does not automatically open the admin door.

Limit what tenants may manage

Tenants often want to create their own topics. You can allow this safely by granting create rights only on their prefix, and backing it with a topic policy that caps size and settings. Do not grant tenants rights over the whole cluster. Altering cluster settings, reassigning partitions, or changing quotas belong to the platform team. Also be careful with the right to read topic settings, since settings can reveal how another team configured its systems.

Keep an audit trail

Kafka’s authorizer can log every allowed and denied request. Send those logs to a central place and search them. A steady trickle of denials from one identity is a useful signal: it may be a bug, a leftover configuration, or someone probing. Also log changes to ACLs, quotas, and topics, with the person and the request that caused them. When something goes wrong, the trail turns guessing into reading.

Review access on a schedule

Access rules tend to grow and never shrink. Every quarter, list each identity and its rules, and ask the owner whether they are still needed. Remove identities of services that no longer exist. Old, unused credentials are the ones attackers love, because nobody notices when they are used.

Access rules stop tenants from reading each other’s data. Quotas stop them from starving each other of resources. Both are needed.

Imagine an apartment with no water meter and no limit. One tenant fills a swimming pool, and the pressure drops for everybody. A quota is a limit on how much of a shared resource a tenant may use per second. Kafka enforces quotas on the brokers, and a tenant that goes past its limit is slowed down instead of being allowed to hurt the others.

The noisy neighbor problem

The classic multi-tenant failure is called the noisy neighbor. One tenant suddenly sends or reads far more than usual, perhaps because of a bug, a replay of old data, or a big batch job. Because brokers share network cards, CPU, and disks, the surge takes capacity away from everyone else. Without quotas, well-behaved tenants see slow responses and failures for reasons that have nothing to do with their own code.

Bandwidth quotas

The most used quotas limit data volume. There are two: a produce rate in bytes per second for writing, and a fetch rate in bytes per second for reading. They are set for a principal, for a client ID, or for a combination of both. The most specific setting wins, and defaults catch everyone with no setting of their own.

Request quotas

Bytes are not the only cost. A tenant sending huge numbers of tiny requests can use up broker processing time even with little data. A request quota limits the share of broker thread time a tenant may consume. It protects against chatty clients, expensive operations, and buggy loops that hammer the brokers.

Controller mutation quotas

Creating and deleting topics and partitions are heavy operations because the controller must update the whole cluster’s map. Kafka 2.7 added a mutation quota that limits how fast one tenant can create or remove topics and partitions. Without it, a script that creates ten thousand topics in a loop can stall the controller for everyone.

Connection quotas

Every connection costs memory and processing time. A program that opens and closes connections in a tight loop can overwhelm a broker with handshakes. Limits on new connections per second, and on total connections per address, cap that damage. Keep in mind that connection limits by address only help when tenants come from separate addresses.

Quotas do not block tenants outright. They slow them just enough to stay under the line. Here is the mechanism, step by step.

sequenceDiagram
  autonumber
  participant P as Tenant Producer
  participant B as Broker
  participant Q as Quota Tracker
  P->>B: Produce request with data
  B->>B: Write data to the log
  B->>Q: Record bytes for this principal
  Q-->>B: Rate over the limit, delay needed
  B-->>P: Response held back for the delay
  P->>P: Sees throttle time and slows down
  P->>B: Next produce request, later
  B->>Q: Record bytes again
  Q-->>B: Rate now within limit
  B-->>P: Response without delay
    

Measure, then delay

The broker keeps a running total of how many bytes each principal has sent or fetched over a short sliding window, about a second long. After handling a request, it compares the rate to the tenant’s limit. If the tenant is over, the broker works out how long it should have taken to stay within the limit, and holds the reply back for that extra time.

Because clients wait for replies before sending more, the delay naturally reduces the rate. The broker does not need to drop data or close connections. The client just experiences slowness. Well-written clients also read a “throttle time” value in the reply, and report it in their metrics, so the tenant can see they are being limited.

Which limit applies

A tenant may match several settings: one for the exact principal and client ID pair, one for the principal, one for the client ID, and defaults for each. Kafka picks the most specific match. Platform teams usually set a default per principal and override for tenants that have paid for more, or for the platform’s own tools.

What throttling does not do

Knowing that quotas exist is easy. Picking the numbers is where teams get stuck. A simple method gets you to sensible starting points.

Let us use a made-up cluster of six brokers. After measuring, the platform team decides each broker can safely handle about 150 MB per second of client writes, once replication traffic and consumer reads are taken into account. Notice that this is a safe working figure, well below the hardware’s raw limit, so that the cluster survives a broker failure and a traffic surge.

Uneven spread

The division above assumes a tenant’s partitions are spread evenly across all brokers. If a tenant only has partitions on two brokers, its quota on those two is the only cap, and those brokers carry all the tenant’s load. So when a tenant’s data is concentrated, either raise the number of brokers involved, or set its quota with the concentrated brokers in mind.

Reads need their own number

Reads are usually a multiple of writes, because several consumer groups may read the same data. A topic written at 10 MB per second and read by four groups causes 40 MB per second of outgoing traffic. Set fetch quotas with this multiple in mind, and leave room for catch-up after a consumer outage, when a group reads faster than usual.

Start gentle, then tighten

A good practice is to first run quotas in observation mode: set them high, watch the real usage for a few weeks, then bring them down to about one and a half to two times the observed peak. This avoids surprising a tenant on day one, and gives each conversation real numbers to talk about.

Disks are the resource most likely to run out quietly. One tenant filling them takes the whole building down.

Kafka keeps events on disk for as long as the retention rules say. A tenant that keeps data far longer than planned, or writes more than expected, consumes disk space that every other tenant shares. Kafka has no single switch that says “this tenant may use at most 2 terabytes”. Instead, you combine several tools.

Retention rules per topic

Each topic can limit how long data is kept, how large it can grow, or both. Time-based retention says “delete data older than seven days”. Size-based retention says “keep at most this much per partition”. Set sensible defaults for the cluster, and set a ceiling that tenants cannot exceed without asking.

Estimating tenant storage

A quick formula helps with planning. Multiply the average write rate by the retention time, then multiply by the replication factor. For example, a tenant writing 5 MB per second and keeping data for 3 days stores about 5 × 86,400 × 3, roughly 1.3 terabytes of unique data. With three copies, that is about 3.9 terabytes of raw disk. Doing this sum for each tenant at onboarding time shows who will be a big spender.

Guardrails that make limits real

Cheaper storage for long retention

Recent Kafka releases support tiered storage, which keeps recent data on fast local disks and moves older data to cheaper remote storage. That lets tenants with long retention needs keep their data without eating premium disk space. If your version supports it, tiered storage turns a scary storage problem into a cost line.

Every topic and partition costs the cluster something, even when it sits empty. Without limits, tenants can exhaust the cluster’s bookkeeping.

Each partition takes up open files, memory, and replication work on the brokers that hold it. Each also adds to the map the controller maintains. When a broker fails, the controller has to move leadership of all its partitions elsewhere. The more partitions there are, the longer that recovery takes. A tenant that creates thousands of partitions “just in case” pushes that recovery time up for everyone.

Cost of a partition

Limits that help

Set a sensible cap on partitions per topic and on partitions per tenant, and enforce it through the topic creation policy. Publish a simple guideline to tenants: choose partitions for the throughput you need soon, not the throughput you might dream of. It is easy to add partitions later, but hard to take them away.

Modern Kafka runs its own controller quorum, called KRaft, which handles far larger partition counts than older designs. That is welcome, but it is not a reason to skip limits. Budget partitions per broker as a shared resource, and watch the total.

Programs that transform data in Kafka create topics and groups you did not ask for. In a shared cluster, those hidden helpers need fences too.

Kafka Streams is a library for building programs that read topics, process events, and write results. To do its job, it quietly creates extra topics. Some hold intermediate data that must be regrouped, called repartition topics. Others hold the saved state of the program so it can recover after a restart, called changelog topics. Their names start with the program’s application ID, which is also the name of its consumer group.

What to enforce

Other stream tools

SQL-style tools that sit on top of Kafka, and stream processors run on outside frameworks, follow the same idea. They create groups, internal topics, and sometimes many at once. Ask the tenant how their tool names its internal objects, then check that the names fall inside their prefix. If a tool insists on a fixed name that would clash with others, give that tenant its own cluster or its own dedicated instance of the tool.

Quotas share machines fairly. Placement goes further by deciding which machines a tenant’s data sits on at all.

By default, Kafka spreads new partitions across brokers to balance load. It does not know about tenants. That is convenient, yet it means every tenant is on every broker, so a bad tenant can stress every machine. Placement means steering a tenant’s partitions onto chosen brokers.

Why place tenants deliberately

How placement is done

Kafka does not have a simple “pin this tenant to these brokers” switch. Teams get the same result in a few ways. When creating a topic, the platform can supply an explicit list of which brokers should hold each replica. Later, a partition reassignment tool can move data between brokers. Balancing tools such as Cruise Control can automate this and follow rules like “keep this tenant’s topics only on brokers in this group”. Rack awareness lets Kafka keep replicas of a partition in different failure zones, so one lost zone does not lose all copies.

The cost of placement

Deliberate placement creates islands of capacity. A broker group reserved for one tenant cannot lend its spare room to another. You give up some of the efficiency that made sharing attractive. Use placement for the tenants that truly need it, and let the rest float on the general pool.

Topics get most of the attention, but tenants share several other things too. Each needs its own fence.

Consumer groups

A consumer group is a team of consumers splitting the work of reading a topic. Each group has a name and a stored position, called its offset. Group names are shared across the cluster, so two tenants using the same name would interfere with each other, joining the wrong team or moving each other’s positions. Enforce tenant prefixes on group names, using the same prefix rule as topics.

Groups also use a broker called the group coordinator to track members. A tenant whose consumers join, leave, and rebalance constantly can overload the coordinator and slow other groups it looks after. Watch for groups that rebalance again and again, and coach the owning team to fix session timeouts and restart patterns.

Transactions

Programs that use Kafka transactions identify themselves with a transactional ID, and Kafka tracks state for each. Apply prefix rules here too, and limit the number of transactional IDs a tenant may hold, since each takes up broker memory and coordinator time.

Schema Registry

Many teams register data formats, called schemas, in a shared registry. Subjects, the names under which schemas are stored, are a shared list. Give tenants their own subject prefixes or separate contexts. Without that, one tenant could register a new version of a shared schema and break another tenant’s readers.

Kafka Connect

Connect moves data in and out of Kafka using ready-made connectors. In a multi-tenant setup, you can run one Connect cluster per tenant for cleaner separation, or a shared one with strict rules. A shared Connect cluster has extra risks: connectors run inside shared processes, and each connector can be configured to use a different Kafka identity. Use per-connector client overrides where your version permits, so a connector uses its tenant’s identity and quotas, not the Connect cluster’s own.

Isolation is also about what tenants can see and what the platform team can see. Privacy needs its own thought.

What tenants should see

A tenant should be able to view its own topics, groups, and usage numbers, and nothing belonging to others. Access rules already hide other topic names. Metrics need the same care. If your dashboards show every tenant’s traffic to every user, tenants can learn their neighbors’ business volumes. Build per-tenant views that filter by prefix or by identity, and give tenants access only to their own.

What the platform team can see

Brokers store data in a form they can read. Anyone with admin rights over the machines or with super user access to Kafka can read every tenant’s data. For tenants with sensitive data, that may be unacceptable, even when it is the platform team looking.

Encryption at different levels

The trade-off of client-side encryption

Encrypting on the client gives real privacy but costs features. Brokers cannot inspect or compress the message contents well, and tools that read messages need the keys. Key management becomes the tenant’s job. Choose it for data that truly needs it, and rely on transit and disk encryption for the rest.

Every shared cluster meets the same handful of problems. Knowing their fingerprints lets you find the culprit fast.

A reliable diagnostic routine

Isolation you cannot see is isolation you cannot trust. Per-tenant measurement is what makes limits, fairness, and billing possible.

Metrics per tenant

Kafka reports many numbers per principal and per client ID, especially for quotas. Tag every graph by tenant by mapping principals or topic prefixes to tenant names. Then you can answer, at any time, the question “who is using what?”

Chargeback and showback

Showback means showing each tenant what they use and what it would cost. Chargeback means actually billing them. Both change behavior. A team that sees a monthly cost for a topic nobody reads soon deletes it. Simple cost models multiply traffic, storage, and partition counts by unit prices. Even rough numbers make tenants thoughtful about waste.

Give tenants a window

Provide a self-service dashboard where each tenant sees its own usage, limits, throttling, lag, and costs. It cuts support requests and helps tenants see their own problems before the platform team has to tell them.

Rules that depend on people remembering them will fail. Rules built into an automated pipeline hold up.

In a busy multi-tenant platform, dozens of teams need new topics, new identities, and changed limits every week. If the platform team handles each by hand, it becomes a bottleneck and a source of mistakes. The answer is a self-service pipeline: tenants describe what they want in a file, a person reviews it, and automation applies it consistently.

flowchart LR
  R["Tenant writes a request file"] --> PR["Review by platform team"]
  PR --> V["Automated policy checks"]
  V -->|"passes"| A["Apply: identity, ACLs, quotas, topics"]
  V -->|"fails"| R
  A --> M["Register in tenant catalog"]
  M --> D["Dashboard and alerts created"]
  D --> N["Tenant notified with connection details"]
    

What the pipeline enforces

Configuration as code

Keep the whole desired state, meaning tenants, identities, topics, ACLs, and quotas, in version control. A change request becomes a reviewed change, and the history shows who approved what. If someone edits the cluster by hand, a drift check flags it and puts things back. This turns isolation into something you can audit rather than something you hope for.

Not every tenant needs the same treatment. Offering a few clear tiers keeps the platform fair and the bills sensible.

A service tier is a package of limits and promises. Instead of negotiating every detail with every team, the platform offers three or four standard packages, and each tenant picks one. The idea is the same as airline seats: economy, business, and first class share one aircraft but get different space, service, and priority.

Keep production and non-production apart

Never share a cluster between production workloads and development or testing. Test environments generate strange traffic: replays, load tests, experiments with tiny mistakes. Those are exactly the noisy neighbors you fear. A separate sandbox cluster costs little and prevents a whole family of outages.

Promises must be measurable

Any promise you make, such as availability or delivery latency, must be something you can measure per tenant. If you cannot measure it, you cannot prove you kept it. Define each promise with a number and a way of measuring it, then publish the results to the tenants.

Sharing is a default, not a religion. Some tenants outgrow it, and knowing the signs saves painful surprises.

flowchart TD
  S["New or growing tenant"] --> Q1{"Legal or contract needs its own hardware or region?"}
  Q1 -- "Yes" --> D["Dedicated cluster"]
  Q1 -- "No" --> Q2{"Traffic near or above a big share of a broker pool?"}
  Q2 -- "Yes" --> D
  Q2 -- "No" --> Q3{"Needs stricter availability than the shared promise?"}
  Q3 -- "Yes" --> D
  Q3 -- "No" --> Q4{"Needs special versions, settings, or plugins?"}
  Q4 -- "Yes" --> D
  Q4 -- "No" --> Q5{"Keeps causing noisy neighbor incidents?"}
  Q5 -- "Yes" --> P["Priority pool or dedicated cluster"]
  Q5 -- "No" --> SH["Stay on shared cluster"]
    

Signs a tenant should move out

Making the move painless

Moving a tenant to its own cluster means copying its topics and switching its clients. Replication tools such as MirrorMaker can copy data between clusters while both run. Prepare by making sure tenants use a bootstrap name they can change without code edits, keeping all configuration outside the code, and keeping the topic naming identical on both sides. Then a move becomes a scheduled, reversible operation.

Multi-tenancy changes how you plan capacity. You are no longer sizing for one workload, but for the sum of many with unpredictable peaks.

Capacity is a portfolio

A single tenant might peak at ten times its average. But tenants rarely peak together, so a shared cluster can be smaller than the sum of every tenant’s worst case. This is the same math that lets airlines sell more seats than one plane could carry if everyone booked the same day. The gain is real, yet so is the risk: if peaks do line up, for example on a sales day, the cluster is caught short.

How to plan

Quota sum versus capacity

A strict approach is to make the sum of all tenant quotas fit inside cluster capacity, so nobody can overload it. That is safe but wasteful, because tenants rarely use their full limits. A looser approach oversubscribes: quotas add up to more than capacity, betting on staggered peaks. Pick your ratio deliberately, and watch the actual combined usage to see whether the bet is paying off.

Scaling out

When the cluster is full, add brokers. But new brokers start empty; Kafka does not automatically move existing partitions onto them. You must run a rebalance. Rebalancing itself uses network and disk, and can slow tenants while it runs. Throttle the data movement, schedule it for quiet hours, and move a little at a time.

On a shared cluster, some failures hit everyone at once. Understanding the blast radius helps you decide what to protect first.

The blast radius of a failure is how much of your system it damages. With a cluster per tenant, the blast radius of most failures is one tenant. With a shared cluster, a broker crash, a bad upgrade, or a full disk touches every tenant with data on it. Your isolation design tries to make the radius smaller.

Shared, tenant-independent protections

A fair look at what sharing gives and what it takes away.

The main trade-offs

Shapes that work, and shapes that look fine until they fail.

Patterns worth copying

Anti-patterns to avoid

A checklist for building a shared cluster, followed by the mistakes that catch most teams the first time.

Before the first tenant arrives

Common mistakes

Isolation that has never been tested is a belief, not a fact. Try to break your own walls before someone else does.

Run the tests as a regular habit

Do the cheap tests, such as access checks, automatically after every configuration change. Do the heavy ones, such as floods and broker loss, in a staging cluster on a schedule, or as planned game days in production for mature platforms. Record the results and keep them as evidence for audits and for tenants who ask, “how do I know I am safe?”

A made-up story to tie everything together. The company, Fabrikam Retail, and its teams are fictional.

Fabrikam Retail starts with one Kafka cluster used by a single team. Within a year, eight teams want to use it: orders, payments, inventory, search, fraud, marketing, analytics, and a data science group. Here is how the platform team turns the single cluster into a safe shared platform.

What the story teaches

How organizations of different shapes typically run shared Kafka. These are common industry patterns, described in general terms, not claims about any one company’s private setup.

Internal platform for a large company

A central platform team runs a small number of big shared clusters for the whole company, sometimes one per region or per environment. Product teams are tenants. The platform team provides a self-service portal, standard client libraries, default quotas, and cost reports. Large operators such as LinkedIn, Uber, and Netflix are known to run Kafka at very large scale for many internal teams, and they share a common lesson: automation and defaults matter more than heroics.

Software-as-a-service provider

A company that sells a product to thousands of business customers often keeps customers in shared topics, with a customer ID in each message. Large or regulated customers get dedicated topics, or dedicated clusters. Customers never connect to Kafka directly. Instead, the company’s own services stand between them and Kafka, which keeps identity and limits under the company’s control.

Managed Kafka services

Cloud and managed Kafka offerings typically let you create separate clusters quickly, and some offer ways to group workloads. Their pricing often nudges you toward several smaller clusters instead of one huge one. Many companies mix the two: a managed cluster per environment and per major domain, with multi-tenant controls inside each.

Federation of many clusters

Some large operators put a thin layer in front of many physical clusters, so tenants see one logical service while the platform spreads them across clusters. This lets the platform move tenants between clusters without tenants noticing, and keeps each cluster small enough to manage comfortably.

Data mesh and domain ownership

In a data mesh style, each business domain owns its data as a product and shares it with others through Kafka topics. The tenant model fits naturally: each domain is a tenant that owns a prefix, publishes read-only topics for consumers, and is responsible for quality and cost. Sharing between tenants happens through explicit, reviewed read permissions.

Every important word from this guide in one place.

Short, direct answers to questions people ask most often.

You now have the full picture of running many tenants on one Kafka cluster without letting them hurt each other.

Sharing a Kafka cluster saves money and effort, but only if each tenant is protected from the others. Protection comes in layers: identity, access, naming, traffic limits, storage limits, placement, and visibility. Each layer covers a gap the others leave open. Automation applies them consistently, monitoring proves they work, and an easy path to a dedicated cluster handles the tenants who outgrow sharing.