Fine-Grained ACL Design at Scale in Kafka
How to decide who can read, write and manage every topic in a shared Kafka platform, and how to keep those rules understandable when you have thousands of topics, hundreds of teams and millions of messages flowing every second.
Imagine a huge office tower where hundreds of companies rent rooms. Everybody enters through the same front door, and inside there are thousands of doors, cabinets and safes. If every visitor could open every door, one careless person could read the payroll files, shred the contracts or unplug the servers. A shared Apache Kafka cluster is exactly this kind of tower. Many teams send their data through the same brokers, and each team should only touch what belongs to it.
Kafka solves this with Access Control Lists, usually called ACLs. An ACL is a simple rule that says “this identity may (or may not) do this action on this resource.” One rule is easy to understand. A hundred thousand rules are not. This guide starts from zero, explains how a single Kafka ACL is built and checked, and then shows how experienced architects design ACL systems that stay simple, fast and safe when the cluster grows very large.
1Why Access Control in Kafka Matters
Before we design anything, we need to know what Kafka is protecting and why the early versions protected nothing at all.
A quick picture of Kafka
Apache Kafka is a system that stores and moves streams of events. An event is a small record of something that happened, such as “order 5521 was placed” or “driver 88 moved to a new location.” Events are written into named containers called topics. Programs that write events are called producers. Programs that read events are called consumers. Consumers that work together to share a topic are placed in a consumer group. The servers that store all this data are called brokers, and a set of brokers working together is a cluster.
Kafka was created at LinkedIn around 2010 to move activity data between internal systems. At that time it lived inside a private, trusted network. Anyone who could reach a broker could write to any topic or read from any topic. That was acceptable for a small team, but it became dangerous as Kafka spread to whole companies and started carrying payments, personal data and business secrets.
Authentication and authorization are different jobs
Authentication answers the question “who are you?” It is the act of proving your identity, for example with a certificate or a password. Authorization answers a different question: “now that I know who you are, what are you allowed to do?” Kafka ACLs belong to the second job. They only work well if the first job is done properly, because a rule about “the billing service” is worthless if anyone can pretend to be the billing service.
Think of a hotel. At check-in the receptionist looks at your passport and gives you a key card. That is authentication. The key card then opens your room, the gym and the lift, but not other guests’ rooms or the manager’s office. That is authorization. An ACL is like the list stored in the hotel computer that says which key card opens which door.
A short history
Security arrived in Kafka version 0.9 in late 2015. That release added encrypted connections, several ways to log in and the first ACL system. Later, Kafka 2.0 added prefixed ACLs, which let one rule cover every resource whose name starts with the same text. This single feature is the reason large clusters are manageable today. Around Kafka 3.2, a new ACL engine was introduced for clusters that no longer use ZooKeeper, and Kafka 4.0 removed ZooKeeper completely, leaving the newer engine as the standard choice.
What “fine-grained” means
A coarse rule says “the analytics team can use Kafka.” A fine-grained rule says “the fraud-detection service can read the topic named payments.card.authorizations.v2, using the consumer group fraud-detector, and nothing else.” Fine-grained control follows the principle of least privilege: every identity gets only the permissions it needs to do its job, and nothing more. If that identity is ever stolen or has a bug, the damage stays small.
Fine-grained is easy with ten topics. The real engineering challenge is staying fine-grained with ten thousand topics without drowning in rules. That challenge is the subject of this whole guide.
What can actually go wrong without ACLs
Security people begin every design by asking a blunt question: what is the worst thing that could happen? For a shared Kafka cluster without good access control, there are four main dangers. The first is data theft: an application, or an attacker who has taken over an application, quietly reads topics that hold customer records. The second is data tampering: someone writes fake events into a trusted topic, for example false payment confirmations, and downstream systems believe them. The third is destruction: a script deletes topics or shortens their retention time, and messages that were never copied elsewhere disappear. The fourth is disruption: one careless program creates thousands of topics or hogs the network and slows everybody else down.
Notice that only the first of these is about secrecy. Three of the four are about integrity and availability. This is why “we encrypt the connections” is never a complete answer. Encryption keeps outsiders from listening on the wire, but it does not stop an insider with a valid login from doing something they should not do. ACLs are the tool that limits what a valid login can do.
Example: the fake refund events
Imagine an e-commerce company where a refund service reads a topic called refunds.approved and sends money back to customers. If any application could write to that topic, a bug in a completely unrelated marketing tool, or a malicious script, could publish fake approvals and drain funds. A single rule that lets only the approval service write to that topic closes the door.
2The Anatomy of a Single ACL
Every Kafka ACL, however complex the system around it, is built from the same few parts.
Kafka documents an ACL as one sentence: “Principal P is Allowed or Denied Operation O from Host H on any Resource R matching a resource pattern RP.” Let us take that sentence apart piece by piece, because every design decision later in this guide comes back to these pieces.
Principal
The identity making the request, written like User:billing-service. It comes from the login step.
Permission type
Either Allow or Deny. A Deny rule always wins over an Allow rule.
Operation
What is being attempted, such as Read, Write, Create, Delete, Alter or Describe.
Host
The network address the request comes from. Often set to a star, meaning any address.
Resource type
The kind of thing being protected: a Topic, a Group, the Cluster, a TransactionalId and so on.
Resource name and pattern
The exact name, a name prefix, or a star for everything, plus how the name should be matched.
Reading a real rule out loud
“User:billing-service is Allowed to Write from any host on the Topic whose name is exactly billing.invoices.v1.” Read that sentence again and you will see all six parts. A second rule might say “User:billing-service is Allowed to Read on the Group whose name starts with billing-.” That rule uses a prefix instead of an exact name, so it covers billing-workers, billing-audit and any other group that starts with that text.
How Kafka decides when many rules apply
When a request arrives, several rules may match it at the same time. Kafka follows a strict order. First it asks whether the caller is a super user, an identity listed in the broker configuration that skips all checks. Then it looks for a matching Deny rule. If one is found, the request is refused immediately. Then it looks for a matching Allow rule. If one is found, the request is permitted. If no rule matches at all, the default behaviour applies, and by default the answer is “no.”
Picture a nightclub doorman with two lists. The first list holds banned guests, and the second list holds invited guests. He checks the banned list first. If your name is there, you are out, even if you also appear on the invited list. Only if you are not banned does he look at the invited list. If you appear on neither, he does not let you in.
Kafka has a broker setting called allow.everyone.if.no.acl.found. When it is true, any resource that has no rules at all becomes open to everybody. It is meant only for a gentle migration. Leaving it on in production silently turns “we forgot to add a rule” into “everyone can read this topic.”
The wildcard and the host field in more depth
Two parts of the sentence look harmless but deserve extra care. The first is the star. A star in the resource name means “every resource of this type,” and a star in the principal means “every authenticated identity.” A rule that allows User:* to Read on Topic:* is the same as saying “anyone with a valid login can read everything.” Sometimes that is exactly what you want for a small development cluster. In production it is almost always a mistake, because it removes the very protection ACLs exist to give.
The second is the host field. It tells Kafka which network address the request must come from. The idea sounds attractive: “the billing service may only write from these three servers.” In real networks this breaks down. Cloud load balancers, service meshes and address translation often replace the real address with their own, so the broker sees the same few addresses for everybody. Containers that restart on new machines also change their addresses all the time. Most teams therefore set the host to a star and rely on strong identities and separate network zones for location-based safety.
Reading an ACL list like a detective
When you look at a list of rules for the first time, ask five simple questions. Who is the principal, and is it a real application or a shared login? Is the permission type Allow or Deny? Is the resource a single name, a prefix or a star? Is the operation broader than the job needs, for example All when Read would do? And does anything look like it was added for a one-time task and never removed? Five questions, asked calmly, will reveal most of the risk in an unfamiliar cluster.
Whenever you add a rule, write it as a full English sentence in the change description. If the sentence sounds strange when read aloud, for example “everyone may do everything on every topic,” the rule is probably wrong.
3Resources, Operations and Pattern Types
The three menus you choose from when writing any rule: what is protected, what action is allowed, and how names are matched.
Resource types
A resource is anything Kafka wants to protect. Each kind of resource has its own name space, so a topic called orders and a group called orders are two different things and need separate rules.
| Resource type | What it protects | Typical example |
|---|---|---|
| Topic | The event containers and their data | payments.card.authorizations.v2 |
| Group | Consumer groups and their saved reading positions | fraud-detector |
| Cluster | Cluster-wide actions such as creating ACLs or inter-broker traffic | The single cluster resource |
| TransactionalId | Producers that write several messages as one all-or-nothing unit | billing-tx-1 |
| DelegationToken | Short-lived login tokens that Kafka itself issues | A token owned by one user |
| User | Actions that create tokens on behalf of a user | User:billing-service |
Operations
An operation is the action being tried. The ones you will meet most often are these. Read lets a program fetch messages from a topic or join a group. Write lets a program append messages to a topic. Create and Delete control making and removing topics. Alter changes a resource, for example adding partitions. Describe lets a program look at a resource’s details, such as how many partitions a topic has. DescribeConfigs and AlterConfigs do the same for settings. ClusterAction is reserved for brokers talking to each other. IdempotentWrite was used for producers that avoid duplicate messages, and All is a shortcut that means every operation.
Some operations imply others. If you allow Read, Write, Delete or Alter, Kafka automatically treats Describe as allowed as well, because you cannot read a topic that you are not allowed to look at. In the same way, AlterConfigs implies DescribeConfigs. This means you do not need to write two rules where one is enough, but it also means a Deny on Describe can produce surprising side effects if you are not careful.
Pattern types: the secret to scale
The third menu is the pattern type, which tells Kafka how to compare the name in the rule with the name in the request.
Literal
The rule applies to exactly one name. Precise, but you need one rule per resource.
Prefixed
The rule applies to every name that begins with the given text. One rule can cover thousands of topics.
Wildcard
A literal star matches every resource of that type. Powerful and risky, so use it sparingly.
Principals can also use a star, written User:*, meaning “any authenticated identity.” Combined with a Deny rule, this is a handy way to block a sensitive topic for everyone and then carve out explicit exceptions with more specific rules, keeping in mind that a Deny always beats an Allow.
Creating topics: two doors, not one
The right to create a topic can be given in two ways. A rule with Create on the Cluster resource lets an identity make a topic with any name. A rule with Create on a Topic resource, usually a prefix, lets it make only topics whose names match. For a shared platform, the second door is nearly always the right one. Combined with a prefix per team, it lets each team create its own topics freely inside its own area and nowhere else.
Automatic topic creation deserves special mention. If it is switched on, a simple metadata request for a topic that does not exist can cause the broker to create it, provided the caller has the Create permission. This means a small spelling mistake in a producer’s configuration can produce a brand new, unowned topic with default settings. Turn automatic creation off on shared clusters and let a controlled pipeline create topics with the right settings and the right rules from the very start.
Cluster-level actions worth knowing
Alter on Cluster
Needed to create or delete ACLs. Whoever holds it can rewrite all permissions, so it belongs only to the control plane and a break-glass account.
ClusterAction
Used by brokers and followers when copying data between themselves. Only broker identities should have it.
Describe on Cluster
Allows viewing basic cluster information. Harmless for most tools, but still worth granting deliberately.
AlterConfigs
Changes settings such as how long a topic keeps data. A bad change here can delete data quietly, so treat it as a powerful right.
The right to alter topic settings can be more dangerous than the right to read. Someone who can shorten a topic’s retention to a few seconds can cause every unread message to vanish. Grant configuration rights only to the owners of a prefix and to the platform team.
A worked example of matching
Suppose three rules exist for the principal User:reporting. Rule one allows Read on the topic named sales.daily, matched literally. Rule two allows Read on every topic whose name starts with sales.regional., matched by prefix. Rule three denies Read on the topic sales.regional.secret, matched literally. Now test some requests. A read of sales.daily is allowed by rule one. A read of sales.regional.europe is allowed by rule two. A read of sales.regional.secret matches both rule two and rule three, and because Deny wins, it is refused. A read of sales.monthly matches no rule at all, so the default refusal applies. Working through small examples like this on paper is the fastest way to build confidence before touching a real cluster.
4Principals: Who Exactly Is Asking?
An ACL is only as trustworthy as the identity it is written for.
Where a principal comes from
A principal is the name Kafka gives to the caller after login. Kafka never invents it. It is produced by whichever login method the connection used. Kafka supports several methods. With mutual TLS (mTLS), both sides show certificates, and the principal is built from the client certificate’s subject. With SASL, the client proves itself using a username and password (PLAIN or SCRAM), a Kerberos ticket (GSSAPI) or a token from an identity provider (OAUTHBEARER). In every case, the result is a text label such as User:payments-api.
The ugly certificate name problem
When mTLS is used, the default principal is the full distinguished name from the certificate, something like CN=payments-api,OU=Payments,O=Example Corp,C=US. Writing ACLs with that long text is fragile, because a small change in the certificate’s fields would silently produce a different principal and lock the application out. Kafka lets you define principal mapping rules that shorten this to simply payments-api. Architects nearly always set these rules up on day one.
Kafka has no groups or roles in its ACLs
This is the most important limitation to understand. In the built-in ACL system, a rule names one principal, or the star. There is no way to say “everyone in the fraud team” or “anyone holding the reader role.” If fifty people need the same access, then either fifty rules exist or they all share one identity. Neither option is comfortable. Several products fill the gap: Confluent Platform adds role-based access control, Apache Ranger can plug in as an external authorizer, and Kubernetes operators such as Strimzi let you describe users and their rules in one file. We will return to these choices in Chapter 10.
Built-in Kafka ACLs are like a building whose security desk only accepts a list of individual names. There is no line saying “all employees of Company X.” If Company X has 200 staff, someone must type in 200 names, and update the list every time somebody joins or leaves.
Service accounts, not people
The solution used almost everywhere is to give each application its own identity, called a service account, instead of giving access to individual human beings. A human engineer normally does not read production topics directly. The payments service does, and its identity is one clean principal with a short list of rules. This gives you three benefits: the rules stay few, the audit trail names the application instead of a person, and rotating that application’s credentials does not disturb anyone else.
Never let several unrelated applications share one principal just to save effort. The moment two services share a login, you can no longer give them different permissions, and you can no longer tell from the logs which one did something.
A note about Amazon MSK
If your Kafka runs on Amazon MSK, you have an extra choice. MSK supports normal Kafka ACLs when clients log in with mTLS or SCRAM. It also offers IAM access control, where clients sign requests with AWS credentials and permissions are written as AWS IAM policies instead of Kafka ACLs. IAM gives you roles and groups for free, but it moves the rules out of Kafka and into AWS, and the two systems should not be mixed carelessly for the same clients. Pick one approach per client type and document it.
Choosing a login method: what each one means for your ACLs
The login method you choose decides how principals look and how hard they are to manage. There is no single winner. The table below compares the four most common choices from the point of view of ACL design.
| Method | Where the principal comes from | Strength | Difficulty at scale |
|---|---|---|---|
| Mutual TLS | Subject of the client certificate | Strong identity, no shared secret sent over the wire | Needs a certificate authority, renewal automation and mapping rules |
| SASL with SCRAM | The username stored in the cluster | Simple, good for many service accounts | Passwords must be stored, distributed and rotated safely |
| SASL with Kerberos | The Kerberos principal, shortened by rules | Fits companies that already run Kerberos | Complex infrastructure and clock sensitivity |
| SASL with OAuth tokens | A claim inside a signed token, such as the subject | Central identity provider, short-lived tokens | Depends on the token issuer being available; claim choice matters |
Short-lived credentials are friends of good ACLs
The longer a credential lives, the longer a stolen copy remains useful. Modern platforms therefore prefer credentials that expire in hours or days and are renewed automatically. On Kubernetes, a common approach is to give each workload its own identity and let a sidecar or operator fetch fresh certificates or tokens. On AWS, workloads running in containers or on virtual machines can use IAM roles, so no long-term password ever sits in a file. Whatever mechanism you choose, make sure the resulting principal name stays stable when the credential is renewed. The rules are written for the name, not for the individual certificate.
Delegation tokens in a sentence
Kafka can also issue its own short-lived delegation tokens, which let a user hand a limited credential to a worker process, such as a batch job, without sharing the user’s real password. The token acts on behalf of its owner, so the owner’s ACLs apply. This is convenient for large fleets of short-lived workers, and the DelegationToken resource type exists so that who may create and view tokens can be controlled as well.
5How a Broker Checks an ACL Internally
Every single request that reaches a broker passes through a small checkpoint. Understanding it explains why ACL design affects speed and reliability.
The journey of one request
Suppose the fraud-detection service asks a broker for new messages from a topic. Here is what happens before any data is returned. The request travels over a network connection that has already been secured and authenticated, so the broker knows the principal. The request is placed in a queue and picked up by a pool of request handler threads. Each thread decodes the request, sees that it wants to Read a particular topic and to use a particular group, and asks the authorizer a yes-or-no question for each resource involved.
sequenceDiagram
participant C as Client App
participant LB as Load Balancer
participant B as Broker Network Layer
participant H as Request Handler Thread
participant A as Authorizer in memory
C->>LB: Connect and log in
LB->>B: Forward connection
B->>B: Authenticate and build principal
C->>B: Fetch request for topic
B->>H: Queue the request
H->>A: Can this principal Read this topic?
A-->>H: Allowed or Denied
H-->>C: Data, or an authorization error
Notice that the authorizer is a component inside the broker, sitting in memory. It does not call another server for each request. That is deliberate. A cluster serving millions of requests per minute cannot afford a network hop for every permission check.
Step by step
Connection and login
The client opens a secured connection. The broker verifies the certificate or password and builds the principal.
Request decoded
A handler thread reads what the client wants: the action and the names of the topics and groups involved.
Super user check
If the principal is on the super user list, the answer is yes and the check ends.
Deny rules, then Allow rules
The authorizer looks for a matching Deny. If none, it looks for a matching Allow. If there is no match, the default is refusal.
Answer returned
Allowed requests continue to the storage layer. Refused requests get a specific error code, such as topic authorization failed or group authorization failed.
Refusals leak less than you think, but still leak
Even a “listing” request is filtered. When a client asks Kafka for the list of topics, the broker only returns those the caller is allowed to Describe. This stops a curious application from discovering the names of other teams’ topics. Still, the error codes tell a caller that an action was refused, so avoid putting secrets in topic names.
A common question is “Where does Kafka enforce ACLs?” The best answer is: on the broker, inside the request handling path, using an in-memory copy of the rules, with the order super user first, then Deny, then Allow, then default.
Which permission is used at which step of a consumer’s life
To design ACLs correctly, it helps to watch a consumer from the moment it starts until it commits its position. Many “mysterious” permission errors are simply one step in this story that nobody planned for.
Discover the topic
The client asks the broker for information about the topic, such as its partitions. This needs Describe on the topic.
Find the group coordinator
The client asks which broker manages its group. This needs Describe on the group.
Join the group
The client joins and receives its share of partitions. This needs Read on the group.
Fetch messages
The client pulls data in a loop. This needs Read on the topic.
Save progress
The client commits its offset, meaning it records how far it has read. This needs Read on the group and Read on the topic.
Broker-to-broker and controller traffic
Brokers are also clients of one another. A follower broker copies data from a leader by sending fetch requests, and these requests are checked as well. That is why the brokers’ own principal needs the ClusterAction operation on the Cluster resource. If you forget to grant this to the inter-broker principal when enabling an authorizer, replication stops and the cluster degrades quickly. Most people avoid the problem by listing the broker principal as a super user, but a more careful design gives brokers only the specific rights they need and keeps the super user list for emergencies.
Put the brokers on their own listener that is reachable only from inside the cluster network, with its own login. Application traffic then never shares a door with replication traffic, and a mistake in the application ACLs cannot break replication.
6Authorizer Implementations and the ACL Lifecycle
Kafka defines the shape of an authorizer but lets different engines do the work. Choosing the engine is a major design decision.
The plug-in model
Kafka defines an Authorizer interface, which is simply a contract: “give me a principal, an operation and a resource, and I will answer yes or no.” Any software that follows this contract can be plugged into the broker through one configuration line. Because of this, four families of engines exist.
| Engine | Where rules live | Best for | Watch out for |
|---|---|---|---|
| Legacy ACL authorizer | ZooKeeper | Older clusters that still use ZooKeeper | Removed together with ZooKeeper in Kafka 4.0 |
| Standard authorizer (KRaft) | Kafka’s own metadata log, replicated by the controllers | Modern clusters with no ZooKeeper | No groups or roles built in |
| Vendor RBAC (for example Confluent) | A central metadata service | Large organizations needing roles and groups | Extra service to run, licensing |
| External policy engines (for example Apache Ranger or Open Policy Agent style plug-ins) | An outside policy store, cached in the broker | Companies that already govern many systems from one place | Cache freshness, plug-in maintenance across Kafka upgrades |
How ACL changes travel
In a modern cluster, an administrator (or an automated tool) sends a “create ACL” request to any broker through the Admin API. The broker forwards it to the active controller, which appends it to the cluster’s metadata log. The log is replicated using Kafka’s Raft-based protocol called KRaft, so a majority of controllers must agree before the change is real. Every broker follows this log and updates its own in-memory copy of the rules. The whole process usually completes in a fraction of a second, but it is eventually consistent, which means for a very short moment two brokers may disagree.
flowchart LR
ADM["Admin tool or GitOps job"] --> BR["Any broker (Admin API)"]
BR --> CTL["Active controller"]
CTL --> LOG[("Metadata log replicated by controller quorum")]
LOG --> B1["Broker 1 in-memory ACLs"]
LOG --> B2["Broker 2 in-memory ACLs"]
LOG --> B3["Broker 3 in-memory ACLs"]
CL["Client requests"] --> B1
CL --> B2
CL --> B3
Imagine head office sends a new “approved visitors” list to every branch of a bank. Head office keeps the master copy, and each branch keeps a printed copy at its own front desk. Guards never phone head office for each visitor. They just check the printed copy. If head office is unreachable for an hour, guards keep working with their last copy, but nobody can change the list until the phone lines return.
What this means for reliability
Because every broker holds a local copy, reads of the rules keep working even if the controllers are briefly unavailable. What you lose is the ability to change ACLs. That is a good trade for a busy platform, since permission checks are frequent and changes are rare.
Migrating from ZooKeeper
Teams moving from the old ZooKeeper design to KRaft must move their ACLs as well. Kafka’s migration process copies existing ACLs into the new metadata log, but you should still export a full list of rules before starting, compare it afterwards, and rehearse the whole procedure in a test cluster. A missing rule after migration means an outage for the application that depended on it.
Choosing an engine: a simple decision guide
Choosing an authorizer is a business decision as much as a technical one. Ask what your organization already owns and how many teams need self-service. The guide below shows how experienced architects usually decide.
Native standard authorizer fits when
- You run open-source Kafka and want the fewest moving parts.
- Your naming conventions are strong enough for prefixed rules to do the heavy lifting.
- You are willing to build or adopt a small control plane.
- You value portability across clouds and vendors.
An external or vendor engine fits when
- Hundreds of humans need role-based access through a graphical tool.
- A central compliance team already governs many data systems in one policy store.
- You can afford to operate and upgrade the extra component.
- Your auditors require centralized reports on access across the entire company.
Sizing and start-up behaviour
Every broker loads all rules when it starts and replays changes as they arrive. On a cluster with a small rule set this takes moments. On a cluster with an enormous set, start-up and restarts become slower, and a rolling upgrade takes longer because each broker must rebuild its memory copy before serving traffic. This is one more reason to keep the total rule count low, and to include rule counts in your capacity dashboards alongside disk and network numbers.
Also plan for Kafka upgrades. Authorizers are code that runs inside the brokers, so a custom plug-in must be tested against each new Kafka version. Native engines travel with Kafka itself, which is a hidden but real advantage when the platform team is small.
Removing access: the forgotten half of the lifecycle
Teams spend most of their energy on granting access and very little on taking it away. Yet a rule that lingers after the need is gone is a hidden risk. Deleting an ACL uses the same path as creating one and takes effect just as quickly for new requests. One detail to remember is that existing long-lived connections are already authenticated, but each new request is still checked against the current rules, so a removed permission blocks the very next request rather than waiting for a reconnect. Even so, if a credential has been compromised, remove the rules and revoke the credential together, because the rules alone do not stop the attacker from connecting and probing.
7The Standard Recipes: What Each Kind of Client Needs
Most of your ACLs will follow a handful of repeating patterns. Learn them once and you can generate them automatically.
The table below lists the usual minimum permissions for common jobs. The exact list can change between Kafka versions and between tools, so always test a new application in a staging cluster before production.
| Client job | Resource | Operations needed |
|---|---|---|
| Simple producer | Topic it writes to | Write (Describe is implied) |
| Producer with duplicate protection | Topic; older versions also the Cluster | Write on the topic; older versions also needed IdempotentWrite on the Cluster |
| Transactional producer | Topic and its TransactionalId | Write on the topic, plus Write and Describe on the TransactionalId |
| Consumer in a group | Topic and Group | Read on the topic and Read on the group |
| Stream processing application (Kafka Streams) | Input topics, output topics, and everything named after its application id | Read on inputs, Write on outputs, and broad rights on names starting with its application id (internal topics and group) |
| Kafka Connect worker | Its internal config, offset and status topics, plus the group | Read, Write, Create on internal topics; Read on the worker group |
| Source or sink connector | Its data topics | Write for source, Read (and group Read) for sink |
| Monitoring tool | Topics, groups, configs | Describe and DescribeConfigs only |
| Platform administrator | Cluster, topics | Create, Delete, Alter, AlterConfigs, and Alter on the Cluster for managing ACLs |
Why consumers need two rules
New engineers are often surprised that a consumer needs permission on both the topic and the group. The topic rule protects the data. The group rule protects the saved reading position, called the offset. If someone else could write into your group, they could move your reading position forward and make you skip messages, or backward and make you process them again.
Stream processors are the special case
A Kafka Streams application creates its own internal helper topics, and their names all start with the application’s id. A prefixed rule based on that id is the natural fit here. It is a perfect example of prefixed ACLs doing their job, because the application may create dozens of internal topics you cannot predict in advance.
Example: an order-enrichment stream
A retail company runs a stream application with the id order-enricher. It reads the topic orders.raw, writes orders.enriched and creates internal topics whose names all start with order-enricher. Its principal receives Read on one topic, Write on another, and one prefixed rule for everything starting with order-enricher. Three rules, and future internal topics need no new rule at all.
Treat each row of this table as a template. When an application is onboarded, your tooling picks the template that matches its job and fills in the names. Humans should almost never type raw ACL rules by hand.
Reading the error messages
Permission errors in Kafka are very specific, and reading them correctly saves hours. The name of the error tells you which resource type was refused, and the message usually names the resource. The table below turns the common ones into plain English.
| What you see | Plain English | Usual fix |
|---|---|---|
| Topic authorization failed | The principal lacks the needed operation on a topic | Add Read or Write for that topic name or prefix |
| Group authorization failed | The principal cannot use that consumer group | Add Read on the group, or change the group name to fit the convention |
| Cluster authorization failed | A cluster-wide action was refused | Check whether the client really needs that action; often a sign of an old client setting |
| Transactional id authorization failed | A transactional producer lacks rights on its id | Add Write and Describe on the transactional id prefix |
| Authentication failed | The login itself did not work | Fix the certificate, password or token, not the ACLs |
Test every recipe before you trust it
Build a small test suite that plays each client role against a staging cluster with a fresh principal: a producer, a consumer, a stream application and a connector. For each role, verify that the allowed actions work, and also verify that the forbidden neighbours are refused, such as reading a topic outside the prefix. A recipe that has been proven both ways is one you can safely automate.
8Why ACLs Break Down at Scale
Small clusters forgive sloppy rules. Large ones do not.
The ACL explosion
Let us use an imaginary but realistic company. It has 400 teams, 30,000 topics, 2,000 applications and 6,000 consumer groups. If each application reads from an average of ten topics and writes to five, and each read needs a topic rule plus a group rule, a rule-per-resource design produces tens of thousands of rules. Add test environments, temporary migrations and copies of topics, and the count can reach several hundred thousand.
What goes wrong
Heavy brokers
Every broker keeps every rule in memory. Hundreds of thousands of rules increase memory use and slow the loading at startup.
Slow bulk updates
Creating rules one at a time through a command-line tool is painfully slow. Big migrations take hours.
Unreadable rule lists
No human can review 200,000 lines. Nobody knows which rules are still needed, so old rules never get deleted.
Typos and ticket delays
A missing character in a topic name means an application fails at 3 a.m. Manual ticket queues also make teams wait days for access.
Picture a hotel that cuts a separate physical key for every guest and every single room they are allowed to enter, including the towel cupboard on each floor. The key cabinet grows huge, guests carry heavy key rings and nobody can tell which keys are still valid. A hotel that instead gives each guest one card that opens “the rooms on my booking” is far easier to run. Prefixed ACLs and templates are that single card.
What people do
They create one literal ACL per application per topic by hand, following support tickets, with no naming rules for topics.
Why it hurts
The rule count grows faster than the topic count, nobody owns cleanup, and the state of the cluster can only be understood by dumping and reading the entire list.
What to do instead
Agree on naming conventions, use prefixed rules for ownership, and generate all rules from a reviewed source of truth. The next chapters show how.
The real cost of a wrong rule
Consider two mistakes. In the first, a rule is too narrow: an application cannot read its topic after a release. The cost is a failed deployment, an alert and a quick fix. Painful but visible. In the second, a rule is too broad: a new analytics application receives Read on every topic starting with an over-short prefix. Nothing fails, and everyone is happy. Months later an audit discovers that the application has been reading customer data it never needed. Too-broad mistakes are silent, and that is exactly why they are so dangerous. A good design makes the safe choice the easy choice and makes over-broad rules easy to detect.
The human cost
Beyond the technical trouble, manual access management wastes time. If every request needs a ticket, a review by a platform engineer and a manual command, a busy company can spend a large number of engineer-hours each month just moving permissions around. Developers wait days, get frustrated and start asking for wildcards “to avoid asking again.” The security design and the developer experience are the same problem. If getting the right access is quick and safe, people stop looking for shortcuts.
Security controls that are painful get bypassed. Measure the time between a team asking for access and the access working. If that number is measured in days, expect shadow arrangements to appear.
9Naming Conventions and Prefixed ACLs
The cheapest and most powerful scaling trick in Kafka security is not a tool. It is a rule about how names are written.
Encode ownership in the name
If every topic name starts with the team that owns it, then one prefixed rule can say “the payments team’s applications may write to anything that starts with payments.” A common layout is domain.team.dataset.version, for example payments.cards.authorizations.v2. Consumer group names and transactional ids follow the same convention, so the same prefix logic works for all three resource types.
| Name part | Example | Why it helps |
|---|---|---|
| Domain | payments | Lets security teams write broad rules for sensitive business areas |
| Owner team or product | cards | Gives each team a private area where it can create topics freely |
| Dataset | authorizations | Describes the data so readers know what they are requesting |
| Version | v2 | Allows a breaking format change without touching the old topic’s rules |
The trailing delimiter trap
Prefix matching is purely about text. A rule for the prefix pay also matches payroll.secrets and payments.cards. A rule for the prefix orders also matches orders-archive and ordersecret. Always finish the prefix with your separator, for example payments. with the dot, so the rule matches only the intended family of names. This single detail has caused many accidental data exposures.
Do not choose a prefix such as data or app. It is too short and will overlap with many other names. A prefix should be long enough and end with a separator so it belongs to exactly one owner.
Two enforcement helpers
Conventions only help if they are followed. Kafka has a create-topic policy hook, which lets platform engineers reject any topic whose name does not match the agreed pattern or whose settings are dangerous. Also turn off automatic topic creation, so a typo in a producer’s configuration cannot quietly create a stray topic outside your naming system. Together these two steps mean every topic belongs to an owner and is covered by a prefix rule.
Ownership rules versus access rules
A helpful way to think is to split rules into two layers. The ownership layer uses prefixed rules: the payments team’s service account may create, write, read and alter everything under its own prefix. The sharing layer uses narrow rules: the fraud team’s service account may Read exactly one topic that belongs to payments. The first layer is few and stable. The second layer is where reviews, approvals and audits should focus, because it represents data crossing team boundaries.
Example: onboarding a new team in one step
A new team named logistics joins the platform. The platform tool adds three prefixed rules for its service account: topics starting with logistics., groups starting with logistics. and transactional ids starting with logistics. That is all. The team can now create hundreds of topics without another ticket, and cannot touch anything outside its prefix.
A full worked design for a mid-sized company
Let us design a naming and rule scheme for an imaginary company with three business domains: payments, logistics and marketing. First decide how environments are separated. Production and non-production should ideally be different clusters, because then no prefix mistake in a test can ever touch production data. Inside each cluster, every name starts with the domain, then the owning team, as in the layout of the earlier table.
| Rule layer | Principal | Resource and pattern | Operations |
|---|---|---|---|
| Ownership | payments-cards service account | Topics starting with payments.cards. | Create, Write, Read, Alter, Describe, DescribeConfigs, AlterConfigs |
| Ownership | payments-cards service account | Groups starting with payments.cards. | Read, Describe |
| Ownership | payments-cards service account | Transactional ids starting with payments.cards. | Write, Describe |
| Sharing | fraud-detector service account | Topic payments.cards.authorizations.v2, literal | Read |
| Sharing | fraud-detector service account | Groups starting with fraud.detector. | Read |
| Guard rail | contractor-analytics service account | Topics starting with payments. (Deny) | Read, Write, so this identity can never touch the payments domain whatever other rules exist |
The last row teaches an important lesson. Because Deny beats Allow, you cannot “deny everyone and then allow one person” on the same resource. Kafka will always refuse the exception. To keep something restricted, do not use Deny as an on-off switch. Instead, place it under a prefix that nobody but its owner has been granted anything on, and let the default refusal do the work. Use Deny only for things that must never be reachable by a particular identity, whatever other rules may say.
What about many environments in one cluster?
Sometimes cost pressure forces development and testing to share a cluster. In that case add the environment to the prefix, for example dev. or staging., and apply the same rules per environment with separate service accounts. Never let a test service account hold rights on a production prefix. The prefix is your only wall, so it must be treated with the same care as a network boundary.
Write the naming convention as a short document with three good examples and three bad ones. Link it from the request form. The best security rule is the one everybody can read in one minute.
What if your names are already messy?
Most real companies do not begin with clean names. They inherit topics called test1, orders_new_final and john-temp. Renaming a live topic is hard, because Kafka has no simple rename, and every producer and consumer must move together. A practical plan has three parts. First, apply the new convention to every new topic from today, enforced by the create-topic policy. Second, create a small legacy zone: a fixed list of old topic names that get literal rules, owned and documented, so the mess stays contained and visible. Third, migrate the legacy topics one at a time as their owners release new versions, by creating a correctly named replacement, copying data across and switching applications over.
Over a year or two, the legacy list shrinks and the prefix system takes over. The important point is to stop the mess from growing. Every new topic that breaks the convention makes the future clean-up more expensive.
10Identity Design: Roles, Groups and the Control Plane
Since Kafka ACLs know nothing about groups and roles, large companies build that missing layer themselves.
The idea of a control plane
A control plane is a management layer that decides what the configuration of a system should be, while the data plane does the real work. In our case the data plane is Kafka, moving events. The control plane is a small service, or a set of scripts, that knows about teams, roles and approvals and turns them into the flat list of ACLs that Kafka understands. Humans work with friendly ideas such as “the fraud team can read the payments card topic.” The control plane compiles that idea into exact rules.
flowchart TB
IDP["Identity provider with teams and groups"] --> CP["Policy compiler in the control plane"]
GIT["Git repository of access requests"] --> CP
APR["Approval workflow"] --> GIT
CP --> DIFF["Compare desired rules with live rules"]
DIFF --> ADMIN["Kafka Admin API"]
ADMIN --> KAFKA["Kafka cluster ACL store"]
KAFKA --> AUD["Audit log and drift report"]
AUD --> CP
Roles as templates
Inside the control plane, a role is just a named bundle of ACL templates. A role called topic-producer might stand for “Write on this topic.” A role called topic-consumer stands for “Read on this topic and Read on a group with this prefix.” A role called stream-app stands for the whole Streams recipe from Chapter 7. When a team asks for access, it names the role, the topic and the service account. Nobody remembers operation names.
A restaurant kitchen does not give each cook a separate list of fifty allowed ingredients. It gives a job title, such as “pastry chef,” and everybody knows what that title can use. The control plane is the manager who turns “pastry chef” into the exact cupboard keys.
Your choices for the engine underneath
Native ACLs plus your own tooling
Lowest moving parts. You build or adopt a tool that produces ACLs from roles. Best when your rules follow strong naming conventions.
Operator-managed users
Tools such as Strimzi let teams describe a Kafka user and its rules in a Kubernetes file that is reviewed like any other code.
Vendor role-based access
Confluent Platform and Confluent Cloud offer roles and service accounts natively, with a central place to see who can do what.
External policy engine
Apache Ranger style plug-ins share one policy store across many data systems. Good for compliance, but they add a component that must stay fresh and available.
If you write your own authorizer plug-in that calls an outside service, cache aggressively and keep answering from the last known good copy if that service is down. A slow or unavailable permission service must never be able to stall every request on every broker.
Credentials live and die with the identity
An identity is not only a name but also a secret: a certificate or password. Automate its creation, store it in a secrets vault, and rotate it regularly. When an application is retired, the control plane should delete the principal’s rules and revoke its credential in the same workflow. Orphaned service accounts with lingering permissions are one of the most common findings in security audits.
Mapping company groups to Kafka permissions
Human engineers still need access sometimes: to inspect a topic during an incident, to test a new consumer or to run administration tasks. Never create individual accounts by hand. Instead, connect Kafka to the company’s identity provider, and let the control plane read group membership. For example, the group kafka-payments-readers in the identity provider can be compiled into read rules for the payments prefix on the non-production cluster. When a person leaves the group, the next reconciliation run removes their access. The identity provider stays the single source of truth for people, and Kafka stays a consumer of that truth.
Workload identity in the cloud
Applications need identities too, and the modern answer is workload identity: the platform itself vouches for which application is running. On Kubernetes on AWS, a workload can be tied to an IAM role, so a pod can prove who it is without any stored password. The control plane maps that role to a Kafka principal name, and the same name is used in every rule. The benefit is huge: nobody copies secrets around, credentials expire quickly, and the audit trail traces every action back to a named workload.
A strong answer to “How would you handle roles when Kafka has none?” mentions three ideas: a control plane that turns roles into ACL templates, an identity provider that owns group membership, and reconciliation that removes access automatically when membership changes.
11ACLs as Code: Governance, Review and Drift
The safest way to change permissions on a busy platform is the same way you change software: through reviewed, tested, versioned code.
Why manual changes fail
When an engineer opens a terminal and adds a rule, nobody else sees it. There is no record of why it was added, no test and no easy undo. Six months later nobody remembers whether it is still needed. Treating ACLs as code fixes these problems. The desired list of rules lives in a Git repository. Every change is a proposed edit that someone else reviews. The history tells you who asked for access, who approved it and when.
A healthy access-request flow
Request
A developer opens a request naming the service account, the topic and the role. A friendly form or a short file is enough.
Owner approval
The team that owns the topic’s prefix approves. Data owners, not the platform team, decide who may read their data.
Automatic checks
A pipeline validates naming, blocks wildcards and warns when a rule is unusually broad.
Apply in batches
A job compares desired rules with the live cluster and applies only the differences using the Admin API, in bulk rather than one at a time.
Verify and record
The job confirms the new rules are active and stores the result in an audit log.
Drift detection
Drift means the live cluster no longer matches the reviewed source of truth, for example because someone used a back door to add a rule. A reconciliation job should run on a schedule, list every live ACL, and compare it with the desired list. Unknown extra rules raise an alert. Missing rules can be restored automatically. This constant loop is what keeps a huge system honest.
A reconciliation job that deletes “everything not in Git” is powerful and dangerous. If the repository is accidentally empty, the job could remove every rule in the cluster and stop the whole company. Build in a limit, such as refusing to delete more than a small percentage of rules in one run without a human confirming.
Break-glass access
Sometimes production is on fire and a person needs quick access. A break-glass process gives a named person temporary, powerful access, alerts the security team and expires automatically after a short time. This is much safer than keeping a shared administrator password in a document.
Expiry and reviews
Give sharing rules an owner and, where sensible, an expiry date. Ask owners every quarter to confirm which consumers still need their data. Many companies discover that a large share of old rules point at applications that no longer exist. Removing them reduces both risk and the size of the rule set that every broker keeps in memory.
Rolling out ACLs on a live cluster without an outage
Many companies do not start with security. They start with an open cluster, grow to hundreds of clients and only later decide to lock things down. Turning on enforcement in a single step would break every application at once. A safe rollout is a careful, staged migration.
Take inventory
Find out who talks to what. Broker metrics per client id, per-topic byte counts and connection logs show which applications read and write which topics.
Add a secured listener
Open a new listener that requires encryption and login, and keep the old one temporarily. Brokers can serve both at the same time.
Create identities and rules first
Generate service accounts and ACLs from the inventory, in the control plane, before anything is enforced. Review the results with the owners.
Move applications gradually
Switch each application to the secured listener, team by team, starting with low-risk ones. Watch denial metrics after every move.
Close the old door
When no traffic remains on the open listener, shut it down, and turn off any permissive fallback setting.
The most important lesson is to build the rules before enforcing them. Teams that enable enforcement first and write rules while the phones are ringing usually end up granting wildcards in a panic and never tighten them again.
12Going Finer Than Topics: Protecting Data Inside a Topic
Kafka ACLs stop at the topic. Sometimes that is not fine enough.
The limit of the tool
A Kafka ACL cannot say “read only messages from the customer’s own country” or “read the order but not the credit card field.” Permission is decided per topic, and a reader who can read a topic gets every message in it. There is no built-in filter for a single partition, a message key or a field inside a message. If a topic mixes ordinary and sensitive data, the only safe reading of the rules is that everyone with access sees everything.
A topic is like a locked filing cabinet drawer. The lock decides who can open the drawer, but once it is open, all the folders inside are visible. If some folders are secret, you should keep them in a different drawer, or black out the secret lines before filing them.
Four design options
Separate topics by sensitivity
Write ordinary events to one topic and sensitive ones to another. Then use normal ACLs. This is the simplest and most reliable approach.
Derived safe topics
Keep the full raw topic locked down. A stream processing job removes or masks sensitive fields and publishes a sanitized topic that wider audiences can read.
Field-level encryption
Encrypt sensitive fields before they reach Kafka, using keys held in a key management service. Only readers with the key can see the real values.
Protocol proxy layer
A proxy such as Kroxylicious sits between clients and brokers and can apply encryption or filtering policies transparently, at the cost of another component.
flowchart LR
P["Payments service"] --> RAW[("payments.cards.raw.v1 restricted")]
RAW --> RED["Redaction stream job"]
RED --> SAFE[("payments.cards.safe.v1 wider audience")]
RAW --> FRAUD["Fraud detection service"]
SAFE --> BI["Analytics and reporting"]
SAFE --> SUP["Customer support tools"]
Do not forget the other places data sits
Kafka data also lives on broker disks, in backups and in copies made by tools such as mirroring or connectors. Protect disks with encryption, and remember that a sink connector copying a locked topic into a database gives that database’s readers the data too. Access control has to follow the data wherever it goes.
Schema tags help governance
Many companies attach labels such as “personal data” or “financial” to fields in their schema registry. Those labels do not enforce anything on their own, but a control plane can read them and refuse to approve broad access to any topic whose schema contains a sensitive label. This is a cheap way to put human judgement in the right place.
The cost side of finer protection
Every extra layer of protection has a price, and good architects say so out loud. Splitting topics doubles some of your topic count and asks producers to route each event correctly. Redaction jobs add a small delay and another application to run and monitor. Field encryption protects even from a stolen disk, but it makes searching and joining on encrypted fields harder, and it brings the new duty of managing keys. A proxy layer adds a network hop and another component that must scale with your traffic. None of these are reasons to avoid them. They are reasons to apply them only to the data that truly needs them, and to accept the simple topic-level ACL for everything else.
Classify your topics into three or four sensitivity levels, for example public, internal, confidential and restricted. Attach the level to the topic’s name or metadata. Then choose the protection style for each level once, and reuse it, instead of debating every topic separately.
Example: customer support and card numbers
A customer support tool needs to know that a payment happened and how much it was, but must never see the card number. The payment topic stays restricted. A redaction job replaces the card number with only its last four digits and publishes the result to a safe topic. The support tool receives read access to the safe topic only. Even if the support tool is hacked, the full card number is not in the data it can reach.
13Performance, Scalability and High Availability of Authorization
Permission checks happen on every request, so they must be nearly free and always available.
Why checks are cheap
Each broker keeps its rules in memory in a structure arranged so that a lookup jumps straight to the small group of rules for the resource in question, instead of scanning the whole list. A well-designed rule set therefore costs very little per request. The expensive parts are elsewhere: the memory used by very large rule sets, the time to load them when a broker starts and the volume of changes flowing through the controllers.
Levers for keeping it fast
Prefer prefixes
One prefixed rule replaces thousands of literal ones, cutting memory and startup time.
Bulk Admin API calls
Send many rules in one request instead of thousands of separate commands, easing the load on controllers.
Remove dead rules
Delete rules for retired applications and topics so brokers keep only what is used.
Split huge clusters
When one cluster serves too many tenants, separate clusters for different business areas reduce the blast radius of any mistake.
High availability
Brokers keep answering permission questions from their in-memory copy even if the controller group is briefly unreachable, so the data path stays alive. Run an odd number of controllers, typically three or five, spread across separate failure zones, so a majority is always available to accept changes. For a cloud deployment on AWS, that means placing controllers and brokers in different Availability Zones so the loss of one zone leaves a majority intact.
If asked “What happens to authorization if the controller quorum goes down?” the strong answer is: reads continue from each broker’s local copy, but nobody can create or delete ACLs until the quorum returns. Then mention that this is exactly why administrators need a break-glass plan that does not depend on ACL edits.
Failure mode: the retry storm
A misconfigured client that is denied access often retries again and again. Multiply that by hundreds of instances during a rollout and the brokers spend real effort rejecting the same request. Set sensible retry back-off in client libraries, use client quotas to limit how much load any one principal can create, and alert quickly on repeated denials so someone fixes the rule or the client.
Example: a bad deployment at rush hour
A team deploys a new version of a service under a slightly different principal name after a certificate change. Every one of its 300 instances now receives authorization errors and retries every second. A dashboard shows the sudden spike of denied requests for that one principal, the platform team sees the mismatch within minutes, and the fix is one line in the mapping rules rather than an hour of guessing.
Quotas complete the picture
Authorization decides what you may do. Quotas decide how much of it you may do. Kafka can limit bytes per second and request time per principal or per client id. In a shared platform, use both: ACLs prevent unauthorized use, and quotas prevent authorized users from squeezing out their neighbours.
A capacity checklist for the authorization layer
Rule count and growth
Chart the total number of rules per cluster every day. A sudden climb often means a misbehaving automation loop.
Broker start-up time
Record how long a broker takes to become ready after a restart. Watch for slow drift as rules pile up.
Authorization error rate
Graph errors per request type. Expect a low, steady background. Investigate spikes.
Handler thread idle time
If request handler threads are always busy, authorization is rarely the cause, but a storm of denials can add to the load.
Test at realistic size
A design that works with a hundred rules in a lab can behave very differently with a hundred thousand. Before rolling out a new approach, create a test cluster and load it with a realistic number of rules, topics and principals. Measure start-up time, memory use and the speed of bulk changes. Also rehearse the failure of one controller and of one Availability Zone, and confirm that applications keep working while the rules remain readable.
Publish an internal service level target for access changes, such as “approved requests take effect within ten minutes.” A clear target keeps the automation honest and gives the platform team something to measure.
14Security Hardening, Monitoring and Audit
ACLs are one lock on the door. Real security is the whole building.
The foundation
ACLs assume that the identity is real and the traffic cannot be tampered with. So first make sure every listener uses TLS encryption, every client authenticates and no plain, unauthenticated listener is reachable from ordinary networks. Also protect the paths used by brokers and controllers to talk to each other with their own dedicated principals, and give those principals only the permissions they need. Keep the list of super users as short as possible, ideally just the broker identities and a break-glass account, because super users bypass every rule.
The host part of an ACL looks attractive, but load balancers and network address translation often hide the real client address. Do not rely on it as a strong control. Use network segmentation and strong identities instead.
Watching for trouble
Brokers can log every authorization decision. Denied decisions are logged at a normal level and allowed ones at a very detailed level, so most companies keep denials on and turn allowed logging on only when investigating. Ship these logs to your central log platform. In addition, the standard request metrics count errors by type, so you can graph the rate of authorization errors per request kind and alert on sudden jumps.
| Signal | What it may mean | Suggested response |
|---|---|---|
| Sudden spike of denials from one principal | Bad deployment, wrong certificate mapping or missing rule | Notify the owning team and check the latest change |
| Denials across many topics from one principal | A compromised credential probing for data | Disable the credential and investigate |
| New wildcard or admin-level rule appears | Someone bypassed the pipeline | Drift alert and immediate review |
| Rules for principals that never connect | Retired application or unused access | Schedule for removal after owner confirmation |
| Failed logins from unknown sources | Password guessing or scanning | Block at the network and rotate exposed secrets |
Test permissions before production
Build automated tests that use a test principal to try both allowed and forbidden actions and confirm the results. Run them whenever the rules change. A forbidden action that suddenly succeeds is a security bug that a good test will catch before an attacker does.
Periodic access review
Each quarter, send every data owner the list of who can read their topics. Owners either confirm or revoke. Auditors love this evidence, and engineers learn to treat access as something that must be kept, not something granted once and forgotten.
Walkthrough: responding to a leaked credential
Suppose a developer accidentally commits a service account’s password to a public code repository. A prepared team follows a calm, practiced path instead of improvising.
Detect
A secret scanner or an alert on unusual login locations notices the exposed password, or denial logs show a principal probing unfamiliar topics.
Contain
The control plane disables or rotates that one credential. Because each application has its own identity, no other application is disturbed.
Measure the damage
The team reads the audit log for the principal and lists every topic it touched. Narrow rules mean a short list.
Recover
A new credential is issued and delivered by the secrets vault. The application restarts with it.
Learn
The team writes a short review and fixes the cause, for example by adding the secret scanner to the code pipeline.
This story shows why fine-grained design pays for itself. If the leaked identity had wildcard rights across all topics, step three would have taken weeks and required telling regulators about a wide exposure. With a tight prefix, the same incident affects a handful of topics and can be closed in an afternoon.
Connecting ACLs to compliance
Auditors rarely ask about Kafka itself. They ask about principles: who can see personal data, how do you know, and how fast can you remove someone? A good ACL design answers each question with evidence. The Git history proves who approved each access and when. The quarterly review records prove that data owners looked at their readers. The reconciliation reports prove that no rule exists outside the process. The denial logs prove that unauthorized attempts are noticed. When rules follow a naming convention, an auditor can even ask for “all readers of the payments domain” and receive an answer produced by a simple query instead of weeks of manual digging.
Keep in mind that laws and standards differ by country and industry, and that the exact obligations of your company should be confirmed with your security and legal teams. The engineering job is to make the facts easy to produce.
15Trade-offs, Patterns, Anti-patterns and Best Practices
Every access-control design trades safety, speed and simplicity against each other. Here is how to choose with open eyes.
Strengths and weaknesses of native Kafka ACLs
Strengths
- Enforced inside the broker, so no client can skip the check.
- Answered from memory, so the cost per request is tiny.
- Prefixed rules let one line cover whole families of resources.
- Works with every login method and every client language.
- Deny rules give a clear way to block sensitive resources.
Weaknesses
- No groups or roles, so large organizations must build that layer.
- Control stops at the topic, with nothing for fields or keys.
- Rule sets grow large without naming discipline.
- No built-in approval workflow, expiry or ownership metadata.
- Host restrictions are unreliable behind load balancers.
Patterns that work
Prefix per owner
Each team owns a name prefix for topics, groups and transactional ids and gets full control inside it.
Role templates
Roles such as producer, consumer and stream app are compiled into exact rules by tooling, never typed by hand.
Deny then allow
Block a sensitive prefix for everyone, then list the few explicit exceptions that come with approval.
Raw and safe pair
Restricted raw topic, sanitized derived topic for broad use.
Reconciliation loop
A scheduled job compares reviewed rules with live rules and reports or repairs drift.
Break-glass
Temporary, alarmed, expiring emergency access instead of shared admin credentials.
What people do
They give every application a wildcard rule for all topics “for now,” or they make a shared account a super user so developers stop asking for access.
Why it hurts
It turns the whole platform into one big trust zone. One leaked password exposes every topic, and audits cannot prove who accessed what.
What to do instead
Make requesting access cheap and fast through automation, so nobody is tempted to hand out shortcuts.
Common mistakes checklist
- Turning on the “allow everyone if no rule is found” setting and forgetting it.
- Writing prefixes without a trailing separator, so they match unintended names.
- Forgetting the group rule for consumers, or the transactional id rule for transactional producers.
- Letting several applications share one principal.
- Relying on host restrictions as a real security control.
- Using human accounts for production applications.
- Never deleting old rules and credentials.
- Skipping tests, so permission errors appear only in production.
Best practices in one place
Give every application its own service account. Agree on a naming convention before the first production topic exists. Use prefixed rules for ownership and narrow literal rules for sharing. Generate every rule from reviewed code. Keep super users to a minimum. Watch denials, review access regularly and plan for failure with quotas, retries and a break-glass path. If those seven habits are in place, most of the risk in a large Kafka platform disappears.
Which model should you choose?
| Situation | Recommended approach | Reason |
|---|---|---|
| One team, one cluster, a few dozen topics | Native ACLs with literal rules, small script | Simplest thing that works; do not over-engineer |
| Several teams sharing one cluster | Native ACLs, prefix per team, ACLs from a Git repository | Gives isolation and history with very little machinery |
| Dozens of teams on Kubernetes | Operator-managed users and rules | Rules live beside the application’s other configuration |
| Hundreds of teams, strict audit needs | Control plane with approvals, or vendor role-based access | Roles, approvals and reports are worth the extra component |
| Amazon MSK with existing AWS governance | IAM access control | Reuses roles, policies and AWS audit tooling |
| Very sensitive fields inside topics | Any of the above plus topic splitting, redaction or field encryption | ACLs alone cannot see inside a message |
A rule of thumb for every choice
Start simple, add layers only when a real pain appears, and always keep the rule count under control. A small, well-understood rule set with strong naming conventions beats an elaborate system that nobody dares to touch.
16Real-World Usage Patterns
The same ideas appear in very different companies, in different clothes.
The shared internal platform
Companies such as LinkedIn, where Kafka was born, and Uber run Kafka as a central service that many product teams use. At this size, hand-managed rules are impossible. The common approach is a platform team that owns the clusters, publishes naming rules, and offers self-service onboarding so product teams get their own area and request sharing with other teams through an approval workflow.
The streaming data pipeline
Netflix uses Kafka as a backbone of its event pipeline, where large numbers of producers and consumers move data between services and analytics. In pipelines like this, producers are typically allowed only to write to their intake topics, and downstream processors read those topics and write to their own outputs, so the direction of data flow is enforced by the rules themselves.
The regulated financial firm
A bank or payment company must prove who can see card and account data. It typically keeps raw sensitive topics in a tightly limited prefix, publishes sanitized topics for analytics, gives auditors the reviewed source of truth, and runs a quarterly access review where data owners confirm every reader.
The software-as-a-service company
A product that serves many customers on shared infrastructure often gives each customer a prefix or even a separate cluster. Prefixed rules keep one customer’s applications away from another’s topics, and quotas stop a noisy customer from harming the rest.
Managed Kafka in the cloud
Confluent Cloud uses service accounts, API keys and role or ACL bindings so customers never manage broker files. On AWS, Amazon MSK lets teams choose between Kafka ACLs with mTLS or SCRAM logins and IAM-based access control with AWS roles. Teams already deep in AWS often prefer IAM because roles, groups and central auditing already exist, while teams that want portability to other Kafka platforms prefer plain ACLs.
flowchart TB
subgraph TEAMS["Product teams"]
T1["Payments team"]
T2["Logistics team"]
T3["Fraud team"]
end
T1 --> REQ["Self-service access requests"]
T2 --> REQ
T3 --> REQ
REQ --> GATE["Owner approval and policy checks"]
GATE --> CTRL["ACL control plane"]
CTRL --> KC["Shared Kafka cluster with prefix ownership"]
KC --> MON["Denial alerts and quarterly review"]
MON --> TEAMS
Lessons that repeat in every company
Looking across these stories, the same lessons appear again and again. Successful platforms make access requests fast, because slow processes create shortcuts. They put ownership in names, because names cost nothing and scale forever. They give the cluster one automated writer of rules, so the list is always explainable. And they treat old access as debt, cleaning it up on a schedule. Failing platforms do the opposite: they rely on tickets, they let names drift and they keep rules “just in case.”
The gap between the two groups is rarely technology. It is discipline. Every tool in this guide is available to every team, and the difference is whether the team decided early that permissions are an engineered system rather than a pile of exceptions.
17Frequently Asked Questions
18Summary and Key Takeaways
What to remember
- An ACL is one sentence: a principal is allowed or denied an operation on a resource, and the order of checking is super user, then Deny, then Allow, then default refusal.
- Identity comes first: give every application its own service account, and shorten certificate names with mapping rules so principals stay stable.
- Prefixed rules and naming conventions are the scaling secret: end every prefix with a separator so it matches only what you intend.
- Do not write rules by hand: compile role templates into ACLs from reviewed code, apply them in bulk and reconcile continuously to catch drift.
- Kafka ACLs stop at the topic: for finer control, split topics, publish redacted copies, encrypt fields or use a proxy layer.
- Authorization is local and fast: brokers check rules in memory, so design for fewer rules, healthy controller quorums and a break-glass path.
- Watch and review: alert on denial spikes, keep super users minimal, add quotas and ask data owners to re-confirm access regularly.