A short refresher on the things that permissions are attached to.

Apache Kafka is a platform for storing and moving streams of events. An event is a small record of something that happened, such as an order being placed or a temperature being measured. Kafka stores these events in an ordered log so that many applications can read them, now or later.

Why security matters more in Kafka than it first appears

Kafka is often the central nervous system of a company. Orders, payments, customer profiles, device readings, and internal alerts all pass through it. That makes it a very attractive target, and also a very easy place for accidents. A developer who mistakenly points a test program at the production cluster could publish garbage into a critical topic. A curious insider could read personal data they have no business seeing. A misbehaving script could delete a topic that took months to fill.

A cluster without authorization is fully open

By default, a fresh Kafka cluster has no authorizer switched on. That means every client that manages to connect can do absolutely everything. Access control is something you must deliberately turn on and configure, and this tutorial explains exactly what that involves.

Authorization is only one part of a bigger picture, and it depends on the other parts to work.

flowchart LR
  A[Client connects] --> B[Encrypted channel with TLS]
  B --> C[Authentication proves identity]
  C --> D[Authorization checks each request]
  D --> E[Allowed request is processed]
  D --> F[Denied request is rejected]
        

Why the order matters

Authorization can only work on an identity that it trusts. If a client can pretend to be anyone, then giving that identity permissions is meaningless, because an attacker will simply claim to be the most powerful user. Therefore authentication must come first, and it must be strong. Likewise, if traffic is not encrypted, someone on the network could steal login details or read data directly, making the permission checks pointless.

The idea in one sentence, followed by what it really means in practice.

Authorization in Kafka is the process by which a broker decides, for every single request it receives, whether the client that sent the request has permission to perform the requested action on the requested resource. The broker checks a set of rules called Access Control Lists, or ACLs. If a matching rule permits the action, the request proceeds. If not, the broker returns an authorization error and does nothing.

What kinds of decisions does it make?

Authorization is checked on every request

Kafka does not check permissions once when a client connects and then trust it forever. Each request, such as a produce request, a fetch request, or a request to create a topic, is checked as it arrives. This means a change to the ACLs takes effect quickly for clients that are already connected, without forcing them to reconnect.

Every ACL rule begins with the question of who it applies to.

In Kafka, the identity of an authenticated client is called a principal. A principal is written with a type and a name, for example “User:alice” or “User:payment-service”. Almost all principals in Apache Kafka are of the type User, even when they represent an application rather than a human being.

Where the principal name comes from

The principal name is produced by the authentication step, and it depends on which method the client used.

Mapping names into something readable

A certificate’s full distinguished name may look like a long string containing a common name, an organisational unit, and a country. Writing ACLs against such strings is error-prone and ugly. Kafka lets administrators define mapping rules that extract just the part that matters, such as the common name, so that ACLs can refer to a short, stable name like “User:orders-service”. Getting this mapping right is one of the most important early steps, because every ACL depends on it.

Every ACL is a small sentence with a fixed structure. Once you can read the sentence, you can read any ACL.

An ACL can be expressed as one plain English sentence: “Principal P is allowed or denied to perform Operation O on Resource R from Host H.” Each of those parts has a name, and together they form the rule.

Reading a few example sentences

Kafka lets you protect several kinds of things, and each has its own personality.

Why groups deserve their own protection

Beginners often focus only on topics and forget consumer groups. A consumer needs read permission on the topic to fetch data, and read permission on its group to take part in the group and record its progress. Without the group permission, the consumer can be allowed to read data but still fails when it tries to join or commit. Groups also matter for safety: without group ACLs, a rogue client could join someone else’s group, steal partitions, and disrupt their processing.

Why the cluster resource is powerful

The cluster resource is a single, special resource that represents the cluster as a whole. Permissions here are very strong. For example, the ability to alter the cluster resource includes the ability to create and remove ACLs, which means it is effectively the power to change all the other rules. Such permissions should be given to very few identities.

An operation names the action being requested. Different resources accept different operations.

Implied permissions

Some operations automatically include others, which saves you from writing many rules. Read, write, delete, and alter each imply describe, because you cannot meaningfully read or write something you are not even allowed to see exist. Alter implies describe, and altering configs implies describing configs. Knowing these shortcuts keeps rule sets small and readable.

flowchart LR
  R[Read] --> D[Describe]
  W[Write] --> D
  DEL[Delete] --> D
  A[Alter] --> D
  AC[AlterConfigs] --> DC[DescribeConfigs]
        

You rarely want to write one rule per topic. Patterns let a single rule cover many resources safely.

Why prefixed patterns are a superpower

Imagine a team that owns dozens of topics, all named with the team’s prefix. Without prefixed rules, every new topic needs new ACLs. With a prefixed rule, one line grants the team the right to manage everything under their naming umbrella, present and future. This turns topic naming conventions into a security tool, which is why mature organisations agree on naming standards long before they write their first ACL.

Wildcards: handle with care

A wildcard resource rule applies to everything of that type, including topics that will be created in the future. Wildcards are often used for legitimate purposes, such as giving a platform administrator the ability to describe all topics, or for granting a monitoring tool read access to metadata. But an allow rule with a wildcard resource and a wildcard principal effectively switches security off for that operation, so such rules deserve a review and a written justification.

When a request arrives, the authorizer follows a clear, predictable procedure.

flowchart TD
  A[Request arrives at the broker] --> B{Is the principal a super user}
  B -->|Yes| C[Allow]
  B -->|No| D{Does any matching deny rule exist}
  D -->|Yes| E[Deny]
  D -->|No| F{Does any matching allow rule exist}
  F -->|Yes| C
  F -->|No| G[Deny by default]
        

The setting that flips the default

There is a broker setting, allow.everyone.if.no.acl.found, which changes step four. When it is true, a resource that has no ACLs at all is open to everybody. It exists mainly to ease migrations from an open cluster, and it is dangerous in production because any new topic, which starts with no ACLs, is instantly world-accessible. Leave it false, which is the default, in any environment that matters.

Using deny rules wisely

Because deny always wins, a deny rule is a powerful tool but also a source of confusion. A broad deny can silently cancel a carefully written allow. Most well-run clusters rely mainly on the default deny together with precise allow rules, and reserve explicit deny rules for rare exceptions, such as blocking one specific account from one specific sensitive topic.

The component that actually enforces the rules is called the authorizer, and it is pluggable.

Kafka does not hard-wire a single way of making authorization decisions. It defines an interface, and the broker configuration names the implementation to use through the setting authorizer.class.name. If this setting is empty, no authorization happens at all. If it names an implementation, the broker consults it for every request.

Where the rules live

In older ZooKeeper-based clusters, ACLs are stored in ZooKeeper, which means ZooKeeper itself must be secured, or anyone with access to it could rewrite the rules. In newer KRaft-based clusters, ACLs live in the cluster’s own metadata log, and ZooKeeper is no longer part of the picture. Recent Kafka versions run only in the KRaft style, so new deployments use the standard authorizer.

flowchart LR
  A[Administrator creates an ACL] --> B[Metadata store]
  B --> C[Broker 1 cache]
  B --> D[Broker 2 cache]
  B --> E[Broker 3 cache]
  F[Client request] --> D
  D --> G{Check cached rules}
        

Fast decisions

Because every request needs a check, the rules are held in memory on each broker, so decisions take a tiny amount of time. When an administrator changes an ACL, the change propagates to all brokers within moments. Very large numbers of ACLs and very complex patterns can add overhead, which is another reason to prefer prefixed rules over thousands of individual ones.

The most practical question is: which ACLs does my application actually need?

The classic surprise: consumer groups

New administrators frequently grant a consumer read on a topic and then find that the consumer still fails. The missing piece is almost always the read permission on the consumer group. The two permissions protect different things: the topic protects the data, and the group protects the coordination and progress tracking around it.

Kafka’s own servers talk to each other, and that traffic must be permitted too.

Brokers replicate data between themselves, controllers manage cluster metadata, and each of these actions is a request that passes through the authorizer, just like a client request. The brokers authenticate as a principal of their own, and that principal needs permission for the cluster-level actions used in replication and coordination.

Super users

The simplest and most common approach is to list the brokers’ principals as super users in the configuration. Super users bypass all ACL checks, so replication never fails because someone forgot a rule. The danger is that super users are all-powerful, so the list must be short and carefully protected.

Keeping the broker identities safe

Because broker principals are so powerful, protect their credentials. Use dedicated certificates or logins that no application ever receives, restrict who can read the files holding them, rotate them on a schedule, and monitor for use of those identities from unexpected addresses.

Good access control comes from a plan, not from a pile of urgent fixes.

flowchart TD
  A[Platform team] -->|manages ACLs and cluster| K[Kafka cluster]
  B[Team Sales] -->|owns and writes sales prefixed topics| K
  C[Team Billing] -->|owns and writes billing prefixed topics| K
  C -->|read only on selected sales topics| K
  D[Analytics team] -->|read only on approved topics| K
        

Least privilege in practice

The principle of least privilege says every identity should hold only the permissions it truly needs, and no more. In Kafka this means a producer receives write on its topic but not read, a consumer receives read but not write, an application that never creates topics does not receive create, and no application receives permission to change ACLs or configuration.

Typing rules by hand works for a demo. It does not work for hundreds of applications.

The command-line tool

Kafka ships with an ACL management tool that can add, remove, and list rules. It is ideal for learning, experiments, and emergency inspection. It connects to the cluster as an administrative principal, and it needs permission to alter the cluster resource to change rules and to describe it to list them.

Why manual management breaks down

Managing ACLs as code

Mature teams describe their desired ACLs in files kept in version control, and apply them through automated pipelines. Popular approaches include infrastructure-as-code tools, Kubernetes operators that treat users and their permissions as declarative objects, and dedicated topology management tools. The benefits are the same as for any code: every change is reviewed, history is preserved, environments stay consistent, and removing an application removes its permissions in the same change.

An ACL can ask not only who you are, but also where you are connecting from.

Each ACL includes a host field, which is the network address that the request must come from. The default and most common value is a wildcard meaning any address. But you can restrict a rule so that it applies only from a specific address, for example allowing an administrative account to change configuration only from a management network.

Limits of host rules

The host in an ACL is the address as seen by the broker. If clients connect through load balancers, network address translation, or proxies, the broker may see the address of the intermediary rather than the real client, making host rules meaningless or accidentally too broad. Addresses can also change in dynamic environments such as containers, where a fixed list of addresses is impractical.

Permissions are only trustworthy if you can see how they are being used.

The authorizer writes log entries about its decisions through a dedicated logger. Denied requests are recorded at a normal informational level, so they usually appear by default. Allowed requests are recorded at a more detailed level, and turning that on can produce a very large volume of logs, so it is usually enabled only for investigations or with careful sizing.

What a log entry tells you

Using logs for security, not just debugging

Ship authorizer logs to a central log system and set alerts for patterns worth noticing. A burst of denials from one principal can mean a misconfigured application or an attacker probing for weaknesses. Requests from a super user identity at unusual times or from unexpected addresses deserve immediate investigation. Changes to ACLs themselves should always generate an alert and an entry in the change record.

Authorization failures show up as specific error types. Recognising them speeds up diagnosis enormously.

A systematic checklist

ACLs are simple and dependable, but they are not the answer to every access control need.

Options when ACLs are not enough

ACLs work best as part of a wider defensive setup.

Auto topic creation and ACLs

If automatic topic creation is enabled, a client that is allowed to create topics can produce new resources simply by mentioning a name. Combined with permissive rules, this can lead to topics appearing with unexpected names and no owner. Many organisations turn off automatic creation and create topics through a controlled process, together with their ACLs, so that every topic has a known owner from birth.

A few habits keep access control healthy, and a few habits quietly undermine it.

Helpful patterns

Anti-patterns to avoid

Here is how ACL thinking applies across different kinds of organisations.

Let us follow a realistic incident from the first complaint to a lasting fix.

A company deploys a new reporting service that reads from the “orders” topic. Immediately after launch, the service logs an authorization error and processes nothing. The developers insist the permissions were requested and granted.

The lesson is that authorization errors are precise messages. Reading them carefully, confirming the identity, and checking the implied resources such as groups solves most incidents in minutes, without ever needing to loosen security.

Authentication proves who a client is, for example with a certificate or a login. Authorization decides what that authenticated client may do, using ACLs. Authorization depends on authentication being reliable.

An Access Control List entry is a rule stating that a principal is allowed or denied to perform an operation on a resource, optionally from a given host. The broker checks these rules for every request.

No authorization is performed, and every client that can connect may do everything, including reading, writing, and deleting any topic. Access control must be enabled deliberately.

Read permission on the topic and read permission on its consumer group. Missing the group permission is one of the most common causes of consumer authorization errors.

Write permission on the topic. It also needs create permission if it is expected to create the topic, and transactional producers need permissions on their transactional identity. Older versions also required a cluster permission for idempotent writes.

Deny wins. If any matching rule denies the request, it is refused regardless of allow rules.

A principal listed in the broker configuration that bypasses all ACL checks. It is typically used for the brokers themselves and should be extremely limited.

A literal ACL matches exactly one named resource. A prefixed ACL matches every resource whose name begins with a given text, which makes it ideal for team or application naming conventions.

Not natively. ACLs attach to individual principals. Some platforms add role-based access control, and external authorizers can provide richer models. Otherwise, manage groupings in your identity system or automation.

No. Changes propagate to brokers quickly and apply to subsequent requests, including those from already connected clients.

No. ACLs control access at the level of topics, groups, and other resources. To protect sensitive fields, separate them into different topics or protect them with encryption or masking.

Common reasons are a principal name that differs from what the broker sees, a wrong resource name or pattern type, a missing related permission such as the consumer group, a host restriction that does not match, or a deny rule that overrides your allow.

Not in production. It makes every resource without ACLs open to everyone, including every newly created topic. It is sometimes used temporarily while migrating an open cluster, and should be switched off as soon as ACLs are in place.

Treat them as code. Keep desired rules in version control, review changes, and apply them with automation such as infrastructure-as-code tools or operators, so that history, consistency, and clean-up are built in.

Follow least privilege with default deny: give each application its own identity and only the exact permissions it needs, and manage those permissions in a reviewed, automated, and audited way.

Authorization in Kafka answers the question of what an authenticated client may do. It is implemented with Access Control Lists, each of which states that a principal is allowed or denied to perform an operation on a resource from a host. The broker checks these rules on every request, follows a clear decision path in which super users pass, denies win, an allow is required, and everything else is refused, and keeps the rules cached for speed.

Strong results come from combining ACLs with encryption and real authentication, giving every application its own identity, using naming conventions with prefixed rules, granting only the operations each client needs, remembering the implied resources such as consumer groups and internal topics, keeping super users to the brokers alone, managing rules as reviewed code, and watching the authorizer logs. Understand the limits too: ACLs do not provide roles and cannot protect individual fields inside a topic.