Before learning any mechanism, it helps to see what an unprotected Kafka cluster allows.

A quick Kafka refresher

Apache Kafka stores streams of events in topics. Producers write events, consumers read them, and brokers, the servers of the Kafka cluster, store and serve them. Around the brokers live many other programs: stream processors, connectors that move data in and out of databases, monitoring tools and administration tools. All of them connect to the brokers over the network.

What happens with no security

A freshly installed Kafka cluster with default settings accepts any connection from anyone who can reach it over the network. There is no question about who the client is. A curious or careless person can read any topic, write fake events to any topic, delete topics, or change settings. A misconfigured test program can accidentally flood production topics. There is also no way to tell later who did what, because everyone looks the same.

People often blur these words together. Kafka treats them as three different layers, each solving a different problem.

flowchart LR
  C["Client connects"] --> E["Encryption: is the channel private?"]
  E --> A["Authentication: who are you?"]
  A --> Z["Authorization: what may you do?"]
  Z --> D["Request allowed or denied"]
        

Where SASL fits

SASL belongs to the first of the three layers. It is a way of proving identity. It does not decide permissions, which are handled by access control lists, and by itself it does not encrypt the traffic, which is handled by TLS. A secure setup normally uses all three together, and this tutorial focuses on the first.

SASL is not a single login method. It is a framework that lets many login methods plug into a protocol.

The name

SASL stands for Simple Authentication and Security Layer. It is an internet standard that separates the question “how does this application protocol carry authentication messages?” from the question “which method is used to prove identity?” A protocol that supports SASL, such as email or directory services, can offer many methods without redesigning itself each time.

How Kafka uses it

Kafka adopted SASL so that brokers can support several authentication methods through one common pattern. Each method is called a mechanism. Kafka defines the steps by which a client and a broker agree on a mechanism and then exchange a series of messages that belong to that mechanism. Adding a new mechanism does not change how the rest of Kafka works.

Mechanisms Kafka ships with

Before authentication can happen, Kafka must know which doors the broker has and how each door is protected.

Listeners

A listener is a network address and port on which a broker accepts connections. A broker can have several listeners at once, for example one for applications inside the company network and another for outside partners. Each listener has a name and is given its own security protocol, so different doors can have different levels of protection.

The four security protocols

Why the last one matters

SASL_PLAINTEXT proves identity but sends everything else, including your event data, unencrypted. Some mechanisms also expose the secret itself in that case. SASL_SSL wraps the entire conversation in TLS encryption first, then runs the SASL identity check inside it. Almost every production setup uses SASL_SSL.

Every SASL login in Kafka follows the same short conversation before any events can flow.

sequenceDiagram
  participant C as Client
  participant B as Broker
  C->>B: Open connection to the listener
  C->>B: Ask which API versions are supported
  B-->>C: Supported versions
  C->>B: Handshake: I want to use mechanism X
  B-->>C: Mechanism X is enabled, continue
  C->>B: Mechanism-specific message 1
  B-->>C: Mechanism-specific reply
  C->>B: Further messages until finished
  B-->>C: Authentication successful
  Note over C,B: Normal Kafka requests can now flow
        

Authentication produces one valuable result: a name the broker can use for permissions and for logs.

What a principal is

After successful authentication, the broker attaches a principal to the connection. A principal is simply the identity name of the client, such as a user name or a service name. Every later request on that connection is tagged with the principal. This is how the authorization layer knows whose permissions to check, and how logs can record who did what.

Where the name comes from

The principal depends on the mechanism. For password mechanisms it is the user name. For Kerberos it is derived from the Kerberos principal, often shortened by mapping rules that strip the realm and host parts. For token mechanisms it is taken from a claim inside the token, such as the subject. Administrators should know exactly which name each mechanism yields, because permissions are written against these names.

flowchart LR
  A["Successful SASL login"] --> P["Principal, for example User:payments-app"]
  P --> ACL["Access control rules match the principal"]
  P --> LOG["Logs and metrics record the principal"]
  P --> Q["Quotas can be applied per principal"]
        

The simplest mechanism sends a username and password, and everything depends on the channel being private.

How it works

The client sends its username and password to the broker in a single message. The broker checks them against its list of known users and replies with success or failure. There are no extra steps. This simplicity makes PLAIN the easiest mechanism to understand and the easiest to test.

Where the passwords live

By default, Kafka’s PLAIN support reads the list of valid users and passwords from the broker’s login configuration, which is a text file or a configuration property. That means adding or changing a user requires editing that configuration, and often restarting or reloading brokers. In real deployments, teams usually replace this default with a custom check that consults a central store, such as a directory service or a secrets database, so passwords are not kept in plain text in files.

sequenceDiagram
  participant C as Client
  participant T as TLS channel
  participant B as Broker
  C->>T: Encrypted tunnel established
  C->>B: Username and password, inside the tunnel
  B->>B: Compare with stored credentials
  B-->>C: Success or failure
        

SCRAM lets the client prove it knows the password without ever sending the password.

Challenge and response

SCRAM stands for Salted Challenge Response Authentication Mechanism. Instead of sending the password, the client and broker exchange random values. The broker sends a challenge, and the client answers with a calculation that only someone who knows the password could perform. The broker checks the answer using stored verification data. Kafka also lets the client verify that the broker knows the right data, so both sides prove themselves to each other.

What the broker stores

The broker does not store the password. It stores a salted, transformed version that lets it check answers but does not reveal the original password. A salt is a random value mixed in so that two users with the same password have different stored data, which defeats pre-made lookup tables. Kafka supports two strengths, named after the hashing methods SHA-256 and SHA-512.

sequenceDiagram
  participant C as Client
  participant B as Broker
  C->>B: Hello, my user name and a random value
  B-->>C: Challenge: salt, iteration count, my random value
  C->>C: Compute proof using the password
  C->>B: Proof of knowing the password
  B->>B: Verify proof against stored data
  B-->>C: Success and broker's own proof
  C->>C: Verify broker's proof
        

Managing SCRAM users

SCRAM credentials are stored inside Kafka’s own cluster metadata and are created and changed with the standard administration tools. Adding, changing or removing a user does not require restarting brokers. This makes day-to-day user management much easier than with file-based PLAIN. Support for managing SCRAM users when the cluster uses Kafka’s own controller quorum instead of an external coordination service arrived in recent versions, so check the version you run.

Kerberos gives an organisation one central place for logins, and Kafka can join in as one of the services.

The cast of characters

Kerberos is a ticket-based authentication system used widely in large enterprises. It has a central server called the key distribution center, which knows every user and service in the organisation. Each user or service has a principal name in a realm, which is the administrative domain. Services such as Kafka brokers have keytab files, which hold their long-term secrets so they can run without a human typing a password.

How a login works

A client first proves its identity to the key distribution center and receives a ticket-granting ticket, a kind of master pass. When it wants to talk to Kafka, it shows that master pass to the center and asks for a service ticket for the Kafka broker. It presents this service ticket to the broker. The broker can verify the ticket using its own secret, without calling the center each time. No password crosses the network to Kafka.

sequenceDiagram
  participant C as Client
  participant K as Key distribution center
  participant B as Kafka broker
  C->>K: Prove identity, ask for master ticket
  K-->>C: Ticket-granting ticket
  C->>K: Show master ticket, ask for Kafka service ticket
  K-->>C: Service ticket for the Kafka broker
  C->>B: Present service ticket
  B->>B: Verify ticket with own keytab
  B-->>C: Authenticated
        

Details that trip teams up

OAUTHBEARER lets clients authenticate with short-lived tokens from a central identity provider.

The token idea

In token-based authentication, a client first obtains a token from an identity provider, a trusted service that knows who is who. A token is a small signed piece of data that says who the holder is and how long the token is valid. The client presents the token to the Kafka broker. The broker checks the signature and the expiry, and if they are good, accepts the client’s identity from the token.

Why organisations like it

Tokens expire quickly, often within minutes or hours, so a stolen token is only briefly useful. The identity provider is a single place to control who may get tokens, and it often already supports login for humans and services across the whole company. Applications running in containers or cloud platforms can obtain tokens from the platform’s own identity system without any long-lived secret stored in files.

sequenceDiagram
  participant C as Client
  participant I as Identity provider
  participant B as Kafka broker
  C->>I: Prove identity, request token
  I-->>C: Signed short-lived token
  C->>B: Present token
  B->>B: Check signature, expiry and claims
  B-->>C: Authenticated as the token's subject
  Note over C,I: Client refreshes the token before it expires
        

Built-in and custom support

Kafka defines the OAUTHBEARER mechanism and, for a long time, shipped only a basic unsecured implementation meant for testing. Recent versions include built-in support for standard token flows, where the client fetches tokens from an identity provider and the broker validates them against the provider’s published signing keys. Many organisations also use vendor-supplied or custom handlers. Whichever route you choose, confirm that your version and client libraries support it, and never use the unsecured testing implementation in production.

SASL is not the only way to identify a Kafka client, and some environments add their own twists.

Mutual TLS with client certificates

Instead of SASL, a client can prove identity using a TLS certificate. The broker is set to request a client certificate, checks that it was signed by a trusted authority, and takes the principal from the certificate’s subject name. This is called mutual TLS because both sides present certificates. It needs a public key infrastructure to issue, distribute and renew certificates, which can be a large task, but it combines encryption and identity in one step.

Delegation tokens

Delegation tokens are a lightweight feature that lets an already authenticated user obtain a temporary token, which other processes can use to authenticate with the same rights. They rely on the SCRAM mechanism. They are useful for distributed processing frameworks that start many worker processes, since the workers do not need access to the original user’s long-term credentials.

Cloud and vendor mechanisms

Managed Kafka services often add their own SASL mechanisms so that clients can authenticate with the cloud platform’s own identity system. One well-known example is a mechanism that lets applications use their cloud identity and access management roles instead of stored passwords. These mechanisms follow the same SASL pattern, but are supported only on the platform that provides them and need matching client libraries.

The right mechanism depends on what your organisation already has and how much complexity it can carry.

flowchart TD
  S["Which mechanism?"] --> Q1{"Already have Kerberos across the company?"}
  Q1 -- "Yes" --> G["GSSAPI"]
  Q1 -- "No" --> Q2{"Central identity provider issuing tokens?"}
  Q2 -- "Yes" --> O["OAUTHBEARER"]
  Q2 -- "No" --> Q3{"Strong certificate infrastructure?"}
  Q3 -- "Yes" --> M["Consider mutual TLS"]
  Q3 -- "No" --> Q4{"Want to avoid sending passwords and extra systems?"}
  Q4 -- "Yes" --> SC["SCRAM"]
  Q4 -- "No, quick test only" --> PL["PLAIN with TLS"]
        

Questions to ask before choosing

You do not need to memorise settings to understand where each piece of authentication configuration lives.

JAAS in one paragraph

Kafka’s Java components use a standard called JAAS, the Java Authentication and Authorization Service, to describe how a login should be performed. A JAAS configuration names a login module, which is the code that performs the mechanism, and gives it options such as the user name and password or the location of a keytab. Kafka lets you supply this either as a separate file or as a property inside its usual configuration, and the property form is usually preferred because it keeps everything in one place and allows different settings per listener.

What the broker decides

What the client decides

Securing only client connections leaves the internal traffic between servers wide open. Kafka lets you protect that too.

Inter-broker traffic

Brokers constantly talk to each other, most importantly to replicate data from partition leaders to followers. Those connections pass through a listener, and the cluster is told which security protocol and which SASL mechanism to use for them. Each broker therefore needs its own credentials as a client of the other brokers, and the other brokers must recognise that identity and treat it as trusted for replication.

Controller communication

Kafka clusters have controllers that manage metadata such as topics and partition leadership. In clusters using Kafka’s own controller quorum, the controllers and brokers also communicate over listeners that can be secured with the same kinds of protocols and mechanisms. Older clusters that rely on an external coordination service need that service protected separately.

flowchart LR
  APP["Applications"] -->|"Client listener, SASL_SSL"| B1["Broker 1"]
  APP -->|"Client listener, SASL_SSL"| B2["Broker 2"]
  B1 |"Inter-broker listener, authenticated"| B2
  B1 |"Controller listener, authenticated"| CT["Controllers"]
  B2 |"Controller listener, authenticated"| CT
        

Permissions for the brokers themselves

Because brokers act on each other’s behalf, their principals need special rights in the access control rules, for example the right to replicate data and manage cluster metadata. When enabling authorization, remember to grant these rights to the broker identities before turning on enforcement, otherwise the cluster may stop replicating and become unstable.

Getting authentication working is the easy half. Keeping credentials safe and fresh over months and years is the harder half.

Where secrets should live

Passwords and keys should not sit in plain text inside configuration files that are stored in shared repositories or copied between machines. Kafka supports external configuration providers that read secrets from protected files or directories at start-up, and many organisations connect these to a dedicated secrets manager. Dynamic broker settings that hold passwords can also be stored encrypted. The goal is that a copy of a configuration file alone does not reveal any secret.

Rotation without downtime

Credentials should be changed regularly and immediately after any suspected leak. For password mechanisms, a safe pattern is to create a new credential, roll it out to clients gradually, confirm nobody uses the old one, and then remove the old one. Mechanisms that support several credentials for the same user, or that use short-lived tokens, make this smoother. Rotation that requires changing everything at the same instant is risky.

Long-lived connections and re-authentication

As noted earlier, authentication happens when a connection is opened, and Kafka connections often last for days. Without further action, a client whose credential has been revoked, or whose token has expired, might carry on using the old connection. Kafka supports forcing re-authentication after a set interval, so the broker requires clients on long-lived connections to prove themselves again. Clients that can refresh credentials, such as those using tokens, then continue smoothly, and clients whose credentials were removed are disconnected at the next check.

flowchart LR
  A["Client authenticates"] --> B["Connection stays open"]
  B --> C{"Re-authentication interval reached?"}
  C -- "No" --> B
  C -- "Yes" --> D["Broker asks client to authenticate again"]
  D --> E{"Credentials still valid?"}
  E -- "Yes" --> B
  E -- "No" --> F["Connection closed"]
        

A single cluster can serve different groups with different doors, each with its own rules.

flowchart TB
  subgraph BR["Kafka broker"]
    L1["Internal listener: SASL_SSL with SCRAM"]
    L2["External listener: SASL_SSL with OAUTHBEARER"]
    L3["Inter-broker listener: SASL_SSL with SCRAM"]
  end
  A["Internal applications"] --> L1
  P["Partner and cloud applications"] --> L2
  O["Other brokers"] --> L3
        

Why mix them

Most teams add authentication to a cluster that is already serving traffic. Doing it safely means changing things in the right order.

flowchart TD
  A["Stage 1: Inventory clients and their libraries"] --> B["Stage 2: Add a new SASL_SSL listener beside the existing one"]
  B --> C["Stage 3: Create an identity for each application"]
  C --> D["Stage 4: Migrate clients one by one to the new listener"]
  D --> E["Stage 5: Move inter-broker traffic to the secured listener"]
  E --> F["Stage 6: Remove the old open listener"]
  F --> G["Stage 7: Turn on authorization and monitoring"]
        

Kafka’s brokers are only the centre. Every companion program also needs to authenticate, and each has its own place for settings.

Authentication is one layer in a stack. These neighbouring practices decide how much protection it actually gives.

If nobody watches the door, nobody notices when it is being tested. A few signals give good visibility.

Alerts worth having

Alert on a sudden rise in failed authentications, on a client that fails repeatedly from a new address, on credentials approaching their expiry date, on new principals appearing that nobody expected, and on any weakening of security settings such as enabling a new mechanism or a plain listener.

Audit logs

Keep logs that record which principal connected from where, and which administrative actions were taken. Send them to a central logging system with restricted access, so that they survive incidents and cannot be quietly edited by the people they describe.

Most authentication problems fall into a small number of familiar patterns.

A calm method for diagnosis

Habits that keep authentication healthy, and traps that quietly weaken it.

Patterns worth adopting

Anti-patterns to avoid

A checklist to keep beside you when designing or reviewing Kafka authentication.

Five common situations show how the choices fit together in practice.

No. SASL is a framework. It defines how authentication messages are exchanged, and specific mechanisms such as PLAIN, SCRAM, GSSAPI and OAUTHBEARER do the actual proving of identity.

Not by itself in Kafka. Encryption comes from TLS. That is why the combination SASL_SSL is the common choice, since it gives both encryption and identity checking.

Yes, with TLS, and often with credentials checked against an external store. It is simple and widely supported. Without TLS, it exposes passwords and should not be used across untrusted networks.

PLAIN sends the password to the broker. SCRAM proves knowledge of the password through a challenge and response calculation and stores only salted verification data. SCRAM is generally the better password-based choice.

Both work. Choose mutual TLS if you already have strong certificate management. Choose SASL if you prefer passwords, tickets or tokens tied to an existing identity system. They can also be combined, with TLS for the channel and SASL for identity.

It is the identity the broker assigns to a connection after successful authentication. Permissions, quotas and logs are all based on it.

Yes. Several mechanisms can be enabled on the same listener, and each client chooses one. This is useful during migrations and for mixed client populations.

The existing connection may continue until it closes or is forced to re-authenticate. Setting a re-authentication interval makes removals take effect within a bounded time.

If you want the internal traffic protected, yes. Configure a secured listener and a mechanism for inter-broker communication, and give the broker identities the internal rights they need.

A common cause is a protocol mismatch, such as a SASL client connecting to a listener that expects plain traffic or the reverse. Check the security protocol and the port.

Use external configuration providers or a secrets manager so that configuration files hold only references. Restrict file permissions, avoid committing secrets to repositories, and rotate them regularly.

No. Authentication only establishes who the client is. Authorization, through access control rules, decides which actions that identity may perform.

An unprotected Kafka cluster accepts connections from anyone, so authentication is the first gate of a secure platform. SASL is the framework Kafka uses to let clients prove who they are, using mechanisms such as PLAIN, SCRAM, GSSAPI with Kerberos and OAUTHBEARER. Each connection performs a short handshake, and on success the broker records a principal that drives permissions, quotas and audit logs. Good designs pair SASL with TLS, follow it with authorization, give every application its own identity and handle credentials with care.

Success depends as much on operations as on settings: staged rollouts, secret storage, rotation, re-authentication, monitoring and quick troubleshooting. If you remember one sentence, let it be this: SASL tells Kafka who is knocking, but only encryption, permissions and disciplined credential habits make the door truly secure.