Before learning the solution, let us understand exactly what can go wrong when Kafka traffic is not protected.

Kafka in one paragraph

Apache Kafka moves streams of messages between applications. Applications that write messages are producers, applications that read them are consumers, and the Kafka servers that store and deliver messages are brokers. Brokers also talk to each other constantly to copy data, and to the management part of the cluster. All of this talking happens over network connections, and that is where the risk lives.

What an attacker on the network can do

Why this matters more than you might think

Many teams believe that “our network is private, so we are safe.” Private networks are still full of servers, contractors, monitoring tools and cloud components. One compromised machine inside the network is enough to see everything. Rules and laws about protecting personal and financial data also often require encryption whenever data moves across a network, even an internal one. Encrypting Kafka traffic is therefore both a technical and a compliance need.

TLS is the standard technology that protects data moving between two computers. You use it every day when you visit a website whose address starts with https.

A short history and a naming quirk

TLS stands for Transport Layer Security. It is the modern successor of an older technology called SSL, or Secure Sockets Layer. The old SSL versions are now considered unsafe and should not be used. Yet the old name stuck around in many places. In Kafka, the settings are still named with “ssl” in front, and the connection type is still called SSL, even though what is really used today is TLS. When you see “SSL” in Kafka, read it as “TLS.”

The three promises of TLS

What TLS does not do

TLS protects data only while it moves between two ends. It does not protect data once it arrives and is written to a disk. It also does not decide what a user is allowed to do. Those are separate jobs: encryption at rest and access control. We will return to this difference later, because mixing them up is a common source of false confidence.

TLS rests on a few ideas. Once you understand them, every setting in Kafka becomes easy to place.

The chain of trust

Certificates are usually issued in chains. A root authority signs the certificate of an intermediate authority, and the intermediate signs the certificate of your broker. When a client receives a broker’s certificate, it follows the chain upward until it reaches a root that it already trusts. If the chain leads to a trusted root and every signature and date checks out, the client accepts the certificate.

flowchart TD
    R[Root Certificate Authority - trusted by the client] --> I[Intermediate Certificate Authority]
    I --> B[Broker certificate]
    B --> V{Client checks: signature, dates, name}
    V -- All good --> T[Trusted connection]
    V -- Any failure --> X[Connection rejected]
        

Self-signed versus properly issued certificates

Kafka needs a place to keep its own identity and a place to keep the list of identities it trusts. These are two separate files.

Keystore: “who am I?”

A keystore is a protected file that holds your own private key and your own certificate, along with the chain of authorities that signed it. A broker uses its keystore to prove its identity to clients. A client uses its keystore only when it needs to prove its identity to the broker, which is the mutual authentication case we will discuss soon.

Truststore: “whom do I trust?”

A truststore is a protected file that holds the certificates of the authorities you trust. When someone presents a certificate, you check whether it was signed by an authority in your truststore. Clients need a truststore to trust the brokers. Brokers need a truststore to trust each other, and to trust clients when client certificates are required.

File formats

These stores come in several formats. The two classic ones are the Java-specific JKS format and the widely used PKCS12 format. Newer Kafka versions can also read certificates and keys directly from PEM files, the plain text format that many non-Java tools use. Which format you choose depends on your tooling and your Kafka version, but the ideas are identical.

Protect the files and their passwords

Keystores are usually protected by passwords, and those passwords appear in configuration. Restrict file permissions so that only the Kafka process can read the keystore, and avoid leaving passwords in plain text in shared repositories or tickets. Kafka supports supplying secret values through safer mechanisms, such as external secret providers and encrypted dynamic configuration, and using them is strongly recommended.

Before any Kafka message is sent, the two sides perform a short ritual called the handshake. It sets up trust and a shared secret key.

sequenceDiagram
    participant C as Client
    participant B as Broker
    C->>B: Hello, these are the TLS versions and ciphers I support
    B-->>C: Hello, I choose this version and cipher, here is my certificate
    C->>C: Verify certificate chain, dates and hostname
    C->>B: Key exchange material
    B-->>C: Confirm, secure channel ready
    C->>B: Encrypted Kafka requests
    B-->>C: Encrypted Kafka responses
        

Handshake cost and connection reuse

The handshake takes a little time and processor power. Kafka clients keep connections open for a long time and reuse them for many requests, so the handshake cost is paid rarely and spread over a huge number of messages. Problems appear only when clients open and close connections constantly, which can happen with badly written applications or misconfigured tools.

A Kafka system has many network connections. To be truly protected, every important path needs encryption, not just one.

flowchart LR
    P[Producers] -->|Client to broker| B1[Broker 1]
    C[Consumers] -->|Client to broker| B2[Broker 2]
    B1 |Broker to broker replication| B2
    B1 |Broker to controller| K[Controllers]
    B2 |Broker to controller| K
    KC[Kafka Connect] -->|Client to broker| B1
    SR[Schema Registry] -->|Client to broker| B2
    A[Admin tools] -->|Client to broker| B1
        

A Kafka broker can listen for connections in several ways at the same time. Each way is called a listener, and each has its own security choice.

What is a listener?

A listener is a combination of a name, a network address and a port on which a broker accepts connections, together with the security protocol used on it. A broker can have several listeners, for example one for internal traffic and another for external applications. This lets you apply different rules to different groups of users.

The four security protocols

Advertised addresses

When a client first connects to a broker, the broker tells it the addresses of all brokers to use next. These are the advertised addresses. They may differ from the addresses the broker actually listens on, for example when a load balancer or a container network is involved. This matters for TLS because the client will check the certificate against the advertised name. If the name in the certificate does not match the advertised name, the connection is refused.

The inter-broker listener

Brokers also need to choose which listener they use to talk to each other. This is a separate setting, and it should always point to an encrypted option. Keeping a dedicated internal listener for broker traffic, separate from the one used by outside applications, is a common and tidy design.

You do not need to memorise every setting. It is enough to know the handful of groups and what each one is for.

Per-listener settings

When a broker has several listeners, you can give each listener its own keystore and its own TLS settings, by adding the listener name to the setting. This is useful when the internal listener should use certificates from an internal authority, while the external listener uses certificates from a public one.

Changing certificates without stopping the broker

Modern Kafka versions allow the keystore and truststore of a listener to be updated while the broker is running, using its dynamic configuration feature. This is very valuable when certificates need to be renewed, because it avoids restarting every broker. Even so, you should test the process in a non-production cluster first, and check the exact supported behaviour for your version.

Every application that talks to a TLS-enabled Kafka needs a few settings too.

What a client must be told

The same ideas everywhere

Producers, consumers, admin tools, stream applications, connectors and the schema registry are all Kafka clients underneath, so they use the same kinds of settings. Non-Java clients, such as those written in other languages, usually work with plain certificate and key files in the PEM format instead of Java-style stores, and name their settings a little differently, but the concepts of trusting an authority, presenting an identity and choosing the encrypted protocol are the same.

TLS can prove the identity of just the server, or of both sides. The choice changes how you manage identities.

One-way TLS

In the common one-way setup, only the broker proves its identity. The client checks the broker’s certificate, but the broker does not ask the client for one. The connection is encrypted and clients know they are talking to a real broker, but the broker does not learn who the client is from TLS alone. If you need to know who each client is, you add another mechanism, such as a username and password login, on top.

Mutual TLS

In mutual TLS, often written mTLS, both sides show certificates. The broker asks the client for a certificate, verifies it against its truststore, and uses the name inside as the identity of the client. This gives strong, certificate-based identity, and it is popular in environments with automated certificate management.

How Kafka uses the certificate name

When a client presents a certificate, Kafka takes the distinguished name inside it, a structured description including the common name and organisation, and turns it into a principal, the identity used for authorisation. Rules can be configured to shorten or reshape this name into something friendlier. Access rules called ACLs then refer to that principal to decide what the client may read or write.

flowchart LR
    A[Client presents certificate] --> B[Broker verifies against truststore]
    B --> C[Broker reads name from certificate]
    C --> D[Name mapped to a principal]
    D --> E[Access rules decide what the principal may do]
        

A trusted certificate is not enough. The certificate must also belong to the machine you meant to reach.

Why names matter

Imagine that an attacker obtains a genuine certificate for their own server, signed by a perfectly trusted authority. If clients only checked the signature, they would accept the attacker’s server as a Kafka broker. To prevent this, clients also check that the name inside the certificate matches the address they connected to. This check is called hostname verification or endpoint identification, and modern Kafka clients enable it by default.

Subject Alternative Names

Certificates list the names they are valid for in a field called Subject Alternative Name, or SAN. A broker’s certificate should list every name and address by which clients may reach it, including its advertised names. Modern verification relies on this field, so a certificate that only has a name in the older common name field may be rejected.

Not all encryption is equally strong. Choosing the versions and methods that are allowed is an important part of the setup.

Protocol versions

TLS has gone through several versions. The oldest ones, and the SSL versions before them, have known weaknesses and must be switched off. Modern deployments allow only the newer versions. Kafka lets you list exactly which versions are enabled. The newest version brings a faster handshake and removes many old and risky features, but it needs a recent enough Java runtime and Kafka release, so check what your environment supports. A sensible rule is to allow the newest versions that all your clients can handle, and to disable everything older.

Cipher suites

A cipher suite is a named bundle of the methods used during a connection: how keys are agreed, how data is encrypted and how integrity is checked. During the handshake, the two sides pick one that both support. If you leave the list to defaults, Kafka and Java choose reasonable options for their versions. If you set a list yourself, prefer modern suites that offer forward secrecy and authenticated encryption, and remove old ones.

Compatibility versus strength

Tightening the allowed versions and suites improves security but may lock out very old clients. Before restricting settings in production, find out which client versions really connect to your brokers, using logs and inventories, and plan upgrades for those that cannot cope. A well-planned tightening is a one-time effort. A surprise lock-out of an important application is an outage.

Security has several layers. TLS is only one of them, and it is easy to confuse it with the others.

TLS with SASL

Many organisations prefer usernames and secrets, or single sign-on tickets, for client identity, and use TLS only to encrypt. That is the SASL_SSL combination: TLS wraps the connection, and SASL performs the login inside it. This is very common, and it explains why the encrypted-with-login protocol is so widely used. Using a login mechanism without TLS underneath is dangerous, because credentials may be exposed on the network.

Choosing between certificate identity and login identity

TLS ends at the broker. Once a message arrives and is written to disk, TLS has done its job and the message is stored as it is.

Encryption stops at the door

TLS is hop-by-hop. A producer encrypts to the broker, the broker decrypts, stores the message on disk in readable form, and later encrypts again when sending to a consumer. Anyone with access to the disk, a backup or the broker’s memory can read stored messages, regardless of TLS. Administrators of the machines can, in principle, read the data too.

flowchart LR
    P[Producer] -->|Encrypted by TLS| B[Broker]
    B --> D[(Disk - readable unless separately encrypted)]
    B -->|Encrypted by TLS| C[Consumer]
        

Options for protecting stored data

Protecting data is not free. The good news is that the cost is usually manageable, if you understand where it comes from.

Where the cost comes from

Ways to keep the cost under control

Certificates expire on purpose. Managing that expiry is the most important day-to-day duty of running TLS.

Why certificates expire

Every certificate has a start date and an end date. This limits the damage if a key is stolen, and forces regular renewal with fresh keys and up-to-date algorithms. But it also creates a risk: on the end date, an expired certificate is rejected, and connections start failing, sometimes across the whole cluster at the same moment.

What a healthy lifecycle looks like

Rotating the authority itself

Sometimes not only the certificates but the signing authority must change, for instance when its own certificate expires. During such a change, truststores must hold both the old and the new authority for a while, so that certificates from either are accepted. The order matters: first add the new authority to all truststores, then issue certificates from it, and only later remove the old one.

Most clusters start unencrypted. Moving them to TLS without an outage needs a careful, staged plan.

The key idea is that a broker can listen on several listeners at once. You add an encrypted listener next to the existing plaintext one, move clients over gradually, and only then remove the plaintext option. Nobody is forced to switch at the same moment.

flowchart TD
    A[Start: plaintext only] --> B[Add encrypted listener to each broker, one at a time]
    B --> C[Verify encrypted listener with a test client]
    C --> D[Move brokers' inter-broker traffic to the encrypted listener]
    D --> E[Move applications to the encrypted listener, team by team]
    E --> F[Watch that plaintext traffic falls to zero]
    F --> G[Remove the plaintext listener]
    G --> H[Done: encrypted only]
        

Kafka rarely runs alone. The tools around it all need their own TLS setup, and each has a few quirks.

Load balancers and proxies

Placing a load balancer between clients and brokers is tricky for Kafka, because clients must reach each individual broker, not just any broker. If a proxy terminates TLS and re-encrypts, the certificate the client sees belongs to the proxy, and names must match. If a proxy passes TLS through without opening it, the brokers’ own certificates must include the names that clients use. Plan names and certificates together with the network design.

Containers and Kubernetes

In container platforms, broker addresses often change, and names are internal service names. Certificates must contain those internal names. Automated tools that issue and renew certificates for pods, and mount them into containers, are the usual way to keep this manageable. Restarts of brokers to pick up new certificates should also follow safe rolling rules.

When TLS fails, the message is often cryptic. A calm, methodical approach quickly finds the cause.

What to monitor

Common problems and their meaning

flowchart TD
    A[Connection fails] --> B{Right port and protocol?}
    B -- No --> B1[Fix client protocol and port]
    B -- Yes --> C{Certificate expired?}
    C -- Yes --> C1[Renew and roll out certificate]
    C -- No --> D{Client trusts the issuing authority?}
    D -- No --> D1[Fix client truststore]
    D -- Yes --> E{Names match the address used?}
    E -- No --> E1[Reissue certificate with correct names]
    E -- Yes --> F{Versions and ciphers overlap?}
    F -- No --> F1[Align allowed versions and suites]
    F -- Yes --> G[Check mutual TLS settings and logs]
        

Habits that keep encryption strong and manageable, and traps that quietly weaken it.

Good habits

Anti-patterns to avoid

Let us see how these ideas play out in situations that teams commonly meet.

No. By default, Kafka listeners can run in plaintext, and data travels readable across the network. You must enable and configure TLS yourself.

SSL is the older name of the technology. Kafka’s setting names and protocol labels kept the old word, but modern setups use TLS versions, and old SSL versions should be disabled.

A keystore holds your own private key and certificate, answering “who am I?” A truststore holds the authorities you trust, answering “whom do I trust?”

Not for one-way TLS, where clients only need to trust the brokers. Clients need their own certificates only when brokers require mutual TLS.

No. TLS protects data only while it moves across the network. To protect stored data, use disk encryption, application-level encryption or both.

It adds processor cost for encryption and removes the zero-copy shortcut for encrypted connections, so processor use on brokers rises. With modern hardware, recent Java and connection reuse, the impact is usually manageable, but you should measure it for your workload.

No. TLS gives encryption and, optionally, identity. You also need authentication for clients and access control rules that limit what each identity can do, plus protection of data at rest where needed.

The certificate does not list the address the client used. Reissue it with the advertised host names, and any other names clients use, in the Subject Alternative Name field.

Yes, in most setups. Modern Kafka can reload keystores dynamically, or you can restart brokers one at a time. Test the procedure in a non-production cluster first.

Yes. Replication carries all of your data, so an unencrypted inter-broker path leaves it exposed even if clients use TLS.

Mutual TLS means both sides present certificates. It suits environments with automated certificate management where you want strong, key-based client identity without shared passwords.

Add an encrypted listener alongside the plaintext one, move broker traffic and then applications gradually, and remove the plaintext listener only after no clients use it.

Kafka does not encrypt its network traffic by default, so messages can be read or altered by anyone who can see the network. TLS solves this with three promises: confidentiality, integrity and authentication. It rests on private keys, certificates and a chain of trust from a certificate authority, kept in keystores and truststores. Brokers expose encrypted listeners, clients are told which protocol to use and whom to trust, and mutual TLS can add strong client identity. Every path, including broker-to-broker replication, should be protected, and certificate names must match the addresses that clients use. TLS does not protect stored data or decide permissions, so it belongs in a larger plan that includes access rules and encryption at rest. Running it well means automating certificate renewal, watching expiry dates, allowing only modern versions and migrating existing clusters in careful stages.