Access controls guard the front door. Encryption at rest protects the data when someone gets around the door.

A quick Kafka refresher

Apache Kafka stores streams of events in topics. Producers write events, consumers read them, and brokers, the servers of the cluster, store and serve them. Unlike many messaging systems that delete a message once it is read, Kafka keeps events on disk for a configured time, which may be days, months or even indefinitely. That means a Kafka cluster is often one of the largest stores of business data in an organisation, containing orders, payments, customer activity and much more.

How stored data leaks

Attackers and accidents do not always come through the network. Stored data can be exposed in several quiet ways.

Security people describe data as being in one of three states, and each needs its own protection.

flowchart LR
  P["Producer memory: in use"] -->|"Network: in transit, protected by TLS"| B["Broker memory: in use"]
  B -->|"Written to disk: at rest"| D[("Log files on disk")]
  D -->|"Read from disk"| B
  B -->|"Network: in transit, protected by TLS"| C["Consumer memory: in use"]
        

The idea is simple to state: store data in a scrambled form that only someone holding the right key can turn back into readable data.

The building blocks

Encryption uses an algorithm, a mathematical recipe for scrambling data, and a key, a secret value that controls the scrambling. The original readable data is called plaintext, and the scrambled result is called ciphertext. With the correct key, decryption turns ciphertext back into plaintext. Without the key, the ciphertext is practically impossible to read, even for someone who knows the algorithm.

Symmetric encryption for stored data

Stored data is almost always protected with symmetric encryption, where the same key locks and unlocks. It is fast, which matters when large volumes of data are read and written. A widely used and trusted algorithm for this purpose is AES, and modern processors include special instructions that make it very fast.

Encryption at rest, in one sentence

Encryption at rest means that data is encrypted before it is written to storage and decrypted only when it is read by someone who is allowed to use the key.

This is the most important fact in the whole topic, and it often surprises newcomers.

Kafka does not encrypt its log files

Apache Kafka, in its open-source form, does not provide built-in encryption of the data it writes to disk. When a broker receives an event, it appends the bytes to a log file on the local file system exactly as received. If the file is copied, anyone with the copy can read the event contents using simple tools.

Why it was designed this way

Kafka’s performance comes from a very simple storage design: it appends to files and, when serving consumers, can pass data straight from the file system cache to the network without touching it. Adding encryption inside the broker would complicate this, cost performance, and raise difficult questions about where keys should live. The project leaves at-rest protection to layers below or above Kafka.

What Kafka does offer nearby

Protecting data starts with knowing every place it is written. The broker’s main log is only part of the picture.

flowchart TB
  subgraph BR["On each broker disk"]
    L["Partition log segments: event data"]
    I["Index and time index files"]
    S["Producer state and snapshots"]
    CP["Checkpoint files"]
    M["Metadata log"]
    IT["Internal topics: offsets, transactions"]
    CF["Configuration and key store files"]
  end
  subgraph AROUND["Around the brokers"]
    BK["Backups and volume snapshots"]
    MR["Mirrored copies in other clusters"]
    TS["Tiered storage in object stores"]
    ST["Stream processing local state"]
    CN["Connect internal topics and offsets"]
    LG["Logs, dumps and exports"]
  end
        

There is no single encryption that stops every threat. Picking the right approach starts with asking who might get at the data, and how.

flowchart TD
  T["Threat"] --> A{"Stolen disk, snapshot or backup?"}
  A -- "Yes" --> V["Volume or storage encryption helps"]
  T --> B{"Administrator on the running broker?"}
  B -- "Yes" --> C["Only encryption before data reaches the broker helps"]
  T --> D{"Storage platform staff?"}
  D -- "Yes" --> E["Customer-controlled keys or client-side encryption help"]
        

Since Kafka leaves the job to other layers, the practical question is which layer to use.

flowchart TB
  L4["Layer 4: Application, client-side encryption of messages"] --> L3["Layer 3: Kafka broker, stores the bytes it receives"]
  L3 --> L2["Layer 2: File system and operating system encryption"]
  L2 --> L1["Layer 1: Disk, volume or cloud storage encryption"]
        

The most common and least disruptive approach is to encrypt the storage underneath Kafka.

How it works

Below the file system sits a layer that encrypts every block of data as it is written to the disk and decrypts it as it is read. Kafka and the file system see ordinary readable data, and the disk holds only ciphertext. On Linux, this is often done with a standard block-level encryption feature of the operating system. In cloud platforms, the provider offers encrypted volumes, where the encryption is turned on when the volume is created and keys are managed through the provider’s key service.

flowchart TB
  K["Kafka broker writes readable data"] --> FS["File system"]
  FS --> E["Encryption layer: encrypts on write, decrypts on read"]
  E --> D[("Disk holds only ciphertext")]
  KM["Key from a key management service"] -.-> E
        

Why it does not disturb Kafka’s speed

Kafka reads and writes files through the operating system’s cache. Volume encryption sits below that cache, so data stays in memory in readable form after it has been read once, and later reads do not need to be decrypted again. Efficient transfers of cached data to the network remain possible. The cost appears only when data moves between the cache and the physical disk, and hardware-accelerated encryption keeps that cost small.

A related option encrypts at the level of files and directories instead of the whole device.

How it differs from volume encryption

Some operating systems can encrypt individual directories, so that only the Kafka data directory is protected while the rest of the machine is not. Files are encrypted when stored, and decrypted for programs that have the right access. It offers more flexibility, for example different keys for different directories, but it is generally less common for Kafka brokers than encrypting the whole volume.

Where it makes sense

If you run Kafka in the cloud or use a managed service, much of the work is done by the platform, but you still choose the key model.

Encrypted cloud volumes

Cloud providers offer encrypted disks, usually with a simple switch at creation time and integration with their key management service. The provider handles the mechanics. For self-managed Kafka on virtual machines, this is often the easiest route to encryption at rest, and many organisations make encrypted volumes mandatory through policy.

Managed Kafka services

Managed Kafka offerings from cloud and Kafka vendors typically encrypt stored data by default, using keys managed by the provider. Many also allow you to bring or control your own keys, so that you can revoke the provider’s ability to read the data. The details, such as which key options exist and whether they cover all storage tiers, differ between providers and change over time, so verify them in the current documentation of your chosen service.

Who holds the key?

The strongest approach encrypts the message before it ever reaches Kafka, so Kafka never sees the readable data.

How it works

The producer application encrypts the sensitive part of each event, or the whole payload, before sending it. Kafka receives, stores, replicates and delivers ciphertext, without knowing what it means. Consumer applications that have access to the right key decrypt the data after reading. Anyone who reads the brokers’ files, the network traffic or the backups sees only ciphertext.

sequenceDiagram
  participant P as Producer
  participant K as Key service
  participant B as Kafka broker
  participant C as Consumer
  P->>K: Request encryption key
  K-->>P: Key
  P->>P: Encrypt the sensitive data
  P->>B: Send ciphertext
  B->>B: Store and replicate ciphertext only
  B-->>C: Deliver ciphertext
  C->>K: Request decryption key
  K-->>C: Key, if permitted
  C->>C: Decrypt the data
        

Where the encryption plugs in

Applications can add encryption in their own code before calling the producer, or use hooks built into Kafka’s client libraries such as serializers or interceptors. Some platforms provide ready-made features for this, for example field-level encryption defined alongside schemas and applied by the client libraries. Details vary by product and version, so check what your platform offers before building your own.

Real systems do not use one key for everything. They use layers of keys, which makes rotation and control practical.

The problem with a single key

If one master key encrypted all data directly, using it would mean sending large amounts of data to the key service, or spreading the master key across many machines, which is risky. Changing that key would mean re-encrypting everything.

The envelope idea

In envelope encryption, data is encrypted with a data key, a key that is generated for a specific purpose such as one topic, one customer or one batch. The data key is then itself encrypted with a stronger key held in a key management service, called the key-encrypting key. The encrypted data key is stored alongside the data, like a sealed envelope stapled to the document. To read the data, an application asks the key service to unwrap the data key, and then uses it locally.

flowchart TB
  MK["Root or master key: inside a key service or hardware module"] --> KEK["Key-encrypting key: wraps data keys"]
  KEK --> DK1["Data key for topic A"]
  KEK --> DK2["Data key for topic B"]
  KEK --> DK3["Data key for customer 42"]
  DK1 --> D1["Encrypted data for topic A"]
  DK2 --> D2["Encrypted data for topic B"]
  DK3 --> D3["Encrypted data for customer 42"]
        

Encryption moves the security problem from the data to the keys. Managing keys well is what decides whether encryption at rest is real or cosmetic.

Where keys should live

Keys belong in a dedicated key management service or hardware security module, a tamper-resistant device or service designed to hold keys and perform cryptographic operations without exposing the keys themselves. They should never be stored on the same disk as the data they protect, or in configuration files in shared repositories.

The lifecycle of a key

Separation of duties

The people who administer Kafka and the people who control the keys should ideally be different, with different accounts and approvals. If a single person can both read the encrypted files and obtain the key, the separation that encryption is meant to create disappears.

Because Kafka can no longer read encrypted parts of a message, some features behave differently. Knowing this early saves painful redesigns.

Kafka logs are designed to be append-only, which makes deleting one person’s data difficult. Encryption offers a clever way around it.

The deletion problem

Privacy rules in many regions give people the right to have their personal data erased. In a Kafka topic, events are stored in segments that are not edited in place, and copies may exist on replicas, in backups, in mirrored clusters and in downstream systems. Finding and deleting every copy of one person’s events is slow and error-prone.

The crypto-shredding idea

If each person’s data is encrypted with a key unique to that person, then deleting that key makes every copy of their data, wherever it sits, permanently unreadable at once. The encrypted bytes may remain, but they are indistinguishable from random noise. This technique is called crypto-shredding.

flowchart LR
  A["Erasure request for customer 42"] --> B["Delete customer 42's key in the key service"]
  B --> C["All copies of customer 42's data become unreadable"]
  C --> D["Logs, replicas, backups and mirrors"]
  B --> E["Other customers unaffected"]
        

Encryption is never completely free, but each layer has a different price, and most of it is small when done well.

Keeping the overhead in check

Data spreads to many places around Kafka. Each one needs its own answer to the at-rest question.

flowchart LR
  BR[("Broker volumes: encrypted")] --> TS[("Tiered storage in object store: encrypted with managed keys")]
  BR --> BK[("Backups and snapshots: encrypted")]
  BR --> MR[("Mirrored cluster volumes: encrypted")]
  BR --> ST[("Stream processing state volumes: encrypted")]
        

Encrypted data is safe only if the passwords and keys that unlock it are not lying around in plain sight.

What secrets Kafka configuration holds

Broker and client configuration files often contain passwords for key stores and trust stores, credentials for authentication, and secrets for connecting to other systems. Left in plain text in files that are shared, copied or committed to repositories, they undermine every other protection.

Ways to reduce exposure

Encryption changes routine operations. Restarts and disasters need thought in advance.

Unlocking volumes at boot

An encrypted volume must be unlocked before Kafka can use it. In the cloud, this often happens automatically when the volume is attached, based on the machine’s permission to use the key. On self-managed hardware, the options range from typing a passphrase, which does not scale, to network-based unlocking services or hardware-backed keys that release the secret only to a trusted, unmodified machine.

sequenceDiagram
  participant M as Machine starts
  participant K as Key service
  participant V as Encrypted volume
  participant B as Kafka broker
  M->>K: Prove identity, request volume key
  K-->>M: Key released if policy allows
  M->>V: Unlock volume
  V-->>M: Volume available
  M->>B: Start Kafka using the volume
        

What if the key service is unavailable?

If a broker restarts while the key service cannot be reached, it may be unable to unlock its volume and will stay down. If many brokers restart at once, for example after a power failure, the cluster cannot recover until the key service returns. Design the key service for high availability, test restarts, and avoid restarting all brokers simultaneously.

Disaster recovery

Most teams adopt encryption at rest on clusters that are already serving traffic, and the trick is to let Kafka’s own replication do the heavy lifting.

flowchart TD
  A["Stage 1: Inventory data locations and choose the approach"] --> B["Stage 2: Set up the key service, policies and access"]
  B --> C["Stage 3: Add new brokers with encrypted volumes"]
  C --> D["Stage 4: Move partition replicas to the new brokers"]
  D --> E["Stage 5: Remove the old brokers and wipe their disks"]
  E --> F["Stage 6: Encrypt backups, mirrors, tiers and state stores"]
  F --> G["Stage 7: Verify, monitor and document"]
        

For many organisations, encryption at rest is not optional. It is a requirement written into policies, contracts and regulations.

Why rules ask for it

Standards and laws covering payment card data, health records, personal data of individuals and financial services commonly expect that stored sensitive data is protected, and that encryption is one accepted way to do so. The exact wording, scope and accepted methods differ between rules and regions, so involve your compliance and legal teams and record how your design meets each requirement.

What auditors usually ask

A control you do not check drifts away. A small set of signals keeps encryption honest.

Regular verification

Schedule periodic checks that inspect the actual state of volumes, buckets and backups, not just the documents describing them. Where possible, test that a copy of a data file is truly unreadable without the key, and test restores from encrypted backups.

Alerts worth having

Alert when a new volume appears without encryption, when key use spikes or comes from a new identity, when a key is scheduled for deletion, when keys pass their rotation dates, and when key service errors rise.

Habits that keep encrypted Kafka platforms safe and workable, and the traps that quietly defeat them.

Patterns worth adopting

Anti-patterns to avoid

A simple path from your risks and requirements to a sensible combination of layers.

flowchart TD
  S["Does the data need encryption at rest?"] --> Q1{"Any sensitive or regulated data?"}
  Q1 -- "No" --> N["Still consider volume encryption as a low-cost default"]
  Q1 -- "Yes" --> V["Enable volume or storage encryption everywhere"]
  V --> Q2{"Must admins or the platform be unable to read data?"}
  Q2 -- "No" --> Q3{"Need per-person erasure by deleting keys?"}
  Q2 -- "Yes" --> C["Add client-side encryption for sensitive fields"]
  Q3 -- "Yes" --> C
  Q3 -- "No" --> Q4{"Need control over key location or revocation?"}
  Q4 -- "Yes" --> K["Use customer-managed keys for the storage layer"]
  Q4 -- "No" --> D["Provider-managed keys may suffice"]
        

A checklist to keep beside you when designing or reviewing encryption at rest for Kafka.

Five common situations show how the choices fit together.

No. Open-source Kafka writes event data to disk exactly as received. Encryption at rest must be added through storage encryption, file system encryption, a managed service or client-side encryption.

No. TLS protects data in transit over the network. Once the broker writes the data to disk, TLS no longer applies and separate at-rest protection is needed.

Encrypting the disks or volumes under the brokers, especially with the built-in options of cloud platforms or the operating system. It needs no changes to Kafka or client applications.

Not on a running broker. The volume is unlocked while in use, so someone with access to the machine can read the files. To defend against that, use client-side encryption so data is unreadable before it reaches the broker.

Producers encrypt data, or selected fields, before sending, and consumers decrypt after reading. Use it for highly sensitive data, when administrators or the platform must not see readable data, or when you need per-subject keys for erasure.

Volume encryption with hardware acceleration usually has a small cost. Client-side encryption adds CPU work and message size on clients. Measure with your own workload to know the real effect.

Encrypted data looks random, and compression works by finding patterns. Compress before encrypting, or encrypt only small sensitive fields and let the rest compress normally.

You can, but be careful. Kafka uses keys to choose partitions and to compact topics. Randomised encryption of keys breaks both. Prefer stable, non-sensitive keys, and encrypt sensitive data in the value.

Encrypting each person’s data with its own key and deleting that key when erasure is required, so all copies of the data become permanently unreadable without rewriting logs or backups.

The data protected by that key becomes unreadable for good. That is why keys need secure backups, controlled deletion and tested recovery procedures.

Add new brokers with encrypted volumes, use partition reassignment to move replicas onto them gradually, then retire and securely wipe the old brokers. Kafka’s replication does the data movement.

Typically yes, by default, with provider-managed keys, and many offer customer-managed key options. The exact features vary by provider and change over time, so check the current documentation for yours.

Kafka keeps large volumes of business data on disk for long periods, and open-source Kafka stores it in readable form. Encryption at rest closes the gap for stolen disks, leaked snapshots and misused storage access. Because Kafka leaves the job to other layers, you choose among encrypted volumes, file system encryption, managed services with managed or customer-controlled keys, and client-side encryption for the most sensitive fields.

The encryption itself is the easy part. The hard parts are the keys, the many copies of data around Kafka, the side effects on compression and partitioning, and the operations of unlocking, rotating, recovering and monitoring. If you remember one sentence, let it be this: encryption is only as strong as the protection of its keys and the completeness of its coverage.