Before touching Kafka, we need to be clear about what we are building. An audit log is not the same thing as the messages your application prints while it runs.

Two very different kinds of logs

Most software writes application logs. These are notes for developers, such as “started payment job” or “database took 400 milliseconds to answer”. They help engineers fix bugs. Nobody promises to keep them for years, and nobody worries if a few lines go missing.

An audit log is different. It is a permanent, trustworthy record of who did what, to which thing, when, from where, and whether it worked. It is written for auditors, investigators and regulators. If an application log is a diary, an audit log is a signed receipt book.

The five questions every audit record must answer

A useful audit record answers five simple questions. Security people call them the “who, what, when, where and outcome” of an event.

• Who acted: a person, a service account, or an automated job.
• What they did: read, create, change, delete, approve, log in, change a permission.
• On what: the customer record, the topic, the file or the setting that was touched.
• When and where: an exact timestamp and the system or network address involved.
• Outcome: whether the action succeeded, failed or was blocked, and why.

Why the law cares

Governments and industry bodies do not trust a company’s word alone. They ask for evidence. When something goes wrong, such as a data breach, a fraud case or a suspicious change to financial numbers, the audit log is the evidence. Without it, a company cannot prove what happened, cannot show it acted responsibly, and often cannot even tell which customers were affected.

You do not need to be a Kafka expert to follow this guide. Here are the handful of ideas that everything else builds on.

What Kafka is

Apache Kafka is a system for moving and storing streams of events. An event is just a small record that says “something happened”, for example “user 42 changed their email address”. Software components called producers write events into Kafka. Other components called consumers read them out, sometimes seconds later and sometimes months later.

Kafka was created at LinkedIn to carry activity data and operational data around a very large website, and it was later open sourced. Today companies such as Netflix, Uber and thousands of banks and retailers use it as the central nervous system between their services.

The building blocks

Why Kafka suits audit logging

Kafka has three natural strengths for this job. First, it is append-only: new events are added at the end of a partition, and the normal way of using it never rewrites earlier ones. Second, it is durable: events are saved to disk and copied to other brokers. Third, it is replayable: an investigator can start reading from any offset and re-live what happened. Those three traits are exactly what an audit trail needs.

They are not enough on their own, though. Append-only in normal use does not mean tamper-proof, and durable does not mean forever. The rest of this guide is about closing those gaps.

People use the phrase “audit logging in Kafka” to mean two related but different jobs. Mixing them up is the most common source of confusion, so let us separate them at the start.

Meaning one: auditing the Kafka platform itself

Here the question is: who did what to my Kafka cluster? Who connected? Who was refused permission? Who created a topic, deleted a topic, changed a retention setting or altered an access rule? Your platform team and your security team both care about this, because an attacker or a careless administrator with the wrong permissions can quietly damage the data of every application on the cluster.

Meaning two: using Kafka as the pipe for application audit events

Here the question is: how do I carry the audit events of my business applications safely? Your banking app, your hospital records system and your admin portal all generate audit events. Kafka becomes the reliable highway that collects them, fans them out to storage, search and alerting, and keeps a replayable copy.

A serious compliance program needs both. If you only build the pipeline, an administrator could still delete your audit topic without leaving a trace. If you only audit the cluster, you have no organized way to collect the events from your applications. The chapters that follow cover each side and then join them together.

You do not need to memorize legal text. You do need to know what each major rule pushes you to build. Please treat this chapter as a friendly map, not legal advice; always confirm the details with your compliance team or lawyers.

The main frameworks

Six ideas hiding behind all the rules

The frameworks use different words, but they keep asking for the same six qualities. Learn these and you can design for any of them.

Notice what these six qualities are about. None of them says “use Kafka”. They are goals. Kafka is a tool that can help you reach several of them, and the remaining chapters show which Kafka features serve which goal.

The quality of your audit trail is decided long before Kafka gets involved. It starts with what each single event contains.

Use one standard envelope

If every team invents its own audit format, investigators will spend their days translating. Instead, define one common envelope: a fixed set of fields that every audit event must carry, plus a flexible section for details that are specific to one application.

Record the decision, not just the success

Failed and denied actions are often more interesting than successful ones. Ten failed logins followed by one success tells a very different story than one success alone. Always audit refusals, not just completions.

Time deserves extra care

Kafka stamps every record with a time, and it can do so in two ways. Create time is set by the producer, using the producer’s clock. Log append time is set by the broker at the moment it stores the record. A producer’s clock can be wrong, or even changed on purpose, so a careful design keeps both: the time the action happened as reported by the source, and the time Kafka received it. When the two differ wildly, that difference is a clue worth investigating.

Keep all machines synchronized with a trusted time service, and store times in one zone (UTC) so nobody has to guess about daylight saving changes.

Lock the format with a schema

A schema is a written contract describing exactly which fields an event has and what type each field is. Tools such as a Schema Registry, used with formats like Avro or Protobuf, refuse events that break the contract and control how the contract may evolve. For audit data this is a gift: it prevents a well-meaning developer from renaming or dropping a field that an auditor relies on.

Choose the message key wisely

In Kafka, the key of a message decides which partition it goes to, and Kafka only promises ordering within one partition. If you key audit events by the id of the thing being changed, such as the account number, then every event about that account lands in the same partition, in the right order. That makes “show me the full history of this account” a straightforward read.

Now let us zoom out and see all the pieces of a complete audit pipeline, from the moment an action happens to the moment an auditor reads the record.

flowchart LR
  A["Applications and Services"] --> B["Audit Producer Library"]
  K["Kafka Cluster Security Events"] --> C
  B --> C["Kafka Audit Topics"]
  C --> D["Stream Processor for Enrichment"]
  D --> E["Kafka Connect Sink"]
  E --> F["Immutable Archive with Object Lock"]
  C --> G["Search Index for Investigators"]
  C --> H["Alerting and SIEM"]
    

Walking through the diagram

Applications and services are where actions happen. They do not write directly to Kafka in a haphazard way. They use a shared audit producer library, a small piece of code every team includes, which builds the standard envelope from Chapter 5, adds identity and time, and sends the event. One library means one place to fix bugs and one consistent format.

The Kafka audit topics are the heart of the design. Events are collected here, copied across brokers, and held for a defined period. The Kafka cluster security events box on the left shows that the platform’s own login and permission records feed into the same stream, so investigators see both stories together.

A stream processor reads the raw events and enriches them. For example, it may attach the department of a user, translate an internal id into a readable name, or mask a sensitive value. Then a Kafka Connect sink, a ready-made tool for moving data out of Kafka, writes the events to an immutable archive such as object storage with write-once protection. Two other consumers read the same topics independently: a search index so investigators can query quickly, and an alerting system that raises the alarm on suspicious patterns.

Why several independent consumers matter

Kafka lets many consumer groups read the same topic without interfering. This is a big advantage for compliance. The archive can be slow and careful, the search index can be fast and disposable, and the alerting tool can be picky, all from one source of truth. If the search index is ever corrupted, you rebuild it by replaying the topic. The audit records themselves stay safe.

Now we turn to the first meaning from Chapter 3: recording what happens on the cluster. This is where Kafka’s built-in abilities and your own additions come together.

Step one: know who is connecting (authentication)

Authentication means proving who you are. Kafka can ask clients to prove identity with certificates (mutual TLS), with usernames and passwords in a safer challenge form (SASL SCRAM), with Kerberos, or with tokens (OAuth bearer). For audit purposes the vital point is that every client must have its own identity. If ten applications share one login, the log can say only “the shared login did it”, and that is useless to an investigator.

Step two: control what they can do (authorization)

Authorization means deciding what an identity is allowed to do. Kafka uses access control lists, usually shortened to ACLs. Each rule says that a certain identity may or may not perform a certain operation, such as read, write, create or delete, on a certain resource, such as a topic, a consumer group or the cluster as a whole.

Step three: record every decision

Kafka brokers include an authorizer log. By default it records requests that were denied. If you raise its detail level, it records allowed requests as well, at the price of much larger log volume. Brokers also have a request log that shows every call clients make, but it is extremely detailed and normally switched on only briefly when troubleshooting, not as a permanent audit feed.

sequenceDiagram
  participant C as Client App
  participant B as Kafka Broker
  participant A as Authorizer
  participant L as Audit Log
  C->>B: Connect with credentials
  B->>B: Verify identity
  B->>L: Record login success or failure
  C->>B: Request to write to a topic
  B->>A: Is this identity allowed
  A-->>B: Allowed or denied
  B->>L: Record the decision
  B-->>C: Send result to client
    

Read the diagram from top to bottom. The client connects and the broker checks who it is, writing down the result. Then the client asks to do something. The broker asks the authorizer whether that identity may do it, and writes down the answer before replying. Two records, two moments, one complete story.

What the built-in logs do not give you

Plain Apache Kafka writes these records into ordinary broker log files, in a text format meant for humans. That is a fine start but it falls short of compliance needs in three ways. The files sit on the same machines an attacker may control. The format is not a tidy structured event. And an administrator with shell access could edit them. So the usual practice is to ship these files off the broker immediately with a log shipper into a protected, separate system, and to turn them into structured events.

Some commercial Kafka distributions add a ready-made feature here. Confluent, for instance, offers audit logs for its platform and cloud service that write structured records, in a standard event format, into a dedicated Kafka topic. Managed services help in other ways: Amazon MSK can deliver broker logs to CloudWatch Logs, S3 or Firehose, and AWS CloudTrail records the management calls made to the MSK service itself, such as creating or deleting a cluster.

The administrative actions you must never miss

• Creating, deleting or changing a topic, including changes to its retention time.
• Adding, removing or altering an ACL.
• Using the delete-records operation, which removes old data from a partition.
• Changing broker or cluster configuration.
• Creating or removing credentials and tokens.

Each of these could quietly destroy or expose audit data. Treat them as high-severity events and alert on them.

Here is a truth that surprises many teams: the hardest part is not storing audit events in Kafka. It is making sure that every event that should be stored actually gets there.

The dual-write trap

Imagine a service that updates a customer’s address in its database and then sends an audit event to Kafka. That is two separate systems and two separate steps. If the database update succeeds but the service crashes before sending the event, you have a change with no audit record. If the event is sent but the database update then fails, you have an audit record of something that never happened. Both are compliance failures, and both are invisible until an auditor looks.

The transactional outbox pattern

The standard cure is the outbox pattern. The service writes the business change and the audit event into the same database, inside one transaction. The audit event goes into a special table, the outbox. Either both are saved or neither is. A separate relay process then reads new rows from the outbox and publishes them to Kafka. Some teams do the relay by watching the database’s own change log, using a tool such as Debezium.

sequenceDiagram
  participant S as Service
  participant D as Database
  participant R as Outbox Relay
  participant K as Kafka
  S->>D: Begin transaction
  S->>D: Save the business change
  S->>D: Save the audit event in the outbox table
  S->>D: Commit both together
  R->>D: Read new outbox rows
  R->>K: Publish audit event and wait for acknowledgement
  K-->>R: Acknowledged by in-sync replicas
  R->>D: Mark row as published
    

If the relay crashes between publishing and marking the row, it may publish the same event twice. That is acceptable, because each event carries a unique id and consumers can spot and ignore duplicates. It is far better to have a harmless duplicate than a missing record.

Producer settings that protect you

When the relay or an application sends events, a few Kafka producer settings decide how safe the send is.

What if Kafka is down?

An audit design must decide, in advance, what happens when Kafka cannot be reached. There are two honest choices. Fail closed means the business action is refused if it cannot be audited. Banks often choose this for sensitive operations such as moving money. Buffer and continue means the outbox keeps the events safely in the database until Kafka returns. With the outbox pattern you get the second choice almost for free, because the events are already saved. Whichever you pick, write it down and test it. Auditors love a documented and rehearsed failure plan.

An audit record that disappears when a server breaks is worthless. Kafka gives you strong tools to survive machine failures, but only if you set them up on purpose.

Replication: many copies, no single point of failure

Each partition has one leader copy that handles reads and writes, and several follower copies on other brokers that stay in step with it. The followers that are fully caught up form the in-sync replica set. A common production layout keeps three copies, each in a different data center zone.

flowchart LR
  P["Producer"] --> L["Leader copy in Zone 1"]
  L --> F1["Follower copy in Zone 2"]
  L --> F2["Follower copy in Zone 3"]
  F1 --> ACK["Acknowledgement to producer"]
  F2 --> ACK
  ACK --> P
    

Three settings that work as a team

• Replication factor of three means three copies exist.
• Minimum in-sync replicas of two (min.insync.replicas) means a write is only accepted if at least two copies can confirm it.
• Producer acknowledgement set to all means the producer waits for those confirmations.

Together they mean that when an audit event is acknowledged, it exists on at least two machines. You can lose one machine at any moment and lose nothing. Also switch off unclean leader election. That setting would let an out-of-date copy become the leader in an emergency, silently discarding events it never received. For audit data, availability is not worth that price.

Retention: how long Kafka keeps events

Every topic has a retention rule, by time or by size, after which old segments are deleted. Audit topics usually need a long time setting, and Kafka allows unlimited retention if you choose it. Two cautions apply here.

Second, long retention on broker disks is costly. Newer Kafka versions support tiered storage, which keeps recent data on fast local disks and moves older data to cheap object storage while still letting consumers read it through Kafka. That allows months of replayable history without paying for enormous broker disks.

Surviving a whole region going dark

Replication across zones protects against a building failing. A regional disaster needs a second cluster in another region, kept in step by a mirroring tool such as MirrorMaker 2, which copies topics between clusters. Because mirroring is not instant, plan for a small gap and combine it with the archive in Chapter 13, which is independent of any cluster.

Auditors do not only ask whether you have logs. They ask whether you can prove nobody changed them. This chapter is about turning “trust us” into “verify it yourself”.

Kafka is append-only, but not tamper-proof

In normal use, Kafka never edits a stored record. But that is a habit of the software, not a law of nature. A user with powerful enough permissions can delete a topic, shorten its retention so the data expires, call the delete-records operation, or even edit files directly on a broker’s disk. Kafka on its own is therefore append-only but not immutable. Compliance needs the second word.

Idea one: hash chaining

A hash is a fingerprint computed from data. Change one letter of the data and the fingerprint changes completely. In a hash chain, each audit event includes the fingerprint of the previous event. To rewrite history, an attacker would have to recompute every fingerprint after the change, and every checkpoint that was already published elsewhere.

flowchart LR
  E1["Event 1 with hash H1"] --> E2["Event 2 stores H1 and has hash H2"]
  E2 --> E3["Event 3 stores H2 and has hash H3"]
  E3 --> S["Periodic signed checkpoint"]
  S --> V["Verifier recomputes the chain"]
    

Idea two: digital signatures on checkpoints

Chains alone can be rebuilt by someone with enough access. So every so often, a separate trusted component takes the latest fingerprint and digitally signs it with a private key held in a protected key service such as a hardware security module or a cloud key management service. The signed checkpoint is stored somewhere the Kafka administrators cannot touch. A verifier can later recompute the chain and check it against the signed checkpoints. AWS CloudTrail uses the same idea with its log file integrity validation, which relies on hashing and digital signing so you can detect changes to delivered log files.

Idea three: write once, read many storage

The strongest protection is to copy audit events to storage that physically refuses changes. Object storage such as Amazon S3 offers Object Lock. In its compliance mode, nobody, not even the account owner, can delete or overwrite a locked object until its retention date passes. AWS describes this as designed to help meet write-once-read-many requirements, such as those in financial record-keeping rules. A lock on the archive means that even if an intruder gains full control of Kafka, the archived copy survives.

Idea four: audit the auditors

Who can change the audit topic’s settings? Who can change the signing keys? Every one of those actions must itself be logged, and the log of those actions must be stored where the same people cannot alter it. This is called meta-auditing, and auditors look for it specifically.

Audit logs have an awkward nature: they are a concentrated record of everything sensitive that happened, so they are a juicy target. Protecting them is part of the job.

Log the fact, not the secret

The first rule is to record what happened without copying the private details. “User changed their password” is an audit fact. The password itself must never appear, not even hashed. Likewise, do not write full card numbers, medical notes or government identification numbers into audit events. Store an identifier, a masked form such as the last four digits, or a token that points to the real value in a protected system.

Encrypt everything, everywhere

• In transit: use TLS between clients and brokers, and between brokers.
• At rest: encrypt broker disks and archive storage with keys you control.
• Field level: for especially sensitive fields, encrypt just that field inside the event, so that only a small group holding the key can read it while everyone else still sees the rest of the record.

The privacy paradox and crypto-shredding

Privacy laws such as GDPR give people the right to have personal data erased in many situations, though there are exceptions, including where the law requires you to keep records. Audit logs are meant to be permanent and unchangeable. These two ideas seem to collide. You cannot edit a locked archive to remove one person’s name.

A clever, widely used answer is crypto-shredding. Each person’s personal fields are encrypted with a key made just for that person, and the keys live in a key service. To “erase” the person, you delete their key. The encrypted words remain in the archive, but they are now unreadable forever, while the rest of the audit trail, who acted and when, stays intact.

Keep secrets out of the plumbing

Do not put access tokens, session cookies or passwords in message headers or keys. Keys and headers are often copied into monitoring tools and are easy to leak. Also review what your stream processor and search index store, because every copy of an audit event is another place that needs protection.

A perfect design can be undone by a single overly generous permission. Access control on the audit topics deserves the same care you give to the vault of a bank.

Least privilege and separation of duties

Least privilege means every identity gets only the permissions it truly needs and nothing more. Separation of duties means no single person can both do something risky and hide the evidence of it. Together they are the backbone of SOX-style controls.

Practical rules

Keeping records for seven years is one thing. Being able to find one specific record inside those seven years, on a deadline, is another.

Think in temperature: hot, warm and cold

Not all audit data is needed equally fast. Recent events are searched often; old ones rarely. Storing everything on the fastest, most expensive system would waste money, and storing everything in the cheapest would frustrate investigators. Tiering solves it.

flowchart TD
  A["Hot tier: Kafka brokers, days to weeks"] --> B["Warm tier: tiered storage or search index, months"]
  B --> C["Cold tier: locked object archive, years"]
  C --> D["Legal hold or disposal review"]
    

Make the archive searchable

Write archived events in an efficient file format, partitioned by date and, if useful, by event type. Then a query service can scan only the small slice needed. Query engines that read directly from object storage, such as Amazon Athena, make it possible to answer “show all deletions by this user in March” without loading anything back into Kafka.

Legal holds and disposal

When a lawsuit or investigation begins, a legal hold freezes the relevant records so they cannot be deleted even when their normal retention ends. Object Lock supports legal holds for exactly this. When the retention period does pass and no hold applies, disposal should also be deliberate, logged and, where required, approved. Keeping data forever “just in case” can itself be a compliance failure, as privacy rules expect you not to hoard personal data.

Rehearse the retrieval

Do a drill. Pick a random event from two years ago and time how long it takes your team to produce it, prove its integrity and hand it over. If the answer is “a week and three engineers”, fix that before an auditor finds out.

The bridge between Kafka and your long-term archive is easy to overlook, yet it is the piece that decides whether your records are still readable in ten years.

What Kafka Connect does

Kafka Connect is a framework for moving data into and out of Kafka without writing custom code. You configure a connector, which is a ready-made plug for a specific system. A source connector pulls data into Kafka, for example from a database change log. A sink connector pushes data out of Kafka, for example into object storage. Connect runs the connector as a set of tasks spread across worker machines, restarts them if they fail and remembers how far each has read.

Choices that matter for audit data

• File format. Column-oriented formats such as Parquet compress well and speed up queries. Row-oriented formats such as Avro keep every field of an event together, which suits full-record retrieval. Many teams keep one of each.
• Partitioning by time. Writing files into folders by date and hour makes searches cheap, and it makes output deterministic, meaning the same events always land in the same files.
• Delivery guarantee. Some sink connectors can offer exactly-once delivery when configured with a deterministic, time-based layout. Otherwise expect at-least-once, and rely on the unique event id to remove duplicates.
• File size and rotation. Too many tiny files slow every query. Rotate files by size or time so that each holds a healthy amount of data.
• Manifests and checksums. Record the name and fingerprint of every archived file. A manifest makes integrity verification and legal exports far easier.

Handling bad records without losing them

Sooner or later a malformed event will appear. The connector can be told to stop the entire pipeline, to skip the bad record, or to send it to a dead-letter topic for humans to inspect. For audit data, skipping silently is the worst option, because a skipped record is a lost record. Prefer the dead-letter topic, alert on anything that lands there, and treat the queue as a to-do list that must be emptied.

Schema evolution without regret

Your event format will change over the years: new fields, new event types, new regulations. A schema registry can enforce compatibility. In backward compatible mode, readers using the new schema can still read old data. In forward compatible mode, readers using an old schema can still read new data. Full compatibility means both. For audit data, the safest habit is simple: only add optional fields with default values, never remove or rename a field, and never reuse a field name for a different meaning.

Now for the trick that people forget. Store the schema together with the data. Avro files carry their schema inside. For other formats, save the schema versions in the archive alongside the events. Ten years from now, your schema registry product may have changed, or the server may be gone, but an archive that explains itself will still be readable.

Large companies operate in many countries, and many countries have rules about where personal data may live. Audit logs are full of personal identifiers, so those rules apply to them too.

Why location matters

Privacy regulations often restrict moving personal data across borders, and some countries require certain data to stay inside their territory. An audit event that says “employee Maria in Berlin viewed the record of customer 8841” contains personal data about at least two people. If you ship every event from every country into one central cluster, you may create a legal problem while solving a technical one.

Three ways to organize it

Practical guidance

• Keep keys local. Use encryption keys held in the same region as the data. Crypto-shredding then works inside the jurisdiction.
• Disaster recovery inside the boundary. If your data must stay in the European Union, put the backup region in the European Union as well.
• Search where the data lives. A central security team can send its queries to each region and receive only the answers it needs, rather than copying everything to one place.
• Do not promise one global order. Clocks in different regions differ slightly, and events from separate clusters have no shared sequence. Order events within each region by sequence numbers and treat cross-region timing as approximate, showing both source times and arrival times.
• Avoid synchronous cross-region writes. Waiting for a distant region to confirm every audit event makes the business slow. Write locally, copy onwards in the background.

To trust Kafka with evidence, it helps to know what actually happens between “the producer sent it” and “it is safe”. The story is simpler than you might fear.

Step by step through a single write

When a producer sends audit events, it does not send them one at a time. It gathers several into a batch for the same partition, often compresses the batch to save space, and sends it to that partition’s leader broker. The leader appends the batch to the end of the partition’s current log segment, which is simply a file on disk. Segments have a maximum size, and when one fills up, Kafka starts a new one. Deleting old data means deleting whole segment files, which is why retention is quick and cheap.

Next, the follower brokers, which hold the other copies, continuously ask the leader for new data. This is a pull design: followers fetch, the leader does not push. As followers confirm they have the data, the leader advances a marker called the high watermark. This marker is the highest offset that every in-sync copy has stored. Consumers can only read up to the high watermark, so they never see an event that could still be lost in a failover.

sequenceDiagram
  participant P as Producer
  participant L as Leader Broker
  participant F1 as Follower One
  participant F2 as Follower Two
  participant C as Consumer
  P->>L: Send batch of audit events
  L->>L: Append to log segment
  F1->>L: Fetch new data
  F2->>L: Fetch new data
  L->>L: Advance high watermark
  L-->>P: Acknowledge with all in-sync copies
  C->>L: Read up to high watermark
    

Does Kafka write to disk instantly?

This surprises many people. By default, Kafka hands data to the operating system, which keeps it in memory and writes it to disk shortly after, rather than forcing a disk flush for every message. Kafka’s design relies on replication for safety, not on flushing a single machine’s disk each time. That is why the settings from Chapter 9 matter so much: with three copies in three zones and a minimum of two confirmations, the chance that all copies lose the same event at once is tiny. Forcing constant flushes would cost a lot of speed for little extra safety.

Who manages the cluster?

Someone must decide which broker leads which partition and what to do when a broker dies. In modern Kafka this is handled by a group of controller nodes running Kafka’s own consensus protocol, called KRaft. Older clusters used a separate coordination system called ZooKeeper. For audit purposes, remember that the controllers hold the cluster’s metadata, including ACLs, so they deserve the same protection as brokers, and changes to them should also be audited.

Where do reader positions live?

A consumer group remembers how far it has read by saving its offset in an internal Kafka topic. The safest habit for an archive connector is to save its position only after the data has been written to the archive successfully. If the connector crashes mid-way, it will re-read a few events and write them again. This is called at-least-once delivery. Combined with unique event ids, a duplicate is harmless, but a missing event is not, so at-least-once is the right side to err on.

Let us follow a single event, from the moment someone clicks a button to the day it is finally retired. Seeing the whole journey makes it clear where each control fits.

What can go wrong at each step

Chapter 7 touched on authentication and authorization. Here we build the full picture, layer by layer, because audit data is only as safe as the weakest layer around it.

Layer 1: the network

Brokers should live in private network segments with no public address. Firewall rules or cloud security groups allow only known clients, known administrators and other brokers to reach them. This reduces who can even attempt to connect, which shrinks the number of events your audit log must explain.

Layer 2: encryption in transit

TLS scrambles traffic so that anyone listening on the network sees only noise. Turn it on for client-to-broker traffic and for broker-to-broker traffic. Rotate certificates before they expire, and treat an expired certificate as an outage-in-waiting, because a broken audit path is a compliance incident.

Layer 3: proving identity

Choose one strong method and use it consistently. Mutual TLS gives each client a certificate. SASL with SCRAM gives each client a salted, challenge-protected password. OAuth tokens let you plug into a company identity provider so that people and services come from a central directory. Kerberos is common in older enterprises. Whatever you pick, the identity name that Kafka sees becomes the principal written into every authorization record, so name them meaningfully, such as the service name and environment.

Layer 4: permissions

ACLs bind principals to operations on resources. A few subtleties matter for audit work:

• Super users are principals that bypass ACL checks entirely. Keep that list very short, and alert whenever it changes.
• Denials override allows, which is useful for placing a hard block on a sensitive topic.
• Consumer groups are resources too. Without controlling them, a reader could join with an unexpected group name.
• Prefixed rules make it easy to protect every topic starting with an audit prefix, but a careless prefix can match more than you intend, so test it.

Layer 5: quotas

Quotas limit how much traffic a client may send or read. They protect the audit topic from a buggy service that suddenly floods it, which could push out other producers or fill disks. A quota breach is also a useful alert signal, since a service suddenly shouting is often either a bug or an attack.

Layer 6: the surrounding services

Kafka rarely runs alone. Kafka Connect exposes a web management interface that can start, stop and reconfigure connectors, so it must require authentication and be reachable only from trusted networks. The same goes for the schema registry, monitoring dashboards and any tool that can read audit topics. Attackers love the forgotten side door more than the main gate.

Layer 7: secrets and keys

Passwords, certificates and encryption keys belong in a secrets manager, never in configuration files stored in plain view or in shared chat channels. Rotate them on a schedule and log every read of a key.

Audit logging must never become the reason a payment is slow or a disk is full. A little arithmetic up front prevents nasty surprises later.

Do the storage math first

Suppose your systems produce 5,000 audit events every second, and the average event is about one kilobyte. That is roughly five megabytes per second, or about 430 gigabytes per day before compression. Audit events are text-like and repetitive, so compression often shrinks them several times over; assume roughly 100 gigabytes per day after compression as a planning figure. With three copies, that becomes about 300 gigabytes of broker disk per day. Keep 90 days on the brokers and you need around 27 terabytes. These are illustrative numbers, not a promise for your data, but the pattern is clear: broker retention gets expensive fast, which is why the tiers of Chapter 13 exist.

Choose partitions with care

More partitions mean more parallel writers and readers. But there is a catch specific to audit data. If you increase the number of partitions later, the mapping from key to partition changes, so new events for an account may land in a different partition than its old ones, which weakens the per-account ordering you designed. It is also impossible to reduce partitions. So estimate growth generously and start with more partitions than you need today. A modest surplus costs little, while a redesign is painful.

Batching and compression

Producers can wait a few milliseconds to fill larger batches, which raises throughput and cuts network use. Audit pipelines are rarely latency-sensitive in the millisecond sense, since an auditor does not care whether an event arrived in 5 or 50 milliseconds. So lean toward larger batches and efficient compression. Just keep the delay small enough that alerts on dangerous actions still fire quickly.

Do not let the audit path block the business

With the outbox pattern, the only cost the application pays is one extra small row written inside a transaction it was already making. The relay, Kafka and every downstream reader work in the background. If a reader slows down, the business flow does not.

Plan for peaks

Audit traffic follows business traffic. A shopping site sees spikes on sale days; a bank sees them at month end. Size for the peak, keep spare disk headroom, and load-test with realistic bursts. Watch consumer lag during tests, because that is where an undersized archive connector reveals itself.

Many teams run Kafka on Amazon Managed Streaming for Apache Kafka, known as MSK. It removes much of the machine-level work and pairs nicely with other AWS services that solve the compliance puzzle.

flowchart LR
  subgraph W["Workload account"]
    S["Application services"] --> M["Amazon MSK across three zones"]
    M --> C["MSK Connect S3 sink"]
  end
  subgraph A["Audit account"]
    B["S3 bucket with Object Lock"]
    K["KMS keys for encryption and signing"]
    Q["Athena for investigator queries"]
  end
  C --> B
  K --> B
  B --> Q
    

The pieces and their jobs

Why two accounts

The most valuable design choice in the diagram is the boundary between the workload account and the audit account. Engineers who run the applications and even the Kafka cluster can send data into the audit bucket, but they cannot delete it, change its lock, or use the signing keys. Only a small security group controls the audit account. This is separation of duties made physical.

Points to remember

• Use a bucket policy that allows only the connector’s role to add objects, and denies delete operations to everyone.
• Enable versioning and Object Lock when the bucket is created; the lock setting is a deliberate, planned decision.
• Choose the retention period from the longest applicable rule, and apply legal holds when needed.
• Turn on CloudTrail data events for the audit bucket, so that every access to it is recorded.
• Encrypt with keys whose policies are managed by the security team.

A design proves its worth when something goes wrong. Here is an illustrative story that shows every layer helping.

The situation

On Monday morning, the finance team notices that the bank details of a supplier were changed last Friday night, and a large payment was then sent to the new account. Nobody remembers making the change. The security team opens an investigation.

What made this possible

Notice that no step required luck. Events were ordered because of the key choice in Chapter 5. Nothing was missing because of the outbox and reconciliation. The evidence was trusted because of the locked archive and signatures. And the story about the stolen account was clear because failed and denied actions had been recorded, not just the successes.

Controls that are never tested tend to fail on the day they are needed. This chapter shows how to test your audit trail and how to be ready for the auditor’s visit.

Four kinds of testing

Map every requirement to a control and to evidence

Auditors think in three columns: what is required, what control you have, and what proof you can show. Build that table yourself, before they ask.

Keep the evidence fresh

Screenshots taken once a year age badly. Automate the collection: configuration exports on a schedule, verifier results stored automatically, access review sign-offs recorded in a ticket system. When the auditor arrives, your job becomes handing over a folder rather than scrambling for screenshots.

People and process matter as much as tools

Name an owner for the audit pipeline. Write a short runbook for the most likely incidents: connector down, disk filling, integrity check failing, lost key access. Rehearse the runbook. Train developers on what belongs in an audit event and what must never be logged. Most audit failures in real life come from missing ownership, not missing technology.

An audit pipeline that fails quietly is worse than none, because everyone believes they are covered. Monitoring turns silent failures into loud ones.

The health signals of Kafka itself

The completeness question: did every event arrive?

Health signals tell you the machinery is running. They do not tell you that nothing was lost. For that, use count reconciliation. Each stage of the pipeline counts how many events it saw in each short time window, and the counts are compared. If the source counted 10,000 events between 10:00 and 10:05 and the archive holds 9,998, you know two events are missing and exactly when to look.

This idea has a long history in the Kafka world. The original LinkedIn paper on Kafka described a monitoring scheme in which producers periodically publish counts so that consumers can check that no data was lost. Uber later built and described a system named Chaperone to audit its Kafka pipelines end to end, counting messages in time buckets at every tier to detect loss, duplication and delay.

Sequence numbers and heartbeats

Two more tricks catch gaps. Give each source a running sequence number; if a consumer sees 41 then 43, event 42 is missing. And have every producing service emit a regular heartbeat audit event, such as one per minute. A silent service that should be busy, or a missing heartbeat, is an alert by itself.

Alerts that matter to compliance

• Any change to audit topic settings, retention or ACLs.
• Any use of delete-records on an audit topic.
• Sudden drops in audit event volume from any source.
• Archive lag beyond your agreed limit.
• Failures of the integrity verifier.
• Repeated denied access attempts against the audit topics.

Here is how the ideas above show up in practice. Some examples describe publicly known systems, and others are illustrative scenarios that show how a typical team would apply the patterns.

Patterns are proven recipes; anti-patterns are recipes for trouble. Knowing both saves months of painful learning.

Patterns worth adopting

Anti-patterns to avoid

No design is free. Here is what you gain and what you pay for when you build compliance audit logging on Kafka.

Decisions you will have to make

If you are starting today, do not try to build everything at once. Here is a sensible order that delivers value early and adds protection step by step.

A rollout plan you can follow

The most common mistakes

• Treating audit logs like debug logs. They need stricter format, retention and protection.
• Forgetting failed and denied actions. Refusals often tell the most important stories.
• Relying on Kafka alone for immutability. Add locked storage and signatures.
• Sharing credentials. Accountability disappears the moment two people use one login.
• Never testing restore and retrieval. An archive you cannot read is not an archive.
• Ignoring privacy law. Long retention and personal data need an erasure plan such as crypto-shredding.
• Letting the schema drift. Undocumented changes make old records unreadable.
• Skipping the meta-audit. Changes to audit settings and keys must themselves be recorded elsewhere.

A quick self-check before your next audit

Ask your team these questions. Any answer that is “I am not sure” points to your next piece of work.

• Can we name every system that must produce audit events, and prove each one does?
• If a broker, a zone or the connector fails tonight, can we show that no events were lost?
• Who can delete or shorten an audit topic, and would we be alerted the moment they tried?
• Could we produce a verified record from three years ago within one working day?
• Are personal fields protected, and do we know how an erasure request would be handled?
• When did we last run a tamper drill, an access review and a restore drill, and where is the evidence?

Quick answers to the questions that come up most when teams start this journey.

Audit logging for compliance is less about clever technology and more about earning trust that can be checked. Kafka gives you a strong foundation. The rest is careful design.