Kafka For Intermediate
The concepts that separate "I can produce and consume messages" from "I can run Kafka reliably for a real system" — for readers already comfortable with Kafka basics.
This guide assumes you already understand topics, partitions, producers, consumers, and consumer groups. It moves into the configuration choices, reliability guarantees, and operational concerns that matter once Kafka is powering a real system with actual throughput, multiple teams, and production uptime expectations.
1Advanced Producer Configuration
Tuning how producers batch, retry, and confirm delivery.
Idempotent Producers
Prevents duplicate messages caused by producer retries after a network error, by having the broker deduplicate based on a producer ID and sequence number.
Batching and linger.ms
Producers group messages into batches before sending; linger.ms adds a small delay to allow more messages to accumulate, trading a little latency for significantly better throughput.
Compression (gzip, snappy, lz4, zstd)
Compressing batches before sending reduces network and storage usage; zstd generally offers the best compression ratio, while lz4 favors speed over ratio.
Retries and max.in.flight.requests
Controls how many unacknowledged requests a producer can have outstanding at once — setting this above 1 without idempotence enabled risks message reordering on retry.
Producer Buffer Memory
The memory pool a producer uses to hold messages awaiting send — if this fills up because the broker can’t keep up, the producer blocks or throws an exception depending on configuration.
2Advanced Consumer Configuration
Controlling exactly how and when consumers read and acknowledge data.
Manual Offset Commits
Disabling auto-commit and committing offsets explicitly after successful processing avoids the common bug of marking a message “processed” before your application logic actually finished handling it.
auto.offset.reset
Determines where a consumer starts reading when no committed offset exists — earliest reads from the beginning of the topic, latest skips straight to new messages only.
Consumer Poll Loop & max.poll.interval.ms
Consumers must call poll() regularly to be considered alive by the group coordinator — exceeding max.poll.interval.ms between polls (e.g., due to slow processing) triggers a rebalance, evicting that consumer from the group.
Static Group Membership
Assigns a persistent identity to a consumer instance, avoiding unnecessary rebalances during brief restarts or network blips that would otherwise trigger a full group rebalance.
Consumer Lag Monitoring
Tracks the gap between the latest offset produced and the offset a consumer group has actually processed — growing lag is the primary early warning sign of a consumer falling behind.
3Partitioning & Data Distribution Strategies
Making sure load is actually spread evenly across a topic.
Custom Partitioners
Lets you override Kafka’s default partitioning logic with business-specific routing rules, useful when the default key-hash behavior doesn’t match your data distribution needs.
Partition Count Planning
Partition count sets the upper bound on consumer parallelism within a group and generally can’t be decreased later — under-provisioning limits scale, over-provisioning increases broker overhead and rebalance time.
Skewed Partitions / Hot Partitions
Occurs when a small number of keys receive disproportionate traffic, overloading the partitions they map to regardless of overall partition count — the same fundamental problem as hotspotting in any partitioned system.
Sticky Partitioner
The modern default partitioning strategy for unkeyed messages, which batches multiple messages to the same partition before switching, improving batching efficiency over strict round-robin.
4Kafka Streams — Intermediate Concepts
Processing data continuously as it flows through Kafka.
KStream vs KTable
A KStream represents an unbounded sequence of individual events; a KTable represents the latest state per key, similar to a continuously updating table derived from a stream.
Stateful vs Stateless Operations
Stateless operations (map, filter) process each record independently; stateful operations (aggregations, joins) require maintaining state across multiple records, backed by a local state store.
Windowing
Groups stream data into time-based buckets (tumbling, hopping, sliding) for aggregations like “count events per 5-minute window,” essential for any time-based analytics on a stream.
State Stores
Local, embedded key-value stores (typically RocksDB) that back stateful operations, with changes also written to an internal Kafka topic for fault-tolerant recovery.
Stream-Table Joins
Joins an incoming stream of events against the current state of a table (like enriching an order event with current customer data), a common pattern for real-time enrichment.
5Schema Management
Keeping producers and consumers agreeing on message format as systems evolve.
Avro, Protobuf, and JSON Schema
The three most common serialization formats used with Kafka; Avro and Protobuf are compact binary formats well-suited to schema evolution, while JSON Schema trades some efficiency for human readability.
Schema Evolution
The practice of changing a message’s structure over time (adding fields, deprecating others) without breaking producers or consumers still using an older or newer version of the schema.
Backward/Forward/Full Compatibility
Backward compatibility means new schemas can read old data; forward compatibility means old schemas can read new data; full compatibility requires both — the choice affects which schema changes are actually allowed.
Schema Registry Enforcement
Configuring compatibility rules in Schema Registry causes it to reject schema changes that would break existing consumers, catching incompatible changes at registration time rather than in production.
6Kafka Connect In Depth
Moving data between Kafka and external systems reliably.
Source vs Sink Connectors
Source connectors pull data from an external system into Kafka; sink connectors push data from Kafka out to an external system — most integration needs map to one of these two patterns.
Connector Configuration & Tasks
A single connector configuration can spawn multiple parallel tasks to scale throughput, with Kafka Connect distributing tasks across available worker nodes automatically.
Single Message Transforms (SMTs)
Lightweight, inline transformations (like renaming a field or masking sensitive data) applied to messages as they pass through a connector, without needing a separate stream-processing job.
Dead Letter Queues in Connect
Routes messages that fail to be processed (due to serialization errors, for example) to a separate topic instead of blocking the entire connector pipeline.
7Reliability & Delivery Guarantees
Understanding exactly what Kafka promises, and what you must still handle yourself.
Idempotence In Depth
Enabled via enable.idempotence=true, this only guarantees no duplicates from producer-side retries — it does not, by itself, guarantee exactly-once processing across a full read-process-write pipeline.
Transactions in Kafka
Allow a producer to write to multiple partitions/topics atomically, so either all writes in a transaction are visible to consumers or none are — the foundation for exactly-once stream processing.
Exactly-Once Semantics (EOS) In Depth
Achieved in Kafka Streams by combining idempotent producers, transactions, and offset commits into the same atomic unit — meaningful only within Kafka-to-Kafka pipelines, not automatically extending to external systems.
min.insync.replicas
Combined with acks=all, this sets the minimum number of in-sync replicas that must acknowledge a write for it to succeed — the actual lever that trades availability for durability guarantees.
8Monitoring & Operations
Knowing whether a Kafka cluster is actually healthy.
Key Kafka Metrics (Broker, Producer, Consumer)
Under-replicated partitions, request latency percentiles, and consumer lag are the three metric categories most predictive of an unhealthy cluster before it becomes an outage.
Consumer Lag Monitoring Tools
Tools like Burrow or Kafka’s own consumer group CLI expose per-partition lag, which is more actionable than an aggregate lag number that can hide a single stuck partition.
JMX Metrics
Kafka exposes detailed internal metrics via JMX, which monitoring tools (Prometheus JMX Exporter, Datadog) scrape to build dashboards and alerts.
Log Retention Monitoring
Tracking actual disk usage against configured retention settings prevents brokers from unexpectedly running out of disk space when traffic volume grows faster than retention was originally planned for.
9Security Basics
Controlling who can connect to a cluster and what they can do.
Authentication (SASL)
Verifies the identity of clients connecting to Kafka, commonly via SASL/SCRAM or SASL/GSSAPI (Kerberos) in enterprise environments.
Authorization (ACLs)
Once authenticated, ACLs determine which specific operations (read, write, create) an identity is allowed to perform on which topics or consumer groups.
Encryption in Transit (TLS)
Encrypts data moving between clients and brokers, and between brokers themselves, protecting against network-level eavesdropping.
Encryption at Rest
Typically handled at the disk or filesystem level (rather than natively by Kafka), protecting stored log segments if the underlying storage is compromised.
10Multi-Cluster & Data Replication
Moving data reliably between separate Kafka clusters.
MirrorMaker 2
Kafka’s built-in tool for replicating topics between clusters, commonly used for disaster recovery, geo-replication, or migrating data to a new cluster.
Cluster Linking
A more modern alternative to MirrorMaker (available in some Kafka distributions) that replicates data while preserving original offsets, simplifying failover compared to MirrorMaker’s offset translation.
Cross-Datacenter Replication Use Cases
Common patterns include active-passive (one cluster serves traffic, the other is a standby) and active-active (both clusters serve traffic with careful conflict handling).
11Performance Tuning Basics
Getting the throughput and latency your workload actually needs.
Throughput vs Latency Trade-offs
Larger batches and higher linger.ms improve throughput but increase per-message latency — the right setting depends on whether your workload favors bulk efficiency or real-time responsiveness.
Batch Size Tuning
batch.size sets the maximum bytes per batch on the producer side — too small wastes the benefit of batching, too large can increase memory pressure and latency for low-traffic partitions.
Broker Configuration Tuning (num.io.threads, etc.)
Thread pool settings for network and disk I/O should scale with available CPU cores and disk parallelism — undersized thread pools are a common, easily overlooked throughput bottleneck.
Disk and Network Considerations
Kafka’s performance is fundamentally sequential-disk-I/O-bound — fast, dedicated disks (not shared with other heavy I/O workloads) and sufficient network bandwidth between brokers matter more than raw CPU power in most cases.
12Kafka in Production Architectures
Design patterns for using Kafka as a core piece of system architecture.
Event Sourcing with Kafka
Stores every state change as an immutable event in a topic, letting the current state of an entity be reconstructed by replaying its event history rather than storing state directly.
CQRS with Kafka
Separates the write path (commands producing events) from the read path (materialized views built by consuming those events), often implemented using Kafka Streams or Connect to build read-optimized stores.
Kafka as a System of Record
Using Kafka’s durable, replicated log as the authoritative source of truth for an event history, rather than just a transient message bus — a design choice with real implications for retention and replication settings.
Designing Topic Naming Conventions
Consistent naming (like domain.entity.event) makes topics discoverable and self-documenting as the number of topics in an organization grows into the hundreds.
Key Takeaways
- Idempotent producers and transactions together are what make exactly-once semantics possible — neither alone is sufficient.
- Schema Registry with enforced compatibility rules is what actually prevents producer/consumer format mismatches at scale, not just documentation discipline.
- Consumer lag is the single most important operational metric — it’s the earliest, clearest signal that something downstream is falling behind.
- Kafka Streams’ KStream/KTable duality is the foundation for almost every real-time processing pattern, from windowed aggregations to stream-table enrichment.
- Kafka’s performance is fundamentally about sequential disk I/O and batching — most tuning decisions trace back to one of these two levers.
- Event sourcing and CQRS are natural architectural fits for Kafka’s durable log model, but they require deliberate retention and replication decisions to serve as a true system of record.