Amazon MSK — Running Kafka Without Running Kafka
A deep, chapter-by-chapter walkthrough of Amazon Managed Streaming for Apache Kafka — how it is built, how data actually moves through it, and how to run it safely at production scale.
Imagine a busy train station where thousands of trains arrive every second, each one carrying passengers who need to reach hundreds of different platforms without ever bumping into each other. Someone has to build the tracks, keep the signals working, make copies of every train’s manifest in case a train breaks down, and make sure the station never closes even if one control room goes dark. That “someone” is Apache Kafka in the world of data. Amazon MSK is what happens when AWS agrees to be the station master, the maintenance crew, and the night-shift electrician — so that engineering teams can focus on which trains to send, not on keeping the tracks bolted down. This tutorial goes chapter by chapter through the intermediate-level machinery of Amazon MSK: its architecture, its internal behavior, its failure modes, and the decisions that separate a stable production deployment from a 3 a.m. incident call.
1Core Concepts, Refreshed
Before going deep into MSK itself, a few Kafka concepts need to be sharp in your mind — not from scratch, but tightened up at an intermediate level.
Topics, Partitions, and Offsets
A Kafka topic is a named stream of records — think “orders”, “clickstream”, or “payment-events”. Every topic is split into partitions, and each partition is an append-only, ordered log. Order is only guaranteed within a single partition, never across the whole topic. Each record inside a partition gets a monotonically increasing offset, which is simply its position number in that log.
Producers, Consumers, and Consumer Groups
Producers write records into partitions, usually choosing a partition based on a key (so that all events for the same customer, for example, land in the same partition and stay in order). Consumers read records back out, and consumer groups let many consumer instances split the work of reading a topic, with each partition assigned to exactly one consumer within a group at a time.
A topic is a whole library. Each partition is one specific shelf in that library, and the offset is the position of a book on that shelf. Two readers (consumers) in the same book club (consumer group) will never be handed the exact same shelf to read — they split the shelves between them so the whole library gets read faster.
Brokers and the Cluster
A Kafka broker is a single server process that stores partitions and serves reads and writes for them. A cluster is a group of brokers working together. In self-managed Kafka, you provision these servers, patch their operating systems, tune their disks, and babysit their coordination layer. Amazon MSK removes exactly that layer of work — AWS provisions, patches, and replaces the broker machines, while you keep full control over topics, partitions, and how your applications produce and consume.
Broker
A server that stores partitions and answers client requests. An MSK cluster is a fleet of these.
Partition
An ordered, append-only log — the true unit of parallelism in Kafka.
Replica
A copy of a partition kept on a different broker for durability.
Consumer Group
A set of consumers sharing the work of reading a topic’s partitions.
2Architecture & Components
An MSK cluster is more than “some Kafka brokers in the cloud” — it is a specific arrangement of AWS-managed pieces working together.
At the center sits the broker fleet, spread across multiple Availability Zones inside your VPC. Each broker runs on an EC2-class instance type that you choose (for example the kafka.m5 or kafka.m7g family), backed by Amazon EBS storage. AWS handles the underlying host, the Kafka software installation, and OS-level patching, while you control instance sizing, storage size, and networking.
For metadata and controller coordination, older MSK clusters use Apache ZooKeeper, a separate ensemble of nodes that tracks broker membership, topic configuration, and partition leadership. Newer MSK clusters can run in KRaft mode, where Kafka’s own built-in controller quorum replaces ZooKeeper entirely, removing an extra moving part and simplifying the architecture.
graph TD
A[Producer Application] -->|writes records| B[MSK Broker - AZ 1]
A -->|writes records| C[MSK Broker - AZ 2]
A -->|writes records| D[MSK Broker - AZ 3]
B |replication| C
C |replication| D
D |replication| B
E[Consumer Application] -->|reads records| B
E -->|reads records| C
E -->|reads records| D
F[Controller Quorum - KRaft or ZooKeeper] -.metadata.-> B
F -.metadata.-> C
F -.metadata.-> D
Client applications reach the cluster through bootstrap broker endpoints, which MSK provides for you in several flavors: plaintext, TLS-encrypted, IAM-authenticated, and SASL/SCRAM-authenticated. Networking runs entirely inside your VPC using elastic network interfaces attached to each broker, so security groups and subnet routing apply just as they would to any other private AWS resource.
Provisioned Mode
You choose the broker count, instance type, and storage per broker up front, and MSK keeps that fleet running. Best when your throughput is predictable and you want fine control over cost and performance.
Serverless Mode
MSK Serverless automatically provisions and scales capacity based on your traffic, charging per throughput and storage actually used. Best for spiky or hard-to-predict workloads where capacity planning is a burden.
3Internal Working
Underneath the managed surface, MSK brokers behave exactly like standard Apache Kafka brokers — which is precisely the point.
Every partition has one broker acting as its leader and zero or more brokers holding follower replicas. All writes and reads for that partition go through the leader; followers continuously pull new records from the leader to stay in sync. The set of replicas that are fully caught up with the leader is called the In-Sync Replica set, or ISR.
Think of the leader broker as the one official notebook a teacher writes homework assignments into. Follower brokers are students who copy every new line into their own notebooks as fast as possible. If a student falls too far behind copying, the teacher stops trusting that student’s notebook as a backup until they catch up.
When a producer sends a record, MSK’s underlying Kafka protocol acknowledges it based on the producer’s acks setting: acks=0 does not wait for any confirmation, acks=1 waits only for the leader, and acks=all waits until every broker in the ISR has the record. This single setting is one of the most important durability knobs an intermediate Kafka user controls.
If the leader broker fails, the controller quorum (KRaft controllers, or ZooKeeper working with a Kafka controller broker) detects the failure and promotes one of the in-sync followers to be the new leader. Clients are redirected automatically once they refresh their metadata, which normally takes a few seconds.
Replication is not the same as backup. Replicas protect against a broker or AZ failure happening right now — they do not protect against a bad message, a schema mistake, or an accidental topic deletion, because the mistake gets replicated everywhere just as fast as the correct data would.
4Data Flow & Lifecycle
A record’s life inside MSK follows a predictable path from the moment it is produced to the moment it disappears.
Production
A producer serializes a record, picks a partition (often by hashing a key), and sends it to that partition’s leader broker.
Commit & Replication
The leader appends the record to its log; followers in the ISR pull and append the same record to their own copies.
Acknowledgement
Depending on the acks setting, the producer receives confirmation once the required number of brokers hold the record.
Consumption
Consumers poll the partition starting from their last committed offset and process records at their own pace.
Retention or Compaction
Old records are eventually deleted based on a time or size retention policy, or — for compacted topics — replaced by only the latest value per key.
Retention is configured per topic, not globally, which means a “raw-clickstream” topic can be set to expire after three days while a “customer-profile-changelog” topic is compacted forever, always keeping only the newest record for each customer key. This flexibility is what lets the same MSK cluster serve very different data lifecycle needs at once.
| Retention Type | Behavior | Typical Use Case |
|---|---|---|
| Time-based | Deletes records older than a configured duration | Event streams, logs, metrics |
| Size-based | Deletes oldest records once a partition exceeds a byte limit | Fixed-storage buffering |
| Compacted | Keeps only the latest record per key, forever | Changelogs, state snapshots |
5Advantages, Disadvantages & Trade-offs
Choosing MSK over self-managed Kafka, or over an alternative like Amazon Kinesis, involves real trade-offs.
Advantages
- No manual broker provisioning, patching, or Kafka version upgrades to script by hand
- Native integration with IAM, VPC security groups, KMS encryption, and CloudWatch
- Full compatibility with the open-source Kafka protocol and client libraries
- Serverless option removes capacity planning almost entirely
- Multi-AZ replication is built into the default cluster layout
Disadvantages / Trade-offs
- Less low-level control than self-hosted Kafka (no custom broker patches, limited OS access)
- Provisioned mode still requires capacity planning and partition-count decisions
- Cross-region replication needs an add-on (MSK Replicator) rather than being automatic
- Cost can exceed self-managed Kafka at very large, steady-state throughput
6Performance & Scalability
Throughput in MSK is mostly a partitioning and instance-sizing problem, not a magic setting to flip.
Partitions are the unit of parallelism: a topic with one partition can only be written to and read from by one producer thread and one consumer at a time per broker, no matter how large your cluster is. Adding partitions lets more producers write in parallel and lets a consumer group spread work across more consumer instances, but partitions cannot easily be reduced later and too many small partitions increase per-broker overhead.
Partitions are like checkout lanes in a supermarket. More lanes let more shoppers check out at once, but if you open fifty lanes for ten shoppers, most cashiers just stand around doing nothing while still costing the store money.
Broker instance type and count set the ceiling for throughput and connection capacity, while EBS volume type and size affect sustained write and read speed, especially for workloads that fall behind and need to read from disk instead of from memory cache. MSK Serverless sidesteps most of this tuning by scaling storage and throughput automatically, at the cost of some pricing predictability.
Choose a partition key that spreads load evenly. A key like “customer_id” usually spreads well; a key like “country” can create a few enormous “hot” partitions if most of your traffic comes from one region.
7High Availability & Reliability
MSK’s default topology is built to survive the loss of a broker, or even an entire Availability Zone, without losing data.
A standard MSK cluster spans three Availability Zones, and Kafka’s replication mechanism spreads each partition’s replicas across different AZs whenever possible. Combined with a replication factor of three and min.insync.replicas set to two, a cluster can lose one full AZ and keep accepting writes without any data loss for topics using acks=all.
sequenceDiagram
participant P as Producer
participant L as Leader Broker (AZ-1)
participant F1 as Follower (AZ-2)
participant F2 as Follower (AZ-3)
P->>L: Write record (acks=all)
L->>F1: Replicate record
L->>F2: Replicate record
F1-->>L: Ack
F2-->>L: Ack
L-->>P: Write acknowledged (ISR satisfied)
Replication Factor
How many total copies of each partition exist. Three is the common production default.
min.insync.replicas
The minimum number of in-sync replicas required before a write with acks=all succeeds.
Unclean Leader Election
Whether an out-of-sync replica may become leader during an outage — usually disabled for durability.
Multi-AZ Placement
The default spread of brokers and replicas that lets a cluster tolerate an AZ-level failure.
8Security
MSK layers several independent controls, so a breach in one layer does not automatically mean a breach everywhere.
Encryption
Data at rest on broker storage is encrypted using AWS KMS keys, either AWS-managed or customer-managed for stricter key control. Data in transit between clients and brokers, and between brokers themselves, can be encrypted with TLS, which is strongly recommended for anything beyond local testing.
Authentication
MSK supports several authentication mechanisms side by side: mutual TLS with client certificates, SASL/SCRAM with usernames and passwords stored in AWS Secrets Manager, and IAM-based authentication that lets you reuse existing IAM roles and policies to control exactly which principal can produce to or consume from which topic.
Network Isolation
Because every broker sits inside your VPC, standard security group rules control which subnets and instances can even reach the cluster’s ports, well before any Kafka-level authentication is checked.
Problem
Leaving a cluster on plaintext, unauthenticated access “just for now” during development, and forgetting to switch it before go-live.
Why It’s Harmful
Anyone who can route traffic to the broker ports can read and write any topic, including sensitive production data, with no audit trail of who did what.
Correct Approach
Enable TLS and IAM or SASL/SCRAM authentication from the very first environment, including development, so security configuration is never a “later” task.
9Monitoring, Logging & Metrics
Kafka clusters fail quietly long before they fail loudly, so visibility into the right metrics matters as much as the architecture itself.
MSK publishes broker, topic, and partition-level metrics directly to Amazon CloudWatch by default, covering things like CPU usage, disk usage, network throughput, and under-replicated partitions. For teams already using Prometheus and Grafana, MSK’s Open Monitoring feature exposes the same underlying JMX metrics in a Prometheus-scrapable format, avoiding a separate metrics pipeline.
UnderReplicatedPartitions
Counts partitions where a follower has fallen out of the ISR — a key early-warning signal.
Consumer Lag
The gap between the latest offset and a consumer group’s committed offset — shows if consumers are falling behind.
CpuUser / CpuSystem
Broker-level CPU pressure, often the first sign a cluster needs more or larger brokers.
KafkaDataLogsDiskUsed
How full broker storage is — running out of disk can halt writes entirely.
Alert on consumer lag trends, not just absolute values — a lag that is steadily climbing over an hour is a much stronger warning sign than a brief spike that recovers on its own.
10Deployment & Cloud Integration
MSK is rarely used alone — most real systems connect it to other AWS services for ingestion, processing, and storage.
Clusters are created through the AWS console, CLI, CloudFormation, or Terraform, specifying VPC subnets, security groups, broker instance type, storage size, and the Kafka version. MSK Connect lets you run Kafka Connect connectors — for example streaming data into Amazon S3, or pulling change-data-capture events from a database — without managing separate connector infrastructure.
flowchart LR
A[Application Services] -->|produce events| B(Amazon MSK)
B -->|MSK Connect| C[Amazon S3]
B -->|stream processing| D[Kinesis Data Analytics / Flink]
B -->|consume| E[Lambda Functions]
D --> F[Amazon Redshift]
E --> G[DynamoDB]
For cross-region needs — disaster recovery, or serving data close to users on multiple continents — MSK Replicator continuously copies topics from one cluster to another, including their configuration and access control lists, without requiring you to build and operate custom replication tooling.
11Design Patterns & Anti-patterns
Certain patterns show up again and again in mature MSK deployments — and so do certain mistakes.
Event Sourcing with Compacted Topics
Using a compacted topic as the durable source of truth for an entity’s current state, replaying it to rebuild application state after a failure or a new service deployment.
Fan-out via Consumer Groups
Multiple independent consumer groups reading the same topic for entirely different purposes — one for real-time alerting, another for batch analytics — without the producer knowing or caring who is listening.
Dead Letter Topics
Routing records that repeatedly fail processing into a separate topic, so a single bad message cannot block an entire partition’s consumer indefinitely.
Problem
Treating Kafka topics as a request-response messaging system, where a producer waits synchronously for a specific consumer’s business-logic reply.
Why It’s Harmful
Kafka is built for durable, asynchronous, one-to-many streaming — forcing synchronous request-response patterns onto it fights the model and adds fragile complexity.
Correct Approach
Use Kafka for what it is good at — event distribution and durable logs — and use a purpose-built RPC or queue mechanism for true request-response interactions.
12Best Practices & Common Mistakes
Most production incidents involving MSK trace back to a handful of recurring oversights.
Right-size partitions upfront
Plan partition counts around expected peak throughput and consumer parallelism, since shrinking partitions later is disruptive.
Monitor disk headroom
A broker that runs out of disk space can stop accepting writes cluster-wide, not just for one topic.
Ignoring consumer lag
Slow consumers eventually cause records to be deleted by retention before they are ever read.
Skewed partition keys
A poorly chosen key can send most traffic to one partition, creating a bottleneck that more brokers cannot fix.
Setting acks=1 everywhere by default without understanding that a leader failure right after acknowledgement can silently lose that record before followers replicate it.
13Real-world & Industry Examples
The underlying Kafka model behind MSK powers some of the largest event-driven systems in the world.
Netflix
Uses Kafka-based streaming pipelines to move playback events, recommendation signals, and operational telemetry across thousands of microservices in near real time.
Uber
Relies on Kafka-style event streaming to coordinate trip state changes, pricing signals, and driver-rider matching across a highly distributed system.
Financial Services
Banks and payment processors commonly use managed Kafka clusters like MSK to stream transaction events into fraud-detection pipelines within milliseconds of a card swipe.
What these examples share is not the specific business domain, but the shape of the problem: many independent producers generating events, many independent consumers needing those events for different purposes, and a requirement that no event silently disappears along the way.
14Frequently Asked Questions
A few questions come up in nearly every team’s first serious MSK evaluation.
Yes — MSK runs the actual Apache Kafka broker software, so any client library or tool that speaks the Kafka protocol works against it without modification.
Choose Provisioned when throughput is steady and you want fine-grained cost and performance control; choose Serverless when traffic is unpredictable or the team prefers not to plan capacity manually.
Data loss is possible if replication settings are weak (for example, replication factor of one) or if acks is set too low for the durability the application actually needs — correctly configured, MSK is built to avoid loss during single-broker or single-AZ failures.
Kinesis is a proprietary AWS streaming service with its own API and shard model, while MSK gives you the open Kafka protocol and ecosystem — the right choice often depends on existing tooling and team familiarity rather than raw capability.
Yes, through MSK Replicator, which continuously copies topics, configuration, and access control between clusters in different AWS Regions.
15Summary and Key Takeaways
Amazon MSK takes the proven architecture of Apache Kafka — partitions, replication, leaders and followers, consumer groups — and removes the operational burden of running the broker fleet yourself. The Kafka concepts do not change; what changes is who is on call for the servers underneath them. Understanding partitioning, replication, acknowledgement settings, and monitoring signals is still entirely the team’s responsibility, because those are application-level and data-level decisions no managed service can make for you.
Key Takeaways
- Partitions drive parallelism — throughput scales with partition count and broker capacity, not by itself.
- Replication protects, backups restore — replicas guard against hardware failure, not against bad data or human error.
- acks and min.insync.replicas define durability — these two settings decide what “acknowledged” actually means for your data.
- Multi-AZ is the default safety net — a well-configured cluster survives the loss of an entire Availability Zone.
- Security is layered — VPC networking, authentication, and encryption each close a different gap, and none of them alone is enough.
- Consumer lag is the canary — it is usually the earliest visible sign that something downstream needs attention.
- Serverless trades control for simplicity — the right choice depends on how predictable your traffic really is.