The building blocks every serious MSK architecture is assembled from.

Amazon MSK is a managed control plane wrapped around genuine, unmodified Apache Kafka brokers — the wire protocol, the partition and replication model, and the broker binary itself are the same Kafka software you’d run yourself, which matters because every Kafka client library, tuning parameter, and operational technique developed anywhere in the open-source ecosystem applies directly to MSK without translation.

Producers, acknowledgments, and delivery semantics

A producer’s `acks` setting is one of the most consequential single configuration values in the entire platform. `acks=0` fires records without waiting for any broker confirmation — fastest, but a broker failure can silently lose data. `acks=1` waits for the partition leader to write the record locally, a middle ground that still risks loss if the leader fails before followers replicate it. `acks=all` (equivalent to `acks=-1`) waits for every in-sync replica to acknowledge the write, the strongest durability guarantee Kafka offers, at the cost of added latency per write. Advanced producer configuration also governs idempotence (preventing duplicate writes from producer retries) and transactions (atomic writes across multiple partitions), together forming the basis of Kafka’s exactly-once semantics when paired with a transactional consumer.

Cluster deployment topology and networking

An MSK cluster is deployed into subnets you choose within a VPC, with each broker occupying a specific Availability Zone — a decision that directly determines fault-tolerance boundaries later, since replicas should be spread across the same AZs the brokers themselves occupy. Client access is typically routed through private subnets with security groups restricting inbound connections to known producer and consumer sources, and for cross-VPC or cross-account access, AWS PrivateLink or VPC peering extends that same private connectivity model without exposing broker endpoints to the public internet at all. This networking layer is easy to treat as a one-time setup task, but it directly shapes later decisions around multi-region DR and cross-account topic sharing covered in later chapters.

Configuration as a first-class concept

MSK exposes both cluster-level configuration (applied to every broker — default replication factor, log retention defaults, compression type) and per-topic configuration overrides (a specific topic’s own retention, cleanup policy, or replication factor). Advanced operators keep a small, well-understood set of cluster-level defaults and override only where a specific topic’s requirements genuinely diverge — a compacted changelog topic needing `cleanup.policy=compact` while the cluster default remains `delete`, for instance — rather than scattering ad hoc per-topic overrides that make the cluster’s actual behavior hard to reason about as topic count grows.

What actually happens inside a broker when a message is produced and consumed.

Each MSK broker is an Amazon EC2 instance (for Provisioned clusters) running the Kafka broker process against dedicated Amazon EBS storage, with AWS handling patching, replacement, and the underlying instance lifecycle while leaving broker-level Kafka configuration under your control. When a producer sends a record, it is routed — by a partitioner, either key-based hashing or round-robin — to a specific partition’s current leader broker, appended to that partition’s on-disk log segment, and only then replicated to follower brokers holding the other replicas.

flowchart LR
    A[Producer] --> B[Partition Leader Broker]
    B --> C[Append to Local Log Segment]
    C --> D[Replicate to Follower Brokers]
    D --> E[Followers Join ISR
Once Caught Up]
    E --> F[Leader Acknowledges
per acks Setting]
    F --> G[Consumer Group
Reads via Offset]
        

Log segments and the commit log model

A partition is not one giant file — it is a sequence of immutable log segments, each rolled over once it reaches a configured size or age. This segmented design is what makes both retention (deleting old segments wholesale) and compaction (rewriting a segment to keep only the latest value per key) efficient operations rather than expensive in-place rewrites of a single enormous file. Sequential, append-only disk writes are also why Kafka achieves such high write throughput on ordinary disks — it deliberately avoids random-access writes, leaning on the operating system’s page cache and sequential I/O patterns that spinning and solid-state disks alike handle far faster than scattered writes.

The shift from ZooKeeper to KRaft

Historically, Kafka relied on an external ZooKeeper ensemble to store cluster metadata and elect a controller broker responsible for partition leader assignment. KRaft mode replaces this with a Raft-based consensus protocol run directly among a subset of the brokers themselves, removing an entire separate distributed system from the operational footprint, shortening controller failover time, and raising the practical ceiling on how many partitions a cluster can host — ZooKeeper’s metadata-propagation overhead was a well-known scaling limit in very large, high-partition-count clusters, and KRaft was designed specifically to remove it.

Leader election mechanics

When a partition leader broker fails, the controller selects a new leader from the partition’s current ISR set — never from an out-of-sync replica, unless unclean leader election is explicitly enabled, which trades consistency for availability by allowing an out-of-date replica to become leader and potentially lose the most recently written, unreplicated records. This single configuration flag is one of the sharper trade-offs in Kafka operations: leaving unclean election disabled (the safer default) means a partition can become temporarily unavailable if every in-sync replica is down, rather than silently losing data by promoting a stale one.

Page cache, zero-copy, and why Kafka is fast

Two implementation details explain most of Kafka’s throughput characteristics. First, brokers rely heavily on the operating system’s page cache rather than managing their own in-process memory cache for recently written data — a design choice that survives broker restarts gracefully, since the OS page cache is repopulated naturally by subsequent reads without any broker-side warm-up logic. Second, when serving a fetch request for data that’s already sequential on disk, Kafka uses a zero-copy transfer (`sendfile`) that moves bytes directly from the page cache to the network socket without ever copying them through the broker’s own application memory — a detail invisible to any client, but the direct explanation for why Kafka can serve read-heavy consumer traffic at throughput levels that would otherwise require far more broker CPU.

Broker-to-broker replication traffic

Replication is not free — every write accepted by a leader must be re-sent to every follower replica, meaning a partition with replication factor 3 generates roughly three times the raw network traffic of an unreplicated write. This replication traffic competes with client produce and fetch traffic for the same broker network interfaces, which is precisely why broker instance and network throughput sizing must account for replication overhead explicitly, not just the raw external produce/consume volume a capacity estimate might start from.

Tracing a record from production through retention, compaction, and eventual deletion.

Compacted topics and the changelog pattern

A log-compacted topic retains only the most recent record for each key indefinitely, rather than expiring by age — effectively turning the topic into a durable, replayable changelog of “current state per key” rather than a pure event stream. This is the exact mechanism Kafka Streams and ksqlDB use internally to maintain state stores that can be rebuilt from scratch by replaying the compacted topic, and it’s a deliberate architectural choice worth making explicitly for any topic representing entity state (a user profile, an account balance) rather than a pure event log.

Tiered storage for long retention

MSK’s tiered storage capability separates “hot,” recently written data kept on broker-local storage from older data automatically offloaded to lower-cost object storage, while keeping both tiers queryable through the same Kafka API a client already uses. This decouples retention length from broker disk sizing — a topic can retain months or years of history economically without provisioning enormous, expensive broker-local storage to hold data that is rarely read once it ages past a few days.

Exactly-once semantics across the pipeline

“Exactly-once” is one of the most frequently misunderstood terms in the Kafka ecosystem — it describes a specific, narrow guarantee (no duplicate or lost writes within a Kafka-to-Kafka transactional pipeline using idempotent producers and read-committed consumers), not a blanket promise that covers arbitrary external side effects. A consumer that reads a message exactly once from Kafka but then calls an external, non-transactional API as a side effect can still, on a crash-and-retry, call that external API twice — the exactly-once guarantee applies to the Kafka read/write path itself, and any external effect layered on top needs its own idempotency handling (a unique request ID, a database upsert instead of insert) to inherit that same guarantee end to end.

Idempotent producers versus transactions

Idempotent producers (enabled per-producer) prevent duplicate writes caused by the producer’s own retries after a transient network error, using sequence numbers the broker tracks per producer session — a narrower, lighter-weight guarantee than full transactions. Kafka transactions extend this to atomic writes spanning multiple partitions or topics, letting a producer publish to several destinations such that either all of them commit or none do, which is the building block Kafka Streams uses internally to guarantee that reading from one topic, transforming, and writing to another all succeed or fail together as a single unit.

Where MSK genuinely wins, and where experienced teams weigh alternatives.

MSK versus its closest AWS alternatives

Amazon Kinesis Data Streams overlaps heavily with MSK in use case — both are durable, ordered, replayable streaming platforms — but Kinesis uses a proprietary API and shard-based scaling model rather than Kafka’s protocol and partition model, making it the simpler choice for teams with no existing Kafka investment and a preference for deeper native AWS service integration, while MSK is the natural choice for teams with existing Kafka expertise, open-source ecosystem tooling (Kafka Streams, ksqlDB, Debezium), or a need for Kafka protocol compatibility with external systems. Amazon SQS and SNS, by contrast, are not real substitutes for either — they are message-queue and pub/sub primitives without Kafka’s ordered, replayable, long-retention log model, better suited to simple task distribution or fan-out notification than to event-sourcing or stream-processing architectures.

These trade-offs also shift over a pipeline’s lifetime, not just at initial selection. A team that starts on MSK Serverless for a low-volume, exploratory pipeline may genuinely outgrow it once throughput becomes large and predictable enough that Provisioned mode’s finer control and typically lower cost per unit of sustained throughput starts to matter — revisiting that choice periodically against real usage, rather than treating the original deployment mode as permanent, is a healthy operational habit rather than a sign the original decision was wrong.

The levers that actually move throughput and latency at scale.

Kafka’s scalability model is horizontal at the partition level — a topic’s maximum consumer parallelism is capped by its partition count, since a single partition can only be actively read by one consumer within a given consumer group at a time. Under-provisioning partitions is the single most common scalability mistake, because increasing partition count later is possible but re-shuffles key-to-partition mapping, which can break any consumer logic relying on a stable partition assignment for a given key.

Broker sizing and storage throughput

In MSK Provisioned, broker instance type and EBS volume throughput jointly determine sustained write and replication throughput per broker — a common oversight is sizing brokers for steady-state average load rather than peak replication traffic during a broker replacement or an unclean-shutdown recovery, when the cluster must simultaneously catch up multiple under-replicated partitions at once, a scenario that can saturate network and disk throughput far beyond ordinary daily peaks.

Consumer-side fetch tuning

Just as producers batch outgoing writes, consumers can tune how much data they request per fetch via `fetch.min.bytes` and `fetch.max.wait.ms` — a consumer configured to wait for a larger minimum batch before returning reduces the number of round trips to the broker at the cost of marginally higher per-record latency, generally a good trade for high-throughput batch-style consumers but a poor one for latency-sensitive consumers reacting to individual events in near real time. `max.poll.records` caps how many records a single poll call returns to the application, which matters directly for consumer group health: a consumer that pulls a very large batch and then takes too long processing it risks missing its next heartbeat and being evicted from the group as presumed dead, triggering an unnecessary rebalance.

Consumer lag and rebalancing cost

Consumer lag — the gap between the latest produced offset and a consumer group’s committed offset — is the primary real-time health signal for any Kafka-based pipeline, and unlike a queue’s simple depth metric, it must be tracked per partition, since one slow partition (often the result of a skewed key distribution) can hide behind an otherwise healthy group-average lag figure. Consumer group rebalancing — triggered whenever a consumer joins, leaves, or is considered dead by a missed heartbeat — briefly pauses partition consumption across the entire group while partitions are reassigned; incremental cooperative rebalancing, now the default in modern Kafka clients, minimizes this pause by only reassigning the specific partitions that need to move rather than stopping the whole group and reassigning everything from scratch.

MSK Serverless auto-scaling behaviour

MSK Serverless removes broker and storage capacity planning by auto-scaling throughput capacity per topic based on observed traffic, at the cost of per-topic throughput quotas and a narrower set of tunable broker-level configurations than Provisioned mode offers — a reasonable trade for unpredictable or bursty workloads, but a genuine constraint for very high, sustained, or highly customized throughput requirements where Provisioned mode’s explicit control remains the better fit.

Producer batching and compression trade-offs

A producer’s `linger.ms` and `batch.size` settings control how long it waits, and how much data it accumulates, before sending a batch of records to the broker — a small `linger.ms` minimizes per-record latency at the cost of smaller, less efficient batches, while a larger value trades a small amount of added latency for meaningfully higher throughput and lower per-record overhead, particularly valuable at high message volume. Compression (typically `lz4` or `zstd` in modern deployments) is applied to the entire batch before sending, which is why batching and compression settings interact directly — a larger batch compresses more efficiently, further amplifying the throughput benefit, making the two settings worth tuning together rather than in isolation.

Cruise Control and automated partition rebalancing

As topics and partitions accumulate over a cluster’s lifetime, partition distribution across brokers can drift away from balanced, leaving some brokers carrying disproportionately more leader partitions (and therefore more client traffic) than others. Cruise Control, an open-source tool commonly paired with Kafka clusters including MSK, continuously analyzes broker resource utilization and automatically proposes or executes partition reassignments to rebalance load — turning what would otherwise be a manual, risky, and rarely-performed maintenance task into a continuous, low-friction background process.

Vertical versus horizontal scaling ceilings

Two distinct scaling questions get conflated too often in practice: how much a single broker can handle, and how much a cluster as a whole can handle. Upgrading broker instance type (vertical scaling) raises the ceiling for a single broker’s CPU, memory, and network throughput, which helps when individual brokers are resource-constrained but does nothing to relieve a partition-count ceiling on consumer parallelism. Adding more brokers (horizontal scaling) increases the cluster’s aggregate capacity and gives more places to spread partition replicas, but only pays off if partition count and key distribution are already designed to actually use that additional spread — a cluster with too few, too-large partitions gains little from more brokers, since the same handful of partitions simply move to a bigger pool of machines rather than genuinely parallelizing further.

Designing for broker failure as a routine event, not an emergency.

MSK clusters are deployed across multiple Availability Zones by design, with partition replicas distributed across those AZs so that a single AZ failure does not take a partition’s every replica down simultaneously — provided replication factor and rack-awareness are configured correctly, which is a deployment-time decision, not something the platform silently guarantees regardless of configuration.

Disaster recovery topology choices

Active-passive DR replicates a primary cluster’s topics into a standby cluster in a second region via MM2, with consumers failing over only during a declared disaster — simpler to reason about, at the cost of idle standby capacity. Active-active DR runs both regions serving live traffic simultaneously with bidirectional MM2 replication, offering better resource utilization and lower failover time, at the cost of needing to handle potential topic-offset divergence and careful conflict-avoidance in how producers and consumers are partitioned across regions. Neither topology is a checkbox MSK enables automatically — both require deliberate MM2 configuration and testing.

Offset translation across MirrorMaker 2

A detail that catches many teams off guard the first time they fail over: MM2-replicated topics do not preserve the exact same offsets as the source cluster, because the target cluster’s topic is populated independently rather than being a byte-for-byte mirror at the log level. MM2 solves this with an offset-translation mechanism (checkpoint topics) that maps a consumer group’s committed offset on the source cluster to the equivalent position on the target, letting a failed-over consumer resume from the right point rather than either reprocessing everything from the beginning or skipping records it never actually read. Testing this translation path — not just the data replication itself — before a real disaster is what separates a DR plan that works from one that only looks like it works.

Authentication, authorization, and encryption as three independent controls.

Choosing an authentication mechanism

IAM access control lets you authenticate and authorize Kafka clients using the same IAM roles and policies already governing the rest of an AWS account, avoiding a separate credential system entirely — the natural default for teams already deeply invested in IAM-based access management. SASL/SCRAM authenticates against usernames and passwords stored in AWS Secrets Manager, a better fit for clients or third-party systems that cannot assume an IAM role directly. Mutual TLS authenticates using client certificates, common in environments with existing PKI infrastructure or strict client-identity requirements that predate or sit outside AWS IAM. All three can technically coexist on one cluster, though most production deployments standardize on one primary mechanism to keep ACL and policy management tractable.

ACL granularity and least privilege

Kafka ACLs (or their IAM-policy equivalent) can scope permissions down to specific operations (Read, Write, Describe, Create) on specific topics or consumer groups, and advanced deployments use this to enforce genuine least privilege — a producer service account granted Write access to only the topics it legitimately publishes to, and a consumer service account granted Read access to only its own consumer group and the topics it’s meant to process, rather than broad cluster-wide access granted for convenience during initial setup and never revisited.

Network-level isolation as a second line of defense

Authentication and ACLs answer “who is this and what can they do,” but network placement answers a different question: “can they even reach the cluster in the first place.” Placing broker endpoints exclusively in private subnets, with security groups scoped to only the specific CIDR ranges or security groups of legitimate producer and consumer workloads, means a credential leak alone is not sufficient to reach the cluster — an attacker would also need network-level access, a meaningfully higher bar than a single leaked secret. This layering — network isolation plus authentication plus authorization plus encryption — is deliberate defense in depth, not redundant overlap, since each layer independently closes a different failure scenario the others don’t cover.

Spotting a struggling cluster before consumers start falling behind visibly.

The metrics that predict incidents before they happen

Under-replicated partitions — partitions where at least one replica has fallen out of the ISR — is arguably the single most important broker-level metric to alarm on, since a sustained rise usually signals a broker under resource pressure, a network partition, or a disk nearing capacity, all of which erode the cluster’s failure tolerance well before an actual outage occurs. Request queue time and produce/fetch latency percentiles (rather than averages, which hide tail latency) reveal broker-side pressure building up under load. On the client side, consumer lag trending upward over a sustained window — not a brief spike during a deploy — is the clearest signal that a consumer’s processing rate can no longer keep pace with production rate, and needs either more consumer parallelism or an investigation into what got slower downstream.

Cost observability alongside performance

MSK Provisioned bills primarily for broker instance hours and provisioned storage, meaning cost visibility is largely a capacity-planning exercise — tracking whether actual broker CPU, network, and storage utilization justify the currently provisioned instance count and type, and right-sizing down when a cluster is meaningfully over-provisioned relative to its real traffic. MSK Serverless bills instead on throughput actually consumed, which shifts the relevant cost question from “are we over-provisioned” to “which topics or producers are driving throughput growth month over month” — a distinction worth tagging and tracking per topic or per owning team, the same way DPU-hours are tracked per job in a batch-processing pipeline, so that a cost increase can be traced to a specific cause rather than showing up only as an unexplained total.

Alerting on combined signals, not isolated metrics

A single metric crossing a threshold is often a weak signal in isolation — broker CPU briefly spiking during a deploy, or a momentary blip in produce latency, happens routinely without indicating a real problem. A composite alarm that fires only when multiple related signals move together — under-replicated partitions rising at the same time as broker disk utilization approaches capacity, for instance — is a substantially stronger indicator of a genuine, developing incident than watching either metric alone, and reduces the alert fatigue that comes from paging on every isolated, ultimately benign metric fluctuation.

Treating cluster and topic configuration as versioned infrastructure.

Mature MSK deployments declare cluster configuration, subnet and security group placement, and even topic definitions as infrastructure-as-code via CloudFormation, CDK, or Terraform, deploying changes through the same CI/CD pipeline as the rest of the platform rather than creating topics ad hoc through a console or a one-off script that nobody remembers running six months later.

sequenceDiagram
    participant Dev as Developer
    participant Repo as Git Repository
    participant CI as CI/CD Pipeline
    participant IaC as CloudFormation/Terraform
    participant MSK as Amazon MSK
    Dev->>Repo: Push cluster config + topic definitions
    Repo->>CI: Trigger pipeline on merge
    CI->>CI: Validate topic configs, run consumer contract tests
    CI->>IaC: Apply cluster/topic/ACL changes
    IaC->>MSK: Create/Update resources via API
    MSK-->>CI: Deployment confirmation
        

MSK Connect for managed source and sink pipelines

MSK Connect runs standard Kafka Connect connectors — Debezium for change-data-capture from relational databases, S3 sink connectors for landing topic data into a data lake, and dozens of community connectors — as a managed service, handling connector worker scaling and infrastructure without a self-managed Connect cluster. This is frequently the bridge between MSK and the rest of a data platform: a Debezium source connector streaming database changes into MSK topics, and a sink connector landing those same topics into S3 for later processing by a service like AWS Glue, forming a continuous change-data pipeline from operational database to analytical data lake.

Schema governance across producers and consumers

AWS Glue Schema Registry, when integrated with MSK producers and consumers, enforces that every message conforms to a registered, versioned schema before it’s accepted — rejecting or flagging a producer that tries to publish a payload violating the current schema contract, which prevents the exact class of silent, gradual schema drift that otherwise surfaces only when a downstream consumer crashes on an unexpected field months later.

Testing topic and connector changes before production

Because a topic’s configuration (partition count, cleanup policy, replication factor) is expensive or impossible to change cleanly after significant data has accumulated, mature pipelines validate proposed topic configurations against a staging cluster — or at minimum, a staging topic on a shared non-production cluster — running realistic consumer contract tests that would catch a partition-count-driven key remapping issue or a cleanup-policy mismatch before it reaches a cluster carrying real traffic. The same applies to MSK Connect connector configuration changes, which can silently alter delivery semantics (at-least-once versus exactly-once) depending on connector-specific settings that are easy to get wrong on a first pass.

Shapes that scale, and shapes that quietly rot.

The habits that quietly prevent the most expensive incidents.

A quieter mistake worth naming directly: under-documenting topic ownership and schema contracts. As the number of topics grows past a few dozen, “which team owns this topic, and what schema changes are safe to make” becomes a real governance question, and a lightweight, enforced convention — schema registry entries with clear ownership metadata, topic naming that encodes the owning domain — pays for itself the first time two teams need to coordinate a breaking schema change.

Equally worth flagging: treating client library defaults as production-ready without review. Default values for `session.timeout.ms`, `max.poll.interval.ms`, and retry/backoff settings are chosen by the client library maintainers to work reasonably across a wide range of unknown use cases, not tuned for any specific pipeline’s actual message size, processing time, or failure tolerance — a consumer doing heavier per-record processing than the default `max.poll.interval.ms` assumes will be evicted from its group repeatedly, and no amount of otherwise-correct application logic will fix a problem rooted entirely in an unreviewed timeout default.

Where these patterns show up outside a tutorial environment.

Financial services and payment platforms commonly use MSK’s strong replication guarantees — replication factor 3, `min.insync.replicas=2`, `acks=all` — as the backbone for transaction event streams feeding fraud-detection and ledger-reconciliation systems, where the cost of a lost or duplicated event is measured directly in money, making the durability trade-offs covered earlier a first-order design decision rather than a tuning afterthought. E-commerce and retail platforms frequently use the change-data-capture bridge pattern to stream inventory and order updates out of operational databases into MSK, feeding real-time search indexing and recommendation systems without those systems ever touching the production database directly. Media and telecommunications companies processing high-volume clickstream, call-detail, or viewing-event data lean on MSK’s partition-based horizontal scaling to absorb traffic that varies enormously between peak and off-peak hours, often pairing MSK Serverless’s auto-scaling behaviour with downstream Kafka Streams or ksqlDB applications for real-time aggregation.

Logistics and IoT-heavy operators, ingesting continuous telemetry from vehicles, sensors, or warehouse equipment, often hit the partition-skew anti-pattern described earlier head-on when a naive partitioning key (device type, rather than device ID) concentrates traffic onto a handful of partitions — a lesson usually learned once, expensively, before key-cardinality planning becomes a standard part of every new topic’s design review. Healthcare and life-sciences platforms handling event streams derived from clinical or research systems typically pair MSK’s mTLS or IAM authentication with strict per-topic ACLs, reflecting the same least-privilege principle covered in the security chapter, since a single shared “read everything” consumer credential is rarely acceptable once patient-adjacent data flows through the same cluster as less sensitive operational events.

Ad-tech and marketing-analytics platforms, reconciling bid, impression, and click events arriving from many advertising partners at extremely high volume and with inconsistent schemas, lean on the schema registry enforcement pattern from the deployment chapter for much the same reason a Glue-based batch pipeline would — a single malformed partner feed should be rejected or quarantined at the producer boundary rather than silently corrupting a shared topic that dozens of downstream reporting consumers depend on. Gaming platforms processing player-event telemetry at extremely bursty volume (a launch day, a live event) are a natural fit for MSK Serverless’s automatic throughput scaling described in the performance chapter, letting infrastructure expand transparently during a spike without the advance capacity planning a fixed Provisioned cluster would require.

No — MSK runs the standard Apache Kafka broker software and wire protocol. Any Kafka client library, and most open-source Kafka ecosystem tools, work against MSK exactly as they would against a self-managed cluster.

Provisioned suits predictable, high, or highly customized throughput needing fine-grained broker configuration. Serverless suits bursty or unpredictable workloads where avoiding capacity planning outweighs having narrower configuration control and per-topic throughput quotas.

Yes — increasing partitions changes the key-to-partition hash mapping for a key-partitioned topic, meaning records with the same key can land on a different partition than before, which can break consumer logic relying on a stable partition assignment per key.

With unclean leader election disabled (the safer default), that partition becomes unavailable for both reads and writes until an in-sync replica returns, rather than promoting a stale out-of-sync replica and risking silent data loss.

No — cross-region replication requires explicitly configuring MirrorMaker 2 between a source and target cluster. Within a single MSK cluster, replication across Availability Zones in one region is automatic, provided replication factor is configured appropriately.

Aggregate CPU or broker-level metrics can look healthy while a single skewed partition overwhelms one consumer instance’s processing capacity — check per-partition lag, not just the group’s average, to find a hidden hotspot.

No — compaction is appropriate for topics representing current entity state (a changelog), where only the latest value per key matters. Pure event or activity streams, where every historical event has value, should use time- or size-based deletion instead.

It’s the simplest choice for clients already running under IAM roles, but SASL/SCRAM or mTLS may fit better for external clients, legacy systems, or organizations with existing certificate infrastructure that doesn’t map cleanly onto IAM.

MM2 does not preserve identical offsets between source and target clusters — it relies on checkpoint-based offset translation. If that translation path wasn’t tested and validated ahead of time, a failover can resume from an unexpected position.

Generally no — consumers end up filtering every message client-side, and a schema change to one event type risks breaking consumers of an unrelated event type sharing the topic. Split genuinely distinct event types into their own topics instead.

Amazon MSK’s real advanced-level value isn’t “Kafka without servers to patch” — it’s the combination of genuine, protocol-compatible Kafka semantics (partitions, replication, ISR, consumer groups) with AWS-managed infrastructure lifecycle, native IAM and KMS integration, and companion services like MSK Connect and Glue Schema Registry that would otherwise require significant self-managed tooling. None of the internals covered here — replication and ISR mechanics, partition-based parallelism limits, consumer rebalancing behaviour, or the acks/durability trade-off — are MSK inventions; they are Apache Kafka’s own architecture, which is precisely why expertise built on MSK transfers directly to any other Kafka deployment, and vice versa. The teams that operate MSK well treat partition key design and durability settings as first-class architectural decisions made deliberately up front, monitor per-partition lag and under-replicated partitions as leading indicators, and enforce schema and ACL discipline before a cluster accumulates dozens of loosely governed topics. Just as importantly, they revisit those early decisions periodically against real production data rather than treating a launch-day configuration as permanent — partition counts, deployment mode, and DR topology all deserve a second look as traffic patterns and organizational needs evolve.