Before touching shards or partition keys, it helps to understand the exact gap Kinesis fills between “a database” and “a message queue.”

A traditional database is built to answer “what is true right now?” A queue like Amazon SQS is built to hand off individual jobs to exactly one worker, once, and then forget about them. Neither is designed for a third job: preserving the exact order and full history of events as they happened, and letting many independent readers replay that history at their own pace. That third job — durable, ordered, replayable event streaming — is what Amazon Kinesis was purpose-built to do, and it is the same job that Apache Kafka does outside of AWS.

AWS launched Kinesis in 2013 as a managed alternative to running and babysitting a self-hosted Apache Kafka cluster. Under the Kinesis umbrella today sit four related but distinct services: Kinesis Data Streams (the raw, low-level streaming engine this guide focuses on), Kinesis Data Firehose (a fully managed delivery pipe into storage and analytics targets), Managed Service for Apache Flink (formerly Kinesis Data Analytics, for real-time stream processing with SQL or Flink), and Kinesis Video Streams (for time-encoded media like camera feeds). Netflix, for example, uses Kinesis-style streaming internally to track playback events across hundreds of millions of devices in near real time, feeding recommendation engines and quality-of-service dashboards within seconds of a viewer pressing play.

It’s worth being precise about what “real time” actually buys an engineering team, because the phrase gets overused. Real time does not mean instantaneous in some magical sense — it means the gap between an event happening in the world and a system reacting to it shrinks from hours or minutes down to single-digit seconds or even milliseconds. That shrinking gap is what turns a fraud detection system from “we noticed the stolen card was used three days later” into “we declined the third fraudulent swipe before it cleared.” It’s what turns a ride-hailing ETA from “recalculated every two minutes” into “recalculated continuously as the car moves.” Kinesis exists because that gap matters commercially, not just technically, and AWS built it as a managed service precisely because operating the alternative — a self-hosted Apache Kafka cluster with its own ZooKeeper or KRaft coordination layer, broker patching schedule, and partition rebalancing logic — is a genuine, ongoing engineering tax that many teams would rather not pay.

There’s also a subtler reason Kinesis matters at the intermediate level specifically: once you’ve written your first producer and consumer and gotten data flowing, the interesting problems stop being “how do I connect to Kinesis” and start being “how do I keep this stream healthy as traffic triples, as I add a fifth consumer, as a partition key I chose eighteen months ago turns out to be lopsided.” Those are architecture and operations problems, not API-syntax problems, and they are exactly what the rest of this guide is built to address.

Four ideas — shards, partition keys, sequence numbers, and retention — govern almost every design decision you will make with Kinesis. Get comfortable with how they interact before going further.

Shards are the unit of throughput and parallelism in a Kinesis Data Stream. Each shard guarantees a fixed capacity: up to 1 MB/second or 1,000 records/second of writes, and up to 2 MB/second of reads. A stream is not one big pipe — it is a bundle of these fixed-capacity lanes. If you need 5 MB/second of write throughput, you need at least five shards, and the way records are spread across those shards is not automatic guesswork; it is controlled entirely by you through the partition key.

A partition key is a string you attach to every record you write — a customer ID, a device ID, a session ID. Kinesis runs that string through an MD5 hash function and uses the resulting hash to deterministically route the record to exactly one shard. This is the single most important design decision in any Kinesis system, because it decides whether your load spreads evenly across shards or piles up on one.

Every record that lands in a shard is stamped with a sequence number — a unique, strictly increasing identifier within that shard that establishes the exact order in which records arrived. This is what lets a consumer say “give me everything after sequence number X” and get a reliable, gapless replay.

Finally, retention defines how long a record stays available for reading after it is written — the default is 24 hours, extendable up to 8,760 hours (365 days) with Kinesis Data Streams’ long-term retention tier. Unlike SQS, where a consumed message disappears, a record in Kinesis is simply read — it stays in the stream for every other consumer and for replay until its retention window expires.

It helps to think of sequence numbers and retention as two different axes of the same idea: sequence numbers answer “in what order did this happen relative to everything else on this shard,” while retention answers “for how long can I still ask that question.” A consumer that crashes and restarts three hours later doesn’t need to guess where it left off — it stores the last sequence number it successfully processed as a checkpoint, and on restart asks Kinesis for an iterator starting immediately after that sequence number. As long as three hours is inside the retention window, every record is still sitting there waiting, untouched, exactly as it was written. This is also what makes Kinesis genuinely useful for reprocessing: if you discover a bug in how your analytics job interpreted a field, and the bug has been live for six hours, you can simply create a new consumer application, point it at a shard iterator from six hours ago, and replay the exact same events through corrected logic — something a traditional queue-based system cannot do once a message has already been consumed and deleted.

One more subtlety worth internalizing: a shard’s hash key range is fixed the moment the shard is created, and it never changes for the lifetime of that shard. When you split or merge shards later, you aren’t editing an existing shard’s range — you’re closing it and creating new shards with new ranges that together cover the same space. This immutability is precisely what lets Kinesis guarantee ordering: because a given partition key’s hash always falls in the same place, and a shard’s range is never silently reassigned, there is never a window where two different shards could simultaneously believe they own the same key.

A working Kinesis system is really three separate concerns wired together: producers, the stream itself, and consumers — each with its own responsibilities and failure modes.

Producers are anything that writes records into the stream: application servers using the AWS SDK’s PutRecord or PutRecords calls, the Kinesis Producer Library (KPL) for high-throughput batching, mobile SDKs, IoT devices, or upstream services like CloudWatch Logs and AWS IoT Core writing directly. A producer’s only real job is to pick a good partition key and hand off the record; everything about durability and ordering downstream is Kinesis’s problem from that point on.

The stream is the managed, distributed core: a named collection of shards, each independently replicated synchronously across three Availability Zones before a write is acknowledged. This is why a successful write response from Kinesis is a strong durability guarantee, not an optimistic one.

Consumers read from shards, and they come in two flavors that matter a great deal for design. A standard consumer polls a shard using GetRecords and shares that shard’s 2 MB/second read throughput with every other standard consumer reading the same shard. An enhanced fan-out (EFO) consumer gets its own dedicated 2 MB/second pipe per shard, pushed to it automatically via HTTP/2, independent of how many other consumers exist.

flowchart LR
    subgraph Producers
      P1[Application Servers]
      P2[Kinesis Producer Library]
      P3[IoT / Mobile Clients]
    end
    P1 -->|PutRecord| S
    P2 -->|PutRecords batch| S
    P3 -->|PutRecord| S
    subgraph S[Kinesis Data Stream]
      SH1[Shard 1]
      SH2[Shard 2]
      SH3[Shard 3]
    end
    SH1 --> KCL1[KCL Consumer App]
    SH2 --> KCL1
    SH3 --> KCL1
    SH1 -->|Enhanced Fan-Out| EFO1[Lambda Function]
    SH2 -->|Enhanced Fan-Out| EFO1
    KCL1 --> DDB[(DynamoDB Checkpoint Table)]
    EFO1 --> DL[(Data Lake / S3)]
        

Sitting alongside these three layers is the Kinesis Client Library (KCL), an open-source framework that handles the unglamorous but critical work of load-balancing shards across multiple worker instances and tracking checkpoints — usually in a DynamoDB table — so that a consumer restarting after a crash knows exactly where it left off, per shard.

The KCL’s job becomes clearer once you imagine running five instances of the same consumer application against a stream with twelve shards. Nobody manually decides “instance 3 handles shards 7 through 9” — the KCL does this automatically through a lease-based coordination model. Each shard has exactly one “lease” that a single worker instance holds at a time, tracked as a row in the DynamoDB checkpoint table alongside that worker’s last processed sequence number. If an instance crashes or is terminated during a scale-in event, its leases become stale after a timeout, and the remaining instances detect this and redistribute the orphaned shards among themselves — no human intervention, no lost checkpoint, no shard left unread. Add a sixth instance, and the KCL will rebalance leases again so the new capacity actually gets used rather than sitting idle.

This lease table is also why KCL-based consumers need IAM and network access to DynamoDB in addition to Kinesis itself — a detail that trips up many first deployments, since the DynamoDB table is created automatically on first run and easy to overlook until a permissions error surfaces it. Teams running the KCL in production typically provision this table with on-demand DynamoDB capacity, since checkpoint write volume scales directly with shard count and consumer polling frequency, not with the size of the data itself.

Understanding what happens between a PutRecord call and a consumer reading that record explains almost every operational quirk you will encounter.

When a producer calls PutRecord, Kinesis first hashes the partition key into a 128-bit value using MD5. Every shard owns a contiguous, non-overlapping range of that 128-bit hash space — this is called the shard’s hash key range. Kinesis looks up which shard’s range contains the hashed value and routes the record there. This is deterministic: the same partition key always maps to the same shard, for as long as the shard exists unchanged.

Once routed, the record is appended to that shard’s log and synchronously replicated across three Availability Zones before Kinesis returns a success response containing the assigned sequence number. This three-AZ replication is what allows Kinesis to survive an entire zone outage without losing already-acknowledged data.

On the read side, a consumer does not “pull” data the way it might poll a folder for new files. It calls GetRecords with a shard iterator — a pointer into a specific position in a specific shard’s log — and Kinesis returns whatever records exist from that position onward, plus a fresh iterator to use on the next call. Standard consumers must poll roughly once every second per shard by default; enhanced fan-out consumers instead have Kinesis push records to them over a persistent HTTP/2 connection, cutting the typical 200 millisecond polling latency down to about 70 milliseconds.

Batching also happens internally on the write path in a way that’s easy to miss. When a producer calls PutRecords with, say, 200 records in a single request, Kinesis does not write all 200 to one shard — it evaluates each record’s partition key independently and can scatter that one API call’s records across every shard in the stream. The response comes back as an array with a per-record success or failure status and the specific shard each record landed on, which is why well-written producer code always checks this per-record result rather than assuming the whole batch succeeded or failed together. A batch can be “successful” as an API call while three of its two hundred records were individually throttled and need to be retried.

The 128-bit hash key range itself is worth visualizing concretely: it runs from 0 to 2^128 minus 1, an astronomically large space, and AWS divides it into contiguous, non-overlapping chunks — one per shard — when the stream is created. A four-shard stream, for instance, might split that space into four roughly equal quarters, each owned by one shard. “Roughly equal” is doing real work in that sentence, because if your partition keys are not evenly distributed across the hash function’s output, an equal division of the hash space does not translate into an equal division of real traffic — which is exactly the trap covered later in the anti-patterns chapter.

Tracing one record from birth to expiry ties the previous two chapters together into a single timeline.

Notice what is absent from this lifecycle: there is no step where a record is “deleted because it was consumed.” This is the fundamental behavioral difference from a queue, and it’s what allows Kinesis to support use cases like feeding both a real-time fraud-detection Lambda and a batch analytics job from the exact same stream, with neither one affecting the other’s view of the data.

At the intermediate level, picking the right Kinesis service matters as much as understanding any single one of them.

A useful rule of thumb: reach for Firehose when the destination is a data lake or warehouse and you don’t need custom per-record logic. Reach for Data Streams directly when you need multiple independent consumers, sub-200ms latency, or full control over checkpointing and replay. Reach for Managed Service for Apache Flink when the processing itself — windowed aggregations, joins across streams, anomaly detection — is complex enough to need a real stream-processing engine rather than a simple consumer application. Samsung SmartThings, for instance, routes IoT telemetry from millions of connected devices through Kinesis Data Streams for real-time device-state processing, while separately using Firehose to archive the same raw telemetry into S3 for long-term analytics.

It’s worth noting these services are not mutually exclusive tiers where you “graduate” from one to the next — they’re building blocks meant to be composed. Firehose itself can be configured to read directly from a Data Streams stream as one of its sources, meaning a very common intermediate-level pattern is to stand up a single Data Streams stream for the demanding, low-latency, multi-consumer real-time path, and then attach a Firehose delivery stream to that same source purely for durable archival, with zero custom code required for the archival half. Similarly, Managed Service for Apache Flink applications read from and can write back to Data Streams streams, letting you chain a raw ingestion stream into a Flink job that enriches or aggregates records, then republishes the results to a second, cleaner stream for downstream consumers — a pattern covered in more detail in the design patterns chapter ahead.

Cost profile is another axis that shapes this decision in practice. Data Streams bills primarily by shard-hour (or by data volume in on-demand mode) regardless of whether every shard is fully utilized, which rewards careful capacity planning. Firehose bills by the volume of data ingested and delivered, with no shard concept at all, which makes it simpler to reason about for bursty or unpredictable archival workloads. Teams frequently underestimate Firehose’s buffering behavior, too — it does not deliver every record instantly, but batches records up to a configurable buffer size or time interval (whichever triggers first, commonly 60 seconds to several minutes), which is perfectly fine for a data lake destination but would be far too slow for a real-time fraud check, reinforcing why Data Streams and Firehose are usually paired rather than chosen as either-or alternatives.

Kinesis is a strong default for AWS-native streaming, but it is not free of real costs and constraints.

Scaling Kinesis is really the act of changing how many shards exist and how the hash key range is divided among them.

Kinesis offers two capacity modes. Provisioned mode requires you to set the shard count explicitly and pay per shard-hour, giving you predictable cost but requiring you to forecast load. On-demand mode automatically scales shard count based on observed throughput over the previous 30 days, trading some cost predictability for zero manual capacity planning — a good fit for unpredictable or spiky workloads like a flash-sale event.

Scaling a provisioned stream up or down is done through two operations. Splitting a shard divides one shard’s hash key range into two child shards, each inheriting half the range and therefore half the traffic — used when a shard is consistently hot. Merging two adjacent shards combines their ranges back into one — used to reduce cost when traffic drops. Both operations are online: the stream keeps accepting writes throughout, though the shards involved briefly become CLOSED to new writes while children take over.

Reads scale differently: standard consumers share a shard’s fixed 2 MB/second read budget among however many of them are attached, so adding a fifth standard consumer to a busy shard slows down the other four. Enhanced fan-out sidesteps this entirely by giving each registered consumer its own dedicated 2 MB/second per shard — Zynga, for example, uses enhanced fan-out consumers to let its game-analytics pipeline and its live-ops alerting system both read the same in-game event stream at full speed, independently.

Switching a stream between provisioned and on-demand mode is itself a low-friction operation that can be done at most a few times per 24-hour period, which matters for teams who want on-demand’s simplicity during unpredictable launch windows but prefer provisioned’s cost predictability during steady-state operation. Under the hood, on-demand mode is not magic — AWS is still running the same split and merge machinery described above, just triggering it automatically based on a rolling window of observed throughput rather than waiting for a human to notice a CloudWatch alarm. This means on-demand mode reacts to sustained trends rather than instantaneous spikes; a genuinely sudden, massive burst (for example, a viral social media moment sending ten times normal traffic within one minute) can still briefly hit throttling before the automatic scaling catches up, which is why extremely spike-prone workloads sometimes still benefit from deliberately over-provisioning a provisioned-mode stream ahead of a known event, such as a product launch or a live sports broadcast.

Reshaping a stream also has a practical limit worth planning around: a single stream can only double its total shard count once every 24 hours through split operations, a safeguard that prevents runaway scaling from a misconfigured application and forces genuinely large, planned capacity increases to happen incrementally rather than all at once.

Kinesis’s durability guarantees come from AWS’s infrastructure; your reliability guarantees come from how you build checkpointing and retry logic on top of it.

Because every write is synchronously replicated across three Availability Zones before acknowledgment, a single AZ failure does not lose acknowledged data and does not stop the stream from accepting new writes — Kinesis simply continues serving from the remaining healthy replicas. This is a stronger default guarantee than many self-managed systems provide out of the box.

The reliability gap that remains is entirely on the consumer side. If a KCL worker crashes mid-processing without having checkpointed, it will re-read records from its last saved checkpoint on restart — meaning consumer logic must be designed to tolerate at-least-once delivery and handle duplicate records gracefully, typically through idempotent processing (for example, using the record’s unique sequence number to deduplicate before writing to a downstream database).

It’s worth being explicit about what Kinesis’s Availability Zone replication does and does not protect against. It protects against a data center losing power, a network partition isolating one AZ, or hardware failure taking down the physical machines a shard’s data happens to live on — none of these should cause acknowledged data loss or stream unavailability, because a healthy replica in one of the other two AZs is always available to serve reads and accept the next writes. What it does not protect against is an entire AWS Region becoming unavailable, since Kinesis streams are inherently regional resources. Teams with strict disaster-recovery requirements — often driven by regulatory obligations in banking or healthcare — commonly run parallel Kinesis streams in a second Region, with producers dual-writing or a cross-Region replication mechanism keeping the two in sync, accepting the added cost and complexity as the price of surviving a full regional outage.

The other reliability dimension worth naming is consumer-side error handling. A KCL-based or Lambda-based consumer that throws an unhandled exception while processing a batch will, depending on configuration, either skip the bad record (risking silent data loss for that one record) or retry it indefinitely (risking a “poison pill” record blocking all subsequent records on that shard forever, since ordering within a shard must be preserved). Mature Kinesis consumers handle this with a dead-letter pattern: after a bounded number of retry attempts, the problematic record is written to a separate error stream or an S3 bucket for manual inspection, and processing advances past it rather than stalling the entire shard indefinitely.

Protection

You may likeAmazon EBS→

10Security

Kinesis security is built from the same three layers used across AWS: identity, network, and encryption — applied specifically to stream and shard-level access.

Identity and access is governed by IAM policies scoped down to specific actions (PutRecord, GetRecords, SubscribeToShard) and even specific streams or consumer ARNs, so a producer service can be granted write-only access with no ability to read the stream it feeds.

Encryption at rest is available using AWS KMS, encrypting every record transparently before it touches disk, with AWS managing key rotation if you use an AWS-managed key, or you managing rotation policy yourself with a customer-managed key for stricter compliance needs. Encryption in transit is enforced by default, since all Kinesis API calls run over HTTPS/TLS.

For network isolation, VPC endpoints (via AWS PrivateLink) let producers and consumers running inside a VPC talk to Kinesis without traffic ever traversing the public internet — a common requirement in financial services and healthcare workloads where PCI-DSS or HIPAA compliance is in scope.

A detail that separates intermediate designs from beginner ones is scoping IAM permissions at the resource level rather than granting blanket Kinesis access. A well-scoped producer policy allows PutRecord and PutRecords on one specific stream ARN and nothing else — it cannot list other streams, read from any shard, or delete the stream it’s writing to. A well-scoped enhanced fan-out consumer policy goes a level deeper, scoping to the specific registered consumer ARN returned when that consumer subscribes, since SubscribeToShard permissions are consumer-specific rather than stream-wide. This granularity matters most in multi-tenant systems, where one team’s producer service should have no ability whatsoever to read data belonging to another team’s stream, even though both streams might live in the same AWS account.

For regulated industries, Kinesis’s compliance posture is generally strong: it falls under AWS’s shared responsibility model and supports workloads requiring PCI-DSS, HIPAA (when configured within a covered account and paired with a signed Business Associate Addendum), and SOC 2 compliance, provided encryption at rest, VPC-only network paths, and least-privilege IAM policies are all correctly configured — the service being compliant-capable does not mean a careless configuration is automatically compliant.

Kinesis exposes its health through CloudWatch, and a handful of metrics matter far more than the rest.

A growing IteratorAgeMilliseconds is usually the first sign of trouble in any Kinesis system — it means records are piling up faster than a consumer can process them. Left unaddressed, it can eventually reach the retention period and cause silent data loss as unread records expire, which is why most production teams alarm on this metric well before it approaches the retention window.

Beyond the four headline metrics, mature Kinesis deployments typically layer on three additional practices. First, enabling enhanced (shard-level) monitoring surfaces per-shard metrics rather than stream-level aggregates, which is essential for spotting a single hot shard hiding inside an otherwise healthy-looking average — a stream-level throughput graph can look perfectly calm while one specific shard is being throttled repeatedly. Second, AWS CloudTrail logs every management-plane API call against a stream — who created it, who changed its shard count, who updated its retention period — which is invaluable during incident review when a stream’s behavior suddenly changes and nobody remembers making a change. Third, teams commonly build a small CloudWatch dashboard combining IteratorAgeMilliseconds, incoming and outgoing bytes per shard, and throttling counts side by side, since these three signals together tell a more complete story than any one of them alone: rising iterator age with flat incoming bytes points to a slow consumer, while rising iterator age with spiking incoming bytes points to genuine traffic growth outrunning current shard capacity.

Kinesis rarely operates alone — its real value shows up in how easily it connects to the rest of the AWS ecosystem.

The most common consumer in modern architectures is AWS Lambda, which can be configured with an event source mapping directly against a Kinesis stream: Lambda automatically polls shards, batches records, retries on function errors, and can even use enhanced fan-out for lower latency, all without you deploying or managing a single server. For heavier stateful processing, applications run the Kinesis Client Library on EC2, ECS, or Fargate, using DynamoDB for checkpoint coordination.

Firehose can itself act as a consumer of a Data Streams stream, providing a low-effort path to land raw or lightly transformed records into S3, Redshift, or OpenSearch Service on a buffered schedule, without writing any consumer code at all. This “Streams feeds Firehose” pattern is extremely common: use Data Streams for the low-latency, multi-consumer real-time path, and let Firehose quietly handle the durable archival path from the same source.

Choosing between a Lambda-based consumer and a KCL application running on EC2, ECS, or Fargate is one of the more consequential intermediate-level decisions. Lambda’s event source mapping is the lowest-operational-overhead option — no servers, no KCL configuration, automatic batching and retry — and it scales naturally with shard count since AWS runs one concurrent Lambda invocation per shard by default. Its trade-off is a hard per-invocation execution time limit and a batch-oriented processing model that fits many workloads well but becomes awkward for consumers that need long-lived in-memory state across records, such as a session-based aggregation that needs to hold data for minutes at a time. A KCL application on a persistent compute platform has no such execution-time ceiling and can maintain arbitrarily long-lived in-memory state per shard, at the cost of managing the compute layer, scaling policy, and deployment pipeline yourself.

A last integration worth naming explicitly is Kinesis Data Streams as a source for Amazon EventBridge Pipes and for direct integrations with services like Amazon OpenSearch Service and Amazon Redshift Streaming Ingestion, both of which can consume directly from a stream without an intermediate Lambda or Firehose hop — useful when the destination system already understands how to poll a Kinesis shard iterator natively and an extra processing layer would only add latency without adding value.

A few recurring shapes show up again and again in Kinesis-based systems — some worth copying, some worth avoiding.

The fan-out pattern — one stream, many independent consumers each doing something different with the same data — is Kinesis’s signature strength and the reason teams choose it over a simple queue. The enrichment pipeline pattern chains streams together: a raw ingestion stream is consumed by a Lambda that enriches or filters records, then republishes them to a second, cleaner stream for downstream consumers — trading a small amount of added latency for much simpler downstream logic.

A third pattern worth naming is the aggregation-then-fan-out pattern, used when producers generate a very high volume of small records — think per-click telemetry from a mobile app — that would waste shard capacity if written one at a time, since each PutRecord call has fixed per-request overhead regardless of payload size. Here, the Kinesis Producer Library aggregates many small user records into a single larger Kinesis record before writing, and consumers use the corresponding deaggregation logic (built into KCL and available as a library for Lambda) to unpack the original individual records on the way out. This can meaningfully increase effective throughput per shard for high-volume, small-payload workloads, at the cost of slightly more producer-side complexity and a small amount of added latency while records wait to be batched together.

Most Kinesis incidents trace back to one of a small set of recurring, avoidable mistakes.

A more subtle mistake is checkpointing too eagerly — saving progress after every single record, which adds unnecessary DynamoDB load and cost, versus checkpointing every few seconds or every batch, which is sufficient for realistic recovery-time requirements in almost every workload.

Two further habits separate teams that operate Kinesis smoothly for years from those that get paged repeatedly. The first is treating shard count as a living parameter tied to a real capacity-planning cadence — reviewing throughput trends monthly or quarterly rather than only reacting after a throttling incident already happened in production. The second is load-testing partition key distribution before launch, not after: generating a realistic sample of expected partition key values, running them through the same MD5 hashing Kinesis uses, and checking the resulting distribution across shards is a cheap, five-minute exercise that catches the single most common category of Kinesis production incident before it ever reaches customers.

Seeing how established companies actually deploy Kinesis makes the abstract concepts concrete.