Amazon Kinesis is a family of AWS services for collecting, processing, and delivering fast-moving (“streaming”) data in near real time. The flagship service, Kinesis Data Streams, works like a durable, ordered, continuously flowing pipe: producers write small records into the pipe, and one or more consumers read those same records back out, independently and at their own pace, without ever removing the data for anyone else reading it.

The word “streaming” is doing real work here, and it’s worth defining precisely. A traditional database write is a single, isolated transaction — write a row, move on. A message queue like Amazon SQS typically delivers each message to exactly one consumer, and then the message is gone. A stream is different on both counts: records arrive continuously in the order they were written, they are retained for a configurable window of time rather than deleted after one read, and multiple independent applications can all read the same records without one consumer “using up” data meant for another.

Kinesis is actually a family of four related services, and beginners often conflate them. Kinesis Data Streams is the core, general-purpose stream you build custom producers and consumers around. Kinesis Data Firehose is a fully managed delivery pipe that automatically loads streaming data into destinations like Amazon S3 or Amazon Redshift, with no consumer code to write. Managed Service for Apache Flink (formerly Kinesis Data Analytics) lets you run SQL-like or Flink-based queries directly against a moving stream. Kinesis Video Streams handles the same real-time-capture idea but for video and audio rather than small data records. This guide focuses primarily on Kinesis Data Streams, since it is the foundation the other services build on.

It also helps to separate two ideas beginners often blend together: “real time” and “batch.” A batch system collects data over a period — an hour, a day — and processes it all at once, which is efficient but means every insight is old by the time it’s produced. Kinesis belongs to the opposite world, often called “stream processing,” where individual records or small groups of records are handled within seconds of arriving. Neither approach is universally better; a monthly billing report is perfectly happy running as a batch job, while a fraud alert or a live dashboard needs the stream approach because value decays the moment the data goes stale.

Another concept worth locking in early is that a Kinesis stream itself stores nothing permanently — it is a temporary, ordered holding area, not a database. This distinction trips up beginners who assume that because data lives in a stream, it is automatically archived somewhere. In reality, if you need to keep streamed data indefinitely for later querying, you deliberately route it into a long-term store, most commonly through Kinesis Data Firehose writing into Amazon S3 or a data warehouse. The stream’s job is to move data fast and reliably for a limited window, not to be your system of record.

Every Kinesis Data Stream is made of one or more shards. A shard is the fundamental unit of capacity and throughput — each shard guarantees a fixed write and read rate, and it stores records in the strict order they were received. When your data volume grows beyond what one shard can handle, you don’t upgrade to a “bigger shard”; you add more shards, spreading incoming data across them.

Two capacity modes control how shards behave. Provisioned mode requires you to explicitly choose the shard count, giving predictable cost and throughput but requiring you to plan for peak load. On-demand mode automatically scales shard capacity up and down based on observed traffic over the last 30 days, trading a little cost efficiency for zero manual capacity planning — a good default for beginners and unpredictable workloads.

graph LR
    P1["Producer:
Mobile App"] -->|"PutRecord"| STREAM
    P2["Producer:
IoT Sensor Fleet"] -->|"PutRecords (batch)"| STREAM
    subgraph STREAM["Kinesis Data Stream"]
        S1["Shard 1"]
        S2["Shard 2"]
        S3["Shard 3"]
    end
    STREAM --> C1["Consumer:
KCL Application"]
    STREAM --> C2["Consumer:
AWS Lambda"]
    STREAM --> C3["Kinesis Data Firehose"]
    C3 --> S3B["Amazon S3"]
    STREAM --> C4["Managed Service
for Apache Flink"]
    C4 --> DASH["Real-Time Dashboard"]
        

Notice in the diagram that the same stream feeds four completely different consumers simultaneously — a custom application, a Lambda function, Firehose (which lands data into S3), and Flink (which powers a live dashboard). None of these consumers compete with each other for records; each maintains its own independent read position, called a shard iterator or, in more modern applications, a checkpoint tracked through the Kinesis Client Library.

When a producer calls the PutRecord or PutRecords API action, it sends a data blob (up to 1 MB) along with a partition key. Kinesis runs the partition key through a hash function, and the resulting hash value determines exactly which shard receives the record — every shard owns a specific range of hash values, similar to how a library assigns each shelf a range of last names. Once the record lands in its shard, Kinesis assigns it a strictly increasing sequence number, guaranteeing that within that shard, records are always readable in the exact order they arrived.

On the read side, a consumer doesn’t get records pushed to it automatically by default — it asks. Using the classic pull-based model, a consumer calls GetShardIterator to establish a starting position in a shard (from the oldest available record, the newest, a specific sequence number, or a specific timestamp), then repeatedly calls GetRecords to pull batches of data, advancing its position each time. For lower latency, Enhanced Fan-Out flips this model: Kinesis pushes records to each registered consumer over a dedicated, private throughput channel, cutting typical delivery latency from around 200 milliseconds down to about 70 milliseconds, and giving every consumer its own full 2 MB/sec read throughput instead of sharing it.

This is why partition key choice matters so much in practice: pick a key that is too uniform (like a fixed constant, or a single popular customer ID), and every record piles onto one shard, creating a “hot shard” while the rest sit idle. Pick a key with good, even distribution, and Kinesis naturally spreads load across all your shards.

It’s worth walking through a concrete example, because the hashing behavior can feel abstract otherwise. Imagine a ride-sharing stream with four shards and a partition key of driver ID. Driver “D-1042” will always hash to the same shard, meaning every location ping from that driver arrives in order, one after another — perfect for a consumer that needs to trace one driver’s exact route. Meanwhile, driver “D-8871” might hash to a completely different shard, and their pings are processed in parallel, on a separate shard, with no bearing on D-1042’s ordering. This is the core trade-off of sharded streaming: you get strong ordering per key, and near-limitless parallelism across keys, but no built-in guarantee of ordering between two different keys on two different shards.

Unlike a queue, where a record’s lifecycle ends the moment one consumer reads it, a Kinesis record’s lifecycle is governed entirely by time. The moment it’s written, a retention clock starts. By default that clock runs for 24 hours; you can extend it up to 7 days at standard cost, or up to 365 days on extended retention pricing. During that entire window, any consumer — even one that starts reading weeks after the data first arrived, as long as it’s within retention — can request records from any point in the shard’s history, a capability called replay.

Replay is not a side feature — it is one of the biggest practical reasons teams choose Kinesis over simpler alternatives. If a downstream analytics job has a bug and produces bad output for six hours, you don’t need to somehow “resend” the original data; as long as it’s still within the retention window, you simply point a corrected consumer back at an earlier sequence number and reprocess it.

Extended retention comes with a genuine trade-off worth naming: storing records for 365 days instead of the default 24 hours increases cost, because AWS is keeping far more data resident and replicated across Availability Zones for far longer. Teams typically reserve long retention windows for streams where replay value is high — compliance-sensitive event logs, or pipelines feeding machine learning models that benefit from reprocessing historical data with an improved algorithm — rather than applying long retention as a blanket default across every stream in an account.

Throughput in Kinesis Data Streams scales almost linearly with shard count, because each shard is an independent unit of capacity. A stream with 10 shards in provisioned mode can absorb roughly 10 MB/sec of writes and serve up to 20 MB/sec of reads under the classic model, or far more with Enhanced Fan-Out spreading reads across dedicated channels per consumer. On-demand mode watches your traffic pattern automatically and adds shard capacity as sustained throughput grows, removing the guesswork for teams that don’t yet know their peak load.

Every shard’s data is synchronously replicated across multiple Availability Zones within a region before a write is acknowledged back to the producer, which is what gives Kinesis its durability guarantee — a successful PutRecord response means the data has already survived a single data center failure. Because this replication happens automatically and is managed entirely by AWS, there is no equivalent of manually configuring a replica set or a leader election, unlike a self-managed streaming system you might run yourself.

A frequent early mistake is under-provisioning shards for bursty, unpredictable workloads and then being surprised by ProvisionedThroughputExceededException errors during traffic spikes. On-demand mode largely absorbs this risk for beginners, while experienced teams running provisioned mode typically build in headroom and use Amazon CloudWatch alarms on shard-level metrics to catch a hot shard before it starts throttling producers.

Access to a Kinesis stream is governed entirely through IAM policies attached to the identities of your producers and consumers. A producer typically needs only kinesis:PutRecord and kinesis:PutRecords permissions scoped to a specific stream, while a consumer needs read-oriented actions like kinesis:GetRecords, kinesis:GetShardIterator, and kinesis:DescribeStream. Following least privilege here matters in practice: a producer role that can accidentally read the stream, or a consumer role that can accidentally write to it, widens your blast radius for no operational benefit.

In transit, all Kinesis API calls travel over HTTPS by default, so data moving between your producer application and the Kinesis service endpoint is encrypted in flight without any extra configuration. For network isolation, Kinesis also supports VPC endpoints, letting producers and consumers running inside a private subnet reach Kinesis without traversing the public internet at all — a common requirement for regulated workloads handling sensitive event data.

Amazon CloudWatch automatically publishes stream-level and shard-level metrics for every Kinesis stream, including IncomingBytes, IncomingRecords, WriteProvisionedThroughputExceeded, and ReadProvisionedThroughputExceeded. A healthy monitoring setup alarms on sustained throttling metrics, since repeated throttling almost always signals either a hot shard from a poorly chosen partition key, or a stream that has genuinely outgrown its current shard count.

Deployment Considerations

For teams just getting started, on-demand mode removes an entire category of early capacity-planning mistakes and is generally the safer default. As traffic patterns become well understood and predictable, moving to provisioned mode can reduce cost, provided the team commits to actively monitoring shard-level metrics and resharding — splitting hot shards or merging underused ones — as load evolves over time.

Netflix uses Kinesis-style streaming ingestion to process enormous volumes of viewing and interaction events, feeding recommendation systems that need to reflect what a viewer just watched, not what they watched yesterday. A ride-sharing platform can stream every driver’s GPS ping through Kinesis Data Streams, with one consumer updating a live map for riders, a second consumer feeding a fraud-detection model, and a third streaming raw location data into S3 through Firehose for long-term analytics — three completely different systems, reading the same stream, without any of them slowing the others down.

A financial services company processing card transactions can rely on Kinesis’s per-shard ordering guarantee to ensure that transactions from a single account are always processed in the exact sequence they occurred, which matters enormously for fraud detection logic that depends on transaction order. An IoT manufacturer collecting telemetry from thousands of factory sensors can use Kinesis Data Streams as the front door for that data, with Managed Service for Apache Flink running continuous queries that flag an equipment anomaly within seconds of it happening, rather than discovering it in tomorrow’s batch report.