A fully managed service that receives, stores for a time, and delivers very large streams of events to many applications.

Starting with the word “event”

An event is a small record that says something happened. It usually carries a timestamp, a few details, and sometimes an identifier. A smart thermostat sending “21.4 degrees at 10:03:05” is an event. So is “customer 482 added a book to the cart”. One event is tiny. The challenge is that there can be millions of them every second, arriving without pause.

What “streaming” means

Streaming means handling data as it arrives, continuously, instead of collecting a big pile and processing it once a day. A stream is like a river: it keeps flowing, and you decide where to dip your bucket. This lets a bank detect fraud while a payment is happening, or a factory spot a failing machine before it breaks.

What Event Hubs does

Azure Event Hubs is a managed Microsoft service built to be the front door for such streams. It:

• Ingests events from a huge number of senders at high speed.
• Stores them in order for a configured retention period, so readers do not have to be online at the exact same moment.
• Delivers them to one or many independent reading applications, each at its own pace.
• Scales by splitting the stream into parallel lanes called partitions.
• Speaks several languages: it supports common protocols, including one compatible with Apache Kafka, so existing tools can connect.

“Managed” means Microsoft runs the servers, the storage, the patching, and the replication. You create the resource, send events, and read events.

To see the value, imagine building a big data pipeline without it.

Problem one: direct connections break down

If every sender talks straight to every receiver, the number of connections multiplies. Ten senders and ten receivers need up to one hundred links. Add one receiver and you must change every sender. A shared middle layer turns this tangle into a simple star: all senders talk to one place, all receivers listen to one place.

Problem two: traffic is bursty

A shopping website may get ten times more traffic during a sale. If the processing system must handle every burst instantly, you must pay for peak capacity all the time. A buffer in the middle absorbs bursts and lets the processing side work at a steady pace. Engineers call this load leveling.

Problem three: receivers fail and restart

Programs crash, get updated, or run slowly. If events were delivered only to a live receiver, anything sent during a restart would vanish. Because Event Hubs stores events for a retention period, a returning reader resumes from where it left off.

Problem four: many teams want the same data

The same click events may feed a live dashboard, a recommendation system, a fraud detector, and a data lake for later study. Each needs the full stream. Event Hubs allows every one of them to read independently, with no extra work for the sender.

flowchart LR
    subgraph WITHOUT["Without a central hub"]
      A1["Sender 1"] --- B1["Receiver 1"]
      A1 --- B2["Receiver 2"]
      A2["Sender 2"] --- B1
      A2 --- B2
    end
    subgraph WITH["With Event Hubs"]
      S1["Sender 1"] --> H["Event Hubs"]
      S2["Sender 2"] --> H
      H --> R1["Receiver 1"]
      H --> R2["Receiver 2"]
    end
    

Five words explain most of Event Hubs. Learn them once and everything else falls into place.

Namespace

A namespace is the container for one or more event hubs. It gives you a unique network address, holds shared settings such as pricing tier, capacity, network rules, and security policies, and acts as the unit you pay for and manage. Think of it as the building that houses several mailrooms.

Event hub

An event hub is one named stream inside a namespace. A company might have one event hub for website clicks, another for payments, and another for sensor readings. Each has its own partitions and retention settings. In Kafka language, an event hub corresponds to a topic.

Partition

A partition is an ordered, append-only sequence of events inside an event hub. New events are added to the end and are never inserted into the middle. A hub is split into several partitions so that many senders and readers can work in parallel. Order is guaranteed inside a partition, but not across partitions.

Consumer group

A consumer group is a named view of the whole event hub. Each application that wants to read the full stream independently uses its own consumer group. Each group keeps its own reading position in every partition. Two groups never disturb each other.

Producer and consumer

A producer (also called a publisher or sender) is any program that sends events in. A consumer (or receiver) is any program that reads events out.

flowchart TD
    NS["Namespace: the building"] --> EH1["Event hub: clicks"]
    NS --> EH2["Event hub: payments"]
    EH1 --> P0["Partition 0"]
    EH1 --> P1["Partition 1"]
    EH1 --> P2["Partition 2"]
    EH1 --> P3["Partition 3"]
    P0 --> CG1["Consumer group: dashboard"]
    P1 --> CG1
    P2 --> CG1
    P3 --> CG1
    P0 --> CG2["Consumer group: fraud detection"]
    P1 --> CG2
    P2 --> CG2
    P3 --> CG2
    

Putting the vocabulary to work: a worked example in words

Imagine a ride-sharing company with a million active drivers. Each driver’s phone sends a location update every few seconds. The company creates a namespace called “rides” and, inside it, an event hub called “driver-locations” with thirty-two partitions. Every update carries the driver’s ID as the partition key, so all updates from one driver always land in the same partition and stay in time order. A live map service reads the stream in a consumer group named “map”. A pricing service, which needs to know how many drivers are near each neighborhood, reads the same stream in a consumer group named “pricing”. A safety team reads it in a group called “safety” to detect unusual stops. Each of the three has its own bookmark in each of the thirty-two partitions. If the pricing service crashes for ten minutes, the map and the safety team do not even notice, and pricing resumes from its own bookmark when it returns.

This small story contains every core word: the namespace is the building, the event hub is the stream of driver locations, partitions are the thirty-two lanes, the driver ID is the key that keeps each driver’s trail in order, and consumer groups are the independent readers. When you can retell a system in this way, you understand Event Hubs.

Partitions give you speed and parallelism. They also decide what ordering guarantees you really have.

How events pick a partition

When a producer sends an event, one of three things decides its partition.

• Automatic spreading: If you give no instruction, the service spreads events across partitions to balance the load. There is no ordering relationship between events on different partitions.
• Partition key: You may attach a value, like a customer ID or device ID. The service converts the key into a number and always sends events with the same key to the same partition. This keeps all events of one customer or device in order.
• Direct partition choice: You may name a partition explicitly. This is rarely needed and makes balancing your job.

The ordering promise

Event Hubs promises order within one partition. If device 17’s events always carry the key “device-17”, they land together and are read in the order they were written. But there is no guaranteed order between device 17 and device 18 if they sit on different partitions, and in most real systems that is perfectly fine.

flowchart LR
    E1["Event: device 17 reading A"] --> KEY{"Hash of partition key"}
    E2["Event: device 18 reading A"] --> KEY
    E3["Event: device 17 reading B"] --> KEY
    KEY -->|"device 17"| PA["Partition 1: A then B in order"]
    KEY -->|"device 18"| PB["Partition 3: separate lane"]
    

Choosing the number of partitions

The partition count sets the maximum useful parallelism for readers: within one consumer group, each partition is read by one active reader at a time. If you have eight partitions, at most eight readers in a group can share the work. Too few partitions limits scale. Too many adds overhead and can make small workloads wasteful. Depending on tier and settings, the number of partitions can be fixed at creation or can only be increased within limits, so plan with growth in mind and check current rules.

Hot partitions

If one key is extremely popular, such as one giant customer, its partition gets more traffic than the others. This is called a hot partition. Choose keys with many distinct, evenly used values, or add a secondary part to the key to spread the load.

The full picture of how events move from sources to destinations.

flowchart LR
    subgraph SRC["Producers"]
      A["Web and mobile apps"]
      B["IoT devices"]
      C["Servers and logs"]
      D["Business systems"]
    end
    subgraph EHN["Event Hubs namespace"]
      H["Event hub with partitions"]
      CAP["Capture to storage"]
      SR["Schema Registry"]
    end
    subgraph CONS["Consumers"]
      E["Stream Analytics"]
      F["Azure Functions"]
      G["Spark or Databricks"]
      I["Custom event processors"]
      J["Kafka clients"]
    end
    A --> H
    B --> H
    C --> H
    D --> H
    H --> CAP
    SR -.-> H
    H --> E
    H --> F
    H --> G
    H --> I
    H --> J
    CAP --> LAKE["Data lake storage"]
    

The main building blocks

Protocols

Event Hubs supports AMQP, an efficient messaging protocol widely used for production senders and receivers, HTTPS, which is simple and useful from environments where AMQP is not possible, and the Kafka protocol, for applications written for Kafka. Software development kits exist for many programming languages, so the application side can be written in the language your team already uses.

Where things physically live

You choose an Azure region when creating a namespace. Events are stored inside that region, replicated across machines (and across availability zones where enabled) so a single machine failure does not lose data. That is why it is important to pick a region near your senders and to follow data residency rules.

Event Hubs is sold in tiers that trade features, isolation, and price. Capacity units set how much traffic you can carry.

The tiers in plain words

Capacity units

Throughput is bought in capacity units. In the Standard tier they are called throughput units, in Premium processing units, and in Dedicated capacity units. Each unit gives a certain amount of ingress (data in) and egress (data out) per second. If you send more than your capacity, the service slows you with throttling, which tells senders to back off and retry. Standard tier namespaces can be configured to grow automatically, which is often called auto-inflate, up to a maximum you choose. That protects you from sudden traffic without leaving you permanently over-provisioned.

How to estimate what you need

1. Estimate events per second at the busiest time, not the average.
2. Multiply by the average event size to get megabytes per second.
3. Add a safety margin for growth and bursts.
4. Count how many consumer groups read, because each adds outgoing traffic.
5. Choose capacity and partition count to match, then test with realistic load.

Everything a sender should know to deliver events reliably and efficiently.

The shape of an event

An event consists of a body, which is the payload, often text formats like JSON or compact binary formats, and optional properties, which are key and value labels you attach, such as event type or version. The service also adds system information, like the time it was received, the offset, and the sequence number within the partition.

Sending in batches

Sending one tiny event per network call wastes effort. Producers should gather events into batches and send them together. Batching greatly increases throughput and lowers cost per event. The client libraries help by building batches that respect the maximum size limit.

Delivery guarantees from the producer side

A producer waits for confirmation that the service has stored the event. If the confirmation does not arrive, the producer might not know whether the event was stored, so it retries. This can create duplicates. Event Hubs aims for at-least-once delivery: an event will arrive one or more times, rarely twice. Consumers should therefore be written so that processing the same event twice does no harm, a property called idempotence.

Handling throttling and failures

• Retry with waiting: when the service says “server busy”, wait a growing amount of time before retrying, rather than hammering it.
• Buffer locally: in case of brief outages, keep a small local queue and send when the connection is back.
• Re-use connections: creating a new connection for every event is slow. Keep long-lived clients.
• Limit event size: keep events compact. Very large payloads belong in storage, with the event carrying only a reference.

sequenceDiagram
    participant P as Producer application
    participant L as Local batch buffer
    participant E as Event Hubs
    P->>L: Add event 1, 2, 3
    L->>E: Send batch with partition key
    E->>E: Store in partition and replicate
    E-->>L: Acknowledge
    P->>L: Add event 4, 5
    L->>E: Send batch
    E-->>L: Server busy, slow down
    L->>L: Wait and retry later
    L->>E: Send batch again
    E-->>L: Acknowledge
    

Reading a stream is not like reading a queue. Events are not removed when read; readers remember where they are.

Offsets and sequence numbers

Every event in a partition has an offset, a position marker, and a sequence number, a counter that increases by one for each event. A reader can start from the beginning of retained data, from the newest event, from a specific time, or from a saved position.

Checkpointing

A checkpoint is a saved note of “I have processed up to here” for a given partition and consumer group. Checkpoints are typically stored in a small durable store, often Azure Blob Storage. If a reader crashes and restarts, it loads the checkpoint and resumes from that point instead of from the beginning. How often you checkpoint is a trade-off: saving after every event is safe but slow, while saving every few hundred events is faster but means that after a crash a few events are processed again.

The event processor

The client libraries include an event processor, a ready-made helper that handles the hard parts of reading. It discovers the partitions, shares them among multiple running instances of your application (a process called load balancing), stores checkpoints, and resumes after failures. You write only the part that says what to do with each event or batch of events.

sequenceDiagram
    participant H as Event Hubs partition
    participant R as Event processor instance
    participant C as Checkpoint store
    participant W as Your processing code
    R->>C: Where did we stop for this partition
    C-->>R: Offset 1200
    R->>H: Read from offset 1201
    H-->>R: Batch of events
    R->>W: Process the batch
    W-->>R: Done
    R->>C: Save checkpoint at offset 1260
    Note over R,C: If the instance crashes here, the next one resumes from 1260
    

Sharing work among readers

Suppose you have eight partitions and start four instances of your application with the same consumer group. The event processors coordinate, usually through the checkpoint store, and each instance takes about two partitions. If one instance dies, the others notice and take over its partitions. If you add more instances, partitions are redistributed. The upper limit is one active reader per partition per group.

Choosing a starting position

A new consumer must decide where to begin. Starting from the earliest retained event lets a new application build its picture of the world from history, which suits analytics and rebuilding a lost data store. Starting from the latest event suits live monitors that care only about what happens from now on. Starting from a specific time is useful after an incident: “reprocess everything since 2 p.m., when the bug was introduced”. Once a checkpoint exists, the normal rule is to resume from it, and the starting choice only applies the first time. Be careful with the “earliest” choice on a very large retained backlog, since the new reader will spend a long time catching up before it reaches the present.

It also helps to understand batching on the reading side. Processors usually hand your code a batch of events at once. Handling a batch together, for example writing all its results to a database in one operation, is much faster than handling events one at a time. Checkpoint after finishing a batch, not in the middle of it, so that the saved position always matches work that is truly complete.

Follow a single event from the moment it is born to the moment it expires.

flowchart TD
    A["Event created"] --> B["Producer batches and sends"]
    B --> C{"Authorized and within capacity"}
    C -->|"no"| D["Rejected or throttled, producer retries"]
    C -->|"yes"| E["Appended to partition"]
    E --> F["Replicated for durability"]
    F --> G["Available to consumer groups"]
    F --> H["Captured to storage if enabled"]
    G --> I["Processed and checkpointed"]
    E --> J["Deleted after retention period"]
    

Retention is not archiving

Retention keeps events for a limited time so readers can catch up or replay. It is not a database. If you need to keep events for months or years, either enable Capture to storage or have a consumer write them to a long-term store. Also, retention is a shared safety net: if a consumer is down for longer than the retention period, the events it missed are gone from the hub.

Replaying

Because events are not removed when read, a new application can start from the oldest retained event and “replay” history, or an existing application can rewind to a time and reprocess after fixing a bug. This is a powerful advantage of the log style of storage.

Three optional capabilities that turn Event Hubs from a pipe into a platform.

Capture: automatic archiving

Capture copies events from the hub into Azure Blob Storage or Azure Data Lake storage without you writing any code. Files are created in time windows or size windows that you configure, and are stored in a compact binary format such as Avro. This gives you a permanent record for batch analytics, auditing, and machine learning. It also lets you treat the stream as both a live feed and a historical store.

Schema Registry

A schema describes the structure of an event: its fields and types. The Schema Registry is a shared place to store schemas and their versions. Producers register or look up the schema and attach its identifier to each event. Consumers fetch the schema and decode correctly. When a team wants to add a field, compatibility rules help prevent a change that would break existing readers.

Kafka compatibility

Apache Kafka is a popular open-source streaming platform. Event Hubs exposes a Kafka-compatible endpoint, so many applications written for Kafka can connect by changing configuration, such as the server address and security settings, rather than rewriting code. This allows teams to keep their Kafka tools while handing over server management to Azure. Not every Kafka feature is identical, and some advanced capabilities may differ or be unsupported, so test your own use case and read the current compatibility notes.

Event Hubs collects events. Other services decide what they mean.

Common consumers

flowchart LR
    SRC["Event sources"] --> EH["Event Hubs"]
    EH --> SA["Stream Analytics: live aggregates"]
    EH --> FN["Functions: react to each event"]
    EH --> SP["Spark: heavy analytics"]
    EH --> CAPT["Capture: archive"]
    SA --> DB["Dashboards and alerts"]
    FN --> API["Business systems"]
    SP --> ML["Models and reports"]
    CAPT --> LAKE["Data lake"]
    

Lambda and Kappa style thinking

Older designs ran two separate pipelines: a fast one for live results and a slow batch one for accurate history. Many modern designs use a single stream as the foundation, with the stored log serving both purposes, replaying history through the same code. Event Hubs with Capture supports either approach.

Choosing a processing service

• If your logic is mostly filtering, grouping, and joining over time windows, a SQL-style stream query service is simplest.
• If you must call other systems for each event, small serverless functions fit well.
• If you run complex analytics, joins with large data sets, or machine learning, a Spark-based platform is stronger.
• If you have unusual requirements or need full control, write a custom processor.

Event streams often carry customer behavior, payments, and device data. Protect the front door, the lanes, and the archive.

flowchart TD
    A["Who: identity and roles"] --> B["Where from: network rules and private endpoints"]
    B --> C["In flight: encrypted connections"]
    C --> D["At rest: encryption of stored events"]
    D --> E["What: schema rules and data minimization"]
    E --> F["Proof: logs and monitoring"]
    

Authentication and authorization

There are two main ways to prove identity. Microsoft Entra ID (identity-based access) uses accounts, applications, and managed identities with Azure role-based access control. Built-in roles let you separate senders, receivers, and owners, for example a role that only allows sending and another that only allows receiving. This is the recommended modern approach because there are no shared secrets to leak, and access can be revoked centrally. Shared access signatures (SAS) use keys or tokens with defined permissions. They are simple and work everywhere, but keys can leak and are harder to track. If you use them, create narrow, per-application policies, rotate keys, and avoid using the all-powerful root key in applications.

Least privilege

• Give producers only permission to send, and consumers only permission to read.
• Scope permissions to a single event hub rather than the whole namespace when possible.
• Use separate identities per application, so one leaked credential does not expose everything.

Network protection

By default, a namespace has a public address. You can restrict it with IP firewall rules and virtual network rules, or turn off public access and use private endpoints, which give the namespace a private address inside your own network. This keeps traffic off the public internet.

Encryption

Connections use TLS encryption in transit. Stored data is encrypted at rest by default with Microsoft-managed keys, and higher tiers allow you to bring your own keys for additional control. Remember that Capture files in your storage account are your responsibility as well: secure and encrypt that account too.

Protecting the data itself

The best way to protect sensitive data is not to put it in events needlessly. Avoid sending full personal records when an identifier is enough. Mask or tokenize sensitive fields before they enter the stream, and be aware that retention keeps copies for days.

What happens when a machine, a zone, or a whole region has a bad day.

Built-in durability

Event Hubs stores each event on multiple machines before acknowledging it. Machine failures are handled automatically. In regions that support availability zones, zone redundancy can spread a namespace across separate buildings with independent power and networking, so even a zone-level problem need not interrupt service. Check which tiers and regions support this.

Surviving a region failure

For the rare event of an entire region becoming unavailable, Event Hubs offers options that depend on tier and configuration.

• Geo-disaster recovery: pairs a primary namespace with a secondary in another region. It copies metadata (the configuration, not the events) and lets you fail over, switching an alias name to the secondary. Events in the failed region that were not yet read are not carried over.
• Geo-replication: a newer capability that replicates event data itself between regions in supported tiers, allowing failover with data continuity subject to its replication mode. Check current availability and behavior.
• Active-active design: senders write to two namespaces in two regions, and consumers read from both, handling duplicates. It is the most resilient and the most complex.

Reliable clients

• Producers retry with waiting and keep a local buffer for brief outages.
• Consumers checkpoint regularly and handle duplicates.
• Applications reach the namespace through an alias or a configurable address, so failover does not need a code change.
• Run failover drills before a real incident forces you to try for the first time.

A stream you cannot see is a stream you cannot trust.

Key metrics to watch

Logs and alerts

Enable diagnostic settings to send operational and runtime logs to Log Analytics, storage, or another destination. Create alerts for sustained throttling, rising error rates, and consumer lag beyond the amount of time your business can tolerate. Alert on lag measured in time (for example, “more than five minutes behind”), since that matches what people care about.

Common problems

A simple health routine

A healthy team follows a small routine. Every morning someone glances at a dashboard showing incoming traffic, throttling, errors, and consumer lag for each important event hub. Every week the team reviews the trends: is traffic growing faster than capacity, is any partition carrying more than its share, and are any consumers repeatedly restarting? Every quarter the team runs a failure drill, such as stopping a consumer for an hour to prove that lag grows, alerts fire, and recovery works. These habits take little time, yet they catch most problems while they are still small and cheap to fix.

When you open a support request, gather the namespace name, event hub name, region, approximate time of the problem with its time zone, the exact error messages, and the client library and version. Precise details save days of back-and-forth.

How to get more from the same money, and how to grow without surprises.

Where speed comes from

Scaling the three sides

• Service capacity: add throughput, processing, or capacity units, or move to a higher tier.
• Producers: add sender instances and make sure each batches efficiently.
• Consumers: add reader instances, up to the number of partitions per consumer group, and speed up the code that handles each event.

Cost drivers

Reusable shapes for streaming systems, and the traps that experienced engineers avoid.

Pattern: fan-out

One event hub, many consumer groups. Each team reads the full stream independently, which keeps teams loosely coupled.

Pattern: partition by entity

Use the identity of the business object (customer, device, order) as the partition key so all events about it stay in order.

Pattern: hot path and cold path

The hot path processes events immediately for live dashboards and alerts. The cold path stores everything (using Capture) for deeper batch analysis later.

Pattern: dead-letter side channel

Event Hubs has no built-in per-message dead-letter queue like some messaging services. If a consumer cannot process an event, it should log it and send it to a separate hub, queue, or storage location for later inspection, instead of blocking the partition forever.

Pattern: claim check

For large payloads such as images, store the large data in blob storage and send only a small event containing the reference.

flowchart TD
    P["Producers"] --> H["Event hub partitioned by customer"]
    H --> HOT["Hot path: stream processing and alerts"]
    H --> COLD["Cold path: Capture to data lake"]
    HOT --> DASH["Live dashboards"]
    HOT --> BAD{"Event cannot be processed"}
    BAD -->|"yes"| DLQ["Side channel for inspection"]
    COLD --> BATCH["Batch analytics and reports"]
    

Anti-patterns

A short path to a healthy streaming system, and the potholes along the way.

Best practices

Common mistakes

Testing a streaming system

Streaming systems are tested differently from ordinary applications because they are continuous and time-dependent. Begin with small tests of the event handlers using sample events, including malformed ones, duplicated ones, and ones arriving out of order across partitions. Next, run an end-to-end test in a non-production namespace where a test producer sends a known set of events and the test verifies the final result in the downstream store. Finally, run a load test that mimics your busiest hour, including bursts, and watch throttling, lag, and consumer behavior. Also test the unhappy paths on purpose: stop a consumer, restart it, revoke a permission, and block the network briefly. A system that has been broken on purpose in a safe place is much less likely to surprise you in production.

These are composite examples that show typical situations. They are illustrations, not reports about specific companies.

Azure has several messaging services. Picking the right one avoids painful redesigns.

flowchart TD
    Q["What are you sending"] --> A{"Huge continuous stream of facts"}
    A -->|"yes"| EH["Event Hubs"]
    A -->|"no"| B{"Commands or business messages needing careful delivery"}
    B -->|"yes"| SB["Service Bus"]
    B -->|"no"| C{"Notifications that something happened in Azure"}
    C -->|"yes"| EG["Event Grid"]
    C -->|"no"| D{"Two-way talk with physical devices"}
    D -->|"yes"| IOT["IoT Hub"]
    D -->|"no"| SQ["Simple queue or another fit"]
    

Working together

These services are often combined. IoT Hub can route device messages into an event hub. Event Grid can notify when a Capture file is created. A processor can read a stream from Event Hubs and place business commands on Service Bus queues. Use each one for what it does best.

Event Hubs is the wide front gate for the river of events that modern systems produce.

We began at a railway station where thousands of people pass through every minute and several teams watch the same flow without interfering. Event Hubs plays that role for software. Producers send events in batches. The service splits them into partitions that keep order, stores them for a retention period, and lets any number of consumer groups read at their own speed, remembering their place through checkpoints. Optional features such as Capture, the Schema Registry, and the Kafka endpoint extend it into a full streaming platform, while tiers and capacity units let you match cost to need.

The deeper lesson is that good streaming design is about agreements: agreements on what an event means, which key keeps related events together, how duplicates are handled, and how fast readers must keep up. The service moves the bytes. Your design decides whether the story they tell is correct.