Some data does not arrive in tidy envelopes. It arrives in floods.

Think about what modern systems produce: clicks on a website, readings from thousands of sensors, application logs, financial ticks, vehicle positions and game actions. A single reading is tiny, but together they can number millions per second, and they never stop. Ordinary databases and queues are not designed to absorb that kind of continuous torrent and let many different programs read it at once.

Azure Event Hubs is a fully managed, real-time data ingestion service built for exactly this. It accepts huge volumes of events from many producers, stores them temporarily in an ordered log, and lets any number of consumers read them independently. You do not manage servers, brokers or clusters. You create an event hub, send data to it and attach readers.

Its purpose is to be the front door for streaming data. It decouples the systems that produce data from the systems that analyse, store or react to it. Producers can send at full speed without waiting for slow consumers, and consumers can be added, removed or replayed without producers knowing.

An event log, not a mailbox

The key idea is that Event Hubs keeps events in an append-only log. Consumers do not remove events when they read them. They simply remember how far they have read. This is very different from a traditional queue, where reading a message takes it away. In a log, ten different consumers can read the same events, and a consumer that fell behind or crashed can go back and read again, as long as the events are still within the retention period.

Choosing the right messaging service starts with the nature of what you are sending.

Azure has three well-known messaging services, and beginners often confuse them. Each is built for a different job, and good architectures often use more than one together.

flowchart TD
  A["What are you moving?"] --> B{"Continuous high-volume stream to be analysed or replayed?"}
  B -->|"Yes"| EH["Event Hubs"]
  A --> C{"A notification that something happened?"}
  C -->|"Yes, react to it"| EG["Event Grid"]
  A --> D{"A command or work item needing reliable single processing?"}
  D -->|"Yes"| SB["Service Bus"]
  EG -.->|"Can deliver into"| EH
  EG -.->|"Can deliver into"| SB
  EH -.->|"Capture to storage"| ST["Blob storage or data lake"]
        

Event Hubs and Apache Kafka

Apache Kafka is a popular open-source platform for the same kind of streaming workload, and many organisations already run it. Event Hubs can speak the Kafka protocol, so existing Kafka producers and consumers can connect to an event hub by changing configuration rather than rewriting code. This lets teams keep their Kafka tools while handing the operation of the platform to Azure. It is not a complete copy of every Kafka feature, so check that the features you depend on are supported.

A short vocabulary unlocks every screen, metric and design conversation.

Event Hubs has a compact set of terms. Spending a few minutes here saves hours of confusion later.

What Azure runs, what you configure and how the tiers differ.

Event Hubs is a managed service, so the brokers, storage, replication and failure handling behind it are Azure’s job. Your decisions concern the namespace, the event hubs inside it, the number of partitions, the retention period, the tier and the clients that send and receive.

The tiers

From a sensor reading to an analytics dashboard.

Following one event through the system explains partitions, offsets, consumer groups and checkpoints in a single story.

flowchart LR
  P1["Producer A"] --> HUB
  P2["Producer B"] --> HUB
  P3["Producer C"] --> HUB
  subgraph HUB["Event hub with four partitions"]
    PA["Partition 0"]
    PB["Partition 1"]
    PC["Partition 2"]
    PD["Partition 3"]
  end
  PA --> CG1["Consumer group: analytics"]
  PB --> CG1
  PC --> CG1
  PD --> CG1
  PA --> CG2["Consumer group: archiving"]
  PB --> CG2
  PC --> CG2
  PD --> CG2
        

sequenceDiagram
  participant Prod as Producer
  participant EH as Event hub partition
  participant Proc as Event processor
  participant CP as Checkpoint store
  Prod->>EH: Send batch of events
  EH-->>Prod: Acknowledge stored
  Proc->>CP: Where did I stop last time?
  CP-->>Proc: Last checkpoint position
  Proc->>EH: Read events after that position
  EH-->>Proc: Events in order
  Proc->>Proc: Process events
  Proc->>CP: Save new checkpoint
        

Almost every Event Hubs design question comes back to partitions.

A single lane can carry only so much traffic and be read by only one reader at a time within a consumer group. Partitions solve both problems by splitting the stream into several lanes that can be written and read in parallel. They are also the unit of ordering, which creates important trade-offs.

What partitions give you

Choosing the number of partitions

The partition count sets the ceiling for read parallelism: a consumer group can have at most one active reader per partition. It also bounds how much traffic the hub can take. Choose it based on peak throughput and the number of parallel consumers you will want in future. In some tiers the count can be changed later, but in others it is fixed at creation, and increasing it can change which partition a given key maps to. Check the rules of your tier, and leave headroom for growth.

Choosing partition keys

Many readers, each with their own bookmark.

The ability for many independent applications to read the same stream is a defining feature. It is achieved through consumer groups and carefully managed positions.

Consumer groups

A consumer group is a named view of the whole event hub. Each group keeps its own position in each partition. An analytics application, an archiving process and a fraud detector can each use their own group, and each will see every event, at its own pace. Applications that need to share the work of one task, such as a fleet of identical workers, belong to the same group.

The event processor pattern

Writing a reliable reader by hand is difficult, so client libraries provide an event processor. It coordinates several instances of your application so that partitions are shared among them, balances ownership automatically when instances join or leave, reads events in batches and records checkpoints in a storage account. Using it is the standard approach for production consumers.

flowchart TD
  subgraph HUB["Event hub with four partitions"]
    P0["Partition 0"]
    P1["Partition 1"]
    P2["Partition 2"]
    P3["Partition 3"]
  end
  subgraph G1["Consumer group: orders-analytics with two instances"]
    I1["Instance 1"]
    I2["Instance 2"]
  end
  P0 --> I1
  P1 --> I1
  P2 --> I2
  P3 --> I2
  I1 --> CK["Checkpoint store in storage account"]
  I2 --> CK
        

Delivery guarantees you should expect

Event Hubs consumers generally operate with at-least-once processing. If a consumer processes events but crashes before saving its checkpoint, the replacement will read those events again. Your processing must therefore tolerate seeing the same event twice. Making handlers idempotent, so that repeating an action does not change the result, is the standard answer.

Speak whichever language your existing tools already know.

Event Hubs supports several ways for applications to connect, and the choice affects both effort and capability.

Why Kafka compatibility matters

Many companies have years of investment in Kafka producers, consumers, connectors and stream-processing frameworks. The Kafka endpoint on Event Hubs lets those applications talk to a managed service without rewriting them. In simple terms, a Kafka topic corresponds to an event hub, a Kafka consumer group to an Event Hubs consumer group, and Kafka offsets to Event Hubs offsets. This can remove the operational burden of running and patching Kafka clusters.

An ingestion service is only useful if something valuable happens afterwards.

Event Hubs deliberately keeps data only for a limited retention period. Real solutions send it onward to places where it can be analysed, stored or acted on. Fortunately, the service integrates with many other tools.

flowchart LR
  S["Devices, apps and logs"] --> EH["Event Hubs"]
  EH --> CAP["Capture to data lake"]
  EH --> SA["Stream Analytics"]
  EH --> FN["Azure Functions"]
  EH --> SP["Spark streaming"]
  EH --> RT["Real-time analytics"]
  CAP --> BATCH["Batch analytics and long-term storage"]
  SA --> DASH["Live dashboards"]
  FN --> ACT["Alerts and actions"]
  SP --> ML["Machine learning features"]
        

Hot path and cold path

A common design splits processing into two paths. The hot path reacts within seconds, using stream processing for alerts and live dashboards. The cold path keeps everything cheaply for later, using Capture to store raw events for batch analysis, reprocessing and audit. Using both gives you immediate insight and a durable record from the same single stream.

A stream with many readers is a contract with strangers.

When one producer sends data that ten different teams read, the structure of that data becomes a shared agreement. Change it carelessly and ten applications break in different ways, often silently.

Schema Registry

The Schema Registry feature gives you a central place to store the definitions of your event structures, usually in formats such as Avro and, in more recent updates, others such as JSON Schema. Producers reference a schema by identifier rather than embedding it in every event, which saves space. Consumers look up the schema to interpret the data. The registry can also enforce compatibility rules, so that a new version of a schema cannot be published if it would break existing readers.

Practical design advice

Capacity is a shared budget, and exceeding it means being throttled.

Each tier measures capacity in its own units, which define how much data can flow in and out. When traffic exceeds the allowed rate, the service slows down or rejects requests, a behaviour called throttling. Producers and consumers must be ready for it.

Ways to scale

Handling throttling gracefully

When a producer is throttled, it should wait and retry with growing delays rather than hammering the service. Client libraries include retry behaviour, and you should keep it enabled. On the consuming side, throttling shows up as slow reads, which often means the total read rate across all consumer groups is exceeding the egress allowance.

Plan for the day a datacentre, or a whole region, is unavailable.

Event Hubs stores events redundantly within a region, and supports availability zones in regions that offer them, so that a failure of one datacentre does not lose data or stop service. Protection against the loss of a whole region needs extra design, and the platform offers more than one approach.

Designing for failure

A data stream often carries the most sensitive records a company has.

Streams of telemetry, transactions and logs can reveal customer behaviour and business secrets. Security therefore needs attention at several layers.

In streaming, falling behind is the most common silent failure.

A streaming pipeline can look healthy while quietly slipping behind, until retention expires and data is lost. Monitoring must therefore cover the service itself, the producers and, most importantly, the consumers.

Service metrics to watch

Other useful measures include successful and failed requests, server errors, user errors such as authorisation problems, active connections, and capture backlog if Capture is enabled. A rising throttled count means capacity is too small or traffic is badly shaped.

Consumer lag

The most important health indicator for consumers is lag: how far behind the newest event each consumer group is, measured by comparing its checkpoint with the latest position in each partition. Growing lag means a consumer cannot keep up. If lag approaches the retention period, data will be lost before it is read. Track it, chart it and alert on it.

Logs, alerts and tracing

Pay for capacity and volume, and avoid paying for waste.

Pricing differs by tier and changes over time, so use the official pricing page for current figures. In broad terms, the lower tiers charge for capacity units by the hour plus a charge per million events ingested, while the Dedicated tier charges for reserved capacity with a minimum period. Capture, geo-replication, extended retention and data transfer can add further charges.

What drives cost

Choosing a tier in practice

A few shapes cover most real-world uses.

Once the building blocks are clear, a handful of patterns appear in almost every streaming solution.

Shapes that scale, and shortcuts that hurt later.

Patterns worth adopting

Anti-patterns to avoid

Habits that separate calm streaming systems from fragile ones.

Best practices

• Pick the right tool. Streams to Event Hubs, notifications to Event Grid and reliable commands to Service Bus.
• Plan partitions deliberately. Base the count on peak throughput and desired parallelism, and understand your tier’s rules for changing it later.
• Batch when sending. Send groups of events and keep payloads compact.
• Use the event processor. Rely on library support for load balancing and checkpointing rather than writing your own.
• Design for duplicates. Idempotent consumers make restarts and failover harmless.
• Archive with Capture. Keep a durable copy of important streams.
• Secure with identities. Managed identities, narrow roles, private networking and customer-managed keys where required.
• Monitor lag, throttling and silence. Alert before data is lost.
• Govern schemas. Register them and enforce compatibility.
• Automate and document. Define namespaces, hubs and consumer groups in code, and keep diagrams and owners current.

Common mistakes

How different organisations put the wide river to work.

The following scenarios are typical patterns rather than descriptions of specific companies. They show how the same building blocks combine for very different needs.

Quick answers to questions that come up again and again.

Event Hubs ingests high-volume streams that many consumers can read and replay. Event Grid routes discrete notifications to handlers. Service Bus provides reliable business messaging where each message is processed and removed.

No. Events stay until the retention period ends, and each consumer group keeps its own position. That is why multiple readers and replay are possible.

A partition is an ordered lane within an event hub. Partitions allow parallel writing and reading, and order is guaranteed only within one partition.

A named, independent view of the event hub. Each consumer group tracks its own position, so different applications can read the same events at their own pace.

Often yes. Event Hubs offers a Kafka-compatible endpoint, so many Kafka clients connect with configuration changes. Test for any features you rely on, and use a tier that supports it.

For a limited retention period that depends on the tier, with longer retention in higher tiers. For permanent storage, use Capture to write events to storage.

Use the event processor with checkpoints, make processing idempotent, monitor lag and make sure retention is long enough for the consumer to recover.

When traffic exceeds the capacity of your units, the service slows or rejects requests. Handle it with retries and backoff, and add capacity or reduce traffic.

Geo-disaster recovery copies metadata only and lets you fail over configuration to a paired namespace. Geo-replication, available for Premium and Dedicated tiers, replicates both metadata and event data. Confirm current details in the documentation.

Prefer Microsoft Entra and managed identities with narrow roles, restrict networks with firewalls and private endpoints, use customer-managed keys where required and never embed the namespace root key in applications.

Usually not. It has a restricted feature set, such as no Capture or Kafka endpoint, and limited consumer groups. Most production workloads start at Standard or above.

Azure Event Hubs is the wide river for streaming data. Producers pour in events at enormous volume, the service spreads them across partitions and keeps them in an ordered, replayable log for a limited time, and any number of consumer groups read them independently, while Capture, stream processing and analytics services turn the flow into archives, dashboards and actions. Success depends on a handful of design choices: picking Event Hubs for streams rather than queues or notifications, planning partitions and keys for scale and ordering, building idempotent consumers with sensible checkpointing, securing access with identities and private networking, choosing the right tier and replication approach, and watching lag, throttling and silence so that no data is lost quietly. Because tiers, replication features and integrations keep evolving, always confirm current capabilities and limits in the official documentation.