A fully managed engine that turns a constant river of events into answers, alerts, and summaries while the data is still fresh.

Batch versus streaming

Most people first meet data as a batch: a big file or table that sits still while you analyze it. A nightly sales report is a batch job. It looks at everything that happened yesterday, and it finishes when it finishes. This is like reading a finished book.

Streaming is different. Data arrives continuously, one small piece at a time, and the pieces never stop. Each piece is called an event: a temperature reading, a click on a website, a card payment, a GPS position. Analyzing a stream is like listening to a live radio broadcast. You cannot wait for the end, because there is no end. You must make sense of each moment as it passes.

What the service does

Azure Stream Analytics, often shortened to ASA, is a managed service for real-time analytics. You tell it three things: where events come from (an input), what to calculate (a query), and where results should go (an output). Together these form a job. Once started, the job runs day and night, reading events, applying the query, and writing results. Microsoft runs the machines, handles failures, and keeps state safe. You do not manage servers.

The query language looks very much like SQL, the standard language for asking questions of tables, with extra features for time. This is a big advantage, because many people already know SQL and can become productive in hours instead of weeks.

What problem does it solve?

Building your own streaming system means handling many hard things: reading from many sources at once, remembering partial results between events, dealing with events that arrive late or out of order, spreading work across machines, restarting safely after a crash without losing or duplicating work, and watching it all. ASA packages these hard problems into settings and built-in behaviors, so you can focus on the question you want answered.

Where it is used

Teams use it for factory and device monitoring, website click analysis, fraud signals on payments, fleet tracking, live dashboards, alerting, and for cleaning and reshaping data before it lands in a database or a data lake.

Streaming pipelines are made of several services. Knowing who does which job prevents confusion and wasted effort.

flowchart LR
    SRC["Devices, apps and services producing events"] --> ING["Ingestion: Event Hubs or IoT Hub"]
    ING --> ASA["Stream Analytics job"]
    REF["Reference data in storage or database"] --> ASA
    ASA --> DB["Database or data lake"]
    ASA --> DASH["Live dashboard"]
    ASA --> ACT["Alerts and automation"]
    

Stream Analytics versus writing code

You can analyze streams with your own code, for example in Azure Functions or on a Spark platform. That gives total freedom, but also total responsibility for state, time, scaling, and recovery. ASA trades some freedom for speed and safety: you write a query, and the service takes care of the hard parts. When your logic is mostly filtering, grouping, joining, and time windows, ASA is often the fastest way to production. When you need heavy custom algorithms or very complex stateful logic, code-based tools may suit better.

A handful of words appear in every Stream Analytics conversation. Learn them once and the rest becomes easy.

Event and stream

An event is a small record of something that happened, usually with a time and a few fields. A stream is an unbounded sequence of events ordered mostly by time. The word unbounded means it has no planned end.

Job

A job is the unit you create, start, stop, and pay for. It contains inputs, a query, outputs, and settings such as how many compute units it gets. A job is either running or not. When stopped, it uses no compute and costs nothing for processing.

Input

An input is a connection to a source of events. There are two kinds. Stream inputs carry the live flow and are read continuously. Reference inputs carry slow-changing lookup data, such as a table of device names and locations, that is loaded into the job’s memory and used to enrich events.

Query

The query is the logic of the job, written in a SQL-like language. It can filter events, calculate over time windows, join streams, call functions, and shape the results. A job may contain several steps, each building on the previous one, and may write to several outputs.

Output

An output is where results go: a database, a storage account, another event stream, a dashboard service, or a message queue.

Streaming unit

A streaming unit, or SU, is the measure of compute a job gets: a combined bundle of processing power and memory. More SUs let a job handle more events per second and keep more state. You choose the number, and you pay by SU and by hour while the job runs.

Window

A window is a slice of time over which the job calculates something, such as “the average temperature in each five-minute slice”. Because streams never end, windows are how you turn an endless flow into finite answers. Chapter 7 covers them in detail.

Partition

A partition is one lane of a stream. Big streams are split into several lanes so that many workers can read them at once. Matching lanes from input to output lets a job scale almost without limit.

flowchart TD
    JOB["Stream Analytics job"] --> IN["Inputs"]
    JOB --> Q["Query with steps"]
    JOB --> OUT["Outputs"]
    JOB --> SU["Streaming units"]
    JOB --> SET["Settings: time policy, error policy, compatibility level"]
    IN --> SI["Stream inputs"]
    IN --> RI["Reference inputs"]
    

Where events come from, where results go, and the small rules that keep the connections healthy.

Stream inputs

Reference inputs

Reference data usually comes from a file in storage or from a table in a database. The job loads it into memory and refreshes it on a schedule, or picks a new file when one appears in a dated folder path. Since it sits in memory, it must be small enough to fit within the job’s resources.

Outputs

flowchart LR
    EH["Event Hubs"] --> J["Stream Analytics job"]
    IH["IoT Hub"] --> J
    BL["Blob storage as stream"] --> J
    RF["Reference data from storage or SQL"] --> J
    J --> O1["SQL database"]
    J --> O2["Data lake"]
    J --> O3["Event Hubs for next stage"]
    J --> O4["Power BI dashboard"]
    

Connection rules worth knowing

• Own consumer group: When reading from Event Hubs, give each job input its own consumer group. A consumer group is a separate reading position on the same stream. Sharing one between readers causes them to disturb each other.
• Managed identity: Prefer letting the job sign in as a managed identity (an automatic identity managed by Azure) over storing connection secrets.
• Output error policy: You decide what happens if an output write fails: retry until it works, which protects data but can stall the job, or drop the failed event, which keeps the job moving but loses that event.
• Write sizing: Outputs batch their writes for efficiency. The batch size and wait time influence delay and cost.

What sits inside the service, seen from a distance, and why that matters for how jobs behave.

You never see the machines behind a Stream Analytics job, but understanding their layout explains many behaviors. A job is run by an engine that reads inputs, executes your query as a graph of steps, keeps state in memory, saves progress markers called checkpoints, and writes outputs.

flowchart TD
    ADM["You using portal, code editor or templates"] --> CP["Control plane: validate and store job definition"]
    CP --> ENG["Job engine on allocated streaming units"]
    SRC["Input partitions"] --> RD["Readers"]
    RD --> PIPE["Query steps and window state in memory"]
    PIPE --> WR["Writers"]
    WR --> SNK["Output destinations"]
    PIPE --> CK["Checkpoint storage"]
    ENG --> MON["Metrics and logs in Azure Monitor"]
    ENG -.-> RD
    ENG -.-> PIPE
    ENG -.-> WR
    

Where jobs can run

Time is the trickiest part of streaming. Get it wrong and your numbers will be quietly wrong.

Two kinds of time

Event time is when something actually happened, according to the device or app that created the event. Arrival time is when the event reached the service. In a perfect world they would be equal. In the real world they differ, sometimes by a lot. A phone in a tunnel may collect readings for ten minutes and then send them all at once.

By default, ASA uses the arrival time at the input source. If you want analysis based on when things really happened, you tell the query which field holds the event time, using a clause that sets the timestamp. For most business questions, such as “orders per minute at the shop”, event time is what you want.

Late and out-of-order events

Events do not always arrive in order. Network delays, retries, and offline devices create two problems. An out-of-order event arrives after a newer event was already seen. A late event arrives long after its time slot, perhaps after the window it belongs to has closed.

The job needs a rule for how long to wait. It cannot wait forever, because the results would never be released. The service keeps a moving marker called the watermark, which is its estimate of “all events older than this time have arrived”. A window produces its result only once the watermark passes the end of the window.

Time policy settings

Five kinds of windows cover almost every need. Pick the one that matches the question you are asking.

To count or average something in a stream, you must choose which events to group together. A window is a rule for that grouping by time. ASA offers five types.

Tumbling window

A tumbling window cuts time into equal, non-overlapping slices, such as every five minutes. Each event belongs to exactly one slice. Use it for regular reports: “orders per five minutes”. It is the simplest and most common.

Hopping window

A hopping window has a size and a hop. For example, a ten-minute window that moves forward every two minutes. Windows overlap, so one event can appear in several. Use it for smooth moving averages: “average over the last ten minutes, refreshed every two”.

Sliding window

A sliding window produces output only when something changes, looking back over a fixed length. It fits conditions such as “alert whenever more than three failures occur in any ten-second period”, because it checks every possible ten-second stretch instead of fixed slots.

Session window

A session window groups events that happen close together and closes after a quiet gap. A visit to a website is the classic case: clicks stay in one session until the visitor is silent for, say, twenty minutes. The size of each window depends on the activity itself.

Snapshot window

A snapshot window groups events that share exactly the same timestamp. It answers questions such as “how many events happened at this very instant?” and is rarely needed, but useful for precise comparisons.

flowchart TD
    Q["Which question are you asking"] --> A{"Regular fixed slices"}
    A -->|"yes, no overlap"| T["Tumbling window"]
    A -->|"yes, overlapping"| H["Hopping window"]
    A -->|"no"| B{"Triggered by activity"}
    B -->|"any recent period"| S["Sliding window"]
    B -->|"until a quiet gap"| SE["Session window"]
    B -->|"same instant"| SN["Snapshot window"]
    

Windows and delay

A window produces its result when it is complete, which means the end of the window plus any time the job waits for stragglers. A one-hour tumbling window therefore cannot give you an answer faster than an hour. If you need quick feedback, use short windows, or a sliding window that responds as events arrive, and keep longer summaries for reporting.

Grouping by key

Windows are usually combined with a grouping key. For example, “average temperature per machine in each five-minute window” keeps a separate running calculation for every machine. The more distinct keys you have, the more memory the job needs, which matters when you choose the number of streaming units.

Follow one event from the moment it is created to the moment it shows up in a result, and see how a job moves through its own states.

sequenceDiagram
    participant Dev as Device or app
    participant EH as Event Hubs
    participant ASA as Stream Analytics job
    participant REF as Reference data in memory
    participant OUT as Output database
    Dev->>EH: Send event with timestamp
    EH->>ASA: Reader pulls event from partition
    ASA->>ASA: Decode and apply time policy
    ASA->>REF: Look up device details
    REF-->>ASA: Location and owner
    ASA->>ASA: Add to window and update running totals
    ASA->>ASA: Watermark passes window end
    ASA->>OUT: Write summarized result in a batch
    ASA->>ASA: Save checkpoint
    

The states of a job

A job moves through states that you can see in the portal and in metrics.

Choosing where to start

When you start a job, you decide at which point in the input stream it begins. It can start from now, from a custom time in the past, or from when last stopped. Starting from a past time lets you rebuild results after fixing a bug, as long as the ingestion service still holds the old events. Starting from “when last stopped” continues where the job left off, so a planned stop does not lose data.

You will not need to memorize a new language. If you know the basics of SQL, you already know most of it.

The query language of Stream Analytics follows the shape of SQL: choose fields, name where data comes from, filter, and group. The additions are about time. Because this guide avoids code samples, here is how each building block works in words.

Selecting and filtering

You choose which fields to pass on and, with a condition, which events to keep. A typical first step is cleaning: drop events with missing device numbers, convert text to numbers, and discard readings that are impossible, such as a negative speed.

Grouping with windows

To summarize, you group events by a key, such as the machine, and by a window, such as five minutes. Then you ask for counts, sums, averages, minimums, or maximums. The result is one row per machine per window.

Joining a stream with reference data

To enrich each event with details from a lookup table, you join the stream to the reference input on a shared key, such as the device number. Because reference data lives in memory, this is fast. Think of adding the customer’s name to each order line.

Joining two streams

You can also join two live streams, for example payments and shipping notices. Because both are endless, the join needs a time limit that says how far apart in time two matching events may be, such as “shipping within one hour of payment”. Without such a limit, the job would have to remember every event forever.

Looking at earlier events

Some questions need the previous event, such as “is this reading higher than the last one from the same device?” Special functions can look back at the last event for each key, optionally within a time limit, and this is the building block for detecting changes and gaps.

Steps and multiple outputs

A long query can be split into named steps, each feeding the next, like stages on an assembly line. A job can also send different results to different outputs, for example detailed rows to a data lake, hourly totals to a database, and alerts to a message queue.

flowchart LR
    IN["Raw events"] --> S1["Step 1: clean and convert"]
    S1 --> S2["Step 2: enrich with reference data"]
    S2 --> S3["Step 3: window and aggregate"]
    S2 --> O1["Output A: detailed rows to data lake"]
    S3 --> O2["Output B: summaries to database"]
    S3 --> S4["Step 4: apply alert rules"]
    S4 --> O3["Output C: alerts to message queue"]
    

Built-in extras

How a job grows from a few events a second to millions, and what blocks it from growing.

Streaming units

A streaming unit is the amount of compute and memory a job gets. More units mean more events per second can be processed and more state held in memory. Windows with many keys, large reference data, and complicated joins all need more memory, so they need more units even when event rates are modest. Newer pricing options allow finer-grained, smaller steps than the original ones, so check the current choices when you size a job.

Parallel jobs

The secret of scaling is parallelism: splitting the work so that many workers handle different lanes at the same time. A job is fully parallel, sometimes called embarrassingly parallel, when each input partition can be processed independently and written to a matching output partition, with no worker needing data from another lane.

flowchart LR
    P0["Input partition 0"] --> W0["Worker 0"] --> O0["Output partition 0"]
    P1["Input partition 1"] --> W1["Worker 1"] --> O1["Output partition 1"]
    P2["Input partition 2"] --> W2["Worker 2"] --> O2["Output partition 2"]
    P3["Input partition 3"] --> W3["Worker 3"] --> O3["Output partition 3"]
    

What keeps a job from being parallel

• Mismatched partitions: The input has a different number of lanes than the output, so events must be reshuffled.
• Grouping by a different key: If the input is split by device but you group by city, events for one city sit in many lanes and must be brought together.
• Global calculations: A single total across all devices needs everything in one place.
• Outputs that cannot take parallel writes: Some destinations serialize writes, which cancels the benefit.

When a global result is necessary, a common trick is a two-stage design: compute partial answers in parallel per lane, then combine the small partial results in a second step. This moves most of the work into the parallel part.

Scaling steps in practice

Machines fail. A good streaming service plans for that, so that nothing is lost and nothing is counted twice.

Checkpoints and recovery

The engine regularly saves checkpoints, which record how far each input has been read and what the in-memory window state looks like. If a machine fails or the service moves your job for maintenance, the job restarts on healthy resources, restores the latest checkpoint, and continues reading from the saved position. Windows that were half-filled continue to fill from where they were.

Delivery guarantees

Three words describe how many times an event takes effect.

The practical advice: assume an output may occasionally see a repeated row after a recovery, and prefer destinations that can absorb repeats safely, for example tables where a key identifies each result, so a second write replaces the first. Check the current documentation for the exact guarantee of each output type, since they differ.

Handling bad data

Real streams contain surprises: broken records, wrong types, missing fields. Decide ahead what to do. Count them using metrics. Filter obviously bad events in the first step of the query. Consider writing rejected events to a separate store, so that someone can study them and fix the producer.

Availability and the service promise

Microsoft publishes an availability promise, a service level agreement, for the service. It covers the platform, not the correctness of your query, and it does not include your inputs and outputs, which have their own promises. Read the current document for exact figures when you need to quote one.

Regional failures and disaster recovery

If an entire region becomes unavailable, a single job in that region stops. A common recovery design runs a second, matching job in a paired or alternative region, reading from a replicated or mirrored input and writing to its own output, which can be kept warm or started during an emergency. Because streams are time-sensitive, decide in advance how much delay and how many repeated or missing results are acceptable during the switch, and rehearse the plan.

flowchart TD
    SRC["Event producers"] --> EH1["Event Hubs in region one"]
    SRC -.-> EH2["Event Hubs in region two"]
    EH1 --> J1["Primary job"]
    EH2 --> J2["Standby job"]
    J1 --> DB1["Output in region one"]
    J2 --> DB2["Output in region two"]
    DB1 -->|"replication"| DB2
    

Event streams often carry customer and device data. Good security is many layers, not a single lock.

flowchart TD
    A["Who can manage the job"] --> B["Role based access control and approvals"]
    B --> C["How the job signs in: managed identity"]
    C --> D["Network path: private endpoints or cluster network integration"]
    D --> E["Encryption in transit and at rest"]
    E --> F["Data minimization inside the query"]
    F --> G["Audit logs and alerts"]
    

Who can change the job

A streaming query decides what is alerted, stored, and discarded, so editing it is powerful. Use role-based access control to limit who can create, edit, start, and stop jobs. Keep the query in version control with reviews, and require the same discipline as for application code.

How the job signs in

The job needs permission to read inputs and write outputs. The best way is a managed identity, an identity that Azure creates and rotates for the job, so no password sits in a setting or a file. Give it only the roles needed, such as read on the input and write on the output, and nothing more. This is the principle of least privilege.

Network protection

By default, jobs talk to inputs and outputs over their public service endpoints, protected by encryption and sign-in. For stricter needs, a dedicated Stream Analytics cluster can connect over private network links, so traffic stays on your own virtual network and the data services need not be exposed to the internet.

Encryption

Data moves between services over encrypted connections. The service also stores job state and settings encrypted. If your rules demand it, you can supply your own key for protecting data stored by the job, so you control its lifecycle.

Use less data

The safest data is data you never carry. Inside the query, drop fields you do not need, mask or hash personal details before they reach long-term storage, and send only summaries to wide-audience dashboards. Think of the query as a privacy filter, not just a calculator.

Make it correct first, then make it fast. Most slowness has a small list of causes.

Backlog: the main warning light

When events arrive faster than the job can process them, a pile of backlogged events builds up in the input. Latency then grows, because each new event waits behind older ones. A small, steady backlog is a signal to act. A growing one means the job is falling behind and, if the input keeps events only for a limited time, data could eventually expire before it is read.

Common causes and cures

Tuning habits

A streaming job runs unattended, day and night. You need alarms that speak before your customers do.

Key metrics

Logs

Metrics say that something is wrong. Logs say what. Through diagnostic settings you can send job logs to a Log Analytics workspace, a storage account, or an event stream. Execution logs describe events that happen while the job runs, such as failures to write. Authoring logs describe actions taken on the job definition, like edits and starts. Together they give an audit trail and a troubleshooting guide.

Alerts

Create alerts for the symptoms users feel, not just for machine health.

• Job state changed to degraded or failed.
• Watermark delay above the limit the business can accept.
• Backlog above a threshold for several minutes.
• Input volume dropped to zero when it should not, which may mean a producer broke.
• Error counts above normal.

The “no events” alert is easy to forget. A job that is healthy but reading an empty stream looks perfectly fine on every machine metric, while your business is blind.

Building the first job is quick. Building a team habit that keeps jobs safe, repeatable, and affordable is the real craft.

Ways to build a job

A healthy development loop

Changing a running job

Many changes to a query require stopping the job, editing, and starting again. Because state in memory is lost on restart unless the job replays input from an earlier time, plan changes with care: choose a start time that covers the period needed to refill windows, and check how outputs handle repeated rows. For critical pipelines, many teams run the new version side by side with the old, writing to a different destination, compare the outputs, and then switch.

Edge deployments

When jobs run on IoT Edge devices, the query is packaged and delivered to devices through the device management system. Plan versioning and staged rollout, because a bad query on thousands of devices is harder to fix than one in the cloud.

Understanding and controlling cost

The model is simple: you pay for streaming unit capacity for every hour the job is running. A stopped job incurs no processing charge. Dedicated clusters have their own capacity pricing, and the services around the job, such as event ingestion, databases, storage, and monitoring, are billed separately. Use Microsoft’s pricing calculator for current numbers.

A pattern is a recipe that worked many times. An anti-pattern is a trap that looks tempting and hurts later.

Helpful patterns

flowchart TD
    SRC["Devices and apps"] --> EH["Event Hubs"]
    EH --> HOT["Stream Analytics: hot path"]
    EH --> CAP["Capture to data lake: cold path"]
    HOT --> AL["Alerts and live dashboard"]
    HOT --> SUM["Summary tables in database"]
    CAP --> LAKE["Data lake with raw history"]
    LAKE --> BATCH["Batch analysis and reprocessing"]
    

Anti-patterns to avoid

A practical checklist to carry into any project.

Mistakes beginners make, and how to dodge them

• Expecting instant results from long windows. A one-hour window answers once an hour. Match the window to the delay the business accepts.
• Testing only with clean data. Real streams contain bad records, late events, and odd values. Test with the ugly ones.
• Overlooking the output. The job can be perfect while the database it writes to is the bottleneck.
• Not planning for replay. Sooner or later a bug will need fixing retroactively. Know how you will rebuild results.
• Monitoring the machine, not the meaning. A healthy job on an empty stream still means a blind business. Alert on missing data too.

Four typical situations, described as realistic example setups, showing how the pieces combine.

How to talk about it in an interview

A strong short answer might sound like this: “Azure Stream Analytics is a managed real-time analytics service. I read from Event Hubs or IoT Hub, write a SQL-like query with the right window type, enrich with reference data, and write to outputs such as a database, a data lake, or an alert queue. I use event time with sensible late-arrival settings, size streaming units with headroom, make the job parallel by aligning partitions, make outputs safe for repeats, and monitor watermark delay, backlog, and errors.” Then be ready to explain the reasoning behind each choice using the ideas in this guide.

You have now followed an event from a device, into an ingestion service, through a job that reads, decodes, orders by time, enriches, windows, and writes, and finally out to a dashboard, a database, or an alert. The thread running through everything is simple: Azure Stream Analytics lets you hand the hard machinery of streaming to Microsoft, while you keep responsibility for clear time rules, well-chosen windows, parallel design, repeat-safe outputs, and honest monitoring.