Begin with the problem of data that never stops arriving, then see how a streaming service answers it.

The problem

A factory has thousands of sensors. Each one sends a temperature reading every second. A bank processes thousands of card payments a minute. A delivery company receives a location from every truck every few seconds. Each single reading is tiny and nearly useless. The value comes from patterns across many readings: a machine that is slowly overheating, a card used in two countries within ten minutes, a truck that has stopped for too long. Finding these patterns after a nightly batch job is often too late. The machine has already broken. The card has already been drained.

What the service is

Azure Stream Analytics is a fully managed real-time analytics service. “Fully managed” means Microsoft runs the machines, patches them and keeps them alive. You describe what you want in a query written in a language that looks like the database language many people already know, and you tell the service where events come from and where results should go. The service then runs your query continuously, day and night, over the flowing events.

What you gain

To see why streaming services exist, look at how data processing grew up.

The batch era

For decades, data was processed in batches. During the day, systems wrote records into files or databases. At night, a big job read everything collected, calculated totals and wrote reports. Managers read the reports the next morning. This worked well when business moved slowly and data volumes were small. A batch job is simple to understand: it has a start, an end and a clear input.

The limits of waiting

As systems grew, three things changed. Volumes became huge, so night jobs began to take longer than the night. Business expectations changed, so a fraud alert tomorrow was worth far less than one right now. And machines began to talk to each other, so a thermostat or a payment terminal needed an answer in a second, not a day.

Micro-batches and true streams

A first step was running batches more often, every hour, then every minute. This is called micro-batching. It reduces delay but adds complexity and cannot easily reach one-second answers. True stream processing treats data as an endless sequence of events and processes each one, or small groups, the moment it arrives, keeping running state in memory.

Streaming in the cloud

Early stream engines needed experts to build and run clusters. Cloud services took that burden away. Azure Stream Analytics grew from this idea: give people a way to write a few lines of query and have a managed engine run it at scale. It sits beside other Azure options for bigger or more custom needs, which a later chapter compares.

Events versus records

An event is a small fact that something happened at a moment in time: “sensor 42 read 71 degrees at 10:00:03.” Unlike a row in a table that you update, an event is never changed. Events pile up in order, like entries in a diary. Streaming systems read the diary as it is written.

Each term below says what it is, why it exists, a picture from daily life and an example.

How the words fit together

Read it as a pipeline: a job reads one or more inputs, optionally enriches them with reference inputs, runs the query over windows measured in event time, and writes to one or more outputs. The job’s size is its streaming units. When the input is split into partitions, the job can work on many lanes in parallel.

Here is where the service sits in a typical real-time pipeline.

flowchart LR
  subgraph SRC["Event sources"]
    D1["Devices and sensors"]
    D2["Applications"]
    D3["Web and mobile clients"]
  end
  D1 --> IOT["IoT Hub"]
  D2 --> EH["Event Hubs"]
  D3 --> EH
  IOT --> SA["Stream Analytics job"]
  EH --> SA
  REF["Reference data in storage"] --> SA
  SA --> DB["Database"]
  SA --> LAKE["Data lake storage"]
  SA --> BUS["Service Bus or Event Hubs"]
  SA --> DASH["Live dashboard"]
  BUS --> FUNC["Alerting function"]

Event sources

Events begin on devices, applications and websites. They are sent into an ingestion service that holds them safely for a short time, in order, until a reader takes them. This holding area, such as an event hub or an IoT hub, is a buffer between the busy world and the job. If the job stops for a while, events wait in the buffer and are read when it returns.

The job

The job is the engine. It reads events from the buffer, runs the query, and keeps in memory the running state that windows and joins need, such as “the count so far in this minute.” It also writes progress markers so it can recover.

Outputs

Results are written to one or more destinations. A database may keep the latest numbers for an application. A data lake may hold everything for later analysis. A message service may carry alerts to other systems. A dashboard service may show live charts.

Supporting pieces

A job needs something to read. There are two kinds of input with very different behavior.

Stream inputs

A stream input is a flow of new events with no end. The most common sources are event hubs, IoT hubs and files appearing in storage. Each event carries data fields and a time. The job reads the stream continuously and never expects it to finish.

Event serialization

Events arrive in a format such as JSON, CSV or a compact binary format. You tell the job which format and character encoding to expect. If a producer sends a different shape than the job expects, you get input errors. Keep formats stable and version them carefully when they change.

Consumer groups

An event hub can be read by several independent readers, each using its own consumer group. Give each job its own group so jobs do not steal events from one another or from other applications. Sharing a group is a common cause of strange missing events.

Reference inputs

A reference input is a table that changes slowly or not at all. It is loaded into memory and used to add meaning to events. Example: an event has a device number, and the reference table maps device numbers to the building and floor where each device lives. The job joins the two so output shows the building and floor. Reference data usually comes from files in storage or from a database, and can be refreshed on a schedule, so the job sees new values over time.

Early arrival and event ordering

Streams from different partitions are not guaranteed to arrive in perfect time order. A device in a poor network area may deliver its events late, in bursts. The job has settings that decide how much disorder and lateness it will tolerate, explained in the chapter on time.

Choosing the right source

The query is the heart of a job. You do not need to memorize code to understand how it thinks.

A language that looks familiar

The language is a dialect of the one used to ask questions of databases. You pick fields from an input, filter which events you care about, group them, and name an output. The difference is that, in a database, a question runs once over stored rows and finishes. Here, the question runs forever over events that keep arriving, and each answer is emitted as time passes.

The usual building blocks

Several outputs from one job

A single job can contain several queries. One may send raw filtered events to a data lake, another may send per-minute averages to a database, and a third may send alert events to a message service. All share the same input reading, which is cheaper than running separate jobs.

Built-in functions

The language has functions for text, dates, numbers, arrays and records, for geographic checks such as “is this point inside this area,” and for simple statistical and machine learning helpers, like spotting unusual values. You can also bring your own functions written in common languages for special logic, though keeping queries simple is wise.

Thinking in streams

When you write a query, ask three questions. What slice of time does each answer cover? What groups am I counting separately? When should an answer be released? Many beginners forget the first question and ask, for instance, for an average without saying over what time. The service will reject such a question for a stream, because an endless stream has no end to average over.

Follow a single event from its birth to a result in a database.

sequenceDiagram
  participant DEV as Device
  participant EH as Event hub
  participant JOB as Stream Analytics job
  participant ST as In-memory state
  participant OUT as Database output
  DEV->>EH: Send reading with event time
  EH->>JOB: Deliver from a partition
  JOB->>JOB: Parse and check format
  JOB->>JOB: Apply filter
  JOB->>ST: Add to window state
  JOB->>JOB: Watermark passes end of window
  JOB->>ST: Compute window result
  ST-->>JOB: Result
  JOB->>OUT: Write result in a batch
  JOB->>JOB: Save checkpoint

Step by step

Why batches inside a stream

Although the idea is event-by-event, the engine works on small groups of events for efficiency, much as a bus carries many passengers at once. These groups are small, so delay stays in the range of seconds.

Backpressure

If an output is slow, results pile up inside the job. If they pile up enough, the job slows its reading, and then events accumulate in the buffer instead. This slowing is called backpressure. It protects the job from crashing, but it shows up as growing delay. The monitoring chapter explains how to see it.

Memory is the real limit

Windows and joins keep state in memory. Counting events per device per minute for a million devices keeps a million small counters. State size grows with the number of groups and with the length of windows. This is why streaming units, which include memory, matter so much.

Time is the hardest idea in streaming, and the part that most often corrupts results. Take this chapter slowly.

Two timestamps

Every event can be placed on two clocks. Event time is when the thing really happened, written by the device or application. Arrival time is when the ingestion service received it. If a phone is offline in a tunnel for ten minutes and then sends a burst of readings, their event times are spread across ten minutes, but their arrival times are all now. If you count by arrival time, you would wrongly say that traffic jumped just now. Counting by event time tells the true story.

Choosing the clock

The job can use either clock for windows. Using arrival time is simple and never has late data, but it reflects delivery delays. Using event time gives correct business answers, but the job must decide how long to wait for stragglers. A field in the query points to the time stamp inside the event, and the job uses it for all time decisions.

Out-of-order and late events

Two settings handle disorder. The out-of-order tolerance says how far an event may be out of sequence within the stream and still be re-sorted into place. The late arrival tolerance says how late, compared with the newest time the job has seen, an event may arrive and still be counted in its proper window. Events beyond the limit are handled by a chosen policy: either they are dropped, or their event time is adjusted to the nearest acceptable time so they are not lost.

flowchart TD
  E["Event arrives with event time T"] --> C1{"Within out-of-order tolerance?"}
  C1 -->|"Yes"| P["Sorted into place and processed"]
  C1 -->|"No"| C2{"Within late arrival tolerance?"}
  C2 -->|"Yes"| P
  C2 -->|"No"| POL{"Policy"}
  POL -->|"Adjust"| ADJ["Event time moved to nearest allowed time"]
  POL -->|"Drop"| DROP["Event discarded and counted"]
  ADJ --> P

The cost of waiting

A bigger tolerance gives more accurate results but delays them. If the late arrival tolerance is five minutes, a one-minute window is released only after the job believes no more events with that window’s times will come, so results trail real time by about that tolerance. Short tolerances give fast results but risk missing stragglers. The right value depends on your devices and networks, so measure real delays before choosing.

Idle sources

If one partition goes quiet, the job cannot be sure no events are coming, and may hold back results. A setting for the maximum wait on idle partitions tells the job when to move on. This prevents one silent device group from stalling all output.

Clock quality

Device clocks drift and can be set wrongly. An event stamped far in the future can push the job’s sense of time ahead and cause genuine events to be treated as late. Validate time stamps at ingestion if possible, and set limits in the job for how far ahead a stamp may be.

A window turns an endless stream into pieces you can calculate on. The service offers five types.

Tumbling windows

Picture a calendar with one box per day. Every event falls in exactly one box. Tumbling windows are the simplest and cheapest, and suit regular reports such as totals per minute.

Hopping windows

These overlap, like a moving average on a chart. A ten-minute window that hops every minute means each event counts in ten different windows. They give smoother numbers but multiply state and output. A tumbling window is a hopping window whose hop equals its size.

Sliding windows

A sliding window looks back a fixed time from every event and reports only when the content changes. It is the right tool for threshold alerts that must react at once, such as too many failures in a short span. It can produce many outputs on busy streams.

Session windows

Sessions have no fixed size. Each starts at an event and extends while more events from the same key arrive within a timeout. When the gap exceeds the timeout, the session ends. They fit user behavior and equipment cycles, such as one machine run.

flowchart LR
  subgraph TUM["Tumbling: fixed, no overlap"]
    T1["0-5 min"] --> T2["5-10 min"] --> T3["10-15 min"]
  end
  subgraph HOP["Hopping: fixed, overlapping"]
    H1["0-10 min"] -.-> H2["5-15 min"] -.-> H3["10-20 min"]
  end
  subgraph SES["Session: grows until quiet gap"]
    S1["Burst of events"] --> S2["Gap longer than timeout ends session"]
  end

Choosing a window

Windows and keys

Windows are almost always combined with a key, such as device or customer. Each key has its own window state. The number of distinct keys therefore drives memory use. A key with millions of unique values and a long window can use large amounts of memory.

Real questions often need two sources of information. Joins let a job combine them.

Joining a stream with reference data

This is the easy and cheap kind of join. For each event, the job looks up matching fields in the reference table, such as turning a device number into a location. No time condition is needed, because the reference table is treated as a fixed list at that moment. Most enrichments use this approach.

Joining two streams

Here, the job matches events from two flows, such as a click and a later purchase from the same user. Because both flows are endless, the match needs a time rule: events match only if they happened within a stated time distance of each other, such as within ten minutes. The job must hold events from both streams in memory for that distance, waiting for partners. A longer distance means more memory.

Kinds of stream join

Refreshing reference data

Reference files are loaded when the job starts and then refreshed on a pattern, usually based on the date and time in the file path. A new file appears for the next period, and the job picks it up at the right moment. Time-versioned reference data lets the job apply the correct values for each event time, which matters when prices or rules change at known moments.

Size limits

Reference data is held in memory for each part of the job, so very large tables cause pressure. Keep the table small, with only needed columns. For very large lookups, consider doing the lookup downstream or using a different tool.

Common join mistakes

Getting results to the right place safely is half of the job. Learn what the service promises and what it does not.

Common outputs

What delivery guarantees mean

A guarantee describes what happens to each result when things fail. At-least-once means every result is delivered, but after a failure some may be delivered twice. At-most-once means none are duplicated, but some may be lost. Exactly-once means each result takes effect once. The service provides exactly-once processing inside the job for its internal state, and, for some outputs, exactly-once delivery. For others, such as message streams, delivery can be at-least-once, so duplicates are possible after recovery.

Designing for duplicates

The safe approach is to make downstream consumers idempotent: applying the same result twice has the same effect as applying it once. For instance, write results with a key made from the window and device, so a repeated result overwrites itself rather than adding another row. Do not assume duplicates never happen.

Batching and output rate

The job groups outputs into batches. Settings control the batch size and, for some outputs, how many writers run in parallel. A slow output, such as a small database, becomes the bottleneck for the whole job. Match the output capacity to expected results, and use partition-aware outputs to spread writes.

Error handling at the output

If an output rejects a row, for example because of a data type problem, the job can either retry forever, which stalls it, or drop the row and continue. The error policy setting decides which. For critical data, choose retry and alert on lag. For metrics that tolerate gaps, drop and log. Whatever you choose, make the choice deliberately.

How fast a job runs depends on how much compute you give it and how well the work can be split.

Streaming units

A streaming unit is a bundle of compute and memory. A job with more streaming units can hold more state and process more events. In many configurations units come in steps, and a job whose query can split into parallel lanes can use many units at once. The service shows a utilization figure that tells you how much memory the job uses relative to what it has.

Parallel jobs

The ideal case is called an embarrassingly parallel job. The input is split into partitions, the query keeps work for each partition separate, and the output is also split into matching partitions. Each lane runs alone with no need to talk to other lanes. Such a job scales almost in a straight line: twice the lanes and units handle roughly twice the events.

flowchart LR
  subgraph IN["Input partitions"]
    P1["Partition 1"]
    P2["Partition 2"]
    P3["Partition 3"]
    P4["Partition 4"]
  end
  P1 --> W1["Worker lane 1"]
  P2 --> W2["Worker lane 2"]
  P3 --> W3["Worker lane 3"]
  P4 --> W4["Worker lane 4"]
  W1 --> O1["Output partition 1"]
  W2 --> O2["Output partition 2"]
  W3 --> O3["Output partition 3"]
  W4 --> O4["Output partition 4"]

What breaks parallelism

You can often repartition explicitly in the query, to spread work and then combine small partial results in a second step. This two-stage approach, computing partial counts in lanes and then summing them, keeps most work parallel.

A worked sizing example

These numbers are illustrative, not official limits. Suppose 20,000 devices each send one event per second, so the job must handle 20,000 events per second. If one lane with a given number of units comfortably handles 2,500 events per second for your query, you need 20,000 divided by 2,500, which is 8 lanes. If the input has 8 partitions and each lane needs, for example, 6 units, you would plan for 8 times 6, or 48 units, then add a margin of about 30 percent for spikes. That gives roughly 62 units, so you might start at 64 and watch utilization. If utilization stays below 40 percent for a week, reduce units; if it often passes 80 percent, add more. Test with your real query, because a complex query changes the numbers greatly.

Autoscale and manual scale

You can change the number of units manually, and some configurations support rules that scale on a schedule or on load. Changing units may need a short restart, so make changes during quiet periods and test them.

A job moves through states, and the way you start it decides what data it processes first.

stateDiagram-v2
  [*] --> Created
  Created --> Starting: start
  Starting --> Running: resources ready
  Running --> Degraded: errors or lag
  Degraded --> Running: recovered
  Running --> Stopping: stop
  Degraded --> Failed: unrecoverable error
  Failed --> Starting: restart
  Stopping --> Stopped
  Stopped --> Starting: start
  Stopped --> [*]: delete

Start modes

Catching up

If a job starts from the past, it will read stored events as fast as it can. During catch-up, windows close quickly and outputs surge. Make sure the outputs can take the burst, and consider extra units during catch-up.

State and restarts

Starting from the last stopped time normally restores saved state, so windows are not lost. But changes to the query may make saved state unusable. When you change a query, test whether the job can resume with state or must start fresh, and plan the consequences for windows that were in progress.

Compatibility levels

The service has numbered compatibility levels that fix certain behaviors, such as how some functions treat data types. New jobs get the recommended level. Existing jobs keep their level until you change it, to avoid breaking results. When you raise the level, test, since small differences can change numbers.

Stopping and deleting

You pay for the job while it exists in a running state, and often for allocated units even when not processing much. A job you do not need should be stopped and, if never to be used again, deleted. Stopped jobs normally do not bill for running time, but check the current billing rules.

Job configuration as code

Query text, inputs, outputs and settings can all be exported as files and kept in version control. Treat them like application code, with review and automated deployment. A job created by hand in the portal is hard to reproduce after an accident.

A job that runs for months will see failures. The design assumes this and recovers automatically.

Checkpoints

The service regularly saves a snapshot of its state and the position it has read in each input partition. If the machine running the job fails or is moved for maintenance, a new one loads the latest checkpoint and continues from there. Events after the checkpoint are read again from the buffer, which is why the buffer must keep events long enough. After recovery the job produces the same results it would have produced without the failure, thanks to the way it replays.

Retention matters

If a job is down longer than the buffer’s retention period, events that expired are gone. Choose retention longer than your worst realistic outage and repair time, and alert when a job stops.

Planned maintenance

The platform updates the underlying machines from time to time. The job restarts from its checkpoint with a short pause. Results may be delayed briefly. Applications that consume outputs should tolerate small gaps and duplicates.

Availability across zones and regions

Some configurations place a job so that it survives the loss of a zone. For loss of a whole region, a common design runs a second job in another region reading from a replicated or duplicated stream. Both write to separate outputs, and consumers switch if the primary fails. This costs double, so reserve it for critical flows.

flowchart LR
  subgraph RA["Region A: primary"]
    EHA["Event hub"] --> JA["Job A"] --> OA["Output A"]
  end
  subgraph RB["Region B: secondary"]
    EHB["Event hub copy"] --> JB["Job B"] --> OB["Output B"]
  end
  EHA -. "geo-replication or dual send" .-> EHB
  OA --> APP["Consumers"]
  OB -. "used if A fails" .-> APP

Failure modes to expect

Streaming jobs often handle sensitive data, such as payments, locations and health readings. Protect each link in the chain.

Identity and access

Use a managed identity for the job rather than stored keys wherever the input or output supports it. The identity is given exactly the rights it needs: read on the input, write on the output, read on the reference data. Humans get roles that let them view or edit jobs; only a small team should be able to start, stop or change production jobs.

Secrets

If a connection string or password is unavoidable, keep it in Key Vault and refer to it from deployment code. Never put secrets in query text, in chat messages or in shared documents. Rotate secrets on a schedule and after any person with access leaves the team.

Network protection

Inputs and outputs have public addresses by default. For sensitive flows, place the services behind private endpoints and use the job’s dedicated cluster option or network integration so the job reaches them privately. Restrict public access on the buffer and outputs when private paths are in place.

Encryption

Data is encrypted in transit between the service and its inputs and outputs, and at rest where it is stored, such as in checkpoints and reference data. Some organizations bring their own keys for stronger control; consider this if rules require it.

Protecting the data inside events

Auditing

The activity log records who changed a job, its query or its settings. Send it to a central workspace, protect it from edits and alert on changes to production jobs. Because a changed query can silently change business numbers, query edits deserve review like any code change.

A job that runs all day needs eyes on it. A short list of metrics tells you almost everything.

Key metrics

Logs

Activity logs record management actions. Diagnostic logs record details of runtime and data errors, such as which event failed to parse. Send them to a central workspace for search and alerts. For events that failed, an option to route them to a separate store lets you inspect and repair them later rather than losing them.

Alerts worth having

Alert when the job state is not running, when watermark delay passes your limit, when utilization passes eighty percent for a sustained period, when errors appear, when input events fall to zero for longer than usual, and when output events fall to zero. Zero events is a classic silent failure: the job looks healthy but nothing flows.

A troubleshooting walk

Costs for streaming come from always-on compute and from the services around the job. Prices change, so use the official calculator for real figures.

A worked monthly example

The numbers below are invented to show the method. Suppose a job uses 12 streaming units and each costs 0.11 dollars per hour. The hourly cost is 12 times 0.11, or 1.32 dollars. A month has about 730 hours, so the job costs 1.32 times 730, or 963.60 dollars. Add an event hub at 150 dollars, a database output at 300 dollars and logs at 40 dollars, and the pipeline costs about 1,453.60 dollars per month. If utilization is always below 30 percent, halving units to 6 saves about 481.80 dollars per month. If a test job runs all month at 6 units, it adds about 481.80 dollars; stopping test jobs outside working hours cuts this by two thirds.

Ways to reduce cost

Right-size units by watching utilization over weeks. Combine several small queries into one job so they share inputs. Stop development and test jobs when not in use. Filter early so less data moves through the system. Reduce hop count in windows and reduce the number of keys in state. Use batching to lower output costs. Archive raw events cheaply and keep only what you need in expensive databases.

Where money is wasted

Typical waste includes jobs sized for peak that never comes, abandoned test jobs, duplicate jobs reading the same stream for similar results, large reference tables and long retention of detailed outputs in costly stores. A monthly review of utilization and tags finds these.

Azure offers several ways to process streams. The right one depends on the complexity of the logic and the skills of your team.

flowchart TD
  S["Need to process a stream"] --> A{"Logic mostly filter, window, join, aggregate?"}
  A -->|"Yes"| SA["Stream Analytics"]
  A -->|"No, custom or complex"| B{"Simple per-event reaction?"}
  B -->|"Yes"| FN["Functions"]
  B -->|"No"| C{"Heavy transformations or machine learning?"}
  C -->|"Yes"| SP["Spark-based or open stream engine"]
  C -->|"No"| D["Revisit requirements"]

When Stream Analytics shines

It shines when the question can be expressed as “filter these events, group by key and time, enrich with a list and send the result there.” Teams that know the database query style can ship a working pipeline in hours. Operation effort is low.

When to look elsewhere

Complex custom state machines, extremely large state, multi-step machine learning, advanced libraries and special protocols often fit better elsewhere. Also, if your whole data platform already runs on another engine and your team is expert in it, a single standard may beat a mix of tools.

Combining tools

Many designs use several: Stream Analytics for fast alerts and windowed metrics, a data lake for full history, and a heavier platform for model training. They share the same event hub, each reading in its own consumer group.

An honest look at what the service does well and where it falls short.

Key trade-offs

The central balance

Almost every streaming design balances three forces: how fast you want answers, how correct you want them, and how much you want to pay. Faster and more correct together cost more. Decide which of the three matters most for each use. A fraud alert may accept some false alarms for speed; a financial report must be correct even if delayed.

Teams that run streaming pipelines well repeat a handful of habits. Teams that struggle repeat a handful of mistakes.

Patterns that work

Anti-patterns to avoid

A checklist to apply before any job carries live traffic.

Best practices

Common mistakes

Walk through failures in advance, and recovery becomes routine.

Scenario one: results are suddenly delayed

Dashboards show numbers ten minutes behind. Likely causes: an output is slow, utilization is high, or a hot key overloads one lane. Recovery: check backlog and utilization, scale units, fix the slow output, and repartition the hot key. Prevention: alert on delay before users notice.

Scenario two: the totals are too low after a network outage

Field devices were offline for twenty minutes, then sent everything at once. The job’s late arrival tolerance was five minutes, so older events were adjusted or dropped. Recovery: replay from the buffer or archived raw events with a larger tolerance into a corrected table. Prevention: measure the longest realistic outage and either widen tolerance or run a corrective batch.

Scenario three: a producer changed its format

A team renamed a field and the job’s input errors soared. Events were dropped. Recovery: revert the producer or update the job, then replay the missed period from the buffer. Prevention: schema versioning, producer testing and a dead-letter path.

Scenario four: duplicates in the database

After a platform restart, some rows appear twice. The output was at-least-once and the table had no unique key. Recovery: remove duplicates by key and rebuild affected totals. Prevention: define a natural key per result and upsert, and test with failure injection.

Scenario five: memory runs out

Utilization hits one hundred percent and the job degrades. A new feature added a request id to a group key and kept long windows. Recovery: remove the unbounded key, shorten windows, add units. Prevention: review state size when changing queries.

Scenario six: the job stopped and nobody noticed

A credential expired on a Friday and the job failed. The buffer retention was one day, so events from the weekend were lost. Recovery: restore access and accept the gap or fill from other sources. Prevention: alerts on job state and on zero events, managed identities, and retention longer than a long weekend.

flowchart TD
  A["Results wrong or late"] --> B{"Job running?"}
  B -->|"No"| C["Check credentials, errors and restart"]
  B -->|"Yes"| D{"Backlog or delay growing?"}
  D -->|"Yes"| E["Check utilization, output speed and hot keys; scale or fix"]
  D -->|"No"| F{"Counts lower than expected?"}
  F -->|"Yes"| G["Check late events, input errors and consumer groups"]
  F -->|"No"| H["Check query logic, windows and reference data"]
  C --> I["Verify outputs and replay if needed"]
  E --> I
  G --> I
  H --> I

Good streaming systems are built, tested and released with the same care as any other software.

Develop locally and in the cloud

You can write and test queries in the portal against sample data, or in a developer tool on your own computer that runs the query on local files. Local testing is faster and lets you use version control naturally. Start with small sample files that include normal, late, malformed and edge-case events.

Test layers

Release pipeline

Upgrades with state

A changed query may or may not resume with saved state. Prefer designs where a restart from a recent time, with a short warm-up, gives correct results. For example, an hourly window can be rebuilt by starting one hour earlier. Plan these warm-ups in the release steps.

Environments

Keep separate development, test and production jobs with their own inputs and outputs. Use parameters for names, connection settings and sizes. Never point a test job at a production output.

These are typical, invented scenarios that show the ideas at work. They are not reports about any named company.

What they share

Each example starts from a clear question, uses the right window for the question, enriches events with small reference lists, writes results somewhere that people or systems act on, and keeps raw events for replay. The tool was simple; the thinking about time and groups was the real work.

Beyond counting and averaging, the service includes helpers that answer richer questions without leaving the query.

Spotting unusual values

Fixed thresholds are easy to write but often wrong. A temperature that is normal for a furnace is alarming for a freezer. Built-in anomaly detection functions learn what is normal in a recent stretch of the stream and flag values that depart from it, either as sudden spikes and dips or as slow changes in trend. The job looks back over a history window you define, scores each new reading, and outputs a flag and a score. You then decide what score deserves an alert.

Tuning anomaly checks

Geographic functions

Events often carry a latitude and longitude. The language can build points and areas and ask whether a point lies inside an area, or how far two points are apart. This allows rules like “alert when a vehicle leaves its permitted zone” or “count visitors currently inside a building’s boundary.” The areas can come from reference data, so planners update zones without editing the query.

Windowed pattern matching

Some questions are about the order of events: a login failure followed by a password reset followed by a successful login from a new place. Analytic functions let a query look back at the previous event, or the last event of a certain type for the same key, within a time limit. This turns a stream of separate facts into a story with a beginning and an end.

Custom functions

When built-in functions are not enough, you can add your own logic written in common programming languages or as a trained model. This is helpful for a special calculation or a scoring model, but it comes with costs. Custom code adds delay, needs testing of its own and can become the slowest part of the job. Keep such functions small, stateless and well tested, and prefer built-in features when they can do the job.

Machine learning scoring

A trained model, hosted as a service, can be called from the query to score each event, for instance to predict the chance of a fault. Calls over the network add delay, so batch events per call when possible, and plan for the model service being slow or unavailable. A fallback value for failed calls keeps the pipeline alive.

Short answers to common questions. Where a detail may have changed, confirm in the current Microsoft documentation.

Azure Stream Analytics is a managed service that reads endless streams of events, computes over time windows and sends results to where people and systems can act at once.