Log Analytics Workspace in Azure: The Complete Advanced Guide

Log Analytics Workspace in Azure: The Complete Advanced Guide

A book-quality walkthrough of the place where Azure keeps its logs: how data arrives, how tables and plans work, how to query, retain, secure, and pay for it, and how to design workspaces that stay useful for years.

Picture a huge city archive. Every day, thousands of clerks from every department drop off reports: traffic counts, building inspections, water usage, security visits. Some reports are urgent and must be searched in seconds. Some are rarely opened but must be kept for a decade because the law says so. A sensible archive has a reception desk that checks and sorts incoming paper, shelves arranged by type, a fast reading room for urgent work, a cheaper basement for old files, guards who decide who may open which cabinet, and a bill at the end of the month. A Log Analytics workspace is that archive for your cloud estate.

This guide teaches the workspace from the very beginning. Each new word is explained when it first appears, each big idea has an everyday comparison, and the later chapters go into the decisions that architects face: how many workspaces to create, which table plan to choose, how long to keep data, how to protect it, how to stop the bill from growing out of control, and how to migrate away from retired collection methods. No earlier knowledge of monitoring is needed.

1Logging Basics You Need First

Before learning the tool, understand the problem it solves.

What a log is

A log is a time-stamped record that a system writes to describe something that happened: a user signed in, a disk filled up, a web request failed, a virtual machine was restarted. Each entry usually carries a time, a source, a severity, and a message. One entry is not interesting. Millions of entries, searched together, tell you what your systems actually did.

Why logs matter

Logs answer questions that nothing else can. Why did the website slow down at three in the morning? Who deleted that storage account? Did the attack reach other servers? Metrics, which are simple numbers sampled over time, tell you that something is wrong. Logs tell you why. In regulated industries, logs are also legal evidence that controls worked.

Simple Analogy

A metric is the warning light on a car dashboard. A log is the mechanic’s detailed diagnostic printout. The light tells you to stop. The printout tells you which part failed and when.

The three pillars of observability

Pillar 1

Metrics

Lightweight numbers over time, such as processor use. Great for fast alerts and trends.

Pillar 2

Logs

Detailed event records. Great for investigation, auditing, and answering unexpected questions.

Pillar 3

Traces

The path of one request through many services. Great for finding where time is lost.

The central collection problem

In a cloud estate, logs are born in hundreds of places: virtual machines, databases, firewalls, containers, identity systems. If every one keeps its own file, nobody can search across them, files vanish when machines are deleted, and attackers can erase their own traces. The answer is to send everything to one central, protected, searchable store. In Azure, that store is the Log Analytics workspace.

i
Key Idea

Collecting logs is cheap to start and expensive to forget. The skill is deciding what to collect, how long to keep it, and how quickly you need to search it.

2What Is a Log Analytics Workspace?

A workspace is the container where Azure Monitor stores log data and lets you query it.

Definition

A Log Analytics workspace is an Azure resource that holds log data in tables and provides the engine to search that data. It belongs to one subscription and one region. It has its own settings for retention, access, pricing, and network access. Think of it as a database built specially for log and event data, hosted and managed by Microsoft, which scales without any server for you to run.

Where it sits in Azure Monitor

Azure Monitor is the family name for Microsoft’s monitoring services. It has two main data stores. The metrics store keeps numbers. The logs store, which is the Log Analytics workspace, keeps records. Features such as log alerts, workbooks, Application Insights, VM insights, container insights, and Microsoft Sentinel all read from workspaces.

flowchart TD
  SRC["Sources: VMs, Azure services, apps, containers, custom feeds"] --> COL["Collection: agents, diagnostic settings, ingestion API"]
  COL --> WS["Log Analytics workspace: tables and retention"]
  WS --> Q["Log Analytics queries"]
  WS --> AL["Log alerts"]
  WS --> WB["Workbooks and dashboards"]
  WS --> SEN["Microsoft Sentinel"]
  WS --> EXP["Data export to storage or event hub"]
        
FIG 1 — A workspace sits between collection and all the ways of using logs

What it gives you

One place to search everything

Logs from many services can be queried together, so one question can span a firewall, a server, and an identity system.

A powerful query language

The Kusto Query Language lets you filter, group, join, and chart millions of records within seconds.

Managed retention

You choose how long to keep data, from days to many years, without managing storage.

Built-in access control

Azure role-based access control decides who may read which data, down to individual tables.

Integration

Alerts, dashboards, security tools, and automation all work from the same data.

Simple Analogy

The workspace is a library with its own catalog and librarians. Books arrive from many publishers, are shelved by subject, and anyone with a card can ask the librarian a question. You do not build the library. You decide what goes on the shelves and who gets a card.

!
Terminology Note

Names have changed over time. Older articles say Operations Management Suite or OMS, and the older machine agent was called the Microsoft Monitoring Agent. Today the product is Azure Monitor Logs, and the workspace is still called a Log Analytics workspace.

Regions, limits, and what a workspace is not

Each workspace sits in a single Azure region, chosen when it is created, and the data stays in that region unless you export it. A workspace is not a backup system and not a long-term document store. It is a query engine over events. It also does not replace metrics for fast numeric alerts, nor traces for following one request across services. Many mature estates use all three together, with the workspace as the place for investigation, audit, and deep analysis. Knowing what the workspace is for prevents the common mistake of forcing it to do every monitoring job.

3What Is Changing in 2026

A good guide must describe the service as it is now, including the changes that affect real estates.

Three table plans

Each table in a workspace now carries a plan that trades price against capability. Analytics is the full-featured plan for data you monitor and alert on. Basic offers reduced ingestion cost for data used for troubleshooting. Auxiliary, which Microsoft’s current documentation labels Auxiliary or Lake, offers the lowest ingestion cost for verbose and compliance data that is rarely queried. All three live in the same workspace. The plans are covered in depth later.

Older collection methods are retired

The HTTP Data Collector API, long used to push custom data, reached the end of support on 14 September 2026 according to Microsoft’s migration guidance, and its endpoint has refused older TLS versions since 1 March 2026. Its replacement is the Logs ingestion API, which sends data through data collection rules. The older Log Analytics agent, known as the Microsoft Monitoring Agent, has been replaced by the Azure Monitor Agent. Estates still using the old paths should migrate now.

!
Lifecycle Warning

Because the support date for the HTTP Data Collector API is already past, any script or product still posting data that way is running on borrowed time. Find these senders first. Also note that the new API accepts smaller requests, up to one megabyte per call, so large senders need changes in how they batch.

Pricing and legacy tiers

Pay-as-you-go and commitment tiers are the modern pricing models. Commitment tiers start at one hundred gigabytes per day. Several legacy pricing tiers still exist only for old subscriptions and do not support the Basic and Auxiliary plans. A workspace on a legacy tier cannot use the cheaper plans, which is a strong reason to move it.

AreaDirectionWhat to do
Table plansThree plans per tableClassify tables by value and move noisy ones to cheaper plans
Custom ingestionData Collector API retired, Logs ingestion API replaces itInventory senders, create collection rules, and convert or recreate tables
AgentsAzure Monitor Agent replaces the older agentMigrate machines and rebuild collection as data collection rules
Legacy pricingOlder tiers block newer plansMove to pay-as-you-go or commitment tiers
Security analyticsSentinel builds on workspaces and a data lakePlan data tiers together with the security team
“The workspace you built three years ago was designed for a cheaper, simpler menu than the one on offer today.”

4Architecture and Data Flow

Understanding the path a log takes explains almost every setting you will meet.

The building blocks

PieceRole
Data sourceWhere the event is born, such as a virtual machine or an Azure service
CollectorThe mechanism that picks up the data: Azure Monitor Agent, diagnostic settings, the Logs ingestion API, or a built-in connector
Data collection ruleA reusable definition of what to collect, how to transform it, and where to send it
Data collection endpointA network entry point used by some senders and by private networking designs
WorkspaceThe storage and query service itself
TableA named set of records with a defined set of columns inside the workspace
ConsumersQueries, alerts, workbooks, Sentinel, export rules

Two ways data gets in

Some data is pulled by an agent running on a machine, which reads event files and counters and sends them on. Other data is pushed by Azure itself: when you enable a diagnostic setting on a resource, the platform streams its logs to the workspace with no agent. Custom applications and tools use the Logs ingestion API. All paths end at a table.

Simple Analogy

Agents are couriers who visit each office and collect parcels. Diagnostic settings are pneumatic tubes built into the walls of certain offices that send parcels automatically. The ingestion API is the loading dock where outside companies deliver on their own. The data collection rule is the instruction sheet that tells the mailroom which parcels to open, trim, and shelve.

flowchart LR
  VM["Virtual machine"] --> AMA["Azure Monitor Agent"]
  AMA --> DCR["Data collection rule: filter and transform"]
  PAAS["Azure resource"] --> DS["Diagnostic setting"]
  APP["Custom application"] --> API["Logs ingestion API"]
  API --> DCR
  DS --> TBL
  DCR --> TBL["Workspace tables"]
  TBL --> USE["Queries, alerts, workbooks, Sentinel"]
        
FIG 2 — Three entry paths, one set of tables
i
Practical Tip

Draw this diagram for your own estate before changing anything. Knowing which sources use which path makes every later decision about cost, security, and migration much easier.

5Tables, Columns, and Schemas

Everything in a workspace lives in a table, and table choices drive cost and capability.

What a table is

A table is a collection of records that share the same set of columns. Each record is one event. Column types include text, numbers, true or false, date and time, dynamic data for flexible structures, and identifiers that are stored and queried as text. Table names are case sensitive, which surprises many beginners.

Kinds of tables

Platform

Azure tables

Created and owned by Azure services, such as activity records, heartbeat signals, security events, and performance counters. You cannot change their core schema.

Yours

Custom tables

Created by you for application or third-party data. They are fed through the Logs ingestion API or collection rules.

Derived

Search results tables

Created by search jobs to hold the results of a large asynchronous query.

Derived

Restored tables

Temporary views that bring old data back into fast storage for analysis.

Derived

Summary tables

Created by summary rules to hold pre-aggregated results as data arrives.

Standard columns

Every record also carries standard columns that Azure adds, such as the time it was generated, a billing flag, and the billed size of the record. The billing columns are useful, since they let you ask the workspace itself which tables and sources cost the most. Some tables, including activity records, heartbeat, usage, and operation tables, are free of ingestion charge, though you should confirm that with the billing flag instead of assuming.

Simple Analogy

A table is a filing cabinet drawer with printed index tabs. Azure tables are drawers delivered ready-made by the city. Custom tables are drawers you build for your own forms. Every card in a drawer has the same boxes to fill in, plus a small stamp showing when it arrived and what filing it cost.

Schema changes

You can add columns to custom tables. Changing a schema carelessly can break senders that still use older rules. For tables fed by the retired Data Collector API, adding columns through the portal or interface can break legacy ingestion, so plan the migration first. The new API does not adjust schemas automatically, so a source that adds a field must be matched by a table change, a transformation that drops the field, or a decision to leave that field uningested.

!
Cost Trap

Values that do not fit the destination schema are still billed even though the table cannot store them. Keep collection rules aligned with table columns, or you pay for data you cannot read.

6Getting Data In: Sources and Collection Methods

There are four main ways to feed a workspace, and each fits a different kind of source.

MethodUsed forNotes
Azure Monitor AgentWindows and Linux machines in Azure, other clouds, and on premisesControlled by data collection rules. Replaces the older agents.
Diagnostic settingsAzure resources such as databases, firewalls, and key vaultsNo agent. Choose log categories and a destination.
Logs ingestion APICustom applications and third-party toolsReplaces the HTTP Data Collector API. Needs a collection rule and identity-based authentication.
Built-in connectorsMicrosoft services and security products, especially with SentinelConfigured by connector pages. Cloud-hosted connector frameworks are replacing older approaches.

Azure Monitor Agent

The Azure Monitor Agent is installed on a machine and, instead of keeping its own configuration, receives instructions from one or more data collection rules associated with the machine. The same machine can follow several rules, one for security events, one for performance counters, and one for text logs. Because rules are separate Azure resources, you can apply one rule to a thousand machines through policy.

Diagnostic settings

Most Azure services produce platform logs and metrics. A diagnostic setting tells the service where to send them: a workspace, a storage account for cheap archival, or an event hub for streaming to other tools. Different settings can send different categories to different places. Many categories can be extremely noisy, so choose with care.

Logs ingestion API

For your own applications, the Logs ingestion API takes batches of records and sends them through a collection rule into a named table. It authenticates with an Entra identity instead of a shared key, which removes the old problem of a workspace key that can write anything. Each call is limited to about one megabyte, so large senders send several calls in parallel.

Simple Analogy

The old Data Collector API was a mailbox with one shared key that anyone could use to post anything. The new design is a front desk that checks each visitor’s badge and consults the instruction sheet before accepting a delivery.

Choose by source

Machines use the agent, Azure services use diagnostic settings, your own software uses the ingestion API, and Microsoft security products use connectors. Mixing them in one workspace is normal.

Avoid duplicates

Sending the same data twice, for example through both an agent and a diagnostic setting, doubles the cost and confuses analysis.

i
Practical Tip

Use Azure Policy to enforce diagnostic settings and rule associations automatically. Manual setup always leaves gaps, and gaps are noticed only when an incident needs the missing log.

Planning a rollout wave by wave

Resist the urge to switch on collection everywhere in one afternoon. Start with the resources whose logs your security and operations teams already use, confirm that the data arrives and is useful, then expand to the next tier of importance. For each wave, record expected daily volume before and after, so you can compare plan against reality. Early waves teach you which categories are noisy, which collection rules need filtering, and which teams need training. Later waves then go quickly because the standard, the templates, and the policy are already proven.

7Data Collection Rules and Transformations

Data collection rules are the control panel for what enters the workspace, and therefore for what you pay.

Anatomy of a rule

1

Data sources

What to collect: which event logs, counters, or files, and which severity levels.

2

Stream declarations

For custom data, the shape of incoming records before processing.

3

Transformation

An optional query that filters rows, removes columns, or reshapes values before storage.

4

Destinations

Which workspace and table receive the result.

Why transformations matter

A transformation runs on the way in, before data is stored and billed. This is the most powerful cost control in the service, because dropped data costs nothing. It can remove noisy events, strip large unneeded columns, mask sensitive values such as account numbers, split one stream into several tables, or convert formats. It can also route low-value data to a table with a cheaper plan.

Simple Analogy

A transformation is the mailroom clerk who opens each envelope, throws away the advertising, blacks out private numbers with a marker, and sends the useful pages to the right department. Paying only for what reaches the shelf is far cheaper than shelving everything and sorting later.

Rule design habits

One rule per purpose

Separate rules for security, performance, and application logs are easier to understand and reassign than one giant rule.

Version through templates

Keep rules in infrastructure templates under source control so that changes are reviewed and repeatable.

Test before wide rollout

Apply a new rule to a few machines, compare volume and content, and then expand.

!
Common Mistake

Filtering aggressively without asking the security team. A log you drop today may be the one an investigator needs next year. Agree on a protected list of events that are never filtered.

8Internal Working: The Life of a Log Record

Follow one record from creation to query to see where delay, cost, and failure come from.

Step one: creation and pickup

An event happens and the source writes it. The agent or the platform collects it, usually in small batches, and sends it over an encrypted connection. Collection is not instant. Batching and buffering add seconds or minutes.

Step two: the rule and transformation

The service receives the batch, matches it to the collection rule, runs the transformation, and checks that the result fits the destination table. Values that do not match the schema are dropped but may still be billed.

Step three: ingestion and storage

The records are written to the table. At this point they are billed by size. They are available in interactive retention, where queries are fast. A short ingestion latency, typically from seconds to a few minutes, separates event creation from visibility in queries.

Step four: use and aging

Queries, alerts, and workbooks read the data. After the interactive retention period, data moves to long-term retention at a reduced price, where it can still be reached by search jobs or restored for deeper analysis. When total retention ends, data is removed.

sequenceDiagram
  participant S as Source
  participant C as Agent or platform
  participant R as Collection rule
  participant W as Workspace table
  participant U as Query, alert, workbook
  S->>C: Event occurs
  C->>R: Send batch over encrypted channel
  R->>R: Filter and transform
  R->>W: Write records, bill by size
  W-->>U: Available for fast queries
  U->>W: Query by time range
  W->>W: After interactive retention move to long term
  U->>W: Search job or restore for old data
        
FIG 3 — The life of a log record
Collect
Batching delay at the source
Ingest
Rule, transformation, write, billing
Use
Queries, alerts, retention aging
Simple Analogy

A log record travels like a letter. It is written, picked up from a mailbox in bundles, sorted at the post office, delivered to a shelf, read by whoever asks, and eventually moved to a distant warehouse where retrieval takes longer.

i
Practical Tip

When an alert fires late or a log seems missing, compare the time the event was generated with the time it was ingested. The gap tells you whether the problem is collection, ingestion, or query timing.

9Querying with Kusto Query Language

The query language is how you turn a mountain of records into answers. Its core ideas are simple.

The pipeline idea

A query starts with a table name and then passes the rows through a series of steps, each one refining the result. One step filters rows, another picks columns, another groups and counts, another sorts, another limits the output, and another draws a chart. Steps are chained one after another, like stations on an assembly line, and each station receives the output of the previous one.

Simple Analogy

A query is a coffee filter stack. You pour in all the grounds, which is the table. The first filter removes anything outside the last hour. The next keeps only error events. The next groups them by server. What drips out is the short list you wanted.

Core operations in plain words

Filter

Where

Keep only rows meeting a condition, such as a time window or a severity.

Shape

Project and extend

Pick the columns you need or add new calculated columns.

Group

Summarize

Count, average, or total values grouped by columns, including time buckets.

Order

Sort and top

Rank results and return only the largest or smallest few.

Combine

Join and union

Link records from two tables or stack their rows together.

Show

Render

Draw results as a line, bar, or pie chart.

Habits that make queries fast

1

Always limit the time range

The time range is the strongest control over how much data is scanned. Use the narrowest range that answers the question.

2

Filter early

Put row filters first so later steps handle less data.

3

Select only needed columns

Dropping unused columns before heavy steps reduces memory use.

4

Prefer exact matches over wide text searches

Searching every column of every table is slow and costly. Name the table and column.

i
Practical Tip

Save good queries as functions, so teams reuse trusted logic. A shared function that parses a messy table once is better than twenty slightly different copies.

Learning the language step by step

A practical way to learn is to begin with a single table and ask one question at a time. First, show a few records to see the columns. Second, keep only the last hour. Third, keep only the rows of interest. Fourth, count them by a useful column. Fifth, draw the count over time. Each step builds on the one before, and mistakes are easy to spot because the result changes a little at a time. Within a few days, most people can answer their own everyday questions, and within weeks they are writing reusable functions that other teams adopt.

10The Query Experience: Modes, Scope, and Search Jobs

The portal offers several ways to ask questions, suited to different people and data ages.

Simple mode and KQL mode

The query interface has two modes. Simple mode lets someone with no query language knowledge pick tables, filter and sort with clicks, and explore results in a spreadsheet-like grid. KQL mode is for writing and editing queries that can be saved, shared, turned into alerts, or placed in workbooks. Beginners start in simple mode and graduate as needs grow.

Query scope

ScopeWhat you seeTypical user
Workspace scopeAll tables the user may read in that workspaceCentral operations and security teams
Resource scopeOnly logs belonging to one resource, such as one virtual machineApplication owners
Resource group or subscription scopeLogs of the resources within it, across workspacesTeam leads with delegated access

Basic and Auxiliary tables can only be queried at workspace scope, while Analytics tables support resource scope as well. This matters for application teams that rely on resource-centric access.

Search jobs and restore

Interactive queries are designed for quick answers on recent data. For very large or old data, two other tools exist. A search job runs asynchronously, scans a large range, and copies matching records into a new search results table, which you can then query quickly. A restore brings a chosen time range of a table back into fast storage temporarily, so heavy analysis can run as if the data were recent. Search jobs work on all three table plans. Restore is available on Analytics and Basic only.

Simple Analogy

An interactive query is looking through the files on your desk. A search job is asking the archive clerk to hunt through the basement and bring you a folder of everything matching your request. A restore is asking for a whole box to be carried upstairs and kept on your desk for a few days.

!
Cost Note

Restores are billed by the data volume and time active, with minimums, and Basic and Auxiliary queries are billed by data scanned. Test on a narrow time range before launching broad exploration on cheaper plans.

Choosing the right tool for the question

A quick matching guide helps beginners. For recent events, ask an interactive query. For a repeating chart, build a workbook on a summary table. For one needle in last year’s haystack, run a search job. For a week of deep investigation across many questions about an old incident, restore the period and work on it. For an application owner who only needs their own system’s logs, use resource scope. Picking the right tool avoids both frustration and unexpected charges, because each tool has a different speed and a different price.

11Table Plans in Depth: Analytics, Basic, and Auxiliary

Choosing the plan for each table is the biggest single cost decision after choosing what to collect.

FeatureAnalyticsBasicAuxiliary or Lake
Intended useHigh-value data for monitoring, detection, and performance analysisTroubleshooting and incident response dataVerbose logs, audit, and compliance data
Ingestion priceStandardReducedLowest
Query speedOptimizedOptimizedSlower, not for real-time analysis
Query chargeIncludedPer gigabyte scannedPer gigabyte scanned
Query scopeResource and workspaceWorkspace onlyWorkspace only
AlertsFullSimple log alertsNot supported
Commitment tiersEligibleNot eligibleNot eligible
RestoreSupportedSupportedNot supported
Data exportSupportedSupportedNot supported
Interactive retentionThirty days by default, up to two yearsNot applicableNot applicable
Total retentionUp to twelve yearsUp to twelve yearsUp to twelve years

How to read the table

Analytics is the plan you reach for when someone will watch the data or write alerts on it. The ingestion price is highest, but queries cost nothing extra and the first thirty-one days of retention are included. Basic gives most of the query power for a lower ingestion price, but you pay per gigabyte scanned and give up some alerting and integrations. Auxiliary is the cheapest way to land large volumes and keep them for years, at the cost of slower queries and fewer features.

Simple Analogy

Analytics is the front shelf in a shop, where the best-selling items stay within arm’s reach and are restocked constantly. Basic is the stockroom just behind the counter, cheaper to maintain, and you fetch things when asked. Auxiliary is the distant warehouse where bulk goods sit at very low rent, and fetching anything takes a trip.

What qualifies for which plan

Not every table can use every plan. Basic and Auxiliary apply to Azure tables that Microsoft has enabled for them, and to custom tables created through data collection rules. Workspaces on legacy pricing tiers cannot use Basic or Auxiliary at all. Check the supported table list before planning savings.

A simple classification method

1

Ask who reads it

If an alert or a dashboard depends on the table, it belongs in Analytics.

2

Ask how often

Tables opened only during incidents are candidates for Basic.

3

Ask how fast

Data needed only for audit within days or weeks of a request can tolerate Auxiliary.

4

Review quarterly

Usage changes. A table nobody queried this quarter may deserve a cheaper plan.

!
Common Mistake

Moving a table to Basic to save money and then discovering that three alerts and a dashboard depended on it. Check dependencies first, since changing back is possible but disruptive.

12Retention: Interactive and Long-Term

Keeping data longer is easy. Paying sensibly for it is the art.

Two stages of life

Data in an Analytics table first lives in interactive retention, where queries are fast and complete. The default is thirty days, with ninety days for workspaces used with Microsoft Sentinel and Application Insights data. You can extend interactive retention up to two years for a prorated fee beyond the included period. After that, data enters long-term retention, which is cheaper, and stays until the total retention period ends, up to twelve years. Basic and Auxiliary tables have no interactive stage and rely on total retention.

flowchart LR
  I["Ingestion"] --> H["Interactive retention: fast queries, default 30 days"]
  H --> L["Long term retention: cheaper, search jobs and restore"]
  L --> D["Deleted when total retention ends"]
  H -. restore .-> H2["Restored table for deep analysis"]
  L -. search job .-> S["Search results table"]
        
FIG 4 — Retention stages for an Analytics table

Setting retention

Retention is set at the workspace level as a default and can be overridden per table. A security table may be kept for seven years while a verbose performance table is kept for forty-five days. Retention is billed daily. Reducing retention lowers future cost, but remember that purging data does not reduce retention charges for data that remains, and deleting a custom table does not delete its data, so retention charges can continue until the data ages out.

Simple Analogy

Interactive retention is the working desk. Long-term retention is the filing room across the corridor. Both hold your papers, but the desk is expensive floor space, so only current work sits there, and the filing room is fine for things you must keep but rarely read.

Deciding how long

Operational troubleshooting

Thirty to ninety days usually covers incidents and trends.

Security investigation

Attackers can be present for months before detection, so many organizations keep security logs for at least a year in a searchable form.

Regulation and audit

Rules may demand one to seven years or more. Use long-term retention or the Auxiliary plan rather than interactive retention.

Immutable archives

If a rule demands data that cannot be changed, export to storage with immutability policies in addition to workspace retention.

i
Practical Tip

Write a retention policy document listing each major table, its owner, its reason for retention, and its period. When an auditor or finance asks why, you have an answer.

A worked retention example

Imagine three tables. A sign-in table feeds security alerts, so it stays in interactive retention for ninety days and in long-term retention until it reaches two years. A web server request table is used for troubleshooting only, so it sits on the Basic plan and is kept for forty-five days. A platform audit table exists purely to satisfy a regulator who may ask questions up to six years later, so it sits on the Auxiliary plan with total retention of six years. Each table has an owner, a reason, and a cost that matches its value. This is the target state: retention that reflects need, not habit.

13Summary Rules, Search Jobs, Restore, and Export

Four tools help you get value from data without scanning everything every time.

Aggregate

Summary rules

Run a query on a schedule as data arrives and write aggregated results to a new table. Dashboards and alerts read the small summary instead of billions of rows.

Find

Search jobs

Asynchronous queries across large or old data that copy matching records into a results table.

Rehydrate

Restore

Temporarily bring a time range of a table back to fast storage. Billed by volume and time, with minimums.

Send out

Data export

Continuously copy chosen tables to storage or an event hub, billed by exported volume.

Summary rules as a cost lever

Summary rules deserve special attention. Suppose a firewall produces billions of connection records, but the dashboard needs only counts per hour per rule. A summary rule writes those counts to a small table. The big table can then sit on a cheaper plan or shorter retention, while the dashboard stays fast and cheap. On Basic and Auxiliary tables, the summary query is limited to a single table.

Simple Analogy

A summary rule is the accountant who reads every receipt as it arrives and writes a one-line daily total in a ledger. Managers read the ledger. The receipts go to storage and are fetched only for audits.

NeedBest tool
Fast dashboards over huge dataSummary rules
Find a needle in old dataSearch job
Run many varied queries on an old periodRestore
Keep a copy outside the workspace for immutability or other toolsData export
!
Common Mistake

Leaving a restore active for weeks after the investigation ended. Restores are billed while active, so dismiss them when done and set a reminder.

14Workspace Design: One or Many?

Microsoft’s guidance is simple: begin with one workspace and add more only for a specific reason.

Reasons to create more than one

ReasonWhy it matters
Microsoft Entra tenantsMany sources send only to a workspace in the same tenant, so you generally need one per tenant
Data residencyEach workspace lives in one region. Rules requiring data to stay in a country need a workspace in that region
Data ownershipSubsidiaries or affiliated companies may need separate stores
Split billingSeparate subscriptions can send separate invoices, though cost reports can often answer the need without splitting
Different retention in the same tableRetention is set per table per workspace, so resources needing different retention for the same table type need separate workspaces
Operations versus securitySeparate teams may want separate ownership, while a combined workspace gives broader visibility and reaches a commitment tier sooner
Legacy agentsThe older Linux agent connects to only one workspace
flowchart TD
  START["Start: one workspace"] --> Q1{"Multiple Entra tenants?"}
  Q1 -- Yes --> A1["One workspace per tenant"]
  Q1 -- No --> Q2{"Data residency rules by region?"}
  Q2 -- Yes --> A2["One workspace per required region"]
  Q2 -- No --> Q3{"Need different retention for same table or strict ownership split?"}
  Q3 -- Yes --> A3["Add workspaces for those groups"]
  Q3 -- No --> A4["Keep a single workspace"]
  A1 --> CL{"Combined 100 GB per day or more?"}
  A2 --> CL
  A3 --> CL
  A4 --> CL
  CL -- Yes --> DC["Consider dedicated cluster"]
  CL -- No --> OK["Stay on standard pricing"]
        
FIG 5 — A decision path for workspace count

The money side of the decision

Combining data can unlock a commitment tier. In Microsoft’s own example, fifty gigabytes per day of operational data and fifty of security data separately miss the hundred gigabyte tier, but together they reach it, earning a discount on monitoring and a larger discount on Sentinel. Splitting by region can reduce bandwidth charges, yet those are often small compared with ingestion, and splitting may lose tier discounts. Duplicate data in several workspaces always adds cost.

Cross-workspace queries

Queries can span workspaces, which softens the downside of having several. Very wide queries over dozens of workspaces become slow, and there is an upper limit on how many workspaces one query can reach. Placing workspaces on a shared dedicated cluster reduces that problem.

Simple Analogy

One workspace is one big library with a single catalog. Several workspaces are branch libraries. Branches make sense when they sit in different countries or serve different owners. Otherwise one library with good shelving serves everyone better and is cheaper to run.

ANTI-PATTERN-01 Avoid
Problem

Every application team creates its own workspace with its own settings, giving hundreds of tiny workspaces.

Why It’s Harmful

No commitment discounts, inconsistent retention and security, hard cross-team investigations, and a large administration burden.

Correct Approach

Use a small number of shared workspaces and give teams access through resource-context permissions and table-level controls.

15Access Control and Permissions

Logs often contain personal data, security events, and secrets. Who can read them is a serious question.

Two access modes

Workspace-context

A user is granted access to the workspace and sees every table they are allowed to read in it. This fits central operations and security staff.

Resource-context

A user with read access to an Azure resource automatically sees that resource’s logs in the workspace, and nothing else. This is the default and suits application owners.

The workspace setting can use resource or workspace permissions, which allows both models together, or require workspace permissions, which turns resource-context access off. Note that resource-context access depends on the resource being identifiable in the records, so some custom data without resource identifiers will not follow the rule.

Table-level access

Azure role-based access control can grant or deny access to individual tables. An audit team might receive only security tables, while an application team may be denied the security tables. Combine table-level rules with resource-context to build a layered model that follows least privilege.

PersonaSuggested access
Central monitoring teamWorkspace reader for operational tables, contributor rights for rules and alerts
Security analystsReader on security tables, plus Sentinel roles where used
Application ownerResource-context read on their own resources
AuditorReader on audit tables only, with time-limited assignment
Automation identityNarrow role scoped to the data it must write or read
Simple Analogy

Workspace access is a master key to the whole archive building. Resource-context access is a key that opens only the drawer holding your own department’s papers. Table-level rules are padlocks on the most sensitive cabinets, even for people who hold the master key.

!
Common Mistake

Giving the broad contributor role at workspace level to many people so they can run queries. Contributors can also change settings, retention, and tables. Readers suffice for querying.

Designing roles around real jobs

Built-in roles cover most needs, such as reader for querying and contributor for management, but some organizations create custom roles that allow only running queries without reading certain tables, or only managing alert rules. Pair roles with groups from the identity directory so that joining or leaving a team updates access automatically. Review assignments every quarter, remove stale ones, and use time-limited elevation for rare administrative work. Access to logs is access to knowledge about your whole estate, so it deserves the same discipline as access to production systems.

16Security, Encryption, and Dedicated Clusters

The workspace stores the evidence of everything else. Protecting it protects the whole investigation trail.

Encryption and protection

Data is encrypted in transit and at rest by default using keys managed by Microsoft. Organizations with stricter rules can use customer-managed keys, which require a dedicated cluster. With a customer-managed key, you hold the key in your own key vault and can revoke access, which makes the data unreadable. Customer Lockbox, available on Analytics and Basic tables, gives you approval control over Microsoft support access to your data in rare support cases.

Dedicated clusters

A dedicated cluster is a managed environment that hosts one or more workspaces and unlocks capabilities such as customer-managed keys and pooled commitment pricing. The minimum commitment is one hundred gigabytes per day, with no pay-as-you-go option. The commitment can be raised at any time but cannot be reduced during its thirty-one day period. Billing starts when the cluster is created, even if no workspace is linked, so do not create one early.

Benefit

Customer-managed keys

Control over encryption keys and the power to cut access.

Benefit

Pooled commitment

Combine the volume of several workspaces to reach a tier discount.

Benefit

Better cross-workspace queries

Workspaces on the same cluster query each other more efficiently.

Cost

Always-on commitment

Billed from creation, with a minimum of one hundred gigabytes a day.

Simple Analogy

A dedicated cluster is a private vault wing inside the bank. You pay a fixed monthly rent whether the wing is full or half empty, but you hold your own keys and you can bring several of your own safes into it.

Protecting the data itself

Mask sensitive values at ingestion

Transformations can remove or hash account numbers, passwords captured by mistake, and personal details before storage.

Purge with care

A purge operation can remove specific records for privacy requests, but it is a controlled and slow process and does not reduce retention charges.

Audit the workspace

Record who queries what and who changes settings, and send those records to a protected location.

i
Practical Tip

Treat the workspace like a production database that contains secrets, because it often does. Restrict write access, monitor changes, and keep a secured copy of critical logs elsewhere.

Thinking like an attacker

Attackers often try to blind defenders before acting, for example by disabling agents, deleting diagnostic settings, or lowering retention. Protect against this by restricting who can change those settings, alerting on any change to collection or retention, and storing a second copy of critical logs in a location the same administrators cannot alter. Watching the watchers is a core security habit. Treat a sudden drop in logs from a sensitive system as a possible incident until proven otherwise, not merely as a technical glitch.

17Networking and Private Connectivity

By default a workspace is reachable over the public internet with strong authentication. Many organizations want it reachable only from their private networks.

Private link scopes

Azure Monitor Private Link Scope, usually shortened to AMPLS, lets you connect private endpoints in your virtual network to Azure Monitor resources, including workspaces. Traffic stays on the Microsoft backbone and public access can be disabled for ingestion, for queries, or for both. A single scope can hold several workspaces and monitoring resources, and the endpoint resolves private addresses for the Azure Monitor domain names.

Design points

SettingMeaning
Ingestion access modeWhether data may be sent only through private links or also over public routes
Query access modeWhether queries may come only through private links or also from the public internet
Scope sharingOne scope shared by many networks can expose more than intended, so plan boundaries carefully
Name resolutionPrivate DNS zones must be configured so that monitoring domain names point to private addresses
Simple Analogy

Public access is a post office counter open to the street. Private link is a secure internal mail chute from your building straight to the post office back room. Once the chute exists, you can lock the street counter.

flowchart LR
  VNET["Virtual network with private endpoint"] --> AMPLS["Azure Monitor Private Link Scope"]
  ONP["On premises via VPN or ExpressRoute"] --> VNET
  AMPLS --> WS1["Workspace A"]
  AMPLS --> WS2["Workspace B"]
  INET["Public internet"] -. blocked when public access disabled .-> WS1
        
FIG 6 — Private connectivity to workspaces
!
Common Mistake

Turning off public query access before checking which tools query the workspace. Portal users on corporate laptops, automation runners, and external dashboards may all break at once.

18Alerts, Workbooks, and Dashboards

Collected data pays back when it triggers action or informs people.

Log alerts

A log search alert runs a query on a schedule and fires when results meet a condition, for example when the count of failed sign-ins crosses a limit. Alerts then trigger action groups that notify people, open tickets, or run automation. Each alert query runs against the workspace and is itself a cost and a load, so design them with narrow time windows and efficient filters. Alerts depend on Analytics tables, with only simple alerts on Basic tables.

Workbooks and dashboards

Workbooks

Rich interactive reports mixing text, parameters, and charts from many data sources. They are the main reporting tool for deep views.

Azure dashboards and Grafana

Pin charts for at-a-glance views. Basic and Auxiliary tables work with workbooks and Grafana but not with the Azure Monitor dashboards.

Power BI

Export query results to business reporting tools for executives.

Simple Analogy

A log alert is a night watchman who walks the same round every few minutes and rings a bell when he sees a particular thing. A workbook is the printed security report you read over coffee. A dashboard is the wall of monitors in the control room.

Good alert hygiene

1

Alert on symptoms users feel

Failed requests and slow responses matter more than a single processor spike.

2

Give every alert an owner and an action

An alert without a response is noise, and noise teaches people to ignore alerts.

3

Use summary tables for heavy alerts

Query a small aggregate, not billions of rows, every few minutes.

4

Review monthly

Retire alerts that never fire or always fire.

Connecting alerts to action

An alert is only as useful as the response behind it. Action groups can email, send mobile notifications, call webhooks, create tickets, or start automation that restarts a service or isolates a machine. For serious conditions, route to an on-call rotation. For informational ones, send a daily digest instead of a flood of messages. Include a link to the exact query and a short note about the first thing to check, so the person who wakes at night does not have to rediscover the system. Test the whole chain regularly with a deliberate, harmless trigger.

19Integrations: Sentinel, Application Insights, Defender, and More

The workspace is the common foundation under several Microsoft products.

Security

Microsoft Sentinel

A cloud security information and event management product built on workspaces. When enabled, Sentinel pricing applies to all data in the workspace, and a longer included retention applies.

Apps

Application Insights

Workspace-based resources store application telemetry in tables of the chosen workspace, sharing its pricing and retention.

Cloud

Defender for Cloud

Sends security events and recommendations, and gives a daily free data allowance per protected server for certain security tables.

Machines

VM insights and container insights

Use the agent and collection rules to fill performance and inventory tables for virtual machines and clusters.

Automation

Logic apps, automation, and playbooks

React to alerts and query results with workflows.

Outbound

Export and APIs

Query interfaces and export rules feed storage, event hubs, and other analysis tools.

Simple Analogy

The workspace is the main electrical panel of a building. Sentinel, Application Insights, and Defender are separate appliances that plug into the same panel. Add too many appliances without checking the panel’s capacity and the bill rises quickly.

The Sentinel effect on design

Enabling Sentinel on a workspace changes the billing for all its data, including operational logs that have nothing to do with security. This is why many organizations keep a dedicated Sentinel workspace for security data, and others combine to reach a larger commitment tier. The right answer comes from modeling the volumes and discounts for your own estate.

i
Practical Tip

Involve the security team before restructuring a workspace. Security products have their own content, such as rules and workbooks, that depend on table names and schemas.

Sharing one foundation across tools

Because many products read the same tables, a change in one place can surprise another. Renaming a column in a custom table can break a security rule. Shortening retention can delete data an investigation needed. Adding a very noisy source can push a shared workspace into a different cost band for everyone. Maintain a simple list of which teams and products depend on each major table, and require a short notice before changes. A modest amount of coordination prevents most of the cross-team incidents that otherwise appear in shared platforms.

20Pricing Models and Commitment Tiers

Costs follow volume, so understanding the meters is essential. Always confirm exact prices on Microsoft’s current pricing page, since they vary by region and change over time.

What you are billed for

Ingestion

The volume of data written, measured in gigabytes of billed size, which is typically a quarter smaller than raw incoming JSON and sometimes half for tiny events.

Retention

The first thirty-one days on Analytics tables are included. Longer retention is billed daily.

Queries on cheaper plans

Basic and Auxiliary queries are billed by data scanned.

Search jobs, restore, and export

Billed by scanned, restored, or exported volume.

Pay-as-you-go versus commitment tiers

ModelHow it worksBest for
Pay-as-you-goPay a flat rate per gigabyte ingested, no commitmentSmall or unpredictable workloads
Commitment tierCommit to a daily volume starting at one hundred gigabytes. Pay a lower rate per gigabyte, with savings that Microsoft documents as up to thirty percent. Overage is billed at the tier’s rateSteady, high-volume Analytics ingestion
Dedicated clusterCommitment tier shared by linked workspaces, minimum one hundred gigabytes a day, no pay-as-you-goLarge estates or those needing customer-managed keys

A commitment tier lasts thirty-one days from selection. You can move up during the period, which restarts the clock, but you cannot move down until it ends, except for a short correction window of about six hours after changing. Commitment tiers apply only to Analytics tables, so moving high-volume data to cheaper plans reduces how much volume counts toward the commitment. Enterprise Agreement and other Azure discounts apply to commitment pricing.

Simple Analogy

Pay-as-you-go is buying electricity at the standard rate. A commitment tier is signing a monthly contract for a fixed daily amount at a discount: wonderful if you really use that much, wasteful if you use less, and any extra beyond the contract is still billed at the same discounted per-unit rate.

!
Common Mistake

Choosing a commitment tier based on a peak week, or before applying cost-saving filters. After optimization, daily volume falls and the unused commitment is paid anyway. Optimize first, then commit based on a stable average.

A worked cost illustration

Suppose a company ingests one hundred and twenty gigabytes a day into Analytics tables, of which forty gigabytes come from a single verbose firewall table. If the firewall data moves to a cheaper plan after summary tables take over its dashboards, Analytics volume falls to eighty gigabytes. The company then no longer reaches the hundred gigabyte tier on that table set alone, so it must decide whether combining with another workspace restores the discount, or whether pay-as-you-go on the smaller volume is now cheaper than committing. The lesson is that every optimization changes the pricing decision, so recalculate after each major change instead of treating the tier as permanent.

21Cost Optimization Playbook

Savings come from a handful of repeatable steps, in a sensible order.

1

Find where the volume is

Use the usage records, the billing columns, and the workspace usage page to rank tables and sources by billed volume. A few tables usually cause most of the cost.

2

Stop collecting what no one uses

Disable unneeded diagnostic categories and remove noisy event levels from collection rules.

3

Transform on the way in

Filter rows and drop large unused columns with collection rule transformations.

4

Move tables to the right plan

Shift troubleshooting data to Basic and verbose or compliance data to Auxiliary, after checking dependencies.

5

Right-size retention

Shorten interactive retention and rely on long-term retention for the rest.

6

Choose the pricing model last

When volumes stabilize, compare pay-as-you-go with commitment tiers and consider combining workspaces or a dedicated cluster.

Simple Analogy

Cutting a household electricity bill works the same way. First find which appliance uses the most. Switch off what nobody uses. Fit efficient bulbs, which are the transformations. Move the freezer to the cheaper night tariff, which is the cheaper plan. Only then pick the best supplier contract for the new, lower usage.

Typical heavy hitters

SourceWhy it is largeRemedy
Windows security eventsVery verbose audit categories collected at the highest levelCollect a defined common level and filter specific noisy event identifiers with security approval
Firewall and network flow logsOne record per connectionSummary rules, cheaper plan, or storage-only export
Container and Kubernetes logsStandard output of every podFilter namespaces and move to Basic
Diagnostic categories on data servicesPer-request and per-query recordsEnable only categories you will use
Debug-level application logsDevelopers leave verbose levels onSet log levels per environment and sample
i
Practical Tip

Make each team see its own cost. When owners get a monthly report of their top tables, behavior changes faster than any central policy.

Building a cost review ritual

A one-time cleanup decays within months, because new applications and settings add data again. Create a monthly review of fifteen minutes with the platform owner and a finance partner. Look at the five largest tables, the biggest changes since last month, any table with rising volume and falling queries, and any workspace approaching a tier boundary. Record each decision and the owner who must act. Over a year, this small habit prevents more waste than any single dramatic project, and it builds a shared understanding that logs are a product with a price, not a free by-product of running systems.

22Daily Caps, Anomalies, and Budget Guardrails

Surprises come from sudden floods of data. Guardrails limit the damage.

Daily cap

A daily cap sets a maximum amount of billable data per day. When the cap is reached, ingestion of most billable data stops until the next day. This protects the budget but also creates blind spots at exactly the moment something unusual is happening, which is when you most need logs. Some data types are exempt from the cap. Use caps as a safety net on non-critical, development, or test workspaces, and consider alerts instead of hard stops for production.

Spotting anomalies early

Ingestion volume alerts

Alert when daily volume exceeds a multiple of the recent average, or when a single table suddenly dominates.

Billing alerts

Use Azure cost management budgets to notify owners when spending crosses thresholds.

Source tracking

When volume jumps, query by source computer, resource, or category to find the cause, often a misconfigured application logging at debug level.

Simple Analogy

A daily cap is a circuit breaker. It prevents a fire when something shorts, but it also switches off the lights. A volume alert is a smoke detector that tells you early and keeps the lights on while you check the wiring.

!
Common Mistake

Setting a daily cap on a security workspace. If an attack floods logs, the cap can silence the very records an investigator needs later.

Forecasting and budgeting

Cost forecasting for logs is easier than it looks. Take the daily billed volume per table over the last ninety days, note growth trends, and add expected changes such as new applications or planned migrations. Multiply by the unit price of the relevant plan and add retention and query charges for cheaper plans. Present the result to owners as a range, with the main assumptions written beside it. When the real number drifts outside the range, investigate quickly, because early drift is nearly always a single misconfigured source rather than a gradual change.

23Query Performance and Reliability

Fast queries save time and money, and reliable ingestion saves investigations.

Why queries are slow

CauseSymptomRemedy
Very wide time rangeLong waits and timeoutsNarrow the range, or use summary tables
Searching all tables or all columnsHeavy scansName the table and column
Late filteringLarge intermediate resultsFilter before joining or grouping
Joins on large tablesMemory limitsReduce each side first, join on a small key set
Parsing text at query timeRepeated heavy workParse at ingestion with a transformation or in a saved function
Auxiliary plan tablesSlow by designUse search jobs or summaries for time-sensitive needs
Simple Analogy

A slow query is like asking a librarian to read every book in the building to find one name. A fast query names the shelf, the year, and the author. The same librarian answers in seconds.

Reliability of the pipeline

Data can fail to arrive for many reasons: an agent stopped, a network path blocked, a rule removed, a daily cap reached, a schema mismatch, or a throttled sender. Build checks that confirm each important source is still reporting. Heartbeat records from machines, expected-volume alerts per table, and ingestion latency queries give you early warning of silent gaps.

Resilience across regions

A workspace belongs to one region. For critical data, you can send logs to workspaces in more than one region, and for some scenarios use workspace replication where supported. Alerts and workbooks do not move automatically if a region fails, so keep their definitions as templates ready to deploy against the alternate workspace.

i
Practical Tip

Monitor the monitor. A missing heartbeat or a sudden drop in a core table is often the first sign that logging itself has broken, and it is much cheaper to learn that today than in the middle of an incident.

Designing for both people and machines

Interactive investigators tolerate a few seconds of waiting, but alert rules and dashboards run constantly and magnify any inefficiency. A query that scans a day of data every five minutes will scan the same records hundreds of times. Move repeating logic into summary tables, keep alert windows short, and avoid expensive joins in scheduled queries. Review the slowest and most frequent queries regularly, since they combine to create most of the load. Faster, cheaper queries also leave room for analysts to explore without worrying about cost or timeouts, which improves the quality of investigations.

24Governance, Policy, and Operating Model

A workspace grows messy unless someone owns it. A light operating model prevents drift.

Roles

Platform owner

Owns workspace design, pricing model, retention policy, and the data collection rule library.

Data owners

Each major table has a named owner who justifies its collection, plan, and retention.

Security liaison

Approves filtering of security data and protected event lists.

Finance partner

Reviews monthly cost, forecasts, and commitment decisions.

Enforcement with Azure Policy

Azure Policy can require a diagnostic setting on supported resources, deploy agents and rule associations to machines, restrict workspace creation to approved regions, and audit workspaces that lack required settings such as private access or retention. Policies turn written standards into automatic behavior, and they fix gaps that appear when new resources are created.

Naming and tagging

Meaningful workspace names show purpose and region at a glance. Tags on workspaces and rules record owner, environment, and cost center, which makes chargeback and cleanup possible.

flowchart TD
  STD["Logging standard: what to collect, plan, retention"] --> POL["Azure Policy enforces settings"]
  POL --> RES["New and existing resources"]
  RES --> WS["Shared workspaces"]
  WS --> REP["Monthly usage and cost report per owner"]
  REP --> REV["Quarterly review of tables, plans, retention"]
  REV --> STD
        
FIG 7 — A governance loop for logging
Simple Analogy

Governance of a workspace is like the rules of a shared kitchen. Everyone may cook, but there are labeled shelves, a cleaning rota, and a monthly check of what has gone off in the fridge. Without them, the kitchen fills with forgotten containers that somebody pays for.

Documenting the standard

A logging standard works best as a short, living document. It lists which sources must send logs, which categories are required, which tables are mandatory for security, what plan and retention each class of data receives, and who approves exceptions. Add examples of good and bad requests, so teams can see how decisions are made. When a new service appears, the standard tells the owner exactly what to do on day one, and policy checks whether they did it. This turns monitoring from a favor requested from a central team into a normal part of delivering any workload.

25Migration from Retired Collection Methods

Moving off the HTTP Data Collector API and the older agent is the most time-sensitive task for many estates.

Step by step

1

Inventory classic tables

List tables whose subtype is classic. Records showing a source system of the REST interface, and columns with the old type suffixes, point to legacy ingestion.

2

Move machine sources to the Azure Monitor Agent

Do this first for any table fed by the older agent. Otherwise data stops flowing into custom fields after table conversion.

3

Create rules and endpoints

Define data collection rules, and endpoints where needed, with stream declarations that treat identifiers as text.

4

Choose a path

Either convert the existing table in place, which is one-way and cannot be rolled back, or build a new table side by side and delete the old one when ready.

5

Switch senders

Point applications at the Logs ingestion API with identity-based authentication, splitting payloads to stay within the per-call size limit.

6

Verify and update dependents

Confirm modern TLS on every client and update rules, workbooks, and security content that read the changed tables.

Simple Analogy

Migrating is like moving a busy shop to a new building. You build the new shelves first, train staff to use the new entrance, move stock in batches, and only close the old door when deliveries have clearly switched. Converting a table in place is a one-way trip, so rehearse with the side-by-side method when the data is important.

!
Lifecycle Warning

Microsoft’s page states support for the HTTP Data Collector API ended on 14 September 2026 and that its endpoint rejects older TLS versions since 1 March 2026. Verify the current status on Microsoft’s migration page and test every sender, because ingestion can fail silently from the sender’s point of view.

Testing a migration safely

Before converting anything important, copy a small sample of data through the new path and compare it with the old. Check record counts, column values, timestamps, and the behavior of every alert, workbook, and saved function that reads the table. Keep a rollback plan for the side-by-side approach, which is simply to leave the old table running until the new one is trusted. Communicate dates to application owners well in advance, because the people who wrote the original senders may have left, and their scripts may be running on forgotten machines.

26Patterns and Anti-Patterns

Experience from many estates shows a few recurring good and bad designs.

PATTERN-01 Recommended
Problem

Security and operations both need logs, but the volume is large and the bill is growing.

Approach

Land everything through collection rules that filter and route. Keep detection-critical tables on the Analytics plan, move troubleshooting data to Basic, store compliance data on Auxiliary, and build summary tables for dashboards.

Result

Fast alerts where they matter, long retention where required, and a bill driven by value rather than volume.

PATTERN-02 Recommended
Problem

Hundreds of resources must send logs consistently, and manual setup misses some.

Approach

Define the logging standard once, enforce it with Azure Policy, and deploy rules and settings through templates stored in source control.

Result

New resources are covered automatically and drift is detected and corrected.

ANTI-PATTERN-02 Avoid
Problem

Every diagnostic category is enabled on every resource “just in case”, and retention is set to the maximum.

Why It’s Harmful

Cost climbs, signal drowns in noise, and nobody can explain why most data is kept.

Correct Approach

Enable categories for named use cases, give each table an owner, and set retention from a documented need.

ANTI-PATTERN-03 Avoid
Problem

Applications send logs using a shared workspace key stored in many scripts.

Why It’s Harmful

A leaked key lets anyone write false records, and rotation breaks everything at once.

Correct Approach

Use the Logs ingestion API with managed identities and role assignments scoped to a single collection rule.

PATTERN-03 Recommended
Problem

Dashboards over very large tables load slowly and cost too much to refresh all day.

Approach

Create summary rules that aggregate the raw table into hourly figures, point dashboards and alerts at the summary table, and keep the raw table on a cheaper plan with long-term retention.

Result

Dashboards respond quickly, alert queries stay small, and raw detail remains available for investigation.

27Real-World Scenarios

Three short stories show how the concepts combine.

Scenario one: a retailer with a runaway bill

A retailer’s monthly log bill doubled in a quarter. Usage queries showed that two tables, network flow records and container output, produced most of the volume. The team filtered flow records at ingestion, built a summary table for the dashboards, moved container output to the Basic plan, and shortened interactive retention on noisy tables. Volume on the Analytics plan dropped sharply, and only then did they select a smaller commitment tier that matched the new, steadier baseline.

Scenario two: a bank with long retention duties

A bank needed seven years of audit evidence searchable within days. It kept ninety days of security data in interactive retention for detection, relied on long-term retention for the remaining years, put verbose platform audit logs on the Auxiliary plan, and used search jobs when investigators asked about older events. It also exported critical tables to immutable storage, and used customer-managed keys on a dedicated cluster, with private access only.

Scenario three: a software company migrating old custom ingestion

A software company discovered dozens of internal scripts posting data to the retired collector API. It listed classic tables, rebuilt senders on the new ingestion API with managed identities, ran old and new tables side by side for two weeks to compare counts, and then removed the old tables. A policy now prevents new workspace keys from being used by application teams.

“Logs are cheap to collect and costly to ignore. The art is knowing which is which before the invoice arrives.”

Scenario four: a manufacturer with plants in several countries

A manufacturer operated plants under data residency rules in three regions. It created one workspace per regulated region and a small central workspace for shared services. Resource-context access let plant engineers see only their own equipment logs, while a central team used cross-workspace queries for company-wide reports. Summary rules sent hourly aggregates to the central workspace, so detailed local data never left its region, and the monthly bill stayed predictable.

28Best Practices and Common Mistakes

A quick reference of what experienced teams do and what they avoid.

Do

  • Start with one workspace and add more only for stated reasons
  • Use collection rule transformations to cut noise before billing
  • Classify every large table by value and choose the matching plan
  • Build summary tables for dashboards and heavy alerts
  • Enforce diagnostic settings and agents through Azure Policy
  • Use identity-based authentication for custom ingestion
  • Apply table-level and resource-context access for least privilege
  • Review usage, plans, and retention every quarter

Avoid

  • Enabling every log category everywhere “just in case”
  • Leaving legacy pricing tiers that block cheaper plans
  • Sending the same data to several workspaces
  • Choosing a commitment tier before optimizing volume
  • Applying hard daily caps to security workspaces
  • Giving broad contributor rights where readers suffice
  • Leaving restores active after the investigation
  • Ignoring the retirement notices for old collection methods

29Troubleshooting Playbook

Most problems fall into a few families. Work through them in order.

SymptomLikely causeFirst action
No data from a machineAgent stopped, rule not associated, or network blockedCheck the agent status, rule association, and connectivity to the endpoints
No data from an Azure resourceDiagnostic setting missing or wrong categoryConfirm the setting, destination, and chosen categories
Custom ingestion failsWrong table name, schema mismatch, payload too large, or old TLSCheck rule stream, table columns, request size, and client TLS version
Data arrives lateBatching at the source or ingestion delayCompare generated time with ingestion time
Query times outWide range or heavy joinsNarrow the range and filter earlier
Unexpected bill spikeNew noisy source or debug loggingRank tables and sources by billed size over recent days
Alert did not fireData not ingested yet, wrong query window, or unsupported planTest the query manually and check table plan and ingestion delay
User sees too little or too muchAccess mode and role assignmentReview workspace access mode, resource permissions, and table rules
flowchart TD
  S["Expected logs are missing"] --> Q1{"Is it one source or many?"}
  Q1 -- Many --> Q2{"Daily cap reached or workspace unreachable?"}
  Q2 -- Yes --> F1["Check cap, network settings, private link modes"]
  Q2 -- No --> F2["Check rule changes and policy deployments"]
  Q1 -- One --> Q3{"Machine or Azure resource?"}
  Q3 -- Machine --> F3["Check agent health, rule association, endpoints"]
  Q3 -- Azure resource --> F4["Check diagnostic setting and categories"]
  Q3 -- Custom sender --> F5["Check identity, rule stream, schema, size, TLS"]
        
FIG 8 — A decision path for missing logs
i
Practical Tip

Keep a short list of health queries: last heartbeat per machine, daily volume per table, ingestion delay per table, and failures reported by collection rules. Running them weekly catches silent problems early.

Escalating with good evidence

When a problem needs support from another team or from Microsoft, gather evidence first. Note the workspace identifier, the table, the time range, the exact symptom, and what you already tried. Include sample records that arrived and records that did not, plus any recent changes to rules, policies, or network settings. Clear evidence shortens resolution dramatically, and writing it down often reveals the answer before anyone else reads it. Afterwards, add the lesson to the troubleshooting playbook so the next person solves the same problem faster.

30Frequently Asked Questions

Short answers to questions that beginners and architects ask most often.

Q1Is a Log Analytics workspace the same as Azure Monitor?
No. Azure Monitor is the overall service family. The workspace is the log store inside it, used by many Azure Monitor features and by Microsoft Sentinel.
Q2How many workspaces should I have?
As few as meet your needs. Split for tenants, data residency, ownership, or retention differences, and combine to reach commitment tiers.
Q3What is the difference between Analytics, Basic, and Auxiliary?
Analytics offers full features for monitoring and alerting. Basic is cheaper to ingest with scan-based query charges. Auxiliary is the cheapest, with slower queries and fewer features, for verbose and compliance data.
Q4How long can I keep logs?
Total retention can reach twelve years within the workspace. Interactive retention on Analytics tables can be extended up to two years, and the rest is long-term retention.
Q5Can I query old data in long-term retention?
Yes, through search jobs, which copy results into a new table, or by restoring a time range on plans that support restore.
Q6What replaced the HTTP Data Collector API?
The Logs ingestion API, which uses data collection rules and identity-based authentication. Microsoft’s migration page says support for the older API ended on 14 September 2026.
Q7What replaced the older Log Analytics agent?
The Azure Monitor Agent, configured through data collection rules.
Q8Is it worth buying a commitment tier?
If your Analytics ingestion is steadily at or above one hundred gigabytes per day, usually yes. Optimize volume first, then choose a tier from a stable average.
Q9Why is my bill higher than the volume I sent?
Common reasons are data that fails the schema but is still billed, duplicate collection, long retention, scans on Basic or Auxiliary plans, restores left active, and Sentinel pricing on the whole workspace.
Q10Do I need a dedicated cluster?
Consider one at about one hundred gigabytes per day or when you need customer-managed keys. Remember that billing starts at creation and the commitment cannot be reduced within its period.
Q11How do I let application teams see only their logs?
Use resource-context access so that read permission on a resource grants its logs, and use table-level controls to block sensitive tables.
Q12Should I set a daily cap?
Use it as a safety net on development and test workspaces. Avoid hard caps on production and security workspaces, and use volume alerts instead.

31Summary and Key Takeaways

A Log Analytics workspace is Azure’s central, queryable home for log data. Good outcomes come from deliberate choices about what enters, where it sits, how long it stays, and who may read it.

  • One archive, many sources. Agents, diagnostic settings, the ingestion API, and connectors all feed tables in the workspace.
  • Control the front door. Data collection rules and transformations filter, shape, and route data before it is billed.
  • Match plan to value. Analytics for monitored data, Basic for troubleshooting, Auxiliary for verbose and compliance data.
  • Retention in two stages. Short interactive retention for speed, long-term retention up to twelve years for obligation, with search jobs and restore for old data.
  • Start simple on design. Begin with one workspace and split only for tenants, regions, ownership, or retention needs.
  • Least privilege. Combine resource-context access with table-level rules, private links, and customer-managed keys where required.
  • Optimize, then commit. Cut volume first, then pick pay-as-you-go, a commitment tier, or a dedicated cluster.
  • Migrate off retired paths. Replace the HTTP Data Collector API and the older agent with the Logs ingestion API and the Azure Monitor Agent, and verify dates on Microsoft’s pages.