Start with the one-sentence idea, then build the picture slowly.

Azure Databricks is a data and artificial intelligence platform that runs inside Azure. It is built by the company Databricks, whose founders created Apache Spark, and it is offered on Azure as a first-party service, which means it is sold, supported, and connected to Azure’s identity, billing, and security as a native Azure service. You use it to store, clean, analyse, and learn from very large amounts of data.

Let us explain the words. Data is recorded facts, such as the time of a sale or a reading from a sensor. Big data means data so large, so fast-growing, or so varied that one normal computer cannot handle it. Analytics is the work of studying data to find patterns and answers. Machine learning is a way of teaching a computer to find patterns in past data and then make predictions about new data. Artificial intelligence, or AI, is the broader field of making computers do tasks that normally need human intelligence.

Apache Spark is an open tool that spreads data work across many computers working together. A group of such computers is called a cluster. Spark is fast because it keeps data in memory while it works. Databricks takes Spark and wraps it in a full platform: a web workspace, managed clusters, a catalogue for data, scheduling for jobs, tools for machine learning, and strong security.

The word lakehouse describes Databricks’ main idea. A data lake is a cheap, huge store of files of any shape. A data warehouse is a tidy, structured store tuned for fast reports. A lakehouse tries to give you both: the cheap, open storage of a lake, with the reliability and speed of a warehouse, in one place.

To see why Databricks helps, look at how big data work used to go.

Not long ago, a company kept its data in a few databases and analysed it on one large server. As data grew, the server slowed down, and buying a bigger one cost a fortune and took months. Eventually even the biggest server could not cope.

The first answer: Hadoop

Engineers found that many cheap computers working together could do what one giant computer could not. A system called Hadoop stored files across many machines and processed them in parallel. It worked, but it was slow because it wrote results to disk between every step, and hard to use because it needed deep technical skill.

Spark makes it faster

Spark improved on this by keeping data in memory between steps, which made many jobs ten to a hundred times faster. It also offered friendlier languages, such as Python and SQL. But running a Spark cluster still meant installing software, tuning settings, handling failures, and keeping security patched. Companies needed whole teams just to keep clusters alive.

Lakes became swamps

Data lakes held the files, but they had no guarantees. If two jobs wrote to the same folder at the same time, files could be half written. If a job crashed midway, it left broken data behind. No one could tell which version of a table a report had used. Many data lakes slowly turned into messy “data swamps” that no one trusted.

Many tools, many copies

Data engineers used one set of tools, analysts another, and data scientists a third. Each group copied the data to its own place. Rules about who could see what were repeated in each tool, and they drifted apart. A single sensitive column might be protected in one place and wide open in another.

The cloud changes the rules

In the cloud, storage is cheap and machines can be rented by the minute. This allows a key design idea: separate storage from compute. Keep the data safely in cheap storage all the time, and start powerful clusters only while you need them. Databricks is built around this idea.

Every service exists because someone had a painful problem. Here are the main ones.

From raw files to trusted answers

A retailer receives millions of sales records every day from stores and websites. Databricks lets the team load them, clean them, join them with product lists, and publish trusted tables, with all steps recorded and repeatable. Reports are then built from the trusted tables, not from messy raw files.

Experiments that can reach production

A data scientist builds a model that predicts which customers may leave. In many companies, the model stays in a laptop notebook and never reaches real use. On Databricks, the same platform can track experiments, store the model, serve it, and watch its quality, making the path from idea to production shorter.

Sharing data safely

Companies often need to share data with partners without handing over copies. Databricks supports open sharing methods so that other parties can read selected tables under controlled rules, without emailing files around.

Ten building blocks. Learn these and the rest becomes easy.

Account and workspace

An account is the top-level container for your organisation in Databricks. A workspace is an environment inside it, where people work with notebooks, clusters, jobs, and data. In Azure, a workspace is an Azure resource that you create in a subscription and region. Large organisations often have several workspaces, for example for development, testing, and production.

Notebook

A notebook is an interactive document that mixes text, code, and results. You write a piece of code in a cell, run it, and see the answer right below. Notebooks support Python, SQL, Scala, and R, and are the main place for exploring and building.

Cluster and compute

A cluster is a group of virtual machines that work together on your data. One machine is the driver, which plans the work, and the others are workers, which do it. Databricks calls this resource compute in modern documentation. Compute can be created manually, defined for a job, or provided by Databricks in a serverless form.

Databricks Runtime

The Databricks Runtime is the software bundle that runs on the cluster: Spark, Delta Lake, common libraries, and performance improvements. New versions appear regularly, and some are marked for long-term support. A part of the runtime called Photon is a faster engine for SQL and data frame work.

Delta Lake

Delta Lake is an open table format that adds reliability to files in the lake. It stores data as files plus a log of changes, giving safe all-or-nothing writes, the ability to see older versions, and better speed.

Unity Catalog

Unity Catalog is the central system for governing data and AI assets. It keeps a catalogue of tables, files, and models, controls who may use them, and records lineage and audit trails across workspaces.

Job and workflow

A job runs work automatically, on a schedule or when triggered. A workflow chains several tasks, such as notebooks, SQL queries, and pipelines, in a defined order with dependencies.

SQL warehouse

A SQL warehouse is compute designed for running SQL queries and dashboards, tuned for analysts and reporting tools.

MLflow

MLflow is an open tool for tracking machine learning experiments and managing models. It is integrated into Databricks.

Storage

Your actual data sits in Azure Data Lake Storage Gen2, which is Azure’s storage account type for analytics, with folders and fine-grained permissions. Databricks reads and writes it directly.

Understanding where things live is the key to understanding security and cost.

Databricks splits its architecture into two main parts. The control plane is the management side, run by Databricks. It contains the web interface, the notebook storage, the job scheduler, and the cluster manager. The compute plane is where your data is actually processed. There are two main kinds of compute plane.

flowchart TD
    subgraph CP["Control plane - managed by Databricks"]
        UI["Web interface and notebooks"]
        JS["Job scheduler"]
        CM["Cluster manager"]
    end
    subgraph SUB["Your Azure subscription - classic compute plane"]
        VNET["Virtual network"]
        DRV["Driver machine"]
        WRK["Worker machines"]
    end
    ADLS["Azure Data Lake Storage Gen2"]
    UI --> CM
    JS --> CM
    CM -->|"Starts and manages"| DRV
    DRV --> WRK
    WRK --> ADLS
    DRV --> ADLS

What lives in each plane

The control plane holds your notebook code, job settings, and metadata about clusters. Your business data does not live there. It stays in your storage accounts and is processed by compute in the compute plane. Query results shown in the notebook screen pass through to your browser, and some are stored for a short time in the control plane, so read the current documentation about what is kept where if you have strict rules.

Managed resource group

When you create a classic workspace, Azure creates a separate managed resource group that holds the workspace’s supporting resources, such as virtual machines, a default storage account for workspace use, and networking pieces. You should not change these by hand, because Databricks manages them.

The default storage and your own storage

The workspace has a built-in storage area for system files. Your real data should live in your own storage accounts, connected through Unity Catalog, so that you keep control over location, retention, and access. Treat the built-in area as temporary and not as the place for important data.

How a cluster starts

When you start a classic cluster, the control plane asks Azure to create virtual machines in your subscription. The machines join your network, download the runtime, and register with the cluster manager. This takes a few minutes. Pools of pre-started machines and serverless options reduce this wait.

What happens when a cluster processes a huge table.

When you ask Spark to do something, such as “total sales per store”, it does not do it on one machine. It splits the data into many pieces called partitions, gives pieces to different workers, lets each worker handle its pieces, and then combines the results.

Driver and executors

The driver is the manager. It reads your instructions, builds a plan, and splits it into tasks. The workers run executors, which are processes that carry out tasks on partitions and keep intermediate data in memory. When all tasks finish, the driver gathers results.

Lazy plans and optimisation

Spark is lazy in a clever way. When you describe a series of steps, such as filter, join, and group, Spark does not run each one immediately. It builds a plan first, then an optimiser finds a faster way to do the whole chain, such as filtering early so less data moves. Only when you ask for a result does the work run. Databricks adds further optimisations and its Photon engine to speed up common operations.

Shuffles: the expensive step

Some operations, like joining two big tables or grouping by a key, need rows with the same key to be on the same machine. Spark must move data across the network to arrange this, which is called a shuffle. Shuffles are the most common reason for slow jobs. Good design tries to reduce the data before shuffling and to avoid unneeded shuffles.

flowchart LR
    A["Your instruction"] --> B["Driver builds a plan"]
    B --> C["Optimiser improves the plan"]
    C --> D["Plan split into tasks"]
    D --> E["Worker 1 - partitions"]
    D --> F["Worker 2 - partitions"]
    D --> G["Worker 3 - partitions"]
    E --> H["Shuffle if needed"]
    F --> H
    G --> H
    H --> I["Combined result to driver"]

Memory and spilling

Executors keep working data in memory for speed. When data is larger than memory, Spark spills extra data to local disk, which is slower. If memory runs out badly, tasks fail. Choosing machine sizes with enough memory, and balanced partition sizes, avoids these troubles.

Skew

Skew happens when some partitions hold far more data than others, for example when one customer has millions of records and others have a few. One task then runs long while others finish and wait. Databricks and Spark have features to reduce skew, but good key choice and design still matter.

Fault tolerance

If a worker machine dies, Spark knows how each partition was created, and recomputes only the lost pieces on other workers. Jobs often continue with no human action. Still, repeated failures or lost drivers can stop a job, so use retries and monitoring.

Choosing the right machines for the job is the most important cost decision.

All-purpose versus job compute

Developers use all-purpose clusters to explore and build. Once the work is ready, it should run on job compute, which exists only while the job runs. Running production work on an always-on all-purpose cluster is one of the commonest ways to waste money.

Autoscaling and auto-termination

Autoscaling lets a cluster add workers when the workload grows and remove them when it shrinks, within limits you set. Auto-termination stops a cluster after it has been idle for a chosen time. Together they prevent paying for idle machines. Set a short idle time for development clusters.

Choosing machine types

Azure offers many virtual machine families. Memory-heavy jobs, such as large joins and caching, like memory-optimised machines. Compute-heavy jobs like compute-optimised ones. Jobs that read the same data repeatedly may benefit from machines with fast local disks that cache files. Machine sizes and names change, so check the options available in your region.

Spot instances

Azure sells spare capacity at large discounts as spot machines, which can be taken back with short notice. Databricks can use spot workers for tolerant jobs, falling back to normal machines if spot is reclaimed. Keep the driver on a normal machine, and use spot only for work that can retry.

Cluster access modes

A cluster has an access mode that decides how users share it and whether Unity Catalog rules apply. Shared modes allow several users with isolation and enforce data permissions, while single-user modes assign the cluster to one person or service. For governed data, use a mode that supports Unity Catalog and the isolation your rules need. Names of modes have changed over time, so read the current documentation.

flowchart TD
    Q["What kind of work is it?"] -->|"Exploring and developing"| A["All-purpose compute with auto-termination"]
    Q -->|"Scheduled production pipeline"| B["Job compute created per run"]
    Q -->|"SQL dashboards and analysts"| C["SQL warehouse"]
    Q -->|"Want no cluster management"| D["Serverless compute"]
    B --> E["Consider spot workers for tolerant jobs"]

The feature that turns a pile of files into tables you can trust.

Raw data files such as Parquet are efficient, but on their own they cannot promise safe changes. Delta Lake adds a layer on top. A Delta table is a folder containing data files plus a transaction log, a record of every change made to the table. Readers use the log to know exactly which files make up the table at any moment.

What Delta gives you

• ACID transactions: changes are atomic (all or nothing), consistent, isolated from each other, and durable once saved. No more half-written tables.
• Time travel: you can query an older version of a table, handy for audits, fixing mistakes, or repeating an old report exactly.
• Updates, deletes and merges: you can change or remove individual records and merge new data into old, which raw files do not support well.
• Schema enforcement: the table refuses data that does not match its structure, protecting quality. Controlled schema evolution lets you add columns on purpose.
• Performance features: data skipping with statistics, file compaction, and layout choices that speed up queries.
• Change data feed: an option to read just what changed, useful for passing changes downstream.

How a write works

When a job writes to a Delta table, it first writes new data files. Then it tries to add a new entry to the transaction log naming those files. This entry is the commit. If another writer committed a conflicting change in the meantime, Delta checks whether the changes can safely coexist, and retries or fails with a clear error. Readers always see either the old version or the new version, never something in between.

sequenceDiagram
    participant W as "Writer job"
    participant S as "Storage files"
    participant L as "Transaction log"
    participant R as "Reader query"
    W->>S: "Write new data files"
    R->>L: "Read latest version of the log"
    L-->>R: "Version 5 - old file list"
    W->>L: "Commit version 6 naming new files"
    R->>L: "Next query reads the log"
    L-->>R: "Version 6 - new file list"

Maintenance

Over time, tables collect many small files and old files that are no longer part of the current version. Regular maintenance helps. Compaction combines small files into larger ones for faster reads. Vacuum deletes old files that are no longer needed once the retention period for time travel has passed. Databricks offers automatic and predictive maintenance options for managed tables, so check what is available and enabled.

Organising data for speed

Partitioning splits a table into folders by a column value, such as date, so queries can skip irrelevant folders. But too many small partitions hurt. Newer layout options, such as clustering of data by chosen columns, can adapt better than fixed folders. Features here have changed over time, so check the current guidance for choosing between them.

Open and portable

Delta is an open format. Other tools can read Delta tables, and the files live in your own storage. That reduces lock-in compared with storing data in a closed system, though features specific to Databricks remain tied to the platform.

One catalogue, one rulebook, many workspaces.

Before Unity Catalog, each workspace had its own list of tables and its own permissions. Rules had to be copied and kept in step by hand. Unity Catalog moves governance to the account level, so many workspaces share one catalogue and one set of rules.

The three-level naming

Unity Catalog organises data in three levels. A metastore is the top container for the account’s region. Inside it, a catalog is the first level, a schema (also called a database) is the second, and a table, view, volume, function, or model is the third. A table is addressed as catalog, schema, and table, joined by dots. Typical catalogs separate environments or business areas, such as one for production and one for development.

flowchart TD
    M["Metastore - one per region"] --> C1["Catalog - sales"]
    M --> C2["Catalog - marketing"]
    C1 --> S1["Schema - orders"]
    C1 --> S2["Schema - customers"]
    S1 --> T1["Table - daily_orders"]
    S1 --> T2["Table - order_items"]
    S2 --> T3["Table - customer_profile"]
    S2 --> V1["Volume - uploaded files"]

Securable objects and permissions

Everything you can protect is a securable object. Permissions are granted to users, groups, or service principals, which are identities for programs. A permission on a catalog flows down to the schemas and tables inside, unless limited. Use groups, not individual people, to keep rules simple. Typical permissions include permission to see an object, to read it, to change it, and to create things in it.

Row filters and column masks

Beyond table-level access, Unity Catalog supports row filters, which hide rows a user should not see, and column masks, which hide or disguise sensitive values, such as showing only part of an identification number. This lets different people use the same table with different views.

Lineage and audit

Lineage shows where data came from and where it goes: which tables feed a report, and which notebook produced a table. This helps with impact analysis, such as “if I change this column, who is affected?”, and with proving the origin of numbers. Audit logs record who accessed what and when.

External locations and storage credentials

Unity Catalog accesses your storage accounts through objects that represent a secure connection. In Azure, an access connector with a managed identity can be given rights on a storage account, and Unity Catalog uses it to reach data on users’ behalf under governance rules. This avoids storing account keys in code.

Managed and external tables

A managed table stores data in a location governed by Unity Catalog, and dropping the table removes its data after a retention window. An external table points at data in a location you manage, and dropping it removes only the table definition. Choose according to who should own the data’s lifecycle.

Sharing outside the organisation

Delta Sharing is an open way to share live data with other organisations or platforms without making copies. Providers choose which tables to share and with whom, and recipients read them with compatible tools. Check current options and limits for your needs.

A tidy way to organise data from raw to polished.

A popular pattern on Databricks is the medallion architecture, which stores data in three quality levels. Data flows through them, improving at each step.

flowchart LR
    SRC["Sources - databases, files, streams"] --> ING["Ingestion with Auto Loader or connectors"]
    ING --> BR["Bronze tables"]
    BR --> SV["Silver tables"]
    SV --> GD["Gold tables"]
    GD --> BI["SQL warehouse and dashboards"]
    GD --> ML["Machine learning"]
    GD --> SH["Delta Sharing"]

Ingestion

Getting data in is the first step. Auto Loader is a Databricks feature that watches a storage folder and loads new files as they arrive, keeping track of what has been processed, and handling changing file structures. For databases and software services, connectors and managed ingestion options exist, and Azure Data Factory or event streaming services can also feed the lake. Check which sources have managed connectors today.

Batch and streaming

Batch processing handles data in chunks, such as the previous day’s records. Streaming processes data continuously as it arrives. Spark Structured Streaming treats a stream as a table that keeps growing, so the same kinds of queries work for both. Streaming gives fresher results but needs more careful design for late data, failures, and cost, since clusters run constantly or in frequent triggers.

Declarative pipelines

Instead of coding every step and its error handling, you can describe the tables you want and the rules for data quality, and let the platform manage the order, retries, and updates. Databricks offers such a pipeline framework, previously known as Delta Live Tables and now presented under a newer product family name, so check current naming. You can attach expectations, which are quality rules such as “this column must not be empty”, and choose whether failures drop rows, warn, or stop the pipeline.

Change data capture

Source systems change records over time. Change data capture means applying inserts, updates, and deletes from the source to your tables, rather than reloading everything. Delta’s merge feature and pipeline tools support it. Handling late and out-of-order changes needs careful keys and sequence columns.

Keep raw data

Always keep bronze data unchanged and labelled with arrival time and source. When a business rule changes or a bug is found, you can rebuild silver and gold from bronze.

Turning working notebooks into dependable, scheduled production processes.

A notebook that works on your screen is not yet a production process. Production needs scheduling, retries, alerts, dependencies, and records of each run. Databricks Jobs, also called Workflows, provide this.

Tasks and dependencies

A job is made of tasks. A task can run a notebook, a script, a SQL query, a pipeline, or a packaged program. Tasks can depend on each other, so one starts only when the earlier ones succeed. You can add branches that run when something fails, loops over a list of items, and conditions that decide the path.

flowchart TD
    TR["Trigger - daily schedule or file arrival"] --> T1["Task 1 - ingest to bronze"]
    T1 --> T2["Task 2 - clean to silver"]
    T2 --> T3["Task 3 - build gold tables"]
    T3 --> T4["Task 4 - refresh dashboards"]
    T1 -->|"On failure"| AL["Send alert"]
    T2 -->|"On failure"| AL
    T3 -->|"On failure"| AL

Triggers

Jobs can start on a schedule, such as every night, when new files arrive in storage, on a table update, or through a call from another system. Continuous mode keeps a job running for streaming. Choose the lightest trigger that meets the freshness need.

Parameters and reuse

Parameters let one job serve many cases, such as different dates or regions. Share common logic through libraries and modules rather than copying notebooks. Use environments that can be reproduced, with fixed library versions, so a job behaves the same next month.

Source control and deployment

Treat data work like software. Keep code in a Git repository and connect it to the workspace. Test changes in a development workspace, then promote to production through a controlled process. Databricks supports packaging and deploying jobs and settings as code through bundles and APIs, so environments can be created repeatedly and reviewed.

Running as a service identity

Production jobs should run as a service principal, an identity for a program, not as a person. If a person leaves, jobs owned by their account may stop. A service principal also lets you give the job only the permissions it needs.

Reliability habits

• Make tasks idempotent: running twice gives the same result as once, so retries do not create duplicates.
• Set retries and timeouts for temporary faults and runaway tasks.
• Alert with context: include which task failed, the run link, and the error.
• Check data quality between layers, not only whether tasks finished.
• Record run identifiers in your tables so numbers can be traced to a run.

Serving analysts and dashboards from the lakehouse.

Not everyone wants to write Python. Analysts know SQL, and managers want dashboards. Databricks SQL is the part of the platform for this audience, with its own query editor, dashboards, alerts, and SQL warehouses.

SQL warehouses

A SQL warehouse is compute tuned for SQL. It can scale out with more clusters when many people query at once, and it can start and stop quickly. Serverless warehouses start in seconds and are managed by Databricks. The size of a warehouse, chosen from small to very large, sets the power of each query, while scaling settings control how many queries run side by side.

Connecting reporting tools

Tools such as Power BI and others connect to warehouses through standard connections, so dashboards read the same governed gold tables. Use identity-based access where possible so that column masks and row filters apply to each viewer. Using a shared password for all report viewers hides who is asking and can bypass fine-grained rules.

Making queries fast

• Query gold tables, which are pre-shaped for the question, instead of raw ones.
• Select only the columns you need. Columnar storage reads less data this way.
• Filter early on columns used for layout so files can be skipped.
• Use materialised results for heavy, repeated aggregations if available.
• Use caching features for repeated queries.
• Keep tables maintained with compaction and up-to-date statistics.

Lakehouse versus warehouse

For years, companies used a separate data warehouse for reports and a data lake for everything else. The lakehouse idea says one copy of data on open files can serve both, avoiding extra copies and sync problems. Whether it fully replaces a classic warehouse depends on your needs for concurrency, response time, and existing tools. Many organisations run both side by side and move gradually.

From raw data to predictions, in one connected workflow.

Machine learning projects have many steps beyond training a model: preparing features, tracking experiments, comparing versions, deploying, and watching behaviour afterwards. Databricks supports this life cycle on the same platform where the data lives.

Preparing data and features

A feature is a measurable property used as input to a model, such as a customer’s number of purchases in the last thirty days. Preparing features is often most of the work. Feature tables stored in the catalogue can be reused by many models and by live serving, which reduces the mismatch between training and production.

Training and tracking

Data scientists train models using popular libraries on single machines or across clusters. MLflow tracking records every experiment: settings, data version, measured quality, and the resulting model. This makes results reproducible and comparable. Without tracking, teams soon forget which settings produced the good model.

The model registry

Good models are registered in a central model registry, with versions, notes, and stage labels. In Unity Catalog, models are governed like tables, with permissions and lineage. A model can be promoted from testing to production after review.

flowchart LR
    D["Gold data and feature tables"] --> TR["Train models with tracked experiments"]
    TR --> EV["Evaluate and compare"]
    EV --> RG["Register the best model in Unity Catalog"]
    RG --> DP["Deploy for batch or real-time serving"]
    DP --> MN["Monitor quality and drift"]
    MN -->|"Quality drops"| TR

Serving predictions

Models can score large tables in batch, for example every night, or answer requests one at a time through a hosted endpoint for live use, such as showing recommendations on a web page. Serverless serving options reduce infrastructure work, though costs and limits should be checked.

Monitoring and drift

Drift happens when the real world changes and a model’s predictions get worse, for example when customer habits shift. Watch input data and prediction quality over time, and retrain when needed. A model without monitoring slowly becomes wrong while still looking confident.

Generative AI and large language models

Databricks also offers tools for building applications with large language models, including hosting and calling models, searching your own documents with vector search, evaluating answers, and governing the whole thing. These features evolve quickly, and product names change often, so rely on the current documentation. Whatever the tools, the same discipline applies: use trustworthy data, test carefully, control access, and watch for wrong or unsafe answers.

Making jobs faster and cheaper without simply buying bigger machines.

Performance tuning means finding why a job is slow and fixing the cause. Often a design change saves more than extra hardware, and it costs less.

A tuning routine

Caching

If you use the same data several times in one job, keeping it in memory can save repeated reads. But caching everything wastes memory. Cache only data that is reused and moderately sized, and release it when done. Disk caching on workers can also speed repeated reads of the same files.

Adaptive features

Recent Spark and Databricks versions can adapt plans while running, for example by merging small partitions, switching join methods, and splitting skewed partitions. Keep runtimes reasonably current to benefit from these improvements, and test upgrades before moving production.

Data platforms hold a company’s most valuable assets. Protect them in layers.

Identity and sign-in

Azure Databricks integrates with Microsoft Entra ID for sign-in, so people use their work accounts with multi-factor checks and conditional access. Users and groups can be synchronised from Entra ID into the account, so that joiners and leavers are handled centrally. Service principals provide identities for automation.

Authorisation layers

Access is controlled in several places: Azure roles for managing the workspace resource, workspace permissions for notebooks, clusters, and jobs, and Unity Catalog permissions for data. A user may need rights in more than one layer, which surprises beginners. Use groups at every layer, grant the least privilege needed, and review regularly.

Network isolation

flowchart TD
    USR["User on corporate network"] -->|"Private Link"| WS["Databricks workspace front end"]
    WS --> CP["Control plane"]
    CP -.->|"Secure cluster connectivity - outbound only"| CL["Cluster in your virtual network"]
    CL -->|"Private endpoint"| ST["Storage account"]
    CL --> FW["Firewall with allow list"]
    FW --> EXT["Approved external services only"]

Protecting data

• Encryption at rest and in transit for storage and connections, with options to manage your own keys.
• Row filters and column masks in Unity Catalog for sensitive fields.
• Secrets management through Azure Key Vault or Databricks secret scopes, so passwords never appear in notebooks.
• Audit logs recording sign-ins, data access, and administrative actions.
• Data exfiltration controls by restricting outbound routes and sharing options.
• Compliance features such as hardened images and monitoring add-ons for regulated industries, offered under specific plans, so check availability.

Security checklist

• Use Entra ID with multi-factor checks and synchronise groups.
• Govern all data with Unity Catalog and grant access to groups.
• Run production as service principals, not people.
• Use VNet injection, secure cluster connectivity, and Private Link for sensitive workloads.
• Keep secrets in a vault, never in notebooks.
• Restrict direct storage access so the catalogue rules cannot be bypassed.
• Use cluster policies to enforce safe settings.
• Collect and review audit logs in a central workspace.

You cannot manage what you cannot see. Here is what to watch.

Monitoring means continuously checking the health and behaviour of your system. Metrics are numbers measured over time. Logs are written records of events. Alerts are automatic messages when a number crosses a line you choose.

Where the information lives

Databricks provides several sources: the Spark UI and cluster metrics for individual jobs, job run pages with task outputs, query history for SQL, and system tables, which are built-in tables in Unity Catalog that record usage, billing, audit, lineage, and compute information. You can query them like any other table to build cost and usage dashboards. Logs and metrics can also be delivered to Azure storage or Azure Monitor and Log Analytics for central analysis. The exact set of system tables grows over time, so check what is available.

Data observability

A job can succeed technically and deliver wrong data. Add checks for row counts, empty values, freshness, and totals compared with the source. Treat these checks as part of the pipeline, and alert on failures just as you would alert on a crash.

Troubleshooting step by step

Powerful platforms can produce powerful bills. Know what drives the cost.

Prices differ by region, plan, and time, so use the official pricing pages and the Azure pricing calculator for real numbers. The structure of the bill is stable enough to learn.

Two bills in one

With classic compute, you pay two parties. You pay Databricks for Databricks Units (DBUs), a measure of processing capability used per hour, with the price per DBU depending on the type of workload and plan. You also pay Azure for the underlying virtual machines, storage, and network. With serverless compute, the charges are combined into one usage-based rate. Understanding which costs apply to which compute type is key to forecasting.

Commitments

Both Databricks and Azure offer discounts for committing to a level of use over a period, such as pre-purchase plans for DBUs and reservations for virtual machines. These can save a lot for steady workloads, but only if your usage really stays at that level. Compare them with the flexibility of pay-as-you-go before committing.

Ways to save

• Move production to job compute and keep all-purpose clusters for development only.
• Set auto-termination to a short idle time for interactive clusters.
• Use cluster policies to cap sizes and force tags.
• Use autoscaling with sensible minimum and maximum worker counts.
• Use spot workers for tolerant, retryable jobs.
• Process incrementally instead of reprocessing all history every run.
• Keep tables compact and well laid out so queries read less data.
• Stop and review idle SQL warehouses, and set auto-stop.
• Tag everything by team and project, and review the cost reports often.
• Set budgets and alerts so surprises show up within days.

Azure and the wider market offer many data platforms. Compare before you choose.

Product lines and positioning change quickly. In particular, check Microsoft’s and Databricks’ current guidance, because Microsoft Fabric and Azure Databricks are both strategic offerings and are designed to work together in many scenarios, for example through shared open Delta tables.

Databricks and Synapse

Synapse combines a SQL warehouse engine with Spark and pipelines in one Azure workspace and suits teams with strong SQL warehouse skills. Databricks offers a deeper and more advanced Spark, lakehouse, and machine learning experience. Some organisations use both, with Databricks for engineering and machine learning and a warehouse for serving SQL users, though the overlap has grown over time.

Databricks and Fabric

Fabric offers a simpler, capacity-based, fully managed experience with a single shared lake, and strong links to Power BI. Databricks offers more control, a mature engineering and machine learning environment, and open governance across clouds. The two can coexist: Delta tables written by Databricks can often be read in Fabric, and the reverse. Evaluate both for new projects against your team skills and goals.

Databricks and Azure Machine Learning

Both support machine learning. Databricks is strongest when machine learning sits close to large-scale data engineering. Azure Machine Learning is strongest as a dedicated model development and operations service. Some teams use both, for example preparing data in Databricks and managing certain deployments elsewhere.

flowchart TD
    Q1{"Main workload?"} -->|"Heavy data engineering and machine learning"| Q2{"Want governance across clouds and rich Spark tools?"}
    Q1 -->|"SQL warehouse and Power BI centred"| Q3{"Prefer simplest managed platform?"}
    Q1 -->|"Small relational application data"| SQLDB["Azure SQL Database"]
    Q2 -->|"Yes"| DBX["Azure Databricks"]
    Q2 -->|"No"| SYN["Consider Azure Synapse Analytics"]
    Q3 -->|"Yes"| FAB["Evaluate Microsoft Fabric"]
    Q3 -->|"No"| SYN2["Consider Synapse or Databricks SQL"]

Every design choice gives something and takes something. Be honest about both sides.

Flexibility versus simplicity

Databricks gives many knobs: compute types, runtimes, table layouts, and governance settings. That flexibility rewards skilled teams but can overwhelm newcomers. Use defaults, policies, and templates to keep things simple until you have a real reason to change them.

Control versus convenience

Classic compute gives you network control and visibility, but you manage more. Serverless is convenient and quick, but you accept Databricks managing the machines and may have less network customisation. Choose according to your security rules and your team’s capacity.

Openness versus integration

Open formats keep your data portable. Platform-specific features, such as certain performance and governance capabilities, give extra benefits but tie you more closely to the platform. Decide consciously which features are worth the tie.

Speed of change

The platform evolves quickly, bringing new capabilities but also renamed products and changed best practices. Plan time to read release notes, test runtime upgrades, and update your habits.

Proven shapes to copy, and common shapes to avoid.

Patterns that work

Anti-patterns

An anti-pattern is a common solution that looks sensible but causes trouble. Here are the usual ones.

A practical list you can print and keep next to your desk.

Best practices

• Start from business questions and design layers, tables, and jobs to answer them.
• Use Delta and Unity Catalog for reliable, governed tables from the start.
• Keep raw data unchanged and refine in clear steps.
• Match compute to work: interactive for exploring, job compute or serverless for production, warehouses for SQL.
• Control cost with policies, auto-termination, autoscaling, tags, and budgets.
• Process incrementally and make every task safe to re-run.
• Maintain tables: compact small files, keep layouts sensible, and vacuum with care.
• Secure in layers: Entra ID, groups, service principals, private networking, and catalogue rules.
• Keep secrets in a vault and give each identity only the rights it needs.
• Treat data work as software: Git, reviews, tests, and automated deployment.
• Monitor jobs, costs, and data quality with system tables and alerts.
• Keep runtimes reasonably current, testing upgrades before production.
• Check current documentation regularly, because names, features, and prices change.

Common mistakes

Four typical situations, described as illustrative examples, showing when the service fits.

The stories below are illustrative composites of the kinds of situations where teams commonly use Azure Databricks. They are not claims about any particular company, but they reflect the reasoning real architects use.

Short answers to the questions beginners ask most often.

Everything important, in one place.

Azure Databricks is a managed data and AI platform on Azure built around Apache Spark. A control plane run by Databricks manages the workspace, while a compute plane, either in your subscription or serverless, does the processing on data that stays in your own storage. Delta Lake makes files behave like reliable tables. Unity Catalog governs data, models, and access across workspaces. Clusters, SQL warehouses, and serverless options give compute that starts and stops on demand. Jobs and pipelines turn notebooks into dependable production processes. MLflow and related tools carry machine learning from experiment to monitored production. Costs follow how long compute runs and how much data is read, so job compute, autoscaling, auto-termination, and good table layout save money.

Remember the workshop we began with: a clear catalogue of every heap, a rulebook for who may touch what, powerful machines that switch on when needed and off when done, and benches where everyone works together. It runs well only if the materials are shelved in good order, the right machine is used for each task, the doors are guarded in layers, and someone watches the running costs. Treat Azure Databricks the same way: organise data in layers, govern it centrally, match compute to work, protect it well, and check the current documentation because the platform keeps changing.