Before meeting any product, understand the problem it solves. Once the problem is clear, every feature looks sensible instead of random.

Three troubles at once

Organisations struggle with data in three ways at the same time. The first trouble is volume: there is simply too much data to fit on one computer or to process in a reasonable time on one machine. The second is variety: the data arrives as tidy database tables, as messy text files, as logs, as images, as streams of events from sensors, and as spreadsheets that someone emailed. The third is scatter: the data lives in many systems owned by different teams, each with its own rules, names, and copies.

The old answers: warehouses and lakes

For decades, companies built a data warehouse: a big, carefully structured database used for reports. A warehouse is fast and trustworthy, because data is cleaned before it goes in. But it is expensive, rigid, and poor at handling messy or unstructured data such as images and logs. When data volumes exploded, companies built data lakes: huge, cheap storage areas where any file can be dropped in its raw form. A lake is flexible and inexpensive, but it can turn into a data swamp, where nobody knows what is there, files are half written, two people overwrite each other’s work, and reports silently give wrong numbers.

Many companies ended up running both: a lake for data scientists and a warehouse for analysts, with brittle pipelines copying data between them. Each copy added cost, delay, and a chance for the two to disagree.

The lakehouse idea

A lakehouse is an approach that tries to give you the best of both. It keeps data in cheap, open storage like a lake, but adds a thin layer of management on top, which brings the reliability, structure, and speed of a warehouse. With a lakehouse, the same data can serve reports, data engineering, and machine learning, without constant copying. Azure Databricks is one of the best-known platforms built around this idea, and its key ingredients are Apache Spark for processing, Delta Lake for reliable storage tables, and Unity Catalog for governance.

Who uses it

An illustrative example

Take a hypothetical streaming-video company with 20 million subscribers. Every play, pause, and search creates an event, about two billion events per day. The company wants nightly reports of watched hours, hourly recommendations, and a model that predicts which subscribers may cancel. With Azure Databricks, event files land in cloud storage, a pipeline cleans them into reliable tables, analysts query the tables with SQL, and data scientists use the same tables to train the cancellation model. Everyone sees the same numbers, and no one maintains a separate copy.

A handful of simple ideas about data and computers make everything else easy. None are difficult, and each has an everyday comparison.

Distributed computing: many hands make light work

One computer has a limited number of processor cores, a limited amount of memory, and a limited disk. Distributed computing means splitting a big job into many small pieces and giving the pieces to many computers that work at the same time. A group of such computers working together is called a cluster, and each computer in it is a node. The challenge is coordination: someone must split the work, hand it out, collect the results, and cope when a computer breaks halfway through.

Apache Spark in one paragraph

Apache Spark is an open-source engine for distributed data processing. You describe what you want to do with the data, such as filtering rows, joining two tables, or adding up amounts by country, and Spark decides how to split the work across the machines in a cluster. It keeps data in memory where possible, which makes it much faster than older systems that wrote everything to disk between steps. You can use it from Python, SQL, Scala, Java, or R. Databricks was founded by the original creators of Spark, and Spark is the heart of the platform.

Files, formats, and columns

Data on disk lives in files, and the file format matters enormously. A CSV file is plain text with values separated by commas. It is easy to read for humans but slow and bulky for computers, and it has no memory of data types. A Parquet file stores data in a columnar layout: all values of one column are stored together, compressed. If a query needs only three columns out of a hundred, the engine reads only those three, which saves huge amounts of time. Parquet files also record the column types. Delta Lake, which we meet in chapter 7, builds on Parquet.

Cloud object storage

Azure keeps files in object storage, a service where each file is stored as an object with a name, scaled to almost any size, and billed cheaply by the amount stored. For analytics, the relevant flavour is Azure Data Lake Storage Gen2 (ADLS Gen2), which adds a real folder structure and fine-grained access control to Azure Blob Storage. In a lakehouse, the data itself lives in ADLS Gen2, separate from the computers that process it. This separation is powerful: you can switch off the computers when they are not needed, and the data stays safe and cheap in storage.

Batch and streaming

Batch processing handles data in large chunks on a schedule, such as every night. Streaming handles data continuously, a little at a time, as it arrives. Batch suits reports that are fine being a few hours old. Streaming suits fraud alerts or live dashboards. Spark and Databricks support both, and Databricks encourages treating streaming as simply “batch that never ends,” so that one logic works for both.

ETL and ELT

ETL stands for extract, transform, load: pull data from a source, reshape it, and then load it into its destination. ELT flips the last two steps: load the raw data first, then transform it inside the platform. Lakehouses favour ELT, because storage is cheap and keeping the raw copy lets you redo transformations later when rules change. The word pipeline describes any chain of such steps.

ACID: what makes data trustworthy

ACID stands for four promises that reliable databases make. Atomicity: a change either happens completely or not at all. Consistency: data always obeys its rules. Isolation: people working at the same time do not see each other’s half-finished work. Durability: once a change is confirmed, it survives crashes. Plain files in a data lake do not offer these promises, which is why swamps happen. Delta Lake adds them back.

Notebooks

A notebook is an interactive document made of cells. Each cell holds a piece of code, such as a SQL query or a few lines of Python, and shows its results, tables, or charts right beneath it. Notebooks mix code, results, and written explanation in one place, so they are ideal for exploring data and sharing findings. Databricks notebooks can combine languages and can be run on a cluster with a single click, or scheduled to run as part of a pipeline.

With the basics in place, we can define the service, see its main parts, and understand how it relates to Azure and to the wider Databricks company.

The short definition

Azure Databricks is a managed, cloud-based data and AI platform, built on Apache Spark and the lakehouse approach, offered as a first-party service in Microsoft Azure. “Managed” means that Databricks and Microsoft run the complicated machinery, such as the software that creates clusters, installs Spark, schedules jobs, and keeps the platform updated, so your team can concentrate on data and not on infrastructure. “First-party” means it is sold, supported, and integrated by Microsoft as part of Azure, with Azure sign-in, Azure billing, and Azure networking.

The main parts of the platform

What makes it different from plain Spark

You could install open-source Spark yourself on a group of virtual machines. People do. But then you must build and maintain the cluster software, scaling, security, notebooks, job scheduling, monitoring, and access control. Azure Databricks provides all of that ready-made, with sensible defaults, an optimised version of Spark, and tight links to Azure. It also adds proprietary performance features and the governance layer. The trade-off is that you rely on a vendor platform and pay for it, which chapter 14 covers.

How it fits in Azure

Azure Databricks lives alongside other Azure data services. Data usually rests in Azure Data Lake Storage Gen2. People sign in with Microsoft Entra ID. Secrets such as passwords are kept in Azure Key Vault. Streams of events often arrive through Azure Event Hubs. Pipelines can be triggered from Azure Data Factory. Results are shown in Power BI. Source code lives in Git repositories such as GitHub or Azure DevOps. Networking uses Azure virtual networks, private endpoints, and firewalls. This tight fit is the main reason many Azure organisations choose it.

Azure Databricks is built from a few clearly separated parts. Understanding who owns which part explains security, networking, and cost.

flowchart LR
  USER["Users and tools"] --> CP["Control plane: web app, jobs scheduler, notebooks, cluster manager"]
  CP --> CMP["Compute plane in your Azure subscription: clusters of virtual machines"]
  CMP --> ADLS["Azure Data Lake Storage Gen2"]
  CMP --> SRC["Other sources: databases, Event Hubs, APIs"]
  CP --> UC["Unity Catalog metastore: permissions and metadata"]
  UC -.-> ADLS
  KV["Azure Key Vault"] -.-> CMP
  ENTRA["Microsoft Entra ID"] -.-> CP

The control plane: the brain

The control plane is the part that Databricks runs for you. It contains the web application you log into, the notebook editor, the job scheduler, the cluster manager that decides when to start and stop machines, and the metadata about your workspace. Your notebook code and workspace settings are stored here. The control plane does not hold your large datasets, and it does not do the heavy processing of your data.

The compute plane: the muscles

The compute plane is where your data is actually processed. In the classic model, the compute plane lives inside your own Azure subscription. When you start a cluster, the control plane asks Azure to create virtual machines in your subscription, installs the Databricks software on them, and sends them work. Your data is read by these machines directly from your storage. Because the machines are in your subscription, they follow your network rules and your Azure billing.

Newer serverless compute options place the machines in an environment run by Databricks, which is shared and managed, and which starts in seconds without you managing cluster settings. The data still lives in your storage, but the processing happens in the vendor-managed area. Serverless trades some control for convenience and fast start, and its availability and features vary by region and by product, so check current documentation.

The workspace

A workspace is a single Databricks environment: a web address where a team logs in to find its notebooks, clusters, jobs, and settings. An organisation often creates several workspaces, such as one for development, one for testing, and one for production, or one per business unit. With Unity Catalog, many workspaces can share the same governed data without copying it.

The managed resource group

When you create a workspace in Azure, Azure also creates a special managed resource group. It holds resources that the platform needs, such as a storage account for workspace system data, networking pieces, and the virtual machines of clusters. This group is managed by the service: you can look at it, but you should not change things inside it, because that can break the workspace. A deny assignment protects it from most edits.

Storage: where the data lives

Your lakehouse data lives in Azure Data Lake Storage Gen2 accounts that you own. Tables managed by Unity Catalog are stored in locations you configure, and permissions to reach them are given through an access connector, a managed identity that Unity Catalog uses, so that people do not need direct storage keys. Remember the key principle: compute and storage are separate. You can resize or switch off the compute without touching the data, and different clusters or warehouses can read the same tables at the same time.

Networking in brief

By default the compute plane’s machines are placed in a network created for you, with each machine given only private addresses and no direct exposure to the internet. For stricter needs, you can inject the compute into your own virtual network, which lets you control routing, firewalls, and connections to on-premises systems. Chapter 13 covers this in detail along with private connectivity for users.

Spark can seem like magic, but its working is quite logical. Following one job from your code to its result explains most of the performance advice later in the guide.

flowchart TD
  CODE["Your code: read, filter, join, group"] --> DRV["Driver builds a plan"]
  DRV --> OPT["Optimizer chooses an efficient plan"]
  OPT --> DAG["Plan split into stages at shuffle points"]
  DAG --> T1["Stage 1: many parallel tasks on partitions"]
  T1 --> SH["Shuffle: data exchanged between executors"]
  SH --> T2["Stage 2: many parallel tasks"]
  T2 --> RES["Result written to a table or returned to the driver"]
  EX1["Executor 1"] --- T1
  EX2["Executor 2"] --- T1
  EX3["Executor 3"] --- T2

Driver and executors

A Spark cluster has one driver and many executors. The driver is the coordinator: it runs your program, builds a plan of the work, splits the plan into pieces, and sends the pieces to the executors. The executors are the workers: each one is a process on a cluster machine that runs the pieces, holds some data in memory, and reports the results back. The more executors and cores you have, the more pieces can run at once.

Partitions: slicing the data

Spark divides a large dataset into partitions, each a slice of the rows small enough for one task to process. A task is the smallest unit of work: one operation on one partition. A one-terabyte dataset in slices of about 128 megabytes has roughly 7,800 partitions. If your cluster has 64 cores, 64 tasks run at the same moment, so the whole stage needs about 122 rounds, which are sometimes called waves. More cores shorten the job, until another bottleneck takes over.

Lazy evaluation: plan first, work later

Spark operations come in two kinds. Transformations, such as filtering rows or joining tables, only describe what you want and do not run anything. Actions, such as writing a result or counting rows, trigger the real work. This is called lazy evaluation. Because Spark waits until it sees the whole plan, it can reorder and combine steps, for example by pushing a filter before a join so that fewer rows are joined. The part of Spark that does this clever rewriting is the Catalyst optimizer.

Jobs, stages, and tasks

When an action fires, Spark creates a job. It divides the job into stages, and each stage into tasks. A stage is a group of steps that can run on each partition independently, without needing data from other partitions. The boundary between stages is a shuffle.

Shuffles: the expensive step

Some operations need rows with the same key to end up together. Adding up sales per country needs all the rows for France in one place. Joining two big tables on customer number needs matching customers on the same machine. To achieve this, Spark must shuffle: executors write data to disk, send pieces across the network, and read the pieces they need. Shuffles are the slowest and most resource-hungry part of most jobs, so much performance tuning is about avoiding or shrinking them.

Fault tolerance

Machines fail. Spark records how each partition was produced, as a lineage of steps, so if an executor dies, the lost partitions can be rebuilt by repeating the steps on the original data. This means big jobs survive the loss of a machine without starting over, and it is part of why Spark is trusted for long-running work. Chapter 14 shows how to reduce the pain of failures further.

Adaptive query execution

Modern Spark includes adaptive query execution (AQE). It watches the actual sizes of data at shuffle points while the job runs, and then adjusts the plan: it can merge many tiny partitions into fewer ones, switch to a faster join strategy when one side turns out to be small, and split out very large partitions that are causing skew. AQE is on by default in modern Databricks runtimes and rescues many jobs without any manual tuning.

Compute is where most of the money goes and where most beginners make costly mistakes. This chapter explains your options and how to choose.

What a cluster is

A cluster in Databricks is a set of virtual machines acting as one Spark engine: one driver machine and zero or more worker machines. You choose the machine size, the number of workers, the Databricks Runtime version, and options such as autoscaling. Starting a classic cluster means asking Azure for virtual machines, so it usually takes several minutes to become ready.

All-purpose and job compute

All-purpose clusters are for interactive work: people attach notebooks, run cells, and explore. They stay up until stopped or until an idle timer ends them, and they are billed at a higher rate. Job clusters, in contrast, are created automatically to run a scheduled job and are destroyed when it finishes. They are billed at a lower rate and start with a clean state each time, which makes them more reliable and cheaper for production. A golden rule is to develop on all-purpose compute and run production on job compute.

Runtimes

The Databricks Runtime is the software image installed on each machine. It bundles Spark, Delta Lake, many popular libraries, and Databricks optimisations. There are versions with long-term support, which are good for production because they stay stable, and newer versions with fresh features. A machine learning flavour adds popular machine learning libraries and graphics processor support. Pinning a specific runtime version in production prevents surprises from silent upgrades.

Autoscaling and auto-termination

Autoscaling lets a cluster grow when work piles up and shrink when it quietens, between a minimum and maximum number of workers you choose. Auto-termination shuts down an idle all-purpose cluster after a set number of minutes. Together they stop you paying for machines nobody is using. A cluster left running over a weekend is among the most common causes of an unexpectedly high bill.

Pools: warm machines on standby

Starting a virtual machine takes time. An instance pool keeps a number of idle machines ready, so new clusters grab them almost instantly. This cuts start times, which matters for jobs that run every few minutes. Idle pool machines still cost the Azure machine price, though not the Databricks software fee, so size pools sensibly.

SQL warehouses

For analysts who write SQL and use dashboards, Databricks offers SQL warehouses: compute tuned for SQL queries and business intelligence tools. They come in a few types, with a serverless option that starts in seconds and scales with the number of concurrent queries. Warehouses use the Photon engine. Analysts choose a size, like a T-shirt size, and the platform handles the rest.

Serverless compute

With serverless options, you do not configure machines at all. You submit a notebook, job, or query, and the platform supplies compute almost instantly, scales it automatically, and bills for what you use. It removes cluster tuning and idle waste, and it is excellent for bursty or occasional work. The costs and limits differ from classic compute, so compare them for steady, heavy workloads.

Choosing machine types

• General purpose machines: a balanced mix of processor and memory, a safe default.
• Memory optimised: more memory per core, good for large joins, caching, and wide aggregations.
• Compute optimised: more cores for heavy processing with small memory needs.
• Storage optimised: fast local disks, helpful for workloads that spill or cache a lot.
• GPU machines: graphics processors for deep learning and some model training.

Cluster policies

Left alone, anyone could create an enormous cluster. Cluster policies are rules set by administrators that limit what users can choose: allowed machine sizes, maximum workers, mandatory auto-termination, required tags for cost tracking, and fixed runtime versions. They give people freedom within safe limits, and are the cheapest cost-control tool you have.

Delta Lake is the quiet hero of the lakehouse. It turns piles of files into real tables with promises you can trust.

The problem with plain files

Imagine two jobs writing into the same folder of Parquet files at the same time. One job crashes halfway, leaving a mess of half-written files. A report reads the folder in the middle of an update and sees some old rows and some new ones. Someone wants to undo yesterday’s wrong update, but the old files were overwritten. These are the everyday hazards of plain data lake files, and they are exactly what the ACID promises from chapter 2 prevent.

What Delta Lake is

Delta Lake is an open-source table format. A Delta table is a folder that contains ordinary Parquet data files plus a special sub-folder named _delta_log, called the transaction log. The log is an ordered record of every change ever made to the table: which files were added, which were removed, and what the table’s columns are. To read a table, the engine reads the log to learn which files make up the current version, and then reads only those files. Because the log is the single source of truth, readers never see half-finished changes.

sequenceDiagram
  participant W as Writer job
  participant S as Storage files
  participant L as Transaction log
  participant R as Reader query
  W->>S: Write new Parquet data files
  W->>L: Try to commit entry listing the new files
  L-->>W: Commit accepted as the next version
  R->>L: Read latest log to find current files
  L-->>R: List of files for this version
  R->>S: Read only those files

How a commit works

A writer first writes new data files to storage. Those files are invisible to readers at this point, because the log does not mention them. Then the writer tries to add a new entry to the log listing the files it added or removed. If the entry is accepted, the whole change becomes visible at once, as the next version of the table. If the writer crashes before the commit, the stray files are never referenced, and the table is unchanged. This is atomicity.

When two writers try to commit at the same time, Delta uses optimistic concurrency control. Each writer assumes no conflict, does its work, and checks at commit time whether another writer changed something it depended on. If there is no real conflict, both succeed. If there is, one retries or fails with a clear message. This is cheaper than locking the whole table.

Time travel

Because old files are not immediately deleted and the log remembers every version, you can query the table as it was at an earlier version number or an earlier time. This is called time travel. It lets you audit changes, reproduce a report exactly as it was last month, compare before and after, and undo a mistake by restoring an older version. The history is limited by retention settings: the log keeps entries for a set period, and the old data files stay until a clean-up command removes them.

Schema enforcement and evolution

A Delta table has a defined schema, the list of columns and their types. Schema enforcement means a write that does not match, such as text in a number column or a surprise extra column, is rejected instead of silently corrupting the table. When a change is intended, schema evolution lets the table accept new columns in a controlled way. This stops the common data lake horror where a column quietly changes meaning, and downstream reports break weeks later.

Upserts and deletes with MERGE

Plain files are hard to update, since you cannot edit a row inside a Parquet file. Delta supports UPDATE, DELETE, and a powerful MERGE command that compares incoming records with the table and, in a single atomic operation, inserts new ones, updates changed ones, and optionally deletes removed ones. This is the standard way to apply change data from a source system, and it makes privacy requests such as “delete this customer” practical.

Keeping tables healthy: OPTIMIZE and VACUUM

Frequent small writes create many tiny files, which slows reading. The OPTIMIZE command compacts small files into larger ones, commonly aiming for sizes in the hundreds of megabytes up to about a gigabyte. It can also reorder data so that rows with similar values sit together, using Z-ordering or the newer liquid clustering, which helps queries skip files they do not need. The VACUUM command permanently deletes data files no longer referenced by recent versions and older than a retention period, with a default of seven days, freeing storage but limiting how far back time travel can go. Newer managed options can run these chores automatically for governed tables, so check what your platform version offers.

Managed and external tables

With Unity Catalog, a managed table lets the platform decide where files are kept and also take care of file lifecycle, so that dropping the table removes its data after a grace period. An external table points to files in a location you control, and dropping the table leaves the files in place. Choose managed tables for most new work because they get the most automatic care, and external tables when other systems must own or share the same files.

Having reliable tables is not enough. You also need a sensible way to organise them and move data through them. The medallion approach is the most common answer.

flowchart LR
  S1["Source files and databases"] --> ING["Ingestion: Auto Loader and connectors"]
  S2["Event streams"] --> ING
  ING --> BR["Bronze: raw data as received"]
  BR --> SI["Silver: cleaned, deduplicated, joined"]
  SI --> GO["Gold: business-ready tables and metrics"]
  GO --> BI["Dashboards and reports"]
  GO --> ML["Machine learning features"]
  SI --> ML

The three layers

The medallion architecture organises data into three quality levels. Bronze tables hold data exactly as it arrived from the source, with little or no change, usually append-only and with extra columns recording when and from where it came. Bronze is your safety net: if a later rule was wrong, you can rebuild from it. Silver tables hold cleaned and conformed data: duplicates removed, bad records handled, data types fixed, and related sources joined. Gold tables hold business-ready data: aggregates, metrics, and models shaped for particular uses, such as daily revenue by region.

Why layers help

• Trust: users know that gold tables passed quality checks, while bronze is raw.
• Reprocessing: when a rule changes, you rebuild silver and gold from bronze without going back to the sources.
• Separation of duties: engineers own bronze and silver, analysts and product teams consume gold.
• Performance: gold tables are small and tuned for queries, so dashboards are quick.
• Debugging: if a number looks wrong, you can walk back through the layers to see where it changed.

Ingestion: getting data in

Data arrives in many ways. Files are dropped into storage by other systems. Databases are read through connectors or captured through change feeds. Events stream in through services such as Azure Event Hubs or Apache Kafka. Software-as-a-service tools offer managed connectors. Whatever the source, the aim is to land it in bronze reliably, once, and without losing anything.

Auto Loader: finding new files

A very common task is to process new files as they land in a storage folder, without reprocessing old ones. Auto Loader is a Databricks feature that does exactly this. It watches a folder, either by listing it efficiently or by listening to storage notifications, remembers which files it has already processed using a checkpoint, and ingests each new file once. It can detect the schema of incoming files and adapt when new columns appear, putting unexpected data into a rescue column instead of dropping it. For folders with millions of files, it is far more scalable than naive approaches.

Streaming and incremental processing

Structured Streaming is Spark’s way of processing data continuously. You write the logic as if the data were a normal table, and Spark applies it to each new batch of arriving records. A checkpoint records progress so that after a restart the stream resumes where it stopped, with no duplicates and no gaps. You can run a stream all the time, or run it on a trigger that processes everything that has arrived and then stops, which is a cost-effective way to get incremental updates every few minutes or hours without a cluster running all day.

Declarative pipelines and data quality

Writing and wiring many steps by hand is laborious. Declarative pipelines, known in Databricks as Delta Live Tables and now evolving into newer pipeline products, let you describe the tables you want and how each is derived, and the platform works out the order, handles dependencies, manages checkpoints, retries failures, and scales the compute. You can add expectations, which are data quality rules such as “the order amount must be positive” or “the customer number must not be empty,” and choose what happens to rows that break them: record a warning, drop the row, or fail the pipeline. This builds data quality into the flow itself.

Handling changes from source systems

Operational databases change rows over time. To mirror them, teams capture change data: a feed of inserts, updates, and deletes. Applying the feed to a silver table with MERGE, or with pipeline features designed for change data capture, keeps the lakehouse in step with the source. Remember to handle out-of-order and duplicate events, which are normal in distributed systems, by using sequence numbers or timestamps to decide which change wins.

Design guidance

• Keep bronze append-only and as close to the source as possible.
• Apply data quality rules at the silver step, and record rejected rows somewhere you can inspect them.
• Build gold tables around business questions, not around source system shapes.
• Make every step idempotent, meaning that running it twice gives the same result as running it once, so retries are safe.
• Use consistent naming and put each layer in its own schema or catalog.
• Partition or cluster large tables thoughtfully, based on how queries filter, not by habit.

When many people share data, the question changes from “can we store it?” to “who may see what, and how do we find and trust it?” Unity Catalog is the answer inside Databricks.

flowchart TD
  MS["Metastore: one per region in an account"] --> C1["Catalog: prod"]
  MS --> C2["Catalog: dev"]
  C1 --> S1["Schema: sales"]
  C1 --> S2["Schema: finance"]
  S1 --> T1["Table: orders"]
  S1 --> T2["View: daily_revenue"]
  S1 --> V1["Volume: raw_files"]
  S2 --> T3["Table: invoices"]
  EXT["External location with storage credential"] -.-> MS

The three-level name

Unity Catalog organises data in a hierarchy. At the top is the metastore, the container for all metadata in a region of your account. Inside it are catalogs, which usually represent an environment or a business unit, such as prod and dev. Inside a catalog are schemas (also called databases), which group related objects, such as sales or finance. Inside a schema are the objects themselves: tables, views, volumes for files that are not tables, functions, and machine learning models. Every object has a three-part name: catalog, schema, then object, such as prod.sales.orders.

Access control: who may do what

Permissions in Unity Catalog are given as privileges on objects, granted to users, groups, or service principals. The common ones are permission to use a catalog or schema (so you can see inside), to select from a table (read), to modify it (write), and to create new objects. Privileges inherit downward: granting read on a schema covers all tables in it, present and future. The best practice is to grant to groups, managed in Microsoft Entra ID, never to individuals, so that people gain and lose access simply by joining or leaving a group.

Fine-grained control

Sometimes table-level control is too coarse. Row filters hide rows a user should not see, for example showing each sales manager only their own region. Column masks hide or scramble sensitive columns, for example showing only the last four digits of a card number to most users. Views can expose a safe subset of a table. These controls are defined once, in the catalog, and apply to every tool that queries the data, so you do not rebuild the same rule in each report.

Storage access without keys

Unity Catalog never asks users to hold storage keys. A storage credential wraps an Azure managed identity, created through an access connector, that has permission on a storage account. An external location ties a storage path to a credential. Users are then granted rights on the external location or on tables built there, and Unity Catalog performs the actual file access on their behalf. This keeps secrets out of notebooks and gives one place to audit file access.

Discovery, lineage, and audit

• Discovery: users can search for tables, read descriptions, and see tags such as “contains personal data” or “certified.”
• Lineage: the platform automatically records which tables and columns feed which others, and which notebooks and jobs touched them. If a number looks wrong, you can trace it back to its sources. If a source changes, you can see everything affected.
• Audit: access and activity are logged and queryable through system tables, so you can answer “who read this table last month?”
• Sharing: Delta Sharing lets you share live tables with other organisations or platforms without copying data.

Why not just use storage permissions?

You could control access with file-level permissions on the storage account, but that works only at folder and file level, and different tools may bypass it. Unity Catalog works at the level of tables, columns, and rows, understands business groupings, and applies to every query engine that goes through it. For anything beyond a small team, central governance saves enormous time and prevents mistakes.

Notebooks are great for exploring. Production needs schedules, order, retries, alerts, and tested code that moves safely from development to production.

Jobs and tasks

A job is a scheduled or triggered unit of work in Databricks. A job contains one or more tasks, and tasks can be notebooks, Python scripts, packaged code, SQL queries, pipelines, or other job types. Tasks can depend on each other, forming a graph: ingest first, then clean, then build gold tables, then refresh a dashboard. The platform runs each task when its dependencies succeed, runs independent tasks in parallel, and lets you re-run only the failed part, a feature often called a repair run.

Triggers

• Schedules: run at set times, such as every night at two o’clock.
• File arrival: run when new files land in a storage location.
• Continuous: keep a job running, restarting it if it fails, for streaming work.
• Manual or API: start by a person, or from another system such as Azure Data Factory or a pipeline tool.
• Table updates: some designs trigger work when upstream tables change.

Reliability features

Jobs can retry failed tasks automatically, with a limit and a delay, which handles temporary problems such as a brief network error. They can enforce time limits, so that a hung task does not run for days and cost money. They send notifications by email, chat, or other channels when a run fails, succeeds late, or exceeds expected duration. Parameters let the same job work for different dates or environments. For important pipelines, also add data quality checks that fail the run when output looks wrong, because a job that succeeds technically but produces nonsense is the worst outcome.

Working alongside Azure Data Factory

Many Azure organisations already use Azure Data Factory to orchestrate movement of data across many systems. Data Factory can call Databricks notebooks and jobs as steps in a wider pipeline, passing parameters and waiting for results. A common split is to let Data Factory handle copying data in from varied sources and triggering, while Databricks handles heavy transformation. Alternatively, Databricks workflows can run the whole thing alone. Choose by what your team already knows and where you want one place to monitor.

Version control and Git folders

Code should live in a version control system such as GitHub or Azure DevOps. Databricks Git folders connect a workspace folder to a repository, so people can commit, branch, and pull from the notebook interface. This gives history, code review, and the ability to roll back. Never treat the workspace itself as the only copy of your code.

Environments and promotion

A healthy setup has separate environments: development, test, and production, typically in separate workspaces and catalogs. Code is developed and reviewed in development, verified automatically in test using sample data, and promoted to production only by an automated process. Production jobs should run as a service principal, an application identity, not as a person’s account, so that a staff change does not break them and so that permissions can be exact.

Infrastructure and deployment as code

Workspaces, clusters policies, jobs, and permissions can be defined in code. The Databricks Terraform provider manages platform resources. Databricks Asset Bundles package notebooks, code, job definitions, and configuration for each environment in a set of text files that a pipeline can deploy. Combined with Azure DevOps or GitHub Actions, this gives repeatable deployments with review and an audit trail.

Testing data code

• Unit tests for functions that transform data, run locally or on a small cluster using a test framework.
• Integration tests that run a whole pipeline on a small, known dataset and compare the results with expected values.
• Data quality tests that run on every production load, checking counts, nulls, ranges, and uniqueness.
• Performance checks that flag a job whose run time suddenly doubles.

The same governed tables can serve analysts who love SQL and scientists who love Python. This chapter shows how both groups work on one platform.

Databricks SQL for analysts

Analysts can use a SQL editor in the browser to write queries against lakehouse tables, save them, schedule them, and turn the results into charts and dashboards. Queries run on a SQL warehouse, which scales with the number of people querying at once. Because the data stays in Delta tables, analysts see the same numbers as engineers, without exporting to a separate warehouse. Alerts can notify a team when a metric crosses a threshold. Natural-language assistants that translate questions into queries are also appearing in the platform; treat their results with the same care as any analyst’s work, and check them against known figures.

Power BI and other tools

Many companies use Power BI for business reporting. Power BI connects to a SQL warehouse, either by importing a copy of the data on a schedule, or by querying the warehouse live each time a user interacts with a report. Importing is fast for users but can go stale, while live querying is fresh but depends on warehouse speed. Standard connectors also exist for Tableau, Excel, and other tools. Good practice is to point reports at gold tables, set row-level security in Unity Catalog or in the reporting tool, and avoid giving report users direct access to raw bronze data.

Machine learning on the same data

Data scientists work in notebooks, read the same governed tables, and use popular libraries for statistics, machine learning, and deep learning. They can scale training across clusters, use GPU machines for large models, and track their work in MLflow.

Moving from experiment to production

A model is only valuable once it is used. The path usually runs: explore data, build features, train and compare models, register the best model, test it against fresh data, deploy it for batch or real-time use, and monitor it. Models decay as the world changes, a problem called drift: customers’ behaviour shifts, and predictions get worse. Track input data and prediction quality over time, and retrain on a schedule or when quality drops. Treat models as governed assets, with owners and documentation.

Responsible use

• Keep training data governed, so that restricted or personal data is not used without permission.
• Record which data and code produced each model, using lineage and tracking.
• Test for unfair outcomes across groups, especially for decisions that affect people.
• Keep a human in the loop for high-stakes decisions.
• Do not send sensitive data to external services without approval and agreed safeguards.

Most slow Spark jobs are slow for a small number of well-known reasons. Learn them and you can fix the majority of performance problems without guesswork.

Start by finding the bottleneck

Never tune blindly. Open the Spark UI, which shows the stages and tasks of a job, how long each took, how much data moved, and whether any tasks ran far longer than the rest. Ask three questions. Is the time spent reading data, shuffling it, or writing it? Are a few tasks much slower than the others? Is the cluster busy or mostly idle? The answers point to the cure. Improving a stage that takes two percent of the run time is wasted effort.

The five usual suspects

Read less: pruning and skipping

Delta tables store simple statistics about each file, such as the minimum and maximum of each column. When a query filters on a column, the engine can skip files whose range cannot contain matching rows. This works best when similar values are stored together, which is what Z-ordering and liquid clustering arrange. Partitioning a table by a column, such as the date, puts each value in its own folder, so a query for one day reads only one folder. But over-partitioning is a trap: partitioning by a column with millions of distinct values creates millions of tiny folders and files. A common guideline is that each partition should hold at least about a gigabyte of data, so low-cardinality columns like date are suitable and high-cardinality ones like customer number are not.

File sizes

The sweet spot for data files is big enough to amortise overhead but small enough to parallelise, in the range of roughly 128 megabytes to about a gigabyte. Streaming jobs and frequent small appends produce tiny files, so schedule regular compaction or enable automatic optimisation features for your tables. After a big delete or update, compaction also restores read speed.

Joins and shuffles

When joining a huge table to a small one, Spark can broadcast the small table to every executor, so the big table does not need to be shuffled at all. By default, tables smaller than about ten megabytes are broadcast automatically, and AQE can choose this at run time when it learns a side is small. For two huge tables, Spark uses a sort-merge join, which shuffles both sides. To make it cheaper, filter and trim both sides first, select only needed columns, and consider storing both tables clustered by the join key.

Dealing with skew

Suppose 40 percent of all orders belong to one placeholder customer called “unknown.” When Spark groups by customer, the task handling that key must process 40 percent of the data while the other tasks finish instantly. The job is as slow as that one task. AQE can split skewed partitions in many joins. Other remedies include removing or separating the hot key, adding a random salt to spread it over many tasks and then combining the results, or aggregating partly before the shuffle.

Shuffle partitions and memory

The number of partitions after a shuffle is controlled by a setting whose default is 200. Too few partitions means each task handles too much data and may spill to disk or run out of memory. Too many means tiny tasks with high overhead. Modern runtimes with AQE adjust this automatically, which is the first thing to rely on. When tasks spill, meaning they run out of memory and write temporary data to disk, you will see it in the Spark UI, and the cures are more memory per core, more partitions, or less data per task.

Caching

If you reuse the same data many times in one notebook, caching it in memory can help. Databricks also offers a disk cache that keeps copies of recently read remote files on the fast local disks of the machines, which speeds up repeated reads from cloud storage without you doing anything on the supported machine types. Do not cache by habit, since caching large data steals memory from the actual work.

Photon and runtime choices

Photon is a vectorised engine written in a low-level language that accelerates many SQL and DataFrame operations, especially large scans, joins, and aggregations. It costs a higher DBU rate per hour but often finishes jobs so much faster that the total cost falls. Measure with your own workload before deciding. Keep the runtime reasonably current, since each release brings performance improvements.

Code habits that help

• Prefer built-in functions over custom row-by-row functions, which are slow because data must travel between the engine and the language process.
• If you need custom logic in Python, use vectorised (pandas) functions that handle batches.
• Avoid collecting large results to the driver, and avoid repeated actions on the same unpersisted plan.
• Write results in a few large files, using appropriate repartitioning before the write.
• Use incremental processing, handling only new data, instead of recomputing history every day.

A data platform holds a company’s most valuable and most sensitive information. Security is built in layers, so that no single mistake exposes everything.

flowchart LR
  USER["Users with Entra ID sign-in"] --> FE["Front-end private endpoint or IP access list"]
  FE --> WS["Databricks workspace control plane"]
  WS --> BE["Back-end private connectivity"]
  BE --> CMP["Compute in your virtual network, no public IPs"]
  CMP --> PE["Private endpoints"]
  PE --> ADLS["Data Lake Storage"]
  PE --> KV["Key Vault"]
  CMP --> FW["Firewall or NAT gateway for controlled outbound traffic"]

Identity: who are you?

Users sign in with Microsoft Entra ID, so the company’s existing accounts, multifactor authentication, and conditional access rules apply. Accounts and groups are synchronised into Databricks, commonly through automatic provisioning, so that joining or leaving a team in Entra ID changes access without manual work. For automation, use service principals, which are application identities, rather than personal accounts. Give each pipeline or application its own service principal with only the permissions it needs.

Authorization: what may you do?

Three levels of permission matter. Workspace permissions control who can use notebooks, create clusters, or run jobs. Unity Catalog privileges, covered in chapter 9, control who can read or change data. Cloud permissions on the storage account and other Azure resources control the underlying infrastructure. A principle runs through all three: least privilege, giving each person and process only what the job requires. Assign permissions to groups, review them regularly, and remove access promptly.

Cluster access modes

Unity Catalog enforces governance only when compute is set up in a mode that supports it. A standard (shared) mode isolates users from each other on one cluster, and a dedicated (single-user) mode assigns a cluster to one user or group. Using an unrestricted legacy mode could let code bypass fine-grained rules, so use cluster policies to require governed modes for everyone.

Secrets

Passwords and keys must never be written in notebooks or code. Store them in Azure Key Vault and read them through a secret scope backed by Key Vault, so code refers to a secret by name and the value is never shown in output. Better still, avoid secrets completely by using managed identities and Unity Catalog’s storage credentials wherever possible.

Network isolation

By default, the compute plane gets a network that Databricks sets up. For tighter control, use VNet injection to place the compute in your own Azure virtual network. This requires two subnets, often called the host (public) subnet and the container (private) subnet, delegated to Databricks. Each worker and driver machine uses one address in each subnet, so subnet sizes limit how big your clusters can be at the same time. Azure reserves a few addresses in every subnet, so allow generously, planning for the peak number of machines across all clusters and all concurrent job runs, plus growth.

Secure cluster connectivity, sometimes called no public IP, means cluster machines have only private addresses and open no inbound ports. They call out to the control plane through a secure relay instead of waiting for connections. This is the recommended setting for nearly all workspaces. Outbound access to the internet, for example to download packages, should pass through a controlled path such as a NAT gateway with a fixed address or a firewall that enforces an allow list.

Private Link

Azure Private Link lets traffic travel over private addresses on Microsoft’s network instead of the public internet. For Databricks it appears in two forms. Front-end Private Link lets users and tools reach the workspace through a private endpoint in your network, often used with corporate VPN or private circuits. Back-end Private Link lets the compute plane reach the control plane privately. Storage and Key Vault can also be reached through private endpoints. Together they let you run a platform with no part exposed to the public internet. Take care with DNS: private endpoints need matching private name resolution, or connections fail in confusing ways.

Encryption

• In transit: connections use TLS encryption by default, and additional encryption between cluster machines can be enabled for sensitive workloads.
• At rest: storage and disks are encrypted with Microsoft-managed keys by default. You can use customer-managed keys in Key Vault for the workspace’s managed storage, managed disks, and some other areas where your policy requires it.
• Column-level protection: masking, hashing, or encrypting specific fields, such as national identity numbers, adds protection if broad access is ever granted by mistake.

Auditing and compliance

Databricks records audit events such as logins, permission changes, cluster creation, and table access. These can be sent to Azure Monitor and queried through system tables. Combine them with alerts on suspicious patterns, such as a user suddenly reading very large amounts of data, or a new service principal getting broad permissions. If you work in a regulated industry, check which compliance programs and regions are supported by the platform and which workspace features, such as enhanced security monitoring, you must switch on.

Powerful platforms can generate surprising bills. Understanding the parts of the cost lets you control it without slowing your teams.

The parts of the bill

Azure Databricks charges are made of several pieces. First, Databricks Units (DBUs): a DBU is a unit of processing capability per hour, and each machine type consumes a certain number of DBUs per hour while it runs. You pay a rate per DBU, and the rate depends on the type of work, such as jobs, all-purpose, SQL, or serverless, and on the pricing tier. Second, the Azure virtual machines themselves, billed by Azure at the normal machine price for as long as they run, in the classic compute model. Third, storage for your data in Data Lake Storage. Fourth, networking and other services such as data transfer, NAT gateways, private endpoints, Key Vault, and monitoring logs. Serverless options bundle the machine cost into one rate.

A worked example

These figures are illustrative arithmetic only, with no prices. Suppose a nightly job runs on a job cluster of nine machines, one driver and eight workers, and each machine uses 2 DBUs per hour. That is 18 DBUs per hour. The job runs for 3 hours, so it consumes 54 DBUs per night, or about 1,620 DBUs in a 30-day month. Multiply by your DBU rate for jobs to get the Databricks charge. Add nine machines for three hours per night at your Azure machine price. If the job could be tuned to run in 90 minutes, both charges halve. Run time is the most powerful cost lever you have.

Levers that save money

Storage costs

Storage is cheap compared with compute, but it grows quietly. Old versions of Delta tables, kept for time travel, and old data files that nobody cleaned up with VACUUM, add up. Intermediate tables created for experiments and never deleted add up too. Use lifecycle policies and cleanup jobs, and decide a retention period that matches your need for history rather than keeping everything forever.

Visibility: knowing where the money goes

Use system tables that record billable usage by workspace, cluster, job, and tag, and build a dashboard of the top spenders. Set budgets and alerts in Azure Cost Management at subscription or resource group level. Make cost a regular agenda item for the team, because costs rise slowly when nobody watches.

Azure offers many services that touch data. Their names and roles overlap, and the lineup keeps evolving. This chapter gives a practical way to think about the choice.

flowchart TD
  A["What is the main job"] --> B{"Large scale data engineering, data science, or lakehouse"}
  B -->|"Yes"| C["Azure Databricks"]
  B -->|"No"| D{"Moving and orchestrating data between systems"}
  D -->|"Yes"| E["Azure Data Factory"]
  D -->|"No"| F{"Self-service analytics inside a Microsoft SaaS platform"}
  F -->|"Yes"| G["Microsoft Fabric"]
  F -->|"No"| H{"Operational app data or simple reporting"}
  H -->|"Yes"| I["Azure SQL or Cosmos DB"]
  H -->|"No"| J["Review the specialist services"]

The neighbours, in plain words

They often work together

These services are not strict rivals. A typical Azure data estate uses Data Factory to ingest from varied sources, Databricks to transform and model data, Power BI to present results, and operational databases as sources. Fabric and Databricks can also share data through open formats and storage shortcuts, so one team may use Databricks while another uses Fabric over the same Delta tables. Because capabilities and product positioning change quickly, review the current Microsoft documentation and your own team’s skills before committing to a design.

Questions to settle the choice

• How big is the data, and how complex is the processing?
• Which skills does the team have: SQL, Python, Spark, or visual tools?
• Do you need machine learning and streaming on the same data as reports?
• How strict are your governance, network, and compliance requirements?
• What is already in place that you must work with?
• How much operational effort can your team spend?

Every platform has bad days. The skill is in finding out quickly what happened, where, and why.

Where to look

Alerts worth setting

• A production job failed, was skipped, or ran far longer than usual.
• A table was not updated by its expected time.
• Data quality rules failed or the dropped-row count jumped.
• Daily spend rose well above the usual level.
• A cluster has been running for an unusual number of hours.
• Large reads, permission changes, or failed sign-ins appear in the audit trail.

Common failures and their usual causes

A calm method for investigating

A data platform is a living service. It must be designed, built, tested, run, watched, and improved, with changes made safely.

flowchart LR
  A["Design platform and governance"] --> B["Provision environments as code"]
  B --> C["Develop in dev workspace"]
  C --> D["Test with sample data"]
  D --> E["Deploy to production by pipeline"]
  E --> F["Run, monitor, and alert"]
  F --> G["Tune cost and performance"]
  G --> C
  F --> H["Govern, audit, retire old assets"]

Design the platform first

Before creating workspaces, write down the decisions that are hard to change: how many workspaces and environments, which regions, the network model and subnet sizes, the Unity Catalog metastore and catalog layout, the storage accounts and their access connectors, the Entra ID groups, the cost-tagging scheme, and the cluster policies. A short design document that is reviewed by security, networking, and data teams will save weeks of rework.

Provision with code

Build the platform with infrastructure as code, using Terraform or Bicep for Azure resources and the Databricks Terraform provider for workspace objects such as policies, groups, catalogs, grants, and jobs. This ensures that development, test, and production match, that changes are reviewed, and that you can rebuild after a mistake. Keep secrets out of the code, and store state files securely.

Develop, test, and promote

Developers work in a development workspace, using Git folders and a feature branch. Automated tests run on every change: unit tests on transformation logic, integration tests on a small known dataset, and data quality checks. After review and approval, a pipeline deploys the code and job definitions to test and then to production, where jobs run as service principals. Production should not be edited by hand: the only road to production is the pipeline, which leaves a clear audit trail and makes rollbacks simple.

Operate every day

• Watch dashboards for job success, data freshness, quality results, and cost.
• Respond to alerts using a run book that explains common failures and who to call.
• Keep runtimes up to date on a schedule, testing new versions in lower environments first.
• Maintain tables: compact files, clear old versions, and review partitioning and clustering as data grows.
• Review access quarterly: groups, service principals, external locations, and shared data.
• Retire what is no longer used: old jobs, tables, clusters, and workspaces, with a documented process.

Disaster recovery

Think about what happens if a region fails or someone deletes something. Data in storage can be replicated across zones or regions using storage redundancy options. Code in Git and infrastructure in code can be redeployed to a second workspace. Delta table history and soft-delete features protect against accidental deletion. Decide your recovery time and recovery point targets, build a plan to restore to another region if needed, and rehearse it. Remember that metadata, permissions, jobs, and secrets also need to be recreated, which is another reason to keep everything in code.

Change management

Schema changes in widely used tables can break dozens of downstream reports. Announce changes in advance, make backward-compatible changes where possible, such as adding columns instead of renaming, and use lineage to find who is affected. Version important data contracts, which are agreements between producers and consumers about the shape and meaning of shared data, so that changes are discussed before they land.

Teams that succeed with Azure Databricks tend to repeat a few good designs. Teams that struggle tend to repeat a few familiar mistakes. Knowing both saves months.

Patterns that work

Anti-patterns to avoid

Four decisions worth recording

Teams that write down their key decisions avoid arguing about them again a year later. These short records use the common format of context, decision, and consequences. They are illustrative examples, not mandates.

This chapter gathers the advice from the whole book into one checklist you can return to before launch, during reviews, and after incidents.

Best practices

Common mistakes

The following scenarios are invented illustrations built from common situations. They are not case studies of real companies, and the numbers are examples chosen to make the ideas concrete.

Short answers to the questions beginners ask most often.

You began with a problem: data that is too big, too messy, and too scattered. You now have the map for solving it with Azure Databricks.

Azure Databricks is a managed lakehouse platform. Its control plane manages the experience, while your compute plane in Azure does the heavy lifting on your own data in your own storage. Apache Spark spreads work across many machines, Photon speeds it up, and Delta Lake gives lake files the reliability of a database. Unity Catalog governs who can see what, Workflows and Git-based deployment keep production safe and repeatable, and SQL warehouses, Power BI, and MLflow carry the results to analysts and applications. Around all of this sit the disciplines that make a platform last: performance tuning guided by evidence, layered security, cost control, monitoring, and a regular operating rhythm.