Start with the big picture: what it does, what it does not do, and where it lives.

Azure Data Factory is a fully managed, serverless data integration service. Let us unpack those words. Data integration means bringing data from many different places together so it can be used as one. Managed means Microsoft runs the machines, applies the updates, and keeps the service available, so you do not look after servers. Serverless means you do not choose or rent machines in advance. You describe the work, and Azure finds the computing power when the work runs and releases it afterward, charging you only for what was used.

What does it actually do? Two main things. First, it moves data: it copies information from a source, such as a database in your office, a file in Amazon storage, or a software service like Salesforce, to a destination, such as Azure storage or a cloud database. Second, it orchestrates work, which means it decides what happens in what order and at what time. It can start a data copy at two in the morning, wait for it to finish, then ask another service to clean the data, then send an email if anything fails. The word orchestration comes from music: a conductor does not play any instrument, but tells each player when to start and stop.

ADF is often described as an ETL and ELT tool. ETL stands for extract, transform, load: pull the data out, reshape it, then put it into its destination. ELT swaps the last two steps: pull the data out, load it first into a powerful destination such as a data lake or warehouse, and let that destination do the reshaping. Modern cloud designs lean toward ELT because cloud destinations are very powerful, and ADF supports both styles.

A short history

Before the cloud, companies moved data with tools such as SQL Server Integration Services, a program that ran on their own servers. When businesses began storing data in the cloud, they needed a cloud-native way to move it that did not depend on a particular server. Microsoft released the first Azure Data Factory in 2015. The current generation, called version 2, arrived in 2018 with a far richer design, including schedules, event triggers, visual data transformation, and the ability to run old SSIS packages in the cloud. The first version has since been retired, so everything today means version 2. Microsoft has also built a related Data Factory experience into Microsoft Fabric, its newer all-in-one analytics platform. Azure Data Factory remains a standalone Azure service, and Microsoft’s current guidance can help you decide which one suits a new project.

Every tool exists because something hurt. Here is the pain that ADF removes.

Imagine a clothing retailer. Sales happen in a point-of-sale system in each shop, orders arrive from a website, stock levels live in a warehouse system, and customer emails sit in a marketing tool. The head of the company asks a simple question: “Which products are selling fastest in which cities, and are we about to run out?” To answer it, someone must gather data from all of those places, line it up, remove errors, and put it somewhere reports can read it. Done by hand, with spreadsheets, this takes days and goes wrong often.

Companies tried to automate it by writing their own scripts, small programs that copy data on a schedule. This works until it does not. A script fails silently at night and nobody notices until the morning report is empty. One person writes it, leaves the company, and nobody understands it. A change in a source system breaks everything. Scripts rarely handle retries, logging, alerts, or scaling, and each one is built differently.

ADF replaces the pile of scripts with a single, visual, monitored system. You draw the flow of work as a diagram, point it at your data sources, set the schedule, and let Azure run it. Retries, logging, alerts, history, and scaling are built in, so the whole team can see what ran, when, and what went wrong.

ADF has a small set of building blocks. Learn these eight words and every diagram becomes easy to read.

The central object is the data factory itself. It is the Azure resource that holds everything else, like the building that houses the factory floor. Inside it you create the following pieces.

Two more words appear everywhere. A pipeline run is one execution of a pipeline, like one shift on the production line. A source is where data comes from and a sink is where it goes. The sink name comes from the idea of a kitchen sink where things flow down and end up.

Now let us see how all the pieces connect, from the data sources on the left to the destinations on the right.

ADF is built in two layers. The control plane is the management side. It stores your pipeline designs, runs the scheduler, tracks every run, and decides what should happen next. It is entirely managed by Azure. The data plane is the working side. It is the integration runtime, the engine that really connects to the sources, moves the bytes, and runs the transformations. Keeping the two apart means the brain of the system stays small and reliable, while the muscle can be placed near your data and scaled independently.

You build your designs in Azure Data Factory Studio, a web-based drawing board. It has several areas called hubs. The Author hub is where you create pipelines, datasets, and data flows. The Monitor hub shows what ran and what failed. The Manage hub holds connections, integration runtimes, triggers, source control settings, and security options.

flowchart LR
    SRC1["On-premises database"] --> IR["Integration runtime - data plane"]
    SRC2["Software service"] --> IR
    SRC3["Cloud files and storage"] --> IR
    CP["Data Factory - control plane"] -->|"starts and tracks runs"| IR
    IR --> SINK1["Data lake storage"]
    IR --> SINK2["Cloud data warehouse"]
    IR --> SINK3["Analytics and reports"]
    

Read the diagram from left to right. Data sits in many sources. When a run begins, the control plane tells the integration runtime what to do. The runtime reads from the sources, optionally reshapes the data, and writes it to one or more sinks. The control plane never touches your actual data. It only sends instructions and collects status reports, which is a useful fact for security teams.

Before any data can move, ADF must know where it lives and how to get in. That is the job of these two objects.

A linked service is a saved connection. It holds the address of a system and the way to prove who you are, such as a name and password, a key, or a managed identity. Linked services come in two kinds. The first kind connects to data stores, such as databases, file storage, and software services. The second kind connects to compute services, such as a big data cluster that ADF asks to do heavy work. A linked service is like a contact in your phone: it saves the number so you do not retype it every time.

A dataset sits one level down. It points to the exact data inside a linked service. If the linked service is a connection to a database server, the dataset names one table. If the linked service is a connection to file storage, the dataset names one folder or one file and says whether it is a spreadsheet-style text file, a compact column format such as Parquet, or a nested format such as JSON. Datasets describe the shape of the data and where it sits, and activities use them as inputs and outputs.

The big benefit is reuse. You set up the connection once, and every pipeline that needs that system uses the same linked service. When a password changes or a server moves, you fix it in one place. Modern ADF also allows parameterized linked services and datasets, which means the server name or file path is filled in at run time. One generic dataset can then serve a hundred different tables.

The pipeline is the heart of ADF. Learn the kinds of steps it can hold and how steps depend on each other.

A pipeline is a logical group of activities that together perform one task, such as “load yesterday’s sales.” Activities are the steps. They fall into three families.

Control flow in plain words

Control flow is what makes a pipeline smart rather than a simple list. ForEach repeats a set of steps for every item in a list, and it can run many items in parallel, up to about fifty at once. If Condition chooses between two branches, like a fork in the road. Switch chooses among several branches. Until repeats until something becomes true, such as waiting for a file to appear. Wait pauses. Lookup reads a small amount of data, often a list of tables or a last-loaded date, and hands it to later steps. Get Metadata inspects a file or folder, for example to see whether it exists or how big it is. Execute Pipeline calls another pipeline, letting you build small reusable blocks. Web calls an external web address, which is handy for sending a message to a chat tool or starting some other service.

Dependencies between activities

Activities are joined by arrows called dependencies. An arrow can be set to follow one of four outcomes: on success, on failure, on completion (whether it worked or not), or on skip. This lets you draw error handling right into the design. For instance, if the copy step succeeds, run the clean-up step; if it fails, send an alert. Because the arrows decide order, activities without arrows between them run at the same time, which is the easy way to get parallel work.

The integration runtime decides where your work runs and which networks it can reach. Choosing the right one is one of the most important design decisions in ADF.

An integration runtime, shortened to IR, is the computing infrastructure that ADF uses to do real work: copy data, run data flows, dispatch jobs to other services, and run packages. There are three kinds.

Azure integration runtime

The Azure IR is the default. It is fully managed and serverless, runs in Azure, and can reach any data store that has a public address on the internet or inside Azure. You do not install anything. It scales by itself, and you can choose the Azure region where it runs. A variation, the Azure IR in a managed virtual network, runs inside a private network that Microsoft creates for your factory. In that mode ADF connects to your data stores using managed private endpoints, which are private doors approved by the data store owner, so data never crosses the public internet.

Self-hosted integration runtime

Some data sits where Azure cannot reach directly, such as a database in your office behind a firewall. For this you install the self-hosted IR, a small program, on a machine inside your own network. It makes only outbound connections to Azure, over the standard secure web port, so you do not need to open any inbound holes in your firewall. The self-hosted IR can also run on several machines together, called nodes, for high availability and more throughput, and it can be shared among several factories.

Azure-SSIS integration runtime

Many companies already own hundreds of packages built with SSIS, SQL Server Integration Services, Microsoft’s older on-premises data tool. The Azure-SSIS IR is a managed group of virtual machines dedicated to running those packages in the cloud without rewriting them, a strategy called lift and shift.

flowchart TB
    ADF["Data Factory control plane"] --> AZIR["Azure IR - managed and serverless"]
    ADF --> SHIR["Self-hosted IR - on your machines"]
    ADF --> SSIS["Azure-SSIS IR - managed virtual machines"]
    AZIR --> CLOUD["Cloud data stores"]
    AZIR --> PE["Private data stores via managed private endpoints"]
    SHIR --> ONP["On-premises and private network data"]
    SSIS --> PKG["Existing SSIS packages"]
    

A pipeline is only a design until something starts it. Triggers are the starters.

A trigger decides when a pipeline runs. You can also start a pipeline by hand, which is useful for testing, but production work is almost always started by a trigger. ADF offers four main kinds.

The tumbling window deserves a closer look because it solves a real puzzle. Suppose a pipeline must process each hour’s data exactly once. With an ordinary schedule, if the system is down at three o’clock, the three o’clock slice may simply be missed. A tumbling window remembers each slice as a separate unit with its own start and end time. If a slice fails, it can retry just that slice. If the factory was off for a day, it can run all the missed slices to catch up, a task called backfill. It also lets one window wait for another to finish first.

flowchart LR
    T1["Schedule trigger - time based"] --> P["Pipeline run"]
    T2["Tumbling window - one run per time slice"] --> P
    T3["Storage event - file arrives"] --> P
    T4["Custom event - message arrives"] --> P
    T5["Manual start - testing"] --> P
    

What actually happens, step by step, when a Copy activity moves a table from a database into a data lake? Let us follow one run.

It starts with the trigger. When a trigger fires, the control plane creates a pipeline run, gives it a unique identifier, and records its start time. The scheduler works out which activities can start first, which are those without waiting arrows, and queues them. For a Copy activity, the control plane hands a job to the chosen integration runtime, together with the connection details from the linked services and the descriptions from the datasets.

The integration runtime now does the real work. It opens a connection to the source using the stored credentials. It asks the source for the data, often splitting the request into several parallel slices so many parts of a large table can be read at once. For example, it may divide a table by ranges of a number column. Data flows through the runtime in memory in small batches, so even a very large table never needs to fit in memory all at once. The runtime converts values from the source’s types into the destination’s types, writes the data to the sink in efficient bulk operations, and reports progress back to the control plane every so often.

When all slices finish, the Copy activity reports the number of rows read, rows written, the speed, and any skipped rows. The control plane records the result, checks which arrows leave this activity, and starts the next activities whose conditions are now satisfied. When the last activity ends, the pipeline run is marked succeeded or failed, and the run is stored so you can inspect it later.

sequenceDiagram
    participant Trig as Trigger
    participant CP as Control plane
    participant IR as Integration runtime
    participant Src as Source database
    participant Snk as Data lake sink
    Trig->>CP: Fire and create pipeline run
    CP->>IR: Send copy job with connection details
    IR->>Src: Open connection and read in parallel slices
    Src-->>IR: Stream batches of rows
    IR->>Snk: Write batches in bulk
    IR-->>CP: Report progress and final counts
    CP-->>Trig: Mark run succeeded or failed
    

Speed controls inside Copy

When data must be reshaped, not just moved, mapping data flows let you do it with boxes and arrows instead of code.

A mapping data flow is a visual way to build data transformations. You drag a source onto a canvas, connect it to transformation boxes, and finish at a sink. Each box does one clear job. Common ones include filter, which keeps only rows that meet a rule; join, which combines two data sets on a shared column; aggregate, which totals or averages rows by group; derived column, which creates or changes a column using a formula; select, which keeps, renames, or reorders columns; pivot, which turns rows into columns; window, which calculates running totals and rankings; lookup, which adds matching information from another data set; and conditional split, which sends rows down different paths depending on a rule.

The visual design is only the surface. When you run it, ADF turns your diagram into code for Apache Spark, a popular engine that splits big data across many machines and works on the pieces at the same time. ADF creates and manages the Spark cluster for you, so you never configure it. This is why data flows can transform very large volumes: the work is spread over many workers. The price of that power is startup time. A fresh cluster can take a few minutes to start, so the Azure IR offers a time to live setting that keeps the cluster warm for a while after a run, so the next run starts quickly.

When not to use data flows

Data flows are wonderful for visual transformations at scale, but they are not always the best tool. If your destination is a powerful warehouse or lakehouse that already has a strong engine, it may be cheaper and faster to copy the data in and let that engine transform it with its own language, the ELT approach. If a team already works with notebooks and code in Databricks or Synapse, ADF can simply call those jobs as activities and focus on orchestration. Use each tool where it is strongest.

The difference between a beginner’s factory and a professional one is reuse. Parameters make one design do the work of a hundred.

A parameter is a value you give a pipeline, dataset, or linked service when it starts, instead of building it into the design. A variable is a value that a pipeline can change while it runs, such as a counter or a message. A global parameter is a value shared by every pipeline in the factory, for example the name of the environment. An expression is a small formula in ADF’s expression language that calculates a value at run time, such as today’s date minus one day, or the file name built from a table name and a time stamp. Expressions turn static designs into flexible ones.

Put these ideas together and you get the most important professional pattern in ADF, the metadata-driven pipeline. Instead of drawing a separate pipeline for each table or file, you keep a small control table, a list that describes what to load: the source system, the table name, the destination folder, and how often to run. One generic pipeline reads that list with a Lookup, loops over it with ForEach, and copies each item using parameterized datasets. To add a new table, you add a row to the list. You do not change or redeploy any pipeline.

flowchart TB
    CT["Control table - list of what to load"] --> LK["Lookup activity reads the list"]
    LK --> FE["ForEach loop over each item"]
    FE --> CP1["Copy table 1"]
    FE --> CP2["Copy table 2"]
    FE --> CP3["Copy table N"]
    CP1 --> LOG["Write result to audit table"]
    CP2 --> LOG
    CP3 --> LOG
    

Copying everything every night is simple, but it wastes time and money. Smart pipelines copy only what changed, and survive being run twice.

A full load copies every row of a table every time. It is simple and fine for small tables. An incremental load copies only rows that are new or changed since the last run. For large tables, this cuts hours to minutes. There are a few common ways to find the changed rows.

The idea of idempotency

A very important word is idempotent. A pipeline is idempotent if running it twice gives the same result as running it once. This matters because things go wrong: a run fails halfway, someone presses the button twice, or a trigger fires again after a repair. If a pipeline simply appends rows every time, a re-run creates duplicates and your totals double. To avoid that, design the sink step to be safe. You can overwrite the folder for that day, use an upsert that updates existing rows and inserts new ones using a key, or delete the slice before loading it again.

Data pipelines run at night, unwatched. Reliability means they cope with trouble and tell you clearly when they cannot.

Every activity has a retry policy. You choose how many times ADF should try again after a failure and how many seconds to wait between tries. Many problems are short-lived, such as a database that is briefly busy, or a network hiccup, and a retry or two cures them. Each activity also has a timeout, the longest it may run before ADF stops it. The default timeout is very long, so it is wise to set a realistic limit. Otherwise a stuck step may hold a pipeline for days.

For failures that retries cannot fix, use failure paths. Draw an arrow from an activity on failure to an alert step, such as a web call to a messaging tool, a stored procedure that writes the error, or an email service. You can also build a catch-all that runs on completion to clean up temporary files whether things worked or not. A word of caution: how ADF decides the final status of a pipeline depends on how the last steps finished, and a handled failure may leave a pipeline looking successful. Test your error paths and check the status you really see.

Failure scenarios and recovery

A data factory touches nearly every important system in a company, so its security matters a great deal.

Start with identity. Every data factory can have a managed identity, an automatic identity in Microsoft Entra ID, the Azure identity system. You give that identity access to the data stores it needs, and ADF logs in as itself, so no password has to be stored anywhere. Where a password or key is unavoidable, store it in Azure Key Vault and let the linked service fetch it at run time. A user-assigned identity can be shared and reused when several services need the same access.

Next, who may change things. Azure role based access control decides who can create, edit, or run in a factory. Built-in roles such as Data Factory Contributor allow designing, while narrower roles can allow only monitoring. Apply the rule of least privilege: people and systems get only the access they need.

Then the network. By default the Azure IR connects over the public internet, protected by encryption. For stricter needs, use the managed virtual network and managed private endpoints, so ADF reaches storage and databases over private addresses. The data store owner must approve each private endpoint, which also protects against data exfiltration, meaning data being sent to places it should not go. For on-premises sources, the self-hosted IR makes only outbound connections, so no inbound firewall holes are needed. You can also disable public network access to the factory itself and reach it through a private endpoint.

Professional teams do not edit production directly. They design in one place, test in another, and release in a controlled way.

ADF connects to Git, the standard system for tracking changes to files, through Azure DevOps or GitHub. Every pipeline, dataset, and trigger is stored as a small text file in a repository, so you get a full history of who changed what and when, and you can work in separate branches without disturbing others. When your work is ready, you merge it into the collaboration branch and publish, which pushes the designs into the live factory and creates a deployment package of the whole factory.

Real projects use several factories, usually called environments: development, test, and production. A release pipeline, built in a tool such as Azure DevOps or GitHub Actions, takes the deployment package from the development factory and applies it to test and production factories. Values that differ between environments, such as server names and Key Vault addresses, are replaced using parameters at deployment time. Triggers are usually stopped before a deployment and restarted after, so that no run begins while the design is half-updated.

flowchart LR
    DEV["Developer branch"] --> PR["Pull request and review"]
    PR --> COL["Collaboration branch"]
    COL --> PUB["Publish creates deployment package"]
    PUB --> TEST["Test factory"]
    TEST --> APPR["Approval"]
    APPR --> PROD["Production factory"]
    

A quiet pipeline is not always a healthy one. Good monitoring tells you about trouble before your users do.

The first place to look is the Monitor hub in Data Factory Studio. It lists every pipeline run, every trigger run, and every activity run, with status, start time, duration, and error messages. You can drill into a failed activity and see its inputs, its outputs, and the exact error. One important fact: ADF keeps this run history for only about forty-five days, so if you need longer history for audits or trend analysis, send it elsewhere.

That is done with Azure Monitor. You can turn on diagnostic settings to send logs to a Log Analytics workspace, a searchable store with a query language, or to storage for long retention. You can also use metrics, which are numbers over time, such as the count of succeeded and failed pipeline runs, failed activity runs, failed trigger runs, and the CPU use of a self-hosted integration runtime. On top of those, set alerts, rules that notify you through email, text, or a chat tool when a number crosses a line.

Slow pipelines cost time and money. Tuning starts with finding the real bottleneck, not guessing.

Every data movement has three parts: reading from the source, carrying the data across the network, and writing to the sink. The whole job can only go as fast as its slowest part. So the first rule of tuning is to measure. After a Copy run, ADF shows a breakdown of how long each stage took and which part was the bottleneck. If the source is slow to answer, adding more copy power does nothing. If the sink is overloaded, you need to relieve the sink.

ADF is pay-as-you-go. That is kind to small projects and can surprise large ones.

ADF billing has several parts, and understanding them is the key to controlling cost. Orchestration is charged by the number of activity runs and trigger runs, in units of a thousand. Data movement is charged for Copy activities by the time they run multiplied by the data integration units used, and the rate differs between the Azure IR and the self-hosted IR. Data flow execution is charged by the cluster’s core count and the time it runs, so a long-running cluster with many cores is the largest single cost for many teams. External activities, which hand work to other services, have a small orchestration charge, while the other service bills its own compute separately. The Azure-SSIS IR is billed for the virtual machines for as long as it is running. There are also small charges for reading and writing factory entities and for monitoring operations. Check the current price list, since rates change.

No tool is right for everything. A fair look at both sides helps you choose with open eyes.

The big trade-offs

Visual simplicity versus fine control. Boxes and arrows are quick to build and share, while code in a notebook gives finer control and easier testing for tricky logic. Orchestration versus transformation. ADF is superb at deciding what runs when, but it is often wiser to hand heavy transformation to a specialist engine. Batch versus streaming. ADF excels at scheduled and event-driven batches, while continuous streams belong in services built for them, such as Event Hubs with Stream Analytics or Spark streaming. Managed convenience versus cost. You pay a premium for not running things yourself, which is usually worth it, but must be watched.

Proven recipes for ADF, and a guide to when other tools fit better.

Pattern: layers of a data lake

A popular design, often called the medallion pattern, organizes a data lake in three layers. The bronze layer holds raw data exactly as it arrived, which keeps a faithful record. The silver layer holds cleaned, de-duplicated, and standardized data. The gold layer holds business-ready tables shaped for reports and analysis. ADF moves data into bronze, orchestrates the steps that build silver and gold, and tells the business when fresh data is ready.

flowchart LR
    S1["Business systems"] --> ADF1["ADF copy"]
    S2["Files from partners"] --> ADF1
    S3["Software services"] --> ADF1
    ADF1 --> B["Bronze - raw data"]
    B --> T1["Clean and standardize"]
    T1 --> SV["Silver - cleaned data"]
    SV --> T2["Join and aggregate"]
    T2 --> G["Gold - business tables"]
    G --> RPT["Reports and dashboards"]
    

Choosing among similar tools

An anti-pattern is a habit that looks sensible but leads to pain.

A short checklist that separates calm operations from late-night emergencies.

Common mistakes

• Forgetting to start triggers. Published triggers that are not started never run.
• Leaving debug sessions on. Debug clusters keep billing until they time out or are stopped.
• Choosing the wrong integration runtime. A private on-premises database cannot be reached by the default Azure IR.
• Ignoring time zones. A schedule set in the wrong zone starts at the wrong hour, and daylight saving changes can shift or repeat runs.
• Overloading the source. Too many parallel copies can slow the very system the business depends on.
• Trusting schema forever. A column added or renamed at the source may break a mapping, so handle schema changes deliberately.
• No owner for alerts. Alerts that go to nobody are the same as no alerts.

Where does Azure Data Factory shine in practice?

Yes, and an ELT and orchestration tool too. It moves data, can transform it with data flows, and coordinates other services that transform it.

Not for most work. You build pipelines visually. Code appears mainly in small expressions and when you call code-based services such as notebooks or stored procedures.

A pipeline is a group of steps that does one job. An activity is one step inside it.

A linked service is the saved connection to a system. A dataset points to the exact data inside that system, such as one table or folder.

Use the Azure IR for cloud and internet-reachable data, a managed virtual network version when it must stay private, the self-hosted IR for on-premises or private networks, and the Azure-SSIS IR to run existing SSIS packages.

When each time slice must be processed exactly once, with retries per slice, dependencies between runs, and the ability to backfill missed periods.

Not as its main purpose. It is built for batch and event-driven work. Use a streaming service for continuous low-delay flows, and let ADF orchestrate around it.

Use the factory’s managed identity where a service supports it, and store any remaining secrets in Azure Key Vault referenced from the linked service.

About forty-five days in the built-in monitor. Send diagnostic logs to Log Analytics or storage if you need longer retention.

Connect Git to the development factory, publish to create a deployment package, and use a release pipeline to apply it to test and production with environment-specific parameters.

A new Spark cluster must start before the flow runs, which can take a few minutes. A time to live setting keeps the cluster warm between runs, at a cost.

They share many ideas and similar building blocks, but live in different products. ADF is a standalone Azure service, Synapse pipelines sit inside a Synapse workspace, and Fabric includes its own Data Factory experience. Check Microsoft’s current guidance when choosing.