Azure Data Factory: The Delivery Network for Your Data
A beginner-friendly guide to Azure Data Factory. Learn how it collects data from many places, cleans and reshapes it, and delivers it on schedule, using pipelines, activities, datasets, linked services, triggers, and integration runtimes.
Imagine a huge city with farms, factories, warehouses, and shops spread far apart. Food grown on farms must be collected, washed, packed, and delivered to shops before breakfast every morning. A good delivery network has trucks, routes, schedules, packing stations, and managers who check that nothing goes missing. In the world of data, companies have the same problem. Data is created in many different places, and it must be collected, cleaned, reshaped, and delivered where it is needed. Azure Data Factory is the delivery network for data.
1What Is Azure Data Factory?
Start with a simple definition and a picture in your mind.
Azure Data Factory, often shortened to ADF, is a cloud service from Microsoft for building data integration workflows. Data integration means gathering data from different sources, combining it, preparing it, and moving it to a place where people and programs can use it. ADF lets you design these workflows visually, schedule them, and watch them run, without managing any servers yourself.
The service is serverless, which means Microsoft provides and scales the computing power behind the scenes. You describe what should happen, and ADF finds the resources to do it. You pay for what you use rather than for machines that sit idle.
ADF does not store your data itself. Think of it as the manager and the trucks, not the warehouse. The data lives in other places such as databases, storage accounts, and applications. ADF moves it and transforms it between those places.
If your data stores are farms, factories, and shops, Azure Data Factory is the logistics company. It owns no farms and no shops. Its job is to pick things up, process them if needed, and deliver them on time, following routes and schedules that you define.
2The Problem Data Factory Solves
A tool makes sense when you know the pain it removes.
Modern organizations keep information in many places at once: sales systems, websites, spreadsheets, sensors, partner files, and old databases in the office basement. To answer a simple business question such as “How did we sell last month across all channels?”, someone must bring all of this data together.
Scattered data
Data lives in cloud services, on-premises servers, files, and applications, each with its own format and access method.
Messy data
Raw data has duplicates, missing values, and different spellings. It must be cleaned before it can be trusted.
Manual work
Copying files by hand or running scripts on someone’s laptop is slow, error-prone, and impossible to rely on.
Scale
Data volumes keep growing. A script that worked for a thousand rows may fail for a billion.
No visibility
When a nightly job silently fails, nobody knows until the morning report is wrong.
Azure Data Factory answers each of these. It connects to many sources, transforms data visually or with code, runs automatically on schedules or events, scales without extra effort, and shows every run with its status and errors.
3ETL and ELT in Simple Words
Two short abbreviations describe most of what Data Factory does.
ETL stands for Extract, Transform, Load. You extract data from a source, transform it into a clean and useful shape, and load it into a destination. The transformation happens in the middle, before the data arrives at its final home.
ELT stands for Extract, Load, Transform. You extract data and load it quickly into a powerful destination such as a data warehouse or data lake, and then transform it there using the destination’s own computing power. This approach is popular in the cloud because modern destinations can process huge amounts of data fast.
| Point | ETL | ELT |
|---|---|---|
| Order of steps | Extract, then transform, then load | Extract, then load, then transform |
| Where transformation happens | In a separate processing engine in the middle | Inside the destination system |
| Best when | Data must be cleaned or protected before landing | The destination is powerful and you want raw data kept |
| Typical in | Traditional data warehouses | Cloud data lakes and modern warehouses |
ETL is like washing and chopping vegetables at the farm, then delivering them ready to cook. ELT is like delivering the vegetables as they are to the restaurant kitchen and letting the chefs prepare them there. Both work. The right choice depends on where the best tools are.
Data Factory supports both styles. It can transform data itself using data flows, or it can move data quickly and then trigger transformations inside services such as Azure Synapse, Azure Databricks, or a SQL database.
4The Core Building Blocks
Six words explain almost everything inside Data Factory.
Pipeline
A group of steps that together do one job, such as “load yesterday’s sales into the warehouse”. A pipeline is the whole delivery route.
Activity
One single step inside a pipeline, such as copying data, running a transformation, or calling a web service.
Dataset
A named description of the shape and location of data that activities read or write, such as a particular table or folder of files.
Linked Service
The connection information needed to reach a data store or compute service, like a saved login and address.
Trigger
The thing that starts a pipeline run, such as a clock schedule, a new file arriving, or a manual click.
Integration Runtime
The computing infrastructure that actually performs the work. It is the engine and the trucks behind the scenes.
flowchart TD
T["Trigger: starts the run"] --> P["Pipeline"]
P --> A1["Activity 1: Copy data"]
P --> A2["Activity 2: Transform data"]
P --> A3["Activity 3: Send notification"]
A1 --> D1["Dataset: source files"]
A1 --> D2["Dataset: destination table"]
D1 --> L1["Linked Service: source connection"]
D2 --> L2["Linked Service: destination connection"]
A1 --> IR["Integration Runtime: does the work"]
A pipeline is a delivery route plan. Activities are the stops and tasks on the route. A dataset describes the goods being picked up or dropped off. A linked service is the key and address for each location. A trigger is the alarm clock that starts the day. The integration runtime is the truck and driver that actually do the work.
5Linked Services and Datasets
Connections say where; datasets say what.
Linked services: the connection
A linked service stores the information required to connect to something. For a database it may include the server name and how to sign in. For a storage account it may include the account address and the identity used to reach it. For a compute service it describes where and how to start work.
You define a linked service once and reuse it in many pipelines. If the password or address changes, you fix it in one place.
Datasets: the shape and place of data
A dataset sits on top of a linked service. It names a specific piece of data, for example “the orders table” or “the folder of daily sales files in CSV format”. It describes the structure, such as the columns and their types, when needed.
| Item | Answers the question | Example |
|---|---|---|
| Linked Service | How do I reach this system? | The address and sign-in details of a SQL server |
| Dataset | Which data inside that system? | The Customers table, or the sales folder with CSV files |
Do not type passwords directly into linked services. Use managed identities where the service supports them, or keep secrets in Azure Key Vault and let the linked service read them from there.
A linked service is the address and key to a warehouse. A dataset is the label on a specific shelf in that warehouse. You need the key to enter, and the shelf label to find the exact boxes.
Creating a separate dataset and linked service for every tiny variation. Use parameters, covered later, so one reusable dataset can serve many folders or tables.
6Integration Runtimes: The Engines Behind the Scenes
Every activity needs somewhere to run. That place is the integration runtime.
An integration runtime, shortened to IR, provides the computing power that moves data, runs transformations, and dispatches work to other services. There are three kinds, each suited to a different situation.
Azure Integration Runtime
Fully managed by Microsoft and used for moving data between cloud services and running data flows. It scales automatically and needs no setup beyond choosing a region.
Self-hosted Integration Runtime
Software you install on a machine you control, such as a server in your office or a virtual machine in a private network. It lets Data Factory reach systems that are not exposed to the internet.
Azure-SSIS Integration Runtime
A managed cluster dedicated to running existing SQL Server Integration Services packages in the cloud, helping companies move older ETL jobs without rewriting them.
| Runtime | Who manages it | Best for |
|---|---|---|
| Azure IR | Microsoft | Cloud-to-cloud copying and data flows |
| Self-hosted IR | You | On-premises or private network data sources |
| Azure-SSIS IR | Microsoft, with you choosing the size | Lifting existing SSIS packages to the cloud |
flowchart LR
subgraph ONP["On-premises Office"]
DB["Local Database"]
SH["Self-hosted Integration Runtime"]
DB --- SH
end
subgraph AZ["Azure Cloud"]
ADF["Azure Data Factory"]
AIR["Azure Integration Runtime"]
LAKE["Data Lake Storage"]
end
ADF -->|Sends instructions| SH
SH -->|Moves data securely| LAKE
ADF --> AIR
AIR --> LAKE
The Azure integration runtime is a fleet of delivery trucks owned by the logistics company, available instantly. The self-hosted one is a local driver you station inside a gated factory, because the main fleet is not allowed through the gate.
The self-hosted runtime makes outbound connections to Azure. This usually means you do not have to open inbound holes in your company firewall, which is a big plus for security teams.
7Activities: The Steps of the Journey
Activities are the individual tasks that make up a pipeline.
Data movement
Copy activity
The workhorse of Data Factory. It reads data from a source and writes it to a destination, handling format conversion, speed tuning, and retries on the way. It supports a very wide range of sources and destinations.
Data transformation
Mapping data flow
A visual way to design transformations such as joining, filtering, aggregating, and cleaning data. Behind the scenes Data Factory runs them on a managed Apache Spark engine, so you get big-data power without writing Spark code.
External compute activities
Activities that hand the work to another service, such as running a notebook in Azure Databricks, a stored procedure in a database, an Azure Synapse job, or an Azure Function.
Control flow
Control activities decide how the pipeline behaves, like traffic lights and road signs on the route.
If Condition
Takes one path or another depending on a true or false check.
ForEach
Repeats steps for every item in a list, such as every table or every file.
Until
Repeats steps until a condition becomes true, useful for waiting on something to finish.
Switch
Chooses among several paths based on a value.
Lookup and Get Metadata
Read small pieces of information, such as a list of tables or the size and date of a file, to guide later steps.
Execute Pipeline
Calls another pipeline, letting you build small reusable parts and combine them.
Web and Webhook
Call an outside web address, for example to send a message or start another system.
Wait, Set Variable, Fail
Pause for a time, store a value for later, or deliberately stop with a clear error message.
Connecting activities with outcomes
Activities are joined by arrows that depend on the result of the previous step: on success, on failure, on completion regardless of result, or on skip. This lets you build error handling, such as “if the copy fails, send an alert and clean up temporary files”.
flowchart LR
S["Start"] --> L["Lookup: list of tables"]
L --> F["ForEach table"]
F --> C["Copy table to data lake"]
C -->|Success| T["Transform with data flow"]
C -->|Failure| N["Send failure alert"]
T --> E["Load into warehouse"]
Control flow activities are the road signs on a delivery route: “go through every street on this list”, “if the bridge is closed, take the other road”, “wait here until the shop opens”. They give the pipeline decision-making ability.
8Data Flows: Visual Transformations at Scale
Transform data without writing programs.
Mapping data flows let you design transformations by dragging boxes onto a canvas and connecting them. Each box represents an action such as choosing columns, filtering rows, joining two sets of data, grouping and summing, or handling slowly changing records. Data Factory translates the design into code that runs on a managed Spark cluster.
Source
Where the data comes in, with its columns and types.
Filter and Select
Keep only the rows and columns you need.
Derived Column
Create or change a column using an expression, such as combining first and last names.
Join and Lookup
Combine data from two streams based on matching values.
Aggregate
Group rows and calculate totals, averages, and counts.
Sink
Where the result is written.
How a data flow runs
Design on the canvas
You connect sources, transformation steps, and sinks visually, and preview the data at each step while debugging.
A pipeline calls the data flow
A data flow activity inside a pipeline starts it, passing in any parameters.
A Spark cluster is prepared
Data Factory starts a managed cluster for the run. Starting can take a few minutes unless you use options to keep clusters warm.
Data is processed in parallel
The cluster splits the work across many workers, reading, transforming, and writing in parallel.
Results and logs are returned
The sink receives the data, and the run reports row counts, timings, and any errors.
Data flows are billed by the compute used and the time the cluster runs, and debug sessions also consume compute. Turn off debug mode when you are not using it, and avoid data flows for simple copying where a Copy activity is cheaper.
A data flow is a factory production line you design on paper. You say: wash here, sort there, join these two streams, pack at the end. Data Factory builds and runs the production line on a big team of workers for as long as the job takes.
9Triggers: Deciding When Pipelines Run
A pipeline does nothing until something starts it.
Manual run
You click a button to run the pipeline now. Perfect for testing and one-off tasks.
Schedule trigger
Runs a pipeline on a clock schedule, such as every night at two in the morning or every Monday. Simple and widely used for regular loads.
Tumbling window trigger
Runs a pipeline for back-to-back, non-overlapping time slices, such as each hour of data. Each window knows its own start and end time, supports waiting for earlier windows, and can rerun a missed window. This is ideal for processing time-based data reliably.
Storage event trigger
Runs a pipeline when a file is created or deleted in storage. For example, as soon as a partner uploads a file, the pipeline starts to process it.
Custom event trigger
Runs a pipeline in response to custom events published through Azure Event Grid, so other systems can start your pipelines.
| Trigger | Starts when | Good for |
|---|---|---|
| Manual | You press run | Testing and one-time jobs |
| Schedule | The clock reaches a set time | Nightly and weekly loads |
| Tumbling window | Each fixed time slice begins | Reliable time-sliced processing with reruns |
| Storage event | A file appears or is removed | Reacting to uploads from partners and systems |
| Custom event | Another system publishes an event | Event-driven designs |
A schedule trigger is an alarm clock that rings every morning. A storage event trigger is a doorbell that rings when a package arrives. A tumbling window is a timetable with one bus for every hour that always tells its driver which hour it is covering.
One trigger can start several pipelines, and one pipeline can have several triggers. Triggers must be published and started before they take effect, which beginners often forget.
10Parameters, Variables, and Expressions
Make pipelines flexible so one design can do many jobs.
Parameters
Values that you pass into a pipeline, dataset, linked service, or data flow when it starts. They do not change during the run. Examples: the table name, the date, or the folder path.
Variables
Values stored inside a pipeline that can change as the pipeline runs, such as a counter or a status text.
Expressions
Small formulas written in Data Factory’s expression language to calculate values on the fly, like building a folder name from today’s date.
System variables
Values provided by Data Factory itself, such as the pipeline name, the run identifier, and the trigger time.
Example: one pipeline for fifty tables
Instead of building fifty copy pipelines, you build one. A lookup activity reads a list of table names, a ForEach loop goes through the list, and a parameterized dataset copies each table. Adding a new table means adding one row to the list, not building a new pipeline.
A parameter is a blank on a form that you fill in before the delivery starts, such as the destination street. A variable is the notepad the driver keeps during the trip. An expression is a little calculator that works out something like today’s date to write on the label.
Think “metadata-driven”. Keep lists of sources, tables, and rules in a small control table and let one smart pipeline read it. This approach scales far better than copying and pasting pipelines.
11A Typical Data Platform Pattern
See where Data Factory sits in a common design.
Many organizations organize their data lake into layers that improve step by step. Raw data lands first, then it is cleaned, and finally it is shaped for reports. Data Factory commonly orchestrates the movement between these layers.
Ingest into the raw layer
Copy activities bring data exactly as it is from sources into a landing area of the data lake, keeping an untouched history.
Clean into the refined layer
Data flows or external compute remove duplicates, fix types, and standardize values.
Shape into the curated layer
Data is joined and summarized into business-friendly tables, ready for reports and analysis.
Serve to consumers
The curated data is loaded into a warehouse or made available to reporting tools and data scientists.
flowchart LR
S1["Sales Database"] --> ADF["Azure Data Factory"]
S2["Website Files"] --> ADF
S3["Partner Uploads"] --> ADF
ADF --> RAW["Raw Layer in Data Lake"]
RAW --> CLEAN["Refined Layer"]
CLEAN --> CUR["Curated Layer"]
CUR --> DW["Data Warehouse"]
DW --> BI["Reports and Dashboards"]
Think of a water supply system. Water is collected from rivers (raw), cleaned at a treatment plant (refined), tested and blended (curated), and finally sent through pipes to homes (reports). Data Factory is the network of pumps and valves that moves the water at every stage.
12Incremental Loading: Moving Only What Changed
Copying everything every night wastes time and money.
Imagine a table with a hundred million rows where only ten thousand change each day. Copying all of it every night is slow and expensive. Incremental loading moves only new or changed rows. This is one of the most important patterns in real projects.
Watermark column
Use a column such as “last modified time” or an increasing identifier. Remember the highest value from the last run, and next time fetch only rows above it.
Change tracking or change data capture
Use the source database’s own feature that records inserts, updates, and deletes, and read only those changes.
File-based by time or name
For files, copy only files that arrived after a certain time or match a date folder, for example yesterday’s folder.
Tumbling window slices
Process each time window once, using its start and end times to select the matching data.
Read the old watermark
A lookup activity reads the last saved watermark value from a small control table.
Read the new watermark
Another lookup finds the current highest value in the source.
Copy the difference
A copy activity moves only rows between the old and new watermark.
Save the new watermark
After a successful copy, update the control table so the next run starts from here.
Saving the new watermark before the copy finishes. If the copy then fails, the next run skips the missed rows forever. Always update the watermark only after success.
Incremental loading is like a newspaper delivery that only brings today’s edition, not every paper ever printed. The watermark is the bookmark showing which edition was delivered last.
13Monitoring and Alerts
A pipeline you cannot watch is a pipeline you cannot trust.
Data Factory includes a Monitor area that shows every pipeline run, trigger run, and activity run. For each run you can see the start time, duration, status, input and output details, and detailed error messages. You can filter by time, status, or pipeline name, and rerun failed pipelines or only the failed part.
Pipeline and activity runs
See which runs succeeded, failed, or are still going, and drill into any step for details.
Trigger runs
Check whether schedules and events fired as expected.
Azure Monitor alerts
Send emails or messages when a pipeline fails or takes too long.
Log Analytics
Send logs and metrics to a central workspace for long-term storage, advanced searches, and dashboards.
Rerun options
Rerun a whole pipeline, or rerun from the failed activity to save time and cost.
The Monitor area is the control room of a delivery company. A big screen shows every truck, whether it is on time, and where it broke down. Alerts are the phone calls that wake the manager when something goes wrong.
Add error-handling paths and alerts to important pipelines from the start. Silent failures are the most dangerous kind, because wrong data keeps flowing into reports without anyone noticing.
14Security in Data Factory
Data pipelines touch sensitive information, so protection must be built in.
Managed identity
The factory gets its own identity in Azure, which can be given permission to storage, databases, and other services, so no passwords need to be stored.
Azure Key Vault for secrets
Store passwords and keys in Key Vault and let linked services fetch them securely when needed.
Role-based access control
Control who can view, edit, or run the factory, using least-privilege roles such as Data Factory Contributor.
Managed virtual network and private endpoints
Run integration runtimes inside a Microsoft-managed network and connect to data stores over private endpoints so traffic stays off the public internet.
Encryption
Data is encrypted in transit, and factory metadata is encrypted at rest. You can bring your own key for extra control.
Auditing
Activity and diagnostic logs record who changed what, supporting reviews and compliance needs.
Problem
Typing database passwords and storage keys directly into linked services and sharing them across the team.
Why It’s Harmful
Secrets can leak through exports, screenshots, or source control, and a leaked password gives direct access to your data stores.
Correct Approach
Use managed identities wherever possible, keep any remaining secrets in Key Vault, and rotate them regularly.
Using a managed identity is like giving the delivery company employee an ID badge that the warehouse recognizes, instead of handing out the physical key to every driver. If the badge is cancelled, access ends instantly and nothing needs to be re-keyed.
15Source Control and Deployment
Treat your pipelines like software, because they are.
Data Factory can connect to Git repositories such as Azure DevOps and GitHub. Your pipelines, datasets, and other items are saved as files in the repository, which gives you version history, branches, reviews, and the ability to undo mistakes.
Work in a feature branch
Developers make changes in their own branch so they do not disturb others or live pipelines.
Test with debug runs
Use debug mode to run pipelines on demand and check results before sharing the change.
Review and merge
A teammate reviews the change through a pull request, and it is merged into the main collaboration branch.
Publish and generate templates
Publishing produces deployable templates that describe the whole factory.
Deploy to test and production
An automated release process applies the templates to other factories, swapping in environment-specific settings such as connection details.
flowchart LR
DEV["Development Factory with Git"] --> PR["Pull Request and Review"]
PR --> MAIN["Main Branch"]
MAIN --> PUB["Publish: deployment templates"]
PUB --> TEST["Test Factory"]
TEST --> PROD["Production Factory"]
Use separate factories for development, test, and production, and keep environment-specific values such as server names as parameters so the same pipeline design works everywhere.
Git is a notebook that records every edit to your delivery plans, with names and dates. You try a new route on a draft copy, a colleague checks it, and only then does it replace the official route book.
16Performance and Cost
Fast pipelines are usually cheap pipelines too.
Data Factory billing is based on how much you use. The main parts are pipeline orchestration, which counts activity runs and trigger runs, data movement, which depends on how much power the copy uses and for how long, and data flow execution, which depends on the cluster size and run time. Always confirm current prices on the official pricing page.
Orchestration
Charged per activity and trigger run, so a loop with thousands of tiny steps can add up.
Data movement
Charged by the compute power used during copying and for how long it runs.
Data flows
Charged for cluster compute time, including time spent starting up and in debug sessions.
Self-hosted runtime machines
The servers you run yourself have their own cost, outside Data Factory billing.
Ways to improve speed and lower cost
Load incrementally
Move only changed data instead of everything every time.
Use the right tool for the job
Use a Copy activity for plain movement, and use data flows only when you need real transformations.
Use parallel copying carefully
Copy several tables at once with ForEach in parallel, but not so many that you overload the source database.
Choose efficient file formats
Columnar formats such as Parquet are smaller and faster to read for analytics than plain text files.
Place resources in the same region
Keep the factory, runtime, and data stores near each other to reduce delay and avoid data transfer charges.
Stop idle debug sessions
Debug clusters keep running and billing until their time to live expires or you turn them off.
Use the Azure pricing calculator for estimates, and review actual costs in Azure Cost Management after the first weeks of real use.
17Real-World Use Cases
See how different organizations use Data Factory every day.
Nightly sales loads for a retailer
A retail chain copies the day’s sales from store databases into a central data lake every night, cleans them, and loads summaries into a warehouse so managers see fresh reports each morning.
Moving an on-premises warehouse to the cloud
A company uses a self-hosted integration runtime to reach its office database and copies years of history into cloud storage, then switches to incremental loads for daily changes.
Processing partner files as they arrive
A logistics firm receives shipment files from many partners. A storage event trigger starts a pipeline the moment a file lands, validating and loading it within minutes.
Bringing in software-as-a-service data
A marketing team pulls data from advertising and customer relationship tools using built-in connectors and stores it alongside website data for combined analysis.
Lifting existing SSIS packages
An enterprise runs its older integration packages in the cloud using the Azure-SSIS runtime, buying time to modernize them gradually.
Preparing data for machine learning
A data science team uses pipelines to collect and clean training data on a schedule, then triggers a notebook that trains or scores a model.
18Data Factory Compared with Related Tools
Knowing the neighbors helps you choose the right tool.
| Tool | Main purpose | Choose it when |
|---|---|---|
| Azure Data Factory | Orchestrating and moving data between many systems | You need scheduled or event-driven data pipelines across many sources |
| Azure Synapse pipelines | Data Factory style pipelines inside the Synapse workspace | You already work mainly in Synapse and want everything in one place |
| Microsoft Fabric Data Factory | Data integration inside the Microsoft Fabric platform | Your analytics platform is built on Fabric. Check current feature differences before choosing |
| Azure Databricks | Large-scale data engineering and machine learning with notebooks and code | Your team prefers code-heavy, advanced processing |
| SQL Server Integration Services | On-premises ETL tool | You have existing packages and a server environment |
| Azure Logic Apps | Workflow automation between applications | You need business process automation rather than heavy data movement |
| Azure Event Hubs and Stream Analytics | Real-time streaming data | Data must be processed continuously within seconds |
Use Data Factory as the manager that schedules and connects things, and use specialized engines such as Databricks or SQL for heavy custom processing when needed. Data Factory often calls those engines as activities inside a pipeline.
19Advantages and Trade-offs
An honest look helps you decide with open eyes.
Advantages
- Serverless, so there are no servers to patch or size
- Large library of connectors to cloud and on-premises sources
- Visual design that makes pipelines easy to understand and share
- Flexible triggers: schedules, time windows, and events
- Built-in monitoring, alerting, and rerun options
- Integrates with Git, Key Vault, managed identities, and private networking
Disadvantages / Trade-offs
- Not a storage or database itself, so it needs other services around it
- Data flow clusters can take time to start and add cost
- Not designed for true real-time streaming
- Large visual pipelines can become hard to manage without good structure
- Costs can grow quietly through many small activities or idle debug sessions
- Some advanced logic is easier in code-first tools
20Best Practices and Mistakes to Avoid
Habits that keep your pipelines reliable, secure, and affordable.
Design for reuse
Use parameters and metadata-driven designs so one pipeline serves many sources.
Make pipelines idempotent
Design so that running a pipeline twice gives the same result, not duplicate rows. Then reruns after failures are safe.
Build small, composable pipelines
Break large jobs into smaller pipelines that a parent pipeline calls, so each part is easy to test and understand.
Handle errors and alert people
Add failure paths, sensible retries, and alerts to every important pipeline.
Use Git and separate environments
Keep development, test, and production apart and deploy through automated releases.
Secure connections properly
Prefer managed identities, store secrets in Key Vault, and use private connectivity for sensitive data.
Name things clearly
Use consistent names that show purpose and type, so teammates can understand pipelines at a glance.
Problem
Copying the full source table every night, even when only a small part changes.
Why It’s Harmful
Runs get slower as data grows, costs rise, and source systems are put under needless load.
Correct Approach
Use incremental loading with watermarks, change tracking, or time windows, and do a full load only when truly needed.
Problem
Building one huge pipeline with hundreds of activities that does everything.
Why It’s Harmful
It is hard to read, hard to test, slow to open, and one failure can block unrelated work.
Correct Approach
Split work into smaller pipelines and orchestrate them with a parent pipeline using Execute Pipeline activities.
Problem
Editing production pipelines directly in the live factory with no source control.
Why It’s Harmful
There is no history, no review, and no easy way to go back after a bad change. Differences between environments also grow unnoticed.
Correct Approach
Connect Git, work in branches, review changes, and deploy to production through an automated release.
Problem
Using data flows for every task, even simple file copying.
Why It’s Harmful
Cluster startup time and compute cost make simple jobs slower and more expensive than they need to be.
Correct Approach
Use Copy activities for movement and reserve data flows for real transformation logic.
Problem
Having no alerts, so failed pipelines are discovered only when a report looks wrong.
Why It’s Harmful
Decisions get made on stale or partial data, and trust in the whole data platform falls.
Correct Approach
Configure alerts for failures and long runs, and review the Monitor area regularly.
21Frequently Asked Questions
No. It moves and transforms data between other services but does not keep your business data itself. It stores only the definitions of your pipelines and run history. Your data stays in sources and destinations such as storage accounts and databases.
Not necessarily. You can build many pipelines and transformations visually. Code is optional and useful for special cases, such as calling notebooks, stored procedures, or writing custom expressions.
A pipeline is the overall workflow made of activities and control logic. A data flow is one kind of activity inside a pipeline that performs visual data transformations on a Spark engine.
Install a self-hosted integration runtime on a machine inside your network. It connects out to Azure, receives instructions from Data Factory, and moves data from the local database securely.
It is designed mainly for batch and scheduled or event-driven processing. For continuous streaming within seconds, services such as Event Hubs and Stream Analytics are usually a better fit, and Data Factory can work alongside them.
Design the pipeline to be idempotent. For example, use upserts or delete-and-reload for the affected time window, and update watermarks only after a successful run. Then rerunning gives the same result instead of duplicates.
Microsoft Fabric includes its own Data Factory experience built into the Fabric platform, with similar ideas such as pipelines and dataflows. Azure Data Factory remains a standalone Azure service. Features differ and evolve, so compare them against your needs in the current documentation before choosing.
You pay for what you use, mainly pipeline orchestration, data movement, and data flow execution, plus any self-hosted machines you run. The total depends on how often pipelines run and how much data they handle. Use the Azure pricing calculator to estimate.
22Summary and Key Takeaways
Azure Data Factory is a serverless cloud service for building, scheduling, and monitoring data integration workflows. Pipelines hold activities that copy, transform, and control the flow of data. Linked services and datasets describe where data lives and what it looks like. Triggers decide when work starts, and integration runtimes provide the computing power, including self-hosted runtimes for private networks. With parameters you can reuse one design for many jobs, with incremental loading you move only what changed, and with monitoring, security, source control, and cost awareness you can run it safely at scale. Think of Data Factory as the conductor that coordinates your data platform, and it will quietly deliver the right data to the right place at the right time.
Key Takeaways
- Data Factory orchestrates, it does not store — it moves and transforms data between other services using ETL and ELT patterns.
- Know the six building blocks — pipelines, activities, datasets, linked services, triggers, and integration runtimes.
- Pick the right runtime — Azure IR for cloud work, self-hosted IR for private networks, and Azure-SSIS IR for existing SSIS packages.
- Choose the right activity — Copy for movement, data flows for visual transformation, and control activities for logic and error handling.
- Make pipelines flexible and efficient — use parameters, metadata-driven designs, and incremental loading instead of copying everything.
- Secure and automate — use managed identities, Key Vault, private connectivity, Git, and automated deployments across environments.
- Watch and tune — set up alerts, monitor runs, make pipelines safe to rerun, and track costs so surprises stay small.