Cloud Data Fusion is a fully managed service on Google Cloud for building and running data integration pipelines through a visual, drag-and-drop interface.

The Short Definition

Data integration means collecting data from many different places, such as databases, files, applications and message streams, cleaning it, reshaping it and delivering it to a destination where people can analyse it. This work is often called ETL, short for extract, transform, load, or ELT when the transformation happens after loading. Cloud Data Fusion lets you design these pipelines on a canvas by dropping boxes, called plugins, and connecting them with arrows. Under the surface, the service turns your drawing into a job that runs on a managed cluster of machines.

It is built on CDAP, an open-source data application framework. That matters for two reasons. First, the pipelines you draw are described in a portable format rather than being locked inside a closed tool. Second, there is a large library of ready-made plugins, so most common sources and destinations already have a box waiting for you.

Why It Exists

Moving and shaping data is one of the most repetitive and error-prone tasks in any organisation. Teams used to hand-write scripts for each source, schedule them with ad hoc tools, and struggle to remember what each script did. When the person who wrote a script left, nobody understood it. Cloud Data Fusion exists so that the structure of a pipeline is visible at a glance, so that people who are not expert programmers can build and maintain pipelines, and so that the infrastructure underneath is created and removed automatically.

Where It Is Used

Typical uses include loading data from operational databases into a warehouse such as BigQuery, consolidating files from many partners into one clean table, preparing data for reports and machine learning, migrating data from on-premises systems to the cloud, and keeping a copy of a database continuously updated for analysis.

Before the tool makes sense, the problem it solves must be clear.

The Journey of Data

Data is created in many places: a shop’s checkout system, a mobile app, a payroll package, a partner’s spreadsheet. Each place stores data in its own format and its own style. One system writes dates as day-month-year, another as year-month-day. One calls a customer “client_id,” another “cust_no.” Reports that mix them are wrong or impossible until the data is aligned.

The Three Steps

Batch versus Streaming

A batch pipeline processes a chunk of data at a time, for example everything from yesterday, on a schedule. A streaming or real-time pipeline processes events continuously as they arrive. Batch is simpler and cheaper. Streaming gives fresher results. Most organisations use mostly batch with a few streaming flows where freshness truly matters.

Ten words unlock almost every conversation and document about Cloud Data Fusion.

flowchart TD
  I["Instance"] --> N1["Namespace: finance"]
  I --> N2["Namespace: marketing"]
  N1 --> P1["Pipeline: daily sales load"]
  N1 --> CN["Connections: database, storage"]
  N2 --> P2["Pipeline: campaign clean-up"]
  P1 --> PL1["Source plugin"]
  P1 --> PL2["Transform plugins"]
  P1 --> PL3["Sink plugin"]
  P1 --> CP["Compute profile"]
        

The most important idea is that designing a pipeline and running it are separate things done by separate machinery.

Design Time and Run Time

When you open Cloud Data Fusion in your browser, you are talking to the instance. The instance stores your pipeline designs, shows the web interface, keeps metadata and history, and decides when to start jobs. It does not itself process your big data. When a pipeline runs, the instance asks another service, most commonly Dataproc, to create a temporary cluster of machines. The pipeline executes on that cluster, using a distributed processing engine, usually Apache Spark. When the run finishes, the cluster is deleted so that you stop paying for it.

The Main Parts

flowchart TD
  U["Developer in browser"] --> IN["Cloud Data Fusion instance: designer, scheduler, metadata"]
  IN --> PR["Compute profile"]
  PR --> DP["Temporary Dataproc cluster in your project"]
  DP --> SRC["Read from sources"]
  SRC --> TR["Transform with Spark"]
  TR --> SNK["Write to sinks"]
  DP --> DEL["Cluster deleted after run"]
  IN --> MD["Metadata and lineage"]
        

Why This Design Is Useful

Because processing happens on separate machines, one huge pipeline cannot freeze the designer for everyone else. Because clusters are temporary, you pay for compute only while data is being processed. Because the pipeline is described independently of the cluster, the same pipeline can run on a small cluster for a test and a large one for production just by changing the compute profile.

Cloud Data Fusion is offered in editions that trade price against capabilities. Choose by what your pipelines need.

The exact list of features in each edition, along with prices and limits, changes over time, so always read the current edition comparison page before you decide. As a general rule, start a trial in the Developer edition, move to Basic for dependable batch work, and choose Enterprise when you need streaming, replication or the strictest availability and governance requirements.

How to Choose

Pipeline Studio is where most of the daily work happens. It feels like drawing a flow chart that actually runs.

The Canvas and the Palette

On the left is a palette of plugins grouped by kind. In the middle is a blank canvas. You drag a source plugin onto the canvas, then drag a transformation, then a destination, and join them with arrows showing the direction of the data. Each box has a settings panel where you give it a name, point it at a connection, choose fields and set options. The designer shows the list of fields that flows out of each box, so you can see how the data changes shape at every step.

Kinds of Plugins

The Life of a Design

Real data is messy. Wrangler lets you see the mess and fix it with clicks instead of guesses.

What Wrangler Does

Wrangler loads a small sample of a dataset into a spreadsheet-like view. You can click on a column and choose actions such as split it into parts, change its type, fix capitalisation, fill missing values, remove rows that do not meet a rule, mask sensitive values or parse structured text. Every action you take is recorded as a step, called a directive. When you are happy, you add the whole recipe to a pipeline as a single transformation box that applies the same steps to every row, not just the sample.

Common Cleaning Tasks

A pipeline is only as useful as the places it can reach.

Reaching Data

Cloud Data Fusion includes plugins for a wide range of systems. Exact availability varies by version and edition, and additional plugins can be installed from a library called the Hub or added by your own developers. The families below are the ones most teams start with.

Connections

Rather than typing the address and password into every box, you can create a named connection once, such as “Sales database,” and reuse it in many pipelines. Changing the password then happens in one place. Connections are also where you test that the instance can actually reach the system.

Custom Plugins and Drivers

If no ready-made plugin exists, your developers can write one and upload it, and database drivers can be uploaded so the generic database plugin can talk to a specific product. This keeps the visual approach available even for unusual systems, though it does need developer time.

Transformations are the heart of a pipeline, where raw becomes useful.

Row-by-Row Changes

Some steps look at one row at a time. Renaming fields, converting a text value to a number, calculating a total from price and quantity, extracting the year from a date, replacing codes with names, and removing rows that fail a rule are all row-level changes. They are cheap and can be spread evenly across the cluster.

Changes Across Rows

Other steps need to see many rows together. Group by collects rows with the same key and calculates totals, averages or counts. Joiner combines two inputs on a matching field, much like matching customers to orders. Deduplicate keeps one row for each key. Distinct removes repeated rows. Window and ranking steps compare each row with its neighbours. These steps require data to be reshuffled across the cluster, so they cost more time and memory.

Working with Joins Wisely

When one input is small, such as a list of country codes, and the other is huge, Cloud Data Fusion can keep the small input in memory on every worker so the huge one does not need reshuffling. This is often called a broadcast or in-memory join. When both inputs are large, a regular join is needed and key balance becomes important: if one key has far more rows than the rest, one worker does most of the work while the others wait.

Using the Warehouse for Heavy Lifting

For very large aggregations, some teams load data into BigQuery first and let it do the heavy transformation with SQL, then use Data Fusion mainly to move and orchestrate. A pushdown feature in some versions lets certain pipeline steps run inside BigQuery directly. Choosing where the heavy work happens is one of the most useful design decisions you will make.

A compute profile tells Cloud Data Fusion where to run a pipeline and how much machinery to use.

Kinds of Profiles

What Happens When You Press Run

sequenceDiagram
  participant D as Developer
  participant I as Data Fusion instance
  participant P as Dataproc
  participant S as Sources
  participant K as Sinks
  D->>I: Start pipeline run
  I->>P: Create cluster from profile
  P-->>I: Cluster ready
  I->>P: Submit pipeline program
  P->>S: Read data
  S-->>P: Records
  P->>P: Transform in parallel
  P->>K: Write results
  P-->>I: Report status and counts
  I->>P: Delete cluster
  I-->>D: Run finished with metrics
        

Start-up Time Matters

Because a cluster is created from nothing, even a tiny pipeline spends minutes starting up. For many short runs a day, this overhead can dominate. In such cases, a long-lived cluster, or combining several small jobs into one run, may be more efficient. For big nightly jobs, the start-up cost is negligible.

One well-designed pipeline can replace dozens of near-identical copies.

Macros

A macro is a named blank in a pipeline setting that is filled in when the pipeline starts. A file path could contain a date placeholder, a database name could be a placeholder, a threshold for rejecting rows could be a placeholder. At run time, values are supplied by a person, by a schedule, by a trigger from another pipeline, or by an external orchestrator.

Why Reuse Beats Copying

Preferences and Secure Values

Values such as environment names can be stored as preferences at instance or namespace level and shared across pipelines. Passwords and keys should not be typed into plain macros. Use the secure storage options provided by the service or integrate with Google Cloud’s secret management so that secrets are not visible in pipeline definitions.

A pipeline is only valuable if it runs reliably without anybody pressing a button.

Built-In Scheduling

Cloud Data Fusion can run deployed pipelines on a time schedule, such as every night at one o’clock, or on a trigger, such as “start this pipeline when that one succeeds.” Triggers let you chain pipelines into small workflows and pass values between them.

When You Need a Bigger Conductor

For complex workflows that span many systems, with branching, retries, dependencies on external events and notifications, many teams use Cloud Composer, a managed workflow orchestrator, to coordinate Data Fusion pipelines along with BigQuery jobs, file checks and other tasks. Composer decides what runs when, and Data Fusion does the data moving.

flowchart LR
  EV["File arrives or timer fires"] --> OR["Orchestrator: Composer or schedule"]
  OR --> P1["Pipeline 1: ingest and clean"]
  P1 --> P2["Pipeline 2: join and aggregate"]
  P2 --> BQ["BigQuery reporting tables"]
  BQ --> CHK["Quality check"]
  CHK --> OK["Notify success"]
  CHK --> BAD["Alert and stop"]
        

When data cannot wait for tonight’s batch, streaming pipelines process events as they arrive.

How They Differ

A real-time pipeline runs continuously on a long-lived cluster. It reads from a stream such as Pub/Sub or a Kafka-style system, processes small groups of events, often called micro-batches, and writes them to a destination such as BigQuery. Because it never finishes, you monitor it rather than wait for completion. Availability of real-time pipelines depends on the edition, so check the current edition comparison.

Design Concerns

Sometimes the goal is not to transform data, but to keep a faithful copy of a database continuously available for analysis.

The Idea of Change Data Capture

Operational databases keep a log of every change. Replication reads that log and applies the same changes to a destination such as BigQuery, so the analytical copy stays close to current without repeatedly reading the whole source. This puts far less load on the source database than frequent full extractions. Cloud Data Fusion offers a replication feature for supported databases, generally in higher editions, and Google Cloud also provides separate services for change data capture, so compare options for your source and destination.

Replication versus Pipelines

Trust in data depends on knowing where it came from and whether it is right.

Metadata and Lineage

Cloud Data Fusion records which datasets were read and written by each run, what the schemas were and how fields flowed through the steps. This is visible as a lineage view. When a number on a report looks wrong, lineage lets you trace it backwards to the source field and the exact transformation that touched it. It also helps with impact analysis: before changing a source field, you can see which pipelines and outputs depend on it.

Data Quality in Practice

Data pipelines touch your most sensitive systems. Security must be designed in from day one.

Who Can Do What

Access to the service is controlled with Google Cloud’s identity and access management. Different roles exist for people who administer instances, those who design and edit pipelines, those who only run them and those who only view results. Giving each person the smallest role that lets them do their job reduces risk. Separately, the service accounts used by the instance and by the clusters need permissions to read sources and write sinks, and these should be limited to exactly the data they need.

Two Identities to Understand

Networking Choices

By default, an instance can be reached through a public interface protected by authentication. For stricter requirements, you can create a private instance that is reachable only from your own private network, using private connectivity between Google’s managed project and yours. Private instances are important when sources are on-premises or when policy forbids public exposure. They require careful planning of address ranges, firewall rules and name resolution, and they should be tested before production.

Protecting the Data

flowchart TD
  USR["Authorised users with IAM roles"] --> INS["Private Data Fusion instance"]
  INS --> NET["Private network connection"]
  NET --> CL["Dataproc cluster with dedicated service account"]
  CL --> SEC["Secret storage for passwords"]
  CL --> SRC["On-premises or cloud sources"]
  CL --> DST["BigQuery with column controls"]
  AUD["Audit logs"] -.-> INS
  AUD -.-> CL
        

Pipelines will fail sometimes. What matters is how quickly you notice and fix them.

Where to Look

Each run has a status, start and end times, record counts for every step and a log. Records in and out per box tell you where data is being lost or multiplied. Logs show errors from the program and from the cluster. Cloud Logging and Cloud Monitoring can collect the same information and drive alerts, dashboards and reports across all your pipelines.

Common Failure Causes

Alerting

Attach alerts to important pipelines so that failures send a message to a team channel or email. Also alert on the absence of success, for example no successful run in the last day, because a pipeline that never starts raises no failure. Include enough context in the message, such as pipeline name, run identifier and a link, so the person on call can act quickly.

Faster pipelines finish earlier, cost less and are easier to schedule.

The Main Levers

Cost comes from two places: keeping the instance available and the compute used when pipelines run.

The Cost Pieces

Exact prices, free allowances and edition rates change and vary by region, so always use the current pricing page and the Google Cloud pricing calculator for estimates. The practical point is that edition choice sets a baseline, while pipeline design determines the rest.

Ways to Save

Pipelines are software, and deserve the same care as application code.

Separate Environments

Use separate instances or at least separate namespaces and projects for development, testing and production. Developers experiment freely in development, a test environment uses realistic but safe data, and production changes are deliberate and reviewed. Keep production credentials out of reach of casual experiments.

Exporting and Versioning

Every pipeline can be exported as a text definition file. Store these files in a version control system so that changes are tracked, compared and reviewed, and so that you can roll back to an earlier version. Automated deployment tools can import the definitions into the next environment with environment-specific values passed in as parameters, giving a repeatable promotion path.

flowchart LR
  DEV["Development instance: design and preview"] --> EXP["Export pipeline definition"]
  EXP --> VC["Version control and review"]
  VC --> TST["Test environment: realistic data checks"]
  TST --> APP["Approval"]
  APP --> PRD["Production instance: scheduled runs"]
  PRD --> MON["Monitoring and alerts"]
  MON -.-> DEV
        

Team Habits That Help

Every approach has strengths and costs. Here is an honest balance sheet.

The Decision in One Sentence

Choose Cloud Data Fusion when you want a visual, governed way to move and prepare data from many different systems, and prefer drawing and configuring pipelines to writing and maintaining code.

Google Cloud offers several ways to move and transform data. Choose by the job and the team.

A Simple Way to Decide

Ask four questions. First, does your team prefer visual design over code? If so, Data Fusion is attractive. Second, is the data already in BigQuery and the changes expressible in SQL? Then use SQL tools and skip a separate integration service. Third, do you need heavily customised streaming logic at very high scale? Look at Dataflow. Fourth, do you only need a continuous copy of a database? Consider Datastream or the replication feature. Many mature platforms combine several of these, each doing what it does best.

A pattern is a proven shape for a solution. These recur in healthy Data Fusion environments.

flowchart TD
  SRC["Sources: databases, files, SaaS"] --> RAW["Raw landing in Cloud Storage or BigQuery"]
  RAW --> CLN["Cleaning pipeline with reject path"]
  CLN --> STG["Staging tables"]
  STG --> QC["Quality checks"]
  QC -->|"pass"| PUB["Published reporting tables"]
  QC -->|"fail"| ALR["Alert and hold"]
  CLN --> REJ["Rejected rows with reasons"]
  PUB --> BI["Dashboards and analytics"]
  PUB --> ML["Machine learning"]
        

An anti-pattern looks like a good idea but quietly causes trouble. These are the most common.

A checklist of habits shared by teams that run Data Fusion calmly.

Start Small, Then Grow

Begin with one valuable pipeline, such as loading a single important table into BigQuery. Get its network, permissions, scheduling, alerts and cost visible and understood. Then copy the proven approach for the next pipeline. Teams that start with fifty pipelines at once find the same setup problems fifty times.

Plan for Change

Sources change: columns are added, renamed or removed. Decide in advance how pipelines will react. Some can tolerate extra fields, others should fail loudly so someone fixes them. Either way, make the behaviour a decision, not an accident.

Plan for Upgrades

The service is managed, but versions of the platform and plugins still evolve. Keep a test environment where you can try an upgrade and run your key pipelines before applying it to production, and read the release notes for changes that affect you.

These are illustrative scenarios showing how organisations typically use Cloud Data Fusion, not claims about specific companies.

No. It is an integration service that moves and prepares data between systems. The data itself lives in your sources and destinations.

Most pipelines can be built visually without code. For unusual sources or very custom logic, developers can write plugins or use steps that accept small snippets of logic.

Usually a temporary Dataproc cluster in your project, running the pipeline as a Spark program. The instance orchestrates and records, and the cluster does the heavy processing.

Because a cluster must be created before the pipeline starts and removed after. For many small frequent jobs, consider combining them or using an existing cluster.

Developer for learning, Basic for production batch pipelines, and Enterprise when you need streaming, replication or the strictest availability and governance. Confirm the current feature list before deciding.

It is an interactive area for exploring a sample of data and building cleaning steps by clicking. The steps become a transformation you can add to a pipeline.

Use macros for values that change, and supply their values at run time through schedules, triggers or an orchestrator.

Yes, using private connectivity between your network and Google Cloud, and a private instance if policy requires. Plan the network path carefully and test it early.

You pay for the instance for the time it exists, at a rate depending on its edition, plus the compute and services used when pipelines run. Use the current pricing page and calculator for estimates.

Use lineage to trace fields back through the pipeline steps to the source. Combine it with good naming and documentation.

Real-time pipelines are available in editions that support them. Consider whether frequent batch runs would meet your need more simply and cheaply.

Data Fusion suits visual, connector-rich work by mixed teams. Dataflow suits developers who want code-level control and serverless scaling, especially for demanding streaming. Many organisations use both for different jobs.

Export the pipeline definition, store it in version control, review it, and import it into the target environment with environment-specific values supplied as parameters.

Skipping design discipline: hard-coding values, ignoring rejected rows, leaving idle instances running and skipping alerts. Build the habits early, while the first pipelines are small.

Cloud Data Fusion is Google Cloud’s managed, visual data integration service, built on the open-source CDAP framework. You design pipelines in Pipeline Studio by connecting plugins for sources, transformations and destinations, and you clean data interactively in Wrangler. The always-available instance stores designs, schedules runs and records metadata, while the actual work happens on temporary Dataproc clusters that appear for a run and disappear afterwards. Editions let you match capabilities and cost to your needs, from learning to enterprise production with streaming and replication.

The habits that matter most are simple. Parameterise with macros, preview with representative samples, filter early, keep rejected rows, secure credentials and use narrow service accounts, monitor and alert on both failures and missing successes, and manage cost by deleting idle instances and right-sizing clusters. Combine Data Fusion with BigQuery for warehouse-side transformation and with an orchestrator when workflows grow. With these practices, a small team can build clear, trustworthy data pipelines that anyone can read at a glance.