Before looking at features, let us understand what Cloud Data Fusion is and the problem it solves.

The one-sentence definition

Cloud Data Fusion is a fully managed, cloud-native data integration service on Google Cloud that lets you build and manage data pipelines using a visual, point-and-click interface instead of writing code. It is built on CDAP, an open-source data integration framework, so the pipelines you create are not locked into a proprietary black box. You design the flow of data visually, and the service runs it on managed processing clusters.

What is data integration?

Data integration means bringing data from many different places into one useful, consistent form. A company may have customer records in a database, orders in files, and web activity in a message feed. To analyze them together, someone must collect, clean, combine and load them into a warehouse. This work is often described as ETL, which stands for extract, transform and load, or ELT, where the loading happens before the transformation.

Who it is for

Understanding the pain it removes makes its design choices easy to follow.

Building data pipelines by hand is slow. Each source needs its own connection code, each transformation needs careful testing, and every pipeline needs logging, error handling and scheduling. Teams ended up with large collections of scripts that only their authors understood, and whenever those authors left, nobody dared to touch them. At the same time, business teams waited weeks for simple new reports because data engineers were overloaded.

Visual data integration tools had existed for years, but many were heavy, expensive products that ran on dedicated servers. The CDAP project offered an open-source, extensible alternative. Google adopted it and offered it as a managed service, so that organizations could get a visual integration tool with the scale and convenience of the cloud and without running their own servers.

A handful of terms appear everywhere in Cloud Data Fusion. Learn these and the rest feels familiar.

flowchart LR
  A[Source Plugin] --> B[Transform Plugin]
  B --> C[Transform Plugin]
  C --> D[Sink Plugin]
  C --> E[Error Collector]
  E --> F[Error Sink]
        

The friendly canvas hides an important split between the design tool and the processing engine. Understanding it explains costs, networking and failures.

Control plane and execution plane

Cloud Data Fusion has two main parts. The first is the instance itself, which hosts the web interface, stores your pipeline designs, tracks metadata and schedules runs. It is managed by Google, and in a typical setup it runs in a separate Google-managed project connected to yours. The second part is the execution environment: when a pipeline runs, Cloud Data Fusion typically translates it into jobs that run on a Dataproc cluster in your own project, using Apache Spark as the processing engine.

flowchart TD
  U[Pipeline Designer in Browser] --> I[Cloud Data Fusion Instance - Design, Metadata, Scheduling]
  I --> P[Compute Profile]
  P --> D[Dataproc Cluster in Your Project]
  D --> S1[Sources]
  D --> S2[Sinks]
  I --> M[Metadata and Lineage Store]
        

Ephemeral clusters

By default, many setups use ephemeral clusters. When a pipeline run starts, Cloud Data Fusion creates a fresh Dataproc cluster, runs the job, and deletes the cluster when it finishes. This means you pay for processing only while pipelines run, and each run gets a clean environment. The trade-off is a startup delay of a few minutes each time a cluster is created. For frequent or time-critical runs, you can instead point pipelines at an existing, long-running cluster.

The path of a pipeline run

Cloud Data Fusion comes in more than one edition. Picking the right one avoids both overspending and missing features.

The exact feature split between editions can change over time, so always confirm the current comparison in Google Cloud documentation before choosing. In general, start with Developer while learning, use Basic for straightforward batch integration in production, and move to Enterprise when you need real-time pipelines, change data replication or stronger availability.

Other instance choices

The studio is where most of your time will be spent, so let us see how a pipeline comes together.

The canvas and the plugin palette

On the left sits a palette of plugins grouped by type. You drag a plugin onto the canvas, connect it to others with arrows, and open its properties to fill in the details, such as which table to read or which column to filter. The studio checks the settings and shows the data structure, called the schema, that flows out of each block, so you can see how data changes along the way.

Plugin types

Common transformations in plain words

Preview before you run

The studio has a preview mode that runs your pipeline on a small sample, without writing to real destinations, and shows the data after each step. This is the fastest way to find mistakes, such as a wrong column name or a join that produces no matches. Use it often while building.

Data from the real world is rarely clean. Wrangler turns cleanup into a visual, step-by-step activity.

How Wrangler works

You open a sample of your data, such as a file or a database table, and see it as a grid. You then apply steps by clicking on columns and choosing actions: split a full name into first and last, fix date formats, drop empty rows, change text case, or parse a messy text column into structured fields. Each action becomes a recorded step, called a directive, and the full list of steps can be attached to a pipeline so it is applied to all your data, not just the sample.

A pipeline is only useful if it can reach your data. Cloud Data Fusion ships with many connectors and lets you add more.

The Hub and custom plugins

The Hub is a catalog from which you can install more plugins, ready-made pipeline templates and sample solutions. If no plugin exists for a system you need, developers can write a custom plugin and upload it, so that analysts can then use it in the studio like any other block. Alternatively, a generic database connector using a driver you upload can reach many systems.

Reaching systems on-premises

To read from databases in your own data center, the compute running your pipeline must have a network route to them. This is normally arranged with Cloud VPN or Cloud Interconnect, together with private networking for the instance and a compute profile in the right network. Planning this early avoids many connection failures later.

Cloud Data Fusion supports more than one style of data movement. Each fits a different need.

flowchart TD
  Q1{Do you only need to mirror a database to a warehouse?} -->|Yes| R[Replication]
  Q1 -->|No| Q2{Must results arrive within seconds?}
  Q2 -->|Yes| S[Real-Time Pipeline]
  Q2 -->|No| B[Batch Pipeline]
        

A pipeline’s speed and cost depend heavily on the cluster it runs on.

What a compute profile controls

A compute profile stores settings such as the type and number of machines, the disk size, the network and subnet, the region and the service account used by the cluster. You can create several profiles, for instance a small one for tests and a larger one for heavy production loads, and choose one for each pipeline or each run.

Tuning a pipeline

Pipelines rarely run once. They run on time, in order, and with the right inputs.

Built-in scheduling

You can schedule a deployed pipeline to run at set times, such as every night at two o’clock. You can also set triggers so that one pipeline starts when another finishes, either always or only when it succeeds or fails. This lets you chain a series of steps, such as ingest, then clean, then load, without outside tools.

Macros and runtime arguments

Hard-coding values such as a date or a file path makes pipelines inflexible. Macros let you leave a blank in a setting, and the value is supplied each run, either manually, from the schedule or from an earlier pipeline. For example, a daily load can receive “yesterday’s date” and read only that day’s files.

Orchestrating bigger workflows

When your workflow includes many different systems, not just Cloud Data Fusion, a dedicated orchestration service such as Cloud Composer, which is based on Apache Airflow, can start Cloud Data Fusion pipelines as one step among many. It also handles dependencies, retries and alerts across the whole workflow.

Knowing where data came from and what happened to it is a major reason teams choose Cloud Data Fusion.

Lineage

Cloud Data Fusion records lineage, a map showing how data moved through pipelines: which sources fed which datasets, which pipelines touched them, and how fields were changed along the way. This includes field-level lineage, which can show that a column in a report was built from specific columns in a source after specific transformations.

Metadata

Datasets and pipelines can be tagged and described, and the system collects technical information such as schemas and run histories. Search tools let people find datasets by name, tag or field, which reduces duplicate work and helps new team members learn the data landscape.

Pipelines touch valuable data, so security needs attention in several layers.

Identity and access

Access is controlled through Identity and Access Management. Predefined roles allow some people to administer instances, others to design and run pipelines, and others only to view. Besides human users, service accounts matter: the Cloud Data Fusion service uses one identity to manage resources on your behalf, and the Dataproc cluster runs as another, which needs permission to read your sources and write your destinations. Give each only what it needs.

Protecting sensitive data in pipelines

Use masking, hashing or removal of sensitive fields as early as possible, and make sure error outputs and logs do not store raw personal data. Applying these steps in a shared, reviewed pipeline gives consistent protection and a lineage record that proves it.

When a pipeline fails at three in the morning, clear signals decide how fast you recover.

A simple troubleshooting routine

Bad data is a fact of life. A good pipeline expects it and handles it gracefully.

flowchart LR
  A[Read Source] --> B[Validate Records]
  B -->|Valid| C[Transform and Enrich]
  B -->|Invalid| D[Error Collector]
  C --> E[Write to Destination]
  D --> F[Error Table for Review]
  F --> G[Fix and Replay]
  G --> B
        

Visual does not mean unmanaged. Professional teams treat pipelines like any other important software asset.

Exporting and importing pipelines

Every pipeline can be exported as a file that describes its design. You can store these files in a source control system, review changes, and import them into another instance. This is how teams move a pipeline from development to testing to production without redrawing it.

Promoting across environments

Cloud Data Fusion costs come from more than one place. Knowing them prevents bill shock.

Ways to keep costs under control

Several Google Cloud services move or transform data. Picking the right one saves effort.

flowchart TD
  Q1{Is data already in BigQuery and SQL enough?} -->|Yes| A1[BigQuery SQL]
  Q1 -->|No| Q2{Only need a database copy into the cloud?}
  Q2 -->|Yes| A2[Datastream or Data Fusion Replication]
  Q2 -->|No| Q3{Prefer visual building and many ready connectors?}
  Q3 -->|Yes| A3[Cloud Data Fusion]
  Q3 -->|No| Q4{Need custom streaming logic or huge scale?}
  Q4 -->|Yes| A4[Dataflow]
  Q4 -->|No| A5[Dataproc for existing Spark code]
        

Patterns capture what experienced teams repeatedly find works; anti-patterns capture what goes wrong.

Helpful patterns

Anti-patterns to avoid

An honest view of strengths, limits and where teams actually use it.

Real-world examples

A checklist to carry into every project, followed by answers to common beginner questions.

Not for most common tasks. You build pipelines visually with ready-made plugins. Coding is only needed for custom plugins or very specialized logic.

Cloud Data Fusion is a managed Google Cloud service built on CDAP, an open-source framework for data integration. This is why plugins and pipelines have an open foundation rather than being fully proprietary.

Typically on Dataproc clusters in your own project, using Apache Spark. The instance only designs, schedules and tracks them.

With ephemeral clusters, a new cluster must be created for each run, which takes a few minutes. Use an existing cluster if you need faster starts, at the cost of paying for it while idle.

Cloud Data Fusion is a visual, plugin-driven tool running on Spark clusters. Dataflow is a code-based, serverless service built for custom logic and advanced streaming. Many organizations use both for different jobs.

Yes, real-time pipelines are supported in higher editions, though the transformations available in streaming mode are more limited than in batch. Check current documentation for details.

Provide a network path with Cloud VPN or Cloud Interconnect, use a private instance and a compute profile in the right network, and make sure firewall rules allow the cluster to reach the database.

Use lineage, including field-level lineage, to trace the report’s columns back through each pipeline and transformation to the original source fields.

Export the pipeline file, keep it in source control, review it, and import it into the production instance through the interface or the API, supplying environment-specific values through arguments.

You decide. With the error collector, bad records can be routed with their error reason to a separate destination while good records continue, and you can set thresholds that stop the run if too many fail.

The instance, billed hourly by edition even when idle, plus the Dataproc clusters used during runs, plus the services you read from and write to, and network transfer.

Often not. The continuous instance fee can outweigh the value. Lighter tools, such as scheduled queries or transfer services, may be more economical for small, simple needs.

Cloud Data Fusion is Google Cloud’s managed, visual data integration service. You design pipelines by dragging plugins onto a canvas, clean data interactively with Wrangler, preview results on samples, and then run the pipelines on managed Dataproc clusters that can be created and deleted for each run. Built on the open-source CDAP framework, it offers a large library of connectors, scheduling and triggers, lineage and metadata for governance, and support for batch, real-time and replication styles in the appropriate editions. Its strengths are speed of development, accessibility and traceability; its trade-offs are the continuous instance fee, cluster startup time and the awkwardness of very custom logic. Success comes from choosing the right edition, designing small and rerunnable pipelines, handling bad data deliberately, securing identities and networks, versioning pipelines like code and watching both run status and record counts.