Before diving into settings and concepts, let us understand what Dataflow is and what job it does.

The one-sentence definition

Dataflow is Google Cloud’s fully managed service for running data processing pipelines. You write your pipeline using the open-source Apache Beam programming model, and Dataflow takes care of starting machines, dividing the work, recovering from failures and scaling up or down. It handles both batch processing, where the data is a fixed, finite set such as yesterday’s log files, and streaming processing, where the data keeps arriving and the pipeline never finishes, such as clicks on a website.

What is a data pipeline?

A data pipeline is a series of steps that moves data from a source, changes it in useful ways, and delivers it to a destination. Typical steps include reading, cleaning, filtering, joining with other data, counting, grouping and writing results. Pipelines power dashboards, alerts, recommendations, fraud checks and machine learning features.

Where Dataflow fits

Dataflow often sits between data sources and data stores. Messages come in from services such as Pub/Sub or Kafka, files come from Cloud Storage, and results go to BigQuery, Bigtable, Spanner, Cloud Storage or other systems. Because it is serverless in feel, you manage the pipeline logic, not the servers.

Understanding the problem Dataflow was born to solve makes its design choices feel natural.

In the early days of big data, processing large datasets meant writing batch jobs that ran overnight. If you also wanted live results, you built a second, separate system for streaming. Teams ended up maintaining two code bases for the same logic, and the results from the two systems often disagreed. Google faced this problem at huge scale and developed internal systems and research, including a well-known paper on the Dataflow Model, that argued for one unified approach: treat batch as a special case of streaming, and give users clear tools to reason about time and late-arriving data.

Google then made Dataflow available as a cloud service, and later donated the programming model to the Apache Software Foundation as Apache Beam. Today, Dataflow runs Beam pipelines, and Beam has become one of the most popular ways to write portable data pipelines.

A few simple ideas explain almost everything you will meet in a Dataflow pipeline.

Bounded versus unbounded data

A bounded dataset has a clear beginning and end, such as a file of last month’s orders. An unbounded dataset has no end, such as a live feed of sensor readings. Beam treats both with the same transforms, which is what lets one pipeline design serve batch and streaming. The main difference is that with unbounded data you cannot wait for “all the data”, so you need extra tools to decide when to produce results. We cover these tools in the chapter on time.

A pipeline is a graph

Behind the scenes, your pipeline becomes a directed graph of steps. Dataflow looks at the graph and optimizes it, for instance by fusing neighboring steps so data does not have to travel between machines more than necessary. This graph is what you see in the Dataflow monitoring page, with each box representing a step and arrows showing data flowing between them.

flowchart LR
  A[Read from Source] --> B[Parse and Clean]
  B --> C[Filter Invalid Records]
  C --> D[Key by Customer]
  D --> E[Group and Combine]
  E --> F[Write to Destination]
  C --> G[Write Rejects to Dead-Letter Store]
        

When you submit a pipeline, several pieces of Google Cloud cooperate to run it.

The main pieces

flowchart TD
  A[Developer Pipeline Code] --> B[Dataflow Service Control Plane]
  B --> C[Worker VM 1]
  B --> D[Worker VM 2]
  B --> E[Worker VM N]
  C --> F[Shuffle or Streaming Engine Back End]
  D --> F
  E --> F
  C --> G[Sources and Sinks]
  D --> G
  E --> G
  B --> H[Monitoring and Logging]
        

What happens when you submit a job

Dataflow handles two styles of processing with the same basic building blocks. Knowing when to use each is the first design decision.

When to choose batch

Choose batch when results are needed on a schedule, when the data is naturally a set of files or tables, or when you must reprocess history. Batch jobs can use cheaper scheduling options and are simpler to reason about, because the whole dataset is available.

When to choose streaming

Choose streaming when freshness matters, such as when you must react within seconds to an event, update a dashboard live, or keep a store continuously in sync. Streaming jobs cost more over time because they run all day, but they deliver answers while they are still valuable.

The same logic, two modes

A strength of Beam is that the same transform logic can often be reused for both. A team may use a streaming job for live data and the same logic in a batch job to backfill history or fix past mistakes. This keeps results consistent and avoids the two-systems problem described earlier.

flowchart TD
  Q1{Must results be available within seconds or minutes?} -->|Yes| S[Streaming pipeline]
  Q1 -->|No| Q2{Is data a fixed set of files or tables?}
  Q2 -->|Yes| B[Batch pipeline]
  Q2 -->|No| Q3{Can you process in small scheduled chunks?}
  Q3 -->|Yes| B
  Q3 -->|No| S
        

This is the chapter that separates beginners from confident Dataflow users. Time in data pipelines is trickier than it looks.

Event time versus processing time

Event time is when something actually happened, for example when a user tapped a button. Processing time is when the pipeline sees the event. These can differ a lot: a phone with no signal may send events minutes or hours late. If you count events by the time they arrive, a tunnel on a train could distort your numbers. Beam lets you work in event time so results reflect reality.

Windows: cutting an endless stream into pieces

You cannot compute “the total” of data that never ends, so you divide it into windows, finite slices of time, and compute results per window.

Watermarks: how the pipeline estimates completeness

A watermark is the system’s best guess of how far event time has progressed, meaning the point before which it believes all data has arrived. It is an estimate, not a promise. When the watermark passes the end of a window, the pipeline considers the window mostly complete and can produce a result. Data that shows up after the watermark has passed is called late data.

Triggers: when to emit results

A trigger decides when results for a window are produced. The default is to emit when the watermark passes the end of the window. You can also emit early, giving partial results before the window is complete, or emit again when late data arrives. Triggers let you balance freshness against accuracy.

Late data and accumulation

You choose how long to accept late data, called allowed lateness. Data later than this is dropped. When late data arrives within the allowed time, the pipeline can fire the window again. The accumulation mode decides whether the new output replaces the old result or contains only the new changes. Downstream systems must be ready for corrections, for instance by overwriting a row rather than appending a duplicate.

sequenceDiagram
  participant E as Events
  participant W as Window 12:00 to 12:05
  participant T as Trigger
  participant O as Output
  E->>W: Events arrive with event times
  T->>O: Early result before window is complete
  Note over W: Watermark passes 12:05
  T->>O: Main result when window is considered complete
  E->>W: Late event arrives within allowed lateness
  T->>O: Updated result for the window
        

A peek under the hood explains why some pipelines are fast and others stall.

Splitting work into bundles

Dataflow divides your data into small chunks called bundles and hands them out to workers. Each worker processes bundles independently. If a worker fails or is slow, its bundle is retried or given to another worker. This is why a failed machine rarely ruins a job: the system just replays the work.

Fusion: keeping steps together

Moving data between machines is expensive. When several steps can run back to back on the same data, Dataflow fuses them into one stage so each element flows through all steps on the same worker without being written out in between. This is a big speed-up, though it can sometimes limit parallelism, which we discuss under performance.

Shuffle: regrouping data

Steps like GroupByKey and Combine need all values for the same key to end up in the same place. This regrouping is called a shuffle, and it is one of the most costly parts of any pipeline. In Dataflow, the shuffle for batch jobs is handled by a managed service rather than by the workers’ own disks, and streaming jobs can use the Streaming Engine for a similar benefit. This makes jobs faster, makes autoscaling smoother, and reduces the load and disk needs of workers.

flowchart LR
  A[Input Splits] --> B[Fused Stage: Read Parse Filter]
  B --> C[Shuffle by Key]
  C --> D[Fused Stage: Combine and Format]
  D --> E[Write Output]
        

Exactly-once processing

Dataflow aims to give exactly-once processing semantics for the data flowing through the pipeline, meaning each element’s effect is reflected once in the pipeline’s internal state even if a bundle is retried. This is different from guaranteeing that outside systems receive exactly one copy. If your code writes to an external system that does not recognize duplicate writes, a retry could cause a duplicate there. Using sinks that support idempotent writes, or including unique identifiers, protects you.

At-least-once streaming mode

For some streaming jobs where occasional duplicates are acceptable, Dataflow offers an at-least-once mode that can lower cost and latency by skipping some of the work needed for exactly-once guarantees. It suits workloads like simple event counting where small duplication is tolerable, or where the sink removes duplicates anyway.

A pipeline is only useful if it can read from and write to the places your data lives.

Reading and writing carefully

Connectors have settings that affect cost and correctness. For example, writing to BigQuery can use bulk file loads, which are cheaper and suit batch-like patterns, or a streaming write method for low latency at a different price. Choosing the right method is part of pipeline design.

Dead-letter handling

Real data is messy. Some records will be malformed or fail validation. Rather than letting one bad record crash the pipeline, a common practice is to send failed records to a separate destination, often called a dead-letter location, with the error details. Teams can inspect and replay them later.

You do not always need to write code. Templates let you launch common pipelines or package your own for reuse.

Google-provided templates

Google offers a library of ready-made templates for popular tasks, such as streaming from Pub/Sub to BigQuery, copying data between Cloud Storage and BigQuery, or moving data between databases. You pick a template, fill in parameters like source and destination, and launch it from the console or through automation. This is perfect for standard jobs and for teams without Beam experience.

Custom templates

You can also package your own pipelines as templates so others can launch them without touching the code. Flex Templates are the modern approach: your pipeline is packaged in a container image with its dependencies, and users launch it by supplying parameters. This separates the people who build the pipeline from the people who run it, and it fits well with automation and release processes.

One of Dataflow’s main promises is that it adjusts resources for you. Knowing how helps you get the best result.

Horizontal autoscaling

Dataflow can automatically add or remove worker machines based on how much work is waiting. In batch jobs it looks at the remaining work and the speed of progress. In streaming jobs it looks at backlog and CPU use. You set a maximum number of workers as a safety limit, so a surprise spike cannot create unlimited cost.

Dataflow Prime and vertical autoscaling

Dataflow Prime is an enhanced version of the service that adds features such as vertical autoscaling, which adjusts the memory of workers to the actual need to avoid out-of-memory failures, and better right-fitting of resources for individual steps. It can also help with resource needs that differ per step, for example a step that needs a GPU while others do not.

What affects speed

Streaming jobs run for weeks or months. You need safe ways to change and stop them.

Pipelines often touch sensitive data, so security deserves attention from the start.

Identity and permissions

Two identities matter. The first is the person or system that submits the job and needs permission to create Dataflow jobs. The second is the worker service account, the identity the worker machines run as, which needs permission to read sources, write sinks and perform Dataflow work. Use a dedicated service account for each pipeline with only the permissions it needs, rather than the broad default account.

Handling sensitive data inside pipelines

Pipelines often process personal data. Reduce risk by removing or masking fields you do not need, tokenizing identifiers, and never logging full records. Be careful with error messages and dead-letter outputs, since they may contain the very data you are trying to protect.

A pipeline that nobody watches will eventually fail quietly. Dataflow provides built-in tools to see what is happening.

Custom metrics

You can add your own counters and distributions inside the pipeline, for example the number of records rejected for each reason. These appear in the monitoring tools and are among the best ways to see data quality problems early.

A simple troubleshooting routine

Beyond classic data processing, Dataflow supports several newer workloads.

Dataflow is billed for resources used, so understanding the meters helps you keep bills predictable.

Ways to reduce cost

Several services overlap with Dataflow in some way. Picking the right one saves effort and money.

Dataflow or Dataproc?

If you already have Spark code and skills, Dataproc lets you keep them. If you are starting fresh, need streaming with strong event-time handling, or want a fully managed experience with minimal cluster care, Dataflow is often the more natural choice.

Dataflow or just SQL in BigQuery?

If your data is already in BigQuery and your transformations can be written in SQL, using BigQuery directly is usually simpler and cheaper. Choose Dataflow when you need to process data before it lands, handle continuous event streams with complex timing, call outside services, or work with sources and sinks beyond the warehouse.

flowchart TD
  Q1{Is the data already in BigQuery and SQL enough?} -->|Yes| A1[Use BigQuery SQL]
  Q1 -->|No| Q2{Existing Spark or Hadoop code?}
  Q2 -->|Yes| A2[Consider Dataproc]
  Q2 -->|No| Q3{Need streaming, windowing or custom logic?}
  Q3 -->|Yes| A3[Use Dataflow]
  Q3 -->|No| Q4{Prefer visual, code-free building?}
  Q4 -->|Yes| A4[Consider Cloud Data Fusion]
  Q4 -->|No| A3
        

Patterns capture what experienced teams repeatedly find works; anti-patterns capture what repeatedly goes wrong.

Helpful patterns

Anti-patterns to avoid

An honest summary of what Dataflow gives you and what it asks in return.

When to choose Dataflow

Choose Dataflow when you need to transform or enrich large or continuous data, want one approach for both batch and streaming, need accurate handling of event time and late data, or want to run pipelines without managing infrastructure.

When not to choose it

Avoid it when SQL in a warehouse is enough, when a built-in connection feature already moves your data, when you must run existing Spark code unchanged, or when the volume is so small that a simple scheduled script would do.

Seeing how organizations use Dataflow makes the concepts easier to remember.

A short checklist to carry into every Dataflow project.

Apache Beam is the open-source programming model you use to write pipelines. Dataflow is Google Cloud’s managed service that runs Beam pipelines, handling machines, scaling and recovery.

Beam supports Java, Python and Go, and Dataflow can run pipelines written in them. Some connectors and features are more mature in certain languages, so check current documentation for your needs.

The same transform logic can often be reused in both modes. The main difference is the source, bounded or unbounded, and the need to think about windows and triggers for streaming.

It is the system’s estimate of how far event time has progressed, meaning the point before which it believes nearly all data has arrived. It helps decide when a window is ready to produce results.

If it arrives within the allowed lateness, it can update the window’s result. If it arrives later than that, it is dropped, so choose your lateness setting deliberately and monitor drops.

It provides exactly-once processing semantics inside the pipeline. For outside systems you write to, you need idempotent writes or deduplication, because retries could otherwise create duplicates.

Drain stops reading new input and finishes processing what has already been received before shutting down. Cancel stops immediately, and buffered data may be lost.

Dataflow adds or removes workers based on backlog and utilization, within the maximum you set. Dataflow Prime adds vertical autoscaling for memory as well.

No. Google provides ready-made templates for common jobs, which you launch by supplying parameters. Writing your own Beam code is needed only for custom logic.

Common causes are hot keys, steps that cannot be parallelized, slow external services, or too much shuffled data. Use the job graph and metrics to find the bottleneck.

Yes. You can run workers with private addresses in your own subnet and use Private Google Access to reach Google services, which is a common security practice.

Set a maximum worker count, right-size machines, use FlexRS for delay-tolerant batch work, reduce shuffle volume, consider at-least-once streaming where suitable, and stop jobs you no longer need.

Dataflow is Google Cloud’s managed engine for running Apache Beam pipelines, giving you one programming model for both batch and streaming data. You describe what should happen to your data as a graph of transforms, and the service handles machines, scaling, shuffling, failures and monitoring. The hardest and most valuable ideas to learn are about time: event time, windows, watermarks, triggers and late data. Good results then come from sound design habits: choosing batch or streaming for real business reasons, pre-aggregating before shuffles, handling bad records with dead-letter paths, using least-privilege identities and private networking, packaging pipelines as templates, and watching lag and freshness. Used for the right jobs, it makes large-scale data processing dependable and surprisingly calm to operate.