A few simple ideas before we meet Dataflow.

Raw data is messy

Data rarely arrives ready to use. Website clicks come as long lines of text. Sensor readings arrive with missing values. Sales files from different countries use different date formats. Before anyone can build a report or train a model, the data must be cleaned, reshaped, combined with other data and stored somewhere convenient. This work is called data processing, or sometimes ETL, which stands for extract, transform and load.

Two shapes of data: batch and streaming

Batch data is a fixed pile that already exists, such as yesterday’s sales file. You process the whole pile and finish. Streaming data never stops arriving, such as payments, taps in a mobile app or readings from machines. You must process each piece as it comes, often within seconds, because the answer is most valuable right away.

Why processing at scale is hard

One computer can handle small data. Once data reaches millions or billions of records, the work must be split across many computers. That brings new problems: dividing the work fairly, recovering when a machine fails, combining partial results correctly and avoiding counting the same record twice. Solving these by hand is slow and error-prone.

A fully managed service for running batch and streaming data pipelines.

Dataflow is Google Cloud’s fully managed service for processing data, both batch and streaming. You write a pipeline, which is a description of the steps your data should pass through, and submit it to Dataflow. Dataflow then provisions the computers needed, divides the work among them, monitors them, replaces any that fail, and adds or removes computers as the workload changes. When the job ends, the computers are released.

Pipelines for Dataflow are written with Apache Beam, an open-source programming model. This is an important point: the code you write is Beam code, and Dataflow is one place, called a runner, where Beam code can run. The same pipeline can be written once and used for both batch and streaming data, which means teams learn one model instead of two.

A short dictionary of Beam and Dataflow words.

The handful of ideas that all pipelines are built from.

Element-by-element work

The most common transform applies a small piece of logic to each item independently, for example cleaning a text line or converting a currency. Because each item is handled on its own, the work can be spread across many workers with no coordination.

Grouping

Sometimes you need to bring related items together, such as all clicks from the same user. Grouping by key sends every item with the same key to the same place. This is more expensive than element-by-element work, because data must be moved between workers, so good pipelines group only when necessary.

Combining

Combining reduces many items into one result, such as a sum, a count or an average. Beam can do partial combining on each worker first and then merge the partial results, which saves a great deal of data movement.

Side inputs

Occasionally a step needs a small reference dataset, such as a list of currency rates. A side input makes that reference data available to every worker without grouping the main data.

Joins and branching

A pipeline does not have to be a straight line. It can split one collection into several branches, process them differently, and merge results back together, forming a graph of steps.

flowchart LR
    SRC[Read from source] --> CLEAN[Clean and parse each record]
    CLEAN --> VALID{Valid record?}
    VALID -->|Yes| KEY[Assign key]
    VALID -->|No| DEAD[Write to dead-letter storage]
    KEY --> GROUP[Group and combine by key]
    GROUP --> OUT[Write results to sink]
        

The ideas that make streaming correct, and the most important section for beginners to understand.

Two kinds of time

Event time is when something actually happened, such as when a customer tapped a button. Processing time is when your system learned about it. These can differ a lot: a phone in a tunnel may send a tap from five minutes ago once it reconnects. If you measure by processing time, your results are wrong. Beam lets you work in event time, which reflects reality.

Windows

A stream never ends, so you cannot wait for “all the data” before computing a total. A window slices the stream into chunks of time, so you can compute results per chunk. Fixed windows are equal, non-overlapping slices, such as every five minutes. Sliding windows overlap, such as the last hour updated every five minutes. Session windows group activity by bursts, closing after a gap of inactivity, which is ideal for user visits.

Watermarks

How does the system know when a window is complete? It uses a watermark, an estimate of how far event time has progressed, meaning “I believe I have now seen essentially all the data up to this moment.” When the watermark passes the end of a window, the system considers the window ready to produce its result.

Triggers and late data

Watermarks are estimates, so some data arrives late. Triggers control when results are emitted: when the watermark passes, earlier as early speculative results, or again when late data shows up. You also choose how long to keep accepting late data, called allowed lateness, and whether later results refine or replace earlier ones. These choices let you balance speed, accuracy and cost.

flowchart TD
    A[Events arrive with event timestamps] --> B[Assigned to windows by event time]
    B --> C{Watermark passed window end?}
    C -->|Not yet| D[Optional early results by trigger]
    C -->|Yes| E[Emit on-time result]
    E --> F{Late data within allowed lateness?}
    F -->|Yes| G[Emit updated result]
    F -->|No| H[Late data dropped or sent aside]
        

What happens between pressing “run” and seeing results.

sequenceDiagram
    participant Dev as Developer
    participant SVC as Dataflow Service
    participant W as Worker VMs
    participant SRC as Source
    participant SNK as Sink
    Dev->>SVC: Submit pipeline
    SVC->>SVC: Build and optimize job graph
    SVC->>W: Start workers
    W->>SRC: Read bundles of data
    W->>W: Transform in parallel
    W->>SNK: Write results
    SVC->>W: Add or remove workers as load changes
        

How Dataflow keeps up with changing workloads efficiently.

Why the numbers can be trusted even when machines fail.

Machines fail, networks hiccup, and workers are restarted. In such a world, how do we avoid losing data or counting it twice? Dataflow checkpoints its progress and state, so when a worker fails, another picks up from the last safe point. For streaming pipelines, Dataflow offers exactly-once processing semantics within the pipeline, meaning each record’s effect on the pipeline’s internal state is applied once, even if work was retried.

There is an important subtlety. Exactly-once inside the pipeline does not automatically mean exactly-once in the outside world. If your pipeline writes to an external system that cannot recognize duplicates, a retried write could appear twice. Good designs use connectors with built-in deduplication or writing methods that are safe to repeat, often called idempotent writes, and use unique identifiers to detect duplicates.

Where data comes from and where it ends up.

flowchart LR
    APPS[Apps and devices] --> PS[Pub/Sub]
    PS --> DF[Dataflow Streaming Pipeline]
    FILES[Files in Cloud Storage] --> DFB[Dataflow Batch Pipeline]
    DF --> BQ[(BigQuery)]
    DFB --> BQ
    DF --> BT[(Bigtable)]
    DF --> DL[Dead-letter storage]
    BQ --> BI[Dashboards and reports]
        

Reusing and sharing pipelines safely.

A template is a pipeline packaged so that it can be launched later, by you or by others, with different settings such as input location, output table or window size. Google provides many ready-made templates for common jobs, such as copying messages from a messaging topic into a warehouse table, moving files between storage systems, or loading data into a database. These let teams without Beam programming experience get started within minutes.

Teams can also build their own templates. The modern style is the flexible template, which packages the pipeline code in a container, so launching it does not require the original development environment. Templates are excellent for standardizing how pipelines run in an organization, controlling versions and letting operations teams launch jobs without touching code.

Choosing the right tool is half the battle.

Where Dataflow earns its keep.

Protecting the data as it flows.

Identity and permissions

Dataflow uses two kinds of identity. The person or system that launches a job needs permission to create it. The workers themselves run as a service account, and that account needs just enough permission to read sources and write sinks. Giving each pipeline its own tightly scoped service account follows the least-privilege principle and limits damage if something goes wrong.

Network protection

Workers can be placed in your own private network, and configured without public IP addresses so they are not reachable from the internet. Private Google Access lets them reach Google services without leaving private paths, and a managed NAT service can provide controlled outbound access to anywhere else. Firewall rules should allow the worker-to-worker traffic that Dataflow needs while blocking everything unnecessary.

Encryption

Data is encrypted at rest and in transit by default. If your rules require it, you can use customer-managed encryption keys to control who can decrypt pipeline data and state.

Data protection inside the pipeline

Pipelines can mask, tokenize or remove sensitive fields, such as card numbers and personal identifiers, before data lands in analytics systems. Dedicated data-protection services can detect sensitive values for you, and dead-letter storage should be protected as carefully as the main output, because it holds the raw bad records.

flowchart TD
    A[Least-privilege service accounts] --> B[Private network and no public worker IPs]
    B --> C[Firewall rules for worker traffic only]
    C --> D[Encryption at rest and in transit with optional customer keys]
    D --> E[Mask or remove sensitive fields in the pipeline]
    E --> F[Audit logs and monitoring]
        

A pipeline that runs 24 hours a day needs eyes on it.

Streaming health signals

Three numbers tell most of the story. Data freshness says how old the newest fully processed data is. System latency says how long items wait in the pipeline. Backlog says how much data is waiting. If freshness worsens and backlog grows, the pipeline is falling behind and needs more capacity or a fix to a slow step.

Updating and stopping jobs safely

Streaming jobs run continuously, so you need safe ways to change them. An in-place update can replace a running job with a compatible new version while keeping its state. Draining stops reading new data but finishes processing everything already received, so no in-flight data is lost, and is the usual way to shut down a streaming job. Cancelling stops immediately and may discard work in progress, so use it carefully.

You pay for the resources your jobs use, so efficiency saves money.

Dataflow is billed mainly for the compute resources workers consume, such as processor time and memory, plus storage attached to workers and, where used, the data handled by managed services such as Streaming Engine and shuffle. You also pay for the services your pipeline touches, such as messaging, storage and warehouse charges. Rates differ between batch and streaming and change over time, so always use Google Cloud’s pricing page and calculator.

A balanced view.

Lessons learned by teams who run pipelines every day.

Best practices at a glance

Anti-patterns to avoid

Apache Beam is the open-source programming model used to write pipelines. Dataflow is Google Cloud’s managed service that runs Beam pipelines. Beam is the recipe, and Dataflow is the kitchen.

No. Dataflow creates, scales and removes the worker machines for you. You choose settings such as region, machine type and maximum workers, but you do not operate a cluster.

Beam’s model is designed so the same transforms can be applied to bounded and unbounded data, though some choices, such as windowing and triggers, matter more for streaming. Many teams reuse core logic across both.

Beam offers software kits for Java, Python and Go, and supports mixing languages through cross-language transforms. Check current documentation for supported versions and feature differences between languages.

A watermark is the system’s estimate of how far event time has progressed, essentially “I believe I have seen all data up to this timestamp.” It decides when windows are considered complete.

You choose. Within an allowed lateness period, late data can trigger an updated result. After that period, late data can be dropped or routed aside for separate handling.

Draining stops reading new data and finishes processing everything already received, so nothing in flight is lost. Cancelling stops immediately and may discard work in progress.

Use Google-provided templates for common tasks, such as streaming messages into a warehouse table or moving files, and supply the input and output settings.

Within the pipeline, Dataflow provides exactly-once processing semantics for streaming. For the final result in an external system, it depends on whether the sink can handle repeated writes safely, so choose connectors and designs accordingly.

If the transformation is easy to express in SQL on data already in the warehouse, SQL is usually simpler. If you have existing Spark or Hadoop jobs, Dataproc may fit better. Choose Dataflow when you need serverless scaling, streaming with event-time correctness, or custom processing across many systems.

Dataflow takes the hardest parts of large-scale data processing and turns them into a managed service. You describe your pipeline with Apache Beam, using a single model for batch and streaming, and Dataflow builds the execution plan, provisions workers, scales them, rebalances work and recovers from failures. Understanding a few concepts, especially event time, windows, watermarks and triggers, is the key to correct streaming results. Reliable pipelines handle bad data gracefully, write to destinations safely, run with least-privilege identities in private networks, and are watched with good monitoring and cost controls. Used thoughtfully, Dataflow lets small teams process enormous volumes of data confidently.