A managed service that loads data into BigQuery on a repeating schedule, with no custom code.

The Short Definition

The BigQuery Data Transfer Service, often shortened to BigQuery DTS or just DTS, is a fully managed Google Cloud service that automates the movement of data into BigQuery. You describe where the data lives, where it should land and how often it should move. Google then runs the transfers for you, keeps a history of every run and retries or reports when something goes wrong.

The key words are managed and scheduled. Managed means you do not create servers, install agents for most sources, or maintain scripts. Scheduled means that after you configure a transfer once, it keeps running by itself every day, every week or at whatever repeating time you chose.

What It Is Not

It helps to know the boundaries early. DTS moves data into BigQuery. It cannot be used to move data out of BigQuery to another system. It is also not a streaming engine: it works in batches, so data arrives in chunks at scheduled times rather than row by row in real time. Finally, it is not a general-purpose transformation tool. It lands data in tables; shaping that data further is the job of SQL, scheduled queries or other services.

Three Ways to Operate It

You can configure and manage transfers from the Google Cloud console, from the bq command-line tool, or through the BigQuery Data Transfer Service API. The console is friendly for first-time setups because it presents forms that match each source. The command line and API are better for repeatable automation, and infrastructure-as-code tools can drive the same API so that transfers are versioned like any other part of your platform.

Why moving data by hand, or with homemade scripts, quietly becomes a full-time job.

The Hidden Cost of Homemade Pipelines

Before managed transfer services, a typical team wrote a small script to pull data from an advertising platform, then another for the CRM, then another for a payment provider. Each script needed credentials, a place to run, a scheduler, error handling, retry logic and monitoring. Each source also had its own quirks: different authentication styles, different page sizes, different rate limits and different ideas of what “yesterday” means.

At first, three scripts feel cheap. A year later, those three scripts have become thirty, the person who wrote them has moved teams, and nobody remembers why one of them skips Sundays. The pipeline is now a risk that sits between the business and its numbers.

What DTS Takes Off Your Plate

Eight words that appear in every DTS conversation, each explained from scratch.

Every technology has its own vocabulary, and DTS is no exception. Learn these eight terms and the rest of the service becomes easy to follow. For each term, remember three things: what it is, why it exists and where you will meet it.

Configuration Versus Run

Beginners often confuse these two ideas, so it is worth pausing. The configuration is permanent and describes intent. The run is temporary and describes one attempt to fulfil that intent. If you edit the schedule, you change the configuration. If you want to see why last night failed, you open the run. Keeping this separation in mind makes troubleshooting far easier.

The moving parts between a source system and a BigQuery table.

The Big Picture

At a high level, the service sits between your sources and your warehouse. On the left are the places data lives. In the middle is the control plane of DTS, which stores configurations, triggers runs and tracks status. On the right is BigQuery, where the data lands and where standard storage and query behavior takes over.

flowchart LR
    A["Sources: SaaS, ads, clouds, databases, buckets"] --> B["Transfer configuration"]
    B --> C["DTS scheduler and run engine"]
    C --> D["Connector reads source data"]
    D --> E["Staging in dataset location"]
    E --> F["BigQuery destination dataset"]
    C --> G["Run history and logs"]
    C --> H["Notifications: email or Pub/Sub"]
        

Read the diagram from left to right. The configuration is the instruction card. The scheduler and run engine act on that card at the right time. A connector, which is the piece of logic that understands one specific source, reads the data. The service stages the data in the same location as your destination dataset and then writes it into BigQuery tables. Alongside this main path, the run engine records history and can send notifications.

Key Components Explained

The family of suppliers DTS can already talk to, grouped by type.

DTS supports a broad and growing set of sources. Google groups them into categories, and the exact list changes as connectors are added, promoted from Preview or updated. Always confirm the current list and each connector’s launch stage in the official documentation before you design around a source. The groups below give you a reliable mental map.

Why the Split Between Google and Non-Google Sources Matters

Google publishes a data delivery objective for automatically scheduled transfers from sources within Google Cloud. That objective does not apply to transfers from non-Google Cloud sources, because outages or slowness in a third-party system can affect delivery and are outside Google’s control. In everyday terms: if the supplier’s truck breaks down, the delivery service cannot promise the groceries will arrive on time. Plan buffers into downstream reports that depend on third-party sources.

From saving a configuration to seeing rows in a table, step by step.

Phase One: Setup

Before anything runs, the project needs the BigQuery Data Transfer API enabled, a destination dataset must exist, and the person or service account creating the transfer needs the right permissions. When you save a configuration, the service validates settings and the credentials needed to read the source. For user-owned transfers against many sources, you will be asked to authorize access through a standard consent screen.

Phase Two: Runs

sequenceDiagram
    participant S as Schedule
    participant D as DTS Control Plane
    participant A as Service Agent
    participant X as Source System
    participant B as BigQuery
    S->>D: Time reached, create run
    D->>A: Request access token for owner
    A-->>D: Token granted
    D->>X: Read data for time window
    X-->>D: Return records
    D->>B: Stage and load into tables
    B-->>D: Load result
    D->>D: Save run status and logs
        

The sequence shows that DTS acts as the coordinator. It never needs you to be online. Each arrow represents a point where something can fail: an expired authorization, an unreachable source, a schema surprise or a permission gap on the destination. Knowing the order of events tells you where to look when a run fails.

Update Deadlines

Each data source has a limit for how long the service will wait for an update before it marks the run as failed. This protects you from runs that hang forever. If a run fails with a timeout-like message, the cause is usually a very large source slice, a slow source system or a connectivity problem rather than a bug in your configuration.

Deciding how often data moves, and how to catch numbers that change after the fact.

Choosing a Schedule

Most configurations use a repeating schedule such as daily, weekly or monthly, and many sources let you choose a time of day. The default for many connectors is once a day, which fits the way most reporting works. The shortest allowed interval depends on the source, so check the connector’s documentation if you need frequent loads. More frequent is not automatically better: each run uses source API capacity and may have cost or quota effects.

Why Recent Data Changes

Advertising and analytics platforms often revise recent numbers. A click may be re-classified as invalid a day later, or a conversion may be attributed to a campaign two days after the fact. If you load yesterday’s data once and never look again, your warehouse can drift away from what the platform itself reports.

To solve this, some connectors support a refresh window. With a sliding window, each run re-reads the last several days and updates them, so late adjustments are folded in automatically. The source’s own settings describe whether it supports this behavior and the default number of days.

Time Zones and Time Windows

Transfers work with time windows, and the boundaries of a “day” depend on how the source and the transfer are defined. Run times are commonly reasoned about in Coordinated Universal Time, while the business may think in local time. Agree on one convention for reporting, document it, and test with a known date to confirm that numbers in BigQuery match the source’s own report for that date.

How to fill gaps, repair outages and load the past after a new transfer is created.

What a Backfill Really Is

A new transfer usually starts loading from today onward. But analysts almost always want history: last quarter, last year, or at least the last few weeks. A backfill is a manual request that tells the service, “run this configuration for these past dates.” Each date in the range becomes its own run, so you can watch progress in run history exactly like a normal scheduled run.

Backfills are also your repair tool. If a source was down for two days, if a credential expired over a weekend, or if a permission was removed by mistake, the schedule will have skipped or failed those days. After you fix the cause, a backfill for the missing dates restores the table without any custom recovery code.

Limits to Remember

How far back you can go is controlled by the source, not by DTS. Each data source keeps only a certain amount of history and exposes only part of it through its reporting interface. Some sources support daily windows only. Date ranges are aligned to day boundaries, and the service treats the start and end as whole days in Coordinated Universal Time. Always read the connector’s page for the maximum history and for any rules about range size.

A Safe Backfill Routine

Two everyday BigQuery features that quietly run on the same service.

DTS is not only about pulling data in from outside. The same engine also powers two operations that happen entirely inside Google Cloud. Many people use them for months without realizing they are using the Data Transfer Service at all.

Why They Belong Together

Ingestion is only the first half of a data platform. The second half is shaping raw tables into trustworthy, ready-to-query tables. By letting the same scheduler handle both, a small team can build a complete daily rhythm: raw data arrives from outside sources, scheduled queries clean and summarize it, and dataset copies publish final tables to consumer projects.

flowchart TD
    A["Source transfers land raw tables"] --> B["Scheduled query cleans and joins"]
    B --> C["Scheduled query builds summary tables"]
    C --> D["Dataset copy publishes to reporting project"]
    D --> E["Dashboards and analysts"]
        

Credentials for Internal Jobs

When scheduled queries or dataset copies run, the service starts BigQuery jobs using the transfer owner’s credentials. That means the owner must keep the right access to both the source tables and the destination. If the owner is a person who later leaves the company or loses access, those jobs can start failing. Using a service account as owner, where supported, removes this dependency on a human.

Loading files when they arrive, instead of waiting for the clock.

The Idea

Schedules are great when data is produced on a rhythm. But files often arrive at unpredictable times: a partner drops a report at noon one day and at four the next. For Cloud Storage sources, DTS offers an event-driven mode. Instead of running on a timer, the transfer reacts when a new file shows up in the bucket.

How the Pieces Connect

Cloud Storage can publish a notification to a Pub/Sub topic whenever an object is created. A Pub/Sub subscription holds those messages. The BigQuery Data Transfer Service service agent reads from that subscription, learns about the new file and starts a load. Because of this chain, correct permissions on Pub/Sub and on the bucket are essential, and the service agent must be allowed to subscribe to the relevant resources.

sequenceDiagram
    participant P as Partner or Pipeline
    participant G as Cloud Storage Bucket
    participant T as Pub/Sub Topic
    participant U as Pub/Sub Subscription
    participant D as DTS Service Agent
    participant B as BigQuery
    P->>G: Upload new file
    G->>T: Object created notification
    T->>U: Message delivered
    D->>U: Pull event
    D->>G: Read the new file
    D->>B: Load into destination table
        

When to Choose Events Over Schedules

Why the place where your dataset lives quietly decides where everything else happens.

Like BigQuery itself, DTS works with locations. The transfer configuration takes the same location as the destination dataset, and the service processes and stages data in that location. In other words, you do not pick a processing region for the transfer separately. You pick the dataset’s location, and the transfer follows it.

Data Can Come From Anywhere

Transfers are supported from data stored in a wide range of places to a destination dataset in the location you choose. Compatibility is documented per source, so always check the official location page for your connector. The practical lesson is to decide dataset locations early, because moving data between locations later requires extra work and planning.

Who is allowed to do what, and which identity actually touches your data.

Three Different Identities

Security in DTS becomes easy once you separate the identities involved. There are three, and each has a different job.

User Owners Versus Service Account Owners

If the owner is a person, that person usually has to approve access through a consent screen so the service can read the source on their behalf. If that person later leaves, loses access or revokes the consent, the transfer stops working. If the owner is a service account, no human consent is needed, and the service agent impersonates the account to obtain tokens. This is more stable and easier to audit, which is why service accounts are the preferred choice wherever the connector supports them.

Cross-Project Authorization

Sometimes the service account that owns a transfer lives in a different project from the one where the transfer service is enabled. In that case, the service agent of the transfer project needs permission to create tokens for that account. Google recommends granting this at the level of the specific service account, not across the whole project, to follow least privilege.

Notifications and Their Permissions

Transfers can send run notifications by email or to a Pub/Sub topic. Email needs no extra permission. Pub/Sub notifications require the person configuring them to be allowed to set the topic’s access policy, so the service can publish to it. Always test a notification by running a transfer once and watching the topic.

Protecting Data Once It Lands

After data reaches BigQuery, ordinary BigQuery security controls apply. You can restrict datasets and tables through IAM, limit sensitive columns with column-level controls, and apply encryption policies that your organization requires. Treat the destination dataset as the security boundary: give analysts access to curated tables, and keep raw landing datasets visible only to the pipeline owners.

Understanding what you pay for and what limits exist, without memorizing numbers that change.

Where Costs Come From

Pricing for transfers depends on the data source, and Google publishes it on a dedicated pricing page. Some connectors are offered at no separate charge, while others have their own pricing model. Because these details change over time, always confirm the current terms for your exact connector before you commit to a design.

Regardless of connector, once data is in BigQuery, normal BigQuery storage and query costs apply. This is the cost most teams actually feel: the loaded tables take up storage, and every analyst query over them consumes compute. A well-designed transfer plan therefore thinks about what to load, not only how.

Slots and Reservations

Jobs triggered by the service use reserved capacity only if your project, folder or organization is assigned to a reservation that covers the relevant job types. If you do not have such an assignment, the jobs run without drawing on reserved slots. Teams with reservations should check that the right job types are included so that transfer-related jobs behave as expected.

Quotas

Like most cloud services, DTS has quotas and limits, such as how many configurations a project can hold and how quickly requests can be made. The values are published on the quotas page and can differ by source. Treat quotas as design inputs: if you plan to create hundreds of configurations, check the limits first and consider combining similar transfers where the connector allows.

Knowing when a transfer is healthy, and finding the cause fast when it is not.

What to Watch

A transfer that silently stops is worse than one that fails loudly, because people keep trusting stale numbers. Build three layers of visibility: per-run status in the transfer’s run history, notifications that reach a human or a system, and freshness checks on the destination tables themselves.

The Usual Suspects When a Run Fails

An honest look at when DTS shines and where it is the wrong tool.

How DTS Compares With Neighboring Tools

Habits that make transfers boring and reliable, and habits that make them fragile.

Patterns That Work

Anti-Patterns To Avoid

A practical checklist you can apply to your very next transfer.

Before You Create a Transfer

After You Create It

Mistakes Seen Again and Again

Four realistic situations showing how the pieces fit together.

flowchart LR
    A["Ad platforms"] --> R["Raw dataset"]
    B["Commerce and payments"] --> R
    C["Partner files in bucket"] --> R
    R --> Q["Scheduled queries"]
    Q --> K["Curated dataset"]
    K --> L["Dashboards and analysts"]
    R --> N["Notifications and freshness checks"]
        

What actually happens to the data between the source and the final table.

From Raw Records to BigQuery Tables

When a run reads data from a source, it does not simply pour rows straight into your final table. The service first gathers the data, stages it in the destination dataset’s location and then asks BigQuery to load it. This two-step approach is familiar to anyone who has moved data before: collect first, then commit. It lowers the chance that a half-finished read leaves a table in a confusing state.

The tables that appear in your dataset are created by the connector. Their names and column lists are decided by the connector’s design, not by you. For reporting sources, you often receive several tables, such as one for campaigns, one for daily performance and one for reference data. For database and storage sources, tables usually mirror the source objects or the destination you name in the configuration.

Schemas and Surprises

A schema is the list of columns in a table and the type of each column. Source systems change over time: a new field is added, a name is adjusted, or a number that used to be whole starts having decimals. Some connectors are designed to adapt to additions, while others expect a fixed structure. For file-based sources, the shape of each file matters a great deal, because a file with extra or missing columns can cause a load to fail or to place values in unexpected columns.

Write Behavior: Replace, Append or Update

Each time a run lands data, one of a few things can happen to the target table. The data may be appended to what is already there, it may replace the table, or it may replace only a specific date partition. Which one applies depends on the connector and sometimes on a setting you choose when creating the transfer. Understanding this is important for safe backfills and for avoiding duplicate rows.

Partitioning and Clustering on Landing Tables

Partitioning divides a large table into smaller date-based pieces so that queries read only the pieces they need. Clustering sorts data inside those pieces by chosen columns so filters become cheaper. Many reporting connectors create date-partitioned tables for you. For tables you manage yourself, such as curated outputs from scheduled queries, you should choose partitioning and clustering on purpose, because they strongly influence both speed and cost.

flowchart TD
    A["Connector reads source slice"] --> B["Data staged in dataset location"]
    B --> C{"Write behavior"}
    C -->|Append| D["Rows added to table"]
    C -->|Replace partition| E["One date partition rewritten"]
    C -->|Replace table| F["Whole table rewritten"]
    D --> G["Run marked succeeded"]
    E --> G
    F --> G
        

The full journey from an empty project to a working, monitored pipeline.

Reading about a service is useful, but nothing beats understanding the exact order of actions. This walk-through is written for the Google Cloud console, but the same ideas apply to the command line and the API. We will use a typical scenario: loading daily files from a storage bucket into a reporting dataset.

Setting Up Through Automation

Once a transfer works in the console, many teams capture the same configuration as code so it can be reviewed, versioned and recreated. Because the service has a public API, infrastructure tools can create and update transfers, and scripts can trigger runs and read history. The benefit is not speed but repeatability: when a project must be rebuilt or cloned for another environment, the transfer comes with it rather than depending on someone’s memory of console clicks.

How each group of sources behaves, and what to expect from each.

Cloud Storage Sources

Storage connectors load files from Cloud Storage, Amazon S3 and Azure Blob Storage. You describe where the files live and which ones to pick up, usually through a path with a pattern. The service handles authentication, pattern matching and loading. Typical issues include files with inconsistent columns, partially uploaded files, and patterns that accidentally match too much or too little.

Database and Warehouse Sources

These connectors are mostly used for migration and recurring replication of tables from systems such as other warehouses and relational databases. They often require network access between Google Cloud and the source, which means firewall rules, private connectivity or a small helper component near the source depending on the connector. Plan time for networking and security reviews, because this is where setup usually takes the longest.

Google Marketing and Analytics Sources

Connectors for advertising, analytics and app platforms load reporting data produced by Google products. They usually need an identity that has access to the account being read, and they land standard report tables. Because these platforms revise recent days, the refresh window concept is especially important here.

SaaS, Marketing and Payment Platforms

Connectors for customer, marketing and payment platforms read data through each vendor’s interface. They are subject to that vendor’s rate limits and availability. Many of these connectors have carried Preview status, so treat their documentation as the source of truth for fields, limits and support terms.

graph LR
    A["Cloud storage files"] --> DTS["Data Transfer Service"]
    B["Databases and warehouses"] --> DTS
    C["Google marketing and analytics"] --> DTS
    D["SaaS and payments"] --> DTS
    DTS --> BQ["BigQuery datasets"]
        

Making sure the numbers in the warehouse deserve the trust people place in them.

Why Loading Successfully Is Not Enough

A run that finishes with a green status only tells you that the mechanics worked. It does not tell you that the numbers are right. A source might have returned an empty slice because of a temporary problem, an account might have been removed from the connection, or a filter might have excluded most rows. Reliable teams add checks that look at the data itself.

Governance Basics

As the number of transfers grows, governance keeps things understandable. This does not need heavy bureaucracy. A few light rules go a long way.

What changes when you go from three transfers to three hundred.

Small setups are forgiving. Large ones are not. When many teams share a platform, small inconsistencies multiply into real operational pain. The ideas below keep a growing estate manageable.

Structure Your Projects and Datasets

Separate projects by environment and, when needed, by business domain. Within a project, keep raw landing datasets apart from curated ones. This structure makes it easy to give analysts access to curated data only, and to measure costs per domain.

Standardize Through Templates

If every team invents its own approach, support becomes impossible. Create a small set of approved patterns, such as “daily reporting source with refresh window” or “partner file drop with event trigger,” and publish a short guide for each. A template is not a restriction; it is a head start that already includes the right owner identity, notifications and checks.

Plan for Quotas and Limits

Review documented limits on the number of configurations and the rate of requests, and combine similar transfers where the connector allows. Stagger schedules so that hundreds of runs do not all start at the same minute and compete for source capacity.

Prepare Runbooks

A runbook is a short document that says what to do when something goes wrong. For transfers, a good runbook contains where to look first, how to restore authorization, how to run a backfill safely, whom to inform and how to confirm recovery.

flowchart LR
    A["Alert fires"] --> B["Open failed run"]
    B --> C{"Access problem?"}
    C -->|Yes| D["Fix identity or role"]
    C -->|No| E{"Source or data problem?"}
    E -->|Source| F["Wait or contact vendor"]
    E -->|Data| G["Fix config or schema"]
    D --> H["Backfill missing dates"]
    F --> H
    G --> H
    H --> I["Verify and close"]
        

How organizations use DTS during warehouse moves, and how different industries apply it.

A Safe Migration Pattern

Moving to BigQuery from another warehouse is not only a technical copy. It is a change that affects every report. The safest approach runs old and new systems side by side for a period, compares results and moves consumers gradually.

How Different Industries Use It

Putting every idea from this guide into one coherent picture.

Let us combine everything into a single design for a mid-sized company that wants a trustworthy daily reporting platform. The company has advertising accounts, an online store, a payment provider and partners who drop files into a bucket. The goals are simple: fresh numbers every morning, no manual loading, safe access for analysts and quick recovery when something breaks.

Layer One: Landing

Each source has its own transfer, owned by its own service account. All transfers write into raw landing datasets inside a single, deliberately chosen location. Raw data is never edited and is kept for a defined period. Analysts have no access here; only the pipeline owners do.

Layer Two: Cleaning

Scheduled queries read the raw tables, standardize names, fix types, remove obvious duplicates and join related sources. They run at a safe gap after ingestion and write to a cleaned dataset with date partitioning and clustering chosen to match the way people filter data.

Layer Three: Curated

More scheduled queries build business-ready tables such as daily revenue, campaign performance and delivery status. A dataset copy can publish these tables to a separate reporting project, where analysts and dashboards have read-only access. This keeps experiments away from production data.

Layer Four: Observability

Every transfer sends failure notifications to a shared channel. A small set of scheduled checks measures freshness, row count sanity and key totals, and writes results to a monitoring table that a dashboard displays. A runbook explains exactly what to do for each type of alert.

flowchart TD
    subgraph Sources
      S1["Ad accounts"]
      S2["Online store"]
      S3["Payment provider"]
      S4["Partner files"]
    end
    S1 --> T["Transfers with dedicated service accounts"]
    S2 --> T
    S3 --> T
    S4 --> T
    T --> RAW["Raw landing datasets"]
    RAW --> CLEAN["Scheduled queries: cleaned datasets"]
    CLEAN --> CUR["Scheduled queries: curated datasets"]
    CUR --> COPY["Dataset copy to reporting project"]
    COPY --> BI["Dashboards and analysts"]
    T --> MON["Notifications and freshness checks"]
    CLEAN --> MON
    CUR --> MON
        

A realistic plan for introducing DTS into a team over the first month.

Adopting a new service works best in small, safe steps. The roadmap below is deliberately modest. Each week ends with something that works and can be shown to colleagues, which builds trust and keeps motivation high.

Skills Worth Building Alongside

Every important term from this guide, defined in one or two simple sentences.

Common Misunderstandings Cleared Up

A Short Pre-Launch Checklist

Before declaring any new transfer finished, walk through these questions with a teammate. If any answer is unclear, pause and resolve it first; ten minutes of review now prevents days of confusion later.

One-Page Decision Guide

No. It is designed to bring data into BigQuery. Exporting data out of BigQuery uses other mechanisms such as export jobs or other services.

Not for normal use. You configure transfers through forms in the console, or through the command line or API if you prefer automation. The connectors themselves are maintained by Google.

No. It works in scheduled batches, with event-driven triggering available for Cloud Storage files. For continuous change capture or streaming, look at services built for that purpose.

The run is marked failed in run history with messages explaining what happened, and notifications can alert you. After you fix the cause, you can start a backfill for the affected dates.

Yes, through a backfill, within the limits of how much history the source makes available. Check the connector’s documentation for the maximum range.

A dedicated service account is the better choice wherever the connector supports it, because it does not depend on any one person. Some sources still require a user account.

Google publishes a data delivery objective for scheduled transfers from sources within Google Cloud. It does not apply to transfers from non-Google Cloud sources, because those depend on outside systems.

In the same location as the destination dataset. The transfer configuration takes that location too.

It depends on the connector, so check the official pricing page for your source. After loading, normal BigQuery storage and query costs apply, and the source provider may charge separately for exports or egress.

Not necessarily. Some are labeled Preview at different times. Review each connector’s launch stage and terms before depending on it for critical reporting.

Yes, through scheduled queries, which run on the same service. They are separate from source connectors, so you load first and then transform.

Common reasons are recently revised data, differences in how a day is defined across time zones, and reading data before the source finished processing. A refresh window and a consistent day convention usually fix this.

The BigQuery Data Transfer Service turns the repetitive, error-prone work of loading data into a managed routine. You describe a source, a destination dataset and a schedule once, and Google runs the transfers, records every run and helps you recover with backfills. Alongside external sources, the same engine powers scheduled queries and dataset copies, giving small teams a complete rhythm for ingesting, shaping and publishing data. Success depends less on clever configuration and more on a few steady habits: choose locations carefully, use dedicated service accounts, watch freshness, and always test with one run before trusting a schedule.