Before the details, let us settle the basic vocabulary and the basic problem.

Start with the word object. In cloud storage, an object is simply a file plus some information about that file, such as its name, size, and type. A photo, a video, a log file, or a database backup can each be an object. Objects live inside a bucket, which is a named container, a bit like a top-level folder that you own. Google’s object storage product is called Cloud Storage.

Now the problem. Data rarely stays in one place. A company may keep years of files in Amazon S3 and decide to analyse them with Google Cloud tools. A hospital may hold scans on servers in its own basement and want a safe off-site copy. A media team may need to copy a bucket from one Google Cloud project to another. Each of these jobs is a data transfer.

You could write your own script for this. That script would need to list millions of files, copy them in parallel, handle network failures, retry, avoid copying twice, check that the copies match, and run again tomorrow. That is a lot of careful engineering for a task that is not your real business. STS exists so you do not have to build it.

What STS is not

STS is not a database migration tool, and it is not a streaming engine for live events. It moves files and objects. If you need to move database tables, Google has other products for that. If you need to analyse data while it moves, you would pair STS with something else. Keeping this boundary clear saves a lot of confusion later.

Knowing how a service grew helps you understand why it looks the way it does today.

STS began in the mid-2010s as a fairly narrow tool. Its first big job was copying data from Amazon S3 into Cloud Storage, along with lists of public web addresses. At that time, many companies were trying to run in more than one cloud, and they needed a dependable bridge.

Over the years the service grew in three directions. First, it learned to read from more cloud sources, including Microsoft Azure storage. Second, it gained a way to reach servers that sit outside any cloud, using small helper programs called agents. Third, it became smarter about timing, adding recurring schedules and later event-driven transfers, which start copying when new files appear rather than waiting for a clock.

Where it sits in the Google Cloud family

Google Cloud has several data movement products, and the names can sound alike. STS moves files and objects. BigQuery Data Transfer Service loads data from marketing and analytics products into BigQuery tables, which is a different job. Transfer Appliance is a physical box that you fill with data and ship to Google. Database Migration Service moves databases. Chapter 14 compares them side by side.

Six words carry most of the meaning in STS. Learn these and the console will make sense.

Other words you will meet

A manifest is a list, stored as a file, that names exactly which objects to copy. A filter is a rule such as “only objects whose names start with this prefix”. A prefix is the beginning of an object’s name, which acts like a folder path. Metadata is information about a file, such as when it was last changed or what type it is. A checksum is a short fingerprint calculated from a file’s contents, and two files with the same fingerprint are almost certainly identical.

What can STS read from, and where can it write? The table below is the map.

Notice the pattern. When both ends are public cloud services that Google can reach directly, no agents are needed. When either end is a machine hidden inside your own network, an agent must sit next to that data. This single rule explains most of the design choices you will make.

A note on web URL lists

The URL list option works from a plain text file, in a tab-separated format, that you store in a place STS can read. Each line holds a web address, and usually the file size and a fingerprint so that STS can confirm the download is correct. It is a handy way to pull a set of public files into a bucket without writing a download script.

Let us open the box and see the main parts, and how they talk to each other.

STS has a control plane and a data plane. The control plane is the brain. It stores your job definitions, decides when runs should start, hands out work, and records results. The data plane is the muscle. It is the set of workers that actually read bytes from the source and write them to the sink.

For cloud-to-cloud transfers, Google runs the data plane for you on its own infrastructure. You never see these workers. For agent-based transfers, your own agents become the data plane on the source or destination side, while the control plane stays in Google Cloud.

flowchart LR
  U["You: Console, CLI or API"] --> CP["STS control plane: jobs, schedules, status"]
  CP --> DP["Google managed workers"]
  CP --> AP["Agent pool"]
  S3["Amazon S3 or Azure Blob"] --> DP
  GCSsrc["Cloud Storage source bucket"] --> DP
  URL["Public URL list"] --> DP
  DP --> GCS["Cloud Storage destination bucket"]
  FS["On-premises file system"] --> AP
  AP --> GCS
  CP --> MON["Cloud Monitoring and Logging"]
  CP --> PS["Pub/Sub notifications"]
    

The supporting cast

Identity and Access Management (IAM) decides who can create jobs and what STS itself may touch. Cloud Monitoring shows charts of bytes and objects moved. Cloud Logging keeps detailed records of what happened to individual objects. Pub/Sub, Google’s messaging service, can send you a message when a transfer finishes or fails, and it is also the channel used for event-driven transfers from Cloud Storage. Secret Manager can hold credentials for outside clouds so they are not pasted into a form.

This is the biggest design choice in STS, and it is easy once you see the reason behind it.

Agentless transfers

In an agentless transfer, nothing is installed anywhere. Google’s workers connect straight to the source over the internet using credentials you provide, and write straight into your bucket. This works when the source is a public cloud service reachable from the internet, such as S3 or Azure Blob Storage.

Agent-based transfers

An agent-based transfer is needed when the data sits on machines that the outside world cannot reach, like a file server in your office. Here you install agents inside your network. Each agent makes outbound secure connections to Google, asks the control plane “is there work for me?”, and then reads or writes files as instructed. Because the agents call out, you do not need to open any incoming ports in your firewall.

flowchart TB
  subgraph OnPrem["Your data centre"]
    NAS["File server or NAS"]
    A1["Agent 1"]
    A2["Agent 2"]
    A3["Agent 3"]
    NAS --> A1
    NAS --> A2
    NAS --> A3
  end
  A1 -->|"HTTPS outbound"| CP["STS control plane"]
  A2 -->|"HTTPS outbound"| CP
  A3 -->|"HTTPS outbound"| CP
  A1 --> GCS["Cloud Storage bucket"]
  A2 --> GCS
  A3 --> GCS
    

How agent pools behave

All agents in a pool are treated as equal helpers. If you add more agents, the pool can move data faster, because more hands share the work. If one agent crashes, the others keep going and the lost work is picked up again. For this reason, teams that care about reliability run more than one agent, commonly three or more, on separate machines.

A pool can also be given a bandwidth limit. This is a cap on how much network capacity the agents may use at once. It is very useful when the same internet link is shared with office staff or customer traffic, because it stops a big migration from slowing everyone else down.

Follow one transfer operation from the moment it starts to the moment it finishes.

Why skipping matters

Step 3 is the hidden hero. Because STS skips objects that are already identical, you can run a big job once to move the bulk of the data, and then run it again right before the final switch-over to catch only what changed. The second run is short, because most of the work was already done.

Every job moves through a small set of states. Knowing them helps you read the dashboard calmly.

A transfer job itself is either enabled, meaning it will run on its schedule, disabled, meaning it is paused, or deleted, meaning it is gone from the active list. Each operation under it moves through its own states: it is queued, then in progress, and finally ends as successful, failed, or aborted. A paused or cancelled run ends as aborted.

flowchart LR
  A["Job created"] --> B["Waiting for schedule or event"]
  B --> C["Operation starts"]
  C --> D["Listing and comparing"]
  D --> E["Copying objects"]
  E --> F{"All objects handled?"}
  F -->|"Yes"| G["Success"]
  F -->|"Some failed"| H["Failed with error summary"]
  C -->|"Stopped by user"| I["Aborted"]
  G --> B
  H --> B
    

Partial success is normal

Beginners are sometimes surprised that an operation can copy ninety-nine thousand objects and still be marked as failed because of a handful of problem files. This is by design. STS reports honestly that something needs attention. The error summary lists the affected objects and the reason, such as a missing permission or an object that changed during the copy.

The next scheduled run will try those objects again, since they are still missing or different at the destination. Often, the problem disappears by itself. If it does not, the error details tell you what to fix.

When does a transfer run, and which objects does it touch? This chapter covers the dials you can turn.

Scheduling

A job can run once, right now or at a time you choose. It can also repeat on a schedule, such as every hour, every day, or every week, with a start date and an optional end date. Recurring jobs are ideal for ongoing backups and for keeping a destination bucket close to a source that keeps changing.

Filters

Filters narrow down what gets copied. You can include only objects whose names begin with certain prefixes, or exclude prefixes you do not want. You can also filter by time, for example “only objects modified in the last seven days”. Filters reduce cost and run time, because STS never touches what you did not ask for.

Manifests

A manifest is a file, stored in Cloud Storage, that lists the exact objects to transfer. Use it when your selection rule is too clever for prefixes, for example “these 40,000 specific files that our audit flagged”. The manifest turns a vague wish into a precise shopping list.

Event-driven transfers

Instead of running on a clock, an event-driven transfer waits for news. When a new object lands in an Amazon S3 bucket, a message goes into an Amazon queue service. When a new object lands in a Cloud Storage bucket, a message goes into Pub/Sub. STS listens to that queue, and copies the new object soon after. The result is a destination that trails the source by a short delay, with no polling schedule to tune.

flowchart LR
  P["Producer uploads new object"] --> SRC["Source bucket"]
  SRC -->|"Change notification"| Q["Queue or Pub/Sub topic"]
  Q --> STS["STS event-driven job"]
  STS --> DST["Destination bucket in Cloud Storage"]
    

These settings decide what happens when the destination already has something, and what happens to leftovers.

When should STS overwrite?

If an object with the same name already exists at the destination, STS needs a rule. There are three common choices. Overwrite if different replaces the destination copy only when the two do not match, which is the usual default and the best choice for most jobs. Never overwrite leaves existing objects alone, which protects the destination. Always overwrite replaces everything every time, which is rarely needed and wastes effort.

Deleting at the destination

You can ask STS to delete objects from the destination that no longer exist at the source. This turns a copy into a mirror: the destination becomes an exact picture of the source. It is powerful and risky in equal measure, because a mistake in the source quickly becomes a mistake in the destination.

Deleting at the source

You can also ask STS to delete source objects after they have been copied successfully. This changes a copy into a move. It is handy for emptying a staging area or finishing a one-way migration, but it needs extra permission on the source, and it should only be used after you have confirmed that the destination is correct.

Metadata

Metadata is the information that travels with each object. When copying between Cloud Storage buckets you can decide what to carry over, such as the storage class, the encryption key used, or the object’s original creation time. When copying from file systems, you can choose to keep details like file permissions and modification times. Carrying metadata across matters when other systems rely on it, for example a retention rule that looks at an object’s age.

Moving data means touching keys, doors, and secrets. Here is how STS keeps them safe.

Who can do what: IAM roles

Google Cloud controls access with IAM, short for Identity and Access Management. Each person or program is given roles, and each role is a bundle of permissions. STS has its own roles. An admin role can create and change jobs. A user role can run and view jobs. A viewer role can only look. A special transfer-agent role is for the account that agents use.

This split lets you follow the principle of least privilege, which means giving each person or program only the permissions they need and nothing more. A person who only needs to check whether last night’s backup worked should be a viewer, not an admin.

The service agent

When a transfer runs, STS acts through a Google-managed identity called a service agent, which belongs to your project. This identity must be allowed to read the source bucket and write to the destination bucket. If it lacks permission, the run fails with a clear message. For delete options, it needs delete permission as well, and you should add that only when you truly use those options.

Reaching outside clouds

To read from Amazon S3 or Azure, STS needs proof that it is allowed in. The simple way is to provide an access key or token. The safer way is federated identity, where the other cloud trusts Google’s identity and hands out short-lived access without a long-lived secret ever being stored. Short-lived access is better, because a stolen key that expires in an hour is far less dangerous than a key that never expires.

Protecting data in motion and at rest

Data moving between systems travels over encrypted connections. At the destination, objects in Cloud Storage are encrypted at rest by default, and you can choose your own encryption keys through Cloud KMS, the key management service, if your rules require it. Agent connections use secure web traffic that starts from your side, so you do not open inbound ports.

Building a fence around the service

Organisations with strict rules can place STS inside VPC Service Controls, which draws a protective perimeter around Google Cloud services and blocks data from being copied to places outside the perimeter. For agent-based transfers over private links such as Cloud Interconnect, traffic can stay off the public internet.

A transfer you cannot see is a transfer you cannot trust. STS gives you four windows.

What to watch

Four numbers tell most of the story. Objects found says how much work existed. Objects copied says how much was moved. Objects skipped says how much was already identical. Objects failed says how much needs attention. For a healthy recurring job, “found” stays steady, “copied” is small after the first run, and “failed” is near zero.

Alerts worth having

Good alerts are few and meaningful. First, alert on any operation that ends as failed. Second, alert when a job that should run daily produces no operation at all, which suggests it was disabled or broke silently. Third, for agent-based jobs, alert when the count of healthy agents in a pool drops below what you need.

How fast will it go, how do you make it go faster, and what will it cost?

What sets the speed

Speed is limited by the slowest link in the chain. That could be the source system’s ability to serve files, the network between source and Google, the number of workers or agents, or the destination’s ability to accept data. For cloud-to-cloud copies, Google adds workers automatically, so the limit is more often the source’s own request limits or the network path. For agent-based copies, your agents and your internet link are the usual limits.

A quick back-of-the-envelope calculation

Planning starts with simple arithmetic. A link of 1 gigabit per second can carry about 125 megabytes each second at best. At that pace, one terabyte takes a little over two hours, and 100 terabytes takes about nine days of non-stop, perfect transfer. A link ten times faster, at 10 gigabits per second, cuts that to about 22 hours. Real life is slower, because links are shared and files have overhead, so always add a safety margin.

Many small files versus few big files

Ten million tiny files are slower than the same amount of data in a thousand big files. Every object has fixed overhead: listing it, comparing it, opening a connection, and checking it. If you control how data is produced, bundling tiny files into larger archives before the transfer can help a lot. If you cannot, plan for the extra time.

Ways to speed things up

• Add more agents for agent-based jobs, and spread them across several machines.
• Split one giant job into several jobs by prefix, so work runs in parallel.
• Run the first big pass early, then use short top-up runs near the cut-over.
• Keep the source and destination in nearby regions when you have a choice.
• Make sure the agent machines have enough memory, processor power, and a fast disk or network connection to the data.

Where the money goes

Think of cost in four buckets. First, source-side charges: Amazon and Microsoft charge for data leaving their clouds, and for the read requests STS makes. This is often the biggest line in a migration. Second, Google-side storage and request charges for the data you land in Cloud Storage. Third, network costs for any private links or cross-region copies on the Google side. Fourth, STS service charges, which depend on the type of transfer. Because prices and free allowances change, check the current Google Cloud pricing page before you promise anyone a number.

STS is excellent, but it is not always the right answer. Here is how it compares.

A simple decision path

If the data set is small and you only need it once, use a copy command from your laptop or a cloud shell. If the data is large, or the job must repeat, or someone needs a report afterwards, choose STS. If the sheer size would take weeks over your network, look at a physical appliance, and use STS afterwards to catch up with changes made during shipping.

flowchart TB
  Q1{"Is the data a database?"}
  Q1 -->|"Yes"| DMS["Use a database migration product"]
  Q1 -->|"No, files or objects"| Q2{"Is it a quick one-off and small?"}
  Q2 -->|"Yes"| CLI["Use a command-line copy"]
  Q2 -->|"No"| Q3{"Would the network take weeks?"}
  Q3 -->|"Yes"| TA["Consider Transfer Appliance then STS for top-ups"]
  Q3 -->|"No"| STS["Use Storage Transfer Service"]
    

Patterns are proven recipes. These four show up again and again in healthy STS setups.

These are the mistakes that appear in post-incident reviews. Recognise them early.

A checklist you can follow before, during, and after any transfer project.

Before you start

• Measure first. Count the objects and total size at the source. Without these numbers you cannot estimate time or check the result.
• Decide the success test. Write down how you will prove the copy is correct before you begin.
• Plan permissions. List who needs which role, and which identity STS will use at each end.
• Pick the right mode. Cloud source with public access means agentless. Hidden servers mean agents.
• Estimate cost. Include source-side exit charges, because they surprise people the most.

While it runs

• Start small. Run a pilot on one prefix and check it carefully.
• Watch metrics. Compare copied bytes against your estimate to see if the pace is on target.
• Read the first error list early. Common problems such as permission gaps show up in the first hour.
• Do not change the source heavily. Objects that change during copying may be reported as failed and retried later.

After it finishes

• Verify. Compare counts and sizes, then sample content.
• Keep the source until you are sure. Hold the old data for an agreed period before removing it.
• Clean up. Disable jobs, shut down agents, remove cross-cloud permissions, and delete temporary buckets.
• Document. Write down what was moved, when, with which settings, so the next person is not guessing.

Common mistakes in one glance

The cases below are illustrative stories that show the patterns in action. They are examples to learn from and are not accounts of specific companies.

Short answers to the questions that come up most often.

If you remember only a few things from this guide, make them these.

Storage Transfer Service takes a hard, repetitive engineering task and turns it into a managed setting. You choose a source, a destination, a schedule, and a few safety options. Google runs the workers, checks the work, and reports back. The harder part is not the tool but the planning: knowing your data, protecting it, watching the run, and proving the result.