Batch is a fully managed Google Cloud service for scheduling, queuing and running batch workloads on Compute Engine virtual machines.

The Short Definition

Google Cloud Batch, usually just called Batch, lets you submit a piece of work called a job. You describe what the job should run, how many copies of it should run, and what kind of machines it needs. Batch then does the boring and difficult parts for you. It creates the virtual machines, starts your work on them, watches the progress, retries failed pieces, writes logs, and deletes the machines when everything is finished.

The word “batch” simply means “a group of work items processed together, without a person watching every step.” Nobody sits at a screen clicking a button for each item. You hand over the whole pile and come back for the result.

Why It Exists

Before managed batch services, a team that wanted to process a million images or run a thousand simulations had to build its own tooling. They created virtual machines by hand, wrote scripts to copy work onto them, wrote more scripts to detect crashes, and then remembered, often too late, to delete the machines. Batch removes all of that glue code and operational burden.

Where It Is Used

Batch is used wherever there is a large amount of computing that can be split into pieces: scientific simulations, genome analysis, video rendering, financial risk calculations, machine learning data preparation, nightly data processing, and engineering workloads such as electronic design checks. The common thread is that the work has a beginning and an end. It is not a website that must stay online forever.

Before learning the tool, it helps to understand the kind of work the tool was built for.

Two Kinds of Computing Work

Computer workloads fall broadly into two families. The first family is services: programs that run all day and answer requests, like a website, a payment API, or a chat server. They must always be ready, and a human or another system is waiting for a quick answer.

The second family is batch jobs: programs that start, do a defined amount of work, and stop. Nobody is waiting for an instant reply. What matters is that the whole pile of work finishes correctly, on time, and at a sensible cost.

The Idea of Splitting Work

Batch computing becomes powerful when work can be split. If one machine needs one hundred hours to process a dataset, one hundred machines may do the same job in about one hour, because each machine handles one hundredth of the data. Work that splits neatly like this is often called “embarrassingly parallel”, which is a funny way of saying the pieces do not need to talk to each other while they run.

Why Not Just Rent Machines by Hand?

You can create virtual machines yourself, but then you become responsible for choosing how many, spreading work among them, detecting stuck work, handling machines that are taken away, and cleaning up. In a large job these chores cost more engineering time than the actual computing. A managed batch service turns the chores into configuration.

Eight words unlock the whole service. Learn these and every document about Batch becomes readable.

How the Pieces Nest

The nesting is easy to picture. A job contains a task group. The task group contains many tasks. Each task follows the same task specification, which lists the runnables, the CPU and memory needed, the retry rules, and the storage to attach. What makes one task different from another is only its index number, which your own program can read and use to pick its slice of the data.

flowchart TD
  J["Job"] --> AP["Allocation policy: machines"]
  J --> LP["Logs policy"]
  J --> NP["Notifications"]
  J --> TG["Task group"]
  TG --> SP["Scheduling policy"]
  TG --> TS["Task specification"]
  TG --> T0["Task 0"]
  TG --> T1["Task 1"]
  TG --> TN["Task N"]
  TS --> R1["Runnable 1"]
  TS --> R2["Runnable 2"]
        

From the moment you submit a job to the moment the last machine disappears, a clear sequence of states unfolds.

Step by Step

Job States and Task States

Both jobs and tasks carry a state, which is simply a label saying where they are in their journey. Watching these labels is the most common way to understand what Batch is doing.

sequenceDiagram
  participant U as You
  participant B as Batch service
  participant C as Compute Engine
  participant A as Agent on VM
  participant L as Cloud Logging
  U->>B: Submit job
  B->>B: Validate and queue
  B->>C: Create VMs
  C-->>A: VM starts, agent boots
  A->>B: Ask for tasks
  B-->>A: Assign tasks
  A->>A: Run runnables in order
  A->>L: Stream logs
  A->>B: Report task state
  B->>C: Delete VMs when done
  B-->>U: Job succeeded or failed
        

Batch is a control layer sitting on top of the same Compute Engine machines you already know.

The Three Layers

It helps to think of Batch as three layers. At the top is the control plane: the managed service that accepts jobs, keeps their state, decides when and where to create machines, and tracks every task. In the middle are the virtual machines from Compute Engine, which are the actual workers. At the bottom are the supporting services that give the workers data and a voice: Cloud Storage, Persistent Disk, Filestore, Artifact Registry, Cloud Logging, Cloud Monitoring and Pub/Sub.

The Agent: The Quiet Worker on Every Machine

When Batch starts a virtual machine, a small agent program comes up with it. The agent is the link between the managed service and your work. It asks the service which tasks it should run, starts your scripts or containers, collects their exit status, sends progress back, and forwards your program output to logging. Because the agent reports in, the service always knows whether a machine is healthy, busy or gone.

This is also why the job’s service account needs a special permission to report back. Without that permission the agent cannot talk to the control plane, and the job stalls.

Regions, Zones and Location Choices

A job is created in a region, such as one in Asia or Europe. Inside your allocation policy you can list the zones where machines are allowed. A wider list gives Batch more chances to find capacity, which helps with big jobs or scarce machine types. A narrow list keeps data close to compute, which helps with speed and network cost. You balance the two based on where your data lives.

flowchart LR
  subgraph CP["Control plane"]
    API["Batch API"]
    SCH["Scheduler and state tracker"]
  end
  subgraph WK["Workers"]
    VM1["VM with agent"]
    VM2["VM with agent"]
    VM3["VM with agent"]
  end
  subgraph SV["Supporting services"]
    GCS["Cloud Storage"]
    FS["Filestore or disks"]
    AR["Artifact Registry"]
    LG["Cloud Logging"]
    PS["Pub/Sub"]
  end
  API --> SCH
  SCH --> VM1
  SCH --> VM2
  SCH --> VM3
  VM1 --> GCS
  VM2 --> FS
  VM3 --> AR
  VM1 --> LG
  SCH --> PS
        

What Batch Does Not Do

Batch does not decide how to split your data. You or your program decide that, usually by using the task number. It also does not understand dependencies between different jobs. If job B must wait for job A, an orchestrator such as Workflows or Cloud Composer coordinates that. Batch focuses on one thing: running one job well.

A runnable is the smallest step you can describe. Tasks are built by lining runnables up in order.

The Three Kinds

Why Containers Are Popular

A container is a sealed box holding your program and everything it needs to run. Because the box is identical everywhere, a task running on machine one behaves exactly like a task on machine five hundred. This removes the classic complaint “it worked on my laptop.” Containers also make it easy to version your work: you point the job at a specific image tag and know exactly what code ran.

Control Switches on a Runnable

Runnables carry behavior switches that make real workloads manageable. You can ignore the exit status of a harmless step so it does not fail the whole task. You can mark a runnable as a background step, such as a helper service that should run alongside your main step. You can mark a step as “always run” so cleanup or upload happens even if earlier steps failed. You can set a time limit on a step so one stuck process cannot block everything. Each runnable can also have its own environment variables.

Telling Each Task Who It Is

Batch gives every task a set of environment variables, which are named values visible to your program. The most important ones tell the task its own index and the total number of tasks, and the number of the current retry attempt. A program that reads its index can compute which slice of the data is its own. For example, task 7 of 100 might process the eighth hundredth of a file list. This tiny idea is how one description becomes thousands of different pieces of work.

Choosing the right machine is the biggest single lever on both speed and cost.

Machine Types

Because Batch runs on Compute Engine, you can choose from the familiar machine families: general purpose, compute optimized, memory optimized, and accelerator optimized. A task also declares how much CPU and memory it needs. Batch uses those numbers to decide how many tasks can share one machine. A task that needs two CPUs on an eight-CPU machine means up to four such tasks can run side by side, depending on your settings.

Operating System Images

Each machine boots from an image, which is a saved copy of an operating system. Batch offers images prepared for batch work, including Debian based and Container-Optimized OS based choices, and images tuned for high performance computing. You may also bring a custom image that already contains your heavy software, which saves startup time because nothing must be installed at the beginning of every task.

Standard, Spot and Preemptible Pricing

Compute Engine sells the same machine at different prices depending on promises. Standard machines stay until you finish. Spot machines are spare capacity sold at a deep discount, often in the range of sixty to ninety percent less, but Google may take them back with only a short warning, about thirty seconds. Batch lets you choose the provisioning model per job.

GPUs and Other Accelerators

A GPU is a chip with thousands of small cores, built for doing the same arithmetic on lots of data at once. Batch lets you attach GPUs to the machines and, where needed, request that the required drivers be installed. This supports machine learning preparation, rendering, scientific simulation, and video work. Because GPU machines are expensive and sometimes scarce, matching the number of tasks to the number of GPUs, and using Spot where safe, matters a great deal.

Three simple numbers decide how your job spreads across machines.

The Three Numbers

Doing the Mental Math

Suppose you have ten thousand tasks, a parallelism of five hundred, and four tasks per machine. At any instant five hundred tasks run, which needs about one hundred twenty five machines. When a task ends, another starts in its place, until all ten thousand have finished. If each task takes ten minutes, the whole job takes roughly twenty waves of ten minutes, so around three hours and twenty minutes. Raise parallelism to one thousand and the time roughly halves, as long as quota and data systems can keep up.

Scheduling Policy: Any Order or In Order

With the default “as soon as possible” policy, tasks start whenever a slot is free, with no guaranteed order. This is fastest and is right for independent work. The “in order” policy runs tasks one by one in index order, which is helpful when each task must see the output of the previous one, though it gives up the benefit of parallel speed.

Tightly Coupled Work and Hosts Files

Some workloads, such as weather models or fluid simulations, run as one big program spread over many machines, with each part constantly talking to the others. This style often uses a message passing library. For such work Batch can provide a file listing all machines in the group, so the parts can find each other. Fast networking, close placement, and barriers become important here. This is quite different from the independent-task style, and it is where the HPC focused images and settings are used.

graph TD
  Q["10000 tasks waiting"] --> W1["Wave 1: 500 tasks running"]
  W1 --> W2["Wave 2: next 500 tasks"]
  W2 --> W3["More waves"]
  W3 --> D["All tasks finished"]
  W1 -.-> N["About 125 VMs at 4 tasks each"]
        

Compute is easy to scale. Getting data in and out efficiently is where most real projects succeed or struggle.

Where Data Can Live

The Typical Flow

Most batch pipelines follow the same rhythm. Raw input sits in a bucket. Each task reads its slice, works in fast local scratch space, and writes its result back to a bucket. A later step combines the partial results. Notice that no task depends on a machine staying alive for its outputs. Anything valuable is saved to durable storage before the task ends.

flowchart LR
  IN["Input bucket"] --> T1["Task reads its slice"]
  IN --> T2["Task reads its slice"]
  T1 --> SC1["Local scratch space"]
  T2 --> SC2["Local scratch space"]
  SC1 --> OUT["Output bucket"]
  SC2 --> OUT
  OUT --> MG["Merge or report step"]
        

Keep Data and Compute Close

Moving data across regions is slower and can cost money. Place the bucket and the job in the same region whenever possible. This simple choice often improves speed and lowers the network bill at the same time.

Every worker machine needs a safe road to its data, its images and its logs.

The Basics

Batch machines live inside a Virtual Private Cloud network, which is your private, isolated slice of Google’s network. By default the project’s standard network is used, but you can pick a specific network and subnetwork. This matters because firewall rules and routes belong to the network, so the network choice decides who may talk to your workers and what your workers may reach.

External Addresses or Not

Machines can be created with or without public internet addresses. Many security teams prefer machines without public addresses, because a machine that cannot be reached from the internet is much harder to attack. When you remove public addresses, your tasks can still reach Google services through a feature called Private Google Access, and you can reach the wider internet, if truly needed, through a managed gateway such as Cloud NAT.

Blocking Outside Access for Containers

For extra safety, a container runnable can be told to block external network access entirely. This is useful when you run code that should only touch mounted data and never call out to the internet, for example when processing sensitive files.

Networking Checklist

A batch job can create hundreds of machines and read vast amounts of data, so identity and permissions deserve careful thought.

Two Different Identities

It is easy to confuse two identities. The first is the person or system that submits the job. This identity needs permission to create jobs, and permission to act as the service account the job will use. The second is the service account the job runs as. This is the identity your tasks carry while they work, and it decides what data they can read and write.

Roles to Know

Least Privilege

Least privilege means giving an identity only the permissions it truly needs. If a job reads from one bucket and writes to another, its service account should have read access on the first and write access on the second, not broad access to every bucket in the project. Using a dedicated service account for each pipeline also makes audits easy: you can see exactly what each pipeline is allowed to touch.

Protecting Secrets

Passwords, tokens and keys must never be written into job definitions or container images. Batch supports passing secrets to tasks by referencing them in Secret Manager, so the value is fetched at run time and not stored in plain sight. Access to the secret is controlled by the same permission system as everything else.

More Layers of Defense

A batch job runs while you are not watching, so it must tell you clearly what is happening.

Logging

The logs policy decides where program output goes. In the usual setup, everything your tasks print is sent to Cloud Logging. There you will find two main streams. One holds the output from your own tasks. The other holds messages from the Batch agent about what it did on each machine. When a task fails, these two streams together usually explain why: your own error message on one side, and the platform’s view of what happened on the other.

Logs can be filtered by job, by task, by time and by severity. Because every line carries the task identity, you can follow one failing task out of ten thousand without reading the rest.

Metrics

Cloud Monitoring collects numbers about the machines: CPU use, memory, disk, and network. You can chart them, set alerts, and build dashboards. Installing the monitoring agent on the workers gives richer memory and disk numbers. Watching CPU use is the simplest way to discover that you picked machines far larger than needed, or that tasks are waiting on data rather than computing.

Notifications

Batch can publish messages to a Pub/Sub topic when the state of a job or a task changes. Other systems subscribe to that topic. A subscriber can post to a team chat, update a ticket, trigger the next pipeline stage, or run a cleanup. This is the clean way to make jobs part of a larger automated system, without a program that keeps asking “are you done yet?”

In big jobs, something will always go wrong. Good design assumes failure and plans for it.

Why Tasks Fail

Tasks fail for many reasons. Your program may have a bug or meet a strange input. A machine may be taken back because it is a Spot machine. A network call may time out. A disk may fill up. A task may run out of memory and be stopped by the operating system. Knowing the reason matters, because some failures are worth retrying and others never will succeed.

Retry Count

Each task specification includes a maximum retry count, a small number that can be set from zero up to ten. With the default of zero, a failed task is not retried. Setting it to two or three covers most temporary problems like network blips or interrupted Spot machines.

Lifecycle Policies: Smart Rules by Exit Code

When a program ends, it returns a small number called an exit code. By convention zero means success, and other numbers mean different kinds of trouble. A lifecycle policy lets you say: “if the task ends with this exit code, retry it; if it ends with that one, fail it immediately.” Batch also treats certain platform events, such as a Spot reclaim, in a way you can plan for. This lets you avoid wasting time retrying a task that failed because of bad input, while still retrying one that failed because its machine disappeared.

flowchart TD
  S["Task starts"] --> R["Runnables execute"]
  R --> E{"Exit status"}
  E -->|"Success"| OK["Task succeeded"]
  E -->|"Failure"| P{"Lifecycle policy matches?"}
  P -->|"Retry action"| C{"Retries left?"}
  P -->|"Fail action"| F["Task failed"]
  C -->|"Yes"| S
  C -->|"No"| F
        

Time Limits

A maximum run duration can be set on a task. If the task is still running when the limit arrives, Batch stops it and treats it as a failure. This protects your budget from a program that has silently frozen. Choose a limit comfortably above the normal runtime, so healthy tasks are never cut short.

Idempotent Tasks Are Safe Tasks

A task is idempotent when running it twice leaves the same final result as running it once. Retries and interruptions mean your task might run more than once. If it writes its result to a fixed, predictable location, a second run simply overwrites the first with identical content. If instead it appends to a shared file, a retry might duplicate data. Designing for idempotence is the most important habit in batch engineering.

Checkpoints for Long Tasks

For tasks that run for hours, especially on Spot machines, save progress at regular intervals to durable storage. When a retry begins, the program checks for a saved checkpoint and resumes from there. A task interrupted at minute 170 of 180 then loses only a few minutes, not the whole three hours.

The service is free, the resources are not. Knowing the bill’s shape lets you shrink it.

What You Pay For

Batch itself carries no additional fee. Your bill is made of the resources your jobs use: virtual machine time for the CPUs, memory and accelerators; disk storage; network traffic, especially when data leaves a region or the cloud; storage for your data in buckets; and logging volume. Because machines exist only while the job runs, you do not pay for idle capacity between jobs.

Ways to Reduce Cost

Quotas: The Invisible Ceiling

Every project has limits on how many CPUs, GPUs, disks and addresses it may use in a region. A job asking for more than the quota allows will wait or be limited. Before launching a very large job, check your quota in the target region and request an increase well in advance. There are also limits on the number of tasks per job and other service limits, which Google documents and may change over time, so check the current documentation before planning an extreme scale.

Google Cloud has several places to run work. Picking the right one avoids pain later.

A Simple Way to Decide

Ask four questions. First, is my code already a Spark or pipeline program? If yes, look at Dataproc or Dataflow. Second, do my tasks fit inside the size and time limits of a serverless container? If yes, Cloud Run jobs is wonderfully simple. Third, do I already operate a Kubernetes cluster that has spare capacity? If yes, Jobs in that cluster may be natural. Fourth, do I need specific machine shapes, GPUs, long runtimes, shared file systems, or scripts that expect a real machine? That is where Batch shines.

One job is a step. Real systems chain many steps together.

Ways to Create Jobs

You can submit jobs by clicking through the Google Cloud console, by using the command line tool, by calling the REST API, by using client libraries in popular languages, or by describing jobs in infrastructure tools such as Terraform. A team often starts with the console for learning, moves to the command line for experiments, and settles on code or templates for production so that every run is repeatable and reviewable.

Orchestrators

Scientific Workflow Engines

Scientific communities rely on workflow engines that describe long pipelines of tools. Popular engines such as Nextflow and Cromwell, along with others, can use Google Cloud Batch as the place where each step runs. The engine decides what to run next, and Batch supplies the machines. This lets researchers keep their familiar pipeline definitions while gaining cloud scale.

flowchart LR
  SCH["Cloud Scheduler"] --> WF["Workflows"]
  WF --> J1["Batch job: prepare data"]
  J1 --> PS["Pub/Sub state message"]
  PS --> WF
  WF --> J2["Batch job: heavy compute"]
  J2 --> BQ["BigQuery results"]
  J2 --> NT["Team notification"]
        

Templates and Reuse

Batch can create machines from an existing Compute Engine instance template, so organizations that have already standardized machine settings, security hardening and images can reuse them. This keeps batch workers consistent with the rest of the company’s virtual machines.

A pattern is a proven shape for a solution. These patterns appear again and again in successful batch systems.

Pattern in Picture: Fan-Out and Fan-In

flowchart TD
  I["Large input"] --> SP["Split into slices"]
  SP --> A["Task 0"]
  SP --> B["Task 1"]
  SP --> C["Task 2"]
  SP --> D["Task N"]
  A --> M["Combine results"]
  B --> M
  C --> M
  D --> M
  M --> O["Final output"]
        

An anti-pattern looks like a good idea but quietly causes trouble. Here are the most common ones in batch work.

A short checklist of habits shared by teams that run batch workloads well.

Labels and Naming

Labels are small key and value tags you attach to jobs. Use them to record the team, the environment, the pipeline name, and the software version. They make cost reports meaningful, help search through logs, and let automated cleanup find old jobs. Clear job names that include the pipeline and date turn the job list into a readable history.

Cleaning Up Old Jobs

Finished jobs remain visible as records. Periodically delete jobs you no longer need so lists stay readable, but keep logs and audit trails as long as your rules require. Deleting a job does not delete the data it wrote to your buckets, so review output retention separately.

Test Locally First

The easiest bugs to fix are the ones found on your own computer. If your task is a container, run it locally against a small sample before sending it to the cloud. Many costly cloud failures turn out to be simple typing errors that a thirty second local test would have caught.

Plan for Observability from Day One

Decide before the first run how you will know it worked. Which log line proves success? Which count should match the input? Which dashboard shows progress? When these are decided early, the team trusts the pipeline and sleeps better.

The same service solves very different problems across industries. These examples are illustrative scenarios of how organizations typically use batch computing.

A calm, step by step path from zero to a dependable production pipeline.

Not deeply. Knowing what a virtual machine, a disk and a machine type are is enough to start. Batch creates and deletes the machines for you.

You do not manage servers, patch them or scale groups yourself, so the experience feels serverless. Underneath, real Compute Engine machines run your work, which is why you can choose their size, type and image.

The job can wait in the queue until resources become available, or run with less parallelism than you hoped. Check and raise quota before large runs.

Yes. A task is a list of runnables, and each runnable can be a script, a container or a barrier. They run in the order you list them.

Every task receives its own index and the total task count through environment variables. Your program uses them to pick its slice, for example by taking every Nth file from a sorted list.

It is safe when tasks are idempotent and either short or checkpointed, because interrupted tasks can be retried. It is a poor fit for tasks that cannot be safely repeated or that have a strict, unmovable deadline with no spare time.

Resource charges begin when machines and disks exist, not while a job waits in the queue. Always confirm current billing details in the official pricing documentation.

Use Cloud Scheduler to trigger a workflow or a small service that creates the job at the chosen times. The orchestrator can also wait for completion and react to the result.

They can, and tightly coupled HPC workloads depend on it, using a hosts file and fast networking. But most batch work is designed so tasks stay independent, which is simpler, faster and more resilient.

Check the job and task states, then read your task logs and the agent logs in Cloud Logging. Also verify permissions, quota, and whether the machines could reach the data and the image registry.

Yes. A custom image with your software pre-installed saves startup time and gives you tighter control, as long as it meets the requirements Batch has for its agent and supported setups. Check current documentation for details.

No. Deleting a job removes the job record, not the files your tasks wrote to buckets or file shares. Manage the retention of output data separately.

A single VM runs your script once, on one machine, and you handle everything around it. Batch automates machine creation, spreads tasks across many machines, retries failures, collects logs, sends notifications and removes the machines afterward.

Launching at full scale before testing small. A tiny canary run reveals most permission, data path and sizing problems for almost no cost.

Cloud Batch turns the hard operational work of large batch computing into a handful of settings. You describe the work as a job made of many tasks, choose the machines and storage, set your retry and security rules, and submit. The service queues the job, creates the machines, runs and monitors every task, handles failures, reports progress, and cleans up. What remains for you is the part that truly matters: designing tasks that are small, repeatable and safe, and feeding them data efficiently.

Mastering Batch is less about memorizing options and more about adopting a way of thinking. Split the work. Make each piece independent and idempotent. Keep data close and durable. Give every identity only the access it needs. Measure on a small scale before spending at a large scale. With these habits, a single person can run computations that once required an entire data center team.