A multi-cloud analytics service that runs BigQuery queries on data stored in Amazon S3 or Azure Blob Storage.

BigQuery is Google Cloud’s data warehouse. A data warehouse is a system designed to answer analytical questions over very large amounts of data, such as “how many orders did each region place last quarter?” Analysts write questions in SQL, a common query language, and BigQuery spreads the work across many machines to answer quickly.

Traditionally, BigQuery works on data stored in Google Cloud. But real companies are rarely that tidy. Logs may sit in Amazon S3 because the application runs on AWS. Files from a partner may arrive in Azure Blob Storage. Moving everything to Google Cloud can be slow, costly and sometimes not allowed.

BigQuery Omni solves this by running the BigQuery query engine in the other cloud, close to the data. Google’s documentation sums up the idea simply: processing happens where the data already sits. You use the same BigQuery console, the same SQL and the same tools, while the heavy reading of files happens inside AWS or Azure.

What it is not

Omni is not a tool for moving all your data permanently, although it offers some cross-cloud transfer features for that purpose. It is also not a full copy of every BigQuery feature, since several capabilities are not available in the Omni regions. We cover those limits in a dedicated chapter, because knowing them early prevents surprises.

Data gravity, cost, rules and speed all push companies toward leaving data where it is.

Before learning the tool, it helps to feel the problem it solves. Companies end up with data in several clouds for ordinary reasons, and each reason makes simple copying harder than it sounds.

Why copying everything is painful

The classic fix is to build pipelines that copy data into one warehouse. That sounds easy until you count the costs. Pipelines need to be built, scheduled, monitored and repaired. Copies go stale between runs. Storing two copies doubles the storage bill. Moving data between clouds incurs transfer fees. And every extra copy is another place where sensitive data can leak or drift from the original.

Eight terms unlock almost every sentence in the official documentation.

Every platform speaks its own dialect. Learn these terms once, and the rest of this guide reads like plain English.

Two more words appear later. Colocated region means the normal BigQuery region that is paired with a given Omni region, and cross-cloud features work only between colocated pairs. Egress means data leaving a cloud, which is where transfer charges come from.

You ask in Google Cloud, the work happens in AWS or Azure, and only results come back.

The simplest way to understand Omni is to split it into two halves. The control plane lives in Google Cloud and is the part you talk to. The data plane lives in the other cloud and does the work. A secure VPN link joins them.

flowchart LR
  U["Analyst or application"] --> CP["Control plane in Google Cloud"]
  CP -->|job sent over VPN| DP["Data plane in AWS or Azure"]
  DP -->|reads files| ST["S3 or Blob Storage"]
  DP -->|results over VPN| CP
  CP --> U
  DP -.->|optional export| ST
        

The control plane

The control plane receives your job from the Google Cloud console, the bq command-line tool, the API or a client library. It checks that the job is valid and sends it to the right data plane. For you, it looks and feels like ordinary BigQuery.

The data plane

The data plane runs inside the chosen AWS or Azure region. It connects back to the control plane over a VPN, reads table data from your bucket or container, runs the query in that region and returns the results over the VPN. Because the reading of large files happens next to the storage, bulk data does not need to cross the internet between clouds.

Access using your own identities

Omni uses the identity systems that already exist in each cloud: AWS IAM roles, or Azure Active Directory principals. You grant the access and you can revoke it whenever you like. Read access is enough to query. Write access is needed only if you want Omni to write results back into your storage.

A limited, growing list of AWS and Azure regions, each paired with a normal BigQuery region.

Omni is available in selected regions rather than everywhere. Your data must sit in a supported region, because the query engine runs there. Google’s documentation lists the AWS regions of Northern Virginia, Oregon, Seoul, Sydney, Ireland and Frankfurt, and an Azure region in the eastern United States. Azure support is narrower than AWS support, and the pricing page also mentions an additional Azure location, so always check the current list before you design anything.

Colocated BigQuery regions

Each Omni region is paired with a normal BigQuery region nearby. For example, the Omni region for AWS Northern Virginia and the one for the Azure East US 2 region are both paired with a United States BigQuery region. This pairing matters for cross-cloud features, which work only between colocated regions. A transfer from a United States Omni region lands in a United States BigQuery location, and a transfer from a European Omni region lands in a European one. This also helps with data residency, because data crossing clouds stays on the same continent.

A connection tells BigQuery which identity in your other cloud it may use to read your storage.

The first practical step with Omni is creating a connection. A connection is a secure object in BigQuery that points to an identity in your AWS or Azure account. When BigQuery needs to read your files, it uses this identity, and your cloud’s own permission system decides what it can touch.

How access is built, in plain steps

Read access versus write access

Querying only needs the ability to read. If you want to export results into your S3 bucket or Blob container, you must also allow writing, ideally to a separate, tightly scoped location. Keeping reading and writing in separate roles limits the damage if something goes wrong.

Two sides of control

Notice that there are two layers of permission. Your AWS or Azure account controls what the connection may read. Google Cloud controls which users may use the connection and which tables they may query. Both must say yes. This gives you two places to enforce least privilege, and two places to remove access quickly.

A table definition turns a pile of files in S3 or Blob Storage into something you can query with SQL.

Files in a bucket are just files. To ask questions with SQL, BigQuery needs a table definition that says which files belong to the table, what format they are in and what columns they contain. In Omni, the main kind of table for this job is a BigLake table. Queries can also use external table definitions created with a statement for external tables.

What a BigLake table gives you

• A SQL view of files. Analysts query familiar table names instead of file paths.
• Governance features. Because access flows through the table and connection, you can manage who sees what in one place, rather than giving every analyst direct access to storage.
• Common formats. Typical analytical file formats and partitioned folder layouts can be described, so large datasets organised by date or region work naturally.

Metadata caching: the secret to speed on big datasets

When a table sits on millions of files, just listing the files can take longer than reading them. Omni can store a cache of file and partition information so BigQuery plans queries quickly without listing storage every time. According to the documentation, you decide how stale the cache is allowed to be, within a range from thirty minutes to seven days. The cache can refresh automatically at intervals, or you can refresh it manually when you know new files have arrived. If it is not refreshed for seven days, it expires.

Follow a single question from the analyst’s keyboard to the files in S3 or Blob Storage and back.

Understanding one query end to end clears up most of the mystery. Everything else in Omni is a variation on this journey.

sequenceDiagram
  participant A as Analyst
  participant CP as Control plane
  participant DP as Data plane
  participant S as S3 or Blob Storage
  A->>CP: Submit SQL query
  CP->>CP: Check permissions and parse job
  CP->>DP: Send job over VPN
  DP->>S: Read only needed files
  S-->>DP: File data
  DP->>DP: Run query in the Omni region
  alt Show results
    DP-->>CP: Return results over VPN
    CP-->>A: Display results
  else Export results
    DP->>S: Write results to storage
    DP-->>CP: Report job complete
  end
        

The journey in plain steps

Showing results versus exporting results

When results come back to you, they are kept for a limited time in the control plane so you can view them again. The documentation notes about twenty-four hours. Very large results, over about twenty gibibytes, should not be pulled back to the screen. Instead, use an export statement to write the results straight into your S3 bucket or Blob container. This way, the results never cross clouds at all, which saves both time and transfer cost.

Sometimes data in one cloud must meet data in another, and Omni offers several ways to do it.

Querying in place is only part of the story. Real questions often combine data from two places, such as customer records in Google Cloud and clickstream events in S3. Omni provides cross-cloud operations for these cases. The documentation describes four main tools.

Cross-cloud joins

In a join, BigQuery brings the needed slice of external data to the other side, combines it and returns the result. It is a convenient way to enrich local data without building a pipeline, but it is meant for modest volumes because of the size limit and because each run may incur transfer fees.

Transfers with SELECT

When you only need part of the external data, a transfer with a query copies exactly that part. You can filter rows and choose columns, so you pay only for the slice you move. Appending is possible for incremental loads. The same size limit of roughly sixty gigabytes of results applies.

Transfers with LOAD

When you want everything, LOAD is the right tool. It moves raw files without filtering and without a size limit, and it does not use query compute, only transfer charges for the bytes moved. This is the tool for deliberate, one-time or scheduled migrations.

Materialized view replicas

A materialized view replica keeps a local copy of selected external data and refreshes it on a schedule you choose, between one minute and one hour. Queries and joins on the local copy are fast and do not repeatedly pull data across clouds. The price is ongoing refresh compute, transfer and storage for the copy.

Rules that apply to all cross-cloud features

• The connection and the destination dataset must belong to the same Google Cloud project.
• Transfers work only between colocated regions, such as a United States Omni region and a United States BigQuery location.
• Very small files, below about five megabytes, are inefficient to move, so combine them first.
• Some data types and destinations, such as certain geospatial formats and clustered destination tables for inserts, are not supported.

The right choice depends on how often you read the data, how much you need and how fresh it must be.

With several tools available, a clear way to choose saves time and money. The decision tree below captures how most experienced teams think.

flowchart TD
  Q["Need to analyze data in S3 or Blob Storage"] --> A{"Must it stay in its cloud?"}
  A -->|Yes| B["Query in place with Omni"]
  A -->|No| C{"Need the whole dataset in Google Cloud?"}
  C -->|Yes| D["Use LOAD for bulk transfer"]
  C -->|No| E{"Need only a filtered slice?"}
  E -->|Yes| F["Use CREATE TABLE AS SELECT or INSERT INTO SELECT"]
  E -->|No| G{"Joined often with local data?"}
  G -->|Yes| H["Use a materialized view replica"]
  G -->|No| I["Use occasional cross-cloud joins"]
        

Questions that guide the decision

The same habits that speed up and cheapen any warehouse query matter even more when files live in object storage.

Because Omni reads files, not warehouse-managed storage, the way you lay out those files has a big effect on speed and cost. A few simple habits go a long way.

Organise files with partitions

Partitioning means arranging files into folders by something you often filter on, usually a date. When a query asks for one week, BigQuery can skip every other folder entirely. Without partitions, it must open far more files than needed. Hive-style folder layouts are a common way to do this, and the metadata cache helps plan queries over them faster.

Choose efficient file formats

Columnar analytical formats let a query read only the columns it needs, instead of whole rows. A query touching three columns out of fifty reads a small fraction of the data. Plain text formats force a full read of every row, which is slower and, on on-demand pricing, more expensive.

Avoid thousands of tiny files

Many small files create overhead, because each file must be located and opened. The documentation specifically warns that files under roughly five megabytes transfer inefficiently. Aim for fewer, larger files by combining small ones in a regular compaction step upstream.

Select only what you need

• Name the columns you need instead of selecting everything.
• Filter early on partition columns so unneeded folders are skipped.
• Aggregate inside the query so that only compact results come back.
• Remember that on on-demand pricing, a row limit does not reduce the amount scanned, because billing depends on the columns read.

Use the metadata cache wisely

For large or heavily partitioned tables, enabling the metadata cache reduces planning time. Match the staleness setting to your freshness needs, and give refresh jobs their own background capacity so that user queries are not slowed.

Omni regions do not offer every BigQuery feature, and some limits shape what you can build.

Omni brings BigQuery to other clouds, but it is not an identical copy of everything BigQuery can do in Google Cloud. The documentation lists restrictions, and a good design accounts for them from the start. Treat the following as a summary and check the current documentation, because capabilities evolve.

How to design around the limits

Most limits point to a clear pattern. Use Omni to explore, filter and summarise external data. When you need features Omni does not offer, such as machine learning or data modification, transfer the needed slice into a normal BigQuery region and work there. Treat Omni as the bridge to the data, and the normal region as the workshop.

Multi-cloud analytics adds a second set of doors, so both clouds must agree on who may enter.

Security in a multi-cloud setup involves identities and policies in two places. Fortunately, Omni uses each cloud’s native tools, so you do not have to invent a new security model.

Least privilege on both sides

• In AWS or Azure, give the connection identity read access only to the specific buckets or containers that need analysis.
• Use separate identities for reading and for writing exports, and restrict writing to dedicated locations.
• In Google Cloud, allow only named groups to use each connection and to query each table.
• Review these grants on a regular schedule and remove those no longer needed.

Governance on tables

Because analysts query tables rather than raw files, you can manage access at the table level in BigQuery instead of handing out storage permissions to every person. This keeps your storage locked down while still giving analysts what they need. BigLake tables are designed to support this style of fine-grained governance on external data.

Data in motion

Jobs and results travel between the control plane and the data plane over a VPN. Bulk file reads happen within the cloud region. When you use cross-cloud transfers, data crosses clouds, so you should confirm that doing so is allowed under your data rules and that the destination region respects residency requirements. Because cross-cloud features work only between colocated regions, data stays on the same continent.

Encryption considerations

Your storage encrypts data under your own cloud account’s settings. Note that customer-managed encryption keys are not supported with datasets and external tables in Omni at the time of the documentation, so teams with strict key control requirements must verify this early.

Auditing

Keep audit logs from both clouds. Google Cloud records who ran which query, and AWS or Azure record who accessed which objects. Together they let you answer the auditor’s favourite question: who looked at what, and when?

Know what slow, failed and expensive queries look like, and where to look first.

Because Omni uses ordinary BigQuery job tracking, you can inspect jobs, run times and bytes processed in the console and through logs. The same discipline you apply to any warehouse applies here, with a few extra things to watch because two clouds are involved.

What to watch

Typical troubleshooting scenarios

You pay for compute in the Omni region, and separately for any data that crosses clouds.

Omni costs fall into two groups: the cost of running queries, and the cost of moving data. Mixing them up is the most common budgeting error.

Query compute

Omni offers the same two broad models as BigQuery. With on-demand pricing, you pay for the amount of data each query scans, at a rate per tebibyte, which at the time of writing is a little over nine dollars. With capacity pricing, you pay for slots, and Omni regions support the Enterprise edition, with pay-as-you-go and commitment options. Current rates, including regional differences, are on Google’s BigQuery pricing page and change from time to time.

Data transfer

Querying in place does not move your source files, and the documentation notes there are no additional analytics charges to your AWS or Azure account for the queries themselves. But when data crosses clouds, through joins, SELECT-based transfers, LOAD transfers or replicas, transfer fees apply. These are charged per gibibyte and differ by source region, with typical figures in the range of several cents to a little over ten cents per gibibyte at the time of writing. Replicas also add storage cost for the local copy and refresh compute.

Keeping the bill sensible

• Partition files and use columnar formats so queries scan less.
• Export large results to storage instead of pulling them across clouds.
• Use LOAD for bulk moves, since it avoids query compute.
• Prefer one planned transfer over repeated joins that move the same data each time.
• Set maximum bytes billed and budget alerts.
• Review replicas regularly, since they cost money every hour they run.

Patterns are habits that repeatedly work; anti-patterns are habits that repeatedly hurt.

Teams that use Omni for a while converge on the same handful of habits. Here are the useful ones, followed by the traps that catch newcomers.

Patterns that work

flowchart LR
  R["Raw history in S3 or Blob"] -->|query in place| O["Omni summaries"]
  O -->|small result| J["Join with local BigQuery data"]
  R -->|filtered slice| H["Hot slice replica"]
  H --> J
  J --> D["Dashboards and models"]
        

Anti-patterns to avoid

Omni is superb for one job; other jobs deserve other tools.

Omni fits a specific situation: you want BigQuery analytics over data that sits in AWS or Azure. For other situations, other options may be better. The table below places Omni among its neighbours at a high level.

Situations where Omni shines

• After an acquisition. Analysts need combined insight before any migration project can finish.
• Large logs in another cloud. The files are too big to copy casually, but answers are small.
• Residency rules. Data must remain where it is, yet teams want a single analytics interface.
• Exploration and proof of value. You want to learn what the data can tell you before investing in pipelines.

Situations where caution is wise

• Data in unsupported regions. Omni runs only in listed regions.
• Very frequent, very heavy queries on the same data. A local copy may be cheaper and faster.
• Strict key-management rules. Customer-managed keys are not supported with datasets and external tables in Omni at the time of the documentation.
• Feature-heavy workloads. Anything depending on unsupported statements or functions needs a local copy.

Start small, prove value, then widen access with guardrails in place.

Adopting Omni is more successful when treated as a small program than as a switch to flip. The roadmap below has helped many teams avoid early disappointment.

Who needs to be involved

Four illustrative situations showing where Omni fits.

The scenarios below are illustrative composites, not case studies of specific named companies.

A short checklist to keep nearby while you work with Omni.

Best practices

• Verify region and feature support first. Check the current lists before designing anything.
• Grant the least access. Narrow connections, separate read and write roles, and named access groups.
• Fix the file layout. Partition by date, use columnar formats, and compact small files.
• Enable metadata caching for big tables. Match staleness to freshness needs and give refresh jobs background capacity.
• Summarise before returning. Aggregate and filter inside the Omni region so results stay small.
• Export large results. Write them straight to storage instead of pulling them across clouds.
• Use the right cross-cloud tool. LOAD for bulk moves, SELECT transfers for slices, replicas for hot joined data.
• Set cost guardrails. Maximum bytes billed, budgets and alerts from day one.

Common mistakes

Quick answers to the questions beginners ask most.

It is a multi-cloud service that lets BigQuery run queries directly on data stored in Amazon S3 or Azure Blob Storage, without copying that data into Google Cloud first.

Not for ordinary queries. The query runs next to the data and only results come back. Copying happens only if you choose a cross-cloud transfer, join or replica.

The control plane is the Google Cloud side that receives your jobs. The data plane is the AWS or Azure side that reads files and runs the query. They are linked by a secure VPN.

Amazon S3 in several AWS regions and Azure Blob Storage in a smaller set of Azure regions. The list changes, so check the current documentation.

Through a connection that uses an identity in your own cloud, such as an AWS IAM role or an Azure Active Directory principal, which you create, scope and can revoke.

A table definition in BigQuery that points at files in external storage and supports governance on top, so analysts query table names instead of file paths.

To speed up queries on large or heavily partitioned tables, because BigQuery can plan from cached file information instead of listing storage every time.

Use a cross-cloud join for modest volumes, or a materialized view replica if you join the same data often. Joins are limited to about sixty gigabytes per transfer.

Use the LOAD approach for raw files. It has no size limit and does not use query compute, though transfer charges apply.

Those statements are not available in Omni regions according to the documentation. Transfer the needed data to a normal BigQuery region first.

You pay for query compute, either per amount of data scanned or through reserved slots, plus data transfer fees whenever data crosses clouds. See the current BigQuery pricing page for rates.

When your data is already in Google Cloud, when it sits in an unsupported region, or when your workload depends on features that Omni regions do not offer.

BigQuery Omni lets you ask BigQuery questions about data that lives in Amazon S3 or Azure Blob Storage, by running the query engine in the other cloud and returning only the results. A control plane in Google Cloud takes your job, a data plane in the other cloud does the work, and your own cloud identities decide what may be read. When data must meet across clouds, Omni offers joins, transfers and replicas, each with its own size limits and costs. It is not a full copy of BigQuery, so you must design around the documented limits.

Remember the researcher at one desk and the librarians in several cities. The books stay on their shelves, only the answers travel, and you move a book only when you have a good reason.