First the basic words, then the service.

Data, metadata, and governance

Data is the actual information: customer records, sales numbers, photos, logs. Metadata is information about data: its name, what it means, who owns it, when it changed, and how good it is. Data governance means the rules and habits that keep data findable, trustworthy, and safe.

The data sprawl problem

A modern company stores data in many places: Cloud Storage buckets, BigQuery tables, databases, and even other clouds. Teams create new datasets every day. Soon, nobody knows what exists, what it means, or whether it can be trusted. This mess is called data sprawl.

So what is Dataplex?

Dataplex is Google Cloud’s intelligent data governance and management service. It sits on top of your data and gives you one place to discover it, describe it, check its quality, trace where it came from, and control who can use it. Importantly, Dataplex mostly works without moving or copying your data. The data stays where it is.

You may also see the name Dataplex Universal Catalog. Google has been bringing its older data catalog features and Dataplex features together under this umbrella. Product names have shifted over time, so check the current documentation for the latest naming.

The pain it takes away.

Five painful questions

• Where is it? Analysts spend hours hunting for the right table.
• What does it mean? A column called “amt” could mean amount, amortization, or something else.
• Can I trust it? Numbers may be missing, duplicated, or wrong.
• Where did it come from? If a report looks strange, who changed what, and when?
• Who may see it? Sensitive data must be protected, but access rules differ in every tool.

The old fix: spreadsheets and meetings

Many teams try to answer these with shared spreadsheets, wiki pages, and long meetings. These go out of date within weeks. A tool that automatically reads, labels, and watches the data scales much better.

The vocabulary of Dataplex in simple words.

Lake, zone, and asset

Dataplex’s original way to organize data uses three levels:

• Lake: a large grouping of data that belongs together, such as everything for one business domain like “Sales”.
• Zone: a section inside a lake with a purpose. A raw zone holds messy, original data in any format. A curated zone holds cleaned, structured data ready for analysis.
• Asset: a real storage place linked to a zone, such as a Cloud Storage bucket or a BigQuery dataset.

Catalog entry

An entry is the catalog’s record for one piece of data, such as a table, a file set, a dashboard, or a model. It stores the name, location, description, and extra details.

Aspects and aspect types

An aspect type is a template for extra information, such as “Data owner: name and email” or “Sensitivity: low, medium, high”. An aspect is that template filled in for one entry. This lets you attach custom, structured labels to any data.

Business glossary

A business glossary is a shared dictionary of business words, such as “Active customer”, with an agreed meaning. You can link glossary terms to columns and tables so everyone uses words the same way.

Data quality, profile, and lineage

• Data quality scan: checks whether data meets rules you define, such as “email must not be empty”.
• Data profile scan: describes what the data looks like, such as the range of values and the share of empty cells.
• Data lineage: a map of where data came from and where it went.

Data discovery

Discovery means Dataplex automatically scans storage, finds files and tables, and registers them in the catalog, so you do not have to list each one by hand.

How the pieces connect.

The main building blocks

• Data sources: BigQuery, Cloud Storage, Pub/Sub, Spanner, Cloud SQL, Looker, Vertex AI, and many more. Some outside systems can be brought in through connectors or imports.
• Universal Catalog: the central store of metadata and its search.
• Discovery and metadata harvesting: automatic reading of sources to create and update entries.
• Governance tools: glossary, aspects, quality scans, profiling, and lineage.
• Security layer: IAM and policy settings that apply across the data.
• Processing and tasks: serverless jobs, often Spark, that Dataplex can run on your data.
• Consumers: analysts, data scientists, dashboards, and AI tools that search and use the data.

flowchart TD
  SRC["Data sources: BigQuery, Cloud Storage, databases, other services"] --> DISC["Discovery and metadata harvesting"]
  DISC --> CAT["Dataplex Universal Catalog"]
  CAT --> GOV["Governance: glossary, aspects, policies"]
  CAT --> DQ["Quality and profile scans"]
  CAT --> LIN["Lineage"]
  CAT --> SEARCH["Search for people and tools"]
  SEARCH --> USERS["Analysts, data scientists, dashboards, AI"]
    

Metadata, not data

A key point: Dataplex mainly handles metadata. It reads descriptions and structure and builds an index. Your actual rows and files remain in BigQuery, Cloud Storage, or wherever they live. This keeps things fast, cheap, and low-risk.

A closer look at the structure that keeps things tidy.

Why organize by domain

A good data setup follows the business. Sales, Finance, and Marketing each have their own data and their own experts. Giving each domain its own lake makes it clear who owns what, and who should fix problems. This idea is part of a modern approach sometimes called data mesh, where each domain owns its data like a product.

Raw and curated zones

How data moves between zones

flowchart LR
  IN["Source systems"] --> RAW["Raw zone: Cloud Storage assets"]
  RAW --> PROC["Cleaning and transformation jobs"]
  PROC --> CUR["Curated zone: BigQuery and Parquet assets"]
  CUR --> USE["Analytics, BI, machine learning"]
  RAW -. "Discovery registers entries" .-> CAT["Catalog"]
  CUR -. "Discovery registers entries" .-> CAT
    

The story of how data gets into the catalog and becomes searchable.

sequenceDiagram
  participant S as Cloud Storage bucket
  participant D as Discovery scan
  participant C as Catalog
  participant O as Data owner
  participant A as Analyst
  D->>S: Scan files and detect structure
  S-->>D: Files and columns found
  D->>C: Create or update entries
  O->>C: Add description, owner, glossary terms
  A->>C: Search for customer orders
  C-->>A: Entry with schema, owner, quality, lineage
    

Importing outside metadata

Not all your data is in Google Cloud. Dataplex offers ways to import metadata from other systems, such as other databases and tools, using connectors and import jobs, so the catalog can describe your whole data estate. Check current documentation for supported sources.

Can you trust what is in the table?

What is data quality?

Data quality is how well data fits its purpose. Good quality data is complete, accurate, consistent, valid, unique, and timely. Poor quality data leads to wrong decisions, such as a sales report that counts the same order twice.

Quality rules

With Dataplex you define rules that describe what good data looks like. Examples in plain words:

• The customer identifier must never be empty.
• Order amounts must be greater than zero.
• The email column must look like an email address.
• No two rows may share the same order number.
• Dates must not be in the future.
• At least ninety-nine percent of the values in this column must be valid.

A data quality scan checks the data against the rules, gives a score, and reports failing rows or columns. Scans can be run once or on a schedule, and results can feed alerts.

Profiling

A profile scan looks at a table and reports what it contains: minimum and maximum values, average, number of distinct values, and the share of empty cells. It helps you understand new data and even suggest sensible quality rules.

Acting on quality results

• Send alerts to the data owner when the score falls.
• Stop a pipeline from publishing bad data until it is fixed.
• Show the score next to the dataset in the catalog so users see how trustworthy it is.
• Track the score over time to see whether things improve.

The family tree of your data.

What is lineage?

Data lineage shows where data came from, how it was changed, and where it is used. It is like a family tree for a table: its parents (source tables), its grandparents, and its children (reports and downstream tables).

Why it matters

• Troubleshooting: when a dashboard shows a wrong number, you can trace it back to the source of the problem.
• Impact analysis: before changing a table, you can see which reports and jobs depend on it.
• Compliance: you can show auditors how sensitive data flows through your systems.
• Trust: users feel safer using data when they can see where it came from.

How it is collected

Lineage is captured automatically for supported Google Cloud services such as BigQuery, and can be sent from other tools using an open standard called OpenLineage. Each job that reads one dataset and writes another leaves a trail, and Dataplex joins these trails into a graph you can explore.

flowchart LR
  A["Raw orders: Cloud Storage"] --> J1["Job: clean orders"]
  B["Customer table: BigQuery"] --> J2["Job: join orders and customers"]
  J1 --> C["Clean orders table"]
  C --> J2
  J2 --> D["Sales summary table"]
  D --> R["Executive dashboard"]
    

Governance includes keeping the right people in and the wrong people out.

IAM roles

IAM (Identity and Access Management) controls who can view the catalog, edit metadata, run scans, and manage lakes. Give roles by job: viewers for most people, editors for data owners, and administrators for a small governance team.

Policy propagation to data

One feature of lakes and zones is that you can set access at the lake or zone level, and Dataplex pushes matching permissions to the underlying buckets and datasets. This avoids setting the same rules by hand in many places.

Fine-grained control

Tools such as BigQuery’s column-level and row-level security work alongside the catalog. Policy tags can mark columns as sensitive, such as “personal data”, so only approved groups can read them. The catalog can show these labels so people know what is protected.

Sensitive data discovery

Google Cloud also offers tools that scan for sensitive items such as identification numbers or card numbers. Their findings can be added to the catalog as labels, helping you apply the right protection.

Audit and compliance

Audit logs record who searched, viewed, or changed metadata and policies. Combined with lineage, they help show regulators that you know where sensitive data lives and who touched it.

Giving data meaning and packaging it for people.

Business glossary

Technical names do not tell the story. A glossary term such as “Net revenue” has a clear business definition, an owner, and related terms. Linking terms to columns lets anyone hovering over a column see what it truly means.

Aspects for custom metadata

Aspects allow you to add your own structured labels, such as data owner, retention period, sensitivity, and cost center. Because aspects are structured, they can be searched and filtered, such as “show all datasets owned by Finance with high sensitivity”.

Data products

A data product is a bundle of related data assets packaged for a clear purpose, with a description, an owner, and access rules, much like a product on a store shelf. For example, a “Customer 360” product may combine several tables and views. Producers publish it, and consumers request access and use it with confidence. The data product idea is newer in Dataplex, so check the current docs for the exact features.

Dataplex can do work, not just describe data.

Tasks

A Dataplex task is a scheduled or on-demand job, often Spark or a notebook-style job, that runs on serverless compute. You can use tasks to clean data, convert file formats, or run checks without managing clusters.

Templates

Dataplex offers ready-made templates for common work, such as converting files to a columnar format, or moving data between systems. Templates save effort for standard jobs.

Automation with APIs and events

Everything in Dataplex can be controlled through APIs and command-line tools, and many actions can be managed as code. Changes in the catalog can trigger notifications, which can start other work, such as sending an alert or kicking off a pipeline.

Working with orchestration tools

Tools such as Cloud Composer and Workflows can start scans and tasks as part of a larger data pipeline. A good pattern is: load data, run quality scans, publish only if the scores are good.

flowchart TD
  L["Load new data"] --> Q["Run data quality scan"]
  Q --> D{"Score above threshold?"}
  D -- "Yes" --> P["Publish to curated zone and notify users"]
  D -- "No" --> A["Alert owner and hold the data"]
  A --> F["Fix the source or the rules"]
  F --> L
    

How it behaves at scale, how to watch it, and what it costs.

Scale

Because Dataplex works mainly on metadata, it can handle very large estates: thousands of datasets and millions of columns. The heavy work, such as scanning data for quality, uses compute that scales with the size of your tables. Plan scans for your most valuable data first.

Monitoring

• Scan results and scores: track quality over time and set alerts on drops.
• Discovery job status: confirm scans are finding new data and not failing.
• Catalog coverage: measure what share of important datasets have owners and descriptions.
• Audit logs: see who changed policies and metadata.
• Cloud Monitoring and Logging: collect metrics and logs from Dataplex jobs.

What drives cost

• Catalog and metadata storage: the amount of metadata stored and the operations performed.
• Scans: data quality, profiling, and discovery scans consume processing based on how much data is read.
• Tasks and processing: serverless compute used by jobs.
• Underlying services: BigQuery queries, Cloud Storage reads, and network charges.

Prices and free allowances change, so use the current pricing page and calculator.

Ways to save

• Scan important tables first, and use sampling or incremental scans for large ones.
• Schedule scans at sensible intervals instead of constantly.
• Filter discovery so it only scans the folders that matter.
• Delete old scans and unused entries.
• Use labels so you can see which team drives the cost.

An honest report card.

Compared with neighbors

Habits that work, and traps to avoid.

Good patterns

Anti-patterns

Your checklist, the usual stumbles, and where Dataplex shines.

Best practices

• Start with a few high-value domains, not the whole company at once.
• Name owners for every important dataset.
• Write short, clear descriptions and link glossary terms.
• Define a standard set of aspects such as owner, sensitivity, and retention.
• Use quality scans on critical data and show scores in the catalog.
• Capture lineage for key pipelines, including non-Google tools through OpenLineage.
• Tag sensitive columns and connect them to access policies.
• Review IAM roles regularly with least privilege in mind.
• Measure adoption, such as the number of searches and the share of documented datasets.
• Automate with code so settings are repeatable and reviewable.

Common mistakes

Real-world uses

You started with a company full of unlabeled filing cabinets and a librarian who could fix the chaos. You now understand that Dataplex is that librarian for Google Cloud: it finds your data, explains it, checks it, traces it, and helps protect it, without moving it.