Dataplex in Google Cloud: Governing Your Data Across Lakes, Warehouses and Teams
A deep, plain-English guide to Dataplex: how it catalogs, organizes, secures, measures and tracks data wherever it lives, so that people can find trustworthy data quickly and companies can prove they are in control of it.
Imagine a huge company with fifty warehouses scattered across many cities. Each warehouse has its own staff, its own labels, its own way of counting stock and its own rules about who may enter. Nobody can answer simple questions: what do we own, where is it, is it any good, and who is allowed to touch it? Dataplex is the head office that walks through every warehouse, writes down what is inside, checks the quality, records where goods came from, and enforces one consistent set of rules, without forcing anyone to move their goods.
1What Dataplex Is and Why It Exists
Dataplex is Google Cloud’s service for data governance, discovery and management across many storage systems.
As companies grow, data spreads. Raw files sit in object storage, analytical tables live in a warehouse, operational data hides in databases, and each team builds its own copies. This spread brings familiar problems. People cannot find the data they need. They find three versions of the same table and do not know which is right. Sensitive columns are exposed because no one knew they were sensitive. Reports disagree because the underlying data was quietly changed. When an auditor asks where a number came from, nobody can say.
Dataplex addresses these problems with a layer that sits above storage and processing. It does not replace your warehouse or your lake. Instead, it gathers information about your data, called metadata, into one place, and adds services on top: search, quality checks, profiling, lineage, business glossaries and consistent access policies. The goal is simple to state: make data easy to find, easy to trust and safe to use.
Google has renamed and reorganized parts of this product family more than once. Features once presented as separate products, such as the data catalog, have been brought under the Dataplex name, and newer documentation may use the label Dataplex Universal Catalog for the central metadata layer. Some older concepts, such as lakes and zones, receive less emphasis than they once did. Treat the names and feature boundaries in this guide as a conceptual map and confirm current names in the official documentation.
A city has many neighborhoods, each with its own shops. A city directory does not move the shops. It lists them, describes them, checks their licenses and tells visitors which ones are open. Dataplex is the directory plus the licensing office for your data.
What Dataplex is not
Dataplex is not a database, not a processing engine and not a place to store the data itself. It does not make slow queries fast. It makes data understood and controlled. Teams that expect a storage product are often confused at first, so keep the picture clear: the data stays where it is, and Dataplex holds knowledge about it and enforces rules over it.
2The Problem of Data Sprawl
To appreciate the cure, look closely at the disease it treats.
Data sprawl is what happens when data creation outpaces data management. It is the natural result of success: more teams, more tools, more sources and more projects. No single decision causes it, and no single fix removes it. Understanding its symptoms helps you decide which Dataplex capabilities matter most to your organization.
Nobody can find anything
Analysts spend more time hunting for data and asking colleagues than analyzing it.
Nobody trusts anything
Several copies of the same table disagree, and no one knows which is authoritative.
Nobody owns anything
When a table breaks, no one is responsible for fixing it, and questions bounce between teams.
Nobody knows what is sensitive
Personal or confidential columns sit in tables that many people can read.
Nobody can trace changes
A report number changes and no one can explain which upstream step caused it.
Rules differ everywhere
Every system has its own permissions model, so policies are inconsistent and hard to audit.
The cost of doing nothing
These problems are not merely annoying. They slow projects, produce wrong decisions and create regulatory risk. A machine learning model trained on poor data produces poor predictions. A dashboard built on a stale table misleads executives. A data protection regulator who asks where personal data is stored expects a real answer, not a month-long scramble. Governance is the discipline that turns sprawl into something manageable, and Dataplex is a toolset that supports that discipline.
A library with no catalog, no labels and no librarian still holds books, but finding one takes luck. Adding a catalog and labels does not add a single book, yet it makes the whole collection vastly more useful.
Governance is as much about people and process as tools. Dataplex can show who owns a table, but the organization must still decide who should own it. Plan for roles and responsibilities alongside the technology.
3Core Concepts and Vocabulary
A compact vocabulary makes every later chapter easier.
Metadata
Data about data: names, descriptions, column types, owners, sensitivity labels, update times. The raw material of governance.
Entry
A catalog record that describes one resource, such as a table, a file set, a dashboard or a stream.
Aspect
A structured piece of metadata attached to an entry, such as an owner, a data classification or a retention rule. Shape is defined by an aspect type.
Entry group
A container that collects related entries and lets you manage access to them together.
Lake, zone and asset
Older organizing concepts. A lake is a logical domain, a zone is a subdivision such as raw or curated, and an asset is a bucket or dataset attached to a zone.
Data profile
A statistical summary of a table: counts, distinct values, ranges, null rates and common values.
Data quality rule
A testable statement about data, such as “this column is never empty” or “values fall between zero and one hundred”.
Lineage
A map of how data flows from sources through transformations to outputs.
Business glossary
A shared dictionary of business terms with agreed definitions, linked to the data that represents them.
Data product
A curated, documented, owned bundle of data assets presented for consumers to use with confidence.
An entry is the library card for one book. Aspects are the stamps on the card: genre, shelf, reading level, loan rules. A glossary is the dictionary kept at the front desk so everyone uses words the same way.
Technical metadata versus business metadata
Technical metadata describes structure: table names, column types, partitions, sizes. Much of it can be collected automatically. Business metadata describes meaning: what a column represents, who is responsible, whether it contains personal data, how it may be used. Business metadata usually needs human knowledge, though automation and suggestions can help. A governance program is only as good as its business metadata, because that is what lets non-engineers understand and trust data.
Assuming that automatic discovery produces meaning. It records that a column named “cust_id” exists and what type it is. It cannot know that the column is a regulated identifier unless someone, or some classification process, says so.
4The Big Picture: Capabilities at a Glance
Dataplex is best understood as a family of capabilities that share a metadata foundation.
flowchart TD
S1["Cloud Storage files"] --> M["Metadata foundation: catalog of entries and aspects"]
S2["BigQuery datasets"] --> M
S3["Databases, streams, dashboards, other systems"] --> M
M --> D["Discovery and search"]
M --> Q["Data quality and profiling"]
M --> L["Lineage"]
M --> G["Glossary and business context"]
M --> P["Access and policy governance"]
M --> DP["Data products and sharing"]
| Capability | Question it answers | Who benefits most |
|---|---|---|
| Catalog and search | What data do we have and where is it? | Analysts, scientists, engineers |
| Automatic discovery | What files and tables exist that nobody registered? | Platform and governance teams |
| Data profiling | What does this data actually look like? | Engineers and analysts |
| Data quality | Can we trust this data right now? | Data owners, consumers, auditors |
| Lineage | Where did this come from, and what depends on it? | Engineers, auditors, impact analysts |
| Glossary | What does this business term mean? | Business users and stewards |
| Policy and access | Who may see what? | Security and compliance teams |
| Data products | Which curated data is ready to use? | Consumers and domain teams |
The strength of the design is the shared foundation. When a quality result is attached to the same catalog entry that carries the owner and the lineage, a person looking at a table sees the whole story in one place instead of hopping across tools.
A hospital chart holds a patient’s history, test results, medications and doctor notes together. A doctor does not visit five buildings to understand one patient. The catalog entry is the chart for a data asset.
You do not need every capability on day one. Many teams begin with catalog and search, add quality checks on the most important tables, and grow into lineage and glossaries as the habit takes root.
5The Universal Catalog: Entries, Aspects and Search
The catalog is the heart of Dataplex, and everything else hangs on it.
The catalog stores an entry for each resource it knows about. Some entries are created automatically for Google Cloud resources such as BigQuery tables and datasets, Pub/Sub topics and other supported services. Others come from custom sources: you can write entries for systems the catalog does not know natively, such as an on-premises database or a third-party application, through metadata import capabilities.
System metadata and custom metadata
Each entry carries system-managed information, such as the source, type and technical schema, which you do not edit. On top of that you attach aspects, which hold your own structured metadata. You define an aspect type with named fields, such as an owner, a sensitivity level, a retention period or a data domain, and then apply aspects of that type to entries. Because aspects are structured rather than free text, they can be searched, filtered and validated consistently.
flowchart LR
AT["Aspect type: defines the fields"] --> A1["Aspect on Table A: owner, sensitivity, domain"]
AT --> A2["Aspect on Table B: owner, sensitivity, domain"]
A1 --> E1["Entry: Table A"]
A2 --> E2["Entry: Table B"]
E1 --> EG["Entry group: Sales domain"]
E2 --> EG
EG --> SR["Search and filtering by aspect values"]
Search that people actually use
Search is the feature most users touch daily. A person types a term and the catalog returns matching entries, drawing on names, descriptions, column names and aspect values. Results respect permissions, so people only see what they are allowed to see. Good search depends on good metadata: descriptive names, clear descriptions and consistent labels turn a mediocre search into a useful one. Facets and filters let users narrow by system, project, domain or aspect values.
Describe every important table
A two-sentence description often does more for findability than any amount of tooling.
Use consistent aspect types
One agreed owner field beats five teams inventing five spellings of it.
Group by domain
Entry groups mirror how the business is organized, making browsing natural.
Mark authoritative data
Label the trusted version so users stop guessing between copies.
A well-run bookstore shelves by category, adds shelf-talkers saying “staff pick,” and keeps a computer where customers can search by title or topic. Aspects are the shelf-talkers, and entry groups are the shelves.
Launching a catalog full of unlabeled, undescribed entries. Users search, find cryptic names, and conclude the catalog is useless. Seed the most important assets with good metadata before announcing it.
6Organizing Data: Lakes, Zones and Assets
The original Dataplex model organized scattered storage into logical domains, and it remains a useful way to think.
In the lake-and-zone model, you create a lake for a business domain, such as sales or logistics. Inside the lake you create zones, which represent stages of data maturity. Into zones you attach assets, which are references to existing Cloud Storage buckets or BigQuery datasets. Importantly, attaching an asset does not move any data. It simply tells Dataplex that this storage belongs to this domain and stage.
flowchart TD
L["Lake: Sales domain"] --> Z1["Raw zone: data as received"]
L --> Z2["Curated zone: cleaned and organized"]
Z1 --> A1["Asset: Cloud Storage bucket"]
Z2 --> A2["Asset: BigQuery dataset"]
Z2 --> A3["Asset: Cloud Storage bucket of Parquet files"]
| Zone type | Purpose | Typical contents | Typical rules |
|---|---|---|---|
| Raw zone | Land data as it arrives | Original files in many formats | Flexible, minimal structure demands |
| Curated zone | Trusted, ready-to-use data | Cleaned tables in analytics-friendly formats | Stricter structure and quality expectations |
Why zones help
The raw and curated split mirrors the way data matures. Raw data is messy but valuable as a faithful record. Curated data is cleaned, conformed and documented. Marking the difference lets you apply different security and quality rules to each. Data engineers can work freely in raw zones, while analysts and applications consume from curated ones.
How to treat this model today
Google’s emphasis has shifted toward the universal catalog, entry groups and aspects as the primary way to organize and describe data, and toward managing data by domain through data products. Existing lake and zone deployments continue to matter and the underlying ideas, domain ownership and maturity stages, remain sound. For a new design, check current guidance on whether to use lakes and zones, and focus on the catalog concepts that apply to every resource.
A farm has a receiving dock where produce arrives dirty, and a packing house where it is washed, sorted and boxed. The dock and the packing house are zones. The produce does not change address when you draw the map.
Whatever structure you choose, align it with how your organization is actually owned. Domains that match team boundaries produce clear ownership. Domains invented by a central committee tend to go stale.
7Automatic Discovery
Cataloging by hand does not scale, so Dataplex can scan storage and register what it finds.
Discovery scans look inside attached storage, most often Cloud Storage buckets, and detect structured and semi-structured files, such as Parquet, Avro, ORC, CSV and JSON. They infer schemas and partitioning layouts, and they publish the results as catalog entries. In many configurations they also create BigQuery external tables, so analysts can query the files with SQL without any manual table definitions.
Point at storage
You attach a bucket or path and set include and exclude patterns to define what should be scanned.
Scan runs
The scan lists files, reads samples and infers formats, schemas and partitions.
Entries published
Each detected table-like file set becomes a catalog entry with its inferred structure.
Tables created
Where enabled, external tables appear in BigQuery pointing at the files.
Repeat on schedule
Later scans pick up new files, new partitions and changed schemas.
Challenges discovery faces
Discovery works best when files are tidy: consistent formats within a folder, sensible partition naming, and predictable layouts. It struggles with folders that mix unrelated file types, files whose schemas change constantly, and enormous numbers of tiny files. When it guesses wrong, you may see odd table boundaries or missing columns. The fix is usually better storage hygiene rather than tool tuning.
Keep one dataset per folder
Mixing unrelated files in one folder confuses schema inference.
Use consistent partition naming
Predictable folder names such as date partitions are detected reliably.
Exclude scratch and temporary paths
Prevents junk from becoming catalog entries.
Review what was found
Discovery proposes. Humans should confirm the important tables are described correctly.
A census taker walks every street and records each household. It is fast and thorough, but if houses have no numbers and families share rooms, the records get messy. Tidy streets make for an accurate census.
Scanning an entire bucket full of mixed content and then complaining about the noise. Scope scans narrowly to areas that hold real datasets.
8Metadata Import and Connecting Outside Systems
Your data estate is bigger than Google Cloud, and the catalog should reflect that.
Most organizations run databases, applications, business intelligence tools and pipelines that Dataplex cannot see natively. To include them, you bring their metadata into the catalog. The usual approach is to extract metadata from the source system with a connector or custom program, format it as import files describing entries and aspects, and load it through a metadata import job. The catalog then shows those external assets next to native ones, searchable in the same way.
flowchart LR
X1["On-premises database"] --> C["Connector or extraction program"]
X2["Third-party BI tool"] --> C
X3["Another cloud's warehouse"] --> C
C --> F["Import files in Cloud Storage describing entries and aspects"]
F --> J["Metadata import job"]
J --> CAT["Universal catalog"]
CAT --> U["Search, glossary, lineage context"]
Full and incremental imports
An import can replace the full set of entries from a source, which keeps the catalog in sync including deletions, or can update selected entries. Frequent small updates keep the catalog fresh, while periodic full syncs correct drift. Design the cadence around how fast the source changes and how much stale information users can tolerate.
Federation versus copy
Some designs keep metadata in the source system and let the catalog reach for it when needed, while others copy it in. Copying makes search fast and consistent but requires keeping it current. Reaching out keeps data fresh but depends on source availability. Most catalogs lean on copying because search must be fast and reliable.
| Approach | Strength | Weakness |
|---|---|---|
| Scheduled full import | Simple, self-correcting, handles deletions | Heavier for large sources |
| Frequent incremental import | Fresh and light | Needs change tracking and can drift |
| Event-driven update | Near real time | More engineering effort |
| Manual entry | No tooling needed | Does not scale and goes stale |
A travel guide covering other countries is compiled from correspondents’ reports. Each edition refreshes the pages. Metadata import is the correspondent network that keeps the guide up to date.
Start with the systems that hold your most important data and the largest user base, not the longest list of systems. A catalog that covers the top ten systems well beats one that covers a hundred poorly.
9Data Profiling
Before you can judge data, you need to see what it actually contains.
A profiling scan examines a table and computes statistics for each column. For every column you can see how many values are missing, how many are distinct, the smallest and largest values, averages and common values. The result is a snapshot of reality that often surprises people. A column documented as “always filled” may be forty percent empty. A “status” field with five intended values may hold eighty different spellings.
Null percentage
How often the column is empty. High values may signal collection problems.
Distinct count and uniqueness
Shows whether a column behaves like an identifier or a category.
Minimum, maximum and average
Reveals impossible values such as negative ages or future dates.
Top values
Shows the most common entries, quickly exposing spelling variants and placeholder junk.
Profiling at scale
Profiling every row of a huge table can be expensive, so scans often work on a sample. Sampling gives a good picture of general behavior but may miss rare problems. Choose the sample size according to table size and the importance of catching rare issues, and profile filtered segments such as recent partitions when only fresh data matters.
From profile to rules
A powerful use of profiling is to propose quality rules. If a profile shows a column is never empty and always within a range, a rule can encode that expectation so future violations are caught. This turns exploratory knowledge into automated protection.
Before buying a used car, you inspect it: mileage, tire wear, fluid levels. Profiling is the inspection report for a table, and quality rules are the checklist you run every time afterward.
Trusting documentation over measurement. Always profile data you plan to depend on. Documentation describes intent, while profiles describe reality.
10Data Quality: Rules, Scans and Scores
Quality checks turn the vague worry “is this data any good?” into a measurable answer.
A data quality scan evaluates a table against a set of rules and reports which rows or checks failed. Each rule expresses an expectation. Results include pass or fail per rule, the share of rows that passed and, often, an overall score. Because scans run on schedules or on demand, you can track quality over time and be alerted when it drops.
| Rule kind | Plain meaning | Example in words |
|---|---|---|
| Non-null | The value must be present | Every order has a customer identifier |
| Uniqueness | No duplicates | Each order number appears once |
| Range | Within allowed bounds | Discount percentage between zero and one hundred |
| Set membership | One of a list of allowed values | Status is open, shipped or cancelled |
| Pattern | Matches a defined format | Email addresses look like email addresses |
| Row-level condition | A logical test across columns | Ship date is never before order date |
| Table-level condition | A test across the table | Today’s row count is not far below the weekly average |
flowchart LR
T["Table or partition"] --> S["Quality scan with rules"]
S --> R{"Results"}
R -- "Pass" --> OK["Score recorded, consumers see healthy status"]
R -- "Fail" --> AL["Alert raised, failed row details saved"]
AL --> FIX["Owner investigates and fixes source or pipeline"]
FIX --> T
Quality dimensions
Quality has several dimensions. Completeness asks whether needed values are present. Validity asks whether values obey format and range rules. Uniqueness asks whether duplicates exist. Consistency asks whether related data agree. Timeliness asks whether data is fresh enough. Accuracy asks whether data matches reality, the hardest to test automatically. Organizing rules by dimension helps explain results to business audiences.
Acting on results
A score is useful only if someone acts on it. Results can be exported to tables for dashboards, can trigger alerts, and can be attached to the catalog entry so consumers see the latest quality status next to the data. Some teams wire failures into pipelines so that downstream steps stop automatically when upstream data is bad, preventing bad data from spreading.
Start small
A handful of high-value rules on critical tables beats hundreds of ignored rules.
Set thresholds
Decide how much failure is tolerable. Perfect data is rare, so define acceptable levels.
Assign owners
Every failing rule needs a named person who responds.
Check at the source
Catching problems early in the pipeline costs far less than repairing reports later.
A factory inspects parts at several points on the line. Catching a cracked part at the first station is cheap, while catching it after the car is assembled is a disaster. Quality scans are inspection stations along your data pipeline.
Creating rules and alerts that nobody reads. An alert that fires daily with no owner trains everyone to ignore it. Fewer rules with clear owners work better.
11Data Lineage
Lineage answers two of the most valuable questions in data: where did this come from, and what will break if I change it?
Lineage records how data moves and transforms. It shows that a revenue report reads from a summary table, which was built from an orders table, which was loaded from a source system. For supported Google Cloud services, lineage can be captured automatically as jobs run, for example when queries in BigQuery create tables from other tables. For other tools, lineage can be reported through an open standard interface so that custom pipelines also appear in the graph.
flowchart LR
SRC["Source system export"] --> RAW["Raw orders table"]
RAW --> CLEAN["Cleaned orders table"]
CLEAN --> SUM["Daily revenue summary"]
CUST["Customer table"] --> SUM
SUM --> REP["Executive dashboard"]
SUM --> ML["Forecasting model input"]
Two directions of use
Looking upstream helps with trust and debugging. When a number looks wrong, you trace backward to find which step introduced the problem. Looking downstream helps with change management. Before renaming a column or retiring a table, you see every report, model and job that depends on it, and warn the owners in advance.
Root cause analysis
Trace a bad figure back to the first step where the data went wrong.
Impact analysis
See everything affected before you change or delete an asset.
Compliance evidence
Show regulators where personal data flows and which systems touch it.
Cleanup of unused data
Find tables with no downstream consumers that can be retired to save cost.
Column-level detail
Table-level lineage says that table B comes from table A. Column-level lineage goes further and says that a specific column in B was computed from specific columns in A. This is especially valuable for sensitive data, because you can follow a personal identifier through every transformation that touches it. Availability of column-level detail depends on the source service, so check what is supported for your tools.
Lineage is a food label that traces a jar of jam back to the farm that grew the fruit, and also tells the farm which shops sell its fruit. If a farm has a contamination issue, you know exactly which jars to recall.
Lineage is only as complete as the tools that report it. Gaps appear wherever data passes through an unsupported tool or manual step. Document known gaps so users do not mistake a missing link for no dependency.
12Business Glossary and Shared Meaning
The most expensive data bugs are not technical, they are disagreements about what words mean.
Ask three departments to define “active customer” and you may get three answers. Finance counts customers who paid this quarter, marketing counts those who opened an email, and support counts anyone with an open account. Each is reasonable, and each yields a different number. A business glossary records agreed definitions for key terms, assigns an owner, and links terms to the data assets that implement them.
Identify contested terms
Start with terms that cause arguments or inconsistent reports, such as customer, revenue, active, churn.
Agree on definitions
Bring owners together to settle one meaning, or to split a term into clearly named variants.
Record in the glossary
Write the definition, the steward and related terms.
Link to data
Connect each term to the tables and columns that hold it.
Maintain
Review terms regularly as the business evolves.
Linking terms to data is where the glossary becomes more than a document. A user viewing a column sees the business term attached to it, with its definition, and can jump from a term to all the data that represents it. Glossaries can also be organized into categories and hierarchies, so broad concepts contain narrower ones.
| Without a glossary | With a glossary |
|---|---|
| Each report defines terms differently | One agreed definition with an owner |
| New staff learn meanings by asking around | Definitions are searchable and attached to data |
| Disputes over numbers take meetings to resolve | Disputes are resolved by reference to the recorded definition |
| Tables named inconsistently | Terms link related columns across tables |
An international team without a shared dictionary will misread each other’s words. A glossary is the shared dictionary that ensures “revenue” means the same thing to everyone in the room.
Treating the glossary as a one-off documentation project. Without owners and review cycles it goes out of date quickly and loses credibility.
13Security and Access Governance
Governance is incomplete unless the rules are actually enforced.
Dataplex works alongside Google Cloud Identity and Access Management and the security features of the underlying storage systems. Several layers matter, and good governance coordinates them.
Who can see metadata
Permissions on catalog entries and entry groups decide who can find and read descriptions. Metadata can itself be sensitive, so control it.
Who can read data
Access to the underlying tables and files is controlled by the storage systems. In lake-based setups, roles granted on a lake or zone can flow down to attached assets.
Column and row controls
Warehouse features such as column-level policies and row-level filters hide sensitive fields and subsets from unauthorized users.
Classification and tagging
Labels such as confidential or personal data drive which controls apply, and let reports show where sensitive data lives.
Encryption and keys
Data is encrypted at rest, with customer-managed keys available in the underlying services for stricter requirements.
Audit logging
Records of who accessed or changed what, essential for investigations and compliance.
Policy by classification
A scalable pattern is to tag data by sensitivity and let policies key off the tags rather than managing permissions table by table. When a column is tagged as personal data, a policy automatically restricts it to approved groups. New tables with the same tag inherit protection, and removing a tag changes access consistently. This reduces the chance that someone forgets to secure a new table.
Least privilege and groups
Grant access to groups, not individuals, and grant the narrowest role that does the job. Separate the people who manage metadata from those who read data, and separate administrators from consumers. Review memberships regularly, because permissions accumulate silently as people change roles.
flowchart TD
CL["Classify column as personal data"] --> TG["Tag stored in catalog"]
TG --> POL["Policy: only approved group can read tagged columns"]
POL --> ENF["Enforced by warehouse column-level security"]
ENF --> AUD["Access logged for audit"]
U1["Approved analyst"] --> ENF
U2["Unapproved user"] --> ENF
An office building uses badges that open only certain doors, and rooms with sensitive files have extra locks. Tagging a room as “restricted” automatically applies the extra lock, no matter which floor it is on.
Remember that hiding a table in the catalog is not the same as securing the data. Always enforce access at the data layer too, so that a person who guesses a table name still cannot read it.
14Sensitive Data Discovery and Privacy
You cannot protect personal data you do not know you have.
Privacy regulations expect organizations to know where personal data lives, why it is kept and who can reach it. Finding it by hand across thousands of tables is unrealistic. Google Cloud offers sensitive data protection services that scan data for patterns such as names, national identifiers, payment card numbers and contact details. Their findings can feed tags into the catalog, so classification becomes part of the metadata that drives policy.
Scan for sensitive patterns
Inspection services look at column contents and names for known sensitive types.
Record findings as tags
Detected types are attached to catalog entries or columns as classification metadata.
Apply policy
Access controls, masking and retention rules key off those tags.
Review and report
Dashboards show where sensitive data lives, who touches it and where gaps remain.
Limits of automated detection
Pattern detection produces false positives, such as a product code that looks like an identifier, and false negatives, such as free-text fields where personal details hide inside sentences. Treat detection as strong evidence needing human review, not as a final verdict. Combine automatic scans with steward confirmation for high-risk data.
Masking
Show partial or replaced values to users who need the shape but not the substance.
Tokenization
Swap sensitive values for stable tokens so joins still work without revealing the original.
Retention rules
Delete or archive data after the period it is needed, as required by policy.
Purpose labels
Record why data may be used, so access matches the permitted purpose.
A security scanner at an airport flags suspicious items, but a human officer inspects the bag. Automated detection flags likely sensitive data, and a steward confirms before heavy restrictions are applied.
Assuming a clean scan means no personal data exists. Unstructured text, images and unusual formats are harder to inspect, so keep process controls even when scans find nothing.
15Data Products and Domain Ownership
Modern governance shifts from controlling data centrally to enabling teams to publish trustworthy data.
A data product is a curated set of data assets packaged for consumers, with a clear owner, documentation, quality expectations and defined access. Instead of users hunting through raw tables, they find a product such as “Customer Orders” that comes with a description, known limitations, a contact person, current quality status and the right to request access.
Discoverable
Listed in the catalog with clear names and descriptions.
Trustworthy
Quality rules run and results are visible.
Owned
A named team is accountable for it.
Documented
Describes contents, refresh rate, limitations and intended use.
Accessible
Has a defined path for approved users to get access.
Stable
Changes are communicated so consumers are not surprised.
Federated governance
In a federated model, a central group sets the standards, such as required metadata, quality expectations and security classifications, while domain teams own their data and apply those standards. This balances consistency with speed. Domain teams know their data best, and central teams ensure that everyone’s products look and behave alike to consumers.
| Model | Strength | Weakness |
|---|---|---|
| Fully centralized | Strong consistency | Bottleneck, slow, far from the data |
| Fully decentralized | Fast and close to the business | Inconsistent, duplicated effort, security gaps |
| Federated | Shared standards with local ownership | Needs good coordination and clear roles |
A food hall has many independent stalls, each cooking its own cuisine, but all must pass the same hygiene inspection and display prices the same way. Each stall is a domain, the hygiene code is central governance, and the menu is the data product.
Launch with a few pilot data products from willing teams, make them excellent, and use them as examples. Examples persuade more than policy documents do.
16Processing Data with Dataplex Tasks
Dataplex can also run data processing jobs, though many teams use other tools for the heavy lifting.
Dataplex includes a way to schedule serverless processing tasks, often based on Apache Spark, and to run curated templates for common jobs, such as converting files to analytics-friendly formats or moving data between storage and the warehouse. These tasks run without cluster management, in the spirit of serverless Spark offerings. They can be attached to lakes and zones, and results are registered back to the catalog.
Format conversion
Convert raw files into columnar formats such as Parquet for faster, cheaper queries.
Light transformation
Clean and reshape data between raw and curated zones.
Scheduled housekeeping
Compaction and routine data maintenance jobs.
Notebook-based exploration
Interactive environments for analysts to explore governed data.
For larger and more complex pipelines, organizations usually rely on dedicated processing and orchestration services and use Dataplex to catalog, check and trace what those pipelines produce. Because processing capabilities and their packaging change over time, verify what is currently supported before building on this feature.
A property manager can do small repairs personally, but for a major renovation hires specialist contractors and then inspects and documents the result. Dataplex tasks handle the small jobs, while specialists handle the big builds.
Treating Dataplex as the main pipeline engine. Its core value is governance, so use it to oversee pipelines, not to replace purpose-built processing services.
17Working with BigQuery and Cloud Storage
Most Dataplex deployments revolve around two storage systems, and each has its own character.
| Aspect | BigQuery | Cloud Storage |
|---|---|---|
| Catalog entries | Created automatically for datasets and tables | Created by discovery scans or attached assets |
| Schema | Defined in the table | Inferred from files |
| Quality and profile scans | Run directly on tables | Run on tables discovered from files |
| Access controls | Dataset, table, column and row policies | Bucket and object permissions |
| Lineage | Captured automatically for supported operations | Depends on the processing tools used |
| Typical role | Curated, query-ready analytical data | Raw landing, lake files, exports |
Bridging the two worlds
A recurring challenge is governing both a lake and a warehouse consistently. Discovery can expose lake files as external tables in BigQuery so one query interface and one security model cover both. Open table formats are also gaining ground for lake data, giving file-based data more warehouse-like behavior such as transactions and schema management. Whatever storage you choose, the catalog should present both as parts of one estate, with the same metadata standards.
Keep layouts discoverable
Use predictable folder structures and file formats in the lake.
Apply the same labels in both
Sensitivity and ownership aspects should look identical across systems.
Match regions
Keep buckets, datasets and scans in compatible locations to avoid friction and cost.
Promote data deliberately
Move proven datasets from raw to curated with clear quality gates.
A company stores bulk materials in a warehouse and finished goods in a showroom. The inventory system must track both under the same rules, even though the buildings are very different.
Check supported regions and locations before you design. Scans, catalogs and the storage they inspect often need compatible locations, and fixing a mismatch later is tedious.
18Monitoring, Logging and Operating Dataplex
Governance tools are themselves systems that need watching.
Scans, imports and tasks run as jobs, and jobs can fail, run late or produce surprising results. Integration with Cloud Logging and Cloud Monitoring lets you see job status and errors, and set alerts. Audit logs record administrative changes and data access, which supports security reviews.
Scan job outcomes
Failed or stuck discovery, profile and quality scans.
Import job results
Rejected records, missing entries and unexpected deletions after imports.
Quality score trends
Gradual declines that signal upstream drift.
Metadata coverage
The share of important assets with owners, descriptions and classifications.
Permission changes
Unexpected grants or removals recorded in audit logs.
Governance metrics worth tracking
Beyond technical health, track whether governance is working. Useful measures include the percentage of critical tables with an owner, the percentage with descriptions, the share of rules passing, the time taken to resolve quality incidents, the number of searches that end in a click, and how many sensitive columns are unprotected. These show whether the program is changing behavior, not only whether the software runs.
| Metric | What it shows | Healthy direction |
|---|---|---|
| Owner coverage | Accountability for important data | Rising toward complete |
| Description coverage | Findability and understanding | Rising |
| Quality pass rate | Trustworthiness of data | Stable or rising |
| Incident resolution time | Responsiveness of owners | Falling |
| Unclassified sensitive columns | Privacy exposure | Falling toward zero |
A school measures not only whether the buildings are open but also whether students are learning. Governance dashboards measure whether data is becoming better managed, not just whether the tools are switched on.
Measuring only tool activity, such as number of scans run. Activity is not outcome. Measure coverage, quality and response time.
19Cost Model and Optimization
Governance should cost far less than the problems it prevents, but costs still deserve attention.
Pricing differs by capability and changes over time, so always consult the current pricing pages. In general, costs come from the processing used by scans, such as profiling and quality checks, which consume compute when they read data; from metadata storage and operations in the catalog; from processing tasks you run; and from the underlying storage and queries your data lives in. Lineage and catalog features may be billed or included in different ways depending on edition and usage.
Practical ways to control spend
Scan what matters
Focus quality and profile scans on critical tables, not every table in the estate.
Use sampling
Sample large tables when full scans add little extra insight.
Scan incrementally
Check only new or changed partitions instead of full history every time.
Match frequency to change
A table that updates daily does not need hourly scans.
Exclude junk paths
Narrow discovery scopes so scans do not read scratch data.
Retire unused data
Use lineage to find unused tables and remove them, saving storage and scan cost.
Where Governance Saves Money
- Less analyst time wasted hunting for data
- Fewer expensive incidents from bad data
- Retiring unused tables and duplicate copies
- Faster, cheaper audits and compliance reviews
- Avoiding duplicated pipelines
Where Governance Can Waste Money
- Scanning everything on tight schedules
- Scanning huge tables fully when sampling would do
- Rules nobody acts on
- Duplicate catalogs and overlapping tools
- Unscoped discovery of scratch storage
Insurance and safety inspections cost money, but far less than a factory fire. Spend inspection effort where the risk and value are highest, not equally on every closet.
Run a small pilot and read the bill before scaling up scans. Real costs depend on data volumes and frequency, and a pilot reveals them cheaply.
20Adoption Roadmap: From Zero to a Working Program
Tools do not create governance, habits do, and habits grow in stages.
Assess and prioritize
List your key data assets, pain points and regulatory duties. Pick a first domain with real business value and a willing owner.
Define standards
Agree on required metadata: owner, description, sensitivity, domain. Create aspect types that capture them.
Catalog and describe
Bring in the pilot domain’s data, run discovery where needed, and fill in descriptions and owners for the key assets.
Add quality and classification
Profile the key tables, create a few high-value quality rules and classify sensitive columns.
Enforce policy
Connect classifications to access controls, and verify with audit logs.
Publish data products
Package the best datasets as documented, owned products and announce them to consumers.
Expand and measure
Repeat for further domains, track coverage and quality metrics, and adjust standards from experience.
Roles that make it work
Data owner
A business leader accountable for a domain’s data and its use.
Data steward
The hands-on person who maintains definitions, quality rules and metadata.
Platform team
Runs the tooling, standards and automation.
Security and compliance
Define policies and verify they are enforced.
Cleaning a cluttered house works room by room, not all at once. Finish one room, keep it tidy, and the habit spreads. Start with one domain, make it exemplary, then move on.
Trying to catalog the entire enterprise before delivering any value. Programs that take a year to show results lose sponsorship. Deliver visible wins in weeks.
21Design Patterns and Anti-Patterns
Teams that succeed share a few patterns, and teams that struggle share a few traps.
Patterns that work well
Standard aspect types for every asset
All entries carry the same small set of required fields: owner, description, domain and sensitivity. Search and policy work uniformly because the shape never varies.
Quality gates between zones
Data moves from raw to curated only after passing defined quality rules. Failures stop the promotion and alert the owner, so the curated layer stays trustworthy.
Tag-driven security
Sensitivity tags drive access policy. New data inherits protection by being labeled correctly, instead of requiring bespoke permission changes.
Lineage-informed change management
Before any schema change, owners consult downstream lineage and notify consumers. Surprise breakages drop sharply.
Federated stewardship
Central teams define standards and provide automation. Domain teams own their data products and meet the standards locally.
Anti-patterns to avoid
Problem
The catalog is populated automatically and never curated by humans.
Why It’s Harmful
Users see thousands of cryptic entries without descriptions or owners and conclude the catalog is useless, so adoption dies.
Correct Approach
Curate the most important assets first, require descriptions and owners for them, and mark trusted data clearly.
Problem
Quality rules are created in bulk with no owners or response process.
Why It’s Harmful
Constant alerts nobody handles create noise, hide real problems and waste scan cost.
Correct Approach
Create fewer, meaningful rules on critical data, assign each an owner and track response time.
Problem
One central team is expected to document, classify and fix everything in the company.
Why It’s Harmful
The team becomes a bottleneck far from the data, knowledge is shallow, and backlogs grow.
Correct Approach
Adopt federated ownership, with central standards and automation but domain teams responsible for their own data.
Problem
Catalog visibility is treated as the security control, with data layers left open.
Why It’s Harmful
Anyone who knows or guesses a table path can read it, since hiding an entry does not protect the data itself.
Correct Approach
Enforce permissions in the storage and warehouse layers, and use the catalog to describe and classify, not as the only barrier.
22Dataplex Compared with Related Options
Governance tools overlap, so it helps to see where each fits.
| Option | What it is | Strength | Limitation |
|---|---|---|---|
| Dataplex | Google Cloud governance, catalog, quality and lineage layer | Deep integration with Google Cloud services and security | Strongest for Google Cloud estates, outside coverage needs import work |
| BigQuery built-in features | Native table metadata, policies and labels | Simple, no extra service for basic needs | Limited cross-system view and governance workflows |
| Third-party enterprise catalogs | Commercial catalog and governance platforms | Broad multi-cloud connectors and rich workflows | Extra vendor, cost and integration effort |
| Open source catalogs and lineage tools | Community-built metadata platforms | Flexible, no license cost | You operate and maintain them |
| Spreadsheets and wikis | Manual documentation | Zero tooling cost to begin | Go stale, no enforcement, no automation |
How to choose
If your data lives mostly in Google Cloud, Dataplex is a natural first choice because discovery, lineage and security integrate with little effort. If your estate spans many clouds and on-premises systems and you already have an enterprise catalog, the question becomes how to connect the two, for example by sending Google Cloud metadata to the enterprise tool or the reverse. Avoid running several competing catalogs where users must check each one, since that recreates the very sprawl you set out to cure.
Mostly Google Cloud data
Dataplex covers most needs natively.
Heavy multi-cloud estate
Consider an enterprise catalog and integrate Dataplex as a source.
Small team, simple needs
Start with native BigQuery features and add Dataplex as needs grow.
Strict engineering culture and skills
An open source catalog may suit, if you accept the operating burden.
A city can have a main public library or a patchwork of private collections. Visitors prefer one directory they can trust. Choose the catalog that lets people ask one place where the data is.
23Best Practices and Common Mistakes
A short checklist separates governance programs that last from those that fade.
Best practices
Start with a business problem
Tie the first domain to a pain people feel, such as slow reporting or audit pressure.
Standardize metadata
Define required aspects once and reuse them everywhere.
Name owners for everything important
Unowned data decays. Every key asset needs a responsible person.
Automate the boring parts
Use discovery, import, profiling and classification scans so humans focus on judgment.
Measure quality continuously
Schedule scans, chart scores and respond to drops quickly.
Drive security from tags
Classification-based policy scales better than manual permission edits.
Use lineage before changing things
Check downstream impact, then notify consumers.
Publish and promote data products
Make the good data easy to find and the path to access clear.
Common mistakes
Treating governance as a tool installation rather than an organizational change. Software without owners, standards and incentives changes little.
Trying to boil the ocean. Cataloging everything before delivering value burns goodwill. Start narrow and visible.
Ignoring location and region constraints. Resources in incompatible locations cause scan and import failures that surface late.
Letting metadata rot. Descriptions, owners and glossary terms need periodic review, or users stop trusting them.
Building on features whose names and packaging may have changed. Always verify current naming, availability and limits in the official documentation before committing a design.
Hold a short monthly review of the top quality incidents, the least-described important tables and any unclassified sensitive columns. Small regular reviews keep the program healthy.
24Real-World and Industry Scenarios
Concrete stories show how the pieces fit under real pressure.
Retail: one version of the customer
A retailer has customer data in an online store database, a loyalty system and a data lake of clickstream files. Discovery and import bring all three into the catalog. The glossary settles what “active customer” means. Quality rules on the customer table catch duplicate records, and lineage shows which dashboards depend on it. Marketing and finance finally report matching numbers.
Banking: proving control to regulators
A bank must show where personal and financial data lives and who can access it. Sensitive data scans tag columns, tag-driven policies restrict access to approved groups, lineage maps flows into reports, and audit logs record access. When auditors ask, the bank answers from the catalog in hours rather than weeks.
Healthcare: protecting patient information
A health provider keeps records in several systems and a research lake. Classification flags patient identifiers, masking gives researchers de-identified views, and retention rules delete data after required periods. Data products offer approved research datasets with clear usage rules.
Manufacturing: trusting sensor data
A manufacturer streams sensor readings into storage. Quality rules check ranges and freshness, flagging broken sensors early. Lineage connects readings to maintenance models, so engineers see which models are affected when a sensor feed goes bad.
Media: finding and reusing datasets
A media company has dozens of teams building overlapping audience datasets. A catalog with domain entry groups, descriptions and owners lets teams discover existing data first, reducing duplicate pipelines and storage cost.
Technology company: safe data for machine learning
A technology company trains models from many sources. Data products with quality scores and lineage give data scientists confidence in inputs, and classification keeps restricted fields out of training sets.
In every story, the data does not move. What changes is that people can find it, trust it and prove they control it, which is the entire promise of Dataplex.
25Frequently Asked Questions
Short answers to the questions teams ask most when adopting Dataplex.
No. Your data stays in Cloud Storage, BigQuery or wherever it lives. Dataplex stores metadata about it and runs governance services over it.
The catalog capabilities have been brought under the Dataplex name, and newer documentation uses terms such as the universal catalog. Check current documentation for the exact naming and any migration guidance for older catalog features.
They remain supported for existing designs and the ideas behind them are sound, but the emphasis has moved toward catalog-wide concepts and data products. For a new design, confirm current guidance.
Yes, by importing metadata from those systems through connectors or custom extraction, formatted as import files and loaded with an import job.
You define rules, such as non-null, unique, in range or matching a pattern. Scans evaluate the data against those rules and report pass rates and failures, which you can schedule, chart and alert on.
Only for supported services and for tools that report lineage through the supported interface. Manual steps and unsupported tools appear as gaps, so document them.
It helps you find and classify sensitive data, apply tag-driven access policies, trace flows with lineage and keep audit records. It supports compliance but does not by itself make an organization compliant.
Pick one valuable domain, catalog it, add descriptions and owners, run profiling and a few quality rules, and classify sensitive columns. Show results in weeks, then expand.
Done well, it speeds them up, because they spend less time finding data, debugging mysterious numbers and handling surprise breakages. Done badly, it becomes paperwork, so automate and keep requirements small.
Automate metadata collection, schedule discovery and imports, assign owners with periodic review duties, and track coverage metrics so decay becomes visible.
26Summary and Key Takeaways
Dataplex gives organizations a way to understand and control data without moving it. A shared metadata foundation powers search, discovery, profiling, quality measurement, lineage, glossaries, classification, policy and data products. The data stays where it is, while knowledge about it, and rules over it, are gathered into one trusted place.
Success depends on habits as much as features. Start with one valuable domain. Standardize a small set of required metadata. Name owners. Measure quality where it matters, classify sensitive data, drive access from tags and use lineage before changing anything. Treat the tool as an enabler of a federated program in which domain teams own their data products and a central group sets standards. Because product names and packaging in this area evolve, verify current documentation before finalizing a design.
Key Takeaways
- Governance without moving data — Dataplex catalogs and governs data in place across Cloud Storage, BigQuery and imported outside systems, rather than storing or relocating it.
- The catalog is the foundation — entries, aspects and entry groups carry technical and business metadata, and every other capability builds on them.
- Discovery, profiling and quality turn guesses into facts — scans find data, summarize what it contains and test it against rules, with owners acting on the results.
- Lineage answers where and what-if — upstream tracing speeds debugging and downstream views prevent surprise breakage and support compliance evidence.
- Meaning and security need people and policy — glossaries align definitions, and classification tags drive consistent access controls enforced at the data layer, not just in the catalog.
- Data products and federated ownership scale — domain teams publish documented, owned, quality-checked products under shared central standards.
- Start narrow, measure, and verify names — begin with one domain, track coverage and quality outcomes, control scan costs, and confirm current product naming and features in the official documentation.