Dataplex in Google Cloud: The Complete Intermediate Guide
Learn how Google Cloud finds, describes, checks, and protects data across your whole organization, from catalogs and glossaries to quality scans, lineage, and governance, explained step by step in plain English.
Imagine a gigantic library that has grown for twenty years. Books sit in many buildings, some on shelves, some in boxes, some in a basement nobody visits. There is no catalog. A researcher who needs one specific report must wander for days, and even after finding something, nobody can say who wrote it, whether it is still correct, or who may read it. Most companies treat their data exactly like that library. Dataplex is the librarian, the catalog, the quality inspector, and the security desk, all in one service.
This guide explains Dataplex from the first idea to production-grade design choices. You do not need any data governance background. Every new word is explained the first time it appears, with an everyday comparison. By the end you will know what the service does, how its catalog is organized, how metadata flows in, how quality checks and lineage work, how access is controlled, and which mistakes to avoid.
1What Dataplex Is and Why It Exists
Dataplex is Google Cloud’s service for discovering, understanding, governing, and trusting data that is scattered across many systems.
The problem: data everywhere, trust nowhere
A modern company stores data in many places at once. Sales figures sit in a data warehouse. Raw logs sit in storage buckets. Customer records live in operational databases. Machine learning features sit in yet another system. Each team builds its own tables, names its own columns, and fixes its own errors. Soon nobody knows what data exists, which copy is correct, where a number came from, or who is allowed to see it.
This causes real damage. Analysts spend more time hunting for data than analyzing it. Two dashboards show different revenue figures, and nobody can explain why. A privacy officer cannot answer which tables contain personal information. An engineer changes a column and silently breaks twelve reports. The root cause is the same each time: the company has data but lacks reliable information about its data.
Think of a supermarket where products have no labels, no price tags, no expiry dates, and no aisle signs. The food might be excellent, but shoppers cannot find it, cannot trust it, and cannot tell whether it is safe. Dataplex puts labels, signs, and inspection stickers on your data.
Information about data is called metadata
Metadata means data about data. The name of a table, the meaning of a column, the owner of a dataset, the date it was last updated, and the quality score of its contents are all metadata. Managing metadata well is the foundation of data governance, because you cannot protect, fix, or share what you cannot describe.
A note about names and history
Names in this area have changed several times, so it helps to know the story. Google first offered a service called Data Catalog for tagging and searching data. It then introduced Dataplex, which organized data into lakes, zones, and assets and added quality checks and discovery. Later the catalog features were presented as Dataplex Universal Catalog. According to Google’s documentation, on April 10, 2026 Dataplex Universal Catalog was renamed Knowledge Catalog. Google states that the API, client library, command line tool, and permission role names did not change, so the technical name Dataplex still appears in endpoints and roles.
Because many engineers, tutorials, and tools still say Dataplex, this guide uses Dataplex as the main name and mentions Knowledge Catalog where it matters. The ideas are the same under either name.
Dataplex does not usually move or copy your data. It builds an index and a layer of knowledge on top of data that stays where it is, such as BigQuery tables or Cloud Storage files.
2Core Concepts You Must Know First
A handful of data governance ideas explain nearly everything in Dataplex.
Governance vocabulary can sound heavy, but each idea is simple. Learn these eight terms and the rest of the guide will feel natural.
Data Asset
Anything that holds or represents data: a table, a file, a topic, a dashboard, or a machine learning model.
Metadata
Facts about an asset, such as its name, owner, schema, description, and last update time.
Data Catalog
A searchable inventory of assets and their metadata, like a library catalog for data.
Business Glossary
An agreed list of business terms, such as “active customer”, with clear definitions linked to the data that represents them.
Data Quality
How accurate, complete, fresh, and consistent data is. Quality rules turn those ideas into automatic checks.
Data Lineage
The recorded journey of data from its sources, through each transformation, to the places it is used.
Data Governance
The rules and habits that decide who owns data, who may use it, and how it must be protected.
Data Product
A curated bundle of data assets packaged for reuse, with an owner and clear meaning, like a product on a shelf.
Technical metadata and business metadata
There are two flavors of metadata. Technical metadata describes the machine side: column names, data types, sizes, and update times. Much of it can be collected automatically. Business metadata describes the human side: what a table means, who is responsible for it, how sensitive it is, and which business term it represents. People usually have to add it, though modern tools can help suggest it. A good catalog holds both, because analysts need meaning as much as structure.
A library record has technical details such as page count and size, and business details such as the topic, the author’s reputation, and which shelf it belongs to. A catalog with only page counts would be nearly useless for a researcher looking for the best book on a subject.
Governance is more than blocking access
Many people hear “governance” and picture a locked door. Good governance is closer to a well-run airport. It makes sure the right passengers reach the right gates quickly, with clear signs and safety checks that people trust. The aim is not to stop data use but to make safe, confident data use easy.
Many teams believe a catalog is a one-time documentation project. A catalog is a living system. If metadata is not updated as data changes, people stop trusting it, and the tool quietly dies.
3The Pieces of the Platform
Dataplex bundles several capabilities that work together around one shared metadata layer.
It helps to see Dataplex as a toolbox rather than a single tool. Each capability solves one problem, and they reinforce each other because they share the same catalog. Google’s use-case documentation groups them under search and retrieval, enrichment and metadata management, aggregation, governance, data quality and profiling, and data lineage.
Catalog and Search
An index of data and AI assets across Google Cloud and other environments, searchable by keyword or natural language.
Discovery
Automatic scans that find data, such as files in Cloud Storage, and register what they contain.
Business Glossary
A taxonomy of business terms linked to columns and assets, so people find data by meaning.
Data Profiling
Statistics about columns, such as value ranges and null counts, that reveal anomalies and suggest quality rules.
Auto Data Quality
Scheduled scans that test tables against rules and produce quality scores and alerts.
Data Lineage
Automatic and imported records of how data flows between sources, jobs, and tables.
Data Products
Curated bundles of assets with owners and descriptions, published for others to find and reuse.
Governance Controls
Centralized policies, such as column-level security, applied consistently across the data estate.
flowchart TD
SRC["Data sources: BigQuery, Cloud Storage, databases, streams"] --> CAT["Shared metadata catalog"]
CAT --> SEARCH["Search and discovery"]
CAT --> GLOS["Business glossary"]
CAT --> DQ["Profiling and data quality"]
CAT --> LIN["Lineage"]
CAT --> GOV["Governance policies"]
CAT --> DP["Data products"]
Read the diagram from the top. Information about your sources flows into one shared catalog, and every other capability reads from and writes to that catalog. This is why a quality score can appear on a table’s catalog page, why lineage can show glossary terms, and why a governance rule can follow a table wherever it appears.
A hospital has one patient record that doctors, nurses, labs, and pharmacists all read and update. Each department does a different job, yet they all trust the same record. The shared catalog is the patient record for your data.
How it relates to other Google Cloud services
Dataplex does not replace storage or processing tools. BigQuery still stores and queries tables. Cloud Storage still holds files. Dataproc and Dataflow still process data. Dataplex sits across them, describing and watching what they do. Google’s documentation lists metadata sources such as BigQuery, Vertex AI, Bigtable, Spanner, Pub/Sub, and Cloud Storage, along with connectors for third-party systems.
You do not need every capability on day one. Many teams begin with search and automatic metadata, add a glossary for key terms, then introduce quality scans on their most important tables.
4How the Catalog Is Organized
The catalog uses entries, aspects, and groups, which work like index cards, fill-in forms, and drawers.
Entries
An entry is the catalog’s record for one data asset. Google’s documentation gives the example of a BigQuery table, which becomes an entry identified by its project, dataset, and table name. Columns appear on the entry when a schema aspect is attached to it. Every searchable thing in the catalog is an entry.
Aspects and aspect types
An aspect is a group of related metadata fields attached to an entry, for example an “ownership” aspect with fields for team and contact, or a “sensitivity” aspect with a classification level. An aspect type is the reusable template that defines which fields an aspect has. Aspects are stored inside the entry they describe rather than as separate objects. Google states that aspect types succeed the tag templates of the older Data Catalog service.
An entry is an index card for a book. An aspect type is a blank form such as “Condition report” with set boxes. An aspect is that form filled in and stapled to the card. Every card can carry several different filled forms.
Entry types and entry groups
An entry type is a template saying which aspect types an entry of that kind must carry, so every BigQuery table entry looks consistent. An entry group is a container for entries. According to Google’s documentation, the entry group is where you set access control, project attribution, and location for the entries inside it. Think of it as a labeled drawer with its own lock.
Entry links
Assets do not live alone, so the catalog also records relationships through entry links. The documented link types are synonym, related, definition, and schema-join. A synonym link says two terms mean the same thing. A definition link connects a glossary term to the asset or column it defines. A schema-join link says two tables can be joined on certain columns. Links can be symmetric or directional, and many can point to a specific column.
flowchart TD
EG["Entry group: a drawer with its own access rules"] --> E1["Entry: sales orders table"]
EG --> E2["Entry: customers table"]
ET["Entry type: required aspect types"] -.-> E1
E1 --> A1["Aspect: schema"]
E1 --> A2["Aspect: ownership"]
E1 --> A3["Aspect: sensitivity"]
E1 -- "schema-join link" --> E2
GT["Glossary term: Customer"] -- "definition link" --> E2
The diagram shows one drawer holding two entries. The orders entry carries three filled forms and a join relationship to the customers entry. A business term links to the customers entry as its definition. Together, these pieces describe not only what exists but how things connect.
| Building block | What it is | Everyday comparison |
|---|---|---|
| Entry | Catalog record for one asset | Index card |
| Aspect type | Template for a set of fields | Blank form |
| Aspect | Filled-in set of fields on an entry | Completed form stapled to the card |
| Entry type | Template listing required aspect types | Card layout for a kind of item |
| Entry group | Container with access, project, and location settings | Locked drawer |
| Entry link | Relationship between entries | String between two cards on a pinboard |
Creating aspect types casually, with overlapping fields and unclear names. Once many entries carry them, changing them is painful. Design a small, clear set of aspect types first and reuse them everywhere.
5How Metadata Gets Into the Catalog
A catalog is only as good as what flows into it, so Dataplex offers automatic, connector-based, and custom ways to load metadata.
Automatic metadata from Google Cloud
For many Google Cloud services, technical metadata arrives without you doing anything. Google’s documentation lists sources including BigQuery (with related items such as sharing listings and Dataform), Dataproc Metastore, Iceberg REST catalogs, Vertex AI, Bigtable, Spanner, and Pub/Sub, with Cloud Storage handled through discovery. AlloyDB and Cloud SQL can be added after enabling the integration, and Looker support is listed as a preview. Because this metadata is collected automatically, new tables appear in the catalog soon after they are created.
Discovery scans for files
Files in Cloud Storage have no built-in table definitions, so Dataplex can run discovery scans that look inside buckets, recognize structured data such as common file formats, work out the columns, and register the results as catalog entries. This turns a pile of files into something people can search and even query.
Third-party and custom sources
Few companies keep everything in Google Cloud. Google’s documentation mentions connectors for third-party systems such as dbt Core, Oracle, MySQL, PostgreSQL, and SQL Server, plus managed connectivity pipelines and custom ingestion through open APIs. A custom import lets you load metadata from an in-house system, so the catalog becomes a single view across the whole company rather than just one cloud.
flowchart LR
A["Google Cloud services: BigQuery, Pub/Sub, Spanner, and more"] --> C["Knowledge Catalog index"]
B["Cloud Storage discovery scans"] --> C
D["Third-party connectors: databases, dbt, and others"] --> C
E["Custom ingestion through APIs"] --> C
C --> F["Search, glossary, quality, lineage, governance"]
C -. "Change feed to Pub/Sub" .-> G["Downstream tools and automation"]
Change feeds keep other tools in sync
Catalog information is useful beyond the catalog itself. Google’s documentation describes change feeds that send near real-time notifications of metadata changes to a Pub/Sub topic. A security tool could react when a table’s sensitivity label changes, or a data platform could refresh its documentation site whenever descriptions are edited.
Metadata ingestion is like a library accepting new books. Some arrive through the front door automatically from the main publisher. Some come in unlabeled boxes that a librarian must open and register. Some arrive from partner libraries. A well-run library welcomes all of them and records each in the same catalog format.
Let machines collect technical metadata and put human effort into business meaning. People are slow at copying column names but irreplaceable when explaining what a column means to the business.
Forgetting that date-sharded tables, such as one table per day, are indexed as separate tables. A catalog can fill up with thousands of near-identical entries. Prefer partitioned tables where possible, and plan for the entry count.
6Search and Discovery
Search is the front door of the catalog. If people cannot find data quickly, nothing else matters.
What search covers
Search looks across entries and their metadata, so a query can match a table name, a column name, a description, or a business term. Google’s documentation says search requires one of the catalog roles, such as Viewer, Editor, or Admin, and that results are limited to the same organization and the same VPC Service Controls perimeter. It also notes that the search API does not search across organizations. Creating and managing catalog resources, search calls, and console search are described as free of charge, while metadata storage is billed.
Keyword and natural language search
Traditional search matches words. Newer capabilities also let people ask questions in plain language, such as “Which tables hold monthly subscription revenue?”, and use AI to find relevant assets. Google’s documentation describes use cases where an agent answers natural-language queries about data assets through the API, and where a single request returns context about assets that a language model can use. These capabilities make the catalog useful not only to people but also to AI applications that need to know which data to trust.
Keyword search is asking a librarian for a book by its exact title. Natural language search is describing what you need and letting the librarian suggest the best three books, with a short note on why each fits.
Making search results trustworthy
Finding an asset is only half the job. A good result page tells the searcher what the asset is, who owns it, how fresh it is, whether it has passed quality checks, which business terms it matches, and where it came from. Without these signals, people find ten similar tables and have no idea which one to use. This is why the catalog connects search with quality scores, lineage, and glossary links.
Description
A plain sentence explaining what the asset contains and how it should be used.
Owner
A named person or team who answers questions and fixes problems.
Quality Score
The share of quality rules passed, so people see at a glance if the data is healthy.
Freshness
When the data was last updated, so nobody builds a report on a stale table.
Sensitivity
A label showing whether the data contains personal or restricted information.
Lineage
The upstream sources and downstream users of the asset, revealing how trustworthy and how important it is.
Ask a few analysts which data they could not find last month, and use their answers to improve descriptions and terms. The best search improvements come from real searches that failed.
Assuming people can see everything in search. Results respect permissions and organizational boundaries. If someone says an asset is missing, check their role and the perimeter before suspecting the index.
7Business Glossary and Enrichment
A glossary gives the whole company one shared language, which is often the biggest source of confusion in data work.
Why a glossary matters
Ask three departments to define “active customer” and you may get three answers. Sales counts anyone who bought this year. Support counts anyone with an open account. Finance counts anyone who paid in the last ninety days. All three are reasonable, yet reports using different meanings under one name cause endless arguments. A business glossary records agreed definitions, so everyone knows which meaning a given number uses.
Google’s documentation describes a glossary as a taxonomy of business terms that you can link to columns and assets, and use to find assets by term. Terms can be grouped into categories, and links such as synonym and definition show how terms relate to each other and to the data.
A glossary is a shared dictionary for a team of travelers from different countries. Without it, the word “football” might mean two different sports, and the team plans the wrong evening. With it, everyone knows exactly what is meant.
Linking terms to data
A glossary becomes powerful when terms point to real columns. If the term “Customer ID” links to the matching columns in five tables, a person searching for the term finds all five at once, and a data steward can see where a definition is actually in use. If a definition changes, the people responsible can find every affected asset.
Enrichment: adding meaning at scale
Adding descriptions to thousands of tables by hand is unrealistic. Google’s use-case page describes enrichment, including an example agent that generates AI-powered overviews of data assets at scale. Generated descriptions give people a useful starting point, which human owners then review and correct. Treat generated text as a draft, not a final fact, particularly for sensitive or business-critical assets.
Pick the critical terms
Start with the dozen terms that cause the most arguments, such as customer, revenue, and active user.
Agree on definitions
Bring owners from each department together and record one clear definition, with examples of what is included and excluded.
Link terms to columns
Connect each term to the actual columns and assets that represent it.
Name stewards
Give each term an owner who approves changes and answers questions.
Review on a schedule
Revisit definitions regularly, because the business changes and glossaries must change with it.
A glossary of fifty well-linked, well-owned terms beats a glossary of five thousand terms that nobody reads. Grow it in response to real confusion.
Writing glossary definitions in vague language, such as “a customer is someone who uses our service”. Good definitions are precise enough that two people applying them to the same record reach the same answer.
8Automatic Data Quality
Quality scans act like inspectors who test your tables against written rules, on a schedule, and report what they find.
What data quality means
Good data is accurate, complete, fresh, consistent, and unique. Bad data looks normal until someone makes a decision with it. A missing country code, a negative price, a duplicated order, or a table that stopped updating three days ago can quietly spoil every report built on top. Automatic data quality lets you turn those worries into checks that run without anyone remembering to look.
How a scan works
Google’s documentation explains that an auto data quality scan targets one table and tests it against rules you define. Scans run on a schedule or on demand, produce quality scores, and log alerts when records fail. Supported table types include BigQuery tables, Iceberg REST catalog tables, SAP Business Data Cloud Delta Lake tables, and Apache Hive tables. The scan runs in a Google-managed tenant project, so you do not provision or maintain any machines for it.
A quality scan is a food inspector who visits a factory every night with a checklist. Is every box labeled? Are the weights within range? Are any items expired? The inspector writes a score on a certificate and raises an alarm if the failures pile up.
The kinds of rules
| Rule kind | What it checks | Example |
|---|---|---|
| Non-null | A column has no empty values | Every order has a customer ID |
| Range | Values fall between a minimum and a maximum | Prices are never negative |
| Set | Values belong to an allowed list | Status is one of a few known codes |
| Regular expression | Text matches a pattern | Email addresses look like email addresses |
| Uniqueness | No duplicate values in a column | Order numbers are not repeated |
| Statistic range | An average, minimum, or maximum stays in range | Average basket value stays reasonable |
| Custom SQL | Any condition you can express in SQL | Shipping date is never before order date |
Google describes row-level rules, which test each row against a passing threshold percentage, and aggregate rules, which test one value across the whole table. Custom SQL rules can be a row condition, a table condition, or an assertion that returns the rows that break a rule. Every rule is assigned to a quality dimension, with the predefined set including freshness, volume, completeness, validity, consistency, accuracy, and uniqueness. A scan can hold up to one thousand rules.
flowchart LR
T["BigQuery table"] --> S["Quality scan with rules"]
S --> R["Scores by job, dimension, column"]
R --> BQ["Results exported to BigQuery"]
R --> CAT["Quality scorecard on catalog page"]
R --> AL["Logs and email alerts when score drops"]
BQ --> L["Looker dashboards"]
Controlling cost and scope
Scanning huge tables repeatedly can be expensive, so scans offer controls. You can sample a percentage of records, apply row filters to limit what is scanned, and run incremental jobs that look only at new data using a date or timestamp column as a marker. Running jobs can be canceled. Google also notes that Private Google Access must be enabled on the subnet used by the scan.
Ten rules on your most important table beat five hundred rules nobody reads. Begin with checks tied to real past incidents, such as a feed that once arrived empty.
Running scans and ignoring the results. A score nobody reviews is decoration. Decide who receives each alert, what they must do when it fires, and how quickly.
9Profiling, Scores, and Acting on Results
Profiling shows what your data really looks like, and results turn quality from a feeling into numbers people can act on.
Profiling first, rules second
Before writing rules you need to know what is normal. Data profiling computes statistics for each column of a table, such as the smallest and largest values, the share of empty values, the number of distinct values, and common patterns. Google’s use-case page describes profiling a BigQuery table for column statistics and anomalies, then configuring and running a quality scan. The documentation also lists profiling-based recommendations as one source of rules, so the profile can suggest sensible starting checks.
Profiling is a doctor’s check-up that records your normal pulse, weight, and blood pressure. Quality rules are the alarms set against those normal values. You cannot set a good alarm without first knowing the normal reading.
Reading scores at different levels
Scores show the percentage of rules passed. According to Google’s documentation, they can be viewed at the level of a whole job, of each quality dimension, and of each column, although column-level scores are available through the API. A table might score ninety-eight percent overall yet only seventy percent on freshness, which tells you the contents are fine but the pipeline is late. Looking at dimensions points you to the kind of problem rather than just its existence.
Finding the bad rows
A failing score is only the start. People then need to know which records broke the rule so they can repair the source. Google describes a query that retrieves failed rows, returning all columns of the table. Teams commonly route failed rows to a repair queue or quarantine table, so that downstream users see only trusted records while owners fix the rest.
Profile the table
Look at ranges, null rates, and distinct counts to learn what normal looks like.
Draft rules from the profile
Turn clear patterns into rules, and have the data owner confirm each one makes business sense.
Schedule the scan
Run it after the pipeline that updates the table, not at a random clock time.
Route alerts to owners
Send email or log-based alerts when scores fall below targets or jobs fail.
Fix the cause, then tighten rules
Repair the upstream problem, and add a rule so the same failure is caught earlier next time.
Publishing scores where people look
Google’s documentation says results can be published as catalog metadata, shown on the table’s data quality tab, and exported to BigQuery for dashboards in Looker. Publishing scores on the catalog page lets an analyst judge trustworthiness at the very moment of choosing a table.
Google describes rule templates for reuse. If twenty tables need the same non-empty email check, define it once and apply it everywhere, so a change to the rule updates all of them.
Setting every threshold to one hundred percent. Real data has small, acceptable flaws. Overly strict rules fire constantly, and people learn to ignore them. Pick thresholds that separate real trouble from normal noise.
10Data Lineage
Lineage is the family tree of your data, showing where each table came from and what depends on it.
What lineage records
When a job reads a table, transforms it, and writes another table, lineage records the link. Over time these links form a graph that shows, for any asset, which sources fed it and which reports, models, or tables rely on it. Google describes lineage as automatically tracking data flows, including how sensitive information moves through pipelines.
Lineage is a food label that also traces ingredients back to farms and forward to every store that sold the product. If a farm reports contamination, the company can instantly find every affected package.
How lineage is captured
Google’s documentation says lineage from BigQuery, Managed Airflow (Cloud Composer), Cloud Data Fusion, and Managed Service for Apache Spark (Dataproc) appears in the same interface. For other tools, an open standard called OpenLineage lets pipelines send events describing which datasets a job read and wrote. Dataplex can import these events, and some services, such as Spark and Airflow, have built-in producers that send events automatically. The documentation also mentions client libraries and an OpenLineage transport for tools such as Airflow, Spark, and Flink.
sequenceDiagram
participant J as Pipeline job
participant P as OpenLineage producer
participant L as Lineage API
participant U as Lineage graph in console
J->>P: Job reads table A and writes table B
P->>L: Send run event with inputs and outputs
L->>L: Store process, run, and lineage event
U->>L: Analyst opens table B
L-->>U: Show table A upstream and reports downstream
What lineage is good for
Impact Analysis
Before changing or deleting a column, see every downstream table and report that depends on it. This turns risky guesses into informed decisions and prevents surprise breakages.
Root Cause Analysis
When a dashboard shows a wrong number, trace upstream to find the table or job where the error entered, instead of searching blindly.
Privacy and Compliance
Google’s use-case page mentions tracing the origin of sensitive data leaks from trusted to untrusted locations. Lineage helps show regulators where personal data travels.
Cost Cleanup
Assets with no downstream users may be unused. Lineage helps find them, which can reduce storage costs after a careful review.
Know the limits
The lineage graph shows inputs and outputs for each run, but not every low-level step, such as individual Spark stages. Google documents limits on imported events, including a maximum message size of five megabytes and caps on the number of links per event. The imported API supports the first major version of the OpenLineage standard. Check current limits when you design large pipelines.
Make “open the lineage graph” a standard step in your change checklist. Five minutes of looking can prevent a long night of repairing broken reports.
Assuming lineage is complete. Jobs that copy data through tools that do not report events leave gaps. Treat the graph as strong evidence, not as proof, and keep important dependencies documented.
11Data Products and Domains
A data product packages related assets with an owner and a purpose so other teams can adopt them with confidence.
From loose tables to products
A shared warehouse often contains hundreds of tables, and nobody is sure which are meant for general use. A data product changes that mindset. Instead of exposing raw tables, a team publishes a curated bundle, such as “Customer 360”, that includes the right tables, clear descriptions, an owner, quality expectations, and information about how to request access. Consumers search for the product, not for a half-remembered table name.
Raw tables are loose ingredients in a warehouse. A data product is a boxed meal kit with a recipe card, a freshness date, and the name of the chef to call if something is wrong. Customers choose the kit, not individual ingredients.
Domains give ownership a home
Large companies group data by business area, called domains, such as sales, marketing, supply chain, and finance. Each domain’s team knows its data best and is best placed to publish and maintain products. Google’s documentation lists data products and data domains within the service, and third-party summaries describe data products as generally available in 2026. Because this area is evolving, check the current status and limits in the documentation before building a large program on it.
Clear Purpose
A one-line statement of the question the product answers.
Named Owner
A team accountable for quality, changes, and support.
Quality Promise
Published expectations, backed by quality scans, about freshness and accuracy.
Access Path
A simple way to request and receive permission, with rules the owner controls.
Ask consumers what they need before building. A data product that nobody asked for is just another table with a nicer name.
Declaring everything a data product. If every table is a product, the word means nothing. Publish a small number of well-maintained products and retire those that are no longer supported.
12Security and Governance
Governance means the right people see the right data, with rules that are written down once and applied everywhere.
Layers of control
Dataplex works alongside Google Cloud’s normal security tools rather than replacing them. Access to catalog resources is controlled through Identity and Access Management, and Google’s documentation notes that roles such as catalog admin, editor, and viewer decide who can search and manage metadata. Entry groups carry their own access control, so a sensitive drawer of entries can be restricted even when other groups are open.
IAM Roles
Decide who can view, edit, or administer catalog resources. Give the smallest role that gets the job done.
Entry Group Permissions
Restrict groups of entries separately, so sensitive metadata stays in a locked drawer.
VPC Service Controls
A boundary around services and data. Search results stay within the same organization and perimeter.
Policy Tags
Labels on columns that drive column-level security, so only approved people can read sensitive columns.
Centralized Policies
Rules applied once and enforced across the lakehouse, as described in Google’s governance use case for Iceberg tables.
Audit Logs
Records of who changed metadata or ran scans, supporting investigations and compliance reviews.
Think of a company building. IAM roles are the badges. Entry group permissions are locks on individual rooms. The service perimeter is the fence around the property. Policy tags are the extra locked cabinets inside the rooms holding the most sensitive papers.
Classify first, then protect
You cannot protect sensitive data you have not identified. A common approach is to tag columns that hold personal information, financial details, or health data, and then attach security rules to those tags. When a new table arrives with a matching column, the same protection can follow. Lineage then adds a safety net by showing where tagged data flows, so you can spot copies that escaped the rules.
Separate who manages from who reads
Managing metadata and reading data are different powers. A person who edits glossary terms does not automatically need permission to read the underlying tables. Likewise, an analyst who reads tables should not be able to rewrite catalog policies. Separating duties limits the damage from mistakes and compromised accounts.
Keep governance settings, such as aspect types, glossary structure, and scan definitions, in version control and deploy them through reviewed automation. Changes become visible and reversible instead of clicks nobody remembers.
Granting broad catalog administrator roles to many people “to avoid access tickets”. Administrators can change shared structure that everybody relies on. Keep that role to a small, accountable group.
13Lakes, Zones, and Lakehouse Foundations
Dataplex began with lakes and zones, ideas that still help explain how to organize data stored in files.
The original model
When Dataplex was introduced, it organized data into three layers. A lake was a logical home for related data, such as everything belonging to a business area. A zone divided a lake by readiness, commonly a raw zone for data exactly as it arrived and a curated zone for cleaned, trusted data. An asset was a Cloud Storage bucket or a BigQuery dataset attached to a zone. This structure let teams apply rules, discovery, and security consistently to data spread across many buckets and datasets.
A lake is a neighborhood, zones are districts such as the loading docks and the showroom, and assets are the individual buildings. Rules for the loading docks, where goods arrive messy, differ from rules for the showroom, where everything must be spotless.
Where this fits today
Google’s current catalog documentation focuses on entries, aspects, glossaries, lineage, quality, and data products, and it notes that lakes, zones, assets, and entities cannot be registered as catalog entries. That means the older lake model and the newer catalog model are distinct. If your organization has existing lakes and zones, check the current documentation to see which features still apply to them and how Google recommends transitioning. For new work, designing around the catalog, data products, and standard Google Cloud resources is the safer direction.
The raw, refined, curated idea lives on
Even when the specific lake and zone objects are not used, the layered thinking remains valuable. Land data raw and untouched, so you can always replay it. Refine it by cleaning and standardizing. Curate it into trusted, well-described products. Each layer has different owners, different quality expectations, and different access rules, and the catalog can record those differences through aspects and entry groups.
flowchart LR
R["Raw layer: data exactly as received"] --> F["Refined layer: cleaned and standardized"]
F --> C["Curated layer: trusted products"]
C --> U["Analysts, dashboards, ML models"]
Q["Quality scans"] -.-> F
Q -.-> C
Open table formats and the lakehouse
Modern lakehouse designs store data as files in open table formats such as Apache Iceberg, which add reliable tables on top of storage. Google’s documentation mentions Iceberg REST catalogs as a metadata source and as a supported target for quality scans, and describes a governance use case of creating Iceberg tables with centralized column-level security and visualizing automated lineage. The catalog is the shared source of truth that lets many engines understand the same tables.
Name datasets and buckets so the layer is obvious, and record the layer as an aspect on each entry. Analysts should never wonder whether they are looking at raw or trusted data.
Letting analysts query the raw layer directly because it is convenient. Raw data contains duplicates, errors, and sensitive fields that later layers are meant to remove or protect.
14Operating, Monitoring, and Troubleshooting
A governance platform needs its own care, with alerts, ownership, and a routine for fixing the problems it reports.
What to watch
Dataplex generates operational signals you can observe like any other service. Google’s documentation says quality scans write entries to Cloud Logging that can drive alerts, and that email notifications can be sent when a score falls below a target, a job fails, or a job finishes. It also notes that scan costs can be tracked through billing export using labels, and that scans do not emit time-series metrics to Cloud Monitoring, so plan to use logs and billing data rather than expecting standard metric charts.
| Signal | Where to see it | What to do |
|---|---|---|
| Quality score drop | Scan results, email, logs | Open failed rows, find the upstream cause |
| Failed scan job | Logs and notifications | Check permissions, network settings, and table access |
| Metadata gaps | Catalog coverage reports you build | Assign owners and request descriptions |
| Lineage gaps | Missing upstream or downstream links | Enable event producers on the tools that lack them |
| Metadata changes | Change feed on a Pub/Sub topic | Trigger reviews or sync downstream tools |
| Spend | Billing export labels | Review scan frequency, sampling, and scope |
An inspection service also needs inspecting. If the inspectors stop turning up or their reports pile up unread, the certificates on the products become meaningless. Monitoring the governance platform keeps its stamps trustworthy.
Troubleshooting routine
Is the asset in the catalog?
If not, check the ingestion route, the integration setting, and whether the asset type is supported.
Can the person see it?
Check their role, the entry group permissions, and the service perimeter before blaming the index.
Did the scan run?
Check the service account, table permissions, and network settings such as Private Google Access on the subnet.
Is the lineage missing?
Confirm the tool sends events, the message size is within limits, and the event format is supported.
Is the cost surprising?
Look at scan frequency, full-table versus incremental runs, and whether sampling is enabled.
Ownership is an operational duty
Every glossary term, aspect type, scan, and data product needs a named owner. When an owner leaves, the item should pass to someone else rather than rot. Many governance programs fail not because the tools are weak, but because nobody feels responsible. Regular review meetings, where owners confirm their items are current, keep the system alive.
Track the percentage of important tables that have an owner, a description, and at least one quality scan. A simple rising number motivates teams more than any policy document.
Waiting for a perfect catalog before announcing it. Launch with the important tables, show quick wins, and expand. People adopt tools that already help them, not tools promised for the future.
15Advantages, Trade-offs, and Cost Thinking
Dataplex brings real strengths, but like any platform it has limits and a bill worth understanding.
Advantages
- Automatic technical metadata from many Google Cloud services, with no scripts to maintain.
- One shared metadata layer that connects search, glossary, quality, lineage, and governance.
- Serverless scans, so there are no machines to size, patch, or monitor.
- Reach beyond Google Cloud through connectors and open APIs for third-party and custom sources.
- Open standards such as OpenLineage and Iceberg, which reduce lock-in.
- Useful context for AI applications that need to find and trust data.
Disadvantages / Trade-offs
- Tools do not create governance. People must define owners, terms, and rules.
- Names, features, and launch stages have changed quickly, so documentation and tutorials can disagree.
- Some automated extraction and features may be limited to built-in Google Cloud services.
- Documented limits apply, such as maximum rules per scan and event sizes for lineage.
- Scans and processing add cost, especially on very large tables scanned often.
- Migration from the older catalog service takes planning and has gaps to check.
How the bill is built
Google’s documentation states that storing metadata is billed, while creating and managing catalog resources, search calls, and console search are free. Processing-heavy features, such as quality scans, profiling, discovery, and lineage, consume processing capacity measured in Data Compute Units. A third-party summary of Google’s pricing, published in September 2026, reports different rates for standard processing such as discovery and for premium processing such as quality, profiling, and lineage, a metadata storage rate, and a small monthly free allowance. It also reports a new charge for data insights features starting in late October 2026. Because these figures come from a secondary source and prices change, always confirm current rates on Google’s pricing page and test with your own data volumes.
Using the catalog is like visiting a public library: browsing and asking questions are free. Storing a large private collection on the library’s shelves has a rent, and hiring inspectors to test every book has a labor charge. The more often and more thoroughly you inspect, the higher the labor bill.
Practical ways to keep cost sensible
Scan What Matters
Put quality scans on critical tables first, not on every table in the warehouse.
Sample Large Tables
Scanning a percentage of records often gives the same signal at a fraction of the cost.
Use Incremental Runs
Check only new data using a date or timestamp column, instead of re-reading history every night.
Match Frequency to Change
A table that updates weekly does not need a daily scan. Align scans with the pipeline schedule.
Label and Track Spend
Use billing export labels to see which scans cost the most, then review whether each earns its keep.
Mind Entry Counts
Date-sharded tables and thousands of temporary tables multiply entries. Clean up or restructure them.
Dataplex compared with other approaches
| Approach | Strength | Limitation | Fits when |
|---|---|---|---|
| Dataplex | Native Google Cloud integration, automatic metadata, serverless scans | Evolving features and naming | Your data is mostly on Google Cloud |
| Third-party catalog | Rich workflows and wide connector libraries | Extra product to buy, integrate, and run | Data is spread across many clouds and tools |
| Spreadsheet or wiki documentation | Simple and free | Goes out of date quickly, no automation | Very small teams and few datasets |
| Custom-built metadata store | Exactly fits your needs | High build and maintenance effort | Unusual requirements nothing else meets |
If most of your important data already lives in BigQuery and Cloud Storage, Dataplex is usually the natural starting point. If your data estate is deeply multi-cloud, compare it with dedicated catalog tools and consider combining them through open APIs.
16Design Patterns and Popular Use Cases
A handful of tested patterns cover most governance needs in the real world.
Enterprise Data Discovery Portal
All analysts search one catalog for tables, files, and dashboards. Each result shows description, owner, freshness, sensitivity, and quality score. Time spent hunting for data drops, and duplicate tables become easier to spot and retire.
Quality Gates in Pipelines
A pipeline writes to a staging table, a quality scan checks it, and only passing data is published to the curated layer. Failed data goes to a quarantine area for repair. Consumers see only trusted data.
Regulatory and Privacy Reporting
Columns holding personal data are tagged, column-level security is applied, and lineage shows where tagged data flows. When a regulator or customer asks where personal data goes, the answer comes from the graph instead of weeks of manual research.
Safe Change Management
Before altering a table, engineers open the lineage graph to list every downstream consumer, notify the owners, and schedule the change. Surprise breakages become rare.
Data Mesh by Domain
Each business domain owns, documents, and publishes its own data products, while a small central team provides shared standards, aspect types, and platform tools. Ownership sits close to knowledge.
Context for AI Applications
Chat assistants and agents query the catalog to find the right tables, learn what columns mean, and check quality before generating answers. Google’s use-case page describes discovery and enrichment agents built on the catalog API, with the catalog acting as shared context for data assets.
Lakehouse Governance
Iceberg tables on Cloud Storage are governed with centralized column-level policies, automatic lineage, and quality scans, so a single set of rules protects the data whichever engine reads it.
flowchart TD
P["Pipeline writes to staging table"] --> Q{"Quality scan passes?"}
Q -- "Yes" --> C["Publish to curated layer"]
Q -- "No" --> X["Quarantine failed rows and alert owner"]
C --> CAT["Catalog shows fresh score and lineage"]
X --> FIX["Owner fixes source, rerun pipeline"]
FIX --> P
The diagram shows the pattern’s heart: bad data never reaches consumers, owners are alerted, and the cycle repeats until the problem is fixed. This one pattern prevents a large share of the “who changed my numbers?” incidents.
17Anti-Patterns to Avoid
Most governance programs stumble over a short list of repeated mistakes.
Problem
The company buys or enables a catalog, loads it once, and assumes the governance problem is solved.
Why It’s Harmful
Metadata decays as data changes. Within months descriptions are stale, owners have left, and people return to asking colleagues. The catalog becomes shelfware.
Correct Approach
Automate technical metadata, assign owners for business metadata, and review coverage regularly as a normal operating routine.
Problem
Hundreds of quality rules are created in bulk, with every threshold set to one hundred percent.
Why It’s Harmful
Alerts fire constantly for trivial flaws. Owners stop reading them, and real problems hide in the noise.
Correct Approach
Start with a few rules tied to real business risk, choose realistic thresholds, and add rules after real incidents.
Problem
Every table, including scratch and temporary ones, is scanned fully every day.
Why It’s Harmful
Processing charges grow quickly, with little benefit, because most tables are not critical and most do not change daily.
Correct Approach
Scan critical tables, use sampling and incremental runs, and align schedules with pipeline timing.
Problem
A central team tries to describe and own all data for the whole company by itself.
Why It’s Harmful
The central team lacks business knowledge, becomes a bottleneck, and produces generic descriptions that help nobody.
Correct Approach
Let domain teams own and describe their data, while the central team sets standards, templates, and tools.
Problem
Governance is treated as a way to block access, with long approval queues for every dataset.
Why It’s Harmful
Teams create shadow copies and unofficial extracts to get around delays, which spreads ungoverned data far more widely than the original.
Correct Approach
Make approved access fast and easy through data products, clear owners, and policy tags, while blocking only what must be blocked.
Problem
Teams trust AI-generated descriptions and lineage as complete facts without review.
Why It’s Harmful
Generated text can be wrong or vague, and lineage may miss tools that send no events. Decisions built on gaps can mislead people.
Correct Approach
Treat generated content as drafts for owners to review, and document critical dependencies independently of automatic graphs.
18Best Practices, Migration, and Common Mistakes
A short checklist of habits separates thriving governance programs from abandoned ones.
Habits that pay off
Start with a business problem
Pick a painful, visible problem, such as conflicting revenue numbers, and show it solved before expanding.
Name owners everywhere
Every table, term, scan, and product has an accountable owner with a backup.
Design a small set of aspect types
Define clear, reusable templates for ownership, sensitivity, layer, and domain, and use them consistently.
Automate what machines do well
Use automatic metadata, discovery, profiling, and lineage, and reserve human time for meaning and decisions.
Put checks inside pipelines
Gate publication on quality results rather than hoping someone looks at a dashboard.
Manage configuration as code
Keep glossary structure, scans, and policies under version control with review.
Measure adoption
Track searches, coverage of owners and descriptions, and the share of key tables with quality scans.
Moving from the older Data Catalog
Many organizations still have tags and templates from the older Data Catalog service. Google publishes a transition guide, and notes that aspect types succeed tag templates. A third-party summary from September 2026 reports that the legacy service began shutting down in June 2026, and lists migration gaps, such as private tag templates having no direct counterpart, regional tag templates becoming global aspect types so name collisions matter, and limits in managing certain custom items through infrastructure tooling. Because the transition details can change and sources describe the timeline differently, read Google’s current guide before planning a migration, inventory your existing tags, and test with a small subset first.
Inventory what you have
List existing tag templates, tags, and the teams that depend on them.
Map to the new model
Decide which templates become aspect types, how to avoid name clashes, and which entry groups will hold the results.
Pilot with one domain
Migrate one team’s metadata, check search and permissions, and collect feedback.
Migrate in waves and retire the old
Move remaining teams in planned waves, confirm access rules carried over, then retire old tags and processes.
Mistakes that appear again and again
No named owner
Items without owners decay first, and nobody notices until an audit.
Ignoring permissions in testing
Everything works for the admin who built it, then analysts cannot see the data they need.
Skipping the glossary
Technical metadata alone cannot settle arguments about what a number means.
Scans with no follow-up
Scores drift down while nobody investigates, and trust in the program drops.
Relying on preview features
Preview capabilities can change. Check launch status before depending on them for critical processes.
Forgetting the people side
Training, communication, and visible wins matter as much as configuration.
Before announcing the catalog, confirm key tables have owners and descriptions, search roles work for ordinary analysts, quality alerts reach a real person, and a quick-start guide explains how to find and request data.
19Real-World Scenarios
Walking through realistic situations shows how the ideas fit together.
The Two Revenue Numbers
Finance and marketing present different revenue totals at a leadership meeting. Using the catalog, an analyst finds two similar tables, sees from lineage that one includes refunds and the other does not, and links both to a new glossary term that defines “net revenue”. The definition is agreed, the old table is marked deprecated, and the argument ends.
The Empty Feed on Monday Morning
A partner’s file arrives empty, and the weekly sales dashboard shows zero. A quality scan with a volume rule on the staging table fails, an email alert reaches the data owner at six, and the pipeline holds back publication. The owner requests a resend from the partner, and executives never see a wrong dashboard.
The Privacy Request
A customer asks a company to delete their personal data. The privacy team searches for the sensitivity tag, finds forty tables, and uses lineage to discover three downstream copies in an analytics project. All are cleaned within the deadline, with an audit trail showing how they were found.
The Column Nobody Dared to Delete
An engineer wants to remove an old column. The lineage graph shows seven downstream tables and two dashboards using it. The engineer messages their owners, schedules a coordinated change, and avoids a silent outage.
The New Analyst’s First Week
A new analyst needs customer churn data. Instead of asking ten colleagues, she searches the catalog, finds a data product with a clear description, a named owner, a recent quality score, and a request button. She starts working in hours instead of weeks.
A Ninety-Day Rollout Plan for a New Team
Teams often ask how to turn all these ideas into a calendar. A realistic first quarter has three phases, each with a small, visible result that builds confidence for the next one.
During the first month, focus on visibility. Enable the catalog in the projects that hold your most important warehouse data, and confirm that automatic metadata appears for tables and files. Choose one business domain, such as sales, and ask its owners to review descriptions for the ten tables people use most. Create a short list of glossary terms for the metrics that cause the most arguments. Teach analysts how to search, and collect their first complaints about missing or confusing results. The goal of this phase is simple: people can find the important data and see who owns it.
During the second month, focus on trust. Run profiling on the critical tables to learn what normal looks like, then add a few quality rules for completeness, uniqueness, and valid ranges. Send alerts to a real mailbox or channel that somebody watches, and agree on who responds and how fast. Connect scan results to the catalog so consumers see scores beside the table description. Review lineage for the most important reports and note any gaps where a tool sends no events. The goal of this phase is that people trust the data more, and problems are noticed before executives notice them.
During the third month, focus on scale and protection. Add sensitivity tags and column-level access rules for personal data, document the approval path for requesting access, and publish the first data products for the domain that went first. Measure adoption through search activity, the share of key tables with owners and descriptions, and the number of quality incidents caught early. Present these numbers to leadership, then use the results to bring the next domain on board. The goal of this phase is a repeatable pattern that other teams can copy without starting from scratch.
Resist the urge to onboard every team in the first quarter. A small program that visibly works earns trust and budget. A giant program that touches everything but fixes nothing earns skepticism and is hard to rescue.
Roles That Make The Program Work
Governance succeeds when responsibilities are clear. A data owner is accountable for a dataset, approves access, and answers questions about meaning. A data steward looks after daily quality, descriptions, and glossary terms, acting as the practical caretaker. A platform engineer maintains the technical setup, such as permissions, scans, and ingestion. A consumer, such as an analyst or scientist, uses the data and reports problems back. When a catalog entry names all four roles, nobody has to guess whom to contact, and problems travel to the right person quickly instead of circulating in chat threads.
Each story is the same well-run library helping a different visitor. One needs the right edition of a book, one needs to know a shipment was damaged, one needs every copy of a recalled title removed. A good catalog, inspector, and security desk solve all three.
20Frequently Asked Questions
Quick answers to the questions that come up most often.
No. It does not store your business data. It catalogs, describes, checks, and governs data that lives in services such as BigQuery and Cloud Storage.
Yes. Google’s documentation says Dataplex Universal Catalog was renamed Knowledge Catalog on April 10, 2026, and that API, library, command line, and role names did not change, which is why Dataplex still appears in technical names.
Data is the content itself, such as customer orders. Metadata is information about that content, such as the table name, owner, column meanings, and quality score.
An aspect type is a reusable template of metadata fields. An aspect is that template filled in and attached to an entry. Aspect types succeed the tag templates of the older Data Catalog.
Google’s documentation says search calls and console search are free, and that creating and managing catalog resources is free, while metadata storage is billed. Processing features such as scans consume billable processing capacity.
According to Google’s documentation, supported table types include BigQuery tables, Iceberg REST catalog tables, SAP Business Data Cloud Delta Lake tables, and Apache Hive tables. Each scan targets one table.
Google documents a limit of one thousand rules per scan. Start with a few meaningful rules rather than approaching the limit.
Automatically for several Google Cloud services such as BigQuery, Managed Airflow, Data Fusion, and Dataproc, and through OpenLineage events imported from other tools that produce them.
Yes, through connectors for several third-party systems, managed connectivity pipelines, and custom ingestion through open APIs, though some automated features may be limited to built-in Google Cloud services.
People with a catalog role such as viewer, editor, or admin. Results are limited to the same organization and service perimeter, and they respect permissions on entry groups.
They were central to the original Dataplex. Google’s catalog documentation says lakes, zones, assets, and entities cannot be registered as catalog entries, so check current guidance if you use them. The raw, refined, and curated layering idea remains useful either way.
Pick one important domain, let automatic metadata fill the catalog, add owners and descriptions to the key tables, define a handful of glossary terms, and add quality scans to the two or three most critical tables.
Yes. Google’s use-case page describes agents that search the catalog with natural language and retrieve context about data assets, so AI applications can find and understand the right data.
21Summary and Key Takeaways
Dataplex is the librarian, inspector, and security desk for your data. It builds a searchable catalog on top of data that stays where it is, adds meaning through glossaries and aspects, checks health through profiling and quality scans, records history through lineage, and applies protection through consistent policies. The platform and its names have evolved, but the goal is stable: make data easy to find, safe to trust, and safe to share. Tools help, but results come from habits: named owners, small clear templates, meaningful checks, and routines that keep metadata alive.
Key Takeaways
- Dataplex governs data it does not store — it catalogs and checks data in BigQuery, Cloud Storage, and other systems, and Google now presents the catalog as Knowledge Catalog while technical names stay the same.
- Learn the catalog building blocks — entries, aspect types, aspects, entry types, entry groups, and entry links describe what exists and how things connect.
- Automate technical metadata, own business metadata — machines collect schemas and updates, while people supply meaning through owners, glossary terms, and descriptions.
- Profile first, then write quality rules — scans with sensible thresholds, sampling, and alerts turn data quality into measurable, actionable scores.
- Use lineage for confidence and safety — impact analysis, root cause tracing, and privacy investigations all depend on knowing how data flows.
- Protect data with layered, centralized controls — IAM roles, entry group permissions, service perimeters, and policy tags keep access consistent and auditable.
- Treat governance as a living practice — assign owners, review coverage, watch cost, check launch status of newer features, and plan carefully when migrating from older catalog tools.