Cloud Monitoring in Google Cloud: Seeing Inside Your Systems Before Your Users Do
A complete, plain-English guide to how Google Cloud collects, stores, visualizes and alerts on the health of your applications, from metrics and scopes to alerting policies, uptime checks and service level objectives.
Imagine you own a busy restaurant. You cannot stand in the kitchen all day, yet you still want to know if an oven is too hot, if orders are piling up, or if the front door is locked when customers arrive. So you place thermometers, order counters and door sensors around the building, and you ask a manager to ring your phone when something looks wrong. Cloud Monitoring is that whole setup for your software. It places “sensors” on your servers, containers, databases and applications, draws their readings on live screens, and calls you when something goes wrong.
1What Cloud Monitoring Is and Why It Exists
Cloud Monitoring is the Google Cloud service that turns raw numbers from your systems into understanding and action.
The simple definition
Cloud Monitoring is a managed service on Google Cloud that collects measurements, called metrics, from your cloud resources, your applications and even systems outside Google Cloud. It stores those measurements as a history, lets you draw charts from them, lets you set rules that raise alarms, and helps you track how reliable your service really is. “Managed” means Google runs the machinery for you. You do not install a database, patch a server or worry about disk space for the monitoring system itself.
Why it exists
Modern applications are built from many small parts spread over many machines. A single web page might touch a load balancer, a container, a database, a cache and a message queue. When the page is slow, no human can guess which part is to blame. Without measurements, teams argue and guess. With measurements, they point at a chart and say “the database is waiting on disk”. Cloud Monitoring exists to replace guessing with evidence.
A car dashboard shows speed, fuel, engine temperature and warning lights. You do not need to open the hood to know something is wrong. Cloud Monitoring is the dashboard for your cloud systems, and the alerting feature is the warning light that turns on before the engine fails.
Where it is used
Site reliability engineers use it to protect uptime. Developers use it to see how a new release behaves. Platform teams use it to track cost-driving resources such as CPU and memory. Managers use dashboards to see whether the business is healthy. Because it is built into Google Cloud, most Google services send their health data to it automatically, without any setup at all.
2Observability, Monitoring and the Wider Google Cloud Family
Monitoring is one member of a team of tools that together make your system observable.
Monitoring versus observability
Monitoring means watching known signals and reacting when they cross a line. Observability is the bigger idea: being able to ask new questions about your system from the outside, even questions you did not plan for. Monitoring answers “is it healthy?” Observability helps answer “why is it unhealthy?” Cloud Monitoring is the part that answers the first question quickly and reliably, and it hands you over to the other tools for the second.
The family of tools
Google groups its observability products under the name Google Cloud Observability. You may still meet the older names Stackdriver and Operations Suite in books and blog posts, and they describe the same family.
Cloud Monitoring
Collects metrics, draws dashboards, runs uptime checks, raises alerts and tracks service level objectives.
Cloud Logging
Stores log entries written by services and applications, and can turn log patterns into metrics.
Cloud Trace
Follows a single request across many services and shows where time was spent.
Cloud Profiler
Shows which functions in running code consume the most CPU and memory.
Error Reporting
Groups application errors and exceptions so that repeated failures appear as one clear item.
Think of a hospital. Monitoring is the heart-rate screen beside the bed, logging is the nurse’s written notes, tracing is the record of which departments a patient visited, and profiling is the detailed scan of one organ. You need the screen to notice trouble, and the other tools to understand it.
These tools are connected. You can jump from a spike on a monitoring chart to the logs from that exact minute, and from an alert to the trace of a slow request. This linking is one of the biggest time savers during an incident.
3Core Concepts: Metrics, Resources, Labels and Time Series
Five small ideas explain almost everything you will ever see in Cloud Monitoring.
Metric
A named measurement, such as CPU utilization or number of requests. It describes what is being measured, not who is being measured.
Monitored resource
The thing being measured, such as a virtual machine, a container, a database or a load balancer. Each resource type has its own set of identifying labels.
Label
A key and value pair that adds detail, such as zone equals asia-south1-a or response code equals 500. Labels let you slice and group data.
Data point
One measurement at one moment, made of a value and a timestamp or a short time interval.
Time series
A list of data points in time order for one exact combination of metric, resource and labels. This is the basic unit that gets stored, charted and alerted on.
Putting the pieces together
Suppose you have ten virtual machines. The metric “CPU utilization” is one idea, but it produces ten separate time series, one for each machine, because each machine is a different monitored resource. If each machine also reports CPU in different states, such as user and system, every state becomes its own series too. This is why the number of time series can grow quickly, a topic we return to when we discuss cost.
A school keeps attendance. The metric is “students present”. The resource is each classroom. The label is the grade. A single time series is the daily attendance of Grade 5, Room 12 across the whole year. Change any one of those three and you are reading a different series.
Metric types and their names
Every metric has a type, which is a unique path-like name. Metrics produced by Google Cloud services usually start with the service name, followed by a path that describes the measurement. Metrics you create yourself begin with a prefix that marks them as custom. This naming rule keeps the thousands of available metrics tidy and easy to search.
When you are unsure what to monitor, open the metric picker in Metrics Explorer and type the name of your service. The list of available metrics is a quick tour of what Google already measures for you.
4Metric Kinds and Value Types
How a number behaves over time decides how you must read it, and Cloud Monitoring labels each metric accordingly.
Three kinds of metrics
The kind of a metric describes how its value relates to the values before it. Getting this wrong leads to charts that look sensible but tell the wrong story.
Gauge
A gauge records a value at one instant, and the value can go up or down freely. Memory in use, queue length and temperature are gauges. You read them directly.
Delta
A delta records how much changed during a short interval, such as the number of requests in the last minute. Each point stands alone and does not depend on earlier points.
Cumulative
A cumulative metric keeps adding up from a fixed starting point, such as the total number of requests since the server started. The value only grows, until the counter restarts. To make it useful you normally convert it to a rate of change.
A gauge is a thermometer: it shows the temperature right now. A delta is a note saying “forty people entered the shop this hour”. A cumulative metric is the turnstile counter at the door that only keeps climbing; to learn how busy it is now, you compare two readings.
Value types
Besides the kind, each metric has a value type. The common ones are true or false values, whole numbers, decimal numbers, text strings and distributions. A distribution is special: instead of one number, it carries a whole histogram, which counts how many measurements fell into each range. Latency is the classic example. One average hides the slow requests, but a distribution lets you ask for the ninety-fifth or ninety-ninth percentile.
Charting the raw value of a cumulative counter produces a line that only climbs and tells you nothing about current health. Always look at the rate, which Cloud Monitoring can calculate for you through alignment, explained later.
5Architecture and Main Components
Behind the friendly console sits a large distributed system built for speed and scale.
The big picture
Cloud Monitoring has an input side, a storage side and an output side. On the input side, many sources push measurements through a single front door, the Cloud Monitoring API. On the storage side, a very large time series database keeps the data and answers queries quickly. On the output side, you get dashboards, charts, the alerting engine, uptime checking, service level objectives and the API for other tools.
flowchart LR
A["Google Cloud services"] --> M["Cloud Monitoring API"]
B["Ops Agent on virtual machines"] --> M
C["Custom and OpenTelemetry metrics"] --> M
D["Managed Service for Prometheus"] --> M
E["Log based metrics from Cloud Logging"] --> M
M --> S["Time series storage"]
S --> Dash["Dashboards and Metrics Explorer"]
S --> Alert["Alerting engine"]
S --> SLO["Service monitoring and SLOs"]
S --> Ext["API and Grafana"]
Alert --> N["Notification channels"]
The storage engine
Google built a time series database called Monarch to hold the measurements of its own huge internal services. Cloud Monitoring and the Prometheus offering on Google Cloud are built on the same family of technology. The practical meaning for you is simple: it is designed to take in enormous numbers of data points per second and still answer a chart query in moments, without you managing shards, replicas or disks.
Main components you will touch
Metrics scope
The boundary that decides which projects’ metrics you can see together on one set of charts and alerts.
Metrics Explorer
The free-form chart builder for asking one-off questions about any metric.
Dashboards
Saved collections of charts for ongoing watching, with ready-made and custom options.
Alerting
Policies, conditions, incidents and notification channels that call people or systems when a rule is met.
Uptime checks
Probes from outside that test whether your endpoint answers correctly and quickly.
Service monitoring
Tools to define services, service level indicators, objectives and error budgets.
Picture a national weather service. Thousands of stations send readings to a central office (the API), a giant archive keeps every reading (the storage), and the public sees forecasts, maps and storm warnings (dashboards and alerts). Each station does not need to know how the archive works.
6Metrics Scopes: Watching Many Projects at Once
A metrics scope is the lens that decides what data you can see in one place.
The problem it solves
Google Cloud organizes resources into projects. Real companies often have dozens or hundreds of projects: one per team, per environment or per application. If monitoring stopped at each project’s edge, an engineer would need to open many screens to understand one product that spans several projects. The metrics scope solves this by letting one project act as a viewing point for many others.
How it works
Every project has a metrics scope. The project that hosts the scope is called the scoping project. The other projects whose data it can read are called monitored projects. The data is not copied. It stays in each monitored project, and the scoping project is simply allowed to query it. Dashboards, alerting policies and uptime checks are created in the scoping project and can use data from all monitored projects. Google documents a maximum number of monitored projects per scope, in the hundreds, so check the current limit when you design a large setup.
flowchart TB
SP["Scoping project: central monitoring"]
P1["Monitored project: web production"]
P2["Monitored project: payments production"]
P3["Monitored project: data platform"]
P1 --> SP
P2 --> SP
P3 --> SP
SP --> V["Shared dashboards and alerts"]
A security control room in a shopping mall shows camera feeds from every shop. The cameras stay inside the shops, but the control room has permission to look through all of them on one wall of screens.
Two common ways to organize
One central scoping project
A dedicated project that contains no workloads and sees everything. This gives one view for the whole company and is popular with central platform and operations teams.
One scoping project per environment or team
For example, a production scope and a separate development scope. This keeps noisy test data away from production alerts and matches access rules more closely.
Putting alerting policies inside a busy workload project and then deleting or restructuring that project. The policies disappear with it. Keep alerting in a stable, dedicated scoping project.
7How Data Gets In: Collection Paths
There are several routes for measurements to reach Cloud Monitoring, and each suits a different situation.
Built-in system metrics
Google Cloud services report their own health automatically, such as virtual machine CPU, load balancer request counts and database connections. No setup is needed and there is no charge for these.
Ops Agent
A small program you install on virtual machines to collect memory, disk and process data, plus application metrics and logs, which the hypervisor cannot see from outside.
Custom metrics
Measurements you invent, such as orders placed or videos processed, sent through the API or through OpenTelemetry from your own code.
Managed Service for Prometheus
A Prometheus-compatible service where Google stores the data, so you keep familiar Prometheus tools and queries without running the storage yourself.
Log-based metrics
Counts and measurements extracted from log entries, such as the number of lines that say a payment failed.
External and hybrid sources
Agents and collectors can send data from on-premises servers and other clouds, giving one view of a mixed environment.
Why an agent is needed on virtual machines
From outside a virtual machine, Google can see how much CPU the machine uses and how many bytes cross its network card. It cannot see how much memory your application holds or how full a particular disk partition is, because those facts live inside the operating system. The Ops Agent runs inside the machine, reads those facts, and sends them out. It combines a logging component and a metrics component built on open source collectors, and it replaces the older separate monitoring and logging agents.
A doctor can look at you and measure your temperature from outside, but to know your blood sugar a small sensor must be placed on your body. The Ops Agent is that sensor inside the machine.
Choosing custom metrics wisely
Custom metrics are powerful because they measure what the business cares about. A shop owner cares less about CPU and more about checkouts per minute. But each custom metric costs money based on how much data is sent, and every extra label value multiplies the number of series. Start with a small set that links to real decisions and add more only when someone asks a question you cannot answer.
Prefer open standards such as OpenTelemetry for custom instrumentation. Your code then stays portable, and you can change where the data goes without rewriting your application.
8Internal Working: Alignment and Aggregation
This is the engine room of every chart and every alert, and understanding it removes most confusion.
Why raw data is not enough
Different sources report at different times and speeds. One machine may send a value every ten seconds, another every minute. If you tried to add them directly, the timestamps would not line up. Also, you rarely want a thousand lines on a chart. You want one clear line that summarizes the group. Cloud Monitoring solves both problems in two steps: alignment, then aggregation.
Alignment, inside each time series
The raw points of one series are cut into equal windows, for example one minute each. A function called an aligner turns all points in a window into one value. Common aligners compute the rate, the mean, the maximum, the sum or a percentile. After this step every series shares the same regular time grid.
Aggregation, across many time series
A function called a reducer combines several aligned series into fewer ones. You choose which labels to group by. For example, group by zone and take the mean, and you get one line per zone instead of one per machine.
Display or evaluation
The final series is drawn on a chart, or it is compared with a threshold by an alerting policy. The same two steps power both uses, so what you see on a chart is what the alert sees.
flowchart LR
R["Raw data points per series"] --> AL["Aligner per series: rate, mean, max or percentile"]
AL --> G["Aligned series on one time grid"]
G --> RE["Reducer across series: sum, mean or count by chosen labels"]
RE --> F["Final series for chart or alert"]
Imagine many cashiers, each writing down the sales they make. Alignment is each cashier totalling their own sales hour by hour. Aggregation is the manager then adding the cashiers of each shop together to get one number per shop.
Choosing the right aligner
For a counter that only rises, use a rate aligner so you see requests per second rather than a running total. For a gauge such as memory, a mean shows the typical level and a maximum shows the peaks. For latency distributions, a percentile aligner shows how bad the slow tail is. The wrong choice can hide a problem: a mean of latency may look calm while the ninety-ninth percentile is terrible.
Averaging percentiles from different groups. The average of two ninety-fifth percentiles is not the ninety-fifth percentile of the combined data. Use distribution metrics and let the system compute percentiles correctly over the merged histogram.
9Data Flow and Lifecycle of a Metric
Follow one measurement from birth to retirement to see every stage it passes through.
Creation
A virtual machine, container, service or piece of code measures something and prepares a data point with a value and a time.
Transport
The point travels to the Cloud Monitoring API, either by Google’s internal paths for built-in metrics or over the network from agents and custom code, which must have permission to write metrics.
Ingestion and storage
The point is checked, attached to the right time series and stored. Created series are identified by their metric type, resource and label values.
Use
Charts query it, the alerting engine evaluates it on a regular schedule, and service level objectives count it toward error budgets.
Ageing and expiry
Recent data is kept at fine detail. Older data is kept at coarser detail and eventually removed after the retention period. Retention rules can change, so read the current documentation for exact durations.
Delay you should expect
Data does not appear instantly. Most metrics become visible within a minute or two, and some take a little longer. This is called ingestion delay. Alerting policies include a short allowance for it so that they do not misread a late arrival as a missing signal. When you build a “no data” alert, keep this in mind.
A newspaper article is written, sent to the press, printed, delivered and read, then archived in the library and eventually stored on microfilm with less detail. A metric follows a similar journey, with its own small delays along the way.
10Metrics Explorer, Dashboards and Query Languages
Charts turn stored numbers into something a human brain can read in a glance.
Metrics Explorer
Metrics Explorer is the place to ask questions. You pick a metric, choose which resource to look at, set filters, choose an aligner and a grouping, and see the chart. It is perfect for investigations because there is nothing to save or set up first. When a chart answers a question you will ask often, you can save it onto a dashboard or turn it into an alerting policy.
Dashboards
A dashboard is a saved board of charts, scorecards, tables, text notes and sometimes log panels. Google provides ready-made dashboards for many services, which light up the moment you start using the service. You can also build custom ones. A good dashboard tells a story from top to bottom, starting with overall health and moving down to detail.
Service overview
Traffic, errors, latency and saturation for one service at the top of a single page.
Infrastructure view
CPU, memory, disk and network across a fleet, grouped by zone or by instance group.
Business view
Orders, sign-ups and revenue-linked custom metrics, readable by non-engineers.
Incident view
A focused board built during an outage, showing only what is relevant to that event.
Ways to ask for data
Cloud Monitoring offers a point-and-click builder for most needs. For more advanced questions it supports query languages. Prometheus Query Language, usually called PromQL, is supported for charts and for alerting, and it is the natural choice if your team already knows Prometheus. Google also built its own Monitoring Query Language for complex joins and ratios. Google has been moving its attention toward PromQL, so check the current guidance before choosing a language for long-lived alerts.
Keep dashboards small and purposeful. A page with sixty charts is a wall of noise. Aim for the handful of charts that answer “are we healthy, and if not, where is the problem?”
Using other tools
Many teams already use Grafana. Cloud Monitoring can act as a data source for it, and through the Prometheus-compatible query interface you can reuse existing Prometheus-style dashboards. This lets you keep favorite visuals without giving up the managed storage.
11Uptime Checks and Synthetic Monitoring
Internal numbers may look fine while real users still cannot reach you. Uptime checks look from the outside.
What an uptime check does
An uptime check is a probe that regularly calls your service the way a user would. It can use web protocols, secure web protocols or a simple connection test. It sends a request from several places around the world and records whether the answer arrived, how fast it came and whether the content was correct. You can require a specific status code or look for a certain piece of text in the response body. Checks can run at intervals of one, five, ten or fifteen minutes.
A shop owner may feel everything is fine inside the store, but a friend standing on the street who tries the door every few minutes can tell whether customers can actually enter. The uptime check is that friend, and there are several friends in different cities.
Public and private checks
Public uptime checks
Probes come from Google’s checker locations on the internet and test endpoints that anyone can reach, such as your public website or API.
Private uptime checks
Probes reach endpoints that live inside your private network, such as an internal load balancer, by connecting through your virtual network. This lets you watch services that are never exposed to the internet.
Synthetic monitors
For more than a single request, Google offers synthetic monitoring, in which you provide a small script that walks through a user journey, such as signing in, adding an item and checking out. If any step breaks, you are told. This catches failures in multi-step flows that a single ping cannot detect.
Connecting checks to alerts
An uptime check produces a metric that records pass or fail from each location. You create an alerting policy that fires when the check fails from more than one location at once. Requiring more than one location prevents a false alarm caused by a small network problem at a single checker.
Checking only the home page. A home page may be served from a cache and stay up while login or payment is broken. Add checks for the pages and endpoints that matter most to the business.
12Alerting Policies in Depth
Alerting is the feature that makes monitoring useful when nobody is looking at the screen.
Parts of an alerting policy
An alerting policy is a rule bundle. It has one or more conditions that describe what “bad” looks like, a list of notification channels that say who to tell, optional documentation that tells the responder what to do, and settings that control how the policy behaves, such as how long to wait and when to close an incident on its own.
Threshold
Fires when a metric stays above or below a value for a set time, for example CPU above ninety percent for ten minutes.
Metric absence
Fires when expected data stops arriving, which can reveal that a service or an agent has died.
Rate of change
Fires when a value grows or shrinks too fast, which can catch a sudden surge before it reaches any fixed limit.
Forecast
Predicts that a value, such as disk usage, will cross a limit within a chosen future period, giving you time to act early.
Query based
Uses a PromQL or Monitoring Query Language expression for ratios and complex logic, such as error percentage.
Log match
Fires directly when a particular log entry appears, useful for rare but serious events such as a security change.
The duration window
Most conditions include a duration. The rule must be true for this whole time before an incident opens. A short duration reacts fast but is noisy because brief spikes trigger it. A long duration is calm but slow. Picking the right balance is part of the craft, and the best answer depends on how quickly the problem hurts users.
A smoke detector that rings the moment a single particle floats by would go off every time you toast bread. One that waits until smoke is steady for a short while is much more useful. The duration window is that waiting time.
Documentation inside alerts
You can attach notes to a policy that appear in every notification: what the alert means, how serious it is, links to dashboards and runbooks, and the first steps to try. A responder woken at three in the morning should not need to hunt for context. Good documentation turns an alarm into a plan.
Give each policy a severity level, such as critical, error or warning, and route them differently. Critical alerts can wake a person. Warnings can simply create a ticket for working hours.
13Notification Channels and the Incident Lifecycle
An alert is only helpful if it reaches the right person or system in the right way.
Where notifications can go
Cloud Monitoring can notify people by email, text message and the Google Cloud mobile app. It can post to team chat tools such as Slack, and it can page on-call staff through incident-management products such as PagerDuty. It can also call a webhook to trigger automation and publish a message to Pub/Sub so that your own code can react, for example by restarting a failed component or opening a ticket in your tracking system.
The life of an incident
When a condition is met, Cloud Monitoring opens an incident. This is a record that something is wrong, with start time, affected resource and the policy that raised it. The incident moves through stages until the problem is gone.
flowchart TD
A["Metric data arrives"] --> B["Condition evaluated"]
B --> C{"Rule true for the whole duration?"}
C -- "No" --> B
C -- "Yes" --> D["Incident opened"]
D --> E["Notifications sent to channels"]
E --> F["Responder acknowledges"]
F --> G["Problem fixed and data returns to normal"]
G --> H["Incident closed automatically or manually"]
Keeping noise under control
Auto-close
Incidents can close by themselves when data shows recovery, or after a set period of silence, so old alerts do not pile up.
Notification rate limits
You can limit how often repeat messages are sent for the same ongoing problem, so the responder is not buried.
Snoozes
During planned maintenance you can pause selected policies for a period, preventing a flood of expected alarms.
Grouping
Related symptoms can be grouped into one incident so the team sees one problem rather than twenty.
Sending every alert to one shared email inbox. People quickly learn to ignore it. Route by severity and ownership, and make sure every notification has one clear owner.
14Service Level Objectives and Error Budgets
Instead of guessing what “reliable enough” means, you can measure it and manage it with numbers.
Three key terms
SLI: service level indicator
A measurement of how well the service is doing, usually a ratio of good events to all events, such as the share of requests answered successfully.
SLO: service level objective
The target for the indicator over a period, such as ninety-nine point nine percent of requests succeed over thirty days.
Error budget
The amount of failure the target still allows. A target of ninety-nine point nine percent leaves a budget of one failure in every thousand requests.
Imagine you promise a delivery company that ninety-five of every hundred parcels will arrive on time. The promise is the SLO. The five late parcels you are allowed are the error budget. If you have used up most of your five late parcels by the middle of the month, you must slow down risky changes until things improve.
How Cloud Monitoring supports this
Service monitoring lets you define a service, such as a Cloud Run service, a service in a service mesh on GKE, an App Engine app, or a custom service. You then choose an indicator. Request-based indicators count good requests out of total requests. Window-based indicators judge each time window as good or bad, for example a minute is good if the latency stayed under a limit. You pick a compliance period, either rolling or calendar based, and the console shows how much budget remains.
Burn-rate alerts
The best way to alert on an SLO is not to wait until the budget is gone, but to watch how fast it is being used. This speed is called the burn rate. A burn rate of one means the budget would be exactly used up at the end of the period. A burn rate of ten means it would be gone in a tenth of the time. A well-known approach from the Site Reliability Engineering community uses several windows together: a fast burn that would consume a couple of percent of the budget in about an hour, which should page someone, and a slower burn that eats a larger share over days, which can simply open a ticket.
flowchart LR
I["SLI: share of good requests"] --> O["SLO: target over the period"]
O --> B["Error budget remaining"]
B --> R["Burn rate calculation"]
R --> F["Fast burn: page the on-call engineer"]
R --> S["Slow burn: create a ticket"]
Choose an SLO a little below what users would actually tolerate, not at the level your system happens to achieve today. Set it too high and you spend money chasing reliability nobody notices. Set it too low and users leave.
15The Four Golden Signals and Monitoring Methods
When you are unsure what to watch, a few trusted frameworks give you a starting point.
Latency
How long requests take, with successful and failed requests looked at separately, since a fast error can hide a slow success.
Traffic
How much demand the system receives, such as requests per second or messages per minute.
Errors
The rate of requests that fail, whether openly through error codes or quietly through wrong content.
Saturation
How full the most limited resource is, such as CPU, memory, disk or connection pool, and how close it is to its ceiling.
At a toll booth, latency is how long each car waits, traffic is how many cars arrive per minute, errors are the cars turned back, and saturation is how long the queue has grown compared to the road length.
Two related methods
The RED method for services
Watch the Rate of requests, the Errors and the Duration for each service. It is simple, user focused and well suited to microservices and APIs.
The USE method for resources
For each resource such as a disk or network card, watch Utilization, Saturation and Errors. It is well suited to machines and infrastructure.
Together, the golden signals and these methods ensure you cover both what users feel and what the machines endure. Start every new service with these few charts and alerts, and add more only when experience shows a gap.
16Monitoring Common Google Cloud Services
Each major service has its own important signals, and Google provides ready-made views for many of them.
Compute Engine virtual machines
Built-in metrics cover CPU, network and disk throughput. Install the Ops Agent to add memory, disk space and process data. Watch CPU saturation, memory pressure and disk fullness, and use metric absence alerts to detect machines that stopped reporting.
Google Kubernetes Engine
System metrics describe clusters, nodes, pods and containers. Managed collection for Prometheus can gather application metrics from pods. Important signals are restarts of containers, pods stuck waiting, node resource pressure and the gap between requested and used resources.
Cloud Run
Metrics include request counts, request latencies, number of container instances and billable time. Watch the response code mix, the ninety-fifth percentile latency and how often new instances must start from cold.
Cloud SQL and databases
Watch CPU, memory, disk use, number of connections and replication lag. Databases fail slowly through filling disks and exhausted connections, so forecast alerts are especially helpful here.
Cloud Load Balancing
Request counts, latency and backend error rates show the customer’s view of the whole application. A good place for both dashboards and service level indicators.
Pub/Sub and messaging
Watch the number of undelivered messages and the age of the oldest unacknowledged message. A growing backlog shows that consumers are falling behind.
Open the service’s own page in the Google Cloud console first. Many services display a built-in monitoring tab with the important charts already arranged, which is a fast way to learn what matters for that service.
17Security and Access Control
Monitoring data can reveal how your systems are built, so it deserves the same care as other sensitive data.
Who can do what
Access is controlled through Identity and Access Management. Predefined roles let you give people exactly the access they need. A viewer role can read metrics and dashboards. An editor role can also change dashboards and alerting policies. An admin role has full control. Narrower roles exist for specific jobs, such as allowing a program only to write metrics, or allowing someone only to manage alerting policies. Follow the principle of least privilege: give the smallest role that gets the job done.
In a hospital, a visitor may look at the waiting-room screen, a nurse may update charts, and only an administrator may change how the whole system is set up. Different keys open different doors.
Protecting what flows in and out
Service accounts
Give agents and applications their own service account with only the permission to write metrics, never broad access.
Service controls
VPC Service Controls can help keep monitoring data inside a defined security boundary, reducing the risk of leaks.
Change history
Audit logs record who created, changed or deleted monitoring settings such as alerting policies, so changes are traceable.
Labels and data hygiene
Never place personal data, passwords or secrets in metric labels or alert documentation, because many people can read them.
Using customer identifiers such as email addresses as label values. This leaks private data and also creates a huge number of time series. Use stable, non-personal categories instead.
You can also watch the watchers. Create log-based alerts for risky actions such as someone deleting an alerting policy or a notification channel, so a quiet attempt to silence your alarms becomes loud itself.
18Cost, Retention and Cardinality
Monitoring is cheap when designed with care and surprisingly expensive when it is not.
What you pay for
Metrics produced by Google Cloud services as part of normal operation are generally free to store and query. Charges apply mainly to metrics that you bring in yourself, such as custom metrics, agent-collected application metrics, external metrics and Prometheus data, usually based on the volume of data ingested. Some usage has a monthly free allowance. Prices and allowances change over time, so always confirm the figures on the official pricing page and use the billing reports to see where your own money is going.
Cardinality: the silent cost driver
Cardinality means the number of distinct time series a metric creates. Each unique mix of label values is a new series. If you add a label with thousands of possible values, such as a user identifier or a request identifier, you multiply the series count and the cost, and you slow down queries too.
A library with ten shelves is easy to search. If every single book got its own private shelf, the library would be enormous, the librarian would be lost, and the rent would be impossible. Labels with unlimited values are like giving every book its own shelf.
Retention
Metric data is kept for a limited time. Recent data is available at full detail for a number of weeks, after which older data is kept at lower detail for much longer, up to about two years in total, according to the documentation at the time of writing. If you must keep data longer for audits or yearly planning, export it to a data warehouse such as BigQuery or Cloud Storage. Check the current limits in the official documentation before relying on them.
Measure first
Use the billing and metrics usage views to find which metrics and which projects produce the most data.
Trim labels
Remove labels with unbounded values and keep only those you really group or filter by.
Reduce sampling where safe
Not every metric needs a reading every few seconds. A slower interval often loses nothing important.
Filter at the source
Drop unneeded metrics inside collectors before they are sent, so you never pay to store them.
19Design Patterns That Work
Teams that run reliable systems tend to follow a few repeated habits.
Alert on symptoms, investigate causes
Page people for things users feel, such as failing requests or slow pages. Use dashboards for causes like high CPU, which may or may not hurt anyone. This keeps alerts rare and meaningful.
Central scope, team ownership
Use a central scoping project for shared visibility, but let each team own and tune its own policies and dashboards through clear naming and labels.
Monitoring as code
Define dashboards, alerting policies and notification channels in infrastructure tools such as Terraform so they are reviewed, versioned and recreated identically in every environment.
Golden dashboard per service
Give every service the same standard first page with the four golden signals. Responders then know where to look no matter which service is failing.
Multi-window burn-rate alerting
Combine a fast window and a slow window so that serious outages page immediately while slow leaks become tickets.
Automated first response
Route selected alerts through Pub/Sub to automation that performs safe first steps, such as restarting a stuck component, while still informing a human.
Anti-patterns to avoid
Problem
Alerts are created for every metric that could possibly be interesting, such as CPU above seventy percent on every machine.
Why It’s Harmful
The team receives constant notifications, most of which need no action. People start ignoring all alerts, and a real outage is missed in the noise.
Correct Approach
Create alerts only for conditions that need a human to act and that matter to users. Move everything else to dashboards and reports.
Problem
Monitoring is set up by hand in the console, with nobody remembering who changed what.
Why It’s Harmful
Environments drift apart, important alerts are lost when a project is rebuilt, and new team members cannot learn from a reviewed history.
Correct Approach
Manage monitoring configuration as code, store it in version control and review changes like any other change.
Problem
Only averages are watched, for example average response time.
Why It’s Harmful
A small group of users may suffer terrible slowness while the average looks fine, so complaints arrive before the alerts do.
Correct Approach
Track high percentiles using distribution metrics and base service level indicators on them.
20Best Practices and Common Mistakes
A short checklist in plain language that you can apply to any project.
Best practices
- Start with the user. Decide what a happy user experiences, then measure exactly that before measuring machines.
- Write down the action. If an alert has no clear action for the responder, it should not page anyone.
- Add runbook links. Put the next steps and dashboard links in the alert documentation.
- Review alerts regularly. After every incident, ask which alerts helped, which were noise, and which were missing.
- Use consistent names and labels. Clear names for policies, dashboards and custom metrics make search and ownership easy.
- Test your alerts. Cause a harmless failure in a test environment and confirm that the right person is told.
- Watch the monitoring itself. Use metric absence and uptime checks to be sure that a silent system is not simply a broken one.
- Plan for growth. Keep an eye on series count and ingestion volume so costs stay predictable.
Common mistakes
Do this
- Alert on user-facing symptoms
- Use percentiles for latency
- Require failures from multiple uptime locations
- Keep configuration in version control
- Give every alert an owner and a runbook
Avoid this
- Paging on every resource spike
- Relying on averages alone
- Trusting one probe location
- Editing alerts by hand with no record
- Sending all alerts to a shared inbox
Forgetting to monitor the agent. If the Ops Agent stops, memory and disk charts go flat or empty, and CPU alerts from the outside may stay quiet. A metric absence alert on a key agent metric protects you from this blind spot.
21Real-World and Industry Examples
Here is how different kinds of organizations might use the same service for very different goals.
Online retail during a festival sale
A shopping site expects ten times its normal traffic. The team watches load balancer request rate, checkout success ratio and database connections on one dashboard. A burn-rate alert on the checkout success indicator pages the engineer if failures spike, and a forecast alert warns when database storage will run out within a day.
A bank running internal APIs
Private uptime checks probe services that are never exposed to the internet. Audit-log alerts tell the security team if anyone changes an alerting policy. Strict roles keep production monitoring read-only for most staff.
A video streaming platform
Custom metrics track videos processed per minute and queue age in Pub/Sub. If the oldest message grows beyond a safe age, an alert triggers automation that adds more workers before viewers notice delays.
A software company on Kubernetes
Engineers use managed Prometheus collection to keep their existing metrics and dashboards while avoiding the work of running storage. Container restart alerts and pod scheduling failures feed a team chat channel for quick triage.
A healthcare technology provider
Synthetic monitors walk through the patient appointment booking journey every few minutes from several regions. A broken step alerts the on-call team long before patients start calling for help.
A start-up with a tiny team
Built-in metrics, a handful of uptime checks and three well-chosen alerts give a two-person team peace of mind without extra tools or servers. They add SLOs only when customers begin asking for reliability guarantees.
22Log-Based Metrics and Log-Based Alerts
Sometimes the clearest signal is not a number the system reports, but a sentence the system writes down.
Why turn logs into metrics
Applications write logs constantly. A payment service may write a line every time a card is declined. An authentication service may write a line every time someone enters the wrong password. These lines are valuable, but reading thousands of them to learn “how often is this happening?” is slow and tiring. A log-based metric solves the problem. You tell Cloud Logging which log entries to look for, and it counts them or measures a value inside them. The result is an ordinary metric that Cloud Monitoring can chart, alert on and include in a service level indicator.
Think of a school librarian who keeps a diary. Each day she writes down every book that was returned damaged. At the end of the term nobody wants to read the whole diary. Instead she puts a tally mark on a chart every time she writes such a line. The tally chart is the log-based metric, and the diary is the log.
Two kinds of log-based metrics
Counter metrics
A counter metric counts how many log entries match a filter. Examples include the number of failed logins, the number of timeouts reported by a worker, or the number of times a specific warning appears. This is the simplest and most common type.
Distribution metrics
A distribution metric extracts a numeric value from each matching entry, such as the time a job took, and records it as a histogram. This lets you chart percentiles of something that was only ever written in text, such as a processing duration in a batch job.
System-defined and user-defined
Google provides some log-based metrics on its own, such as the number of log entries received and the number of bytes ingested, which are useful for watching your logging volume and cost. You create user-defined ones for your own needs. A user-defined metric can also use labels that are pulled out of the log entry, such as the name of a service or the severity, so you can group the counts. The same warning about labels applies here: choose labels with a small, stable set of values.
Log-based alerts
Besides alerting on a metric, Cloud Monitoring can alert directly when a matching log entry appears. This is called a log-based alert, and it is best for rare and serious events where you want to know right away and a count would add little. Good examples are a change to a firewall rule, a change to an access policy, or a message that a critical job failed. Because log-based alerts react to a single event, they are usually limited in how often they notify, so one noisy event does not flood your phone.
Use a log-based alert when one event is enough to matter. Use a log-based metric with a threshold when the question is “how many times, or how fast?” The first reacts to a single occurrence, the second to a pattern.
Expecting a new log-based metric to show history. A metric counts only the entries that arrive after it has been created, so it will look empty for the past. Create the metrics you will need before an incident, not during one.
When to prefer a real metric
If your own code can emit a proper metric, that is usually cleaner and cheaper than turning text into numbers. A log line must be written, shipped, stored and then matched, and the wording of a message can change when a developer edits the code, silently breaking your filter. Log-based metrics shine for software you do not control, for quick wins and for security events that already appear in audit logs.
23Managed Service for Prometheus and OpenTelemetry in Practice
Many teams already speak Prometheus or OpenTelemetry. Google Cloud lets them keep those habits and hand the heavy lifting to a managed backend.
What Prometheus is, in simple words
Prometheus is a popular open source system for collecting metrics, especially in Kubernetes. Applications expose their numbers on a small web page. A Prometheus server visits that page at a regular interval, which is called scraping, and stores the numbers. Teams then write queries in PromQL to draw charts and trigger alerts. Prometheus is loved for being simple and for its huge community, but running it at scale is hard. One server can run out of memory, storing years of data needs extra tools, and combining many clusters needs more extra tools.
Imagine a teacher who walks around the class every few minutes and writes down what each student says. That is scraping. It works well for one classroom. For a whole university you need many teachers, a central office and a big archive, and that is the part Google can run for you.
How the managed offering works
Managed Service for Prometheus keeps the Prometheus way of collecting and querying but stores the data in Google’s global time series storage. You can use managed collection, where Google runs and updates the collectors inside your Kubernetes cluster and you describe what to scrape through simple Kubernetes resources. You can also use self-deployed collection, where you keep running your own collectors and let them send data to Google, or you can use the OpenTelemetry Collector. Queries are written in PromQL, and the same data is available in Cloud Monitoring charts and alerting policies.
Global view
Data from many clusters and regions lands in one place, so one query can span them all without extra federation tools.
Long retention
Metrics are kept for a long period without you managing disks or object storage for the database itself.
Familiar tools
Existing exporters, dashboards and PromQL knowledge continue to work, which lowers the cost of moving.
One alerting system
Prometheus data can feed Cloud Monitoring alerting policies, so teams use one notification and incident system for everything.
OpenTelemetry in the picture
OpenTelemetry is an open standard for producing metrics, logs and traces from your applications. Its biggest promise is portability. If your code uses the standard, you can change the backend later without rewriting the code. In Google Cloud you can send OpenTelemetry metrics to Cloud Monitoring and traces to Cloud Trace through a collector. The Ops Agent itself uses an OpenTelemetry-based collector for its metrics, which shows how central the standard has become.
Choosing between the paths
| Situation | Suggested path | Reason |
|---|---|---|
| Kubernetes team already uses Prometheus exporters and PromQL | Managed collection with Managed Service for Prometheus | Least change and no storage to operate |
| New application code that needs custom business metrics | OpenTelemetry instrumentation | Standard, portable and backend independent |
| Virtual machines running databases or web servers | Ops Agent with application integrations | Ready-made collection for popular software |
| Quick check on a text-only third-party tool | Log-based metric | No code change is possible, so use what it writes |
Scraping everything because it is available. Prometheus exporters can expose thousands of series per pod. Because charges are based on the volume of samples, choose what to collect deliberately and drop what nobody uses.
24Monitoring as Code and Automation
Clicking in a console is fine for learning. For a real company, monitoring must be written down, reviewed and repeatable.
Why code beats clicking
Imagine a team of ten engineers who each add alerts by hand over three years. Nobody knows which alerts exist, which are outdated, or who owns them. When a project is rebuilt, the alerts vanish. When a new region is added, someone must remember to copy everything over. Monitoring as code fixes these problems by describing dashboards, alerting policies, uptime checks and notification channels in text files stored in a version control system, exactly like application code.
A recipe written in a book can be cooked again and again by anyone, and improvements can be tracked page by page. A dish made from memory by one cook is hard to repeat and disappears when the cook leaves. Monitoring as code is the written recipe.
Tools you can use
Terraform
A widely used infrastructure tool with resources for alerting policies, notification channels, uptime checks, dashboards and service level objectives.
Cloud Monitoring API
The programming interface behind the console. Teams use it from scripts and from their own tools to create and update monitoring objects.
Command line tools
The gcloud tool can create and manage many monitoring objects, which suits quick automation and one-time migrations.
JSON definitions
Dashboards can be exported and imported as text definitions, which makes it easy to review and copy them between projects.
A healthy workflow
Draft in the console
Experiment with a chart or alert until it answers the question. This is the fastest way to find the right metric and settings.
Capture it as code
Export or write the final definition into your repository, with a clear name, an owner and a link to its runbook.
Review like any change
A teammate reads the proposed alert, checks the threshold and duration, and confirms that it matches the service’s objectives.
Deploy through a pipeline
An automated pipeline applies the change to test, then to production, so every environment receives identical monitoring.
Detect drift
Run regular comparisons to find manual changes made in the console and bring them back under control.
Automation driven by alerts
Alerts can do more than inform humans. Through Pub/Sub or a webhook, a notification can trigger a small automated workflow. A disk-space warning might trigger a cleanup job. A sustained backlog might scale up workers. A security log alert might open a ticket and add the details to it automatically. The golden rule is to automate only safe and reversible actions, to record everything that the automation does, and to still tell a human so that nobody is surprised.
Use templates for common patterns. If every service gets the same four golden signal charts and the same burn-rate alerts, a single template can create them for a new service in minutes, with consistent names and labels.
25Capacity Planning and Forecasting
Monitoring is not only about the present. The history you collect lets you predict what you will need next month and next year.
From watching to planning
Stored metrics tell a story over time. Memory use that grows slowly every week will eventually fill a machine. A database that doubles its storage every quarter will need a bigger plan. Traffic that jumps every festival season will need extra capacity ahead of time. By reading these trends, you can buy or reserve resources before there is a shortage, and avoid paying for large unused capacity when demand is low.
A farmer who tracks rainfall every year can decide when to plant and how much water to store. A farmer who looks only at today’s weather is surprised by every drought. Historical metrics are your rainfall records.
Tools for the job
Forecast conditions in alerting
A forecast condition uses recent behavior to predict whether a metric will cross a limit within a chosen period. A disk that is only half full today may still trigger a warning if it is predicted to be full in three days. This turns a surprise outage into a calm task.
Longer time ranges on dashboards
Viewing weeks or months of a metric reveals daily and weekly cycles, seasonal peaks and slow growth that short windows hide. Compare this week with the same week last month to see real changes.
Exports for deeper analysis
When you need years of history or want to combine metrics with business data such as sales, export the data to BigQuery. Analysts can then build models, reports and cost reviews with familiar tools.
Signals worth trending
Always plan around peaks and not averages. A system that averages thirty percent CPU may still hit ninety-five percent for ten minutes every evening, and those ten minutes are when users complain. Keep a safety margin for sudden growth, and review the plan after every major launch or marketing event.
Forecasting from a short or unusual window. A forecast built on a quiet weekend may badly underestimate a busy weekday, and one built on an incident week may exaggerate. Use enough history to include normal cycles.
26An Incident Response Workflow Using Cloud Monitoring
When an alert fires, a calm and repeatable routine saves time, stress and money.
The flow from alert to learning
A good response does not begin when the alert arrives. It begins earlier, with clear ownership, helpful documentation and well-built dashboards. When the alert does arrive, the responder follows a path that moves from detection to understanding, then to a fix, then to learning so the same failure is less likely to return.
flowchart LR
A["Alert fires and incident opens"] --> B["On-call engineer acknowledges"]
B --> C["Open linked dashboard and runbook"]
C --> D["Find the failing layer using golden signals"]
D --> E["Jump to logs and traces for the same minutes"]
E --> F["Apply fix or roll back change"]
F --> G["Confirm recovery on charts"]
G --> H["Incident closes and review is written"]
H --> I["Improve alerts, dashboards and runbooks"]
Step by step
Acknowledge quickly
Acknowledging tells teammates that someone is looking, which prevents duplicate effort and further escalation.
Check the scope of impact
Look at the service level indicator and the load balancer view to learn how many users are affected and since when.
Look for recent changes
Most outages follow a change. Compare the start of the problem with deployments, configuration updates and traffic changes.
Narrow down the layer
Use latency, traffic, errors and saturation to decide whether the problem is in the front end, an application service, a database or a dependency.
Prefer a safe, fast mitigation
Rolling back a release or shifting traffic is usually better than a clever repair in the middle of the night. Fix the root cause after users are safe.
Verify and communicate
Confirm that charts have returned to normal and tell stakeholders what happened, in plain language and without blame.
Learn from it
Write a short review that asks what signal came first, what signal was missing, which alert was noisy and what should change.
A fire brigade does not invent a plan when the bell rings. It arrives with a drill, checks if anyone is in danger, finds the source, puts out the flames, confirms nothing is smoldering, and later studies how to prevent the next fire. Good incident response is the same drill for software.
After every incident, ask one more question: “Could we have known sooner?” The answer often leads to a new uptime check, a better threshold or a missing burn-rate alert.
27Hybrid and Multi-Cloud Monitoring
Few companies live entirely in one place. Monitoring must follow the workload wherever it runs.
The reality of mixed environments
A bank may keep its core ledger in its own data center, run its mobile app on Google Cloud and use a software service hosted somewhere else. A single customer action may travel through all three. If each environment has its own separate monitoring tool, an engineer must jump between screens, and no single chart shows the whole journey. The aim of hybrid and multi-cloud monitoring is to bring the essential signals together without trying to replace every local tool.
A traveller flying through three countries may hold three separate boarding passes. A good travel app puts all three in one place so you always know which gate is next. Cloud Monitoring can be that app for your systems.
Ways to include outside systems
Agents on outside machines
The Ops Agent and OpenTelemetry collectors can run on servers outside Google Cloud and send metrics and logs to a Google Cloud project, as long as they are given proper credentials and network access.
Prometheus from any cluster
Kubernetes clusters in other clouds or in your own data center can send Prometheus metrics to the managed service, so one place can answer questions about every cluster.
Uptime checks from outside
Public uptime checks work on any reachable internet address, wherever it is hosted. Private checks can reach services over a private network connection into your own network.
Integration with other tools
Notifications can flow to chat, paging and ticketing tools, and data can be pulled into third-party platforms through the API, so teams that use a different main tool can still share signals.
Things to plan for
Benefits
- One place to see end-to-end health
- Consistent alerting and incident process
- Shared service level objectives across environments
- Less switching between tools during outages
Challenges
- Network paths and credentials for outside agents need careful setup
- Data transfer and ingestion add cost
- Naming and labels must be consistent across environments
- Some provider-specific metrics may remain easier to read in the provider’s own tools
A sensible approach is to bring in the signals that define user experience, such as request rate, errors and latency, and the most important saturation signals, and to leave deep provider-specific detail in the tools that produce it. This keeps cost and complexity under control while giving everyone a shared picture.
28Troubleshooting Cloud Monitoring Itself
When the monitoring does not behave as expected, a few common causes explain most problems.
A chart is empty
Check the time range, the selected project and scope, the filters and the metric name. Also confirm the resource is really sending data and that the time series has not simply been inactive for a long time.
Agent metrics are missing
Confirm the Ops Agent is installed and running, that the machine has network access to Google APIs, and that its service account has permission to write metrics and logs.
An alert never fires
Check the condition filter, the aligner, the threshold and the duration. Compare the policy chart preview with the real chart, and make sure the policy is enabled and has a working notification channel.
An alert fires too often
Lengthen the duration, change the aligner to reduce noise, raise the threshold, or move the signal to a dashboard if it needs no action.
No notification arrives
Verify the channel is confirmed and not paused, check spam filters, test the channel from the console and confirm that rate limits or snoozes are not hiding messages.
The bill is higher than expected
Look at ingestion by metric and by project, find labels with many values, and remove unused custom or Prometheus metrics at the source.
If a smoke alarm stays silent during a fire, you test the battery, the wiring and the sound. Monitoring problems are solved in the same way: check each link in the chain from sensor to message in turn.
A simple order of investigation
Is the data being produced?
Check the source: is the service running, is the agent healthy, is the application emitting the metric?
Is the data arriving?
Look in Metrics Explorer for the raw series and check permissions, network paths and the correct project.
Is the query shaped correctly?
Check filters, grouping, aligner and time window, since a slightly wrong setting can hide real data.
Is the message getting out?
Check the policy state, the incident list, the channel health and any snoozes.
29A Practical Adoption Roadmap
You do not need to build everything at once. A staged plan delivers value early and avoids waste.
Week one: foundations
Choose a scoping project, set up access roles with least privilege, create notification channels for email and team chat, and look at the built-in dashboards of the services you already run.
Weeks two to three: user-facing basics
Add uptime checks for the most important pages and endpoints, install the Ops Agent where needed, and create a small number of symptom-based alerts for errors and latency.
Month two: golden dashboards and runbooks
Give each service a standard dashboard with the four golden signals, attach runbooks to every alert and agree on severities and who owns what.
Month three: objectives and budgets
Define service level indicators and objectives for the most important services, and add multi-window burn-rate alerts to replace many older threshold alerts.
Month four: code and cost control
Move configuration into version control, review ingestion and cardinality, trim waste and set a regular review of alert quality.
Ongoing: learn and refine
After every incident, improve something. Retire alerts that nobody acts on, add the one that would have helped, and keep dashboards focused.
How to know it is working
Good signs include fewer pages at night, shorter time to find the cause of an incident, fewer incidents found first by customers, a steady and understood monitoring bill, and engineers who can open any service’s dashboard and immediately know where to look. If these are improving, your monitoring is doing its job.
30Metrics, Logs and Traces: Choosing the Right Signal
Cloud Monitoring is strongest when you know which kind of signal answers which kind of question.
Beginners often ask which of the three main signals to use. The honest answer is that they work as a team. Metrics tell you that something is wrong and how big it is. Logs tell you what exactly happened at a certain moment. Traces tell you where a single request spent its time. Using them in that order during an incident is a reliable habit.
| Signal | Question it answers | Strength | Weakness |
|---|---|---|---|
| Metrics | Is something wrong, and how much? | Cheap, fast, ideal for trends and alerts | Little detail about individual events |
| Logs | What exactly happened at this moment? | Rich detail and context for each event | Large volume, slower to search and costlier to keep |
| Traces | Where did this request spend its time? | Shows the path across many services | Usually sampled, so not every request is recorded |
| Profiles | Which code uses the most resources? | Pinpoints expensive functions | Needs a running workload and some expertise to read |
When a patient arrives at a hospital, the nurse first reads the vital signs, which are the metrics. If something looks wrong, the doctor reads the detailed notes, which are the logs. To understand a complicated case that touched many departments, the doctor follows the patient’s journey sheet, which is the trace.
Detect with metrics
An alert on error rate or latency tells you quickly that users are affected, and the dashboard shows the size and start time of the problem.
Explain with logs
Open the logs for the same minutes and the same service to read the actual error messages and find the failing component.
Locate with traces
Look at a slow request across services to see which step added the delay, such as a slow database call or a dependency timeout.
Optimize with profiles
If the cost lies inside your own code, the profiler shows which functions deserve attention after the incident is over.
31Glossary of Cloud Monitoring Terms
A quick reference for the words used throughout this guide, listed in alphabetical order.
Alerting policy
A saved rule bundle made of conditions, notification channels, documentation and behavior settings. It decides when a problem exists and who is told about it.
Aligner
A function that turns the many points inside one time window of a single series into one value, such as a rate, a mean, a maximum or a percentile.
Reducer
A function that merges several aligned series into fewer ones, for example by summing or averaging across machines while grouping by a chosen label.
Burn rate
How fast the error budget is being consumed compared with a steady, even rate. A value of one uses the whole budget exactly by the end of the period.
Cardinality
The number of distinct time series a metric produces. It grows with every new label value and strongly affects cost and query speed.
Compliance period
The span of time over which a service level objective is judged, either a rolling window that moves forward or a fixed calendar period.
Cumulative metric
A metric that keeps adding up from a starting point, such as total requests since a server began. It is usually read as a rate.
Custom metric
A metric you define yourself to measure something specific to your business or application, such as orders per minute.
Dashboard
A saved board of charts, tables and notes that gives a quick picture of the health of a service or a set of resources.
Delta metric
A metric that reports how much changed during a short interval, with each point standing on its own.
Distribution
A value type that carries a histogram of many measurements, which allows accurate percentiles such as the ninety-ninth.
Error budget
The amount of failure that a service level objective still permits during its period before the promise is broken.
Gauge metric
A metric that records a value at one instant and can rise or fall freely, such as memory in use or queue length.
Incident
A record opened when an alerting condition is met. It shows the start time, the affected resource and the policy that raised it.
Ingestion delay
The short wait between a measurement being taken and it becoming visible in charts and available to alerting.
Label
A key and value pair attached to a metric or a resource that adds detail and allows filtering and grouping.
Log-based metric
A metric created from log entries by counting matching entries or by extracting a numeric value from them.
Metric type
The unique, path-like name of a metric, which identifies exactly what is being measured.
Metrics scope
The boundary that decides which projects can be viewed together in one place for charts, dashboards and alerts.
Monitored resource
The thing being measured, such as a virtual machine, a container or a database, described by its own set of identifying labels.
Notification channel
A destination for alerts, such as email, text message, chat, a paging service, a webhook or a Pub/Sub topic.
Ops Agent
The program installed on virtual machines to collect operating system, application and log data and send it to Google Cloud.
PromQL
The query language of Prometheus, supported in Cloud Monitoring for charts and alerting conditions.
Scoping project
The project that hosts a metrics scope and from which monitored projects can be viewed together.
SLI
Service level indicator: a measurement of service quality, usually the share of good events out of all events.
SLO
Service level objective: the target value for an indicator over a chosen period, such as ninety-nine point nine percent.
Time series
An ordered list of data points for one exact combination of metric, monitored resource and label values.
Uptime check
A probe from outside your system that regularly tests whether an endpoint responds correctly and quickly.
32Advantages, Disadvantages and Trade-offs
No tool is perfect. Knowing the trade-offs helps you decide where Cloud Monitoring fits and where you may need help from other tools.
Advantages
- Fully managed, with no monitoring servers or storage to run
- Automatic metrics from most Google Cloud services at no extra charge
- Tight links to logging, tracing and error reporting for faster diagnosis
- Built-in support for service level objectives and burn-rate alerts
- Open standards support through Prometheus and OpenTelemetry
- Scales to very large volumes and many projects through metrics scopes
Disadvantages / Trade-offs
- Costs can grow quickly with custom metrics and high-cardinality labels
- Deepest value is on Google Cloud, so mixed environments need extra setup
- Retention at full detail is limited, so long-term analysis needs exports
- Ingestion delay means it is not a tool for sub-second reactions
- Query language choices and feature changes require keeping up with documentation
Compared with other approaches
| Approach | Strength | Cost of ownership | Best fit |
|---|---|---|---|
| Cloud Monitoring | Integrated and managed for Google Cloud | Low operations effort, usage-based cost | Teams running mainly on Google Cloud |
| Self-run Prometheus and Grafana | Full control and wide community | You operate storage, scaling and upgrades | Teams that need deep customization |
| Managed Service for Prometheus | Familiar Prometheus experience with Google-run storage | Low operations effort, cost per sample | Kubernetes teams with existing Prometheus habits |
| Third-party observability suites | Single view across many clouds and tools | License cost and data transfer | Large multi-cloud organizations |
Many companies combine approaches. They may keep Cloud Monitoring for infrastructure and service level objectives while sending selected data to a third-party suite for company-wide views. The best choice depends on team skills, existing tools and where your workloads run.
33Frequently Asked Questions
Not for basic use. Most Google Cloud services send their health metrics automatically, and the dashboards appear as soon as you use the service. You only install the Ops Agent when you want deeper machine-level data such as memory and disk space from virtual machines.
Monitoring deals with numbers over time, such as CPU or request rate, and is best for spotting trends and raising alarms. Logging stores detailed text records of events and is best for finding the exact reason something happened. They are designed to work together, and you can turn log patterns into metrics.
Every project has a metrics scope automatically. You only need to plan it carefully when you want to view several projects together, in which case you pick one scoping project and add the others as monitored projects.
The metric is probably cumulative, which means it adds up from a starting point. Change the aligner to a rate so the chart shows how fast the counter is growing instead of its running total.
Alert only on conditions that need a person to act and that affect users, use a sensible duration window, route by severity, enable notification limits and review your policies after each incident. Move informational signals to dashboards.
Yes. Agents, OpenTelemetry collectors and the API can send data from on-premises servers and other clouds. Uptime checks can also probe any reachable public endpoint, and private checks can reach endpoints inside your own network.
The indicator is the measurement, such as the share of successful requests. The objective is the target for that measurement over a period. The error budget is the amount of failure the target still allows before you break your promise.
Metrics generated by Google Cloud services are generally free to use, and some usage has a free monthly allowance. Charges mainly apply to data you bring in yourself, such as custom, agent-collected, external and Prometheus metrics. Always confirm current prices on the official pricing page.
Data is kept for a limited time, with fine detail for recent weeks and coarser detail for older data, for up to roughly two years according to current documentation. Export the data to a warehouse if you need to keep it longer or to analyze it in other tools.
Use the point-and-click builder for common needs, because it is quick and easy to share. Choose PromQL if your team already knows Prometheus or if you need logic that the builder cannot express, such as ratios between two metrics.
By default a threshold condition simply has nothing to judge. If silence itself is a problem, add a metric absence condition so you are told when the data stops, which often reveals a failed service or agent.
Yes. A notification channel can call a webhook or publish to Pub/Sub, and your own automation can react by restarting a component, scaling a service or opening a ticket. Start with safe, reversible actions and always keep a human informed.
34Summary and Key Takeaways
Cloud Monitoring gives you eyes and ears across your Google Cloud environment and beyond. It gathers measurements from many sources, stores them as time series, lets you shape them with alignment and aggregation, shows them on dashboards, watches them with alerting policies and uptime checks, and turns reliability into a measurable promise through service level objectives. The tool itself is simple to start with and deep enough to grow with you. The craft lies in choosing what to measure, keeping alerts meaningful and keeping costs under control through careful labels and sampling.
If you remember one thing, remember the restaurant from the beginning: sensors are cheap, but the value comes from placing them where they matter and calling the right person at the right time.
A short self-check for your own environment
Before you close this guide, take a few minutes to test your own setup against the ideas you have just read. Ask whether you can name the single most important user journey in your product, and whether an uptime check or synthetic monitor walks through it from outside. Ask whether the four golden signals for your busiest service sit together on one dashboard that a new teammate could understand in a minute. Ask whether every alert that can wake a person has a clear owner, a stated severity and a link to a runbook, and whether anybody has acted on each of those alerts in the last quarter.
Then look at the less glamorous side. Check that the people and programs with access to monitoring hold the smallest roles that allow them to work. Check which custom or Prometheus metrics create the most series and whether anyone still reads them. Check that your configuration lives in version control, so that rebuilding a project does not erase your safety net. Finally, check that an alert on silence exists for the most important agents and services, because a quiet screen is only good news when you know the sensors are alive.
Pick one service and apply the whole guide to it end to end: a golden dashboard, two symptom-based alerts, one uptime check, one service level objective with burn-rate alerts, and a short runbook. When that single service feels calm and well understood, repeat the pattern for the next one.
Key Takeaways
- Everything is a time series — a metric, a monitored resource and labels together define one series, which is the unit stored, charted and alerted on.
- Alignment then aggregation — first summarize each series over time, then combine series across labels; the same pipeline powers charts and alerts.
- Use metrics scopes — one scoping project can watch many monitored projects, giving a single view without copying data.
- Alert on symptoms — page people for what users feel, use dashboards for causes, and give every alert an owner and a runbook.
- Measure reliability with SLOs — indicators, objectives, error budgets and multi-window burn-rate alerts replace guesswork with clear decisions.
- Watch cardinality and cost — unbounded label values multiply series and spending, so keep labels few, stable and non-personal.
- Treat monitoring as code — define dashboards, policies and channels in version control, secure them with least-privilege roles and audit changes.