Before any details, let us form one clear picture of what Cloud Monitoring is, what it does, and why it matters.

Cloud Monitoring is a managed service on Google Cloud that collects, stores, displays, and analyzes metrics from your cloud resources, your applications, and even systems outside Google Cloud. A metric is simply a measurement taken again and again over time, such as “this server is using forty percent of its processor right now”.

It does four main jobs.

• Collect: Gather measurements from many sources, automatically for Google Cloud services and with a little set-up for your own programs.
• Show: Draw the measurements as charts and group them into dashboards that people can read at a glance.
• Alert: Compare measurements against rules and notify people or systems when something looks wrong.
• Judge: Track whether your service meets the reliability promises you made, using service level objectives.

You do not install or run a monitoring server. Google runs the storage, the query engine, and the alerting system. You decide what to watch and what counts as trouble.

What people use it for

A small vocabulary from the world of operations makes everything else easy to follow.

The three pillars of observability

Cloud Monitoring is mainly about metrics, but it works hand in hand with Cloud Logging for logs and Cloud Trace for traces. Together they are often called Google Cloud’s observability tools.

Understanding how monitoring grew up explains why Cloud Monitoring looks the way it does.

Where Cloud Monitoring sits in Google Cloud

• Cloud Logging stores logs, and can turn log patterns into metrics.
• Cloud Trace follows requests across services to find slow steps.
• Error Reporting groups similar errors from your applications into readable lists.
• Cloud Profiler shows which parts of your code use the most processor and memory.
• Managed Service for Prometheus collects and stores Prometheus-style metrics without you running the storage.
• Pub/Sub, email, chat, and paging tools receive notifications.
• Every other Google Cloud service automatically sends its metrics to Cloud Monitoring.

A handful of ideas explain how almost everything in Cloud Monitoring fits together.

How a measurement is named

Every metric type has a long name that looks like a path. The first part says who owns it, and the later parts say what it measures. Google Cloud’s own metrics begin with the address of the service that produces them. Your own custom metrics live under a special custom prefix. A name such as “compute virtual machine, instance, processor utilization” tells you the service, the kind of resource, and the measurement. You do not need to memorize these. The console lets you search the catalog by plain words.

Automatic and manual

Most Google Cloud services send metrics to Cloud Monitoring on their own, with no setup, as soon as you create them. These are called system metrics or built-in metrics. Anything you add yourself, such as “orders placed per minute”, is a custom metric and needs a little work to produce.

Retention and resolution

Measurements are not kept at full detail forever. Recent data is stored at its original sampling interval so you can zoom into a short spike. Older data is kept for a long time as well, but at a coarser resolution, which is enough for trends and capacity planning. The exact periods depend on the metric type and have changed over the years, so if you must answer questions about last year, check the documented retention for the metrics involved. If you need to keep data for a very long time or analyze it deeply, you can export it to another store such as BigQuery for further analysis.

Groups and labels for organization

As your estate grows, you will want to talk about “all the production web servers” instead of naming each one. Cloud Monitoring lets you use resource labels, user-defined labels, and groups to select sets of resources. Choosing a consistent labeling scheme for environment, team, and application on every resource pays off later, because dashboards and alerts can then filter on those labels and automatically include new resources as they are created.

Here is the bigger picture of how measurements travel from your systems to a chart on a screen or a phone call at night.

flowchart LR
    SRC1["Google Cloud services: built-in metrics"] --> ING["Monitoring ingestion"]
    SRC2["VMs with the Ops Agent"] --> ING
    SRC3["Kubernetes with Prometheus collection"] --> ING
    SRC4["Your code with custom metrics or OpenTelemetry"] --> ING
    SRC5["Cloud Logging: log-based metrics"] --> ING
    ING --> STORE["Time-series storage"]
    STORE --> QRY["Query engine"]
    QRY --> DASH["Dashboards and Metrics Explorer"]
    QRY --> ALERT["Alerting engine"]
    ALERT --> CH["Notification channels"]
    CH --> PEOPLE["People and systems"]
    STORE --> SLO["SLO tracking"]
    

The main components

Push and pull

Measurements can arrive in two ways. In push style, the thing being measured sends numbers to the monitoring system, like a patient’s sensor sending readings to the nurse station. In pull style, the monitoring system visits the thing at regular intervals and reads its numbers, like a nurse walking the ward and checking each bedside chart. Google Cloud’s built-in metrics arrive by Google’s own internal means. Agents and the Prometheus collection can scrape numbers by pulling from your applications and then send them on to Cloud Monitoring.

Let us follow a single measurement on its journey, and watch what happens when it crosses an alert line.

sequenceDiagram
    participant R as Resource or Agent
    participant I as Ingestion
    participant S as Time-series Storage
    participant A as Alerting Engine
    participant N as Notification Channel
    participant P as On-call Person
    R->>I: Send measurement with labels and time
    I->>I: Validate name, labels and type
    I->>S: Store as a point in a time series
    A->>S: Evaluate policy conditions regularly
    S-->>A: Recent values
    A->>A: Condition true for the required duration?
    A->>N: Open incident and send notification
    N->>P: Email, chat message or page
    

The stages in plain words

Delay, in honesty

Monitoring is close to real time but not instant. There is a small delay between something happening and its measurement being visible, and another small delay for an alert to be evaluated. For many metrics the total is on the order of a minute or two, and some metrics are reported less often than others. Do not expect second-by-second precision from every chart, and design alerts with this delay in mind.

Not all numbers behave alike. Knowing the kind of a metric tells you how to read and combine it correctly.

Metric kinds

Cumulative values are usually turned into a rate before you look at them. The total number of requests since last year is not very helpful, but requests per second is. Cloud Monitoring’s tools can do this conversion for you through the aligner setting, explained in the querying chapter.

Value types

Why distributions matter

The average hides pain. Suppose a website answers 99 requests in one tenth of a second and one request in ten seconds. The average looks fine, but one visitor had a terrible time. A distribution lets you ask about percentiles. The ninety-fifth percentile is the response time that ninety-five percent of requests beat. The ninety-ninth percentile shows the slow tail. Experts watch these tail percentiles because the slowest requests are what unhappy customers remember.

Anatomy of a time series

• Identity: metric type, resource type, resource labels, and metric labels together identify one stream.
• Points: each point has a value and a time, or a start and end time for interval kinds.
• Cardinality: the number of different streams that exist. Every new combination of label values creates another stream. This number must be kept in check, as explained in the cost chapter.

Data can enter Cloud Monitoring in several ways. Choosing the right one for each source keeps things simple.

Log-based metrics in more detail

Sometimes the number you need is hidden in logs. For example, your application writes a log line each time a payment fails. A log-based metric counts those lines and turns the count into a time series, so you can chart it and alert on it like any other metric. There are two shapes. A counter simply counts matching entries. A distribution extracts a number from each matching entry, such as the time taken, and builds a spread of values. Google also provides some system log-based metrics automatically.

Choosing the right path

Installing and trusting the Ops Agent

The Ops Agent runs as a small background service on each virtual machine. It needs two things to work: permission to write metrics and logs, which comes from the machine’s service account, and a network path to the Google Cloud interfaces. Teams with many machines usually install it by policy, so every new machine gets the agent without anyone remembering. Google provides ways to roll it out and keep it updated across a fleet. After installation, check that data appears for the new machine within a few minutes, and confirm that the agent itself is monitored, so you hear if it stops.

Sending your own numbers

When your code sends a custom metric, it must choose a name under the custom prefix, decide the kind and the value type, pick a small set of labels, and decide how often to report. A good habit is to report at a steady, modest interval and to name metrics so that a stranger can guess their meaning. A name like “orders placed” is better than “counter7”. Avoid changing the meaning of an existing metric, because history and alerts built on it will quietly become wrong.

Raw numbers are not very useful until you slice and combine them. This chapter shows how queries turn data into answers.

Metrics Explorer is the workbench in the console for exploring any metric. You pick a metric, choose how to filter and group it, and see a chart immediately. It is also the best place to prototype a chart before putting it on a dashboard or turning it into an alert.

The main knobs

flowchart TD
    A["Many raw time series"] --> B["Filter: keep the streams you care about"]
    B --> C["Align: one value per time window using an aligner such as rate or mean"]
    C --> D["Reduce: combine streams using a reducer such as sum, with a group by"]
    D --> E["Result: a few clear lines"]
    E --> F["Show on a chart or use in an alert"]
    

Query languages

The point-and-click builder covers most needs. For advanced work there are query languages. Cloud Monitoring supports PromQL, the popular query language of the Prometheus world, which lets you query both Prometheus metrics and many Google Cloud metrics. Google has also offered its own Monitoring Query Language in the past. Google’s guidance on which language is preferred has been shifting toward PromQL, so check the current documentation before investing in the older one.

Worked idea: error rate

Suppose you want to know what share of web requests are failing. You take the count of requests with an error response, aligned as a rate, summed across all servers. You do the same for all requests. Then you divide the first by the second. The result is a ratio between zero and one that you can show as a percentage. The same idea powers alerts and reliability objectives.

A good dashboard answers “is everything okay, and if not, where do I look?” in a few seconds.

Where dashboards come from

Common building blocks

• Line charts for trends over time.
• Stacked area and bar charts for showing parts of a whole, such as traffic by region.
• Heatmaps for distributions, such as response time buckets over time.
• Scorecards for single big numbers, such as the current error rate with a color that turns red above a threshold.
• Gauges for a value within a range.
• Text panels for notes, links to runbooks, and explanations.
• Log panels and alert lists to show recent logs and open incidents beside the charts.

Designing a dashboard people use

Alerts turn monitoring from something you look at into something that looks after you.

An alerting policy is a saved rule with three parts: a condition that describes what bad looks like, one or more notification channels that say who hears about it, and optional documentation that explains what to do. When the condition is met, Cloud Monitoring opens an incident, and when it clears, the incident closes.

Kinds of conditions

flowchart TD
    M["Metric is evaluated each interval"] --> C{"Condition true?"}
    C -->|"No"| OK["Healthy: nothing happens"]
    C -->|"Yes"| D{"True for the whole duration window?"}
    D -->|"No"| W["Wait and re-check"]
    D -->|"Yes"| I["Open incident"]
    I --> N["Notify channels, with documentation"]
    N --> R{"Condition cleared?"}
    R -->|"No"| K["Incident stays open, may re-notify"]
    R -->|"Yes"| X["Close incident and send recovery message"]
    

Why “for how long” matters

Many harmless blips look like emergencies for a few seconds. A processor that jumps to ninety-five percent for one moment is normal. A processor that stays there for fifteen minutes needs attention. The duration setting makes the condition wait before firing. Longer durations mean fewer false alarms but slower reaction. Choose by asking: “how long can this be bad before a customer is hurt?”

Good alert documentation

Every alert should carry a short note that says what is wrong, why it matters, how to check it, and what to try first, with links to the dashboard and the runbook. The documentation text can include placeholders that fill in details, such as the name of the failing resource. A person woken at night should not have to guess.

Severity and ownership

Not all alerts are equal. Give each alerting policy a severity level, such as critical, error, or warning, and use it to decide the channel. A critical alert means customers are hurt right now and a person must act immediately. A warning means something is trending the wrong way and can be handled during working hours. Alongside severity, every policy should have an owner, a named team that is responsible for it. Alerts without owners become orphans: they fire, nobody feels responsible, and eventually they are ignored.

Testing your alerts

An untested alert is only a hope. You can test a policy by temporarily lowering its threshold, by sending a test notification to the channel, or by deliberately causing a harmless version of the problem in a test environment. Check not only that the notification arrives, but that it contains the right link, the right documentation, and enough information for a half-asleep person to act. Repeat such tests after big changes to the system or to the monitoring setup.

An alert is only useful if it reaches the right person, in the right way, with the right urgency.

Channels

Match urgency to channel. A page that wakes someone should be reserved for problems that need action right now. Warnings that can wait for morning belong in email or chat.

Life of an incident

stateDiagram-v2
    [*] --> Open: condition met
    Open --> Acknowledged: person takes ownership
    Acknowledged --> Investigating: look at dashboards and logs
    Investigating --> Mitigated: user pain stopped
    Mitigated --> Resolved: condition cleared
    Open --> Resolved: condition clears by itself
    Resolved --> Reviewed: follow-up learning
    Reviewed --> [*]
    

Staying sane: noise control

• Snoozes: Temporarily silence a policy during planned work, so the team is not flooded by expected alerts.
• Notification rate limits: Control how often a still-open incident sends reminders.
• Auto-close: Close incidents automatically after a period, so forgotten ones do not stay open forever.
• Grouping: Combine related alerts into one incident, rather than ten separate ones for the same root cause.
• Review regularly: Look at which alerts fire most and which are ignored. Fix or delete the noisy ones.

After the incident

The best teams hold a short review after serious incidents, sometimes called a postmortem. It is blameless, meaning it looks for how the system and process allowed the problem, not for a person to punish. Typical outputs are a timeline, the root cause, and actions such as a new alert, a better dashboard, or a fix to the design. This is how monitoring improves over time.

Internal numbers can look perfect while customers cannot reach your site at all. Uptime checks look at your service the way a customer does: from the outside.

An uptime check is a probe that Cloud Monitoring sends to your service at regular intervals from several places around the world. It tries to connect, and records whether it succeeded and how long it took. You can alert on failures, so you learn about an outage within minutes, even if every internal metric still looks healthy.

What a check can do

flowchart LR
    P1["Probe location A"] --> SVC["Your service"]
    P2["Probe location B"] --> SVC
    P3["Probe location C"] --> SVC
    SVC --> RES["Responses and timings"]
    RES --> MON["Cloud Monitoring: uptime metrics"]
    MON --> POL{"Failing from enough locations?"}
    POL -->|"Yes"| ALERT["Alert and incident"]
    POL -->|"No"| FINE["No action"]
    

Design tips

• Point checks at a lightweight health address that actually touches the important dependencies, such as the database, so it reports real health.
• Do not check only the home page. Include one or two important journeys, such as login or search.
• Alert when several locations fail, not just one, to avoid false alarms from a single network hiccup.
• Allow the probes through firewalls if your service is restricted, and take care not to expose private pages in a public check.
• Combine external checks with internal metrics. External tells you that it is broken, and internal helps tell you why.

What to do with the results

Uptime check results are metrics too, so you can chart them, alert on them, and include them in SLOs. Many teams build an availability indicator from them: the share of probes that succeeded over a month. Because probes come from outside, this number is close to what real visitors experience. Compare it with the internal error rate. If the outside view is much worse than the inside view, something between your servers and your visitors, such as a network setting, a certificate, or a content delivery layer, is probably at fault.

The most mature use of monitoring is not “is the server busy?” but “are we keeping our promise to users?”

The building blocks

flowchart TD
    S["Define the service"] --> I["Choose an SLI: good events divided by all events"]
    I --> O["Set an SLO target and window"]
    O --> B["Error budget = allowed bad events"]
    B --> M["Monitor remaining budget and burn rate"]
    M --> D{"Budget healthy?"}
    D -->|"Yes"| R["Ship features and take reasonable risks"]
    D -->|"No"| F["Slow releases and focus on reliability work"]
    F --> M
    R --> M
    

How Cloud Monitoring helps

Cloud Monitoring lets you define a service, which is a description of a thing you operate (for example a Cloud Run service, a Kubernetes workload, or a custom service), and attach one or more SLOs to it. It can calculate the indicator from built-in metrics for common service types, or from your own metrics. It then shows the current attainment, the remaining error budget, and how fast it is burning. You can create alerts on burn rate.

Why burn-rate alerts are smart

A simple alert on “error rate above one percent” can be too noisy or too slow. A burn-rate alert asks a better question: “at the current pace, will we break our promise?” Teams often use two alerts together. A fast alert catches a sudden big problem, using a short window and a high burn rate, and pages someone at once. A slow alert catches a gentle leak that would drain the budget over days, using a long window and a lower burn rate, and opens a ticket instead of paging.

Choosing good objectives

• Pick indicators users actually feel: availability and speed of important actions.
• Do not aim for one hundred percent. It is nearly impossible and extremely expensive, and it leaves no room to change anything.
• Set the target slightly better than what users will tolerate, not better than the best you have ever achieved.
• Agree on objectives together with product owners, so the budget becomes a shared decision tool.

Monitoring data can reveal a lot about your business. Decide carefully who can see it, change it, and receive it.

Metrics scopes: one view over many projects

Most organizations have many Google Cloud projects, for example one per team or per environment. A metrics scope is the group of projects whose metrics you can see together. One project is chosen as the scoping project that hosts the scope, and other projects are added as monitored projects. From the scoping project you can build dashboards and alerts across all of them.

Limits on how many projects one scope can hold exist, and they can change, so check the documentation when planning big estates.

Who can do what: IAM

IAM, Identity and Access Management, is Google Cloud’s system for deciding who may do what. For monitoring, roles typically fall into groups.

Practical protection

• Least privilege: Give agents and applications only the writer permission, not editor or admin.
• Sensitive data in labels: Never put personal data, passwords, or customer secrets into metric labels or alert text. Metrics are widely visible and long-lived.
• Alert content: Notifications can pass through chat tools and email. Keep details short and link to the console, where access is controlled.
• Notification channels: Treat webhooks and paging tokens as secrets. Anyone who can change a channel can redirect alerts, or silence them.
• Audit logs: Changes to policies, channels, and access are recorded, so you can see who silenced or changed an alert.
• Network and perimeter controls: Organizations with strict rules can include the monitoring service in a protective boundary, so data cannot easily leave.
• Review access on a schedule, and remove people when roles change.

Separating production from everything else

Test and development systems are noisy. They restart often, break on purpose, and produce odd numbers. If their metrics and alerts mix with production in the same scope, charts become cluttered and a test failure can wake an engineer for no reason. A common arrangement keeps production in its own scope with strict access and urgent channels, while development gets a separate scope with relaxed rules and gentle channels. This also makes it easier to give developers wide access to their own environment without exposing production.

Metrics tell you that something is wrong. Logs, traces, and error reports help you find out why. The tools are designed to be used together.

sequenceDiagram
    participant O as On-call Engineer
    participant A as Alert
    participant D as Dashboard
    participant T as Trace
    participant L as Logs
    participant E as Error Reporting
    A->>O: Error rate above threshold
    O->>D: Open dashboard, see when it started
    D-->>O: Spike after a deployment
    O->>T: Look at slow or failed requests
    T-->>O: Database call is slow in one service
    O->>L: Read logs of that service at that time
    L-->>O: Connection errors
    O->>E: Check grouped errors
    E-->>O: New error type since the release
    

How the tools connect

• Alert to dashboard: Alert notifications can link straight to relevant charts, showing the problem in context.
• Chart to logs: From a spike on a chart you can jump to the logs from the same resource and time range.
• Logs to metrics: Log-based metrics turn events into numbers you can chart and alert on.
• Traces to services: Traces show the path of requests and which hop was slow.
• Errors grouped: Error Reporting collects similar errors, counts them, and shows when each first appeared, which often points to a particular release.
• Profiler: For code performance issues, a profiler shows which functions use the most time or memory.

The four golden signals

Site reliability engineers often recommend watching four signals for any user-facing service. They are a good starting point for dashboards and alerts.

Two related shortcuts are often used. RED covers rate, errors, and duration for services. USE covers utilization, saturation, and errors for resources such as machines and disks.

Monitoring is not free at scale. A little understanding stops surprising bills.

How charges usually work

• System metrics from Google Cloud services are generally free to ingest.
• Custom, agent, external, and Prometheus metrics are typically charged based on the volume ingested, measured by data size or by the number of samples, depending on the metric source. There is usually a monthly free allowance.
• Reading data through the programming interface beyond a free allowance may be charged.
• Alerting policies and uptime checks can have their own pricing, such as a charge per condition or per check execution, beyond some free amount.
• Logs in Cloud Logging, which feed log-based metrics, are charged separately by ingestion volume and retention.

Pricing structures, units, and free allowances change over time and differ by product. Always read the current pricing page and use the pricing calculator.

The cardinality trap

The biggest cost and performance risk is high cardinality: having a huge number of different time series. Each unique combination of label values creates a new stream. If you add a label such as “user identifier” or “request identifier” to a metric, you may create millions of streams, each needing storage and processing. Costs balloon, queries slow, and quotas are hit.

Ways to lower the bill

Quotas and limits

Cloud Monitoring has limits, such as the number of alerting policies per project, conditions per policy, notification channels, custom metric descriptors, label counts per metric, and the rate of writes and reads. These change over time. Check the quotas page, set alerts as you approach limits, and request increases early when you expect growth.

No tool is right for everyone. An honest comparison helps you decide when Cloud Monitoring fits.

Key trade-offs

Alternatives to consider

• Prometheus with Grafana, self-run, is the popular open-source choice for containers. You manage storage and scaling yourself, or use Google’s managed Prometheus service to avoid that.
• Datadog, New Relic, Dynatrace, Splunk Observability and similar commercial platforms offer rich analysis and many integrations across clouds, usually at a higher price.
• Amazon CloudWatch and Azure Monitor are the native equivalents on other clouds, useful if you run mainly there.
• The Elastic stack and OpenSearch are strong for logs and search, with some metrics support.
• Zabbix and Nagios are long-standing tools for traditional infrastructure monitoring.

Many organizations mix tools. A common pattern is to use Cloud Monitoring for Google Cloud metrics and alerts, and a dedicated vendor tool for deep application analysis, with OpenTelemetry keeping the instrumentation portable.

Teams who run production systems for years end up repeating the same good ideas, and avoiding the same bad ones.

Patterns that work

Anti-patterns to avoid

Making alerts self-explaining

A well-designed alert message answers five questions without opening anything: what is wrong, how bad it is, since when, where it is happening, and what to do first. Include the name of the service, the region or cluster, the current value and the limit, a link to the dashboard, and a link to the runbook. Use plain words, not codes known only to the author. A message such as “Checkout error share is 6 percent, above the 2 percent limit for 10 minutes, in the Mumbai region; open the checkout dashboard and the rollback guide” saves minutes in a real incident.

A checklist of good habits, a list of beginner mistakes, and a map of what to try when monitoring itself misbehaves.

Best practices

• Start with built-in metrics and uptime checks, then add custom metrics where you have real questions.
• Use the four golden signals for every user-facing service.
• Define one or two SLOs for key user journeys and alert on burn rate.
• Give every alert a clear name, a severity, an owner, and documentation with a runbook link.
• Choose durations that filter blips but still protect users.
• Route by urgency: page, chat, or ticket.
• Use percentiles for latency, not just averages.
• Keep labels low-cardinality and never put personal data in them.
• Install the Ops Agent on virtual machines where you need memory and disk detail.
• Define dashboards and alerts as code and review changes.
• Monitor the monitoring: heartbeats, absence alerts, and notification channel tests.
• Review alerts and dashboards on a schedule, and delete what is unused.
• Keep an eye on ingestion volume and budgets.

Common beginner mistakes

Troubleshooting guide

Each kind of service has a few measurements that matter most. Here is a starter list you can adapt.

A starter recipe for a new service

These illustrative situations show how the ideas combine. They are examples to learn from, not claims about any real company.

An example decision record

Short answers to the questions people ask most.

You began with a car that had no dashboard and ended with charts, alerts, uptime checks, and reliability promises. Here is the journey in a few lines.

Cloud Monitoring is Google Cloud’s managed service for collecting, storing, showing, and acting on measurements. Built-in metrics arrive automatically, the Ops Agent, Prometheus, OpenTelemetry, custom metrics, and log-based metrics fill the gaps. Metrics Explorer and dashboards turn raw streams into understanding. Alerting policies, notification channels, and incident handling make sure the right person hears about trouble. Uptime checks look from the outside, and service level objectives with error budgets measure what really matters, whether users are happy. Metrics scopes give one view across many projects, IAM protects who can see and change things, and careful label design keeps cost under control. Throughout, fewer and better alerts beat many noisy ones.

Because product names, query languages, limits, free allowances, and prices change from time to time, confirm the current details in the official Google Cloud documentation before you finalize your design.