Cloud Logging is a fully managed service that ingests, stores, searches, analyzes and routes log data from Google Cloud, other clouds and on-premises systems.

The problem it solves

A log is a time-stamped message that a program or system writes to describe something that happened. A user signed in. A payment failed. A disk filled up. A request took four seconds instead of forty milliseconds. In the old days, every server kept its own log files on its own disk. To investigate a problem, an engineer connected to each machine one by one and searched through text files. That approach collapses in the cloud, where servers appear and disappear within minutes, containers live for seconds, and a single user request may touch ten services on ten different machines.

Cloud Logging removes the need to visit machines at all. Logs flow automatically to a central place, they are kept even after the machine that created them is deleted, and they can be searched across the whole organization in seconds. It is serverless, which means you do not install, size or patch any logging servers. You simply send logs and Google handles the storage, indexing and scaling.

What it is used for

Teams use Cloud Logging for troubleshooting failures, investigating security incidents, proving compliance to auditors, measuring application behavior, triggering alerts, and feeding data into analytics systems. It is deeply connected to the rest of Google Cloud Observability, which also includes Cloud Monitoring for metrics, Cloud Trace for request timing and Error Reporting for grouping crashes.

Before going deep, learn the small vocabulary that every later chapter depends on.

Every record in Cloud Logging follows one standard shape called the LogEntry, and understanding its fields is the key to searching well.

The fields that matter most

A LogEntry is a structured object, not just a line of text. The log name tells you which log it belongs to. The resource field describes the source. The timestamp says when the event happened, while a separate receive timestamp says when Cloud Logging got it. These two can differ if a machine was offline and sent its backlog later. The insert identifier is a unique tag that lets Cloud Logging remove duplicates when the same entry is sent twice.

The payload is where the actual content lives, and it comes in three flavors. A text payload is a plain string. A JSON payload is a structured set of named fields, which is by far the most useful because each field becomes searchable. A protocol payload is used mostly by Google services, and audit logs use it to carry rich structured information about who did what.

Not all logs come from the same place, and knowing the families helps you decide what to keep, what to protect and what to drop.

Cloud Logging is made of a small number of clearly separated pieces, each with one job.

The big picture

On the left are the sources: your applications, Google services and agents. They all talk to the Logging API, which is the single front door. Behind that door sits the Log Router, which acts as the traffic controller. The router sends copies of entries to log buckets for storage and to sinks that forward data elsewhere. On the right are the tools people use: the Logs Explorer for searching, Log Analytics for SQL, log-based metrics for numbers and alerts for notifications.

flowchart LR
  A["Applications and Services"] --> API["Logging API"]
  B["Google Cloud Services"] --> API
  C["Ops Agent on VMs"] --> API
  API --> R["Log Router"]
  R --> F["Exclusion Filters"]
  F --> BK["Log Buckets"]
  R --> S["Sinks"]
  S --> BQ["BigQuery"]
  S --> GCS["Cloud Storage"]
  S --> PS["Pub/Sub"]
  BK --> LE["Logs Explorer"]
  BK --> LA["Log Analytics"]
  BK --> LM["Log-based Metrics"]
  LM --> AL["Alerting"]
        

Each component in one line

The Logging API accepts entries and checks permissions. The Log Router evaluates routing rules. Buckets hold the data. Sinks copy data to other places. The Logs Explorer is the search screen. Log Analytics lets you ask questions in SQL. Log-based metrics turn logs into numbers. Alerting watches those numbers or the logs themselves and notifies people.

Follow a single log line from the moment a program writes it until it becomes searchable.

sequenceDiagram
  participant App as Application
  participant API as Logging API
  participant Router as Log Router
  participant Bucket as Log Bucket
  participant Sink as Sink Destination
  App->>API: Write log entry
  API->>API: Check permission
  API->>Router: Pass entry
  Router->>Router: Apply exclusions and match sinks
  Router->>Bucket: Store copy
  Router->>Sink: Forward copy
  Bucket-->>App: Entry searchable in seconds
        

The router is where you gain control over cost, compliance and where your data lives.

Two built-in sinks

Every project starts with two sinks already created. The required sink sends audit logs that Google guarantees to keep into a special bucket you cannot change or delete. The default sink sends almost everything else into a general-purpose bucket. You may edit or disable the default sink, but not the required one.

What a sink is made of

A sink has a destination, an inclusion filter and optional exclusion filters. The inclusion filter is a query that describes which entries to copy. If you leave it empty, every entry matches. Exclusion filters then remove entries from that match. Each sink works independently, so the same entry can be copied to several destinations at once without any conflict.

Aggregated sinks

A normal sink only sees logs from its own project. An aggregated sink is created at a folder or organization and collects logs from every project beneath it, including projects created in the future. This is how large companies guarantee that all audit logs land in one protected place, regardless of what individual teams do.

The writer identity

Each sink gets its own service identity, called the writer identity. The destination must grant this identity permission to write. A very common cause of “my sink is not working” is simply that nobody granted that permission on the destination.

flowchart TD
  ORG["Organization"] --> FOLD["Folder"]
  FOLD --> P1["Project A"]
  FOLD --> P2["Project B"]
  ORG -. "Aggregated sink" .-> SEC["Central Security Project Bucket"]
  FOLD -. "Aggregated sink" .-> ARC["Archive in Cloud Storage"]
  P1 -. "Project sink" .-> BQ["BigQuery Dataset"]
  P2 -. "Project sink" .-> PS["Pub/Sub Topic"]
        

Buckets decide how long logs live, where they physically sit and who can see them.

The built-in buckets

The required bucket holds Admin Activity, System Event and similar audit logs, and keeps them for 400 days. You cannot change that period or switch it off. The default bucket holds most other logs and keeps them for 30 days unless you change it. Retention on custom buckets can be set anywhere from one day up to about ten years, and the numbers can change over time, so always confirm current limits in the official documentation.

User-defined buckets

You can create your own buckets for special needs. A bucket for security logs might keep data for several years and allow only the security team to read it. A bucket for noisy debug logs might keep data for only three days to save money. Because routing is done by sinks, you simply create a sink that sends the right logs to the right bucket.

Location matters

Each bucket lives in a chosen region, or in a global setting where Google picks the placement. Data residency rules in some countries or industries require logs to stay inside a specific geography, so choosing the bucket location early is important. Location cannot be changed after the bucket is created, so decide before you start storing data.

Locking a bucket

A bucket can be locked. Once locked, its retention period can never be shortened and the bucket cannot be deleted until every entry has aged out. This is the tool for regulations that demand tamper-proof, write-once retention. It is permanent, so use it only when you are sure.

Audit logs answer the most important security question: who did what, where and when.

Cloud Audit Logs are a special family of logs written by Google Cloud services. They do not describe application behavior. They describe actions taken against Google Cloud resources, and each entry includes the identity of the caller, the method they invoked, the resource affected, the network address they came from and whether the call succeeded.

Why Data Access logs need a decision

Data Access logs can be enormous because every read creates a record. They are also essential for investigating data leaks. The sensible approach is to enable them for sensitive projects and services, such as those holding customer or financial data, and to leave them off for low-risk, high-volume ones. You can also exempt specific trusted identities, like a known monitoring account, to reduce noise.

Different compute platforms collect logs in different ways, and knowing which applies saves a lot of confusion.

Virtual machines and the Ops Agent

Compute Engine virtual machines do not send application logs by themselves. You install the Ops Agent, a single agent that collects both logs and metrics. It reads system logs by default and can be configured to read application log files, such as a web server or database log. It handles buffering, so if the network drops briefly, entries are held and sent later. An older, separate logging agent still exists in some environments, but Google recommends the Ops Agent for new work.

Google Kubernetes Engine

On GKE, logs written by containers to standard output and standard error are collected automatically by a node-level component. System logs, workload logs and control plane logs can each be switched on or off at the cluster level. Each entry is tagged with the cluster, namespace, pod and container, which makes it easy to pin down a misbehaving workload.

Cloud Run, Cloud Functions and App Engine

Serverless platforms collect anything printed to standard output or standard error with no setup at all. If the printed line is valid JSON, it is understood as a structured entry, and special field names let you set the severity, attach a trace and add labels. Request logs are written automatically by the platform too.

Other sources

Client libraries exist for popular languages and can write directly to the Logging API. This is useful for programs running outside Google Cloud, including other clouds, laptops and on-premises servers.

Collecting logs is useless unless you can find the right five lines among billions.

The Logs Explorer

The Logs Explorer is the main screen for searching. It shows a histogram of log volume over time, a query box, a list of matching entries and a panel for filtering by resource, log name and severity. You can drill into any entry to see every field, and you can jump from an entry to its related trace or to other entries from the same request.

The Logging query language

The query language is built from simple comparisons joined by AND, OR and NOT. You compare fields such as severity, resource type, log name or any field inside a structured payload to a value. You can test for equality, greater-than, containment and pattern matches. You can combine these to say things like “all entries from this service, at error level or above, in the last hour, whose message mentions a timeout”.

Saving and sharing

Useful queries can be saved and shared with teammates so that common investigations, such as “failed logins in the last day”, become one click. Queries can also be turned into log-based metrics or alert policies directly from the screen.

Log scopes and views

Searches normally cover one project, but a scope can combine several projects or buckets into a single search. Log views, covered later, restrict which entries a person can see inside a bucket.

Log Analytics adds the power of SQL on top of the same log data you already store.

The Logs Explorer is excellent for finding individual entries. It is less suited to questions like “which ten API methods produced the most errors this week, grouped by customer tier?” For that you need aggregation, and that is what Log Analytics provides. When you upgrade a log bucket to use Log Analytics, you can run SQL-style queries directly on its contents, and charts can be built from the results.

Linked datasets

A bucket with Log Analytics can also be linked to BigQuery. The link creates a read-only dataset that points at the same data, so BigQuery users can query logs and join them with other business data without copying anything and without paying to store a second copy. This is different from a sink to BigQuery, which writes a real copy of the entries into tables.

Logs tell stories; metrics tell trends. Log-based metrics turn the first into the second.

Two kinds of log-based metrics

A counter metric counts how many entries match a filter, for example the number of payment failures per minute. A distribution metric extracts a numeric value from each matching entry, such as response time, and records how those values are spread, so you can chart averages and high percentiles. Both can be broken down by labels you extract from the logs, such as the service name or error type.

System and user-defined

Google provides some system metrics automatically, such as the number of entries received and bytes ingested. You create user-defined metrics for application-specific signals. Once created, a metric appears in Cloud Monitoring like any other metric, ready for dashboards and alerts.

Alerting from logs

There are two ways to be notified. A metric-based alert watches a log-based metric and fires when it crosses a threshold, such as more than fifty errors in five minutes. A log-based alert fires directly when a matching entry appears, which suits rare, important events like a project owner being added or a firewall rule being deleted. Alerts can notify by email, chat tools, paging systems or webhooks.

flowchart LR
  L["Incoming Logs"] --> M["Log-based Metric"]
  M --> D["Monitoring Dashboard"]
  M --> A1["Metric Alert Policy"]
  L --> A2["Log-based Alert Policy"]
  A1 --> N["Notification Channels"]
  A2 --> N
  N --> P["On-call Engineer"]
        

Logs often contain the most sensitive information in your company, so protecting them is as important as collecting them.

Identity and access roles

Access is controlled through Identity and Access Management. A few roles cover most needs. A log writer can send entries but not read them, which is the right role for applications. A viewer can read most logs. A private logs viewer can additionally read the sensitive Data Access audit logs. A configuration writer can create sinks, buckets and metrics. The logging admin can do everything. Follow the principle of least privilege and grant only the smallest role that does the job.

Log views for fine-grained control

A log view is a filtered window onto a bucket. By granting someone access to a view instead of the whole bucket, you can let the payments team see only payment logs while the rest of the bucket stays hidden from them. This is how a shared central bucket can safely serve many teams.

Encryption

All logs are encrypted at rest by default with Google-managed keys. For stricter control, buckets can use customer-managed encryption keys, so your organization controls when the key is disabled or destroyed. Organizations can also set default encryption for all new buckets within a folder or organization.

Protecting sensitive content

Applications sometimes log things they should not, such as passwords, tokens, card numbers or personal data. Once written, such data is stored and searchable. The best protection is prevention: never log secrets, and scrub or mask sensitive fields in the application. Google also offers sensitive data discovery tools that can scan logs for patterns and help you find leaks.

Network perimeter and integrity

VPC Service Controls can place Cloud Logging inside a security perimeter, limiting data exfiltration. Bucket locking and aggregated sinks to a restricted project protect audit evidence from tampering by project owners.

A log passes through distinct life stages, and every stage offers a decision about cost, speed and retention.

graph TD
  G["Generate"] --> C["Collect"]
  C --> R["Route"]
  R --> H["Hot storage in buckets"]
  R --> X["Export to BigQuery, Storage, Pub/Sub"]
  H --> U["Search, metrics, alerts"]
  H --> E["Expire after retention"]
  X --> AR["Long-term archive or analytics"]
        

Experienced teams repeat a handful of proven layouts, and each solves a specific organizational problem.

flowchart LR
  PRJ["Many Projects"] --> AGG["Aggregated Sink"]
  AGG --> SB["Security Bucket, long retention, locked"]
  AGG --> ARC["Cloud Storage archive"]
  PRJ --> LOC["Local Default Bucket, 30 days"]
  SB --> SEC["Security Team Only"]
  LOC --> DEV["Developers"]
        

Anti-patterns to avoid

Logging cost is driven mainly by how much data you ingest, so cost control is really volume control.

What you pay for

At the time of writing, pricing has three main parts. First is ingestion, charged per gibibyte of log data stored into buckets, with a monthly free allowance for each project. Second is storage beyond the free retention period, charged per gibibyte per month. Third is query cost for certain analytics features and any downstream cost in services such as BigQuery, Cloud Storage and Pub/Sub that receive exported data. Some logs, like Admin Activity audit logs, are free. Exact prices and free allowances change, so always check the current official pricing page before budgeting.

Where the volume comes from

The biggest bills usually come from three sources: verbose debug logging left on in production, high-volume network logs such as VPC Flow Logs and load balancer logs captured at full rate, and Data Access audit logs on busy data services.

A short checklist of habits that separate calm production teams from stressed ones.

Best practices

Common mistakes

Concrete stories show how the pieces combine when real teams face real pressure.

No tool is perfect, and a mature team knows both sides before committing.

Choosing between destinations

Quick answers to the questions beginners ask most often.

For serverless platforms and most Google Cloud services, no. For virtual machines you install the Ops Agent to collect application and system logs. Outside Google Cloud, you use an agent or a client library.

It depends on the bucket. Audit logs in the required bucket are kept for 400 days, while the default bucket keeps entries for 30 days unless you change it. Custom buckets use the retention you set.

You can set very long retention on buckets or export logs to Cloud Storage or BigQuery, where you control retention yourself. Consider the storage cost and any privacy obligations before keeping data indefinitely.

A bucket stores log entries inside Cloud Logging. A sink is a routing rule that decides which entries go to which destination, and a bucket is just one possible destination.

Data Access audit logs are disabled by default for most services. You must enable them for the services you care about, and entries only exist from the moment you turn them on.

The most common causes are a missing permission for the sink’s writer identity on the destination, a filter that matches nothing, or an exclusion filter removing everything. Check the sink’s details for error messages.

Logging deals with individual event records, while Monitoring deals with numeric measurements over time. They work together, and log-based metrics are the bridge that turns log events into measurements.

Yes. Aggregated sinks collect logs into a central bucket, and log scopes let a person search across several projects or buckets, provided they hold permission on each.

The required bucket always keeps Admin Activity and System Event audit logs, and you cannot exclude them from it. You can, however, exclude entries from the default bucket and from your own sinks.

Reduce volume at the source, exclude or sample noisy low-value entries, tune network log sampling, shorten retention on noisy buckets, export long-term data to cheaper storage, and alert on sudden ingestion spikes.

Yes, always at rest and in transit. You can choose customer-managed keys for extra control over buckets.

Prefer structured logs. They make filtering exact, enable metrics from named fields and allow SQL analysis, while plain text relies on fragile word searching.

Cloud Logging is the memory of your Google Cloud environment. It gathers logs from platforms, applications, agents and security systems through one API, sends each entry through the Log Router, and stores copies in buckets and sinks according to rules you control. On top of that data you search with the Logs Explorer, analyze with SQL in Log Analytics, measure with log-based metrics and react with alerts. Surrounding all of it are governance choices: who can read what, how long data lives, where it physically sits, and how much you are willing to pay. Teams that treat logging as a designed system, with structured entries, central audit collection, deliberate retention and cost awareness, find problems faster, pass audits more easily and spend less. Teams that leave it on defaults tend to learn its limits during their worst incident.