Cloud Trace is a distributed tracing system that tracks how long requests take as they pass through your application.

The problem with modern applications

Old applications were often one big program running on one machine. When something was slow, you looked at that one program. Modern applications are different. A single click on a web page may trigger a web server, an authentication service, a product service, a payment service, a database and a message queue. These pieces may run on different machines, written by different teams, in different languages. When the page is slow, nobody can say which piece is to blame, and each team insists their part is fast.

What the service does

Cloud Trace collects timing data from your applications and shows the full journey of a request. It answers questions such as: how long do requests take overall, which step takes the most time, has latency grown over time, which services does my application depend on, and where do errors begin. It works for traditional services and, increasingly, for applications that use generative AI models and tools.

Twelve words you will meet in every tracing conversation.

Hold on to the picture: a trace is a tree of spans, each span is a stopwatch reading, and context propagation is what ties them into one story.

Observability rests on three kinds of data, and each answers a different question.

Observability means being able to understand what a system is doing by looking at the data it produces. Three signals are used together. Metrics are numbers over time, such as requests per second or error rate. Logs are text records of events, such as a user logged in or a payment failed. Traces are the journeys of individual requests, with timing for every step. Each has strengths and blind spots.

flowchart LR
M["Metrics: something is slow"] --> T["Traces: where it is slow"]
T --> L["Logs: why it is slow"]
L --> F["Fix the cause"]
        

A trace looks like a tree, and shows up on screen as a waterfall of bars.

Imagine a customer opens a product page. The web frontend receives the request, which becomes the root span. To build the page, the frontend asks the product service for details, and asks the recommendation service for suggestions. The product service reads the database. The recommendation service calls a cache first, then a machine learning model. Each of these steps is a span, and each span knows its parent. Together they form a tree. Shown on a timeline, each span becomes a horizontal bar, with the width showing duration and the position showing when it began.

flowchart TD
R["Root span: Frontend handles page request"] --> P["Span: Product service call"]
R --> C["Span: Recommendation service call"]
P --> D["Span: Database query"]
C --> K["Span: Cache lookup"]
C --> ML["Span: Model prediction"]
        

Reading the waterfall

How spans travel from your application to a screen in the console.

A tracing client inside your application records spans while requests are handled. These spans are batched and exported, either directly to Google Cloud or through a collector running beside the application. They arrive at Google Cloud’s ingestion endpoint, are stored in your project, and appear in the Trace Explorer, where you can filter, aggregate and inspect them. Your data stays in your own Google Cloud project.

flowchart LR
APP["Application with tracing client"] --> EXP["Exporter"]
EXP --> COL["OpenTelemetry Collector (optional)"]
COL --> ING["Telemetry API: OTLP ingestion"]
EXP --> ING
ING --> STORE["Trace storage in your project"]
STORE --> EXPL["Trace Explorer"]
STORE --> ANA["Log Analytics: SQL over traces"]
        

OpenTelemetry is the open standard that teaches your applications to emit traces.

OpenTelemetry is an open source, vendor neutral framework for producing telemetry, meaning traces, metrics and logs. It is the recommended way to instrument applications for Cloud Trace. Because it is a standard, the same instrumentation can send data to Cloud Trace today and to another tool tomorrow, without rewriting your application. Older Cloud Trace specific client libraries still exist, but new work should use OpenTelemetry.

Automatic and manual instrumentation

Without a shared trace identity, spans from different services remain strangers.

When service A calls service B, how does B know it is part of A’s trace? The answer is context propagation. Service A adds a small piece of information to the outgoing request, usually in a request header. This carries the trace identifier, the identifier of the calling span, and a flag that says whether the trace is being recorded. Service B reads this header, creates its spans as children of A’s span, and passes the same information on to its own downstream calls.

sequenceDiagram
participant U as User
participant A as Service A
participant B as Service B
participant D as Database
U->>A: Request
A->>A: Start root span, create trace ID
A->>B: Call with trace context in headers
B->>B: Read context, start child span
B->>D: Query with same trace
D-->>B: Result
B-->>A: Response
A-->>U: Response and finish root span
        

Recording every request is expensive, so most systems record only a share of them.

A busy service can handle thousands of requests per second. Recording a full trace for each would create huge volumes of data and cost, with little extra insight, since most requests look alike. Sampling decides which traces to keep. A small, well chosen sample still shows the typical behavior and the rare problems, as long as it is chosen with care.

There are two doors for sending trace data to your project, and one is preferred.

Google Cloud offers two ways to send traces. The Telemetry API implements the OpenTelemetry protocol, called OTLP, and works directly with the open source ecosystem. It is the recommended option for new work, and its limits are generally more generous than those of the older, Google specific Cloud Trace API, which remains available. The Telemetry API also supports VPC Service Controls. The two APIs are enabled separately, and the Cloud Trace API must be enabled on your project for trace data to be stored, even if you send data through the Telemetry API.

Why use a collector?

You can export directly from your application, but a collector brings advantages. It lets the application send data quickly and move on, while the collector handles batching, retries and encryption. It can add resource information, drop noisy spans, mask sensitive attributes and perform tail sampling. It is also a single place to change where data goes, so you can switch or add destinations without touching application code.

The main window into your trace data, built for both the big picture and the fine detail.

The Trace Explorer shows aggregate information about your traces and lets you open individual ones. A latency heatmap displays how request durations are spread over time, so you can see at a glance whether most requests are fast, whether a slow group exists, or whether latency crept up after a release. You can click and drag on the heatmap to select a region, such as the slowest requests, and see the matching traces. Filters narrow results by service, route, status, attribute or application.

A trace is much more powerful when you can jump from it to the logs and numbers around it.

When your logs record the trace and span identifiers of the request being handled, Google Cloud can link them. In the trace details you can open the log entries that belong to a span, and in the log explorer you can jump from a log entry to its trace. This turns a mystery into a short walk: you see a slow span, open its logs and read the warning about a retry or a timeout. Default scopes decide which projects are searched when you jump between the two views.

Analytics with SQL

Trace data can also be analyzed with SQL through Log Analytics. You can ask questions that the explorer cannot answer easily, such as the average duration of a database call by customer type, or how many requests passed through a particular service last week. Queries can join trace and log data, and results can be shown as tables or charts. The older analysis reports page has been replaced, so use the Trace Explorer and Log Analytics for analysis.

flowchart TD
TR["Slow trace in Trace Explorer"] --> SP["Select slow span"]
SP --> LG["Open its log entries"]
LG --> CA["Find cause: timeout or retry"]
TR --> AN["Query many traces with SQL"]
AN --> PAT["Find pattern: one customer type or region"]
        

Where traces come from on Google Cloud, with and without your own code.

Experienced engineers recognize a handful of shapes that appear again and again.

flowchart TD
S["Request is slow"] --> Q1{"One span dominates?"}
Q1 -->|Yes| A1["Optimize or cache that dependency"]
Q1 -->|No| Q2{"Many similar small spans?"}
Q2 -->|Yes| A2["Batch queries, fix N plus one"]
Q2 -->|No| Q3{"Spans run one after another?"}
Q3 -->|Yes| A3["Run independent calls in parallel"]
Q3 -->|No| A4["Look for gaps: cold starts, queues, pools"]
        

Tracing is not just for emergencies, it is a daily habit for healthy teams.

Teams that use tracing well do not wait for outages. They review traces after every major release to confirm that latency has not crept up. They compare traces from before and after a change. They study the slowest one percent of requests, because those are the ones real users complain about. They connect tracing to service level objectives, which are promises about speed and reliability, such as ninety five percent of checkouts finishing within two seconds. Metrics show whether the objective is being met, and traces explain why it is not.

Traces describe your system in detail, so protect them like any other sensitive data.

Who can see and send traces

Access is controlled through identity and access management. Roles decide who may view traces, who may write trace data, and who may administer settings. Applications authenticate with service accounts, and each application should have only the permission to write trace data, not to read everything. Developers who investigate issues need view access to the projects they work on, and trace scopes let you grant views across several projects in a controlled way.

What goes into attributes

Attributes are powerful, and dangerous. It is tempting to attach a user’s email, a full request body or a query with real values. These may contain personal or secret data, which would then be stored in your trace backend and visible to everyone who can read traces. Record identifiers that are safe and useful, such as a route name, a status code or an internal customer tier, and avoid secrets, passwords, tokens and personal data.

Trace data is not kept forever, and there are limits on how much you can send.

Cloud Trace stores spans for a limited retention period, so it is designed for troubleshooting and recent analysis, not long-term archives. The exact period is stated in the current documentation, so check it when planning investigations. If you need history for longer, such as for audits or seasonal comparisons, summarize important findings into metrics or reports, or export the data to a store that you control.

There are also quotas and limits, such as how many spans you can ingest per minute, how big a span or an attribute can be, and how many attributes a span may carry. The Telemetry API has more generous limits than the older API, and limits can differ by region. When you exceed limits, spans may be rejected or dropped, which creates incomplete traces.

You pay for the number of spans you send and the number you scan, so volume is the lever.

Cloud Trace pricing is part of Google Cloud Observability pricing. Charges are based on the number of trace spans ingested and the number of spans scanned when you query them. A monthly free allowance has historically been offered, but amounts and terms change, so use the official pricing page and calculator. The key insight is that cost follows span count, and span count follows traffic, number of spans per request and sampling rate.

Where tracing quickly turns confusion into answers.

The following are common patterns rather than named customer stories, because specific architectures change over time.

Tracing gives sharp visibility, and asks for instrumentation effort and cost discipline.

Habits that make traces useful, and habits that bury the signal in noise.

Patterns that work

Strong teams standardize on OpenTelemetry and the W3C context format across all services. They name spans consistently, using stable names for operations and putting variable data in attributes. They add trace and span identifiers to every log line. They instrument the critical user journeys first, then widen the coverage. They use a collector to batch, redact and sample. They review traces after releases and keep a short guide of common patterns for new engineers.

A checklist to make your tracing helpful, safe and affordable.

Mistakes beginners make most often

The first is instrumenting only some services, which breaks traces. The second is losing context at queues and proxies. The third is recording every request at full rate. The fourth is reading only averages. The fifth is putting sensitive data into attributes. The sixth is creating too many tiny spans. The seventh is waiting for an outage before turning tracing on for the first time.

Several tools overlap with Cloud Trace, and each answers a slightly different question.

flowchart TD
Q1{"What do you need to know?"} -->|"Is the system healthy over time?"| MON["Cloud Monitoring"]
Q1 -->|"Which service or step is slow?"| TRC["Cloud Trace"]
Q1 -->|"What exactly happened?"| LOG["Cloud Logging"]
Q1 -->|"Which code inside a service is slow?"| PRO["Cloud Profiler"]
Q1 -->|"Which errors are happening?"| ERR["Error Reporting"]
        

Because Cloud Trace accepts the open OpenTelemetry standard, you are not locked in. The same instrumentation can send data to Cloud Trace, to a self-hosted backend or to a third party platform, and many teams send to more than one while they evaluate.

It is Google Cloud’s distributed tracing service, which records how long each step of a request takes across your services so you can find bottlenecks and errors.

A span is one timed operation, such as a database call. A trace is the collection of all spans that belong to one request, arranged as a tree from the first service to the last.

Usually a little. Automatic instrumentation with OpenTelemetry covers popular frameworks with minimal change, and some Google Cloud services create spans for you. Custom spans add detail for your own business steps.

OpenTelemetry is recommended. It is an open standard that keeps your instrumentation portable, and Google Cloud supports it through the Telemetry API.

Context is probably not being propagated, or other services are not instrumented. Check that trace headers are passed in outgoing requests and not stripped by proxies, and that each service reads them.

Sampling means recording only a share of requests. Most production systems need it to control volume and cost. Start with a small fraction, and use tail sampling if you want to keep slow and failing requests.

Make your logs include the trace and span identifiers. Then you can open the logs of a span from the trace view, and open the trace from a log entry.

For a limited retention period stated in the current documentation. It is meant for troubleshooting and recent analysis, so save summaries elsewhere if you need long-term history.

It is billed by the number of spans ingested and the number scanned by queries, under Google Cloud Observability pricing, usually with some free allowance. Check the official pricing page for current figures, and control cost with sampling.

No, not secrets or personal data. Record safe identifiers and redact sensitive attributes in the collector, and restrict who can view traces.

Yes. Trace data can be queried through Log Analytics with SQL, and queries can join trace and log data. Use it for questions beyond what the explorer filters provide.

Tracing shows which service or step in a request is slow. Profiling shows which functions inside one service consume the most CPU or memory. Use tracing first to find the service, then profiling to look inside.

Cloud Trace is Google Cloud’s managed distributed tracing service. Applications instrumented with OpenTelemetry create spans, pass trace context between services, and export data through the Telemetry API, optionally via a collector. The Trace Explorer, linked logs and SQL analytics turn that data into answers about where time and errors come from. Good results depend on consistent instrumentation, preserved context, sensible sampling, careful attributes, and a habit of reading traces before problems become outages.