Cloud Trace is Google Cloud’s tool for following a single request on its journey through your system and showing exactly where the time went.

When you order something online, the shop gives you a tracking number. With that number you can see that the parcel left the warehouse, reached the sorting center, rode on a truck, and arrived at your door, with a time stamp at every stop. If the parcel is late, the tracking page shows which stop was slow. Cloud Trace gives your software the same kind of tracking page.

Each request that enters your application, such as a click on “Buy now,” can be given a tracking number. As the request travels from your website to a login service, then to a shopping cart, then to a database and a payment provider, every step records when it started and when it finished. Cloud Trace collects those records and draws them as one timeline. You can see at a glance which step was slow.

Cloud Trace is a distributed tracing system. “Distributed” means your application is spread over many programs, containers or machines. “Tracing” means following one request across all of them. It belongs to Google Cloud Observability, the family of tools that also includes Cloud Logging and Cloud Monitoring.

What Cloud Trace is good at

• Showing where a slow request spent its time, not only that it was slow.
• Revealing hidden chains of calls between services that nobody remembered existed.
• Helping teams agree on facts during a slowdown, because everyone looks at the same timeline.
• Pointing to the right team, since each step belongs to a service with an owner.

A short history in plain words

Tracing grew up with big websites. When a company has hundreds of small services, one slow page can involve dozens of them, and no single person knows the whole picture. Large companies built internal tracing tools to cope. Twitter shared one called Zipkin in 2012, and Uber later shared Jaeger. In 2019, two open projects merged into OpenTelemetry, which is now the common standard way to create trace data. Google Cloud supports sending OpenTelemetry data into Cloud Trace.

Product names, screens and limits change over time. This guide teaches ideas that stay stable, and it points out places where you should confirm current details in the official Google Cloud documentation.

Finding slowness was easy when an application lived in one program. It becomes a detective story once the application is split into many parts.

In the old days, many applications were a single big program, called a monolith. If a page was slow, a developer could run a profiler inside that one program and see which function was guilty. Everything happened in one place, so one tool could see everything.

Modern systems often use microservices, which are many small programs that each do one job and talk to each other over the network. This brings freedom, because teams can change their own service without waiting for others. It also brings a new problem. A single click may touch ten services, and each service only knows about its own small piece of the story.

Why logs and metrics are not enough alone

Logs are written messages from each service. They tell you what that service saw, but they are scattered over many places and hard to line up in time. Metrics are numbers such as average response time. They tell you that something is slow, but an average hides the details of any single request.

Suppose your checkout page has an average response time that looks fine, yet some customers wait ten seconds. Metrics show the average, not the unlucky ones. Logs from ten services show ten partial stories. A trace is the one thing that ties the stories into a single timeline for one request.

The cost of not knowing

Without tracing, teams argue. The frontend team says the database is slow, the database team says the network is slow, and the network team says the frontend asks for too much. Hours pass. With tracing, the timeline is shared evidence, and the discussion becomes “step seven took four seconds, who owns step seven?”

A handful of words carry almost all of tracing. Learn them once and every screen, document and tool will make sense.

Trace

A trace is the complete record of one request as it travels through your system. It has a unique trace ID, a long string of letters and numbers that works like a tracking number. In Cloud Trace and in the common standards, a trace ID is a 32-character hexadecimal value, which means it uses the digits zero to nine and the letters a to f.

Span

A span is one timed piece of work inside a trace. It has a name such as “GET /checkout” or “query orders table,” a start time, an end time, and a span ID of its own. A trace is a collection of spans, much like a book is a collection of chapters.

Parent, child and root

Spans form a family tree. The first span, which represents the whole request arriving, is the root span. When that work calls another piece of work, a child span is created, and the first one becomes its parent. Each child remembers its parent’s span ID, which is how the tree is rebuilt later.

Attributes, events and status

Attributes (older tools call them labels) are small notes attached to a span, such as the HTTP method, the page route or the database name. Events are time-stamped moments inside a span, like “cache miss” or “retry started.” Status says whether the span succeeded or failed, so errors can be found quickly.

Latency and percentiles

Latency is the time something takes. Because averages hide problems, engineers use percentiles. The 50th percentile, or p50, is the typical request: half are faster and half are slower. The 95th percentile, p95, means that ninety-five out of a hundred requests are faster than this value. The 99th percentile, p99, shows the slow tail that your unluckiest one percent of users experience.

Context propagation

Context propagation means passing the trace ID and the parent span ID from one service to the next, usually inside a request header. Without it, each service would start a new unrelated trace, and the timeline would fall apart into pieces.

Instrumentation and sampling

Instrumentation is the act of adding tracing to your code, either automatically through a helper library or manually by creating spans yourself. Sampling means keeping only a portion of all traces to save cost and storage, for example one request out of every hundred.

The best way to understand tracing is to follow one imaginary request through a small online shop.

A customer presses “Place order.” The request reaches a load balancer, which hands it to a frontend service. The frontend asks a cart service for the items, and the cart service reads from a database. Then the frontend asks a payment service to charge the card, and the payment service calls an outside payment provider. Finally the frontend answers the customer.

flowchart LR
  U["User browser"] --> LB["Load balancer"]
  LB --> FE["Frontend service"]
  FE --> CT["Cart service"]
  CT --> DB[("Database")]
  FE --> PY["Payment service"]
  PY --> EXT["External payment provider"]

Now look at the same journey in time order. Each arrow below will become part of a span in the trace.

sequenceDiagram
  participant U as User
  participant F as Frontend
  participant C as Cart service
  participant D as Database
  participant P as Payment service
  U->>F: Place order
  F->>C: Get cart items
  C->>D: Read items
  D-->>C: Items
  C-->>F: Cart
  F->>P: Charge card
  P-->>F: Approved
  F-->>U: Order confirmed

The waterfall view

Cloud Trace shows a trace as a waterfall. Each span is a horizontal bar. The bar starts where the span started and ends where it ended, so longer bars mean more time. Child spans are drawn below their parent and slightly indented. The top bar, the root span, covers the entire request, and the smaller bars show how that time was divided.

Reading a waterfall is a skill that comes quickly. Look first for the longest bars. Then look for gaps, which are places where the parent was waiting and no child was working. Then look for bars stacked one after another that could run side by side. These three clues explain most slow requests.

In this made-up example, the request took 1,150 milliseconds. The cart service used about 140 of them, which is fine. The payment step used 950, and nearly all of that was the call to the outside provider. The trace makes the answer obvious: the slow part is not your database, it is the external provider.

A tracing system has a few simple parts: something that creates spans, something that sends them, and something that stores and shows them.

flowchart TD
  APP["Your application code"] --> SDK["OpenTelemetry SDK creates spans"]
  SDK --> EXP["Exporter for Google Cloud"]
  SDK --> COL["OpenTelemetry Collector optional"]
  COL --> EXP
  EXP --> API["Google Cloud trace ingestion"]
  GCP["Some Google Cloud services"] --> API
  API --> STORE["Trace storage with limited retention"]
  STORE --> TE["Trace Explorer in the console"]
  STORE --> LOGS["Links with Cloud Logging"]
  STORE --> MON["Links with Cloud Monitoring"]

Part 1: the code that creates spans

Inside your application, a tracing library watches important moments. When a request arrives, it starts a root span. When your code calls a database or another service, it starts a child span. When the work ends, it closes the span and records the finish time. Today, the most common library family for this is OpenTelemetry, often shortened to OTel.

Part 2: the exporter

Finished spans are not sent one by one, because that would slow your app. They wait in a small memory buffer, and an exporter sends them in groups to Cloud Trace in the background. Google provides an exporter for OpenTelemetry. There is also a newer route that accepts OpenTelemetry’s own data format directly, so check current documentation to see which route fits your language and platform.

Part 3: the optional collector

The OpenTelemetry Collector is a small separate program that receives spans from many applications, can process them, and forwards them. Teams use it to centralize settings, to add or remove attributes, to keep fewer traces by smart sampling, and to avoid putting export logic inside every service. It is optional, but it grows more helpful as a company grows.

Part 4: ingestion, storage and the screens

Cloud Trace receives spans through an API, checks permissions, groups spans by trace ID, and stores them for a limited time, which is about thirty days at the time of writing. You then search and view traces in Trace Explorer in the Google Cloud console. Links connect traces to logs and metrics, so you can move between the three without losing your place.

If the tracking number is lost between two services, the trace breaks in half. Propagation is the habit that prevents this.

When service A calls service B, A adds a small piece of information to the request, usually in an HTTP header. This piece says “you are part of trace number so-and-so, and my span is the parent.” Service B reads it, makes its own spans inside the same trace, and passes the same information on when it calls service C.

sequenceDiagram
  participant A as Service A
  participant B as Service B
  participant C as Service C
  Note over A: Creates trace ID and root span
  A->>B: Request with trace context header
  Note over B: Reads header and creates child span
  B->>C: Request with same trace ID and new parent span
  C-->>B: Response
  B-->>A: Response

The headers in use

The modern standard is called W3C Trace Context, and its main header is named traceparent. It carries a version, the trace ID, the parent span ID and a flag that says whether the trace is being kept. Google Cloud also has an older header named X-Cloud-Trace-Context, and Google Cloud load balancers and some other products can add it to incoming requests. Modern libraries can read and write both, so mixed systems can still produce connected traces.

Where propagation breaks

• A service that forgets to pass the header on. A custom HTTP client or a proxy that drops unknown headers makes the trace split into two separate traces.
• Different standards in different services. If one service writes only an old format and the next reads only the new one, the link is lost.
• Background work. A job picked from a queue starts with no request around it, so it needs the context to be saved with the message.
• Third-party services. Outside providers usually do not continue your trace, so you see one span for the whole call and nothing inside it.

Messages and queues

Many systems hand work from one service to another through a message queue, such as Pub/Sub. The sender is not waiting for the receiver, so no live request carries the header. The usual solution is to put the trace context inside the message’s attributes. The receiver reads it and either continues the same trace or starts a new trace that is linked to the original, which is a way of saying “this work was caused by that request.”

There are several ways to add tracing to an application. Choosing a path early saves a lot of confusion later.

Automatic instrumentation

Automatic instrumentation uses ready-made helpers that watch popular libraries, such as web frameworks, HTTP clients and database drivers, and create spans without changes to your business code. In some languages, a helper can be attached when the program starts. This is the quickest way to get a first useful trace and is a good starting point for beginners.

Manual instrumentation

Manual instrumentation means you create spans yourself around work that matters, such as “calculate shipping” or “apply discount rules.” You choose the span names and add attributes that make sense for your business. It takes more effort, but it shows your own logic and not only the technical plumbing.

The standards and older libraries

OpenTelemetry is the recommended standard for new work. An older project called OpenCensus was a predecessor and has been retired, so new code should avoid it. Google Cloud also offered its own earlier client libraries for Cloud Trace. Treat those as older routes and prefer OpenTelemetry unless a platform guide says otherwise.

What makes a good span

• A stable, low-variety name. Use the route pattern such as “GET /orders/{id}” and not the actual order number. Otherwise you create millions of different names that cannot be grouped.
• Useful attributes. Add the region, the customer plan type, or the feature flag in use, since these help filtering later.
• The right size. A span should cover a meaningful piece of work, not every tiny loop turn.
• Honest status. Mark a span as failed when it fails, so you can search for errors.

Spans are only as useful as their names and notes. A few simple rules turn a noisy trace into a clear story.

When you open a waterfall, the first thing you read is the span names. If the names are clear, you understand the story at a glance. If they are vague or random, you must open every span to learn anything. Good naming is the cheapest improvement you can make to a tracing system.

Naming rules

• Describe the operation, not the data. “GET /orders/{id}” is a good name. “GET /orders/84721” is a bad one, because it changes with every request.
• Keep the number of different names small. A system should have hundreds of span names, not millions. Names are used for grouping and searching.
• Use a verb and a noun for business steps. “calculate shipping” and “apply discount” read like plain sentences.
• Be consistent across services. If one team writes “db query” and another writes “database call,” shared filters will miss half the spans.

Attribute rules

Attributes are the notes that make filtering possible. Choose them by asking “what question will someone ask during an incident?” Typical questions are: Which region? Which version of the service? Which type of customer? Which feature switch was on? Which database was used? Add attributes that answer these questions.

The OpenTelemetry project publishes common naming conventions for attributes such as the HTTP method, response status and database system. Following these shared conventions helps tools understand your data and helps new teammates read it. Because the conventions evolve, check the current version when you set a team standard.

The path to a first visible trace is short. Follow the order, and each step prepares the next.

Two groups of identities touch Cloud Trace: programs that write spans and people who read them. Each needs its own narrow permission.

Writers: the programs

Your application runs under an identity, commonly a service account. To send spans, that identity needs permission to write traces. The ready-made role for this is Cloud Trace Agent. On Cloud Run and GKE, the identity attached to the service or workload is the one that matters, so check which service account the platform is really using, not the one you assume.

Readers: the people

Engineers who investigate need to read traces. The ready-made role Cloud Trace User allows viewing, and Cloud Trace Admin allows more control. Most investigators need only the user role. Granting it to a group, and not to individuals, keeps reviews and removals simple.

Common permission problems

• Permission denied on export. The app’s service account lacks the writer role. Grant the agent role to that account.
• API not enabled. The error mentions that the Cloud Trace API has not been used. Enable it in the project.
• Engineer sees an empty page. The engineer is looking at the wrong project, or lacks the reader role.
• Cross-project confusion. Services in one project may send traces to another project. Check which project the exporter points to, because traces appear only in the project that received them.

Millions of requests may flow through a system each day. Trace Explorer helps you pick out the few that matter.

Trace Explorer is the main screen in the Google Cloud console for working with traces. Its layout may change over time, but the ideas stay the same: choose a time range, filter down to the requests you care about, look at how they are spread out in time, then open one to read its waterfall.

Step 1: set the time range

Choose the period of the problem, such as “the last hour” or “yesterday between two and four in the afternoon.” Narrow ranges load faster and make patterns clearer. If an incident has just ended, widen the range a little on both sides to include normal behavior for comparison.

Step 2: filter

You can filter by service, by span name, by status, by minimum latency, and by attributes you added. A useful first filter is “slow requests to one route.” Another is “failed spans in the payment service.” Good attributes such as region or customer plan make these filters much more powerful, which is why meaningful attributes matter so much.

Step 3: look at the shape

The screen shows a chart where each trace is a point placed by time and latency. Clusters tell stories. A thin band high above the rest suggests a special slow case. A cloud that suddenly moves up at one moment suggests a change, such as a deployment or an outage in a dependency. A second band running parallel to the first may suggest two different code paths with different speeds.

Step 4: open a trace

Click a point to open its waterfall. Look for the longest bar, the empty gaps, and the stacked steps. Open span details to read attributes and events. Follow links to related logs, so you see the exact messages that were written during this very request.

Step 5: compare

One trace is a clue, but two make a comparison. Open a slow trace beside a fast trace of the same route. The span that grew is your suspect. This habit turns “it feels slow” into “the inventory call took 40 ms normally and 1,900 ms in the slow case.”

Slow requests are slow in a handful of recognisable ways. Learn the shapes, and diagnosis becomes quick.

Two ideas worth explaining

N plus one queries. Imagine a page that shows a list of fifty orders. The code asks the database for the list once, and then asks again for each order’s customer, one at a time. That is one plus fifty queries. In a trace it looks like a comb of fifty small bars. The fix is often a single combined query. This pattern is invisible in averages and very clear in a waterfall.

The critical path. When some work happens in parallel, only the longest branch decides the total time. That branch is called the critical path. Speeding up a step that is not on the critical path changes nothing for the user. Traces show which steps are on the path, so you do not waste effort tuning the wrong thing.

Most of the time in a typical request is spent waiting for someone else. Tracing these waits is where the biggest wins hide.

Databases

A database is a program that stores and finds data. Your application sends it a query, which is a question such as “give me all orders for this customer.” Automatic helpers can create a span for each query and record the type of database and a shortened form of the query. A slow query appears as a long bar. Many repeated queries appear as a comb.

Be careful about recording the full text of queries. Query text may contain customer data if values are written inside it. Prefer recording a template of the query with placeholders and not the actual values, and follow your privacy rules.

Caches

A cache is a fast storage place for answers you may need again soon, such as a stored copy of a product page. A cache lookup normally takes a tiny fraction of the time of a database query. Spans for cache lookups, with an attribute that says “hit” or “miss,” let you measure how often the cache saves time. A sudden growth in misses often explains a sudden slowdown, because every miss now falls through to the slower database.

Outside calls

Your service may call a payment company, a mapping service or an email provider. You cannot see inside them, but you can see how long you waited. Create a span for each such call, and record the status and the number of retries. When an outside provider has a bad day, your traces will prove it with clear evidence you can share with them.

flowchart TD
  REQ["Incoming request"] --> CK{"Is the answer in the cache?"}
  CK -->|Hit| FAST["Return the cached answer quickly"]
  CK -->|Miss| DBQ["Query the database"]
  DBQ --> SAVE["Store the answer in the cache"]
  SAVE --> RESP["Return the answer"]
  FAST --> RESP

Keeping every trace of a busy system is expensive and often unnecessary. Sampling is the art of keeping enough to learn and no more.

A busy service may handle thousands of requests every second. Each request can create many spans, and every span has a cost to create, send and store. Most requests look nearly identical. So systems keep only a portion, and that portion is chosen by a rule called a sampling policy.

flowchart TD
  R["Request arrives"] --> Q{"Does the incoming context already say sampled?"}
  Q -->|Yes| K["Keep and record spans"]
  Q -->|No decision| S{"Does a random draw fall below the sampling rate?"}
  S -->|Yes| K
  S -->|No| D["Do not record spans"]
  K --> E["Export spans to Cloud Trace"]

Head-based sampling

In head-based sampling, the decision is made at the very start of the request, before anyone knows how it will turn out. A random draw decides, for example, to keep one request in a hundred. The decision is then passed along in the trace context, so every service in the chain agrees. This is simple and cheap, but it may miss the rare slow or failed requests you most want to see.

Tail-based sampling

In tail-based sampling, the decision is made after the request finishes, when its outcome is known. A collector can hold the spans for a short time and then keep every trace that was slow or failed, plus a small portion of normal ones. This keeps the interesting traces, but it needs more memory and a more careful setup, because all spans of a trace must reach the same collector.

Choosing a rate

A common approach is to begin with a higher rate in development and test, such as keeping everything, and a small rate in production. Increase the production rate if you find yourself unable to locate examples. Lower it if costs or overhead grow. Many teams also choose a rule such as “always keep traces with errors” when their tools allow it.

Follow a single span from the moment it is born to the day it expires, and you will understand every setting along the way.

flowchart LR
  A["Request starts: root span created"] --> B["Child spans for calls and queries"]
  B --> C["Spans end and wait in a buffer"]
  C --> D["Batch export runs in the background"]
  D --> E["Cloud Trace ingests the spans"]
  E --> F["Trace is stored and searchable"]
  F --> G["Engineer investigates in Trace Explorer"]
  F --> H["Data expires after the retention period"]

Why the delay matters

Spans are exported in the background and from different services at slightly different moments. A trace may therefore look incomplete if you open it extremely soon after the request, with some services still missing. If a trace looks short, wait a minute and reload before concluding that propagation is broken.

A short look at what happens inside your application explains why tracing is light, and why it sometimes loses data.

The tracer and the current context

The library keeps track of “which span am I currently inside” for each running piece of work. This is the current context. When your code starts a new span, the library looks at the current context and makes the new span a child of it. When the span ends, the context steps back to the parent. In programs that handle many requests at once, each request has its own context so that spans never get mixed up.

The span processor and the buffer

When a span ends, it is handed to a span processor. The usual choice is a batching processor. It stores spans in a queue of limited size and sends them in groups after a short time or when the group is full. This is efficient, because sending many spans in one network call is much cheaper than sending each separately.

What can go wrong

• Full buffer. If a service creates spans faster than it can export them, the queue fills, and new spans are dropped. This protects your app, but you lose data.
• Shutdown before export. If the program exits right after a request, spans still waiting in the buffer may never be sent. Short-lived jobs must flush the buffer before they exit.
• CPU-throttled instances. On some serverless platforms, background work may be slowed when no request is active, which can delay exports. Check platform guidance on this.
• Clock differences. Different machines have slightly different clocks. A child span may appear to start a little before its parent. Do not over-read tiny differences between services.

The overhead

Creating a span takes very little time, usually a tiny fraction of a millisecond, and most of the work is done in the background. Even so, overhead is not zero. Adding thousands of spans to one request, or storing large attributes on each, can slow the service and increase costs. Measure the effect in a test environment, and keep spans meaningful.

Not every request waits for an answer. Some work is handed off and finished later, and tracing must follow it across the gap.

In an event-driven system, one service publishes a message, such as “order placed,” and other services react to it whenever they are ready. The sender does not wait. This makes systems flexible and resilient, but it also breaks the simple picture of a request moving in a straight line.

Two ways to connect the pieces

Continue the same trace. The receiver reads the trace context stored in the message and creates spans inside the original trace. The waterfall then shows the delay between the message being published and being processed. This is simple to read for short, one-to-one chains.

Start a new trace with a link. The receiver starts a fresh trace and adds a link that points back to the original span. This suits long delays, fan-out to many receivers, or batches that combine many messages, because one giant trace with hours of waiting would be unreadable.

What to look for

• Queue delay. The time between publish and start of processing is often the biggest part of the end-to-end time. It grows when receivers are overloaded.
• Retries and redelivery. A message that fails may be delivered again. Several processing spans for one message show repeated failure.
• Fan-out. One event can trigger many receivers. Links help you see them all as part of the same cause.
• Batch jobs. A scheduled job has no incoming request. Create a root span for each run and record the number of items processed as an attribute.

A trace tells you which step was slow. A log tells you what that step was saying while it was slow. Together they close the case.

The key idea is simple: write the trace ID and span ID inside every log line. Cloud Logging has special fields for these values. When a log entry carries the trace field in the expected format, which includes the project name and the trace ID, Google Cloud can link the log line to its trace. In the console, you can then jump from a span to the logs written during it, and from a log line to its trace.

Many logging libraries can add these fields for you when tracing is active. On some Google Cloud platforms, such as Cloud Run, there are also ready ways to connect logs written to standard output with the incoming trace header. Check the current platform documentation for the exact conditions, because the details depend on how logs are written.

Cloud Trace is strongest when used as part of a team of tools. Each answers a different question, and they hand work to each other.

Metrics tell you when to look

Cloud Monitoring watches numbers such as request counts, error rates and latency percentiles. An alert on p99 latency tells you that users are suffering right now. The alert gives you a time and a service, and those become your filters in Trace Explorer. Some metric charts can also show example traces for a data point, a feature often called exemplars. Availability depends on the type of metric and on the platform, so check current documentation.

Error tools tell you what failed

Error Reporting groups similar crashes and exceptions, showing how many times each occurred. When an error group looks important, you can search for a trace that contains a failed span from the same service and time to see what the request was doing when it broke.

Profilers tell you why code is slow

A trace shows that one span was slow, but not which lines of code inside it burned the time. Cloud Profiler samples your running program continuously and shows which functions use the most processor time or memory. Use the trace to find the slow step, then use the profiler to look inside it.

Tracing is not only for emergencies. It is also a planning tool that helps teams decide how fast each part of a request is allowed to be.

A service level objective, or SLO, is a promise about quality that a team makes to itself, such as “ninety-nine percent of checkout requests finish in under one second.” The measurable number behind such a promise is called a service level indicator. Latency percentiles are common indicators, and they come from metrics. Traces explain what stands behind those numbers.

The latency budget

A latency budget splits the total allowed time among the steps. If checkout may take one second, the team might allow 100 milliseconds for the frontend, 200 for the cart, 500 for payment and 200 for everything else. A trace shows how real requests spend the budget. When one step regularly uses more than its share, the team knows where to invest effort.

Why percentiles and traces work together

A metric might say that p99 latency is 2.4 seconds. That is a symptom. The next move is to open traces from the slowest one percent and ask which span is consistently the longest. If the answer is always the payment call, you have a clear target. If the answer varies, the cause may be shared, such as slow start-ups after scaling up.

Error budgets and change

If an SLO allows one percent of requests to be slow, that allowance is an error budget. When the budget is nearly used up, teams slow down risky releases and work on reliability. Traces from the slow and failed requests that used the budget become the to-do list for that work.

The idea is the same everywhere, but each platform has its own ways of helping and its own details to watch.

Some Google Cloud products can create basic trace data on their own, and some pass trace headers to your code. Others leave everything to you. Because these behaviors change as products evolve, treat the table below as a guide to ideas and confirm specifics in current documentation.

Special notes

• Serverless instances come and go. Spans must be exported soon after they finish, because an instance can be frozen or removed right after the response.
• Service meshes see the network, not your code. They add spans for calls between services but cannot see inside a service. Pair them with application spans.
• Virtual machines need the right scope and identity. A machine with no permission to write traces will export without errors being obvious to the user.

Real companies rarely have one tidy project. Traces must cross boundaries between projects, teams and even different clouds.

Many Google Cloud projects

Large organizations split systems across several projects, one per team or environment. Traces are stored in the project that receives them. If service A in project one calls service B in project two, and each exports to its own project, the full trace may be split between two places. Teams solve this in one of two ways. One way is to send all traces to a central observability project, and the other is to use features that let you view data from several projects together. Check current documentation for the options available to your setup.

Many teams

When many teams share one tracing setup, naming and access rules matter even more. Use clear service names, a common set of required attributes, and a reader role that includes engineers who may be called to help another team. Remember that wide read access increases privacy risk, so mask sensitive data centrally.

Other clouds and on-premises systems

If part of your system runs on another cloud or in your own data center, you can still use OpenTelemetry there and send spans to Cloud Trace, or use a collector to forward them. The service account or credentials used must be handled carefully, since the program is outside Google Cloud and cannot use an attached platform identity. Some teams prefer to send all spans to a central collector, and then forward them to one or more backends. This also keeps the door open if you later change tools.

Traces are a detailed diary of how your system works. That diary must be protected like any other sensitive record.

Trace data can reveal internal service names, server addresses, database tables, error messages and even customer information if careless attributes are added. In the wrong hands, this is a map for attackers. For this reason, a few layers of defense are needed.

Layer 1: control who can read and write

Grant the writer role only to the identities that run your applications, and the reader role only to the groups who investigate problems. Avoid broad roles. Review the lists on a regular schedule, and remove people when they change teams.

Layer 2: keep sensitive data out of spans

Decide in advance what must never appear in a span name or attribute: passwords, tokens, full card numbers, health details, government identifiers and anything your privacy rules protect. Use harmless identifiers, such as an internal customer number or a hashed value, if you need to filter by customer. Review code in pull requests with this in mind.

Layer 3: clean data centrally

A collector can remove or mask attributes before spans leave your network. This works as a safety net when a developer forgets a rule. It should never be your only defense, but it is a strong second one.

Layer 4: respect boundaries and regulations

Organizations with strict rules about where data may live and who may reach it can place Google Cloud services inside protected perimeters and apply organization policies. Confirm in current documentation which controls apply to Cloud Trace in your setup. Also be aware that stored traces expire, but data in exports or screenshots may not.

Layer 5: protect the pipeline

Keep the collector and exporters updated, run them with minimal permissions, and avoid sending traces over untrusted networks. Prefer attached platform identities over key files.

A tracing system that quietly stops working is worse than none, because people believe it. Watch the watcher.

Tracing can fail without anyone noticing. A permission change might block exports. A new release might drop the propagation code. A full buffer might silently discard spans. Because traces are mostly used during incidents, the worst time to find out that they are missing is during an incident.

Signs of a healthy pipeline

• Fresh traces appear for each important service every hour, even at a low sample rate.
• A test request that crosses all services forms one trace with all expected services in it.
• Export errors in your application logs are close to zero.
• The count of ingested spans is steady, apart from expected daily and weekly patterns.
• Collectors report low or zero dropped spans.

Good habits

• Run a small synthetic request, a scheduled fake user action, that always produces a trace. If that trace disappears, an alarm rings.
• Alert on sudden drops in ingested spans, and on repeated export failures in logs.
• After every major change to networking, permissions or deployment tools, send one test request and look at the trace.
• Include tracing checks in your release checklist for new services.

A decision path for “no traces appear”

flowchart TD
  S["No traces appear"] --> A{"Is the Cloud Trace API enabled?"}
  A -->|No| A1["Enable the API in the project"]
  A -->|Yes| B{"Can the service account write traces?"}
  B -->|No| B1["Grant the Cloud Trace Agent role"]
  B -->|Yes| C{"Is the app exporting without errors?"}
  C -->|No| C1["Fix exporter setup and read the app logs"]
  C -->|Yes| D{"Is sampling too low or the time range wrong?"}
  D -->|Yes| D1["Raise sampling and widen the time range"]
  D -->|No| E["Check network paths, quotas and the project that receives traces"]

Tracing is inexpensive when planned, and surprisingly costly when ignored. A little arithmetic in advance prevents bad surprises.

What you pay for

Cloud Trace is generally billed by the number of spans ingested each month, with a free allowance and a price per additional amount of spans. Prices and allowances change, so read the current pricing page before you plan. Viewing and querying traces through the console is typically not the main cost driver. Span volume is.

Span volume is easy to estimate. Multiply the requests per second by the number of spans in each trace, then by the fraction you sample, then by the seconds in a month. A service with 500 requests per second and ten spans per request produces 5,000 spans every second if you keep everything, which is a very large monthly number. Keeping one percent produces fifty spans per second, a much smaller number.

How to keep volume sensible

• Lower the sampling rate for very busy, low-risk services, and keep higher rates for critical paths.
• Avoid creating spans inside tight loops. One span for the loop is usually enough.
• Drop noisy spans such as health checks and static file requests at the collector.
• Keep attributes small and few, since oversized values add storage and network cost.
• Use tail sampling to keep errors and slow requests while discarding most normal ones.

Performance overhead

The performance cost inside your app has three parts: the small time to create each span, the memory for the buffer, and the network and processor time for exporting. For most services this is small, but it grows with span count. Test with realistic traffic. If tracing noticeably slows a service, you are probably creating too many spans or sampling too much.

Every tool is a bargain. Look at both sides, and choose with open eyes.

Cloud Trace compared with other choices

Trade-offs to think about

Detail versus cost. More spans reveal more but cost more. Begin with automatic spans plus a few business spans, and add detail only where questions repeat.

Simplicity versus control. A managed service is easy but gives fewer knobs. A self-run tool gives knobs, and also the work of keeping it healthy.

Coverage versus effort. Tracing every service gives complete timelines, but half-traced systems give half-useful traces. Plan the rollout so that each request path is covered end to end before moving on.

Patterns are shortcuts earned by other teams’ experience. Anti-patterns are the traps they fell into.

Six helpful patterns

Three costly anti-patterns

Small habits make traces trustworthy. Most mistakes are cheap to fix once you know them.

Best practices

• Name your services clearly. A trace with ten spans named “app” is almost useless. Use a unique, readable name per service, and include the version.
• Start with one critical journey. Pick checkout, sign-in or search, and trace it completely.
• Use standard names for common attributes. Shared conventions allow shared dashboards and filters.
• Mark errors properly. Set span status to failed and record the error message as an event.
• Set time limits on outgoing calls. A call with no limit can turn a slow dependency into a hung request. Traces will show it, but limits prevent it.
• Flush before exit. In batch jobs and functions, make sure spans are sent before the program ends.
• Test propagation after each infrastructure change. Proxies, gateways and libraries are the usual culprits.
• Review sampling and costs monthly. Systems change, and old settings may no longer fit.

Common mistakes and quick fixes

Both your application and the tracing pipeline can fail. Knowing the common failures keeps you calm when they appear.

A calm response path

Teams use Cloud Trace in a few repeating ways. Find the one that looks like your team.

The wider industry picture

Distributed tracing became common because large internet companies needed it. Twitter shared Zipkin, and Uber shared Jaeger, and both ideas flowed into the OpenTelemetry standard now supported by almost every major observability tool. Large companies such as Spotify, Snap and The Home Depot are well-known Google Cloud customers running systems made of many services. We name them to show the scale at which tracing matters, not to claim how they use Cloud Trace specifically, because public information may not say.

The deeper lesson is about culture. Teams that can see their system make calmer decisions, argue less, and fix problems faster. A trace is a shared picture, and shared pictures end many arguments.

Let us follow a made-up engineer named Meera through one stressful afternoon, using everything above.

Meera is on call for a bookshop’s website. At two in the afternoon, a monitoring alert says that the 99th percentile latency of the checkout route has doubled. The average has barely moved, which is why nobody complained loudly yet. She knows the time window and the service, so she has her first two filters.

She opens Trace Explorer, sets the time range to the last two hours, and filters to the checkout route. The cloud of points shows most requests are fast, but a band of requests has risen sharply since one-thirty. That moment matches a release that went out at the same time. The shape suggests a change, not random bad luck.

Meera opens one slow trace and one fast trace from the same route. In the fast trace, the inventory check takes forty milliseconds. In the slow trace it takes nearly two seconds, and the waterfall shows it called the stock service three times in a row. The service version attribute shows that the slow traces all come from the new release.

She clicks through to the logs linked to the slow span. A warning appears saying that the new code asks for stock one item at a time. The developer who made the change has accidentally created the N plus one pattern. Meera rolls back the release, and within minutes new traces show the old, fast shape again.

The next morning, the team writes a short report that includes the slow trace, the fast trace and a note about the one span that grew. They add a rule to their review checklist: new loops that call other services must show a single span in a test trace. They also add an alert that compares latency by service version, so the next regression is spotted within minutes of release.

Tracing succeeds when a team shares a few simple rules. Without them, every service becomes a different dialect.

Agree on a standard

Decide that all services use OpenTelemetry, which header format they propagate, and how services are named. Publish a short starter template with tracing already configured, so new services begin correct.

Agree on attributes and names

Create a short list of required attributes, such as service version and environment, and a list of forbidden content, such as personal data. Review it once a quarter, and keep it short enough that people remember it.

Assign ownership

Someone must own the collector, the sampling rules and the cost dashboard. Often this is the platform team. Application teams own their own spans and attributes, and they fix gaps in their services.

Make traces part of daily habits

Include a trace link in incident reports and in pull requests that change performance. Teach new engineers to read a waterfall in their first week. Hold a short monthly session where someone shows an interesting trace to the group.

You do not need to learn everything at once. Move from simple and safe to deep and powerful.

Keep a small notebook of every odd waterfall you meet, with a one-line explanation. After a month, you will recognise shapes at a glance, and that quick recognition is the real skill behind good tracing.

Short, honest answers to the questions beginners ask most often.

If you remember only a few ideas from this long guide, make them these.