Start with the big picture, in plain words.

The simple definition

A program is a list of instructions that a computer follows. Those instructions are grouped into named pieces called functions. When your program runs, some functions run for a long time and others finish in a blink. A profile is a report that tells you how much of a resource, such as processor time or memory, each function used. The act of collecting such a report is called profiling.

Cloud Profiler is a Google Cloud service that does continuous profiling. The word continuous matters. Instead of running a special tool once on your laptop, Profiler collects data all the time from your real, live applications and stores it so you can look at it later. It then draws the results as interactive pictures that link each cost back to the exact function in your source code.

Where it sits in Google Cloud

Cloud Profiler belongs to the family of tools Google calls its observability suite, which also contains Cloud Monitoring for numbers over time, Cloud Logging for text messages, Cloud Trace for the path of one request, and Error Reporting for crashes. Each answers a different question. Profiler answers one very specific question: inside my running code, which lines are using the most resources?

Why it exists

Software that works fine on a developer’s laptop often behaves differently in production. Real traffic is larger, messier, and more surprising than any test. A slow function that never showed up in testing can suddenly dominate when a million real users arrive. Profiling on a laptop cannot reproduce that. Profiling in production can. Cloud Profiler lets teams do it safely because its overhead is designed to be low enough to leave on all the time.

A short history

The idea did not start as a cloud product. In 2010 Google engineers published a research paper describing an internal system called Google-Wide Profiling. It sampled programs across Google’s enormous fleets of computers and showed which code burned the most resources. The lesson was powerful: even tiny wins, such as making a common function two percent cheaper, saved huge amounts of computing power when multiplied across thousands of machines. Years later, Google packaged the same idea for its customers as Cloud Profiler, so that any team could do what Google had done internally.

Around the same time, a visual style called the flame graph, invented by the performance engineer Brendan Gregg, became popular across the industry because it shows a profile in one glance. Cloud Profiler uses a flame graph as its main view.

Five small ideas make the rest of this guide easy.

CPU time

The CPU, or processor, is the part of a computer that actually carries out instructions. CPU time is how long the processor was busy working for your function. If a function does lots of math, it consumes lots of CPU time. High CPU time usually means slow responses or high bills.

Memory and the heap

Memory is the short-term workspace where a running program keeps its data. The part of memory used for objects that your program creates and later discards is called the heap. If a program keeps creating objects and never lets go, the heap grows until the machine runs out of space. This problem is called a memory leak.

Allocation

An allocation is a request for a piece of memory. Even if a program never leaks, constantly allocating and discarding many objects forces the system to spend time cleaning up, a job called garbage collection. A function that allocates too much can slow the whole application, even when its own math is quick.

Waiting time

Not all slowness is the processor working hard. Often a program is simply waiting, for a database to answer, for a file to load, or for another thread to release a shared item. Time spent waiting is sometimes called wall time, because it is the time you would measure by looking at a clock on the wall, including all the idle moments.

Latency and throughput

Latency is how long one request takes from start to finish. Throughput is how many requests the system can handle per second. Profiling helps with both: cheaper functions mean quicker answers and room for more requests on the same machines.

Profiling versus monitoring, logging, and tracing

Beginners often wonder why they need yet another tool. The easiest way to see the difference is to ask each tool a question.

The vocabulary of the service, explained one word at a time.

Agent

The profiler agent is a small piece of code that lives inside your application, either added as a library or switched on as an agent. It does the actual collecting. Think of it as the stopwatch-holding observer in the kitchen. It runs beside your own code, wakes up now and then, records what is happening, and sends the notes to Google.

Sampling

Sampling means taking snapshots at intervals instead of recording every moment. If you wanted to know how a crowd spends its day, you would not follow each person constantly. You would take a photograph every few minutes and count what people are doing. Given enough photographs, the picture of the crowd becomes accurate. Sampling is the secret to Cloud Profiler’s low overhead.

Stack trace

When a function calls another function, which calls another, the computer keeps a list of this chain called the call stack. A stack trace is a snapshot of that list at one instant. It reads like: the web handler called the order checker, which called the price calculator, which called the discount lookup. A profile is built from thousands of these snapshots.

Profile type

A profile type says which resource was measured. Examples are CPU time, heap usage, allocated memory, wall time, contention, and threads. Different programming languages support different types, which we cover in a later chapter.

Service and version

In Profiler, a service is the name you give to one logical application, such as “checkout-api”. A version is a label for a specific release, such as “v42” or a build number. Profiler groups data by these two labels. This makes it possible to compare today’s release with yesterday’s and to see whether a change made things better or worse.

Flame graph

A flame graph is a picture made of horizontal bars. Each bar stands for a function. The wider the bar, the larger its share of the measured resource. Bars stacked above or below each other show who called whom. A glance tells you which function is the widest and therefore the most expensive.

Self cost and total cost

Every function has two numbers. Self cost is the resource used by the function’s own lines. Total cost includes the function plus everything it called. If a function has high total cost but low self cost, the real culprit is hiding in one of its helpers. If its self cost is high, the function’s own work is the problem.

Follow the notes from your running code to the screen of your browser.

The system has three simple parts: the agent inside your application, the Cloud Profiler service on Google’s side, and the interface where you view and compare the results. Between them flows a steady trickle of small profile files.

flowchart LR
  A["Your application running in production"] --> B["Profiler agent library inside the app"]
  B --> C["Short sampling window per profile type"]
  C --> D["Profile data with stack traces"]
  D --> E["Cloud Profiler API"]
  E --> F["IAM permission check"]
  F --> G["Profile storage managed by Google"]
  G --> H["Profiler interface with flame graph"]
  H --> I["Engineer compares versions and finds hotspots"]
    

Reading the diagram

Your application running in production. This is any program on Compute Engine, Google Kubernetes Engine, Cloud Run, App Engine, or even a machine outside Google Cloud, as long as it can reach Google over the internet.

Profiler agent library inside the app. The agent starts when your program starts. It registers with Google, announcing the project, the service name, and the version. Then it waits for instructions about when to collect.

Short sampling window per profile type. The service tells each agent when to take a profile. The window is short, roughly a few seconds of measurement per profile, and happens only now and then. Outside these windows the agent costs almost nothing.

Profile data with stack traces. The agent packs its snapshots into a compact file, a format commonly known as pprof, which is an open standard for profile data.

Cloud Profiler API. The agent uploads the file by calling Google’s Profiler API over a secure connection.

IAM permission check. Google verifies that the identity behind the upload is allowed to send profiles to this project. The role designed for that purpose is the agent role.

Profile storage managed by Google. The data is stored and indexed so it can be filtered by service, version, time, and type. Profiles are kept for a limited retention period, so very old data eventually disappears.

Profiler interface with flame graph. When you open the Profiler page in the Google Cloud Console, it fetches the stored data and draws it for you.

Different questions need different measurements.

Cloud Profiler supports several programming languages, and each language runtime exposes different information. The most commonly used languages are Go, Java, Node.js, and Python. Because Google’s lists change as the product evolves, treat the table below as a learning map and confirm the exact current support in the official documentation before you plan a rollout.

The main profile types

Not every language offers every type. For example, CPU time is widely available, while contention and thread profiles are tied to specific runtimes. This is a property of the language runtime, not a limit of the Google service, because Profiler can only report what the runtime is able to measure.

Choosing the right type

Lift the lid and watch the gears turn.

Statistical profiling

Cloud Profiler uses statistical profiling. That means it does not measure each function call exactly. Instead, it checks the program at many random-ish moments and notes which function was running. If a function appears in one hundred of one thousand snapshots, the profile concludes it used about ten percent of the time. With enough snapshots, the estimate is very close to reality, and the effort needed is tiny compared with recording every call.

CPU sampling

For CPU profiles, the runtime asks the operating system to interrupt the program at a steady rhythm, for example many times per second while the profile window is open. Each interruption records the current stack trace. Functions that are running more often are caught more often. A function that never runs is never caught, which is exactly right.

Heap sampling

For memory profiles, runtimes do not record every single allocation. Instead they record a sample of allocations, such as one for every fixed number of bytes created. Large or frequent allocations are therefore more likely to be recorded. The recorded stack trace tells which function asked for the memory. For heap-in-use profiles, the runtime also tracks which of the sampled objects are still alive when the snapshot is taken.

Wall-time sampling

For wall-time profiles, the sampling also records threads that are waiting. This reveals functions that are slow because of time spent blocked, not because of heavy calculation. This is why wall time is the best friend of anyone chasing slow database calls or slow network calls.

How Google schedules the collection

Each agent registers with the Profiler service and then asks, “Is it my turn to profile?” The service spreads the work across all agents of the same service so that, at any moment, only a few instances are actually collecting while the others carry on without any extra effort. Over time, many small profiles from many instances are merged into a single combined view. This spreading is what keeps overhead low for each individual machine, while still producing a rich picture for the whole fleet.

sequenceDiagram
  participant App as "Application instance"
  participant Ag as "Profiler agent"
  participant API as "Cloud Profiler API"
  participant Store as "Profile storage"
  Ag->>API: "Register service name and version"
  loop "Repeated cycle"
    Ag->>API: "Any profile requested for me?"
    API-->>Ag: "Please collect a CPU profile for a short window"
    Ag->>App: "Sample stack traces during window"
    App-->>Ag: "Stack samples"
    Ag->>API: "Upload packed profile"
    API->>Store: "Save, index by service, version, type, time"
  end
    

Symbols and names

A running program works with raw addresses, which are numbers. To show the human-friendly name of a function, the profile must carry a symbol table, which is a dictionary from address to name. For languages like Java, Python, and Node.js, the runtime itself knows the function names. For compiled languages like Go, the binary contains the information. If you strip the symbols from a binary to make it smaller, your flame graph may show unreadable numbers instead of names, so be careful with such build options.

Aggregation

Finally, Profiler aggregates profiles. It adds together the samples from many instances and many time windows for the chosen service and version. This is why the flame graph you see is usually a summary over a chosen period, not the story of one moment. If you want to understand one burst of trouble, narrow the time range.

A calm, ordered path that avoids the usual surprises.

Step one: choose a project and switch on the API

Every Google Cloud resource lives in a project, which is a folder for your cloud things plus billing and permissions. You must turn on the Cloud Profiler API for the project that will receive the profiles. Without it, the agent’s uploads are rejected, and the error message will say the API is not enabled.

Step two: give the application an identity with permission

The agent must prove who it is when uploading. On Google Cloud, applications normally run as a service account, which is an identity for programs rather than people. That account needs the Cloud Profiler Agent role. Following the principle of least privilege, grant only that role, not a broad one such as owner. People who only need to view profiles need a viewing role, which is separate from the agent role.

Step three: add the agent to the application

Each language has its own small package or agent. You add it to the application and call a start function as early as possible, giving it a service name and an optional version. For Java, the usual approach is to attach an agent when the runtime starts. For Go, Node.js, and Python, you include an official library and start it in your code. Because exact package names, flags, and versions change over time, use the official quickstart for your language when you apply this step.

Step four: deploy and generate traffic

A profile only shows what the program does. An idle program produces an empty picture. Send realistic traffic, from a test tool or from real users, so that functions actually run. After a few minutes you should see the first profiles appear.

Step five: open the Profiler page

In the Google Cloud Console, open the Profiler page. Choose the service, the profile type, and the time range. The flame graph appears. If it is empty, jump to the troubleshooting chapter.

flowchart TD
  A["Create or choose a project"] --> B["Enable Cloud Profiler API"]
  B --> C["Create service account for the app"]
  C --> D["Grant Cloud Profiler Agent role"]
  D --> E["Add language agent and start it in the app"]
  E --> F["Deploy to Cloud Run, GKE, Compute Engine, or App Engine"]
  F --> G["Send realistic traffic"]
  G --> H["Open Profiler page and view flame graph"]
    

Where the agent can run

Learn to read the picture in under five minutes.

What each bar means

Every bar in the flame graph represents one function. The width of the bar shows how much of the selected resource that function and everything it called is responsible for. The stacking of bars shows the call chain: a bar sits next to the function that called it. Colors help tell different pieces apart, but the shapes and widths carry the meaning.

How to find the hot spots

Look for the widest bars. A wide bar means a big share. Then look at the bars it contains. If the wide bar is made of many narrow children, the cost is spread thin and no single child is the culprit. If one child takes almost the whole width, you have found a likely hotspot. If a wide bar has almost nothing inside it, the function’s own lines are expensive.

flowchart TD
  A["handleRequest - 100 percent"] --> B["validateInput - 5 percent"]
  A --> C["buildResponse - 90 percent"]
  A --> D["writeLog - 5 percent"]
  C --> E["queryRecommendations - 70 percent"]
  C --> F["formatJson - 20 percent"]
  E --> G["rankItems - 60 percent"]
  E --> H["fetchFromCache - 10 percent"]
    

In the example, a beginner might be tempted to speed up the logging, because logging is easy to understand. But logging is only five percent. The tree makes it plain that ranking items carries sixty percent of the cost. Fix that first, and the whole request becomes faster. This is the main gift of profiling: it protects you from optimizing the wrong thing.

The controls you will use

• Service: choose which application to view.
• Profile type: choose CPU, heap, wall time, and so on.
• Version: narrow to one release, or view all.
• Zone and other filters: narrow to where the code ran when such filters are offered.
• Time range: view the last hour, a day, or a custom window. Wider windows give smoother results, and narrow windows capture specific incidents.
• Weight: some profile types let you choose which weighting to display, for example how to interpret the numbers.
• Focus and highlight: select a function to see only paths that include it, or to see where else it appears.
• Compare: overlay two time ranges or two versions and see which functions grew or shrank.

Compare mode: the before and after tool

Compare mode is the most useful feature for teams that care about regressions. A regression is a change that makes things worse. After a release, you compare “this week” against “last week” for the same profile type. Functions that grew are flagged in a distinct way, which leads you directly to the change that caused the slowdown.

From the first sample to the day the data is deleted.

stateDiagram-v2
  [*] --> Registered: "Agent starts"
  Registered --> Idle: "Waiting for a request"
  Idle --> Collecting: "Service asks for a profile"
  Collecting --> Uploading: "Window ends"
  Uploading --> Idle: "Upload accepted"
  Uploading --> Idle: "Upload failed - retry later"
  Idle --> [*]: "Application stops"
    

What happens when an instance disappears

Modern services scale up and down. A container may live only a few minutes. If an instance is shut down in the middle of a collection window, that one small profile may be lost, but profiles from other instances continue to arrive. Because the final picture is built from many instances, losing a few windows hardly changes it.

Data volume

Each profile file is small, usually a few kilobytes to a few hundred kilobytes, depending on the program. Because collection is spread across the fleet, the total upload stays modest even for large systems.

Four stories that show the tool at work, with no code needed.

Workflow one: the slow API with a hot CPU

Your team notices that an API takes much longer to answer during busy hours, and the CPU graph on the dashboard is high. You open Profiler, pick the CPU profile type, and select the last day. The widest bar leads through a function that sorts a large list again and again for every request. You change the code so the list is sorted once and reused. After the next release, you use compare mode and see the sort function shrink to a thin sliver. Response times drop, and fewer machines are needed during peak hours.

Workflow two: the slowly leaking memory

A service restarts itself every few days because it runs out of memory. Dashboards show memory rising in a steady staircase. In Profiler, you open the heap profile and compare a view from the morning with a view from the evening. One function that stores recent results in a list has grown wider and wider, because nothing ever removes old entries. You add a size limit to that list. The staircase flattens and the restarts stop.

Workflow three: slow requests but low CPU

Users complain about slowness, yet the processor is nearly idle. This is the classic sign of waiting. You switch to a wall-time profile where available. The flame graph shows a wide section inside a function that talks to a database, called many times in a loop for one request. The fix is to ask the database for all the data at once instead of one item at a time. This pattern is known as the N plus one problem, and it is among the most common causes of slow web pages.

Workflow four: too much cleanup work

A service sometimes pauses briefly and then speeds up again, a rhythm that matches garbage collection. The allocated heap profile shows one function creating huge numbers of short-lived temporary objects while building text messages. You change it to reuse a buffer. Allocation drops, pauses shrink, and the long tail of slow requests improves.

flowchart TD
  A["Users report slowness or bill rises"] --> B["Check Monitoring for CPU, memory, latency"]
  B --> C{"What looks abnormal?"}
  C -- "CPU high" --> D["Open CPU profile"]
  C -- "CPU low but latency high" --> E["Open wall-time profile"]
  C -- "Memory rising" --> F["Open heap profile and compare over time"]
  C -- "Frequent pauses" --> G["Open allocated heap profile"]
  D --> H["Find widest function and fix"]
  E --> H
  F --> H
  G --> H
  H --> I["Deploy new version"]
  I --> J["Use compare mode to confirm improvement"]
    

An honest look at what Profiler is great at, and where it is not the right tool.

Profiler versus other approaches

Trace and Profiler work as a pair. A trace says, “this request spent two seconds in the recommendation service”. Profiler then says, “inside that service, most of the effort goes into the ranking function”. Together they take you from a user complaint to a line of code.

What does it cost to watch? Very little, by design.

Why the overhead is low

Three design choices keep the slowdown small. First, sampling takes snapshots instead of recording every event. Second, short windows mean the agent is active for only a few seconds at a time, not continuously. Third, fleet-wide scheduling spreads collection so that only a small fraction of instances profile at any moment. Google’s documentation describes the typical overhead as small, in the range of a few percent or less of CPU, and the memory use of the agent as modest. Exact numbers depend on language and workload, so measure it for your own system during a trial.

How to test the overhead yourself

1. Pick a service with steady traffic, such as one canary instance.
2. Run it for an hour with the agent, then for an hour without it, under similar load.
3. Compare CPU use, memory use, and latency between the two hours.
4. If the difference is within your comfort level, roll out wider.

Scaling to large fleets

Because the work is shared across instances, adding more instances does not multiply the collection cost. In fact, more instances give more samples and a smoother, more reliable picture. Large systems with thousands of containers can profile every service without drowning in data.

Impact on cost

Profiling often pays for itself. A small percentage saved in a hot function, multiplied by the hundreds of machines running that code, can cut the number of machines needed. Companies that run large fleets have long reported that continuous profiling finds savings that no manual review would catch. Always check Google’s current pricing page for the service itself, since pricing terms may change.

Profiles describe your code, so treat them with care.

Two different jobs, two different roles

Google Cloud uses IAM, short for Identity and Access Management, to decide who may do what. For Profiler there are two main jobs. One is sending profiles, which is the job of the application’s service account. The other is viewing profiles, which is the job of engineers. Keep these separate. The application should be able to upload but not to browse, and engineers should be able to browse but not impersonate the application.

What a profile contains

A profile contains function names, file names, line numbers, and counts. It does not normally contain your users’ data, request bodies, or passwords. That is one reason profiling is considered low risk. However, function and file names can reveal how your code is organized, and sometimes names include sensitive words such as a customer or a project code name. Treat profile access like source code access and give it to people who are already allowed to see the code.

Authentication for the agent

Inside Google Cloud, the best way is an attached service account, with no key file anywhere. On Kubernetes, use Workload Identity so each pod receives short-lived credentials. Outside Google Cloud, prefer workload identity federation, which swaps your existing proof of identity for a short-lived Google token. Avoid long-lived key files, because anyone who copies a key can pretend to be that service account until it is deleted.

Network and encryption

Uploads use encrypted HTTPS connections. The agent only needs outbound access to Google’s API address. If your company limits outbound traffic, allow the Profiler API address, or use Private Google Access so that instances without public addresses can still reach Google APIs. In organizations that use VPC Service Controls, which draw a protective boundary around Google services, make sure the Profiler API is included correctly in the perimeter so that agents inside it can upload.

Auditing

Administrative changes, such as granting a role or enabling the API, are recorded in audit logs. Review these records when you investigate who gained access to profile data and when.

When the flame graph is empty or looks wrong, walk this checklist.

Common symptoms and likely causes

Watching the health of profiling

Because profiling is an observability tool, it deserves a little observability of its own. Review your application logs for agent warnings, check that new versions still show up in Profiler after each deployment, and add a simple team habit of looking at the profile after major releases. If profiles stop arriving, treat it like any other monitoring gap.

Profiler is strongest when it talks to its neighbors.

With Cloud Monitoring

Monitoring tells you when something is wrong. Profiler tells you why. A neat team habit is to link the dashboard for each service to its Profiler page, so that anyone who sees a CPU spike can jump straight to the flame graph for the same time range.

With Cloud Trace

Trace finds the slow service in a chain of services. Profiler then explains the slowness inside that service. Together they take you from “the checkout page is slow” to “this exact function in the pricing service is the cause”.

With Cloud Logging and Error Reporting

Logs describe events, and errors show crashes. If a log line reveals that something is retrying over and over, Profiler may show the retry loop eating CPU. If out-of-memory crashes appear in Error Reporting, heap profiles show which function holds the memory.

With deployment pipelines

If you include the version label in every deploy, Profiler can compare releases. Some teams add a review step after each release: open compare mode between the new and previous versions, and look for newly wide bars. This turns performance into a routine check instead of a rare emergency.

With budgets and cost reviews

When the finance team asks why compute costs rose, a flame graph can show which function grew between months. This changes the conversation from “buy fewer machines” to “make this function cheaper”.

flowchart LR
  A["Cloud Monitoring alert: latency high"] --> B["Cloud Trace: slow service identified"]
  B --> C["Cloud Profiler: hot function identified"]
  C --> D["Developer fixes code"]
  D --> E["Pipeline deploys new version"]
  E --> F["Profiler compare mode confirms improvement"]
  F --> G["Monitoring shows latency back to normal"]
    

Habits that help, and habits that hurt.

Helpful patterns

Anti-patterns to avoid

A short, practical rulebook you can pin to the wall.

Best practices

• Start the agent early and keep it on. Continuous data is the whole point.
• Label versions clearly. Use a build number or a short commit identifier so releases can be compared.
• Profile under realistic load. An idle service tells you nothing.
• Pick the right profile type. Match the type to the symptom, as in the chapter on types.
• Change one thing at a time. Then measure again.
• Check the overhead on a canary. Verify it on your own workload before wide rollout.
• Keep build symbols. Names make flame graphs readable.
• Give least-privilege access. Separate the agent role from the viewing role.
• Review profiles after each major release. Make it a routine, like reading test results.
• Write down what you learned. Notes survive longer than profile data.

Common mistakes beginners make

How different teams put continuous profiling to work.

flowchart TD
  A["Agent baked into base image"] --> B["Every service sends profiles"]
  B --> C["Weekly review of widest bars"]
  C --> D["Fixes tracked as small tickets"]
  D --> E["New version deployed"]
  E --> F["Compare mode confirms savings"]
  F --> G["Capacity plan and budget updated"]
  G --> C
    

Notice what the stories share. Profiling is not a one-time rescue operation. It becomes a quiet routine that keeps software lean, keeps incident counts low, and keeps cloud bills under control. The tool is simple. The habit is what creates the value.

Quick answers to the questions beginners ask most.

It is a Google Cloud service that continuously samples your running applications and shows which functions use the most CPU, memory, or waiting time, drawn as flame graphs.

It is designed to add only a small overhead by using sampling, short windows, and fleet-wide scheduling. Google’s documentation describes the overhead as low, but you should measure it on your own workload with a canary before wide rollout.

The commonly supported languages are Go, Java, Node.js, and Python. Support can change over time and varies by profile type, so confirm the current list in the official documentation.

No. It works best there, but an application running elsewhere can also send profiles if it has network access to Google and valid credentials, preferably through federation.

Trace follows one request across services and shows which step is slow. Profiler looks inside a service and shows which functions use the most resources. They work well together.

The usual causes are a disabled API, a missing agent role, the wrong project, an agent that did not start, blocked network access, or an idle application. Follow the troubleshooting checklist in order.

It means that function, including everything it calls, accounts for a large share of the measured resource. Start your investigation with the widest bars.

Self cost counts only the function’s own lines. Total cost counts the function plus everything it calls. High total cost with low self cost means the real problem is in a helper below it.

Start with CPU time for slow, busy services. Use wall time when users wait but the processor is idle. Use heap profiles for growing memory and allocated heap for heavy cleanup work.

Profiles contain function names, file names, line numbers, and counts, not request contents. Still, treat access like access to source code, because names can reveal how your system is built.

Profiles are kept for a limited retention period and then removed automatically. Check the current documentation for the exact window and save important findings in your own tickets.

Pricing terms can change, so check Google’s pricing page for the current answer. Also remember that the machines and traffic your application uses are billed in the usual way.

No. Keep a stable service name and use the version field for the release. This keeps history together and lets compare mode work well.

Use compare mode to place the version before the fix next to the version after it, with similar traffic. The target function should be visibly narrower.

No. Load testing creates realistic pressure. Profiling shows where the pressure goes. Use them together for the best results.

Everything important, in one place.