strace & perf with Containers in Docker
How two of Linux's most powerful, decades-old diagnostic tools reach past a container's illusion of isolation to show you exactly which system call hung, or exactly which line of code is quietly burning your CPU.
Imagine a car mechanic who is only ever allowed to look at the dashboard — never allowed to pop the hood. They can see the speedometer, the fuel gauge, the check-engine light, but if the car makes a strange noise nobody can quite place, there’s no way to actually watch the engine while it runs. For a long time, that’s roughly how debugging a misbehaving container felt: application logs and basic resource metrics were the dashboard, but the actual moment-by-moment behavior of the process — which files it was trying to open, which system calls were blocking, which function was eating all the CPU — stayed hidden under a hood nobody could open. strace and perf are the tools that let you pop that hood, even though the “engine” in question is running inside a container. This article explains exactly how that’s possible, given that containers are supposed to be isolated from the host in the first place, and how to use these tools thoughtfully once you have access to them.
1Core Concepts
Before touching either tool, it helps to understand exactly what each one watches, and why that’s possible at all inside a container.
What strace actually does
Every time a running program needs something from the operating system — opening a file, reading from the network, allocating memory, creating another process — it doesn’t do that work itself. It asks the Linux kernel to do it, through a request called a system call. strace is a tool that intercepts and prints every single one of these requests a process makes, along with its arguments and its result, giving you a live, honest transcript of everything that program asked the operating system to do, and exactly how the operating system answered.
Think of a customer at a restaurant who never talks to the kitchen directly — every request goes through a waiter. strace is like standing next to that waiter with a notepad, writing down every single order that gets passed to the kitchen and every plate that comes back, even though you can’t see into the kitchen itself. That transcript alone often tells you exactly where something is going wrong — a request that never gets a response, an order that keeps getting sent back, or one plate taking dramatically longer than every other.
What perf actually does
Where strace focuses on system calls specifically, perf is a broader profiling tool that can sample what the CPU is actually doing, at very high frequency, often using hardware performance counters built directly into the processor. Instead of a transcript of kernel requests, perf gives you a statistical picture: “over the last ten seconds, this function was running 40% of the time, that one 25%, this other one barely at all” — exactly the kind of insight you need to answer “why is this container using so much CPU,” which strace, focused purely on system calls, generally can’t answer well on its own.
strace
Traces system calls: what a process asked the kernel to do, with what arguments, and what came back. Best for hangs, errors, and unexpected file/network access.
perf
Samples CPU activity, often via hardware counters, to show where processing time is actually being spent. Best for “why is this slow” and “why is this using so much CPU.”
Why this is even possible with containers at all
Here’s the concept that unlocks everything else in this article: a container is not a separate, independent operating system kernel — it is a normal Linux process (or group of processes), given the *illusion* of isolation through kernel features called namespaces and control groups, but running on top of the exact same shared kernel as everything else on that host. Because strace and perf are tools that observe things at the kernel’s own level — system calls, CPU scheduling, hardware counters — the same single kernel underlying a container is perfectly capable of exposing that container’s activity to a tool running with the right permissions, container boundary or not.
A container’s isolation is mostly about what a process *sees* — its own filesystem, its own process list, its own network interface. It says very little about what the *kernel itself* can see, and the kernel sees everything, on every container, all the time. strace and perf are simply asking the kernel to share some of what it already knows.
2Architecture & Components
Let’s look at exactly which kernel and Docker mechanisms make this observability possible, and what stands in the way by default.
Why containers are locked down by default
By default, Docker containers run with a restricted set of Linux capabilities — fine-grained permissions that split up what used to be an all-or-nothing “root” concept into dozens of individual powers. One specific capability, related to the underlying ptrace system call, governs whether a process is allowed to attach to and observe another process’s execution — exactly what strace needs to do its job. Docker deliberately omits this capability from containers by default, as a security precaution, since the ability to trace another process is also, in the wrong hands, a tool for interfering with or extracting secrets from that process.
SYS_PTRACE capability
The specific Linux capability required for one process to trace another’s system calls — needed by strace, absent from containers by default.
Seccomp profile
Docker’s default seccomp filter also restricts which system calls a container can make at all, including some that tracing tools themselves rely on.
The shared host kernel
The single kernel instance underlying every container on the host, which is what actually holds all the process, scheduling, and system-call information these tools read.
perf_event_paravirt / hardware counters
The CPU’s own built-in performance-measuring registers that perf reads from, accessed through a kernel subsystem that also has its own permission requirements inside containers.
Two different ways to get access: from inside, or from outside
There are two broad architectural approaches to using these tools with a container, and understanding the difference matters for choosing the right one for a given situation.
flowchart LR
subgraph Approach1["Approach A: Trace from inside"]
A1["Container started with extra capability"] --> A2["strace/perf installed and run inside the container"]
end
subgraph Approach2["Approach B: Trace from the host"]
B1["Host has strace/perf installed"] --> B2["Attach to the container's process ID as seen from the host"]
end
style Approach1 fill:#111111,stroke:#dc2626,color:#ffffff
style Approach2 fill:#111111,stroke:#dc2626,color:#ffffff
Fig. 1 — You can either grant a container the permissions to trace itself, or use the host’s own view of the same shared kernel to trace it from outside.
Approach A requires deliberately relaxing a container’s default restrictions (typically by explicitly granting the relevant capability when starting it) and installing the diagnostic tool inside the container’s own image or a companion debugging container. Approach B takes advantage of the fact that, from the host’s perspective, a container’s processes are just ordinary host processes with ordinary, host-visible process IDs — meaning a sufficiently privileged process on the host can trace them directly, without ever needing to modify the container itself.
A container’s process ID as seen *inside* the container (often something small, like PID 1) is not the same number as its process ID *as seen from the host* (often some much larger, host-wide number). Tools run from the host need the host-side PID, not the one the containerized application itself reports.
3Internal Working
Let’s walk through, mechanically, what happens when each tool is actually used against a containerized process.
How strace attaches to a process, step by step
Underneath its friendly command-line interface, strace relies on the kernel’s own ptrace mechanism, which allows one process (the tracer) to intercept and inspect the system calls of another (the tracee). When you run strace against a target process, the kernel begins pausing that process momentarily at each system call boundary, handing control briefly to strace so it can record the call’s name, arguments, and eventual return value, before letting the traced process continue exactly as it otherwise would.
It’s similar to a security guard standing at a single doorway that every person must pass through to enter a building. The guard doesn’t stop anyone from entering, but writes down everyone’s name and the time they passed by. The building’s normal business continues completely unaffected, but now there’s a full, accurate log of who went through that one crucial doorway, in order.
Why this still works across a container’s boundary
Because a container’s processes are, from the kernel’s perspective, still just ordinary Linux processes with real, host-level process IDs and real entries in the kernel’s own process and scheduling structures, the ptrace mechanism doesn’t inherently care that a target process happens to be running inside a container’s namespaces. What actually gates whether tracing succeeds is a combination of permission checks — namely, whether the tracer has the necessary capability and privilege level relative to the target — not whether a container boundary happens to exist between them.
How perf samples a running system
perf typically works by configuring the CPU’s hardware performance counters (or, when unavailable, software-based equivalents) to periodically interrupt whatever is currently running and record what was executing at that exact instant — which process, and often which specific function inside it. Repeated thousands of times per second, these samples build up into a statistically meaningful picture of where CPU time is actually going, even though no single sample, by itself, tells the whole story.
TYPICAL FOR CPU PROFILING
ACROSS ALL CONTAINERS
(INSIDE VS. FROM HOST)
Why perf’s view naturally spans container boundaries too
Because the kernel’s scheduler is the single component deciding which process runs on which CPU core at any given moment, across every container on the host, perf’s hardware-counter sampling is inherently a host-wide, kernel-level activity. Depending on the exact command used, perf can be filtered down to just the process IDs belonging to one specific container, so you see only that workload’s profile, while the actual sampling mechanism itself is, structurally, working at the shared, host level the entire time.
Turning raw samples into a readable picture
Raw perf output on its own is a large volume of individual samples, which is why it’s almost always processed afterward into a more digestible form — most famously, a “flame graph,” a visualization where the width of each bar represents how much sampled time was spent in that function, making it immediately obvious, at a glance, which parts of a program are consuming the most CPU.
4Data Flow & Lifecycle
Let’s trace a full, realistic diagnostic session from the first symptom to a resolved understanding of the problem.
A symptom appears
A containerized service is unexpectedly slow, hung, or consuming far more CPU than expected, and ordinary application logs don’t explain why.
Choosing the right tool
A hang, an unexpected file or network access pattern, or a mysterious error points toward strace; sustained high CPU usage with no clear cause points toward perf.
Identifying the right process
The relevant container’s process is located, and its host-visible process ID (as opposed to its in-container PID) is identified for tools run from the host.
Attaching the diagnostic tool
strace or perf is run against that process, either from a privileged host session or from inside a container explicitly granted the needed capability.
Capturing a representative window
Data is gathered for a period genuinely representative of the problem — capturing a hang as it happens, or a CPU spike while it’s actively occurring.
Analyzing the output
A raw system call transcript is read for stuck or repeatedly failing calls; raw CPU samples are turned into a flame graph or similar summary view.
Root cause identified and fixed
The specific system call, code path, or resource contention responsible is pinpointed, and a genuine fix — rather than a guess — is applied.
Why capturing the right moment matters so much
Both tools are only as useful as the window of time they actually observe. Attaching strace after a hang has already resolved itself tells you nothing about what caused it; sampling perf during a quiet period when CPU usage happens to be normal will show you a flame graph of completely unremarkable activity. Timing the observation to overlap with the actual problem — sometimes requiring a slightly longer capture window to reliably catch an intermittent issue — is often the single most important factor in getting a useful result from either tool.
For intermittent issues, it’s often more effective to start tracing proactively and let it run continuously (accepting some overhead) until the problem reoccurs, rather than trying to react and attach a tool at the exact moment symptoms appear.
5Advantages, Disadvantages & Trade-offs
These are powerful tools, but using them against production containers is a deliberate trade, not a free action.
Advantages
- Reveals ground-truth behavior — exactly what a process actually did, or where its CPU time actually went — rather than relying on inference from indirect symptoms.
- Works on essentially any containerized application, regardless of programming language, since it observes at the kernel and CPU level rather than needing language-specific instrumentation.
- Requires no changes to the application’s own code to gain deep, precise visibility.
- Extremely effective at resolving genuinely mysterious problems — hangs, unexpected file access, CPU spikes with no obvious cause — that other, higher-level monitoring often can’t explain.
- Both tools are mature, stable, and extremely well-documented, having been core parts of the Linux toolkit for a very long time.
Disadvantages & Trade-offs
- strace, in particular, can add meaningful overhead to a traced process, since it pauses execution briefly at every single system call — potentially masking or altering timing-sensitive behavior.
- Both tools generally require elevated privileges or explicitly relaxed container restrictions, which is itself a deliberate security trade-off, especially in production.
- Output can be extremely verbose (strace) or require post-processing tooling to be genuinely readable (perf), adding a learning curve before either tool becomes quickly useful.
- Using them meaningfully requires some familiarity with how the underlying operating system and hardware actually work, more so than many higher-level, application-focused observability tools.
- Attaching directly to a live production process, if not done carefully, could theoretically disrupt a sensitive, latency-critical workload.
When reaching for these tools is clearly worth it
Any situation where standard application logs, metrics, and higher-level tracing have already been exhausted, and a problem remains genuinely mysterious — an unexplained hang, an intermittent CPU spike, a process that seems to be doing something nobody in the application code can account for — is exactly the situation these tools were built for. Their overhead and complexity are a reasonable price to pay for actually solving a problem that would otherwise remain a guess.
When lighter-weight tools should come first
For everyday, ongoing monitoring, dedicated application performance monitoring or standard resource metrics are almost always the better starting point — they’re lower overhead, easier to interpret at a glance, and designed for continuous use. strace and perf shine as targeted, deep-dive tools for specific, hard problems, not as a replacement for routine observability.
Rule of Thumb
Reach for strace when you need to know *what a process is asking the kernel to do*; reach for perf when you need to know *where its CPU time is actually going*. If you’re not sure which question you’re asking, that uncertainty itself is worth resolving first.
6Security
Granting tracing capabilities is a genuine security decision, not a purely operational one — let’s look at exactly what’s being traded.
Why Docker restricts this by default
The capability that allows tracing one process from another is powerful precisely because tracing isn’t a read-only activity in the strictest sense — a sufficiently privileged tracer can, in principle, inspect sensitive data a traced process handles (including things like credentials passed as command-line arguments or read from files) and, in some configurations, even influence the traced process’s execution. This is exactly why Docker omits this capability from containers by default: an attacker who compromises one container shouldn’t automatically gain the ability to trace, and potentially extract secrets from, every other process on that container’s ability to see.
Secrets exposure via tracing
A traced process’s system calls can reveal sensitive arguments or data it handles, which is why tracing capability should never be granted casually or broadly.
Expanded attack surface
Granting the relevant capability to a container widens what a compromised process inside it could potentially do, beyond the container’s originally intended boundary.
Cross-container visibility
Tracing tools run from a sufficiently privileged host context can, depending on configuration, potentially observe processes belonging to other containers too, not just the one intended.
Scoping access as narrowly as possible
The generally preferred approach is to avoid granting tracing capabilities to production containers as a standing configuration, and instead use privileged, carefully audited host-level access for one-off diagnostic sessions, granted only for the duration of the investigation, by operators with an appropriate level of trust. This keeps the powerful capability available when genuinely needed, without leaving it as a permanent, broad expansion of every container’s normal attack surface.
The Mistake
Baking the tracing capability permanently into a production container’s standard configuration, “just in case it’s needed for debugging later.”
Why It’s Risky
This permanently widens that container’s effective privilege and potential blast radius if compromised, for the sake of a diagnostic need that arises only occasionally, if ever, in normal operation.
The Better Approach
Keep production containers at their default, restricted capability set, and use a deliberate, time-boxed, audited process to grant elevated access only when an actual investigation requires it.
Interviewer’s Corner
A good system design or platform-engineering interview question: “Why does Docker restrict process tracing by default, and how would you still support legitimate production debugging needs without permanently weakening every container’s security posture?” A strong answer recognizes the fundamental tension between observability and isolation, and proposes a solution built around narrowly scoped, time-limited, audited elevated access for genuine investigations — rather than either permanently disabling the restriction, or permanently forbidding any deep debugging at all.
7Monitoring, Logging & Metrics
Because these tools are themselves diagnostic instruments, “monitoring” them mostly means using them well and keeping a record of when and why they were used.
| Signal | What it tells you | Where to look |
|---|---|---|
| strace system call transcript | Reveals exactly which calls are slow, failing, repeating unexpectedly, or simply never returning | strace’s own output stream during a capture session |
| perf sample summary / flame graph | Shows which functions or code paths are consuming the majority of observed CPU time | Post-processed perf report output |
| Container capability configuration | Confirms whether a given container currently has tracing-related capabilities granted, intentionally or otherwise | The container’s runtime configuration, inspected via Docker’s own tooling |
| Access and audit logs for elevated sessions | Records who ran a tracing tool against which container, and when, for accountability | Host-level session logging and access-control audit trails |
Treating diagnostic sessions as auditable events
Because tracing tools require elevated access, and because that access is powerful enough to warrant real scrutiny, mature operational practice treats every such diagnostic session as an event worth recording — who initiated it, against which container, for how long, and ideally why. This isn’t about bureaucracy for its own sake; it’s the same principle that applies to any other use of elevated, sensitive access to a production system.
It’s similar to a hospital’s policy around who can access a patient’s full medical record versus a routine visit summary. Full access is sometimes genuinely necessary for proper care, but every instance of using it is logged, precisely because the power to see everything comes with a responsibility to account for when and why it was used.
Practical operational checklist
- Log every elevated diagnostic session, including which container, which tool, and the operator who initiated it.
- Prefer time-boxed, revocable access grants over standing, permanent capability configuration for tracing.
- Review flame graphs and strace transcripts for evidence of genuinely sensitive data before sharing them outside the immediate investigation team.
- Periodically audit which, if any, production containers currently have tracing-related capabilities enabled, and question any found unexpectedly.
8Design Patterns & Anti-patterns
How experienced platform and SRE teams build reliable, safe workflows around these powerful tools.
Dedicated, ephemeral debug sessions
Use a purpose-built, temporary debugging approach — such as attaching a specially privileged, short-lived diagnostic container or session to the host, used and torn down for one investigation only.
Sample first, trace second
Reach for lightweight, host-level CPU sampling as a first pass to identify a likely culprit, then bring in more detailed, heavier tracing only once the investigation is meaningfully narrowed.
Reproduce in staging when possible
Whenever a problem can be reliably reproduced outside production, prefer running deep tracing there, where elevated access and its overhead are far less consequential.
Permanent elevated debug capabilities in production
Leaving tracing capabilities permanently enabled across a production fleet “for convenience” trades a large, standing security cost for a benefit that’s only ever needed occasionally.
The “trace everything, understand nothing” anti-pattern
A common trap for newcomers to these tools is capturing enormous volumes of raw strace or perf output without a specific hypothesis in mind, then feeling overwhelmed trying to find the needle in that haystack. Forming even a rough hypothesis first — “I suspect this is stuck on a network call” or “I suspect this loop is the hot path” — and using that to scope both the capture and the analysis dramatically improves how quickly these tools actually lead to an answer.
The Pattern
Maintain a documented, pre-approved runbook for how to safely attach these tools to a production container — including who can authorize it, how access is granted and revoked, and how findings should be handled — rather than improvising the process during an active incident.
Why It Works
It turns a potentially risky, ad hoc use of powerful access into a well-understood, low-friction, pre-vetted procedure, so teams aren’t forced to choose between “debug slowly and safely” and “debug quickly but recklessly” during a real incident.
9Best Practices & Common Mistakes
Concrete, field-tested guidance for using these tools well against real containerized workloads.
Best Practices
- Identify the correct host-visible process ID before attaching any tool from outside a container, to avoid tracing the wrong process entirely.
- Scope strace captures narrowly (specific system calls, specific processes, limited duration) rather than tracing everything indiscriminately.
- Convert raw perf samples into a flame graph or similar visualization before trying to interpret results manually.
- Treat any elevated access granted for tracing as strictly temporary, and revoke it immediately once the investigation concludes.
- Document findings from each diagnostic session, so the same investigative work doesn’t need repeating for a recurring issue.
Common Mistakes
- Confusing a container’s internal process ID with its host-visible process ID, leading to tracing the wrong target entirely.
- Running an unscoped, full strace capture against a high-throughput production process and being surprised by the resulting overhead.
- Granting broad tracing capability to a container permanently, rather than for the specific, time-boxed investigation that actually needed it.
- Skipping the step of forming a hypothesis first, and drowning in undifferentiated raw output as a result.
- Forgetting that captured output can contain sensitive data, and sharing it more broadly than the investigation genuinely requires.
Before attaching either tool to a genuinely critical production process, rehearse the exact same investigative steps against a similar, non-critical workload first, so you’re confident in the commands and the expected overhead before it matters.
10Real-world & Industry Examples
Where this kind of deep, kernel-level diagnostic work shows up in real operational practice across the industry.
Large-scale infrastructure and platform companies
Companies operating vast fleets of containerized services, including Netflix, have publicly described building sophisticated internal performance-analysis practices around kernel-level tooling like perf and flame graphs, precisely because application-level metrics alone couldn’t always explain subtle, systemic performance issues at their scale.
Site reliability and platform engineering teams
Teams responsible for keeping large, complex containerized systems running reliably commonly maintain internal runbooks and tooling specifically built around safe, controlled use of strace and perf for incident response, reflecting how central these decades-old tools remain even in thoroughly modern, container-based infrastructure.
Kernel and container runtime developers themselves
Engineers building and maintaining the container runtimes and kernel features this entire article depends on rely on exactly these same tools, at an even deeper level, to understand and optimize the isolation mechanisms — namespaces, cgroups, seccomp — that make containers work in the first place.
BEHIND STRACE AND PERF
CROSS-CONTAINER TRACING POSSIBLE
NEEDED TO USE EITHER TOOL
The pattern across every one of these examples is the same: no matter how many layers of abstraction — containers, orchestration, service meshes — sit on top of a running system, the actual truth of what’s happening still lives at the kernel and hardware level, and the tools capable of reading that truth directly remain some of the most valuable, if occasionally underused, instruments in a serious engineer’s toolkit.
11Frequently Asked Questions
12Summary and Key Takeaways
What to remember
- strace traces every system call a process makes — its arguments and results — making it ideal for diagnosing hangs, errors, and unexpected file or network activity.
- perf samples CPU activity at high frequency, often via hardware counters, revealing exactly where processing time is actually being spent — ideal for understanding unexplained CPU usage.
- Both tools work against containers because a container is, to the kernel, just an ordinary process running on the one shared host kernel — isolation hides things from other processes, not from the kernel itself.
- Docker restricts the relevant tracing capabilities by default, as a security measure, since tracing is powerful enough to potentially expose sensitive data or widen a compromised container’s reach.
- Access can come from inside a container (with explicitly granted capabilities) or from a privileged host session targeting the container’s real, host-visible process ID.
- Both tools carry real overhead and complexity, and should be used as targeted, deliberate, time-boxed diagnostic instruments — not as standing, everyday monitoring.
- Mature teams treat elevated tracing access as an auditable, narrowly scoped event, governed by a clear runbook, rather than an ad hoc or permanently granted capability.