Control Groups (cgroups) in Docker — The Complete Deep Dive
How the Linux kernel feature that quietly polices every container's CPU, memory, disk, and process count actually works underneath Docker — explained from first principles, with no assumed background.
Imagine an apartment building where every tenant shares the same water tank, the same electrical panel, and the same elevator. If one tenant decides to run every tap in their flat at once, or leaves every light on for a month, everyone else’s water pressure drops and the electricity bill spikes for the whole building. A well-run building needs a caretaker who can say, “You get this much water, this much electricity, this much elevator time — no more.” On a Linux server running Docker, dozens or hundreds of containers share the exact same CPU cores, the exact same RAM chips, and the exact same disks. Control groups, almost always shortened to cgroups, are that caretaker. They are a Linux kernel feature that measures and limits how much of the machine’s shared resources each group of processes is allowed to consume, and Docker leans on them for nearly every promise it makes about isolating one container from another.
WChapter 1 — What Are cgroups?
Before touching Docker at all, it helps to understand cgroups purely as a Linux kernel feature, because Docker did not invent them — it simply automates them.
Cgroups (Control Groups) are a feature built directly into the Linux kernel that let you take a collection of processes, put them into a named “group,” and then attach rules to that group about how much CPU time, memory, disk bandwidth, network bandwidth, or number of processes it is allowed to use. The kernel itself enforces these rules at the lowest level, which is what makes cgroups so reliable — a misbehaving process cannot simply ignore the limit, because the enforcement happens inside the operating system’s own scheduler and memory manager, not in some external watchdog script that could be bypassed or could crash.
The name itself is a clue to what problem cgroups solve: a group of processes that is put under control. Before cgroups existed, Linux could isolate what a process could see (through mechanisms like chroot) fairly well, but it had almost no built-in way to say “these forty processes, taken together, may use at most 512 megabytes of memory and 20% of one CPU core.” Every process competed for resources on an equal footing, and a single runaway process — a memory leak, an infinite loop, a fork bomb — could starve every other program on the machine, including critical system services.
Think of a school cafeteria during lunch. Without any rules, the biggest, fastest students could pile their trays with every dish available, leaving nothing for the smaller kids who queue politely. A cgroup is like a lunch lady standing at the serving line with a rule book: “Table 4 gets at most two servings of rice and one serving of chicken, no matter how loudly anyone shouts.” She does not care how the food is used within Table 4 — students at that table can swap dishes among themselves — but the table as a whole cannot exceed its allotment. That is precisely how a cgroup treats a container: it does not care what happens between the processes inside, only what the group consumes as a whole from the shared kitchen.
Formally, a cgroup does three things. First, it groups processes — you can add any number of process IDs to a cgroup, and any child processes they spawn automatically inherit membership in that same group. Second, it accounts for resource usage — the kernel keeps running counters of exactly how much CPU time, memory, and I/O each group has consumed, which is the same data Docker reads when you run a statistics command. Third, it limits and prioritizes — you can set hard ceilings that a group is never allowed to cross, soft targets that the scheduler tries to respect under contention, or relative weights that decide who wins when the machine is busy.
Process Collections
Every process in a cgroup, and every child it spawns, is tracked as one unit for resource purposes.
Usage Metering
The kernel maintains live counters of CPU cycles, memory pages, and I/O operations per group.
Hard & Soft Ceilings
Administrators set maximum usage or relative priority, and the kernel enforces it directly.
One of Two Pillars
Cgroups control “how much” a container can use; namespaces control “what” it can see. Together they make a container.
It is worth being precise about a common point of confusion: cgroups are not what makes a container’s filesystem, hostname, or network stack look separate from the host. That job belongs to a different kernel feature called namespaces. Cgroups and namespaces are frequently mentioned in the same breath because Docker uses both together, but they solve two completely different problems. Namespaces answer the question “what can this process see?” while cgroups answer the question “how much can this process use?” A container without namespaces but with cgroups would see the entire host filesystem and process list, but would still be capped on CPU and memory. A container without cgroups but with namespaces would have its own private view of the world, but could still consume every last byte of RAM on the machine. Docker needs both to deliver what people commonly call “containerization.”
WChapter 2 — Why cgroups Exist: The History and Motivation
Cgroups were not built for Docker — Docker did not exist yet when the feature was created. Engineers at Google, most notably Paul Menage and Rohit Seth, began the work around 2006 under the internal name “process containers,” aiming to solve a very concrete problem inside Google’s own data centers: thousands of unrelated jobs, batch processing tasks, and web-serving processes were all being packed onto the same physical machines to save cost, and Google needed a way to guarantee that a low-priority batch job could never starve a high-priority, revenue-generating web server sharing the same box. The feature was merged into the mainline Linux kernel in 2008 under the renamed label “control groups,” and it has been a core, actively maintained part of the kernel ever since.
The motivation traces back to a much older tension in computing: the gap between wanting to run many independent workloads on one machine for cost efficiency, and needing those workloads to not interfere with one another for reliability. Virtual machines had already solved this problem by giving each workload an entire simulated computer, complete with its own virtual CPU, virtual memory, and virtual disk — but that came at a heavy cost. Every virtual machine needed its own full operating system kernel booted up, its own device drivers, and megabytes to gigabytes of memory overhead just to exist before running a single line of useful application code. Google, running at a scale of hundreds of thousands of machines, could not afford to waste that much overhead multiplied by every workload.
Cgroups offered a cheaper answer: instead of giving every workload its own kernel, give every workload a fenced-off share of the one kernel and the one set of physical resources that are already running. A process inside a cgroup is a completely ordinary Linux process — same kernel, same scheduler, same page cache — it is simply metered and capped. This is why containers start in milliseconds while virtual machines take tens of seconds: a container is not booting a new operating system, it is just asking the already-running kernel to spawn a new, accounted-for group of processes.
Whenever someone asks “why are containers faster to start than VMs,” the honest one-line answer is: containers reuse the host’s already-booted kernel and rely on cgroups plus namespaces for isolation, while VMs boot an entirely separate kernel from scratch.
By the time projects like LXC (Linux Containers) and later Docker arrived, cgroups had already matured inside the kernel for several years. Docker’s real innovation in 2013 was not inventing new isolation primitives — it was wrapping the existing, somewhat fiddly kernel interfaces for cgroups and namespaces inside a simple command-line tool and a portable image format, so that a developer typing one command could get the isolation that previously required hand-editing kernel filesystem entries.
CChapter 3 — cgroups v1 versus cgroups v2
Anyone reading about cgroups today will run into two versions, and the difference matters because it changes which files you would look at on a server, and how Docker itself is configured. cgroups v1 is the original design from 2008. cgroups v2 is a full rewrite, merged into the kernel in 2016 and gradually adopted as the default by major Linux distributions and by Docker itself over the years that followed.
The single biggest architectural difference is how the two versions organize their controllers — the individual modules responsible for a specific resource, such as the CPU controller or the memory controller. In cgroups v1, each controller got its own completely separate hierarchy of groups. This meant a process’s CPU limit lived in one tree of directories, its memory limit lived in a totally different, unrelated tree of directories, and its I/O limit lived in yet another tree. Nothing forced these trees to mirror each other, so a sufficiently complex system could end up with CPU groups and memory groups that did not line up cleanly, and tools that wanted a holistic view of “this container” had to stitch together data from several unrelated trees.
cgroups v2 fixed this by using a single, unified hierarchy. There is exactly one tree of cgroup directories, and every controller you enable — CPU, memory, I/O, PIDs — applies within that same tree. A container’s cgroup is one folder, and every resource rule for that container lives inside that one folder, alongside the live usage statistics for every resource at once. This single-hierarchy design also closed several long-standing correctness problems in v1, particularly around how memory accounting interacted with the page cache, and how tricky it was to safely move a running process between different resource trees without briefly escaping enforcement.
| Aspect | cgroups v1 | cgroups v2 |
|---|---|---|
| Hierarchy structure | One separate tree per controller | Single unified tree for all controllers |
| Released / merged | 2008 | 2016 (matured through later kernel releases) |
| Memory accounting | Known edge cases around page cache and swap | More accurate, unified accounting model |
| Interface files | Controller-specific, inconsistent naming | Standardized naming across all controllers |
| Docker default | Older Docker Engine versions and older kernels | Default on modern Docker Engine with a v2-capable kernel and distro |
| Delegation to unprivileged users | Very limited, considered unsafe | Designed from the start to support safe delegation |
For most people running Docker day to day, this version difference is invisible — Docker Engine detects which version the host kernel is using and adapts automatically. But it becomes very relevant the moment you try to debug resource limits by hand, because the file paths and file names you inspect on disk are different between the two versions, and instructions written for one version will not directly apply to a host running the other.
TChapter 4 — The Cgroup Filesystem and Hierarchy
One of the most elegant design decisions behind cgroups is that they are exposed to the rest of the operating system as an ordinary filesystem, specifically a special filesystem type called cgroupfs, conventionally mounted at a path such as /sys/fs/cgroup. This means that, in principle, reading a resource limit or a live usage statistic is as simple as reading a text file, and setting a new limit is as simple as writing a number into a text file. No special system calls, no custom binary protocol — just files and directories, which is exactly the kind of interface the Unix philosophy has favored for decades.
Inside that filesystem, cgroups are organized as a tree, much like a folder structure on your own computer. There is a root cgroup at the very top, representing the entire machine, and every other cgroup is a subdirectory somewhere beneath it, representing a smaller and smaller slice of the machine’s resources. Critically, a child cgroup can never exceed the limits inherited from its parent — if a parent directory caps memory at 4 gigabytes, no child folder beneath it can ever be configured to use more than that 4 gigabytes combined, no matter what number an administrator tries to write into the child’s own limit file. This inheritance rule is what allows an administrator to draw a hard boundary around, say, “all Docker containers on this host,” and know with certainty that nothing beneath that boundary can breach it, regardless of how individual containers are configured.
graph TD
A["/sys/fs/cgroup (root — entire machine)"] --> B["system.slice (host OS services)"]
A --> C["docker (Docker Engine's own group)"]
C --> D["container_a1b2c3 (one running container)"]
C --> E["container_d4e5f6 (another running container)"]
D --> F["cpu.max"]
D --> G["memory.max"]
D --> H["pids.max"]
D --> I["io.max"]
E --> J["cpu.max"]
E --> K["memory.max"]
style A fill:#111,stroke:#dc2626,color:#fff
style C fill:#111,stroke:#dc2626,color:#fff
style D fill:#dc2626,stroke:#111,color:#fff
style E fill:#dc2626,stroke:#111,color:#fff
Simplified cgroup v2 hierarchy on a Docker host — every running container gets its own subdirectory under Docker’s parent cgroup, and every subdirectory inherits whatever ceiling its parent already imposed.
Within any cgroup directory sit a set of specially named files that the kernel treats as a live control panel rather than ordinary stored text. Files ending in names such as .max hold the limit you can write to change enforcement, files such as those reporting current usage are read-only windows into the kernel’s own live counters, and a file named cgroup.procs lists exactly which process IDs currently belong to that group. Writing a process ID into that file moves the process into the group; reading it tells you the group’s current membership. This tiny set of files is the entire interface — there is no separate cgroups daemon or database, the filesystem itself is the API, which is part of why cgroups have remained fast and dependable for over a decade.
People sometimes assume cgroups create a copy of the resource — as if a container “gets its own” slice of RAM chips carved out physically. In reality nothing is physically partitioned; the same RAM, same CPU cores, and same disks are shared by every process on the box. Cgroups only enforce accounting rules about how that single shared pool may be divided, entirely in software, inside the kernel.
Docker itself creates its own cgroup subtree, typically visible as a directory literally named docker beneath the root, and every container Docker starts gets its own child cgroup nested underneath that, usually named after the container’s unique ID. This is precisely why running two commands — one that lists Docker’s own live statistics, and one that reads the raw cgroup files by hand — will show consistent numbers: both are reading from, or derived from, that exact same tree of files maintained by the kernel.
HChapter 5 — How Docker Actually Uses cgroups Internally
This is where the abstract kernel feature and the everyday Docker command line finally connect.
Docker Engine does not talk to cgroups directly for most operations. Instead, it delegates the actual work of creating namespaces, setting up cgroups, and launching the container’s first process to a lower-level component called a container runtime, most commonly runc, which is itself an implementation of an industry-standard specification maintained by the Open Container Initiative (OCI). When you type a command to start a container, Docker’s daemon translates your flags — memory limits, CPU shares, and so on — into a structured configuration file, hands that file to containerd (Docker’s mid-level supervisor process), which in turn hands it to runc, which is the component that finally performs the low-level work of creating the cgroup directory, writing the limit files, and launching the container’s process inside both the new namespaces and the new cgroup simultaneously.
sequenceDiagram
participant U as User (docker run)
participant D as Docker Daemon (dockerd)
participant C as containerd
participant R as runc
participant K as Linux Kernel
U->>D: docker run --memory=512m --cpus=1 myapp
D->>D: Translate flags into OCI runtime spec
D->>C: Create and start container task
C->>R: Invoke runc create + runc start
R->>K: mkdir cgroup directory for container
R->>K: Write memory.max, cpu.max limit files
R->>K: Create new namespaces (PID, net, mnt, ...)
R->>K: Move container's first process into cgroup
K-->>R: Process running, resource-limited
R-->>C: Container running
C-->>D: Status: running
D-->>U: Container ID printed
The chain of components between a Docker command and the kernel actually creating the cgroup — Docker delegates the low-level work to containerd and runc, which speak directly to the kernel’s cgroup filesystem.
The practical upshot of this layered architecture is that familiar Docker command-line flags are really just friendly translations of raw cgroup controller files. A memory flag on the command line ultimately becomes a number written into the memory controller’s maximum-usage file for that container’s cgroup. A CPU flag becomes numbers written into the CPU controller’s quota and period files. A flag limiting the number of processes a container may spawn becomes a number written into the PIDs controller’s maximum file. Docker’s contribution is entirely about making these mappings convenient, validated, and consistent — the enforcement itself is 100% the Linux kernel’s own cgroup machinery, not anything Docker computes or polices in user space.
Think of Docker as a friendly hotel receptionist, containerd as the hotel’s floor manager, and runc as the porter who actually walks up with the master key and physically locks a guest’s door to the agreed level of access. You, the guest, just say “I’d like a room with a mini-fridge limited to soft drinks only” at the front desk in plain English. The receptionist writes that down in hotel-policy language, the floor manager passes the instruction along, and the porter is the one who actually turns the physical lock. You never touch the lock yourself, but every convenience you requested ultimately becomes a physical restriction enforced by someone standing at the door.
It also explains why cgroups are so central to Docker’s security and stability story: even if a bug existed somewhere in Docker’s own daemon code, the actual enforcement of “this container cannot use more than 512 megabytes” lives inside the kernel itself, in code that has been reviewed, hardened, and battle-tested across the entire Linux ecosystem — not just by Docker’s own engineers. A crash or bug in the Docker daemon does not remove a running container’s resource limits, because those limits are not something the daemon is actively enforcing moment to moment; they are baked into the kernel’s cgroup files, and the kernel keeps enforcing them regardless of whether Docker’s own process is even alive.
CChapter 6 — Data Flow and the Container Lifecycle
Tracing a container from creation to termination makes it easier to see exactly when cgroups enter and leave the picture, because it is not a one-time setup step — cgroups are actively read from and written to throughout the entire life of a container.
Image Pull and Configuration Parsing
Docker reads any resource-related flags you supplied and validates them — for instance, rejecting a memory limit that is nonsensically low for the base image being used.
Cgroup Directory Creation
runc creates a brand-new subdirectory in the cgroup filesystem, nested under Docker’s own parent cgroup, named after the container’s unique identifier.
Limit Files Are Populated
Every requested constraint — memory ceiling, CPU quota, maximum process count, I/O weight — is written into its corresponding controller file inside that new directory.
Namespaces Are Created
In parallel, runc creates the isolated namespaces that give the container its own private view of the process tree, network stack, and filesystem mounts.
The Container’s First Process Is Launched Inside Both
The container’s entrypoint process is started already living inside the new cgroup and the new namespaces simultaneously — there is no moment where it runs unrestricted.
Every Child Process Inherits Membership
If the container’s application forks additional worker processes or threads, each one automatically becomes a member of the same cgroup with no extra configuration needed.
Continuous Live Accounting
For as long as the container runs, the kernel keeps updating usage counters in the cgroup’s read-only files on every scheduling tick, memory allocation, and disk operation.
Cleanup on Exit
When the container stops, its process exits the cgroup, and once the group is empty, Docker (via runc) removes the now-unused cgroup directory, freeing the accounting structures.
Step 5 is the detail most people miss: a container process is never launched “naked” and then wrapped in restrictions afterward. The cgroup and namespaces exist before the container’s first line of code ever runs, which closes off any window where a fast-starting process could act unrestricted.
MChapter 7 — Memory Cgroups in Depth
Memory is usually the resource people care about most when setting Docker limits, because an application that runs out of memory does not simply slow down — it typically gets killed. The memory controller inside cgroups tracks every page of memory a group’s processes touch, including the trickier categories: shared memory, memory-mapped files, and cached file data. When a container’s total memory usage reaches the ceiling set in its cgroup, the kernel does not politely ask the container to free some memory. Instead, a mechanism called the OOM killer (Out-Of-Memory killer) steps in and forcibly terminates a process inside that cgroup to bring usage back under the limit — this is the origin of the infamous “OOMKilled” status that Docker users encounter when a container mysteriously dies.
Picture a parking garage with a strict capacity sign reading “Maximum 50 cars.” Cars are memory pages, and the garage is the container’s memory cgroup. As long as fewer than 50 cars are parked, everyone comes and goes freely. The moment a 51st car tries to enter, the attendant does not let it squeeze in somewhere — they tow away a car that is already parked (typically the biggest offender) to make room. That tow truck is the OOM killer, and the car it removes is usually the process using the most memory inside that group, though the exact choice depends on a scoring heuristic the kernel maintains for every process.
A subtlety that trips up many engineers is the distinction between a container’s memory limit and the memory limit of an individual process running inside it. Setting a container-wide ceiling of, say, one gigabyte does not automatically mean any single process inside gets a guaranteed slice of that gigabyte — if a container runs three processes and one of them leaks memory unrestrained, it can consume the entire group’s budget and get OOM-killed even while its two siblings were behaving perfectly normally. This is why applications with predictable memory ceilings of their own, such as a Java virtual machine with an explicit heap size, tend to behave far more predictably inside a memory-limited container than applications that assume they can use however much memory the operating system reports as available.
This leads to one of the most common real-world pitfalls: many programming language runtimes historically asked the operating system “how much total memory does this machine have?” rather than asking “how much memory does my own cgroup allow me to use?” Older versions of the Java virtual machine, for example, would size their default heap based on the host’s full physical memory rather than the container’s much smaller cgroup limit, leading to a JVM that assumed it had access to sixty-four gigabytes of RAM while actually running inside a two-gigabyte cgroup — a recipe for constant, confusing OOM kills. Modern versions of major runtimes have since become “cgroup-aware” and correctly read the container’s actual limit, but this history is exactly why memory limits deserve careful testing rather than blind trust that “the application will just adapt.”
Setting a memory limit without also configuring the application inside to respect a similar or slightly smaller ceiling is one of the most frequent causes of mysterious container crashes in production. The container is not “broken” — it is doing exactly what it was told, killing the process that exceeded the budget it was given.
Beyond the hard maximum, the memory controller also exposes a softer concept often called a memory “low” or “reservation” boundary: a level below which the kernel will try hard to protect the group’s memory from being reclaimed even under system-wide pressure, without being an absolute hard limit the group can never exceed. This lets administrators express priority — “under a memory crunch, protect this group’s memory before protecting that one” — which is heavily used by orchestration systems like Kubernetes to distinguish workloads that must never be starved from workloads that can tolerate being squeezed first.
CChapter 8 — CPU Cgroups in Depth
CPU is a fundamentally different kind of resource from memory, and that difference shapes how its cgroup controller is designed. Memory is a fixed pool you either have available or you don’t — once a gigabyte is in use, it is simply gone until freed. CPU time, by contrast, is a continuously renewing resource: every single second, the machine produces a fresh second of processing time on every core, whether anyone uses it or not. This means CPU limiting is less about a hard ceiling on a stockpile and more about controlling how a continuously flowing resource gets divided among competitors.
The CPU controller offers two complementary approaches to this division problem. The first is relative weighting, historically exposed through a value commonly called CPU “shares.” This does not set any absolute cap at all — instead, it says “when multiple groups are actively competing for the same CPU core at the same moment, divide the available time proportionally according to these weights.” A group with double the weight of another gets roughly double the CPU time only when both are actually competing; if the other group is idle, the higher-weighted group can freely use the entire core, because there is no hard ceiling being enforced, only a tie-breaking rule for contention.
The second approach is absolute quota and period limiting, which is what most people mean when they set a specific “number of CPUs” on a Docker container. This mechanism defines a recurring time window (the period, commonly one-tenth of a second) and a maximum amount of actual CPU time the group may consume within that window (the quota). A container capped at one full CPU is, roughly speaking, being told “out of every hundred milliseconds that pass, you may use at most one hundred milliseconds of total processing time across all your cores combined” — and if the container tries to use more, the kernel simply pauses (throttles) its processes for the remainder of that window, resuming them only when the next window begins.
Relative CPU shares work like a shared photocopier during a busy office morning: if only one person needs it, they can print as much as they want uninterrupted. The moment three colleagues show up wanting to print at the same time, the office’s informal etiquette (“senior staff get first priority”) decides who gets more turns — but nobody is ever told they can’t use the machine at all if it happens to be free. Absolute CPU quota, on the other hand, works like a strict photocopier booking system: “you are allowed exactly six minutes of copier time every hour, whether anyone else needs the machine or not.” Even if the copier sits completely idle the rest of the hour, you cannot exceed your six minutes.
This distinction explains a frequently confusing symptom: a container configured with an absolute CPU limit can appear artificially slow or “throttled” even on a mostly idle server with plenty of spare CPU capacity sitting unused, because the limit is a hard ceiling regardless of contention, not merely a fairness rule that only kicks in when the machine is busy. Engineers debugging unexpectedly sluggish containers on quiet servers should always check whether an absolute CPU quota, rather than actual hardware contention, is the real culprit.
If a workload has occasional short bursts of CPU-intensive work but is idle most of the time, relative CPU shares usually produce smoother behavior than a strict absolute quota, because shares only restrict the workload when there is genuine competition for the core rather than throttling it in isolation.
PChapter 9 — PIDs, Block I/O, and Device Cgroups
Beyond the two headline resources of memory and CPU, several other cgroup controllers quietly protect a Docker host from equally dangerous failure modes, and each deserves its own explanation because each guards against a different category of real-world incident.
The PIDs controller limits the total number of processes and threads a cgroup may create at once. This might sound like a minor detail, but it defends against one of the oldest and simplest attacks against any shared computer: the fork bomb, a program that does nothing but continuously spawn copies of itself until the operating system runs out of process table slots entirely, freezing the entire machine for every user and workload, not just the offending one. Before the PIDs controller existed, a single misbehaving or malicious container could exhaust the host’s global process table and bring down every other container on that host, even ones with generous memory and CPU limits, simply because there were no process “slots” left anywhere on the machine. Capping the number of PIDs per container closes this entire category of attack, containing the damage to the single misbehaving group.
Real Incident Pattern
A common accidental fork bomb happens when a script has a typo in a recursive function — for example, calling a shell script from within itself with no exit condition. With a sensible PIDs limit configured, that container simply hits its process ceiling and new process-creation calls start failing loudly, which is a far gentler and more diagnosable failure than an entire multi-tenant host silently grinding to a halt.
The block I/O (blkio, or the unified io controller in cgroups v2) controller governs how much disk read and write throughput, and how many disk operations per second, a cgroup may perform. Disks — even fast solid-state drives — have a finite amount of bandwidth and a finite number of operations they can service per second, and unlike CPU cores, a single disk is often shared across an entire host rather than duplicated per core. A single container performing constant heavy disk writes, such as an application logging excessively or a database running an unthrottled bulk import, can saturate the shared disk and cause every other container on that host to experience painfully slow file access, even if those other containers have plenty of free CPU and memory. The I/O controller lets an administrator cap a container’s read and write throughput, or assign relative I/O priority weights, using exactly the same relative-versus-absolute philosophy already described for CPU.
The devices controller takes a different shape from the others — rather than limiting an amount, it controls whether a cgroup is permitted to access a given device file at all, such as a specific disk, a GPU, or a sound card. This is less about fair sharing and more about security and correctness: it prevents a container from directly reading or writing to raw hardware devices that were never intended to be exposed to it, which would otherwise be a straightforward way for a compromised or malicious container to read data belonging to a completely different container sharing the same host.
Process Count Ceiling
Prevents fork bombs and runaway thread creation from starving the host’s entire process table.
Disk Throughput Fairness
Caps read/write bandwidth and operations-per-second so one noisy container cannot saturate shared disks.
Hardware Access Gate
Grants or denies access to specific raw device files rather than metering an amount of usage.
Traffic Classification
Tags a group’s network packets so external routers and firewalls can apply their own bandwidth policies.
AChapter 10 — Advantages, Disadvantages, and Trade-offs
Like every engineering decision, relying on cgroups for isolation and resource control involves genuine trade-offs, and understanding both sides is essential for anyone designing a production system rather than just following defaults.
Advantages
- Enforcement happens inside the kernel itself, making it extremely fast and reliable compared to a user-space watchdog process.
- Containers start in milliseconds because there is no separate operating system kernel to boot, unlike a virtual machine.
- Fine-grained control over CPU, memory, disk I/O, process count, and device access, all from one consistent filesystem-based interface.
- Prevents “noisy neighbor” problems, where one workload’s resource hunger degrades every other workload sharing the same machine.
- Live, low-overhead usage accounting is a natural byproduct of the same mechanism used for enforcement, making monitoring nearly free.
Disadvantages & Limitations
- Cgroups alone do not provide the same strength of isolation as a full virtual machine, because containers still share one kernel.
- Memory limits interacting with legacy application runtimes that are not “cgroup-aware” can cause confusing, hard-to-diagnose crashes.
- CPU quota throttling can create latency spikes that are difficult to distinguish from genuine hardware contention without careful monitoring.
- cgroups v1 and v2 differ enough in file layout that tooling and documentation written for one version can mislead engineers on the other.
- Cgroups control resource quantity but say nothing about what a process can see — namespaces, and separate security tooling, are still required.
The deepest trade-off is philosophical rather than technical: cgroups (together with namespaces) buy containers their famous speed and density precisely by sharing one kernel among many workloads, but that same shared kernel is also the single point through which a sufficiently severe kernel-level vulnerability could, in theory, affect every container on the host at once. A virtual machine trades away that speed and density in exchange for a stronger isolation boundary, because each virtual machine runs its own separate kernel that a bug in one guest generally cannot reach across to affect another. Neither choice is universally “better” — it is a genuine engineering trade-off between density and cost on one side, and isolation strength on the other, and many modern platforms increasingly blend the two by running lightweight virtual machines that still present a container-like interface, precisely to get closer to both benefits at once.
SChapter 11 — Security Implications
Cgroups were originally designed to solve fairness and resource-allocation problems, not to be a security boundary in the way a firewall or an encryption scheme is — but in practice, they play a meaningful supporting role in Docker’s overall security posture, and misunderstanding that role is a common source of risky assumptions.
The most important security benefit cgroups provide is denial-of-service containment. Without resource limits, any single container — whether compromised by an attacker, or simply misbehaving due to a bug — can consume unlimited memory, CPU, disk bandwidth, or process slots, degrading or completely disabling every other workload on the same host, including unrelated tenants who have nothing to do with the misbehaving container. This is sometimes called the “noisy neighbor” problem when accidental, or a resource-exhaustion attack when deliberate, and cgroups are the primary defense against both, by ensuring that even a fully compromised container cannot exceed the resource envelope it was assigned, no matter what code an attacker manages to run inside it.
Cgroups limit how much a container can use, not what a container can access or do. Running Docker containers without explicit resource limits is a genuine security gap on any shared, multi-tenant host, because a single container can effectively deny service to every other container simply by exhausting a shared resource — this is true even if every container is otherwise perfectly well-behaved code, since bugs and unexpected load spikes happen to honest applications too.
There is also a subtler, historically significant security angle: certain features of cgroups themselves, particularly a legacy mechanism in cgroups v1 called the “release agent,” have been exploited in real-world container escape research as a path for a privileged process inside a container to execute arbitrary code on the underlying host, effectively breaking out of the container entirely. This is precisely why running containers with unnecessary elevated privileges, and running an outdated container runtime, are treated as serious risks by security-conscious operators — the cgroup filesystem, if a container is given write access to sensitive administrative files within it, can itself become an attack surface rather than only a protective mechanism. Modern Docker deployments mitigate this by keeping containers unprivileged by default and by adopting cgroups v2, whose redesigned architecture removed several of these historically exploitable corners.
The practical security guidance that follows from all of this is straightforward: always set explicit memory, CPU, and PIDs limits on containers running in any shared or production environment, never run containers with broader privileges than the specific task requires, and keep the underlying Docker Engine, container runtime, and host kernel patched, since the security guarantees cgroups provide are only as strong as the kernel code enforcing them.
MChapter 12 — Monitoring, Logging, and Metrics
Because cgroups already maintain live, continuously updated accounting data as a core part of how they enforce limits, monitoring a container’s resource usage is essentially free — there is no separate instrumentation layer required, only a question of which tool reads the existing numbers and how it presents them.
At the most direct level, an administrator can read the raw cgroup files for any running container by hand, seeing exactly the same numbers the kernel itself is using to make enforcement decisions — current memory usage, the count of times the memory limit has been hit and triggered a reclaim, current CPU time consumed, and the number of periods in which the container was throttled by its CPU quota. This raw view is invaluable when debugging a specific mystery, such as confirming whether a container’s slowness truly stems from CPU throttling rather than something else entirely, because it shows the exact ground truth the kernel is enforcing rather than an approximation calculated by a monitoring agent one layer removed.
For everyday use, most engineers rely on Docker’s own built-in statistics command, which is itself simply a friendly, human-readable summary of the same underlying cgroup files, refreshed continuously. It shows memory usage against the configured limit, CPU usage as a percentage, network throughput, and block I/O, all per container, without requiring any additional software to be installed. This is usually the fastest way to get an immediate read on whether a specific container is close to hitting one of its ceilings.
Cgroup Filesystem
Reading limit and usage files directly gives the exact ground-truth numbers the kernel itself is using.
Docker Stats Command
A friendly, continuously refreshed summary derived directly from the same cgroup files, no extra tooling required.
cAdvisor
An open-source agent, originally built at Google, that exports per-container cgroup metrics for dashboards and alerting.
Prometheus & Grafana
A common pairing where cAdvisor-style exporters feed a time-series database, visualized on dashboards, for historical trend analysis.
At production scale, teams typically layer a dedicated collection agent on top of the raw cgroup data, one of the most widely used being cAdvisor (Container Advisor), originally built at Google, which continuously scrapes every running container’s cgroup files and exposes the resulting metrics in a format that time-series databases such as Prometheus can ingest, store historically, and graph on dashboards such as Grafana. This layered approach — kernel accounting at the bottom, a lightweight scraping agent in the middle, and a full observability stack on top — is what lets platform teams set automated alerts, such as paging an engineer whenever a container’s memory usage sits above ninety percent of its limit for more than five minutes, entirely built on data that originates from the same simple cgroup files described earlier in this article.
When a monitoring dashboard and Docker’s own statistics command disagree, trust the raw cgroup files as the tiebreaker — they are the single source of truth every other tool is ultimately built on top of.
DChapter 13 — Design Patterns and Anti-Patterns
Over more than a decade of production Docker usage, certain recurring patterns for using cgroup-based limits have proven reliable, while certain recurring mistakes have proven costly enough to earn their own well-known names among practitioners.
What It Is
Every container that will ever run outside of a developer’s personal laptop is given an explicit memory limit, CPU limit, and PIDs limit before it is ever deployed, sized from real measurement rather than guesswork.
Why It Works
It converts an unbounded, unpredictable workload into a workload with a known worst-case footprint, which is the single piece of information a shared host, a capacity planner, and an incident responder all need most.
What It Is
Running several containers on the same shared host with no resource limits configured at all, relying on the assumption that “there’s usually enough capacity to go around.”
Why It Fails
Any single container that spikes — from legitimate load, a bug, or an attack — can silently degrade every other workload on that host, and because nothing was ever limited, there is no clear signal pointing at which container caused the incident.
What It Is
Copying a memory or CPU limit from an unrelated project’s documentation or a colleague’s old configuration without measuring the actual workload’s real requirements.
Why It Fails
A limit set too low causes constant OOM kills or CPU throttling under perfectly normal load, producing an unstable service that looks broken even though nothing is actually wrong with the application code itself.
What It Is
Distinguishing between a soft, guaranteed baseline of resources a container should always be able to count on, and a harder, looser ceiling it is allowed to burst up to if the host has spare capacity.
Why It Works
It lets a host pack more workloads together economically during quiet periods, while still guaranteeing every workload a protected floor during busy periods — this exact pattern, built directly on relative cgroup weights alongside absolute cgroup limits, is the foundation of how Kubernetes’ own “requests and limits” system operates.
BChapter 14 — Best Practices and Common Mistakes
Measure Before You Limit
Run the workload under realistic load first and observe its true resource ceiling before setting a production limit, rather than guessing a round number.
Set All Three: Memory, CPU, PIDs
Memory limits alone leave a host exposed to CPU starvation and fork bombs; all three controllers protect different failure modes.
Make Application Runtimes Cgroup-Aware
Confirm the language runtime or framework inside the container actually reads the container’s cgroup limit rather than the host’s total physical memory.
Alert on Approaching Limits, Not Just Breaches
An alert firing only once a container is already OOM-killed is too late to be useful; alert well before the ceiling is reached.
Confusing Throttling With a Crash
A CPU-throttled container looks slow, not dead — engineers who assume a crash and restart the container repeatedly can mask the real underlying quota problem.
Ignoring Page Cache in Memory Accounting
File-backed cache pages can count toward a container’s memory usage under certain configurations, making usage appear higher than the application’s own working set.
Setting Identical Limits Across Very Different Services
Copy-pasting one memory or CPU value across a whole fleet of unrelated services ignores that each workload has a genuinely different footprint.
Forgetting cgroups v1/v2 Differences When Scripting
A monitoring script that reads raw cgroup file paths hard-coded for one version will silently fail or report nothing useful on a host running the other version.
RChapter 15 — Real-World and Industry Examples
Kubernetes, the industry-standard container orchestration platform originally built at Google and now maintained by the Cloud Native Computing Foundation, is arguably the largest and most consequential consumer of cgroups in the world today. Kubernetes’ well-known “requests” and “limits” configuration for every workload is, underneath the surface, a direct translation into exactly the same cgroup controller files described throughout this article — a Kubernetes memory “limit” becomes a cgroup memory maximum, and a Kubernetes CPU “request” becomes a relative cgroup CPU share. Kubernetes also introduces its own “Quality of Service” classification for workloads, built entirely on how tightly a workload’s requests and limits line up, and that classification directly determines which containers the kernel’s OOM killer will choose to sacrifice first on a memory-starved node, again driven by the exact same cgroup mechanisms.
Netflix, which runs an enormous fleet of containerized microservices to deliver streaming video to hundreds of millions of subscribers, has published extensively about tuning cgroup-based CPU and memory limits as a core part of keeping its infrastructure both cost-efficient and resilient to noisy-neighbor effects across the thousands of services co-located on its shared compute fleet. Precisely calibrated resource limits let Netflix safely pack many independent services onto shared hosts, substantially reducing wasted, unused capacity compared to giving every service its own dedicated, oversized machine.
Google, the very company that originated cgroups to solve exactly this problem inside its own internal cluster manager (a system called Borg, the direct ancestor of Kubernetes), continues to rely on the same underlying kernel feature — now refined over more than fifteen years of continuous production use — to safely pack a staggering diversity of workloads, from web-serving front ends to massive batch analytics jobs, onto the same shared machines across its data centers worldwide.
Even smaller-scale, everyday platforms benefit the same way: any shared hosting provider or platform-as-a-service company that lets customers deploy Docker containers side by side on the same physical servers is, whether or not the customer ever notices, relying entirely on cgroups to guarantee that one customer’s runaway workload cannot degrade a completely unrelated customer’s application running on the very same machine.
FChapter 16 — Frequently Asked Questions
Entirely outside, from the container’s own point of view. A cgroup is a kernel-level accounting and enforcement structure that sits at the host level; a process running inside the container generally cannot even tell it is being limited by a cgroup unless it specifically checks for the presence and values of the relevant cgroup files.
Under a normal, unprivileged container configuration, no — the process does not have permission to modify the limit files governing its own group, because that would defeat the entire purpose of the mechanism. This protection weakens significantly if a container is run with elevated privileges or excessive capabilities, which is exactly why minimizing container privileges is treated as a core security practice.
By default, Docker still creates a cgroup for every container, but without explicit limits configured, that container is effectively free to consume as much memory, CPU, and I/O as the host machine has available, competing on equal footing with every other unrestricted container and the host’s own processes.
No. A limit is a hard ceiling the container can never cross, enforced by the OOM killer if breached. A reservation (or “low” boundary) is closer to a protected floor, expressing that the kernel should try to defend that much memory for the group even under system-wide memory pressure, without being an absolute cap on how much more it could use if available.
This commonly happens because the statistics snapshot was taken a moment before a sudden spike, because certain memory categories such as kernel-side buffers are accounted differently than user-visible application memory, or because a legacy, non-cgroup-aware runtime inside the container briefly attempted to allocate far more than it ultimately used, triggering the OOM killer before settling back down.
The accounting and enforcement overhead itself is extremely small, generally negligible for the vast majority of workloads, because it is implemented directly inside performance-critical kernel code paths that were already running regardless. Any slowdown people experience almost always comes from an actual limit being reached — CPU throttling or memory pressure — rather than from the mere existence of the cgroup mechanism itself.
No — cgroups are a general-purpose Linux kernel feature used by many technologies beyond Docker, including other container runtimes such as Podman and LXC, orchestration platforms such as Kubernetes and systemd’s own service management, and even non-container use cases like limiting resource usage for ordinary system services.
SChapter 17 — Summary and Key Takeaways
Key Takeaways
- Cgroups are a Linux kernel feature, originally built at Google, that group processes together and meter, limit, and prioritize their use of CPU, memory, disk I/O, and process count.
- Cgroups answer “how much,” namespaces answer “what.” Docker’s isolation depends on both working together, not either one alone.
- Two versions exist — v1 uses separate trees per controller, v2 uses one unified tree, and modern Docker hosts default to v2.
- Cgroups are exposed as an ordinary filesystem, typically at
/sys/fs/cgroup, where reading and writing plain text files is the entire control interface. - Docker delegates the actual work to containerd and runc, which translate command-line flags into cgroup controller files and enforce them via the kernel.
- Memory limits are enforced by the OOM killer, which terminates a process rather than gracefully throttling memory usage, making cgroup-aware application runtimes important.
- CPU limiting comes in two flavors — relative shares for fairness under contention, and absolute quota for a hard, unconditional ceiling — each suited to different workload patterns.
- The PIDs controller prevents fork bombs from exhausting the host’s entire process table, protecting every other container sharing that host.
- Cgroups are essential for denial-of-service protection on any shared host, but they are not a full security boundary on their own, and misconfigured privileges can weaken their guarantees.
- Live monitoring is nearly free because usage accounting is a natural byproduct of enforcement — Docker stats, cAdvisor, and Prometheus dashboards all ultimately read the same underlying cgroup data.
- Kubernetes, Netflix, and Google all build core infrastructure decisions directly on top of cgroups, proving the mechanism scales from a single developer’s laptop to some of the largest compute fleets on the planet.