Before we go anywhere near Docker, kernels, or copy-on-write, let’s build the idea from nothing.

A regular filesystem is a single, flat source of truth. When you open a file, the operating system looks in exactly one place: the disk (or partition, or volume) where that filesystem lives. A union filesystem breaks that assumption. Instead of one source of truth, it lets you stack several directories — called layers — on top of each other, and then presents them to every program on your computer as though they were a single, unified directory. The word “union” is used in the mathematical sense: just like the union of two sets combines their members into one collection, a union filesystem combines the contents of several directories into one visible view.

Here is the part that makes it genuinely useful rather than just a curiosity: the layers do not have to be equal. Typically, one or more layers are marked read-only, and exactly one layer on top is marked writable. Any program that reads a file walks down through the stack until it finds that file in the topmost layer that has it. Any program that writes or deletes a file only ever touches the writable layer on top — the read-only layers underneath are never modified, no matter what happens above them.

This single idea — many layers, one visible view, changes isolated to the top — is the entire foundation of union filesystems. Everything else in this guide is really just working out the consequences of that one idea: what happens when you edit a file that lives in a lower layer, what happens when you delete something, what happens when two layers have a file with the same name, and how operating systems make all of this fast enough to use in production at massive scale.

Good technology usually exists because copying gigabytes of data over and over was too slow, too wasteful, or both.

Imagine you want to run ten copies of the same web server on one machine, each with slightly different configuration files. Without a union filesystem, the naive approach is to copy the entire base operating system — every library, every binary, every configuration file — ten separate times onto disk. If that base system is 500 megabytes, you have just spent 5 gigabytes of disk space to store what is, in reality, the same data ten times over with only a handful of files actually different between the copies. Worse, creating each copy takes real time, because the operating system has to physically write every one of those files to disk before the new instance can start.

This exact problem showed up again and again across computing history, in slightly different clothes each time:

Notice the pattern: every generation of this technology was solving the same underlying problem — how do you give many consumers the illusion of their own private, writable copy of something, without physically duplicating the data behind the scenes? Live CDs needed it so a read-only disc could feel interactive. Containers needed it so thousands of near-identical application environments could start instantly and share almost everything on disk. That shared motivation is why union filesystems became one of the quiet, foundational technologies of modern computing.

Modern union filesystems, especially Linux’s OverlayFS, are built from a small handful of named directories that each have a very specific job.

Rather than the vague word “layer,” real implementations give each layer role a precise name. Understanding these names is the difference between reading a Docker inspect output as gibberish and reading it as a clear map of what is happening on disk.

graph TB
    subgraph Stack["Union Mount Stack"]
        L3["lowerdir 3 (base OS image layer)"]
        L2["lowerdir 2 (application dependencies layer)"]
        L1["lowerdir 1 (application code layer)"]
        UP["upperdir (this container's writable layer)"]
    end
    WD["workdir (internal scratch space)"]
    MERGE["merged / mount point (what the running process actually sees)"]
    L3 --> MERGE
    L2 --> MERGE
    L1 --> MERGE
    UP --> MERGE
    WD -. supports .-> UP
    MERGE --> APP["Application process reads and writes here"]
    

This is where the “magic” of union filesystems becomes a small set of concrete, learnable mechanics.

Copy-Up: How Editing a Read-Only File Actually Works

Suppose a file called config.txt lives only in a lowerdir, and a running container tries to open it for writing. The lowerdir is read-only, so the kernel cannot simply edit the bytes in place. Instead, OverlayFS performs an operation called copy-up: it copies the entire file from the lowerdir into upperdir, byte for byte, and only then applies the write to that new copy in upperdir. From that moment forward, every read of config.txt finds the version in upperdir first, because upperdir is always checked before any lowerdir. The original file in the lowerdir is left completely untouched, which is exactly why the same base image can be safely reused by a thousand other containers running at the same time.

Whiteouts: How “Deleting” a File in a Lower Layer Works

Deletion is trickier than it sounds, because the file you are “deleting” might physically live in a read-only lowerdir that the current layer has no permission to touch. OverlayFS solves this with a whiteout: instead of removing the file, it creates a special character device file (traditionally with device numbers 0,0) inside upperdir, using the same filename. When the union filesystem builds its merged view and sees a whiteout marker for a name, it hides that name entirely from every layer below, even though the real file is still sitting untouched in the lowerdir. Delete a file, and from the outside it simply disappears — but underneath, nothing in the read-only layer was ever removed.

Opaque Directories: Replacing, Not Merging

Sometimes a container needs to completely replace a directory’s contents rather than adding new files alongside old ones. If you delete every file inside a directory one by one, OverlayFS would normally still merge in anything a lower layer contributes to that same directory name. To prevent this, OverlayFS marks the directory in upperdir as opaque using an extended file attribute. An opaque directory tells the merge logic: “stop looking in lower layers for this directory’s contents — everything you need is right here in upperdir.” Docker relies on this exact mechanism whenever a Dockerfile instruction removes an entire directory and replaces it with fresh contents.

Once you know the pieces from the last two chapters, tracing exactly what happens for each kind of file operation becomes mechanical.

sequenceDiagram
    participant App as Application Process
    participant Overlay as OverlayFS Merge Logic
    participant Upper as upperdir (writable)
    participant Lower as lowerdir (read-only)

    App->>Overlay: open("app.conf", READ)
    Overlay->>Upper: file present?
    Upper-->>Overlay: not found
    Overlay->>Lower: file present?
    Lower-->>Overlay: found, return contents
    Overlay-->>App: return contents from lowerdir

    App->>Overlay: open("app.conf", WRITE)
    Overlay->>Upper: file present?
    Upper-->>Overlay: not found
    Overlay->>Lower: copy-up file to upperdir
    Overlay->>Upper: apply write to new copy
    Overlay-->>App: write succeeds on upperdir copy

    App->>Overlay: delete("old.log")
    Overlay->>Upper: create whiteout marker for old.log
    Overlay-->>App: file now hidden from merged view
    

The lifecycle of a typical container image tells this story end to end. A Dockerfile with five instructions that touch the filesystem produces five lowerdirs, stacked in build order. When you run a container from that image, Docker adds one fresh, empty upperdir and workdir on top, and mounts the whole stack as a single merged directory that becomes the container’s root filesystem. The application inside genuinely believes it has its own private, writable disk. In reality it is looking through five read-only layers shared with every other container using that image, plus one thin, private layer of its own changes. When the container stops and is removed, that thin private layer simply vanishes, and the five shared layers — untouched the entire time — are ready to back the next container instantly.

Knowing the mechanism is one thing; knowing how to design systems around it well is another.

Pattern: Layered, Cacheable Build Images

Docker’s build system is really just an application of union filesystems to software packaging. Each instruction in a Dockerfile — installing a package, copying application code, setting an environment variable — produces exactly one new lowerdir layer. Because layers are content-addressed (identified by a hash of their contents), if you rebuild an image and only the last instruction changed, every layer above the unchanged instructions is reused straight from cache, and only the changed layer and anything above it needs to be rebuilt. This is why placing rarely-changing instructions (installing system dependencies) near the top of a Dockerfile and frequently-changing instructions (copying your application’s own source code) near the bottom dramatically speeds up rebuilds — the expensive, stable layers stay cached.

Pattern: Read-Only Base, Writable Session (Live Media)

Live Linux distributions apply the same layering idea in reverse order of intent, but identically in mechanism: the entire operating system sits in a read-only lowerdir sourced from a compressed disk image, and a small writable upperdir — often backed by RAM, or by a small writable partition on the USB drive — captures every change made during that session. This lets a completely read-only, tamper-proof base image feel fully interactive and writable to the user, without ever needing to modify the original protected image.

Anti-Pattern: Excessive Layer Counts

Every additional lowerdir adds a small amount of lookup overhead to every filesystem operation, because the kernel may need to check multiple layers before finding — or confirming the absence of — a file. Dockerfiles that chain dozens of tiny, separate RUN instructions rather than combining related steps into a single instruction create unnecessarily deep layer stacks. This does not just slow lookups slightly; it also bloats the image, since intermediate layers created by instructions like installing and then removing temporary files can leave “ghost” data taking up space in earlier layers even after later layers mark it deleted with a whiteout.

Anti-Pattern: Treating a Lowerdir as Mutable

A subtler mistake is architecting an application to assume it can write directly to what is actually a lowerdir path — for instance, bypassing the union mount and mounting a lower layer’s underlying directory directly for “faster” writes. This breaks the entire isolation model: changes become visible to every other consumer of that layer, defeating the purpose of the read-only base and potentially corrupting the shared image used by many other containers or sessions.

Like every real engineering decision, union filesystems trade one set of costs for another set of benefits.

None of these trade-offs make union filesystems a poor choice — quite the opposite, since the industry’s overwhelming adoption of them for containers proves the benefits outweigh the costs for the vast majority of workloads. But they do explain why certain specialized workloads, particularly databases that perform constant, heavy in-place writes to large files, are usually configured to use dedicated volumes that bypass the union mount entirely rather than writing through several layers of copy-up logic on every update.

Because union filesystems sit directly underneath container isolation, their security properties matter for the entire platform built on top of them.

Shared Read-Only Layers as an Attack Surface

Because many containers share the exact same lowerdirs, a vulnerability that allows one container to somehow write directly into a lowerdir path — bypassing the intended union mount — would compromise every other container sharing that layer, not just the attacker’s own container. This is precisely why container runtimes enforce strict permission boundaries and mount namespaces around upperdir and lowerdir paths, ensuring a compromised container process genuinely cannot reach outside its own upperdir.

Whiteout and Opaque Directory Trust

Whiteout markers and opaque-directory attributes are just files and metadata like any other — which means an image containing maliciously crafted whiteouts could, in theory, be used to hide files or make a scanning tool believe certain paths do not exist, when in fact the data is still present in an underlying layer. This is one reason security scanning tools for container images increasingly inspect each layer independently, rather than only inspecting the final merged view, so that nothing hidden by a whiteout in a later layer goes unnoticed.

Permission and Ownership Consistency

When copy-up duplicates a file from a lowerdir into upperdir, the file’s ownership and permission bits are preserved by default. However, misconfigured user-namespace remapping (used to make container processes run as an unprivileged user on the host, even if they believe they are root inside the container) can create situations where copied-up files end up with unexpected ownership, which has historically been a source of subtle privilege-related bugs in container platforms.

You cannot manage disk pressure on a container host without understanding what a union filesystem’s numbers actually mean.

A common source of confusion is that traditional disk usage tools were never designed with layered storage in mind. Running a plain directory-size check on a container’s merged view can report a number close to the entire image size, even though almost all of that data is shared, read-only, and already counted once for every other container using the same base layers. Platform teams instead rely on tools built specifically for layered storage, such as Docker’s own system df reporting, which separately reports the space consumed by images, the space consumed by each container’s own writable layer, and reclaimable space from stopped containers and unused layers.

What matters most in production monitoring is tracking the growth of upperdir directories specifically, since that is the only part of the storage footprint that grows unpredictably during a container’s lifetime — the lowerdirs are fixed the moment the image is built. Teams running many long-lived containers typically alert on individual upperdir growth rather than total merged filesystem size, because a single misbehaving process writing an ever-growing log file directly into the container’s writable layer, instead of to a mounted volume, is a frequent and easily missed cause of a host quietly running out of disk space.

Most union-filesystem problems in production trace back to a small, well-known set of avoidable mistakes.

Union filesystems are not a niche curiosity — they quietly run some of the most widely used software delivery systems on the planet.

No — OverlayFS is one specific, modern implementation of the broader union filesystem concept, and the one built directly into the mainline Linux kernel today. Earlier implementations such as UnionFS and AUFS applied the same underlying idea using different internal mechanisms and had different trade-offs and compatibility characteristics.

For typical read-heavy workloads, the overhead is small enough to be unnoticeable, since reads that hit a file already present in upperdir, or in the topmost lowerdir containing it, are close to as fast as a normal filesystem lookup. The overhead becomes more noticeable for workloads with heavy write patterns to large files stored only in a lowerdir, due to the copy-up cost described in Chapter 4.

The layer closer to the top of the stack wins. If lowerdir 2 sits above lowerdir 3 and both contain a file named settings.conf, the merged view shows the version from lowerdir 2, and the version in lowerdir 3 is simply shadowed — it still exists physically, but it is invisible in the merged view.

In the common OverlayFS model, no — there is exactly one upperdir per mount. If you need what looks like multiple independent writable views of the same base, you create multiple separate union mounts, each with its own upperdir and workdir but sharing the same underlying lowerdirs, which is precisely how many containers share one base image simultaneously.

Because each unchanged Dockerfile instruction maps to an unchanged, already-built lowerdir layer, the build system can reuse those cached layers directly instead of recomputing them. Only the layers from the first changed instruction onward need to be rebuilt, which is why editing only your application’s final “copy source code” step typically results in a near-instant rebuild.

Not by the act of “deleting” it from within a container. A whiteout only hides the file from the merged view; the bytes remain on disk in the lowerdir until that entire image, and every layer that makes it up, is explicitly removed and garbage collected by the container runtime.