Docker’s classic builder executed a Dockerfile’s instructions one line at a time, strictly top to bottom, in a single sequence. Every instruction produced a new layer, and if an instruction changed, every instruction after it had to re-run too — even instructions that had nothing to do with the change. There was no way to build two independent parts of an image at the same time, no way to securely pass a secret into a build step without leaving traces behind, and no clean way to reuse cache from a remote build server shared across a whole team.

Docker began developing BuildKit around 2017 as a ground-up rewrite of the build engine, designed from the start around a dependency graph rather than a linear script. It was donated to the Moby project and later became a genuinely independent, vendor-neutral open-source project with contributors from Docker, Amazon, Google, and many others — eventually becoming the default builder in Docker itself starting with Docker Engine 23.0, and the engine underneath tools like docker buildx, GitHub Actions’ container build steps, and several cloud CI platforms.

The single most important shift to internalize is this: BuildKit does not think in terms of “lines in a Dockerfile.” It thinks in terms of a graph of operations with dependencies between them, where a Dockerfile is just one possible way of describing that graph.

graph TB
    A["Client\n(docker build / buildctl)"] -->|"Build request"| B["buildkitd daemon"]
    B --> C["Frontend\n(dockerfile.v0, custom frontends)"]
    C -->|"Produces"| D["LLB Graph\n(dependency graph of ops)"]
    D --> E["Solver\n(schedules & parallelizes)"]
    E --> F["Cache Manager\n(checks cache keys)"]
    F -->|"Cache miss"| G["Executor\n(runs step via runc)"]
    F -->|"Cache hit"| H["Reused Result"]
    G --> I["Snapshotter\n(filesystem diff)"]
    I --> E
    E --> J["Exporter\n(OCI image, tarball, registry push)"]
    

Client and Daemon Split

A build is requested by a client — most commonly docker build itself, or the more flexible docker buildx and raw buildctl tools — which sends the build request to the buildkitd daemon over a gRPC connection. This daemon can run locally, remotely, or even distributed across a cluster of build workers, which is what lets teams share one powerful, centrally cached build service instead of every developer’s laptop rebuilding everything from scratch.

Frontends

A frontend’s only job is translation: take some build definition and turn it into an LLB graph. The default frontend, dockerfile.v0, understands standard Dockerfile syntax. Because frontends are pluggable and themselves distributed as container images, BuildKit can support entirely different build definition languages — for example, a Nix-based frontend that produces images from Nix package definitions — without changing anything in the solver or executor beneath it.

The Solver

The solver is the scheduling brain. It receives the LLB graph, examines the dependency edges between operations, and determines which nodes can be executed in parallel because they do not depend on each other’s output. This is the architectural feature responsible for BuildKit’s headline speed improvement: multi-stage Dockerfiles with independent stages (for example, one stage building a frontend bundle and another compiling a backend binary) can now genuinely run at the same time instead of waiting on each other.

Executors and Snapshotters

Once the solver decides a node must actually run (rather than being served from cache), an executor spins up an isolated environment — using an OCI-compliant runtime under the hood — to run that step’s command. The snapshotter then captures exactly what changed on the filesystem as a result, storing it as a new layer-like snapshot that later steps, or the final image export, can build on.

LLB is best understood as a directed acyclic graph (DAG), where each node is a discrete operation — such as “run this command,” “copy these files,” or “pull this base image” — and each edge represents a dependency: node B cannot run until node A’s output is available. This is a fundamentally different data structure than a Dockerfile’s flat, ordered list of instructions.

Each node in the graph carries a content-based cache key, computed from its inputs: the exact command being run, the digest of the base filesystem it starts from, and the digests of any files being copied in. Before executing a node, the solver checks whether a result already exists in the cache for that exact key — locally, or in some setups, in a shared remote cache store. If it does, BuildKit reuses that cached result instantly, skipping execution entirely. This is far more precise than the classic builder’s layer cache, which mostly just compared instruction text and file timestamps.

Multi-stage builds map particularly cleanly onto this model: each FROM instruction in a Dockerfile effectively starts a new, potentially independent subgraph, and the solver is free to interleave execution across stages wherever their dependency edges allow it, rather than being forced to finish one stage completely before starting the next.

One of BuildKit’s most concrete security improvements over the classic builder is dedicated, first-class support for build secrets. Previously, passing a credential (like a private package registry token) into a build step meant embedding it in an instruction, where it risked being baked permanently into an image layer even if a later step deleted the file — because deleting a file in a later layer does not remove it from the earlier layer’s history. BuildKit’s --mount=type=secret mechanism mounts a secret into the build environment only for the duration of a single command, and it never becomes part of any layer at all.

BuildKit executors also run build steps inside isolated environments backed by the same namespace and cgroup isolation used by OCI-compliant container runtimes, and support rootless operation, letting an entire build run without the build daemon itself requiring root privileges on the host — significant for shared CI infrastructure where a compromised build should not automatically mean a compromised host.

BuildKit exposes structured, per-node build progress rather than a single scrolling text log, which is why modern build output shows a tree of concurrent steps with individual timers instead of one flat stream — this structure comes directly from the underlying LLB graph, where each node’s start and completion is tracked independently. Tools consuming this progress stream (including Docker’s own CLI, and CI dashboards built on top of buildx) can render genuinely parallel step timing rather than an artificial sequential log.

For fleet-level observability, buildkitd exposes Prometheus-compatible metrics covering build duration, cache hit ratios, and worker resource usage, letting platform teams running a shared, centralized build cluster monitor overall build health and cache effectiveness across every team using it, rather than each developer’s build performance being an isolated mystery.

Pattern: Multi-Stage Builds for Small Final Images

Using one build stage to compile or bundle an application with all its heavy build-time tooling, then copying only the final compiled artifact into a second, minimal stage, keeps the shipped image small while letting the solver still parallelize the two stages wherever their dependencies allow.

Pattern: Ordering Instructions by Change Frequency

Placing rarely-changing operations (installing system dependencies) before frequently-changing ones (copying application source) maximizes the number of graph nodes that hit BuildKit’s cache on each subsequent build, even though the underlying cache mechanism is more precise than the classic builder’s.

Anti-pattern: One Giant RUN Instruction

Chaining an entire setup process into a single massive RUN instruction collapses what could have been several independently cacheable graph nodes into one large, coarse-grained node — meaning a tiny change anywhere inside it invalidates the entire block’s cache at once.

Anti-pattern: Ignoring Remote Cache Export in CI

Running CI builds without configuring a shared remote cache (via --cache-to / --cache-from) means every CI run effectively starts cold, throwing away one of BuildKit’s biggest advantages over the classic builder in exactly the environment where it matters most.