The engine is the working heart of Docker. Everything else, from the laptop app to the cloud services, sits on top of it.

The Docker Engine is the software that creates and runs containers on a computer. A container is an app plus all the files, libraries and settings it needs, wrapped up so that it runs in its own private space. The app inside believes it has the whole computer to itself, even though many containers may be sharing one machine.

The engine is a client-server program made of three parts. The first is a long-running background program called the daemon, whose command name is dockerd. A daemon is simply a program that keeps running quietly in the background and waits for work. The second is an API, which is a set of agreed messages that other programs can send to the daemon to ask for things. The third is the command-line client, the docker command you type in a terminal. When you type a command, the client turns it into an API message and sends it to the daemon, and the daemon does the real work.

The problem it was created to solve

Before containers, teams usually installed apps directly on servers. Two apps on the same server might need different versions of the same library, and they would fight each other. Moving an app from a developer laptop to a test server to a production server meant repeating a long list of setup steps, and someone always forgot one. Virtual machines helped, but they were heavy and slow to start. Containers gave teams a lighter, faster way to get the same repeatable result.

A short history you can remember

Why the ecosystem around it grew so quickly

Part of the reason Docker spread so fast is that it launched alongside Docker Hub, a public registry where anyone could publish a ready-made image for free. Instead of writing installation steps for a database or a web server from scratch, a developer could simply download an image someone else had already built, tested and shared. Within a few years, official images existed for almost every popular piece of software, and companies began publishing their own images for internal reuse. This network effect, an easy engine paired with a huge shared library of images, is a large part of why containers moved from a niche developer tool to something used by nearly every technology company within about a decade of that first public demonstration.

To understand why containers are popular, compare them with the older idea they often replace.

A virtual machine, or VM, is a whole pretend computer running inside a real one. A special layer of software called a hypervisor carves a physical server into several virtual servers. Each VM carries its own full operating system, its own memory and its own virtual hardware. That gives strong separation, but each VM may need several gigabytes of disk space and can take from tens of seconds to a few minutes to boot.

A container skips the pretend computer. All containers on a host share the kernel of the host’s operating system. The kernel is the core part of an operating system that talks to the hardware and decides which program gets the processor, memory and disk. Because there is no second operating system to boot, a container is usually just a few megabytes to a few hundred megabytes and can start in well under a second to a few seconds. A tiny base image such as Alpine Linux is only around five megabytes.

The best of both worlds

Containers and virtual machines are not enemies. Most cloud companies use both. Amazon’s AWS Fargate, for example, runs each customer task inside its own lightweight virtual machine built with a technology called Firecracker. The customer still thinks in containers, while AWS gets the strong wall of a VM between different customers. You will see this idea of layering again in the security chapter.

Eight small ideas explain almost everything in Docker. Learn these words and the rest becomes easy to follow.

Image versus container, the most important pair

Beginners often mix these up, so let us be very clear. An image never changes once it is built. It is only a file collection sitting on disk. When you ask the engine to run an image, it adds a thin writable layer on top and starts a process. That running package is the container. Delete the container and the image is still there, unchanged, ready to make a new one.

Programmers may recognise this pattern: an image is like a class, and a container is like an object created from that class. If you do not write code, the recipe-and-cake picture works just as well.

The engine is not one big lump. It is a chain of small programs, each with one clear job.

When you type a Docker command, your request travels through a short chain of programs before a container appears. The diagram below shows the chain from left to right.

flowchart LR
    CLI["Docker CLI: the docker command"] -->|"REST API calls"| API["Engine API: Unix socket or secured TCP"]
    API --> D["dockerd: the Docker daemon"]
    D --> BK["BuildKit: the image builder"]
    D --> NET["Networking: bridges and firewall rules"]
    D --> VOL["Volumes: persistent storage"]
    D -->|"gRPC"| C["containerd: container supervisor"]
    C --> SH["containerd-shim: one per container"]
    SH --> R["runc: OCI runtime"]
    R --> P["Your app process inside the container"]
    D -->|"pull and push images"| REG[("Registry: Docker Hub or Amazon ECR")]

Reading the diagram

On the left, the Docker CLI (command-line interface) is the program you type into. It does almost no real work. It packages your request and sends it over the Engine API. By default this message travels through a local Unix socket, which is a special file that programs on the same machine use to talk to each other. The engine can also listen on a network port, but that must be protected with encryption and login checks, as you will see in the security chapter.

The daemon, dockerd, is the manager. It keeps track of images, containers, networks and volumes. It also talks to registries when images must be downloaded or uploaded. But the daemon does not start containers by itself. It hands that job down the chain to a helper called containerd.

containerd (say “container-dee”) is a container supervisor. It pulls image content, stores it in a way the system can use, and manages the life of every container. It is an independent project that Kubernetes also uses directly. For each container it launches a tiny program called the shim. The shim stays alive as long as the container does. It keeps the container’s input and output connected and remembers the exit code when the container stops. Thanks to the shim, you can restart the daemon for an upgrade while containers keep running, if the live-restore setting is on.

runc is the worker at the very end. It follows the Open Container Initiative runtime standard. It asks the Linux kernel to build the private space, applies the safety limits, and starts your app. Once the app is running, runc’s job is done and it exits.

Other pieces around the core

One more point matters for the rest of this guide. Every link in the chain follows an open standard: the OCI image and runtime specifications. That is why an image built by Docker can run under containerd, Podman or Kubernetes, and why you are never locked into a single tool.

Containers feel like separate computers, but the trick is done with three ordinary Linux features working together.

A container is not magic. It is a normal Linux process that the kernel has been told to treat specially, using three ingredients: namespaces, control groups, and a security layer.

Namespaces: giving each container its own view

A namespace is a Linux feature that gives a process its own private view of one type of resource, while hiding everything else on the machine. There are several kinds. The PID namespace makes a container see only its own processes, so its first process looks like process number 1, even though the host sees it as, say, process number 4522. The network namespace gives the container its own private network card, IP address and routing table. The mount namespace gives it its own view of the file system, so it cannot see the host’s files unless they are deliberately shared. There are also namespaces for host names, for inter-process communication, and for user ID numbers.

Control groups: sharing resources fairly

Control groups, usually shortened to cgroups, are a Linux feature that limits and measures how much processor time, memory and disk activity a process may use. Without cgroups, one busy container could use all the memory on a server and starve every other container next to it. With a memory cgroup limit set, the kernel simply stops that container from allocating more, and the container may be safely stopped and restarted instead of freezing the whole machine.

A security layer on top

On top of namespaces and cgroups, the engine can apply extra rulebooks such as seccomp, which blocks dangerous low-level requests a container is allowed to make to the kernel, and AppArmor or SELinux, which restrict what files and actions a container’s processes may touch. These are covered in more depth in the security chapter.

Understanding layers explains why Docker builds and downloads are usually so fast after the first time.

An image is built from a text file called a Dockerfile. Each instruction in that file, such as copying in your code or installing a package, produces one read-only layer. The final image is simply an ordered stack of these layers, plus a small file called a manifest that lists them.

flowchart TB
    L1["Layer 1: base operating system files"] --> L2["Layer 2: system packages installed"]
    L2 --> L3["Layer 3: application dependencies"]
    L3 --> L4["Layer 4: application source code"]
    L4 --> L5["Writable container layer: created only when a container starts"]

Layers are stored once and reused. If ten different images all start from the same base layer, that layer is downloaded and stored on disk only once, and every image simply points to it. This is why pulling a second image that shares a base with one you already have often finishes in seconds instead of minutes.

Layers are also the reason build order matters. Docker checks whether an instruction and the files it depends on have changed since the last build. If nothing changed, it reuses the old layer instantly instead of redoing the work; this is called the build cache. Because of this, experienced teams place instructions that rarely change, like installing system packages, near the top of the Dockerfile, and instructions that change constantly, like copying in the latest source code, near the bottom. That way, an ordinary code change only rebuilds the last one or two layers.

How a container’s writable layer works

When a container starts, the engine adds one thin, writable layer on top of the image’s read-only layers, using a technique called copy-on-write. If the container’s process wants to change a file that exists in a lower, read-only layer, the storage driver copies that file up into the writable layer first, then changes the copy. The original layer, and every other container sharing it, is untouched. This is exactly why deleting a container does not affect the image or any other container built from it, and why many containers can share one image’s files in memory and on disk at the same time.

One image, many processor types

Modern images can also be multi-architecture, meaning a single image name actually points to several different builds behind the scenes, one for each processor family such as the common Intel and AMD design or the Arm design used in many phones and newer laptops. When you pull an image, the engine automatically selects the correct build for your own machine, using a small index file called a manifest list. This is what lets the same docker pull command work correctly whether it is run on a traditional data-centre server or on a newer Arm-based cloud instance, without the person typing the command needing to know or care which processor family is underneath.

Follow one command from your keyboard all the way to a running, and later a stopped, container.

sequenceDiagram
    participant U as User
    participant CLI as Docker CLI
    participant API as Engine API
    participant D as dockerd
    participant C as containerd
    participant R as runc
    U->>CLI: docker run myapp
    CLI->>API: POST create + start container
    API->>D: forward request
    D->>D: check image locally, pull if missing
    D->>C: create and start task
    C->>R: run OCI bundle
    R->>R: build namespaces and cgroups
    R-->>C: process started, runc exits
    C-->>D: container running
    D-->>API: success
    API-->>CLI: container ID
    CLI-->>U: prints container ID

When you run docker run myapp, the CLI first checks whether an image called myapp already exists on your machine. If not, the daemon downloads, or pulls, it from a registry, layer by layer, skipping any layers already present locally. The daemon then asks containerd to create a container from that image, containerd asks runc to actually start the process inside fresh namespaces and cgroups, and runc exits once the process is running, leaving the shim to watch over it.

The states a container passes through

Containers need to talk to each other and to the outside world. The engine gives each container its own virtual network card and several ways to connect them.

By default, the engine creates a private virtual network called bridge on the host. Every new container that does not ask for something different is attached to this bridge and given its own private IP address, similar to how a home Wi-Fi router hands out addresses to phones and laptops. Containers on the same bridge can reach each other by IP address, and if you create your own named bridge network, they can also reach each other simply by container name, because the engine runs a small internal DNS server for you.

Reaching the outside world

A container’s private IP address is not reachable from outside the host by default. To let outside traffic in, you publish a port, for example telling Docker to send anything arriving on the host’s port 8080 to port 80 inside the container. Behind the scenes, the engine sets up network address translation rules using the host’s firewall so that traffic is quietly redirected to the right container.

A less common but useful option is the macvlan driver, which gives a container its own genuine hardware-level address on the physical network, making it appear to the rest of the building’s network as if it were a separate physical device rather than something hidden behind the host. This is handy for older applications that expect to own a real network address directly, but it is used far less often than the default bridge network, since it also means the host’s usual firewall protections no longer sit between the container and the network.

Containers are meant to be thrown away and rebuilt often, so anything that must survive needs a home outside the container.

A volume is a storage area managed entirely by the Docker Engine, stored on the host’s disk but kept outside any single container’s writable layer. When a container writes to a location where a volume is attached, that data is really being written into the volume, and it stays there even after the container is stopped, removed, or replaced by a newer version.

Volumes can also be shared between several containers at once, which is how, for example, a web server container and a separate log-collecting container can both read and write the same log files without copying them back and forth.

Volume drivers and cloud storage

On a single machine, a volume is normally just a managed folder on the local disk. However, the engine also supports pluggable volume drivers, which let a volume actually be backed by something other than local disk, such as a network file share or a cloud storage service. This matters most once containers move across several machines in a cluster: if a container with important data is rescheduled from one server to another, a local-only volume would be left behind, but a volume backed by shared network or cloud storage can follow the container to its new home. This is one of the details orchestrators such as Kubernetes handle for you through their own storage systems, building on the same underlying volume idea introduced here.

No technology is free of trade-offs. Knowing the honest downsides makes you a stronger engineer, not a weaker one.

Because containers share a kernel, security has to be built in layers rather than relied on as one single wall.

Built-in defences

Practices every team should follow

• Run the process inside the container as a non-root user whenever the application allows it.
• Keep base images small and current, since fewer installed packages means fewer known vulnerabilities to patch.
• Scan images for known vulnerabilities before they are deployed, using a vulnerability scanner as part of the build pipeline.
• Sign and verify images so a team can trust that the image running in production is exactly the one that was built and tested.
• Never mount the host’s own Docker socket into a container casually, since a process with access to it can effectively control every container on that host.
• Protect the Engine API. If it must be reachable over the network, always require mutual TLS encryption and authentication rather than leaving it open.

You cannot fix what you cannot see. The engine gives you basic visibility out of the box, and larger systems build much more on top.

Every container’s standard output and standard error, the two text streams a program normally prints to, are captured automatically by the engine through a logging driver. The default driver simply writes these streams to local files in JSON format, which the docker logs command reads back for you. In real production systems, teams usually switch to a driver that forwards logs straight to a central logging platform, so that logs from hundreds of containers across many servers can be searched in one place instead of one host at a time.

For live resource usage, the docker stats command shows processor use, memory use, network traffic and disk activity for running containers, pulled directly from the same cgroup counters the kernel already keeps. This is useful for a quick manual check, but production systems usually export these numbers continuously to a metrics platform such as Prometheus, so trends can be graphed and alerts can fire automatically when something drifts out of range.

As the number of containers grows, teams usually add one more layer of visibility on top of logs and metrics: distributed tracing. A single user request in a modern application might pass through a dozen small services, each running in its own container, before a response is sent back. Tracing tools attach a shared identifier to that request as it travels, so that when something is slow or fails, an engineer can see the exact path the request took across every container involved, rather than searching through a dozen separate log files by hand and guessing which ones are related.

A single Docker Engine on one laptop is where most people start. Real products usually run across many machines, coordinated by an orchestrator.

An orchestrator is a system that decides which container runs on which machine, restarts containers that fail, and spreads traffic across many copies of the same app. Docker ships its own simple orchestrator, called swarm mode, built directly into the engine. Most large organisations instead choose Kubernetes, a separate, more feature-rich orchestrator that has become the industry standard, even though it no longer talks to the Docker Engine directly and instead uses containerd or a similar runtime underneath.

flowchart TB
    Dev["Developer writes Dockerfile"] --> Build["CI pipeline builds image"]
    Build --> Push["Image pushed to a registry"]
    Push --> Orc["Orchestrator schedules containers across a cluster"]
    Orc --> N1["Node 1: containerd + runc"]
    Orc --> N2["Node 2: containerd + runc"]
    Orc --> N3["Node 3: containerd + runc"]

Every major cloud provider offers managed container services built on these same open standards. Amazon Web Services offers Amazon ECS, its own orchestrator, and Amazon EKS, a managed version of Kubernetes, alongside AWS Fargate, which runs containers without the customer having to manage any servers at all. Google Cloud and Microsoft Azure offer comparable managed Kubernetes services of their own. In every case, the image you built with the Docker Engine on your laptop is the same image these cloud services run, because they all speak the same open container format.

GitOps: letting the cluster read its own instructions

Many teams take this one step further with an approach called GitOps. Instead of an engineer running deployment commands by hand, the desired state of the whole system, which images should be running and how many copies of each, is written down as plain configuration files stored in a version-controlled repository, the same kind of system used to store source code. A small automated controller inside the cluster continuously compares that stored description with what is actually running, and quietly corrects any difference it finds. This means every change to production has a written history, can be reviewed before it happens, and can be undone simply by reverting the configuration file to an earlier version.

Certain shapes of using containers work well again and again. Others cause pain that looks different every time but has the same root cause.

A short, practical checklist that catches most of the problems beginners run into.

These ideas are not theoretical. Some of the largest technology companies in the world run on containers every single day.

A common thread across all of these companies is the same one described at the start of this guide: containers let a huge number of independent teams package their own piece of software once and trust that it will run the same way everywhere it is deployed, from a laptop to a global fleet of servers.