Union Filesystems and How Docker Image Layers Stack

A Docker image is not a VM disk or a monolithic tarball. It’s a stack of thin, read-only layers plus one thin writable layer per container. The mechanism that stitches those layers into what looks like a single filesystem to your process is a union filesystem — and understanding it explains a surprising amount of Docker behavior: why rm in a RUN step doesn’t shrink your image, why five containers from one image cost almost nothing extra, and why your database is slow inside a container.

What a union filesystem actually does

A union filesystem mounts several directories so they appear as one merged view. With the overlay2 driver — the default on most Linux hosts — Docker lays out each container roughly like this:

• Lower layers (lowerdir): the image layers, stacked oldest at the bottom, newest on top. All read-only.
• Upper layer (upperdir): the container’s writable layer. Every write the container makes lands here.
• Merged view: the directory the container’s processes actually see as /.

When a process reads a file, overlay2 walks the layers from top to bottom and returns the first copy it finds. A newer layer shadows an older one. When it writes, the driver does a copy-on-write (CoW): it copies the file up into the writable top layer, then modifies that copy. The read-only layers below are never touched.

Every filesystem instruction is a layer

As the Docker docs put it, an image is built from a series of layers, and each layer represents an instruction in the Dockerfile. Instructions that modify the filesystem — RUN, COPY, ADD — produce a new layer. Metadata-only instructions — LABEL, ENV (in terms of file content), CMD, EXPOSE — don’t:

# syntax=docker/dockerfile:1

FROM ubuntu:22.04
LABEL org.opencontainers.image.authors="org@example.com"
COPY . /app
RUN make /app
RUN rm -r $HOME/.cache
CMD python /app/app.py

Here FROM, COPY, and both RUNs create layers. LABEL and CMD only change image metadata. Each layer stores only the difference from the layer below it — which is why layers are small compared to a full root filesystem.

The catch: deleting files doesn’t shrink the image

Each layer is a delta. The RUN rm -r $HOME/.cache step writes a deletion into a new top layer, but the cache files still exist in the layer created by the earlier RUN. The union view hides them; the bytes remain in the image. docker image history shows this — the make step carries a size, and the rm step adds its own layer on top.

This is the single most common source of bloated images. The fix isn’t clever rm commands; it’s ordering and multi-stage builds so throwaway build artifacts never make it into the final image’s layer stack.

Copy-on-write: cheap until it isn’t

When you start a container, Docker only creates the thin writable layer on top of the existing image layers — it doesn’t copy the image. That’s why container start is fast and why five idle containers from one image share 100% of the read-only data. docker ps --size makes it visible: freshly started containers report 0B in the size column with the image size shown as virtual.

The first write to a given file triggers copy_up. For overlay2 the sequence is: search the image layers newest-to-oldest for the file, copy the first hit into the writable layer, then apply the modification there. The docs note two things that trip people up:

• Only the first modification per file pays the cost. Subsequent writes hit the already-copied file in the upper layer.
• Metadata-only changes also trigger copy_up. Changing permissions or ownership on a large file in a lower layer duplicates that file into the writable layer.

For a web app writing logs this is invisible. For a database engine that reads and rewrites large existing files, the copy-up behavior on top of the read-only image layer is a real performance problem. That’s why Docker’s own guidance is: use volumes for write-intensive data, data that must outlive the container, and data shared between containers. Volumes live outside the union stack entirely and get native filesystem performance.

Layer sharing: why images stay small on disk

Layers are content-addressed and shared. If image B is built FROM image A, B contains A’s layers plus its own — and the daemon stores A’s layers only once. The same applies across unrelated images that happen to share a base: two containers from different images still share any common layers, so you can’t just total up virtual size to estimate disk usage.

The sharing extends to the network. When you docker pull, layers download individually and are stored under the daemon’s storage area (usually /var/lib/docker/, in a directory per driver such as overlay2/). If you already have a layer locally, it isn’t re-fetched. You can inspect the layer stack of any image:

$ docker image inspect --format "{{json .RootFS.Layers}}" my-app:1.0
[
  "sha256:72e830a4dff5f0d5225cdc0a320e85ab1ce06ea5673acfe8d83a7645cbd0e9cf",
  "sha256:07b4a9068b6af337e8b8f1f1dae3dd14185b2c0003a9a1f0a6fd2587495b204a",
  "sha256:cc644054967e516db4689b5282ee98e4bc4b11ea2255c9630309f559ab96562e"
]

Those digests are exactly what a registry serves and what the manifest references — the layer stack in the image and the blobs in the registry are the same thing, keyed by content hash.

A note on the containerd image store

Docker’s storage-driver docs carry an important caveat: Docker Engine 29.0 and later uses the containerd image store by default for fresh installations, which uses snapshotters instead of the classic storage drivers. The layer concepts on this page are unchanged, but the on-disk layout and some commands differ. If you’re on a fresh install of a recent Engine, treat overlay2 paths and ls /var/lib/docker/overlay2 as explanations of the classic model, and check which store you’re using before trusting any disk-forensics advice.

Practical gotchas

• rm in a RUN step is not a size optimization. It hides files in a new layer; the old bytes remain. Reorder the build or use a multi-stage build instead.
• COPY . . early kills your cache. Any change to the build context invalidates every layer from that point on. Copy dependency manifests and install first; copy source last.
• Write-heavy workloads belong on volumes. Database files, upload directories, and anything that gets rewritten in place will trigger copy-up overhead sitting on top of the image layer.
• The writable layer is per-container and dies with the container. Anything you need later — logs, artifacts, state — must go to a volume, bind mount, or external store, not the container filesystem.
• Deep stacks have lookup costs. Overlay2 lookups walk layers; very many layers (dozens-plus) can make file lookups slower. Conversely, too few layers forces expensive cache invalidation. Balance, don’t dogma.
• Container metadata isn’t free. docker ps --size reports the writable layer, but each container also consumes space for its config and log files under the daemon’s directories — logs especially, if rotation isn’t configured.

Sources

• Docker Engine storage
• Storage drivers
• containerd image store
• Volumes