Storage Drivers Deep Dive: overlay2

How Docker's overlay2 driver stores image layers and container writes: lower, upper, merged, copy-up, on-disk anatomy, kernel options, and where it performs badly.

Published on • October 16, 2026

AI Assistant

Ask a Docker host what is holding its images together and the answer comes back in one line: Storage Driver: overlay2. That single string hides a union filesystem, a copy-on-write protocol, a directory tree under /var/lib/docker/overlay2, and a set of kernel features that you are better off not touching. This post unpacks all of it so you can read mount output, explain a slow write, and know when to move data out of the container layer entirely.

Note on versions: this covers the classic overlay2 storage driver. Docker Engine 29.0 and later uses the containerd image store (with the overlayfs snapshotter) by default on fresh installs; the layer concepts are the same, but the on-disk layout and some commands differ. Check where you stand with docker info.

Prerequisites and why overlay2 wins

Overlay2 needs Linux kernel 4.0 or higher, or the RHEL/CentOS 3.10.0-514 kernel or higher. It works on xfs backing filesystems only with d_type=true; verify with xfs_info that ftype is 1, and format with mkfs.xfs -n ftype=1 when creating the filesystem. Backing filesystems supported for overlay2 are xfs (ftype=1), ext4, btrfs, and a few others.

It is the default classic driver on Ubuntu, Debian, CentOS, Fedora, SLES 15, and RHEL because it needs no extra configuration, is the most widely compatible, and (in Docker’s own words) provides the highest stability. It also natively supports up to 128 lower layers, which is better for docker build and docker commit performance and consumes fewer inodes than stacking via symlinks.

Lower, upper, merged: the three views of every container

OverlayFS layers directories on one host and presents them as a single directory. The read-only stack (image layers) is lowerdir, the container’s writable layer is upperdir, and the unified view the container actually sees is merged. workdir is OverlayFS’s scratch space and must live on the same filesystem as upperdir.

$ mount | grep overlay
overlay on /var/lib/docker/overlay2/9186877c.../merged type overlay (rw,relatime,
lowerdir=l/DJA75GUWHWG7...:l/B3WWEFKBG3PL...:l/6Y5IM2XC7TSN...,
upperdir=9186877c.../diff,
workdir=9186877c.../work)

Reads are cheap and follow three rules: a file absent from the container layer comes from the image (minimal overhead); a file only in the container layer is read there; a file in both is read from the container layer, because upper always wins and obscures the same-named lower file.

On-disk anatomy under /var/lib/docker/overlay2

Pull a five-layer image and you get six directories plus a symlink farm:

$ ls /var/lib/docker/overlay2
223c28641754...  3a36935c9df3...  4e9fa83caff3...  e8876a226237...  eca1e4e16942...  l
$ ls -l /var/lib/docker/overlay2/l
6Y5IM2XC7TSNIJZZFLJCS6I4I4 -> ../3a36935c9df3.../diff
B3WWEFKBG3PLLV737KZFIASSW7 -> ../4e9fa83caff3.../diff

The l directory holds short symlinks to each layer’s diff; short names exist to avoid hitting the page size limit on mount arguments. Inside a layer directory:

diff/ holds that layer’s file contents.
link holds the short identifier used by the l/ symlink.
lower (second layer and up) names the parent layer, for example l/6Y5IM2XC7TSNIJZZFLJCS6I4I4.
merged/ and work/ are the union view and OverlayFS scratch space.

A running container’s own directory has a lower-id file pointing at the image’s top layer, plus upper/, merged/, and work/.

The layer content directories are only half the story. The daemon’s image database under /var/lib/docker/image/overlay2/layerdb/sha256/ maps logical chain IDs to physical storage: each entry holds cache-id (the directory name to look up under /var/lib/docker/overlay2/), diff, parent, size, and tar-split.json.gz. That is the bridge between docker inspect ... .RootFS.Layers and the folders on disk. Do not manipulate any of these files directly; they are managed by Docker.

Writes: copy-up, whiteouts, opaque directories

The first time a container writes to an existing file, overlay2 performs a copy_up: the whole file is copied from the image into the container’s upper layer, and the write lands there. Two properties matter in practice:

It happens once per file. Later writes hit the upper copy directly.
OverlayFS works at the file level, not the block level, so a 2 GB database file is copied in full even if you change one byte. Changing metadata (permissions, ownership) also triggers copy-up.

Deletion does not touch the read-only image at all. Deleting a file creates a whiteout in the upper layer; deleting a directory creates an opaque directory entry. The kernel implementation uses either a character device with 0/0 device numbers or a zero-size file carrying the trusted.overlay.whiteout xattr, and marks opaque directories with trusted.overlay.opaque=y.

Renaming is the trap. rename(2) on a directory is allowed only when both source and destination sit on the top layer; otherwise you get EXDEV (“cross-device link not permitted”). Applications must handle EXDEV and fall back to copy-and-unlink. Related POSIX wrinkle: open a file read-only, then open the same path read-write, and copy-up makes the two descriptors point at different files. touching files before the build (what yum-plugin-ovl does for rpm databases) forces the copy-up early.

metacopy and redirect_dir: why they stay off

Two kernel features would optimize overlayfs further, and a stock Docker setup deliberately runs without them.

metacopy (metadata-only copy-up) copies only metadata on chown/chmod and defers the data copy until a write-open, marking the upper file with a metacopy xattr. The kernel documentation is blunt: do not use metacopy=on with untrusted upper/lower directories, because handcrafted REDIRECT and METACOPY xattrs can point at files on lower layers; and metacopy=on conflicts with redirect_dir={off|nofollow|follow} and nfs_export=on. Docker’s documented behavior confirms the feature is not in play: copy-up copies the entire file.

redirect_dir lets overlayfs handle directory renames across layers by copying up the directory and stashing trusted.overlay.redirect instead of failing. With it disabled, rename(2) on a lower or merged directory returns EXDEV, which is exactly what Docker’s overlay2 docs describe as the supported behavior. Both features are toggled through kernel config (CONFIG_OVERLAY_FS_METACOPY, CONFIG_OVERLAY_FS_REDIRECT_DIR), module parameters, or per-mount options; leave them alone unless you know why you need them.

Performance characteristics

Page cache sharing. Multiple containers reading the same file share one page cache entry. This is why overlay2 is efficient in memory and a good fit for high-density use cases such as PaaS.
Copy-up latency. The first write to a large lower-layer file costs real time; subsequent writes do not. Large files, many layers, and deep directory trees all make lookups and copy-ups more expensive.
Backing filesystem. Format XFS with -n ftype=1; on ext4 (which has d_type enabled by default) nothing special is needed. Use SSDs. If you want per-container size caps on overlay2, docker run --storage-opt size=120G works only when the backing filesystem is XFS mounted with pquota.
Inodes. Each layer is a directory tree of its own. Busy hosts that build many images can exhaust inode budgets on ext4 long before they run out of bytes; watch df -i /var/lib/docker.
Volumes beat the driver for writes. Docker’s guidance is unambiguous: write-intensive data belongs in volumes, which bypass the storage driver and its copy-on-write overhead entirely.

Debugging disk usage

$ docker system df
TYPE                TOTAL   ACTIVE   SIZE       RECLAIMABLE
Images              5       2        16.43 MB   11.63 MB (70%)
Containers          2       0        212 B      212 B (100%)
Local Volumes       2       1        36 B       0 B (0%)

$ docker system df -v | less

-v breaks it down per image (SHARED SIZE, UNIQUE SIZE, which images share layers) and per container (writable-layer size). Follow up with the raw tree when one container looks fat:

$ sudo du -x -d 1 /var/lib/docker/overlay2 | sort -h | tail
$ sudo du -sh /var/lib/docker/containers/*
$ df -i /var/lib/docker

Remember that container size in docker ps -s counts only the writable layer, and virtual size double-counts shared image data across containers, so summing it overestimates real usage. Build cache is a separate consumer: docker builder prune reclaims it, docker system prune sweeps broadly.

When overlay2 is the wrong tool

overlay2 operates at file level, which is memory-efficient but means the writable layer can grow large under write-heavy load. Block-level drivers (btrfs, zfs) handle write-heavy workloads better, at the cost of more memory and setup, with zfs a good choice for high-density environments. vfs copies the whole image per container and is for debugging only. In practice, though, the answer is rarely “switch drivers”: put databases, logs, and build artifacts in named volumes and keep the writable layer for ephemeral scratch data.

Key Takeaways

overlay2 stacks read-only image layers as lowerdir, the container’s writable layer as upperdir, and exposes the union at merged; upper always wins on name collisions.
The first write to a file copies the entire file into the upper layer; deletes are recorded as whiteouts and opaque directories, never by editing image layers.
Directory rename() across layers returns EXDEV, and metacopy/redirect_dir stay off for good security and consistency reasons; handle EXDEV in your app.
XFS needs -n ftype=1, XFS + pquota enables --storage-opt size=, and docker info is where you verify driver, backing filesystem, and d_type support.
Write-heavy data belongs in volumes; docker system df -v, du on overlay2, and df -i are the debugging toolkit.

References

docker storage overlayfs linux performance