Union Filesystem Layers

Delete a 50 MB file and a credentials file in the last RUN of your Dockerfile — then read the credentials back out of the shipped image, and find the 50 MB still sitting on your disk. An image is a stack of immutable diffs; a container is one thin layer on top.

Concept

A Docker image is not a disk image. It's a stack of tarballs — one per build step — and the filesystem your container sees is a union mount that overlays them, topmost wins. Every layer is immutable and content-addressed, so unrelated images share the ones they have in common. A running container adds exactly one thin writable layer on top, which is why docker diff can tell you what changed and why two containers from the same image cost almost nothing extra.

This exercise makes all of that literal. You'll build an image whose last RUN deletes a 50 MB file and a credentials file — and then read that credentials file back out of the shipped image. You'll watch a two-byte append cost 4.6 MB. And you'll end up looking at the actual kernel overlay mount, with its six lowerdir= paths and a character device standing in for a deleted file.

Mental model: a stack, a union, and a tombstone

Four terms carry the whole exercise:

The shape you're about to see on disk:

Deleting layer 3's file from layer 1 changes the top row. It does not change row 3.

Setup

One terminal is enough. You need Docker, and roughly 70 MB of free image space. Everything here is tagged layers-demo:* and named layers-* so you can clean up precisely at the end.

Work in a scratch directory, not in a repo:

mkdir -p /tmp/layers && cd /tmp/layers

Create Dockerfile. Note that each RUN is deliberately its own step, and the last one is the "cleanup" line you've seen in a hundred real Dockerfiles:

FROM alpine:3.20

RUN echo "build-id=1" > /etc/app-release

RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null

RUN echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials

RUN rm -f /opt/build-cache.bin /root/.aws-credentials

CMD ["sleep", "3600"]

/dev/urandom rather than zeros on purpose: incompressible bytes, so nothing downstream can quietly make the problem look smaller than it is.

Part 1 — one instruction, one layer

What this part tests: the claim that the image is a stack of per-step diffs, not a single blob. If that's true, docker history should show one row per instruction with a size attached, and those sizes should add up.

docker build -t layers-demo:leaky .
docker history layers-demo:leaky

docker image inspect layers-demo:leaky --format '{{.Size}}'

docker run --rm layers-demo:leaky sh -c 'ls -la /opt; cat /root/.aws-credentials'

Part 2 — the file you deleted is still in the image

What this part tests: the mental model's headline — a whiteout hides, it doesn't remove. The union view above is honest; it just isn't the whole artifact. docker save gives you the artifact, exactly as a registry would receive it.

mkdir -p save && docker save layers-demo:leaky -o save/leaky.tar
cd save && tar -xf leaky.tar
for f in blobs/sha256/*; do
  if tar -tf "$f" 2>/dev/null | grep -q 'aws-credentials'; then
    echo "FOUND IN: $f"
    tar -xOf "$f" root/.aws-credentials 2>/dev/null
  fi
done

tar -tvf blobs/sha256/33b9bea4...   # the layer that created the files
tar -tvf blobs/sha256/8c147ab8...   # the layer that "deleted" them

FROM alpine:3.20

RUN echo "build-id=1" > /etc/app-release

RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null \
 && echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials \
 && rm -f /opt/build-cache.bin /root/.aws-credentials

CMD ["sleep", "3600"]

docker build -f Dockerfile.fixed -t layers-demo:squashed .
docker image inspect layers-demo:squashed --format '{{.Size}}'

Part 3 — layers are content-addressed, and therefore shared

What this part tests: the claim that a layer is named by its contents. If so, two independently-built images that happen to produce identical layers should list the same digest, and Docker should store one copy.

docker image inspect layers-demo:leaky    --format '{{json .RootFS.Layers}}'
docker image inspect layers-demo:squashed --format '{{json .RootFS.Layers}}'

docker build -t layers-demo:leaky .           # unchanged, for contrast
sed -i '' 's/build-id=1/build-id=2/' Dockerfile
docker build --progress=plain -t layers-demo:rebuilt .

Part 4 — the writable layer is genuinely thin

What this part tests: that a running container adds only a diff. Per the mental model, the upperdir starts empty and gains one entry per file you touch — which is exactly the information docker diff reports.

docker run -d --name layers-box layers-demo:leaky sleep 3600
docker diff layers-box

docker exec layers-box sh -c '
  echo "build-id=2" > /etc/app-release
  mkdir -p /srv/data && echo hello > /srv/data/new.txt
  rm /etc/motd'
docker diff layers-box

docker ps -s --filter name=layers-box --format 'table {{.Names}}\t{{.Size}}'

docker run -d --name layers-box2 layers-demo:leaky sleep 3600
docker exec layers-box2 cat /etc/app-release
docker ps -s --filter name=layers-box --format '{{.Names}} {{.Size}}'

docker exec layers-box2 sh -c 'ls -l /lib/libcrypto.so.3'
docker exec layers-box2 sh -c 'echo x >> /lib/libcrypto.so.3'
docker ps -s --filter name=layers-box2 --format '{{.Names}} {{.Size}}'
docker diff layers-box2

Part 5 — look at the actual mount

What this part tests: everything above, at the level the kernel sees it. docker inspect will hand you the real paths.

docker inspect layers-box --format '{{json .GraphDriver}}'

docker inspect layers-box2 --format '{{.GraphDriver.Data.LowerDir}}' | tr ':' '\n'

docker run --rm --privileged --pid=host alpine \
  nsenter -t 1 -m -- <command>

docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- \
  find /var/lib/docker/overlay2/5cd5505b…/diff -exec ls -ld {} +

docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- \
  sh -c 'grep 5cd5505b /proc/mounts'

docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- sh -c \
  'for d in p3nn7a2jvcy7m982o8hidm1bx 1y7kex3p13fatta38d9kgg45w \
            1dp70kjleo19gjnqbcpoz793k 3srqfagieyzqzsxt0pcvln6vc; do
     echo "=== $d ==="
     find /var/lib/docker/overlay2/$d/diff -exec ls -ld {} +
   done'

docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- \
  cat /var/lib/docker/overlay2/1y7kex3p13fatta38d9kgg45w/diff/root/.aws-credentials

What you should see

A docker history whose per-layer sizes sum to exactly the image size (8,822,697 + 11 + 52,428,800 + 21 + 0 = 61,251,529), with the RUN rm step contributing 0B. A container that agrees the deleted files are gone, and a docker save tarball that hands you AKIAEXAMPLESECRETKEY anyway, alongside two zero-byte .wh. tombstones. The same build with one && instead of three RUNs at 8,822,708 bytes — 7× smaller for identical behavior. Two images sharing their first two layer digests character-for-character, and a one-character edit invalidating every layer above it. A running container whose writable layer is 17B (virtual 61.3MB), a second one at 0B, and a two-byte append that costs 4.6 MB. And finally a real overlay line in /proc/mounts with six lowerdirs, a character device at …/diff/etc/motd, and a 52,428,800-byte file alive in a layer directory that nothing can reach.

Why

Because the layer is the unit of transfer, and immutability is what makes transfer cheap. If a layer could be edited, its digest would change, every image referencing it would need updating, and the "already have that one" check that makes docker pull fast would have nothing stable to compare. So a layer is frozen the moment it's written, named by its contents, and shared by anyone whose stack happens to contain the same bytes. Everything else follows from refusing to break that.

Deletion is where the cost shows up. You cannot remove a file from a read-only layer, so the only honest thing an upper layer can do is record "stop looking below this path" — a whiteout. The union honors it, ls honors it, your application honors it. The registry does not, because the registry ships layers, not the union. RUN rm on a later line therefore buys you nothing but two tombstones, and this is not a Docker bug to be fixed; it is the direct price of the property that makes images shareable at all. The fix is always to arrange for the bytes never to be committed to a layer in the first place — one RUN, a multi-stage build, or a build mount.

Copy-up is the same trade in the other direction. Making an image layer writable per-container would mean copying it per-container, which would undo the sharing. So the layer stays read-only and the copy happens lazily, per file, on first write — near-zero cost for a container that mostly reads, and a full file copy the instant it isn't. 17B (virtual 61.3MB) and 4.6MB for two bytes are the two ends of that one mechanism.

Go deeper

• Rebuild :leaky as a multi-stage build — do the dd and the credentials work in a FROM alpine AS build stage, then FROM alpine and COPY --from=build only what you want. Check RootFS.Layers: the discarded stage contributes no layers at all, because it was never part of the final image's stack. This is the actual answer to Part 2's problem.
• Try RUN --mount=type=secret,id=creds cat /run/secrets/creds with docker build --secret id=creds,src=./creds. The file is present during the step and absent from the layer — verify with the Part 2 tarball scan and confirm nothing matches.
• Add WORKDIR, ENV, LABEL and EXPOSE lines and re-read docker history. All 0B: they're recorded in the image config (docker inspect --format '{{json .Config}}'), which is itself a blob in the tarball. The config plus the ordered layer list is the entire image — there is no other file.
• Write a file to a -v volume from inside layers-box and re-run docker diff. It doesn't appear, because a volume is a bind mount over the union, not part of it — which is the whole reason volume data survives docker rm.
• Count your machine's real layer reuse: docker system df reports images TOTAL and SIZE, and the gap between the sum of docker images sizes and that SIZE is layer sharing, in bytes.

Sources: Linux kernel: Overlay Filesystem — its "whiteouts and opaque directories" section defines the 0,0 character device you found · OCI Image Layer Specification for the .wh. tar convention · Docker Docs: the overlay2 storage driver

Cleanup

docker rm -f layers-box layers-box2
docker rmi layers-demo:leaky layers-demo:squashed layers-demo:rebuilt
rm -rf /tmp/layers