# Docker: union filesystem layers ## Concept A Docker image is not a disk image. It's a *stack of tarballs* — one per build step — and the filesystem your container sees is a union mount that overlays them, topmost wins. Every layer is immutable and content-addressed, so unrelated images share the ones they have in common. A running container adds exactly one thin writable layer on top, which is why `docker diff` can tell you what changed and why two containers from the same image cost almost nothing extra. This exercise makes all of that literal. You'll build an image whose last `RUN` deletes a 50 MB file and a credentials file — and then read that credentials file back out of the shipped image. You'll watch a two-byte append cost 4.6 MB. And you'll end up looking at the actual kernel `overlay` mount, with its six `lowerdir=` paths and a character device standing in for a deleted file. ## Mental model: a stack, a union, and a tombstone Four terms carry the whole exercise: | Term | What it actually is | | --- | --- | | **Layer** | A tar archive of the filesystem *changes* one build step made — files added or modified, plus tombstones for files removed. Immutable once written, and named by the SHA-256 of its contents (its *diff ID*), which is what makes sharing possible. | | **Union / overlay mount** | The kernel stacking those layers into one directory tree. In overlay2's vocabulary: the read-only image layers are `lowerdir`s, the container's writable layer is the `upperdir`, and the combined view is the `merged` directory — which is what becomes the container's `/`. | | **Whiteout** | How you delete a file you don't own. Since lower layers are read-only, an upper layer records a marker at that path — a character device with major/minor `0,0` on disk, or a `.wh.` entry inside a layer tarball — and the union hides everything below it. | | **Copy-up** | How you *modify* a file you don't own. On first write, overlayfs copies the entire file from the lower layer into the upper layer, then edits the copy. Costs are per-file, not per-byte-changed. | The consequence that costs people real money: > **A later layer can hide a file, but it can never remove it.** The bytes > stay in the earlier layer forever, and that layer still ships. `RUN rm` > in its own step reduces your image size by zero and reduces the > recoverability of your secret by zero. The shape you're about to see on disk: ``` container / ← what your process sees (the "merged" view) ═══════════════════════════════════════════ upperdir writable layer [C /etc/app-release] [A /srv/data] [D /etc/motd] ─────────────────────────────────────────── ← container starts here lowerdir 1 RUN rm ... whiteouts only, 0 B lowerdir 2 RUN echo creds /root/.aws-credentials, 21 B lowerdir 3 RUN dd 50MB /opt/build-cache.bin, 52,428,800 B lowerdir 4 RUN echo build /etc/app-release, 11 B lowerdir 5 FROM alpine:3.20 the whole distro, 8.82 MB ``` Deleting layer 3's file from layer 1 changes the top row. It does not change row 3. ## Setup One terminal is enough. You need Docker, and roughly 70 MB of free image space. Everything here is tagged `layers-demo:*` and named `layers-*` so you can clean up precisely at the end. Work in a scratch directory, not in a repo: ```bash mkdir -p /tmp/layers && cd /tmp/layers ``` Create `Dockerfile`. Note that each `RUN` is deliberately its own step, and the last one is the "cleanup" line you've seen in a hundred real Dockerfiles: ```dockerfile FROM alpine:3.20 RUN echo "build-id=1" > /etc/app-release RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null RUN echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials RUN rm -f /opt/build-cache.bin /root/.aws-credentials CMD ["sleep", "3600"] ``` `/dev/urandom` rather than zeros on purpose: incompressible bytes, so nothing downstream can quietly make the problem look smaller than it is. Transcripts below come from Docker Engine 29.5.2, `linux/arm64`, storage driver `overlay2`. Digests and layer directory IDs are per-machine — yours will differ, and you'll substitute your own throughout Part 5. ## Steps ### Part 1 — one instruction, one layer What this part tests: the claim that the image is a stack of per-step diffs, not a single blob. If that's true, `docker history` should show one row per instruction with a size attached, and those sizes should add up. 1. **Predict**: the Dockerfile has five instructions after `FROM`. How many rows will `docker history` show, and what size will the last `RUN` (the `rm`) report? 2. Build and look: ```bash docker build -t layers-demo:leaky . docker history layers-demo:leaky ``` 3. **Observe**: ``` IMAGE CREATED CREATED BY SIZE COMMENT 53d3c1c45693 5 seconds ago CMD ["sleep" "3600"] 0B buildkit.dockerfile.v0 5 seconds ago RUN /bin/sh -c rm -f /opt/build-cache.bin /r… 0B buildkit.dockerfile.v0 5 seconds ago RUN /bin/sh -c echo "AKIAEXAMPLESECRETKEY" >… 21B buildkit.dockerfile.v0 6 seconds ago RUN /bin/sh -c dd if=/dev/urandom of=/opt/bu… 52.4MB buildkit.dockerfile.v0 6 seconds ago RUN /bin/sh -c echo "build-id=1" > /etc/app-… 11B buildkit.dockerfile.v0 3 months ago CMD ["/bin/sh"] 0B buildkit.dockerfile.v0 3 months ago ADD alpine-minirootfs-3.20.10-aarch64.tar.gz… 8.82MB buildkit.dockerfile.v0 ``` Seven rows: your five plus the two the base image contributed — history is inherited, so you're reading `alpine`'s build steps at the bottom. `CMD` rows are `0B` because they change metadata, not files. And the `rm` step is `0B` too, which is the first hint of the whole lesson. 4. Now check the arithmetic: ```bash docker image inspect layers-demo:leaky --format '{{.Size}}' ``` → `61251529` The layer sizes are `8,822,697 + 11 + 52,428,800 + 21 + 0`, which is `61,251,529` exactly. The image's size *is* the sum of its layers' diffs; there's nothing else in there. 5. **Predict**: the last `RUN` deleted both files. From inside a container, is `/opt/build-cache.bin` there? 6. Look at the union view: ```bash docker run --rm layers-demo:leaky sh -c 'ls -la /opt; cat /root/.aws-credentials' ``` 7. **Observe**: ``` total 8 drwxr-xr-x 1 root root 4096 Jul 22 15:55 . drwxr-xr-x 1 root root 4096 Jul 22 15:55 .. cat: can't open '/root/.aws-credentials': No such file or directory ``` Both gone, exactly as a filesystem should behave. The image is still 61 MB. ### Part 2 — the file you deleted is still in the image What this part tests: the mental model's headline — a whiteout hides, it doesn't remove. The union view above is honest; it just isn't the whole artifact. `docker save` gives you the artifact. 8. **Predict**: `docker save` writes the image exactly as it would be pushed to a registry. Can you get `AKIAEXAMPLESECRETKEY` out of it? 9. Unpack it and grep the layers: ```bash mkdir -p save && docker save layers-demo:leaky -o save/leaky.tar cd save && tar -xf leaky.tar for f in blobs/sha256/*; do if tar -tf "$f" 2>/dev/null | grep -q 'aws-credentials'; then echo "FOUND IN: $f" tar -xOf "$f" root/.aws-credentials 2>/dev/null fi done ``` 10. **Observe**: ``` FOUND IN: blobs/sha256/33b9bea4ef890bff4fcf2cd6fcd24261cea15040f2f27e81450bbae8fa0adf90 AKIAEXAMPLESECRETKEY tar: root/.aws-credentials: Not found in archive ``` Two layers matched the *name*; only one has the file. The credential is plaintext in a layer that ships with the image, retrievable by anyone who can pull it — no exploit, no privilege, just `tar`. 11. Look at what the two matching layers actually contain: ```bash tar -tvf blobs/sha256/33b9bea4... # the layer that created the files tar -tvf blobs/sha256/8c147ab8... # the layer that "deleted" them ``` 12. **Observe**: ``` --- 33b9bea… : the layer that created the files --- drwxr-xr-x 0 0 0 0 Jul 22 23:55 etc/ drwx------ 0 0 0 0 Jul 22 23:55 root/ -rw-r--r-- 0 0 0 21 Jul 22 23:55 root/.aws-credentials --- 8c147ab… : the layer that "deleted" them --- drwxr-xr-x 0 0 0 0 Jul 22 23:55 etc/ drwxr-xr-x 0 0 0 0 Jul 22 23:55 opt/ -rw------- 0 0 0 0 Jul 22 23:55 opt/.wh.build-cache.bin drwx------ 0 0 0 0 Jul 22 23:55 root/ -rw------- 0 0 0 0 Jul 22 23:55 root/.wh..aws-credentials ``` There it is, in plain sight: the `rm` layer contains two **zero-byte files whose names begin with `.wh.`**. That is what a delete is in an image. `rm` didn't produce a smaller image, it produced a *slightly larger* one — two extra tombstones on top of everything it was hiding. (`root/.wh..aws-credentials` has two dots because the original filename already started with one: prefix `.wh.` plus `.aws-credentials`.) 13. **Predict**: move all four commands into a single `RUN` chained with `&&`, so the file is created and deleted inside one step. Does the image get smaller? 14. Build the alternative as `Dockerfile.fixed`: ```dockerfile FROM alpine:3.20 RUN echo "build-id=1" > /etc/app-release RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null \ && echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials \ && rm -f /opt/build-cache.bin /root/.aws-credentials CMD ["sleep", "3600"] ``` ```bash docker build -f Dockerfile.fixed -t layers-demo:squashed . docker image inspect layers-demo:squashed --format '{{.Size}}' ``` 15. **Observe**: `8822708`. | | `:leaky` (rm in its own layer) | `:squashed` (one RUN) | | --- | --- | --- | | Image size | 61,251,529 B (61.3 MB) | 8,822,708 B (8.82 MB) | | Layers | 5 | 3 | | 50 MB cache file on disk | yes, layer 3 | no | | Secret recoverable from image | yes | no | Byte-identical container filesystems, 7× the size. The commands that ran are the same, in the same order, producing the same result. The only difference is *where the layer boundaries fell*, and layer boundaries are the unit at which content becomes permanent. Its `docker history` shows the combined step as `0B` — because that step's net diff really is nothing: the file it created never outlived the step, so no tombstone was needed either. ### Part 3 — layers are content-addressed, and therefore shared What this part tests: the claim that a layer is named by its contents. If so, two independently-built images that happen to produce identical layers should list *the same digest*, and Docker should store one copy. 16. **Predict**: `:leaky` and `:squashed` are separate images built from separate Dockerfiles. How many of their layer digests match? 17. Compare: ```bash docker image inspect layers-demo:leaky --format '{{json .RootFS.Layers}}' docker image inspect layers-demo:squashed --format '{{json .RootFS.Layers}}' ``` 18. **Observe**: ``` leaky: squashed: 88b4fba61c4c714a2fc173ddf7e9324a… 88b4fba61c4c714a2fc173ddf7e9324a… ← same 891abf759b795a947180c4a402e06377c… 891abf759b795a947180c4a402e06377c… ← same 8c35f8546e65b2ec544623a065bfac1ba… 7dc425ddeb23c628fc015249884decf6… 33b9bea4ef890bff4fcf2cd6fcd24261c… 8c147ab8f86c88e8d6420070a4ee8b9f8… ``` The first two are identical strings: the alpine rootfs, and the `echo "build-id=1"` layer. Nothing coordinated that — the two builds produced byte-identical diffs, so they hash to the same name, so they *are* the same layer. Pull a hundred alpine-based images and you download `88b4fba…` once. 19. **Predict**: rebuild `:leaky` unchanged. Then change *only* the first `RUN` to `build-id=2` and rebuild. Which steps re-run? 20. Try both: ```bash docker build -t layers-demo:leaky . # unchanged sed -i '' 's/build-id=1/build-id=2/' Dockerfile docker build --progress=plain -t layers-demo:rebuilt . ``` 21. **Observe** — the unchanged build: ``` #5 [2/5] RUN /bin/sh -c echo "build-id=1" > /etc/app-release #5 CACHED #6 [3/5] RUN /bin/sh -c dd if=/dev/urandom of=/opt/build-cache.bin … #6 CACHED #7 [4/5] RUN /bin/sh -c echo "AKIAEXAMPLESECRETKEY" > /root/.aws-… #7 CACHED #8 [5/5] RUN /bin/sh -c rm -f /opt/build-cache.bin /root/.aws-cred… #8 CACHED ``` and after the one-character edit: ``` #4 [1/5] FROM docker.io/library/alpine:3.20@sha256:d9e853e87e55526f… #4 CACHED #5 [2/5] RUN /bin/sh -c echo "build-id=2" > /etc/app-release #5 DONE 0.1s #6 [3/5] RUN /bin/sh -c dd if=/dev/urandom of=/opt/build-cache.bin … #6 DONE 0.3s #7 [4/5] RUN /bin/sh -c echo "AKIAEXAMPLESECRETKEY" > /root/.aws-… #7 DONE 0.1s #8 [5/5] RUN /bin/sh -c rm -f /opt/build-cache.bin /root/.aws-cred… #8 DONE 0.2s ``` Steps 3, 4 and 5 have identical text and identical inputs from your point of view, and all three re-ran. The digests confirm it — only the base layer survives: ``` leaky: rebuilt: 88b4fba61c4c714a2fc173ddf7e9324a… 88b4fba61c4c714a2fc173ddf7e9324a… ← same 891abf759b795a947180c4a402e06377c… 2134c0a566affbfe97d243d9881e2cf8… 8c35f8546e65b2ec544623a065bfac1ba… b23fa4651b8337aa627b457e795d8ad3… 33b9bea4ef890bff4fcf2cd6fcd24261c… 65702316331fc7dcbd7bb33847e7eb48… 8c147ab8f86c88e8d6420070a4ee8b9f8… 502e066b81faad051a7c4bf2ab27db71… ``` A layer's cache key includes the layer it was built *on*. Change one, and every layer above it is a different stack, so none of them can be reused. That's the reason `COPY package.json` comes before `COPY . .` in every well-written Node Dockerfile: put the thing that changes every commit as high in the stack as you can. ### Part 4 — the writable layer is genuinely thin What this part tests: that a running container adds only a diff. Per the mental model, the `upperdir` starts empty and gains one entry per file you touch — which is exactly the information `docker diff` reports. 22. Start a container and diff it before touching anything: ```bash docker run -d --name layers-box layers-demo:leaky sleep 3600 docker diff layers-box ``` 23. **Observe**: nothing at all. Empty output. A container that has done no writes has an empty writable layer — the process is running on a stack of read-only layers and a mount point. 24. **Predict**: modify one existing file, create one new file, delete one existing file. What three letters does `docker diff` report, and what does it say about the *directories* containing them? 25. Do it: ```bash docker exec layers-box sh -c ' echo "build-id=2" > /etc/app-release mkdir -p /srv/data && echo hello > /srv/data/new.txt rm /etc/motd' docker diff layers-box ``` 26. **Observe**: ``` C /srv A /srv/data A /srv/data/new.txt C /etc C /etc/app-release D /etc/motd ``` `A`dded, `C`hanged, `D`eleted. Note `/srv` and `/etc` are `C`, not `A` — they already existed in the image; the writable layer had to materialize its own copy of each directory to hold the new entries. That's copy-up applied to directories. 27. **Predict**: you wrote 17 bytes of content. What does Docker report as the container's size? 28. ```bash docker ps -s --filter name=layers-box --format 'table {{.Names}}\t{{.Size}}' ``` 29. **Observe**: ``` NAMES SIZE layers-box 17B (virtual 61.3MB) ``` `17B` is the writable layer — `11` for `app-release` plus `6` for `new.txt`; the tombstone for `/etc/motd` is free. `virtual` is that plus the shared image. Start a second container and it costs `0B`: ```bash docker run -d --name layers-box2 layers-demo:leaky sleep 3600 docker exec layers-box2 cat /etc/app-release docker ps -s --filter name=layers-box --format '{{.Names}} {{.Size}}' ``` ``` build-id=1 layers-box2 0B (virtual 61.3MB) layers-box 17B (virtual 61.3MB) ``` Two containers reporting 61.3 MB each; 17 bytes of actual new disk between them. And `layers-box2` still sees `build-id=1` — the first container's edit lives in *its* upperdir, invisible to anyone else, because the layer they share is read-only for both. 30. **Predict**: now append two bytes to a 4.6 MB file that came from the image. What happens to the writable layer? 31. ```bash docker exec layers-box2 sh -c 'ls -l /lib/libcrypto.so.3' docker exec layers-box2 sh -c 'echo x >> /lib/libcrypto.so.3' docker ps -s --filter name=layers-box2 --format '{{.Names}} {{.Size}}' docker diff layers-box2 ``` 32. **Observe**: ``` -rwxr-xr-x 1 root root 4597896 Apr 9 22:19 /lib/libcrypto.so.3 layers-box2 4.6MB (virtual 65.8MB) C /lib C /lib/libcrypto.so.3 ``` Two bytes written, 4.6 MB consumed. Copy-up has no notion of a partial file: to make one byte writable, overlayfs copies all 4,597,896 of them into the upper layer first. This is the mechanism behind "why did my container balloon overnight" — a log rotation or an in-place edit of large image files, not new data. It's also why databases in containers want a volume: a volume bypasses the union entirely. ### Part 5 — look at the actual mount What this part tests: everything above, at the level the kernel sees it. `docker inspect` will hand you the real paths. 33. ```bash docker inspect layers-box --format '{{json .GraphDriver}}' ``` 34. **Observe** (reformatted): ``` "Name": "overlay2", "LowerDir": "/var/lib/docker/overlay2/5cd5505b…-init/diff :/var/lib/docker/overlay2/p3nn7a2jvcy7m982o8hidm1bx/diff :/var/lib/docker/overlay2/1y7kex3p13fatta38d9kgg45w/diff :/var/lib/docker/overlay2/1dp70kjleo19gjnqbcpoz793k/diff :/var/lib/docker/overlay2/3srqfagieyzqzsxt0pcvln6vc/diff :/var/lib/docker/overlay2/f9488352a1dfcbea…/diff", "MergedDir": "/var/lib/docker/overlay2/5cd5505b…/merged", "UpperDir": "/var/lib/docker/overlay2/5cd5505b…/diff", "WorkDir": "/var/lib/docker/overlay2/5cd5505b…/work" ``` Six lower directories, topmost first: a per-container `-init` layer (where Docker stakes `/etc/hosts` and friends), then your five image layers in reverse order. One upper. That colon-separated string is passed straight to `mount -t overlay`. Compare with `layers-box2`: ```bash docker inspect layers-box2 --format '{{.GraphDriver.Data.LowerDir}}' | tr ':' '\n' ``` The five image directories are character-for-character the same paths; only the `-init` and upper differ. "Sharing a layer" means literally mounting the same directory into two containers. **On macOS/Windows, those paths are inside the Docker VM**, not on your machine — `ls /var/lib/docker` from your shell will fail. On native Linux, just `sudo ls` them and skip the wrapper below. Everywhere else, reach into the VM's mount namespace: ```bash docker run --rm --privileged --pid=host alpine \ nsenter -t 1 -m -- ``` `--pid=host` puts you in the VM's PID namespace so PID 1 is its init; `nsenter -t 1 -m` then enters that process's *mount* namespace, which is where `/var/lib/docker` lives. **Substitute your own directory IDs** from the `docker inspect` output above — they are randomly generated per machine. 35. **Predict**: `docker diff` reported six paths for `layers-box`, including `D /etc/motd`. What is physically in the `UpperDir`? 36. ```bash docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- \ find /var/lib/docker/overlay2/5cd5505b…/diff -exec ls -ld {} + ``` 37. **Observe** (paths abbreviated): ``` drwxr-xr-x 4 root root 4096 …/diff drwxr-xr-x 2 root root 4096 …/diff/etc -rw-r--r-- 1 root root 11 …/diff/etc/app-release c--------- 2 root root 0, 0 …/diff/etc/motd drwxr-xr-x 3 root root 4096 …/diff/srv drwxr-xr-x 2 root root 4096 …/diff/srv/data -rw-r--r-- 1 root root 6 …/diff/srv/data/new.txt ``` Six entries for a 61 MB filesystem. And look at `/etc/motd`: leading `c`, size `0, 0`. That is a **character device with major and minor both zero** — the on-disk whiteout convention. Deleting a file added something. The `D` in `docker diff` is Docker translating this back into a word. 38. The kernel's own view, to prove none of this is a Docker abstraction: ```bash docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- \ sh -c 'grep 5cd5505b /proc/mounts' ``` ``` overlay /var/lib/docker/overlay2/5cd5505b…/merged overlay rw,relatime, lowerdir=/var/lib/docker/overlay2/l/DZ2KQHZZEJ45FLLCCAJ5C76FIT :/var/lib/docker/overlay2/l/4K57F42FM4YJYIMLIWNIYX55OO :/var/lib/docker/overlay2/l/7R6Q2D6DTQY25ZCIEHY72XSMCH :/var/lib/docker/overlay2/l/AZPWFUXZRP4C44DBAOXUNEYHDB :/var/lib/docker/overlay2/l/SP6LUSCCBQ43LVGIUMLMFDXXRT :/var/lib/docker/overlay2/l/7XATOXVVNXECXOSKERCYNBHVWW, upperdir=/var/lib/docker/overlay2/5cd5505b…/diff, workdir=/var/lib/docker/overlay2/5cd5505b…/work 0 0 ``` A plain `overlay` mount in `/proc/mounts`, indistinguishable from one you'd make by hand. Docker's whole "container filesystem" is this single line. (The `l/` paths are short symlinks Docker keeps so the option string stays under the kernel's page-sized mount-data limit — a real constraint you hit around 120 layers.) 39. **Predict**: walk the image layers now, in the same order. `1dp70kjleo…` is the `dd` layer, `p3nn7a2jvc…` the `rm` layer. Is the 50 MB file on your disk right now? 40. ```bash docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- sh -c \ 'for d in p3nn7a2jvcy7m982o8hidm1bx 1y7kex3p13fatta38d9kgg45w \ 1dp70kjleo19gjnqbcpoz793k 3srqfagieyzqzsxt0pcvln6vc; do echo "=== $d ===" find /var/lib/docker/overlay2/$d/diff -exec ls -ld {} + done' ``` 41. **Observe** (directory lines trimmed; every layer also carries the parent directories of the files it touched): ``` === p3nn7a2jvcy7m982o8hidm1bx === ← RUN rm c--------- 3 root root 0, 0 …/diff/opt/build-cache.bin c--------- 3 root root 0, 0 …/diff/root/.aws-credentials === 1y7kex3p13fatta38d9kgg45w === ← RUN echo …credentials -rw-r--r-- 1 root root 21 …/diff/root/.aws-credentials === 1dp70kjleo19gjnqbcpoz793k === ← RUN dd 50MB -rw-r--r-- 1 root root 52428800 …/diff/opt/build-cache.bin === 3srqfagieyzqzsxt0pcvln6vc === ← RUN echo build-id -rw-r--r-- 1 root root 11 …/diff/etc/app-release ``` All 52,428,800 bytes, sitting in a directory on your disk, under a path no container can see. And one command finishes the story: ```bash docker run --rm --privileged --pid=host alpine nsenter -t 1 -m -- \ cat /var/lib/docker/overlay2/1y7kex3p13fatta38d9kgg45w/diff/root/.aws-credentials ``` ``` AKIAEXAMPLESECRETKEY ``` Same secret you pulled out of the tarball in Part 2, now read straight off the host filesystem, from an image whose every container reports the file as missing. ## What you should see A `docker history` whose per-layer sizes sum to exactly the image size (`8,822,697 + 11 + 52,428,800 + 21 + 0 = 61,251,529`), with the `RUN rm` step contributing `0B`. A container that agrees the deleted files are gone, and a `docker save` tarball that hands you `AKIAEXAMPLESECRETKEY` anyway, alongside two zero-byte `.wh.` tombstones. The same build with one `&&` instead of three `RUN`s at `8,822,708` bytes — 7× smaller for identical behavior. Two images sharing their first two layer digests character-for-character, and a one-character edit invalidating every layer above it. A running container whose writable layer is `17B (virtual 61.3MB)`, a second one at `0B`, and a two-byte append that costs 4.6 MB. And finally a real `overlay` line in `/proc/mounts` with six `lowerdir`s, a character device at `.../diff/etc/motd`, and a 52,428,800-byte file alive in a layer directory that nothing can reach. ## Why Because the layer is the unit of *transfer*, and immutability is what makes transfer cheap. If a layer could be edited, its digest would change, every image referencing it would need updating, and the "already have that one" check that makes `docker pull` fast would have nothing stable to compare. So a layer is frozen the moment it's written, named by its contents, and shared by anyone whose stack happens to contain the same bytes. Everything else follows from refusing to break that. Deletion is where the cost shows up. You cannot remove a file from a read-only layer, so the only honest thing an upper layer can do is record "stop looking below this path" — a whiteout. The union honors it, `ls` honors it, your application honors it. The registry does not, because the registry ships layers, not the union. `RUN rm` on a later line therefore buys you nothing but two tombstones, and this is not a Docker bug to be fixed; it is the direct price of the property that makes images shareable at all. The fix is always to arrange for the bytes never to be committed to a layer in the first place — one `RUN`, a multi-stage build, or a build mount. Copy-up is the same trade in the other direction. Making an image layer writable per-container would mean copying it per-container, which would undo the sharing. So the layer stays read-only and the copy happens lazily, per file, on first write — near-zero cost for a container that mostly reads, and a full file copy the instant it doesn't. `17B (virtual 61.3MB)` and `4.6MB for two bytes` are the two ends of that same mechanism. ## Go deeper - Rebuild `:leaky` as a multi-stage build — do the `dd` and the credentials work in a `FROM alpine AS build` stage, then `FROM alpine` and `COPY --from=build` only what you want. Check `RootFS.Layers`: the discarded stage contributes no layers at all, because it was never part of the final image's stack. This is the actual answer to Part 2's problem. - Try `RUN --mount=type=secret,id=creds cat /run/secrets/creds` with `docker build --secret id=creds,src=./creds`. The file is present during the step and absent from the layer — verify with the Part 2 tarball scan and confirm nothing matches. - Add `WORKDIR`, `ENV`, `LABEL` and `EXPOSE` lines and re-read `docker history`. All `0B`: they're recorded in the image *config* (`docker inspect --format '{{json .Config}}'`), which is itself a blob in the tarball. The config plus the ordered layer list is the entire image — there is no other file. - Write a file to a `-v` volume from inside `layers-box` and re-run `docker diff`. It doesn't appear, because a volume is a bind mount over the union, not part of it — which is the whole reason volume data survives `docker rm`. - Count your machine's real layer reuse: `docker system df` reports images `TOTAL` and `SIZE`, and the gap between the sum of `docker images` sizes and that `SIZE` is layer sharing, in bytes. - Read the kernel's own description of whiteouts and copy-up: https://docs.kernel.org/filesystems/overlayfs.html — the "whiteouts and opaque directories" section defines the `0,0` character device you found, and the `.wh.` convention is Docker's tar-level equivalent, specified in the OCI image layer spec. ## Cleanup Remove **only** what this exercise created — by exact name and tag, never with a prune: ```bash docker rm -f layers-box layers-box2 docker rmi layers-demo:leaky layers-demo:squashed layers-demo:rebuilt rm -rf /tmp/layers ```