# Docker: cgroup memory limits ## Concept `docker run --memory=50m` does not tell your program anything. It writes a number into a file — `/sys/fs/cgroup/memory.max` — and from that moment the Linux kernel accounts every page the container touches against it. When the container can't stay under the number, the kernel does not raise an exception, call a handler, or return `NULL` from `malloc`. It sends `SIGKILL`. Your process gets no turn. This exercise makes the whole chain visible: the limit as a file the container can read, the kernel's live counter climbing toward it in `docker stats`, the kill, and the kernel's own tally of what it did. Along the way three things will probably not match your prediction — a container limited to 50 MB will allocate 90 MB before dying, `docker stats` will report 3% while the kernel says 89%, and a `dd` whose own memory footprint is 1 MB will get OOM-killed. ## Mental model: five files and one signal Everything below is cgroup **v2** (`/sys/fs/cgroup/memory.*`). If you find `memory.limit_in_bytes` on your machine you're on v1 and the file names differ; check with `docker info | grep -i cgroup`. | File | Docker flag | What the kernel does at this threshold | Reach for it when… | | --- | --- | --- | --- | | `memory.max` | `--memory` / `-m` | **Hard wall.** On the allocation that would cross it, the kernel first tries to reclaim (evict page cache, swap out anonymous pages). If reclaim can't free enough, it invokes the cgroup OOM killer and `SIGKILL`s a task in the cgroup. | Always, on anything you don't fully trust. This is the only knob that actually bounds a container's blast radius — the one that stops a leaking sidecar from taking the node down with it. | | `memory.high` | *not exposed by Docker* | **Soft wall.** The kernel reclaims aggressively and *throttles* the allocating task (puts it to sleep proportionally to the overage). It never kills. | When you'd rather a memory-hungry batch job crawl than die. Kubernetes doesn't expose it either; systemd does, as `MemoryHigh=`. Set it below `memory.max` so you get backpressure before the cliff. | | `memory.low` | `--memory-reservation` | **Reclaim protection, not a limit.** Under *host*-level memory pressure the kernel tries to avoid reclaiming this cgroup below the value. It never throttles and never kills; you can sail past it freely. | Best-effort prioritization on a packed host. Worth knowing mostly because `--memory-reservation` is routinely mistaken for a soft limit — it is not one. | | `memory.swap.max` | `--memory-swap` (as a *combined* memory+swap total) | Caps how much of this cgroup's memory may live in swap. Docker's default is `--memory-swap = 2 × --memory`, i.e. an equal amount of swap on top of RAM. | Set `--memory-swap` equal to `--memory` to switch swap off, which is the only way to make `--memory=50m` mean "50 MB total". Part 2 is what happens when you don't. | | `memory.current` / `memory.events` | (`docker stats`, partly) | Not thresholds — the kernel's live byte counter and its running tally of `high` throttles, `max` wall-hits, and `oom_kill`s. | Diagnosis. `memory.events` is the only place that tells you the kernel killed something *and* how hard it fought first. | Two things about that accounting that catch people out: **`memory.current` is not RSS.** It counts anonymous memory *and* page cache, kernel slab, socket buffers, and tmpfs — every page charged to the cgroup, whoever touched it first. A container that reads a large file has a high `memory.current` while its processes allocate nothing. Part 5 pushes this until something dies. **The kill is a signal, not an error.** `SIGKILL` cannot be caught, blocked, or handled. There is no `MemoryError`, no `std::bad_alloc`, no `OutOfMemoryError`, no chance to flush a log line — the language runtime is never consulted, because the kernel refuses the page rather than the allocation. What you get is an exit status of **137 = 128 + 9**, the shell's convention for "terminated by signal 9". A JVM's own `OutOfMemoryError` is the opposite thing entirely: that's the runtime enforcing its *own* `-Xmx` ceiling in userspace, and it fires whether or not a cgroup exists. ``` your code malloc / new / bytearray() │ ▼ page fault ────▶ kernel charges the page to memory.current │ ├── under memory.max ──────────────▶ page granted │ └── would exceed memory.max │ ├─ reclaim succeeds ──────▶ page granted, memory.events "max" +1 │ (evict page cache, swap out anon) │ └─ reclaim fails ─────────▶ SIGKILL, memory.events "oom_kill" +1 exit code 137 ``` ## Setup Two terminals side by side. Everything runs in containers named `cg-*`, so cleanup at the end is by name only. Make a working directory and write the allocator: ```bash mkdir -p ~/cgroup-lab && cd ~/cgroup-lab cat > alloc.py <<'EOF' import os, sys, time MB = 1024 * 1024 pause = float(sys.argv[1]) if len(sys.argv) > 1 else 0.0 def rd(name): with open("/sys/fs/cgroup/" + name) as f: return int(f.read()) // MB held = [] for i in range(1, 41): held.append(os.urandom(5 * MB)) print(f"allocated {i * 5:3d} MB memory.current = {rd('memory.current'):3d} MB" f" memory.swap.current = {rd('memory.swap.current'):3d} MB", flush=True) time.sleep(pause) print("done: allocated 200 MB, never killed") EOF ``` It allocates 5 MB at a time up to a hard ceiling of 200 MB, holding every chunk, and after each one prints the kernel's own counters *read from inside its own cgroup*. The 200 MB ceiling is deliberate — a bounded allocator is both safer than `tail /dev/zero` and better teaching material, because you can see exactly how far it got. `os.urandom` rather than zeros is also deliberate: zero-filled pages get special-cased on their way to swap, which makes the arithmetic in Part 2 come out wrong by tens of megabytes. Incompressible bytes keep the accounting honest. The script is piped in on stdin, so nothing is mounted and no image is built: ```bash docker run --rm -i --memory=50m python:3-alpine python - < alloc.py ``` ## Steps ### Part 1 — the limit is a file, and the container can read it What this part tests: where the limit actually lives. Per the mental model it's `memory.max`, a plain file in the container's own cgroup — not a property of the Docker daemon, and not something your runtime is told about. 1. **Predict**: a container started with `--memory=50m`. Can a process *inside* it discover that number? And what will `free` report? 2. **Terminal A**: ```bash docker run --rm --name cg-see --memory=50m alpine sh -c ' cat /sys/fs/cgroup/memory.max cat /sys/fs/cgroup/memory.current cat /sys/fs/cgroup/memory.events free -m' ``` 3. **Observe**: ``` 52428800 1351680 low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0 total used free shared buff/cache available Mem: 15973 618 8698 2 6657 15103 Swap: 2048 1 2047 ``` `52428800` is exactly 50 × 1024 × 1024. The limit is right there, readable by anything in the container, and `memory.current` (1351680 ≈ 1.3 MB) is the kernel's live tally of what the container is using *right now*. `memory.events` is all zeros: nothing has hit any threshold yet. And then `free -m` says **15973 MB**. `/proc/meminfo` is not namespaced — it reports the host's RAM, and the container is 320× wrong about its own budget. This is precisely why the JVM needed `UseContainerSupport` (on by default since JDK 10) and why Node's old default heap sizing killed containers: every runtime that sized itself from `/proc/meminfo` sized itself to the host. ### Part 2 — allocate past it (and don't die when you expect to) What this part tests: the hard wall. `memory.max` is 50 MB and the script wants 200 MB, so the container cannot possibly finish. The question is *where* it stops. 4. **Predict**: at roughly what "allocated N MB" line does the script die? 5. **Terminal A**: ```bash docker run -i --name cg-swap --memory=50m python:3-alpine python - < alloc.py echo "exit: $?" ``` 6. **Observe**: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 13 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 19 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 24 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 29 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 34 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 39 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 44 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB allocated 50 MB memory.current = 50 MB memory.swap.current = 17 MB allocated 55 MB memory.current = 49 MB memory.swap.current = 17 MB allocated 60 MB memory.current = 49 MB memory.swap.current = 17 MB allocated 65 MB memory.current = 49 MB memory.swap.current = 20 MB allocated 70 MB memory.current = 49 MB memory.swap.current = 26 MB allocated 75 MB memory.current = 49 MB memory.swap.current = 32 MB allocated 80 MB memory.current = 49 MB memory.swap.current = 36 MB allocated 85 MB memory.current = 49 MB memory.swap.current = 42 MB allocated 90 MB memory.current = 49 MB memory.swap.current = 48 MB exit: 137 ``` It allocated **90 MB inside a 50 MB limit** before dying. Read the two counters as a pair and it's obvious what happened: - Up to 45 MB, `memory.current` tracks the allocation exactly. - At 50 MB it hits the wall. `memory.current` stops climbing — and *never exceeds 50 again for the rest of the run*. That flat line is the limit being enforced, allocation by allocation. - From that point `memory.swap.current` does the climbing instead: 0 → 17 → 26 → 36 → 48 MB. Every 5 MB the script asks for, the kernel reclaims 5 MB by pushing older pages out to swap. - The kill comes when both budgets are full: 49 MB resident + 48 MB swapped = 97 MB, against a total budget of 100 MB. 100 MB, because `--memory-swap` defaults to **twice** `--memory`, and `--memory-swap` is the *combined* ceiling. `--memory=50m` alone quietly grants the container 50 MB of swap on top of its 50 MB of RAM. 7. **Terminal A**: confirm the default: ```bash docker run --rm --name cg-see --memory=50m alpine \ sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)' docker run --rm --name cg-see --memory=50m --memory-swap=50m alpine \ sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)' ``` ``` max=52428800 swap=52428800 max=52428800 swap=0 ``` ### Part 3 — the clean kill What this part tests: the same wall with swap removed, so the number in the flag is the whole budget. Also what the *process* experiences, which per the mental model is nothing at all. 8. **Terminal A**: baseline first — the identical script with no limit: ```bash docker run --rm -i --name cg-nolimit python:3-alpine python - < alloc.py ``` ``` ⋮ allocated 195 MB memory.current = 199 MB memory.swap.current = 0 MB allocated 200 MB memory.current = 204 MB memory.swap.current = 0 MB done: allocated 200 MB, never killed ``` Straight through all 200 MB, `memory.current` tracking it the whole way, exit 0. Nothing in the program changes for the rest of this exercise. 9. **Predict**: now with `--memory=50m --memory-swap=50m`. Where does it stop, and what does the *output* look like at the end — a traceback? a `MemoryError`? an exit message? 10. **Terminal A**: ```bash docker run -i --name cg-hard --memory=50m --memory-swap=50m python:3-alpine python - < alloc.py echo "exit: $?" docker inspect cg-hard --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' ``` 11. **Observe**: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 14 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 19 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 24 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 29 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 34 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 39 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 44 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137 OOMKilled=true ExitCode=137 ``` Nine lines, then nothing. No traceback, no `MemoryError`, no "killed" message from Python — the output simply stops mid-stream, because the 45 MB line was flushed and the process was gone before it could print a tenth. `memory.swap.current` stayed at `0` the entire run, so the 50 MB in the flag really was the whole budget this time. `137` is the only thing your orchestrator sees: 128 + 9, terminated by `SIGKILL`. `OOMKilled=true` is Docker's own record, read back out of the container's state afterwards — the one place that distinguishes "the kernel shot it" from "it exited 137 for some other reason". ### Part 4 — watching it from the outside What this part tests: that the enforcement is observable live, from a different terminal, by something that knows nothing about the program. Per the mental model, `docker stats` reads `memory.current` and `memory.events` is the kernel's own tally. This part needs **both terminals**. 12. **Terminal A**: start a long-lived container so it survives the kill, and check the tally is clean: ```bash docker run -d --name cg-watch --memory=50m --memory-swap=50m python:3-alpine sleep 3600 docker exec cg-watch cat /sys/fs/cgroup/memory.events ``` ``` low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0 ``` 13. **Terminal B**: start watching. Leave this running. ```bash docker stats --format '{{.Name}} {{.MemUsage}} {{.MemPerc}}' cg-watch ``` 14. **Terminal A**: run the allocator *inside* the existing container, with a 0.6 s pause per chunk so B can keep up: ```bash docker exec -i cg-watch python - 0.6 < alloc.py echo "exit: $?" ``` 15. **Predict**: what does B show at the moment of the kill, and what does it show one second later? 16. **Observe** — A: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB ⋮ allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137 ``` and B, refreshing once a second: ``` cg-watch 14.05MiB / 50MiB 28.11% cg-watch 29.04MiB / 50MiB 58.07% cg-watch 44.07MiB / 50MiB 88.15% cg-watch 552KiB / 50MiB 1.08% cg-watch 548KiB / 50MiB 1.07% ``` The climb is visible right up to **88.15%**, and the next sample is **1.08%**. There is no intermediate state — no "shutting down", no "cleaning up". One second the memory is charged, the next it isn't, because the process holding it ceased to exist between samples. 17. **Terminal A**: ask the kernel what it did. ```bash docker exec cg-watch cat /sys/fs/cgroup/memory.events docker inspect cg-watch --format 'Running={{.State.Running}} OOMKilled={{.State.OOMKilled}}' ``` 18. **Observe**: ``` low 0 high 0 max 35 oom 1 oom_kill 1 oom_group_kill 0 Running=true OOMKilled=true ``` Three numbers worth reading carefully: - `max 35` — the cgroup hit `memory.max` **thirty-five times** and survived, by reclaiming. The kill was not the first time it touched the wall; it was the first time reclaim came up empty. - `oom 1` — the OOM killer was invoked once. - `oom_kill 1` — it killed exactly one task. And `Running=true OOMKilled=true`: the container is alive (the `sleep` was never touched), but Docker has flagged it. The OOM killer picks a victim within the cgroup by score, and a `sleep` holding 500 KB is a poor candidate next to a Python process holding 49 MB. In a real container this is the failure mode where PID 1 survives and a worker vanishes — the container looks healthy and half of it is dead. 19. **Terminal A** (optional): read the kernel's own account of it. ```bash docker run --rm --privileged --name cg-dmesg alpine dmesg | grep -B5 'Killed process' ``` ``` [21953.480027] Tasks state (memory values in pages): [21953.480027] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [21953.480029] [ 22630] 0 22630 420 180 0 180 0 49152 0 0 sleep [21953.480030] [ 22653] 0 22653 676 468 256 212 0 45056 0 0 dd [21953.480032] oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/5b298d32...,task=dd,pid=22653,uid=0 [21953.480075] Memory cgroup out of memory: Killed process 22653 (dd) total-vm:2704kB, anon-rss:1024kB, file-rss:848kB, shmem-rss:0kB, UID:0 pgtables:44kB oom_score_adj:0 ``` (This capture is from the Part 5 `dd` kill rather than the Python one — same report, and the numbers in it are the point of Part 5.) `constraint=CONSTRAINT_MEMCG` is the kernel saying this was a *cgroup* OOM, not the host running out of memory, and `oom_memcg=/docker/5b298d32…` names the container. The scoring table above it lists every task that was a candidate. ### Part 5 — page cache counts, and `docker stats` lies about it What this part tests: the accounting caveat from the mental model — `memory.current` is not RSS. This is the part most likely to break your intuition, so take the predictions seriously. 20. **Terminal A**: a fresh container, and write a 40 MB file into it. `dd` with `bs=1M` allocates a *one megabyte* buffer. ```bash docker run -d --name cg-cache --memory=50m --memory-swap=50m python:3-alpine sleep 600 docker exec cg-cache sh -c ' dd if=/dev/urandom of=/big bs=1M count=40 sync echo memory.current=$(cat /sys/fs/cgroup/memory.current) grep -E "^(anon|file|inactive_file) " /sys/fs/cgroup/memory.stat' ``` 21. **Predict**: `dd` allocated 1 MB. What is `memory.current` now, out of a 50 MB budget? And what will `docker stats` say? 22. **Observe**: ``` 41943040 bytes (40.0MB) copied, 1.115645 seconds, 35.9MB/s memory.current=44449792 anon 102400 file 41943040 inactive_file 41943040 ``` `memory.current` is **44449792 — 42.4 MB, or 89% of the limit** — for a container running `sleep` and a `dd` that allocated 1 MB. `anon` is `102400`, a hundred kilobytes: essentially none of it is anybody's heap. `file 41943040` is exactly the 40 MB file. The page cache is charged to the cgroup that faulted it in, and there is no distinction in `memory.max` between "cache" and "heap". 23. **Terminal B**: ask Docker. ```bash docker stats --no-stream --format '{{.Name}} {{.MemUsage}} {{.MemPerc}}' cg-cache ``` 24. **Observe**: ``` cg-cache 1.715MiB / 50MiB 3.43% ``` **3.43%.** The kernel says 89%, Docker says 3%. Neither is lying: Docker subtracts `inactive_file` from `memory.current` before reporting, on the theory that reclaimable cache isn't "really" used. That is a defensible dashboard choice and a terrible debugging one, because the number `memory.max` is compared against is the *unsubtracted* one. If you are chasing an OOM kill, `docker stats` is the wrong instrument; read `memory.current`. 25. **Predict**: `memory.current` is at 42 MB of 50. Now allocate 30 MB of anonymous memory in the same container. There is nowhere near 30 MB free. Does it get killed? 26. **Terminal A**: ```bash docker exec cg-cache python -c " import os x = os.urandom(30*1024*1024) print('allocated 30 MB, still alive') print('memory.current =', open('/sys/fs/cgroup/memory.current').read().strip()) for l in open('/sys/fs/cgroup/memory.stat'): if l.split()[0] in ('anon','file','pgscan','pgsteal'): print(l.strip()) print(open('/sys/fs/cgroup/memory.events').read().strip()) " ``` 27. **Observe**: ``` allocated 30 MB, still alive memory.current = 52166656 anon 35164160 file 15507456 pgscan 6464 pgsteal 6464 low 0 high 0 max 101 oom 0 oom_kill 0 oom_group_kill 0 ``` It survived. `file` fell from 41943040 to 15507456 — the kernel threw away 26 MB of clean page cache to make room — while `anon` rose to 35164160. `pgscan 6464 / pgsteal 6464` is the reclaim: 6464 pages scanned, 6464 successfully freed, a 100% hit rate because clean file pages are free to drop. And `max 101, oom_kill 0`. It hit the wall a hundred and one times and was never killed once. **`memory.max` is not "kill at 50 MB". It's "never exceed 50 MB", and killing is only what happens when the kernel runs out of other ways to obey.** `memory.current = 52166656` — 49.75 MB — is the cgroup sitting flat against its ceiling, working. 28. **Predict**: so if clean cache is free to reclaim, writing a *300* MB file into a 50 MB container should also be fine — `dd` still only holds 1 MB at a time. 29. **Terminal A**, in a fresh container so nothing above muddies it: ```bash docker run -d --name cg-ddkill --memory=50m --memory-swap=50m alpine sleep 600 docker exec cg-ddkill dd if=/dev/urandom of=/big bs=1M count=300 echo "exit: $?" docker exec cg-ddkill sh -c ' ls -lh /big echo memory.current=$(cat /sys/fs/cgroup/memory.current) grep -E "^(anon|file) " /sys/fs/cgroup/memory.stat cat /sys/fs/cgroup/memory.events' ``` 30. **Observe**: ``` exit: 137 -rw-r--r-- 1 root root 46.9M Jul 22 16:02 /big memory.current=52215808 anon 172032 file 49229824 low 0 high 0 max 29 oom 1 oom_kill 1 oom_group_kill 0 ``` `dd` was **OOM-killed with an `anon` footprint of 172032 bytes — 168 kilobytes.** The kernel's own report from step 19 puts it even more starkly: `Killed process (dd) total-vm:2704kB, anon-rss:1024kB`. A process whose entire heap is one megabyte, killed for exceeding a fifty megabyte limit. The difference from step 27 is *dirty*. Reclaiming a clean page means dropping it; reclaiming a dirty page means waiting for writeback to finish first. `dd` was producing dirty pages at 350 MB/s and the disk could not retire them that fast, so the cgroup filled with cache the kernel was not yet allowed to drop, hit the wall 29 times trying, and then killed the only task making it worse. The file stops at 46.9 MB of the requested 300. This is the shape of a whole class of production incidents: a container that is fine all day and dies during the nightly export, backup, or log rotation. Nothing leaked. It wrote a file. ### Part 6 — the same number as a soft limit What this part tests: the hard/soft distinction from the mental model, with everything else held constant. `memory.high` is the same 50 MB threshold, enforced by throttling instead of killing. Docker doesn't expose `memory.high`, so this needs `--privileged` to remount the cgroup filesystem read-write and set the file by hand. That is a lab technique, not a deployment pattern — in the real world this is systemd's `MemoryHigh=`. 31. **Predict**: `memory.high = 50M` with `memory.max` left at `max`. The same script wants 200 MB. What happens? 32. **Terminal A**: ```bash docker run --rm -i --privileged --name cg-soft python:3-alpine sh -c ' mount -o remount,rw /sys/fs/cgroup echo 50M > /sys/fs/cgroup/memory.high python - echo "python exit: $?" cat /sys/fs/cgroup/memory.events' < alloc.py ``` 33. **Observe**: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB ⋮ allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB allocated 50 MB memory.current = 45 MB memory.swap.current = 19 MB ⋮ allocated 95 MB memory.current = 49 MB memory.swap.current = 53 MB ⋮ allocated 140 MB memory.current = 49 MB memory.swap.current = 96 MB ⋮ allocated 195 MB memory.current = 49 MB memory.swap.current = 153 MB allocated 200 MB memory.current = 48 MB memory.swap.current = 159 MB done: allocated 200 MB, never killed python exit: 0 low 0 high 101 max 0 oom 0 oom_kill 0 oom_group_kill 0 ``` (Trimmed — the run prints all forty lines; `memory.current` never leaves the 45–49 range after the tenth.) Same program, same 50 MB number, opposite outcome. All 200 MB allocated, **exit 0**, and `memory.current` pinned in the high forties for the entire run — the ceiling was honoured just as strictly as `memory.max` honoured it. The difference is what happened at the ceiling: `memory.swap.current` climbed to 159 MB, three times the RAM limit, because with no `memory.max` there was nothing to cap it. And the tally is the mirror image of Part 4: | | `memory.max = 50M` | `memory.high = 50M` | | --- | --- | --- | | `max` | 35 | 0 | | `high` | 0 | 101 | | `oom_kill` | 1 | 0 | | exit code | 137 | 0 | | allocated before stopping | 45 MB | 200 MB | `high 101` counts the times the kernel throttled the allocating task — put it to sleep to let reclaim catch up. That is the entire mechanism: the same threshold, applied as backpressure rather than as a wall. ## What you should see A container that reads its own limit out of `/sys/fs/cgroup/memory.max` as `52428800` while `free -m` insists it has 15973 MB. An allocator that gets 90 MB into a 50 MB container before dying, because `--memory=50m` silently grants 50 MB of swap as well — and `memory.current` that flattens at 49 and never once exceeds it. With swap off, a clean kill at 45 MB with no traceback, exit `137`, `OOMKilled=true`. `docker stats` climbing to `88.15%` and reading `1.08%` one second later. `memory.events` reporting `max 35, oom_kill 1` — thirty-five survived collisions before the fatal one. A `dd` with a 1 MB buffer pushing `memory.current` to 89% of the limit while `docker stats` reports `3.43%`, and then getting SIGKILLed at `anon-rss:1024kB`. And the same 50 MB expressed as `memory.high` letting the identical script finish all 200 MB with `high 101, oom_kill 0, exit 0`. ## Why Because the limit is enforced at page-fault time, in the kernel, against a counter the kernel maintains — and there is no path from there back into your program. When a fault would push `memory.current` past `memory.max`, the kernel's options are to make room or to stop the cgroup growing. Making room means reclaim: drop clean page cache, write back dirty pages, swap out anonymous ones. That succeeded 35 times in Part 4 and 101 times in Part 5, invisibly, which is why `max` counts in the dozens and hundreds on a container that is merely *busy*. Only when reclaim returns empty-handed does the second option apply, and the kernel cannot implement it by failing an allocation. The allocation already succeeded — `os.urandom` returned, `malloc` returned, the page table entry exists. What's failing is the *fault*, on memory the process already believes it owns. Handing that back as an error would mean inventing a failure at an instruction that has no error path. So the kernel removes the process instead, with the one signal that cannot be caught, and your program learns nothing because there is nobody left to tell. That is also why `memory.high` can exist at all. Throttling is something you can do *to* a running task — sleep it, let reclaim catch up, resume it — without ever needing an error path. It's strictly more graceful, it's what you actually want for batch work, and Docker exposes no flag for it. And the page cache results follow from the same accounting rather than from any special rule. Charging cache to the cgroup that faulted it in is the only choice that makes limits meaningful — otherwise any container could pin unlimited host memory by reading files. The consequence is that `memory.current` is a measure of pages the cgroup is responsible for, not pages its processes allocated, and the OOM killer's victim is chosen by RSS among tasks that may have had nothing to do with the pressure. `dd` at `anon-rss:1024kB` was not the cause of the problem. It was just the biggest thing in the room. ## Go deeper - Drop the `sync` in step 20 and re-run the 30 MB allocation immediately after the `dd`. With the 40 MB still dirty and in flight, the same allocation that survived at step 27 gets `exit 137` instead — the same distinction as step 30, at a scale small enough to run in a second. It is timing-dependent; run it a few times. - Swap the allocator's `os.urandom(5 * MB)` back to `bytearray(5 * MB)` and re-run Part 2. Zero-filled pages get special-cased in the swap path, and the container survives to ~130 MB instead of 90 — the same limit, honoured identically, with the arithmetic thrown off by page content. A good reminder that synthetic memory benchmarks are easy to get wrong. - Try `--oom-kill-disable` on Part 3's run and read the warning: `WARNING: Your kernel does not support OomKillDisable. OomKillDisable discarded.` The flag is a cgroup **v1** feature (`memory.oom_control`, which paused the cgroup at the limit instead of killing it). v2 removed the ability to disable the OOM killer, Docker keeps the flag for compatibility, and it silently does nothing. Worth knowing before you find it in someone's `docker-compose.yml` and assume it's protecting anything. - Run two allocators in the same container at once and watch which one the OOM killer picks, in `dmesg`'s scoring table. Then bias the choice: `echo 500 > /proc/self/oom_score_adj` in one of them makes it the preferred victim. Note the asymmetry — *raising* the score works unprivileged, but `echo -1000` (the way you protect a supervisor from its own workers) fails with `Permission denied` unless the container has `CAP_SYS_RESOURCE`. You may volunteer to die; you may not opt out. - Compare with the CPU side: `--cpus=0.5` is enforced by the CFS bandwidth controller, which *throttles* rather than kills, because a process can always be given less CPU later but cannot be given back memory it already wrote to. Same cgroup, two very different enforcement stories. - Read the kernel's own documentation for these files: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files — in particular the `memory.high` section, which spells out the throttling behaviour Part 6 counts. ## Cleanup ```bash docker rm -f cg-watch cg-cache cg-ddkill cg-swap cg-hard ``` (`cg-see`, `cg-nolimit`, `cg-soft` and `cg-dmesg` were all `--rm` and are already gone. The allocator lives in `~/cgroup-lab/alloc.py`; delete the directory if you're done.)