cld-toys › Guided exercises › Docker / containers

cgroup Memory Limits

--memory=50m is not advice to your program — it's a number in a kernel file, enforced with SIGKILL. Watch a 50 MB container allocate 90 MB before dying, docker stats report 3% while the kernel says 89%, and a dd holding one megabyte get OOM-killed.

Concept

docker run --memory=50m does not tell your program anything. It writes a number into a file — /sys/fs/cgroup/memory.max — and from that moment the Linux kernel accounts every page the container touches against it. When the container can't stay under the number, the kernel does not raise an exception, call a handler, or return NULL from malloc. It sends SIGKILL. Your process gets no turn.

This exercise makes the whole chain visible: the limit as a file the container can read, the kernel's live counter climbing toward it in docker stats, the kill, and the kernel's own tally of what it did. Along the way three things will probably not match your prediction — a container limited to 50 MB will allocate 90 MB before dying, docker stats will report 3% while the kernel says 89%, and a dd whose own memory footprint is 1 MB will get OOM-killed.

Mental model: five files and one signal

Everything below is cgroup v2 (/sys/fs/cgroup/memory.*). If you find memory.limit_in_bytes on your machine you're on v1 and the file names differ; check with docker info | grep -i Cgroup.

FileDocker flagWhat the kernel does at this thresholdReach for it when…
memory.max --memory / -m Hard wall. On the allocation that would cross it, the kernel first tries to reclaim (evict page cache, swap out anonymous pages). If reclaim can't free enough, it invokes the cgroup OOM killer and SIGKILLs a task in the cgroup. Always, on anything you don't fully trust. This is the only knob that actually bounds a container's blast radius — the one that stops a leaking sidecar from taking the node down with it.
memory.high
not exposed by Docker		 — Soft wall. The kernel reclaims aggressively and throttles the allocating task (puts it to sleep proportionally to the overage). It never kills. When you'd rather a memory-hungry batch job crawl than die. Kubernetes doesn't expose it either; systemd does, as MemoryHigh=. Set it below memory.max so you get backpressure before the cliff.
memory.low --memory-reservation Reclaim protection, not a limit. Under host-level memory pressure the kernel tries to avoid reclaiming this cgroup below the value. It never throttles and never kills; you can sail past it freely. Best-effort prioritization on a packed host. Worth knowing mostly because --memory-reservation is routinely mistaken for a soft limit — it is not one.
memory.swap.max --memory-swap
(a combined memory+swap total)		 Caps how much of this cgroup's memory may live in swap. Docker's default is --memory-swap = 2 × --memory, i.e. an equal amount of swap on top of RAM. Set --memory-swap equal to --memory to switch swap off, which is the only way to make --memory=50m mean "50 MB total". Part 2 is what happens when you don't.
memory.current
memory.events		 (docker stats, partly) Not thresholds — the kernel's live byte counter and its running tally of high throttles, max wall-hits, and oom_kills. Diagnosis. memory.events is the only place that tells you the kernel killed something and how hard it fought first.
memory.current is not RSS It counts anonymous memory and page cache, kernel slab, socket buffers, and tmpfs — every page charged to the cgroup, whoever touched it first. A container that reads a large file has a high memory.current while its processes allocate nothing. Part 5 pushes this until something dies.
The kill is a signal, not an error SIGKILL cannot be caught, blocked, or handled. There is no MemoryError, no std::bad_alloc, no OutOfMemoryError, no chance to flush a log line — the language runtime is never consulted, because the kernel refuses the page rather than the allocation. What you get is an exit status of 137 = 128 + 9, the shell's convention for "terminated by signal 9". A JVM's own OutOfMemoryError is the opposite thing entirely: that's the runtime enforcing its own -Xmx ceiling in userspace, and it fires whether or not a cgroup exists.
your code malloc / new / bytearray() │ ▼ page fault ────▶ kernel charges the page to memory.current │ ├── under memory.max ──────────────▶ page granted │ └── would exceed memory.max │ ├─ reclaim succeeds ──────▶ page granted, memory.events "max" +1 │ (evict page cache, swap out anon) │ └─ reclaim fails ─────────▶ SIGKILL, memory.events "oom_kill" +1 exit code 137

Setup

Two terminals side by side. Everything runs in containers named cg-*, so cleanup at the end is by name only.

Make a working directory and write the allocator:

mkdir -p ~/cgroup-lab && cd ~/cgroup-lab
cat > alloc.py <<'EOF'
import os, sys, time

MB = 1024 * 1024
pause = float(sys.argv[1]) if len(sys.argv) > 1 else 0.0

def rd(name):
    with open("/sys/fs/cgroup/" + name) as f:
        return int(f.read()) // MB

held = []
for i in range(1, 41):
    held.append(os.urandom(5 * MB))
    print(f"allocated {i * 5:3d} MB   memory.current = {rd('memory.current'):3d} MB"
          f"   memory.swap.current = {rd('memory.swap.current'):3d} MB", flush=True)
    time.sleep(pause)
print("done: allocated 200 MB, never killed")
EOF

It allocates 5 MB at a time up to a hard ceiling of 200 MB, holding every chunk, and after each one prints the kernel's own counters read from inside its own cgroup. The 200 MB ceiling is deliberate — a bounded allocator is both safer than tail /dev/zero and better teaching material, because you can see exactly how far it got.

Why os.urandom and not zeros Zero-filled pages get special-cased on their way to swap, which makes the arithmetic in Part 2 come out wrong by tens of megabytes. Incompressible bytes keep the accounting honest. (Try it the other way in "Go deeper".)

The script is piped in on stdin, so nothing is mounted and no image is built:

docker run --rm -i --memory=50m python:3-alpine python - < alloc.py

Part 1 — the limit is a file, and the container can read it

What this part tests: where the limit actually lives. Per the mental model it's memory.max, a plain file in the container's own cgroup — not a property of the Docker daemon, and not something your runtime is told about.

Step 1 · ask the container what it's allowed
Terminal A 
docker run --rm --name cg-see --memory=50m alpine sh -c '
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.events
free -m'
Predict: can a process inside the container discover the 50 MB number? And what will free report? Click to check.
52428800 1351680 low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0 total used free shared buff/cache available Mem: 15973 618 8698 2 6657 15103 Swap: 2048 1 2047

52428800 is exactly 50 × 1024 × 1024. The limit is right there, readable by anything in the container, and memory.current (1351680 ≈ 1.3 MB) is the kernel's live tally of what the container is using right now. memory.events is all zeros: nothing has hit any threshold yet.

And then free -m says 15973 MB. /proc/meminfo is not namespaced — it reports the host's RAM, and the container is 320× wrong about its own budget. This is precisely why the JVM needed UseContainerSupport (on by default since JDK 10) and why Node's old default heap sizing killed containers: every runtime that sized itself from /proc/meminfo sized itself to the host.

Part 2 — allocate past it (and don't die when you expect to)

What this part tests: the hard wall. memory.max is 50 MB and the script wants 200 MB, so the container cannot possibly finish. The question is where it stops.

Step 2 · run the allocator under the limit
Terminal A 
docker run -i --name cg-swap --memory=50m python:3-alpine python - < alloc.py
echo "exit: $?"
Predict: at roughly what "allocated N MB" line does the script die? Click to check.
allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 13 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 19 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 24 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 29 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 34 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 39 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 44 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB allocated 50 MB memory.current = 50 MB memory.swap.current = 17 MB allocated 55 MB memory.current = 49 MB memory.swap.current = 17 MB allocated 60 MB memory.current = 49 MB memory.swap.current = 17 MB allocated 65 MB memory.current = 49 MB memory.swap.current = 20 MB allocated 70 MB memory.current = 49 MB memory.swap.current = 26 MB allocated 75 MB memory.current = 49 MB memory.swap.current = 32 MB allocated 80 MB memory.current = 49 MB memory.swap.current = 36 MB allocated 85 MB memory.current = 49 MB memory.swap.current = 42 MB allocated 90 MB memory.current = 49 MB memory.swap.current = 48 MB exit: 137

It allocated 90 MB inside a 50 MB limit before dying. Read the two counters as a pair and it's obvious what happened:

  • Up to 45 MB, memory.current tracks the allocation exactly.
  • At 50 MB it hits the wall. memory.current stops climbing — and never exceeds 50 again for the rest of the run. That flat line is the limit being enforced, allocation by allocation.
  • From that point memory.swap.current does the climbing instead: 0 → 17 → 26 → 36 → 48 MB. Every 5 MB the script asks for, the kernel reclaims 5 MB by pushing older pages out to swap.
  • The kill comes when both budgets are full: 49 MB resident + 48 MB swapped = 97 MB, against a total budget of 100 MB.

100 MB, because --memory-swap defaults to twice --memory, and --memory-swap is the combined ceiling. --memory=50m alone quietly grants the container 50 MB of swap on top of its 50 MB of RAM.

Step 3 · confirm the default
Terminal A 
docker run --rm --name cg-see --memory=50m alpine \
  sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)'
docker run --rm --name cg-see --memory=50m --memory-swap=50m alpine \
  sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)'
max=52428800 swap=52428800 max=52428800 swap=0

Part 3 — the clean kill

What this part tests: the same wall with swap removed, so the number in the flag is the whole budget. Also what the process experiences, which per the mental model is nothing at all.

Step 4 · a control run with no limit
Terminal A 
docker run --rm -i --name cg-nolimit python:3-alpine python - < alloc.py
lines removed allocated 195 MB memory.current = 199 MB memory.swap.current = 0 MB allocated 200 MB memory.current = 204 MB memory.swap.current = 0 MB done: allocated 200 MB, never killed

Straight through all 200 MB, memory.current tracking it the whole way, exit 0. Nothing in the program changes for the rest of this exercise.

Step 5 · the same script, swap off
Terminal A 
docker run -i --name cg-hard --memory=50m --memory-swap=50m python:3-alpine python - < alloc.py
echo "exit: $?"
docker inspect cg-hard --format 'OOMKilled={{.State.OOMKilled}}  ExitCode={{.State.ExitCode}}'
Predict: where does it stop — and what does the output look like at the end? A traceback? A MemoryError? An exit message? Click to check.
allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 14 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 19 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 24 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 29 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 34 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 39 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 44 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137 OOMKilled=true ExitCode=137

Nine lines, then nothing. No traceback, no MemoryError, no "killed" message from Python — the output simply stops mid-stream, because the 45 MB line was flushed and the process was gone before it could print a tenth. memory.swap.current stayed at 0 the entire run, so the 50 MB in the flag really was the whole budget this time.

137 is the only thing your orchestrator sees: 128 + 9, terminated by SIGKILL. OOMKilled=true is Docker's own record, read back out of the container's state afterwards — the one place that distinguishes "the kernel shot it" from "it exited 137 for some other reason".

Part 4 — watching it from the outside

What this part tests: that the enforcement is observable live, from a different terminal, by something that knows nothing about the program. Per the mental model, docker stats reads memory.current and memory.events is the kernel's own tally.

This part needs both terminals Open them side by side. Terminal A runs the container and the allocator; Terminal B does nothing but watch.
Step 6 · a container that survives the kill
Terminal A 
docker run -d --name cg-watch --memory=50m --memory-swap=50m python:3-alpine sleep 3600
docker exec cg-watch cat /sys/fs/cgroup/memory.events
low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0

A clean tally to compare against.

Step 7 · start watching, then allocate
Terminal B — leave this running: 
docker stats --format '{{.Name}}   {{.MemUsage}}   {{.MemPerc}}' cg-watch
Terminal A — run the allocator inside the existing container, with a 0.6 s pause per chunk so B can keep up: 
docker exec -i cg-watch python - 0.6 < alloc.py
echo "exit: $?"
Predict: what does B show at the moment of the kill, and what does it show one second later? Click to check.
Terminal A — the allocator
allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB lines removed allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137
Terminal B — docker stats
cg-watch 14.05MiB / 50MiB 28.11% cg-watch 29.04MiB / 50MiB 58.07% cg-watch 44.07MiB / 50MiB 88.15% cg-watch 552KiB / 50MiB 1.08% cg-watch 548KiB / 50MiB 1.07%

The climb is visible right up to 88.15%, and the next sample is 1.08%. There is no intermediate state — no "shutting down", no "cleaning up". One second the memory is charged, the next it isn't, because the process holding it ceased to exist between samples.

Step 8 · ask the kernel what it did
Terminal A 
docker exec cg-watch cat /sys/fs/cgroup/memory.events
docker inspect cg-watch --format 'Running={{.State.Running}}  OOMKilled={{.State.OOMKilled}}'
Predict: how many times did this cgroup hit its limit? (It was killed once.) Click to check.
low 0 high 0 max 35 oom 1 oom_kill 1 oom_group_kill 0 Running=true OOMKilled=true
  • max 35 — the cgroup hit memory.max thirty-five times and survived, by reclaiming. The kill was not the first time it touched the wall; it was the first time reclaim came up empty.
  • oom 1 — the OOM killer was invoked once.
  • oom_kill 1 — it killed exactly one task.

And Running=true OOMKilled=true: the container is alive (the sleep was never touched), but Docker has flagged it. The OOM killer picks a victim within the cgroup by score, and a sleep holding 500 KB is a poor candidate next to a Python process holding 49 MB. In a real container this is the failure mode where PID 1 survives and a worker vanishes — the container looks healthy and half of it is dead.

Step 9 · optional — the kernel's own account
Terminal A 
docker run --rm --privileged --name cg-dmesg alpine dmesg | grep -B5 'Killed process'
[21953.480027] Tasks state (memory values in pages): [21953.480027] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [21953.480029] [ 22630] 0 22630 420 180 0 180 0 49152 0 0 sleep [21953.480030] [ 22653] 0 22653 676 468 256 212 0 45056 0 0 dd [21953.480032] oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/5b298d32...,task=dd,pid=22653,uid=0 [21953.480075] Memory cgroup out of memory: Killed process 22653 (dd) total-vm:2704kB, anon-rss:1024kB, file-rss:848kB, shmem-rss:0kB, UID:0 pgtables:44kB oom_score_adj:0

(This capture is from the Part 5 dd kill rather than the Python one — same report, and the numbers in it are the point of Part 5.) constraint=CONSTRAINT_MEMCG is the kernel saying this was a cgroup OOM, not the host running out of memory, and oom_memcg=/docker/5b298d32… names the container. The scoring table above it lists every task that was a candidate.

Part 5 — page cache counts, and docker stats lies about it

What this part tests: the accounting caveat from the mental model — memory.current is not RSS. This is the part most likely to break your intuition, so take the predictions seriously.

Step 10 · write a 40 MB file with a 1 MB buffer
Terminal A 
docker run -d --name cg-cache --memory=50m --memory-swap=50m python:3-alpine sleep 600
docker exec cg-cache sh -c '
  dd if=/dev/urandom of=/big bs=1M count=40
  sync
  echo memory.current=$(cat /sys/fs/cgroup/memory.current)
  grep -E "^(anon|file|inactive_file) " /sys/fs/cgroup/memory.stat'
Predict: dd allocated 1 MB. What is memory.current now, out of a 50 MB budget? Click to check.
41943040 bytes (40.0MB) copied, 1.115645 seconds, 35.9MB/s memory.current=44449792 anon 102400 file 41943040 inactive_file 41943040

memory.current is 44449792 — 42.4 MB, or 89% of the limit — for a container running sleep and a dd that allocated 1 MB. anon is 102400, a hundred kilobytes: essentially none of it is anybody's heap. file 41943040 is exactly the 40 MB file. The page cache is charged to the cgroup that faulted it in, and there is no distinction in memory.max between "cache" and "heap".

Step 11 · now ask Docker
Terminal B 
docker stats --no-stream --format '{{.Name}}   {{.MemUsage}}   {{.MemPerc}}' cg-cache
Predict: the kernel says 89% of 50 MB. What percentage does docker stats report? Click to check.
cg-cache 1.715MiB / 50MiB 3.43%

3.43%. The kernel says 89%, Docker says 3%. Neither is lying: Docker subtracts inactive_file from memory.current before reporting, on the theory that reclaimable cache isn't "really" used. That is a defensible dashboard choice and a terrible debugging one, because the number memory.max is compared against is the unsubtracted one.

If you are chasing an OOM kill, docker stats is the wrong instrument Read memory.current. A container can sit at 3% in your dashboard and be one page away from the wall.
Step 12 · allocate 30 MB into a container that has 8 MB "free"
Terminal A 
docker exec cg-cache python -c "
import os
x = os.urandom(30*1024*1024)
print('allocated 30 MB, still alive')
print('memory.current =', open('/sys/fs/cgroup/memory.current').read().strip())
for l in open('/sys/fs/cgroup/memory.stat'):
    if l.split()[0] in ('anon','file','pgscan','pgsteal'): print(l.strip())
print(open('/sys/fs/cgroup/memory.events').read().strip())
"
Predict: memory.current is at 42 MB of 50, and this wants 30 MB more. Does it get killed? Click to check.
allocated 30 MB, still alive memory.current = 52166656 anon 35164160 file 15507456 pgscan 6464 pgsteal 6464 low 0 high 0 max 101 oom 0 oom_kill 0 oom_group_kill 0

It survived. file fell from 41943040 to 15507456 — the kernel threw away 26 MB of clean page cache to make room — while anon rose to 35164160. pgscan 6464 / pgsteal 6464 is the reclaim: 6464 pages scanned, 6464 successfully freed, a 100% hit rate because clean file pages are free to drop.

max 101, oom_kill 0 It hit the wall a hundred and one times and was never killed once. memory.max is not "kill at 50 MB". It's "never exceed 50 MB", and killing is only what happens when the kernel runs out of other ways to obey. memory.current = 52166656 — 49.75 MB — is the cgroup sitting flat against its ceiling, working.
Step 13 · the same trick at 300 MB
Terminal A — a fresh container so nothing above muddies it: 
docker run -d --name cg-ddkill --memory=50m --memory-swap=50m alpine sleep 600
docker exec cg-ddkill dd if=/dev/urandom of=/big bs=1M count=300
echo "exit: $?"
docker exec cg-ddkill sh -c '
  ls -lh /big
  echo memory.current=$(cat /sys/fs/cgroup/memory.current)
  grep -E "^(anon|file) " /sys/fs/cgroup/memory.stat
  cat /sys/fs/cgroup/memory.events'
Predict: clean cache was free to reclaim in step 12, and dd still only holds 1 MB at a time. So this is fine, right? Click to check.
exit: 137 -rw-r--r-- 1 root root 46.9M Jul 22 16:02 /big memory.current=52215808 anon 172032 file 49229824 low 0 high 0 max 29 oom 1 oom_kill 1 oom_group_kill 0

dd was OOM-killed with an anon footprint of 172032 bytes — 168 kilobytes. The kernel's own report from step 9 puts it even more starkly: Killed process (dd) total-vm:2704kB, anon-rss:1024kB. A process whose entire heap is one megabyte, killed for exceeding a fifty megabyte limit.

The difference from step 12 is dirty. Reclaiming a clean page means dropping it; reclaiming a dirty page means waiting for writeback to finish first. dd was producing dirty pages at 350 MB/s and the disk could not retire them that fast, so the cgroup filled with cache the kernel was not yet allowed to drop, hit the wall 29 times trying, and then killed the only task making it worse. The file stops at 46.9 MB of the requested 300.

The shape of a whole class of production incidents A container that is fine all day and dies during the nightly export, backup, or log rotation. Nothing leaked. It wrote a file.

Part 6 — the same number as a soft limit

What this part tests: the hard/soft distinction from the mental model, with everything else held constant. memory.high is the same 50 MB threshold, enforced by throttling instead of killing.

Docker doesn't expose memory.high, so this needs --privileged to remount the cgroup filesystem read-write and set the file by hand. That is a lab technique, not a deployment pattern — in the real world this is systemd's MemoryHigh=.

Step 14 · 50 MB, softly
Terminal A 
docker run --rm -i --privileged --name cg-soft python:3-alpine sh -c '
  mount -o remount,rw /sys/fs/cgroup
  echo 50M > /sys/fs/cgroup/memory.high
  python -
  echo "python exit: $?"
  cat /sys/fs/cgroup/memory.events' < alloc.py
Predict: memory.high = 50M, memory.max left at max, same script wanting 200 MB. What happens? Click to check.
allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB lines removed allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB allocated 50 MB memory.current = 45 MB memory.swap.current = 19 MB lines removed allocated 95 MB memory.current = 49 MB memory.swap.current = 53 MB lines removed allocated 140 MB memory.current = 49 MB memory.swap.current = 96 MB lines removed allocated 195 MB memory.current = 49 MB memory.swap.current = 153 MB allocated 200 MB memory.current = 48 MB memory.swap.current = 159 MB done: allocated 200 MB, never killed python exit: 0 low 0 high 101 max 0 oom 0 oom_kill 0 oom_group_kill 0

(Trimmed — the run prints all forty lines; memory.current never leaves the 45–49 range after the tenth.)

Same program, same 50 MB number, opposite outcome. All 200 MB allocated, exit 0, and memory.current pinned in the high forties for the entire run — the ceiling was honoured just as strictly as memory.max honoured it. The difference is what happened at the ceiling: memory.swap.current climbed to 159 MB, three times the RAM limit, because with no memory.max there was nothing to cap it.

memory.eventsmemory.max = 50Mmemory.high = 50M
max350
high0101
oom_kill10
exit code1370
allocated before stopping45 MB200 MB

high 101 counts the times the kernel throttled the allocating task — put it to sleep to let reclaim catch up. That is the entire mechanism: the same threshold, applied as backpressure rather than as a wall.

What you should see

A container that reads its own limit out of /sys/fs/cgroup/memory.max as 52428800 while free -m insists it has 15973 MB. An allocator that gets 90 MB into a 50 MB container before dying, because --memory=50m silently grants 50 MB of swap as well — and memory.current that flattens at 49 and never once exceeds it. With swap off, a clean kill at 45 MB with no traceback, exit 137, OOMKilled=true. docker stats climbing to 88.15% and reading 1.08% one second later. memory.events reporting max 35, oom_kill 1 — thirty-five survived collisions before the fatal one. A dd with a 1 MB buffer pushing memory.current to 89% of the limit while docker stats reports 3.43%, and then getting SIGKILLed at anon-rss:1024kB. And the same 50 MB expressed as memory.high letting the identical script finish all 200 MB with high 101, oom_kill 0, exit 0.

Why

Because the limit is enforced at page-fault time, in the kernel, against a counter the kernel maintains — and there is no path from there back into your program. When a fault would push memory.current past memory.max, the kernel's options are to make room or to stop the cgroup growing. Making room means reclaim: drop clean page cache, write back dirty pages, swap out anonymous ones. That succeeded 35 times in Part 4 and 101 times in Part 5, invisibly, which is why max counts in the dozens and hundreds on a container that is merely busy.

Only when reclaim returns empty-handed does the second option apply, and the kernel cannot implement it by failing an allocation. The allocation already succeeded — os.urandom returned, malloc returned, the page table entry exists. What's failing is the fault, on memory the process already believes it owns. Handing that back as an error would mean inventing a failure at an instruction that has no error path. So the kernel removes the process instead, with the one signal that cannot be caught, and your program learns nothing because there is nobody left to tell.

That is also why memory.high can exist at all. Throttling is something you can do to a running task — sleep it, let reclaim catch up, resume it — without ever needing an error path. It's strictly more graceful, it's what you actually want for batch work, and Docker exposes no flag for it.

And the page cache results follow from the same accounting rather than from any special rule. Charging cache to the cgroup that faulted it in is the only choice that makes limits meaningful — otherwise any container could pin unlimited host memory by reading files. The consequence is that memory.current is a measure of pages the cgroup is responsible for, not pages its processes allocated, and the OOM killer's victim is chosen by RSS among tasks that may have had nothing to do with the pressure. dd at anon-rss:1024kB was not the cause of the problem. It was just the biggest thing in the room.

Go deeper

Sources: Linux kernel docs: cgroup v2 memory interface files — the memory.high section spells out the throttling behaviour Part 6 counts · Docker: runtime options with memory, CPUs, and GPUs for --memory-swap's combined-total semantics

Cleanup

docker rm -f cg-watch cg-cache cg-ddkill cg-swap cg-hard

cg-see, cg-nolimit, cg-soft and cg-dmesg were all --rm and are already gone. The allocator lives in ~/cgroup-lab/alloc.py; delete the directory if you're done.

Source of truth for this exercise: docker-cgroup-memory-limits.md. Every terminal transcript above was captured from live containers on Docker's cgroup v2 engine before publishing.