Small Python programs that each teach exactly one mechanism of infrastructure software — a rate limiter, a write-ahead log, an LSM tree. Each builds a working mental model of how the thing works, never a usable reimplementation.
A toy is under 300 lines of core code — most are nearer 150 — small enough to hold in your head at once. Every toy here comes with a commentary: a study guide that explains the concept from scratch, walks the real source in reading order, derives the demo's result arithmetically, and points at the background reading. Click a card to read one.
A key-value store that survives crashes by writing its log first. Then the discovery that the experiment you'd run to prove it works cannot fail: with fsync switched off entirely, kill -9 still returns all five records. Turning fsync on changes nothing — because the bytes were in the kernel, and the kernel isn't what died.
Token bucket and sliding window log, tuned to the same average rate, fed one identical trace — and they disagree twice, in opposite directions. The bucket's generosity at t=2.5 turns out to be the exact cause of its stinginess at t=5.0.
A store where a delete is a write, and where compaction — the 3am housekeeping chore — can hand a deleted key back. compact(newest 2, drop_tombstones=True) takes get(apple) from None to 'red'; keeping the tombstone, or widening the merge to all three segments, leaves it dead. The 11 bytes that compaction reclaimed were the tombstone — the only evidence the delete ever happened.
The same 100,000 keys indexed twice — and the sorted load builds the worse index. Ascending inserts only ever split the rightmost leaf, sealing every left half at exactly 32/63 = 50.8% full, while shuffled inserts settle on ln 2 = 69.3%: 38% more pages, and at this N one extra level, so every lookup costs 4 page reads instead of 3 forever. Postgres has a special case in nbtsplitloc.c for exactly this; adding it here takes the sorted index from worst to best.
One boolean query, the same five documents, three ways to find them — 110 steps, 6,204, or 12,409. The 113× spread comes down to a single line, if len(a) > len(b): a, b = b, a, deciding which postings list drives the loop. Then the part that places your own system: on two common terms the clever strategy loses to the naive merge by 1.9×, because selectivity skew — not galloping — was the resource being exploited all along.
"Readers don't block writers" is true — 50 writers finish while one reader sits idle. But that reader read only key a, and it pins all 51 versions of b, a key it never touched. The real reclamation rule frees 0 of 52 versions; a precise rule would free 49 at the same instant. Commit the idle reader, change nothing else, re-run the identical sweep: 50 vanish.
Eight nodes and a flawless hash function, and one node holds 4003 of 10,000 keys while another holds 54 — raising the key count to a million doesn't help, it just converges on the same 3.18×. Then the part nobody predicts: adding a 9th node to relieve the hot shard leaves it completely untouched in 64 of 100 placements. Virtual nodes barely change how many keys move (14.80% → 10.77%); what they change is who donates them, from 1 node to all 8.
LFU beats LRU by 11 points on a stationary Zipf trace, exactly as everyone expects. Then the hot set shifts once — same code, same capacity — and LFU falls from 87.21% to 4.16% while LRU doesn't move. It's holding 19 keys with counts of 851–938 that will never be requested again, and a newcomer admitted at count 1 is evicted by the very next miss: 19 frozen slots and one revolving door, still frozen 400,000 requests later.
A "1% false positive rate" is not a 1% chance per query — it is 1% of your key space broken on 100% of its queries, forever. One key, key:431, returns positive on all 10,000 lookups while an equally absent key:100 returns zero. Retry can never help, because the seven bit positions are a pure function of the key; only rehashing does, and it just curses a different set of strangers instead.
A valid inclusion proof for a block that was never in the file, forged with two concatenations and a list slice — no collision search, no key. With 64-byte blocks, H(blk0) ‖ H(blk1) is itself a legal block whose leaf hash is the node above it, so it verifies against the real root with a proof of length 1 instead of 2. The fix is one byte per hash — and it turns out to fix only half the problem.
An entry sits on three of five servers — a real majority — and the leader is forbidden to call it committed, because it was created two terms ago. Two steps later it exists on no server at all. Both commit rules run the identical schedule; the only difference is one clause, self.log[n - 1].term != self.term. Raft does not prevent the overwrite — it prevents the lie. And weakening the election restriction to a length comparison changes nothing, so the rule everyone quotes is not the rule doing the work.
One NFA, two ways to walk it: 61 states, the same answer, and 13,631,486 steps against 841. Then the part that relocates the blame — swapping the backtracker's two adjacent stack.append lines, which arrow of a SPLIT it tries first, takes the same input from 13.6 million steps to 41 without changing a single answer. The exponent was never in the backtracking; it was in the order. Delete the simulator's seen set and it reproduces the backtracker's count to the digit.
The half-open probe is a measurement taken at a load of one and acted on at a load of everything. Against a dependency that is simply down, that inference is sound: failed calls fall 2200 → 41 for free. Against one that fails because of load, the probe succeeds every single time and is refuted the very next tick — a limit cycle completing 3.5 requests per tick against a 5/tick arrival rate, so a 10-tick spike becomes an unbounded outage on a dependency with 4× headroom. The knob that decides it is the cooldown, not the error threshold, which is inert from 0.30 to 0.99.
DistributedClient A holds the lease, pauses past its expiry, B legitimately acquires, and A's write still lands and erases B's. The lock servers are provably innocent — every grant they ever issued is printed as an interval and no two overlap, OVERLAPPING LEASES: NONE, right beside the corrupted value. Nothing you can do to the lock fixes it: N = 1 through 51 servers corrupt identically, no lease length is safe against an unbounded pause, and re-checking the lease just before writing still corrupts, because every check during the lease truthfully says you hold it.
One token on one line of the parse loop decides associativity, and flipping it turns 2-3-4 into 3. The lesson is how little notices: 324 of 432 three-operand expressions agree either way, every divergence traced to just four operator pairs, and a plausible 18-test calculator suite catches it 4 times. Invisibility decays with length — 87.5% agree at three operands, 33% at five. And the same edit that is a bug for - is the fix for ^.
A task asking for sleep(1000ms) beside one 200ms-per-await neighbour fires every 1400ms, forever, and nothing raises. The timer is not late — the loop's own instrumentation reports zero lateness on every delivery; the 400ms lives in the queue behind it, where no instrument points. Real asyncio.sleep costs 200ms more than call_later because it resolves a Future first, measured on CPython. One line does it: replace the ready-queue snapshot with while self.ready: and the timer task gets 0 ticks in 10,000 virtual seconds.
Write a secret, delete it, write a 3-byte file that recycles its block — then read 61 bytes of the deleted plaintext back out through the filesystem's own read(), on a file you own, after a perfectly legal truncate. No hex editor, no raw device, no privilege. Zeroing on free fixes it and turns rm of a 4KB file from 0 block writes into 64, which is why no mainstream filesystem does it — ext4 flags unwritten extents instead, and the 2012 proposal to let apps opt out was rejected.
The client is told its write FAILED. One read later that write is the permanent, durable value — read-repair promoted it. Before the read, exactly the 6 of 10 read sets touching the one replica that took it return it; after, all 10, and no read can ever lower that again. Then the part that inverts the instinct: raising W makes failed-but-permanent writes strictly more common, W−1 of them across W=1..5, so the safest-looking setting maximises writes whose reported outcome is the opposite of the truth.
Two responses from the same raw-socket server, differing only in framing headers. One prints the whole body and curl exits 28; the other exits 0 and hands back 'the framin'. The exit code reports on the framing, not on the body — and Content-Length 24–27 truncates silently while 28 errors loudly, one digit apart. Connection: close as a header changes nothing; the FIN does. Driven by real curl, because a client that graded its own framing would prove nothing.
Steal from the "wrong" end of the deque — violating the rule every work-stealing scheduler follows — and on an uneven bag of tasks it costs nothing: −4.8 ticks on average, and the wrong end wins 271 of 500 orderings. Remove the size gradient entirely and the two ends become literally interchangeable, same makespan, same steal count, same failures, to the integer. Then the resolution: on a divide-and-conquer tree the far end holds the largest undivided subtree, one steal moves 121 ticks instead of 33, and the rule is worth 1.62×.
OS-ishEvery byte in the arena is free — 256 of 256, zero live allocations — and malloc(96) returns NULL, with 2.67× the requested space available. Free the same four blocks in the opposite order and it succeeds. The cause is that coalescing is local and a header-only heap can only merge forward, so ascending frees find every neighbour still live and merge nothing. Exactly 1 of 24 free orders fails — and it is the ascending one, which is what a loop that allocates then drains front-to-back actually produces.
The adaptive detector adapts by being wrong once. On the single heartbeat where the network changes regime, phi-accrual convicts a node that is alive and still sending — 173 of 200 traces fire on that exact beat, 45% of all its false alarms. Four beats later, having learned, it shrugs at a longer silence than the one it killed for: 132.6 ms guilty, 142.8 ms innocent, and 80.5% of traces show that inversion. A dumb 250 ms timeout beats it on both axes at once, and the fix production ships is a hard-coded floor under sigma.
NetworkingDrop the newest message or evict the oldest — the overflow policy you agonise over cannot change how much you lose. Both lose exactly 791 of 1000, in all 530 configurations tested, because both leave the queue at exactly capacity so the occupancy trajectory is policy-independent. Backpressure loses the same 791 too, just at the publisher. What the policy does decide is which messages survive, and whether your subscriber runs 2.0 ticks behind reality or 9.7.
DistributedEveryone knows one participant voting NO aborts the transaction. That is the advertised contract, and it is also the safe case: crash the coordinator at every step of the schedule and the one-NO run blocks 0 of 23 timings, while the all-YES run — the one that succeeds — blocks 10 of 25. The NO voter never entered the uncertainty period, so it stays a live oracle. In 9 of those 10 blocked timings the coordinator had decided nothing and abort was legal all along; the participants hold their locks anyway, because they cannot tell those 9 apart from the 1.
SecurityTamper one byte and the signature fails — that is a MAC working, and every reader predicts it. Here is what they do not: a verifier that allowlists only HS256 and RS256, two strong and entirely real algorithms, still hands over role: admin. The attacker signs with the server's public key as the HMAC secret, the server recomputes the same HMAC, and both sides do the arithmetic correctly. Zero secret bits required. alg:none is famous and dies at the reflex fix; this one survives it.
A job that gives up the CPU just before its slice expires gets 980 of 1000 ticks; two identical jobs that run flat out get 10 each. 98×, for sacrificing one tick in ten. The scheduler is not fooled, it is blind: the gamer and a genuinely interactive job hand the demotion rule byte-identical evidence across 96 dispatches and receive 20.6× different CPU. Fixing the accounting kills the exploit and mugs the honest job, 8.4 to 32.2 ticks of latency — and the textbook priority boost, applied alone, fixes nothing at all.
ObservabilityOne backend breaks and starts failing in 1 tick instead of serving in 10. To a latency-aware balancer it is now the fastest, least-loaded machine in the fleet, so it gets everything: goodput collapses 4806 → 5 of 6407 while p99 improves 41 → 3. The metric ranks the policies backwards. Both worlds are bit-identical to the balancer — same routing vector, same EWMA — so no latency metric can ever see it, and the boundary is exact: the effect reverses the moment errors take longer than successes.
ConcurrencyA library function five frames down gains one line — it now suspends between reading a row and returning it. Nothing above it is edited. The stackful runtime returns 150 where 200 is the only correct answer; the stackless one refuses to run at all, with a TypeError about adding a generator to an int. Colour all four frames to make it run and it returns 150 too: the colouring bought visibility, not correctness. Against real asyncio and real threads, the cooperative version is wrong on 20 of 20 trials and the preemptive one on 0 of 20 — cooperative scheduling didn't make the race rarer, it made it the only behaviour the program has.
Point two clients at one pooled upstream connection and the proxy does not multiplex them — it misdelivers. B asks for beta and is handed alpha's value; A gets nothing and hangs. Every byte was forwarded, in order, exactly once, so there is no bug to fix: a raw TCP reply carries no information about whose request it answers. Wrong on 23 of 24 responses, from a relay that is behaving correctly. The pool's checkout boundary turns out to be a parse — and the identical byte-shuffler is 24/24 correct at concurrency 1, which is where the idea is true and worthless.
The O(log n) everyone quotes is true, which is exactly why it is not the lesson: the ratio is flat at ~1.82 from 8 nodes to 1024, just as you would predict. What it hides is where the cost goes. Push reaches half the cluster in 10 rounds and needs 8 more for the last node, and rounds that begin above 90% coverage burn 63.6% of all messages to inform 5.2% of the cluster — 492 calls per node in the final 1% against 1.2 in the first quarter. The mechanism is derivable: push divides the residual by e each round, pull squares it.
"They converge without a coordinator" is the definition of a CRDT, and it held in ~30,000 runs — including the runs where the merge rule was deliberately broken. A claim that survives breaking the mechanism is not teaching it. Here is one that bites: gossip 60× harder and the OR-Set's error falls 15×, while a last-write-wins store does not move at all. Its converged value was identical on 200 of 200 traces across nine sync schedules, because the network never appears in the answer — only the clocks do.
DistributedA Snowflake ID packs a timestamp, a machine id and a sequence into 64 bits, and uniqueness lives in none of those fields — it lives in the generator's memory of its own last millisecond. Step the clock back 50ms and the naive one mints 400 duplicates. Twitter's fix refuses to serve instead, which is correct until a liveness probe restarts the stalled process: then it mints 390. It bought ten IDs, exactly 400 − K, because a refusal converts one duplicate and then deletes the state that made it correct.
Compile a 2,048-leaf expression and the compiler emits 4,095 instructions for 4,095 AST nodes — ratio 1.0000, and zero deviation across 2,000 programs. Compiling removed no work at all; it moved it. The dispatch loop turns out to be the least valuable thing you got: compiling the same tree to nested closures, with no opcodes and no loop, beats every bytecode variant by more than 2×. The real wins are that the loop is a thing you can write in C, and that the IR is a thing you can fold — 4,095 instructions down to one.
SecurityA 9-block chain mined with 21 hashes outranks an honest 8-block chain that cost 2,638,572 — 125,646× cheaper, and validate() returns True. Nothing is forged: SHA-256 is intact, every pointer links, every declared difficulty target is genuinely met. It is an accounting error, not a cryptographic one, and exactly one of four validator/fork-rule combinations is broken by it. The whitepaper's "longest chain" means the heaviest one, and a reader who implements it literally builds the broken cell.
One trace, one reservoir, 2,000 seeds: the reported p99 swings 35.50× while the p50 swings 1.54×, and the median reading is less than half the true p99 — with no error bar. Everyone blames sloppy tail sampling. That is measurably false: the p99 slot is located 3.59× more tightly than the p50 slot. The damage is the steepness of the distribution under it, not the sampler above it. Spend the same 100 units of memory on fixed buckets and you get the right answer, deterministically.
All 35 entries in BACKLOG.md are built. Every one was prototyped as a throwaway first and only then written up, which turned out to matter more than it sounds: of the 45 designs prototyped, not one survived contact with a real run unchanged. Forty-two were sharpened or replaced outright, and three — failure-detector, circuit-breaker and tcp-proxy — were falsified by the prototype meant to confirm them.
The falsifications are the ones worth reading. A fixed timeout beats phi-accrual on both axes; a circuit breaker's error threshold is inert across most of its range; and a pooled TCP proxy does not multiplex connections at all, it misdelivers them. Each of those pages leads with the thing its own author expected to find and did not.
New toys will show up here when a mechanism turns out to have an aha worth 150 lines. The bar is a specific behaviour a careful reader would predict wrongly — not a topic. If you have a candidate, the ground rules for what counts are in README.md.
The full list, with the aha each demo should produce, is in BACKLOG.md. If you'd rather read the finished ones in order than pick at random: wal-kv → lsm-tree → btree-index → consistent-hashing → mvcc-store, which runs single-node durability → storage engine → on-disk index → distributed routing → concurrency control.