Commentary: circuit-breaker

One breaker, one trace, two dependencies. The half-open probe is a measurement taken at a load of one, acted on at a load of everything — and only one of the two dependencies makes that inference safe. A study guide for circuit_breaker.py.

cd circuit-breaker
python3 demo.py                  # the aha (§6)
python3 test_circuit_breaker.py  # pins the trace this page describes

1. Orientation

A circuit breaker watches the calls you make to a dependency, and when too many of them fail it stops making them: closed (calls pass), open (calls are rejected without being attempted), half-open (one call is let through to see whether the dependency is back). That much is a familiar diagram, and if the toy stopped there it would only be teaching you to read a state chart.

The thing worth building it for is the half-open probe, and what it assumes. The probe is a single call. The breaker takes that one call's outcome as a verdict on the whole dependency and, if it succeeds, immediately resumes sending everything. That inference — from one call at essentially zero load, to the health of a dependency about to receive a herd — is only valid for a dependency whose health does not depend on load.

So this toy runs one breaker, with one configuration, against one arrival trace, and swaps out only the dependency:

• DeadDependency — down for ten ticks. Whether it works is a function of the clock and nothing else.
• OverloadedDependency — serves 20 calls a tick and rejects the rest. Whether it works is a function of the offered load and nothing else. It is never "down".

Both start failing at exactly tick 10. The breaker cannot tell them apart, because the only measurement it ever takes is a single call. Against the first, that probe is a valid sample and the breaker is free money. Against the second, every probe succeeds, is contradicted by the very next tick, and the breaker converts a ten-tick surge into an outage that never ends.

By the end you should be able to:

• trace the closed → open → half-open → closed cycle by hand against a request trace, and say which line moved the state;
• derive a breaker's effective throughput from its duty cycle, and use that number to predict whether a system under it recovers or diverges;
• name the knob that actually decides that (it is not the one with "threshold" in its name), and show that the obvious knob does nothing;
• say, for a dependency you actually operate, whether a breaker in front of it will help or hurt — by asking one question about why it fails.

2. The problem this mechanism exists to solve

You call a service over a network. It stops working. What should your code do?

The naive answer — keep calling, let each call fail — is genuinely bad, and for a reason that is easy to underrate. A remote call that fails fast costs you almost nothing. A remote call that hangs costs you a thread, a connection, a slot in a pool, and a multi-second timeout, for every call, for as long as the dependency is unwell. Michael Nygard's original framing in Release It! is that this is how a single sick dependency takes down the system in front of it: the callers run out of threads waiting on it, and then they fail for requests that never needed the sick dependency at all. The breaker's first job is to keep a dead dependency from consuming its callers.

There is a second job, which is where the trouble starts: protecting the dependency. A service that is falling over is often falling over because of the traffic it is receiving, and the argument goes that removing the traffic gives it room to recover.

A dependency failing for its own reasons — a bad deploy, a dead node, an expired certificate — is serving zero requests successfully. Sending it nothing costs nothing. A dependency failing because of load is, by definition, still serving as much as it can. Sending it nothing throws that away.

And the breaker has to decide which situation it is in using the only instrument it has: the outcome of calls it made. When it is open it is making no calls, so it has no information at all — which is why it must eventually guess, let one call through, and generalise from it.

The competing goals that make more than one design defensible:

• React fast vs. don't flap. A low threshold trips on a blip; a high one leaves the caller bleeding threads for longer.
• Recover fast vs. don't re-break it. A short cooldown retries the dependency often; a long one keeps traffic off it but, as §6 shows, spends most of its time refusing to use capacity that exists.
• Cheap and local vs. informed. A breaker is a client-side heuristic with no knowledge of the server. The alternatives (§7.5) require the server to say something about itself, which is more accurate and much more work.

3. Background you need

None of this is deep, but the commentary below leans on it.

The two that carry the result are duty cycle and congestion collapse. Everything in §6.4 is one arithmetic step away from the first, and the second is the name for what that arithmetic produces. If you only internalise one row, take the duty cycle: a breaker that is open four ticks in six has cut its dependency's throughput by two thirds, and it did that on purpose.

4. The mental model

Before any code. The state machine first, since it is the part you already half-know:

Now the part that is actually the toy. Put the same breaker in front of two dependencies that both start failing at tick 10:

5. Reading the source

The file is 200 lines. Read it in this order.

5.1 The two dependencies — six lines that are the entire experiment

class DeadDependency:
    """Fails every call between tick `down` and tick `up`, and succeeds
    every call outside that interval. `k` is ignored: how much load it is
    offered has no bearing on whether it works.
    """

    def __init__(self, down, up):
        self.down, self.up = down, up

    def call(self, now, k):
        if self.down <= now < self.up:
            return 0, k
        return k, 0

class OverloadedDependency:
    """Serves `capacity` calls per tick and rejects the rest. `now` is
    ignored: the offered load `k` is the only thing that decides whether a
    call fails. It is never "down" — at k <= capacity it is perfect.
    """

    def __init__(self, capacity):
        self.capacity = capacity

    def call(self, now, k):
        ok = min(k, self.capacity)
        return ok, k - ok

Start here, because everything else in the toy is machinery and these are the claim. Read the two call bodies as a pair:

• DeadDependency.call branches on now and ignores k.
• OverloadedDependency.call branches on k and ignores now.

They are deliberately built as each other's complement. Each takes exactly one of the two arguments seriously, so "why did this fail?" has a single-word answer in each case, and the two answers are the only two answers there are. Anything you can measure about the rate of failures is identical between them at tick 10 — both go from perfect to badly broken — so no amount of cleverness in the breaker's trip logic can distinguish them. The only thing that separates them is a question the breaker never asks: does load change the answer?

OverloadedDependency is deliberately the kindest possible overloaded service. It degrades gracefully: it serves its full 20 and fast-rejects the overflow, with no queueing, no thrashing, no slow-down under load, and it recovers the instant the load drops. A real overloaded service is worse than this in every direction. The toy uses the flattering version so that when the breaker makes things worse, you cannot blame the dependency's own dynamics.

5.2 CircuitBreaker.__init__ — the knobs, and where they came from

    def __init__(self, window=20, min_calls=10, error_rate=0.5, cooldown=5,
                 probes=1):
        self.window = window
        self.min_calls = min_calls
        self.error_rate = error_rate
        self.cooldown = cooldown
        self.probes = probes
        self.state = "closed"
        self.outcomes = deque(maxlen=window)
        self.opened_at = None
        self.probe_tick = None
        self.probes_used = 0

These are not invented. Each maps one-to-one onto a knob in a breaker people actually run:

That table is worth more than it looks, because §7.1 and §7.2 are about which of these rows matter. Hystrix's documentation describes the half-open state as letting "the next single request" through, with no knob at all; resilience4j exposes one and defaults it to 3. §7.2 measures what those choices are worth here.

self.outcomes = deque(maxlen=window) is the whole memory of the breaker when closed: a bounded ring of booleans. maxlen does the eviction, so there is no explicit trimming anywhere — appending the 21st outcome silently drops the 1st.

5.3 allow — the only place open → half_open happens

    def allow(self, now):
        """Gate one call. The only place open -> half_open happens."""
        if self.state == "open":
            if now - self.opened_at < self.cooldown:
                return False
            self.state = "half_open"
            self.probe_tick = now
            self.probes_used = 0

        if self.state == "half_open":
            if now != self.probe_tick:      # a fresh tick gets a fresh budget
                self.probe_tick = now
                self.probes_used = 0
            if self.probes_used >= self.probes:
                return False
            self.probes_used += 1
            return True

        return True

Three decisions here are worth slowing down for.

The cooldown expiry is checked lazily, on a call. There is no timer and no background task; open → half_open happens inside a request, the first time one arrives after the cooldown has elapsed. This is the same lazy-refill trick a token bucket uses, and it has the same payoff: a breaker guarding a dependency nobody is calling costs nothing, and there is no clock to skew. It also has a consequence people trip over in production: a breaker for an endpoint with no traffic never leaves the open state, because leaving it requires a request to arrive and ask.

The two ifs are sequential, not elif. The first block can set half_open, and the second block then runs in the same call — so the request that discovers the cooldown has expired is itself the probe. Making the second an elif would cost one extra request-time before probing.

< self.cooldown, not <=. With cooldown=5 and opened_at=0, ticks 1 through 4 are rejected and tick 5 probes. I ran the one-character counterfactual rather than reasoning about it:

One character delays the dead dependency's recovery by two ticks (21 → 23) and costs the overloaded one 190 completed requests over 400 ticks. Note the direction: the character makes the cooldown one tick longer, and the overloaded case gets worse. That is the first hint of §6.4 — with this breaker, on this dependency, time spent open is not free, it is throughput you deleted.

test_open_rejects_until_exactly_cooldown (test_circuit_breaker.py lines 61–70) pins the boundary.

5.4 record — one probe decides everything

    def record(self, ok, now):
        """Feed back the outcome of a call that allow() admitted."""
        if self.state == "half_open":
            # A probe is a verdict on the whole dependency, either way.
            if ok:
                self.state = "closed"
                self.outcomes.clear()
            else:
                self._trip(now)
            return

        self.outcomes.append(ok)
        if len(self.outcomes) < self.min_calls:
            return
        if self.outcomes.count(False) / len(self.outcomes) >= self.error_rate:
            self._trip(now)

This is the load-bearing method, and the load-bearing branch is the first one. Look at the asymmetry between the two halves:

• To go closed → open, the breaker demands at least min_calls outcomes and a sustained error rate across them. It is statistically careful.
• To go half-open → closed, it accepts one call. No volume floor, no rate, no second opinion. And the state it transitions into is "send absolutely everything."

The care on the way down and the credulity on the way up are hard to justify together. The usual defence is that a probe is expensive — you are deliberately sacrificing a real user's request to a possibly-dead service, so you take as few as you can. That defence is about the cost of the sample. It says nothing about whether the sample is representative, and §6 is entirely about the gap between those two things.

self.outcomes.clear() on close (and in _trip) means the breaker never carries evidence across a state change. It reads like a load-bearing line — surely the failures still sitting in the ring from before the trip would re-trip the breaker the moment it closed? — so I wrote both counterfactuals and ran them:

Identical, all six. Neither clear() matters on this trace, and the reason is maxlen=20: the tick after a close delivers hundreds of outcomes, so the ring is completely overwritten before anything can be read from it. The line is correct defensive hygiene — it would matter for a dependency called a few times a tick, where 20 stale failures could survive several ticks — but on this workload it is not where the behaviour lives. Worth knowing before writing a paragraph claiming otherwise.

5.5 simulate — where the herd comes from

        for attempts in queue:
            if breaker.allow(now):
                admitted.append(attempts)
            elif max_attempts is not None and attempts + 1 >= max_attempts:
                dropped += 1
            else:
                pending.append(attempts + 1)

A request the breaker refuses is not discarded — it goes on pending and is offered again on the next tick. That is the single most consequential line in the driver, and it is there because it is what real clients do. A user whose page failed reloads it; an SDK with a retry policy retries; a queue consumer that couldn't process a message leaves it on the queue.

max_attempts exists so you can turn that off and measure the difference (§8.1); the demo leaves it at None.

6. The demo, and what it proves

demo.py builds one trace — 5 requests a tick, surging to 40 for ticks 10 through 19 — and one breaker configuration, and runs both dependencies through them.

BASE = 5            # requests per tick in steady state
SPIKE = 40          # requests per tick during the surge
SURGE = (10, 20)    # ticks [10, 20) — also exactly when the dead one is down
CAPACITY = 20       # the overloaded dependency's ceiling, 4x the baseline

Note CAPACITY = 20 against BASE = 5. The overloaded dependency has four times the headroom it needs in steady state. It is not a marginal service; it is a comfortably provisioned one that gets briefly hit with 40.

def breaker():
    return CircuitBreaker(window=20, min_calls=10, error_rate=0.5,
                          cooldown=5, probes=1)

One factory, called for both columns. Same window, same threshold, same cooldown, same single probe.

6.1 The transcript

python3 demo.py

The retry column is the number of requests sitting in the caller waiting for another go. On the left it returns to 0 at tick 21 and stays there. On the right it never returns to 0 again.

6.2 The dead dependency: the breaker is free money

Derive the two headline numbers.

Calls delivered to a dead service, without a breaker: 2775. The dependency is down for ticks 10–19. At tick 10, 40 requests arrive and all 40 fail; at tick 11 those 40 retry alongside 40 new ones, so 80 calls fail; and so on. That is 40 × (1 + 2 + … + 10) = 40 × 55 = 2200 — which is exactly the failed column. Add the calls that worked: 5 a tick for ticks 0–9, the 405 that drain at tick 20 (400 backlogged plus 5 new), and 5 a tick for ticks 21–44. 50 + 405 + 120 = 575, which is the completed column, and 2200 + 575 = 2775. Every one of those 2200 failures occupied a connection and waited out a timeout for nothing.

With a breaker: 616 calls, of which 41 failed. The 41 is the whole story: 40 calls at tick 10 (the breaker needs min_calls=10 outcomes before it can trip, and it gets them mid-tick, but the tick's calls were all admitted before any outcome came back), plus one failed probe at tick 15. That is a 98.1% reduction in doomed calls — 1 − 41/2200.

And it cost nothing. Look at completed in both rows: 575 and 575 over 45 ticks, 2350 and 2350 over 400. Not one request was lost by shedding, because a dead dependency was completing zero requests anyway. The breaker's only price is one tick of recovery lag — over at 20 becomes over at 21 — because the dependency comes back at tick 20 and the breaker's probe finds out at tick 20, closing in time for tick 21 to drain the entire 409-deep backlog in one go.

The two probes are the reason it works, and they are worth reading as measurements:

Two probes, two correct verdicts. At tick 15 the service is still down and the probe fails — correctly. At tick 20 it is back and the probe succeeds — correctly. And the tick after that success, when 409 calls arrive instead of 1, all 409 succeed. The one-call sample generalised to a 409-call load, because for this dependency the number of calls was never part of the answer.

6.3 The overloaded dependency: every probe succeeds, every probe is wrong

The same measurement, same breaker, other column:

Five probes. Five successes. Five refutations, each one tick later.

The probe at tick 15 sends 1 call. min(1, 20) is 1, so it succeeds — and it should, because the dependency genuinely is healthy at a load of 1. The breaker closes on that evidence and at tick 16 delivers the 259 requests it has been storing. min(259, 20) is 20. Two hundred and thirty-nine fail, the error rate is 92.3%, and the breaker trips again.

Nothing here is a bug in the breaker. It measured correctly, drew a valid conclusion about the load level it measured, and that load level ceased to exist the instant it acted on it. test_probe_verdict_generalizes_only_for_the_dead_dependency (test_circuit_breaker.py lines 104–128) asserts all ten of those tuples.

6.4 The arithmetic: why it never ends

The interesting claim is not that the breaker flaps. It is that the flapping is stable and lossy, and you can compute exactly how lossy.

From tick 21 the trace settles into a cycle. Here it is at tick 300, long after the surge is a memory:

The period is 6 ticks: four open, one half-open, one closed. Count what the dependency completes in one period:

So the breaker's effective throughput is 21/6 = 3.50 requests per tick. Arrivals are 5 per tick, or 30 per period. The deficit is 30 − 21 = 9 requests per cycle, and you can watch it in the retry column: 777 at tick 300, 786 at 306, 795 at 312. Exactly 9, forever. test_limit_cycle_arithmetic (test_circuit_breaker.py lines 151–162) asserts all of it.

Now put that number next to the dependency's actual capacity:

The dependency had 4× headroom. The breaker gave it a 30% deficit. It did this not by malfunctioning but by working exactly as designed — spending five ticks in six not using a service that was, throughout, ready to serve 20 a tick. The surge lasted 10 ticks. Without a breaker the backlog drains and the incident is over at 33. With one, at tick 400 the backlog is 935, and at tick 3000 it is 4822 — still rising, at the same 1.5 a tick.

6.5 The boundary condition — where the effect vanishes

Two boundaries, both measured, and you need both to place your own system.

Boundary one: no overload, no divergence. If the surge never exceeds capacity, there are no failures, the breaker never trips, and the two arms are byte-identical:

test_boundary_no_overload_no_divergence (test_circuit_breaker.py lines 184–191) asserts the two totals() dicts are equal. A breaker in front of a dependency that never saturates is inert, and its configuration does not matter at all.

Boundary two — the sharp one: the baseline rate versus 3.50/tick. The limit cycle is only unbounded because arrivals (5/tick) exceed the breaker's effective throughput (3.50/tick). Lower the baseline and the same breaker, the same dependency and the same surge recover:

The measured backlog slope is the arithmetic and nothing else: at 5/tick the deficit is 5 − 3.50 = 1.50, and the measured slope is +1.50. At 4/tick it is +0.50. At 3/tick the baseline is below 3.50, the slope is flat, and the incident ends at tick 701.

6.6 The trace is pinned by tests

python3 test_circuit_breaker.py

The demo output is byte-identical across runs and across PYTHONHASHSEED values, because there is no clock and no RNG anywhere in the toy — see §7.4.

7. Design decisions and roads not taken

7.1 The knob everyone tunes does nothing

error_rate is the knob with "threshold" in its name in every real implementation, the one that shows up in every tuning guide, and the one an on-call engineer reaches for after an incident like this. Here is what it is worth on the overloaded dependency, over 400 ticks:

Not "barely moves" — identical, to the request, across a threshold range from 30% to 99%. test_cooldown_is_the_load_bearing_knob (test_circuit_breaker.py lines 165–181) asserts the totals() dicts are equal for all five.

The reason is worth having, because it generalises well beyond this toy. Once the herd exists, the tick after each close delivers 259, 373, 382, 391, 400 calls against a capacity of 20 — error rates of 92%, 95%, 95%, 95%, 95%. Any threshold below 92% trips on that. The threshold is a knob for distinguishing degrees of unhealthiness, and a saturated dependency does not offer degrees: it offers capacity successes and everything else. You cannot tune your way out of this with the trip threshold, because the trip is correct. It is the close that was wrong.

7.2 The knob that does everything is the cooldown, and it has a cliff

§6.4 gives the model: the breaker's effective throughput is one closed tick's worth of capacity plus one probe, spread over the whole cycle.

and the system is stable exactly when that exceeds the baseline arrival rate. With capacity=20, probes=1 and a baseline of 5, that predicts a cliff between cooldown=3 (5.25/tick, stable) and cooldown=4 (4.20/tick, unstable). Measured against 3000 ticks:

The predicted surplus and the measured backlog slope are the same number with the sign flipped, at every row. The cliff is exactly where the formula puts it: cooldown=3 drains, slowly, finishing at tick 1315; cooldown=4 never does. test_cooldown_is_the_load_bearing_knob pins both.

7.3 The probe budget: tuning against a number you cannot see

probes is resilience4j's permittedNumberOfCallsInHalfOpenState, and it is the other half of the same fraction. Raising it raises the numerator. Over 400 ticks on the overloaded dependency:

Note probes=3 — resilience4j's default — is still catastrophic, and probes=40 is catastrophic again after probes=20 worked. Sweeping every value from 1 to 59:

There is a correct setting, and it is a window bracketing the dependency's capacity of 20. Below 16 the duty cycle is too thin to keep up with the baseline. Above 29 the probe batch itself exceeds what the dependency can serve, so the probe fails and the breaker re-opens without ever closing — here is probes=40 failing inside its own probe tick:

So the knob is tunable in principle and untunable in practice: the target is a number on the server side, that the client has no way to observe, that changes with every deploy and every autoscaling event. This is the deep problem with the whole approach, and §7.5 is about what people do instead.

7.4 Roads not taken inside the breaker

Requiring several consecutive probe successes before closing. The obvious fix for a bad sample is more samples, so I built it and ran it:

It makes the overloaded case worse (1415 → 1175 completed), and the dead case worse too (over at 21 → 23). Both for the same reason: the extra probes are admitted one per tick, so they lengthen the cycle without measuring anything new. Three samples at a load of 1 tell you precisely what one sample at a load of 1 told you, and you paid two more open ticks for them. This is the clearest single result in the toy: the problem is not the number of observations, it is the load they were taken at.

Not refreshing the probe budget each tick (making probes a budget for the whole half-open period rather than per tick). At probes=1 this is measurably identical (1415 completed, backlog 935, both dependencies unchanged), so the if now != self.probe_tick reset in allow is not a load-bearing line at the demo's settings. Worth knowing before writing a paragraph claiming it is.

A Half-open class, or a state pattern. Three states, three string literals, and transitions in two methods. An enum would be tidier and a state class per state would be a file's worth of ceremony hiding the thing worth seeing, which is that of the four transitions allow owns exactly one (open → half_open) and record owns the other three.

No clock, no RNG. allow(now) takes integer ticks and the dependencies are pure functions of (now, k). The usual reason is testability, but here there is a stronger one: the aha is a limit cycle, and a limit cycle observed under jitter is indistinguishable from noise. Making the toy fully deterministic is what lets §6.4 say "exactly 9 per cycle, forever" and have a test assert it. The output is byte-identical across PYTHONHASHSEED=0, 1, 42 and 12345.

7.5 The famous alternatives, and why they are not here

• Load shedding at the server. Instead of the client guessing, the server rejects excess work itself, cheaply and immediately, keeping goodput at capacity instead of at capacity × duty cycle. This is what the overloaded dependency in this toy already does internally — and it is exactly why the no-breaker column recovers.
• Adaptive concurrency limits (Netflix's concurrency-limits, TCP-Vegas-style). The client measures latency and adjusts an in-flight limit continuously, so it converges to the dependency's real capacity instead of bracketing it by hand as in §7.3. This is the actual answer to this toy's problem, and implementing it would have made the toy about congestion control rather than about breakers.
• Bulkheads. A bounded pool per dependency, so a slow dependency can only consume its own budget of threads. This addresses the caller-protection half of §2 without touching load at all, and it does not have the failure mode in §6.4, because it never goes to zero.
• Retry budgets and backoff with jitter. §8.1 measures how much of this toy's headline belongs to the retry policy.

A rate limiter is the closest neighbour, and this repo has one: see rate-limiter. A token bucket in place of this breaker would have throttled the surge to a steady 20/tick instead of alternating between 0 and 400 — the same "reduce load" goal with a continuous knob rather than a switch.

8. What's simplified vs. the real thing

8.1 The retry policy is a co-author of the headline — measured

This is the objection to make first, so §6.4's headline gets stated honestly: isn't this just congestion collapse wearing a breaker costume? Partly yes. The simulate loop retries forever (§5.5), and an unbounded retry policy is independently capable of destroying a system with no breaker in sight.

max_attempts exists to measure the split. Capping every request at 3 attempts, over 400 ticks:

Two things are true at once, and the page needs both:

1. Bounding retries ends the catastrophe. "Backlog 935 and climbing at tick 400" becomes "drained at tick 22, 358 dropped". The unbounded limit cycle of §6.4 requires the retry policy as much as it requires the breaker. If you take one operational lesson from this page, take that one: an unbounded retry loop in front of a breaker is the actual hazard.
2. The breaker still owns its share. With retries bounded identically in both arms, the overloaded dependency drops 358 requests with a breaker against 160 without — 2.2× worse — while the dead dependency drops 359 against 320, a difference of 39 that is essentially the one failed probe and its retries. So the breaker's own contribution is real, it is specific to the load-induced case, and it survives the fix.

The mechanism behind that residual 2.2× is not the retries. It is that a breaker's only load-management move is all-or-nothing, and it spends most of an incident in "nothing" in front of a dependency that was serving its full 20 a tick throughout. test_bounded_retries_are_a_co_author (test_circuit_breaker.py lines 194–214) pins all four numbers.

8.2 The other simplifications

• Failure is binary and instant. A real call fails by hanging — that is most of why breakers exist (§2). Modelling latency would strengthen the breaker's case in both columns, because the wasted calls in §6.2 also cost wall-clock time and a connection each, which the toy charges nothing for. resilience4j's slowCallRateThreshold exists precisely because "slow" and "failed" need to be counted together.
• One breaker, one dependency, one process. Real deployments run a breaker per host, so an outage sees N independent breakers each probing on its own schedule — which, incidentally, is a partial accidental fix for §7.3, since N hosts each sending one probe deliver N probes. It is also a new problem: with no jitter they synchronise, and the herd in §6.3 arrives from every host at once.
• One key, one shard. The toy's dependency is a single undifferentiated service. Marc Brooker's objection (§10) is the sibling of this one: in a sharded or cell-based system, failures of the A–H shard tell a breaker nothing about the S–Z shard, so a breaker that trips takes down keys that were fine. That is the same "the sample does not generalise" problem, across keys instead of across load levels.
• No fallback. Hystrix's design assumes an open circuit returns a degraded answer — a cached value, an empty list, a default — not an error. Where a fallback exists, the calculus in §6.4 changes completely, because the open ticks are not lost work.
• The trip is instantaneous and global. _trip clears the window and flips a flag. Real breakers must do this across concurrent threads, so the ring buffer becomes an atomic or striped structure, and dozens of in-flight calls will still land after the trip.
• Ticks are not seconds. The whole toy runs in discrete ticks with all outcomes resolving inside the tick they were issued in. Real breakers see outcomes arrive out of order and delayed by exactly the timeout that made them interesting.

9. Check yourself

Answer before expanding. Each answer is derivable from the source.

10. Further reading

Every link below was fetched and confirmed live when this was written.

• Marc Brooker: Will circuit breakers solve my problems? — read this one first if you read only one. It reaches the same conclusion from a different direction: in a sharded or cell-based system, failures of one shard tell a breaker nothing about the others, so "is this database down?" has no answer the breaker can act on. That is this toy's "the sample does not generalise", across keys instead of across load levels. It also contains the retry argument that §8.1 measures.
• Martin Fowler: CircuitBreaker — the canonical statement of the pattern, and the source of the framing in §2 about remote calls that hang rather than fail. Short, and worth reading as the case for, since this page is largely the case against.
• Netflix Hystrix: How it Works — the implementation this toy's CircuitBreaker is modelled on, knob for knob (§5.2). Note the sentence "the next single request is let through": the half-open probe count is not configurable at all, which is §7.3's problem baked into the design. Also read the Isolation section on bulkheads, which is the part of Hystrix that solves §2's caller-protection problem without any of §6.4's cost.
• resilience4j: CircuitBreaker — the modern successor, and where permittedNumberOfCallsInHalfOpenState (default 3) comes from. Its slowCallRateThreshold is the answer to §8.2's "failure is binary and instant".
• AWS Builders' Library: Timeouts, retries, and backoff with jitter — the retry side of §8.1: retry budgets, exponential backoff, and why jitter is the part people skip and shouldn't. Directly applicable to Q5.
• AWS Builders' Library: Using load shedding to avoid overload — the server-side alternative from §7.5, including the goodput-versus-offered-load curve that §6.4 walks down.
• Google SRE Book: Handling Overload — criticality, adaptive throttling and the client-side request rejection formula. Their throttling probability is a continuous version of this toy's binary switch, which is precisely the fix.
• Netflix concurrency-limits — TCP-congestion-control ideas applied to RPC concurrency: the client converges on the dependency's capacity from latency signals instead of bracketing it by hand (§7.3). The road not taken, and the one worth taking.
• Wikipedia: Network congestion § Congestive collapse — the name for what §6.4 measures, and the original 1980s internet version of it. Useful for the vocabulary: goodput, offered load, and why the two can move in opposite directions.