Commentary: load-balancer

One fleet, one trace, four policies. A latency-aware balancer routes to whichever backend answers fastest — and the fastest answer in a fleet is an error. Every latency metric ranks the policies backwards. A study guide for load_balancer.py.

cd load-balancer
python3 demo.py                 # the aha (§6) — about 40 seconds
python3 test_load_balancer.py   # pins every number this page claims

1. Orientation

A load balancer takes a stream of requests and a set of backends and decides, per request, which backend gets it. The policies are famous and the comparison is a standard interview question: round-robin is dumb and fair, least-connections adapts to load, and latency-aware policies adapt to speed. The obvious lesson — different policies give different tail latency — is true, easy to demonstrate, and close to worthless, because it is predicted by the word "policy."

This toy exists to show you that the comparison runs the other way. It builds one fleet of four backends and routes one arrival trace through it with four policies, and the thing worth seeing is that every latency metric you could put on a dashboard ranks the four policies in the exact reverse of their goodput. The policy with the best p99 delivers 5 successful requests out of 6407.

The mechanism underneath is one sentence: a latency-aware balancer sends traffic to whichever backend answers fastest, and the fastest answer in a fleet is usually an error. A backend that has fallen over does not hang; it returns 503 in a microsecond, from a code path that touches no database. To a policy that measures round-trip time, that backend is not sick, it is excellent, and it gets promoted. The healthier a backend actually is, the slower it looks, and the less traffic it receives.

Google's SRE book names this exact failure and calls it sinkholing:

if a task is seriously unhealthy, it might start serving 100% errors. Depending on the nature of those errors, they may have very low latency; it's frequently significantly faster to just return an "I'm unhealthy!" error than to actually process a request.

So the toy runs one fleet, one trace, one set of policies, and flips exactly one boolean: is the fast backend genuinely fast, or is it broken? §6.1 shows that the balancer cannot tell — not "struggles to tell," but makes provably identical routing decisions in both worlds, down to the ninth decimal of its internal state.

By the end you should be able to:

• say what signal each policy actually reads, and what each one is therefore blind to;
• derive the traffic share a broken backend attracts under power-of-d choices, from d and N alone, and explain why more choices is worse;
• explain why a latency dashboard is structurally incapable of showing you this incident, even after you exclude the errors from it;
• decide, for a fleet you actually operate, whether latency-aware balancing will save you or sink you — by asking one question about how your backends fail.

2. The problem this mechanism exists to solve

You have one stream of requests and N machines that can serve them. Someone must choose. The choice is made millions of times a second, usually in a proxy or a client library, and it has to be cheap.

The naive answer, round-robin, hands out requests in rotation. It is O(1), needs no state beyond a counter, and is perfectly fair in request count. Its weakness is that request count is not the thing you care about. A backend on older hardware, a backend sharing a host with a noisy neighbour, a backend that just started and has a cold cache — all of them get exactly as much work as the fastest machine in the fleet, and the slow ones build queues while the fast ones idle.

So every serious balancer reads some runtime signal and steers by it:

• least-connections steers by the number of requests currently in flight to each backend. A backend that is slow accumulates in-flight requests, so it stops being chosen. This is nginx's least_conn and (via P2C) Envoy's LEAST_REQUEST.
• least-response-time / peak-EWMA steers by observed latency. Finagle's formulation keeps a moving average of round-trip time and multiplies it by the outstanding request count. It reacts to a degraded backend faster than least-connections, because latency rises before the queue does.

The competing goals that make more than one design defensible:

• Adapt vs. don't trust. A signal that lets you route around a slow backend is the same signal that lets a lying backend route traffic to itself. Round-robin cannot adapt, and for exactly that reason cannot be fooled.
• React fast vs. don't oscillate. Latency is a leading indicator, which is why it is attractive; it is also noisy, and steering hard on it makes fleets flap. The alpha in an EWMA is entirely about this trade.
• Global knowledge vs. O(1). Scoring every backend on every request is fine for four and unacceptable for four thousand, which is why power-of-d-choices exists.

And here is the seam this toy pries open. Every one of those signals is a proxy for the question you actually want answered, which is "how much useful work can this backend do for me?" Latency is a proxy for it. In-flight count is a proxy for it. Both proxies are excellent while every backend is trying to serve you, and both inverted the instant one stops.

Note what is deliberately not in this toy: retries, health checks, and circuit breaking. Those are the remediations, and this page is about the signal. ../circuit-breaker/ already owns retry storms and the all-or-nothing load shedder; adding retries here would let you blame the goodput collapse on the retry policy, which would be wrong and would hide the mechanism.

3. Background you need

None of this is deep, but the commentary below leans on it.

The two that carry the result are goodput vs. throughput and peak EWMA. If you only internalise one row, take the first: this toy's four policies all deliver about 6407 responses, and the number of them worth having ranges from 4806 to 5. Every metric in §6.2 that fails to see the incident is a metric computed over responses rather than over useful responses.

4. The mental model

Before any code. What each policy can see, and what it therefore cannot:

Now the fleet, and the one bit that changes:

The slogan to carry: a latency-aware balancer is a machine for finding the fastest backend, and a broken backend is the fastest thing in your fleet. Speed is being used as a proxy for capacity, and that proxy is exactly inverted by failure — because failing is the cheapest thing a server can do.

5. Reading the source

The file is 218 lines. Read it in this order.

5.1 Backend — the one equivalence the whole toy rests on

class Backend:
    """One server: FIFO, `slots` requests in service at once, work-conserving.

    `latency` is how long a call takes, in ticks. `healthy=False` means every
    call *fails* after exactly that long -- a broken backend that is, to
    anything measuring response times, indistinguishable from a fast one.
    That equivalence is the whole toy, and `dispatch` is where it lives: the
    duration does not depend on `healthy`, only the verdict does.
    """

Start here, because everything else is machinery and this is the claim. Then read dispatch, which is where the docstring is cashed out:

    def dispatch(self, now, rng):
        """Accept one request at tick `now`. Returns (latency, ok).

        The request waits for the earliest-free slot, so a saturated backend
        makes new arrivals queue and its observed latency climbs.
        """
        i = min(range(self.slots), key=lambda k: self.free_at[k])
        start = max(now, self.free_at[i])
        done = start + self.draw(rng)
        self.free_at[i] = done
        self.outstanding.append(done)
        self.sent += 1
        if self.healthy:
            self.ok += 1
        else:
            self.err += 1
        return done - now, self.healthy

Read the branch on self.healthy carefully, and note where it is not. done — the completion tick, and therefore the latency — is computed on line 71 from self.draw(rng), which never looks at healthy. The branch on healthy appears only afterwards, and only to increment a counter and pick the second element of the return tuple.

That is the modelling decision the whole page rests on, so it is worth defending. It says: a broken backend takes exactly as long to fail as a healthy backend of the same configured latency takes to succeed. In the demo the broken backend is configured at 1 tick against the healthy 10, which is the realistic direction — a 503 from a connection-refused or a tripped-breaker code path is faster than real work, not slower. The toy lets you set the failure latency to anything, and §6.5 sweeps it, which is where the boundary comes from.

test_failing_takes_exactly_as_long_as_succeeding (test_load_balancer.py:29-34) pins it: Backend("g", 7, True).dispatch(0, None) returns (7, True) and Backend("b", 7, False).dispatch(0, None) returns (7, False) — same 7.

5.2 RoundRobin — the control, and why it is immune

class RoundRobin:
    """The dumb one. Reads no signal at all, so no signal can mislead it."""

    name = "round-robin"

    def __init__(self):
        self.i = -1

    def pick(self, fleet, now, rng):
        self.i = (self.i + 1) % len(fleet)
        return fleet[self.i]

Eleven lines, and it wins §6. pick takes fleet, now and rng and uses len(fleet) and nothing else — no backend attribute is read anywhere. It cannot route around a slow backend, and it cannot be lured by a fast one. This is the sense in which the toy's result is not "round-robin is good": it is that round-robin's immunity and its stupidity are the same property, and you cannot buy one without selling the other.

5.3 ScorePolicy.pick — one method, three policies, and the d knob

    def pick(self, fleet, now, rng):
        cands = fleet if self.d is None else rng.sample(fleet, min(self.d, len(fleet)))
        self.last = cands               # so the demo can count won-vs-sampled
        return min(cands, key=self.score)

Everything except round-robin is "compute a score, take the minimum," so the toy writes that once and lets subclasses supply score. The d parameter turns any of them into its power-of-d-choices variant over the same score, which is what makes §6.3 an apples-to-apples measurement rather than a comparison of two different algorithms.

Two details that are load-bearing in ways worth flagging now:

min is a first-wins tie-break. When several backends tie on score, min returns the earliest in the list. For LeastConn at light load, ties are the common case — most backends are idle — so its behaviour is decided by list order, and §7.2 measures how much: the same fleet gives least-conn a 33.20% or an 85.38% share depending only on where the broken backend sits. PeakEwma almost never ties, and is position-independent (99.92% either way). This is why least-conn is a supporting character on this page.

self.last is instrumentation, not mechanism. It records the candidate set so demo.py can count how often the broken backend won a sample it appeared in (§6.3). Deleting it would not change a single routing decision.

5.4 PeakEwma — where the toy's thesis lives

class PeakEwma(ScorePolicy):
    """Finagle's peak-EWMA: an exponentially weighted mean of observed
    latency, scaled by the queue the next request would join. This is the
    policy the toy is about, and `observe` is why -- it learns from the
    duration of a call and never from its outcome.
    """

    base = "peak-ewma"

    def __init__(self, d=None, alpha=0.2):
        ScorePolicy.__init__(self, d)
        self.alpha = alpha

    def score(self, b):
        return b.ewma * (b.inflight() + 1)

    def observe(self, b, latency, ok):
        b.ewma = (1 - self.alpha) * b.ewma + self.alpha * latency

This is not invented. Finagle's own documentation describes it as a "moving average over an endpoint's round-trip time (RTT) that is highly sensitive to peaks," which is "then weighted by the number of outstanding requests" — line for line, score and observe.

observe takes ok and ignores it. That signature is the toy's whole argument in one line. The parameter is there because the simulator has the outcome and hands it over; the policy simply has no use for it. Every real latency-aware balancer has the same shape, because RTT is what a client can measure without agreeing on anything with the server.

test_the_balancer_never_reads_the_outcome (test_load_balancer.py:37-45) proves it directly rather than by inspection: feed two backends the identical latency with opposite verdicts, and their ewma and their score come out equal.

The (b.inflight() + 1) term is the only thing standing between this policy and unconditional winner-take-all, and it is worth knowing exactly how much it buys. With ewma at 1 for the broken backend and 10 for the healthy ones, the broken backend keeps winning until its in-flight count reaches 9 — at which point 1 × 10 = 10 finally ties a healthy backend's 10 × 1. At the demo's arrival rate the broken backend's in-flight count is essentially never above 1, so the term never fires. §7.5 measures what happens when you let backends serve more than one request at a time, and it is worse, not better.

5.5 PeakEwmaOverSuccess — the fix, as a diff

class PeakEwmaOverSuccess(PeakEwma):
    """The fix (§7.3): divide the score by the backend's observed success
    rate, Laplace-smoothed so a backend with no history is not divided by
    zero. One term, and it is the only term in the toy that reads `ok`.
    """

    base = "ewma/success"

    def score(self, b):
        rate = (b.ok + 1) / (b.ok + b.err + 2)
        return b.ewma * (b.inflight() + 1) / rate

The subclass overrides score and nothing else, so the diff against the broken policy is exactly one division. A backend failing everything has a smoothed success rate near 0, so its score goes to infinity and it stops being chosen. §7.3 measures it: goodput 4806 → 5 → 5871.

The Laplace smoothing (+1 over +2) is not decoration. Without it a brand-new backend has 0/0, and with a plain ok/(ok+err) a backend whose first request happened to fail would be divided by zero and permanently exiled on a sample of one.

5.6 simulate — the loop, and the order of the three lines that matter

    for now in range(ticks):
        for b in fleet:
            b.retire(now)
        for _ in range(poisson(rng, lam)):
            b = policy.pick(fleet, now, rng)
            latency, ok = b.dispatch(now, rng)
            policy.observe(b, latency, ok)
            samples.append((latency, ok))

retire runs first, so in-flight counts reflect completions up to now before any routing decision is taken. Then arrivals: pick, dispatch, observe.

The thing to notice is that observe is called with the latency of a request that has just been scheduled, not one that has finished. The balancer learns the request's eventual latency immediately. This is a deliberate simplification and it is generous to the balancer — a real client only learns an RTT when the response arrives, so its EWMA lags. Giving the policy instant, perfect latency feedback removes "the signal was stale" as an explanation for anything on this page. The black hole in §6.1 happens to a balancer with better information than any real one has.

6. The demo, and what it proves

demo.py builds four backends — three answering in 10 ticks, one in 1 tick — and routes one Poisson arrival trace (λ=0.32, seed 1, 20000 ticks, 6407 requests) through them with four policies.

The offered load is worth a sentence. Three backends at 10 ticks each serve 0.30 requests/tick between them; the arrival rate is 0.32. So the healthy part of the fleet is just saturated and needs the fourth backend's help — this is not a fleet with acres of spare capacity where routing could not matter.

6.1 Act 1 — the twin worlds

Read the fast3= column down both blocks. Every policy routes identically in both worlds — 24.99%, 33.20%, 50.40%, 99.92%, twice. The p99 columns are identical too. The only column that moves is goodput.

Then read the side-by-side block, which makes it exact rather than approximate: the same [4, 1, 0, 6402] split, and the same EWMA vector to nine decimal places, 1.303659381. Not "similar behaviour" — the balancer's entire observable universe is bit-identical in a world where it is doing the best possible job and a world where it has caused a total outage. test_broken_backend_is_invisible_to_the_balancer (test_load_balancer.py:50-66) asserts all of it, and test_all_four_policies_route_identically_in_both_worlds extends it to the other policies.

Now derive the two headline numbers.

Goodput 5. Not a statistic — the literal sum of the three healthy backends' request counts, 4 + 1 + 0 = 5. Every request that succeeded is a request that peak-EWMA sent somewhere other than fast3, and it did that five times in 20000 ticks. Note ok2 received zero requests in the entire run: a healthy, working machine that the balancer never used once.

Round-robin's 4806. Round-robin gives each backend a 1-in-4 share, so three quarters of the traffic lands on a healthy backend: 6407 × 3/4 = 4805.25, and the integer split gives 4806.

4806 / 5 = 961.2. Choosing the sophisticated policy over the dumb one cost three orders of magnitude of goodput.

And it did so while improving p99 from 41 ticks to 3 — a 13.67× win on the metric, delivered by the same decisions that caused the outage.

6.2 Act 2 — every latency metric ranks the policies backwards

Three orderings of the same four runs. The first two are the exact reverse of the third: best p99 is worst goodput, worst p99 is best goodput. test_p99_ranking_is_the_reverse_of_the_goodput_ranking asserts the two endpoints of that reversal.

The reason the first ranking is upside-down is not subtle once you see it: a fast error is a fast response, and p99 is computed over responses. Sending 99.92% of your traffic to something that answers in 1 tick is a magnificent way to improve a latency percentile.

The second ranking is the one worth sitting with, because it is the obvious objection and it fails. "Fine — exclude the errors and measure only the requests that worked." Peak-EWMA still comes first, at 10 ticks against round-robin's 43. The five surviving requests went to idle healthy backends and were served immediately, with no queueing, precisely because all the other traffic had been diverted into the black hole. The policy's successes get faster as it destroys more of them.

So there is no filter on a latency histogram that reveals this incident. The only metric that moves is the count of useful responses, and that is a different measurement, not a different cut of the same one. test_throwing_the_errors_away_does_not_help pins the four values [43, 19, 23, 10].

6.3 Act 3 — power-of-d-choices is the blast radius, and it is exactly d/N

The standard fix when a "least X" policy misbehaves is power-of-d-choices: sample d backends at random, take the best of those. Mitzenmacher, Richa and Sitaraman's survey gives the famous result — with d = 2 "the maximum load is log log n / log d + Θ(1)", so two choices is a huge improvement over one, "while each additional choice beyond two decreases the maximum load by just a constant factor."

Here is the same knob on the same fleet.

There are no diminishing returns and no sweet spot at 2. The broken backend's share is a straight line in d, and the derivation is one step.

Look at won|sampled: 1692/1692, 3255/3255, 4871/4871, 1577/1577. Of every sample that contained the broken backend, the broken backend was chosen — all of them. It is not likely to win a comparison, it is guaranteed to, because its score is 1 × (0 + 1) = 1 against a healthy backend's 10 × (q + 1) ≥ 10. So the only question is whether it appears in the sample at all, and for a uniform sample of d from N that probability is d/N:

d=3, N=4 predicts 75.00% and measures 75.98%. d=8, N=16 predicts 50.00% and measures 50.24%. d=12, N=16 predicts 75.00% and measures 75.59%. test_power_of_d_choices_is_the_blast_radius asserts the N=4 row to two decimals, and test_broken_backend_wins_every_sample_it_appears_in asserts the 4871/4871.

The two rows that end in 6313/6315 and 6441/6449 are the only ones where the broken backend ever lost a sample — 2 times in 6315, 8 in 6449. Those are the ticks where its in-flight count did briefly reach 9 and the (inflight+1) term fired, exactly as §5.4 predicts.

The classic result is not wrong; it is answering a different question. It concerns the maximum queue length when every server is trying to serve you, and there d=2 is close to optimal. This toy adds one server that is not trying, and the same knob becomes a dial on how much of your traffic finds it. Sampling more widely is only a virtue when the thing you are sampling for is real.

6.4 The fix, and what it costs

One division (§5.5) takes goodput from 5 to 5871 — and past round-robin's 4806, which is the satisfying part. ewma/success is the only policy here that beats the dumb one, because it keeps the latency steering that makes peak-EWMA good in the healthy world and stops the steering from being inverted by failure.

Note its p99 is 70, the worst on the page. Having stopped throwing 99.92% of requests into a 1-tick hole, it has to serve them on backends that take 10 ticks and now have queues. That is not a regression; it is the price of the work actually getting done, and it is a compact demonstration that on this fleet p99 and goodput are in direct tension — you can buy any p99 you like by discarding enough traffic.

The broken backend still gets 8.37% rather than 0%, because the smoothed success rate needs evidence and the score is a soft penalty, not an ejection. Real implementations add the missing piece: Envoy's outlier detection ejects a host outright on consecutive 5xx or on statistical success-rate outliers, and gRPC's weighted-round-robin computes weight = qps / (utilization + eps/qps * error_utilization_penalty) — an error term in the denominator, which is the same move as this toy's division.

6.5 The boundary condition — where the effect vanishes, and where it reverses

Everything above depends on failing being faster than succeeding. Sweep that, and the effect does not merely fade — it crosses zero and changes sign.

Read the e = s column: 23.9%, 25.3%, 25.0%, 25.0%. The crossover is exactly where the error latency equals the service latency, for every service time from 5 to 40, and at that point the share is 1/N — peak-EWMA has become round-robin, because a failure that takes as long as a success is invisible to a signal made of durations. The goodput agrees: 4785 against round-robin's 4806, a 0.4% difference. test_boundary_error_latency_equal_to_service_latency pins it.

Past the crossover the sign flips. At e=50 against s=10 — a backend that fails by hanging, which is the other classic failure mode — the broken backend's share falls to 5.95% and peak-EWMA delivers 6026 against round-robin's 4806, a 25.4% win. The policy that caused the outage in §6.1 is, on the same fleet with the failure mode reversed, the best thing you own: it detects and routes around a hanging backend that round-robin keeps feeding forever. test_the_sign_flips_past_the_boundary pins 6026, 4806 and 5.95%.

So the honest statement, and the one that lets you place your own system:

Latency-aware load balancing is the best failure detector you own for failures that are slow, and the worst thing you own for failures that are fast. The dividing line is one question: does failing take longer than succeeding?

And the uncomfortable part is which side of that line modern infrastructure sits on. Fail fast is advice everyone follows. Connection refused is instant. A tripped circuit breaker (see ../circuit-breaker/) rejects without attempting the call, which is the point of it. A pod with no ready container RSTs immediately. Every one of those is a system that has been carefully engineered to fail on the left-hand side of the boundary — which is the side where the balancer aims traffic at the failure.

6.6 The trace is pinned by tests

The demo output is byte-identical across runs and across PYTHONHASHSEED values 0, 1, 42 and 12345, because there is no clock anywhere and the single RNG is explicitly seeded — see §7.6.

7. Design decisions and roads not taken

7.1 Static weights are immune, because nothing can move them

The backlog entry that produced this toy asked for weighted round-robin as a third policy. It is not one, because on this fleet it is not a different policy. Smooth weighted round-robin with equal weights emits exactly the round-robin sequence:

Identical request-for-request. Give the broken backend a lower weight and goodput rises to 5766 — but only because a human happened to guess right in advance. That is the point worth keeping: a static weight is immune to this failure for the same reason it is useless against every other one. It is not reading anything.

7.2 Least-connections is decided by list order, which is why it is not the headline

I expected least-connections to be the villain — a backend that fails instantly has no queue, so it should look permanently idle. It does, but the size of the effect turns out to be an artifact of tie-breaking rather than a property of the policy, so it could not carry the page:

Same policy, same fleet, same trace: 33.20% or 85.38% depending only on where the broken backend sits in a list. At this load most backends are idle most of the time, so least-connections spends its life in ties, and min's first-wins rule (§5.3) decides the outcome. Peak-EWMA is unmoved by the same permutation, because it almost never ties — its score is a float built from a latency history, not a small integer.

This is worth knowing for its own sake: if you run least_conn in nginx with an idle-ish fleet, a meaningful part of your traffic distribution is decided by the order of lines in a config file.

7.3 Power-of-d over in-flight is a much weaker beacon than over latency

Envoy's LEAST_REQUEST is P2C over active request count — "selects N random available hosts... and picks the host which has the fewest active requests." Running the identical sweep over in-flight instead of latency:

It saturates around 47% instead of climbing to 100%, and it does not follow d/N. The reason is the difference between a signal that self-corrects and one that does not: sending a request to the broken backend raises its in-flight count, so in-flight-based policies push back on themselves. Latency does not — serving one more instant error does not make the next error slower, so the beacon never dims.

That is a real ranking of the two signals, and it is the opposite of the usual one. Latency is the more sophisticated signal, reacts sooner to genuine degradation, and is exactly the one with no negative feedback when the "degradation" is failure.

7.4 Knobs that do not save you

The EWMA smoothing factor. The obvious tuning response to "the balancer over-reacted to a latency signal" is to smooth it harder. Every value fails:

Goodput ranges from 0 to 22 out of 6407 across two orders of magnitude of alpha. Slower smoothing is worse, not better, because the broken backend's true latency really is 1 tick — there is no noise here for smoothing to remove. The signal is not noisy, it is valid and irrelevant.

A bigger fleet. The intuition that more capacity dilutes a bad backend is correct for round-robin and false for peak-EWMA:

Round-robin's blast radius is 1/N and shrinks all the way to 2.00% at N=50. Peak-EWMA's is pinned at 99.92% and its goodput is 5 at every fleet size from 3 to 50. Scaling out is not a mitigation; the gap widens from 801× at N=2 to 1256× at N=50, because you added healthy capacity the balancer will not use. test_a_bigger_fleet_does_not_dilute_a_black_hole pins both columns.

Service-time jitter. Real backends do not answer in exactly 10 ticks, and you might hope variance blurs the signal. At a coefficient of variation of 2.0 — extremely heavy-tailed — the share is still 94.44% and goodput 361. test_service_time_variance_does_not_save_it pins it.

7.5 Per-backend concurrency makes it strictly worse

The most serious objection to the model is that these backends serve one request at a time, so the (inflight+1) term has an unrealistically easy job of saturating. Real servers handle hundreds of concurrent requests. Setting slots higher:

Goodput is exactly zero — not one request in 6407 reached a healthy backend. With more than one slot the broken backend's in-flight count never climbs enough for (inflight+1) to overcome a 10× latency advantage, so the one brake in the score never touches the pedal. The single-slot model in the demo is the conservative choice, and it is the only configuration in which the black hole leaks at all. test_concurrency_makes_it_strictly_worse pins the zeros.

7.6 Roads not taken inside the toy

Health checks and retries. Both are absent on purpose. Retries would make the goodput collapse look like a retry storm, which is a different mechanism that ../circuit-breaker/ already teaches; active health checks are a remediation, and putting one in would answer the question the page is asking. §8 says what they buy.

Ejecting the backend rather than down-weighting it. PeakEwmaOverSuccess applies a soft penalty, which is why the broken backend still gets 8.37% rather than 0%. Real outlier detection ejects. Soft penalty was the right choice for the toy because it is a one-line diff against the broken policy (§5.5) — the reader can see that the entire difference is one division — whereas ejection needs a state machine, timers and an un-ejection rule, which is another toy.

Giving the balancer delayed latency feedback. observe is called at dispatch, so the policy learns each RTT instantly (§5.6). Modelling the lag would be more realistic and would only strengthen the result; the toy gives the balancer perfect information so that nobody can attribute the outcome to stale data.

No clock, no unseeded RNG. simulate takes seed and threads one random.Random through arrivals, service draws and the P2C sample; time is an integer tick counter. The payoff here is specific: the aha in §6.1 is an exact identity between two runs, and an identity is not observable under jitter. It is what lets test_broken_backend_is_invisible_to_the_balancer assert [4, 1, 0, 6402] and 1.3036593805578662 rather than "approximately the same." Output is byte-identical across PYTHONHASHSEED 0, 1, 42, 12345.

The alpha and cv knobs live in the shipped code, not in a fork, because §7.4 needs to measure that they are inert rather than assert it.

8. What's simplified vs. the real thing

• No health checking. This is the big one. Every real balancer polls /healthz and takes failing backends out of rotation, and a health check would catch this toy's broken backend immediately. The catch is that the black hole is a load-balancing failure, not a detection failure: it occurs in the window before a check fires, it occurs for backends that pass a shallow check while failing real requests (the classic "/healthz returns 200 because it doesn't touch the database"), and it occurs for partial failures where only some requests error. What §6.5 tells you is that during any such window, the balancer is not neutral — it is actively aiming traffic at the fault, and 99.92% of it.
• Binary health. A backend here is perfectly fine or fails everything. Real degradation is fractional and drifting, which is why Envoy's outlier detection is statistical (eject hosts whose success rate is an outlier against the cluster) rather than a threshold.
• One balancer, one process. Real deployments run the policy in every client or proxy independently, each with its own EWMA table built from its own traffic. That is worse than the toy in one specific way: all of them converge on the same wrong conclusion simultaneously, since they are all measuring the same fast errors.
• No cost model for an error. Goodput here counts successes. In production a 1-tick error and a 10-tick success are not equal-and-opposite: the error propagates to a user, may be retried (amplifying load), and may poison a cache. The toy's accounting is generous to the broken backend.
• Arrivals are Poisson and open-loop. Requests arrive on a schedule that does not depend on how the system is doing. Real load is closed-loop — users whose requests fail retry or leave — which changes the arrival rate during the incident in both directions.
• In-flight count is exact and free. inflight() is a len() on a deque. A real client's view of in-flight is exact for its own requests and totally blind to everyone else's, which is the actual reason power-of-d-choices exists (Marc Brooker's post in §10 is about balancing on stale load data, which the toy does not model at all).
• Ticks are not milliseconds, and 6407 requests over 20000 ticks is a small fleet under modest load. The ratios are the result; the absolute numbers are not.

9. Check yourself

Answer before expanding. Each answer is derivable from the source.

    def observe(self, b, latency, ok):
        b.ewma = (1 - self.alpha) * b.ewma + self.alpha * latency

10. Further reading

Every link below was fetched and confirmed live when this was written.

• Google SRE Book: Load Balancing in the Datacenter — read this one first. It names the failure this toy reproduces and calls it sinkholing: "if a task is seriously unhealthy, it might start serving 100% errors... it's frequently significantly faster to just return an 'I'm unhealthy!' error than to actually process a request." Their fix is the production-grade version of §6.4 — count recent errors as if they were active requests, so failing costs the backend the same in-flight credit that succeeding would.
• Finagle: Clients § load balancing — the source of PeakEwma (§5.4), described there as "a moving average over an endpoint's round-trip time (RTT) that is highly sensitive to peaks," which is "then weighted by the number of outstanding requests." Note their own stated caveat — that it "assumes loaded endpoints take time recovering" — which is the assumption §6.5 breaks from the other direction.
• Envoy: Supported load balancers — LEAST_REQUEST is P2C by default: "selects N random available hosts... and picks the host which has the fewest active requests." That is §7.3's policy, and the reason it saturates at 47% instead of 100%.
• Envoy: Outlier detection — the fix §6.4 approximates, done properly: ejection on consecutive 5xx, on consecutive gateway failures, and on statistical success-rate outliers. This is the component that makes latency-based balancing safe to run, and the reason to read it is that it is a separate subsystem from the load balancer — the balancer itself still cannot see the outcome.
• gRPC A58: client-side weighted round robin — backends report their own load via ORCA, and the weight is qps / (utilization + eps/qps * error_utilization_penalty). An errors-per- second term in the denominator: the same shape as PeakEwmaOverSuccess (§5.5), reached independently and with a knob for how hard to penalise.
• Mitzenmacher, Richa & Sitaraman: The Power of Two Random Choices: A Survey of Techniques and Results — the theory §6.3 inverts. Their result: d=2 "yields a large reduction in the maximum load over having one choice, while each additional choice beyond two decreases the maximum load by just a constant factor." Worth reading precisely to see what it assumes — that every server is trying to serve you — and therefore why share = d/N is not a contradiction of it.
• Marc Brooker: Load Balancing: The Power of Two Random Choices — the practitioner's version, and the argument for P2C that this page is in tension with. His concern is stale load data and the herd behaviour it causes; the toy's balancer has perfectly fresh data and fails anyway, which is a usefully different failure to have in your head.
• Wikipedia: Goodput — the vocabulary for §6.2. Once you have the word, the question "is this dashboard showing me throughput or goodput?" becomes askable, and on most dashboards the answer is the first one.