A forward proxy that shuffles bytes between many clients and one upstream connection — and the discovery that it multiplexes flawlessly right up to the instant a second request is in flight, and then hands one client's data to another with no bug, no corruption, and every byte delivered in order exactly once.
Python 3.15.0a8 (main, Apr 14 2026, 14:20:41) [Clang 22.1.3], over TCP on loopback.
cd tcp-proxy
python3 demo.py # the aha (§6) -- byte-identical every run
python3 test_tcp_proxy.py # 16 tests
python3 demo.py --race # the same questions with no scripting (§6.8)This toy exists because of a claim that turned out to be false. The entry it was built from read:
a forward proxy that shuffles bytes between client and upstream, with basic connection pooling. Aha: watch it multiplex many client connections onto few upstream connections.
Build that literally — one shared upstream socket, recv on one side, sendall on the other, no idea what a protocol is — and it does not multiplex. It misdelivers. Not because the relay is buggy: §6.2 shows two well-formed requests going in, two well-formed replies coming back, every byte forwarded in order exactly once, and one client receiving the other client's data.
By the end you should be able to:
A TCP connection is expensive in a way that is easy to forget on loopback: a three-way handshake before the first byte, a TLS handshake after that in anything real, a file descriptor and two kernel buffers for as long as it lives, and an entry in a conntrack table on every middlebox in between. A service fronting ten thousand clients does not want ten thousand connections to its database. So it keeps a small pool and lends connections out.
That is a resource-management story, and it is the wrong one. Here is the question a pool actually has to answer:
A connection was lent to a borrower. When is it free again?
Return it too early and the next borrower reads the tail of somebody else's response. Return it too late and the pool is smaller than you think it is. There is no third option and no way to defer the question — every checkin is an assertion that the previous exchange is over.
Now notice what kind of assertion that is. "This exchange is over" is identical to "this message ended here", which is the framing question, and TCP does not answer it. TCP promises a reliable ordered byte stream: you get every byte, in order, and no statement whatsoever about which bytes belong together. So a pool that does not parse the protocol has exactly one way to guess that a connection is free — wait until the upstream stops talking for a while — and that is a guess about the upstream's scheduling, not about the message.
The competing goals that make more than one design defensible:
nc, socat, stunnel and every load balancer's TCP mode exist.You cannot have both, and this toy is the demonstration. The dumb relay in §6.2 is not a worse multiplexer than the parsing pool in §6.5 — it is not a multiplexer at all, and the mechanism the backlog entry asked us to admire turns out (§6.8) to be serialisation wearing a multiplexing costume.
| Concept | Where it's used in the toy | One link |
|---|---|---|
| TCP is a byte stream with no message boundaries | Upstream._handle's while b"\n" in buf (lines 111–113) — one recv can carry half a request or three of them | Python socket HOWTO |
| Message framing by delimiter | FramedProxy._read_reply, lines 331–337. This is the line that carries the result. | RFC 9112 §9.3.2, Pipelining |
| Connection pooling: checkout, checkin, cap | FramedProxy._checkout (lines 273–285) and the finally: on line 303 | Envoy: connection pooling |
| Request/response correlation | absent from NaiveProxy by construction — that absence is §6.2. Also load-bearing, for what it makes impossible. | RFC 9113 §5, Streams and Multiplexing |
recv returns what is buffered, not what was sent | every recv(4096) in the file; the split reply in §6.6 arrives as two | Python socket HOWTO |
| Head-of-line blocking | FramedProxy._checkout's blocking self.free.get(), line 285 | HAProxy: keep-alive, pipelining, multiplexing and connection pooling |
Half-close (shutdown(SHUT_WR)) vs. close | PassthroughProxy._relay, lines 378–387 — appendix material | RFC 9293 §3.6.1, Half-Closed Connections |
| Condition variables as a determinism tool | every wait_* method; §6.7 | threading.Condition |
The two flagged rows are the whole page. The second row is where the result lives; the fourth is why there is nothing you can do about it anywhere else.
The sentence to carry away: a pool's checkin is a parse. Everything else about a pool — the cap, the queue, the eviction policy — is bookkeeping you could write in an afternoon. The only hard part is knowing that the last exchange is over, and nothing below layer 7 knows that.
def _handle(self, conn, cid):
buf = b""
while True:
try:
data = conn.recv(4096)
except OSError:
break
if not data:
break
with self.cv:
self.bytes_in += len(data)
self.cv.notify_all()
buf += data
# The upstream frames on "\n" like everyone else has to. One
# recv can carry half a request, or three of them.
while b"\n" in buf:
line, buf = buf.split(b"\n", 1)
self._respond(conn, cid, line)
conn.close()Start here because it is the honest half of the toy, and because the inner while is the thing the proxies are about to get wrong. The upstream keeps its own buffer and splits on "\n" — it never assumes a recv gave it exactly one request, and it never assumes a recv gave it at most one either. The accounting (self.bytes_in) is bumped inside the lock before the parse, which is what lets the demo say "wait until 16 bytes have arrived" without a sleep; see §6.7.
The important consequence is in the loop's blindness. This upstream will happily parse b"GET alpGET beta" as a request, because that is what arrived between two newlines. It is not being naive: there is no other rule available. A newline-delimited protocol says a newline ends a request, and two clients' bytes concatenated on one connection are indistinguishable from one client's bytes. That is §6.1.
def handle(self, conn):
self.clients.append(conn)
while True:
try:
data = conn.recv(4096)
except OSError:
break
if not data:
break
self.last = conn # <-- the routing decision, such as it is
self.up.sendall(data)
with self.cv:
self.forwarded += len(data)
self.cv.notify_all()Fourteen lines, and there is nothing wrong with any of them. recv, sendall, count the bytes. self.up is a single socket created once in __init__ and shared by every client thread — which is the backlog entry's "multiplex many client connections onto few upstream connections", implemented as literally as it can be implemented.
self.last = conn is the only line here that is a decision rather than plumbing, and it is the line a reader's instinct blames when the output turns out wrong. §6.3 measures it. It is inert.
def _pump_upstream(self):
while True:
try:
data = self.up.recv(4096)
except OSError:
return
if not data:
return
if self.routing == "last":
targets = [self.last] if self.last else []
else:
targets = list(self.clients)
for target in targets:
try:
target.sendall(data)
except OSError:
pass
with self.cv:
self.routed += 1
self.cv.notify_all()This is the whole downstream path, and the thing to notice is what it does not have access to. data is bytes. There is no request id, no correlation token, no sequence number, and — because the upstream connection is shared — not even a socket identity to fall back on. The two branches are the only two policies you can write with the information available: send it to whoever spoke last, or send it to everybody.
A third option would need a fact that is not on the wire. That is why §6.3 sweeps both and both lose: the failure is not in this function, it is in the protocol's silence about who asked.
def _checkout(self):
try:
return self.free.get_nowait()
except queue.Empty:
pass
with self.made_lock:
if self.made < self.pool_cap:
self.made += 1
return (socket.create_connection(self.upstream_addr), b"")
# At the cap and nothing free: this is head-of-line blocking, and it
# is the price the pool charges for not opening a connection here.
self.blocked_gets += 1
return self.free.get()Three tiers, in the order every real pool uses them: reuse an idle connection, else open a new one if you are under the cap, else wait. The get_nowait() first is what makes §6.5's cap=16 row read 8 instead of 16 — a borrower that finds a free connection never reaches the "open a new one" branch, so the pool only ever grows to the number of borrowers that were simultaneously unsatisfied. That number is the concurrency level, not the client count and not the cap.
Note that a connection is carried as (socket, leftover_bytes). The leftovers are the reason the pool works at all across a recv that over-read; §5.6.
def handle(self, conn):
buf = b""
while True:
while b"\n" not in buf:
try:
data = conn.recv(4096)
except OSError:
return
if not data:
return
buf += data
line, buf = buf.split(b"\n", 1)
sock, rest = self._checkout()
try:
sock.sendall(line + b"\n")
reply, rest = self._read_reply(sock, rest)
finally:
self.free.put((sock, rest))
try:
conn.sendall(reply)
except OSError:
returnCompare this against §5.2 and the whole toy is visible in the diff. The naive relay's loop body is "read some bytes, write some bytes". This one's is "read one request, borrow, write one request, read one reply, return, write one reply". The connection is held for exactly the span of one exchange, and that span is delimited at both ends by a parse.
Two details worth the ink:
buf survives the loop iteration (line, buf = buf.split(b"\n", 1)), so a client that pipelines two requests into one write is handled without a second recv. Drop that and you have written a proxy that works in testing and deadlocks against a client that batches.
The finally: puts the connection back even if _read_reply raised. That looks like ordinary hygiene and is actually a small lie the toy tells for brevity — see §8, because returning a connection whose state you do not know is how real pools leak corruption.
def _read_reply(self, sock, rest):
"""Read exactly one reply off a pooled connection.
The `timeout` branch is pooling without parsing: it returns when the
upstream goes quiet, a guess about the upstream's scheduling. The
`framed` branch returns at the delimiter, a fact about the message,
and keeps any bytes past it with the connection.
"""
if self.release == "timeout":
sock.settimeout(self.release_timeout)
out = rest
try:
while True:
data = sock.recv(4096)
if not data:
break
out += data
except (socket.timeout, OSError):
pass
sock.settimeout(None)
return out, b""
while b"\n" not in rest:
data = sock.recv(4096)
if not data:
return rest, b""
rest += data
line, rest = rest.split(b"\n", 1)
return line + b"\n", restBoth branches answer the same question — "has this exchange finished?" — and they are not two implementations of one idea. They are two different questions that happen to agree most of the time.
The timeout branch asks: has the upstream been silent for 50 milliseconds? That is a statement about the upstream's scheduler, its disk, its GC pause and the network between you. The framed branch asks: have I seen the delimiter? That is a statement about the message, and it is true or false for reasons that have nothing to do with timing.
§6.6 runs both against an upstream that pauses 120ms in the middle of a 19-byte reply. The delimiter does not notice. The timeout gets every single answer wrong.
The last three lines are the other half of the trick, and they are easy to skim past. rest — everything that arrived after the delimiter — is returned to the caller and stored with the socket in the free queue. Without that, a recv that happened to pull in the front of the next reply would silently discard it, and the pool would corrupt itself at exactly the moments it is busiest.
# --- sequencing hooks: wait on a real event, never on a sleep ----------
def wait_bytes(self, n):
"""Block until the upstream has received at least n bytes."""
with self.cv:
self.cv.wait_for(lambda: self.bytes_in >= n, timeout=5)
def wait_requests(self, n):
"""Block until the upstream has parsed at least n whole requests."""
with self.cv:
self.cv.wait_for(lambda: len(self.requests) >= n, timeout=5)A toy about concurrency whose transcript changes between runs cannot make an argument, because the reader cannot tell a result from a coincidence. These hooks — wait_bytes and wait_requests here, wait_forwarded and wait_routed on NaiveProxy, wait_eofs on PassthroughProxy — are how demo.py gets a byte-identical transcript with zero sleep() calls anywhere: every step waits on a counter of an event that actually happened.
They are also the toy's largest honest cost. They exist for the demo, not for the mechanism, and they are part of why this file is over the line ceiling. See §6.7 for the one that turned out to matter and the appendix for the budget.
Everything in §6.1–§6.6 is a verbatim slice of one run of python3 demo.py. That run is byte-identical across repeats (§6.7).
The arithmetic, because the relay is not losing anything: A wrote 7 bytes then 3, B wrote 9, so 7 + 3 + 9 = 19 bytes reached the upstream. What the upstream parsed was GET alpGET beta (15) + \n + ha (2) + \n = 19 bytes. Nothing was dropped, duplicated or reordered. (test_naive_relay_ forwards_every_byte_exactly_once asserts exactly this: up.bytes_in == px.forwarded == 19.)
Two clients sent two syntactically perfect requests. The upstream received two syntactically perfect requests. They were not the same two.
The interleave here is scripted — A stops mid-request, B writes, A finishes — because a race is not reproducible and a page you cannot re-run is a page you cannot check. The scripting is honest about what it is: this ordering is entirely legal under TCP and the relay has no defence against it. §6.8 runs the unscripted version and it happens on its own.
This is the section that should hurt. No partial writes, no clever timing — two complete requests, sent one after the other:
Read the two lines at the bottom carefully.
Client B asked for beta and was handed alpha's value. Not a truncated value, not a corrupted one — the correct, complete answer to a question B never asked, from a client B has never heard of. alpha=AAAAAAAAAAAA\n is 19 bytes and beta=BBBBBBBBBBBB\n is 18; B received all 37 of them, in order.
Client A, who asked for alpha, received nothing at all and will sit in recv until it times out.
There is no bug to fix. Walk the relay: A's 10 bytes went upstream (in order, once); the upstream parsed them and produced 19 bytes of reply; the relay received those 19 bytes on the shared socket and had to decide where to send them. By then B had written, so self.last was B. Every individual step is correct. The composition is a cross-client data disclosure produced by a correct byte relay — which, if the values were session tokens instead of AAAAAAAAAAAA, is the entire class of bug.
The precise reason is worth stating plainly: the reply alpha=AAAAAAAAAAAA\n carries no information about whose request it answers. Not "insufficient information" — none. Request/response correlation is a protocol feature (HTTP/2 calls it a stream id, RFC 9113 §5), and a raw TCP reply does not have one.
The instinct on reading §6.2 is that self.last = conn is the bug and a smarter routing rule would fix it. It is not, and it would not. Here is the other defensible policy, same script, same everything:
broadcast fixes A's starvation and makes the disclosure total: now both clients hold both answers. Under load it is no better — the swept numbers are in §6.8, and across 20 trials of 8 clients × 3 requests, last scored 1–4 right out of 24 and broadcast 0–6. The gap between them is noise.
There is no third policy, and that is the point of the paragraph rather than a throwaway. Any correct rule would have to map a reply to a request, and §5.3 shows the function has nothing to map with. This is a line that looks like a decision and carries none of the behaviour. Knowing which of your lines are inert is worth as much as knowing which are load-bearing; it is the difference between fixing a bug and rewriting a protocol.
24 of 24 correct, and a multiplexing ratio of exactly 1.0:1 — which is to say, none. This is PassthroughProxy, the relay socat writes: one upstream connection per client connection, bytes copied both ways, no protocol knowledge required and no protocol knowledge possible.
It is not a failure. It is the honest price of layer-4 generality, and it is what every TCP-mode load balancer does. What it cannot do is amortise anything: ten thousand clients means ten thousand upstream connections.
8 clients × 3 requests = 24 requests, and the wrong column is zero on every row. The same 24 requests that the byte relay gets almost entirely wrong are answered correctly through a pool of one connection, because the pool knows where each reply ends.
The requests/conn column is just 24 divided by the connection count: 24/1 = 24.0, 24/2 = 12.0, 24/4 = 6.0, 24/8 = 3.0. And the last row is the interesting one — cap=16 opened 8 connections, not 16, so its ratio is 24/8 = 3.0, identical to cap=8.
That is not a safety net; it is what §5.4 makes inevitable. A pool only grows when a borrower finds nothing free, and at most 8 borrowers can be unsatisfied at once because there are only 8 clients each waiting for a reply before sending again. So:
upstream connections = min(pool_cap, requests in flight)
The multiplexing ratio is clients : min(cap, in-flight), and the cap is only a lever below the concurrency level. Above it, "tune your pool size" stops meaning anything — a second measured boundary, and a cheap one to check in production before you spend a week on pool tuning.
The claim in §5.6 is that _read_reply's two branches are two different questions. Here they are, crossed with two upstream write patterns. Identical proxy, identical pool (cap=1), identical clients.
8 of 8 → 0 of 8, and the only thing that changed between those two rows is whether the upstream emitted its reply in one sendall or paused in the middle. Nothing about the proxy, the pool, the cap or the clients moved.
The arithmetic: the reply alpha=AAAAAAAAAAAA\n is 19 bytes, and the split upstream sends 6 of them (alpha=), waits 120ms, then sends the remaining 13. The borrower's release timeout is 50ms. 50 < 120, so the borrower gives the connection back holding alpha= and no newline — a checkin asserting an exchange had ended when 13 bytes of it were still in the future. The next borrower sends its request onto that connection and reads whatever comes next, which is the tail of somebody else's answer.
Note what the framed row shows: the delimiter reader is completely unaffected by the pause. It is not more robust, it is asking a question the pause cannot influence.
And note the trap in the top-right of the table. timeout + one write scores 8 of 8. A timeout-based pool passes its tests, passes staging, and passes production until the day the upstream gets slow — which is precisely the day your traffic is highest and your logs are least readable. The bug is correlated with load.
A toy about concurrency has an obvious problem: the repo requires byte-identical transcripts, and threads do not provide them. What was done, concretely:
http-from-sockets — the only other toy here on real sockets — also uses a fixed port, but its port number leaks into every transcript through the Host: header, which is why one of its counterfactual request heads is 157 bytes and another is 156. This toy's protocol has no field that can carry a port, so no port number can reach the output at all.http-from-sockets starts a server in a subprocess and connect-polls it, and pays for that with a phantom "connection 1" in its own log. Here the servers are in-process and listen() has already returned before the constructor does, so there is no probe connection, no phantom, and no poll.http-from-sockets too (its two otherwise-identical runs differ only in 1001 vs 1002 milliseconds). Nothing on this page is timed.wait_bytes, wait_requests, wait_forwarded, wait_routed. There is not one sleep() in demo.py.Upstream(gate=…)) that holds replies until N requests have arrived. It exists only to pin the pool's connection count in §6.5, and it is scaffolding rather than mechanism, so it is named here rather than hidden. §6.8's unscripted run shows what the ungated count does.The hook that mattered. wait_routed is not padding — §6.1 was subtly raced without it, and looked stable. To see this yourself: delete the single line px.wait_routed(1) from interleave() in demo.py and run the scenario 40 times, collecting distinct transcripts. One such run:
Without the line, the same scenario produces two different transcripts: sometimes A gets both errors and B gets nothing, instead of one each. How often is itself a race — two runs of this check gave 2 in 40 and 8 in 40 — while the requests the upstream parsed stay identical either way, which is exactly why the flaw is invisible on a casual re-run. With the line, both runs of the check gave 40 identical transcripts out of 40.
That is the worst kind of wrong to ship: stable enough to believe on the first three runs, and false often enough to burn whoever checks it later.
The full demo, with every hook in place: 10 consecutive runs byte-identical, 5 079 bytes, SHA-256 ea5bd807bd7fd3b40e207364928fabf6b5f75a8190905c477aa5ac7e860d70b7.
Here is the same protocol-blind relay from §6.1 and §6.2. Same class, same routing policy, same single shared upstream connection. The only change is that no two requests are ever in flight at once:
24 of 24 correct, 8 client connections onto 1 upstream connection, a ratio of 8/1 = 8.0:1 — from a proxy that has never heard of a protocol. The backlog entry's aha is true, exactly here.
And it is worth nothing here, which is the sharpest form of the result. With one request in flight, "whoever wrote last" is not a guess: there is only one candidate, so the routing rule is trivially correct. The thing being admired is not multiplexing. It is serialisation — the same amortisation you would get from a mutex around a single connection, with the same throughput ceiling, and none of the concurrency that made you want a proxy.
Is it a slope or a cliff? Run python3 demo.py --race, which re-asks the question with no gate and no scripting at all. These numbers are a distribution and are not reproducible, which is why they are behind a flag and quarantined here rather than in the demo proper — one run:
A cliff. One client: 0 wrong out of 3, on 20 trials out of 20. Two clients: 2 to 4 wrong out of 6 — between a third and two thirds of all answers, on the very first step. There is no gentle degradation to monitor your way through and no traffic level below which you are "mostly fine": you are perfect at concurrency 1 and broken at concurrency 2.
The same run also confirms that the parsing pool does not depend on the demo's gate for its correctness — only for the reproducibility of its connection count:
60 trials, 24 requests each — 1 440 requests, zero wrong answers, with nothing scripted. The conns seen column is exactly the thing the gate exists to pin: at cap=8 the pool opened 7 or 8 connections depending on how the race fell, because "requests in flight" is itself a raced quantity.
So, to place your own system on this: if it has one request outstanding per upstream connection at a time, a byte relay is safe and you are getting serialisation, not multiplexing. The moment two requests can be outstanding on one connection, you need a protocol that can tell them apart — and if you have that, you did not need the byte relay.
The backlog entry's aha was replaced, because the prototype disproved it. The entry promised "watch it multiplex many client connections onto few upstream connections". Built literally, it does not multiplex; it misdelivers. Rather than quietly build something else, the toy keeps the literal implementation (NaiveProxy) as the evidence, and the replacement aha is proved by the same program: the pool's checkout boundary is a parse. That is a better result than the one asked for, and it is only available because the first one was run.
A newline-delimited toy protocol, not HTTP. HTTP would make the toy realistic and the transcript illegible — the framing would be buried in headers and the reader would spend their attention on Content-Length rules that another toy already covers. GET key\n → key=value\n puts the delimiter on screen where the argument needs it.
Threads, not selectors or asyncio. An event loop is the right way to write a proxy and the wrong way to teach this one: it would roughly double the line count and spend all of it teaching readiness notification, which is a different mechanism. Threads also make the determinism hooks ordinary condition variables rather than loop internals.
Three proxies in one file, rather than one proxy plus prose. The comparison is the toy — the same call rate-limiter makes carrying two algorithms and lru-cache makes carrying two policies. A single correct pool would be about ninety lines and would teach the thing everyone already believes.
release="timeout" ships as a real code path, not a described one. It would have been cheaper to write "you could release on an idle timeout, but that would break". Half of such claims are wrong. This one is a branch you can run, and §6.6 is its output.
The gate is scaffolding and is admitted as such. Upstream(gate=…) exists only so §6.5's connection counts are pinned. --race shows the ungated numbers, so a reader can see exactly what the scaffolding bought and what it did not change.
Rejected as the headline: head-of-line blocking. _checkout's blocking self.free.get() is real and important, and a reader predicts its shape — ⌈requests/cap⌉ rounds of upstream latency — correctly before running it, which fails the aha test. It is also a wall-clock measurement, so it could not live in a byte-identical demo without breaking the one property §6.7 is built to protect. It is named in §3 and deliberately left unmeasured rather than asserted: no number appears on this page for it, because none was taken.
PassthroughProxy was nearly cut. It costs 56 lines, it takes the file further over the repo's ceiling, and it teaches a second mechanism. It ships anyway — see the appendix for what it buys and what that cost.
_checkout hands back whatever was in the queue. Real pools ping (SELECT 1, an HTTP/2 PING), cap idle age, and retry on a connection that died while parked — because the most common pooling failure in production is not corruption, it is lending out a socket the peer closed ten minutes ago.finally: returns a connection whose state is unknown. If _read_reply raises halfway through a reply, the socket goes back into the pool with unread bytes ahead of it. A real pool destroys a connection on any error rather than returning it, precisely to avoid the §6.6 failure mode arriving by a different route. Two lines in the toy, and a class of incident in production.self.free.get() blocks forever; there is no checkout timeout and no bound on how many borrowers can be waiting. Real pools have both, and the checkout timeout is usually the first latency signal you see when an upstream degrades.2 × pool_cap. Pool sizing is a fleet property, and this toy cannot show you that.A byte-relay proxy shares one upstream connection between two clients. Both send complete, well-formed requests, and nothing is interleaved on the wire. What does client B receive?
Client A's answer. That is §6.2: A asked for alpha, B asked for beta, and B received alpha=AAAAAAAAAAAA\nbeta=BBBBBBBBBBBB\n while A received b''. Every byte was forwarded in order exactly once, so the relay is not buggy — the reply simply carries no information about whose request it answers, and the relay's only available rule ("whoever wrote last") points at B.
Would routing replies to the client that has been waiting longest fix it?
No, and neither does anything else you can write in that function. §5.3 shows _pump_upstream receives nothing but bytes: no request id, no correlation token, and — because the upstream connection is shared — not even a socket identity to distinguish senders. §6.3 sweeps both defensible policies: broadcast gives everyone both answers, and under load (§6.8's swept run) last scored 1–4 of 24 and broadcast 0–6 of 24, a difference that is noise.
The information needed to route was never on the wire. This is a protocol problem being mistaken for a routing problem.
You set pool_cap = 16 with 8 clients, one request outstanding each. How many upstream connections open, and why?
8. _checkout (§5.4) tries self.free.get_nowait() first and only opens a connection when nothing is free, so the pool grows to the number of borrowers that were simultaneously unsatisfied — the concurrency level. With 8 clients each waiting for a reply before sending again, that is 8. §6.5's cap=16 row reads 8, with the same 3.0 requests/conn as cap=8.
Generally: upstream connections = min(pool_cap, requests in flight). The cap is only a lever below the concurrency level.
Your pool releases a connection when the upstream has been quiet for 50ms, because you did not want to parse the protocol. Your tests pass. What breaks it, and when will you find out?
Any upstream that pauses mid-response for longer than 50ms — a GC pause, a slow disk read, a large response written in several sends. §6.6: the same pool scores 8 of 8 when the upstream replies in one write and 0 of 8 when it pauses 120ms in the middle of a 19-byte reply. The borrower checks the connection back in holding alpha= and no newline, and the next borrower reads the remaining 13 bytes as its own answer.
You will find out under load, because upstream pauses correlate with load. The timeout is a guess about the upstream's scheduler; the delimiter is a fact about the message.
The same byte-relay proxy scores 24/24 with an 8:1 connection ratio. What changed?
Concurrency dropped to 1. §6.8 runs the identical NaiveProxy with 8 clients and 24 requests, one in flight at a time: 24 correct, 1 upstream connection, 8.0:1. With one outstanding request, "route to whoever wrote last" has exactly one candidate and is trivially right.
That mechanism is serialisation, not multiplexing — the same amortisation a mutex around a single connection would give you, with the same throughput ceiling. And it is a cliff, not a slope: the swept run in §6.8 shows 0 of 3 wrong at one client and 2–4 of 6 wrong at two.
Why does _read_reply return leftover bytes alongside the reply, and what breaks if it returns only the reply?
Because a recv can pull in the end of this reply and the start of the next one in a single call. The framed branch splits at the first \n and returns the remainder as rest, which handle stores with the socket in the free queue (self.free.put((sock, rest)), line 304) so the next borrower starts from it.
Drop it and the pool silently discards whatever had been read ahead — producing exactly the §6.6 corruption, but only at the moments the connection is busiest, which is the hardest possible version to reproduce.
Reaching past the toy. You run four replicas of a service, each with a connection pool of 25 to one Postgres configured for max_connections = 100. Traffic doubles and you raise the pool to 50. What have you actually done?
You have raised the fleet's demand to 4 × 50 = 200 against a limit of 100, so half your checkouts will fail at the database rather than queue at the pool — turning a latency problem into an error. Pool size is a fleet property, and the toy cannot show you that (§8: one process, in-memory state).
And per §6.5, the raise may buy nothing even if it fits: connections opened is min(pool_cap, requests in flight), so if each replica only ever has 20 requests outstanding, 25 and 50 are the same pool. Measure in-flight concurrency before you touch the cap — it is the quantity that binds.
recv until your message is complete. Read the myreceive example next to FramedProxy._read_reply.NaiveProxy does not have, and the reason HTTP/1.1 pipelining is effectively dead.keepalive upstream directive — a pool cap you can set in one line, with the documentation's warning about what happens when it is too high. §6.5's cap=16 row is why the warning is about wasted connections rather than wrong answers.socket.shutdown and RFC 9293 §3.6.1, "Half-Closed Connections" — the appendix's mechanism, from the API side and the protocol side.This is not part of the argument above. The main thread of this page is one mechanism: a pool's checkout boundary is a parse. This appendix is a separate result that fell out of PassthroughProxy, kept because it is worth keeping and flagged because it is a second mechanism in a repo whose whole rule is one mechanism per toy.
PassthroughProxy relays bytes in both directions with a thread per direction. When one direction sees EOF, it has to tell the other side. There are two plausible calls:
def _relay(self, src, dst):
while True:
try:
data = src.recv(4096)
except OSError:
break
if not data:
break
try:
dst.sendall(data)
except OSError:
break
if self.half_close:
try:
dst.shutdown(socket.SHUT_WR) # "no more from this side"
except OSError:
pass
else:
try:
dst.close() # ...and the reply dies with it
except OSError:
pass
with self.cv:
self.eofs += 1
self.cv.notify_all()A client that sends its last request and then calls shutdown(SHUT_WR) is doing an ordinary, legal thing: "I have no more to say, I am still listening." Here is what each of the two calls does to that client, with the upstream gated so it is still holding the reply when the FIN arrives:
One call, and the entire response. close() tears down the whole upstream connection, so the reply the upstream was about to write has nowhere to go and the client gets b''. shutdown(SHUT_WR) closes only the write direction, passing the client's "I am done asking" through as a FIN while leaving the reply path open — and all 19 bytes arrive.
TCP connections are two independent half-duplex streams (RFC 9293 §3.6.1) and close() is the operation that forgets that. The mistake is easy to make and essentially invisible in testing, because it only shows up against a client that half-closes and an upstream that had not replied yet.
tcp_proxy.py is 433 lines, or 357 excluding blanks and comments. The repo's ground rules say:
Aim at ~150 lines for the core implementation and treat 300 as the hard ceiling … a toy that wants more than 300 is two toys.
This file is over that, by a lot, and the rule is right about why. Where the lines went — these are exact spans, and they sum to 433 with the 12 blank separator lines between them:
| region | lines | count | why it is here |
|---|---|---|---|
docstring, imports, VALUES, free_port | 1–46 | 46 | |
Upstream | 49–151 | 103 | a real origin server with a framing loop of its own — the toy needs something that can be correctly wrong |
_Listener | 154–181 | 28 | one accept loop instead of three |
NaiveProxy | 184–249 | 66 | the evidence for §6.1–§6.3: the backlog entry implemented literally |
FramedProxy | 252–337 | 86 | the mechanism the page is actually about |
PassthroughProxy | 340–395 | 56 | this appendix, plus the honest 1:1 baseline in §6.4 |
| client helpers | 398–433 | 36 |
Cutting across that table, about 60 lines exist only so the page can be re-run: the five wait_* methods and their banners, the counters and notify_all() blocks that feed them, the gate, and the reply_split/reply_delay knobs. That is the price of putting a concurrent toy in a repo that demands byte-identical transcripts, and it is worth paying — §6.7 shows one missing hook quietly producing two different transcripts from the same scenario.
PassthroughProxy's 56 lines are the genuine luxury. They buy §6.4's 1:1 baseline and this appendix, and the appendix is a second mechanism.
The rule's own advice would be to split: a tcp-proxy toy about the pool boundary, and a separate toy about half-close and connection lifetime. That is probably still the right call, and this page notes it rather than pretending 433 and 300 are compatible numbers. (test_the_toy_is_over_the_repos_line_ceiling pins both figures, so this admission cannot go stale without a test failing.)