I run zpl.tools, a ZPL-to-PDF rendering API written in Rust, on Fly.io. While benchmarking it I found a step in the latency curve between 40 and 45 pages that had nothing to do with the renderer.
http_service · k6 from a Hetzner cpx12 in FSN1, median of 10 requests.
40 pages take ~0.6 s, 45 pages take ~3.5 s, with no measured processing-time difference on the server side. That pointed me towards fly.io.
The step lands at ~10 MB of request body, and my benchmark labels cross that line between 40 and 45 pages.
After searching Fly's forums I found a mention of similar behaviour. It confirmed that fly-proxy's HTTP handler buffers the first 10 MB of a request body so it can retry or fly-replay it. Once that buffer fills it stops reading from the client until the app answers, or a hardcoded 3 s timeout fires.
That exactly explains what I observed. A render cannot answer before it has the whole label, so there is no early response for the proxy to see, and every request over 10 MB paid the full 3 s.
Then I wrote a post on Fly's community forum, where Fly confirmed the behaviour and pointed me in the next direction for the mitigation.
The workaround suggested in that earlier thread is that the pause ends when the proxy gets a response from the instance. An interim response looked ideal: it wouldn't force me to commit a status code before parsing the body. So I tested the entire 1xx space: 100, 102, 103, 199. fly-proxy keeps waiting through all of them: an interim response doesn't release the buffer.
Fly suggested to us HTTP trailers, and they would work. It means we would have to commit to a response code before processing the full request. A label can fail to render, exceed a quota, or turn out malformed, and none of that is visible before processing the full request.
API clients expect a standard status code, so I'd have needed a Cloudflare Worker in front to read the trailer and reconstruct the real response. It would have worked, but it is a lot of moving parts to keep a workaround invisible, and I decided against it.
Two http_service features need that buffer.
retry: covers Fly's own routing rather than your app failing. The proxy can route to an intermediary host that has just died and that it hasn't learned about yet. With the body buffered it sends the request somewhere healthy and the client never notices. Without the body it can't.
fly-replay allows you to read a request (up to 10MB in size), then hand it to a different machine or region.
The retry is a genuinely good property, and at enough traffic you would hit that window regularly. I have not hit that scale. And a render is idempotent (there's no state to corrupt by doing it twice), so a failed request is safe to send again. I would rather the client retry, or retry under my own policy, than delay every large request.
I never used fly-replay.
So the trade is a fixed 3 s on every request over 10 MB, against a rare failure I can absorb and a routing feature I do not need.
I decided to use a raw TLS service. Fly terminates TLS and pipes the TCP connection to the app. Skipping the whole fly.io http processing.
The TTFB dropped instantly. 45 pages went 3.51 s → 538 ms.
http_service and over raw TLS · k6 from a Hetzner cpx12 in FSN1. Up to 40 pages the two curves are one measurement: those requests are under the threshold, so nothing changed for them.
The original autoscaling was based on the request count. Dropping to raw TLS makes it possible to scale only by TCP connection count. So I tried that. Behind Cloudflare it is not viable.
Cloudflare doesn't close an origin connection after a response. Its connection limits documentation puts the Proxy Idle Timeout at 900 s (15 minutes) and states that it "reuses open TCP connections up to the Proxy Idle Timeout limit after the last HTTP request", and lists it as not configurable.
Based on my observation, each Cloudflare edge machine holds its own connection pool. So the count fly-proxy sees tracks Cloudflare's edge fan-out and idle timers, not how much work is in flight.
Under sustained load the autoscaling either scales too far, or hits the hard limit, which reaches clients as 520s on a quarter of all requests in my benchmark, while the machine sat nearly idle.
Responding with Connection: close could help. But it also costs a fresh TLS handshake on essentially every request, plus a cold congestion window on every large upload. That is a tax on exactly the requests I had just made fast.
I measured the handshake on its own, on a body small enough to have no upload ramp to hide it: about 100 requests each, pooled against closed.
Handshake cost on a small request22 ms, and on a large upload the cold congestion window is charged on top of it.
No setting makes socket count mean "work in progress." To get that signal back, something has to count dispatched requests.
We started using Cloudflare primarily for better visibility into customer latency. We add a timestamp to the request at the edge, which lets us measure the latency the end user actually observes. Fly has a header carrying one too, but it isn't in the documented request headers, and it is not documented where in the pipeline it gets applied.
So Cloudflare stays and connection counting stays broken. The scaling decision has to move somewhere that understands the workload.
After another response from the Fly.io team, who confirmed that calling the Machines API yourself costs the same as the built-in autostart, because fly-proxy does that internally, I decided to add my own proxy.
The app is now split into two process groups: a proxy and a worker.
The proxy, built on Pingora, terminates the public surface and owns auth and the billing gate. Making the workers stateless rendering nodes. Previously the app was a single monolithic container, and every copy of it carried auth and billing along with the renderer. This lets us independently scale the worker nodes.
The connection problem went away with it. An always-on proxy is the thing Cloudflare pools its sockets against, and those sockets are no longer anybody's scaling signal: the proxy counts dispatched requests and starts workers on that. The same hour-long benchmark that had failed a quarter of its requests now runs at a zero error rate.
And because the proxy is now the thing holding the public surface, keep-alive came back. Measured from the same Hetzner client across the hosted benchmark suite, against the Connection: close build:
| comparison | median ratio | median delta |
|---|---|---|
vs the Connection: close run | 0.828 | −22 ms |
| vs the original keep-alive run | 1.028 | +2 ms |
The −22 ms is almost exactly the per-request handshake I had been paying. Removing Connection: close gave back what it cost.
We managed to cut the request time from ~3.5 s down to ~0.52 s, and streamline the app scaling at the same time.
The proxy suspends idle workers rather than stopping them. A stopped machine has to boot; a suspended one restores a memory snapshot. Across the measured cold requests, a stopped machine processed the request in ~1.51 s and a suspended one in ~0.59 s, 61% faster. A warm worker answers in ~150 ms. The snapshot's storage cost is negligible.
What a cold worker costs the requestThe split moved a ceiling rather than removing it. Every request now goes through one proxy machine, and a Fly machine gets 1 Gbit of internal network. That gigabit used to belong to a single renderer; it is now shared by the whole cluster sitting behind the proxy.
At 10 MB a label it takes a lot of concurrent uploads to saturate, and I have not come close. But it is a real limit, and it is the kind you only meet under load.
The fix is the lever the split already bought me: run more than one proxy. It is a small machine, 1 shared CPU and 512 MB, so a second one is cheap, and it brings another gigabit with it.
Three runs in August 2026: the cliff on the 9th, the raw TLS service on the 10th, and the proxy split on the 12th. k6 drove the API one request at a time from a Hetzner cpx12 in FSN1 against Fly machines in Frankfurt. Before the split the API was a single machine on 2 dedicated CPUs and 2 GB of RAM; it is now a proxy on 1 shared CPU with 512 MB and renderers on dedicated compute. Each point is the median of 10 requests, 5 on the heaviest, and the renderer itself is unchanged throughout: the only thing that moves is the network path in front of it.
Next I want to look at machine types for the workers. Shared CPUs are a lot cheaper than dedicated ones, but a sustained render workload runs over its CPU share and gets throttled for it, so I have stayed on dedicated. The proxy is in a position to change that. It already sees the size of every request, so it can predict roughly what a render will cost and track what each shared machine has left, instead of handing work to one that has already burned its budget.