Sending every LLM request twice beats paying 2x for priority tier
A simple fix for LLM tail latency
At HOAi, we built a voice agent that answers phone calls, and slow LLM responses caused awkward silences that made callers hang up. We benchmarked two fixes: upgrading to OpenAI's priority tier (2x cost) versus sending each request twice on the standard tier and taking the faster response. Replaying 50 production requests, the double-send approach cut worst-case time to first token from 4.2s to 1.2s and worst-case complete response from 9.8s to 3.5s, beating the priority tier on every tail metric. The trick works when slow responses are rare and independent, making it unlikely both copies are slow. Before paying for a faster tier, test this simple alternative.
Sending the request twice clearly outperformed the priority tier.
- nine_k
Sending two identical parallel requests is the classic approach. But, logically speaking, it should also double the cost.
I would send a second request if the first request fails to return the first token within, say, 1 second. Then there's a chance the first request is stalling, which is an infrequent event.
I wonder if higher-availability tiers of LLM providers do a similar thing internally.
- dvaplima
Nice turn around, does anyone has a benchmark regarding other types of requests (priority vs send twice) other than voice/call? Or the tests already test that?
- ramon156
for a tier thats twice the cost i would expect >2x the speed. somewhere 5-10x
e.g. 1.40m would become 0.30s.
do people really pay for these priority plans?