Nari Labs' Qwen3-TTS hits sub-50 ms audio response, 10 RPS on one H100

How We Made a Text-to-Speech Model Respond in Sub-50 ms

Nari Labs' implementation of Qwen3-TTS 1.7B CustomVoice achieves sub-50 ms p95 time-to-first-audio (TTFA) at 10 requests per second on a single NVIDIA H100 SXM, while maintaining real-time playback with zero underruns. The team compares five serving engines, tunes them for low latency, and open-sources both the implementation and benchmark. Key optimizations include a unified scheduler for the three-model architecture, dynamic leading-silence trimming, state-cached incremental decoding for the codec, and CUDA graphs. At $4.29/hour for the H100, this translates to roughly $2 per 1M characters, far cheaper than commercial APIs like ElevenLabs V3 ($100/1M) or Cartesia Sonic 3.5 ($49/1M).