RealtimeVAD - Faster speech-off detection for voice pipelines
Show HN: RealtimeVAD – trading false positives for 110ms earlier end-of-speech
RealtimeVAD solves a critical bottleneck in voice assistants: the wait for speech to end. It keeps Silero VAD for speech detection but adds a second, faster check every 10 ms that compares audio levels to the room noise and speaker's own level. This delivers end-of-speech events a median 100 ms earlier than Silero alone, accelerating ASR, LLM calls, and TTS. The trade-off is more false positives on short pauses, but these can be used to start cancellable work. It runs on one CPU thread with only numpy and onnxruntime, and includes a LiveKit plugin and drop-in replacement for Silero's VADIterator.
Know that the speaker stopped, about 100 ms before Silero does.
- nowittyusername
You can also avoid the problem all together. I made a "speculative regeneration" at least thats what i call it pipeline i been using for many months now and it works great. basically as long as you have access to the inference server you are good to go. idea is as such, every n milliseconds of silence (usually 300ms) for me, you send the asr transcript for the llm to process immediately. you let the model start answering , but if the user starts talking again before the end of turn token is reached you cancel the generation and again wait for the 300ms. rinse repeat. there is a safety window you have to set up as well that way the final llm answer gets finalized regardless of its inference speeds. works like a charm. basically the user gets 300ms latency at the highest for that "silence" window. so the only other thing you have to worry about as faras latency is llm inferance speeds and asr and tts latencies. in my pipeline the moment the user stops talking, the model ,is already responding within about 900ms - 1.2 sec.. its my gemma 26b moe model , 4 bit quant thinking on... with mtp getting about 300 tokens per second... asr and tts are both under 350ms for total... anyways, speeds could be improved if turning off thinking but i like my voice agent to be smart as he has tool calling and common sence capability is reduced when thinking is off.... fun times!
- com2kid
This is the 3rd or so time I've posted this -
Run a tiny shitty llm, one that is super fast. Or better yet try out a decision model now days.
Feed it the previous conversation turn and all the STT (ASR output) you have so far and just ask it for a probability that the user is done speaking.
Use that probability to shorten your VAD time.
It isn't as good as a true speech to speech model but this will cut a few hundred ms compared to pure VAD. Oh as a bonus it also lets you wait longer when the user isn't done talking yet.