Apple's SpeechAnalyzer Beats Whisper: The First Real On-Device Benchmark
Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
We benchmarked Apple's new SpeechAnalyzer against Whisper models and found it delivers superior accuracy and speed on Apple hardware. Our tests show SpeechAnalyzer significantly outperforms both the legacy SFSpeechRecognizer and Whisper Small, making it the best on-device option for English transcription. We updated our own Inscribe app to prioritize this new engine, proving that the built-in solution is no longer a compromise.
For English transcription on a current iPhone or Mac, the days of Whisper being the automatic accuracy pick are over.
- satvikpendem
Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
- ashivkum
Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.
- mchusma
I will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference.
Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.
- summarity
Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
- mvkel
One tangible thing this doesn't touch on: SpeechAnalyzer supports streaming, so you can see what it's hearing from you as you talk. A massive UX improvement. Many of the other models force you to record, then it transcribes the audio to text as a single job, then returns the entire blob of text. It's slow, and frustrating if you're talking, only to realize it stopped listening after 2 minutes.
- modeless
Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?
- atonse
Just this week I built a live subtitles app for my mother in law who is hard of hearing (she has a hearing aid but still has trouble decoding, but can still read faster than me)
So I ended up organically testing and ending up with SpeechAnalyzer because it was not only fast and accurate enough, but you also see live results as you talk. It also has speaker identification and people can register their voices. And it does all processing on device.
It also had the best model for Indian accented English, given she lives in India.
So I was quite impressed, but the holy grail to me is transcription that does speaker identification but also works in a standard family conversation, where multiple people interrupt each other all the time.
I will say though, I'm really curious as to what Claude Code Desktop uses for their voice mode, because it seems even better than Apple's, and it provides realtime feedback. Maybe they're using apple's model?
- MBCook
Impressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.