Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
Also, this test is English-only, while a strong point of other models is to understand different languages without first having to say which one (so you don't need 3 different keyboard shortcuts if you wanna dictate in 3 languages day-to-day)
Reminds me of the time my neighbours must have wondered if I was having some kind of a breakdown when trying out really basic MacOS voice recognition back in the early 2000s. There was a keyboard shortcut and you could say something like "phone number for firstname lastname" and it would theoretically show you that phone number. Thing is it didn't seem to like a British accent, so I spent a good hour trying out different accents, rotating through various US accents, Australian, South African, Canadian and so on. It seemed to respond best to some kind of a melange of Californian / Australian.
Not too far off what happened, although thankfully I wasn't actually trying to do anything other than test it. Going to take the opportunity afforded by Scottish TV comedy here, and make a very tenuous link to intercultural exchange so I can post my favourite Rab C Nesbitt scene, hands across the sea indeed: https://www.youtube.com/watch?v=uKxPH_QH940
Interesting - I don’t think I’ve ever seen anyone from the UK refer to talking in a “British accent” before since we are normally aware of the wild regional variations.
Fair point! I think it's a tic from being English and having lived in Scotland for quite a while so I autocorrect "English" to "British", but I've over-corrected here. (Also perhaps something to do with "British English").
I use Systran/faster-whisper-medium for real-time subtitling, but you need to get used to the context it's used it and the weirdness it translates into. Parakeet has great mandarin>CN text, but running that + a translation model has been tricky and I never got it fast.
I run it offline too, don't want to depend on separate services.
I use TranscriptionSuite, which is focused on offline transcription, and supports Parakeet as well as whisper and others (https://github.com/homelab-00/TranscriptionSuite)
The dev was sweet enough to improve the live mode GUI so that I and some friends can use it when playing on Chinese mmo servers.
As an Australian, Apples voice models have always sucked. I've tried using stt (again) more recently and its improved, but i'm so tired of having to Americanize my voice to get it to figure out what the hell i'm saying.
As a Texan first, American second, I sympathize with this statement. Siri can't understand me probably 25% of the time. I use STT for iMessage while in the car, and half the time it will take 3+ times to either get it right or me give up, and hope to remember to text them by hand when I next stop.
What does this mean? When I pledged my oath to become a citizen, I had to promise to put America above all other allegiances. Is it in relation to allegiance in the general sense or some weird statement that only relates to STT?
Texans (generally) consider themselves Texans firsts. Similar to born and raised New Yorkers (NYC). That said, I really meant it tongue-in-cheek that my accent was too thick, and doesn't work well for Siri's STT model. ChatGPT's does way better, but I really don't like talking to an LLM.
The Two Yoots problem. Do you use d's in place of t's such as dees/dems/dose/dere? I have a heavy queens accent so you'll hear me say things like "deres tree uh dem ova dere."
> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia
Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however.
I think that's parakeet doing its job there. That is a closer reflection of what you've actually said. The trick is then throwing that output through some additional deterministic and non-deterministic steps to tidy it up however you prefer. It's exactly what I do with my free and open source dictation app (dictator.robgough.net) for Mac+iOS. And of course, everything stays entirely on-device. Gemma E4B is really wonderful for that second step, it's great at language – but takes up 6-7GB RAM.
Site could identify our device and send iOS visitors to the iOS page (or maybe that’s against the vibe and we should tap it ourselves).
App might be able to launch the keyboard settings directly but I suppose Apple doesn’t like devs using those undocumented URIs (uhg but maybe can understand part of it).
Keyboard, given manual app switchbacking + manual pasting, is less convenient for some of my use cases compared to an action button shortcut. (Reference Spokenly w/Local-Only Mode.)
Separating dictation and the history, and having a syncable scratch pad, are some welcome innovations!
>Gemma E4B … takes up 6-7GB RAM.
Google has a spyware-adjacent dictation app (maybe not really, but they demand to connect to servers after you enable their offline toggle). Just like yours, they thought of a cute name too (Google AI Edge Eloquent). Do you know what language model they install on the iPhone? Not a very good one but I’m sure over the next year or two…
I really wanted to have background audio and make it so the keyboard would directly record audio etc, but my first pass didn't make it through app review (and that was just keeping background audio listening AFTER you'd already started a recording). I could maybe have fought it, but figured if I was already butting up against app review there was little point as they'd likely reject in a future release anyway.
Re: analytics, It is quite weird having no idea how many people are using the app. It does leave you a little blind, but I figure people will get in touch if they have big enough problems with it.
For the Google app, I believe Gemma E2B and E4B both have audio input, so I suspect they're using one of those.
Analytics: if you generate them on my device, and bury an option to manually send them to you when I have a bug--or...
you instead bury an opt-in to automatic telemetry, and let us CC ourselves each ~month when the analytics get sent in to you (so we can verify it's all boring data)...
It's hard for me not to opt in to stuff like that, at least periodically. If I'm opted-in by default? Ehhhhh.... I totally get it, weird "flying blind" (quoting a different dev w/similar philosophy), but I guess I'm still weird myself and hence looking for that autonomy or something? Oh and if somebody makes something opt in, and then certainly if they furthermore stick the option somewhere slightly off the beaten path, that seems pretty darn trustworthy.
(I wonder if we'd have enough public data for a decent statistician to calculate a likely number range of how many users you have by extrapolating from the 1% to 10% or 20% of users who'd chose to opt in...)
PS: "bury" concept likely not actually important :)
Parakeet is certainly faster on my machine (m3 max), but I can't stand using it over Whisper for dictating my prompts. It makes a lot more mistakes, possibly because (like you mentioned) large portions of the speech will pause / stutter as I think about what to include.
With whisper v3 turbo, I can almost always live with the few mistakes because the overall stream-of-thought word-salad I provide is still clear at a high level. The bits and pieces of context seem to help, that I might leave out if typing and focused more on traditional conciseness / clean writing. With parakeet, I needed to do frequent editing even for shorter bits of speech.
I realize some applications prioritize the latency.
The near-instantaneous nature of Parakeet has led me to keep using it and occasionally simply re-dictating a second time as needed.
For round two after a typo-laden transcript, I’m dictating and annunciating with great passion as I read the first transcript to make sure I don’t miss a beat. It’s kind of fun because it’s a little performance.
It sounds like post processing should be the job of an LLM. I would like the voice model to be faithful to what was said and then that output can be smoothed over or postprocessed as needed for the use case
To be clear, I'm talking about high word error rate with parakeet vs whisper, not post processing and cleaning up my speech. Re: being faithful to what was said, one small example, Whisper will often put ellipses when I pause.
Thanks, was looking at a better diarization model.
Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.
I'm using it offline. But it's much faster than realtime so it should be usable for streaming. I just asked Codex / Sol to integrate FireRedVAD with Whisper...
Thanks, I didn't see vanch007 version at first (only ~30 downloads), I usually look at mlx-community. For the size I was looking at the wrong model (TTS not transcribe-diarize), thanks for the corrections.
From my experience, Speech-to-text falls way short of Wispr flow and I would assume the ones that are said to be better than that. It lacks context awareness and formatting
I've never been able to record a voice memo on my watch and have it sync to my Mac. It's never once happened. Not sure if I have it configured wrong or what.
> However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
What's insane to me is that you have all of these low-quality me-too apps, and literally no one could bother to read the damn Human Interface Guidelines or follow iOS design conventions.
Doing so is literally LESS WORK than trying to make your own custom awful iOS UI.
Not if your app is a Web wrapper, which so many of these are.
If you use SwiftUI (the native recommendation by Apple), it severely penalizes you, if you want to paint outside the lines (which is a big reason that I don't use SwiftUI for shipping apps). It's insanely easy to write a native app that is 100% in line with HIG.
Whisper v3 is still the best (by far) when it comes to poor quality input (say background audio from a security camera), though remains more susceptible to hallucination so it's a bit of a tradeoff.
Yeah, apple will be optimizing a model to work on ANE and then turn it into a native app. My only hope is that it has a reasonable api so that I can use that as a generic input source across iOS / macOS that’s equivalent to the ubiquity of the keyboard.
One would hope, though I suspect they may want to make things a bit flashier than “we made the audio transcription on the keyboard not terrible” in the changelog given the amount of work that’s gone in.
> RIP to a lot of the paid apps that simply wrap Whisper
I started using a few open source apps for transcription and eventually subscribed to a paid one...
On paper, it's not hard to compete, but for this use case, a few rough edges make it really frustrating to use. Like a keyboard that sometimes doubles the letter "e"
Automatic dictionary, seamless language switch, no issues with accents, etc... Putting the effort in the last mile makes a world of difference.
If anyone has better options, I'm willing to have a look. The best open source solution I found was Handy, and I currently use Wispr Flow
I built my own because I was frustrated with a lot of the free options. Largely because a lot of them had an upsell to be able to do the secondary post-processing step with an LLM. And it wouldn't pick up things like emojis properly or say numbers. Because of that, I left quite a lot of options in there for customising and adding additional steps, etc. Feel free to take a look: dictator.robgough.net
My initial Mac version actually had three additional steps that you could toggle, obviously at the cost of some speed. That is what the website talks about, although nowadays for my own use I've reduced that to just one step and found that it's pretty great. I've got a new version in test to tidy that up, but still lets you define as many steps as you want.
The difference using an mp3 seems to be smaller: yap seems to use about the same time but fluidaudio seems to take twice as long. Do you happen to know why?
Investigated this and it turned out to be an amusing bug: audio decoding was happening three times instead of just once lol. I've put up a PR to remove the wasteful redundant decoding:
With the updated PR code, ran a test comparing transcribing (using Parakeet V3) a 1 hr stereo 44.1 kHz mp3 vs the same audio in 16 kHz mono wav format. The result was about 21.3% slower with the mp3 vs the wav, i.e. that's the overhead of decoding + resampling.
Currently the decoding + resampling is done up front. If it was done in a pipelined fashion with the inference, that overhead can be eliminated. This is what I did in a recent app I made:
It uses FluidAudio as well, but I forked it and replaced the audio decoding code to (a) use mpg123 instead of the native Apple API and (b) do audio decoding and inference in a pipelined fashion. These two changes effectively eliminated the overhead. mpg123 is quite a bit faster than the native Apple API at mp3 decoding (has some very optimized arm64 assembly routines), and the pipelining ensures that the inference is never starved by the mp3 decoding.
Contributing this pipelined setup to FluidAudio would be good.
I use fluidaudiocli and it's unfortunate that it doesn't support streaming (e.g. from a named pipe); that would have been an easy workaround to both the pipelining problem and the faster-decoder problem.
Yep, agreed, streaming would be great to have. Although via pipes I think is better suited for a macOS. Since FluidAudio has to support both macOS and iOS, I think using Swift concurrency primitives would be the better fit. That's what I did for my app (TaskGroup, actor, AsyncSequence).
That app was exactly what I was looking for, something like SponsorBlock but for podcasts but I suppose using AI for finding the ads works too. Any chance it'll release on Android?
Yep it's something I wanted for a while too; there were existing apps that did this, but had two issues: they were paid, and the UI was subpar. So for mine, I made sure it's fully free and that the UI is on par with Apple Podcasts, Spotify, etc.
Making the ad-finding cheap enough such that I could make it free turned out to be harder than expected. The main issue you run into is dynamic, location-targeted ads. So I came up with a novel technique that uses Shazam-style audio fingerprints for accurate matching, instead of their normal use case, which is identification. This technique is what allows the ad finding to be very cheap, allowing me to make it free.
The SponsorBlock model would actually not work for podcasts, due to dynamic ads. I.e. the location and content of the ads in episodes these days varies by download location. You need the media to be static, like YouTube, for SponsorBlock model to work. Therefore, using an LLM to find the ads + the fingerprints matching in combination is an efficient technique.
Android has def been the most requested thing thus far haha. It'll be a decent undertaking due to me having written the app fully in Swift, i.e. it'll be a complete rewrite. I'll also need to replace FluidAudio with some good, fast Android equivalent.
The goal of making this app was to create something impressive so that I could get a job. Haven't gotten a job yet, but if and when I do, then I'll have time & resources to think about doing an Android version. Currently a bit stressed and occupied from the job search lol.
Could you speak more on the Shazam style part? Don't you need a database of all ads to figure out if some snippet is an ad or not? That's how Shazam works for songs at least.
A native implementation I had thought of was running the audio through a STT LLM that then detects the ad timestamps then returns that to the UI to block.
Yeah that's what I do (STT -> LLM -> timestamps to block), but if you did that for every episode download, due to location-targeted dynamic ads, your LLM API bill would run up very fast. So that's where the fingerprints matching comes in, for further versions of an episode (version = same episode but with different dynamic ads inside).
I don’t know how Apple divides computation between the GPU and the Neural Engine, but one major benefit, especially for real-time transcription on laptops, is the improved power and thermal efficiency. I noticed better accuracy after switching my app to SpeechAnalyzer, and I suspect part of that improvement for me came from the microphone no longer having to compete with jet-engine fan noise.
I’ve regularly used an M1 Max, M4 Pro, and M5 Max. They all get pretty loud when driving local LLMs. “Jet-engine” would be hyperbolic, but it’s definitely noisy.
This particular product used Whisper, so that was obviously the right model to compare it against. Further this is explicitly on device, and Nemotron 3.5, as one example, is 2.5GB for the model.
And if someone were broadly comparing all on-device models (instead of just looking at how this new on-device ones compares to what a specific product uses), Nemotron 3.5's WER are actually a bit higher than what they report for SpeechAnalyzer, for both tests.
> RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers
I'd love this, but updated spotlight did not obviate my need for Raycast. I question Apple's ability to make good software at this point.
Being Apple’s model, it will support like 8 languages and leave the rest hanging for 10 years, just like Apple impotently ignored 10 or 20 million countries even with basic “just download open dictionary and run a deploy script” keyboard autocomplete.
The canadian government will provide lots of historical data for curious citizens, many of which are recordings of interviews from decades and decades ago. For a book project this allows me to make a hours of audio searchable through a GUI application I have developed that has a voxtral backend.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.