Cactus Whistle vs Whisper Base CPU benchmark shows tiny model holds its own
A 16.9 MB speech‑to‑text model leads greedy Whisper base on harder LibriSpeech splits, but no significant gap exists against beam search; the pinned engine also silently drops the middle of longer audio clips beyond ~30 seconds.
What you could do with it
- Researchers could verify the model's claims by running the same benchmark on their own computers using the published code and pinned weights 1.
- Developers could compare Whistle against OpenAI's Whisper models on identical audio clips to check transcription quality and speed on their own hardware 1.
- Power users could run voice transcription locally on everyday devices instead of sending audio to the cloud, making the process faster and more private 1.
Tiny model, big claims
Cactus Compute has released Whistle, a 16.9 MB speech-to-text model that runs on a CPU and transcribes seven languages while returning word-level timestamps and confidence probabilities. The model card claims it beats OpenAI’s Whisper base on the LibriSpeech benchmark, a standard test of speech recognition. A new coding guide puts those claims to the test on a single Apple M4 chip.
Setting up a reproducible benchmark
The guide pins the exact versions of Whistle’s weights, the underlying Needle engine, and the supporting Needle 3 weights by their Hugging Face revision hashes. This ensures anyone can reproduce the run. The author installs the cactus-needle package, disables anonymous telemetry, and extracts the engine library from a wheel file. All hash values and file sizes are printed so readers can verify each component.
Accuracy across subsets
Whistle is compared with Whisper tiny and Whisper base on identical audio clips from the LibriSpeech test-clean and test-other sets. The benchmark uses paired speaker-cluster intervals with Holm correction, a statistical method that reduces false positives when making multiple comparisons. On the harder test-other subset, Whistle leads greedy-decoded Whisper base, but the gap closes when Whisper uses beam search—a more thorough decoding technique that explores multiple possible transcriptions. On test-clean, the models perform similarly.
Speed and scaling
Whistle shows a speed advantage on short audio clips, but that advantage narrows as clip length grows. The tutorial also looks into a 30‑second stretch where the pinned engine quietly drops the central part of longer audio clips. Streaming latency, timestamp accuracy, keyword biasing (making the model favor certain terms), and performance in other languages are examined. The guide notes that noise robustness and embedding quality are left for future work.
Beyond word error rate
Word error rate (WER)—the percentage of words incorrectly transcribed—is the main metric, but the benchmark also tests speech-to-tool calls, where spoken input is converted into function invocations for agentic AI systems. The guide provides full Python code so developers can run the same tests on their own CPU hardware.
This story reports on a trend for information only. It is not advice or encouragement to do any of it.
Key facts
- Whistle is a 16.9 MB CPU‑run speech‑to‑text model that transcribes seven languages with word timestamps.
- On LibriSpeech test‑other, Whistle leads greedy‑decoded Whisper base, but beam‑search Whisper matches it.
- Speed advantage shrinks as audio length increases; a 30‑second window causes silent dropping of middle audio.
- Benchmark is reproduced by pinning exact Hugging Face revisions for weights and engine.
Sources
If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work
Keep reading
Apps and productsYogurt lets you record and summarize meetings with AI that runs entirely on your own computerHerald of AI
Yogurt lets you record and summarize meetings with AI that runs entirely on your own computer
A free open-source app turns your calls into transcripts and summaries without sending any data to the cloud.
How it worksHow Meta boosts engineering productivity by 3× using agents to propagate ad model techniquesHerald of AI
How Meta boosts engineering productivity by 3× using agents to propagate ad model techniques
Meta's new AI agent system adapts small improvements across thousands of ad models automatically, saving millions in computing costs.
How it worksA large AI model learned to fix its own brain without being retrainedHerald of AI
A large AI model learned to fix its own brain without being retrained
Researchers found a 27-billion-parameter model can edit its own internal settings to become more accurate — no human teacher needed.