Cactus Whistle vs Whisper Base CPU benchmark shows tiny model holds its own
A 16.9 MB speech‑to‑text model leads greedy Whisper base on harder LibriSpeech splits, but no significant gap exists against beam search; the pinned engine also silently drops the middle of longer audio clips beyond ~30 seconds.
Tiny model, big claims
Cactus Compute has released Whistle, a 16.9 MB speech-to-text model that runs on a CPU and transcribes seven languages while returning word-level timestamps and confidence probabilities. The model card claims it beats OpenAI’s Whisper base on the LibriSpeech benchmark, a standard test of speech recognition. A new coding guide puts those claims to the test on a single Apple M4 chip.
Setting up a reproducible benchmark
The guide pins the exact versions of Whistle’s weights, the underlying Needle engine, and the supporting Needle 3 weights by their Hugging Face revision hashes. This ensures anyone can reproduce the run. The author installs the cactus-needle package, disables anonymous telemetry, and extracts the engine library from a wheel file. All hash values and file sizes are printed so readers can verify each component.
Accuracy across subsets
Whistle is compared with Whisper tiny and Whisper base on identical audio clips from the LibriSpeech test-clean and test-other sets. The benchmark uses paired speaker-cluster intervals with Holm correction, a statistical method that reduces false positives when making multiple comparisons. On the harder test-other subset, Whistle leads greedy-decoded Whisper base, but the gap closes when Whisper uses beam search—a more thorough decoding technique that explores multiple possible transcriptions. On test-clean, the models perform similarly.
Speed and scaling
Whistle shows a speed advantage on short audio clips, but that advantage narrows as clip length grows. The tutorial also looks into a 30‑second stretch where the pinned engine quietly drops the central part of longer audio clips. Streaming latency, timestamp accuracy, keyword biasing (making the model favor certain terms), and performance in other languages are examined. The guide notes that noise robustness and embedding quality are left for future work.
Beyond word error rate
Word error rate (WER)—the percentage of words incorrectly transcribed—is the main metric, but the benchmark also tests speech-to-tool calls, where spoken input is converted into function invocations for agentic AI systems. The guide provides full Python code so developers can run the same tests on their own CPU hardware.
This story reports on a trend for information only. It is not advice or encouragement to do any of it.
Key facts
- Whistle is a 16.9 MB CPU‑run speech‑to‑text model that transcribes seven languages with word timestamps.
- On LibriSpeech test‑other, Whistle leads greedy‑decoded Whisper base, but beam‑search Whisper matches it.
- Speed advantage shrinks as audio length increases; a 30‑second window causes silent dropping of middle audio.
- Benchmark is reproduced by pinning exact Hugging Face revisions for weights and engine.
Sources
- A Coding Guide to Cactus Whistle: Does a 16.9 MB Speech Model Hold Up Against Whisper Base? Word Error Rate, Speed, Stre — marktechpost.com
Every fact in this story comes from the source above.