Model benchmark

I tested four open-source speech recognition models for TalkTuck on Mac

Xuesheng Guo

This article preserves the product version, test conditions and views at the time of writing. Features, pricing, data handling and download options may have changed; consult the current product page. The original charts were not supplied; the verifiable figures in the text are retained.

I've recently been improving the Mac version of TalkTuck, the recording tool I build. Once you use speech-to-text in practice, there is a considerable distance between “it recognizes speech” and “people can rely on it.”

Someone may watch live captions during a meeting or search an hour-long recording afterwards. Captions that arrive too late are awkward during a recording; missing keywords in the final transcript make later search difficult.

My question was: on my Mac, how do several open-source speech recognition models balance recognition quality and resource use? I compared four model and runtime combinations on short Chinese and English clips, speed and memory. In this test Qwen 1.7B had the best recognition results and good batch throughput; SenseVoice stood out for speed and memory. The results have clear limits, particularly for overlapping speakers and long recordings: a leaderboard alone is not enough.

Test conditions

The test device was an Apple M6 Mac with 16 GiB of memory. The four combinations were:

The material came from public audio and reference text, not private user recordings. All four processed the same audio. Long recordings were cut into approximately 20-second clips; this was not a whole-recording long-context transcription test.

This compares combinations of model, quantization, runtime and hardware. Qwen 1.7B used the GPU and the others the CPU, so speed differences cannot all be attributed to model size. The machine was not completely dedicated to the tests.

Chinese: 169 short utterances

To avoid interference from overlapping speakers, I first examined a relatively easy-to-align Mandarin set: 169 utterances from 22 recordings, with 2,751 characters of reference text.

CER is character error rate: roughly the proportion of deletions, insertions and substitutions relative to the reference. Lower is better. On these utterances, CER was 7.16% for Qwen 1.7B, 8.36% for Fun-ASR-Nano, 9.05% for Qwen 0.6B and 9.31% for SenseVoice. Qwen 0.6B used automatic language detection here.

This gives an initial direction, but these were selected non-overlapping utterances, not a representation of every meeting, noisy environment or multi-speaker conversation.

A larger Chinese sample

The completed expanded data came from 17 distinct Chinese audio sources, totaling 1,422 clips of roughly 20 seconds. The four combinations processed the same audio; outputs were concatenated in time order and compared with references.

Approximate CER was 12.20% for Qwen 1.7B, 14.69% for Qwen 0.6B, 15.43% for SenseVoice and 16.75% for Fun-ASR-Nano. Here Qwen 0.6B was explicitly set to Chinese, unlike automatic detection in the earlier group.

These are approximate comparisons. Some material contains overlapping speakers and sentences crossing clip boundaries; the time-concatenated reference and recognition output may not be in exactly the same order. Some Chinese data remained unprocessed, so this is not a result for the full dataset.

English: approximate comparison on meeting audio

The English portion came from 17 distinct audio sources, totaling 932 clips of roughly 20 seconds. Again, all four processed the same audio. Qwen 1.7B's approximate word error rate was 14.53%.

WER is word error rate; lower is better. The material was primarily meetings, with overlapping speakers and sentences crossing clip boundaries. These are approximate results after concatenation, not a precise error rate for clear single-speaker English.

Among the completed data, Qwen 1.7B ranked first on both Chinese and English comparisons. That informed my choice, but does not mean it will be best for every accent and environment.

Performance: a smaller model isn't always faster in this setup

I also had the four combinations process 34 Chinese and English recordings, about 13 hours of audio, one at a time, and recorded processing time and sampled model-process memory footprint. For clarity, I converted measured throughput into approximate time to process one hour of audio.

SenseVoice took roughly 0.9 minutes per hour of audio, with a maximum sampled process footprint of about 0.85 GiB. Qwen 1.7B took roughly 3.9 minutes and peaked at about 3.6 GiB in the samples. Despite fewer parameters, Qwen 0.6B took about 5.5 minutes on CPU; it and the GPU 1.7B used different paths, so parameter count alone doesn't determine speed.

These times describe batch throughput on short clips, not live-caption latency. Memory is sampled model-process footprint, not whole-machine peak memory. The three ONNX models used two CPU threads, while Qwen 1.7B used the GPU. I did not obtain directly comparable whole-machine power measurements.

Fast while recording, accurate afterwards

In TalkTuck I want captions quickly while recording and an accurate final transcript afterwards. For Mandarin and English, processing has roughly three stages: live captions, a quick correction of a complete utterance, and a final transcript from original audio.

In a file replay of about eight minutes of public Chinese audio, the three-stage comparison yielded 13.35% CER for live captions, 9.21% after quick correction, and 6.40% for the final transcript. In this sample, correction improved the text visible during recording. But a “final processing only” comparison on the same audio also produced 6.40%, with the same final text.

The supported conclusion is that quick correction improved the interim reading experience in this sample; it did not further change the final result in that comparison. This was one file-replay sample, not a microphone recording, and cannot establish the same result for all long recordings.

Choosing a quality-first approach for TalkTuck

Based on the completed tests, I chose Qwen 1.7B MLX—the best-recognizing combination in this test—as the quality-first transcription path for TalkTuck on Mac. On this 16 GiB machine, Chinese and English results were relatively good and batch throughput acceptable.

SenseVoice remains interesting because of its resource use. Before offering it as a performance-first option, I would still need to test live captions, hotwords, time alignment and integration with the existing speaker workflow.