We turned EmbeddingGemma 2 into a speech-to-text model

Google's EmbeddingGemma 2, an embedding model capable of handling text, images, audio, and video data, has been repurposed for speech-to-text functionality. This transformation allows an hour of audio to be processed into text in just 13 seconds on a MacBook Air, all performed directly on the device without cloud processing. EmbeddingGemma 2's unique ability to convert speech into written text is demonstrated when you speak a word out loud; its output aligns closely with the spoken word.

To achieve this, the model was trained under two hours using 8×H100 GPUs on openly licensed English speech data. Despite not being specifically designed for speech-to-text tasks, the base EmbeddingGemma 2 model was successfully adapted for this purpose. The entire processing time required is roughly one second of audio processed for every second of processing, as showcased in rows 1-2 of the tests conducted on an M3 MacBook Air.

For comparison, FluidAudio published benchmarks on larger, more recent Macs and varied audio formats in rows 3-5 of the tests. While the word error rate currently stands at around 5%, which is significantly higher than the 2-3% error rate found in leading speech models, it is anticipated that with further training, the gap between these two sets of error rates could be narrowed.

The model's simplicity is highlighted by the user-friendly interface, allowing users to simply drop an audio file into the model or activate speech-to-text by holding Option and speaking, with the audio processing occurring directly on the Mac's Neural Engine without requiring the audio data to leave the device.