I Tried to Build a Bengali Voice Dialer for Android. Here Is What Actually Happened and How I Finally got it right.

A personal project aimed at creating a voice dialer app for an elderly, non-English-speaking mother led to a long and challenging journey through various Whisper models, Android's scoped storage limitations, and the quest for on-device AI solutions. After testing different Whisper models on a MacBook, the turbo variant of the large-v3 model proved to be the most successful on Mac, producing clean output in either Bengali script or Devanagari.

However, porting the model to Android presented significant challenges due to the latency introduced by the encoder pass, which took 60-80 seconds for a three-second voice clip. Attempts to optimize the encoder by reducing the window size and using the OpenCL GPU backend were unsuccessful. The author then switched to using sherpa-onnx, a production ASR framework, which provided a significant speedup, reducing the transcription time to about 2 seconds.

Despite this improvement, the app still faced issues with language detection, as Whisper's internal language classifier randomly misidentified Bengali utterances as Hindi, English, or Spanish. By writing a custom mapper to normalize the output and extending the intent parser, the app was eventually able to transcribe Bengali speech accurately.

The final breakthrough came when the author discovered a Bengali-specific model published in the sherpa-onnx release assets, which drastically improved the accuracy and performance of the voice dialer app.