The models small enough to fit on a phone are the worst at understanding children

When creating a voice product for children, the decision between processing locally on the device or in the cloud becomes more than just a matter of saving money or reducing latency. For products aimed at children aged three to five, the choice is driven by market access and the product's overall market strategy. The study conducted on WhyWave, a voice companion for this age group, revealed an unsettling trend: smaller models consistently performed poorly in understanding children's speech.

One key rule prohibiting children's apps from transmitting identifiable information is Rule 1.3 of the App Store Review Guidelines. However, this rule creates a conflict with the usual voice processing pipeline, which sends audio straight to the cloud for recognition and synthesis. This setup directly contravenes the rule, yet Apple's developer documentation attempts to mitigate the issue by referencing explicit parental consent, a loophole that has not been officially resolved.

Furthermore, major model vendors like Google, OpenAI, and Anthropic have also established strict guidelines for serving minors, which include data retention, verification, and audit rights.

The research highlighted how local processing dramatically alters the legal landscape. When the audio never leaves the device, there's no collection or third-party involvement, removing the possibility of rule 1.3 violations. Consequently, vendors' terms and conditions are irrelevant, as there is no third-party in the equation. The local pipeline has evolved significantly over the past few years, with WhisperKit's Whisper Large v3 Turbo running on the Neural Engine achieving near-cloud-level accuracy at 2.2% word error.

This was measured on a MacBook Pro with an M3 Max chip, but the caveat remains that performance may vary on other devices.

Expanding on the performance of models designed for children, a benchmark table revealed that even the smallest model, Whisper tiny.en, which weighs just 27.1 million parameters, achieved a word error rate of 12.66 against Whisper's tiny.en baseline of 12.81. This discrepancy becomes more pronounced as the material becomes harder for the model to process.

The study also noted that while local processing solves the legal compliance issue, it comes at the expense of accuracy. The models that fit seamlessly on a device with minimal effort often fail to deliver accurate results when handling spoken words by children.

Despite these challenges, the fine-tuning of models on child-specific datasets has shown promising results. Whisper-tiny, for instance, experienced a significant drop in word error rate from 40.1% to 2.7% after fine-tuning on the OGI Kids corpus. Similarly, Base models saw improvements from 38.0% to 2.0%. This indicates that the quality of the data used to train the models plays a more critical role than the sheer size of the model.

The research concludes that while local processing may solve legal issues, the quality of the model, as determined by the data it has been trained on, is of paramount importance for accurate voice recognition in children.