Putting sign language AI into users’ hands

Google has unveiled a groundbreaking model, sign-language-to-text (SL2T), designed to bring sign language AI to everyday consumer devices. This pioneering technology enables Deaf and hard of hearing users to converse with their phones using their native sign languages, marking a significant leap forward in accessibility. With SL2T, powered by AI, users can now sign to type messages, search the web, and even ask voice assistants like Gemini to execute tasks.

The model's strength lies in its ability to understand the complex movements that convey meaning in sign languages, which differ fundamentally from spoken languages. Unlike sign language gloves that merely translate hand gestures into English, SL2T comprehensively processes the full-body movements involved in sign languages, including those of the arms, torso, head, and face.

To build SL2T, Google combined a culturally sensitive approach with extensive data scaling, training the model on over 100,000 hours of sign language data from more than 50 different languages. This inclusive training approach resulted in a model that outperforms single-language alternatives in translation quality. A standout feature of SL2T is its commitment to privacy, processing sign language as a sequence of pose landmark locations rather than a raw video feed.

This allows the technology to immediately discard the original video, ensuring user data remains secure. Crucially, SL2T translates directly from these geometric coordinates into text, bypassing the need for intermediate annotations known as "glosses." This streamlined process enhances translation accuracy and scalability. In rigorous benchmark tests, SL2T has demonstrated remarkable performance, achieving a zero-shot score of 70 on the BLEURT scale, outperforming previous attempts in sign language translation.

What sets SL2T apart is its focus on usability and community involvement. Google collaborated closely with the Deaf community throughout the development process, from initial conceptualization to impact assessment. A dedicated AI Sign Language Advisory Committee, composed of global Deaf organizations and experts, ensured that the technology was built with the needs and perspectives of Deaf users in mind.

This collaborative approach, combined with cutting-edge technology, makes SL2T a landmark achievement in making sign language AI accessible to the masses.