Putting sign language AI into users’ hands

Google DeepMind's SL2T model lets users turn American Sign Language into English text in Gboard and Live Transcribe, initially on Pixel 11. Its announcement describes signing messages or conversational replies instead of typing them.

This requires translation because sign languages have their own grammar and convey meaning through coordinated body and facial movements. The phone first converts those movements into geometric landmarks; a server translates the coordinates into text. The raw video can be discarded on the device, but translation still involves server processing. Training across more than 50 sign languages should not be confused with the narrower ASL-to-English product launch.

“Yet this technological revolution has not reached the world’s more than 200 sign languages — and the estimated 70 million Deaf and hard of hearing people who use them.”

Google DeepMind Sign Language Team

“The model is trained on over 100,000 hours of data across more than 50 sign languages — with roughly a quarter of the data in ASL. Training jointly on diverse languages, dialects, and proficiency levels causes the model to learn shared underlying structures, outperforming single-language models in our experiments.”

Google DeepMind Sign Language Team

“SL2T achieves a remarkable zero-shot score of 70 BLEURT, which is significantly higher than any previously reported score.”

Google DeepMind Sign Language Team

“To protect user privacy, SL2T sees sign language as a sequence of pose landmark locations rather than a raw camera feed.”

Google DeepMind Sign Language Team

“we worked hard on practical issues like minimizing streaming latency, preventing hallucination on non-signing inputs, ensuring fairness for the 10% of signers who are left-handed, and improving performance for one-handed signing, which is used while holding a smartphone in the other hand.”

Google DeepMind Sign Language Team