Sony Patents In-Game Sign Language Translator

Sony is pioneering accessibility for deaf gamers with a newly patented in-game sign language translator. This innovative technology aims to bridge communication gaps between players using different sign languages.

Sony Patents ASL to JSL Translator for Video Games

Utilizing VR and Cloud Gaming for Seamless Translation

Sony's patent, "TRANSLATION OF SIGN LANGUAGE IN A VIRTUAL ENVIRONMENT," details a real-time sign language translation system for video games. Imagine playing with someone who uses American Sign Language (ASL), while you communicate in Japanese Sign Language (JSL)—this patent makes it possible. The system facilitates in-game conversations by translating sign language gestures in real time. This involves a three-step process: first, translating sign gestures into text; second, converting the text to the target language; and finally, translating the translated text back into the target sign language for display.

As Sony explains in the patent: "Implementations of the present disclosure relate to methods and systems for capturing sign language of one user (e.g., Japanese), and translating the sign language to another user (e.g., English). Because sign languages vary depending on geographical origins, sign language is not universal. This provides a need for appropriately capturing the sign language of one user, understanding the native language, and generating new sign language as output for another user in their native sign language."

Sony envisions this system working with VR headsets or head-mounted displays (HMDs). These HMDs would connect to a user's device (PC, game console, etc.), providing an immersive gaming experience. The patent also proposes a network-based solution, with a game server facilitating communication between players. This server maintains the game's state and synchronizes it across user devices.

Furthermore, Sony suggests integrating this technology with cloud gaming, where the game server would handle rendering and video streaming, enabling seamless cross-platform interaction. This setup allows players to share and interact within the same virtual environment over a shared network or server, regardless of their location or the sign language they use.