TL notes
The article originally spelled the guy's name as Kostas but it seems he prefers Costas in English.
The article originally spelled døv with a lowercase d; I translated this as uppercase Deaf rather than lowercase deaf because the article is specifically about signing Deaf people. A few people do make the lowercase-uppercase d/Døv distinction in Norwegian, but this practice isn't nearly as common in Norwegian as in English, just because of Norwegian's regular rules for capitalization.
I didn't include any of the pictures nor their captions. There's also an apparent typo in the original version of the article, "Den bidra" ("It contribute") instead of "Den bidrar" ("It contributes"), but I'm not translating things word for word, anyways. I merged a lot of the paragraphs, too.
The article was originally a lot more repetitive, too, but I got rid of some of the fluff in the translation.
Finally, because this article uses technical terms related to machine learning, which isn't something I'm super knowledgeable about, you may want to compare with a machine translation. Obviously I think this is a perfectly serviceable translation, but for all my translations I want people to be aware that I am a fallible and partial human being.
If they succeed, it will improve communication between the hearing and Deaf.
THIS ARTICLE IS PRODUCED AND FINANCED BY SINTEF.^[One of 80+ owners of Forskning.no; their communication staff deliver content to Forskning.no. This article is marked to clearly separate it from independent content.]
The 430 million Deaf or hard-of-hearing people in the world often experience difficulties when communicating with the hearing. Researchers have now investigated a solution which uses machine learning to instantaneously translate sign language into written text or machine-generated speech. This solution can make dialog simpler, contributing to a greater inclusion of Deaf and hard-of-hearing people in society.
AI proposed collaboration with researchers
It was Tone Ervik and Pål Rudshavn of Statped [State Special Pedagogical Service] in Trondheim who had the idea of using AI to translate sign language to text or machine-generated speech. They saw that AI was continuously improving at translating speech to text, so maybe AI with new language models could also be used to translate sign language?
—"I asked ChatGPT about how we could get further with this idea. It suggested that we should contact Sintef, so that's what we did," Ervik says.
The researchers immediately jumped on the idea of building a tool both very useful for society, yet simultaneously technologically challenging to develop.
—"We saw this as an amazing opportunity. With how quickly AI is progressing, we wanted to use the technology for something that can actually make a meaningful difference in society," says Costas Boletsis, supported by his colleague Zia Uddin.
With support from the Dam Foundation [Stiftelsen Dam], they started the project AI-based Norwegian Sign Language Translator [KI-drevet Norsk Tegnspråkoversetter] in February of 2024.
Facts
Norwegian Sign Language was first recognized as a complete and independent language in 2009. There are 16,500 people who communicate using sign language in Norway. The WHO expects the number of people with hearing impairments to increase in the coming years. A technology which can read Norwegian Sign Language (NSL/NTS) and translate it to text or speech would reduce the communication gap between Deaf and hearing people.
Researchers from Sintef Digital are currently developing an AI-based sign language translator. The first part of the project had a budget of 400,000 NOK [~39K USD] after contributions from the Dam Foundation.
The researchers see this project as having three stages:
- Development of a machine learning (AI) based method for video analysis of sign language which may be used for Norwegian Sign Language as well as other sign languages.
- An initial prototype which can read NSL and transform it into text.
- The project will develop the basis for a system for real-time translation of Norwegian Sign Language to text.
US-based researchers have come a pretty long ways with their own tool, which can interpret sign language in real time with the help of machine learning. Norwegian Sign Language, however, is unique, and therefore a new model must be developed for and in our own country.
16,500 people communicate using sign language in Norway, according to the Norwegian Association of the Deaf.
Boletsis and Uddin decided to start with getting a computer to automatically recognize the Norwegian Sign Language signs for the numbers 0 to 10, following advice from Statped.
Facts
- The researchers used the tool MediaPipe from Google to extract important information from videos of Statped's sign language teachers producing the signs.
- MediaPipe is an open source framework by Google. It makes it easy to use machine learning on mobile phones, web browsers and built-in systems. It offers complete solutions for hand tracking, facial recognition and object identification among other tasks.
- After extracting the information, the researchers used Long Short-Term Memory (LSTM) to identify the signs. LSTM is a type of neural network that remembers information over time. It is often used in language and time series analysis because it can detect both short-term and long-term patterns.
- A neural network is a computer model inspired by the brain, using layers of "neurons" which learn patterns in data, in order to recognize pictures, language or numbers.
- The dataset included 1,059 short videos.
—"We focused on the numbers 0-10 because we had to start somewhere, as Norwegian Sign Language is different from other sign languages. It could've been any other 11 gestures," Zia Uddin says.
He explains that they can expand the system with supplementary analysis. The basic approach will remain the same, just at a larger scale with more complex algorithms.
Tested the system in real time
Through their own tests, the researchers have found that the system they have developed is showing good results. It has a test accuracy of 95%, and the researchers believe this shows that the solution can handle variations in style, speed and camera angle.
We're meeting the researchers one year after the start of their project. It is now finally time to test the AI-based system in real time.
Twelve signers have arrived at Statped's offices, and one after another they stand in front of the computer and produce the signs for the numbers 0-10. The computer program uses hand and mouth markers to distinguish between signs with identical handshapes, such as the signs for 3 and 8.
Although the model performed generally well during this demonstration, it still conflated a few signs. The researchers will use this information to make improvements.
—"The aim is to develop a learning app for real-time recognition and evaluation of NSL. Users will get an immediate translation using an avatar. This will help signers communicate with hearing people in settings like grocery stores, hairdressers, airports et cetera. The results of today's test indicate that this will be a very useful tool in the future," Costas Boletsis says.
The researchers say that further development should focus on expanding vocabulary, and that the system should be tested in different situations, such as different lighting, camera angles and speeds. The system should furthermore use more types of sensor data to get a better spacial awareness.
The goal is an app
—"When the scope of our project is so big, it's only natural that the work will take several years. Artificial intelligence will evolve in parallel at the same time. The core of this type of project is data. We need to develop a dataset, a corpus, where we have a lot of information and many videos for each and every sign in use," Zia Uddin says.
He explains that they can then begin using AI models at a large scale. They can train the models effectively and investigate if they can handle a much broader range of expressions than what they've been trained on thus far.
The researchers' dream is an app or software which could be installed on e.g. a mobile phone, and which can instantaneously translate central words and phrases of sign language.
—"Sign language is extremely important for Deaf and hard-of-hearing people. With the progress being made in AI, particularly in photo and video analysis, we believe we can make a tool which can make a big difference for many people," Zia Uddin says.
REFERENCES
Zia Uddin, Costas Boletsis og Pål Rudshavn: Real-Time Norwegian Sign Language Recognition Using MediaPipe and LSTM. Multimodal Technol. Interact, 2025. Doi.org/10.3390/mti9030023
The few points I'd bring up are: