Robotic sign language interpreter for Deaf Māori in development

Robotics and language researchers at the University of Auckland are building a robotic interpreter for Turi Māori - Māori who are deaf.

The project, led by Dr Ho Seok Ahn and Dr Tauwehe Tamati, uses a sign language underpinned by Māori worldview, which can be interpreted into text and spoken language.

The signs have been developed in partnership with Turi Māori, alongside whānau, Iwi and Hapū.

"I knew a methodology would be required to pull together all the threads involved in creating a new language," says Tamati, who has been working on the project alongside Iwi and Hapū for half a decade. 

Tamati, who is bilingual in Te Reo Māori and English, used both languages and whānau-based home sign with her father. He was hard of hearing for most of his life and became profoundly deaf in his final 20 years.

"Our dad grew up in Te Ao Māori. He knew he was Māori and he was a hard worker at our pā. The cultural, social, and collective life on the marae continually reinforced his identity and sense of belonging," says Tamati.

Over time, however, he stopped going to the pā and didn’t like relatives coming around.

"We’d be making up excuses because he was deaf. We couldn’t help him get through that. So when my engineering colleagues approached me about the project, I just thought of him and said, I’ll do this for dad."

Māori culture is rich in waiata, haka, tauparapara (chants) and pūrākau: ancestral narratives that hold stories, values and concepts passed down through generations.

In her conversations with Iwi and Hapū, Tamati heard concerns about deaf whānau becoming disconnected from the mātauranga (knowledge) of Te Ao Māori and staying insular with their nuclear families, because they didn’t have the opportunity to engage with that knowledge and their wider whānau.

Tamati recounts that Iwi and Hapū were saying the same thing about their Turi Māori relatives of their respective Iwi and Hapū.

"We want them back, we want them to know who they are. Our stories tell us great things our ancestors have done, things to be strengthened and encouraged by.

"It was never about language for me, even though my PhD is in linguistics," says Tamati.

"It was always about: What does a language do to support your identity if you can’t hear it?"

This thinking led Tamati to develop the research methodology for the project: "Whakamana Mātauranga", which means giving expression and voice to Mātauranga Māori knowledge.
 

The participation and inclusion focus of the Whakamana Mātauranga methodology brought together trilingual interpreters, Turi Māori and their extended whānau to co-design the signs for the selected ancestral narrative.

Once the signs were developed, the next stage was teaching the technology how they look and feel when expressed by people.

"Facial expressions, the hands, the position of the head – it's equivalent to tone, pitch and volume in speech," says Tamati.

Māori sign-language role models were photographed and filmed so researchers could capture the different elements of communication, including emotions, lip movements, posture and mannerisms.

Cameras and motion sensors captured the movements and expressions, while computer vision and lip-reading technology provided additional information.

An advanced AI system called a Weighted Multimodal Network combined these different streams of information to help the system understand the meaning and nuance of the signs.

That information helped Dr Ho Seok Ahn and his team develop the technology: robots that can recognise Māori sign language to communicate and teach it through sign, text and spoken language.

The robots are now fully built and functioning within the research lab, including humanoid robots around the height of an average adult, one designed for primary school-aged children and a smaller robot intended for use in kōhanga reo and early childhood centres.

The technology is intended to provide an additional resource for Turi Māori, rather than replace human interpreters.

The project began half a decade ago as a seed project through the Science for Technological Innovation National Science Challenge, with additional support from the Vision Mātauranga Fund. In 2024, the team received a $1 million Smart Ideas grant through the Government’s Endeavour Fund to scale the technology over three years.

The ultimate goal of the research is to bridge communication barriers for Turi Māori and create highly advanced, exportable digital systems that preserve indigenous concepts in non-verbal interaction.

For Tamati, that means bringing Turi Māori back home to Te Ao Māori.

"It’s about supporting them to feel a sense of belonging. It’s about reconnection, bringing them home so every baby born deaf in our community, in our Iwi, in our Hapū, feels embraced."