Now everyone can read subtitles, whenever they want

A new technology developed by researchers at the Auckland Bioengineering Institute allows the hearing-impaired to watch a movie at whatever cinema they like, and never miss a line of dialogue.

Marco Schneider (left) and Tony Tse wearing the Vivify headset, which allows the hearing-impaired to watch a movie, wherever and whenever they like.

The device, called the Vivify headset, uses augmented reality to project subtitles below the cinema screen. A commercial trial of the headset is currently being run through Event cinemas in Newmarket, Auckland.

The system was developed by Marco Schneider and Tony Tse when they were both studying their PhD at the Institute.

Tony was investigating an underwater hearing aid for divers inspired by the mechanism of fish hearing, and Marco was developing a computational framework for analysing joint health from medical imaging data. (Marco is now a research fellow at the Institute, and Tony is completing his PhD.)

The idea for the device came out of a late-night conversation, when they were planning to watch a movie and Tony, who is hearing impaired, expressed his wish that more movies came with subtitles.

Subtitle data is included in most movies, but cinemas in New Zealand rarely use it, so cinematic options for the hearing-impaired are extremely limited – Event cinemas in Auckland only shows subtitles on Mondays, at certain sessions.

Their three-in-the-morning idea went on to win the Velocity Challenge at the University of Auckland in 2015 worth $1,000 which they used to develop the first prototype.

The most challenging aspect of the project in the beginning stages was finding someone in the cinema industry to talk to, to find out if their idea was viable and if the industry would support such technology being used in cinemas. Eventually they managed to present their idea to Event cinemas, which thought it a good one and encouraged them to develop the headsets.

The pair established the start-up company, Vivify, in 2017.

The initial prototype was “was pretty bulky”, says Marco, but has had several iterations since, with changes being made in response to feedback from people participating in pilot trials in which participants watched a movie wearing the headsets.

Changes included finding a way to make the subtitles stay in place when people turned their head to, for example, talk to their neighbor. “The subtitles would follow you, which was annoying and distracting,” says Tony. “So we’ve been trying to make it as seamless as possible.”

While their current prototype involved customising off-the-shelf headsets, they plan to build headsets from scratch, make them more aesthetically pleasing and lighter.

The current prototype model is connected to an iPhone that comes with the headset, but if they can dispense with the phone it make the headset much lighter. They are also working on adding other languages and expect to have Chinese subtitles working by the end of June.

The current trial may reveal the need for further refinements but there is already good evidence that the headsets will be very much in demand.

“We’ve received a lot of requests from across New Zealand asking us to run trials in cinemas in their cities,” says Marco. “It’s very encouraging – people clearly want this technology.”

If you would like to use the device when you next visit, you can book a pair of headsets through the Vivify website. The pair also plan to run free trials over the next few months which, for interested participants, will include a free movie admission and the free use of the headsets. Find out more about Vivify Free Trials.

Margo White I Media adviser

DDI 09 923 5504
Mob 021 926 408


Want to help us make a difference?

There are many ways you can make a difference and support the Auckland Bioengineering Institute - you can make a donation or bequest, or even subscribe to our newsletter.