Bringing touch to the virtual world

Interacting with each other digitally is pretty much an everyday experience. We see and hear others through our screens; put on a virtual or augmented reality headset and the visual and auditory experience can be near-real.

But touch? There’s nothing. I can see the shirt you are wearing through my VR goggles, but I can’t feel the silky fabric; can hear the sound of the sea from the beach you are standing on, but not feel the sand running through your fingers.

In the future that might change, if Moritz Messerschmidt and collaborators around the world have anything to do with it.

Messerschmidt did his masters degree in his native Germany, moved to the Auckland Bioengineering Institute for his PhD, and has now joined the Augmented Human Lab at the University of Singapore.

“New types of human-computer interfaces are changing the way we communicate with each other,” Messerschmidt says. “We’re going to get more used to not just interacting via desktop screens, but through head-mounted displays.”

Already the visual part of those VR or AR interactions – how real the world we see is – is pretty good, he says, and the sound quality is even better.

“The missing part is the touch dimension, and that’s what I’m focusing on. We will not only communicate, but we will hopefully be able to touch things in the virtual world, and it will feel real. We will be able to share and collaborate remotely via these systems, and it will feel incredibly real.”

It’s not hard to imagine real-world applications for such systems – remote surgery, for example, or a technician being able to install or repair equipment thousands of kilometres away. 

But there are some serious practical obstacles to providing a sense of touch in a mixed or augmented reality situation. Your eyes and ears are fixed in one position on your body, but touch – that could be anywhere.

Even if you focus on the hands, you have to find a way of stimulating the skin and providing natural feelings and sensations over a wide area, Messerschmidt says.

If you attach a touch feedback device to the tip of someone’s finger, that’s going to hinder the actual feeling they get from that finger. But if you attach it to the top side of your finger, where it won’t be in the way, will that sensation feel real?

Moreover, because with augmented reality, digital information is being overlaid onto the real world, any digital touch feedback has to be provided unobtrusively, so it doesn't get in the way of the real world environment.

“It’s pretty challenging to provide haptic feedback on the body and allow the user to move around. It’s also hard to give a new type of haptic feedback using mechanical stimuli that isn’t just vibration but perhaps stretching or dragging your skin to enable more natural feeling sensation."

Messerschmidt was involved in a number of augmented touch-related projects during his PhD and afterwards, says his supervisor Associate Professor Suranga Nanayakkara, now based at the National University of Singapore and founder of its Augmented Human Lab.

These often involved researchers in different parts of the world and from different disciplines, including medicine, psychology, design, and the startup business space.

“The value of my group is learning to work with people from different fields; building a portfolio as a group," Nanayakkara says. "Each researcher is like an individual tower in a big building.” 

Messerschmidt says his experience at the Auckland Bioengineering Institute was an incredible introduction to working in a collaborative environment.

“We had people from Spain, Colombia, the US, Australia, from India, Sri Lanka, China, and we bonded quite quickly. I started working with a colleague on a project and felt instantly part of the map. 

“I met so many people who are now distributed across the world. We meet at conferences and also through Suranga, who has a huge network, so there’s a lot of collaboration going on.”

Nanayakkara is still an honorary professor at the Auckland Bioengineering Institute and returns to New Zealand regularly, as well as joining the institute online for research meetings; he regrets other commitments means he won’t be able to watch Messerschmidt graduate.

“Moritz was always coming up with new ideas and was a fantastic person to work with in the lab. His PhD work, ‘Designing expressive on-body haptic feedback for mixed reality environments with unobtrusive technology', was driven by his amazing excitement for exploring new ways to create touch experiences,” he says. 

“Moritz had a knack for taking complicated ideas and turning them into really interesting projects. He's shared his brilliant findings in top research spots like CHI [the Conference on Human Factors in Computing Systems] and [academic journal] ToCHI. 

“I am so proud of everything he's achieved and can't wait to see what he does next!”