Reading emotions from brain sensors: one step closer

Imagine a wearable device that monitors your emotional state, using brain sensor technology to detect early signs of stress, anxiety or low mood before you’ve even noticed them yourself. That kind of technology could help people manage conditions like depression or burnout, or alert healthcare providers when someone needs support.

New Auckland Bioengineering Institute graduate Alireza Farrokhi Nia’s PhD has taken a step closer to that goal. He has developed a system that avoids facial expression or tone of voice (signals that people can easily hide), instead going straight to the source: the brain.

Unlike other research in this space, Reza’s work combined two types of brain-monitoring technology: EEG, which captures electrical activity, and fNIRS, which measures changes in blood oxygen levels.

He recorded volunteers’ brain signals while they listened to music, everyday sounds (parents fighting, glass shattering, a running river), and watched videos (including funny cat compilations, the joyful moment a deaf baby hears her sister’s voice for the first time, news footage of famine, and neutral clips like a barbershop haircut) designed to trigger a full range of emotional responses.

Reza wasn't trying to measure whether someone felt ‘happy’ or ‘sad’ in any simple sense, he says – human emotion is too complex for that.

Instead, he measured emotion along broad dimensions based on how pleasant the feeling was and how intense.

Collecting the data was a huge undertaking. More than 100 people took part, each spending around two hours in the lab. Every detail had to be controlled   from lighting to the words spoken to participants, even down to Reza wearing the same cologne for months so nothing unintentionally influenced the subjects’ mood. 

Once the recordings were complete, the real challenge began: building AI   models that could interpret the brain signals. Harder still, he needed models that worked across different people, not just one individual. It’s a problem that continues to challenge this field of research, Reza says – one person’s emotional response doesn’t necessarily look the same in the brain as another’s.

“We can watch Mr. Bean together, and we can both be happy, but the way that our brain works could be different.”

A piece of the puzzle

Reza’s research showed combining two types of brain sensors gave better results than using either alone.

There’s plenty more work to be done before machines are going to be helping people deal with depression or anxiety, but sitting at the intersection of neuroscience, engineering, and artificial intelligence, his PhD contributes a piece of the puzzle.

“As wearable technology becomes smaller and more capable, the ability to passively monitor brain signals in everyday life is becoming increasingly realistic,” he says.

“My research showed how emotional states can be decoded from brain data and how the challenge of building models that generalise across different people can be tackled. This is essential for any real-world application of this technology.

 “It’s still an open question, but at least now we know which directions might work.”

New projects

Now a research fellow at the Auckland Bioengineering Institute, Reza's post-PhD focus is on translational research – the process of turning scientific discoveries into real solutions for real people.

It’s a concept he was introduced to while taking part in the University’s entrepreneurship programme – the Velocity Challenge.

"This is not about money. This is about how you can make this reach the people who are in need."

Currently, Reza is focused on two interconnected conditions — diabetes and cognitive decline — working alongside clinicians and specialists at Middlemore Hospital and Alzheimer's New Zealand in Gisborne.

His goal is to develop accessible tools which can monitor these conditions through physiological and behavioural signals, with the aim of slowing or preventing their progression. 

As with his PhD, Reza’s new work brings together people and machines – two things that captivated him as a child growing up in Iran and spending time at his grandparent’s farm.

"Growing up around my grandparents' farm, I was always surrounded by people and curious about them — how they think, how they communicate, what makes them different from one another. And honestly, I still am."

Watching his uncle fix any machine or tool on the farm got him interested in how things worked and ultimately directed him to engineering – electrical at first and then a shift to biomedical engineering. 

Auckland Bioengineering Institute’s Professor Mark Billinghurst. one of Reza’s PhD supervisors, says Reza was largely self-directed.

“There were times I wouldn't see him for weeks and then he would show me some exciting new results, or a draft of a strong paper he'd been working on.

“He also worked n a number of side projects while completing his PhD, connecting with a range of people both inside and outside the Auckland Bioengineering Institute.”

“He was an outstanding student and I’m keen to see where his research goes in the future.”

Article written by Sally Gallaugher