How to navigate media polluted with bad science

By Rebekah White

‘Why sniffing dark chocolate will help you lift heavier weights at the gym’, was the headline of a story on The Independent in July.

The article was just one among many on news websites around the world that covered a newly published exercise-science study, which connected the smell of chocolate with more reps.

Trouble is, the study was a shoddy one: it only involved around 20 people, all of them men, and didn’t compensate for the placebo effect, as Australian exercise scientist Hunter Bennett explains.

The participants knew when they were smelling chocolate – it’s impossible to disguise – and might have tried harder as a result.

It’s a silly example, but it illustrates a darker problem.

If you want a scientific study to back up your point of view, you can probably find one among the three-million-odd peer-reviewed studies published around the world so far.

But you won’t be able to guarantee it’s any good.

Bad ‘science’ keeps making the news and doing the rounds on social media.

Some of it’s funny, like the chocolate study.

Some of it’s actively harmful, like a spurious study claiming, with no evidence, that different gender identities are more prone to violence (which epidemiologist Gideon Meyerowitz-Katz debunks here).

How do we protect ourselves against false information – especially when it comes from legitimate-seeming sources, or when it all looks scientific?

How do we know what to trust?

Be on guard: we’re surrounded by polluted information

Think of the landscape of information around you as contaminated, says Dame Juliet Gerrard, a professor in the Faculty of Science at the University of Auckland.

While in office as the Prime Minister’s Chief Science Adviser, Gerrard watched misinformation – which she prefers to call “polluted information” – spread rapidly on a range of topics: 5G phone connectivity, water fluoridation, the cannabis referendum, and the Covid-19 virus and vaccines.

Good-quality information isn’t readily available to everyone, Gerrard discovered.

“The information some people get is very different to the information we get as scientists,” she says.

“We don’t make that any more simple by locking lots of the good studies up behind a paywall.

"So a lot of this easy information is incorrect, and the peer-reviewed, rigorously assessed information is very hard to access.”

Often, accessing better-quality information involves more than getting behind a paywall and understanding scientific language: it requires being able to understand what separates a good-quality scientific study from a bad-quality one.

But there’s something even more important that people can do when they’re assessing the facts in front of them – and that’s to separate values from evidence.

When values and facts get mixed up 

The topic of water fluoridation is a good example of a bad debate, says Gerrard.

People end up talking past each other because they’re arguing about values, not evidence.

“It’s completely legitimate to say, ‘I don’t want anything added to my water’,” she says.

“That’s a values position and there’s nothing wrong with it.

"So I think people tend to argue back with science when the person who’s objecting is not actually talking about the science – they’re talking about something completely different, which is their right to control what’s in their drinking supply and whether that should be a local decision or a national decision.”

Whether to add fluoride to the public water supply: that’s a different question from whether it’s safe to do so, or whether it helps to prevent tooth decay.

There’s evidence around the second and third questions: yes, it’s safe in small amounts, and yes, populations that drink fluoridated water have fewer cavities.

The first question is one of values.

Addressing these separately is crucial, says Gerrard.

“So often people are having a values conversation through a science debate, which is not helpful.”

Another feature of the fluoridation debate is whether or not a study is relevant to the question at hand.

Opponents to fluoridation often cite studies showing a connection between fluoride and brain damage.

But these studies involve extremely high levels of fluoride – amounts no one could consume from drinking New Zealand tap water.

When there isn’t enough science – or the science isn’t conclusive

Before the cannabis referendum of 2020, Gerrard was tasked with assembling information to help members of the public decide how to vote.

But there was so little research on cannabis that it was hard to find scientific consensus on many issues.

“There’s not much data out there because cannabis is illegal,” she says.

“Nobody fills in questionnaires saying how many joints they smoked last week.”

Instead, what Gerrard had to work with was a limited set of studies with different, often contradictory, results. It meant that people for and against cannabis legalisation each had studies to cite in support of their position.

“People tend to clutch onto studies that show harm or show no harm, because it’s supporting their values position,” says Gerrard.

“So, what we did instead was say what might happen if you vote yes and what might happen if you vote no.”

That allowed her team to arrange the best-available evidence according to people’s values.

People could then look at potential outcomes, and how certain or uncertain they were, based on those values.

“It was like, ‘If you’re worried about social justice, look here. If you’re worried about biomedical impact on young men, look here'.

"And so, you could say, what’s more important to me is social justice: let me have a look what happens if I vote yes or no.”

Understanding how evidence is created

You don’t need to know a lot about science to spot a bad study: a little understanding goes a long way.

One principle that makes it easy to discern which studies to pay attention to is: a single study on its own doesn’t mean much.

Scientific knowledge is based on many people finding out the same thing over and over again.

Lone voices of dissent don’t count as evidence – yet. Not until other scientists try the same thing and get similar results. (Those dissenting voices are, instead, more like signposts to other researchers: ‘Try this!’)

“Why science moves forward is consensus,” says Gerrard.

“It’s not the lone ‘genius’.”

During Covid, many peddlers of misinformation claimed they were lone truth-tellers being silenced by the scientific establishment.

In fact, other scientists had already tested out their claims and found that they didn’t work.

The Intergovernmental Panel on Climate Change is an example of consensus-based science: hundreds of scientists agreeing with each other, adding in new bits of knowledge to each report.

“They all sit in those rooms, and it’s not the most exciting work, and they nudge it forward and nudge it forward and nudge it forward,” says Gerrard.

“But consensus is what drives things.”

We could teach more about science – and misinformation

Research from the University of Cambridge shows that teaching people about some of the elements common to misinformation and conspiracy theories helps people develop scepticism towards potential manipulation and junk claiming to be science.

For instance, it would be possible to bolster critical thinking in science at school, says Stuart McNaughton, a professor of education at the University of Auckland and the co-author, with Gerrard and others, of a report on misinformation and young people.

Overall, New Zealand students don’t excel at science, according to international rankings.

“We have had relatively low levels of science achievement and progress between year four and year eight in national assessments,” says McNaughton.

“So, the basis for science knowledge has not been very strong.”

Developing media and information skills at a foundational level could help address the spread of misinformation.

“The polluted information we talk about in that report, everything from environmental science issues through to health, you name it – we need kids who are able to do the critical thinking about that information.”

• The world is facing unprecedented environmental challenges. Planetary Solutions, an initiative of the Sustainability Hub at Waipapa Taumata Rau, University of Auckland, and Newsroom, explores these issues – and the practical ways we can all be part of the solution.