What Does a 226% Improvement Smell Like?

Let me first express my thanks to Andrew Gelman of Columbia whose blog (see above) first brought this to my attention, so I can bring it to yours.

This is another of those ‘if it seems too good to be true it probably is’ research papers that I so love. The paper is titled ‘Overnight olfactory enrichment using an odorant diffuser improves memory and modifies the uncinate fasciculus in older adults’ and it was published in Frontiers in Neuroscience in July of 2023.

It reports on a study in which participants – all elderly, like me – were given scent diffusers to take home and use for two hours each night, starting when they went to bed – the diffusers automatically shut off two hours after they were started. The participants were given cognition tests and a functional MRI (that’s where the uncinate fasciculus bit in the title comes from) before they started the experiment and again six months later, after they had used the diffusers for that period of time.

The ‘treatment group’ got a set of 7 genuine essential oils to use in their diffusers, while the control group got ‘de minimis amounts of odorant’ according to the researchers. (Do you suppose those in the Control Group noticed that their diffusers produced no scent? But I digress.)

In the end there were a total of 43 participants in the two groups, and the headline result of this research was, quoting the paper,

“ A statistically significant 226% improvement was observed in the enriched group compared to the control group on the Rey Auditory Verbal Learning Test….”

Pretty impressive, eh? Six months of smelling essential oils for two hours/night at bedtime, that’s all it took to get that huge improvement in Auditory Verbal Learning.

Anyone smell anything?

Here are some facts about this ‘controlled’ experiment. First, 43 participants? A bit more than 20 in each group? That is what statisticians call a small sample. But wait, there’s more. If you look at the flow chart of how they recruited and screened participants for this study, you find that 132 subjects passed the initial screening. Of these, only 68 were included in the Control and Treatment groups that were used in the statistical analysis of the results, and of those 68, 25 dropped out during the study. That leaves the 43 whose results are reported on, of the 132 who passed the screen.

Smell anything yet? Why did those 25 people drop out? That’s 36% of the 68 whose results were analyzed and reported. What does that drop out rate imply for the credibility of the results? People don’t drop out randomly, they do it for reasons.

And, as one of the readers of the Gelman blog pointed out, that 226% improvement claim comes from the control group scoring on average 0.73 points worse post-treatment than pre-treatment on a particular test, while the treatment group scored 0.92 points better on average. So you have a difference of 1.65 points in the two groups’ average ‘improvement’ on the test, and 1.65 is 2.26 times 0.73.

Interesting arithmetic. I think Gelman’s reader is right, as that 226% number doesn’t come up anywhere else in the paper. However, note that 1.65 being 2.26 times 0.73 is not the same as 1.65 being a 226% ‘improvement’ over 0.73. The latter would mean that it was more than three times greater, and it is not. Neuroscientists don’t do a lot of basic arithmetic, I guess. That detail aside, just looking at the difference in average scores for the two groups – what does a ‘point’ mean in this context, anyway? Is it big? How ‘big’ is a 1.65 point improvement on this test? What does that actually translate into, memory-wise? The researchers do not say.

One last thing. At the end of the paper there is a section called Funding, under which is written ‘This research was sponsored by Procter and Gamble.’

There, now you smell it, eh?

I know nothing about the journal Frontiers in Neuroscience, but if their editors did not stop and wonder about that 226% claim, well, I don’t think I’ll subscribe. Actually, it’s an open access journal so I can read it for free, as I did this smelly article. But I won’t. Read the journal, I mean. Unless maybe I’m looking for something else to write about here.