I have posted more than a few articles about bad research in the social and medical sciences here. These have generally focused on research that involved analyzing data to understand things about human behavior and/or illness, but where the research is badly done, including, in some cases, the relevant data being falsified.

If the research in the now-retracted paper (written about here) on getting people to more honestly report the mileage they drove their cars had been carefully done, it might be of some use to somebody. It wasn’t Nobel-prize stuff (I do hope) but insurance companies would have some reason to try having their clients sign an honesty pledge at the top of their forms rather than the bottom, and other types of organizations needing info from their ‘stakeholders’ might also have followed suit.

It might even be that putting the pledge at the top has the impact the researchers claimed it did; however, their research provided no evidence that it did, and that’s the problem. So, that is science, or, if you prefer, Science. Science doesn’t prove things, it looks for evidence that the world works a certain way, and no conscientious scientist thinks they have the world figured out after one experiment or data analysis. That analysis needs to be replicated more than once before anyone has any faith in it – or should. One of my early posts here was about a general belief among researchers – mostly economists – that careful research had established that there is no such thing as a ‘hot hand’ in shooting baskets, you may recall. It took over 30 years for another pair of researchers to figure out that all the evidence gathered about it had involved a statistical miscalculation, and that correcting it implied that all that evidence actually suggested that a hot hand did in fact exist.

Science, then. Data, experiment or observation, analysis, resulting in evidence for or against some clear hypothesis about the world. And, the understanding that mistakes get made all the time, and the world is full of randomness, so don’t be too certain about anything.

However, a lot of what gets published in academic journals looks nothing like that. It’s only connection to science is that it results in papers written by academics being published in academic journals. It does not, however, involve any of data, evidence or statistical analysis. It ‘says things’. In complex and hard to understand language, to be sure, but still just ‘saying things’.

This was brought to mind by an official email I got last week from NEST, the Network for Economic and Social Trends, about a talk they were sponsoring, which the email invited me to attend. The academic was one Dr. Farah Mihlar, who (I found her web page) is a Senior Lecturer at Oxford Brookes University, UK in Human Rights Policy and Practice. She is apparently in the School of Architecture, which is just….well, bizarre. What her work has to do with architecture I cannot possibly imagine.

Anyway, the title of her talk was ‘Compounded Suffering in Post-War’, and the email included an ‘abstract of her presentation’, which I include in total here:

“Suffering is strongly associated with war but not its aftermath. In spite of its ambiguity, the post-war represents a transformation from humanitarian crisis to development and peacebuilding. Consequentially, references to suffering are replaced by political, economic, social ‘challenges’ and ‘issues’ that fail to accurately and fully represent the experience of most war survivors, especially women, who often make up a larger percentage of the population. Responding to this gap, this research conceptualises compounded suffering as the complex amalgamation of multiple forms of suffering: mental, physical and emotional; grief, pain, agony, torture, hurt, trauma, anxiety, stress, difficulty, produced by the unique conditions of post-war, which in coming together, aggravates and intensifies. The post-war context produces a typology of suffering that is distinct in form and extent fundamentally because it is shaped through an acute, real, and continuing threat to human life and security, precluded by the trauma and experience of extensive death, loss, and destruction. Suffering is compounded by women’s ‘entanglement’ with power specific to the post-war context, where structural and other forms of violence are pervasive and ideologies and methods of war remain amidst the pretext of change and transformation.”

Yes, it is very very long, not what I think of when I see the word ‘abstract’. But, I don’t want my readers to think I make shit up.

I will say a few things about this; enough, I hope, so you see what I mean about ‘just saying things’.

The first sentence is “Suffering is strongly associated with war but not its aftermath.”

This is a statement about the world, it would seem. Humans strongly associate suffering with war, but not its aftermath. I am down with the first clause, but then – ‘but not its aftermath’? Really? Who are these people who do not associate the aftermath of war with suffering? Or is it just that people don’t associate it ‘strongly’ enough?

I am about to do a presentation in my Geezer’s Study Group, and as it happens, my talk will include discussions of two wars: the First Indochina War and The Algerian War of Independence. None of the sources I’ve read on these two sad events suggest that there was no suffering after those wars ended. People lose contact with loved ones and don’t find out until the war ends what happened to them. Others had to flee from combat areas, and come back after the truce is signed to find destruction, disease and loss. Combatants must deal with living with their injuries after the war ends.

Who are these people that Professor Fihlar asserts do not realize this?

It is an assertion, it is ‘saying things’ and it is deeply important that particular thing be said, else there is no research project, and no talk.

Here is the key sentence in the abstract, explaining what the research is all about.

“Responding to this gap, this research conceptualises compounded suffering as the complex amalgamation of multiple forms of suffering: mental, physical and emotional; grief, pain, agony, torture, hurt, trauma, anxiety, stress, difficulty, produced by the unique conditions of post-war, which in coming together, aggravates and intensifies.”

Yes, a very long sentence. These sort of folks tend to write like this, in my experience. In this case, I would suggest that it’s very important to write out all of those different types of suffering. After all, the researcher’s starting point is that most humans don’t realize they even happen after  a war. Then, what this research does is it “conceptualises compounded suffering as….”

Now, how does one conceptualise something? The dictionary defines ‘conceptualise’ as an intransitive verb which means ‘to form an idea or principle in your mind’.

So, how do other researchers determine if the idea or principle formed in this researcher’s mind is, uh….useful? Well, not by gathering data, or doing an experiment, I am sure. I expect they talk about it. Or, possibly, argue about it, but ‘arguing’ is generally frowned on in the 21^st century academic world. That could be regarded as creating an unsafe space, and this is not tolerated.

I could go on about that big sentence above. People in this sort of business are not particularly good on grammar or logic. Consider that last clause

“…produced by the unique conditions of post-war, which in coming together, aggravates and intensifies.”

This refers back to that long list of types of suffering, but how can it be said these are produced by the unique conditions of post-war? Everything on that list occurs during war, too, so what does ‘unique’ mean here?

And then   ‘….which in coming together, aggravates and intensifies.”

What exactly is coming together here? Everything on that long list? And ‘aggravates and intensifies’ needs an object. Just what is being aggravated and intensified?

I found another paper by this same academic, titled ‘Coloniality and the inadequacy of localisation’, and here’s the first bit of the abstract of this paper:

“This article uses coloniality as an analytical framework to critique the concept of localisation. It argues that localisation is inadequate to respond to the asymmetrical power dynamic that it seeks to dislodge. Fundamentally, this is because localisation does not account for coloniality, which is the underlying logic of colonialism embedded within the humanitarian sector.”

So, more concepts. I don’t know what ‘coloniality’ or ‘localisation’ mean, or whether Mihlar or someone else ‘conceptualized’ them. A look through the actual paper shows lots of words, and the word ‘evidence’ appears twice in the paper’s 12 pages (yes, I checked). However, nothing one could call evidence, or even examples, is presented or cited. The paper ends with more than 20 references to other works, and I cannot swear there is no evidence in any of those. No, I didn’t check.

If people want to talk about things, and say ‘this thing here is bad’ that’s ok. If people want to discuss whether idea A is better than idea B, or what is the best way to conceptualize Trump’s approach to tariffs, go for it. I talk about things all the time over drinks/dinner with my friends, and there is little if any evidence ever brought to bear on what we say. We just don’t think of what we’re doing as being academic. Or research. Or science. Certainly, no one pays us to do it. Hell, I mostly ‘say things’ on this blog – but it ain’t science.

There is an even more basic way to see the distinction I am making between science and this ‘saying things’.

In the research on honesty, useless as it was, there was a scientific hypothesis being tested. That is, there was a claim made about how the world (more particularly, how the humans in that world) behaves. The claim of interest was:

“If you have humans sign a pledge of honesty at the start of filling out a form asking for unverified information, you will get more honest answers than if you have them sign the same pledge at the end of the form.’

The bad research actually done on this provided no evidence that this claim about the world is true, because it used fake data. The claim might still be true, and other researchers could devise new experiments designed to provide evidence. But, if you do the experiments right, there is the possibility that the evidence will not be found, or it might even be that you find evidence for the opposite- maybe signing at the bottom generates more honest answers.

Scientists (albeit not the data-fakers) put something on the line, and realize that their hypothesis, no matter how much they like it, may not in the end be supported by evidence.

The ‘saying things’ people face no such test. They say things, and other people might disagree, but those people are just saying different things.

I am not the first, by any means, to point out that the ‘thing-sayers’ don’t face this scientific test of the things they say. Thus, the thing-sayers have devised a very clever response. They say that the whole experiment-data-evidence approach is evil- white, male, hegemonic, colonial, etc. Saying this establishes – for them – the superiority of saying things. Ya gotta hand it to them.

When I was supervising undergrad theses in my last years of teaching, students were very fond of writing things like ‘as Flimblat has proven, X’, when all that meant was that that the student had read a paper by Flimblat in which he said ‘X’. Saying ain’t proving, I would correct them. They did not like that. In my experience, the people who write these ‘saying things’ papers are quite convinced of everything they write, and the students, sadly, take confidence for proof.

I leave it to you to decide which is worse for humanity. Academics following scientific procedures of experiment, observation and evidence, but doing it so badly or so fraudulently that it is meaningless or even deceitful,

or

academics just saying things.

Tough call from my perspective, but I do hope Professor Mihlar is not helping design buildings. Perhaps her job is just to keep them free from coloniality. Or to conceptualize how they might be.

Here is a quote from a comment posted on my favourite stats blog by someone who is not the author of said blog:

“The stand out example for me is nutrition science. A lot of the big, obvious effects have been picked through and now so much of it is simmering in noise with strong incentives to find various different things by getting significance. Alcohol/chocolate/coffee does, doesn’t, does, doesn’t, does, doesn’t cause increased mortality. I don’t know how we could expect that discipline to turn around. There is good work being done there here and there, but so much of it is GIGO.”

…by which is meant Garbage In Garbage Out. The writer is Joe B Bak-Coleman, who describes himself as a computational social scientist. You can check him out here.

This point gets made repeatedly on Gelman’s stats blog by him and many other contributors: much of social and medical and health science research is based on bad statistical analysis of data that is fundamentally ‘noisy’, in which case there is little reason, statistically or logically speaking, to believe that whatever ‘influence of X on Y’ the research purports to have found really exists.

[I hear you wondering – what is ‘noisy’ data? Check out the end of this post.]

To put it in more human terms, a researcher who has a big data set and is determined to find a very particular effect of some X on some Y lurking in that data set, can almost always find it, with attached statistical tests that are (too often) taken to imply that the effect is not just the result of random noise in the data.

And, if that X on Y effect can be portrayed as ‘cool’ or ‘unexpected’ or, sadly, politically convenient, the researcher can become a minor celebrity and get asked to give a TED talk or be interviewed by a podcaster or written up in the NYT.

A different phenomenon, equally important, is when public policy bureaucrats, like, say, the US Surgeon General, make statements about research on, say, the health effects of alcohol consumption, that indicate they did not even read the research they are citing. A topic for another post.

However, what I want to stress here is that the incentives in research are almost perfectly upside down. If a researcher plugs along and tries to develop credible careful evidence that the unsurprising ‘X influences Y’ statement is true, then you will likely never know about it. The stuff the public hears about are the sexy results, such as

1. signing an honesty pledge at the top of a form promotes honesty more than signing it at the bottom of the form.
2. cold showers can change your life and reduce body fat. (That one, with a sample size of 49, was eventually retracted).
3. people who sit at a wobbly work station rather than a stable one are more likely to see their personal relationships as being likely not to last.
4. student subjects who are told with great seriousness that they are putting a lucky golf ball sink more putts than those who are not told that. (An attempt to replicate this experiment by other researchers failed)
5. Adverse Infant Health Outcomes Increased After the 2016 U.S. Presidential Election Among Non-White U.S.-born and Foreign-born Mothers ….that’s actually the title of a 2024 paper published in Demography which, conveniently, also tells you what the authors claim to have ‘found’. This is an example of the ‘politically convenient’ sort of result, and was arrived at using a method called ‘regression discontinuity’, which is so prone to abuse even I could explain why. Maybe I’ll try one day….

And, just to round things off, here are some quotes from Retraction Watch, a fun (but depressing) site which I have mentioned before, that keeps an eye on research studies that get retracted because, well, for many reasons.

i. “Our list of retracted or withdrawn COVID-19 papers is up past 500.”

ii. “And have you seen our leaderboard of authors with the most retractions lately — or our list of top 10 most highly cited retracted papers?

iii. “The retraction of “a final batch” of 678 articles concludes Sage’s investigation into questionable peer review, citation manipulation, and other signs of paper mill activity at one of its journals, according to the publisher.” [For the curious, the journal in question is Journal of Intelligent and Fuzzy Systems. Yes, really.]

iv. “A Scientist Is Paid to Study Maple Syrup. He’s Also Paid to Promote It.”

My point is not a new one. The world is full of bullshit, and that includes that part of the world in which highly educated researchers are supposed to be trying to improve our understanding of the world.

Appendix (sort of) on Noisy Data

This refers to data that is only weakly connected to the actual thing it is supposed to be a measure of. Here’s a good example from a seminar I recently attended. The researcher was studying the impact of parental attention on how their kids did in school. Said researcher had data on a number of things, including, as one example, the time spent by parents reading to their pre-kindergarten children. However, this data came from a survey in which parents were asked to recall how many times in a typical week during the previous year they had read to their kids.

How good is any parent going to be at figuring out what a typical week is, or recalling how many times they read to their kid over the past year? The actual fact that the researcher needs, the impact of which on kids outcomes they are trying to study, is the time parents actually spend reading to their kids. The answers parents give to that survey question will be related to that actual fact, but only imperfectly. Parents will over and/or under state what they actually did when they answer that question. A lot of studies in social and health science rely on survey data, and it all tends to be noisy in this way. People have imperfect memories, and there is also variation in what people think a particular survey question is asking them. There are of course other things that can make data ‘noisy’ when it is not the result of some objective, regularized measurement process.

Suppose you asked 1,000 people how many miles they drove their car last year, recorded the results, then put them into a graphic (a ‘histogram’ in this case). What do you think it would look like?

Something like this:

A few people drive hardly at all, less then 250mi/year in this case, and a very few people drive more than 50,000 miles/year, but most are grouped around a median number between 2,000 and 10 thousand miles per year. This gives the very familiar ‘bell curve’ shape to the distribution of results, as you see above. This general shape is so ubiquitous that statisticians have come to call the continuous version of this a ‘normal’ distribution. (Strictly speaking, the normal distribution has tails at either end that go on forever at lower and lower levels. The distribution above is ‘truncated’ at either end, in stats lingo.)

Very many of the distributions that one gets from observations of occurrences that are influenced by many factors end up looking like this, whether the data is from medical studies (height , weight, systolic blood pressure), education (SAT scores), or nature (rabbits have 1-12 kits per litter, with an average of around 6.)

In 2012 Lisa L Shu, Nina Mazar, Francesca Gino, Max Bazerman and Daniel Ariely published a paper in The Proceedings of the National Academy of Sciences titled ‘Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end’ (a mouthful, I know) which reported on, among other things, the following field experiment.

Subjects who were clients of an auto insurance company were asked to fill out a form in which they were asked how many miles they drove their cars last year. All had to sign a statement saying they had answered honestly, but on some forms the statement of honesty was at the top of the form, and in the others it was at the bottom. The startling result was that the clients who signed at the beginning were claimed to report more honestly, because customers assigned to the ‘sign-at-the-top’ condition reported driving 2,400 more miles (10.3%) on average than those assigned to the ‘sign-at-the-bottom’ condition.

Such a simple thing can make people more honest, that’s truly remarkable, eh? Actually, it is mostly bullshit, and said paper was in fact retracted by the PNAS a year or two ago, due to the efforts by some scholars at a place with the wonderful name of ‘Data Colada’, who determined that the research was suspect. Really suspect.

The data reported in the paper regarding how clients answered this mileage question, put into a histogram like the one above, looked like this:

The ‘Car #1’ refers to the fact that some clients owned more than one car, and this is their report on the first (or only) car they owned. N=13,488 refers to the fact that there were 13,488 clients included in this study, all of whom of course reported on at least one car’s mileage. The histogram above is what statisticians would call a (approximately) uniform distribution. Data on miles driven in a year does not look like this, period. It looks like the first graph above.

Understand that this paper was refereed by other ‘experts’ in the field. Anyone who has been a social scientist for longer than 20 minutes would know that this sort of data should never produce this kind of histogram.

This is by no means the only anomaly the Data Colada sleuths turned up regarding this paper. You can read their entire write-up about it here, in which they conclude that in fact much of the data used in the paper was faked and as I said, their efforts eventually (meaning years later) led to PNAS retracting the paper. Retractions of papers by academic journals are rare occurrences, albeit they are happening more then they used to.

The pre-publication ‘peer review’ of this paper was supposed to be a mark of credibility. Other experts supposedly went through this paper carefully and pronounced it sound and worthy of publication in a professional outlet. Except…..they missed the simple point made above. If reviewers are not looking at the most basic attributes of the data used in the paper, what the hell are they doing?

Very often, not much, sadly. Indeed, it is common when an Important Academic gets a paper to review for a professional journal they give it to one of their grad students to do. I cannot say this happened in this case, but that it happens is well-known. And, to be sure, grad students might do a more careful job, as they want to impress their supervisor. The point is that Important Academics (as well as unimportant ones) often do not take the task of refereeing papers very seriously.

There are two names in the list of the authors of that paper who are particularly noteworthy. Dan Ariely and Francesca Gino. Ariely is a professor at Duke University’s Fuqua School of Business, and Gino is currently on unpaid administrative leave from her position as a tenured professor at Harvard Business School.

Ariely has written many books, including Predictably Irrational in 2008, a copy of which sits on my bookshelf here at home. He has given numerous TED talks, wrote an advice column in the Wall Street Journal for a time, and his life was the basis for an NBC TV crime drama, The Irrational, that ran for two seasons. He’s a big deal.

Gino was certainly a rising academic star on the study of ‘honesty’ until being put on unpaid leave by Harvard in 2023. Tenured prof at The Harvard Business School, and head of HBS’s Negotiations, Organizations and Markets unit. Harvard suspended Gino after undertaking an internal investigation of their own, an investigation that also resulted in three other papers with Gino as an author being retracted from three other journals, and Gino suing Harvard and the Data Colada investigators..

Gino’s defamation suit against the Data Colada people has since been dismissed by a judge, along with some aspects of her suit against Harvard. However, suits accusing Harvard of breach of contract and sexual discrimination are both proceeding as I type this. She has decided that a good offense is the best defense, it would seem.

By no means is the insurance-form paper the only one with Ariely as a co-author that has been thought suspicious. However, unlike Gino, he has to date avoided any sanctions. Some quotes from the Wikipedia page on him –

“In 2021, a 2012 paper written by Francesca Gino, Max H. Bazerman, Nina Mazar, Lisa L Shu, and Ariely was discovered to be based on falsified data and was subsequently retracted.^[5][6] In 2024, Duke completed a three-year confidential investigation, and according to Ariely, concluded that “data from the honesty-pledge paper had been falsified but found no evidence that Ariely used fake data knowingly.”

“In November 2022, the Israeli TV investigative show Hamakor (Channel 13), aired an episode^[30][31] questioning a number of Ariely’s studies that were not reproducible or whose reliability was dubious in terms of the way they were carried out, the data collected, or whether the studies were carried out at all. For example, Ariely claimed that data for his “Ten Commandments” study were collected in 2004–2005 at UCLA with the assistance of Aimee Drolet Rossi. However, despite being thanked in the 2004 paper for collecting the data almost 20 years later, Rossi denies having run the study,^[32][33] and UCLA has issued a statement that the study did not take place there.”

I dunno, does that seem odd to you? On the blogs and other sites I read, other papers in Ariely’s body of work come in for regular skepticism, if not derision. Yet, he sails on, doing TED talks.

I conjecture that Gino’s rather harsher treatment by Harvard is at least partly due to all the bad press Harvard got about their former president in the recent past, but I cannot say that with any confidence.

A particularly disturbing sidelight to all this – at least to me as a former academic – is the following related story, which you can read in more detail here.

It is a story written by one Zoe Ziani, who, as a  graduate student in France working on her PhD, dove deeply into another paper, ‘The Contaminating Effects of Building Instrumental Ties: How Networking Can Make Us Feel Dirty’, published in Administrative Science Quarterly in 2014, and written by Gino and two others. Ziani got into this paper because her dissertation was about networks, and this paper was, in Ziani’s own words, ‘…a cornerstone of the literature.’ on this topic.

She initially came to doubt that the results in the paper could possibly be correct, then eventually concluded that the data used in the study had been fabricated. Seen this movie before? I note this paper was not one of those retracted after Harvard’s investigation, and that in fact you can go here to the ASQ website and find it, although anything beyond the abstract is behind a paywall.  You will also find a link there labelled ‘correction’, and if you click it, you will find that the two authors of the paper whose name is not Gino voluntarily contacted the editors of ASQ with corrected versions of the data for three of the four studies in the paper, which led to changes in some of the results.

Getting back to Ziani, she was an aspiring researcher, a PhD student, and when she brought her concerns about this important paper to the faculty members who were supervising her work, they told her to drop it. Ziani persisted, however, writing up a 10-page criticism of the paper to include in the first chapter of her dissertation. To quote from Ziani’s blog:

“The three members of my committee (who oversaw the content of my dissertation) were very upset by this criticism. They never engaged with the content: Instead, they repeatedly suggested that a scientific criticism of a published paper had no place in a dissertation. After many frustrating exchanges, I decided to write a long letter explaining why I thought it was important to document the issues I had discovered in CGK 2014. This letter stressed that I was not criticizing the authors, only the article, and encouraged the members of my committee to highlight anything in my criticism that they viewed as inaccurate, insufficiently precise, or unfair.

The three committee members never replied to this letter.”

Then, after she had defended her thesis, a ritual all PhD candidates go through, two of the three members of her committee made it clear that they would not sign off on her dissertation if she did not remove from it all traces of her criticism of the Gino et al paper (which she refers to as CGK 2014).

So, fearing for her career, worried that she would not actually be granted a PhD after all those years of work, she did as they insisted.

That working academics would suggest that criticism of another paper had no place in a dissertation is simply appalling. Science advances in no small part when researchers realize that what was thought to be correct is not, and our understanding of the world needs to change. Criticism and correction of past research is at the heart of science.

Now, all I have here is Ziani’s account of the appalling behavior of her faculty supervisors, and nothing from them. Sadly, my 42 years in the business makes it very easy to believe her account.  The paper she thought was deeply flawed, if not indeed a complete fraud, was published by three noted scholars from three prestigious institutions and published in a prestigious academic outlet. It had been refereed by other experts. Many, many academics take the view that you do not push against that, you do not criticize in that instance, as it could go hard on you in the future. (Not all academics, fortunately, but too many.) After all, you have to submit your own papers to these journals, and if the editor or referees who deal with your submission turn out to be any of those people – or their friends/colleagues/admirers – you might worry that you will find your own papers getting rejected a lot.

This is not how science advances, and this is not how scholars should behave. But it happens.

Note: Data Colada also took a run at the paper that Ziani picked apart (i.e., CGK 2014), and you can read their analysis here. They say they were aided in their investigation by ‘early career researchers’. I rather suspect Ziani was one of those, although they remain anonymous on the Data Colada website.

Conclusion: This, and a host of other things about the way 21^st century academia operates, bothers me a lot. My own research was highly theoretical, so no one much cared about it, and certainly it was never of any use to anyone trying to formulate government policy or get their company to run better. But plenty of academic research is used for those purposes, and plenty of it (no, not all of it) is crappy. It may not be fraudulent in the way that the stuff above seems to be, but it is very badly done, and done in a way designed to deliver pre-ordained findings.

I will be writing more about all of this in the future (this post is already too long), because it does indeed piss me off, probably because I was – sort of – in the same business for 42 years.

For now, just remember that what ‘studies say’ may well be bullshit. Fraudulent bullshit, at that.

Let me first express my thanks to Andrew Gelman of Columbia whose blog (see above) first brought this to my attention, so I can bring it to yours.

This is another of those ‘if it seems too good to be true it probably is’ research papers that I so love. The paper is titled ‘Overnight olfactory enrichment using an odorant diffuser improves memory and modifies the uncinate fasciculus in older adults’ and it was published in Frontiers in Neuroscience in July of 2023.

It reports on a study in which participants – all elderly, like me – were given scent diffusers to take home and use for two hours each night, starting when they went to bed – the diffusers automatically shut off two hours after they were started. The participants were given cognition tests and a functional MRI (that’s where the uncinate fasciculus bit in the title comes from) before they started the experiment and again six months later, after they had used the diffusers for that period of time.

The ‘treatment group’ got a set of 7 genuine essential oils to use in their diffusers, while the control group got ‘de minimis amounts of odorant’ according to the researchers. (Do you suppose those in the Control Group noticed that their diffusers produced no scent? But I digress.)

In the end there were a total of 43 participants in the two groups, and the headline result of this research was, quoting the paper,

“ A statistically significant 226% improvement was observed in the enriched group compared to the control group on the Rey Auditory Verbal Learning Test….”

Pretty impressive, eh? Six months of smelling essential oils for two hours/night at bedtime, that’s all it took to get that huge improvement in Auditory Verbal Learning.

Anyone smell anything?

Here are some facts about this ‘controlled’ experiment. First, 43 participants? A bit more than 20 in each group? That is what statisticians call a small sample. But wait, there’s more. If you look at the flow chart of how they recruited and screened participants for this study, you find that 132 subjects passed the initial screening. Of these, only 68 were included in the Control and Treatment groups that were used in the statistical analysis of the results, and of those 68, 25 dropped out during the study. That leaves the 43 whose results are reported on, of the 132 who passed the screen.

Smell anything yet? Why did those 25 people drop out? That’s 36% of the 68 whose results were analyzed and reported. What does that drop out rate imply for the credibility of the results? People don’t drop out randomly, they do it for reasons.

And, as one of the readers of the Gelman blog pointed out, that 226% improvement claim comes from the control group scoring on average 0.73 points worse post-treatment than pre-treatment on a particular test, while the treatment group scored 0.92 points better on average. So you have a difference of 1.65 points in the two groups’ average ‘improvement’ on the test, and 1.65 is 2.26 times 0.73.

Interesting arithmetic. I think Gelman’s reader is right, as that 226% number doesn’t come up anywhere else in the paper. However, note that 1.65 being 2.26 times 0.73 is not the same as 1.65 being a 226% ‘improvement’ over 0.73. The latter would mean that it was more than three times greater, and it is not. Neuroscientists don’t do a lot of basic arithmetic, I guess. That detail aside, just looking at the difference in average scores for the two groups – what does a ‘point’ mean in this context, anyway? Is it big? How ‘big’ is a 1.65 point improvement on this test? What does that actually translate into, memory-wise? The researchers do not say.

One last thing. At the end of the paper there is a section called Funding, under which is written ‘This research was sponsored by Procter and Gamble.’

There, now you smell it, eh?

I know nothing about the journal Frontiers in Neuroscience, but if their editors did not stop and wonder about that 226% claim, well, I don’t think I’ll subscribe. Actually, it’s an open access journal so I can read it for free, as I did this smelly article. But I won’t. Read the journal, I mean. Unless maybe I’m looking for something else to write about here.

Anyone vaguely familiar with basketball has heard of the ‘hot hand’ phenomenon. Someone on the team gets a hot shooting streak going, they can’t seem to miss, and their teammates start looking to get the hot-handed player the ball. I played backyard hoops a lot in my youth, and there were (very few) times when it happened to me; every shot I threw up seemed to go in – briefly.

Well, academics got wind of this long ago also, and decided to investigate whether there was anything to it. Yea, sure, players talk about experiencing it, or seeing it, but it could easily be just a matter of perception, something that would disappear into the ether once subjected to hard-nosed observation and statistical analysis.

The canonical paper to do this analysis was published in 1985 in Cognitive Psychology, authored by Gilovich, Tallone and Tversky. The last of this trio, Amos Tversky, was a sufficiently notable scholar that young economists like me were told to read some of his work back in the day. He died young, age 59, in 1996, six years before his frequent co-researcher, Daniel Kahnemann, was awarded the Nobel Prize in Economics. The work the Nobel committee cited in awarding the prize to Kahnemann was mostly done with Tversky, so there is little doubt Tversky would have shared the prize had he lived long enough, but Nobels are, by rule, not given to the dead.

Now, as a research question, looking for a basketball hot hand is in many ways ideal: the trio used data on shots made and missed by players in the NBA, which tracks such data very carefully, and beyond that, they did their own controlled experiment, putting the Cornell basketball teams to work taking shots, and recording the results. Good data is everything in social science, and the data doesn’t get much better than that. Well, bear with me here, this is most of the Abstract of that 1985 paper:

“Basketball players and fans alike tend to believe that a player’s chance of hitting a shot are greater following a hit than following a miss on the previous shot. However, detailed analyses of the shooting records of the Philadelphia 76ers provided no evidence for a positive correlation between the outcomes of successive shots. The same conclusions emerged from free-throw records of the Boston Celtics, and from a controlled shooting experiment with the men and women of Cornell’s varsity teams. The outcomes of previous shots influenced Cornell players’ predictions but not their performance. The belief in the hot hand and the “detection” of streaks in random sequences is attributed to a general misconception of chance according to which even short random sequences are thought to be highly representative of their generating process.”

That is, a player who hits a shot expects he is likely to hit the next one, too. When he does, he files this away as ‘having a hot hand’, but the actual frequency with which he hits the second shot is not actually higher than when he had missed his previous shot. Standard ‘cognitive bias’ causes the player – and fans – to see it that way, that’s all. They remember when the second shot is made more than they remember it being missed.

Damn scientists are always messing with our hopes and dreams, right? No Easter Bunny, no extra-terrestrials in Mississauga, and no hot hand. Is nothing sacred?  Other researchers went looking for evidence of a hot hand over the ensuing years, but it became known in academic circles as ‘the hot hand fallacy’, the general consensus being that it did not exist in the real world of basketball.

33 years later

But wait, it’s now 2018 and a paper by Miller and Sanjurjo appears in Econometrica, the premier journal for economic analysis involving probability and/or statistics. It’s title is “Surprised by the hot-hand fallacy? A truth in the law of small numbers”

Here’s some of what their Abstract says:

We prove that a subtle but substantial bias exists in a common measure of the conditional dependence of present outcomes on streaks of past outcomes in sequential data…. We observe that the canonical study [that is, Gilovich, Tallone and Tversky] in the influential hot hand fallacy literature, along with replications, are vulnerable to the bias. Upon correcting for the bias, we find that the longstanding conclusions of the canonical study are reversed.

It took over 30 years for two economists to figure out that ‘the canonical study’ of the hot hand did its ciphering wrong, and that once this is corrected, it’s findings are not just no longer true, they are reversed. The data collected in 1985 do provide evidence of the existence of a hot hand.

Think about this. In 1985 some very clever academics showed there was no such thing as a hot hand in the real world of basketball, and the academics who peer-reviewed their work agreed with them. Thirty-plus years later, some other clever academics realized that first set had gotten something wrong, and that fixing it reversed the previous findings – and the academics who peer-reviewed their work agreed with them.

Ain’t social science wonderful? A question for which there is excellent data, a situation rarer than hen’s teeth in social science, is investigated and a conclusive answer arrived at, and thirty years later that answer is shown to be not just wrong but backwards.

No one did anything shady here. There was no messing with data, the 2018 guys used the same data used in 1985. A mistake, a subtle but significant mistake, accounts for the turnaround, and it took 33 years to discover it. One can hardly blame the 1985 researchers for not seeing the mistake, given that no one else did for such a long time.

The Lesson?

So, in case my point is not yet obvious, science is not a set of settled facts. Those do exist – sort of – but anyone who understands the process of science even a little understands that settled facts are settled only until they are overturned. And if that is true for such a clean research question as an investigation of a basketball hot hand, think about a more typical social science question in which two things are almost always true. One, the data is not at all what the researchers need, so they make do with what they can actually gather. Two, the right way to analyze that data – among endless possibilities – is a matter of disagreement among respectable social scientists. Following that kinda science will make you dizzy, my friends.

A teaser: think about this social scientific question. It is arguably of more importance than basketball shooting.

Does the availability of bricks-and-mortar adult entertainment establishments have a positive, negative, or no effect on the commission of sex crimes in the surrounding neighborhood?

Whaddya think is the right answer?

For extra credit: what kind of data would a researcher need to gather to answer that question?

Now that’s real (i.e., messy) social science.

Stay tuned, because a couple of economists set out to investigate the question above, and I’ll have a go at what they did and their findings in a future post.