Academics, Fraud and Integrity
Suppose you asked 1,000 people how many miles they drove their car last year, recorded the results, then put them into a graphic (a ‘histogram’ in this case). What do you think it would look like?
Something like this:
A few people drive hardly at all, less then 250mi/year in this case, and a very few people drive more than 50,000 miles/year, but most are grouped around a median number between 2,000 and 10 thousand miles per year. This gives the very familiar ‘bell curve’ shape to the distribution of results, as you see above. This general shape is so ubiquitous that statisticians have come to call the continuous version of this a ‘normal’ distribution. (Strictly speaking, the normal distribution has tails at either end that go on forever at lower and lower levels. The distribution above is ‘truncated’ at either end, in stats lingo.)
Very many of the distributions that one gets from observations of occurrences that are influenced by many factors end up looking like this, whether the data is from medical studies (height , weight, systolic blood pressure), education (SAT scores), or nature (rabbits have 1-12 kits per litter, with an average of around 6.)
In 2012 Lisa L Shu, Nina Mazar, Francesca Gino, Max Bazerman and Daniel Ariely published a paper in The Proceedings of the National Academy of Sciences titled ‘Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end’ (a mouthful, I know) which reported on, among other things, the following field experiment.
Subjects who were clients of an auto insurance company were asked to fill out a form in which they were asked how many miles they drove their cars last year. All had to sign a statement saying they had answered honestly, but on some forms the statement of honesty was at the top of the form, and in the others it was at the bottom. The startling result was that the clients who signed at the beginning were claimed to report more honestly, because customers assigned to the ‘sign-at-the-top’ condition reported driving 2,400 more miles (10.3%) on average than those assigned to the ‘sign-at-the-bottom’ condition.
Such a simple thing can make people more honest, that’s truly remarkable, eh? Actually, it is mostly bullshit, and said paper was in fact retracted by the PNAS a year or two ago, due to the efforts by some scholars at a place with the wonderful name of ‘Data Colada’, who determined that the research was suspect. Really suspect.
The data reported in the paper regarding how clients answered this mileage question, put into a histogram like the one above, looked like this:
The ‘Car #1’ refers to the fact that some clients owned more than one car, and this is their report on the first (or only) car they owned. N=13,488 refers to the fact that there were 13,488 clients included in this study, all of whom of course reported on at least one car’s mileage. The histogram above is what statisticians would call a (approximately) uniform distribution. Data on miles driven in a year does not look like this, period. It looks like the first graph above.
Understand that this paper was refereed by other ‘experts’ in the field. Anyone who has been a social scientist for longer than 20 minutes would know that this sort of data should never produce this kind of histogram.
This is by no means the only anomaly the Data Colada sleuths turned up regarding this paper. You can read their entire write-up about it here, in which they conclude that in fact much of the data used in the paper was faked and as I said, their efforts eventually (meaning years later) led to PNAS retracting the paper. Retractions of papers by academic journals are rare occurrences, albeit they are happening more then they used to.
The pre-publication ‘peer review’ of this paper was supposed to be a mark of credibility. Other experts supposedly went through this paper carefully and pronounced it sound and worthy of publication in a professional outlet. Except…..they missed the simple point made above. If reviewers are not looking at the most basic attributes of the data used in the paper, what the hell are they doing?
Very often, not much, sadly. Indeed, it is common when an Important Academic gets a paper to review for a professional journal they give it to one of their grad students to do. I cannot say this happened in this case, but that it happens is well-known. And, to be sure, grad students might do a more careful job, as they want to impress their supervisor. The point is that Important Academics (as well as unimportant ones) often do not take the task of refereeing papers very seriously.
There are two names in the list of the authors of that paper who are particularly noteworthy. Dan Ariely and Francesca Gino. Ariely is a professor at Duke University’s Fuqua School of Business, and Gino is currently on unpaid administrative leave from her position as a tenured professor at Harvard Business School.
Ariely has written many books, including Predictably Irrational in 2008, a copy of which sits on my bookshelf here at home. He has given numerous TED talks, wrote an advice column in the Wall Street Journal for a time, and his life was the basis for an NBC TV crime drama, The Irrational, that ran for two seasons. He’s a big deal.
Gino was certainly a rising academic star on the study of ‘honesty’ until being put on unpaid leave by Harvard in 2023. Tenured prof at The Harvard Business School, and head of HBS’s Negotiations, Organizations and Markets unit. Harvard suspended Gino after undertaking an internal investigation of their own, an investigation that also resulted in three other papers with Gino as an author being retracted from three other journals, and Gino suing Harvard and the Data Colada investigators..
Gino’s defamation suit against the Data Colada people has since been dismissed by a judge, along with some aspects of her suit against Harvard. However, suits accusing Harvard of breach of contract and sexual discrimination are both proceeding as I type this. She has decided that a good offense is the best defense, it would seem.
By no means is the insurance-form paper the only one with Ariely as a co-author that has been thought suspicious. However, unlike Gino, he has to date avoided any sanctions. Some quotes from the Wikipedia page on him –
“In 2021, a 2012 paper written by Francesca Gino, Max H. Bazerman, Nina Mazar, Lisa L Shu, and Ariely was discovered to be based on falsified data and was subsequently retracted.[5][6] In 2024, Duke completed a three-year confidential investigation, and according to Ariely, concluded that “data from the honesty-pledge paper had been falsified but found no evidence that Ariely used fake data knowingly.”
“In November 2022, the Israeli TV investigative show Hamakor (Channel 13), aired an episode[30][31] questioning a number of Ariely’s studies that were not reproducible or whose reliability was dubious in terms of the way they were carried out, the data collected, or whether the studies were carried out at all. For example, Ariely claimed that data for his “Ten Commandments” study were collected in 2004–2005 at UCLA with the assistance of Aimee Drolet Rossi. However, despite being thanked in the 2004 paper for collecting the data almost 20 years later, Rossi denies having run the study,[32][33] and UCLA has issued a statement that the study did not take place there.”
I dunno, does that seem odd to you? On the blogs and other sites I read, other papers in Ariely’s body of work come in for regular skepticism, if not derision. Yet, he sails on, doing TED talks.
I conjecture that Gino’s rather harsher treatment by Harvard is at least partly due to all the bad press Harvard got about their former president in the recent past, but I cannot say that with any confidence.
A particularly disturbing sidelight to all this – at least to me as a former academic – is the following related story, which you can read in more detail here.
It is a story written by one Zoe Ziani, who, as a graduate student in France working on her PhD, dove deeply into another paper, ‘The Contaminating Effects of Building Instrumental Ties: How Networking Can Make Us Feel Dirty’, published in Administrative Science Quarterly in 2014, and written by Gino and two others. Ziani got into this paper because her dissertation was about networks, and this paper was, in Ziani’s own words, ‘…a cornerstone of the literature.’ on this topic.
She initially came to doubt that the results in the paper could possibly be correct, then eventually concluded that the data used in the study had been fabricated. Seen this movie before? I note this paper was not one of those retracted after Harvard’s investigation, and that in fact you can go here to the ASQ website and find it, although anything beyond the abstract is behind a paywall. You will also find a link there labelled ‘correction’, and if you click it, you will find that the two authors of the paper whose name is not Gino voluntarily contacted the editors of ASQ with corrected versions of the data for three of the four studies in the paper, which led to changes in some of the results.
Getting back to Ziani, she was an aspiring researcher, a PhD student, and when she brought her concerns about this important paper to the faculty members who were supervising her work, they told her to drop it. Ziani persisted, however, writing up a 10-page criticism of the paper to include in the first chapter of her dissertation. To quote from Ziani’s blog:
“The three members of my committee (who oversaw the content of my dissertation) were very upset by this criticism. They never engaged with the content: Instead, they repeatedly suggested that a scientific criticism of a published paper had no place in a dissertation. After many frustrating exchanges, I decided to write a long letter explaining why I thought it was important to document the issues I had discovered in CGK 2014. This letter stressed that I was not criticizing the authors, only the article, and encouraged the members of my committee to highlight anything in my criticism that they viewed as inaccurate, insufficiently precise, or unfair.
The three committee members never replied to this letter.”
Then, after she had defended her thesis, a ritual all PhD candidates go through, two of the three members of her committee made it clear that they would not sign off on her dissertation if she did not remove from it all traces of her criticism of the Gino et al paper (which she refers to as CGK 2014).
So, fearing for her career, worried that she would not actually be granted a PhD after all those years of work, she did as they insisted.
That working academics would suggest that criticism of another paper had no place in a dissertation is simply appalling. Science advances in no small part when researchers realize that what was thought to be correct is not, and our understanding of the world needs to change. Criticism and correction of past research is at the heart of science.
Now, all I have here is Ziani’s account of the appalling behavior of her faculty supervisors, and nothing from them. Sadly, my 42 years in the business makes it very easy to believe her account. The paper she thought was deeply flawed, if not indeed a complete fraud, was published by three noted scholars from three prestigious institutions and published in a prestigious academic outlet. It had been refereed by other experts. Many, many academics take the view that you do not push against that, you do not criticize in that instance, as it could go hard on you in the future. (Not all academics, fortunately, but too many.) After all, you have to submit your own papers to these journals, and if the editor or referees who deal with your submission turn out to be any of those people – or their friends/colleagues/admirers – you might worry that you will find your own papers getting rejected a lot.
This is not how science advances, and this is not how scholars should behave. But it happens.
Note: Data Colada also took a run at the paper that Ziani picked apart (i.e., CGK 2014), and you can read their analysis here. They say they were aided in their investigation by ‘early career researchers’. I rather suspect Ziani was one of those, although they remain anonymous on the Data Colada website.
Conclusion: This, and a host of other things about the way 21st century academia operates, bothers me a lot. My own research was highly theoretical, so no one much cared about it, and certainly it was never of any use to anyone trying to formulate government policy or get their company to run better. But plenty of academic research is used for those purposes, and plenty of it (no, not all of it) is crappy. It may not be fraudulent in the way that the stuff above seems to be, but it is very badly done, and done in a way designed to deliver pre-ordained findings.
I will be writing more about all of this in the future (this post is already too long), because it does indeed piss me off, probably because I was – sort of – in the same business for 42 years.
For now, just remember that what ‘studies say’ may well be bullshit. Fraudulent bullshit, at that.