Not My Job   

I write a fair bit on here – too much for some of you, I expect – about academics and their research. Besides the fact I was in that business for 42 years, my motivation is simple: I want to convince you – indeed, I would like to convince the world – that you should ignore almost every claim made by every academic researcher that you come across.

They are surely not all bogus, but the fact is that the most interesting and surprising ones are those most likely to be publicized by an uncritical press, and are also those most likely to be bogus. As part of my ongoing effort in this regard, I offer you a look into the thinking of one very esteemed academic as regards his responsibility to keep bogus research from being published.

Yes, it’s statistician Gelman again, and he recently posted a short article titled ‘I don’t see journal review as a gatekeeping process that will keep erroneous articles from being published’

The title kinda says it all, and you can click on that title and read for yourself what he wrote, but I will give you the gist.

‘Journal review’ refers to the review by other academics of research articles that are submitted to journals for publication. Most people think – as did I at one time – that this process is designed to make sure that the research was competently done, and that any claimed findings are sound and backed up by good scientific practice. Seems kinda important, eh?

One would like to believe that if researchers publish a paper that finds that, for example –

Offering substantial performance-based compensation to highly effective teachers in the lowest-achieving schools leads to dramatic improvements in student achievement, nearly closing the gap between the targeted schools and the district average

…then that finding is legit. In fact, that is (a close paraphrase of) the claim from a recent article published in an Econ journal.

Publication in that journal means it was peer-reviewed, and the reviewers found no errors or other reason to think the finding is not valid. The data analyzed is indeed evidence that the claim is true.

Gelman says – nah, you can’t expect that. At least not if Gelman reviewed the paper.

A quote from his post –

I don’t think it is a requirement of a reviewer for a journal to investigate a submission in detail. Ultimately the correctness of an article is the responsibility of the author.

To put it another way, I don’t see journal review as a gatekeeping process that will keep erroneous articles from being published; rather, I see it as providing some information to the author and editor.

See, his loyalties are to the academic(s) who submitted the article to be published. If he/she/they think it is fine, it’s not up to reviewer Gelman to say otherwise, he just provides some helpful info to the author(s) and the editor of the journal.

A more succinct statement of the ‘I scrub your back you scrub mine’ mentality of academics is not possible.

I already noted in a previous post Gelman’s statement that it was not up to him to file a formal complaint that the APSA had given an award to a massively plagiarized book. Not his thing, either.

There is a common thread in these two separate positions of Professor Gelman, and although I witnessed it up close throughout my career, I did not grok it until I left – academics are not much interested in putting in effort on anything other than getting their own research published.

Teaching, reviewing others’ research, running the university so the bureaucrats cannot – none of that is worth their attention, time or effort.

I can only say again – if you read that ‘a new study shows X’, then put your money on ‘not X’. The author of that study was likely willing to do all kinds of dodgy shit to produce ‘X’, and no one in the academic community has any incentive to look for such dodginess.

Coda

Angry as Gelman’s utter self-interest makes me, I should not really claim that all academics refuse to take their journal article reviewing duties seriously. Many in my old discipline of Econ did, and I was one of them. I am guilty of much during my career, for sure. However, my referee reports to editors ran to pages and pages, as I did put in hours going through whatever article I was sent, looking for bullshit, and telling the editor whenever I found any. Of course, the articles I reviewed, as well as those I wrote myself, were purely theoretical, so their impact on anyone outside of my intellectual circle was nil. Nada. As in, zilch. That is not so for research like the article from the econ journal that I cited above, or the kind of articles Gelman gets to review. That kind of work can be used to construct or modify actual government policy that affects thousands, if not millions. Someone should be checking that it is goddam correctly done.

I hope the percentage of Gelmans in academia is small. I fear it is not, because much bullshit does get through.