People? Who Needs People?

I say this often:

There is no idea so bad that one cannot find some politician who will support it.

Exhibits 1 to 26 for the truth of this – most of what London City Council has done in the last three years, plus most of the policies espoused by Z Mamdani for NYC.

I have always also thought, though I say it less often, that there is a corollary to that axiom:

There is no idea so bad that one cannot find an academic to support it, if it will give them an excuse to write more research papers.

This corollary came streaking across the night sky and back into my brain as I was reading one of the latest posts from Gelman’s stats blog. You know the one. Gelman has a team of people who contribute posts from time to time, and this one is from one Jessica Hulman on Dec 19. It describes an idea that is breathtaking in its ignorance.

However, I will have to back up into some history of social science research to explain the whole mess. Do hang in there, I promise not to get geeky.

Sometime after I joined the ranks of economists, a new area of study in Econ was born called ‘experimental economics’. It had two branches, and the legitimate one involved doing field experiments. One such that I remember well was about charitable giving, an area in which I used to work. [The details here may be a bit off, it’s been awhile, but I trust researchers Andreoni and company will forgive me. I just want to give you the idea of how such research works.]

These researchers collaborated with the Salvation Army to put their kettles just inside the doors of a big retail store – might have been Walmart. The bell-ringers operated in two different ways. In Situation 1 they stood silently by their kettles and thanked anyone who might drop money in. In situation 2 they rang their bells, yelled ‘Merry Christmas’ and did their best to engage with the shoppers going in and out.

They then checked to see how much money was in the kettles in each situation, but the clever part was that in each situation they put kettles at every entrance/exit but one. A door around the side of the store had no kettle at all, but it did have someone noting how many people used that door. This allowed them to tell whether any difference in money taken in was partly due to people dodging the kettles more often in one Situation than in the other.

Not world-changing research, but you can imagine that Sally Ann and other charities would be very interested in the results (no, I don’t remember what they found).

That is a social science field experiment. One goes out into the field and tries something out to see how humans react. They are not easy to do, take a lot of time and money, and this one in particular required the researchers to get the cooperation of both Sally Ann and Walmart.

The other branch of experimental economics does lab experiments. That is what it sounds like, these researchers bring the participants into a ‘lab’ to be part of an experiment. What that usually involves is sitting them down at a computer where they interact with other participants and some software package. These experiments require the participants to make choices at their keyboards, and those choices comprise the data that is gathered from the ‘experiment’.

To be concrete, a classic example of such an experiment, also used to study people’s charitable behaviour, is to have them participate in a ‘public good game’. Each subject is given a pile of electronic tokens to ‘spend’. Any tokens they still hold at the end of the experiment are redeemed for actual money before they leave the lab. The participants are given the opportunity to contribute some tokens to a ‘public good’. Each participant of course gets to keep any tokens they do not contribute, but they also get some tokens paid out to them by the software, and the amount they get increases as the total amount donated by all participants goes up. Thus, each participant has to decide how many of his original tokens to keep, and now many to contribute to the public good, knowing that his contributions increase the payoffs to all participants, including himself.

The point here is that using the computer software, you can vary in a million ways how the tokens accumulate and are paid out. For example, you can mimic a ‘matching donation’ regime in which some outsider pledges to match the total number of tokens donated to the public good, thereby increasing in some measure the payoffs each donor gets. You can also put in a pseudo government that takes some percentage of each participant’s tokens, but also gives a tax deduction for any tokens each contributes to the public good.

You can have the participants contribute simultaneously, so no one knows how much anyone else gave when they make their own donation, or you can have people contribute in some order, so each knows how much has been given by those who gave already.

The possibilities are indeed endless, it’s all a matter of software and then writing down what happens.

There is a technical name for this sort of experiment, I’m trying to remember it…..oh, yea – Horseshit.

Ok, that’s my name for it, and here is why. Whatever happens when you put people in a room to participate in such a public good game, the idea that it tells us anything about how people’s real-world donations to charity respond to actual tax policies or actual matching programs or….anything – is laughable. The reasons are many.

The public good game is not having people support an actual charity. It is not,full stop, so their attitudes toward charity in general, or what a charity does with their donation, play no role.

The people are not spending their own real income in the game. The get some tokens they can keep if they don’t contribute, but the amounts are trivial. A subject might walk out of that experiment with an extra $20 in their pocket. Indeed, ethical considerations typically require that subjects not come out of the experiment with less money than they went in with, a real problem if you are trying to study gambling or other risk-taking behaviour in a ‘lab’.

Because the amounts earned or lost are trivial, there is nothing to guarantee the subjects are not just trying to fuck up the experiment in some way. It costs them virtually nothing to do so.

If researchers gave people $5,000 worth of tokens to start with that would not be much of a problem, and then they would be making real decisions about significant amounts of money. But that never happens because it would make the experiments frightfully expensive to run. And, it would still be true that people typically treat money they had to earn differently from money someone gives them that they can then try to keep.

The point then is: whatever subjects do in those labs, there is no good reason to believe it tells us anything about how they behave when they give to actual charities using their actual earned income. The experiments have zero ‘external validity’, as academics like to say. And, the point of social science research is not supposed to be to publish papers, it is to understand society. You know, the one we all live in.

Yet, thousands of such ‘lab experiments’ in Econ have been done since the 90s, and the results get published. There is now even an Economic Science Association which publishes a journal called Experimental Economics solely devoted to this stuff.

Why the popularity of this type of research?   l have an answer. It is easy. You just get your university to build you a ‘Behavioural Economics Lab’, which consists of 30 or so computer terminals in a room, and some software, and you can do as many experiments as you can dream up…..and publish as many papers in EE as you can write up. For the universities that expense is minor, and if their faculty can publish tons of papers based on what happens in it, it seems like a good idea. Cuz what U-bureaucrats want is for their faculty to publish lots and lots of papers. Whether any of them are worth a plugged nickel is not the bureaucrats’ concern. They wouldn’t know, anyway.

Oh, and another key thing about this experimental research is that one cheap way to get subjects for your experiment is to require all the students in Econ 101 to sign up to participate in at least one experiment in order to pass the course. My Dept had none of these experimentalists so this did not happen, but the UWO Psych Dept had that requirement in their Intro Psych course for years. I don’t know if they still do.

One wonders how much of the field of psychology – and now, Econ – is based on the responses of disgruntled undergrads to surveys and computer games.

However – this has all been a description of history. The journal called EE marches on, you can get access to it here, no charge. [The phrase ‘You get what you pay for’ comes to mind, but that’s just mean, right?]

So much for background. The lightning bolt that prompted me to write this post was a post on the Gelman stats blog titled  ‘Validating language models as study participants: How it’s being done, why it fails, and what works instead’

You can click on that title if you want read it yourself.

Quoting from the first paragraph of the post:

Earlier this year, I started paying attention to proposals to use LLMs to simulate participants in surveys and behavioral experiments. The idea is that LLMs can be prompted with experiment or survey instructions and a participant persona (e.g., demographic description), making it possible to simulate target human samples without the cost and headache of recruiting real people.

Yep, that’s right folks. Ms Hulman is looking into the use of LLMs (Large Language Models) like ChatGPT to be used as responders to surveys and to participate in behavioural experiments like the one described above. You just give them a ‘persona’ and see how they respond to your survey or observe what choices they make in a public good game. All without the ‘cost and headache of recruiting real people’.

We can do social science – the study of human society – without any humans at all. I ask you: is that just  Fucking Fabulous or what?

It certainly is if you are an academic looking for ways to write more and more papers, which is how you get ahead in your career.

I can hardly wait to see the policy prescriptions that come out of this emerging new wave of research – and you better know, folks, that no self-respecting social scientist does research these days that does not generate a ringing endorsement of some policy or other.

Moreover, with that ‘cost and headache of recruiting real people’ gone there is truly no end to the number of papers any single social scientist can write. Hell, they can get an LLM to write them (actually, that’s already been happening for some time….).

As I said, this is breathtakingly ignorant of what social science is supposed to be. But academics have their eyes focussed on what matters to them – writing papers and getting them published. Three days after this article was published on the Stats blog, there have been four comments on it from other blog readers, and not one of them has suggested there is anything problematic here.

To sum up this admittedly long post –

The original experimentalists choose to ignore the fact that their human subjects are being observed in completely artificial situations, rather than in actual social settings. Now the LLMers want to ignore the fact that those observed subjects are not human.

Ya gotta love the academic mind, eh?