This is going to be a post without a strong message, prompted by a graph that I happened on while looking into something completely unrelated. The graph is below, and it shows changes in spending per pupil in the US state of Oregon, along with changes in the NAEP scores Oregon students earned in Reading and Math, all between 2013 and 2024. NAEP stands for National Assessment of Educational Progress, which is a set of tests given typically to grade 4 and 8 students in the US. NAEP is administered by a section of the US federal Department of Education. I am showing the Oregon results only because they are what I stumbled across. One can go to the website and see results for all other states too, but if you are so interested, I would suggest you go here instead.

Edunomicslab.org is a website of the McCourt School of Public Policy at Georgetown University, devoted to education finance, and it is the org that produced the graph below, using the stats provided by the Department of Education. It is simply more reader-friendly and organized than the US gov website.

Again, not to pick on Oregon, you can find a similar graph for each of the US states, my quick perusal said that Oregon is not atypical. Spending on education has increased at a rate much greater than inflation in Oregon, at the same time that NAEP scores have decreased.

Below is the same graph for the US as a whole, which certainly indicates that most US states look like Oregon, since the national graph does.

Finally, here is another graph from the same source, which compares staffing on a national basis to student numbers.

So, to note the obvious, staffing in US elementary schools has increased by close to 12% over this 10-year period, while the number of students has decreased by 2%. The graph does not indicate, and I could not find, any stats telling us how many of those added staff actually work in classrooms, rather than in the bureaucracy. I know what I would bet on in that regard, but cannot find corroboration, so far.

Anyway, the aggregate message here is clear. Over the covered period, US primary education has seen spending increase at a rate much greater than inflation, while also seeing staffing increase while student numbers fell (a primary reason for the spending increase, one must think) all resulting in worse student NAEP outcomes. It says nothing good about the current state of (largely publicly funded and administered) primary education in the US. I hasten to add, as does Edunomicslab, that there are US states in which the picture is better, in that student NAEP outcomes have gotten at least a bit better over this period. Their takeaway from that is that what states do with regard to education can make a difference.

I am not going to dive into state-by-state comparisons, I don’t live in the US. I am interested in how things look along these same dimensions in Canada, and here I run into the usual problem. No org in Canada does anything like the data gathering and organization of important public stats that you find in the US. StatsCan in the early years of my professional life had a reputation as one of the best such orgs on the planet. No more. Getting things from their website is difficult, they no longer produce user-friendly graphs and tables of such things, and on top of that, Canada does not have the same density of non-profit think tanks and such which take up the slack. The one exception to that latter  statement that I know about is The Fraser Institute, whose efforts are none the less dwarfed by what is done by edunomicslab, among others.

My complaining aside, I did what I could to find something comparable for Canada, and succeeded partially, with the results below.

A Fraser Institute Report from August of 2024 provides the following stats, among others:

– Per-student spending adjusted for inflation (price changes) increased by 5.1% nationally from 2012/13 to 2021/22.

–   Student enrolment across Canada increased by an average of 5.1% from 2012/13 to 2021/22. Three provinces saw a decrease in enrolment: Newfoundland & Labrador (5.8%), New Brunswick (1.8%), and Ontario (0.1%).

–  Compensation remains the largest and costliest aspect of education spending and has contributed the largest portion to the growth of total education spending in Canada.

The time period here is close to the one in the US graphs above, but it aggregates primary and secondary school stats together. I could not find any stats on staffing level changes in Canada.

Canada has a variety of provincial testing regimes for its students, but what I found readily available were results on the PISA scores in Canada. PISA is the Program for International Student Assessment (PISA), a triennial survey run by the Organization for Economic Cooperation and Development (OECD) that assesses 15-year-olds’ abilities in reading, mathematics, and science. Thus, it is given to secondary students every three years in most countries. I don’t know what percentage of students in every country take it, but get the impression that it is high. It has the advantage of making Canada US comparisons easy, and the last round of PISA test was done in 2022.

I found the following:

Average mathematics scores have dropped consistently in Canada, from an average of 532 in 2003 to an average of 497 in 2022.

PISA reading scores in Canada went from 520 in 2018 to 507 in 2022, just four years.

PISA science scores in Canada went from 528 in 2015 to 515 in 2022

The Canadian average scores in 2022 in Math and Science were notably better than for the US, the reading scores in the two countries were almost the same. It is worth mentioning that Canada’s average score over the three areas was the eighth highest of all countries in 2022, while the US’s average ranked it eighteenth.

So my take, based on this admittedly non-optimal data, is that the situation in the two countries is similar, but rather more stark in the US than in Canada. Neither country’s educational systems are performing well in producing student outcomes from the resources they use in the sense that outcomes have been getting worse while spending and staffing have increased. No doubt some of the declines in achievement can be attributed to school lockdowns during the Covid era, but they started before that, so that is not the entire story.

None of this points clearly at what to do about any of this, the data is too aggregate for that. However, if more is being spent per student to hire more personnel per student, and outcomes are getting worse, I don’t see any way to avoid the implication that the increased spending and personnel are being badly deployed.

….it’s not thinking.

I write these posts and most everything else in Word, like most of the human race, at least in North America. Every time I start a document, and then every time I start a new line in a document, a symbol pops up in Word on the left hand margin letting me know that Copilot is ready and waiting to make my writing better and easier.

I cannot get rid of this annoying symbol, because the Word program I am using is not mine, I am just renting it from Microsoft, and I have very few choices about how it works. I expect before I die that Word will no longer give me the option to ignore Copilot, just as my Microsoft Windows OS no longer gives me many real choices about how my laptop operates. We who use computers -as in, almost everyone – are all just dancing to Microsoft’s tune, nowadays.

Well, I want to report here on a blog post I just came across on a website called Mind Matters, written by Gary Smith, an economist at Pomona College. It has nothing to do with Econ, trust me, and if you want to skip the middleman (me) and read his long-ish post yourself you can do that here.

Assuming you are still with me, I’ll give you the basic story and then some related commentary based on other stuff I’ve come across on the interweb thingie regarding LLMs (Large Language Models).

Smith is reporting on what happened when he put a question to four AI LLMs – Deepseek, Copilot, Gemini Flash and OpenAIo1.

Here’s the initial question, which Smith asked in two parts –

Gary:  I’d like to discuss a new version of tic-tac-toe called 90-degree tic-tac-toe.

The four chatbots all respond enthusiastically with some variation on – ‘sounds interesting, tell me more’

Gary: It is still a 3×3 grid but the grid is rotated 90-degrees to the right.

Smith says he edited the LLMs responses down a bit, as they do go on and on, and here are the four responses.

OpenAI:  Interesting concept! By rotating the grid 90 degrees to the right, you’re visually changing the perspective of the classic 3×3 Tic-Tac-Toe board. The gameplay itself might remain the same, but the rotated perspective could create a novel experience for players.

DeepSeek: Ah, I see! So in 90-degree tic-tac-toe, the 3×3 grid is rotated 90 degrees to the right….This could be a fun twist because the rotated grid might disorient players initially, making it harder to spot winning opportunities or block opponents.

Copilot:  That sounds like a fun twist on the classic game! By rotating the grid 90 degrees to the right, it gives players a fresh perspective and challenges them to think differently about their moves.

Gemini:  This rotation doesn’t change the core concept of getting three of your symbols in a row, but it does significantly alter the strategy and how players approach the game.

Philosophers have an accepted definition of ‘bullshitting’. One is honest when one says what they believe to be true, one is a liar when one says things they believe are false, and one is bullshitting when one is unconcerned about the truth or falsity of what one says.

LLMs bullshit, because it is all they can do. They have no idea what is true. One writer on AI that I read said that what comes out of an LLM is simply the result of a billion incredibly fast coin tosses. Not exactly right, I suspect, but close enough.

As for LLMs learning as they go, Smith goes on from here and drags more BS out of the LLMs. He asks them if they think it would be better to rotate the grid right or left, and they have definite opinions about that, as well as on the question of which direction of rotation will make the game more difficult for expert players.

Smith notes some key features of how LLMs generate answers –

They write hundreds of words when a dozen would do, they are always positive and enthusiastic (lots of !s), reluctant to say ‘I don’t know’, and they lack common sense.

His bottom line: ‘If you know the answer, you don’t need to ask an LLM; if you don’t know the answer, you can’t trust an LLM.’

All this made me realize something. LLMs are algorithms that rather faithfully replicate the behaviour of all the bad students I taught over the years.

I never asked painfully obvious questions about tic-tac-toe during my years in the trenches, but I almost always asked a question or two that could be answered with a few simple lines by any student who had studied. (I never asked multiple choice questions in my last 20 or so years of teaching.) And, there were always some number of students who gave long and ludicrous answers to those questions, because their lack of effort in the course meant they had no idea there was a simple clear answer. They were left trying to guess at words and phrases that would match up well with the words in the question. They were, in a word, bullshitting, in hopes that something they wrote would get them some points, in spite of their utter incomprehension.

I don’t suppose any company is likely to come out with a general use LLM called ‘BadStudent’, though.

A further (last, I promise) thought. I bet if you put Smith’s rotated tic-tac-toe question to a hundred second-year undergrads, some positive number of them would answer with bullshit not too dissimilar to what the LLMs spit out. However, those students would be doing that because they were not confident that they weren’t missing something about the question, perhaps reinforced by a lack of confidence in their own thinking. The LLMs have no ‘reasons’ for bullshitting, it’s just what they are designed to do.

I sometimes write posts for this blog and then file them without posting, either thinking I will get back to them later and improve/shorten them, or just deciding that the post isn’t up to snuff. I have a looong one sitting in limbo right now on grade inflation, specifically in US universities. I think it’s an issue in Canadian universities (and secondary schools) also, but there is better data on it for the US.

Anyway, I may post that still-simmering article at some point, as it is a topic about which I care, but the Harvard Crimson, that august institution’s student paper, has published an article about grade inflation at Harvard that provides a nice introduction to the topic – and a shorter post.

Kurt Vonnegut is reported to have said ‘Most kids can’t afford to go to Harvard to be misinformed.’, and he wrote that long before Harvard’s most recent troubles – in 1987, in fact. It turns out that if a kid can afford it, he is at least all but assured of getting an A while being misinformed.

   or……….  

The Crimson article is from 2023, and titled ‘Harvard Report Shows 79% A-Range Grades Awarded in 2020-21, Sparking Faculty Discussion’

The Crimson has no paywall, so you can find the article and read it yourself, I expect, but here’s the graph that it opens with.

So, nearly 80% of the course grades awarded in 2020/1 were As or A minuses. Wow, is all I can say. I thought we gave marks that were too high at UWO back in the day, but I don’t think we ever approached 70+% As.

As is always the case, the percentage of As awarded varies by discipline. Here’s another graph from the Crimson report, which also goes further back into Harvard history.

SEAS is the School of Engineering and Applied Science, and I have to admit that I am surprised that Social Science profs were just as tough graders as were those in Science. (Economists, almost always housed in Social Science, are typically tough graders, but this is normally more than made up for by the Sociologists, Psychologists, Political Scientists, etc. Not at Harvard, apparently.) That the A & H faculty were the easiest graders should surprise no one. They would almost certainly look tough compared to the faculty in the Faculty of Education – if Harvard has one. And, of course, I am using ‘tough’ here ironically. The 60% A marks handed out in SEAS is anything but ‘tough’.

Yet the truly remarkable thing here is that the percentage of A grades awarded has more than doubled in 20 years in all faculties.

Now why do you think that happened?

I can tell you that there were administrators at Western in my day who attributed the ever-rising admission average to get into the Faculty of Social Science to the fact that ‘Western is just attracting better and better students.’ Secondary School grade inflation? – oh, no, certainly not.

The Crimson has some quotes from various administrators at Harvard about this, one of them from Dean of Undergraduate Education Amanda Claybaugh:

In the meeting, Claybaugh said that the “report establishes we have a problem — or rather, we have two: the intertwined problems of grade inflation and compression.”

By ‘grade compression’ academics mean the fact that all grades awarded are concentrated on a few possibilities, making it hard to distinguish really outstanding students from others. I find it encouraging – remarkable, even, given my own interactions with administrators – that the Dean thinks this is a problem. She is later quoted again:

“There is a sense that giving a wider range of grades would give students better information about their performance, and it would give us better information about where they are ranked against other students,” Claybaugh said in an interview after the meeting.

Claybaugh said that the evidence for the existence of grade inflation was less clear, as many student grades are well-deserved and faculty have increasingly focused on learning objectives.

Yea, that sounds more like an administrator – these grades are well-deserved. At least, ‘many’ of them are.

The next quote actually makes me grumpy. Well, grumpier…..

Nonetheless, she said it seems, as one faculty member put it, external “market forces” are influencing grading, particularly as faculty rely on positive course evaluations from students for professional advancement, she said in the interview.

Ok, I’m an economist, I am used to the fact that everyone wants to blame those awful ‘market forces’ when things go bad, but c’mon, unnamed faculty member. If faculty are giving high marks to get higher student evaluations there is nothing ‘market’ about that, it is an internal Harvard decision to use those (almost entirely uninformative, imho) student evaluations for ‘professional advancement’. You could stop doing that, you know. I’m talking to you, Harvard.

Here’ my favourite quote:

Claybaugh said she would defer to the full faculty to decide whether or not to implement concrete reforms to Harvard’s grading policies, but said she would be “interested in exploring” changes “that put more information on the transcript that put the grade in context.”

The Crimson then mentions some of the things that ‘the full faculty’ considered in their meeting. Someone from A&H (natch) suggested grading simply be abolished, but in the end nothing was decided. Shocking. A faculty meeting in which nothing was decided.

Here’s a fact, from my time in the trenches, that is never mentioned in the article. Giving grades that use the entire grading range, from F to A at Harvard, from 0 to 100 at Western, is hard. It is hard for faculty to give students low marks, partly because they will come to your office and bitch and moan and plead, and their parents may call you (yes, they do) and plead on their behalf, but also because it is not fun. It is not pleasant. Few faculty are sadists, few want to be known as The Grim Reaper of Harvard – or Western.

But the thing is, it is part of the job. That is, it is part of the job to give accurate information to students about their learning, and giving almost everyone an A is not doing that. It is also part of the job to inform the world outside academia how good a student is. Giving almost everyone an A is not doing that, either.

This is, if you ask me – and I know you didn’t, but it’s my damn blog – the fundamental reason why over the last twenty years marks in secondary and post-secondary institutions all over North America have gone up and up and up. Instructors (most of them) don’t want to do that part of their job. And, since they don’t have to, they don’t. Giving lots of As is waaaaay easier on everyone, so that is what happens.

It is the ultimate slogan of the 21st century. ‘Easy is best.’

Standards in general, and in academia in particular, are a keen concern of mine, and I will be writing about them frequently here. This post is about an open letter written by a faculty member in the Department of Computer Science at the University of Regina to the Head of the Department, a letter which he shared in a publication to which I subscribe.

The letter concerns an email sent to all CS students and faculty at UR on August 30, 2023, in which the following sentence appears – “In an effort to provide timely feedback on student work, some of our courses will be moving to a sample-based marking approach.”

The email goes on to explain what that means – that when a student turns in an assignment or test, not all of it will be marked. The parts to be marked will not be revealed until after the marked work has been returned to the students, and their grade will be determined only by that which is marked. So if a test consists of 6 problems, perhaps only three will be marked and feedback provided; the same three for all students.

The reason given for this is the increase in the size of CS classes, driven in turn by an increase in the number of CS majors, said to be nearly 1000 in the email.

The faculty member wrote his open letter to the Head of his Department (Computer Science) decrying this new grading approach, and explaining why he thought it would lead to a decline in academic standards.

I will first just record here that ‘sample-based marking’ as described, is in itself a reduction in academic standards. When I taught at University my assignments and tests were conceived as a whole, different parts of it designed to test different parts of the material, but also different abilities. Some questions could not be answered well without having the ability to write clearly and concisely about something complex, while other questions were designed to test one’s ability to deal with more formal logical or technical issues. To mark or provide feedback on only some aspects of the work is to ignore some part of what the course is about.

I understand well that the idea behind this is that, because the students don’t know up-front which parts of their work will be marked, they still have an incentive to work hard learning all of it. This does not change the fact they will get no feedback on some of their work, a primary point of marking. But in addition, anyone who knows students knows this will lead to a cottage industry in figuring out which parts of work any given instructor is likely to mark, which is not in any way part of what higher education is supposed to teach students.

This policy is, in the end, a further piece of evidence as to what University administrators’ goal is. Get as many students through to a degree as possible, at as low a cost as possible. So far as I can tell, their political masters in Canada are perfectly in agreement with this goal.

This is why sample-based marking is being implemented, rather than the solution suggested by the CS faculty member who objected to it; hiring more faculty to accommodate the growing number of students. Faculty are expensive. And, note that the letter did not indicate that CS students would be seeing a discount on their tuition bill to accompany this sample-based marking initiative.

Imagine a McDonald’s franchise-holder, or local restauranteur, who found themselves with a (delightful) increase in patronage, and responded by filling only part of all food orders, rather than hiring more workers, while charging for everything ordered.

One final note. The original email laying out how this scheme is envisioned working at UR also says that the parts of any student work that are not marked will none the less have solutions posted for the non-marked parts, or they will be gone over in class so as to provide students with the correct solutions. So, there’s your all-around feedback, eh?

Right. In a university atmosphere in which students feel free – indeed, are encouraged – to argue for higher marks for most any reason they can think up, this will open up a whole new area of student appeals. To wit: “ I got the parts of the exam you did not mark nearly perfect, according to your own solution key, so I deserve much more than the 63 I received, which is based solely on the parts you did mark.”…followed by the ever-popular ‘This isn’t fair.’