Epic Fail (2)

I doubt an esteemed professor of education would use the term 'Epic Fail' to describe the Scottish Government's performance over this year's school exam results.

Yet Lindsay Paterson, professor of education policy at Edinburgh University, still concludes that Scotland's school children have been failed by a system which adjusted exam grades in a way that has 'opened up an avalanche of unfairness.   

Read Professor Parterson's comment piece via the link below to The Times. 

  

https://www.thetimes.co.uk/edition/scotland/scottish-qualifications-authority-has-failed-children-in-every-way-c99zm8ggc

TIMES COMMENT - EXAM RESULTS

Scottish Qualifications Authority has failed children in every way

The SQA has been unfair to poorer pupils in adjusting grades and it refused to show its working until it was too late


By Lindsay Paterson - The Times

Hundreds of thousands of school pupils have their results and Scotland’s exam body has invited us to judge its actions this year by three criteria: fairness, integrity, and safety. After yesterday’s announcement, the verdict has to be that the Scottish Qualifications Authority (SQA) failed on all three.

Fairness refers to the central principle of any system of national assessment: that every student should have the same opportunities to show their potential. There are severe doubts that this was achieved. To see why, we have to start at the beginning. When the exams were cancelled, the SQA decided to ask teachers to estimate what each student would have achieved. That was itself dubious. Much research has shown that none of us, as teachers, are very good at reliably estimating our own students’ grades. That is why we have standardised assessments in the first place.

And that is how it has turned out, in the light of the SQA’s next step. They compared teachers’ estimates last year with the outcomes, and found that fewer than half of teachers’ estimates were confirmed by the eventual result. This led to the crucial decision to “moderate” this year’s estimates. On that process, everything else announced yesterday depends, so the technical details matter enormously, obscured though they are by the SQA’s turgid prose.

Essentially what they have done is to adjust the teacher estimates to be in line with the recent record of their school. The SQA does allow for random variation from year to year, so that the distribution of this year’s results does not have to be precisely in line with last year’s, but the core idea is that results this year have not been allowed to drift too far away from where they have been for the past few years.

This has opened up an avalanche of unfairness. A quarter of all estimates were adjusted, and many more were reduced than increased. This might not matter too much if the mis-estimates were random, but the SQA’s own data shows that the reductions have affected students from poor areas more than from richer ones. Whether that’s because the adjustments are biased against poverty or teachers biased the other way, these patterns do indicate that fairness has not been achieved.

There’s a further aspect of unfairness. The adjustment method makes it better to be outstanding in a small school or an unpopular subject than in contexts where you are compared with many more people. This is because the degree of random variation which the SQA has permitted is wider where the data is sparser. Their document is infuriatingly vague about the extent of this problem, but they do report one striking set of facts.

At National 5 level, nearly two thirds of students are in subject groups of 25 or fewer. At Higher, the proportion is three quarters, and at Advanced Higher it is 98 per cent. If you’re unlucky enough to be in the minority who are in large groups, then your teacher’s high estimate for you would be more likely to be pulled down.

That’s capricious (and would suggest that students should take minority subjects in case this whole technique is applied again next year). And it’s all on top of the unreliability of the teacher rankings in the first place, which lie at the heart of the SQA’s model and which are treated as, in their words, “sancrosanct”. No matter how the adjustments went, you could never this year overtake your higher-placed peers in your school. Where you were in March is now your destiny.

So that’s why fairness has not been achieved. On integrity, perhaps if the SQA really had shown a sufficient awareness of what that requires in practice, some of these data problems would have been spotted in advance, but from April onwards they have been relentlessly opaque. Having suggested to the Scottish parliament that they would explain their methods and discuss adjustments with teachers, they changed tack in late June after everyone had gone on holiday. Their statistical methods were released only yesterday, two hours after students had received their awards.

The SQA claims administrative difficulties, but that doesn’t explain why their performance has been so much less transparent than the analogous examining bodies in England. There, a Commons inquiry was able to stimulate a range of well-informed discussions.

Of the SQA’s three principles, that leaves safety. One interpretation of that is the safety of young people’s future. So many estimates have been reduced by the SQA’s adjustments that there is likely to be a flood of appeals. These students will not be likely to feel safe about their future prospects until they are reassured that they have been awarded what their previous attainment would suggest they are capable of.

Safety might also mean health, holding out the prospect that next year’s exams could be cancelled as well. That prospect is dismaying. At the very least, though, there would have to be a great deal of work done to improve the SQA’s system. This year’s students have been let down. It ought to be unthinkable that this should happen again.

Lindsay Paterson is professor of education policy at the University of Edinburgh.

Epic Fail! (05/08/20)

Most of the daily newspapers agree that the Scottish Government has made a real hash of this year's school exam results, but the Daily Record has the best front page by far. 

One high-flying student from St Andrew's High School in Coatbridge went from a predicted A to an F - hence the Daily Record's headline with the play on the words 'Epic Fail.