The Yorkshire Whistler Wrap
A summary of the 2024-25 season, as Willie Collum reshaped how the SFA refereeing service communicated with supporters.
This was the fourth season I had an independent, South Yorkshire-based, Sheffield Wednesday supporting, professional, and practising referee adjudicate on the big calls impacting the matches of Celtic and The Rangers throughout the league season.
The Yorkshire Whistler analysis – he studies footage of key incidents (goals scored, disallowed, sending’s off, etc) and provides expert analysis and explanation as to what should have been the correct call.
If the officials are deemed to have made a big call incorrectly, an expected points model, based on the score and the game time, calculates a projected impact of the wrong call in terms of points lost or gained. The precision of the expected points model is less relevant than the fact both clubs are judged equally by it, and therefore, on a relative basis, the results are comparable.
This forms part of a wider analysis driven by public data into Scottish refereeing outcomes. Today we are concerned with the Yorkshire Whistler project only, and I will recap on where this leaves us apropos Scottish refereeing in a later piece.
Error Rate
The first aspect to note is simply the number of incidents referred for review. And then the percentage of errors uncovered by the Yorkshire Whistler.
The number of issues forwarded for review was increasing year on year under the management of refereeing from the previous incumbent, Crawford Allan. It reached 124 last season, in effectively 76 matches.
This period has seen the implementation of Virtual Assistant Referee (VAR) into Scottish football’s topflight, and the error rate has been falling from a remarkable high of 32% in 2021-22.
Under Willie Collum’s stewardship of the Scottish Football Association (SFA) refereeing service, the error rate has fallen to 13% from a massively reduced referral rate of only 47 incidents in 76 matches by the top two clubs.
Fewer contentious issues post-match, and fewer errors identified. S’all good man!
Expected Points Impact
Alongside any public data applying objective analysis you wish to do on major calls in Scotland’s top tier over the last four years, including the Yorkshire Whistler independent analysis, one team benefited in a statistically significant way. That being The Rangers. Meaning all other teams in the league were being unexpectedly penalised to some extent. A clear pattern of assistance for one side.
This analysis focuses on the top two, given the historical gaps back to the third and beyond.
Back to the YW analysis only.
Here is the updated pattern.
There were clear gaps of over four, then six, then three points in favour of The Rangers. Meaning wrong decisions impacting Celtic were estimated to have had a negative impact on points and a positive impact on the Ibrox club.
This season has seen the smallest gap between the sides – 1.29 expected points, and the first time in Celtic’s favour.
Of the seven wrong calls impacting Celtic, four harmed their chances and three benefited, but the aggregate expected points total was a net benefit for the Champions.
The Rangers saw two calls in their favour and two that were harmful. The net effect as regards expected points was negative.
Taken together, this is what a “normal” distribution should look like. Over 38 matches, neither side benefited nor was penalised to any great extent. The ol’ “it all evens up over a season” appears to be working and in good order.
Is this a one-season blip, or has something changed?
Wonky Willie and the Refereeing Factory
As a referee, Willie Collum was as poor as any of them, given the part-time role in a full-time professional sport. Famously able to detect penalties with his back turned, in truth, there isn’t a club in the land without a Willie Collum sob story. Collum liked to distribute his poor decision-making equally across the league.
And despite his skills as a referee, as an administrator and leader of the refereeing services, he is perhaps following a similar path.
But all good-natured knockabout bantz aside, Collum has made some important changes.
Firstly, he publishes the results of an independent panel of “experts” review of VAR calls for all supporters to peruse. The Independent Review Panel (IRP) concluded only 26 out of 1181 VAR reviews were deemed incorrect. A much higher success rate than the Yorkshire Whistler, although the parameters of operation are different, with the IRP having the VAR evidence to hand. This transparency is welcome and highlights how different “experts” can view the same incident very differently. These opinions are directional, not definitive.
The definitive view of how an incident should have been refereed is reserved for Collum and his match assessors.
And so Collum takes to the airwaves every month with BBC football correspondent Gordon Duncan on the VAR Review Show on Youtube.
If you are a referee, you DO NOT want to be publicly upbraided by your boss on TV. The performance of Greg Aitken in Celtic’s opening win at home to Kilmarnock set the tone. Aitken’s commentary whilst adjudicating on VAR in response to Kyogo Furuhashi being taken out by a Kilmarnock player sounded more like a fan of Celtic’s opposition, dismissing any claim for foul play.
Card marked, the season soon settled down with errors made, for sure, but they appeared to impact all clubs more or less equally. In other words, the standard of refereeing may have improved somewhat due to the VAR second chance, but the continuing errors appeared to be randomly distributed. Something that had not happened under the closed-shop approach of Crawford Allan.
There is nothing like transparency and increased public scrutiny to moderate performance. The referees have not changed. Their unconscious biases have not changed and remain largely uncontrolled. And the culture of the SFA has not changed. But refereeing behaviour does appear to have changed due to this increased transparency.
Whether standards have improved is one for the refereeing experts; all we ask for as fans is a level playing field.
It is too early to declare victory on that. As is normal, more data is needed!
But Collum’s transparency initiatives do seem to have moderated the worst excesses of the Scottish refereeing fraternity. Collum deserves great credit for radically changing how the SFA refereeing service communicates with fans. Now to improve service quality.
The YW project has, in my view, been revolutionary in demonstrating that cool analysis could be used to provide a sound basis for discussion of a topic that has for decades been ill-served by obfuscation, misinformation and confusion.
It is outstanding simply as a piece of journalism. It shows what can be done with an even hand, diligence and a proper respect for facts. The use of video, expert analysis and data analytics contrasts sharply with the argy-bargy and to-my-mind-he's-offside waffle we endure from our sports broadcasters.
It would have been easy to shy away from the complexity of the use of Expected Points s a metric. Probabilistic parameters are a ready target for the ignorant or those who risk being exposed as lazy or worse. It's to your credit that you stuck with it.
The results are a triumph. You've tracked and measured the - positive - impacts of VAR and the new transparency regime. Willie Collum should be seeking you out and thanking you.
It's not the first time the alternative media has produced work of a quality that puts traditional sources to shame, and it almost certainly won't be the last, but it's a piece of work that raises the standard of our discourse and it deserves the highest praise.
Alan, how did the error rate vary over the course of the season? My feeling last year was the refs were decent for the first 2-3 months of the season then started to make some mad calls during the busy Nov-Dec period where it’s two games a week.
It’s maybe something the SFA need to consider, given refs are part time, whether they should be given two games a week on repeat like that on top of being full time professionals with families.