Software Errors… September 30, 2020
Posted by irishelectionliterature in Uncategorized.trackback
I can’t believe that the Software wasn’t tested properly (over 10% of the students are impacted by these errors), that the Department signed off on the Software having presumably tested it.
The worst part is probably that they have known this for a week. A week? I gather some CAO offers went out late last week. The delay is really quite incredible.
The Cedar Lounge Revolution 
This is a dreadful and preventable mistake to make. Clearly the test cases were inadequate and not properly verified and it may well be a good enough reason for the minister to go but she shouldn’t be alone.
LikeLiked by 3 people
It’s beyond belief. For something so crucial for people’s careers etc. Genuinely shocking.
LikeLike
” For something so crucial for people’s careers”
Maybe that’s the problem we need to look at?
LikeLiked by 2 people
Agree completely. I’ve a real problem with the reification of certain forms of educational process over others.
LikeLiked by 1 person
I don’t often agree with Simon Jenkins but on the money here:
https://www.theguardian.com/commentisfree/2020/oct/02/abolish-school-exams-league-tables-assessment
LikeLike
And of course the greatest cleanest secret of them all… you don’t need a LC to get to college… go as a mature student at 23*…before that get a trade, travel, do community work, live, read, love… etc
* limited numbers but still… isn’t it great?
Son … what was your quote about people being so much more than their culture and who said it? Ta!
LikeLike
‘People are, even now, better than their culture’
Theodor Adorno in Minima Moralia (Late 1940s – maybe now was better then?)
LikeLike
Everything about this is bad. Not least Norma Foley’s casual “I will address the Dail in a few days” at her press conference. That contempt didn’t last too long, I believe she’s in tomorrow afternoon.
Being frank, I think she should be gone. This is a very poor showing. The fact that other ministers have resigned ahead of her might paradoxically save her: they can’t afford another loss of face.
LikeLiked by 2 people
Also 50’000 lines of code isn’t actually that many. And why are Junior Cert results involved in this process at all?!
LikeLiked by 1 person
50,000 lines of new code is very odd given the likely recursive nature of the grading calculations. Unless it was all for reporting and visualisations rather than the calculations.
LikeLike
IEL did a great summary of how a process like this should have been checked and verified on twitter earlier (which is worth repeating here, if you can?). Offering a lay-man’s reading, I think the gist of it was: this isn’t just one of those things that can happen, it isn’t the sort of thing that can’t be predicted, and it is a cock up of unusually large proportions.
LikeLiked by 1 person
And from my reading, by the way, this doesn’t address the issue of how grades were depressed to an unusual extent in high achieving classes. So that may still be working its way through the courts over the next few weeks and could well open up a whole new set of grade-changes.
On other thing occurs to me too. From what I could follow of the whole thing, the examinations commission – who usually run the Leaving and Junior Cert – were not involved in this process. I wonder if they had a quick look at it, and knowing what they do about how difficult it can be to assign grades even when students have sat the exams, decided that they wanted no part of this and slipped quietly away. Leaving the department of education to organise the whole thing.
Nobody could have been prepared for what they had to put together over the summer, but it seems likely to me that the DES were trying to do this without any in-house expertise on grading and without any in-house experience of coding.
LikeLiked by 1 person
LikeLiked by 2 people
IEL, any chance you could copy & paste the thread into a comment here? I deleted my twitter account a while ago so can’t see the thread on that platform, and I’m interested in seeing your thoughts on this, as I’m also a techie as you know, but on the development rather than the testing side of things.
LikeLiked by 1 person
Just threw it up as a post with additional observations
LikeLiked by 1 person
https://weaponsofmathdestructionbook.com/
‘the mathematical models that pervade modern life — and threaten to rip apart our social fabric’
LikeLiked by 1 person
“Algorithms often fail to capture unquantifiable concepts such as workers’ motivation and care, and discriminate against the poor and others who can’t so easily game the metrics…..
“Models are opinions embedded in mathematics,”…
it is important to recognize that the word “meritocracy,” coined by the British sociologist Michael Young in his 1958 book The Rise of the Meritocracy, originally described not some idealized state of perfect fairness, but a cruel dystopia….
A mindset of exhaustive quantified evaluation naturally appeals to elite tech companies like Amazon….
One human-resources executive summed up the office culture, approvingly, as “purposeful Darwinism.”…
“The privileged,” O’Neil notes, “are processed more by people, the masses by machines.”….
When society fetishizes measurement and idolizes individual merit at the expense of other things… it reinforces a go-it-alone mentality that is ultimately harmful to ….egalitarian ideals.
“objective” models provide socially-acceptable excuses to blame certain people—most often, the poor and people of color—for a past that, once digitally noted, is never really forgotten or forgiven….
A norm of constant performance reviews focuses people’s energies not on structural reforms but on the ups and downs of their individual ratings, scores, and tallies. It turns some into modern-day Pharisees—expecting perfection, despising failure, excusing nothing—and deepens the despair of those they scorn”
https://www.theatlantic.com/business/archive/2016/10/extreme-meritocracy/505358/
LikeLiked by 1 person
It’s curious how the concept of meritocracy, which Young thought was a very bad thing, has come to be seen as desirable across the political spectrum.
LikeLiked by 1 person
” Education theorist Bill Ayers, commenting on the limitations of standardized testing, writes that “Standardized tests can’t measure initiative, creativity, imagination, conceptual thinking, curiosity, effort, irony, judgment, commitment, nuance, good will, ethical reflection, or a host of other valuable dispositions and attributes. What they can measure and count are isolated skills, specific facts and function, content knowledge, the least interesting and least significant aspects of learning.”
https://en.wikipedia.org/wiki/Meritocracy
LikeLike
Toby Young is the son of Michael Dunlop Young, which perhaps disproves the notion that certain traits are inherited.
LikeLiked by 1 person
“The errors are being rectified and the process is being re-checked with a series of independent checks now underway.
A US firm ETS Educational Testing Service, which specialises in educational measurement, is reviewing aspects of the coding.”
https://www.irishtimes.com/news/education/leaving-cert-q-a-how-did-the-error-happen-and-will-students-get-new-cao-offers-1.4368792
‘ETS has been criticized for being a “highly competitive business operation that is as much multinational monopoly as nonprofit institution”
Problems administering England’s national tests in 2008 by ETS Europe were the subject of thousands of complaints recorded by the Times Educational Supplement.[42] Their operations were also described as a “shambles” in the UK Parliament, where a financial penalty was called for…..
(AETR) claims that ETS is violating its non-profit status through excessive profits, executive compensation, and governing board member pay ”
https://en.wikipedia.org/wiki/Educational_Testing_Service
LikeLiked by 1 person
“Universities have warned that they do not have the capacity to offer places to all students who may receive a new CAO offer on foot of upgraded Leaving Certificate results.”
Another round of court hearings for the state to drag themselves into.
LikeLike
“At a press conference with the minister yesterday, it was confirmed that the “original value” of the contract with the Canadian company was €75,000.
An official from the Department of Education said that “the expenditure to date” is €160,000.”
https://www.thejournal.ie/calculated-grades-contract-5219348-Oct2020/
To be honest that doesn’t sound like an awful lot of money to develop and test 50k lines of code. Now I’m a hardware rather than a software developer myself so I could be wrong, but I’d have imagined that with the type of loaded labour rates in the industry, and the likely profit margins, that 75k or even 160k would map to less than a person year of effort. I mean 50k lines of code, in 250 workdays, 200 lines of code per day, developed and tested? Ambitious.
LikeLike
Where was the quality control and checking? Surely running a few known ones through the system would have turned up errors.
LikeLike
I’ll defer to IEL’s expertise in this area but from observing one of my colleagues who’s a software developer it seems that test coverage is something that’s not straightforward to measure or achieve. There are tools which can automate it but to go back to IEL’s comment above about the Product Requirements Document the difficulty can be in correlating the tests against the requirements. You can relatively easily automate the checks on whether or not all lines of code are exercised during the testing, but it’s not so easy to automate whether the testing is exercising all of the requirements in their various combinations.
LikeLike
And of course the greatest cleanest secret of them all… you don’t need a LC to get to college… go as a mature student at 23*…before that get a trade, travel, do community work, live, read, love… etc
* limited numbers but still… isn’t it great?
Son … what was your quote about people being so much more than their culture and who said it? Ta!
LikeLike
Folks the 50,000 lines is a number sent out to bamboozle and add complexity to a situation.
It is almost certain that there was no 50,000 lines of new code written by someone or anyone for this.
At it’s core this should have required a level of iterative logic that would be barely in the few thousand lines if even that with standard libraries used to do all the usual stuff of data load and validation and generating the output in whatever the desired format was.
And lines of code is a metric that isn’t used to measure much in software since the days of KLoCs and IBM using them as method of paying for work. For Christ’s sake, people wrote games with 48K of memory in the 80s that were fantastic for the time (still are in many cases), and it was the amount of lines they wrote that matter.
LikeLike
“wasn’t the amount” sorry.
LikeLike
Two lessons, maybe?
1. Outsourcing isn’t always a good idea.
2. Neither are algorithms.
LikeLiked by 2 people
I agree with your point about out-sourcing but I wonder does the expertise exist within the Department, or within any sector of the civil service, to undertake and deliver this type of project in-house? You could argue of course that the expertise didn’t exist in the out-source shop either, but that’s at a different level. Of course the expertise does exist within the wider public service, most obviously within the third level education sector, but with the all pervasive neoliberal model of putting everything out to tender I doubt that any entity within third level would have the capability to bid. Unless they did it as a commercial third party, so back to square one really, the private sector and market pricing for work. You’d imagine that a body like the OPW might be a suitable vehicle for amassing the sort of technical know how required to do these types of projects, if that was part of their role?
About your second point, I realize you’re being flippant but as someone who earns their living working in the engineering industry, and not having had a good night’s sleep last night and therefore being a bit grumpy over my porridge this morning, I’d prefer if people didn’t dismiss the efforts that scientists and engineers make to keeping society and the economy running. Taking the piss out of maths is dismissive of that effort. I’d like to see how well you’d be getting on without relying on algorithms. Where’s my tea, another strong mug please.
LikeLike
” You’d imagine that a body like the OPW might be a suitable vehicle for amassing the sort of technical know how required to do these types of projects, if that was part of their role?”
I know of an OPW site where, until recently enough, bookings were taken by fax and the internet was turned off at the weekend…so ‘technical know-how’? Possibly not.
LikeLike
Fair enough, SoS, I can imagine that happening given the nature of many of the sites managed by the OPW, but I was thinking more so of the core competency aspect within the organization, which would be civil engineering, and therefore it wouldn’t be a huge stretch to use that project-oriented approach to build a parallel competency in computer engineering or IT in general. In a separate tack, given the importance of all things IT to the functioning of the state, society at large, and the economy generally, perhaps it’s time for nation states or supranational entities to usurp the role of transnational capitalism and take over the management of the IT sector on behalf of the common good? I know it’s a bit of a stretch but there are historical examples of this sort, in transportation, in electrical power, when those technologies were the newly disruptive forces driving progress to the next step. We take it for granted that IT is what it is but is it really so unimaginable that it could be completely nationalised? As a side note, I’ve just recently decided to end my decade long presence on Facebook, mostly because of the issues around manipulation and addiction that are covered in Susanna Zubhof’s (spelling?) book The Age Of Surveillance Capitalism and the Netflix film The Social Dilemma. The technology just isn’t serving people well, and that will never change until the profit motive underlying it is eliminated and replaced by a different set of cultural values and a different economic model for the industry and the organizations that serve it.
LikeLike
Well, note the word ‘always’ in my flippant/not flippant comment, alanmyler.
There were many other ways to deal with the Leaving Cert results issue this year – without engaging private sector spoofers and without using algorithms.
Unfortunately, the political leadership and probably the Dept of Ed Civil Service leadership decided it would look like they were really taking it very seriously if they bought in outside ‘expertise’ who it turned out weren’t very good at creating algorithms.
LikeLike
There was a splatter of porridge on my screen over the word “always” in your original comment Joe, apologies.
But about algorithms. Any decision making process is an algorithm. Whether done manually or automated using IT. I’m not sure that I get what you’re proposing as an alternative here. Not about the outsourcing question, as I say I agree with you about that, buy about the decision making process?
LikeLike
Alan. I’m not proposing anything. Haven’t got the time or the inclination to examine the whole thing to the extent that I’d have a proposal.
This is why it’s a good idea to get off FB and the internet generally – it’s full of the likes of me, banging off criticisms without putting forward alternatives. I may join you in getting off FB indeed.
I do think that there can be a tendency to make things, or at least to make things appear, more complicated than they need to be. I’m not a luddite. More a supporter of Shaw’s line about professions being a conspiracy against the layperson.
Nothing personal against IT workers but I reckon a good working group of civil servants, teacher reps, student reps and parent reps could have come up with a process to allocate Leaving Cert results this year, without automated algorithms.
Easy for me to type that last paragraph. Wouldn’t have been so easy for that hypothetical working group.
LikeLiked by 2 people
Many thanks Son… appreciate it…
Fergal
LikeLike
At it’s core this should have required a level of iterative logic that would be barely in the few thousand lines if even that with standard libraries used to do all the usual stuff of data load and validation and generating the output in whatever the desired format was.
LikeLike