jump to navigation

Software Errors… September 30, 2020

Posted by irishelectionliterature in Uncategorized.
trackback

The news today that there were two massive errors in the calculated grades system calculations is not unpexected

I can’t believe that the Software wasn’t tested properly (over 10% of the students are impacted by these errors), that the Department signed off on the Software having presumably tested it.

The worst part is probably that they have known this for a week. A week? I gather some CAO offers went out late last week. The delay is really quite incredible.

Comments»

1. Daniel Sullivan - September 30, 2020

This is a dreadful and preventable mistake to make. Clearly the test cases were inadequate and not properly verified and it may well be a good enough reason for the minister to go but she shouldn’t be alone.

Liked by 3 people

2. WorldbyStorm - September 30, 2020

It’s beyond belief. For something so crucial for people’s careers etc. Genuinely shocking.

Like

sonofstan - October 1, 2020

” For something so crucial for people’s careers”

Maybe that’s the problem we need to look at?

Liked by 2 people

WorldbyStorm - October 2, 2020

Agree completely. I’ve a real problem with the reification of certain forms of educational process over others.

Liked by 1 person

sonofstan - October 2, 2020

I don’t often agree with Simon Jenkins but on the money here:
https://www.theguardian.com/commentisfree/2020/oct/02/abolish-school-exams-league-tables-assessment

Like

Fergal - October 2, 2020

And of course the greatest cleanest secret of them all… you don’t need a LC to get to college… go as a mature student at 23*…before that get a trade, travel, do community work, live, read, love… etc
* limited numbers but still… isn’t it great?
Son … what was your quote about people being so much more than their culture and who said it? Ta!

Like

sonofstan - October 2, 2020

‘People are, even now, better than their culture’
Theodor Adorno in Minima Moralia (Late 1940s – maybe now was better then?)

Like

3. NFB - September 30, 2020

Everything about this is bad. Not least Norma Foley’s casual “I will address the Dail in a few days” at her press conference. That contempt didn’t last too long, I believe she’s in tomorrow afternoon.

Being frank, I think she should be gone. This is a very poor showing. The fact that other ministers have resigned ahead of her might paradoxically save her: they can’t afford another loss of face.

Liked by 2 people

NFB - September 30, 2020

Also 50’000 lines of code isn’t actually that many. And why are Junior Cert results involved in this process at all?!

Liked by 1 person

Daniel Sullivan - September 30, 2020

50,000 lines of new code is very odd given the likely recursive nature of the grading calculations. Unless it was all for reporting and visualisations rather than the calculations.

Like

4. 6to5against - September 30, 2020

IEL did a great summary of how a process like this should have been checked and verified on twitter earlier (which is worth repeating here, if you can?). Offering a lay-man’s reading, I think the gist of it was: this isn’t just one of those things that can happen, it isn’t the sort of thing that can’t be predicted, and it is a cock up of unusually large proportions.

Liked by 1 person

6to5against - September 30, 2020

And from my reading, by the way, this doesn’t address the issue of how grades were depressed to an unusual extent in high achieving classes. So that may still be working its way through the courts over the next few weeks and could well open up a whole new set of grade-changes.

On other thing occurs to me too. From what I could follow of the whole thing, the examinations commission – who usually run the Leaving and Junior Cert – were not involved in this process. I wonder if they had a quick look at it, and knowing what they do about how difficult it can be to assign grades even when students have sat the exams, decided that they wanted no part of this and slipped quietly away. Leaving the department of education to organise the whole thing.

Nobody could have been prepared for what they had to put together over the summer, but it seems likely to me that the DES were trying to do this without any in-house expertise on grading and without any in-house experience of coding.

Liked by 1 person

irishelectionliterature - September 30, 2020

Liked by 2 people

alanmyler - October 1, 2020

IEL, any chance you could copy & paste the thread into a comment here? I deleted my twitter account a while ago so can’t see the thread on that platform, and I’m interested in seeing your thoughts on this, as I’m also a techie as you know, but on the development rather than the testing side of things.

Liked by 1 person

irishelectionliterature - October 1, 2020

Just threw it up as a post with additional observations

Liked by 1 person

5. CL - September 30, 2020

https://weaponsofmathdestructionbook.com/

‘the mathematical models that pervade modern life — and threaten to rip apart our social fabric’

Liked by 1 person

CL - October 1, 2020

“Algorithms often fail to capture unquantifiable concepts such as workers’ motivation and care, and discriminate against the poor and others who can’t so easily game the metrics…..
“Models are opinions embedded in mathematics,”…
it is important to recognize that the word “meritocracy,” coined by the British sociologist Michael Young in his 1958 book The Rise of the Meritocracy, originally described not some idealized state of perfect fairness, but a cruel dystopia….
A mindset of exhaustive quantified evaluation naturally appeals to elite tech companies like Amazon….
One human-resources executive summed up the office culture, approvingly, as “purposeful Darwinism.”…
“The privileged,” O’Neil notes, “are processed more by people, the masses by machines.”….
When society fetishizes measurement and idolizes individual merit at the expense of other things… it reinforces a go-it-alone mentality that is ultimately harmful to ….egalitarian ideals.

“objective” models provide socially-acceptable excuses to blame certain people—most often, the poor and people of color—for a past that, once digitally noted, is never really forgotten or forgiven….
A norm of constant performance reviews focuses people’s energies not on structural reforms but on the ups and downs of their individual ratings, scores, and tallies. It turns some into modern-day Pharisees—expecting perfection, despising failure, excusing nothing—and deepens the despair of those they scorn”
https://www.theatlantic.com/business/archive/2016/10/extreme-meritocracy/505358/

Liked by 1 person

sonofstan - October 1, 2020

It’s curious how the concept of meritocracy, which Young thought was a very bad thing, has come to be seen as desirable across the political spectrum.

Liked by 1 person

CL - October 1, 2020

” Education theorist Bill Ayers, commenting on the limitations of standardized testing, writes that “Standardized tests can’t measure initiative, creativity, imagination, conceptual thinking, curiosity, effort, irony, judgment, commitment, nuance, good will, ethical reflection, or a host of other valuable dispositions and attributes. What they can measure and count are isolated skills, specific facts and function, content knowledge, the least interesting and least significant aspects of learning.”
https://en.wikipedia.org/wiki/Meritocracy

Like

CL - October 1, 2020

Toby Young is the son of Michael Dunlop Young, which perhaps disproves the notion that certain traits are inherited.

Liked by 1 person

6. CL - September 30, 2020

“The errors are being rectified and the process is being re-checked with a series of independent checks now underway.
A US firm ETS Educational Testing Service, which specialises in educational measurement, is reviewing aspects of the coding.”
https://www.irishtimes.com/news/education/leaving-cert-q-a-how-did-the-error-happen-and-will-students-get-new-cao-offers-1.4368792

‘ETS has been criticized for being a “highly competitive business operation that is as much multinational monopoly as nonprofit institution”
Problems administering England’s national tests in 2008 by ETS Europe were the subject of thousands of complaints recorded by the Times Educational Supplement.[42] Their operations were also described as a “shambles” in the UK Parliament, where a financial penalty was called for…..
(AETR) claims that ETS is violating its non-profit status through excessive profits, executive compensation, and governing board member pay ”
https://en.wikipedia.org/wiki/Educational_Testing_Service

Liked by 1 person

7. NFB - October 1, 2020

“Universities have warned that they do not have the capacity to offer places to all students who may receive a new CAO offer on foot of upgraded Leaving Certificate results.”

Another round of court hearings for the state to drag themselves into.

Like

8. alanmyler - October 1, 2020

“At a press conference with the minister yesterday, it was confirmed that the “original value” of the contract with the Canadian company was €75,000.

An official from the Department of Education said that “the expenditure to date” is €160,000.”

https://www.thejournal.ie/calculated-grades-contract-5219348-Oct2020/

To be honest that doesn’t sound like an awful lot of money to develop and test 50k lines of code. Now I’m a hardware rather than a software developer myself so I could be wrong, but I’d have imagined that with the type of loaded labour rates in the industry, and the likely profit margins, that 75k or even 160k would map to less than a person year of effort. I mean 50k lines of code, in 250 workdays, 200 lines of code per day, developed and tested? Ambitious.

Like

Jim Monaghan - October 1, 2020

Where was the quality control and checking? Surely running a few known ones through the system would have turned up errors.

Like

alanmyler - October 1, 2020

I’ll defer to IEL’s expertise in this area but from observing one of my colleagues who’s a software developer it seems that test coverage is something that’s not straightforward to measure or achieve. There are tools which can automate it but to go back to IEL’s comment above about the Product Requirements Document the difficulty can be in correlating the tests against the requirements. You can relatively easily automate the checks on whether or not all lines of code are exercised during the testing, but it’s not so easy to automate whether the testing is exercising all of the requirements in their various combinations.

Like

Fergal - October 2, 2020

And of course the greatest cleanest secret of them all… you don’t need a LC to get to college… go as a mature student at 23*…before that get a trade, travel, do community work, live, read, love… etc
* limited numbers but still… isn’t it great?
Son … what was your quote about people being so much more than their culture and who said it? Ta!

Like

Daniel Sullivan - October 1, 2020

Folks the 50,000 lines is a number sent out to bamboozle and add complexity to a situation.

It is almost certain that there was no 50,000 lines of new code written by someone or anyone for this.

At it’s core this should have required a level of iterative logic that would be barely in the few thousand lines if even that with standard libraries used to do all the usual stuff of data load and validation and generating the output in whatever the desired format was.

And lines of code is a metric that isn’t used to measure much in software since the days of KLoCs and IBM using them as method of paying for work. For Christ’s sake, people wrote games with 48K of memory in the 80s that were fantastic for the time (still are in many cases), and it was the amount of lines they wrote that matter.

Like

Daniel Sullivan - October 1, 2020

“wasn’t the amount” sorry.

Like

9. Joe - October 1, 2020

Two lessons, maybe?
1. Outsourcing isn’t always a good idea.
2. Neither are algorithms.

Liked by 2 people

alanmyler - October 2, 2020

I agree with your point about out-sourcing but I wonder does the expertise exist within the Department, or within any sector of the civil service, to undertake and deliver this type of project in-house? You could argue of course that the expertise didn’t exist in the out-source shop either, but that’s at a different level. Of course the expertise does exist within the wider public service, most obviously within the third level education sector, but with the all pervasive neoliberal model of putting everything out to tender I doubt that any entity within third level would have the capability to bid. Unless they did it as a commercial third party, so back to square one really, the private sector and market pricing for work. You’d imagine that a body like the OPW might be a suitable vehicle for amassing the sort of technical know how required to do these types of projects, if that was part of their role?

About your second point, I realize you’re being flippant but as someone who earns their living working in the engineering industry, and not having had a good night’s sleep last night and therefore being a bit grumpy over my porridge this morning, I’d prefer if people didn’t dismiss the efforts that scientists and engineers make to keeping society and the economy running. Taking the piss out of maths is dismissive of that effort. I’d like to see how well you’d be getting on without relying on algorithms. Where’s my tea, another strong mug please.

Like

sonofstan - October 2, 2020

” You’d imagine that a body like the OPW might be a suitable vehicle for amassing the sort of technical know how required to do these types of projects, if that was part of their role?”

I know of an OPW site where, until recently enough, bookings were taken by fax and the internet was turned off at the weekend…so ‘technical know-how’? Possibly not.

Like

alanmyler - October 2, 2020

Fair enough, SoS, I can imagine that happening given the nature of many of the sites managed by the OPW, but I was thinking more so of the core competency aspect within the organization, which would be civil engineering, and therefore it wouldn’t be a huge stretch to use that project-oriented approach to build a parallel competency in computer engineering or IT in general. In a separate tack, given the importance of all things IT to the functioning of the state, society at large, and the economy generally, perhaps it’s time for nation states or supranational entities to usurp the role of transnational capitalism and take over the management of the IT sector on behalf of the common good? I know it’s a bit of a stretch but there are historical examples of this sort, in transportation, in electrical power, when those technologies were the newly disruptive forces driving progress to the next step. We take it for granted that IT is what it is but is it really so unimaginable that it could be completely nationalised? As a side note, I’ve just recently decided to end my decade long presence on Facebook, mostly because of the issues around manipulation and addiction that are covered in Susanna Zubhof’s (spelling?) book The Age Of Surveillance Capitalism and the Netflix film The Social Dilemma. The technology just isn’t serving people well, and that will never change until the profit motive underlying it is eliminated and replaced by a different set of cultural values and a different economic model for the industry and the organizations that serve it.

Like

Joe - October 2, 2020

Well, note the word ‘always’ in my flippant/not flippant comment, alanmyler.

There were many other ways to deal with the Leaving Cert results issue this year – without engaging private sector spoofers and without using algorithms.
Unfortunately, the political leadership and probably the Dept of Ed Civil Service leadership decided it would look like they were really taking it very seriously if they bought in outside ‘expertise’ who it turned out weren’t very good at creating algorithms.

Like

alanmyler - October 2, 2020

There was a splatter of porridge on my screen over the word “always” in your original comment Joe, apologies.

But about algorithms. Any decision making process is an algorithm. Whether done manually or automated using IT. I’m not sure that I get what you’re proposing as an alternative here. Not about the outsourcing question, as I say I agree with you about that, buy about the decision making process?

Like

Joe - October 2, 2020

Alan. I’m not proposing anything. Haven’t got the time or the inclination to examine the whole thing to the extent that I’d have a proposal.
This is why it’s a good idea to get off FB and the internet generally – it’s full of the likes of me, banging off criticisms without putting forward alternatives. I may join you in getting off FB indeed.
I do think that there can be a tendency to make things, or at least to make things appear, more complicated than they need to be. I’m not a luddite. More a supporter of Shaw’s line about professions being a conspiracy against the layperson.
Nothing personal against IT workers but I reckon a good working group of civil servants, teacher reps, student reps and parent reps could have come up with a process to allocate Leaving Cert results this year, without automated algorithms.
Easy for me to type that last paragraph. Wouldn’t have been so easy for that hypothetical working group.

Liked by 2 people

10. Fergal - October 2, 2020

Many thanks Son… appreciate it…
Fergal

Like

11. rahat345678 - October 5, 2020

At it’s core this should have required a level of iterative logic that would be barely in the few thousand lines if even that with standard libraries used to do all the usual stuff of data load and validation and generating the output in whatever the desired format was.

Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: