Rubenerd

Algorithms determine more of our lives than I think people realise. It’s hard to think of anything that hasn’t had machine learning or even a cold decision tree affect it somewhere along the way, and the trend is only accelerating with ubiquitous compute power, storage, and capture devices.

Gabby Bush, Henrietta Lyons, and Tim Miller wrote in the University of Melbourne’s Pursuit magazine about the limitations of these systems for predicting grades in the UK, but it’s questions on ethics and accuracy are broadly applicable. This paragraph in section two so elegantly summarised the promise:

Algorithms, in their most utopian sense, could ensure that decision-making is more efficient, fair and consistent. Algorithms could, if designed and regulated with ethical considerations in mind, make decisions more objective and less biased.

With ethical considerations in mind being the operative phrase. They expanded a few paragraphs down:

[..] predictive algorithms don’t account for a multitude of factors that might determine a person’s success.

Nor do I think they ever could. Algorithms inherently carry the acknowledged or unknown biases of their developers and the datasets they’re trained against. No developer can be perfecly neutral, and data cannot have perfect coverage. We don’t even fully understand all the factors in a decision or event; algorithms can help us surface them, but then we’re potentially falling into the same trap. There are potential legal implications, and we may not approve of algorithmic outcomes on the human values of equality and fairness.

Algorithms, when applied appropriately, can save time and even lives. But neither negate the need for human oversight and careful application. You’d think this would all be obvious in 2020, and yet here we are:

Ultimately, the UK government backed down and students’ results will now be based on the grades that were predicted by teachers based on their students’ progress. However, this is the latest disturbing precedent of how governments’ careless use of algorithmic decision-making can have devastating consequences for citizens and particular social groups.

It’s our responsibility in this industry to keep sight of this, and inform decision makers. They can appeal to ignorance; we can’t.