I really have a hard time deciding if that is the scandal the article makes it out to be (although there is some backpedaling going on). The crucial point is: 8% of the decisions turn out to be wrong or misjudged. The article seems to want us to think that the use of the algorithm is to blame. Yet, is it? Is there evidence that a human would have judged those cases differently? Is there evidence that the algorithm does a worse job than humans? If not, then the article devolves onto blatant fear mongering and the message turns from "algorithm is to blame for deaths" into "algorithm unable to predict the future in 100% of cases", which of course it can't...

The algorithm itself is just a big "whatever". The key issue here is that some assumptive piece of shit decided to conclude, based on partial information, that those women would be safe in the future.

The police accepted the software’s judgment and Ms. Hemid went home with no further protection.

This is what happens when you rely on your Nintendos, instead of using your damn brains.

The way to use these kinds of systems is to have the judge came to an independent decision, then, after that's keyed in, the AI spits out theirs and whichever predicts more danger is then acted on.

Relatedly, the way you have an AI select people and companies to get spot-checked by tax investigators is not to show investigators the AI scores, but mix in AI suspicions among a stream of randomly selected people.

Relatedly, the way you have AI involved in medical diagnoses is not to tell the human doctor results, but suggest additional tests to be made. The "have you ruled out lupus" approach.

And from what I've heard the medical profession actually got that right from the very beginning. They know what priming and bias is. Law enforcement? I fear we'll have to ELI5 them the basics for the next five hundred years.

I remember years ago when they said the value of our lives would be determined by a panel of people.

Now its by a machine.

The algorithm:

isSafe = random();

if isSafe >.5 println ("everything is fine\n");

Having worked in making software for almost 3 decades, including in Finance both before and after the 2008 Crash, this blind reliance on algorithms for law enforcement and victim protection scares the hell out of me.

An algorithm is just an encoding of whatever the people who made it think will happen: it's like using those actual people directly, only worse because by need an algorithm has a fixed set of input parameters and can't just ask more questions when something "smells fishy" as a person would.

Also making judgements by "entering something in a form" has a tendency to close people's thinking - instead of pondering on it and using their intuition to, for example, notice from the way people are talking that they're understating the gravity of the situation, people filling form tend to mindlessly do it like a box-ticking exercise - and that's not even going into the whole "As long as I just fill the form my ass is covered" effect when the responsability is delegated to the algorithm that leads people to play it safe and not dispute the results even when their instincts say otherwise.

For anybody who has experience with modelling, using computer algorithms within human processes and with how users actually treat such things (the "computer says" effect) this shit really is scary at many levels.

Computers are only at fault when its convenient to blame them.

Advocates: take survivors of abuse seriously.
Society: Let's have computers tell us what to do!

I mean I guess the risk of repeated murder-suicide is pretty low...

Why do we live in a dystopian hellscape

In the late 1970s (I was a kid) the computer is always right was a common sarcastic parody of all the people who actually believed it.

We'd discover in the 1980s it was possible to have missing data, insufficient data or erroneous data.

About 20 new cases of gender violence arrive every day, each requiring investigation. Providing police protection for every victim would be impossible given staff sizes and budgets.

I think machine-learning is not the key part, the quote above is. All these 20 people a day come to the police for protection, a very small minority of them might be just paranoid, but I'm sure that most of them had some bad shit done to them by their partner already and (in an ideal world) would all deserve some protection. The algorithm's "success" in defined in the article as reducing probability of repeat attacks, especially the ones eventually leading to death.

The police are trying to focus on the ones who are deemed to be the most at risk. A well-trained algorithm can help reduce the risk vs the judgement of the possibly overworked or inexperienced human handling the complaint? I'll take that. But people are going to die anyway. Just, hopefully, a bit less of them and I don't think it's fair to say that it's the machine's fault when they do.

Minority Report: the beta test

The computer response should be treated as just an indication and in all cases a human needs to decide to override that

Otherwise we’ll all become useless pieces of a simulation

I went to the bank to ask a loan and then it got rejected because the computer said I didn’t met the parameters by just 40 euro. Ah ok, I told the clerk, just lower the amount that I’m asking or spread it over a longer period. No, because after the quote is done and I signed the authorization for the algorithm to perform credit score, it can’t do it again in 3 months. What?? Call a supervisor and let them override it, 40 euro is so minimal that it’s not that big issue. No, impossible. So that means each single employee in the bank is just an interface to the computer and can be fired at will?

Minority Don't Report

Pedantic Mathematician here.

If it failed, then it was a heuristic, rather than an algorithm.

Clearly, that's the most important thing about this post.

You're welcome.

Despite this article, I'm still not convinced that the algorithms aren't better. The policy states that people need to use their best judgement and can override the algorithms. The article argues that the algorithms are being over relied on. The article mentions in passing, however, that the statistics were worse before the algorithm was introduced.

The point of the matter is, best judgement can be shitty. Your average cop has no idea what questions to ask without a list and how important they are per research. Some suggestions are too continue using the tool but use things like psychologists to administer it. The only way you could reasonably have a psych on call for every police station is to make it a remote interview, which frankly doesn't seem better to me.

In the end, the unstated problem is resources and how best to utilize them to prevent the violence. I'm sure Spain's policy could be improved but shoring it up with an algorithm is a good practice.

More data for the algorithm then.