digital justice
© mattjeacock/Getty
Bias and injustice hidden in AI can be hard to root out.
Something is rotten at the heart of artificial intelligence. Machine learning algorithms that spot patterns in huge datasets, hold promise for everything from recommending if someone should be released on bail to estimating the likelihood of a driver having a car crash, and thus the cost of their insurance.

But these algorithms also risk being discriminatory by basing their recommendations on categories like someone's sex, sexuality, or race. So far, all attempts to de-bias our algorithms have failed.

But a new approach by Niki Kilbertus at the Max Planck Institute for Intelligent Systems in Germany and colleagues claims to offer a way to bake fairness right into the process of training algorithms.

Machine learning systems improve with examples. The rough idea is that if you want to predict the likelihood of someone reoffending, you load up an AI with previous cases of people going through the criminal justice system. After viewing enough examples, the algorithm can then predict what will happen when presented with a fresh case.

But there are certain things that, as a society, we don't want an algorithm like this to take into consideration, such as someone's race - bail shouldn't be determined based on the colour of someone's skin.

"Loan decisions, risk assessments in criminal justice, insurance premiums, and more are being made in a way that disproportionately affects sex and race," says Kilbertus.

Reading between the lines

But unfortunately, the obvious quick fix of just removing sensitive categories from the data the algorithms are trained on doesn't work.

Firstly, non-sensitive categories - unrelated to things that are normally considered discriminatory - can be an approximation for those that are.

For example, an algorithm that uses location data to understand someone's driving habits, may end up drawing a link between people who regularly visit a gay bar and how they drive. Not everyone who visits a gay bar is gay, but it is likely to be a reasonable proxy for that group.

And secondly, if we remove all sensitive data from the mix it becomes almost impossible to detect this type of indirect discrimination. Almost every definition of fairness that could be used to guarantee that an algorithm is not discriminatory needs access to sensitive data too. Normally, tests of fairness involve checking that all other things equal, two people with different sexualities, say, get the same prediction.

To counter this, Kilbertus and colleagues suggest a neat way to include sensitive data in the process without producing biased AIs. It relies on introducing an independent regulator and using the mathematics of encryption.

When training the AI, a company can use as much non-sensitive data as is necessary, but they only get sent sensitive data in an encrypted form, as does the regulator.

This is enough for the regulator to check whether the AI is making decisions that are influenced by anything it has inferred from the non-sensitive data, and is therefore biased.

Once satisfied the regulator can then award a fairness certificate to the company - essentially, guaranteeing that the system they have developed is not discriminatory. This approach has the advantage that the regulator doesn't need to know the inner workings of the AI, so the company can keep any trade secrets secret.

Angela Zhou at Cornell University says that the work is "exciting to see". And that she hopes "what is possible in theory becomes possible in practice". She also says that it should enable greater accountability of algorithms in situations where it's not possible to fully inspect the inner workings of a system.

Journal Reference: