Police using AI risks making discrimination worse, researchers warn

Researchers warn that the increasing use of artificial intelligence in policing could exacerbate discrimination against minority groups. Studies indicate that crime hotspot predictions based on AI may be derived from biased data, amplifying existing prejudice and creating a feedback loop. Dr Carolyn McKay, a digital justice expert at the University of Sydney Law School, emphasizes that AI systems should be subject to human oversight and scrutiny, as existing data often contains human bias.

Similar to a map-maker relying on outdated maps, AI systems are trained on data that has already contained human bias, potentially leading to self-fulfilling prophecies.

Sarah Brayne, a sociologist at the University of Texas at Austin, found that the Los Angeles Police Department increasingly used big data for crime prediction rather than solely responding to crime. However, the data used in these systems was skewed, as crimes in public places were more likely to be seen and recorded, and police focus was disproportionately on black neighborhoods.

This led to AI systems creating a feedback loop in which they not only predict events such as crime but also contribute to their future occurrence.

Dr Dimitrios Kollias, an AI expert at Queen Mary University of London, argues that modern AI systems can take the bias of historical data into account and generate novel insights that were not explicitly present in the training examples. However, the real challenge lies in dealing with often flawed information stored by authorities, which AI systems may treat as a pattern to repeat.

In New York, researchers found that nine out of 13 US areas used police data from periods of unlawful or biased policing to train AI systems. This flawed data cannot be recognized by the system as biased, as it simply treats it as a pattern to replicate.

The solution, according to McKay, is to have independent experts review the data before AI systems are deployed, ensuring its accuracy and removing any incorrect lines. However, cleaning up "dirty data" is a complex task, as the mistakes are not minor errors but rather deeply embedded in the system, much like a map drawn on top of existing bias. The New York University researchers highlight that technical fixes offered by AI companies are insufficient without independent checks on how the data was produced.