Police using AI risks making discrimination worse, researchers warn

In his award-winning essay, young British science writer Adrian Dredge explored the potential downsides of AI in policing. The essay earned him the 2026 Young Science Writer Award, chosen annually by the Association of British Science Writers. Researchers warn that the increasing use of AI in policing could exacerbate discrimination against minority groups.

Studies indicate that predictions of future crime hotspots might be based on biased data, which can amplify existing prejudice and create a dangerous cycle. Experts are urging a review of AI use in policing with increased human oversight, involving the checking of data to identify and eliminate bias, as Dr Carolyn McKay, a digital justice researcher at the University of Sydney Law School, points out.

The public functions of state policing should be subject to human oversight and scrutiny, she emphasizes. The issue lies in the fact that AI systems are trained on data that already harbors human bias, potentially leading to self-fulfilling prophecies. Envision AI as a map-maker tasked with drawing a city's map they have never seen.

With limited access, all they possess are outdated maps. Those following the new map might encounter nonexistent walls and non-existent roads. The map-maker merely replicated the old maps, not creating anything new. Now, suppose the map-maker receives a list of past crimes and their locations and is asked to forecast where future crimes will occur.

Policemen subsequently patrol the marked areas, leading to more crimes being recorded there simply because of increased police presence. Conversely, areas without markings appear less crime-ridden as fewer eyes are on them. When the next map is drawn, the same areas are highlighted again, making the crime figures a mix of recorded crimes and areas with heightened police presence.

The problem is as old as policing but AI may be accelerating it. Sociologist Sarah Brayne, studying the Los Angeles Police Department's use of big data, found that they increasingly relied on data to predict crime instead of merely responding to it. The data feeding these systems was skewed, as crimes in public spaces were more readily seen and recorded, with police focus disproportionately concentrated on black neighborhoods.

Once incorporated into the system, the predictions appeared impartial, but they inadvertently contributed to their future occurrence, creating a feedback loop. However, Dr Dimitrios Kollias, an AI expert at Queen Mary University of London, challenges this view, stating that modern AI systems account for historical data bias in novel ways, generating outputs not explicitly present in the training examples.

While this poses a separate risk, he emphasizes the need to ensure AI behavior remains safe, reliable, and under meaningful human control as these systems become more capable of generating autonomous decisions. However, the more insurmountable challenge lies in dealing with flawed information authorities store, which AI systems draw from.

Researchers at New York University discovered that many predictions rely on what they term "dirty data," representing corrupt, biased, or unlawful practices, including rigged crime figures and arrests of innocent individuals. The system treats this flawed data as a pattern to repeat, just as the map-maker follows the bias of past officers.

The map-maker believes they have identified new crime hotspots when, in reality, they have merely followed earlier officers' bias. McKay proposes independent checks on the map before its use and removing incorrect lines. Yet, eradicating dirty data is not a simple task; the errors are not minor scratches but a complete map drawn atop them.

The technical fixes offered by companies selling these systems are insufficient if independent checks are not performed on how the data was produced. As AI perceives the world solely through the data provided, it can make grave mistakes with harmful consequences. For instance, a 2019 study in the journal Science examined an AI system in US healthcare that predicted patients needing extra support based on the most expensive care cases.

Due to inherent bias in the data, less was spent on black patients, partly due to poorer access to care. By treating higher costs as a sign of greater need, the system unfairly favored white patients. The researchers altered the system to predict patient illness rather than care costs, demonstrating the AI's ability to perpetuate flawed data patterns, much like the map-maker following the bias of past officers.