Why Can't AI Replace Judges in a Courtroom?

Artificial intelligence (AI) is being utilized in judicial practices in several countries, including the United States, to assist with case analysis, argument construction, risk prediction, and decision-making support. However, digital practitioners have raised concerns about the compatibility of AI with the principles of justice, citing a significant disparity between algorithms and human judges.

Pratama Persadha, Chair of the Communication and Information System Security Research Center (CISSReC), explained that AI models operate by processing statistical patterns from data and generating outputs based on probabilistic relationships learned during their training. This contrasts with the decision-making process of human judges, who consider legal considerations through a series of traceable steps.

One example from the United States is the use of risk assessment algorithms like COMPAS, which has been criticized for accuracy issues and error gaps, particularly among different racial groups. In a study of over 7,000 COMPAS scores in Broward County, Florida, the algorithm's ability to accurately predict the likelihood of recidivism was only correct about 20 percent of the time.

Similarly, in Colombia, the use of ChatGPT by judges in early 2023 led to a discourse about its potential misuse. A judge in Cartagena admitted to using the OpenAI chatbot to generate questions in a case involving a child with autism regarding medical care financing. Additionally, US citizen Roberto Mata faced a $5,000 fine after his lawyer unintentionally included non-existent legal precedents generated by ChatGPT in a lawsuit against Avianca airlines in 2022.

The black box nature of AI poses a significant challenge in legal contexts, as these systems can provide specified answers without necessarily offering justifiable reasoning. Pratama emphasized that if AI is to be used in legal proceedings, the system must be auditable, documented, monitored, and placed within a human oversight mechanism.

Furthermore, the Executive Director of Information and Communication Technology, Heru Sutadi, stressed the importance of ensuring the identity of AI users in the legal domain to maintain accountability. He also pointed out that AI recommendations should not be the sole basis for decisions concerning someone's rights, as legal decisions require evidence, transparency, accountability, and reasons.

The integration of AI in justice systems presents an accountability gap, as AI tends to prioritize speed, automation, optimization, and the ability to work on a large scale, which may conflict with the requirements for legal decisions.