Column | When machine enters the courtroom: A case for regulating AI?

Artificial intelligence (AI) has seen significant advancements through the work of visionaries in mathematics, computer science, research, and engineering since the 1950s. Today, AI is prevalent in various industries, including the legal and judicial sectors, where it is primarily used for research, drafting, case filing and management, transcribing hearings, summarizing submissions, and identifying risks in sentencing decisions.

The primary concern revolves around determining the boundaries of AI's role and the responsibility for its failures and incidents of malfunctioning.

Legal professionals argue that AI can alleviate many issues, including human limitations, fatigue, and unconscious biases that can impact case backlogs and the right to timely justice. AI has shown promising results in foreign jurisdictions like China's Hangzhou internet courts, which report a more than 50 percent reduction in hearing times through AI-assisted case management.

Similarly, Estonia automates transcription and redaction through "Salme," freeing judicial time for decision-making. In India, the Sikkim High Court was the first to become fully paperless, testing AI's assistive functions at district courts.

However, cautionary evidence highlights the risks of AI. Bias in AI does not disappear when it takes over; rather, it becomes embedded in the training data. AI risk assessment tools in the USA have been found to assign harsher scores to black defendants, while a Dutch welfare-fraud detection system penalizes applicants based on nationality.

Another issue is the generation of plausible yet fabricated legal citations by AI language models, which can lead to a cascade of consequences, including wrongful policing, denial of rights, and breach of privacy expectations. A study by Stanford and Yale shows that even legal-specific AI tools have a 17 to 34 percent margin of hallucination, exacerbating the risks.

The lack of transparency in AI reasoning, coupled with the absence of disclosures around design, further compounds the problem.

To address these concerns, courts and regulators worldwide have responded to penalties, sanctions, disbarments, and revised conduct codes. The Supreme Court of India has demanded a "zero-tolerance" approach to faulty AI outputs, equating the issue to a noxious injection into the justice delivery system that remains "invisible," "insidious," and "catastrophic" for a long time.

The court released draft regulations for the use of AI in courts, emphasizing human primacy, transparency, accountability, data protection, and judicial autonomy. These guidelines prohibit AI tools from making judicial decisions and limit their use to assistive functions only. Legal professionals and institutions must ensure that AI decisions are cross-checked against primary sources and scrutinized based on the stakes involved, the nature of disputes, and the likelihood of moving beyond assistive to actual decision-making domains.

As India grapples with a backlog of over 60 million cases, a nimble mechanism must be put in place to make and review regularly informed choices about AI use, accounting for the stakes at hand, dispute nature, and the need to preserve judicial independence and due process rules.