Our research shows how AI is getting better at exams. Here’s how unis can respond

A recent study has revealed significant advancements in AI's ability to perform well on legal exams. By testing nine generative AI models from five providers on two compulsory law subjects at the University of Wollongong, researchers found that AI performance has improved considerably since a 2023 experiment. The AI models, which had internet access and an "enhanced reasoning" feature, outperformed 82.5% of students in criminal law and 61% in torts.

Notably, seven AI-generated papers ranked within the top 10% of student performance. While AI models have become more competent, their capabilities remain inconsistent. Some models excel in certain subjects while underperforming in others, and they often produce inconsistent results across different prompts and assessments. The study emphasizes the need for universities to adopt a multi-pronged approach to assess students' ability to work with AI responsibly.

One approach suggests maintaining some AI-free, in-person examinations to ensure students develop foundational knowledge and reasoning skills. Another proposes incorporating AI collaboration into assessments to evaluate students' ability to work with AI models effectively. Lastly, a "relay" model could be implemented, where students initially submit a draft in a supervised setting and then collaborate with AI to refine and resubmit their work, demonstrating both their own critical analysis and AI-assisted improvements.