Teo Nie Ching warns of AI-powered ‘sympathy scams’ using fake distress images to solicit donations

Seoul, November 2 - A significant portion of South Korean judges have encountered court cases where false or nonexistent legal precedents generated by artificial intelligence (AI) have been cited, according to a recent Supreme Court survey. Conducted among 360 judges from October 6 to 16, the survey reveals that around 80% of judges working on civil, family, and administrative cases reported experiencing instances of AI hallucination - the presentation of false information by AI models as if it were true.

Out of those, 80.7% witnessed cases involving nonexistent legal precedents, while 65.4% encountered false precedents. Additionally, 16.9% of the judges faced situations where false or nonexistent laws and regulations were cited, and 4.1% saw citations of false or nonexistent theses and publications. A total of 164 judges admitted to having cited false precedents themselves, with 57.6% of respondents stating that lawyers representing parties in court had done the same.

Nearly 92% of the respondents called for institutional measures to tackle the issue, including the creation of legal grounds for imposing fines, verification requirements, and disciplinary measures against lawyers who repeatedly cite false information. A National Court Administration official emphasized that such AI-generated false precedents could lead to delays in court proceedings, emphasizing the urgent need for effective countermeasures based on the survey findings.