Clinical AI needs safeguards against hallucinations, data leaks and overreliance, review finds
Large language models (LLMs), including the models behind ChatGPT and Claude, as well as numerous other systems developed specifically for medical applications, are increasingly used in clinical workflows. They support medical documentation, summarize knowledge and assist with clinical decision-making. However, their adoption is outpacing the development of systems for oversight and safety.
Clinical AI systems, such as large language models (LLMs), are being integrated into medical workflows to assist with tasks like documentation and decision-making. However, their rapid adoption is outpacing the creation of oversight mechanisms, leading to potential risks. An interdisciplinary team of researchers from the Else Kröner Fresenius Center (EKFZ) for Digital Health at TUD Dresden University of Technology, University Hospital Dresden, and other institutions have conducted a comprehensive review of these risks, published in Nature.
The review, co-authored by Dr. Jan Clusmann and Professor Jakob N. Kather, highlights that LLMs can support clinical professionals but must be used responsibly. Risks can arise at various stages, from model development to real-world use, including security breaches, data leaks, hallucinations, and overreliance. The researchers emphasize the need for secure development processes, careful training data selection, systematic evaluation, continuous monitoring, and clear institutional responsibilities.
They also stress the importance of human oversight and coordinated efforts from research, clinical practice, and regulatory bodies. The review recommends establishing local AI oversight teams and centralized Security Operations Centers (SOCs) for AI to detect incidents and enable coordinated responses across institutions. Additionally, the researchers call for more robust regulatory approaches to address the lack of formal approval for many AI systems and the evolving nature of this technology.
Addressing these challenges is crucial to ensuring patient safety and reaping the benefits of AI in healthcare.
Written by urgent.news from Medical Xpress's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
