Missing chatbot records leave doctors struggling to trace harm from AI health advice
As more people turn to AI chatbots for quick, 24/7 medical advice, a new perspective paper co-authored by faculty at Binghamton University, State University of New York, explores the risks of this practice that are largely invisible to users—and health care systems.
This article explores the risks and challenges associated with AI chatbots providing medical advice. Researchers from Binghamton University and other institutions argue that the harm caused by erroneous or dangerous advice from these chatbots is difficult to detect. The conversation between the user and the chatbot often remains hidden within the AI company's platform, making it challenging for users to report issues or for outside researchers to measure the damage.
The article provides an example of a 60-year-old man who asked ChatGPT for dietary advice and replaced table salt with sodium bromide, resulting in bromide toxicity and hospitalization. The man was unable to retrieve his conversation with the chatbot, making it nearly impossible for his doctors to understand what he was told. This lack of visibility into the health advice generated by AI chatbots prevents oversight by clinicians, researchers, and regulators.
The researchers note that AI chatbots can provide incorrect or misleading information, and this risk extends to people who are not actively seeking answers. With the increasing presence of AI summaries in search engines, social media platforms, and other online spaces, people may encounter incorrect health information even when they are not actively searching for it.
The authors propose several solutions to increase transparency and accountability for AI companies. These include allowing users access to their own health conversations, developing a disclosure system for health guidance, and improving the clarity of labels on AI-generated health content on social media and search engines. They also suggest that search engines use only vetted information in their summaries and that policymakers extend physician malpractice liability to AI chatbot companies.
Written by urgent.news from Medical Xpress's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
