¿Podemos fiarnos de la nueva IA médica?

Kayla Secrest, a newly graduated doctor starting her residency in a Michigan hospital, had a frustrating first encounter with AI. A new algorithm was implemented to analyze patient histories every 15 minutes and send alerts for sepsis warning signs. While this innovation automatized a vital and routine task, Secrest soon realized the alerts were triggering for all her patients and began ignoring them.

Over time, she and her colleagues lost confidence in the tool and stopped responding urgently to the notifications. It later turned out that the system had been rushed into hundreds of hospitals without thorough testing due to the busy nature of these centers. Promises about the impact of AI on healthcare were made, and many overwhelmed doctors and hospitals relied on it to compensate for the growing demand on their services.

However, Secrest's experience and that of other professionals suggest there is a gap between AI developers' work and medical professionals' supervision: often, there is little independent oversight of what happens when an AI tool moves from the controlled laboratory environment to the unpredictable world of healthcare. Experts believe that in some cases, this lack of control poses a real risk to patients.

Moreover, it can make it difficult to prove that certain tools improve treatment or justify the investment they represent. Jess Morley, a Yale Digital Ethics Center research associate and former AI official for the UK Department of Health and Social Care, says tech companies tend to focus on statistical validation, i.e., the precision of the tool more than its clinical effectiveness.

What matters in healthcare, though, is that second aspect: we want to know if the tool has a real impact on patient outcomes, she stresses. Evidence that AI influences patient outcomes is scant. AI proponents argue that this technology is being adopted in healthcare at a faster pace than in other sectors and is improving care, partly by alleviating doctors' administrative workload, for example, by automating note-taking.

For over a decade, it has become evident that we have lost professionals to the administrative burden, lamented Kimberly Powell, Nvidia's health area head, in June. OpenAI reported an 80% reduction in time spent on certain administrative tasks by using ChatGPT for Healthcare, and an independent evaluation of its collaboration with Penda Health in Kenya showed a 16% decrease in diagnostic errors among professionals using its AI assistant.

AdventHealth reported an 80% reduction in time spent on certain administrative tasks after using AI. Matt Rainey/AdventHealth Other potential applications include patient monitoring through wearables, analyzing large data volumes to predict future diseases, and accelerating drug development processes. In some areas, such as diagnostics and medical imaging, the impact of AI is irrefutable.

Eric Topol, director of the Scripps Research Translational Institute, cites a 2024 study showing that colonoscopies performed by gastroenterologists with AI assistance detected many more polyps than those done without the technology. However, instead of adopting proven AI tools, Topol points out that healthcare managers have been seduced by the novelty of generative AI.

This advancement, driven by the 2022 launch of ChatGPT, allowed the technology to, for example, summarize extensive medical literature to help doctors make decisions. The regulatory gaps and the lack of performance data make it difficult to determine whether the benefits of generative AI outweigh its risks. The current priority should be to establish that relationship between benefits and risks in real medical practice, Topol suggests.

Topol perceives little interest from major AI groups in measuring performance against data collected after model development to verify if results are reproducible in real-world conditions. Major tech giants are quick to create models and test them in the laboratory, as well as promising them in the lab, but they show little enthusiasm or financial support for the type of real-world trials we would like to see.

The lack of transparency is a concern, says Andrew Wong, a doctor and researcher. He analyzed the faulty sepsis tool during his time at the University of Michigan and worked with developers to improve its performance. In his opinion, companies often make very specific claims about how AI tools improve workflow and generate economic savings but have no obligation to disclose information that would facilitate verification of these benefits.

This could include details on how the model was trained, what type of data set was used, what methodology was employed for development, and where the model was validated to obtain the reported statistics.