Stable answers help medical AI flag diagnoses clinicians can trust

Researchers at the Else Kröner Fresenius Center for Digital Health at TU Dresden and Dresden University Hospital have created an on-premises medical AI system to address challenges in clinical decision-making. This system aims to maintain sensitive patient data under institutional control and enable clinicians to assess the reliability of AI-generated diagnoses.

The AI agent, built on the MIRA AI agent, simulated interactions between a physician and a patient to evaluate the system's diagnostic accuracy in cases such as appendicitis, cholecystitis, pneumonia, pulmonary embolism, and urinary tract infections. The best-performing AI model achieved a 90% correct diagnosis rate in one benchmark and 84% in the other.

Physicians reviewed 181 randomly selected cases, with the combined automated evaluation and physicians' consensus agreeing in over 90% of cases. The researchers determined diagnostic trust based on the stability of AI-generated diagnoses across multiple runs. The more stable the diagnosis, the more likely it was correct. The model's internal likelihood scores were less useful for predicting correct diagnoses.

In stress tests, removing reliable information from the system substantially reduced diagnostic accuracy. The study proposes a framework for collaboration between clinicians and AI, distinguishing cases with stronger reliability signals from more uncertain ones, which would be deferred to medical professionals for review. The AI system operates entirely on locally operated infrastructure, providing technical control over data processing, model versions, access rights, and monitoring processes within the institution.

This on-premises infrastructure supports data protection management and aligns with Europe's commitment to responsible medical AI development, emphasizing technical control and data sovereignty.