When AI Makes Medical Decisions: Designing Human-in-the-Loop Clinical Systems

In recent years, a Midwest hospital introduced a sepsis prediction model in its intensive care units (ICUs). However, nurses quickly muted the alerts as the model produced too many false alarms, leading to a loss of trust and disengagement. This scenario exemplifies the common challenge of deploying AI to make clinical decisions without proper design. The real issue lies not in the mathematics, but in the design around the mathematics.

While creating a model that predicts patient deterioration, flags suspicious masses on scans, or recommends drug dosages is technically easier, the harder part is determining the appropriate role of the human in the loop and the level of authority to grant the AI. Automating the entire process does not survive the scrutiny of hospital settings, where false negatives can lead to patient fatalities and false positives can result in unnecessary treatment, costs, and loss of patient trust.

Medical professionals stress that errors in healthcare are not tolerable, unlike in other industries. Moreover, the distribution shift problem often goes underestimated. A model trained on data from one hospital, with its unique patient demographics, lab equipment, and charting habits, may not work effectively in another hospital.

Therefore, full automation is not the solution, at least for now, and maybe not for a long time. Instead, a human-in-the-loop design is being developed. This approach does not merely involve a doctor clicking "approve" after reviewing the AI's output but instead requires the system to be built so that the human can meaningfully evaluate, question, or override the AI's output, with sufficient context to perform these tasks.

The pipeline of such a system typically starts with patient data from electronic health records (EHRs), monitoring devices, labs, and other relevant sources. The inference engine generates a prediction, but crucially, it is accompanied by a confidence or uncertainty estimate instead of a bare output. This confidence score is then fed into an escalation router, which plays a critical role in the system's architecture.

Low-uncertainty, low-stakes predictions may simply be logged quietly, while high-uncertainty predictions or predictions in inherently high-stakes categories (such as dosing, sepsis, or changes in code status) are routed to a clinician review interface. This interface displays the reasoning behind the AI's prediction, rather than just the verdict. The clinician's decision and any overrides are logged, with the log serving not only compliance purposes but also feeding back into monitoring and retraining.

This approach allows the clinician to only review a fraction of the AI's outputs, preventing the alert fatigue problem observed in the sepsis prediction model rollout. However, the clinician is not presented with a black box solely when it is convenient for the vendor's liability position. The routing logic triages the clinician's intervention, and this decision has a significant impact on the system's overall design.

The escalation router deserves substantial attention in system design, as threshold tuning is not a one-time calibration exercise but rather an ongoing negotiation between sensitivity and clinician trust. Setting the threshold too conservatively risks alert fatigue, while setting it too loosely may lead to missing critical cases, which is the more severe failure mode in healthcare settings.

Designing systems with a clear human-in-the-loop approach, where the human can meaningfully evaluate, question, or override the AI's output, is essential for ensuring patient safety and maintaining trust. This approach sets the stage for more responsible and effective integration of AI in clinical decision-making.