FDA Weighs Clinician-Style Tests for Generative AI Medical Devices

The Food and Drug Administration (FDA) is contemplating a new approach to assessing generative artificial intelligence (AI) medical devices. The agency released a discussion paper on Tuesday, August 18, aiming to gather public input on how to evaluate the safety, effectiveness, and risk of such devices. Premarket testing, post-deployment monitoring, foundation models, and agentic systems capable of executing complex tasks are among the topics covered.

Generative AI presents unique challenges to traditional medical software, which performs predictable functions with specific inputs and outputs. Generative AI can process open-ended instructions, generate various responses to similar prompts, and adapt as its model, safeguards, or data sources evolve. Testing every conceivable interaction may be unfeasible. Therefore, the FDA is considering whether manufacturers could demonstrate their devices' competence in performing intended clinical tasks.

The proposed process would start with nonclinical benchmarking, evaluating the device's clinical knowledge, analytical ability, safety behavior, communication, and performance across diverse patient groups and operating conditions. Clinical confirmation would follow, showing the device's compatibility with real patients, clinicians, and workflows, potentially without requiring a prospective clinical trial in every scenario.

The level of evidence would depend on the device's intended use and the severity of potential harm from an incorrect output.

The FDA's proposal draws an analogy to medical training, where regulators do not anticipate every situation a physician will encounter but instead test underlying knowledge, observe performance in clinical settings, and require ongoing oversight. The agency aims to apply a similar model to AI while addressing the distinct technical and legal issues surrounding medical devices.

The final assessment would occur when the device is used by patients and clinicians, rather than testing the underlying foundation model in isolation. This distinction could be crucial as a single general-purpose model can support multiple products with varying prompts, interfaces, safeguards, and clinical applications.

Post-approval oversight may involve periodic retesting, clinician review of real-world outputs, and monitoring for performance deterioration. Updates or changes made by third-party foundation-model providers could trigger additional assessments. These measures could impact healthcare payments, as payers and finance companies may need records of which AI system version performed a service, whether it remained within its validated role, and whether monitoring detected any decline in performance.

While the FDA proposal does not impose requirements on financial institutions, the concept could serve as a useful model. A bank could test a fraud, underwriting, or collections agent against defined operational competencies and monitor whether it stays within those limits. This approach may be more practical than attempting to prove the safety of a general-purpose model for every conceivable use.

The FDA's discussion paper represents a potential shift in regulatory thinking, with AI systems increasingly evaluated based on their reliable performance, boundaries, and long-term competence.