The benefits of medical AI assistance vary based on user expertise

A new study from MIT and elsewhere reveals that the efficacy of artificial intelligence (AI) assistance in disease diagnosis varies significantly based on the expertise level of the user. The research, led by Marzyeh Ghassemi, Roxana Daneshjou, and Orson Xu, highlights that while AI assistance generally improved the accuracy of both non-experts and clinicians in diagnosing skin diseases, the impact of AI explainability methods depended on the users' knowledge level.

For non-experts, AI explainability methods led to increased diagnostic accuracy, largely due to their deference to the AI system. These users trusted explanations generated by language models (LLMs), whether they were accurate or not, and found vague or generic explanations more convincing. However, this reliance on AI explanations can be detrimental when the AI is incorrect, as it may lead to increased errors in diagnosis.

On the other hand, clinicians were not swayed by incorrect AI assistance and performed best when given only the model's prediction without accompanying explanations. This suggests that clinicians are more capable of critically evaluating AI outputs and relying on their own medical training and expertise. The study underscores the importance of designing AI systems that cater to the specific knowledge levels and needs of their users, as overreliance on AI explanations can mislead those with less medical knowledge.

The researchers tested non-experts and primary care providers in skin disease diagnosis, comparing their performance with and without the help of different explainable AI systems. They found that non-experts' diagnostic accuracy improved with AI assistance, primarily due to their deference to the AI system. This deference effect was most pronounced with LLM-based explanations, causing non-experts to be more confident in their incorrect diagnoses.

In contrast, clinicians were resilient to incorrect AI explanations and benefited the least from LLM explanations. The study concludes that explainability methods must encourage critical thinking rather than promote overreliance on AI models. As AI systems become more prevalent in healthcare, it is crucial to design them with user expertise in mind to prevent errors and ensure accurate diagnoses.