Does ChatGPT Measure Up to Doctors When It Comes to Questions About IVF?

A recent study compared ChatGPT's responses to patient questions about in vitro fertilization (IVF) with those from professional society education materials, such as those from the American Society for Reproductive Medicine (ASRM). The results showed that while ChatGPT's clarity was comparable to ASRM's documents, it scored lower in terms of accuracy, completeness, safety, and quality.

Dr. Nicole Fischer, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, presented the findings at the ASRM annual meeting. She explained that half of ChatGPT's answers contained unsupported claims, and 4.2% contained misinformation, while ASRM's responses contained none. The study also revealed that 94.1% of ASRM's responses adhered to a checklist, compared to 94.1% for ChatGPT.

Moreover, ChatGPT's responses were at an eighth-grade reading level, whereas ASRM's documents were at a 12th-grade level. The study also found that ChatGPT's success estimates for IVF varied compared to CDC and SART data, overestimating live birth rates at age 35 and underestimating rates at 37 and 40.

Fischer emphasized the importance of patients' questions' wording, as it can affect the readability and comprehensibility of the responses they receive. She also noted that newer and specialized LLMs, such as ChatGPT Health, should be tested to see if they provide better and more accurate information.

The study had some limitations, including its small sample size and the use of only one version of a single LLM. However, it highlights the need for clinicians to be cautious when patients rely on chatbots for information about IVF and ART, and encourages them to educate their patients about these tools' limitations.