Version-aware question answering over living oncology guidelines

Living clinical guidelines regularly update their recommendations as new evidence comes to light, leaving AI-generated answers potentially outdated to a specific version of the guideline. To study this issue, we utilized ASCOBench, a dataset containing 288 unique questions spanning 96 three-turn conversations based on versioned American Society of Clinical Oncology (ASCO) breast and prostate cancer guidelines. The dataset also included oncologist-reviewed reference answers for comparison.

Oncologists evaluated version-sensitive answers generated by a frontier model, finding that they corrected one in four of these answers, while fewer than one in ten corrected factual answers. Across the entire guideline corpus, 19 recommendations changed between versions, seven of which were reversals, yet only one out of 14 superseded documents explicitly states that it has replaced the previous version.

When strong models performed retrieval over several guideline versions, the frequency of stale answers increased fourfold compared to answering without retrieval. SentryLine, a verification-first system, reduced incorrect answers from 9.7-22.9% for baselines to 4.2-5.2% across three models. However, identifying changes in guidelines before they are officially announced remains an open challenge.