{
  "id": 13858628,
  "title": "An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph",
  "url": "https://urgent.news/2026/10/11/an-expert-curated-benchmark-for-frontier-and-local-llms-querying-a",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-11T00:00:00.000Z",
  "source": {
    "name": "bioRxiv",
    "slug": "biorxiv",
    "url": "https://www.biorxiv.org/content/10.64898/2026.10.04.756532v1?rss=1"
  },
  "original_language": "en",
  "account": "A new benchmark has been developed to measure how frontier and local language models interact with biomedical knowledge graphs. Researchers utilized GeneNetwork's 1TB structured database, its metadata as RDF graph, and an accompanying literature corpus for this purpose. The benchmark employed four architectures: document-based, graph-based, agentic retrieval-augmented generation, and a combination of the three.\n\nTo assess the performance of these systems, 40 domain-specific expert-curated questions were presented. A novel metric, the satisfaction frequency psi, was introduced. Psi, like precision, recall, and the F1 score, measures the share of questions answered with at least 70% precision, recall, or an F1 of at least 60%. Psi is simpler to understand and evaluate for humans compared to traditional metrics.\n\nThe findings revealed a significant impact of system architecture on performance. When Claude Opus 4.8 was used in full, the document RAG of the GeneNetwork Query Answering (GNQA) system achieved a psi of 36%. However, with the integration of additional tools, this score increased to 86%. Moreover, delegating most of the query work from Claude Opus 4.8 to a local model reduced query costs by half while still maintaining a psi of 59%.",
  "summary": "We combine GeneNetwork's 1TB structured database with its metadata as an RDF graph and an associated literature corpus. We benchmark frontier and local LLMs querying it through four architectures: document, graph and agentic retrieval-augmented generation (RAG), and a combination of the three. We evaluate them on 40 domain specific expert-curated questions with a new metric: the satisfaction…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}