An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph

Authors

Medagbe, J. F. P., Kilyungi, B. M., Isaac, A., Kabua, A., Lisso, F., Wambua, S., Williams, R. W., Prins, P.

Abstract

We combine GeneNetwork's 1TB structured database with its metadata as an RDF graph and an associated literature corpus. We benchmark frontier and local LLMs querying it through four architectures: document, graph and agentic retrieval-augmented generation (RAG), and a combination of the three. We evaluate them on 40 domain specific expert-curated questions with a new metric: the satisfaction frequency psi (psi, i.e, the share of questions answered with precision and recall of at least 70% or an f1 of at least 60%). psi ranks systems the same way precision, recall and f1 -- a machine learning metric built from the harmonic mean of precision and recall -- but is easier to read and assess for humans. System architecture had a large effect: with full use of Claude Opus 4.8 the plain document RAG of our earlier GNQA system had a psi of 36% against 86% for our new combined tools. Delegating most of the work from Claude Opus 4.8 to a local model halved query costs while still reaching a psi of 59%.

Preprint server: bioRxiv
The authors list and abstract were imported from bioRxiv on 12 Oct 2026.

Stats

1-terrible, 9-excellent. How would you rate this preprint? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.