The abstract is read at the publisher; the summary is JClub's.
BenchmarkHub enables reproducible evaluation of 12 open-weight language and vision-language models across 15 biomedical benchmarks, with pooled accuracy spanning 46.1% to 80.9%.
- Why it matters: Biomedical LLM benchmark scores are often not reproducible because prompts, inference settings, and model builds are inconsistently reported.
- What they did: The MIT-licensed Python/Django platform records fixed-seed question sampling, tool calls in a sandbox, and deterministic grading via browser, CLI, and REST API.
- The result: Re-grading with answer-extraction lowered accuracy by up to 33.1 points and reordered the leaderboard, while published scores matched BenchmarkHub in only 4 of 23 comparisons.
The findingWhy it mattersWhat they didThe result
AI summary of the abstract · may contain errors · not medical advice · read the paper.