EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models

The top EM-BENCH configuration achieved 207 of 210 correct guideline-concordant answers (98.6%) when applying ACEP emergency medicine policies to binary clinical vignettes.

• Why it matters: There is no accepted standard to measure how reliably large language models follow clinical guidelines in decision support, limiting trustworthy benchmarking.
• What they did: EM-BENCH used 210 physician-validated yes-or-no vignettes spanning 70 critical questions across 25 ACEP policies, scoring 149 LLM configurations from 79 base models deterministically.
• The result: Closed-access models had higher median accuracy than open-weight models (91% vs. 63%; adjusted P<0.001), with accuracy linked to model size and release year, while thinking mode and reasoning effort showed no consistent gains.