Benchmarking Large Language AI Models Against Human Doctors Using an Emergency Medicine Simulation Approach

Eight LLMs significantly outperformed nine emergency medical doctors in emergency medicine simulated tasks, including history taking, investigation, diagnostic accuracy, and treatment planning.

• Why it matters: Standard Q&A benchmarks miss key clinical skills needed for healthcare agents, so adoption in emergency medicine requires broader, skill-based evaluation.
• What they did: The study used a three-part scheme: expert scoring of physician skills, intern-scored simulated emergency room visits, and a comprehensive written exam to measure general knowledge.
• The result: AI also showed superior comprehensibility, credibility, and empathy; between-model differences were marginal in simulated visits but mostly pronounced on the written examination.

AI summary of the abstract · may contain errors · not medical advice · read the paper.