Calibrated large language models demonstrate a consistent advantage in a randomized longitudinal medical education intervention

In a year-long randomized medical education intervention using Neurology Grand Rounds, receiving AI summaries improved quiz performance in 9 of 11 clerkship cohorts (p = 0.026).

• Why it matters: Trainees use large language models, but the educational value depends on prompt and model choices that are rarely measured in controlled settings.
• What they did: The study built a calibrated pipeline that converted Neurology Grand Rounds recordings into summaries and quizzes, scoring Phase 1 outputs with 2,692 rubric annotations across prompts and two models.
• The result: In Phase 2, within 11 clerkship cohorts, students randomized to get a summary before each quiz reported more review and faster studying and scored higher in 9 of 11 cohorts, with a small between-student gain (+0.28 of 10 points; P = 0.28).

AI summary of the abstract · may contain errors · not medical advice · read the paper.