DiaBench: A Benchmarking Framework to Evaluate the Performance of LLMs in T2D Patient Education

DiaBench benchmarked seven LLMs on 96 T2D patient-education Q&As and found accuracy often stayed ≥95% for configurations 1–2, while self-consistency dropped for some models.

• Why it matters: Clinicians face time limits in T2D education, and it is unclear how reliably LLMs answer across patient health literacy levels.
• What they did: DiaBench used GPT-5 to convert each clinician-authored Q&A into MCQs under three configurations (1–3), then a five-member panel validated rephrasing and correctness before testing seven LLMs across three runs.
• The result: Expert validation showed high MCQ transformation consensus (e.g., configuration 1: 98.1% appropriate answers and 98.1% correct MCQ answers), but Mistral-Large-2.0, Llama 4, and Perplexity Sonar had lower self-consistency (60.4%–91.7%), and all models worsened on configuration 3.

AI summary of the abstract · may contain errors · not medical advice · read the paper.