A decision-only model as a low-cost judge of medical AI answers

TypeSafe Jev 1.13, a decision-only judge, matched expert labels at 64.7% across six clinical benchmark tasks, and was non-inferior to frontier judges on four of six.

• Why it matters: Medical AI evaluation is constrained by clinician capacity, and LLM judges are costly; decision-only models could scale judging while providing calibrated uncertainty.
• What they did: In an OSF pre-registered evaluation across five validated benchmarks (25,091 units; 213,913 model calls), Jev’s plain agreement was compared with expert rater agreement and frontier judges (OpenAI Decisions, GPT-6 Luna, GPT-6.1 Sol, Claude Opus 5.5).
• The result: Jev agreed with experts at 66.1% on 3,614 matched units versus 65.0% for Opus and 60.7% for Sol, judged the full pool in 17 minutes for US$1.46, and missed 55.6% of planted omissions at the same false-alarm rate.

AI summary of the abstract · may contain errors · not medical advice · read the paper.