RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment

RoB2-Evaluator, a rule-constrained GPT-5.2 system, achieved 90% agreement with human reviewers and 94.3% stability in Cochrane Risk of Bias 2 assessments across 10 RCTs.

• Why it matters: Accurate and efficient bias assessment is crucial for evidence-based medicine but is often resource-intensive and variable, highlighting the need for reliable AI assistance in systematic reviews.
• What they did: The system was developed using GPT-5.2 with fixed guidance, calibrated on seven RCTs, and evaluated on ten RCTs with repeated assessments after one week to measure stability and agreement with expert judgments.
• The result: RoB2-Evaluator demonstrated high temporal stability and agreement with human reviewers, with 85.7% full conceptual alignment, supporting its potential for larger independent validation and integration into review processes.