In Search of Ethical Procedures for LLM-Assisted Systematic Review Production: A Proof-of-Concept Evaluation of Selected Review Components

LLMs can effectively assist in specific, well-defined tasks within systematic review processes, achieving up to 86.2% agreement with human decisions when criteria are carefully reworded.

• Why it matters: Understanding how to responsibly integrate LLMs into evidence synthesis is crucial to ensure accuracy, transparency, and ethical standards in scientific research.
• What they did: The study compared LLM screening decisions to human judgments using a published umbrella review and evaluated LLM-generated reviews against human-authored ones, involving 6 hematopathologists and 2 reviews.
• The result: Findings indicate LLMs perform well in bounded tasks under supervision, with some limitations like citation errors, leading to the development of ReviewFoundry to support human-AI collaborative review workflows.