Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis

GPT-5 substantially matched a radiologist in labeling seven of nine specific incidental findings in 300 Dutch lung cancer screening CT reports, with Cohen’s kappa up to 0.97.

• Why it matters: Automated, consistent identification of incidental findings in screening CT reports is needed to reduce manual workload and improve scalable report annotation.
• What they did: In a retrospective dataset of 300 CT reports from a Dutch academic hospital (19 Mar 2003–28 Dec 2020), GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, and keyword search were prompted (with and without chain-of-thought) to label nine SIFs against radiologist judgments.
• The result: GPT-5 reached substantial-to-almost-perfect agreement for seven SIFs (kappa 0.81–0.97), while interstitial lung abnormalities and mediastinal mass showed only moderate agreement (kappa 0.48 and 0.42), and no single model–prompt pair led across all SIFs.

AI summary of the abstract · may contain errors · not medical advice · read the paper.