RadMultiBench: A Reproducible Dual-Task Benchmark Exposing the Limits of Pre-Trained Vision-Language Models Across Thoracic and Breast Imaging

ResNet-LSTM achieves near-perfect classification (99.5% accuracy) on RadMultiBench’s 6,148 thoracic and breast imaging cases, while only it generates coherent reports (BLEU-4 = 0.021).

• Why it matters: General-domain vision-language models may not transfer to high-stakes clinical report generation and classification, and prior work lacks reproducible benchmarks to expose where models fail.
• What they did: The RadMultiBench pipeline standardizes ingestion, de-identification, splits, and evaluation across two tasks, comparing ResNet-LSTM, CLIP-GPT, and BioViL-T on five diagnostic categories.
• The result: Both prefix-projection models catastrophically fail generation (CLIP-GPT BLEU-4 ≤ 0.0001), and BioViL-T also fails classification (11.5% accuracy), enabling targeted ablations on alignment and architecture.

AI summary of the abstract · may contain errors · not medical advice · read the paper.