A generalist CT model raised reader sensitivity, but a reliability review warns that test-set performance does not guarantee deployed performance

One model covered many abdominal CT findings and lifted radiologists' sensitivity

A single vision-language model trained on routine abdominal CT reports improved the sensitivity of 26 radiologists in a reader study, according to a paper published in Science this week. In a reader study, RADAR assistance increased the diagnostic sensitivity of 26 radiologists by ~10%. [1] The abstract does not give the baseline sensitivity, the post-assistance figure, or whether the gain is absolute or relative. It also does not report the number of cases read or any change in specificity. Without those details, we cannot say whether the gain came at the cost of more false positives.

The training approach is the main methodological claim. The authors developed RADAR, a generalist vision-language model trained on more than 400,000 contrast-enhanced abdominal computed tomography (CT) examinations and 15 million anatomy-wise image-text pairs, learning directly from clinical reports without manual annotation. [1] The motivation is that existing supervised strategies remain limited in scope, because each task typically needs its own labelled dataset. [1] Learning from reports removes the annotation bottleneck. It also means the model learns whatever errors and omissions those reports contain. The abstract does not say how report quality was checked.

The evaluation spanned centres, but the abstract does not give the numbers

The scope of the evaluation is wide. Throughout internal and external evaluations across multiple centers and varied clinical scenarios, RADAR achieved high diagnostic performance for 18 anatomical structures and 146 imaging findings. [1] The study design is model development followed by retrospective internal and external validation, plus a reader study. The abstract does not report the number of centres, the external sample sizes, or per-finding performance. It therefore cannot show how performance varies across the 146 findings, and rare findings are where a generalist model is most likely to fall short.

The authors conclude that RADAR demonstrat[es] that generalist AI can match human experts in general and complicated radiology tasks. [1] The reader study, as summarised, supports a narrower claim: radiologists performed better with the tool than without it. "Matching experts" and "improving experts when used as an aid" are different findings, and they would lead to different deployment decisions. The main limitation is that all the evidence in the abstract is retrospective. There is no prospective deployment, no patient outcome, and no workflow measure.

A narrative review argues that technical fixes for unreliability fall short

The Cell paper is a narrative review rather than an empirical study, and it sets a stricter standard for success. The authors define reliability as clinically acceptable performance across contexts such as patient and clinician populations and over time under real-world deployment conditions. [2] They group the failures they see into three classes: erroneous model outputs, clinically unjustified performance differences across patient populations, and performance degradation under changing deployment conditions. [2]

The central argument is that engineering alone is not enough. We argue that current technical solutions are often insufficient to address the challenges that lead to unreliable AI, thus motivating the need for lifecycle-aware evaluation, continuous monitoring, and governance. [2] The review covers both predictive and generative AI settings, which matters because RADAR is a vision-language model and spans both categories. [2] The JClub summary refers to more than 20 failure modes, but the abstract itself gives no count. The main limitation of any narrative review also applies here: the abstract does not describe how the literature was selected, so we cannot judge how complete the synthesis is.

The two papers disagree on what multi-centre validation proves

The two papers differ on how much multi-centre validation shows. RADAR presents external testing across centres as evidence of generalisation. [1] The Cell review treats generalisation across sites as only part of reliability. In its framing, a model must also hold up across clinician populations and over time as deployment conditions change. [2] A retrospective external test set captures one point in time, so it cannot show whether performance degrades as scanners, protocols or case mix change.

The review's second failure class, unjustified performance differences across patient populations, raises a question the RADAR abstract does not answer. [2] The abstract gives no subgroup results by sex, age, body habitus or scanner vendor. These are the analyses the Cell authors would ask for before calling a model reliable. They are not necessarily missing from the full paper, but we cannot tell from the abstract.

The two papers do agree on something. Neither claims that a model's performance at release is the end of the evaluation. RADAR's reader study implicitly accepts that the tool's value depends on how clinicians use it. The review states directly that monitoring and governance have to continue after release. [2]

The JAMA guidance cannot be assessed from what is available

The third paper is JAMA's updated guidance on authors' use of AI in medical publication, by Flanagin, Perlis and Bibbins-Domingo. Abstract: not available [3] We do not know yet what the guidance changes, what it requires of authors, or how it relates to earlier policy. We will not guess at its content. Its appearance in the same week as the other two papers suggests that journals, as well as clinical researchers, are setting rules for AI use. That reading rests on timing, not on anything the document states.

What to watch for next

For RADAR, the full paper should answer several questions. What were the baseline and assisted sensitivities, and how did specificity change? How many cases did the readers interpret? How does performance vary across the 146 findings, particularly rare ones? Are there subgroup results by patient population and scanner? The more decisive evidence would come from a prospective deployment that tracks performance over time. That is the test the Cell review argues current evaluations skip. [2]

For the reliability agenda, the open question is what "lifecycle-aware evaluation" looks like in practice for a model covering 146 findings. Monitoring one prediction target is already demanding, and monitoring many at once is a larger problem that neither paper addresses. For the JAMA guidance, the next step is simply to read the full text. Once it is available, it will be worth checking whether it says anything about reporting standards for AI tools evaluated in research, or only about AI used in writing manuscripts.

This is one of the early evidences of use of AI and promising results in healthcare system. The way AI is advancing now it is not far that we will have reliable applications to interpret complex radiology results in the smartphones. The question is how will it be regulated?