ShortKit-ML: an open toolkit for measuring shortcut potential in medical imaging embeddings

A calibrated conditional linear probe measures shortcut potential on frozen medical imaging embeddings, rejecting the null at 0.02 on synthetic data and detecting attribute information in 21/21 CheXpert cells.

• Why it matters: Shortcut reliance can persist when a classifier uses sensitive attributes not explained by task labels, but measuring this from embeddings alone with a controlled false positive rate is difficult.
• What they did: ShortKit-ML tests dependence between an embedding E and sensitive attribute A given task label Y using a conditional logistic probe with out-of-fold AUROC, calibrated by permuting A within Y levels across patients.
• The result: On CheXpert, probe scores across 42 cells correlated with the true positive rate gap of confounded heads (Spearman 0.60, 95% CI 0.31 to 0.80), and mitigation changed the gap by at most 0.01.

AI summary of the abstract · may contain errors · not medical advice · read the paper.