A Unified Framework for Statistical Inference and Study Design of Comparative F1 and Fβ Scores under Paired Classifier Evaluations

psF1pair improves statistical inference for paired classifier evaluations by accounting for correlations, reducing error and increasing power in biomedical classification studies.

• Why it matters: Accurate assessment of classifier performance is vital in healthcare, especially with imbalanced biomedical data, but current methods often ignore dependencies between classifiers, risking overestimated errors and reduced statistical power.
• What they did: The authors developed psF1pair, an advanced statistical framework that explicitly models correlations between classifier pairs, supported by extensive simulations and real-world imaging data analysis, and provided as a free R package.
• The result: Incorporating correlations leads to narrower confidence intervals and higher statistical power, enabling more reliable evaluation and study design of predictive models in biomedical research.