# Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study

- **DOI:** [10.64898/2026.11.02.26367135](https://doi.org/10.64898/2026.11.02.26367135)
- **Rating:** Sound (3.1 out of 5)
- **Verdict:** A full-text audit extends evidence on prediction-metric clustering and reporting mismatches, but its attribution to intentional hacking exceeds the descriptive design.
- **Framework version:** v5.13
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-16
- **Evaluation page:** https://www.nabu.science/paper/10.64898%2F2026.11.02.26367135

## Key claims

1. not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice
   - Support: Tentative
   - Type: Primary finding
   - Basis: cross-sectional distributions cannot distinguish intentional hacking from rounding, overfitting or publication selection
   - Location: Abstract, Conclusion
2. We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias.
   - Support: Tentative
   - Type: Secondary
   - Basis: descriptive residual patterns lack a rounding benchmark capable of distinguishing the proposed mechanisms
   - Location: Abstract, Results; Section 3.0
3. Around 1 in 40 AUC values were reported outside of their confidence intervals [227 of 9,002 AUC values, 2.5%]
   - Support: Well-supported
   - Type: Secondary
   - Basis: the reported count directly supports the descriptive fraction without establishing that each instance represents an error
   - Location: Abstract, Results; Section 3.0
4. around 1 in 8 ROC curves did not match their reported AUC value [by ≥ 0.02, with no evidence of systematic upward alteration, mean difference 0.0017].
   - Support: Tentative
   - Type: Secondary
   - Basis: selected curve sampling and uncalibrated digitisation error limit interpretation of the discrepancy rate
   - Location: Abstract, Results; Section 3.2
5. more than two-thirds of articles not reporting confidence intervals [for AUC values]
   - Support: Tentative
   - Type: Secondary
   - Basis: the extracted reporting audit gives the opposite complement, with 33.6% rather than more than two-thirds omitting intervals
   - Location: Conclusions

## Strengths

- **Full texts broaden the empirical audit** The study examines 15,656 AUC values from 2,001 articles and extends the earlier abstract-only analysis to sensitivity and specificity.
  - Dimension: contribution
  - Location: Introduction; Methods 2.4; Results 3.0
- **Graphical cross checks add distinct evidence** Comparing reported AUCs with estimates digitised from ROC curves adds a separate check on numerical reporting. Differences occur in both directions rather than showing systematic upward alteration.
  - Dimension: contribution
  - Location: Methods 2.7; Results 3.2
- **Distribution models account for repeated values** Negative binomial spline models include article-level random intercepts, and alternative boundary knots examine fitting artefacts near an AUC of 1.
  - Dimension: methodological rigour
  - Location: Methods 2.5

## Limitations

- **Clustering does not identify intentional hacking** The Abstract and Conclusions attribute threshold concentrations to hacking, but the descriptive design cannot separate manipulation from rounding, small-sample effects or publication selection. Reporting precision is not assessed as a source of the observed spikes.
  - Dimension: methodological rigour
  - Location: Methods 2.5; Discussion 4.0; Conclusions
- **ROC mismatch estimates retain unexamined uncertainty** The 672 subsequently included articles were excluded from ROC extraction and validation, and digitisation error is not calibrated against the 0.02 threshold. Calling the resulting discrepancies incorrectly reported models generalises beyond what the selected curve comparison establishes.
  - Dimension: methodological rigour
  - Location: Results 3.0; Methods 2.7; Results 3.2; Conclusions
- **Conclusion reverses confidence interval reporting prevalence** The Conclusions say more than two-thirds of articles omitted confidence intervals, but Results report intervals in 1,329 of 2,001 articles. The omitted proportion is 33.6%, approximately one-third, not more than two-thirds.
  - Dimension: reporting
  - Location: Results 3.0; Conclusions

## Reliability checks

Caveats (4 of 4 checks).

- **Results check:** Noted
  - more than two-thirds of articles not reporting confidence intervals: "more than two-thirds of articles not reporting confidence intervals" (Conclusions) vs "Confidence intervals were reported for 9,002 of the 15,656 AUC values (58%) in 1,329 of the 2,001 articles (66%)." (Results 3.0). Subtracting 1,329 from 2,001 gives 672 articles without reported AUC confidence intervals, or 33.6%, not more than two-thirds. → Not supported by the paper's own data
  - around 1 in 8 ROC curves did not match their reported AUC value: "From 200 ROC curves, 189 (95%) AUC values were able to be calculated" (Section 3.2 ROC Curves, first sentence) vs "from a random sample of 187 ROC curves" (Figure 5 legend). The text and caption give different denominators; the 26 discrepant curves represent 13.8% of 189 or 13.9% of 187, approximately one in seven. → Conclusion holds
  - Minor reporting notes (1):
    - Descriptive: "After screening, there were 2,166 out of 3,996 articles remaining. One-hundred and seventy articles were excluded during full text data collection" (Results 3.0, article flow) vs "There were 2,001 articles included for analysis." (Results 3.0, analysis sample). Subtracting the 170 stated exclusions from 2,166 yields 1,996, five fewer than the stated analysis sample. → Effect on result unclear
- **Reporting and transparency:** Noted. Protocol registration followed the literature search; its timing relative to analysis is not established. Ethics exemption and public data and code availability are declared.
  - Conflict of interest: not stated · Ethics approval: declared · Data availability: declared · Code availability: declared
- **Reference integrity:** Clean
  - 46 references in manuscript
  - 45 of 45 checkable references found in an index
  - 1 are books, websites or datasets — counted, but not index-checkable
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-16
