Full texts broaden the empirical audit
The study examines 15,656 AUC values from 2,001 articles and extends the earlier abstract-only analysis to sensitivity and specificity.
↳ Introduction; Methods 2.4; Results 3.0
Background Previous research has shown that clinical prediction models have likely been 'hacked' to meet arbitrary performance thresholds. However, this work only assessed abstracts, could not identify models with a perfect AUC (area under the receiver operating characteristic curve) of 1, and did not examine other performance metrics. Methods We collected data from full texts and validated a large language model to extract data from a random sample of Q1 journal articles for 2025. We examined if hacking occurred in the AUC value, sensitivity and specificity, and checked if articles correctly reported the AUC value from corresponding receiver operating characteristic (ROC) curves. Results We identified 2,001 prediction model articles that reported 15,656 AUC values. We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias. Around 1 in 40 AUC values were reported outside of their confidence intervals, and around 1 in 8 ROC curves did not match their reported AUC value. Conclusion We have shown that not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice. Excess models that are not representative of their true performance may misguide clinicians, distort the literature, and make it harder to find prediction models that are useful for practice.
not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice
cross-sectional distributions cannot distinguish intentional hacking from rounding, overfitting or publication selection
We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias.
descriptive residual patterns lack a rounding benchmark capable of distinguishing the proposed mechanisms
Around 1 in 40 AUC values were reported outside of their confidence intervals [227 of 9,002 AUC values, 2.5%]
the reported count directly supports the descriptive fraction without establishing that each instance represents an error
around 1 in 8 ROC curves did not match their reported AUC value [by ≥ 0.02, with no evidence of systematic upward alteration, mean difference 0.0017].
selected curve sampling and uncalibrated digitisation error limit interpretation of the discrepancy rate
more than two-thirds of articles not reporting confidence intervals [for AUC values]
the extracted reporting audit gives the opposite complement, with 33.6% rather than more than two-thirds omitting intervals
Derived from the full evaluation — not a separate score.
Strengths
The study examines 15,656 AUC values from 2,001 articles and extends the earlier abstract-only analysis to sensitivity and specificity.
↳ Introduction; Methods 2.4; Results 3.0
Comparing reported AUCs with estimates digitised from ROC curves adds a separate check on numerical reporting. Differences occur in both directions rather than showing systematic upward alteration.
↳ Methods 2.7; Results 3.2
Negative binomial spline models include article-level random intercepts, and alternative boundary knots examine fitting artefacts near an AUC of 1.
↳ Methods 2.5
Limitations
The Abstract and Conclusions attribute threshold concentrations to hacking, but the descriptive design cannot separate manipulation from rounding, small-sample effects or publication selection. Reporting precision is not assessed as a source of the observed spikes.
↳ Methods 2.5; Discussion 4.0; Conclusions
The 672 subsequently included articles were excluded from ROC extraction and validation, and digitisation error is not calibrated against the 0.02 threshold. Calling the resulting discrepancies incorrectly reported models generalises beyond what the selected curve comparison establishes.
↳ Results 3.0; Methods 2.7; Results 3.2; Conclusions
The Conclusions say more than two-thirds of articles omitted confidence intervals, but Results report intervals in 1,329 of 2,001 articles. The omitted proportion is 33.6%, approximately one-third, not more than two-thirds.
↳ Results 3.0; Conclusions
The full-text audit meaningfully extends the earlier abstract-based work by adding sensitivity, specificity and ROC comparisons. Article-level random intercepts address repeated metric values, but the distributional analysis does not distinguish intentional manipulation from ordinary rounding or selection. The ROC discrepancy estimate is further limited by uncalibrated digitisation error and exclusion of the 672 subsequently included articles from validation and graphical checks. Relevant guidance and alternative explanations are discussed, but those alternatives are not sufficiently weighed before manipulation is preferred. The conclusion also reverses the reported prevalence of confidence-interval omission, creating a substantive reliability concern.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Full-text extraction, inclusion of sensitivity and specificity, and ROC comparisons meaningfully extend the prior abstract-only AUC analysis. The advance is descriptive rather than an identification of hacking mechanisms.
The audit combines large-scale extraction, clustered distribution models and graphical cross-checks, but unassessed rounding, digitisation uncertainty and selection limit interpretation. Intentional hacking is not identified by the design, and the reversed CI-prevalence conclusion is separately surfaced through the reliability flag rather than a reliability cap.
The organisation makes the main analyses accessible, but the conclusion reverses CI-reporting prevalence and turns selected curve discrepancies into general model-reporting errors. Inconsistent flow and ROC counts further reduce numerical precision.
The paper identifies the closest predecessor and engages prediction-model reporting guidance and alternative explanations. Rounding is not explicitly addressed, and the discussion does not adequately weigh the alternatives before preferring manipulation.
Caveats4 of 4 checks
Protocol registration followed the literature search; its timing relative to analysis is not established. Ethics exemption and public data and code availability are declared.
Conflict of interest: not stated · Ethics approval: declared · Data availability: declared · Code availability: declared
46 references in manuscript 45 of 45 checkable references found in an index 1 are books, websites or datasets — counted, but not index-checkable
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
The claim is tested directly on a large real-world sample of 2,001 published Q1 articles with 15,656 extracted AUC values, the exact population the claim concerns.
“Journals should not recommend but mandate future prediction model development, validation and impact assessments be reported in full against the TRIPOD+AI guidelines”
The paper names journals and policy makers as actors and specifies the decision: mandating full TRIPOD+AI reporting and prioritising real-world outcome evidence for model use in practice.
Journal editors and guideline bodies would mandate TRIPOD+AI-compliant reporting and appraisal, keeping inflated prediction models out of clinical practice.
The observational design cannot distinguish deliberate manipulation from publication bias and overfitting, and journal incentives to publish impressive metrics resist editorial reform.
Evaluated 16 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Jesse I. Fleming, Bryan G. Cook · 2026 · Education
publicly accessible articles were associated with 26% (β = 0.26, p < 0.001) and 41% (β = 0.41, p < 0.001) more citations than closed articles in WoS and GS, respectively, after controlling for other variables
SupportedGordon, Viganola, Dreber, et al. · 2021 · Other social sciences
prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103)
Well-supportedTatiana Chakravorti, Robert Fraleigh, Timothy Fritton, et al. · 2026 · Computer and Information Sciences
hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts
Tentative