nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-76D8EDF1·Framework v5.13·Pipeline v0.4·Evaluated 16 Nov 2026·OECD: 3.3 Health sciences·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Cross-sectional study

Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study

Alexander D Gibson et al.Alexander D Gibson, David N Borg, Nicole M White et al.·2026·medRxiv·10.64898/2026.11.02.26367135 ↗

Primary claim

not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice

Tentative · cross-sectional distributions cannot distinguish intentional hacking from rounding, overfitting or publication selection
In brief

A full-text audit extends evidence on prediction-metric clustering and reporting mismatches, but its attribution to intentional hacking exceeds the descriptive design.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Unlikely to hold — the claim's weakest requirement is that it works through the path described. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

Background Previous research has shown that clinical prediction models have likely been 'hacked' to meet arbitrary performance thresholds. However, this work only assessed abstracts, could not identify models with a perfect AUC (area under the receiver operating characteristic curve) of 1, and did not examine other performance metrics. Methods We collected data from full texts and validated a large language model to extract data from a random sample of Q1 journal articles for 2025. We examined if hacking occurred in the AUC value, sensitivity and specificity, and checked if articles correctly reported the AUC value from corresponding receiver operating characteristic (ROC) curves. Results We identified 2,001 prediction model articles that reported 15,656 AUC values. We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias. Around 1 in 40 AUC values were reported outside of their confidence intervals, and around 1 in 8 ROC curves did not match their reported AUC value. Conclusion We have shown that not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice. Excess models that are not representative of their true performance may misguide clinicians, distort the literature, and make it harder to find prediction models that are useful for practice.

Key Claims

Primary finding · Tentative

not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice

Unlikely to hold — the claim's weakest requirement is that it works through the path described. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

cross-sectional distributions cannot distinguish intentional hacking from rounding, overfitting or publication selection

Abstract, Conclusion
Quote verified
Secondary · Tentative

We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias.

descriptive residual patterns lack a rounding benchmark capable of distinguishing the proposed mechanisms

Abstract, Results; Section 3.0
Quote verified
Secondary · Well-supported

Around 1 in 40 AUC values were reported outside of their confidence intervals [227 of 9,002 AUC values, 2.5%]

the reported count directly supports the descriptive fraction without establishing that each instance represents an error

Abstract, Results; Section 3.0
Quote verified
Secondary · Tentative

around 1 in 8 ROC curves did not match their reported AUC value [by ≥ 0.02, with no evidence of systematic upward alteration, mean difference 0.0017].

selected curve sampling and uncalibrated digitisation error limit interpretation of the discrepancy rate

Abstract, Results; Section 3.2
Quote verified
Secondary · Tentative

more than two-thirds of articles not reporting confidence intervals [for AUC values]

the extracted reporting audit gives the opposite complement, with 33.6% rather than more than two-thirds omitting intervals

Conclusions
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Full texts broaden the empirical audit

The study examines 15,656 AUC values from 2,001 articles and extends the earlier abstract-only analysis to sensitivity and specificity.

↳ Introduction; Methods 2.4; Results 3.0

#contribution

Graphical cross checks add distinct evidence

Comparing reported AUCs with estimates digitised from ROC curves adds a separate check on numerical reporting. Differences occur in both directions rather than showing systematic upward alteration.

↳ Methods 2.7; Results 3.2

#methodological rigour

Distribution models account for repeated values

Negative binomial spline models include article-level random intercepts, and alternative boundary knots examine fitting artefacts near an AUC of 1.

↳ Methods 2.5

Limitations

#methodological rigour

Clustering does not identify intentional hacking

The Abstract and Conclusions attribute threshold concentrations to hacking, but the descriptive design cannot separate manipulation from rounding, small-sample effects or publication selection. Reporting precision is not assessed as a source of the observed spikes.

↳ Methods 2.5; Discussion 4.0; Conclusions

#methodological rigour

ROC mismatch estimates retain unexamined uncertainty

The 672 subsequently included articles were excluded from ROC extraction and validation, and digitisation error is not calibrated against the 0.02 threshold. Calling the resulting discrepancies incorrectly reported models generalises beyond what the selected curve comparison establishes.

↳ Results 3.0; Methods 2.7; Results 3.2; Conclusions

#reporting

Conclusion reverses confidence interval reporting prevalence

The Conclusions say more than two-thirds of articles omitted confidence intervals, but Results report intervals in 1,329 of 2,001 articles. The omitted proportion is 33.6%, approximately one-third, not more than two-thirds.

↳ Results 3.0; Conclusions

Does this match your read? Add your assessment ↓

Nabu Commentary

The full-text audit meaningfully extends the earlier abstract-based work by adding sensitivity, specificity and ROC comparisons. Article-level random intercepts address repeated metric values, but the distributional analysis does not distinguish intentional manipulation from ordinary rounding or selection. The ROC discrepancy estimate is further limited by uncalibrated digitisation error and exclusion of the 672 subsequently included articles from validation and graphical checks. Relevant guidance and alternative explanations are discussed, but those alternatives are not sufficiently weighed before manipulation is preferred. The conclusion also reverses the reported prevalence of confidence-interval omission, creating a substantive reliability concern.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.1±0.2

Contribution
3.8

Full-text extraction, inclusion of sensitivity and specificity, and ROC comparisons meaningfully extend the prior abstract-only AUC analysis. The advance is descriptive rather than an identification of hacking mechanisms.

Introduction; Methods 2.4–2.7; Results 3.0 and 3.2
Methodological Rigour
2.7

The audit combines large-scale extraction, clustered distribution models and graphical cross-checks, but unassessed rounding, digitisation uncertainty and selection limit interpretation. Intentional hacking is not identified by the design, and the reversed CI-prevalence conclusion is separately surfaced through the reliability flag rather than a reliability cap.

Methods 2.4–2.7; Results 3.0 and 3.2; Conclusions
Reporting
3.0

The organisation makes the main analyses accessible, but the conclusion reverses CI-reporting prevalence and turns selected curve discrepancies into general model-reporting errors. Inconsistent flow and ROC counts further reduce numerical precision.

Results 3.0; Section 3.2 and Figure 5; Conclusions
Positioning
3.1

The paper identifies the closest predecessor and engages prediction-model reporting guidance and alternative explanations. Rounding is not explicitly addressed, and the discussion does not adequately weigh the alternatives before preferring manipulation.

Introduction; Discussion 4.0; Limitations 4.1

Reliability

Caveats4 of 4 checks


Results checkNoted
  • more than two-thirds of articles not reporting confidence intervals: "more than two-thirds of articles not reporting confidence intervals" (Conclusions) vs "Confidence intervals were reported for 9,002 of the 15,656 AUC values (58%) in 1,329 of the 2,001 articles (66%)." (Results 3.0). Subtracting 1,329 from 2,001 gives 672 articles without reported AUC confidence intervals, or 33.6%, not more than two-thirds. → Not supported by the paper's own data
  • around 1 in 8 ROC curves did not match their reported AUC value: "From 200 ROC curves, 189 (95%) AUC values were able to be calculated" (Section 3.2 ROC Curves, first sentence) vs "from a random sample of 187 ROC curves" (Figure 5 legend). The text and caption give different denominators; the 26 discrepant curves represent 13.8% of 189 or 13.9% of 187, approximately one in seven. → Conclusion holds
Reporting and transparencyNoted

Protocol registration followed the literature search; its timing relative to analysis is not established. Ethics exemption and public data and code availability are declared.

Conflict of interest: not stated · Ethics approval: declared · Data availability: declared · Code availability: declared

Reference IntegrityClean

46 references in manuscript 45 of 45 checkable references found in an index 1 are books, websites or datasets — counted, but not index-checkable

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 16 Nov 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 3
Type of advanceEmpirical
Route to useSpecified
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The claim is tested directly on a large real-world sample of 2,001 published Q1 articles with 15,656 extracted AUC values, the exact population the claim concerns.

From the paper
“Journals should not recommend but mandate future prediction model development, validation and impact assessments be reported in full against the TRIPOD+AI guidelines”

The paper names journals and policy makers as actors and specifies the decision: mandating full TRIPOD+AI reporting and prioritising real-world outcome evidence for model use in practice.

Pathway

Journal editors and guideline bodies would mandate TRIPOD+AI-compliant reporting and appraisal, keeping inflated prediction models out of clinical practice.

Barrier

The observational design cannot distinguish deliberate manipulation from publication bias and overfitting, and journal incentives to publish impressive metrics resist editorial reform.

Evaluated 16 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Examining a citation advantage for publicly accessible articles in special education journals

    Jesse I. Fleming, Bryan G. Cook · 2026 · Education

    Primary claim

    publicly accessible articles were associated with 26% (β = 0.26, p < 0.001) and 41% (β = 0.41, p < 0.001) more citations than closed articles in WoS and GS, respectively, after controlling for other variables

    Supported
    Evidence Score
    3.3±0.1
    Reliability
    Caveats
  2. Predicting replicability-Analysis of survey and prediction market data from large-scale forecasting projects

    Gordon, Viganola, Dreber, et al. · 2021 · Other social sciences

    Primary claim

    prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103)

    Well-supported
    Evidence Score
    3.6±0.1
    Reliability
    Caveats
  3. Human-AI Collaboration for Estimating Scientific Replicability

    Tatiana Chakravorti, Robert Fraleigh, Timothy Fritton, et al. · 2026 · Computer and Information Sciences

    Primary claim

    hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts

    Tentative
    Evidence Score
    3.3±0.2
    Reliability
    Caveats

How the rubric works →