nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-4533E70B·Framework v5.13·Pipeline v0.4·Evaluated 18 Nov 2026·OECD: 5.8 Media and communications·Subfield: Sociology and Political Science·Basis: Full text·Type: Cross-sectional study

Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?

2025·Learned Publishing·10.1002/leap.2018 ↗

Primary claim

[When asked to evaluate 217 retracted or concerning articles,] none of its 6510 reports mentioned that the articles were retracted or had relevant errors

Supported · repeated quality evaluations support the descriptive result, with unmeasured sensitivity of the fixed-string search
In brief

An exploratory audit documents retraction non-mention and frequent claim affirmation by ChatGPT 4o-mini, but its prompts cannot establish lack of retraction awareness.

ReliabilityCaveats4 of 4 checks completeRead the 3 entries ↓
Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

ABSTRACT Large language models (LLMs) like ChatGPT seem to be increasingly used for information seeking and analysis, including to support academic literature reviews. To test whether the results might sometimes include retracted research, we identified 217 retracted or otherwise concerning academic studies with high altmetric scores and asked ChatGPT 4o‐mini to evaluate their quality 30 times each. Surprisingly, none of its 6510 reports mentioned that the articles were retracted or had relevant errors, and it gave 190 relatively high scores (world leading, internationally excellent, or close). The 27 articles with the lowest scores were mostly accused of being weak, although the topic (but not the article) was described as controversial in five cases (e.g., about hydroxychloroquine for COVID‐19). In a follow‐up investigation, 61 claims were extracted from retracted articles from the set, and ChatGPT 4o‐mini was asked 10 times whether each was true. It gave a definitive yes or a positive response two‐thirds of the time, including for at least one statement that had been shown to be false over a decade ago. The results therefore emphasise, from an academic knowledge perspective, the importance of verifying information from LLMs when using them for information seeking or analysis.

Key Claims

Primary finding · Supported

[When asked to evaluate 217 retracted or concerning articles,] none of its 6510 reports mentioned that the articles were retracted or had relevant errors

Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

repeated quality evaluations support the descriptive result, with unmeasured sensitivity of the fixed-string search

Abstract; Section 3.1
Quote verified
Secondary · Supported

[ChatGPT 4o-mini, asked 10 times whether each of 61 retracted-article claims was true,] gave a definitive yes or a positive response two-thirds of the time

404 of 610 responses support the approximate fraction, with subjective claim selection and response coding limiting generalisation

Abstract; Section 3.2; Table 1
Quote verified
Secondary · Unsupported

The results overall suggest that ChatGPT does not have awareness of retractions or other indicators of problematic content for high-profile academic research

quality-scoring prompts cannot distinguish lack of awareness from failure to volunteer retraction information

Discussion and Conclusions
Quote verified
Secondary · Supported

The average ChatGPT research quality scores for each of the 217 articles in the retracted and potentially concerning articles set had a very weak correlation with their altmetric rank

reported rank correlation supports the descriptive association, but altmetric attention does not isolate retraction awareness

Section 3.1, paragraph 4
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Paired tests document retraction related behaviour

The study combines 6,510 article-quality reports with 610 responses about extracted claims, separating spontaneous retraction warnings from affirmation of claims drawn from retracted work.

↳ Sections 3.1–3.2

#methodological rigour

Repeated queries and independent response coding

The methods identify the model version and query period, repeat each article evaluation 30 times and each claim query 10 times, and report agreement with an independent response coder.

↳ Sections 2.3–2.4

#positioning

Training cutoff limitation receives quantitative examination

The discussion checks concern dates against the model's training cutoff, finding that only eight of 217 articles first attracted serious concerns afterward. It also distinguishes retraction from proof that an underlying claim is false.

↳ Introduction; Sections 3.2 and 4

Limitations

#methodological rigour

Quality prompts cannot establish retraction awareness

Asking for quality scores from titles and abstracts tests whether the model volunteers retraction information, not whether it can supply that information when asked. No direct status probe or comparison set separates those possibilities.

↳ Section 2.3; Section 4

#methodological rigour

Response measurement leaves important uncertainty unresolved

Most extracted claims lack independently established truth status, response coding achieves Kappa=0.598, and disagreements are finalised by the first author. The fixed-string search also has no assessment of missed alternative wording.

↳ Sections 2.3–2.4 and 3.2

#positioning

Conclusions exceed stated model and date

The final discussion generalises to ChatGPT and June 2025 despite testing one model in September 2024 and acknowledging that different prompts could reveal retraction knowledge.

↳ Section 4, opening limitations and final paragraph

Does this match your read? Add your assessment ↓

Nabu Commentary

The paired audit provides a bounded descriptive result: the searched quality reports did not flag retractions, and positive answers accounted for roughly two-thirds of the claim responses. Repeated sampling, a specified model version and independent coding make the procedures traceable. The principal limitation is identification: quality-scoring prompts cannot establish that the model lacks retraction knowledge, while largely unverified claim truth prevents interpreting affirmation as general factual error. Section 4's training-cutoff analysis adds useful context, but its final generalisation exceeds the stated model, date and prompt restrictions.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.0±0.1

Contribution
3.4

The paired article-level and claim-level audit supplies a concrete descriptive finding about retraction-related behaviour. Its advance remains bounded by one compact model, selected high-attention articles and prompts that do not identify awareness.

Introduction; Sections 2.3–2.4 and 3.1–3.2
Methodological Rigour
2.8

Pinned model identification, repeated queries and independent response coding support a reproducible descriptive audit. Indirect awareness testing, unassessed search sensitivity and uncertain claim validity limit the conclusions obtainable from those procedures.

Sections 2.3–2.4; Section 3.2
Reporting
3.5

The two research questions, query procedures and corresponding results are readily distinguishable. The partial-truth percentage error and dense sample-breakdown prose reduce precision without obscuring the overall sequence.

Sections 1–3; Section 2.2; Section 3.2 and Table 1
Positioning
2.8

The paper engages post-retraction citation research, distinguishes reasons for retraction and quantifies the training-cutoff limitation. Its final generalisation about ChatGPT and June 2025 does not preserve the stated single-model, single-period and prompt-dependent scope.

Introduction; Section 4, limitations, cutoff analysis and final paragraph

Lower confidence on Contribution — domain match limited.

Reliability

Caveats4 of 4 checks


Results checkNoted
  • "partially true (9.5%), or consistent with research (9.5%)" (Section 3.2, paragraph 1) vs "The statement is/can be considered somewhat/partially true/accurate." (Table 1, row 3). Table 1 gives 79/610 × 100 = 12.95%, rounding to 13.0%, for partially true responses, while its three positive categories total 404/610 × 100 = 66.23%. The result itself is unaffected.
Reporting and transparencyNoted

The privacy or ethical restriction cited for withholding public bibliographic inputs and model outputs is not explained. Data are available on request, and conflicts of interest are disclosed.

Conflict of interest: declared · Data availability: declared

Reference IntegrityNoted

46 references in manuscript 38 of 40 checkable references found in an index 6 have no canonical index record — counted, but not index-checkable 2 references confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 18 Nov 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaProducts and technology · SDG 9
Type of advanceEmpirical
Route to useSuggested
AdvanceAdvances
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessControlled
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The claim was tested through 6,510 controlled API queries to one fixed ChatGPT version in a single period, pinning down the effect under researcher-controlled conditions without independent confirmation.

From the paper
“The results therefore emphasise, from an academic knowledge perspective, the importance of verifying information from LLMs when using them for information seeking or analysis.”

The paper names a real-world practice of verifying LLM information for information seeking and analysis but identifies no specific profession, organisation or setting that should apply it.

Pathway

AI developers such as OpenAI would build retraction awareness into LLM systems, while researchers and students using LLMs verify outputs before relying on them.

Barrier

Evidence comes from one LLM version, 217 articles and a single period, so it may not generalise to other models or survive rapid LLM evolution.

Evaluated 18 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

    Rodionov, Assylbekov · 2026 · Computer and Information Sciences

    Primary claim

    Across 30 models and 10 repeated runs, aggregate IFR-a is 0.93 ± 0.004 and aggregate IFR-i is 0.809 ± 0.009.

    Supported
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  2. 'Put the illness in a box': a longitudinal interpretative phenomenological analysis of changes in a sufferer's pictorial representations of pain following participation in a pain management programme

    Nizza, Smith, Kirkham · 2017 · Clinical medicine

    Primary claim

    as control is regained, a sufferer’s relationship with their chronic pain can visibly change

    Supported
    Evidence Score
    3.2±0.2
    Reliability
    No concerns
  3. Real-Time Inequality

    Thomas Blanchet, Léon Mouillet, Emmanuel Saez, et al. · 2022 · Economics and business

    Primary claim

    our methodology accurately predicts growth during downturns.

    Supported
    Evidence Score
    3.9±0.1
    Reliability
    Caveats

How the rubric works →