# Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?

- **DOI:** [10.1002/leap.2018](https://doi.org/10.1002/leap.2018)
- **Rating:** Sound (3.0 out of 5)
- **Verdict:** An exploratory audit documents retraction non-mention and frequent claim affirmation by ChatGPT 4o-mini, but its prompts cannot establish lack of retraction awareness.
- **Framework version:** v5.13
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-18
- **Evaluation page:** https://www.nabu.science/paper/10.1002%2Fleap.2018

## Key claims

1. [When asked to evaluate 217 retracted or concerning articles,] none of its 6510 reports mentioned that the articles were retracted or had relevant errors
   - Support: Supported
   - Type: Primary finding
   - Basis: repeated quality evaluations support the descriptive result, with unmeasured sensitivity of the fixed-string search
   - Location: Abstract; Section 3.1
2. [ChatGPT 4o-mini, asked 10 times whether each of 61 retracted-article claims was true,] gave a definitive yes or a positive response two-thirds of the time
   - Support: Supported
   - Type: Secondary
   - Basis: 404 of 610 responses support the approximate fraction, with subjective claim selection and response coding limiting generalisation
   - Location: Abstract; Section 3.2; Table 1
3. The results overall suggest that ChatGPT does not have awareness of retractions or other indicators of problematic content for high-profile academic research
   - Support: Unsupported
   - Type: Secondary
   - Basis: quality-scoring prompts cannot distinguish lack of awareness from failure to volunteer retraction information
   - Location: Discussion and Conclusions
4. The average ChatGPT research quality scores for each of the 217 articles in the retracted and potentially concerning articles set had a very weak correlation with their altmetric rank
   - Support: Supported
   - Type: Secondary
   - Basis: reported rank correlation supports the descriptive association, but altmetric attention does not isolate retraction awareness
   - Location: Section 3.1, paragraph 4

## Strengths

- **Paired tests document retraction related behaviour** The study combines 6,510 article-quality reports with 610 responses about extracted claims, separating spontaneous retraction warnings from affirmation of claims drawn from retracted work.
  - Dimension: contribution
  - Location: Sections 3.1–3.2
- **Repeated queries and independent response coding** The methods identify the model version and query period, repeat each article evaluation 30 times and each claim query 10 times, and report agreement with an independent response coder.
  - Dimension: methodological rigour
  - Location: Sections 2.3–2.4
- **Training cutoff limitation receives quantitative examination** The discussion checks concern dates against the model's training cutoff, finding that only eight of 217 articles first attracted serious concerns afterward. It also distinguishes retraction from proof that an underlying claim is false.
  - Dimension: positioning
  - Location: Introduction; Sections 3.2 and 4

## Limitations

- **Quality prompts cannot establish retraction awareness** Asking for quality scores from titles and abstracts tests whether the model volunteers retraction information, not whether it can supply that information when asked. No direct status probe or comparison set separates those possibilities.
  - Dimension: methodological rigour
  - Location: Section 2.3; Section 4
- **Response measurement leaves important uncertainty unresolved** Most extracted claims lack independently established truth status, response coding achieves Kappa=0.598, and disagreements are finalised by the first author. The fixed-string search also has no assessment of missed alternative wording.
  - Dimension: methodological rigour
  - Location: Sections 2.3–2.4 and 3.2
- **Conclusions exceed stated model and date** The final discussion generalises to ChatGPT and June 2025 despite testing one model in September 2024 and acknowledging that different prompts could reveal retraction knowledge.
  - Dimension: positioning
  - Location: Section 4, opening limitations and final paragraph

## Reliability checks

Caveats (4 of 4 checks).

- **Results check:** Noted
  - "partially true (9.5%), or consistent with research (9.5%)" (Section 3.2, paragraph 1) vs "The statement is/can be considered somewhat/partially true/accurate." (Table 1, row 3). Table 1 gives 79/610 × 100 = 12.95%, rounding to 13.0%, for partially true responses, while its three positive categories total 404/610 × 100 = 66.23%. The result itself is unaffected.
- **Reporting and transparency:** Noted. The privacy or ethical restriction cited for withholding public bibliographic inputs and model outputs is not explained. Data are available on request, and conflicts of interest are disclosed.
  - Conflict of interest: declared · Data availability: declared
- **Reference integrity:** Noted
  - 46 references in manuscript
  - 38 of 40 checkable references found in an index
  - 6 have no canonical index record — counted, but not index-checkable
  - 2 references confirmed by manual review
  - Could not be verified automatically — queued for manual review:
    - 1741487, 2025, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/ep.2018 by UNIVERSITY OF SHEFFIELD, Wiley Online Library on [07/08/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use. OA articles are governed by the applicable Creative Commons License. (Cited DOI resolves in no index (Crossref, DataCite), but the title search could not corroborate it (title_search_result='not_attempted'); left unresolved for manual review.)
    - 1741487, 2025, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/bitp.2018 by UNIVERSITY OF SHEFFIELD, Wiley Online Library on [07/08/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use. OA articles are governed by the applicable Creative Commons License. (Cited DOI resolves in no index (Crossref, DataCite), but the title search could not corroborate it (title_search_result='not_attempted'); left unresolved for manual review.)
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-18
