# Performance of AI Tools in Citing Retracted Literature : Content Analysis

- **DOI:** [10.2196/88766](https://doi.org/10.2196/88766)
- **Rating:** Limited (2.9 out of 5)
- **Verdict:** A nine-tool benchmark documents inconsistent handling of retracted papers, but its composite endpoint cannot isolate retraction-detection performance.
- **Framework version:** v5.13
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-18
- **Evaluation page:** https://www.nabu.science/paper/10.2196%2F88766

## Key claims

1. None of the evaluated AI tools consistently handled retracted articles correctly. No model achieved perfect accuracy across all question sets.
   - Support: Supported
   - Type: Primary finding
   - Basis: observed composite failures across the tested tools within a small, selected article sample
   - Location: Abstract, Results
2. ChatGPT 5 performed best, defined by the primary outcome of achieving fully correct responses to all five predefined tasks (5/5) for the highest number of retracted articles, correctly answering all five questions for 8 of 15 articles (53.3%).
   - Support: Supported
   - Type: Secondary
   - Basis: the reported composite count supports the observed ranking, with precision limited by selected articles and model-generated keywords
   - Location: Abstract, Results
3. Research-focused tools (SciSpace, ScienceOS, and Consensus) failed to produce a single fully correct response set.
   - Support: Supported
   - Type: Secondary
   - Basis: zero complete sets in Table 1 within a small sample and a workflow with tool-task fit limitations
   - Location: Abstract, Results
4. Retracted articles were frequently included in topic overviews without warning, with error rates exceeding 40% in several tools.
   - Support: Tentative
   - Type: Secondary
   - Basis: the selected-sample counts establish unflagged inclusion but contradict the clause that several tools exceeded 40%
   - Location: Abstract, Results
5. OpenEvidence only reported data for a subset of our retracted articles as it is only used in health care literature. It demonstrated strong performance in topic overviews but low accuracy in identifying retracted articles.
   - Support: Tentative
   - Type: Secondary
   - Basis: no subset size or corresponding performance results are reported in the main Results section
   - Location: Abstract, Results

## Strengths

- **Cross-platform comparison extends earlier single-model studies** The benchmark applies common tasks to six general-purpose and three research-focused tools, making their observed workflow performance directly comparable.
  - Dimension: contribution
  - Location: Methods, Study Design; Table 1; Discussion, Comparison With Previous Literature
- **Repeated prompts and independent human ratings** Duplicate queries, new chat sessions for each article, and two independent raters make the evaluation workflow inspectable. Overall interreviewer kappa is reported as 0.73.
  - Dimension: methodological rigour
  - Location: Methods, Data Extraction; Results, Reliability of Tools
- **Explicit discussion of consequential design limitations** The discussion acknowledges the small medical-heavy sample, tested-model keyword generation, ambiguity of article nonretrieval, and newer model versions in the retrospective control.
  - Dimension: positioning
  - Location: Discussion, Strengths and Limitations

## Limitations

- **Abstract performance statements exceed reported support** Only SciSpace exceeds a 40% topic-overview error rate, at 8/15, while four tools are exactly at 40%, contradicting the abstract's claim that several exceed it. OpenEvidence performance is also asserted without corresponding counts or subset size in the Results.
  - Dimension: reporting
  - Location: Abstract, Results; Table 1; Results
- **Composite success obscures retraction detection ability** Failure on the five-task composite does not identify failure on explicit retraction questions. Topic-overview scoring also conflates nonretrieval with correct handling, while ChatGPT-generated keywords leave comparison bias unquantified.
  - Dimension: methodological rigour
  - Location: Methods, Study Aims and Retracted Articles; Table 1; Discussion, Strengths and Limitations
- **Universal conclusions outrun the selected sample** Fifteen selected articles, 13 medical, cannot establish performance across all free-access systems and scientific fields. Categorical inability wording also overstates results showing partial success on the tested workflow.
  - Dimension: positioning
  - Location: Abstract, Conclusions; Discussion, Strengths and Limitations; Conclusions

## Reliability checks

Caveats (4 of 4 checks).

- **Results check:** Noted
  - "Retracted articles were frequently included in topic overviews without warning, with error rates exceeding 40% in several tools." (Abstract, Results) vs "SciSpace, on the other hand, had the most incorrect cases with 8 out of 15 cases" (Results, Retracted Articles in Topic Overviews). Counting Table 1's topic-overview results gives one tool above 40% (SciSpace, 8/15 = 53.33%) and four exactly at 40% (6/15). The claim is not supported by the paper's own data.
  - Minor reporting notes (1):
    - "calculating the percentage of conflicting ratings between reviewers for each AI tool" (Methods, Statistical Analysis) vs "Differences between repeated answers resulting in a rating change occurred in each GenAI tool." (Results, Reliability of Tools). Methods defines reviewer disagreement, whereas Results describes disagreement between repeated tool responses. The effect on the result is unclear.
- **Reporting and transparency:** Noted. OpenEvidence performance, hallucination screening, and equality between article subsets are asserted or described without corresponding results in the supplied main text. Registration occurred after data acquisition began.
  - Conflict of interest: declared · Ethics approval: declared · Preregistration: not stated · Data availability: declared
- **Reference integrity:** Clean
  - 33 references in manuscript
  - 31 of 31 checkable references found in an index
  - 2 are books, websites or datasets — counted, but not index-checkable
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-18
