nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-AE6A47B3·Framework v5.13·Pipeline v0.4·Evaluated 18 Nov 2026·OECD: 5.8 Media and communications·Subfield: Education·Basis: Full text·Type: Cross-sectional study

Performance of AI Tools in Citing Retracted Literature : Content Analysis

Sebastian Labenbacher et al.Sebastian Labenbacher, Maximilian Niederer, Sascha Hammer et al.·2026·Journal of Medical Internet Research·10.2196/88766 ↗

Primary claim

None of the evaluated AI tools consistently handled retracted articles correctly. No model achieved perfect accuracy across all question sets.

Supported · observed composite failures across the tested tools within a small, selected article sample
In brief

A nine-tool benchmark documents inconsistent handling of retracted papers, but its composite endpoint cannot isolate retraction-detection performance.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Possible — the claim's weakest requirement is that the effect exists at all. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

Background: Generative artificial intelligence (GenAI) tools are increasingly used in scientific research to support literature searches, evidence synthesis, and manuscript preparation. While these systems promise substantial efficiency gains, concerns have emerged regarding their reliability, particularly their tendency to cite inaccurate, fabricated, or retracted literature. The unrecognized inclusion of retracted studies poses a serious risk to research integrity and evidence-based decision-making. Whether commonly used GenAI tools can reliably detect, exclude, or transparently communicate the retraction status of scientific publications remains unclear. Objective: This study aimed to evaluate the ability of freely available GenAI tools to correctly handle retracted scientific articles during literature searches. Primary and secondary outcomes focused on accuracy, reliability, and consistency in recognizing retracted literature. Methods: In this pragmatic trial, nine widely used free-access GenAI tools (ChatGPT 4, ChatGPT 5, Claude, Gemini, Perplexity, Microsoft Copilot, SciSpace, ScienceOS, and Consensus) were evaluated. Each tool was asked five predefined, standardized questions addressing topic overview, article identification, article summarization, and explicit assessment of retraction status. Overall, 15 retracted articles (the 10 most cited and 5 most recently retracted as of May 23, 2025) were selected from the Retraction Watch database. All questions were repeated twice to assess intratool consistency. Responses were independently rated as correct or incorrect by 2 researchers. Descriptive statistics summarized performance, and comparisons between general-purpose and research-focused AI tools were conducted using descriptive statistics. Interreviewer agreement was assessed using Cohen kappa coefficient. Results: None of the evaluated AI tools consistently handled retracted articles correctly. No model achieved perfect accuracy across all question sets. ChatGPT 5 performed best, defined by the primary outcome of achieving fully correct responses to all five predefined tasks (5/5) for the highest number of retracted articles, correctly answering all five questions for 8 of 15 articles (53.3%). Research-focused tools (SciSpace, ScienceOS, and Consensus) failed to produce a single fully correct response set. Retracted articles were frequently included in topic overviews without warning, with error rates exceeding 40% in several tools. When specifically asked about retraction status, most systems failed to provide correct or complete information. OpenEvidence only reported data for a subset of our retracted articles as it is only used in health care literature. It demonstrated strong performance in topic overviews but low accuracy in identifying retracted articles. Conclusions: Freely available GenAI tools are currently not able to detect, exclude, or appropriately flag retracted scientific literature. The widespread and confident reproduction of retracted studies represents a substantial threat to research integrity, particularly in medical and evidence-based fields. Until retraction-aware verification mechanisms are systematically integrated, independent source checking remains essential when using AI-assisted literature tools.

Key Claims

Primary finding · Supported

None of the evaluated AI tools consistently handled retracted articles correctly. No model achieved perfect accuracy across all question sets.

Possible — the claim's weakest requirement is that the effect exists at all. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

observed composite failures across the tested tools within a small, selected article sample

Abstract, Results
Quote verified
Secondary · Supported

ChatGPT 5 performed best, defined by the primary outcome of achieving fully correct responses to all five predefined tasks (5/5) for the highest number of retracted articles, correctly answering all five questions for 8 of 15 articles (53.3%).

the reported composite count supports the observed ranking, with precision limited by selected articles and model-generated keywords

Abstract, Results
Quote verified
Secondary · Supported

Research-focused tools (SciSpace, ScienceOS, and Consensus) failed to produce a single fully correct response set.

zero complete sets in Table 1 within a small sample and a workflow with tool-task fit limitations

Abstract, Results
Quote verified
Secondary · Tentative

Retracted articles were frequently included in topic overviews without warning, with error rates exceeding 40% in several tools.

the selected-sample counts establish unflagged inclusion but contradict the clause that several tools exceeded 40%

Abstract, Results
Quote verified
Secondary · Tentative

OpenEvidence only reported data for a subset of our retracted articles as it is only used in health care literature. It demonstrated strong performance in topic overviews but low accuracy in identifying retracted articles.

no subset size or corresponding performance results are reported in the main Results section

Abstract, Results
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Cross-platform comparison extends earlier single-model studies

The benchmark applies common tasks to six general-purpose and three research-focused tools, making their observed workflow performance directly comparable.

↳ Methods, Study Design; Table 1; Discussion, Comparison With Previous Literature

#methodological rigour

Repeated prompts and independent human ratings

Duplicate queries, new chat sessions for each article, and two independent raters make the evaluation workflow inspectable. Overall interreviewer kappa is reported as 0.73.

↳ Methods, Data Extraction; Results, Reliability of Tools

#positioning

Explicit discussion of consequential design limitations

The discussion acknowledges the small medical-heavy sample, tested-model keyword generation, ambiguity of article nonretrieval, and newer model versions in the retrospective control.

↳ Discussion, Strengths and Limitations

Limitations

#reporting

Abstract performance statements exceed reported support

Only SciSpace exceeds a 40% topic-overview error rate, at 8/15, while four tools are exactly at 40%, contradicting the abstract's claim that several exceed it. OpenEvidence performance is also asserted without corresponding counts or subset size in the Results.

↳ Abstract, Results; Table 1; Results

#methodological rigour

Composite success obscures retraction detection ability

Failure on the five-task composite does not identify failure on explicit retraction questions. Topic-overview scoring also conflates nonretrieval with correct handling, while ChatGPT-generated keywords leave comparison bias unquantified.

↳ Methods, Study Aims and Retracted Articles; Table 1; Discussion, Strengths and Limitations

#positioning

Universal conclusions outrun the selected sample

Fifteen selected articles, 13 medical, cannot establish performance across all free-access systems and scientific fields. Categorical inability wording also overstates results showing partial success on the tested workflow.

↳ Abstract, Conclusions; Discussion, Strengths and Limitations; Conclusions

Does this match your read? Add your assessment ↓

Nabu Commentary

The common prompts, repeated queries, and independent ratings provide a useful descriptive comparison across nine tools. Table 1 supports inconsistent success on the complete workflow, but that composite cannot isolate performance on explicit retraction questions. Topic-overview scoring and model-generated keywords add uncertainty to between-tool comparisons. The discussion acknowledges consequential limitations, yet the universal conclusion exceeds the selected sample, and the abstract's threshold claim conflicts with the table.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
2.9±0.2

Contribution
3.2

The common-task comparison extends cited single-model and single-field studies to general-purpose and research-focused tools. Its contribution remains a descriptive snapshot based on 15 selected, predominantly medical articles.

Introduction; Discussion, Comparison With Previous Literature; Table 1
Methodological Rigour
2.8

Standardized prompts, duplicate runs, and independent ratings support an inspectable comparison, but the composite endpoint and retrieval-sensitive scoring limit inference about retraction detection. The Red flag separately identifies the abstract's incorrect cross-tool threshold statement.

Methods, Study Aims and Data Extraction; Table 1; Discussion, Strengths and Limitations
Reporting
3.0

The aims, workflow, and primary table are accessible. Categorical inability wording, unsupported OpenEvidence performance statements, and inconsistent terminology reduce reporting precision.

Abstract, Results and Conclusions; Methods, Statistical Analysis; Table 1
Positioning
2.8

The discussion engages relevant earlier studies and explicitly acknowledges sampling, keyword, retrieval, and version limitations. Its universal conclusion nevertheless extends beyond those stated boundaries.

Discussion, Comparison With Previous Literature and Strengths and Limitations; Conclusions

Reliability

Caveats4 of 4 checks


Results checkNoted
  • "Retracted articles were frequently included in topic overviews without warning, with error rates exceeding 40% in several tools." (Abstract, Results) vs "SciSpace, on the other hand, had the most incorrect cases with 8 out of 15 cases" (Results, Retracted Articles in Topic Overviews). Counting Table 1's topic-overview results gives one tool above 40% (SciSpace, 8/15 = 53.33%) and four exactly at 40% (6/15). The claim is not supported by the paper's own data.
Reporting and transparencyNoted

OpenEvidence performance, hallucination screening, and equality between article subsets are asserted or described without corresponding results in the supplied main text. Registration occurred after data acquisition began.

Conflict of interest: declared · Ethics approval: declared · Preregistration: not stated · Data availability: declared

Reference IntegrityClean

33 references in manuscript 31 of 31 checkable references found in an index 2 are books, websites or datasets — counted, but not index-checkable

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 18 Nov 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaHealth · Generic health relevance
Type of advanceEmpirical
Route to useSpecified
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim is demonstrated in a real-world pragmatic evaluation of nine tools with 675 prompts, but it is a single-team, single-time-point snapshot with no independent replication.

From the paper
“Journals and peer reviewers should require explicit disclosure of any AI assistance in literature searches and verify cited sources independently.”

The discussion names concrete actors (AI developers, journals, peer reviewers) and specifies the decisions and methods they should adopt, such as requiring disclosure and verifying cited sources independently.

Pathway

Journal editors, peer reviewers, and researchers would act by mandating disclosure of AI-assisted searches and independently verifying citations before findings enter evidence syntheses and clinical guidance.

Barrier

AI models change rapidly so the performance snapshot may not persist, and fixing the problem requires developers to integrate live retraction databases into their tools.

Evaluated 18 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Scientific misconduct, questionable research practices, and research climate: empirical approaches to integrity problems

    Nazarovets · 2026 · Other social sciences

    Primary claim

    aggregate harm is rarely defined explicitly and is instead inferred from perceived prevalence and severity of practices

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats
  2. Fake publications in biomedical science: red-flagging method indicates mass production

    Sabel, Knaack, Gigerenzer, et al. · 2025 · Clinical medicine

    Primary claim

    Applying Bayes' rule resulted in an estimate of 5.8% actual fakes in the biomedical literature.

    Tentative
    Evidence Score
    2.5±0.1
    Reliability
    Caveats
  3. What is an interpretative phenomenological analysis case study?

    Daniel Rodger, Jonathan Ashley Smith · 2025 · Health sciences

    Primary claim

    While IPA is frequently used with small samples, single-participant IPA case studies can offer unique insights into deeply personal or rare experiences.

    Tentative
    Evidence Score
    3.1±0.1
    Reliability
    Caveats

How the rubric works →