Multiple search outcomes within one protocol
The study combines hallucination, relevance, duplication, and cross-user consistency across six tools, extending beyond a single hallucination-rate comparison.
↳ Introduction; Methods, Research Questions
Abstract
OBJECTIVE: This investigation compared six AI chat tools (ChatGPT, Claude, Copilot, Gemini, OpenEvidence, and Perplexity) to each other and to PubMed to assess their utility in medical literature searches.
METHODS: Each tool was prompted with two standardized prompts simulating an academic and a clinical literature need. Prompts were repeated three times per tool, and results recorded for each prompt iteration.
RESULTS: Main findings include the range in number of real and hallucinated articles for each AI tool, from no hallucinations in OpenEvidence to more than 80% hallucinated citations in ChatGPT. Across all tools, most real articles were assessed as relevant. Continued prompting of all tools produced duplicate results. Most tools showed very little consistency in citations provided to different researchers, suggesting low probability of replicating search results.
CONCLUSIONS: Though limited by small sample size, inconsistent data, and the rapid pace of change in LLM technology, this study suggests some advantages and drawbacks of using AI tools for literature searching compared with PubMed. AI tools with few or no hallucinations and high relevance, such as OpenEvidence and Perplexity, performed well in terms of simply supplying a few article citations on the topic, but small results sets and lack of consistency and transparency limit their value for thorough research. These results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for systematic reviews or other in-depth literature searches but may be adequate for limited information needs or as a complement to database or web searches.
Key Claims
Primary findingTentative
Our results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for use in any in-depth literature searches
Unlikely to hold — the claim's weakest requirement is that the effect exists at all. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
two topics and mixed, superseded versions cannot establish present-day suitability across in-depth search tasks
Conclusions
SecondarySupported
Every tool except OpenEvidence returned hallucinated citations
verified citations support the observed pattern within two topics and mixed-version sessions
Discussion, Hallucinations
SecondaryTentative
Overall relevance ranged from 78% (Copilot) to 100% (OpenEvidence): most tools returned at least 90% relevant citations.
two-topic relevance judgments concern only real citations and use broad criteria without reported agreement statistics
Results, Relevant Citations
SecondaryTentative
PubMed retrieved 93 irrelevant results, for a relevancy score of 31%, lower than any AI tool (Figure 1).
conditional AI denominators and simple PubMed queries do not establish a comparable end-to-end relevance ranking
Results, Comparing AI and PubMed Results
SecondaryTentative
tools with little evidence of hallucination (in this study, OpenEvidence and Perplexity AI) may be adequate for limited information needs or as a complement to database or web searches
practical adequacy is inferred from two-topic descriptive results rather than validated user workflows
Conclusions
Derived from the full evaluation — not a separate score.
Strengths · Limitations
Strengths
Explicit prompts and citation verification rules
The Methods specify prompt wording, PubMed queries, citation verification, relevance adjudication, and the consistency formula, allowing readers to understand how the outcomes were constructed.
↳ Methods, Prompts, Data Collection, and Evaluation Criteria
Concrete relevance to library search instruction
The discussion distinguishes limited article needs from in-depth searching and identifies students and librarians as potential users of the findings.
↳ Discussion, Potential Implications for Information Seekers; Conclusions
Limitations
Relevance ranking relies on incompatible denominators
AI relevance excludes hallucinated citations, whereas PubMed relevance includes all retrieved results. On an all-citation basis, ChatGPT could have at most 18.8% relevant citations, below PubMed's 30.6%, contradicting the reported ordering under that common denominator.
↳ Results, Real and Hallucinated Articles; Comparing AI and PubMed Results
Repeated observations lack justified inferential treatment
The ANOVA does not establish an appropriate error structure for repeated, unequal observations, and missing OpenEvidence responses are mean-imputed. Mixed tool versions and access-based participation further complicate tool-level comparisons.
↳ Methods, Study Design and Participants; Data Analysis
Current suitability conclusion exceeds dated evidence
The Limitations explicitly describe a baseline rather than an assessment of current performance. The conclusion nevertheless states what AI tools are not presently viable for across any in-depth literature searching.
↳ Discussion, Limitations; Conclusions
Nabu Commentary
The common six-tool protocol adds practical comparative evidence on citation existence, relevance, duplication, and consistency. Transparent prompts and verification rules support interpretation, but heterogeneous versions and the treatment of repeated observations limit inferential precision. The relevance comparison excludes hallucinations from AI denominators, so it cannot establish the reported ordering on a common all-result basis. The statement that these data do not assess current performance also conflicts with the present-day replacement conclusion. The findings therefore function best as a dated instructional baseline rather than a general rule for current tools.
Check another paper
Author Note / Field Feedback
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Dimension breakdown
Study Quality
Sound3.0
Confidence mediumContribution
3.6
The common six-tool protocol adds evidence on citation existence, relevance, duplication, and cross-user consistency. This is a meaningful incremental extension of earlier citation-hallucination studies, bounded by two topics and a time-specific tool sample.
Introduction; Methods, Research Questions
Methodological Rigour
2.7
Prompts, verification procedures, and data handling are described, but repeated observations, mean imputation, access-based selection, and unquantified measurement uncertainty limit execution. The denominator-incompatible PubMed ranking is surfaced separately through the Red reliability flag, without a reliability cap on Methodological Rigour.
Methods, Evaluation Criteria and Data Analysis; Results, Comparing AI and PubMed Results
Reporting
3.5
The progression from research questions to findings is readily followed. Central language nevertheless invites broader current-use and relevance-ranking conclusions than the benchmark establishes.
Abstract; Results, Comparing AI and PubMed Results; Conclusions
Positioning
2.8
The article engages prior hallucination research and discusses multiple design limitations. Its present-day replacement judgment conflicts with the explicit statement that the study does not assess current performance, making the contextual ceiling binding.
Introduction; Discussion, Limitations; Conclusions
Reliability
Caveats4 of 4 checks
Results checkNoted
- PubMed retrieved 93 irrelevant results, for a relevancy score of 31%, lower than any AI tool (Figure 1): "PubMed retrieved 93 irrelevant results, for a relevancy score of 31%, lower than any AI tool (Figure 1)." (Results, Comparing AI and PubMed Results) vs "Considering only unique citations, ChatGPT still produced most hallucinations, with 125/154 article citations (81%) hallucinated." (Results, Real and Hallucinated Articles). ChatGPT's maximum all-citation relevant fraction is (154−125)/154=18.8%, below PubMed's 41/134=30.6%, so the reported ordering does not hold with comparable denominators. → Not supported by the paper's own data
Reporting and transparencyClean
The article names an OSF repository for associated data, with no affirmative research-conduct concern established. Uninspected repository materials limit assessability but do not establish a reliability defect.
Data availability: declared
Reference IntegrityClean
23 of 23 checkable references verified
29 references in manuscript 6 are books, websites or datasets — counted, but not index-checkable
References not verified automatically are resolved manually within 24 hours.
Post-Publication RecordClean
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Last checked: 27 Oct 2026
Sources: Crossref (incl. Retraction Watch) ✓
Readiness
Where this paper’s evidence sits on the path from initial observation to real-world use.
Demonstrated
The findings can supply preliminary training examples and cautionary guidance. Superseded versions and lack of operational systematic-review validation prevent stronger readiness claims.
Methods, Study Design and Participants; Discussion, Limitations and Consistency
AI-generated, human-governed. Something look off? Contact us to request a review.