Loading evaluation data…
Abstract Retrieval-augmented and LLM-based (RAG-LLM) evidence-search tools, including Consensus, Ai2 Paper Finder, ChatGPT, Gemini, and Claude, are increasingly used by clinicians and researchers. Whether a realistic query reliably surfaces relevant evidence or leaves systematic gaps that could shape clinical evidence and research synthesis remains unclear. Using a prospectively assembled, non-public gold-standard corpus to avoid benchmark contamination, we assessed five platforms across 15 query formulations. Primary outcomes were formulation-level recall (evidence retrieved per query formulation) and single-query zero-retrieval probability (formulations returning no relevant evidence from a domain); pooled platform recall (evidence retrieved at least once across all formulations) was a secondary capacity benchmark. Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%. For the largest evidence category, single-query zero-retrieval probability ranged from 47% to 80% across platforms; one platform showed a marked pre-2016 evidence gap; and 12.0% of evidence was never retrieved by any platform, with never-retrieval significantly higher for conference proceedings than journal articles (38.9% vs 4.6%; p < 0.001). Evidence gaps varied by platform, evidence category, publication year, and venue type, highlighting potential retrieval bias and visibility blind spots, and supporting domain-specific evaluation before RAG-LLM outputs are used in clinical or research workflows.
Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.
directly measured descriptive ranges across matched formulations within the 83-study benchmark and June 2025 evaluation window
12.0% of evidence was never retrieved by any platform, with never-retrieval significantly higher for conference proceedings than journal articles (38.9% vs 4.6%; p < 0.001).
consistent benchmark counts support the venue association, although sparse proceedings and venue–year confounding limit interpretation
By substance category, tobacco-related studies (n = 28), the largest evidence category, showed a 33-percentage-point spread in single-query zero-retrieval probability (p = 0.043), ranging from 47% for Ai2 Finder to 80% for ChatGPT-4.1.
the descriptive spread is supported, but 15 non-random formulations and uncorrected stratum testing limit inferential certainty
The strongest two-platform combination in this benchmark, Ai2 Finder and Claude-Sonnet-4.0, retrieved 83.1% of the gold-standard studies, compared with 72.3% for the strongest individual platform
observed union coverage demonstrates complementarity within this corpus, not a generally optimal platform pair
Derived from the full evaluation — not a separate score.
Strengths
The 83-study reference corpus was assembled before testing through seven-database searching, independent screening, and arbitration. Keeping the curated list non-public during evaluation reduces the risk of testing reproduction of an existing benchmark list.
↳ Methods, Gold standard development
Formulation-level recall, pooled capacity, domain-level absence, precision, and run consistency distinguish what users receive from what platforms can retrieve across repeated searches.
↳ Methods, Study design and Performance analysis; Results, retrieval performance and complementarity
The discussion presents the results as a warning rather than a general ranking and explicitly separates observed gaps from hypothetical mechanisms and longer-term visibility effects.
↳ Discussion, benchmark scope, mechanisms, and limitations
Limitations
The corpus concerns NLP identification of substance use disorders in electronic health records, with testing confined to June 2025 browser interfaces. The results do not establish coverage patterns across other clinical topics or platform versions.
↳ Methods, Gold standard development and Platform selection; Discussion, limitations
Domain analyses report tobacco p = 0.043 and cannabinoid p = 0.012 without adjustment across the tested strata or uncertainty intervals. These findings warrant more caution than the directly observed overall recall ranges.
↳ Results, Single-query zero-retrieval probability by evidence domain; Methods, Statistical analysis
There is no matched conventional-search comparator, and venue differences are not disentangled from publication-period and related corpus characteristics. The relationship between exhaustive eligible-study recall and the intended rapid-overview use case also remains incompletely examined.
↳ Methods, Study design; Discussion, publication period and venue patterns, use-case framing, and limitations
The independently assembled corpus and matched 225-execution design provide a substantial basis for describing retrieval coverage within this benchmark. Adjudicated matching, repeated-run consistency measures, and the Ai2 output-cap sensitivity analysis strengthen that interpretation. The subgroup analyses merit greater caution because multiplicity control does not extend across the domain strata and uncertainty intervals are absent. The discussion appropriately avoids causal explanations and general platform rankings, although venue–year confounding and comparison with conventional search remain incompletely addressed.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Strong3.9
Confidence mediumThe benchmark separates evidence available from one formulation from pooled platform capacity and documents complementary retrieval and persistent gaps. Its advance is bounded by one clinical NLP topic and a short platform-evaluation window.
Corpus construction, matched triplicate testing, adjudicated title matching, and the output-cap sensitivity analysis support the central descriptive comparison. Uncorrected testing across domain strata, absent uncertainty intervals, and remaining output-volume differences reduce confidence in finer subgroup interpretations.
The paper makes the distinction between formulation-level retrieval and pooled capacity visible and repeatedly limits interpretation to the benchmark. The risk-of-bias title and single-query label for triplicate-pooled outputs introduce bounded terminology ambiguities.
Relevant citation-bias, reproducibility, and bibliographic-coverage literature informs the interpretation, while unknown mechanisms and changing interfaces are explicitly acknowledged. Prior search-recall evaluations, venue–year confounding, and the fit between exhaustive corpus coverage and rapid-overview use receive less attention.
No concerns4 of 4 checks
The reported numbers and statistics are consistent with the stated results.
Ethics approval is declared unnecessary for analysis of public outputs and published literature, and competing interests, data availability, and code availability are disclosed. Availability declarations were not independently verified.
Conflict of interest: declared · Ethics approval: declared · Data availability: declared · Code availability: declared
74 references in manuscript 53 of 55 checkable references found in an index 19 have no canonical index record — counted, but not index-checkable 3 references confirmed by manual review
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
Testing actual browser interfaces and reporting reusable prompts, outputs, and analysis code supports practical reuse. The evaluation does not validate a clinical deployment protocol or performance in operational evidence-search workflows.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Colavizza, Hrynaszkiewicz, Staden, et al. · 2020 · Other social sciences
We also find an association between articles that include statements that link to data in a repository and up to 25.36% (± 1.07%) higher citation impact on average, using a citation prediction model.
SupportedMert Demirer, Leon Musolff, Liyuan Yang · 2026 · Mechanical engineering
Sida Peng, Eirini Kalliamvakou, Peter Cihon, et al. · 2023 · Computer and information sciences