nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-DAD9BC0F·Framework v5.10·Pipeline v0.4·Evaluated 20 Oct 2026·OECD: 1.2 Computer and information sciences·Subfield: Information Systems·Basis: Full text·Type: Dataset or benchmark paper

Blind spots in AI-assisted healthcare evidence search: multiplatform evaluation of clinical retrieval gaps and risk-of-bias

Farrag et al.Farrag, Soliman, Azubuike et al.·2026·npj Digital Medicine·10.1038/s41746-026-03277-y ↗

Primary claim

Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.

Well-supported · directly measured descriptive ranges across matched formulations within the 83-study benchmark and June 2025 evaluation window
In brief

A systematic multiplatform audit demonstrates incomplete AI-assisted evidence retrieval, with a single clinical topic limiting the reach of its findings.

ReliabilityNo concerns4 of 4 checks completeSee what was checked ↓
Abstract

Abstract Retrieval-augmented and LLM-based (RAG-LLM) evidence-search tools, including Consensus, Ai2 Paper Finder, ChatGPT, Gemini, and Claude, are increasingly used by clinicians and researchers. Whether a realistic query reliably surfaces relevant evidence or leaves systematic gaps that could shape clinical evidence and research synthesis remains unclear. Using a prospectively assembled, non-public gold-standard corpus to avoid benchmark contamination, we assessed five platforms across 15 query formulations. Primary outcomes were formulation-level recall (evidence retrieved per query formulation) and single-query zero-retrieval probability (formulations returning no relevant evidence from a domain); pooled platform recall (evidence retrieved at least once across all formulations) was a secondary capacity benchmark. Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%. For the largest evidence category, single-query zero-retrieval probability ranged from 47% to 80% across platforms; one platform showed a marked pre-2016 evidence gap; and 12.0% of evidence was never retrieved by any platform, with never-retrieval significantly higher for conference proceedings than journal articles (38.9% vs 4.6%; p < 0.001). Evidence gaps varied by platform, evidence category, publication year, and venue type, highlighting potential retrieval bias and visibility blind spots, and supporting domain-specific evaluation before RAG-LLM outputs are used in clinical or research workflows.

Key Claims

Primary finding · Well-supported

Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.

directly measured descriptive ranges across matched formulations within the 83-study benchmark and June 2025 evaluation window

Abstract; Results, Single-query and overall platform retrieval performance
Quote verified
Secondary · Supported

12.0% of evidence was never retrieved by any platform, with never-retrieval significantly higher for conference proceedings than journal articles (38.9% vs 4.6%; p < 0.001).

consistent benchmark counts support the venue association, although sparse proceedings and venue–year confounding limit interpretation

Abstract; Results, Single-query zero-retrieval probability by evidence domain
Quote verified
Secondary · Supported

By substance category, tobacco-related studies (n = 28), the largest evidence category, showed a 33-percentage-point spread in single-query zero-retrieval probability (p = 0.043), ranging from 47% for Ai2 Finder to 80% for ChatGPT-4.1.

the descriptive spread is supported, but 15 non-random formulations and uncorrected stratum testing limit inferential certainty

Results, Single-query zero-retrieval probability by evidence domain
Quote verified
Secondary · Supported

The strongest two-platform combination in this benchmark, Ai2 Finder and Claude-Sonnet-4.0, retrieved 83.1% of the gold-standard studies, compared with 72.3% for the strongest individual platform

observed union coverage demonstrates complementarity within this corpus, not a generally optimal platform pair

Discussion, Combining platforms improved coverage but did not achieve complete retrieval
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#methodological rigour

Independent corpus supports a distinct retrieval benchmark

The 83-study reference corpus was assembled before testing through seven-database searching, independent screening, and arbitration. Keeping the curated list non-public during evaluation reduces the risk of testing reproduction of an existing benchmark list.

↳ Methods, Gold standard development

#contribution

Complementary outcomes expose different retrieval limitations

Formulation-level recall, pooled capacity, domain-level absence, precision, and run consistency distinguish what users receive from what platforms can retrieve across repeated searches.

↳ Methods, Study design and Performance analysis; Results, retrieval performance and complementarity

#reporting

Conclusions retain the benchmark's interpretive boundaries

The discussion presents the results as a warning rather than a general ranking and explicitly separates observed gaps from hypothetical mechanisms and longer-term visibility effects.

↳ Discussion, benchmark scope, mechanisms, and limitations

Limitations

#contribution

One topic limits broader performance conclusions

The corpus concerns NLP identification of substance use disorders in electronic health records, with testing confined to June 2025 browser interfaces. The results do not establish coverage patterns across other clinical topics or platform versions.

↳ Methods, Gold standard development and Platform selection; Discussion, limitations

#methodological rigour

Subgroup inference exceeds the uncertainty reporting

Domain analyses report tobacco p = 0.043 and cannabinoid p = 0.012 without adjustment across the tested strata or uncertainty intervals. These findings warrant more caution than the directly observed overall recall ranges.

↳ Results, Single-query zero-retrieval probability by evidence domain; Methods, Statistical analysis

#positioning

Comparative context leaves important distinctions unresolved

There is no matched conventional-search comparator, and venue differences are not disentangled from publication-period and related corpus characteristics. The relationship between exhaustive eligible-study recall and the intended rapid-overview use case also remains incompletely examined.

↳ Methods, Study design; Discussion, publication period and venue patterns, use-case framing, and limitations

Does this match your read? Add your assessment ↓

Nabu Commentary

The independently assembled corpus and matched 225-execution design provide a substantial basis for describing retrieval coverage within this benchmark. Adjudicated matching, repeated-run consistency measures, and the Ai2 output-cap sensitivity analysis strengthen that interpretation. The subgroup analyses merit greater caution because multiplicity control does not extend across the domain strata and uncertainty intervals are absent. The discussion appropriately avoids causal explanations and general platform rankings, although venue–year confounding and comparison with conventional search remain incompletely addressed.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.9±0.2

Contribution
4.0

The benchmark separates evidence available from one formulation from pooled platform capacity and documents complementary retrieval and persistent gaps. Its advance is bounded by one clinical NLP topic and a short platform-evaluation window.

Methods, Study design and Gold standard development; Results, Single-query and overall platform retrieval performance
Methodological Rigour
3.8

Corpus construction, matched triplicate testing, adjudicated title matching, and the output-cap sensitivity analysis support the central descriptive comparison. Uncorrected testing across domain strata, absent uncertainty intervals, and remaining output-volume differences reduce confidence in finer subgroup interpretations.

Methods, Performance analysis and Statistical analysis; Results, Single-query zero-retrieval probability by evidence domain
Reporting
4.2

The paper makes the distinction between formulation-level retrieval and pooled capacity visible and repeatedly limits interpretation to the benchmark. The risk-of-bias title and single-query label for triplicate-pooled outputs introduce bounded terminology ambiguities.

Title; Methods, Study design; Discussion, limitations and concluding synthesis
Positioning
3.9

Relevant citation-bias, reproducibility, and bibliographic-coverage literature informs the interpretation, while unknown mechanisms and changing interfaces are explicitly acknowledged. Prior search-recall evaluations, venue–year confounding, and the fit between exhaustive corpus coverage and rapid-overview use receive less attention.

Introduction; Discussion, mechanisms, benchmark scope, and limitations

Reliability

No concerns4 of 4 checks


Results checkClean

The reported numbers and statistics are consistent with the stated results.

Reporting and transparencyClean

Ethics approval is declared unnecessary for analysis of public outputs and published literature, and competing interests, data availability, and code availability are disclosed. Availability declarations were not independently verified.

Conflict of interest: declared · Ethics approval: declared · Data availability: declared · Code availability: declared

Reference IntegrityClean

74 references in manuscript 54 of 54 checkable references found in an index 20 have no canonical index record — counted, but not index-checkable 5 references confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 20 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaHealth · Generic health relevance
Type of advanceEmpirical
Route to useSpecified
AdvanceAdvances
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

All evidence comes from a single controlled evaluation on one 83-study non-public benchmark during one evaluation window, with no independent replication across other clinical topics or platform versions.

From the paper
“For clinicians and researchers seeking a rapid initial overview, these findings support checking more than one evidence-search platform when feasible”

The paper names clinicians and researchers as the actors, the rapid-initial-overview setting, and the concrete decision to check multiple platforms rather than trust one query; it also directs platform developers to treat coverage and reproducibility as safety features.

Pathway

Clinicians, researchers, and systematic reviewers would cross-check multiple evidence-search platforms before acting on single-query results, while platform developers add coverage disclosure and reproducible retrieval modes.

Barrier

The evidence rests on one 83-study corpus covering a single topic in one evaluation window, so gaps may differ across topics, languages, and changing platform versions.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Artificial intelligence in language learning: biometric feedback and adaptive reading for improved comprehension and reduced anxiety

    Hang Yuan · 2025 · Educational sciences

    Primary claim

    Quantitative results revealed significant improvements in reading comprehension, motivation, anxiety reduction, and cognitive load management among participants using the AI platform compared to the control group.

    Tentative
    Evidence Score
    2.2±0.2
    Reliability
    Concerns
  2. The citation advantage of linking publications to research data

    Colavizza, Hrynaszkiewicz, Staden, et al. · 2020 · Other social sciences

    Primary claim

    We also find an association between articles that include statements that link to data in a repository and up to 25.36% (± 1.07%) higher citation impact on average, using a citation prediction model.

    Supported
    Evidence Score
    3.8±0.2
    Reliability
    Caveats
  3. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

    Mert Demirer, Leon Musolff, Liyuan Yang · 2026 · Mechanical engineering

    Evidence Score
    3.9±0.2
    Reliability
    Caveats

How the rubric works →