# Blind spots in AI-assisted healthcare evidence search: multiplatform evaluation of clinical retrieval gaps and risk-of-bias

- **DOI:** [10.1038/s41746-026-03277-y](https://doi.org/10.1038/s41746-026-03277-y)
- **Rating:** Strong (3.9 out of 5)
- **Verdict:** A systematic multiplatform audit demonstrates incomplete AI-assisted evidence retrieval, with a single clinical topic limiting the reach of its findings.
- **Framework version:** v5.10
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-10-20
- **Evaluation page:** https://www.nabu.science/paper/10.1038%2Fs41746-026-03277-y

## Key claims

1. Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.
   - Support: Well-supported
   - Type: Primary finding
   - Basis: directly measured descriptive ranges across matched formulations within the 83-study benchmark and June 2025 evaluation window
   - Location: Abstract; Results, Single-query and overall platform retrieval performance
2. 12.0% of evidence was never retrieved by any platform, with never-retrieval significantly higher for conference proceedings than journal articles (38.9% vs 4.6%; p < 0.001).
   - Support: Supported
   - Type: Secondary
   - Basis: consistent benchmark counts support the venue association, although sparse proceedings and venue–year confounding limit interpretation
   - Location: Abstract; Results, Single-query zero-retrieval probability by evidence domain
3. By substance category, tobacco-related studies (n = 28), the largest evidence category, showed a 33-percentage-point spread in single-query zero-retrieval probability (p = 0.043), ranging from 47% for Ai2 Finder to 80% for ChatGPT-4.1.
   - Support: Supported
   - Type: Secondary
   - Basis: the descriptive spread is supported, but 15 non-random formulations and uncorrected stratum testing limit inferential certainty
   - Location: Results, Single-query zero-retrieval probability by evidence domain
4. The strongest two-platform combination in this benchmark, Ai2 Finder and Claude-Sonnet-4.0, retrieved 83.1% of the gold-standard studies, compared with 72.3% for the strongest individual platform
   - Support: Supported
   - Type: Secondary
   - Basis: observed union coverage demonstrates complementarity within this corpus, not a generally optimal platform pair
   - Location: Discussion, Combining platforms improved coverage but did not achieve complete retrieval

## Strengths

- **Independent corpus supports a distinct retrieval benchmark** The 83-study reference corpus was assembled before testing through seven-database searching, independent screening, and arbitration. Keeping the curated list non-public during evaluation reduces the risk of testing reproduction of an existing benchmark list.
  - Dimension: methodological rigour
  - Location: Methods, Gold standard development
- **Complementary outcomes expose different retrieval limitations** Formulation-level recall, pooled capacity, domain-level absence, precision, and run consistency distinguish what users receive from what platforms can retrieve across repeated searches.
  - Dimension: contribution
  - Location: Methods, Study design and Performance analysis; Results, retrieval performance and complementarity
- **Conclusions retain the benchmark's interpretive boundaries** The discussion presents the results as a warning rather than a general ranking and explicitly separates observed gaps from hypothetical mechanisms and longer-term visibility effects.
  - Dimension: reporting
  - Location: Discussion, benchmark scope, mechanisms, and limitations

## Limitations

- **One topic limits broader performance conclusions** The corpus concerns NLP identification of substance use disorders in electronic health records, with testing confined to June 2025 browser interfaces. The results do not establish coverage patterns across other clinical topics or platform versions.
  - Dimension: contribution
  - Location: Methods, Gold standard development and Platform selection; Discussion, limitations
- **Subgroup inference exceeds the uncertainty reporting** Domain analyses report tobacco p = 0.043 and cannabinoid p = 0.012 without adjustment across the tested strata or uncertainty intervals. These findings warrant more caution than the directly observed overall recall ranges.
  - Dimension: methodological rigour
  - Location: Results, Single-query zero-retrieval probability by evidence domain; Methods, Statistical analysis
- **Comparative context leaves important distinctions unresolved** There is no matched conventional-search comparator, and venue differences are not disentangled from publication-period and related corpus characteristics. The relationship between exhaustive eligible-study recall and the intended rapid-overview use case also remains incompletely examined.
  - Dimension: positioning
  - Location: Methods, Study design; Discussion, publication period and venue patterns, use-case framing, and limitations

## Reliability checks

No concerns (4 of 4 checks).

- **Results check:** Clean. The reported numbers and statistics are consistent with the stated results.
- **Reporting and transparency:** Clean. Ethics approval is declared unnecessary for analysis of public outputs and published literature, and competing interests, data availability, and code availability are disclosed. Availability declarations were not independently verified.
  - Conflict of interest: declared · Ethics approval: declared · Data availability: declared · Code availability: declared
- **Reference integrity:** Clean
  - 74 references in manuscript
  - 54 of 54 checkable references found in an index
  - 20 have no canonical index record — counted, but not index-checkable
  - 5 references confirmed by manual review
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-20
