# General-purpose large language models outperform specialized clinical AI tools on medical benchmarks

- **DOI:** [10.1038/s41591-026-04431-5](https://doi.org/10.1038/s41591-026-04431-5)
- **Rating:** Strong (3.9 out of 5)
- **Verdict:** Delivers the first independent head-to-head evidence that frontier LLMs match or beat specialized clinical AI tools, constrained by a single-institution real-query benchmark.
- **Framework version:** v5.0
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-06-17
- **Evaluation page:** https://www.nabu.science/paper/10.1038%2Fs41591-026-04431-5

## Strengths

- **Independent blinded real-query clinical comparison** The RCQ benchmark uses 100 de-identified live physician queries scored by 12 blinded clinicians producing 1,800 annotations, providing real-world evidence that is free from training-set contamination and not dependent on industry-authored benchmarks.
  - Dimension: contribution
  - Location: Methods 'Real clinical queries'; Fig. 1; Fig. 2c-g
- **Transparent, pre-specified statistical analysis** Analysis uses a pre-specified cumulative-link model with rater fixed effects as the primary regression, with linear mixed-model and Kruskal-Wallis sensitivity checks that are reported as concordant, plus bootstrap confidence intervals and explicit inter-rater statistics.
  - Dimension: methodological rigour
  - Location: Methods 'RCQ statistical analysis'; Extended Data Figs. 3-4
- **Directly actionable for procurement decisions** By comparing tools already in clinical use against freely available frontier models and naming procurement, reimbursement, and regulatory oversight as decision contexts, the findings speak directly to stakeholders actively making these choices.
  - Dimension: application pathway
  - Location: Discussion final paragraph; Abstract

## Limitations

- **Single-institution benchmark limits generalizability** The most decision-relevant evidence comes from 100 queries at one US academic medical center rated by US clinicians, and the title's broad framing extends beyond what this narrow, private sample can support.
  - Dimension: contribution
  - Location: Methods 'Real clinical queries'; Title; Discussion limitations
- **LLM-judge and benchmark-developer overlap on HealthBench** Frontier models served as both evaluated systems and HealthBench judges, and the top scorer GPT-5.2 is from the developer of that OpenAI-built benchmark, introducing potential grading and developer-overlap bias. The authors disclose this and appropriately demote HealthBench to supplementary evidence.
  - Dimension: methodological rigour
  - Location: Methods 'HealthBench scoring'; Discussion limitations
- **Complete-case dependence after refusal exclusions** Headline pairwise comparisons rest on 74 complete-case questions after excluding 32 refusals, with one clinical tool refusing 19% of queries, complicating interpretation of comparative superiority even though a concordant sensitivity analysis on all 568 items is reported.
  - Dimension: methodological rigour
  - Location: Methods 'RCQ statistical analysis'; Fig. 2e

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. Numerical values are consistent across abstract, figures, and methods, and per-model n values reconcile to the stated 568 non-refusal items. Watch-outs concern headline generalization beyond a single-site 100-query benchmark, complete-case dependence after refusal exclusions, and benchmark-developer/LLM-judge overlap on HealthBench, all of which are disclosed.
- **Research conduct:** Noted. Ethics approval is declared with an IRB protocol identifier (declared-but-unverified) and funding is disclosed, but the most decision-relevant RCQ dataset is private and proprietary tool architectures are opaque, limiting full independent reproducibility. No explicit conflict-of-interest statement is present despite evaluating and using vendor models as judges.
- **Reference integrity:** Clean. All 12 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17