Specialized clinical artificial intelligence (AI) tools are entering medical practice despite scarce independent evaluation. We quantitatively evaluate two clinical AI tools, OpenEvidence and UpToDate Expert AI, built on large language models (LLMs) against three frontier LLMs: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Our evaluation has three stages: (1) 500 MedQA questions testing medical knowledge, (2) 500 HealthBench items measuring alignment with clinicians and (3) the real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment. For the RCQ benchmark, 12 US clinicians performed randomized, blinded review of model outputs, producing 1,800 model-question annotations. Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ. These findings highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.
Strengths
The RCQ benchmark uses 100 de-identified live physician queries scored by 12 blinded clinicians producing 1,800 annotations, providing real-world evidence that is free from training-set contamination and not dependent on industry-authored benchmarks.
↳ Methods 'Real clinical queries'; Fig. 1; Fig. 2c-g
Analysis uses a pre-specified cumulative-link model with rater fixed effects as the primary regression, with linear mixed-model and Kruskal-Wallis sensitivity checks that are reported as concordant, plus bootstrap confidence intervals and explicit inter-rater statistics.
↳ Methods 'RCQ statistical analysis'; Extended Data Figs. 3-4
By comparing tools already in clinical use against freely available frontier models and naming procurement, reimbursement, and regulatory oversight as decision contexts, the findings speak directly to stakeholders actively making these choices.
↳ Discussion final paragraph; Abstract
Limitations
The most decision-relevant evidence comes from 100 queries at one US academic medical center rated by US clinicians, and the title's broad framing extends beyond what this narrow, private sample can support.
↳ Methods 'Real clinical queries'; Title; Discussion limitations
Frontier models served as both evaluated systems and HealthBench judges, and the top scorer GPT-5.2 is from the developer of that OpenAI-built benchmark, introducing potential grading and developer-overlap bias. The authors disclose this and appropriately demote HealthBench to supplementary evidence.
↳ Methods 'HealthBench scoring'; Discussion limitations
Headline pairwise comparisons rest on 74 complete-case questions after excluding 32 refusals, with one clinical tool refusing 19% of queries, complicating interpretation of comparative superiority even though a concordant sensitivity analysis on all 568 items is reported.
↳ Methods 'RCQ statistical analysis'; Fig. 2e
The scores reflect a competently executed, timely evaluation whose central finding is robustly triangulated across three independent stages, with a clinician-rated real-query benchmark that the authors correctly designate as primary evidence over the LLM-judge-scored HealthBench. Methodological Rigour sits in the upper-middle band because the strong design carries genuine execution constraints: low item-level inter-rater agreement, browser-interface asymmetry for the clinical tools, a temporal access gap between tools and frontier models, and dependence on a 74-question complete-case subset after refusal exclusions. Reporting is high given the clear pipeline schematic and explicit evidence prioritization, tempered by headline phrasing broader than the single-site sample warrants. Impact potential is led by high traction and trajectory for procurement and regulatory decisions but held down by genuinely moderate transferability from a single-institution, 100-query benchmark. The reliability picture is Amber: the principal concerns are all disclosed and do not overturn the core finding, but the dual-role judge design, private benchmark, and absent conflict-of-interest statement are non-trivial watch-outs.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Strong3.9
Confidence highProvides the first independent, quantitative head-to-head comparison of two commercial clinical AI tools against three frontier LLMs across knowledge, expert-alignment, and a novel real-clinical-query benchmark, with the RCQ component partially addressing benchmark contamination. The advance is solid and timely but bounded by a brief-communication scope of only two clinical tools and a 100-query single-site benchmark.
an independent, quantitative comparison of clinical AI tools against frontier LLMs using real-world physician queries
Triangulated three-stage design with blinded randomized clinician review, a pre-specified cumulative-link primary regression, multiple concordant sensitivity analyses, and reported inter-rater statistics supports the main finding. Low item-level inter-rater reliability (Krippendorff alpha 0.10-0.20), browser-interface asymmetry for clinical tools, a temporal access gap, and LLM-judge dependence on HealthBench keep execution below the top tier.
item-level agreement was fair (alpha = 0.10-0.20)
The argument flows clearly from motivation through a three-stage pipeline to results and limitations, aided by an effective schematic and compact-letter-display significance notation, with explicit primary/supplementary evidence designation. Minor imprecision arises from broad headline framing and describing HealthBench as 'alignment with clinicians' when it is operationalized via LLM-judge rubric scoring.
snapshot of a rapidly evolving landscape rather than a permanent ordering
Engages substantively with RAG limitations, safety frameworks (NOHARM), and health-system-grounded evaluation, and offers a non-perfunctory limitations section addressing contamination, judge bias, interface asymmetry, and omitted deployment factors. Positioning is not maximal because some references are preprints and the framing still leans toward a broad conclusion from a narrow set of tested products and tasks.
industry-created benchmarks may systematically favor the systems developed by their creators
Caveats4 of 4 checks
Numerical values are consistent across abstract, figures, and methods, and per-model n values reconcile to the stated 568 non-refusal items. Watch-outs concern headline generalization beyond a single-site 100-query benchmark, complete-case dependence after refusal exclusions, and benchmark-developer/LLM-judge overlap on HealthBench, all of which are disclosed.
Ethics approval is declared with an IRB protocol identifier (declared-but-unverified) and funding is disclosed, but the most decision-relevant RCQ dataset is private and proprietary tool architectures are opaque, limiting full independent reproducibility. No explicit conflict-of-interest statement is present despite evaluating and using vendor models as judges.
Flags: 3 declared / 5 total
All 12 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
Findings are validated in operational-adjacent conditions using real physician queries and blinded clinician scoring of already-deployed systems, but stop short of testing actual clinical workflows, latency, citation quality, or patient outcomes. The evidence informs decisions today without constituting a standalone deployment framework.
our evaluation did not assess response latency or citation quality
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Dominikus Brian, Xavier Isidro Pereira Hernández, Hao Yin, et al. · 2026 · Computer and information sciences
Our findings suggest that LLMs can identify methodological errors in reports of randomized clinical trials that human peer review may miss.
TentativeEttien, Downing, Drouhard, et al. · 2026 · Media and communications
Our results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for use in any in-depth literature searches
TentativeDiogo Machado · 2026 · Other social sciences
Novelty is the only evaluation dimension that systematically predicts disagreement between the two evaluation systems.
Supported