We explored the capabilities of frontier language models in sequential diagnosis, a diagnostic paradigm that captures the iterative nature of clinical reasoning. Given a brief clinical presentation, clinicians selectively acquired information to reach a diagnosis, enabling assessment of diagnostic accuracy and workup cost. We developed an interactive benchmark and an artificial intelligence (AI) system that simulates a panel of physicians, continually revising the differential diagnosis and selecting cost-effective tests. Using this benchmark, the system achieved 83.2% accuracy on New England Journal of Medicine clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, outperforming generalist physicians (36.1 and 46.8%, respectively). Although these findings were obtained in a controlled benchmark setting, they highlight the promise of language models for supporting more accurate, lower-cost diagnoses.
DxO without budget constraints achieved an average diagnostic accuracy of 80.3% at a mean cost of $4979 per ED case.
directly reported ED benchmark outcome under a text-only oracle interface rather than real clinical encounters
DxO with GPT-5 established a new Pareto frontier that outperformed the participating physicians at every cost or accuracy level.
public-CPC exposure and an individual-generalist comparator constrain the frontier claim
Across all of the models tested, DxO orchestration boosted the diagnostic accuracy on the ED cases and CPC cases by an average of 4.4 and 7.9 percentage points, respectively.
comparison with minimally prompted models does not isolate orchestration-specific gains against a stronger baseline
Derived from the full evaluation — not a separate score.
Strengths
SDBench measures information gathering and diagnostic stopping decisions alongside accuracy and cumulative estimated testing costs, extending evaluation beyond a complete vignette supplied at once.
↳ Introduction; Results, benchmark description
The evaluation combines CPC and independently sourced ED cases, a held-out CPC physician set, multiple model families, and a GPT-5.5 rerun without prompt changes.
↳ Results, Clinicopathological conference studies; Emergency medicine studies; Studies of model capabilities and errors
The paper distinguishes controlled diagnostic performance from superiority over specialists, teams, or real-world care, and explains the missing safety and longitudinal dimensions.
↳ Introduction; Discussion, limitations
Limitations
The CPC benchmark uses published NEJM cases without a reported contamination evaluation. Holding cases out of engineering does not establish that the underlying models had not encountered them during pretraining.
↳ Results, Clinicopathological conference studies; Discussion
The reported orchestration gains are measured against minimally prompted models. This establishes a difference between the evaluated configurations, not a clean isolation of orchestration-specific benefits against a stronger comparator.
↳ Results, Studies of model capabilities and errors
Text-only oracle interaction, pathological case selection, and US chargemaster pricing leave substantial uncertainty about performance and cost trade-offs in routine care. The Judge does not assess therapeutic safety or longitudinal outcomes.
↳ Discussion, limitations
SDBench's main contribution is a cost-aware, sequential evaluation rather than another static diagnostic accuracy test. The two case sources and repeated model evaluations broaden the evidence, but the public CPC cases lack a reported contamination check. The minimal-prompt model comparator also limits attribution of the measured gains specifically to orchestration. Explicit clinical scope restrictions support the paper's clarity, while omission of public-case exposure limits its contextual interpretation.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Limited2.8
Confidence mediumSDBench extends static vignette evaluation to sequential information acquisition with diagnostic accuracy and estimated test costs. Two case sources, physician comparisons, and model-family evaluations establish a delineated contribution, while novelty relative to other interactive evaluations is less fully developed.
Two datasets, held-out physician cases, and repeated model evaluations provide multiple checks on benchmark performance. Unaddressed public-CPC exposure and the minimal-prompt comparator materially constrain interpretation of the claimed capability gains.
The narrative connects sequential-diagnosis motivation, benchmark design, measured outcomes, and explicit scope restrictions. The unclear CPC denominator and expansive frontier wording prevent a stronger score.
The Discussion connects numerous simulation and clinical-comparison limitations to the scope of the findings. Its omission of possible pretraining exposure to public CPC cases leaves a central alternative explanation unaddressed.
Caveats4 of 4 checks
The reported numbers and statistics are consistent with the stated results.
The main text does not declare ethics approval or consent for the physician study and leaves the DxO CPC evaluation subset unclear. Competing interests and dataset access conditions are disclosed.
Conflict of interest: declared · Ethics approval: not stated · Data availability: declared · Code availability: declared
9 of 9 checkable references verified
12 references in manuscript 3 have no canonical index record — counted, but not index-checkable 8 references confirmed by manual review
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
Released benchmark code enables research reuse, but licensed or conditionally released datasets constrain access. Clinical use remains beyond the demonstrated text-based simulation.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
A. Ali Heydari, Ken Gu, Vidya Srinivas, et al. · Computer and information sciences
End users ranked PHA as their top choice nearly half the time (46.7% on overall preference and 44.4% on response quality)
SupportedFarrag, Soliman, Azubuike, et al. · 2026 · Computer and information sciences
Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.
Well-supportedHo, Liu, Tran, et al. · 2026 · Computer and information sciences
After one recursive step, increasing exposure from 0% to 100% makes recommendation ratings more concentrated, independently generated reviews of the same paper about 11% more similar, and reviews across different papers about 5% less diverse.
Supported