nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-EE9C7199·Framework v5.13·Pipeline v0.4·Evaluated 19 Nov 2026·OECD: 3.2 Clinical medicine·Subfield: Health Informatics·Basis: Full text·Type: Non-randomized or early-phase trial

Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study

2026·The Lancet·10.1016/s0140-6736(26)01535-7 ↗

Primary claim

this study demonstrates the initial feasibility, safety, and user acceptance of conversational AI in a real-world setting

Supported · selected patients and continuous physician oversight restrict the finding to this supervised feasibility setting
In brief

AMIE's supervised urgent-care deployment establishes operational feasibility, but compromised clinical comparisons prevent conclusions about parity with physicians.

ReliabilityCaveats3 of 4 checks completeRead the 2 entries ↓
Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract not available — view at DOI ↗

Key Claims

Primary finding · Supported

this study demonstrates the initial feasibility, safety, and user acceptance of conversational AI in a real-world setting

Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

selected patients and continuous physician oversight restrict the finding to this supervised feasibility setting

Abstract, final sentence
Quote verified
Secondary · Unsupported

AMIE's differential diagnosis (DDx) included the final diagnosis, per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy.

the partly AI-influenced chart reference yields 81.6% exact matches, not 90%, with exact top-three accuracy unreported

Abstract; Section 4.3; Tables A.6–A.7
Quote verified
Secondary · Unsupported

Blinded assessment of AMIE and PCP DDx and management (Mx) plans suggested similar overall DDx and Mx plan quality, without significant differences for DDx (p = 0.6) and appropriateness and safety of Mx (p = 0.1 and 1.0, respectively).

AI-exposed clinicians and non-equivalence testing cannot establish similar clinical quality

Abstract; Section 4.3, Figure 3
Quote verified
Secondary · Tentative

Patients reported high satisfaction and their attitudes towards AI improved after interacting with AMIE (p < 0.001).

uncontrolled, compensated participation with a physician debrief leaves material demand and attribution uncertainty

Abstract; Section 4.4, Figure 4
Quote verified
Secondary · Supported

Human safety supervisors monitored all patient-AMIE interactions in real time and did not need to intervene to stop any consultations based on pre-defined criteria.

complete real-time monitoring directly documents zero predefined safety stops in the included cohort

Abstract; Section 4.2
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Prospective deployment advances beyond simulated encounters

The study observes 100 completed patient interactions within an active urgent-care workflow, providing direct evidence about implementation barriers and supervised feasibility rather than relying solely on simulated patients.

↳ Introduction; Sections 3.1, 4.1, and 6.1

#methodological rigour

Continuous oversight uses explicit stopping criteria

Physicians monitored every interaction against four predefined stopping criteria, following a ten-patient pilot and safety pause. Section 4.2 reports both zero stops and three supervisor clarifications.

↳ Sections 3.4–3.5 and 4.2

#reporting

Detailed reporting makes the workflow traceable

The paper distinguishes research-only management plans from material shared with clinicians and supplies survey instruments, rating rubrics, and participant-flow denominators.

↳ Sections 3.4, 3.9, and 4.1; Appendix A.1–A.5

Limitations

#methodological rigour

Diagnostic inclusion conflates exact and near matches

The reported 90% diagnostic inclusion combines 80 exact and eight near matches among 98 cases. Exact matching is 81.6%, and exact top-three accuracy cannot be recovered from the reported counts.

↳ Abstract; Section 3.9.2; Tables A.6–A.7

#methodological rigour

Shared AI information compromises clinical comparisons

PCPs received AMIE's transcript and differential before their visits, and non-significant tests do not establish equivalent quality. This exposure also weakens the independence of presumptive chart diagnoses, an implication not fully traced in the discussion.

↳ Sections 3.4, 3.9.2–3.10, 4.5.1, and 6.4; Conclusion

#methodological rigour

Pooled results obscure mid-study model changes

The base model changed from Gemini 2.5 Pro to Flash after 50 encounters, but outcomes are not stratified by model. Compensation and interview timing also changed during the study, complicating interpretation of pooled experience measures.

↳ Sections 3.3, 3.6, and 3.8

Does this match your read? Add your assessment ↓

Nabu Commentary

The prospective workflow, explicit stop criteria, and complete live supervision provide concrete evidence for feasibility in selected urgent-care patients. The clinical comparison is less informative because PCPs received AMIE material before generating their own assessments, and non-significant differences do not establish equivalent quality. Diagnostic inclusion is also overstated: the 90% figure combines exact and near matches, whereas exact matches account for 81.6% of cases. Detailed appendices improve traceability, but pooled model changes and incomplete tracing of survey and reference-standard limitations constrain interpretation.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.2±0.1

Contribution
3.6

The study extends simulated conversational-AI evaluations into a prospective urgent-care workflow and documents completion, oversight, and patient experience. Its advance is a selected, supervised feasibility demonstration rather than evidence of autonomous safety or clinical equivalence.

Introduction; Sections 3.1–3.4, 5.2–5.3, and 6.1
Methodological Rigour
2.8

Prospective registration, continuous supervision, stopping criteria, and transparent participant flow support the feasibility assessment, but a non-independent comparator and absence of equivalence testing undermine the similarity conclusion. The separate reliability concern is that reported diagnostic coverage combines exact and near matches while the abstract states inclusion of the final diagnosis.

Sections 3.4–3.5, 3.9–3.10, and 4.1–4.3; Tables A.6–A.7
Reporting
3.5

The workflow and outcome hierarchy are followable, with detailed instruments and rating counts. Flat similarity language and identifiable caption, label, and cross-reference errors reduce precision.

Figures 1–2 and 4; Sections 3.9, 4.3–4.4, and 7; Table 1B; Appendix A.1–A.5
Positioning
3.3

Related work covers diagnostic support, patient-facing AI, oversight, and patient perspectives. The discussion does not fully carry demand effects, selection, and AI-influenced presumptive diagnoses through to the certainty of its trust, generalizability, and diagnostic interpretations.

Sections 5.1–5.4, 6.1, 6.3–6.4, and 6.7; Table A.10

Lower confidence on Reporting, Positioning — domain match limited.

Reliability

Caveats3 of 4 checks


Results checkNoted
  • "AMIE's differential diagnosis (DDx) included the final diagnosis, per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy." (Abstract) vs "For scores of 5 (‘The actual diagnosis was suggested in the differential’) and 4 (‘The suggestions included something very close, but not exact’)" (Section 3.9.2). Table A.7 gives 80 exact and eight near matches among 98 cases, so the 88 reported hits yield 89.8% exact-or-near matching, whereas exact matching is 80/98 = 81.6%. The claim is not supported by the paper's own data.
  • "A manual audit of 270 outputs by the study team ensured semantic integrity" (Section 3.9.2) vs "the number of cases out of 30 cases total (30 AMIE outputs and 30 PCP outputs respectively)" (Table A.5). The stated audit of 270 outputs and the fidelity table covering 30 outputs from each source cannot be reconciled from the supplied text. The effect on the result is unclear.
Reporting and transparencyNoted

Ethics approval, consent, prospective registration, funding, and competing interests are declared. The availability statement addresses development benchmarks but not the clinical study data, and the implementation code and system prompts are not released.

Conflict of interest: declared · Ethics approval: declared · Preregistration: declared · Data availability: not stated · Code availability: not stated

Reference IntegrityNot checked

Reference integrity check could not run: all 1 references in this manuscript are books, websites, datasets, or other non-journal sources that cannot be verified against an index.

1 reference in manuscript 1 are books, websites or datasets — counted, but not index-checkable

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 19 Nov 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaHealth · Generic health relevance
Type of advanceEmpirical
Route to useSpecified
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

A single-center, single-arm feasibility study of 100 patients with real-time physician oversight showed the system working once in a real clinic, without independent or multi-site confirmation.

From the paper
“for conducting clinical history-taking and providing potential diagnoses to patients presenting with urgent concerns in a real-world academic primary care practice setting”

The paper names the actors and setting, patients, PCPs and supervising physicians in an ambulatory primary care urgent care clinic, and describes the concrete mechanism of pre-visit text-chat intake with outputs delivered to PCPs before appointments.

Pathway

Primary care clinics and health systems could deploy supervised conversational AI for pre-visit history taking, with transcripts and potential diagnoses delivered to PCPs before appointments.

Barrier

Larger multi-site comparative trials, EHR integration, scalable safety oversight, and regulatory validation are still needed before routine clinical deployment.

Evaluated 19 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. On-premise medical AI agents for reliable clinical decision-making

    2026 · Clinical medicine

    Primary claim

    Diagnostic behavioral consistency provided the strongest discrimination of correctness (area under the curve (AUC) = 0.860) and remained informative under stress testing (AUC = 0.875).

    Supported
    Evidence Score
    3.7±0.2
    Reliability
    Caveats
  2. Large language models detect methodological errors that human peer review misses in clinical trials

    Dominikus Brian, Xavier Isidro Pereira Hernández, Hao Yin, et al. · 2026 · Computer and information sciences

    Primary claim

    Our findings suggest that LLMs can identify methodological errors in reports of randomized clinical trials that human peer review may miss.

    Tentative
    Evidence Score
    3.7±0.2
    Reliability
    Caveats
  3. Comparing AI chat tools and PubMed for medical literature searches

    Ettien, Downing, Drouhard, et al. · 2026 · Media and communications

    Primary claim

    Our results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for use in any in-depth literature searches

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats

How the rubric works →