# Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study

- **DOI:** [10.1016/s0140-6736(26)01535-7](https://doi.org/10.1016/s0140-6736(26)01535-7)
- **Rating:** Sound (3.2 out of 5)
- **Verdict:** AMIE's supervised urgent-care deployment establishes operational feasibility, but compromised clinical comparisons prevent conclusions about parity with physicians.
- **Framework version:** v5.13
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-19
- **Evaluation page:** https://www.nabu.science/paper/10.1016%2Fs0140-6736(26)01535-7

## Key claims

1. this study demonstrates the initial feasibility, safety, and user acceptance of conversational AI in a real-world setting
   - Support: Supported
   - Type: Primary finding
   - Basis: selected patients and continuous physician oversight restrict the finding to this supervised feasibility setting
   - Location: Abstract, final sentence
2. AMIE's differential diagnosis (DDx) included the final diagnosis, per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy.
   - Support: Unsupported
   - Type: Secondary
   - Basis: the partly AI-influenced chart reference yields 81.6% exact matches, not 90%, with exact top-three accuracy unreported
   - Location: Abstract; Section 4.3; Tables A.6–A.7
3. Blinded assessment of AMIE and PCP DDx and management (Mx) plans suggested similar overall DDx and Mx plan quality, without significant differences for DDx (p = 0.6) and appropriateness and safety of Mx (p = 0.1 and 1.0, respectively).
   - Support: Unsupported
   - Type: Secondary
   - Basis: AI-exposed clinicians and non-equivalence testing cannot establish similar clinical quality
   - Location: Abstract; Section 4.3, Figure 3
4. Patients reported high satisfaction and their attitudes towards AI improved after interacting with AMIE (p < 0.001).
   - Support: Tentative
   - Type: Secondary
   - Basis: uncontrolled, compensated participation with a physician debrief leaves material demand and attribution uncertainty
   - Location: Abstract; Section 4.4, Figure 4
5. Human safety supervisors monitored all patient-AMIE interactions in real time and did not need to intervene to stop any consultations based on pre-defined criteria.
   - Support: Supported
   - Type: Secondary
   - Basis: complete real-time monitoring directly documents zero predefined safety stops in the included cohort
   - Location: Abstract; Section 4.2

## Strengths

- **Prospective deployment advances beyond simulated encounters** The study observes 100 completed patient interactions within an active urgent-care workflow, providing direct evidence about implementation barriers and supervised feasibility rather than relying solely on simulated patients.
  - Dimension: contribution
  - Location: Introduction; Sections 3.1, 4.1, and 6.1
- **Continuous oversight uses explicit stopping criteria** Physicians monitored every interaction against four predefined stopping criteria, following a ten-patient pilot and safety pause. Section 4.2 reports both zero stops and three supervisor clarifications.
  - Dimension: methodological rigour
  - Location: Sections 3.4–3.5 and 4.2
- **Detailed reporting makes the workflow traceable** The paper distinguishes research-only management plans from material shared with clinicians and supplies survey instruments, rating rubrics, and participant-flow denominators.
  - Dimension: reporting
  - Location: Sections 3.4, 3.9, and 4.1; Appendix A.1–A.5

## Limitations

- **Diagnostic inclusion conflates exact and near matches** The reported 90% diagnostic inclusion combines 80 exact and eight near matches among 98 cases. Exact matching is 81.6%, and exact top-three accuracy cannot be recovered from the reported counts.
  - Dimension: methodological rigour
  - Location: Abstract; Section 3.9.2; Tables A.6–A.7
- **Shared AI information compromises clinical comparisons** PCPs received AMIE's transcript and differential before their visits, and non-significant tests do not establish equivalent quality. This exposure also weakens the independence of presumptive chart diagnoses, an implication not fully traced in the discussion.
  - Dimension: methodological rigour
  - Location: Sections 3.4, 3.9.2–3.10, 4.5.1, and 6.4; Conclusion
- **Pooled results obscure mid-study model changes** The base model changed from Gemini 2.5 Pro to Flash after 50 encounters, but outcomes are not stratified by model. Compensation and interview timing also changed during the study, complicating interpretation of pooled experience measures.
  - Dimension: methodological rigour
  - Location: Sections 3.3, 3.6, and 3.8

## Reliability checks

Caveats (3 of 4 checks).

- **Results check:** Noted
  - "AMIE's differential diagnosis (DDx) included the final diagnosis, per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy." (Abstract) vs "For scores of 5 (‘The actual diagnosis was suggested in the differential’) and 4 (‘The suggestions included something very close, but not exact’)" (Section 3.9.2). Table A.7 gives 80 exact and eight near matches among 98 cases, so the 88 reported hits yield 89.8% exact-or-near matching, whereas exact matching is 80/98 = 81.6%. The claim is not supported by the paper's own data.
  - "A manual audit of 270 outputs by the study team ensured semantic integrity" (Section 3.9.2) vs "the number of cases out of 30 cases total (30 AMIE outputs and 30 PCP outputs respectively)" (Table A.5). The stated audit of 270 outputs and the fidelity table covering 30 outputs from each source cannot be reconciled from the supplied text. The effect on the result is unclear.
  - Minor reporting notes (3):
    - "approximately 10% of all urgent care visits were enrolled in the study" (Section 6.1) vs "During the study period, 1,452 urgent care visits were scheduled at HCA." (Section 4.1). The 100 completed interactions represent 100/1,452 = 6.9% of visits, while 140 verbal consents represent 140/1,452 = 9.6%, making the statement dependent on the meaning of enrolled. The result itself is unaffected.
    - "followed by pairwise Wilcoxon post-hoc tests (p < 0.001 for omnibus and pairwise tests)." (Figure 4 legend) vs "Pre-AI vs. Post-AI: p < 0.001; Post-AI vs. Post-Provider: p = 0.86" (Section 4.4). The caption's blanket p < 0.001 statement conflicts with the overall post-AI versus post-provider p = 0.86, without changing the significant pre-AI versus post-AI result. The result itself is unaffected.
    - "| Total | 98 | 100% | 100% |" (Table A.10, Total row) vs "| Other (Rare/Complex Codes) | 8 | 8.16% | 20.50% |" (Table A.10, Other row). The 21 displayed study-category counts sum to 89 rather than 98, leaving nine cases unaccounted for in the category rows. The effect on the result is unclear.
- **Reporting and transparency:** Noted. Ethics approval, consent, prospective registration, funding, and competing interests are declared. The availability statement addresses development benchmarks but not the clinical study data, and the implementation code and system prompts are not released.
  - Conflict of interest: declared · Ethics approval: declared · Preregistration: declared · Data availability: not stated · Code availability: not stated
- **Reference integrity:** Not checked. Reference integrity check could not run: all 1 references in this manuscript are books, websites, datasets, or other non-journal sources that cannot be verified against an index.
  - 1 reference in manuscript
  - 1 are books, websites or datasets — counted, but not index-checkable

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-19
