Assembling the evidence…
Rigorous evidence on the performance of large language models (LLMs) in real-world, low-resource clinical settings remains limited. Here we conducted a pragmatic, cluster-randomized trial in 16 primary care facilities in Kenya. Clinical officers were randomized to use the electronic medical record with or without LLM assistance. The primary outcome was an expert-adjudicated composite of treatment failure events experienced within 14 days of enrollment. Between 22 April and 16 July 2025, 9,691 patients were enrolled, overseen by 103 clinical officers (52 in the LLM-assisted arm and 51 in the control arm). Treatment failure occurred in 102/4,693 patients (2.2%) in the intervention arm and 94/4,654 (2.0%) in the control arm (adjusted odds ratio 0.77, 95% confidence interval 0.55 to 1.08, P = 0.13). The primary outcome did not differ significantly between groups. No serious adverse events were judged related to the intervention, and independent review of the adverse events did not identify a safety signal. In this trial, LLM assistance was safe but did not reduce treatment failure within 14 days and any benefit, if present, is probably modest.Pan-African Clinical Trials Registry: 202502499779176.
Strengths
The trial randomized 103 clinical officers across 16 Kenyan primary-care facilities and analyzed 9,347 encounters, providing rare prospective evidence for LLM decision support in routine LMIC care.
↳ Abstract; Results, Patient disposition; Methods, Design
Randomization at the clinical-officer level, patient and assessor blinding where feasible, independent expert adjudication, and mixed-effects modeling align with the intervention’s workflow-based deployment.
↳ Methods, Randomization, allocation and blinding; Methods, Outcomes; Methods, Statistical analysis
The Discussion frames the primary result as bounded inference, noting that large clinically meaningful effects are unlikely while smaller effects cannot be ruled out because the trial had limited precision for modest effects.
↳ Discussion, first paragraphs
The paper reports per-patient LLM cost, workflow integration, clinician autonomy, safety monitoring, and post hoc cost signals, making the findings useful to implementers even without demonstrated clinical efficacy.
↳ Methods, Interventions; Results, Patient disposition; Results, Post hoc analyses
Limitations
The Methods state protocol deviations were retained in ITT, while Results and Fig. 1 describe exclusions before the primary analysis, including 254 encounters with potential exposure or allocation misclassification. This does not invalidate the primary result but complicates interpretation of the stated ITT set.
↳ Methods, Statistical analysis; Results, Patient disposition; Fig. 1
The trial reports 921 protocol deviations and violations, including unconsented clinical officers, encounters involving multiple or mixed-arm clinicians, failed EMR linkage, and temporary control-arm AI access. The deviations were disclosed and partly handled analytically, but they reduce the cleanliness of execution.
↳ Results, Protocol deviations and violations
The Discussion describes strong evidence that the intervention improved documentation and care, despite a null primary endpoint and multiple unadjusted secondary tests. The documentation findings are useful process signals but should not be read as demonstrated clinical benefit.
↳ Discussion; Fig. 3 legend
All sites were within one private urban clinic network with EMR infrastructure and structured quality-improvement processes. The article acknowledges that results may not transfer directly to rural, periurban, public-sector, or less digitized settings.
↳ Discussion, limitations
The scores are driven by a strong pragmatic trial design and an unusually relevant deployment setting for LLM-based clinical decision support. Methodological Rigour is lower than the other quality dimensions because the article contains unresolved tension between the stated ITT principle and the exclusions described in Results and Fig. 1, alongside substantial protocol deviations and incomplete subsample-selection detail for secondary outcomes. The contribution remains high because the null primary result usefully bounds clinical benefit in a field dominated by simulated or early-stage evidence. Impact potential is substantial but operational rather than deployment-ready: the intervention was tested in routine care, yet it did not significantly reduce 14-day treatment failure and was studied in one private urban network.
Strong3.9/5.0
The paper addresses a clear evidence gap by testing LLM-enabled clinical decision support prospectively in routine LMIC primary care rather than in vignettes or retrospective benchmarks. Its null primary result is still informative because it bounds likely clinical effects while documenting process and safety outcomes.
prospective interventional evidence from real-world clinical studies, particularly in LMICs, remains limited
The cluster-randomized design, blinded follow-up/adjudication, DSMB oversight, and mixed-effects modeling fit the pragmatic intervention question. The score is moderated by population-flow and ITT ambiguity, substantial protocol deviations, facility-level imbalance, and limited description of how documentation-quality and satisfaction subsamples were selected.
These encounters were excluded from per-protocol analyses but retained in the ITT analysis.
The article clearly reports the null primary outcome and generally calibrates its interpretation around bounded inference rather than claiming demonstrated clinical benefit. Reporting is reduced by unresolved enrolled/analyzed count differences and discussion language that gives strong weight to secondary process findings.
any true effect, if present, is likely to be modest
The paper situates the study in LMIC primary-care workforce constraints, prior LLM evaluation literature, and the limitations of earlier rule-based decision support. It also discusses contamination, single-network generalizability, high baseline quality, model-version dependence, rare-event power, and short follow-up.
Conducting the trial within a single private network of urban clinics in Nairobi may limit generalizability
Lower confidence on Methodological Rigour — domain match limited.
Watch-outs4 of 4 checks
The paper is broadly internally coherent on the main endpoint and interpretation, but several population-flow and analysis-set details are not fully reconciled. The raw primary event rates favor control while the adjusted odds ratio favors intervention, which is statistically possible under adjustment but would benefit from clearer explanation.
Ethics approvals, trial registration, consent, blinded adjudication, DSMB oversight, and conflict disclosures are reported, but several conduct and reproducibility constraints remain. Data and analytic code are deferred or request-mediated, core system components are proprietary, and OpenAI provided disclosed in-kind support to the implementing organization.
Flags: 4 declared / 5 total
All 38 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓PubPeer (coming soon)
High3.9/5.0
The study targets a concrete operational problem: supporting clinical officers in Kenyan primary care where physician availability and senior consultation are constrained. The user group, workflow, organizational setting, and mechanism of intervention are explicitly specified.
clinical officers—mid-level practitioners who complete a 3-year diploma in clinical medicine
The intervention was tested in routine care across 16 facilities with real patients, clinicians, costs, follow-up, and safety monitoring. Readiness is limited by the absence of significant clinical benefit on the primary endpoint and by the proprietary EMR/LLM implementation.
comparing routine clinical officer-led consultations supported by an LLM to those conducted without LLM assistance
The multi-facility pragmatic design supports some transferability within similar urban EMR-enabled primary-care networks. Generalization is constrained by the single private network, high baseline quality-improvement infrastructure, and commercial/proprietary system components.
single private network of urban clinics in Nairobi
The article contributes to an active evidence trajectory by extending LLM clinical decision-support evaluation from simulated and pilot contexts into a large pragmatic trial. It also identifies endpoint and sample-size challenges for future studies of general-purpose AI in primary care.
there remains an open question around what is the most appropriate primary outcome
Lower confidence on Generalizability — domain match limited.
Nabu’s assessment, alongside the field’s view.
Evaluation Metadata
AI-powered, human-governed. Something look off? Contact us to request a review.