nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-FC02F176·Framework v5.0·Pipeline v0.4·Evaluated 10 Jul 2026·OECD: 3.2 Clinical medicine·Subfield: Health Informatics·Basis: Full text·Type: Randomized controlled trial

Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial.

Agweyu et al.Agweyu, Mwaniki, Menon et al.·2026·Nature medicine·10.1038/s41591-026-04503-6 ↗

In brief

A policy-relevant pragmatic RCT that bounds LLM decision-support benefits, constrained by null clinical efficacy in one urban private network.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

Rigorous evidence on the performance of large language models (LLMs) in real-world, low-resource clinical settings remains limited. Here we conducted a pragmatic, cluster-randomized trial in 16 primary care facilities in Kenya. Clinical officers were randomized to use the electronic medical record with or without LLM assistance. The primary outcome was an expert-adjudicated composite of treatment failure events experienced within 14 days of enrollment. Between 22 April and 16 July 2025, 9,691 patients were enrolled, overseen by 103 clinical officers (52 in the LLM-assisted arm and 51 in the control arm). Treatment failure occurred in 102/4,693 patients (2.2%) in the intervention arm and 94/4,654 (2.0%) in the control arm (adjusted odds ratio 0.77, 95% confidence interval 0.55 to 1.08, P = 0.13). The primary outcome did not differ significantly between groups. No serious adverse events were judged related to the intervention, and independent review of the adverse events did not identify a safety signal. In this trial, LLM assistance was safe but did not reduce treatment failure within 14 days and any benefit, if present, is probably modest.Pan-African Clinical Trials Registry: 202502499779176.

Strengths · Limitations

Strengths

#contribution

Real-world randomized evidence at scale

The trial randomized 103 clinical officers across 16 Kenyan primary-care facilities and analyzed 9,347 encounters, providing rare prospective evidence for LLM decision support in routine LMIC care.

↳ Abstract; Results, Patient disposition; Methods, Design

#methodological rigour

Pragmatic design matches implementation question

Randomization at the clinical-officer level, patient and assessor blinding where feasible, independent expert adjudication, and mixed-effects modeling align with the intervention’s workflow-based deployment.

↳ Methods, Randomization, allocation and blinding; Methods, Outcomes; Methods, Statistical analysis

#reporting

Null result interpreted with restraint

The Discussion frames the primary result as bounded inference, noting that large clinically meaningful effects are unlikely while smaller effects cannot be ruled out because the trial had limited precision for modest effects.

↳ Discussion, first paragraphs

#application pathway

Operational details support future implementation

The paper reports per-patient LLM cost, workflow integration, clinician autonomy, safety monitoring, and post hoc cost signals, making the findings useful to implementers even without demonstrated clinical efficacy.

↳ Methods, Interventions; Results, Patient disposition; Results, Post hoc analyses

Limitations

#methodological rigour

Analysis population is not fully reconciled

The Methods state protocol deviations were retained in ITT, while Results and Fig. 1 describe exclusions before the primary analysis, including 254 encounters with potential exposure or allocation misclassification. This does not invalidate the primary result but complicates interpretation of the stated ITT set.

↳ Methods, Statistical analysis; Results, Patient disposition; Fig. 1

#methodological rigour

Protocol deviations complicate intervention fidelity

The trial reports 921 protocol deviations and violations, including unconsented clinical officers, encounters involving multiple or mixed-arm clinicians, failed EMR linkage, and temporary control-arm AI access. The deviations were disclosed and partly handled analytically, but they reduce the cleanliness of execution.

↳ Results, Protocol deviations and violations

#reporting

Secondary findings receive strong interpretive weight

The Discussion describes strong evidence that the intervention improved documentation and care, despite a null primary endpoint and multiple unadjusted secondary tests. The documentation findings are useful process signals but should not be read as demonstrated clinical benefit.

↳ Discussion; Fig. 3 legend

#application pathway

Generalizability remains setting dependent

All sites were within one private urban clinic network with EMR infrastructure and structured quality-improvement processes. The article acknowledges that results may not transfer directly to rural, periurban, public-sector, or less digitized settings.

↳ Discussion, limitations

Does this match your read? Add your assessment ↓

Nabu Commentary

The scores are driven by a strong pragmatic trial design and an unusually relevant deployment setting for LLM-based clinical decision support. Methodological Rigour is lower than the other quality dimensions because the article contains unresolved tension between the stated ITT principle and the exclusions described in Results and Fig. 1, alongside substantial protocol deviations and incomplete subsample-selection detail for secondary outcomes. The contribution remains high because the null primary result usefully bounds clinical benefit in a field dominated by simulated or early-stage evidence. Impact potential is substantial but operational rather than deployment-ready: the intervention was tested in routine care, yet it did not significantly reduce 14-day treatment failure and was studied in one private urban network.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.9±0.1

Contribution
4.1

The paper addresses a clear evidence gap by testing LLM-enabled clinical decision support prospectively in routine LMIC primary care rather than in vignettes or retrospective benchmarks. Its null primary result is still informative because it bounds likely clinical effects while documenting process and safety outcomes.

Introduction; Results, Primary outcome; Discussion

prospective interventional evidence from real-world clinical studies, particularly in LMICs, remains limited

Methodological Rigour
3.7

The cluster-randomized design, blinded follow-up/adjudication, DSMB oversight, and mixed-effects modeling fit the pragmatic intervention question. The score is moderated by population-flow and ITT ambiguity, substantial protocol deviations, facility-level imbalance, and limited description of how documentation-quality and satisfaction subsamples were selected.

Methods, Randomization; Methods, Statistical analysis; Results, Patient disposition; Results, Protocol deviations and violations; Table 1

These encounters were excluded from per-protocol analyses but retained in the ITT analysis.

Reporting
4.2

The article clearly reports the null primary outcome and generally calibrates its interpretation around bounded inference rather than claiming demonstrated clinical benefit. Reporting is reduced by unresolved enrolled/analyzed count differences and discussion language that gives strong weight to secondary process findings.

Abstract; Results, Patient disposition; Discussion

any true effect, if present, is likely to be modest

Positioning
4.0

The paper situates the study in LMIC primary-care workforce constraints, prior LLM evaluation literature, and the limitations of earlier rule-based decision support. It also discusses contamination, single-network generalizability, high baseline quality, model-version dependence, rare-event power, and short follow-up.

Introduction; Discussion, limitations paragraphs

Conducting the trial within a single private network of urban clinics in Nairobi may limit generalizability

Lower confidence on Methodological Rigour — domain match limited.

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The paper is broadly internally coherent on the main endpoint and interpretation, but several population-flow and analysis-set details are not fully reconciled. The raw primary event rates favor control while the adjusted odds ratio favors intervention, which is statistically possible under adjustment but would benefit from clearer explanation.

Research ConductNoted

Ethics approvals, trial registration, consent, blinded adjudication, DSMB oversight, and conflict disclosures are reported, but several conduct and reproducibility constraints remain. Data and analytic code are deferred or request-mediated, core system components are proprietary, and OpenAI provided disclosed in-kind support to the implementing organization.

Flags: 4 declared / 5 total

Reference IntegrityClean

All 38 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaHealth · Generic health relevance
Type of advanceEmpirical
Route to useNamed
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim was tested directly in a large pragmatic cluster-randomized trial across 16 real-world primary care facilities with 9,691 patients and blinded expert adjudication.

From the paper
“involving operational leadership within the implementing healthcare organization, Ministry of Health leadership and other local stakeholders to support service improvement”

The paper names the implementing organization's leadership and Ministry of Health leadership as recipients of the findings for service improvement, identifying actors and purpose.

Pathway

Primary care networks, implementing organizations and health ministries in low-resource settings could use this evidence when deciding whether to embed LLM-based decision support in routine consultations.

Barrier

The primary outcome showed no significant benefit and limited precision, so larger adequately powered trials are needed before wide deployment decisions.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Large language models detect methodological errors that human peer review misses in clinical trials

    Dominikus Brian, Xavier Isidro Pereira Hernández, Hao Yin, et al. · 2026 · Computer and information sciences

    Primary claim

    Our findings suggest that LLMs can identify methodological errors in reports of randomized clinical trials that human peer review may miss.

    Tentative
    Evidence Score
    3.7±0.2
    Reliability
    Caveats
  2. Comparing AI chat tools and PubMed for medical literature searches

    Ettien, Downing, Drouhard, et al. · 2026 · Media and communications

    Primary claim

    Our results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for use in any in-depth literature searches

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Generative AI bias against scientific novelty: a cautionary tale from a small-sample evaluation of research proposals

    Diogo Machado · 2026 · Other social sciences

    Primary claim

    Novelty is the only evaluation dimension that systematically predicts disagreement between the two evaluation systems.

    Supported
    Evidence Score
    3.6±0.1
    Reliability
    Caveats

How the rubric works →