User needs guide integrated system design
Analysis of 1,370 queries, 555 survey responses and a 14-expert workshop links the specialist architecture to identifiable consumer health needs.
↳ Results, User-centred analysis of health needs; Fig. 1
Large language models are enabling a new generation of health agents, yet their ability to address the diverse needs of individuals in everyday health settings remains underexplored. Here we developed a comprehensive personal health agent that reasons over multimodal data from consumer wellness devices and medical records to provide personalized health support. Through analysis of web search and health forum queries, together with user-centred research involving end users and health experts, we identified four key user journeys: general health knowledge, personal data interpretation, wellness advice, and symptom assessment. These journeys informed three specialist sub-agents: (1) a data science agent that analyses longitudinal wearable and medical data, incorporating population-level statistics to contextualize numerical insights; (2) a health domain expert agent that integrates personal and contextual data with health knowledge to generate personalized insights; and (3) a health coach agent that synthesizes insights, supports multi-turn interaction and goal setting, applies specified psychological strategies and tracks progress. We integrated these capabilities in the personal health agent, an open-source multi-agent framework for dynamic, personalized health interactions. We evaluated the sub-agents and full system across 11 benchmark tasks using automated and human evaluation, with more than 7,000 annotations and 1,350 h of effort from health experts and end users. This represents the most comprehensive evaluation of a personal health agent thus far and provides a concrete foundation for improving access to high-quality personalized health support more broadly.
End users ranked PHA as their top choice nearly half the time (46.7% on overall preference and 44.4% on response quality)
blinded role-play comparisons support preference within the evaluated personas, not real-world effectiveness
the DS agent significantly outperformed the baseline overall (average score of 75.6 ± 1.4% versus 53.7 ± 1.8%, P < 0.001)
paired benchmark gains are substantial, but Gemini-based autorating and a plain-backbone comparison temper evidentiary strength
the DE agent improves the accuracy of the base Gemini across nearly all domains, with overall 83.6% versus 81.8%
the same-backbone MCQ comparison supports a bounded 1.8-percentage-point accuracy gain
Experts gave PHA the highest overall quality rating (72.8 ± 1.4% versus 64.3 ± 1.3% for parallel multi-agent and 54.4 ± 1.4% for PHIA, all pairwise P < 0.001, GLMM with log link function).
blinded transcript ratings support comparative quality, with generalization constrained by three expert raters
Our analysis shows that the PHA demonstrates strong competency across all four CUJs
symptom assessment was evaluated at sub-agent level rather than established for the integrated PHA
Derived from the full evaluation — not a separate score.
Strengths
Analysis of 1,370 queries, 555 survey responses and a 14-expert workshop links the specialist architecture to identifiable consumer health needs.
↳ Results, User-centred analysis of health needs; Fig. 1
Blinded, counterbalanced human evaluations combine end-user and expert judgments with expert-authored code tests and statistical models for repeated ratings.
↳ Methods, DS agent evaluation; HC agent evaluation; PHA evaluation; Statistical analysis
The named implementation, prompt templates, persona subset and benchmark provide a basis for comparing alternative agents and backbones.
↳ Discussion, PHA advantages and user journeys; Data availability; Code availability
Limitations
Routing, reflection, memory and collaboration are not individually isolated. The parallel-agent comparison tests a bundle of differences rather than identifying which mechanism produces the gains.
↳ Methods, PHA development: multi-agent collaboration; PHA evaluation
Short-term persona role-play measures conversational performance rather than sustained behaviour change or health outcomes. DE summaries received potential-harm ratings in 17% of cases, including 2% moderate-to-severe harm, without a corresponding full-system safety evaluation.
↳ Results, Multimodal clinical health summaries; Methods, PHA evaluation; Discussion, Limitations and conclusion
The claim of strong competency across all four user journeys extends beyond the integrated PHA evaluation for symptom assessment. Sub-agent diagnostic performance does not establish equivalent capability or safety in the orchestrated system.
↳ Discussion, PHA advantages and user journeys; Results, Accuracy of diagnostic conversations
The user-needs analysis and evaluations across specialist and integrated tasks establish a meaningful systems contribution. Blinded comparisons and repeated-rating analyses support conversational-performance gains, but the absence of targeted ablations prevents attribution to individual orchestration mechanisms. Explicit prototype and longitudinal-outcome limitations appropriately bound much of the interpretation, whereas the claim of competency across all four user journeys remains too broad for the integrated evidence. The reusable implementation and benchmark offer a clearer near-term impact pathway than direct consumer deployment.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
The score reflects integration of user-derived needs, three specialist agents and an evaluated orchestration framework with reusable implementation and benchmarks. The advance is primarily systems engineering over existing personal-health-agent work rather than evidence of improved health outcomes.
Blinded, counterbalanced comparisons, expert-authored unit tests and repeated-rating analyses support the execution. Coupled orchestration mechanisms, role-play evaluation, small expert panels and autorater dependence leave meaningful uncertainty about mechanism and external validity.
The needs-to-design-to-evaluation structure makes the system and comparisons understandable. The assertion of competency across all four user journeys exceeds the integrated-system evidence for symptom assessment despite explicit prototype and outcome qualifications elsewhere.
The paper engages closely related wearable, coaching, diagnostic and multi-agent systems and explicitly limits interpretation to conversational quality and short-term task efficacy. Safety harms and real-world outcome evidence could be integrated more fully into the interpretation, but deployment boundaries and validation requirements are substantively stated.
Caveats4 of 4 checks
One reported value does not reconcile, but the key claims do not rest on it.
Ethics approvals, consent, funding and competing interests are declared. Code availability is described using both future and present tense.
Conflict of interest: declared · Ethics approval: declared · Preregistration: not stated · Data availability: declared · Code availability: declared
32 references in manuscript 27 of 27 checkable references found in an index 5 have no canonical index record — counted, but not index-checkable 2 references confirmed by manual review
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
The integrated PHA was shown working in controlled live user studies with 45 end users and expert raters plus synthesized conversations, but has not been tested in independent real-world deployment or routine operation.
“provides a concrete foundation for improving access to high-quality personalized health support more broadly”
The paper names a real-world use, personalized health support for consumers, but explicitly frames itself as a research prototype and open-source foundation rather than naming who would deploy it in practice.
Consumer health and wearable companies would adapt the open-source multi-agent framework into wellness products delivering personalized health insights and coaching.
The prototype lacks longitudinal validation of health outcomes, safety safeguards, privacy controls and computational efficiency required before real-world consumer deployment.
Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Zhu, Lin, Shi, et al. · 2026 · Computer and information sciences
On Harvey LAB's Firm Knowledge benchmark, Recursa raises the all-pass rate on held-out test tasks from 30.0% to 55.0% with DeepSeek-V4.1-Flash and from 35.0% to 46.0% with Qwen3.8-Flash.
TentativeYaochen Wang, Chenguang Wang, Ming Li, et al. · 2026 · Computer and information sciences
Major paper improvements usually do not lead to higher AI review scores. Even revisions annotated as major improvements usually receive unchanged scores.
SupportedTartaglini, Wu, Wiedemer, et al. · 2026 · Computer and information sciences
simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems
Supported