Immune-related adverse events (irAEs) affect up to 40% of patients receiving immune checkpoint inhibitors, yet identification requires laborious, inconsistent manual chart review. Here we show that agentic artificial intelligence coupled with human verification could enhance efficiency, performance and consistency for irAE assessment. We developed and evaluated an agentic large language model system to extract the presence, temporality, severity grade, attribution and certainty of six irAE types from clinical notes. Retrospectively, the system achieved a macro-averaged F 1 of 0.92 for current-event detection and 0.66 for multi-class severity grading; self-consistency improved F 1 by 0.14. The best-performing configuration cost approximately US$0.02 per note. In prospective silent deployment over 3 months, the binary F 1 was 0.72–0.79. In a randomized crossover study of clinical trial staff, agentic assistance reduced annotation time by 2.71 min ( P < 0.001), increased complete-match accuracy (odds ratio 1.45; 95% confidence interval 1.01–2.08; P = 0.045) and improved interannotator agreement (Krippendorff’s α from 0.22–0.51 to 0.82–0.85).
In a randomized crossover study of clinical trial staff, agentic assistance reduced annotation time by 2.71 min (P < 0.001)
randomized within-participant comparison supports time savings for the bundled interface, though repeated notes and comparator differences limit attribution
Retrospectively, the system achieved a macro-averaged F1 of 0.92 for current-event detection and 0.66 for multi-class severity grading
unstated encounter-level split separation and absent-class grading credits limit interpretation of retrospective performance
In prospective silent deployment over 3 months, the binary F1 was 0.72–0.79
temporally prospective testing supports the reported range within one system, with mostly single-annotator reference labels
increased complete-match accuracy (odds ratio 1.45; 95% confidence interval 1.01–2.08; P = 0.045)
bundled workflow comparison with participant-only clustering yields a borderline secondary result without multiplicity adjustment
improved interannotator agreement (Krippendorff's α from 0.22–0.51 to 0.82–0.85)
shared model suggestions increase measured agreement but do not separate independent verification from anchoring
Derived from the full evaluation — not a separate score.
Strengths
The system extracts temporality, severity, attribution and certainty rather than only event presence, addressing information needed for detailed adverse-event curation.
↳ Introduction; Results, phase 1; Table 1
The crossover study compares both workflows within participants and reports a model-estimated reduction of 2.71 minutes per note. This is the clearest evidence for practical workflow benefit.
↳ Methods and Results, phase 3; Table 3
The paper reports lower prospective performance rather than relying solely on retrospective benchmarks. It connects this decline to monitoring and recalibration and explicitly states that no phase establishes external validation.
↳ Results, phase 2; Table 2; Discussion
Limitations
The development and test notes come from a corpus of 289 notes drawn from 100 encounters, without stated encounter- or patient-level separation. Later prospective testing does not resolve this exposure in the retrospective benchmark.
↳ Methods, Note-level irAE system development and evaluation (phase 1)
Phase-1 and phase-2 F1 estimates lack uncertainty intervals, while the retrospective grade macro-F1 assigns 1.0 to classes without examples. These choices complicate interpretation of the headline grading score.
↳ Table 1 footnote; Figure 3 legend; Results, phases 1–2
The assisted arm combines model suggestions, evidence highlighting and interface features against spreadsheet review. Complete-match accuracy improves only marginally in an unadjusted secondary analysis, and grade-level accuracy is not significantly improved.
↳ Methods, phase 3; Results, phase 3; Table 3
The study's main advance is a staged evaluation of granular irAE extraction, with randomized evidence that the assisted workflow reduces annotation time. The retrospective benchmark receives less weight because encounter-level split separation is unstated, grade macro-F1 credits absent classes, and phase-1/2 performance lacks uncertainty intervals. Prospective attenuation and mostly single-annotator reference labels constrain the performance evidence, while the bundled interface prevents isolating an agentic contribution to workflow gains. The paper appropriately identifies external validation as unresolved, and its evidence supports further human-verified curation studies rather than a generally validated clinical detection instrument.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.2
Confidence mediumThe study extends binary chart-level irAE identification to note-level clinical attributes and combines retrospective, prospective and randomized workflow evaluation. The advance is primarily applied and evaluative rather than a separately established architectural breakthrough.
Comparator coverage, a self-consistency ablation and staged testing support assessment, but the retrospective development/test split lacks stated encounter-level separation. Missing F1 uncertainty, the absent-class metric convention and mostly single-annotated prospective labels further limit execution.
The phase-based presentation and endpoint definitions make the central results followable. Broad claims about clinical detection and accuracy are less precise than the specific curation outcomes and null grade-level comparisons.
The paper distinguishes its advance from prior systems and traces prospective attenuation, documentation ambiguity and anchoring to interpretive consequences. Engagement with automation-bias literature and the null grade-accuracy result is less developed.
Caveats4 of 4 checks
Ethics, consent, competing interests, code release and restricted clinical-data access are declared. Transparency notes concern inconsistent power-calculation inputs, no reported preregistration and an adjusted accuracy result asserted without numerical estimates.
Conflict of interest: declared · Ethics approval: declared · Data availability: declared · Code availability: declared
43 references in manuscript 40 of 40 checkable references found in an index 3 have no canonical index record — counted, but not index-checkable 1 reference confirmed by manual review
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
Silent live-data processing and a randomized study using a working interface establish testing in an operational context. They do not establish patient benefit or readiness for autonomous clinical use.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Julia Gallini, Michael L. Alosco, Bobak Abdolmohammadi, et al. · 2026 · Clinical medicine
the estimated bias-adjusted prevalence of CTE neuropathology among all former NFL players who died between 2008 and 2023 was 74.5% (95% CI, 69.6%-78.3%).
TentativeOlaf Blanke, Polona Pozeg, Masayuki Hara, et al. · 2014 · Clinical medicine
participants experienced being in the presence of another person in the asynchronous versus synchronous stimulation condition (p < 0.01) and experienced being touched by that invisible presence behind them (p < 0.01).
SupportedMajani, Chillo, Akida, et al. · 2025 · Clinical medicine
POX demonstrated a sensitivity of 50.0% (95% CI: 32.1-67.9), a specificity of 99.5% (95% CI: 99.4-99.7), a false-positive rate of 0.4%, and an overall accuracy of 99.5% (95% CI: 99.2-99.5)
Tentative