nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Readiness
Report #NBU-14E0E8BF·Framework v5.9·Pipeline v0.4·Evaluated 08 Oct 2026·OECD: 3.1 Basic medicine·Subfield: Molecular Biology·Basis: Full text·Type: Dataset or benchmark paper

Using GPT-4 to annotate the severity of all phenotypic abnormalities within the human phenotype ontology.

Murphy et al.Murphy, Schilder, Skene·2026·Frontiers in digital health·10.3389/fdgth.2026.1794934 ↗

Primary claim

Benchmarking demonstrated strong performance across all clinical characteristics, with true positive recall rates ranging from 89% to 100% (mean = 97%).

Tentative · positive-only HPO-branch recall omits false-positive performance and includes lexically trivial or very small test sets
In brief

A large-scale HPO severity resource fills a real metadata gap, but positive-only ontology benchmarking does not establish expert-equivalent annotation accuracy.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

INTRODUCTION: The Human Phenotype Ontology (HPO) provides a unified framework cataloguing over 17,500 phenotypic abnormalities across more than 8,600 rare diseases, defining hierarchical relationships between them. For example, classifying missing arms and missing legs as both abnormalities of the limb. This structure enables phenome-wide analyses, including the prioritisation of phenotypes as candidates for gene therapy. However, the HPO currently lacks sufficient metadata describing the clinical severity of these phenotypes. Manual expert curation at this scale would be prohibitively labour-intensive, creating a need for automated approaches to systematically annotate phenotypic severity.

METHODS: GPT-4, a large language model (LLM) developed by OpenAI, was employed to annotate the severity of all phenotypic abnormalities catalogued in the HPO. Severity was operationalised using nine clinical characteristics: congenital onset, reduced fertility, sensory impairments, impaired mobility, immunodeficiency, physical malformations, cancer, intellectual disability, and death. Each characteristic was further qualified by frequency of occurrence across four levels: never, rarely, often, and always. To assess annotation quality, GPT-4's outputs were benchmarked against ground-truth labels embedded within the HPO itself. For instance, phenotypes residing in the "Cancer" HPO branch were expected to be annotated as cancer-causing. A novel severity scoring system was then developed that integrates both the nature of each clinical characteristic and its frequency of occurrence.

RESULTS: Benchmarking demonstrated strong performance across all clinical characteristics, with true positive recall rates ranging from 89% to 100% (mean = 97%). This indicates that GPT-4 can replicate expert-level curation with high fidelity. The resulting severity scoring system produced quantitative severity metrics for phenotypic abnormalities across the HPO, incorporating both the type and frequency of associated clinical characteristics.

DISCUSSION: These findings demonstrate that LLMs can automate the large-scale curation of clinical metadata with a high degree of accuracy, substantially reducing the burden of manual expert annotation. The severity metrics generated here provide a foundation for systematically ranking human phenotypes by their impact on health and quality of life, enabling more principled prioritisation of targets for therapeutic intervention, particularly in the context of rare diseases where evidence is sparse and resources for curation are limited. Future work may extend this framework to incorporate additional clinical dimensions or validate annotations against independent clinical datasets.

Key Claims

Primary finding · Tentative

Benchmarking demonstrated strong performance across all clinical characteristics, with true positive recall rates ranging from 89% to 100% (mean = 97%).

positive-only HPO-branch recall omits false-positive performance and includes lexically trivial or very small test sets

Abstract, Results
Quote verified
Secondary · Tentative

This indicates that GPT-4 can replicate expert-level curation with high fidelity.

no human-expert comparison or independent clinical validation was performed

Abstract, Results
Quote verified
Secondary · Supported

For the less stringent metric, duplicated phenotypes were annotated consistently at a rate of at least 80%, and for the more stringent metric, the lowest consistency rate was 57% for congenital onset.

repeat annotations directly support the stated rates, though settings and repeat counts are incompletely reported

Results, “Annotation Consistency and Recall”; Figure 2A
Quote verified
Secondary · Supported

The categorical approach showed a strong degree of positive correspondence with the continuous severity score.

the reported association directly compares the two severity representations, though both derive from the same annotations

Results, “Severity Classes”
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Near-complete ontology annotation resource

The workflow produced severity-related annotations for 17,502 of 17,548 HPO terms, with code, CSV output, and an R accessor claimed to be publicly available.

↳ Introduction, final paragraph; Data Availability Statement

#methodological rigour

Consistency and sensitivity checks reported

The study re-annotated 793 phenotypes using strict and lenient agreement measures and recalculated severity after recoding indirect causal terms to “never.”

↳ Results, “Annotation Consistency and Recall”; Supplementary Figure S6

#positioning

Severity limitations described concretely

The Discussion identifies treatment availability, within-phenotype variation, pain, hospitalisation, psychological burden, and social functioning as dimensions omitted from the score.

↳ Discussion, third-to-last paragraph

Limitations

#methodological rigour

Validation cannot establish expert equivalence

The benchmark measures recall only among expected-positive HPO branches and includes lexically trivial cases and very small denominators. Results also state that the annotations were not validated by human experts.

↳ Results, “Annotation Consistency and Recall”; Methods, “Calculating the True Positive Rate”

#reporting

Headline accuracy language exceeds evidence

The Abstract infers expert-level curation from 89–100% recall, while the Discussion claims efficiency without substantially compromising accuracy. Neither claim is supported by a full error assessment or direct expert comparison.

↳ Abstract, Results; Discussion, paragraph 3

#methodological rigour

Severity construction remains internally unclear

Methods says weights range from 1–6 with 6 representing highest severity, then assigns the most severe Lazarin tiers a value of 1. This ambiguity affects interpretation of the composite score.

↳ Methods, “Quantifying Phenotypic Severity”

Does this match your read? Add your assessment ↓

Nabu Commentary

The main contribution is a near-complete, reusable set of GPT-4-generated severity annotations for the HPO. The score is driven downward by the validation design: expected-positive ontology branches establish recall but do not measure false positives, overall accuracy, or agreement with expert severity judgements. The manuscript’s readable structure and detailed discussion of omitted severity dimensions do not resolve repeated claims of expert-level fidelity and actionable prioritisation. The resource is therefore best treated as a hypothesis-generating curation aid pending independent validation.

Check another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Study Quality

Limited2.6

Confidence high
Contribution
3.5

The paper releases severity-related annotations for 17,502 HPO terms and constructs continuous and categorical severity representations, addressing a documented metadata gap. The advance is meaningful, although the resource’s practical value depends on validation not supplied here.

Introduction, final two paragraphs; Results, “Quantifying Phenotypic Severity”; Data Availability Statement

“less than 1% of terms within the HPO contain metadata such as time course and severity”

Methodological Rigour
1.8

The workflow uses a versioned HPO release, repeats 793 terms, reports cost and throughput, and includes an indirect-causation sensitivity analysis. Its central evaluation lacks an independent comparator or negative-class assessment, remains vulnerable to HPO training exposure, and reports recall without precision, specificity, or uncertainty.

Methods, “Calculating the True Positive Rate”; Results, “Annotation Consistency and Recall”; Discussion, paragraph 3

“our annotations were not validated by human experts”

Reporting
3.1

The manuscript follows a readable progression from annotation through consistency, recall, scoring, and severity classes. However, the Abstract and Discussion repeatedly describe positive-only recall as expert-level accuracy, materially overstating the evidence.

Abstract, Results; Discussion, paragraphs 3 and conclusion

“GPT-4 can replicate expert-level curation with high fidelity”

Positioning
2.8

The paper situates the framework against HPO curation and the Lazarin severity tiers and discusses omitted severity dimensions in detail. It nevertheless retains strong claims about accuracy and actionable prioritisation after acknowledging no expert validation and substantial score limitations.

Introduction; Results, “Annotation Consistency and Recall”; Discussion, final three paragraphs

“These limitations extend beyond the annotations to the construction of the severity score itself”

Lower confidence on Contribution, Positioning — domain match limited.

Reliability

Caveats4 of 4 checks


Results checkNoted
  • Benchmarking demonstrated strong performance across all clinical characteristics, with true positive recall rates ranging from 89% to 100% (mean = 97%): "Severity was operationalised using nine clinical characteristics: congenital onset, reduced fertility, sensory impairments, impaired mobility, immunodeficiency, physical malformations, cancer, intellectual disability, and death." (Abstract, Methods) vs "We sought to evaluate whether each phenotype directly caused a given severity-related clinical characteristic, including: intellectual disability, death, impaired mobility, physical malformations, blindness, sensory impairments, immunodeficiency, cancer, reduced fertility, and/or had a congenital onset." (Methods, Annotating the HPO Using OpenAI GPT-4). Methods enumerates ten characteristics, including blindness separately from sensory impairments, whereas the Abstract lists nine, but the supplied text does not permit reconstruction of the benchmark mean under a reconciled characteristic set. → Effect on result unclear
Reporting and transparencyNoted

The congenital-onset ranking repeats branch labels instead of identifying distinct ranked branches, and a correlation p-value is reported as zero.

Data availability: declared · Code availability: declared

Reference IntegrityClean

33 references in manuscript 33 of 33 checkable references found in an index

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Readiness

Where this paper’s evidence sits on the path from initial observation to real-world use.

Demonstrated

Annotations, code, CSV output, and an R accessor are claimed to be publicly available, allowing research use now. Clinical or prioritisation use remains preliminary because neither annotations nor the composite score were independently validated.

Data Availability Statement; Results, “Annotation Consistency and Recall”; Discussion, final three paragraphs
Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Tig1 regulates proximo-distal identity during salamander limb regeneration

    Oliveira, Knapp, Elewa, et al. · 2022 · Biological sciences

    Primary claim

    Tig1 reprogrammes distal cells to a proximal identity, upregulating Prod1 and inhibiting Hoxa13 and distal transcriptional networks

    Tentative
    Study quality
    Strong3.8
    Reliability
    Caveats
  2. HDACi mediate UNG2 depletion, dysregulated genomic uracil and altered expression of oncoproteins and tumor suppressors in B- and T-cell lines

    Iveland, Hagen, Sharma, et al. · 2020 · Basic medicine

    Primary claim

    UNG2 degradation appeared to be ubiquitous and was observed across several mammalian cell lines of different origin and with several HDACis.

    Supported
    Study quality
    Sound3.4
    Reliability
    Caveats
  3. RPA2 winged-helix domain facilitates UNG-mediated removal of uracil from ssDNA; implications for repair of mutagenic uracil at the replication fork

    Kavli, Iveland, Buchinger, et al. · 2021 · Biological sciences

    Primary claim

    we demonstrate that uracil-DNA glycosylase (UNG), but not SMUG1 efficiently excises uracil from replication protein A (RPA)-coated ssDNA

    Supported
    Study quality
    Strong3.8
    Reliability
    Caveats

How the rubric works →