# Using GPT-4 to annotate the severity of all phenotypic abnormalities within the human phenotype ontology.

- **DOI:** [10.3389/fdgth.2026.1794934](https://doi.org/10.3389/fdgth.2026.1794934)
- **Rating:** Limited (2.6 out of 5)
- **Verdict:** A large-scale HPO severity resource fills a real metadata gap, but positive-only ontology benchmarking does not establish expert-equivalent annotation accuracy.
- **Framework version:** v5.9
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-10-08
- **Evaluation page:** https://www.nabu.science/paper/10.3389%2Ffdgth.2026.1794934

## Key claims

1. Benchmarking demonstrated strong performance across all clinical characteristics, with true positive recall rates ranging from 89% to 100% (mean = 97%).
   - Support: Tentative
   - Type: Primary finding
   - Basis: positive-only HPO-branch recall omits false-positive performance and includes lexically trivial or very small test sets
   - Location: Abstract, Results
2. This indicates that GPT-4 can replicate expert-level curation with high fidelity.
   - Support: Tentative
   - Type: Secondary
   - Basis: no human-expert comparison or independent clinical validation was performed
   - Location: Abstract, Results
3. For the less stringent metric, duplicated phenotypes were annotated consistently at a rate of at least 80%, and for the more stringent metric, the lowest consistency rate was 57% for congenital onset.
   - Support: Supported
   - Type: Secondary
   - Basis: repeat annotations directly support the stated rates, though settings and repeat counts are incompletely reported
   - Location: Results, “Annotation Consistency and Recall”; Figure 2A
4. The categorical approach showed a strong degree of positive correspondence with the continuous severity score.
   - Support: Supported
   - Type: Secondary
   - Basis: the reported association directly compares the two severity representations, though both derive from the same annotations
   - Location: Results, “Severity Classes”

## Strengths

- **Near-complete ontology annotation resource** The workflow produced severity-related annotations for 17,502 of 17,548 HPO terms, with code, CSV output, and an R accessor claimed to be publicly available.
  - Dimension: contribution
  - Location: Introduction, final paragraph; Data Availability Statement
- **Consistency and sensitivity checks reported** The study re-annotated 793 phenotypes using strict and lenient agreement measures and recalculated severity after recoding indirect causal terms to “never.”
  - Dimension: methodological rigour
  - Location: Results, “Annotation Consistency and Recall”; Supplementary Figure S6
- **Severity limitations described concretely** The Discussion identifies treatment availability, within-phenotype variation, pain, hospitalisation, psychological burden, and social functioning as dimensions omitted from the score.
  - Dimension: positioning
  - Location: Discussion, third-to-last paragraph

## Limitations

- **Validation cannot establish expert equivalence** The benchmark measures recall only among expected-positive HPO branches and includes lexically trivial cases and very small denominators. Results also state that the annotations were not validated by human experts.
  - Dimension: methodological rigour
  - Location: Results, “Annotation Consistency and Recall”; Methods, “Calculating the True Positive Rate”
- **Headline accuracy language exceeds evidence** The Abstract infers expert-level curation from 89–100% recall, while the Discussion claims efficiency without substantially compromising accuracy. Neither claim is supported by a full error assessment or direct expert comparison.
  - Dimension: reporting
  - Location: Abstract, Results; Discussion, paragraph 3
- **Severity construction remains internally unclear** Methods says weights range from 1–6 with 6 representing highest severity, then assigns the most severe Lazarin tiers a value of 1. This ambiguity affects interpretation of the composite score.
  - Dimension: methodological rigour
  - Location: Methods, “Quantifying Phenotypic Severity”

## Reliability checks

Caveats (4 of 4 checks).

- **Results check:** Noted
  - Benchmarking demonstrated strong performance across all clinical characteristics, with true positive recall rates ranging from 89% to 100% (mean = 97%): "Severity was operationalised using nine clinical characteristics: congenital onset, reduced fertility, sensory impairments, impaired mobility, immunodeficiency, physical malformations, cancer, intellectual disability, and death." (Abstract, Methods) vs "We sought to evaluate whether each phenotype directly caused a given severity-related clinical characteristic, including: intellectual disability, death, impaired mobility, physical malformations, blindness, sensory impairments, immunodeficiency, cancer, reduced fertility, and/or had a congenital onset." (Methods, Annotating the HPO Using OpenAI GPT-4). Methods enumerates ten characteristics, including blindness separately from sensory impairments, whereas the Abstract lists nine, but the supplied text does not permit reconstruction of the benchmark mean under a reconciled characteristic set. → Effect on result unclear
  - Minor reporting notes (1):
    - Descriptive: "To investigate this further, clinical characteristic justifications were searched for terms indicating indirect causation (“indirectly”, “does not directly”), identifying 4,495 HPO terms. The majority (n = 3,516) were classified as “never” directly causing the associated clinical characteristic, meaning that the indirect cause was not taken into account and only described in the justification." (Results, Annotation Consistency and Recall) vs "1,385 terms rarely caused the associated clinical characteristic. For example, “While this condition (Third degree atrioventricular block) does not directly cause intellectual disability, complications such as cardiac arrest and insufficient oxygen delivery to the brain can cause damage leading to cognitive impairments”. Only 20 phenotypes “often” caused the associated clinical characteristic" (Results, Annotation Consistency and Recall). The category counts sum to 3,516 + 1,385 + 20 = 4,921, exceeding 4,495 by 426, but a phenotype can have different frequency classifications for different characteristics, so the text does not establish that these are mutually exclusive counts. → Effect on result unclear
- **Reporting and transparency:** Noted. The congenital-onset ranking repeats branch labels instead of identifying distinct ranked branches, and a correlation p-value is reported as zero.
  - Data availability: declared · Code availability: declared
- **Reference integrity:** Clean
  - 33 references in manuscript
  - 33 of 33 checkable references found in an index
  - Citation details diverge for 1 reference:
    - #31 Schilder BM Murphy KB Dash H Zhang Y Gordon-Smith R Chapman J Cell type-specific contextualisation of the human phenome: towards the systematic treatment of all rare diseases . medRxiv . ( 2025 ). 10.1101/2023.02.13.23285820 PMC13411100 42363298 (DOI resolves to: “Cell type-specific contextualisation of the human phenome: towards the systematic treatment of all rare diseases” — Schilder, 2023. Differs from the DOI’s record on: year) https://doi.org/10.1101/2023.02.13.23285820
  - 1 DOI could not be checked:
    - #16 Nori H Lee YT Zhang S Carignan D Edgar R Fusi N Can generalist foundation models outcompete special-purpose tuning? Case study in medicine . arXiv . ( 2023 ). 10.48550/arXiv.2311.16452 37396608 (lookup failed (Crossref unavailable) — not the same as wrong)
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
