# Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications.

- **DOI:** [10.1371/journal.pone.0265381](https://doi.org/10.1371/journal.pone.0265381)
- **Rating:** Sound (3.4 out of 5)
- **Verdict:** A practical medical altmetrics framework separates social, scholarly, and societal impact, but its value weights reflect a narrow sponsor-centered panel.
- **Framework version:** v5.5
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-07-23
- **Evaluation page:** https://www.nabu.science/paper/10.1371%2Fjournal.pone.0265381

## Key claims

1. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics
   - Support: Tentative
   - Type: Primary finding
   - Basis: conceptually plausible but not tested against an independent impact criterion
   - Location: Abstract; Conclusions
2. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact)
   - Support: Supported
   - Type: Secondary
   - Basis: framework construction and final scoring description directly establish the components
   - Location: Abstract; Results, Final Scoring Framework
3. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore
   - Support: Supported
   - Type: Secondary
   - Basis: reported correlation analysis supports the direction with acknowledged construct limits
   - Location: Abstract; Results, Characterization of the EMPIRE Index; Table 5
4. the societal impact component providing unique insights
   - Support: Supported
   - Type: Secondary
   - Basis: low correlations and notable-article analyses support distinctiveness but not full practical superiority
   - Location: Abstract; Results, Characterization of the EMPIRE Index; Fig 6
5. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not
   - Support: Supported
   - Type: Secondary
   - Basis: 1-year update results directly show differential metric accumulation over time
   - Location: Abstract; Results, 1-Year Update Phase III Sample Characterization; Fig 4

## Strengths

- **Distinct multidimensional metric structure** The EMPIRE Index separates social, scholarly, and societal impact rather than reducing all article-level activity to one score. This structure is supported by correlation and factor-analysis logic and is summarized in the final scoring framework.
  - Dimension: contribution
  - Location: Results, Final Scoring Framework; Table 5
- **Large and staged development pipeline** Framework construction uses 2,891 Phase III clinical trial publications, a 74-article NEJM benchmark sample, a 1-year update sample, and NEJM notable articles. This gives the paper more than a single retrospective development set for characterizing how the index behaves.
  - Dimension: methodological rigour
  - Location: Methods, Sample Acquisition; Results, Characterization in Prospectively Collected Samples
- **Transparent acknowledgement of limitations** The discussion explicitly notes subjective weighting, narrow publication scope, temporal changes in metrics, cross-sectional predictor development, total-score interpretability problems, and proprietary data dependence. These disclosures make the framework easier to interpret within its intended boundaries.
  - Dimension: positioning
  - Location: Discussion, Weaknesses

## Limitations

- **Single-company value weighting panel** The value assessment was conducted by 14 Novartis employees, mostly from Scientific Communications and Medical Affairs, using open single-round voting. Because the framework is described as value-based, this narrow panel materially limits the generality of the resulting weights.
  - Dimension: methodological rigour
  - Location: Methods, Value Assessment; Results, Weighting
- **Superiority claim lacks external validation** The article argues that EMPIRE provides a richer assessment than standalone metrics, but the evidence is based mainly on internal correlations, benchmark behavior, update patterns, and descriptive notable-article analyses. It does not test the index against an independent criterion of publication impact or decision utility.
  - Dimension: contribution
  - Location: Abstract; Results, Characterization of the EMPIRE Index; Conclusions
- **Transferability remains narrowly demonstrated** The framework is built around English-language Phase III medical publications and benchmarked to NEJM Phase III articles. The paper acknowledges that assumptions may not apply to other publication types, disease areas, time periods, or stakeholders with different values.
  - Dimension: application pathway
  - Location: Methods, Sample Acquisition; Discussion, Weaknesses; Conclusions

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. The numerical reporting and method-result alignment are generally coherent, with no material statistical impossibilities identified. The main internal-coherence watch-out is claim proportionality: statements about a richer assessment and characterizing impactful research go beyond the mainly internal and descriptive validation presented.
- **Research conduct:** Noted. The conduct concern is not about apparent fabrication or required ethics approval, but about the subjective value-weighting process being conducted by a small internal Novartis panel and illustrated with Novartis-sponsored publications. This is disclosed in the text, but it narrows how broadly the value-based framework can be interpreted.
- **Reference integrity:** Clean. 63 references verified via PubMed, Crossref, or OpenAlex.
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
