nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-366BABAB·Framework v5.5·Pipeline v0.4·Evaluated 23 Jul 2026·OECD: 3.2 Medical and Health Sciences - Clinical medicine·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Dataset or benchmark paper

Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications.

Pal et al.Pal, Rees·2022·PloS one·10.1371/journal.pone.0265381 ↗

Primary claim

The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics

Tentative · conceptually plausible but not tested against an independent impact criterion
In brief

A practical medical altmetrics framework separates social, scholarly, and societal impact, but its value weights reflect a narrow sponsor-centered panel.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

Article-level measures of publication impact (alternative metrics or altmetrics) can help authors and other stakeholders assess engagement with their research and the success of their communication efforts. The wide variety of altmetrics can make interpretation and comparative assessment difficult; available summary tools are either narrowly focused or do not reflect the differing values of metrics from a stakeholder perspective. We created the EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index, a value-based, multi-component metric framework for medical publications. Metric weighting and grouping were informed by a statistical analysis of 2891 Phase III clinical trial publications and by a panel of stakeholders who provided value assessments. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact), each incorporating related altmetrics indicating a different aspect of engagement with the publication. These are averaged to provide a total impact score and benchmarked so that a score of 100 equals the mean scores of Phase III clinical trial publications in the New England Journal of Medicine (NEJM) in 2016. Predictor metrics are defined to estimate likely long-term impact. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore, with the societal impact component providing unique insights. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not. Analysis of NEJM 'notable articles' showed that observational studies had the highest total impact and component scores, except for societal impact, for which surgical studies had the highest score. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics and may enable medical researchers to assess the impact of publications easily and to understand what characterizes impactful research.

Key Claims

Primary finding · Tentative

The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics

conceptually plausible but not tested against an independent impact criterion

Abstract; Conclusions
Secondary · Supported

The EMPIRE Index comprises three component scores (social, scholarly, and societal impact)

framework construction and final scoring description directly establish the components

Abstract; Results, Final Scoring Framework
Secondary · Supported

The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore

reported correlation analysis supports the direction with acknowledged construct limits

Abstract; Results, Characterization of the EMPIRE Index; Table 5
Secondary · Supported

the societal impact component providing unique insights

low correlations and notable-article analyses support distinctiveness but not full practical superiority

Abstract; Results, Characterization of the EMPIRE Index; Fig 6
Secondary · Supported

Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not

1-year update results directly show differential metric accumulation over time

Abstract; Results, 1-Year Update Phase III Sample Characterization; Fig 4

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Distinct multidimensional metric structure

The EMPIRE Index separates social, scholarly, and societal impact rather than reducing all article-level activity to one score. This structure is supported by correlation and factor-analysis logic and is summarized in the final scoring framework.

↳ Results, Final Scoring Framework; Table 5

#methodological rigour

Large and staged development pipeline

Framework construction uses 2,891 Phase III clinical trial publications, a 74-article NEJM benchmark sample, a 1-year update sample, and NEJM notable articles. This gives the paper more than a single retrospective development set for characterizing how the index behaves.

↳ Methods, Sample Acquisition; Results, Characterization in Prospectively Collected Samples

#positioning

Transparent acknowledgement of limitations

The discussion explicitly notes subjective weighting, narrow publication scope, temporal changes in metrics, cross-sectional predictor development, total-score interpretability problems, and proprietary data dependence. These disclosures make the framework easier to interpret within its intended boundaries.

↳ Discussion, Weaknesses

Limitations

#methodological rigour

Single-company value weighting panel

The value assessment was conducted by 14 Novartis employees, mostly from Scientific Communications and Medical Affairs, using open single-round voting. Because the framework is described as value-based, this narrow panel materially limits the generality of the resulting weights.

↳ Methods, Value Assessment; Results, Weighting

#contribution

Superiority claim lacks external validation

The article argues that EMPIRE provides a richer assessment than standalone metrics, but the evidence is based mainly on internal correlations, benchmark behavior, update patterns, and descriptive notable-article analyses. It does not test the index against an independent criterion of publication impact or decision utility.

↳ Abstract; Results, Characterization of the EMPIRE Index; Conclusions

#application pathway

Transferability remains narrowly demonstrated

The framework is built around English-language Phase III medical publications and benchmarked to NEJM Phase III articles. The paper acknowledges that assumptions may not apply to other publication types, disease areas, time periods, or stakeholders with different values.

↳ Methods, Sample Acquisition; Discussion, Weaknesses; Conclusions

Does this match your read? Add your assessment ↓

Nabu Commentary

The article is strongest as a transparent applied framework for organizing medical publication metrics into social, scholarly, and societal components. Its construction uses a large Phase III reference sample and multiple characterization samples, but the decisive value judgments come from a small internal Novartis panel and open voting process. The societal impact component is the most distinctive element, especially because it captures guideline, policy, and patent mentions that are distinct from AAS and CiteScore. The main claims are therefore credible as framework-development claims, but claims of richer practical assessment remain provisional until tested against independent impact or decision-use criteria.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.4±0.2

Contribution
3.5

The score reflects a meaningful methodological contribution: a medicine-specific, three-component altmetrics framework that distinguishes social, scholarly, and societal uptake. The contribution is moderated because the strongest claim of a richer assessment is supported mainly by correlations, benchmarking, and descriptive samples rather than independent validation against a ground truth for impact.

Abstract; Results, Final Scoring Framework; Results, Characterization of the EMPIRE Index; Table 5

“social, scholarly, and societal impact”

Methodological Rigour
3.1

The score reflects competent execution for a metric-development study, including a 2,891-publication reference sample, NEJM benchmark sample, 1-year update, factor analysis, correlations, and regression-based predictor scores. Methodological uncertainty remains because value weighting was shaped by a small internal Novartis panel, voting was open and single-round, validation was largely internal or descriptive, and the pipeline depends on proprietary aggregators.

Methods, Sample Acquisition; Methods, Statistical Analysis; Methods, Value Assessment; Results, Predictor Scores

“Voting was conducted openly and in a single round.”

Reporting
3.8

The score reflects a clear structure from motivation through framework construction, benchmarking, characterization, and limitations, with the component definitions and scoring framework explicitly summarized. Precision is reduced by broad phrases such as richer assessment and by minor textual or formula-rendering issues, but the paper remains readily interpretable.

Methods, Approach to Developing the Scoring System; Results, Final Scoring Framework; Discussion, Weaknesses

“An overview of the final EMPIRE Index framework is shown in Fig 2.”

Positioning
3.6

The score reflects substantive engagement with JIF critiques, AAS limitations, prior weighted altmetric approaches, factor-analytic literature, and indicator-design principles. It is moderated because the implications of relying on a single-company stakeholder panel and the broader risks of composite metrics are acknowledged but not deeply contextualized.

Introduction; Discussion, Weighting; Discussion, Weaknesses; Conclusions

“grouping and value weighting have a large subjective component”

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The numerical reporting and method-result alignment are generally coherent, with no material statistical impossibilities identified. The main internal-coherence watch-out is claim proportionality: statements about a richer assessment and characterizing impactful research go beyond the mainly internal and descriptive validation presented.

Research ConductNoted

The conduct concern is not about apparent fabrication or required ethics approval, but about the subjective value-weighting process being conducted by a small internal Novartis panel and illustrated with Novartis-sponsored publications. This is disclosed in the text, but it narrows how broadly the value-based framework can be interpreted.

Flags: 0 declared / 5 total

Reference IntegrityClean

63 references verified via PubMed, Crossref, or OpenAlex.

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaResearch capability
Type of advanceMethodological
Route to useNamed
AdvanceAdvances
CitationsFWCI 2.3
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The framework was shown working on its development samples, an independent 1-year update, NEJM notable articles, and two tracked case examples, yet the authors themselves state fuller validation requires ongoing prospective and cross-sectional studies.

From the paper
“it will enable individuals involved in medical research to assess the impact of related publications easily and to understand what characterizes impactful research”

The conclusion names a user group, medical researchers, and a concrete purpose, assessing publication impact, which is a real research-assessment practice rather than further research alone.

Pathway

Medical researchers and pharmaceutical publication teams would use the index to monitor publication impact, target communication efforts, and inform journal selection.

Barrier

The weighting reflects a single company stakeholder panel, the index depends on proprietary metric providers, and the authors state fuller validation is still ongoing.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Suspected distortion of citations in high-impact cancer journals

    Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications

    Primary claim

    These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.

    Tentative
    Evidence Score
    3.5±0.1
    Reliability
    Caveats
  2. The citation trap: Papers published at year-end receive systematically fewer citations

    Chao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics

    Primary claim

    papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Quantitative Analysis of the Open Access Citation Advantage

    Masashi Kawai, Koichi Ojiro, Makoto Asaoka, et al. · 2022 · Other social sciences

    Primary claim

    日本の機関リポジトリによるオープンアクセス論文の数は限られるものの,その被引用数と様々なタイプのオープンアクセス論文の被引用数の比較からは,日本の機関リポジトリによるオープンアクセスのメリットは確認できなかった.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats

How the rubric works →