Assembling the evidence…
Article-level measures of publication impact (alternative metrics or altmetrics) can help authors and other stakeholders assess engagement with their research and the success of their communication efforts. The wide variety of altmetrics can make interpretation and comparative assessment difficult; available summary tools are either narrowly focused or do not reflect the differing values of metrics from a stakeholder perspective. We created the EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index, a value-based, multi-component metric framework for medical publications. Metric weighting and grouping were informed by a statistical analysis of 2891 Phase III clinical trial publications and by a panel of stakeholders who provided value assessments. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact), each incorporating related altmetrics indicating a different aspect of engagement with the publication. These are averaged to provide a total impact score and benchmarked so that a score of 100 equals the mean scores of Phase III clinical trial publications in the New England Journal of Medicine (NEJM) in 2016. Predictor metrics are defined to estimate likely long-term impact. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore, with the societal impact component providing unique insights. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not. Analysis of NEJM 'notable articles' showed that observational studies had the highest total impact and component scores, except for societal impact, for which surgical studies had the highest score. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics and may enable medical researchers to assess the impact of publications easily and to understand what characterizes impactful research.
The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics
conceptually plausible but not tested against an independent impact criterion
The EMPIRE Index comprises three component scores (social, scholarly, and societal impact)
framework construction and final scoring description directly establish the components
The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore
reported correlation analysis supports the direction with acknowledged construct limits
the societal impact component providing unique insights
low correlations and notable-article analyses support distinctiveness but not full practical superiority
Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not
1-year update results directly show differential metric accumulation over time
Derived from the full evaluation — not a separate score.
Strengths
The EMPIRE Index separates social, scholarly, and societal impact rather than reducing all article-level activity to one score. This structure is supported by correlation and factor-analysis logic and is summarized in the final scoring framework.
↳ Results, Final Scoring Framework; Table 5
Framework construction uses 2,891 Phase III clinical trial publications, a 74-article NEJM benchmark sample, a 1-year update sample, and NEJM notable articles. This gives the paper more than a single retrospective development set for characterizing how the index behaves.
↳ Methods, Sample Acquisition; Results, Characterization in Prospectively Collected Samples
The discussion explicitly notes subjective weighting, narrow publication scope, temporal changes in metrics, cross-sectional predictor development, total-score interpretability problems, and proprietary data dependence. These disclosures make the framework easier to interpret within its intended boundaries.
↳ Discussion, Weaknesses
Limitations
The value assessment was conducted by 14 Novartis employees, mostly from Scientific Communications and Medical Affairs, using open single-round voting. Because the framework is described as value-based, this narrow panel materially limits the generality of the resulting weights.
↳ Methods, Value Assessment; Results, Weighting
The article argues that EMPIRE provides a richer assessment than standalone metrics, but the evidence is based mainly on internal correlations, benchmark behavior, update patterns, and descriptive notable-article analyses. It does not test the index against an independent criterion of publication impact or decision utility.
↳ Abstract; Results, Characterization of the EMPIRE Index; Conclusions
The framework is built around English-language Phase III medical publications and benchmarked to NEJM Phase III articles. The paper acknowledges that assumptions may not apply to other publication types, disease areas, time periods, or stakeholders with different values.
↳ Methods, Sample Acquisition; Discussion, Weaknesses; Conclusions
The article is strongest as a transparent applied framework for organizing medical publication metrics into social, scholarly, and societal components. Its construction uses a large Phase III reference sample and multiple characterization samples, but the decisive value judgments come from a small internal Novartis panel and open voting process. The societal impact component is the most distinctive element, especially because it captures guideline, policy, and patent mentions that are distinct from AAS and CiteScore. The main claims are therefore credible as framework-development claims, but claims of richer practical assessment remain provisional until tested against independent impact or decision-use criteria.
Moderate3.4/5.0
The score reflects a meaningful methodological contribution: a medicine-specific, three-component altmetrics framework that distinguishes social, scholarly, and societal uptake. The contribution is moderated because the strongest claim of a richer assessment is supported mainly by correlations, benchmarking, and descriptive samples rather than independent validation against a ground truth for impact.
“social, scholarly, and societal impact”
The score reflects competent execution for a metric-development study, including a 2,891-publication reference sample, NEJM benchmark sample, 1-year update, factor analysis, correlations, and regression-based predictor scores. Methodological uncertainty remains because value weighting was shaped by a small internal Novartis panel, voting was open and single-round, validation was largely internal or descriptive, and the pipeline depends on proprietary aggregators.
“Voting was conducted openly and in a single round.”
The score reflects a clear structure from motivation through framework construction, benchmarking, characterization, and limitations, with the component definitions and scoring framework explicitly summarized. Precision is reduced by broad phrases such as richer assessment and by minor textual or formula-rendering issues, but the paper remains readily interpretable.
“An overview of the final EMPIRE Index framework is shown in Fig 2.”
The score reflects substantive engagement with JIF critiques, AAS limitations, prior weighted altmetric approaches, factor-analytic literature, and indicator-design principles. It is moderated because the implications of relying on a single-company stakeholder panel and the broader risks of composite metrics are acknowledged but not deeply contextualized.
“grouping and value weighting have a large subjective component”
Watch-outs4 of 4 checks
The numerical reporting and method-result alignment are generally coherent, with no material statistical impossibilities identified. The main internal-coherence watch-out is claim proportionality: statements about a richer assessment and characterizing impactful research go beyond the mainly internal and descriptive validation presented.
The conduct concern is not about apparent fabrication or required ethics approval, but about the subjective value-weighting process being conducted by a small internal Novartis panel and illustrated with Novartis-sponsored publications. This is disclosed in the text, but it narrows how broadly the value-based framework can be interpreted.
Flags: 0 declared / 5 total
63 references verified via PubMed, Crossref, or OpenAlex.
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓PubPeer (coming soon)
Medium3.1/5.0
The score reflects a recognized user problem in interpreting diverse publication metrics, with named use cases such as monitoring publication programmes, journal choice, and communication effectiveness. Relevance is moderated because the clearest demonstrated user base is internal pharmaceutical and medical communications stakeholders rather than a broad external community.
“monitor large numbers of publications”
The score reflects a specified framework with weights, benchmark logic, predictor scores, and case examples showing applied tracking of publications. Readiness is constrained by lack of public code or a deployable calculator and dependence on Altmetric Explorer, PlumX, and other proprietary sources.
“depends on metrics aggregated by two different proprietary systems”
The score reflects limited portability: the framework was developed for English-language medical publications, especially Phase III clinical trials, and benchmarked to NEJM Phase III articles. Some within-medicine extension is shown through NEJM notable articles, but the paper itself states direct relevance depends on stakeholders sharing the panel’s perspective.
“only directly relevant to stakeholders who share broadly similar perspectives to our panel”
The score reflects a cumulative pathway in an active altmetrics literature, with the paper explicitly positioning its process as reweightable by other interested parties. The trajectory remains contingent on fuller prospective and cross-sectional validation and on whether independent groups adopt or adapt the framework.
“any interested party can reweight the EMPIRE index”
Nabu’s assessment, alongside the field’s view.
Evaluation Metadata
AI-powered, human-governed. Something look off? Contact us to request a review.