nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-CCC6305E·Framework v5.9·Pipeline v0.4·Evaluated 28 Sept 2026·OECD: 5.8 Other social sciences·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Cross-sectional study

Open access research outputs receive more diverse citations

Chun-Kai Huang et al.Chun-Kai Huang, Cameron Neylon, Lucy Montgomery et al.·2024·Scientometrics·10.1007/s11192-023-04894-0 ↗

Primary claim

we found a robust association between open access and increased diversity of citation sources by institutions, countries, subregions, regions, and fields of research, across outputs with both high and medium–low citation counts

Supported · large observational corpus and extensive robustness checks support direction, but residual selection and coverage limitations remain
In brief

A large-scale citation-diversity analysis meaningfully extends open-access evidence, but observational confounding limits claims that access itself widens research use.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

Abstract The goal of open access is to allow more people to read and use research outputs. An observed association between highly cited research outputs and open access has been claimed as evidence of increased usage of the research, but this remains controversial. A higher citation count also does not necessarily imply wider usage such as citations by authors from more places. A knowledge gap exists in our understanding of who gets to use open access research outputs and where users are located. Here we address this gap by examining the association between an output’s open access status and the diversity of research outputs that cite it. By analysing large-scale bibliographic data from 2010 to 2019, we found a robust association between open access and increased diversity of citation sources by institutions, countries, subregions, regions, and fields of research, across outputs with both high and medium–low citation counts. Open access through disciplinary or institutional repositories showed a stronger effect than open access via publisher platforms. This study adds a new perspective to our understanding of how citations can be used to explore the effects of open access. It also provides new evidence at global scale of the benefits of open access as a mechanism for widening the use of research and increasing the diversity of the communities that benefit from it.

Key Claims

Primary finding · Supported

we found a robust association between open access and increased diversity of citation sources by institutions, countries, subregions, regions, and fields of research, across outputs with both high and medium–low citation counts

Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

large observational corpus and extensive robustness checks support direction, but residual selection and coverage limitations remain

Abstract; Results
Quote verified
Secondary · Supported

Open access through disciplinary or institutional repositories showed a stronger effect than open access via publisher platforms.

GREEN generally exceeds GOLD across reported comparisons, though overlapping categories and residual selection limit interpretation

Abstract; Results, Comparing OA categories
Quote verified
Secondary · Tentative

It also provides new evidence at global scale of the benefits of open access as a mechanism for widening the use of research and increasing the diversity of the communities that benefit from it.

observational citation diversity cannot identify OA-caused wider use or the full communities benefiting

Abstract; Conclusion
Quote verified
Secondary · Supported

the citation diversity advantage also accrues preferentially to traditionally prestigious centres of research.

regional citation-flow patterns support the direction, with incomplete bibliographic coverage limiting geographic inference

Results, Comparing geographies; Discussion
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Global scale across multiple dimensions

The analysis covers 19 million outputs and 420 million citation links, measuring diversity by institutions, countries, subregions, regions, and fields of research.

↳ Contribution and Method, pp. 827–833

#methodological rigour

Extensive internal robustness analyses

The reported pattern is checked across publication years, two diversity indices, OA categories, equal-sized samples, fields, and 14 citation-count bins.

↳ Robustness of results, pp. 837–840; Supplementary Figures A–N

#positioning

Substantive engagement with competing explanations

The paper discusses randomized null results, author self-selection, prestige, discipline, and resource access, then explicitly limits its conclusion to an observational association.

↳ Introduction, p. 826; Results, p. 836; Discussion, pp. 840–841

Limitations

#methodological rigour

Residual OA selection remains unresolved

Citation-count stratification reduces one source of confounding but does not independently adjust for discipline, institutional prestige, author characteristics, or research-resource context.

↳ Method, p. 827; Results, pp. 834–836; Discussion, pp. 840–841

#methodological rigour

Coverage bias is not quantified

Crossref-only DOI coverage, MAG inclusion, and the exclusion of outputs with fewer than two citations constrain regional and disciplinary generalization without quantitative bias bounds.

↳ Input data, pp. 832–833; Data limitations, pp. 841–842

#reporting

Benefit language exceeds the design

The Abstract describes OA as a mechanism widening use, and the Conclusion treats diverse citing outputs as wider use, although the study identifies an observational association in a narrow proxy.

↳ Abstract, final sentence; Conclusion, pp. 842–843

Does this match your read? Add your assessment ↓

Nabu Commentary

The study’s contribution rests on combining global scale with a citation-diversity framing that extends beyond conventional citation counts. The methodology gains credibility from repeated analyses across time, measures, OA categories, fields, samples, and citation-count bins, but those checks do not fully adjust for selection into OA or quantify bibliographic coverage bias. The paper’s literature positioning and limitations discussion are substantive, including explicit treatment of randomized null evidence and regional data gaps. Scores are moderated chiefly because some Abstract and Conclusion wording turns an observational citation association into stronger claims about OA-driven benefits and wider use.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.6±0.2

Contribution
3.9

The study extends citation-diversity analysis from narrow prior samples to 19 million outputs and five institutional, geographic, and disciplinary groupings. The advance is meaningful but uses standard diversity indices in an observational extension rather than introducing causal identification or a new analytic method.

Contribution, pp. 827–828; Relevant research, pp. 826–827

“making it the largest study of this type to date”

Methodological Rigour
3.3

The workflow defines the corpus, two diversity indices, and extensive checks across years, measures, citation bins, samples, fields, and OA categories. Crossref/MAG selection and classification errors are not quantitatively bounded, and citation-count stratification does not resolve all residual OA-selection confounding.

Method, pp. 827–834; Robustness of results, pp. 837–840; Data limitations, pp. 841–842

“our analysis is not able to confirm the exact causal links”

Reporting
3.6

The article has a coherent progression from motivation and methods through results, robustness, limitations, and conclusion. Reporting is reduced when the Abstract and Conclusion describe causal benefits and wider use more strongly than the observational citation proxy establishes.

Abstract, final sentence; Results, p. 836; Conclusion, pp. 842–843

“benefits of open access as a mechanism for widening the use of research”

Positioning
3.9

The paper engages randomized null evidence, self-selection, prestige, disciplinary confounding, and the two closest diversity studies. Its limitations discussion connects coverage problems to regional findings, although the magnitude of those effects on the headline association is not quantitatively bounded.

Introduction, p. 826; Relevant research, pp. 826–827; Discussion and Data limitations, pp. 840–842

“there may be limitations in our coverage of certain areas of Asia, Sub-Saharan Africa and other regions”

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The reported corpus, methods, and descriptive findings are internally consistent, and the paper explicitly disclaims causal identification. Some Abstract, Results, and Conclusion wording nevertheless shifts from association in citation-based proxies toward causal benefits and wider use.

Research ConductNoted

Funding and competing interests are disclosed, the study uses bibliographic metadata rather than human-subject data, and processed data and analysis code are claimed available through Zenodo and GitHub. One or more identifiers named in the paper's availability or registration statements did not resolve when checked on 2026-09-28T16:08:08.243051+00:00: https://​github.​com/​Curtin-​Open-​Knowl​edge-​Initi​ative/​citat​ion-​diver​sity. Recorded as a discrepancy between the paper's claims and the cited sources; not an assessment of the underlying research.

Flags: 3 declared / 5 total

Reference IntegrityClean

21 of 21 checkable references verified

25 references in manuscript 4 are books, websites or datasets — counted, but not index-checkable

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 10
Type of advanceEmpirical
Route to useSuggested
AdvanceExtends
CitationsFWCI 25.1
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The association is demonstrated on population-scale real-world bibliographic data (19 million outputs, 420 million citation links), consistent across years, regions, fields, measures, and citation-count strata.

From the paper
“a potential insight into where the benefits of OA are being seen and a guide to improving our policy implementation of OA for wider access”

The conclusion names a real-world use, guiding open-access policy implementation, but names no specific actor, organisation or setting that would act on the findings.

Pathway

Open-access policymakers, funders and university administrators could use citation diversity evidence to justify repository and open-access mandates that widen who uses research.

Barrier

The observational design cannot rule out confounding such as author self-selection and institutional prestige, leaving the causal benefit of open access unproven.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Suspected distortion of citations in high-impact cancer journals

    Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications

    Primary claim

    These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.

    Tentative
    Evidence Score
    3.5±0.1
    Reliability
    Caveats
  2. The citation trap: Papers published at year-end receive systematically fewer citations

    Chao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics

    Primary claim

    papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Quantitative Analysis of the Open Access Citation Advantage

    Masashi Kawai, Koichi Ojiro, Makoto Asaoka, et al. · 2022 · Other social sciences

    Primary claim

    日本の機関リポジトリによるオープンアクセス論文の数は限られるものの,その被引用数と様々なタイプのオープンアクセス論文の被引用数の比較からは,日本の機関リポジトリによるオープンアクセスのメリットは確認できなかった.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats

How the rubric works →