nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-D90942FF·Framework v5.9·Pipeline v0.4·Evaluated 28 Sept 2026·OECD: 1.2 Computer and information sciences·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Cross-sectional study

Do open access journal articles experience a citation advantage? Results and methodological reflections of an application of multiple measures to an analysis by WoS subject areas

Isabel Basson et al.Isabel Basson, Jaco P. Blanckenberg, Heidi Prozesky·2020·Scientometrics·10.1007/s11192-020-03734-9 ↗

Primary claim

OA journal articles experience a citation advantage in very few subject areas and, in most of these subject areas, the citation advantage was found on only a single measure of citation advantage, namely whether the article was cited at all.

Well-supported · The complete subject-area analysis identifies six qualifying fields, four showing an advantage only for citedness.
In brief

A broad, methodically transparent test finds little OA-journal citation advantage, constrained by retrospective access labels and unmodeled journal-level differences.

ReliabilityCaveats4 of 4 checks completeRead the entry ↓
Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

Abstract This study is one of the first that uses the recently introduced open access (OA) labels in the Web of Science (WoS) metadata to investigate whether OA articles published in Directory of Open Access Journals (DOAJ) listed journals experience a citation advantage in comparison to subscription journal articles, specifically those of which no self-archived versions are available. Bibliometric data on all articles and reviews indexed in WoS, and published from 2013 to 2015, were analysed. In addition to normalised citation score (NCS), we used two additional measures of citation advantage: whether an article was cited at all; and whether an article is among the most frequently cited percentile of articles within its respective subject area (pptopX %). For each WoS subject area, the strength of the relationship between access status (whether an article was published in an OA journal) and each of these three measures was calculated. We found that OA journal articles experience a citation advantage in very few subject areas and, in most of these subject areas, the citation advantage was found on only a single measure of citation advantage, namely whether the article was cited at all. Our results lead us to conclude that access status accounts for little of the variability in the number of citations an article accumulates. The methodology and the calculations that were used in this study are described in detail and we believe that the lessons we learnt, and the recommendations we make, will be of much use to future researchers interested in using the WoS OA labels, and to the field of citation advantage in general.

Key Claims

Primary finding · Well-supported

OA journal articles experience a citation advantage in very few subject areas and, in most of these subject areas, the citation advantage was found on only a single measure of citation advantage, namely whether the article was cited at all.

Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

The complete subject-area analysis identifies six qualifying fields, four showing an advantage only for citedness.

Abstract; Results, pp. 469–470; Tables 1–2
Quote verified
Secondary · Supported

Access status accounts for little of the variability in the number of citations an article accumulates.

Bivariate effect sizes are generally small, though retrospective labels and unmodeled correlates limit interpretation.

Abstract; Discussion, pp. 471–472
Quote verified
Secondary · Well-supported

OA journal articles are more likely than subscription journal articles to be written in a language other than English (φ = 0.16).

A direct full-population comparison reports φ = 0.16 and motivates the English-only restriction.

Results, Analysis of language as a potentially confounding variable, pp. 467–468
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Population-wide subject-area coverage and reporting

The analysis covers approximately 3.67 million English-language records across 250 WoS subject areas and reports the complete subject-area results rather than only favorable findings.

↳ Results, pp. 468–470; Table 2

#methodological rigour

Multiple explicitly defined citation indicators

NCS, citedness, and top-percentile measures use stated citation windows, self-citation rules, coefficients, and effect-size thresholds. This reduces dependence on mean citation counts alone.

↳ Methodology, pp. 463–465

#positioning

Limitations integrated with interpretation

The discussion traces retroactive OA labeling and the Gold DOAJ restriction to the scope of the conclusions and converts them into specific recommendations for future studies.

↳ Discussion, pp. 470–472

Limitations

#methodological rigour

Retrospective labels create exposure uncertainty

An article classified as Gold DOAJ in the 2018 metadata may not have been open at publication or during citation accumulation. Restricting publication years reduces but does not remove this uncertainty.

↳ Publication Year, pp. 466–467; Discussion, pp. 470–472

#methodological rigour

Important citation correlates remain unmodeled

Language is addressed, but journal prestige, publication and review policies, collaboration, and other author or article characteristics are deferred to future work. The findings therefore characterize association rather than a causal OA effect.

↳ Discussion, pp. 471–472

#reporting

Exact Art coefficient is inconsistent

The Results report an NCS coefficient of −0.10 for Art, while Table 1 reports −0.12. The difference does not change its direction or threshold classification but should be corrected.

↳ Results, p. 469; Table 1, p. 470

Does this match your read? Add your assessment ↓

Nabu Commentary

The population-wide design, three citation measures, and complete subject-area table support a quality score near the upper end of the competently executed band. Methodological Rigour does not reach the airtight-methodology range because OA status was assigned retrospectively and potentially influential journal, author, and article characteristics were not jointly modeled. The paper's calibrated association-level framing preserves the central conclusion despite those limitations. Its strongest practical role is to improve later bibliometric evaluations rather than directly determine OA policy.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
4.0±0.1

Contribution
4.0

The study meaningfully extends OA citation research through population-scale coverage, three citation indicators, and disaggregation across 250 subject areas. Its advance is primarily methodological and consolidative because related studies had already reported limited or absent OA advantages.

Introduction, pp. 461–463; Results, pp. 469–470; Tables 1–2

“across all the measures, only 6 of the 250 subject areas in WoS experienced an OA citation advantage”

Methodological Rigour
3.9

The access categories, citation windows, normalization, effect-size thresholds, and language restriction are explicitly operationalized and applied across the full dataset. Retrospective OA labeling, unmodeled journal and author characteristics, and absent confidence intervals prevent placement in the airtight-methodology band.

Methodology, pp. 463–467; Discussion, pp. 470–472

“an article classified as an OA journal article in the 2018 version of the WoS metadata, may not have been OA”

Reporting
4.3

The paper maintains a traceable sequence from research questions through three defined measures to subject-area synthesis, with largely associational language. Precision is slightly reduced by the Art NCS discrepancy and the stronger discussion statement about other factors' effects.

Research Questions, p. 463; Results, pp. 469–470; Table 1

“OA journal articles experience a citation advantage in very few subject areas”

Positioning
4.2

The paper presents the literature as contested, compares its findings directly with methodologically similar studies, and connects major limitations to interpretation and future work. Coverage of the most recent framework-level literature is less extensive than the otherwise substantive engagement.

Conceptual and Empirical Review, pp. 460–463; Discussion, pp. 471–472

“there is still no consensus whether an OA citation advantage exists”

Lower confidence on Methodological Rigour — domain match limited.

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

A minor inconsistency affects the exact NCS coefficient reported for Art, although its direction and substantive interpretation are unchanged.

Research ConductClean

The study uses bibliometric metadata and therefore does not require human-participant ethics approval; funding and absence of conflicts are disclosed. No suspicious data, code, or availability claim is present.

Flags: 1 declared / 5 total

Reference IntegrityClean

34 of 34 checkable references verified

41 references in manuscript 7 have no canonical index record — counted, but not index-checkable citation details diverge from the cited records for 1 reference 1 reference confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice
Type of advanceEmpirical
Route to useSuggested
AdvanceExtends
CitationsFWCI 5.5
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim is tested on the entire population of WoS-indexed articles from 2013 to 2015 across 250 subject areas, which constitutes large-scale real-world bibliometric data, though from a single database and time window.

From the paper
“we believe that the lessons we learnt, and the recommendations we make, will be of much use to future researchers”

The paper names a real-world use of its findings and recommendations for future researchers using WoS OA labels, but does not specify which organisations or decision settings would act on them.

Pathway

Research funders, universities and journal policy bodies would use these findings to inform open access publishing policies and mandates, and bibliometric researchers would apply the methodological recommendations.

Barrier

The findings rely on retrospective WoS OA labels and a single database, and the paper itself recommends further research on journal-related confounding factors before firm policy conclusions can be drawn.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Suspected distortion of citations in high-impact cancer journals

    Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications

    Primary claim

    These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.

    Tentative
    Evidence Score
    3.5±0.1
    Reliability
    Caveats
  2. The citation trap: Papers published at year-end receive systematically fewer citations

    Chao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics

    Primary claim

    papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Quantitative Analysis of the Open Access Citation Advantage

    Masashi Kawai, Koichi Ojiro, Makoto Asaoka, et al. · 2022 · Other social sciences

    Primary claim

    日本の機関リポジトリによるオープンアクセス論文の数は限られるものの,その被引用数と様々なタイプのオープンアクセス論文の被引用数の比較からは,日本の機関リポジトリによるオープンアクセスのメリットは確認できなかった.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats

How the rubric works →