# Do open access journal articles experience a citation advantage? Results and methodological reflections of an application of multiple measures to an analysis by WoS subject areas

- **DOI:** [10.1007/s11192-020-03734-9](https://doi.org/10.1007/s11192-020-03734-9)
- **Rating:** Strong (4.0 out of 5)
- **Verdict:** A broad, methodically transparent test finds little OA-journal citation advantage, constrained by retrospective access labels and unmodeled journal-level differences.
- **Framework version:** v5.9
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-09-28
- **Evaluation page:** https://www.nabu.science/paper/10.1007%2Fs11192-020-03734-9

## Key claims

1. OA journal articles experience a citation advantage in very few subject areas and, in most of these subject areas, the citation advantage was found on only a single measure of citation advantage, namely whether the article was cited at all.
   - Support: Well-supported
   - Type: Primary finding
   - Basis: The complete subject-area analysis identifies six qualifying fields, four showing an advantage only for citedness.
   - Location: Abstract; Results, pp. 469–470; Tables 1–2
2. Access status accounts for little of the variability in the number of citations an article accumulates.
   - Support: Supported
   - Type: Secondary
   - Basis: Bivariate effect sizes are generally small, though retrospective labels and unmodeled correlates limit interpretation.
   - Location: Abstract; Discussion, pp. 471–472
3. OA journal articles are more likely than subscription journal articles to be written in a language other than English (φ = 0.16).
   - Support: Well-supported
   - Type: Secondary
   - Basis: A direct full-population comparison reports φ = 0.16 and motivates the English-only restriction.
   - Location: Results, Analysis of language as a potentially confounding variable, pp. 467–468

## Strengths

- **Population-wide subject-area coverage and reporting** The analysis covers approximately 3.67 million English-language records across 250 WoS subject areas and reports the complete subject-area results rather than only favorable findings.
  - Dimension: contribution
  - Location: Results, pp. 468–470; Table 2
- **Multiple explicitly defined citation indicators** NCS, citedness, and top-percentile measures use stated citation windows, self-citation rules, coefficients, and effect-size thresholds. This reduces dependence on mean citation counts alone.
  - Dimension: methodological rigour
  - Location: Methodology, pp. 463–465
- **Limitations integrated with interpretation** The discussion traces retroactive OA labeling and the Gold DOAJ restriction to the scope of the conclusions and converts them into specific recommendations for future studies.
  - Dimension: positioning
  - Location: Discussion, pp. 470–472

## Limitations

- **Retrospective labels create exposure uncertainty** An article classified as Gold DOAJ in the 2018 metadata may not have been open at publication or during citation accumulation. Restricting publication years reduces but does not remove this uncertainty.
  - Dimension: methodological rigour
  - Location: Publication Year, pp. 466–467; Discussion, pp. 470–472
- **Important citation correlates remain unmodeled** Language is addressed, but journal prestige, publication and review policies, collaboration, and other author or article characteristics are deferred to future work. The findings therefore characterize association rather than a causal OA effect.
  - Dimension: methodological rigour
  - Location: Discussion, pp. 471–472
- **Exact Art coefficient is inconsistent** The Results report an NCS coefficient of −0.10 for Art, while Table 1 reports −0.12. The difference does not change its direction or threshold classification but should be corrected.
  - Dimension: reporting
  - Location: Results, p. 469; Table 1, p. 470

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. A minor inconsistency affects the exact NCS coefficient reported for Art, although its direction and substantive interpretation are unchanged.
- **Research conduct:** Clean. The study uses bibliometric metadata and therefore does not require human-participant ethics approval; funding and absence of conflicts are disclosed. No suspicious data, code, or availability claim is present.
- **Reference integrity:** Clean. 34 of 34 checkable references verified
  - 41 references in manuscript
  - 7 have no canonical index record — counted, but not index-checkable
  - citation details diverge from the cited records for 1 reference
  - 1 reference confirmed by manual review
  - Citation details diverge for 1 reference:
    - #11 Cronin, B. (1984). The citation process. London: Taylor Graham. https://doi.org/10.1016/0306-4573(85)90080-9. (Differs from the cited DOI’s record on: title, authors)
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
