# The open access citation premium may depend on the openness and inclusiveness of the indexing database, but the relationship is controversial because it is ambiguous where the open access boundary lies

- **DOI:** [10.1007/s11192-019-03221-w](https://doi.org/10.1007/s11192-019-03221-w)
- **Rating:** Sound (3.1 out of 5)
- **Verdict:** A focused bibliometric test reveals database-dependent OA citation patterns, but non-random case selection prevents broad inference.
- **Framework version:** v5.9
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-09-28
- **Evaluation page:** https://www.nabu.science/paper/10.1007%2Fs11192-019-03221-w

## Key claims

1. The relationship takes place in two out of the three analyzed case studies, while it is essentially absent in the other case study.
   - Support: Supported
   - Type: Primary finding
   - Basis: the descriptive pattern appears across three cases, though purposive selection limits inference beyond them
   - Location: Conclusions
2. Statistically significant evidence can be achieved only by setting low to medium citation thresholds, thus the relationship fades away when looking at the documents that accrue many cites.
   - Support: Supported
   - Type: Secondary
   - Basis: the threshold tables show this pattern, though shrinking samples and repeated testing temper certainty
   - Location: Conclusions
3. While focusing on the Gold OA route leads to inconclusive evidence, the joint consideration of Gold and Green OA provides some confirmation of the tested hypothesis.
   - Support: Tentative
   - Type: Secondary
   - Basis: the three-case design is limited and confirmation relies on multiple unadjusted threshold-specific tests
   - Location: Conclusions

## Strengths

- **Specific and identifiable research question** The paper distinguishes database dependence from the broader OA citation-advantage debate and operationalises it through the Google Scholar-to-Scopus citation ratio.
  - Dimension: contribution
  - Location: Abstract; Methodological setting
- **Transparent reporting across all thresholds** Search queries, matching procedures, OA classification, and the complete threshold series are reported, including null and conflicting results rather than only favourable comparisons.
  - Dimension: methodological rigour
  - Location: Data gathering process; Appendix Tables 4–9
- **Balanced engagement with contested evidence** The background separately organises research supporting and questioning the OA citation advantage, then connects self-selection, early-view effects, and database quality to the study's interpretation.
  - Dimension: positioning
  - Location: Background studies; Tables 1–2; Conclusions

## Limitations

- **Confounding remains analytically uncontrolled** The OA and non-OA comparisons are not matched or adjusted for known citation drivers such as self-selection, field, article age, or author prominence, limiting interpretation of the observed ratio differences.
  - Dimension: methodological rigour
  - Location: Related issues of the citation advantage: open access definition; Methodological setting
- **Non-random cases restrict generalisation** The analysis covers one journal, one topic in one publication year, and one highly prolific author. The data-gathering section acknowledges that the limited, non-random sample reduces generalisability.
  - Dimension: contribution
  - Location: Case studies; Data gathering process
- **Repeated tests lack multiplicity strategy** Tables 4–9 report many nested threshold-specific p-values without a stated multiplicity approach, reducing confidence in isolated significant thresholds and Gold-versus-Green contrasts.
  - Dimension: methodological rigour
  - Location: Results; Appendix Tables 4–9

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. The reported numerical series is internally coherent and includes all examined thresholds, but inferential emphasis across numerous nested tests lacks a stated multiplicity strategy. This affects the weight placed on threshold-specific significance rather than indicating fabricated or contradictory results.
- **Research conduct:** Clean. The study uses public bibliometric records, describes manual matching and OA classification, and acknowledges database errors and missing raw-data availability. No ethics requirement, availability misrepresentation, or other conduct concern is established.
- **Reference integrity:** Clean. 102 of 102 checkable references verified
  - 113 references in manuscript
  - 11 have no canonical index record — counted, but not index-checkable
  - 9 references confirmed by manual review
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
