Loading evaluation data…
Abstract Existent scientific literature diverges on the issue of open-access citation advantage, as available studies often employ different methods and were carried out at different periods. To reconcile previous divergent results, we ran several analysis concerning the proportion between open-access and non-open-access articles, and the citation advantage in Elsevier’s Hybrid journals which were active in the years Sci-Hub was being updated (from 2011 to dec/2021) and after it ceased updating (from jan/2022 to 2025). We relied on databases downloaded from the Directory of Open Journals (DOAJ) and Web of Science (WoS), including 233 journals and nearly 298.000 articles published from 2011 to 2025. The statistical tools employed consisted of linear univariate regression for trend analysis and the Wilcoxon rank sum for evaluating differences. We conclude the number of open-access articles per journal has been increasing steadily in this period and that 97% of the articles in our sample from 2011 to 2021 labelled as paywalled are available at Sci-Hub. Concerning citation advantage of open-access articles, we found no explanation for why some journals have a citation advantage in certain years, because they could not be grouped by JIF (Journal Impact Factor) percentile nor by journal size, so the advantage appears to be random. Additionally, when the advantage was significant, the effect size was small which, in our opinion, suffices to declare that we found no traces of a citation advantage for Hybrid Gold articles.
we found no traces of a citation advantage for Hybrid Gold articles
heterogeneous journal results, unresolved selection, and immature post-freeze citations make the categorical wording stronger than the analysis
97% of the articles in our sample from 2011 to 2021 labelled as paywalled are available at Sci-Hub
direct DOI matching supports the sample-specific percentage, though the Sci-Hub list came from a single intermediary
the number of open-access articles per journal has been increasing steadily in this period
journal-level trend models show predominantly positive coefficients across the sampled period
we found no explanation for why some journals have a citation advantage in certain years, because they could not be grouped by JIF percentile nor by journal size
only a limited set of candidate explanations was examined, leaving selection, funder policies, and other factors unresolved
paying for open-access publication is not a reasonable action for authors if their sole aim with such is a hope to achieve a higher citation count
the observational comparison does not identify the causal citation return from paying an APC
Derived from the full evaluation — not a separate score.
Strengths
The study analyzes approximately 298,000 articles from 233 journals and compares citation ranks within journals and publication years, reducing field and citation-window heterogeneity.
↳ Methods, “Filtering out years with an unbalanced proportion”; Results, “Are open-access articles’ citation counts higher?”
The DOI merge reports that 157,417 of 162,310 sampled articles from 2011–2021 were available through Sci-Hub, directly addressing a source of practical access overlooked in many earlier comparisons.
↳ Methods, “Identifying articles available at sci-hub”; Results, Figure 2
The review discusses systematic reviews and empirical studies reporting both positive and null citation advantages, including prior work that explicitly considered Sci-Hub.
↳ Literature on citation advantage of open-access articles
Limitations
The article recommends that authors prefer non-open-access publication, although OA selection was not randomized and the paper acknowledges that selection bias was not controlled. The analysis supports descriptive associations, not the causal return from paying an APC.
↳ Results, paragraphs following Figure 4; Discussion, opening paragraphs
The paper states that the analysis may have been conducted too soon and proposes rerunning it in 2030, yet still draws a firm conclusion about the limited effect of Sci-Hub’s freeze.
↳ Discussion, “Another possible conclusion” paragraph
The analysis covers selected Elsevier hybrid journals, excludes journal-years outside the 10%–90% OA range, and has high mean JIF percentiles. Transfer to other publishers, lower-JIF journals, or particular disciplines is not demonstrated.
↳ Methods, “Journal selection procedures” and filtering section; Discussion; Limitations
The study’s scale, within-journal/year comparisons, and rank-based analysis provide useful descriptive evidence on hybrid OA citation patterns. Figure 2 adds a concrete Sci-Hub availability result, while Figures 3–4 show substantial heterogeneity rather than a stable citation advantage. The main score constraint is the move from those observational patterns to advice that authors should avoid APCs despite acknowledged selection bias. The conclusion about Sci-Hub’s limited effect is also premature relative to the Discussion’s statement that post-2021 citation windows may not yet be mature.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Limited2.8
Confidence mediumThe study adds a large within-journal/year analysis and a pre/post-2021 Sci-Hub comparison to an already contested literature. Its advance remains incremental because it does not identify what drives the heterogeneous journal-level results.
“no actual overarching pattern emerged from our findings”
Within-journal/year stratification, rank-based testing, and effect-size reporting fit skewed citation data, but article-level selection remains unresolved. The observational analysis cannot sustain the recommendation that choosing paid OA will not improve citations.
“The only confounding factor we did not try to control was ‘selection bias’”
The question, data construction, analysis, and findings are presented in a followable sequence. Precision falls where heterogeneous associations become the practical claim that authors should avoid APCs.
“authors should prefer to publish as non-open-access”
The article discusses positive, null, and Sci-Hub-sensitive findings and identifies several methodological differences. Its acknowledged selection and citation-maturity limitations are not fully integrated into the strength of the final recommendation.
“this study was conducted much too soon”
Caveats4 of 4 checks
The reported sample sizes and analytical sequence are broadly coherent, but several disclosed design and measurement uncertainties temper confidence in the conclusions. None establishes fabrication or an internal contradiction severe enough to invalidate the reported analysis.
The bibliometric design does not visibly require human-subject ethics review, and no image-integrity concern is apparent. Transparency is limited by reliance on a privately obtained Sci-Hub DOI dataset and the absence of explicit code, processed-data, conflict-of-interest, and ethics-status statements.
Flags: 0 declared / 5 total
23 of 23 checkable references verified
35 references in manuscript 12 are books, websites or datasets — counted, but not index-checkable
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The analysis uses real journal and citation records across many years, placing it beyond a purely conceptual observation. Readiness is limited because no validated decision framework or context-specific implementation guidance is provided.
“paying for open-access publication is not a reasonable action”
AI-generated, human-governed. Something look off? Contact us to request a review.