Crunching the numbers. Responsibly.
This paper empirically studies the effect of Open Access on journal CiteScores. We have found that the general effect is positive but not uniform across different types of journals. In particular, we investigate two types of heterogeneous treatment effect: (1) the differential treatment effect among journals grouped by academic field, publisher, and tier; and (2) differential treatment effects of Open Access as a function of propensity to be treated. The results are robust to a number of sensitivity checks and falsification tests. Our findings shed new light on Open Access effect on journals and can help stakeholders of journals in the decision of adopting the Open Access policy.
becoming an open access journal will significantly improve the CiteScore of a journal by 0.147 on average
matched DiD estimates and falsification checks support the direction, though causal certainty remains assumption-dependent
the OA effect is more pronounced in lower ranked journals than in high-ranking journals, suggesting a “long tail” effect
rank-stratified estimates support the pattern, but subgroup inference and a single outcome metric limit generality
The average effect of OA on the journals from Big Five publishers is 0.309, whereas the effect on the journals from other publishers is 0.0742
direct subgroup estimates and a reported coefficient-difference test consistently support the comparison
the journals more likely to become OA derive a greater benefit when they actually become OA
cross-sectional propensity stratification depends on selection-on-observables and does not establish the proposed mechanism
Derived from the full evaluation — not a separate score.
Strengths
Treatment journals are matched to controls within minor subject and rank category, then analyzed with journal-by-cohort and cohort-by-year fixed effects. This improves substantially on unadjusted OA/non-OA comparisons.
↳ Data and Descriptive Statistics, paragraph 4; Statistical Methodology, Eq. 2
The paper reports pre-treatment leads, a 2,000-allocation placebo distribution, a propensity-based bias adjustment, and an exclusion of controls with substantial NIH-supported OA content.
↳ Results, Table 4 and Figures 2–3; Robustness Check, Table 10
Results are separated by publisher, discipline, rank, and propensity, directly engaging conflicting long-tail and superstar findings in the prior literature.
↳ Literature Review, final paragraph; Results, Tables 5–9
Limitations
Claims that OA “will significantly improve” CiteScore and that parallel trends are not violated are stronger than nonsignificant leads and matching can establish under the untestable common-shocks assumption.
↳ Results, Open Access Effects; Statistical Methodology, Difference-in-Difference
The paper acknowledges reliance on one citation indicator but explicitly declines to examine its limitations, despite using rank-specific CiteScore changes to support the long-tail conclusion.
↳ Data and Descriptive Statistics, paragraph 2; Results, Table 7
Differentiated OA pricing is proposed as a use for the findings, but no pricing rule, operational test, or adoption evidence is supplied.
↳ Conclusions and Discussion, paragraph 3
The matched-control difference-in-differences design, fixed effects, lead tests, placebo analysis, and sensitivity checks place the methodology within the competently executed band. The main deductions reflect the explicit refusal to trace CiteScore limitations and incomplete treatment of selection into and out of the Scopus frame. The literature review and heterogeneity analyses provide a clear incremental contribution by addressing publisher, discipline, rank, and propensity differences. Causal wording in the Results and Conclusions was treated as an in-band clarity limitation because the design remains dependent on parallel-trends, common-shocks, and unconfoundedness assumptions.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.6
Confidence mediumThe paper moves beyond simple OA/non-OA comparisons through matched controls, difference-in-differences estimation, and heterogeneity analyses by publisher, field, rank, and propensity. The advance is meaningful but incremental because the overall positive OA association largely confirms earlier findings.
“We have found that the general effect is positive but not uniform across different types of journals.”
Fixed effects, pre-treatment leads, a 2,000-allocation placebo exercise, and sensitivity analyses provide a credible assumption-dependent design. The score incorporates the untraced limitations of CiteScore and incomplete treatment of selection into and out of the Scopus sampling frame.
“The insignificance of coefficients for the leads of 1-3 year prior mitigates the concern”
The paper follows a coherent sequence from motivation and design to heterogeneity results and robustness checks. Statements that OA “will significantly improve” CiteScore and that parallel trends are not violated express more certainty than the observational diagnostics establish.
“becoming an open access journal will significantly improve the CiteScore”
The literature review directly engages conflicting long-tail, superstar, and disciplinary findings, while the discussion traces hybrid-OA and Scopus-entry concerns. CiteScore limitations and the common-shocks assumption are acknowledged without being fully carried into the certainty of the conclusions.
“It is beyond the scope of this study to discuss the limitations of CiteScore-like indicators.”
Caveats4 of 4 checks
The numerical results and methods-results correspondence are internally consistent, but the description of direct versus hybrid OA leaves treatment and control composition somewhat ambiguous. The ambiguity is partly mitigated by a later exclusion analysis.
The study uses identifiable journal-level secondary data and presents no human-subjects ethics requirement or contradictory availability claim. No code or replication-data availability statement is provided, but this is a transparency gap rather than a reliability concern.
Flags: 0 declared / 5 total
55 of 55 checkable references verified
60 references in manuscript 5 are books, websites or datasets — counted, but not index-checkable
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The findings come from large-scale observational journal data with several identification checks, making them policy-informative. No pricing model, implementation test, or evidence of use in negotiations is reported.
AI-generated, human-governed. Something look off? Contact us to request a review.