Crunching the numbers. Responsibly.
We demonstrate conclusively that there is no "Open Access Advantage" for papers from the Astrophysical Journal. The two to one citation advantage enjoyed by papers deposited in the arXiv e-print server is due entirely to the nature and timing of the deposited papers. This may have implications for other disciplines.
We demonstrate conclusively that there is no “Open Access Advantage” for papers from the Astrophysical Journal.
publication year and access status vary together, while the null comparison lacks formal inference
The two to one citation advantage enjoyed by papers deposited in the arXiv e-print server is due entirely to the nature and timing of the deposited papers.
the design addresses self-archiving bias but does not establish an exhaustive causal decomposition
There is no significant difference in the citation histories of the two sets, in particular there is no significant difference for the first 36 months following publication.
statistical significance is asserted from visual inspection without a reported test or uncertainty interval
Derived from the full evaluation — not a separate score.
Strengths
The study uses ApJ’s January 1998 subscription change to compare citation histories under open and closed access while restricting both groups to papers not deposited in arXiv.
↳ Introduction, final two paragraphs; Sections 2–3
Both focal cohorts consist of authors who did not self-archive, directly addressing the selection mechanism that complicates ordinary arXiv versus non-arXiv comparisons.
↳ Section 3, paragraph describing the bottom two curves
The Introduction presents OA, early-access, and quality-bias explanations and discusses work supporting both sides, including the Brody et al. response.
↳ Introduction
Limitations
The open cohort is from 1997 and the closed cohort from 1998, but the paper does not test or adjust for changes in article mix, authorship, subfields, or citation practices between years.
↳ Sections 2–3
Section 3 declares no significant difference based on inspection of the citation curves without reporting a test, effect estimate, confidence interval, or equivalence margin.
↳ Section 3, paragraph beginning “As is clear by inspection”
The Abstract and Discussion state that the result is conclusive even though the analysis is an unadjusted cross-year visual comparison. This materially overstates the certainty of the central finding.
↳ Abstract; Section 3; Discussion
The paper’s main contribution is the use of ApJ’s 1998 subscription change to compare non-arXiv papers under different access regimes. Holding self-archiving status constant improves on a direct arXiv versus non-arXiv comparison, but the focal groups also come from different publication years and their comparability is not examined. Section 3 relies on visual overlap while using statistical and causal language that would ordinarily require effect estimates, uncertainty intervals, and checks for secular differences. The literature framing is substantive, but the Discussion does not carry the year-confounding limitation into the certainty or scope of its conclusion.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Limited2.8
Confidence highThe paper introduces a distinct access-policy contrast intended to separate open access from author self-archiving, meaningfully extending the cited OA-citation debate. Its contribution remains bounded by a single journal and two adjacent cohorts.
“We have found a dataset which does not have this bias”
The comparison holds non-arXiv status constant and establishes that the access regime preceded citation accumulation. However, access status is perfectly aligned with publication year, cohort comparability is not tested, and the null result is based on visual inspection without reported inference.
“As is clear by inspection of the figure”
The Introduction-to-Discussion sequence makes the comparison and inferential chain easy to follow. The terms “conclusively” and “no significant difference” communicate greater certainty than the visual comparison and unreported statistical procedure support.
“We demonstrate conclusively”
The Introduction presents both OA-advantage arguments and alternative early-access and quality-bias explanations, including a recent rebuttal. The Discussion does not acknowledge that publication year and access regime are inseparable in the central comparison.
“reasserting their claims that OA is the dominant effect”
Lower confidence on Methodological Rigour, Positioning — domain match limited.
Caveats4 of 4 checks
The central null comparison is described as statistically non-significant, but the paper reports no inferential procedure or numerical uncertainty supporting that wording. This limits verification of the headline conclusion without creating an internal contradiction.
The study uses aggregate bibliometric records from a named database, and no design-specific ethics requirement or visible conduct inconsistency is identified. Ethics, conflict, registration, data-availability, and code statements are not provided, but their absence does not establish misconduct for this study type and era.
Flags: 0 declared / 5 total
1 of 12 checkable references verified
12 references in manuscript
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The analysis uses real citation behaviour surrounding an actual journal access-policy change, giving it direct policy relevance. Readiness is limited by the visual-only comparison, absent uncertainty estimates, and need for corroboration.
“One was erected on January 1, 1998.”
AI-generated, human-governed. Something look off? Contact us to request a review.