Assembling the evidence…
Taken together these figures suggest that, in astronomy, there is a strong EA effect and a strong SB effect; there is also no indication of any OA effect.
multiple analyses support the direction, though concurrent temporal changes limit clean causal separation
we can thus reject the hypothesis that the citation distribution of ArXiv submitted vs. non-ArXiv submitted articles is caused by...the combined EA+OA effect alone at the 99.9% confidence level
the reported Monte Carlo null supports rejection conditional on its measured-ratio assumptions
Figure 1 shows clearly that increasing access to existing articles does not increase the probability that they will be cited.
the flat aggregate trend coincides with multiple unseparated changes in access and citation practice
This can be understood as changing the mean effective publication date by about five or six months.
the estimate follows the plotted lag comparison but lacks quantified uncertainty
Derived from the full evaluation — not a separate score.
Strengths
The Introduction distinguishes OA, EA, and SB rather than treating online availability as a single mechanism, and §§3–4 assign separate empirical tests to those explanations.
↳ Introduction §1; Technique §§3.1–3.2
The paper reports the 2,592-article ApJ sample, the 1,909/683 ArXiv split, the 2.11 citation ratio, and the steps used to generate the Figure 3 null distribution.
↳ Data §2.2; Technique §3.2; Results §4.2
The Analysis limits the interpretation to astronomy and related well-funded, high-barrier disciplines, while stating that the broader desirability of open access should not be decided by this result.
↳ Analysis §5
Limitations
ADS digitisation, ArXiv growth, online journals, and evolving scholarly practices all occurred during the 1990s comparison period. The temporal trends therefore do not cleanly identify access as the source of observed changes.
↳ Introduction §1; Technique §3.1
Figures 1–2 are interpreted from visual changes in level or slope without reported confidence intervals or formal trend tests, and the number of Monte Carlo repetitions is unspecified.
↳ Technique §§3.1–3.2; Results §4.1; Figures 1–2
The inference that no significant population of publishing astronomers lacks sufficient access is drawn from aggregate citation trends rather than direct measurement of researchers' access.
↳ Analysis §5
The paper's main contribution is its explicit separation of OA, EA, and SB and its use of different analyses for the temporal and selection questions. The Figure 3 simulation is transparently tied to the observed sample composition and citation ratio, although its result remains conditional on the null construction. Figures 1–2 are less conclusive because several access and publishing changes occurred over the same period and no uncertainty estimates accompany the plotted trends. The discussion appropriately limits much of the interpretation to astronomy, but it does not treat those concurrent changes as a central limitation on causal inference.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.3
Confidence mediumThe OA, EA, and SB decomposition meaningfully advances prior aggregate comparisons of online and offline citation rates. Its contribution remains bounded to astronomy, and the proposed explanation for the absent OA signal is interpretive rather than directly tested.
“two experiments designed to distinguish to what extent each of the postulated explanations holds true”
The paper defines a publication-volume-normalised citation statistic and constructs the SB simulation from measured sample parameters. The temporal OA/EA comparison coincides with several changes in access and publishing practice, while Figures 1–2 lack quantified uncertainty and §3.2 reports only that simulation was repeated “many times.”
“We repeat this procedure many times, to acquire the statistical distribution.”
The progression from three postulates through two tests to the synthesis in §5 is readily traceable, and the principal conclusion is scoped to astronomy. Statements that increased access does not increase citation probability and implies sufficient individual access are more decisive than the aggregate temporal design supports.
“Taken together these figures suggest that, in astronomy, there is a strong EA effect and a strong SB effect”
The paper engages Lawrence and Brody directly and limits its broader interpretation to well-funded, high-barrier disciplines. However, it does not trace the concurrent development of ADS, ArXiv, online journals, and changing scholarly practice through as a central limitation on the access-effect inference.
“four main developments stand out”
Caveats4 of 4 checks
The sample counts, citation ratio, concordance figures, and simulation results are internally consistent across the Data, Results, and Analysis sections.
The bibliometric design does not require human-participant ethics approval, and the named ADS and ArXiv sources are appropriate for the study period. A potential institutional interest is not explicitly disclosed because the author group is associated with ADS while analysing data and access changes involving that system.
Flags: 0 declared / 5 total
11 of 19 checkable references verified
22 references in manuscript 3 are books, websites or datasets — counted, but not index-checkable
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The study analyses real ADS, ArXiv, journal, and citation records, giving it practical evidentiary relevance. It does not test an implemented policy or intervention, so the results are not directly deployment-ready.
“we use data concerning the 2592 articles published in 2003”
AI-generated, human-governed. Something look off? Contact us to request a review.