Assembling the evidence…
Once we additionally control for institution quality and citations to RePEc pre-prints, the marginal HOA citation advantage turns out to insignificant and drops to 0.4%.
the reported estimate is consistent across models but depends on a heavily reduced complete-case sample
Our empirical evidence therefore suggests that HOA does not constitute a significant advantage in the competition for citations.
the observational affiliation-based comparison and substantial complete-case attrition limit inference from the null estimate
HOA turns out to be a significant predictor of citations with marginal effects ranging between 22% and 26%.
the estimate comes from specifications later attenuated by institution and preprint controls
the coefficient for citations to OA pre-prints on RePec is positive and significant at the 1% level
the conditional association is directly reported in specification 7, though based on the reduced subsample
Derived from the full evaluation — not a separate score.
Strengths
The analysis uses institutional pilot agreements under which eligible articles were generally made openly available without authors paying from their own research budgets, reducing the author-choice selection problem motivating prior critiques.
↳ Introduction, pp. 2–4; Analysis, pp. 7–8
Tables 3–6 show how the positive HOA coefficient attenuates after institution quality and RePEc preprint citations are added. Reporting both PQML and negative-binomial results makes this sensitivity visible.
↳ Tables 3–6; §§3.1–3.2
The paper ties citation outcomes to an approximately $3,000 hybrid-open-access fee and identifies authors, universities, funders, and publishers as stakeholders in the underlying decision.
↳ Title; Introduction, pp. 1–3
Limitations
The PQML analysis falls from 1,295 observations in specification 1 to 246 in specification 7, while the negative-binomial analysis retains 249. The paper does not assess whether this complete-case subsample is representative.
↳ Tables 2, 3, and 5
Pilot eligibility is linked to institutional affiliation rather than randomized assignment. Institution rank and author-quality controls address part of this concern but do not establish comparability on other institution-, author-, or article-level citation determinants.
↳ Introduction, pp. 2–3; Analysis, pp. 7–12
The conclusion presents the fully adjusted null as evidence of no significant HOA advantage without discussing the severe sample reduction or narrowing the inference around residual confounding.
↳ Tables 3 and 5; Conclusion, §4
The pilot-agreement setting and economics-specific dataset provide a meaningful contribution because they address author-choice selection more directly than ordinary voluntary-access comparisons. Tables 3–6 transparently show that the estimated HOA association weakens after institution quality and RePEc preprint citations are introduced, and the two count-model families yield similar qualitative patterns. The decisive specification, however, retains only 246–249 observations from the original 1,329 and the paper does not examine whether that complete-case sample is selected. Strong engagement with the open-access literature is therefore offset by incomplete discussion of the limitations governing the central null conclusion.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.4
Confidence highThe paper contributes an economics-specific test of hybrid-open-access citation differences using pilot agreements intended to reduce author-choice selection. The design and hand-collected multi-journal data meaningfully extend the cited literature, although they do not decisively identify a causal fee return.
This unique set-up helps us to avoid the author-driven selection bias
PQML and negative-binomial models, journal indicators, clustered errors, and progressive controls provide a coherent count-data analysis. The preferred null rests on 246–249 complete cases, while selection into that subsample and possible error in manually constructed measures are not assessed.
Observations 1,295 783 783 695 695 565 246
The paper proceeds transparently from design and data construction through seven progressively adjusted specifications, and both positive and null estimates are reported. The title and recurring effect or fee-return language are stronger than the observational design warrants, although the abstract presents the final null prominently.
the marginal HOA citation advantage turns out to insignificant and drops to 0.4%
The introduction engages both positive open-access findings and work attributing those findings to selection or omitted quality. However, the prose does not discuss the representativeness of the 246–249-observation final subsample or fully trace residual institutional confounding into the conclusion.
Our empirical evidence therefore suggests that HOA does not constitute a significant advantage
Lower confidence on Methodological Rigour, Positioning — domain match limited.
Caveats4 of 4 checks
The numerical results are coherent across Tables 2–6, but some headline language gives the institution-linked observational coefficients a stronger causal interpretation than the design securely supports.
Funding is acknowledged and the bibliometric analysis raises no evident ethics issue, but the ancillary email survey lacks an explicit ethics or consent statement.
Flags: 0 declared / 5 total
16 of 18 checkable references verified
23 references in manuscript 5 are books, websites or datasets — counted, but not index-checkable
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The analysis uses actual publication and citation data from operating pilot agreements, so it can directly inform deliberation. Observational uncertainty and the small fully adjusted sample prevent the findings from independently determining policy.
the statistically insignificant marginal effect of HOA on net citations drops to 0.4%
AI-generated, human-governed. Something look off? Contact us to request a review.