nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-337B3338·Framework v5.9·Pipeline v0.4·Evaluated 28 Sept 2026·OECD: 5.8 Economics·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Cross-sectional study

The Hybrid Open Access Citation Advantage: How Many More Cites is a $3,000 Fee Buying You?

Frank Mueller-Langer et al.Frank Mueller-Langer, Richard Watt·2014·SSRN Electronic Journal·10.2139/ssrn.2391692 ↗

Primary claim

Once we additionally control for institution quality and citations to RePEc pre-prints, the marginal HOA citation advantage turns out to insignificant and drops to 0.4%.

Supported · the reported estimate is consistent across models but depends on a heavily reduced complete-case sample
In brief

A distinctive pilot-agreement study shows hybrid-access citation estimates collapsing under richer controls, but its decisive null rests on a heavily reduced sample.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Fragile — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

We study the hybrid open access (HOA) citation effect. Under HOA pilot agreements, HOA is assigned for all articles of eligible authors. We use unique data on 208 (1,121) HOA (closed access) economics articles. We control for the quality of journals, articles and institutions and citations to RePec pre-prints. Performing Poisson quasi-maximum likelihood regressions, HOA turns out to be a significant predictor of citations with marginal effects ranging between 22% and 26%. However, once we additionally control for institution quality and citations to RePec pre-prints, the marginal HOA citation advantage turns out to be insignificant and drops to 0.4%.

Key Claims

Primary finding · Supported

Once we additionally control for institution quality and citations to RePEc pre-prints, the marginal HOA citation advantage turns out to insignificant and drops to 0.4%.

Fragile — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

the reported estimate is consistent across models but depends on a heavily reduced complete-case sample

Abstract; Table 4, specification 7
Quote verified
Secondary · Tentative

Our empirical evidence therefore suggests that HOA does not constitute a significant advantage in the competition for citations.

the observational affiliation-based comparison and substantial complete-case attrition limit inference from the null estimate

Conclusion, §4
Quote verified
Secondary · Tentative

HOA turns out to be a significant predictor of citations with marginal effects ranging between 22% and 26%.

the estimate comes from specifications later attenuated by institution and preprint controls

Abstract; Tables 3–4, specifications 1–5
Quote verified
Secondary · Supported

the coefficient for citations to OA pre-prints on RePec is positive and significant at the 1% level

the conditional association is directly reported in specification 7, though based on the reduced subsample

§3.1, p. 15; Table 3, specification 7
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Distinctive pilot-agreement research setting

The analysis uses institutional pilot agreements under which eligible articles were generally made openly available without authors paying from their own research budgets, reducing the author-choice selection problem motivating prior critiques.

↳ Introduction, pp. 2–4; Analysis, pp. 7–8

#methodological rigour

Transparent progression across model specifications

Tables 3–6 show how the positive HOA coefficient attenuates after institution quality and RePEc preprint citations are added. Reporting both PQML and negative-binomial results makes this sensitivity visible.

↳ Tables 3–6; §§3.1–3.2

#application pathway

Concrete policy and payment question

The paper ties citation outcomes to an approximately $3,000 hybrid-open-access fee and identifies authors, universities, funders, and publishers as stakeholders in the underlying decision.

↳ Title; Introduction, pp. 1–3

Limitations

#methodological rigour

Decisive model uses reduced sample

The PQML analysis falls from 1,295 observations in specification 1 to 246 in specification 7, while the negative-binomial analysis retains 249. The paper does not assess whether this complete-case subsample is representative.

↳ Tables 2, 3, and 5

#methodological rigour

Residual institutional confounding remains

Pilot eligibility is linked to institutional affiliation rather than randomized assignment. Institution rank and author-quality controls address part of this concern but do not establish comparability on other institution-, author-, or article-level citation determinants.

↳ Introduction, pp. 2–3; Analysis, pp. 7–12

#positioning

Limitations not traced into conclusion

The conclusion presents the fully adjusted null as evidence of no significant HOA advantage without discussing the severe sample reduction or narrowing the inference around residual confounding.

↳ Tables 3 and 5; Conclusion, §4

Does this match your read? Add your assessment ↓

Nabu Commentary

The pilot-agreement setting and economics-specific dataset provide a meaningful contribution because they address author-choice selection more directly than ordinary voluntary-access comparisons. Tables 3–6 transparently show that the estimated HOA association weakens after institution quality and RePEc preprint citations are introduced, and the two count-model families yield similar qualitative patterns. The decisive specification, however, retains only 246–249 observations from the original 1,329 and the paper does not examine whether that complete-case sample is selected. Strong engagement with the open-access literature is therefore offset by incomplete discussion of the limitations governing the central null conclusion.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.4±0.1

Contribution
3.7

The paper contributes an economics-specific test of hybrid-open-access citation differences using pilot agreements intended to reduce author-choice selection. The design and hand-collected multi-journal data meaningfully extend the cited literature, although they do not decisively identify a causal fee return.

Introduction, pp. 2–4; Data, §2; Analysis, §3

This unique set-up helps us to avoid the author-driven selection bias

Methodological Rigour
3.3

PQML and negative-binomial models, journal indicators, clustered errors, and progressive controls provide a coherent count-data analysis. The preferred null rests on 246–249 complete cases, while selection into that subsample and possible error in manually constructed measures are not assessed.

Data, pp. 5–10; Tables 2–6

Observations 1,295 783 783 695 695 565 246

Reporting
3.8

The paper proceeds transparently from design and data construction through seven progressively adjusted specifications, and both positive and null estimates are reported. The title and recurring effect or fee-return language are stronger than the observational design warrants, although the abstract presents the final null prominently.

Title; Abstract; §3.1–3.2; Conclusion, §4

the marginal HOA citation advantage turns out to insignificant and drops to 0.4%

Positioning
2.8

The introduction engages both positive open-access findings and work attributing those findings to selection or omitted quality. However, the prose does not discuss the representativeness of the 246–249-observation final subsample or fully trace residual institutional confounding into the conclusion.

Introduction, pp. 2–4; Tables 3 and 5; Conclusion, §4

Our empirical evidence therefore suggests that HOA does not constitute a significant advantage

Lower confidence on Methodological Rigour, Positioning — domain match limited.

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The numerical results are coherent across Tables 2–6, but some headline language gives the institution-linked observational coefficients a stronger causal interpretation than the design securely supports.

Research ConductNoted

Funding is acknowledged and the bibliometric analysis raises no evident ethics issue, but the ancillary email survey lacks an explicit ethics or consent statement.

Flags: 0 declared / 5 total

Reference IntegrityClean

17 of 17 checkable references verified

23 references in manuscript 6 have no canonical index record — counted, but not index-checkable 2 references confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice
Type of advanceEmpirical
Route to useSuggested
AdvanceExtends
CitationsNot available
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim rests on observational regression analysis of 1,329 real published articles across 15 journals, with two independent model specifications (PQML and negative binomial) supporting the same result, though not an independent replication.

From the paper
“HOA does not constitute a significant advantage in the competition for citations”

The finding speaks to whether authors and institutions should pay hybrid open access fees, a real-world decision, though no specific actor is named.

Pathway

Authors, universities and funders deciding whether to pay hybrid open access fees could use this evidence to guide publication-fee and open-access mandate decisions.

Barrier

The evidence is observational, limited to economics journals and a small subsample in the key specification, so broader replication across disciplines is needed before confident application.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Suspected distortion of citations in high-impact cancer journals

    Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications

    Primary claim

    These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.

    Tentative
    Evidence Score
    3.5±0.1
    Reliability
    Caveats
  2. The citation trap: Papers published at year-end receive systematically fewer citations

    Chao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics

    Primary claim

    papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Quantitative Analysis of the Open Access Citation Advantage

    Masashi Kawai, Koichi Ojiro, Makoto Asaoka, et al. · 2022 · Other social sciences

    Primary claim

    日本の機関リポジトリによるオープンアクセス論文の数は限られるものの,その被引用数と様々なタイプのオープンアクセス論文の被引用数の比較からは,日本の機関リポジトリによるオープンアクセスのメリットは確認できなかった.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats

How the rubric works →