nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-807F6B3D·Framework v5.9·Pipeline v0.4·Evaluated 01 Oct 2026·OECD: 5.8 Other social sciences·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Cross-sectional study

The Sci-Hub effect on papers’ citations

Juan C. Correa et al.Juan C. Correa, Henry Laverde-Rojas, Julián Tejada et al.·2021·Scientometrics·10.1007/s11192-020-03806-w ↗

Primary claim

articles downloaded from Sci-Hub were cited 1.72 times more than papers not downloaded from Sci-Hub

Tentative · the coefficient is framed multiplicatively and causally despite unresolved demand confounding in an observational design
In brief

A novel, broad-sample association between Sci-Hub downloads and citations is undermined by unresolved confounding and causal interpretation.

ReliabilityConcerns4 of 4 checks completeRead the entry ↓
Fragile — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract not available — view at DOI ↗

Key Claims

Primary finding · Tentative

articles downloaded from Sci-Hub were cited 1.72 times more than papers not downloaded from Sci-Hub

Fragile — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

the coefficient is framed multiplicatively and causally despite unresolved demand confounding in an observational design

Abstract; Results, Appendix Table 9
Quote verified
Secondary · Supported

the number of downloads from Sci-Hub was a robust predictor of future citations

the positive association persists across specifications, although citation timing and causal direction remain incompletely established

Abstract; Results, Appendix Tables 2–11
Quote verified
Secondary · Supported

the number of figures in a manuscript consistently predicts its future citations

figure count remains associated with citations across reported models, though automated extraction was not independently validated

Abstract; Results, Appendix Tables 7–11
Quote verified
Secondary · Tentative

limited access to publications may limit some scientific research from achieving its full impact

the access interpretation requires a causal pathway that the selected observational comparison cannot identify

Abstract
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Novel Sci-Hub citation association

The study moves beyond descriptive accounts of Sci-Hub use by estimating its association with citations across a sizeable article sample.

↳ Introduction, Related works; Materials and method, Data acquisition

#methodological rigour

Extensive specification and diagnostic checks

The paper reports OLS, robust regression, heteroscedasticity-consistent estimates, generated-instrument models, GAMLSS, outlier diagnostics, and blockwise sensitivity analyses.

↳ Statistical analyses; Appendix, Tables 2–11 and Figs. 3–8

#methodological rigour

Transparent data and code declaration

The datasets and analytical code are declared available through a named OSF repository, supporting reproducibility and follow-up work.

↳ Data Availability statement

Limitations

#methodological rigour

Causal identification remains unresolved

The observational comparison does not rule out that already popular or highly cited papers attract more Sci-Hub downloads. Generated instruments based on heteroscedasticity do not provide a clear exogenous source of download variation, and two specifications fail the Hansen test.

↳ Introduction, final paragraphs; Appendix, Tables 6 and 9

#methodological rigour

Impossible descriptive statistics remain unexplained

Appendix Table 1 reports a mean of 1488 for Tablessup despite a maximum of 39, as well as implausible maxima for pages and authors. Because these characteristics enter adjusted models, the unresolved errors create a material reliability concern.

↳ Appendix, Table 1

#reporting

Headline language overstates the evidence

The abstract presents a regression coefficient as “1.72 times more,” while the Methods and Discussion label the design quasi-experimental and say Sci-Hub almost doubles citations. These formulations imply multiplicative and causal conclusions not established by the design.

↳ Abstract; Materials and method, Data acquisition; Discussion, final paragraph

Does this match your read? Add your assessment ↓

Nabu Commentary

The paper makes a genuine contribution by estimating a previously underexplored Sci-Hub download and citation association across 12 journals. Its multiverse-style analyses consistently recover a positive coefficient, but the observational comparison and heteroscedasticity-generated instruments do not resolve whether pre-existing article demand drives both downloads and citations. The causal framing therefore carries more weight than the design can support, particularly in the Abstract and Discussion. Appendix Table 1 further lowers confidence because several descriptive values are impossible or implausible and their relationship to the analytical data is unexplained.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
2.7±0.1

Contribution
3.6

The study supplies an early empirical estimate of the association between Sci-Hub downloads and citations across 12 journals and four disciplinary groupings. This meaningfully extends descriptive Sci-Hub research, although it does not identify the causal effect of access.

Introduction, Related works; Materials and method, Data acquisition

“Sci-Hub downloads’ role as a relevant predictor of citations remains neglected.”

Methodological Rigour
2.0

The analysis uses multiple estimators, diagnostics, journal and discipline controls, and sensitivity specifications, but generated instruments do not resolve the principal demand and reverse-causality problem. Methodological Rigour is also affected by unresolved impossible descriptive statistics for analytical covariates.

Statistical analyses; Appendix, Tables 1 and 7–9

“We were able to verify the validity of the instruments, except for models 1 and 3.”

Reporting
3.2

The paper follows a clear progression from motivation and data construction to diagnostics, robustness analyses, and implications. However, “1.72 times more,” “quasi-experimental,” and “almost doubles” communicate a multiplicative or causal conclusion that the reported coefficient and observational design do not establish.

Abstract; Materials and method, Data acquisition; Discussion

“Sci-Hub almost doubles the citations of articles accessed through it.”

Positioning
2.8

The paper engages relevant Sci-Hub usage and citation-prediction literature and explicitly identifies reverse causality as an alternative explanation. It then dismisses that alternative as no formal impediment and proceeds to causal-strength conclusions without tracing the limitation through.

Introduction, final paragraphs; Discussion, paragraphs 1 and 4

“the most popular and most cited articles are more downloaded from Sci-Hub because of their previous citations”

Lower confidence on Contribution, Methodological Rigour — domain match limited.

Reliability

Concerns4 of 4 checks


Internal CoherenceConcern

Appendix Table 1 contains arithmetically impossible and highly implausible descriptive statistics for covariates used in the analyses. The paper does not establish whether these are presentation errors or errors in the analytical data.

Research ConductClean

The study uses bibliometric data for which human-subject ethics approval is not expected and declares public repositories for its data and code. No conduct concern grounded in the supplied text was identified.

Flags: 2 declared / 5 total

Reference IntegrityClean

57 of 57 checkable references verified

59 references in manuscript 2 have no canonical index record — counted, but not index-checkable 3 references confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 9
Type of advanceEmpirical
Route to useNamed
AdvanceExtends
CitationsFWCI 4.7
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessFoundational
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The evidence is a single observational quasi-experiment on real citation records across twelve journals, with no independent confirmation of the claimed effect.

From the paper
“Researchers could exploit this finding as a mechanism to maximize the impact of their research.”

The discussion names researchers as actors who could use the findings (e.g., including figures) to increase their research impact, but does not specify a concrete decision setting.

Pathway

Researchers, journals and research evaluators could use Sci-Hub download counts as early indicators of article impact, guiding publication and evaluation practice.

Barrier

The quasi-experimental observational design cannot fully rule out reverse causality, and Sci-Hub data carries legal and ethical concerns that would deter institutional adoption.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Suspected distortion of citations in high-impact cancer journals

    Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications

    Primary claim

    These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.

    Tentative
    Evidence Score
    3.5±0.1
    Reliability
    Caveats
  2. The citation trap: Papers published at year-end receive systematically fewer citations

    Chao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics

    Primary claim

    papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Quantitative Analysis of the Open Access Citation Advantage

    Masashi Kawai, Koichi Ojiro, Makoto Asaoka, et al. · 2022 · Other social sciences

    Primary claim

    日本の機関リポジトリによるオープンアクセス論文の数は限られるものの,その被引用数と様々なタイプのオープンアクセス論文の被引用数の比較からは,日本の機関リポジトリによるオープンアクセスのメリットは確認できなかった.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats

How the rubric works →