Peer review is the main institution responsible for the evaluation and gestation of scientific research. Although peer review is widely seen as vital to scientific evaluation, anecdotal evidence abounds of gatekeeping mistakes in leading journals, such as rejecting seminal contributions or accepting mediocre submissions. Systematic evidence regarding the effectiveness--or lack thereof--of scientific gatekeeping is scant, largely because access to rejected manuscripts from journals is rarely available. Using a dataset of 1,008 manuscripts submitted to three elite medical journals, we show differences in citation outcomes for articles that received different appraisals from editors and peer reviewers. Among rejected articles, desk-rejected manuscripts, deemed as unworthy of peer review by editors, received fewer citations than those sent for peer review. Among both rejected and accepted articles, manuscripts with lower scores from peer reviewers received relatively fewer citations when they were eventually published. However, hindsight reveals numerous questionable gatekeeping decisions. Of the 808 eventually published articles in our dataset, our three focal journals rejected many highly cited manuscripts, including the 14 most popular; roughly the top 2 percent. Of those 14 articles, 12 were desk-rejected. This finding raises concerns regarding whether peer review is ill--suited to recognize and gestate the most impactful ideas and research. Despite this finding, results show that in our case studies, on the whole, there was value added in peer review. Editors and peer reviewers generally--but not always-made good decisions regarding the identification and promotion of quality in scientific manuscripts.
Strengths
The study leverages a dataset of 1,008 manuscripts submitted to three elite medical journals, including rejected submissions that are almost never accessible, enabling citation comparisons across editorial decisions that published-only data cannot support.
↳ Materials and Methods; Abstract
The result that the 14 most-cited articles in the dataset were rejected by the focal journals, with 12 desk-rejected without peer review, is both provocative and clearly supported by the citation-rank analysis.
↳ Results, Fates of Rejected vs. Accepted Articles; Table 2
The argument is logically structured and consistently frames findings as case studies, describing associations rather than causal effects and openly acknowledging that citations are an imperfect proxy for merit.
↳ Abstract; Discussion
Limitations
The core desk-rejected versus peer-reviewed comparison does not adjust for manuscript quality, author characteristics, or article type, all of which plausibly influence both editorial decisions and citations, leaving causal interpretation ambiguous.
↳ Materials and Methods; Results
Citations are influenced by journal prestige, which is itself a product of peer review, so using citations to judge gatekeeping effectiveness embeds a confound the paper acknowledges but does not resolve analytically.
↳ Discussion
The data come from three elite medical journals and 2003-2004 submissions only, so transfer to other disciplines, journal tiers, and contemporary review practices is speculative and untested.
↳ Materials and Methods; Discussion
The evaluation centered on a genuine tension between an exceptional data resource and a limited analytical design. The rare access to rejected manuscripts and the striking finding that the most-cited articles were disproportionately rejected anchor the paper's contribution, while the absence of multivariate confounding control and the circularity of using citations to evaluate decisions that shape citations constrain the craft score. Assessments converged on the strength of the writing and the honesty of the limitations discussion, and diverged mainly on how substantive the literature engagement was and on whether the paper's clear relevance to editors translates into a concrete application pathway. The accepted-manuscript analyses rest on small subsamples (n=40) with non-significant correlations, which weighed against stronger claims, and the descriptive nature of the findings limits impact readiness despite the high-traction topic.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.4
Confidence highThe score reflects a genuinely novel empirical contribution built on a rare dataset of rejected manuscripts from three elite medical journals, demonstrating that desk-rejection and reviewer scores correlate with citation outcomes while the most-cited articles were disproportionately rejected. It is bounded by the narrow sample and the structural reliance on citations as the sole quality metric, a confound the paper acknowledges but does not resolve.
our three focal journals rejected many highly cited manuscripts, including the 14 most popular
The cross-sectional observational design is fit for comparing citation outcomes across editorial decision categories, with appropriate use of log-transformed citations and an impact-factor control, but confounding control is minimal: the core desk-rejected versus peer-reviewed comparison does not adjust for manuscript quality, and the small accepted-manuscript subsample (n=40) yields non-significant correlations. The use of citations as both outcome and definition of quality introduces circularity.
innate quality of initial submissions explains at least some of the citation gap
The argument moves logically from research question through dataset and editorial-decision analysis to a nuanced conclusion that gatekeeping was generally but imperfectly effective, with informative figures and consistently careful hedging. The only minor friction is the switching between 14 and 15 most-cited cases, which is explained in text but not flagged in the abstract.
Editors and peer reviewers generally--but not always--made good decisions
The paper situates itself substantively in the literature on peer review reliability, citation behavior, the Matthew Effect, and innovation suppression, and discusses the central confounds honestly rather than perfunctorily. The score is held below the top band because the limitations discussion does not fully resolve how the confounds affect the direction or magnitude of the main findings.
citations are not necessarily a perfect indicator of scientific merit
Lower confidence on Contribution — domain match limited.
Caveats4 of 4 checks
Numerical reconciliation across abstract, methods, and results is internally consistent (1,008 = 946 rejected + 62 accepted; 808 eventually published), but interpretive framing of citation differences as evidence of gatekeeping effectiveness extends beyond what an observational design supports, and emphasis shifts between 14 and 15 top-cited cases due to one article rejected by two journals.
Ethics approval from the UCSF Committee on Human Research is declared but not independently verifiable from the text, and the underlying dataset is declared confidential and accessible only to the research team, so the core findings rest on a proprietary single-source dataset that cannot be externally inspected. No conflicts are disclosed and no fabrication signals are present.
All 60 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The work is an archival observational characterization of gatekeeping effectiveness rather than a deployment-ready intervention or validated procedure, providing decision-relevant evidence without an implementable tool. It characterizes a phenomenon and raises but does not answer the operational question of how editors should adjust strategy.
the need for follow-up studies of peer review in different journals and disciplines
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications
These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.
TentativeChao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics
papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year
TentativeMasashi Kawai, Koichi Ojiro, Makoto Asaoka, et al. · 2022 · Other social sciences
日本の機関リポジトリによるオープンアクセス論文の数は限られるものの，その被引用数と様々なタイプのオープンアクセス論文の被引用数の比較からは，日本の機関リポジトリによるオープンアクセスのメリットは確認できなかった．
Tentative