# Measuring the effectiveness of scientific gatekeeping

- **DOI:** [10.1073/pnas.1418218112](https://doi.org/10.1073/pnas.1418218112)
- **Rating:** Sound (3.4 out of 5)
- **Verdict:** A rare-data study showing peer review adds value on average yet systematically rejects the most-cited work, with observational design limiting causal inference.
- **Framework version:** v5.0
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-06-05
- **Evaluation page:** https://www.nabu.science/paper/10.1073%2Fpnas.1418218112

## Strengths

- **Rare access to rejected manuscript data** The study leverages a dataset of 1,008 manuscripts submitted to three elite medical journals, including rejected submissions that are almost never accessible, enabling citation comparisons across editorial decisions that published-only data cannot support.
  - Dimension: contribution
  - Location: Materials and Methods; Abstract
- **Striking, well-documented omission-error finding** The result that the 14 most-cited articles in the dataset were rejected by the focal journals, with 12 desk-rejected without peer review, is both provocative and clearly supported by the citation-rank analysis.
  - Dimension: contribution
  - Location: Results, Fates of Rejected vs. Accepted Articles; Table 2
- **Clear writing with disciplined hedging** The argument is logically structured and consistently frames findings as case studies, describing associations rather than causal effects and openly acknowledging that citations are an imperfect proxy for merit.
  - Dimension: reporting
  - Location: Abstract; Discussion

## Limitations

- **Minimal control for confounding** The core desk-rejected versus peer-reviewed comparison does not adjust for manuscript quality, author characteristics, or article type, all of which plausibly influence both editorial decisions and citations, leaving causal interpretation ambiguous.
  - Dimension: methodological rigour
  - Location: Materials and Methods; Results
- **Citation-as-quality circularity unresolved** Citations are influenced by journal prestige, which is itself a product of peer review, so using citations to judge gatekeeping effectiveness embeds a confound the paper acknowledges but does not resolve analytically.
  - Dimension: methodological rigour
  - Location: Discussion
- **Narrow sample limits generalisability** The data come from three elite medical journals and 2003-2004 submissions only, so transfer to other disciplines, journal tiers, and contemporary review practices is speculative and untested.
  - Dimension: application pathway
  - Location: Materials and Methods; Discussion

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. Numerical reconciliation across abstract, methods, and results is internally consistent (1,008 = 946 rejected + 62 accepted; 808 eventually published), but interpretive framing of citation differences as evidence of gatekeeping effectiveness extends beyond what an observational design supports, and emphasis shifts between 14 and 15 top-cited cases due to one article rejected by two journals.
- **Research conduct:** Noted. Ethics approval from the UCSF Committee on Human Research is declared but not independently verifiable from the text, and the underlying dataset is declared confidential and accessible only to the research team, so the core findings rest on a proprietary single-source dataset that cannot be externally inspected. No conflicts are disclosed and no fabrication signals are present.
- **Reference integrity:** Clean. All 60 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17