# Reviewer bias in single- versus double-blind peer review.

- **DOI:** [10.1073/pnas.1707323114](https://doi.org/10.1073/pnas.1707323114)
- **Rating:** Strong (4.1 out of 5)
- **Verdict:** A first-of-its-kind within-subject field experiment proving single-blind reviewers favour famous authors and top institutions, constrained by single-conference scope.
- **Framework version:** v5.0
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-06-05
- **Evaluation page:** https://www.nabu.science/paper/10.1073%2Fpnas.1707323114

## Strengths

- **First within-subject experimental design on blinding** By randomly splitting the program committee and assigning two single-blind and two double-blind reviewers to every paper, the study creates a within-paper comparison that controls for intrinsic paper quality — the first design of its kind for this question in computer science.
  - Dimension: contribution
  - Location: Materials and Methods, experimental design list
- **Substantive, well-quantified bias effects** Single-blind reviewers are significantly more likely to recommend acceptance for papers from famous authors, top universities, and top companies, with tangible odds multipliers of 1.63, 1.58, and 2.10 reported against a blinded quality covariate.
  - Dimension: methodological rigour
  - Location: Results, Modeling Reviews; Table 2
- **Concrete, commissioned policy decision context** The study was undertaken specifically to advise WSDM 2018 chairs on whether to adopt double-blind review, giving it an identifiable stakeholder pathway directly relevant to ongoing conference-reviewing policy debates.
  - Dimension: application pathway
  - Location: Materials and Methods, opening paragraph

## Limitations

- **Reviewer pools not identically distributed** Because bidding dynamics differ across conditions, the single-blind and double-blind reviewers who end up reading a given paper are not identically distributed, so observed score differences conflate evaluation bias with selection into reviewing — a limitation the authors disclose but cannot resolve.
  - Dimension: methodological rigour
  - Location: Discussion, Reviewing Behavior
- **No confidence intervals on effect sizes** The odds multipliers are reported as point estimates with p-values only, without confidence intervals or a full regression table with standard errors, limiting assessment of precision.
  - Dimension: methodological rigour
  - Location: Table 2
- **Single-venue scope limits generalisation** Findings come from one selective computer-science conference with a tech-specific institution definition and a particular management workflow, and the study stops at reviewing behaviour rather than the final acceptance decision, which occurred after the experiment closed.
  - Dimension: application pathway
  - Location: Discussion, Reviewing Behavior; Discussion, Final decisions

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. The paper is internally consistent across abstract, methods, results, and conclusion, with reported odds multipliers (2.10, 1.63, 1.58) and the 22% bidding reduction matching between abstract and Results. The Conclusion's recommendation that chairs 'seriously consider' double-blind review is somewhat broader than the directly measured reviewing-behaviour endpoint, though it is hedged.
- **Research conduct:** Clean. Ethics approval and informed consent are declared (Study Approval). Data sharing limitations are explicitly disclosed with sufficient statistics provided in lieu of raw data for participant-privacy reasons. The authors' role as program co-chairs is disclosed within the paper. No image or data integrity concerns are evident.
- **Reference integrity:** Clean. All 27 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
