nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-6DA79986·Framework v5.0·Pipeline v0.4·Evaluated 05 Jun 2026·OECD: 5.8 Other social sciences·Subfield: Information Systems·Basis: Full text·Type: Field experiment

Reviewer bias in single- versus double-blind peer review.

Tomkins et al.Tomkins, Zhang, Heavlin·2017·Proceedings of the National Academy of Sciences of the United States of America·10.1073/pnas.1707323114 ↗

In brief

A first-of-its-kind within-subject field experiment proving single-blind reviewers favour famous authors and top institutions, constrained by single-conference scope.

ReliabilityCaveats4 of 4 checks completeRead the entry ↓
Abstract

Peer review may be "single-blind," in which reviewers are aware of the names and affiliations of paper authors, or "double-blind," in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.

Strengths · Limitations

Strengths

#contribution

First within-subject experimental design on blinding

By randomly splitting the program committee and assigning two single-blind and two double-blind reviewers to every paper, the study creates a within-paper comparison that controls for intrinsic paper quality — the first design of its kind for this question in computer science.

↳ Materials and Methods, experimental design list

#methodological rigour

Substantive, well-quantified bias effects

Single-blind reviewers are significantly more likely to recommend acceptance for papers from famous authors, top universities, and top companies, with tangible odds multipliers of 1.63, 1.58, and 2.10 reported against a blinded quality covariate.

↳ Results, Modeling Reviews; Table 2

#application pathway

Concrete, commissioned policy decision context

The study was undertaken specifically to advise WSDM 2018 chairs on whether to adopt double-blind review, giving it an identifiable stakeholder pathway directly relevant to ongoing conference-reviewing policy debates.

↳ Materials and Methods, opening paragraph

Limitations

#methodological rigour

Reviewer pools not identically distributed

Because bidding dynamics differ across conditions, the single-blind and double-blind reviewers who end up reading a given paper are not identically distributed, so observed score differences conflate evaluation bias with selection into reviewing — a limitation the authors disclose but cannot resolve.

↳ Discussion, Reviewing Behavior

#methodological rigour

No confidence intervals on effect sizes

The odds multipliers are reported as point estimates with p-values only, without confidence intervals or a full regression table with standard errors, limiting assessment of precision.

↳ Table 2

#application pathway

Single-venue scope limits generalisation

Findings come from one selective computer-science conference with a tech-specific institution definition and a particular management workflow, and the study stops at reviewing behaviour rather than the final acceptance decision, which occurred after the experiment closed.

↳ Discussion, Reviewing Behavior; Discussion, Final decisions

Does this match your read? Add your assessment ↓

Nabu Commentary

The adjudicated scores rest on a design that genuinely isolates the effect of author-identity access by assigning matched single- and double-blind reviewers to every paper and controlling for a blinded quality score, with appropriate statistical tests for each outcome. The substantive odds multipliers (2.10, 1.63, 1.58) for top companies, famous authors, and top universities, reported in Table 2, anchor the contribution and craft scores in the upper-competent band. They are held below the top band by the absence of confidence intervals on those multipliers, the structural non-equivalence of the two reviewer pools acknowledged in the Discussion, and an untested four-firm definition of 'top companies'. Impact scores are lower and deliberately decoupled from quality: the explicit commissioning context for WSDM 2018 supports a strong traction reading, while translation and transferability are pulled down by the single-venue scope and the study's stopping point at reviewing behaviour rather than final acceptance. Reported null results on gender, discussion-phase score changes, and conflict rates appear alongside the significant findings, supporting internal consistency.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
4.1±0.1

Contribution
4.2

The score reflects the first within-subject controlled field experiment comparing single- and double-blind reviewing in a live computer-science conference, filling an explicitly identified gap with claims kept close to the bidding and scoring outcomes actually measured. Substantive effect sizes for famous-author, top-university, and top-company favouritism are reported with appropriate caveats.

Materials and Methods, opening paragraph; Results, Modeling Reviews

we discovered no within-subject experimental study of the question

Methodological Rigour
4.0

The score reflects a strong design-execution fit — randomised PC split with two reviewers per condition assigned to every paper, controlling for paper-level quality, and analyses (logistic regression, Mann-Whitney, Fisher exact) appropriate to each outcome. It is held below the top band by absent confidence intervals on the odds multipliers, the structural non-equivalence of reviewer pools, an untested four-firm 'top companies' definition, and a modest 0.38 inter-rater correlation.

Materials and Methods, experimental design list; Table 2; Materials and Methods, Blinded Paper Quality Score

The achieved correlation between the two double-blinded raters is 0.38

Reporting
4.2

The score reflects a logically sequenced presentation moving from design through bidding and review results to limitations, with qualifying language that consistently separates reviewing behaviour from final acceptance decisions. A key gender meta-analytic claim is deferred to a companion 'expanded paper' and cannot be fully evaluated from this text.

Results, Summary of Results; Results, The Matilda Effect; Discussion, Reviewing Behavior

we cannot make statistical statements about the final acceptance decision

Positioning
4.0

The score reflects substantive engagement with the peer-review literature, including a 600-plus-item survey and named bias mechanisms (Matilda, Matthew, institutional) with originating citations, plus a non-perfunctory limitations section and balanced treatment of practical double-blind difficulties. A minor gap is the absence of engagement with the demand-characteristics literature relevant to one stated limitation.

Introduction; Discussion, Practical Issues with Double-Blind Reviewing and Methodological Questions

A detailed survey reviews over 600 pieces of literature on reviewing

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The paper is internally consistent across abstract, methods, results, and conclusion, with reported odds multipliers (2.10, 1.63, 1.58) and the 22% bidding reduction matching between abstract and Results. The Conclusion's recommendation that chairs 'seriously consider' double-blind review is somewhat broader than the directly measured reviewing-behaviour endpoint, though it is hedged.

Research ConductClean

Ethics approval and informed consent are declared (Study Approval). Data sharing limitations are explicitly disclosed with sufficient statistics provided in lieu of raw data for participant-privacy reasons. The authors' role as program co-chairs is disclosed within the paper. No image or data integrity concerns are evident.

Flags: 3 declared / 5 total

Reference IntegrityClean

All 27 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 9
Type of advanceEmpirical
Route to useNamed
AdvanceExtends
CitationsFWCI 79.0
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim rests on a single controlled experiment embedded in one conference, with no independent replication at other venues reported.

From the paper
“program and general chairs of conferences should seriously consider the advantages of using double-blind reviewing”

The conclusion names conference program and general chairs as the actors who would use these findings when choosing a reviewing model, though it stops short of specifying the decision fully.

Pathway

Conference program chairs and journal editors use these findings to justify adopting double-blind review, reducing bias against less-famous authors and institutions.

Barrier

Evidence comes from one computer science conference, and practical anonymization problems such as arXiv preprints leave double-blind reviewing unresolved and hard to generalize.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Artificial intelligence in language learning: biometric feedback and adaptive reading for improved comprehension and reduced anxiety

    Hang Yuan · 2025 · Educational sciences

    Primary claim

    Quantitative results revealed significant improvements in reading comprehension, motivation, anxiety reduction, and cognitive load management among participants using the AI platform compared to the control group.

    Tentative
    Evidence Score
    2.2±0.2
    Reliability
    Concerns
  2. Blind spots in AI-assisted healthcare evidence search: multiplatform evaluation of clinical retrieval gaps and risk-of-bias

    Farrag, Soliman, Azubuike, et al. · 2026 · Computer and information sciences

    Primary claim

    Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.

    Well-supported
    Evidence Score
    3.9±0.2
    Reliability
    No concerns
  3. The citation advantage of linking publications to research data

    Colavizza, Hrynaszkiewicz, Staden, et al. · 2020 · Other social sciences

    Primary claim

    We also find an association between articles that include statements that link to data in a repository and up to 25.36% (± 1.07%) higher citation impact on average, using a citation prediction model.

    Supported
    Evidence Score
    3.8±0.2
    Reliability
    Caveats

How the rubric works →