# A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants.

- **DOI:** [10.1371/journal.pone.0014331](https://doi.org/10.1371/journal.pone.0014331)
- **Rating:** Sound (3.5 out of 5)
- **Verdict:** The first quantitative meta-analysis of peer-review inter-rater reliability delivers sophisticated multilevel benchmarks, but its publication-bias and credibility claims outrun the supporting evidence.
- **Framework version:** v5.0
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-06-05
- **Evaluation page:** https://www.nabu.science/paper/10.1371%2Fjournal.pone.0014331

## Strengths

- **First quantitative synthesis of peer-review reliability** The paper provides the first meta-analytic pooling of inter-rater reliability in journal peer review, drawing on 48 studies and 19,443 manuscripts to replace narrative summaries with quantitative estimates (mean ICC/r around .34, mean Kappa .17).
  - Dimension: contribution
  - Location: Abstract; Results, Comparison of Average Effects
- **Appropriate multilevel modeling of nested data** A three-level random-effects model correctly handles the dependence among multiple coefficients reported by single studies, and the meta-regression identifies covariates that reduce between-study variance to approximate homogeneity.
  - Dimension: methodological rigour
  - Location: Materials and Methods, Statistical Procedure; Results, Meta-Regression
- **Honest engagement with competing interpretations** The discussion surfaces views that low reviewer agreement may reflect a competent, diverse panel rather than a defect, weighing these against the publication-bias reading rather than ignoring them.
  - Dimension: positioning
  - Location: Discussion (Bailar, Langfeldt, Cole)

## Limitations

- **Interpretive overreach on bias and credibility** The finding that larger-sample studies report lower reliability is presented as 'a distinctly marked publication bias' and extended to the claim that high-reliability studies are 'less credible,' which goes beyond what the sample-size covariate alone establishes.
  - Dimension: contribution
  - Location: Discussion, paragraph beginning 'The main conclusion of our meta-analysis...'; Results, Meta-Regression Model 1
- **Incomplete meta-analytic documentation** The search is not fully reproducible (no explicit search string, grey literature unaddressed), independent dual screening is not documented, and no validated risk-of-bias tool is applied, with study quality inferred instead from whether the rating system was reported.
  - Dimension: methodological rigour
  - Location: Materials and Methods, Literature Search and Covariate (7) Rating System
- **Kappa studies excluded from moderator analysis** The 26 studies reporting Cohen's Kappa could not be entered into the meta-regression, so the determinants analysis rests on the 44 ICC/r coefficients and does not cover the full evidence base.
  - Dimension: methodological rigour
  - Location: Results, Meta-Regression

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. Sample counts, pooled estimates, and variance-explained figures are internally consistent and arithmetically plausible, but the Discussion extends a sample-size association into strong claims of publication bias and reduced credibility of high-IRR studies that outpace the direct evidence.
- **Research conduct:** Noted. No ethics, conflict-of-interest, data, or code problem is evident, and ethics approval is not ordinarily required for a meta-analysis of published studies; however, ethics is unmentioned and the study-quality inference relies on an indirect, unvalidated proxy (presence/absence of rating-system information).
- **Reference integrity:** Clean. All 111 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
