nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-1EC255C9·Framework v5.0·Pipeline v0.4·Evaluated 05 Jun 2026·OECD: 5.8 Other social sciences·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Meta-analysis

A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants.

Bornmann et al.Bornmann, Mutz, Daniel·2010·PloS one·10.1371/journal.pone.0014331 ↗

In brief

The first quantitative meta-analysis of peer-review inter-rater reliability delivers sophisticated multilevel benchmarks, but its publication-bias and credibility claims outrun the supporting evidence.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

BACKGROUND: This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree.

METHODOLOGY/PRINCIPAL FINDINGS: Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2=.34, mean Cohen's Kappa=.17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed.

CONCLUSIONS/SIGNIFICANCE: Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).

Strengths · Limitations

Strengths

#contribution

First quantitative synthesis of peer-review reliability

The paper provides the first meta-analytic pooling of inter-rater reliability in journal peer review, drawing on 48 studies and 19,443 manuscripts to replace narrative summaries with quantitative estimates (mean ICC/r around .34, mean Kappa .17).

↳ Abstract; Results, Comparison of Average Effects

#methodological rigour

Appropriate multilevel modeling of nested data

A three-level random-effects model correctly handles the dependence among multiple coefficients reported by single studies, and the meta-regression identifies covariates that reduce between-study variance to approximate homogeneity.

↳ Materials and Methods, Statistical Procedure; Results, Meta-Regression

#positioning

Honest engagement with competing interpretations

The discussion surfaces views that low reviewer agreement may reflect a competent, diverse panel rather than a defect, weighing these against the publication-bias reading rather than ignoring them.

↳ Discussion (Bailar, Langfeldt, Cole)

Limitations

#contribution

Interpretive overreach on bias and credibility

The finding that larger-sample studies report lower reliability is presented as 'a distinctly marked publication bias' and extended to the claim that high-reliability studies are 'less credible,' which goes beyond what the sample-size covariate alone establishes.

↳ Discussion, paragraph beginning 'The main conclusion of our meta-analysis...'; Results, Meta-Regression Model 1

#methodological rigour

Incomplete meta-analytic documentation

The search is not fully reproducible (no explicit search string, grey literature unaddressed), independent dual screening is not documented, and no validated risk-of-bias tool is applied, with study quality inferred instead from whether the rating system was reported.

↳ Materials and Methods, Literature Search and Covariate (7) Rating System

#methodological rigour

Kappa studies excluded from moderator analysis

The 26 studies reporting Cohen's Kappa could not be entered into the meta-regression, so the determinants analysis rests on the 44 ICC/r coefficients and does not cover the full evidence base.

↳ Results, Meta-Regression

Does this match your read? Add your assessment ↓

Nabu Commentary

The evaluation centers on a paper that is methodologically sophisticated for its era yet makes claims that run slightly ahead of its design. The three-level multilevel model is the right tool for the nested coefficient structure and is competently executed, and the meta-regression's reduction of between-study variance to approximate homogeneity is a real analytical result. Assessments diverged most on how to weigh the Discussion's leap from a sample-size association to assertions of publication bias and reduced credibility of high-reliability studies, an interpretive step that the data support only indirectly. They also diverged on impact, where the paper's descriptive, benchmark-setting character, with the reader system discussed but not validated, limits its translation potential. Documentation gaps in search reproducibility, screening independence, and risk-of-bias assessment, together with exclusion of the Kappa studies from the moderator analysis, hold Methodological Rigour below the upper band.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.5±0.1

Contribution
3.8

The score reflects a genuine first-of-kind quantitative synthesis of peer-review inter-rater reliability across 70 coefficients from 48 studies and 19,443 manuscripts, advancing beyond prior narrative reviews and adding a meta-regression of determinants. It is held below the upper band because the headline descriptive finding (low IRR) largely confirms the pre-existing narrative consensus.

Abstract; Introduction; Results, Comparison of Average Effects and Meta-Regression

the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews

Methodological Rigour
3.3

The three-level multilevel random-effects model is appropriate for the nested coefficient structure and is correctly executed, with heterogeneity testing and BIC-based model comparison. The score is constrained by non-reproducible search documentation, undocumented dual screening, absence of a validated risk-of-bias tool, and exclusion of the 26 Kappa studies from meta-regression.

Materials and Methods, Statistical Procedure and Literature Search; Results, Meta-Regression

For studies with a Kappa coefficient ... no analyses could be performed

Reporting
3.7

The argument moves cleanly from question to pooled estimates to moderator analysis to implications, with consistent terminology and supporting tables and forest plot. A minor deduction reflects the potentially confusing 'ICC/r2' notation and the absence of a PRISMA-style flow diagram for the inclusion process.

Abstract; Results; Tables 1-3 and Figure 1

mean ICC/r2=.34, mean Cohen's Kappa=.17

Positioning
3.5

The paper engages substantively with named prior work (Marsh/Bond/Jayasinghe, Weller, Cicchetti, Cole) and honestly surfaces competing views that high agreement may be undesirable (Bailar, Langfeldt). It is held mid-band because limitations are embedded rather than explicitly labeled and a broad normative conclusion rests partly on an indirect, proxy-driven evidence base.

Discussion

Too much agreement is in fact a sign that the review process is not working well

Lower confidence on Methodological Rigour, Positioning — domain match limited.

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

Sample counts, pooled estimates, and variance-explained figures are internally consistent and arithmetically plausible, but the Discussion extends a sample-size association into strong claims of publication bias and reduced credibility of high-IRR studies that outpace the direct evidence.

Research ConductNoted

No ethics, conflict-of-interest, data, or code problem is evident, and ethics approval is not ordinarily required for a meta-analysis of published studies; however, ethics is unmentioned and the study-quality inference relies on an indirect, unvalidated proxy (presence/absence of rating-system information).

Flags: 0 declared / 5 total

Reference IntegrityClean

All 111 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaResearch capability
Type of advanceSynthesis
Route to useSuggested
AdvanceExtends
CitationsFWCI 8.4
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The claim is supported by pooled real-world data from 48 independent studies totalling about 19,443 manuscripts across many journals, disciplines and decades.

From the paper
“According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).”

The conclusion names a real-world use, improving peer assessment reliability via reader systems, but identifies no actor or setting that would implement it.

Pathway

Journal editors and publishers would use the low-reliability evidence to reform peer review, for example adopting reader systems or more reviewers per manuscript.

Barrier

No intervention is shown here to raise reviewer agreement, and disagreement may partly reflect legitimate diversity of views at the research frontier.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study

    Alexander D Gibson, David N Borg, Nicole M White, et al. · 2026 · Health sciences

    Primary claim

    not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice

    Tentative
    Evidence Score
    3.1±0.2
    Reliability
    Caveats
  2. Suspected distortion of citations in high-impact cancer journals

    Baptiste Scancar, Jennifer Anne Byrne, David Causeur, et al. · 2026 · Media and communications

    Primary claim

    These findings suggest that paper mills inflate the citation metrics of supported publications and affected journals at scale.

    Tentative
    Evidence Score
    3.5±0.1
    Reliability
    Caveats
  3. The citation trap: Papers published at year-end receive systematically fewer citations

    Chao Ma, Yiwei Li, Feng Guo, et al. · 2019 · Economics

    Primary claim

    papers published between October and December on average get as much as 18.5% fewer citations than those published in the other months in the year

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats

How the rubric works →