Crunching the numbers. Responsibly.
Research assessment relies on expert evaluations, yet human judgement is noisy, and it is unclear whether differences in assessment arise primarily from differences in genuine research quality or from unwanted differences between evaluators. While numerous studies highlight disagreement and biases in research assessment, they have not quantified judge-related noise relative to variation in the evaluated works. Here we show, in a large post-publication peer review database, that research assessment is driven more by differences between evaluators than by difference in the evaluated research. We partition variance in 239,521 research quality ratings assigned by 12,649 judges to 193,128 papers from the H1 Connect post-publication peer review platform. Using multilevel models, we decomposed judge-related variation into differences in overall severity and differences in the weighting of scientific attributes. We found that judge-related effects accounted for substantially more variance in ratings than the evaluated papers. In our most detailed model, judge-level effects and judge-specific slopes explained 61% of the total variance, whereas combined paper and journal-level effects accounted for only 7%. By contrast, examined measures of directional bias, such as author gender and global affiliation, explained less than 1% of the variance. We conclude that assessment outcomes were shaped more by the judges than by the papers themselves. Our results demonstrate the necessity of noise audits in high-stakes scientific evaluation.
judge-level effects and judge-specific slopes explained 61% of the total variance, whereas combined paper and journal-level effects accounted for only 7%
directly estimated in the detailed multilevel model and stable across robustness checks
research assessment is driven more by differences between evaluators than by difference in the evaluated research
supported for H1 Connect ratings, but phrased more broadly than the single-platform design identifies
Variation in ratings attributable to judge severity exceeded variation attributable to the papers themselves, 26% to 18% respectively
direct baseline random-effects comparison, before adding journal and judge-slope structure
examined measures of directional bias, such as author gender and global affiliation, explained less than 1% of the variance
appendix models show negligible marginal R² for measured bias variables, though unmeasured biases remain possible
Our results demonstrate the necessity of noise audits in high-stakes scientific evaluation
policy implication is plausible, but deployment need is inferred from one observational platform
Derived from the full evaluation — not a separate score.
Strengths
The study analyses 239,521 ratings by 12,649 judges across 193,128 papers and 5,038 journals, giving the variance-partitioning analysis substantial empirical scale. This enables comparisons between judge, paper, journal, and judge-slope components that smaller inter-rater studies cannot make.
↳ Abstract; Results, opening paragraph
The modelling strategy directly matches the research question by estimating paper and judge random intercepts, adding journal effects, and then adding judge-specific random slopes for latent paper dimensions. This lets the paper separate level noise from pattern noise rather than treating all disagreement as residual variation.
↳ Methods, Multilevel models; Results, Unmasked pattern noise
The paper tests the main pattern using no-singletons restrictions, richer-classification restrictions, permutation testing, multiple optimisers, and an ordinal Bayesian model. The permutation result, where judge-slope variance falls from 49% to 0.6%, is especially informative for ruling out model-flexibility artefacts.
↳ Results, Findings were robust across model specifications; Figure 1
The Discussion translates the empirical finding into a concrete proposal: routine noise audits that partition variation between judges and research outputs. This makes the policy relevance legible for funders, journals, and research assessment exercises.
↳ Discussion, implications for research assessment
Limitations
The direct evidence comes from H1 Connect, a signed, curated, biomedical post-publication recommendation platform. The paper acknowledges that gatekeeping peer review, grant review, and binary decisions may have different variance structures.
↳ Discussion, H1 Connect limitations
Because experts choose or are associated with particular papers in an observational platform dataset, judge effects cannot be interpreted as cleanly as they could under random assignment. This complicates the leap from variance in observed ratings to intrinsic paper quality versus unwanted evaluator noise.
↳ Methods, The H1 Connect dataset; Discussion, H1 Connect limitations
The paper reduces classification tags to six latent factors explaining 65% of classification-use variance, but the supplied text does not provide factor loadings or enough detail to fully audit factor validity. Because those tags underpin the judge-slope pattern-noise analysis, this limits analytic transparency.
↳ Methods, The H1 Connect dataset; Results, Unmasked pattern noise
The no-singletons sample size is reported as both 76,698 and 76,398 cases, and the Discussion restates the detailed model as 60% to 8% after Abstract/Results report 61% to 7%. These do not overturn the central finding but justify caution about numerical precision.
↳ Methods, Robustness analyses; Abstract; Discussion, first paragraph
The paper’s score is driven by the fit between its central question and its variance-partitioning design: the multilevel models directly estimate judge, paper, journal, and judge-slope components in a large H1 Connect dataset. Robustness analyses in the Results and Figure 1 support the direction of the main finding, especially the permutation collapse of judge-slope variance. The main downward pressure comes from interpretive boundaries: H1 Connect is a selected, signed, post-publication biomedical platform, and the judge-generated classification tags make pattern-noise interpretation less clean than a fully independent paper-characteristic measure would be. Minor internal inconsistencies in sample counts and rounded percentages affect reliability and clarity but do not contradict the core result.
Strong3.8/5.0
The paper makes a clear empirical and conceptual contribution by comparing judge-related variance with paper- and journal-related variance in a large post-publication review dataset. The contribution is limited by the single-platform biomedical setting and by policy language that sometimes reaches beyond the direct evidentiary base.
judge-level effects and judge-specific slopes explained 61% of the total variance
The score reflects appropriate use of crossed multilevel models, variance partitioning, permutation testing, restricted-sample checks, multiple optimisers, and an ordinal Bayesian robustness model. It is tempered by non-random paper-judge assignment, latent factors built from judge-applied classification tags, missing factor-loading and interval details, and limited modelling of selection mechanisms.
variance attributable to judge-specific slopes from 49% to 0.6%
The paper presents the bias-versus-noise framing, progressive model sequence, and robustness checks in a readable structure supported by Figure 1 and explicit R model syntax. Its clarity is reduced slightly by broad phrasing that can blur H1 Connect rating variance with research assessment generally and by minor wording/numerical inconsistencies.
Each horizontal stacked bar shows the proportion of variance in ratings
The paper situates the study in peer-review disagreement, bias, and noise literatures and acknowledges that H1 Connect differs from gatekeeping systems such as grant and journal review. The contextual treatment is less complete on range restriction, the compressed three-level rating scale, and alternative interpretations of paper-level variance in a selected post-publication environment.
H1 Connect is a recommendation rather than gatekeeping system
Lower confidence on Methodological Rigour, Positioning — domain match limited.
Watch-outs4 of 4 checks
The central variance-partitioning claim is coherent across the Abstract, Results, Figure 1, and Discussion, but several text-level numerical and reporting inconsistencies reduce confidence in precision. The concerns are cautionary rather than evidence that the main finding is contradicted by the paper’s own data.
No text-visible evidence indicates suspicious ethics, conflict-of-interest, data, code, or image/manipulation conduct problems. The study analyses an existing post-publication platform dataset, and the absence of ethics or COI statements is treated as a reporting/open-science limitation rather than a conduct concern from the supplied text alone.
Flags: 0 declared / 5 total
1 of 2 cited references could not be matched to the publication their identifier points to — flagged for manual verification. All other checked references verified. 3 references (books, websites, grey literature, or unindexed preprints) could not be checked — this is expected and does not affect the flag.
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓PubPeer (coming soon)
Medium3.7/5.0
The paper addresses a live institutional problem for research assessment systems that depend on expert judgement, including funders, journals, hiring and promotion processes, and REF-like exercises. Its proposed noise-audit framing gives a concrete stakeholder pathway.
routine noise audits... should be a standard component
The study demonstrates a practical audit template using a large operational dataset and provides model formulas that other evaluators could adapt. Readiness remains partial because the paper does not specify implementation protocols, decision thresholds, or governance procedures for institutional deployment.
partitioning variation between the judges and the judged
The modelling approach is portable, but the observed effect sizes may not transfer cleanly beyond a signed, curated, biomedical post-publication recommendation platform. The paper itself notes that gatekeeping systems and binary decisions may have different judge- and paper-variance structures.
The relative contributions of judge- and paper-related variation may differ
The work connects empirical H1 Connect evidence to established reviewer-disagreement literature and to current noise-audit modelling for research assessment. That positioning creates a plausible trajectory toward comparative audits and institutional reform studies.
judge-related noise is... a measureable property of evaluation systems
Recorded from the paper’s own text at evaluation time; not independently verified. Re-check to verify.
Nabu’s assessment, alongside the field’s view.
Evaluation Metadata
AI-powered, human-governed. Something look off? Contact us to request a review.