Shared corpus enables direct model comparisons
All four models reviewed the same 200 manuscripts, enabling direct descriptive comparisons and paired tests of framing and flaw-injection effects.
↳ Methods §§2.1–2.3; Results §§3.1–3.3
Abstract not available — view at DOI
Key Claims
Primary findingWell-supported
605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001)
Likely to hold — the claim's weakest requirement is that the effect exists at all. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
consistent counts and manuscript-blocked analysis directly support heterogeneity within the tested corpus and protocol
Abstract; §3.1
SecondaryTentative
When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001)
flaw validity and detection labels rely on automated judgments without independent expert validation
Abstract; §3.3
SecondarySupported
Re-reviewing all 200 manuscripts under a prompt identical except for an opening sentence stating that the text had already been through one round of review, every model became more lenient (Figure 4).
consistent paired shifts support aggregate leniency changes, although single stochastic outputs limit precision
§3.2, first paragraph
SecondaryTentative
Planting a verifiable flaw nonetheless barely moved the recommendation itself.
baseline recommendation ceilings and unvalidated flaw severity limit interpretation of the small observed change
§3.3
SecondaryTentative
All tested LLMs lacked both sensitivity to errors and specificity, recommending major revision or rejection for papers that had nevertheless been published in top journals.
publication status does not establish the first-submission specificity standard, and one model gave favorable recommendations in 82.5% of cases
Discussion, first paragraph
Derived from the full evaluation — not a separate score.
Strengths · Limitations
Strengths
Explicit prompts and clustered robustness checks
Verbatim prompts make the protocol inspectable, while Friedman and manuscript-clustered GEE analyses address dependence in the principal model comparisons.
↳ Methods §2.5; Appendix A; Results §§3.1 and 3.3
Findings address an active editorial need
The paper identifies editors and publishers as users and connects model-dependent behavior to continuous local benchmarking rather than autonomous editorial decisions.
↳ Introduction; Discussion §4; Conclusion §4.1
Limitations
Publication status does not establish specificity
A published article is not a validated negative control for a first-submission review decision. Treating major revision as a false positive therefore supports a stronger interpretation than the benchmark establishes.
↳ Introduction, final paragraph; Abstract; Discussion §4
Automated evaluation lacks independent expert validation
LLMs generate planted flaws and adjudicate detection and review quality without expert validation. Single outputs also leave run-to-run uncertainty unmeasured, while ceiling effects constrain interpretation of unchanged recommendations.
↳ Methods §§2.2–2.4; Results §3.3
Incomplete inputs narrow the review interpretation
Rendered figures and supplementary files were excluded even though they can contain core evidence and methods. The limitations do not trace this restriction into claims about reviewing published articles.
↳ Methods §2.1; Discussion §4
Nabu Commentary
The shared-corpus design and paired framing experiment establish substantial model-dependent variation in recommendations (§§3.1–3.2). Exact prompts and clustered robustness analyses make the descriptive comparisons inspectable. The specificity interpretation is less secure because prior publication does not establish the appropriate first-submission recommendation, and excluded visual and supplementary evidence further narrows that inference. Automated flaw construction and judging, together with single outputs per condition, limit the capability conclusions and keep the practical contribution at the benchmarking stage.
Check another paper
Author Note / Field Feedback
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Dimension breakdown
Study Quality
Sound3.1
Confidence mediumContribution
3.6
The common corpus and paired manipulations meaningfully extend evidence on model-dependent recommendations, framing sensitivity, and planted-flaw detection. The advance is narrower than the specificity interpretation because publication status and automated judgments do not establish independent validity.
Introduction; Methods §§2.2–2.4; Results §§3.1–3.3
Methodological Rigour
2.9
Verbatim prompts, paired comparisons, and manuscript-clustered robustness checks support the descriptive audit. Single outputs, unreported decoding settings, and unvalidated flaw generation and adjudication leave meaningful uncertainty about the capability interpretations.
Methods §§2.2–2.5; Appendix A
Reporting
3.4
The experimental sequence and numerical reporting are readily followable. The central low-specificity wording overstates what unfavorable recommendations on published manuscripts establish.
Abstract; Results §§3.1–3.3; Discussion §4
Positioning
2.8
The paper engages both human-review variability and directly related AI-review evidence. Its limitations do not address the first-submission decision standard or trace excluded images and supplements into the broad interpretation.
Introduction; Methods §2.1; Discussion §4
Reliability
No concerns4 of 4 checks
Results checkClean
The reported numbers and statistics are consistent with the stated results.
Reporting and transparencyClean
AI assistance and competing interests are disclosed. Data and code are promised for future deposit, not represented as currently accessible.
Conflict of interest: declared · Data availability: not stated · Code availability: not stated
Reference IntegrityClean
21 references in manuscript 18 of 19 checkable references found in an index 2 are books, websites or datasets — counted, but not index-checkable
References not verified automatically are resolved manually within 24 hours.
Post-Publication RecordClean
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Last checked: 23 Oct 2026
Sources: Crossref (incl. Retraction Watch) ✓
Readiness
Where this paper’s evidence sits on the path from initial observation to real-world use.
Demonstrated
The protocol can inform local benchmarking but is not an operationally validated editorial procedure. Expert validation and prospective testing remain necessary.
Methods §§2.3–2.4; Discussion §4; Conclusion §4.1
AI-generated, human-governed. Something look off? Contact us to request a review.