# Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

- **DOI:** [10.48550/arxiv.2611.02244](https://doi.org/10.48550/arxiv.2611.02244)
- **Rating:** Strong (3.8 out of 5)
- **Verdict:** Real manuscript trajectories reveal limited AI score responsiveness and framing-sensitive gains, but single-model improvement labels constrain interpretation.
- **Framework version:** v5.10
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-06
- **Evaluation page:** https://www.nabu.science/paper/10.48550%2Farxiv.2611.02244

## Key claims

1. Major paper improvements usually do not lead to higher AI review scores. Even revisions annotated as major improvements usually receive unchanged scores.
   - Support: Supported
   - Type: Primary finding
   - Basis: single-model improvement labels and absent repeat-review measurements limit interpretation beyond the observed majority of unchanged scores
   - Location: Section 1 Finding 1; Section 5; Abstract; Table 10
2. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers.
   - Support: Supported
   - Type: Secondary
   - Basis: heterogeneous associations and grouped held-out prediction support bounded guidance, while exploratory feature selection limits certainty
   - Location: Abstract
3. Even when a revised paper receives a higher neutral score, a negative user opinion can erase or reverse that advantage without any change to the paper.
   - Support: Tentative
   - Type: Secondary
   - Basis: single reviews and selection on positive neutral gaps leave regression to the mean inseparable from opinion-associated loss of advantage
   - Location: Section 1 Finding 3; Section 7; Table 21

## Strengths

- **Natural revisions connect evidence to scoring** The dataset links 6,959 accepted-paper trajectories to dimension-level revision descriptions and separate overall-quality judgments. This enables assessment of real author changes rather than only controlled rhetorical rewrites.
  - Dimension: contribution
  - Location: Sections 1–3; Appendix A
- **Dependence-aware analyses test competing statistical explanations** Whole-paper bootstraps, grouped cross-validation, common-sample checks, and conditional permutations address repeated revisions and coverage differences. The initial-rating-conditioned permutation analysis also reports that seven unanimous increases fall within its reference range.
  - Dimension: methodological rigour
  - Location: Appendices E–G, especially F.2 and F.6
- **Interpretive qualifications accompany the central findings** The text explicitly distinguishes unchanged scores from failure to notice improvements and revision associations from intervention effects. It also explains why comparisons between initial-advantage groups do not establish causal protection against user opinions.
  - Dimension: reporting
  - Location: Sections 5, 6.2, and 7.3

## Limitations

- **Major improvement labels lack direct validation** One model supplies the central quality labels, while human confirmation accepts either major or minor improvement. Exact human agreement on degree is 62.0%, with ordinal alpha 0.521, leaving the major-versus-minor boundary incompletely validated.
  - Dimension: methodological rigour
  - Location: Section 3.3; Appendix B.5
- **Repeat-review noise remains unmeasured across conditions** Each manuscript is reviewed once per condition, without a repeated-neutral reference. Selecting initially positive score gaps can therefore introduce regression to the mean, limiting attribution of lost advantages specifically to user opinions.
  - Dimension: methodological rigour
  - Location: Appendix G.1
- **Accepted AI papers constrain external interpretation** The sample covers accepted, arXiv-matched papers from five AI conferences and specific reviewer models. Generalization to rejected papers, other disciplines, human reviewing, and actual conference workflows is not established.
  - Dimension: application pathway
  - Location: Appendices B.1 and C; Sections 5–7

## Reliability checks

No concerns (3 of 4 checks).

- **Results check:** Clean. The reported numbers and statistics are consistent with the stated results.
- **Reporting and transparency:** Clean. The text identifies a project page and documents human-validation procedures, prompts, analysis artifacts, and experimental costs. No suspicious availability statement or specific conduct finding is established.
  - Conflict of interest: not stated · Data availability: declared · Code availability: declared
- **Reference integrity:** Not checked. Reference integrity check could not run: references could not be reliably parsed from the manuscript; verification skipped rather than risk false verdicts.
  - 0 references in manuscript

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-06
