# Do AI Scientists Know What's New?

- **DOI:** [10.48550/arxiv.2610.36511](https://doi.org/10.48550/arxiv.2610.36511)
- **Rating:** Sound (3.6 out of 5)
- **Verdict:** Claim-level novelty profiles reduce benchmark false alarms, but proxy-defined ground truth limits claims about reliable novelty detection.
- **Framework version:** v5.10
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-03
- **Evaluation page:** https://www.nabu.science/paper/10.48550%2Farxiv.2610.36511

## Key claims

1. CLAIMNOV catches 97% of disguised papers at a 1.5% false-alarm rate
   - Support: Supported
   - Type: Primary finding
   - Basis: observed benchmark counts support the rates within engineered disguises and model-labeled controls
   - Location: Abstract; Table 1
2. [CLAIMNOV's] claim-level novelty profile cuts the LLM judge's false alarms from 11.8% to 1.5%
   - Support: Supported
   - Type: Secondary
   - Basis: same-judge ablation and clustered tests support the reduction, with false alarms defined by proxy labels
   - Location: Abstract; Table 1 ablation; Table 19
3. CLAIMNOV also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check.
   - Support: Supported
   - Type: Secondary
   - Basis: constructed hybrids and noisy reviewer-dispute labels support preservation within these tasks, with 76 referring to the strongest balanced audited comparator
   - Location: Abstract, final sentence
4. checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers
   - Support: Tentative
   - Type: Secondary
   - Basis: constructed non-novel tasks and model-labeled controls show a tendency with a material balanced-performance exception
   - Location: Abstract; §5.1; Table 1
5. checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal
   - Support: Supported
   - Type: Secondary
   - Basis: evaluated whole-proposal checks return prior-art names only alongside rejections
   - Location: Abstract, sentence beginning 'and observe that checks'

## Strengths

- **Temporal benchmarks separate distinct novelty behaviors** Four datasets distinguish duplicate detection, source-removal behavior, claim attribution, and preservation of partially novel proposals. This makes the comparative contribution more informative than a single whole-proposal novelty score.
  - Dimension: contribution
  - Location: §§4.4–4.5; Tables 1–3
- **Controlled ablations isolate the claim profile** The same-judge ablation reduces reported false alarms from 11.8% to 1.5% when a claim profile replaces nearest abstracts. Clustered tests, repeated runs, and alternative-judge analyses probe the result's stability.
  - Dimension: methodological rigour
  - Location: Tables 1, 19, and 24; Appendix H
- **Results distinguish attribution from proposal rejection** Shared-ranking comparisons separate the decision to preserve a proposal from the ability to identify prior art. The paper also reports that its native claim-level attribution trails dense ranking on disputed proposals.
  - Dimension: reporting
  - Location: §§5.2–5.3; Tables 2–3 and 20

## Limitations

- **Retrieval coverage assertion conflicts with validation** Appendix O.2 says only 5 of 837 matched papers are never retrieved, but Table 33 reports 82% coverage at fixed k=40. That corresponds to approximately 151 absent papers, incompatible with near-complete coverage from a heuristic capped at 40.
  - Dimension: methodological rigour
  - Location: Appendix O.2; Table 33
- **Proxy labels limit real novelty interpretation** The human audit confirms only 28 of 45 sampled later-realized ideas as novel relative to their pools, a stratum-weighted estimate of about 60%. CLAIMNOV rejects none of the 17 judged already done, limiting extrapolation from engineered-disguise detection to natural duplicates.
  - Dimension: methodological rigour
  - Location: Appendix F.1; Table 13
- **Headline conclusions exceed the tested setting** The broad conclusion about AI scientists' novelty knowledge goes beyond fixed-pool, constructed AI-research tasks. Acknowledged label uncertainty is not sufficiently reflected in the abstract's interpretation of false alarms.
  - Dimension: reporting
  - Location: Abstract; Conclusion; Limitations

## Reliability checks

Caveats (4 of 4 checks).

- **Results check:** Noted. 2 reported values do not reconcile, but the key claims do not rest on them.
  - In this set, using this depth rule means that only 5 of the 837 matched papers are never retrieved: "only 5 of the 837 matched papers are never retrieved" (Appendix O.2, paragraph 'Retrieval depth') vs "In profile | 44% | 57% | 69% | 77% | 82% | 91%" (Table 33, first 'In profile' row, fixed k=40 column). The printed 82% coverage at fixed k=40 implies approximately 837 × 0.18 = 151 absent papers, and the stated heuristic capped at 40 cannot retrieve more than fixed k=40 under the same procedure. → Changes the size materially
  - Minor reporting notes (1):
    - Descriptive: "Table 36 compares the judge's use of the 1–5 entailment scale with that of three human annotators on n = 50 pairs" (Appendix O.3, paragraph 'Graded entailment judge') vs "Graded entailment against the human majority label (37 pairs)." (Table 36 caption). The prose specifies 50 annotated pairs while the corresponding table caption specifies 37, without explaining the analyzed subset. → Effect on result unclear
- **Reporting and transparency:** Noted. Appendix analyses use 347 disguises without defining their relationship to the 380-disguise main analysis. Ethics and code/data availability declarations are present.
  - Ethics approval: declared · Data availability: declared · Code availability: declared
- **Reference integrity:** Clean. 61 of 61 checkable references verified
  - 63 references in manuscript
  - 2 have no canonical index record — counted, but not index-checkable
  - 2 references confirmed by manual review
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-03