A major challenge facing AI scientific discovery pipelines is how to determine when a generated idea is meaningfully novel. Ideally, novelty checks in these pipelines must be able to reject non-novel ideas while not rejecting too many novel ideas, and they should also be able to distinguish which parts of a proposal are non-novel, as opposed to opaquely discarding entire proposals. We design an analytical framework for evaluation of novelty checks with temporal ground truth at both the proposal- and claim-level, using four datasets of 1,060 proposals built from published papers, reviewer disputes, and generated ideas matching ones the field went on to publish. Auditing existing novelty checks with this framework, we find that checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers, and observe that checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal. We therefore introduce ClaimNov, a fine-grained, interpretable claim-level novelty module. ClaimNov catches 97% of disguised papers at a 1.5% false-alarm rate, and its claim-level novelty profile cuts the LLM judge's false alarms from 11.8% to 1.5%. Because its verdict rests on claim-level evidence, ClaimNov also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check. Code and data are available at https://github.com/emmatliu/claimnov.
CLAIMNOV catches 97% of disguised papers at a 1.5% false-alarm rate
observed benchmark counts support the rates within engineered disguises and model-labeled controls
[CLAIMNOV's] claim-level novelty profile cuts the LLM judge's false alarms from 11.8% to 1.5%
same-judge ablation and clustered tests support the reduction, with false alarms defined by proxy labels
CLAIMNOV also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check.
constructed hybrids and noisy reviewer-dispute labels support preservation within these tasks, with 76 referring to the strongest balanced audited comparator
checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers
constructed non-novel tasks and model-labeled controls show a tendency with a material balanced-performance exception
checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal
evaluated whole-proposal checks return prior-art names only alongside rejections
Derived from the full evaluation — not a separate score.
Strengths
Four datasets distinguish duplicate detection, source-removal behavior, claim attribution, and preservation of partially novel proposals. This makes the comparative contribution more informative than a single whole-proposal novelty score.
↳ §§4.4–4.5; Tables 1–3
The same-judge ablation reduces reported false alarms from 11.8% to 1.5% when a claim profile replaces nearest abstracts. Clustered tests, repeated runs, and alternative-judge analyses probe the result's stability.
↳ Tables 1, 19, and 24; Appendix H
Shared-ranking comparisons separate the decision to preserve a proposal from the ability to identify prior art. The paper also reports that its native claim-level attribution trails dense ranking on disputed proposals.
↳ §§5.2–5.3; Tables 2–3 and 20
Limitations
Appendix O.2 says only 5 of 837 matched papers are never retrieved, but Table 33 reports 82% coverage at fixed k=40. That corresponds to approximately 151 absent papers, incompatible with near-complete coverage from a heuristic capped at 40.
↳ Appendix O.2; Table 33
The human audit confirms only 28 of 45 sampled later-realized ideas as novel relative to their pools, a stratum-weighted estimate of about 60%. CLAIMNOV rejects none of the 17 judged already done, limiting extrapolation from engineered-disguise detection to natural duplicates.
↳ Appendix F.1; Table 13
The broad conclusion about AI scientists' novelty knowledge goes beyond fixed-pool, constructed AI-research tasks. Acknowledged label uncertainty is not sufficiently reflected in the abstract's interpretation of false alarms.
↳ Abstract; Conclusion; Limitations
The temporal datasets and shared evaluation contract provide a useful comparison of detection, attribution, and preservation behavior. The claim-profile ablation supports reduced benchmark false alarms, while clustered comparisons place CLAIMNOV statistically level with CoI-Agent on whole proposals. Human auditing and weak detection of naturally already-done ideas restrict the meaning of the headline rates, and these restrictions are not fully carried into the conclusion. The retrieval-depth assertion conflicts with Table 33, requiring correction independently of the directly reported main-table outcomes. Released reconstruction materials support reuse, but the in-the-loop pilot does not establish operational benefit.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.6
Confidence mediumThe temporal, proposal- and claim-level evaluation framework and claim-profile module meaningfully extend existing novelty-check comparisons. The advance is bounded by constructed tasks and statistical parity with CoI-Agent on whole-proposal performance.
The shared evaluation contract, component ablations, clustered tests, and robustness analyses support competent execution, while model-mediated labels and weak natural-duplicate detection limit construct validity. The retrieval-depth reporting conflict is surfaced through the Red reliability flag rather than a numerical cap.
The method-to-results sequence and separation of detection, attribution, and preservation outcomes make the argument traceable. Headline false-alarm and general capability wording nevertheless conveys broader validation than the proxy-labeled benchmark supplies.
Related work distinguishes traditional novelty measures, literature-grounded assessment, and AI-scientist pipelines, with relevant limitations and robustness checks included. The human audit's implications for interpreting the headline false-alarm rate are not carried through sufficiently.
Caveats4 of 4 checks
2 reported values do not reconcile, but the key claims do not rest on them.
Appendix analyses use 347 disguises without defining their relationship to the 380-disguise main analysis. Ethics and code/data availability declarations are present.
Ethics approval: declared · Data availability: declared · Code availability: declared
61 of 61 checkable references verified
63 references in manuscript 2 have no canonical index record — counted, but not index-checkable 2 references confirmed by manual review
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
Released implementation materials and retrospective benchmark tests establish the Demonstrated readiness stop. The small in-the-loop pilot cannot establish improved committed ideas or operational benefit.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Kubicek, Caciolai, Peng, et al. · 2026 · Computer and information sciences
At 10M tokens, one attention layer of D-LSA prefiles 22.7× and decodes 4.6× faster than sparse attention in GLM-5.3-FLASH, and 72.4× and 42.5× faster than Full Attention in QWEN-3.8-27B.
SupportedSon, Han, Koh, et al. · 2026 · Computer and information sciences
Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines.
SupportedLu, Chen, Wu, et al. · 2026 · Computer and information sciences
Across multiple datasets, task types, and LLMs, the Procedural Graph delivers consistent gains over memory-based baselines
Supported