Loading evaluation data…
Abstract Generative AI (GenAI) tools, and large language models (LLMs) in particular, are being rapidly adopted across the scientific research process, including in funding evaluation and peer review. While their efficiency gains are well documented, their systematic biases remain comparatively understudied. A central concern is whether LLMs can recognise and value scientific novelty, an essential property for the long-run productivity of science. This paper examines 139 monodisciplinary research proposals submitted in 2021 to the Austrian Science Fund (FWF) in mathematics, physics and astronomy, chemistry, and biology. Each proposal carries expert-assigned scores for novelty, scientific quality, team capacity, and feasibility, along with the actual binary funding decision. Using GPT-4o, we generate AI-based funding decisions through within-field pairwise comparisons aggregated into field-level rankings, simulating a competitive selection process. AI-based funding decisions matched human funding decisions in 60% of cases. Conditional logistic regression with field fixed effects shows that, among the four review dimensions, novelty is the only one significantly associated with disagreement. Conditional and multinomial logit analyses further show that novelty systematically predicts the direction of disagreement. Proposals rated as more novel by human reviewers are substantially more likely to be funded by humans and not by the LLM. The observed pattern is consistent with a conservative tendency in LLM-based evaluation, potentially reflecting the statistical logic of next-token prediction trained on past scientific outputs. While the design does not allow us to identify the underlying mechanism, the results raise concerns that LLM-assisted evaluation may under-select proposals that human reviewers identify as highly novel.
Novelty is the only evaluation dimension that systematically predicts disagreement between the two evaluation systems.
setting-specific regressions support the association, but the single unablated evaluation pipeline limits inference
Proposals rated as more novel by human reviewers are substantially more likely to be funded by humans and not by the LLM.
directional models agree, though the key conditional analysis uses only 52 disagreement cases
AI-based funding decisions matched human funding decisions in 60% of cases.
the directly calculated Results figure is 62.6%, not the repeated 60% headline value
The observed pattern is consistent with a conservative tendency in LLM-based evaluation, potentially reflecting the statistical logic of next-token prediction trained on past scientific outputs.
the observed association is compatible with this account, but the design cannot test the proposed mechanism
Derived from the full evaluation — not a separate score.
Strengths
The empirical strategy distinguishes whether human and LLM decisions disagree from which system funds the proposal. Table 2 then tests the directional asymmetry using both the disagreement subset and the full sample.
↳ Data and methodology, “Empirical strategy”; Results, Table 2
Every proposal is compared with every other proposal in its field, pair order is randomised, and field-specific selectivity is preserved. Alternative Bradley–Terry and Elo aggregations produce highly similar rankings.
↳ Data and methodology, “Pairwise comparisons and ranking”; footnote 1
The Discussion identifies the small sample, single funder and model, fixed pipeline choices, potential novelty confounding, and lack of rubric guidance. The Conclusion correspondingly frames the finding as context-specific rather than general LLM conservatism.
↳ Discussion, “Limitations”; Conclusion
Limitations
Prompt framing, forced choice, model settings, and other pipeline choices were held constant. Without a rubric-guided comparison, the study cannot determine whether the novelty pattern persists when the LLM receives the criteria used by human reviewers.
↳ Discussion, “Limitations”
The analysis covers 139 monodisciplinary proposals from one funder and year using one GPT-4o configuration. This prevents confident extension to other agencies, disciplines, models, or prompting strategies.
↳ Data and methodology, “Data”; Discussion, “Limitations”
The Abstract, Discussion, and Conclusion state 60% agreement, while Results reports 87 of 139 cases, which equals 62.6%. The discrepancy is minor but affects the precision of a prominent descriptive result.
↳ Abstract; Results, “Overall agreement”; Discussion; Conclusion
The paper earns its strongest marks for contextual positioning, clear qualification, and the separation of disagreement from disagreement direction in Tables 1 and 2. Its methodological construction is transparent, including exhaustive pairwise comparisons, field-specific selection rates, held-out human scores, and alternative aggregation checks. Methodological Rigour is held below the stronger dimensions because the general prompt, forced-choice format, rubric omission, and model configuration were not systematically varied, leaving meaningful uncertainty about pipeline dependence. The context-specific framing prevents these limitations from undermining the reported association, but they block broader claims about LLM evaluation systems.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.6
Confidence highThe paper advances prior agreement-focused work by separating disagreement from its direction and identifying a novelty-specific pattern. The advance remains exploratory because it rests on 139 proposals from one funder, year, and model configuration.
“It identifies a structured pattern of disagreement within a specific funding context and model configuration.”
The exhaustive within-field tournament, held-out human scores, field-conditioned regressions, and alternative aggregation checks support the setting-specific association. The score is reduced because prompt framing, forced choice, model configuration, and rubric guidance were not systematically varied.
“These choices were held constant in the present analysis but not systematically varied.”
The paper clearly separates overall disagreement from directional disagreement and consistently qualifies its mechanism and generality claims. Numerical precision is weakened by reporting 60% in the Abstract and Conclusion but 62.6% in Results.
“the findings should be interpreted as an empirical indication from a specific funding context”
The paper engages directly relevant and complicating evidence on LLM evaluation, stylistic sensitivity, grant review, and human conservatism. Its limitations section traces the implications of sample, model, prompt, measurement, and generalisability constraints into the conclusion.
“rather than as definitive evidence of generalised LLM conservatism”
Lower confidence on Contribution, Methodological Rigour — domain match limited.
Caveats4 of 4 checks
The headline agreement rate is reported inconsistently, although the underlying Results calculation is arithmetically coherent and the discrepancy does not change the substantive novelty analysis.
The paper declares its ethics rationale, competing interests, confidentiality restrictions, and conditional availability of aggregated data and code. No text-grounded conduct concern was identified.
Flags: 4 declared / 5 total
25 of 25 checkable references verified
27 references in manuscript 1 have no canonical index record — counted, but not index-checkable 1 could not be checked — not the same as no match found 2 references confirmed by manual review
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The study provides an early warning from historical proposals and a simulated counterfactual selection process rather than a tested intervention. Deployment-ready guidance would require operational and rubric-guided validation.
“this choice limits what the study can say about rubric-guided LLM evaluation”
AI-generated, human-governed. Something look off? Contact us to request a review.