nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-0236F501·Framework v5.9·Pipeline v0.4·Evaluated 29 Sept 2026·OECD: 3.4 Medical and Health Sciences - Health Sciences (other)·Subfield: Developmental and Educational Psychology·Basis: Full text·Type: Computational simulation

Can large language models reproduce higher education grade bands? Cross-model study of calibration and grading bias in authentic student writing

Logayna Kerwat et al.Logayna Kerwat, Daniel L. Donaldson, Aziza Mahomed·2026·Assessment & Evaluation in Higher Education·10.1080/02602938.2026.2734796 ↗

Primary claim

performance differed significantly across model configurations with some models aligning more closely with the human-grades while others showed clear patterns of undergrading

Well-supported · paired tests and Tables 2–4 directly show substantial configuration differences and patterned undergrading
In brief

A useful matched-cohort benchmark reveals sharp cross-model grading bias, but single-run evidence from a two-band corpus limits broader grading conclusions.

ReliabilityCaveats4 of 4 checks completeRead the entry ↓
Possible — the claim's weakest requirement is that the effect exists at all. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract

Large Language Models (LLMs) are becoming increasingly used to support higher education assessment, yet evidence of their capability on reproducing authentic institutional grade-band labels remains limited. There is also limited understanding of how different model families vary in their grading behaviour, calibration, and tendency to produce systematic bias under consistent assessment conditions. This study examines whether LLMs can accurately grade university level student assignments using eight distinct pre-trained LLMs. The approach was tested using 114 human-marked university essays, and found that performance differed significantly across model configurations with some models aligning more closely with the human-grades while others showed clear patterns of undergrading. In some cases, models predicted Pass or Fail despite the essays being human-labelled as Merit or Distinction, showing the risk of systematic grading bias under consistent assessment conditions. Overall, the study indicates that large language models may be useful as supervised assessment-support tools, but they are not yet ready to be used independently for marking higher education student writing. Any future use in higher-stakes assessment would require further fine-tuning, a more directive assessment-specific knowledge base, and sustained human supervision.

Key Claims

Primary finding · Well-supported

performance differed significantly across model configurations with some models aligning more closely with the human-grades while others showed clear patterns of undergrading

Possible — the claim's weakest requirement is that the effect exists at all. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

paired tests and Tables 2–4 directly show substantial configuration differences and patterned undergrading

Abstract; Results, Tables 2–4
Quote verified
Secondary · Well-supported

In some cases, models predicted Pass or Fail despite the essays being human-labelled as Merit or Distinction, showing the risk of systematic grading bias under consistent assessment conditions

Table 3 directly documents frequent below-reference-band predictions in several GPT-OSS configurations

Abstract; Results, Table 3
Quote verified
Secondary · Supported

large language models may be useful as supervised assessment-support tools, but they are not yet ready to be used independently for marking higher education student writing

best-run performance remained modest, but one corpus and two reference bands limit the deployment conclusion

Abstract; Discussion, Implications for supervised assessment support; Conclusion
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#methodological rigour

Locked cohort enables direct comparisons

All eight configurations evaluate the same 114 essays, preventing shifting denominators and supporting paired Friedman, McNemar, and Wilcoxon analyses.

↳ Methodology, Analytic design and Statistical analysis

#contribution

Bias metrics reveal consequential failures

Prediction distributions, grade-distance error, mean bias, and Distinction-specific metrics show that similar aggregate accuracy can conceal systematic undergrading.

↳ Results, Tables 2–4 and Figures 2–3

#positioning

Limitations constrain the interpretation

The paper explicitly reframes the task as Merit–Distinction discrimination and explains how the single corpus, common prompt, incomplete reasoning comparison, and absent moderation limit generalisation.

↳ Limitations and future research

Limitations

#methodological rigour

Single runs leave stochastic uncertainty unmeasured

Each model-temperature cell is represented by one output per essay, so run-to-run variability is not quantified, including at temperature 0.5.

↳ Methodology, Model configurations and Analytic design; Table 1

#methodological rigour

Model-family attribution remains partly confounded

The compared families differ simultaneously in parameter scale and reported reasoning settings, while reasoning effort is not analysed as an independent factor.

↳ Methodology, Model configurations; Table 1

#application pathway

Evidence transfers poorly beyond this cohort

The reference cohort contains only Merit and Distinction essays from one filtered corpus under one shared prompt, limiting conclusions about full-spectrum grading and other contexts.

↳ Methodology, Data source and essay selection; Limitations and future research

Does this match your read? Add your assessment ↓

Nabu Commentary

The locked 114-essay cohort and paired analyses provide a sound basis for comparing the eight configurations, while Tables 2–4 make directional undergrading visible rather than relying on aggregate accuracy. The principal methodological deductions arise from single-run generation and from attributing differences to model family when parameter scale and reasoning settings also vary. The paper's literature engagement and limitation handling are stronger than its transferability, because the evaluation covers one filtered corpus, one prompt, and only Merit and Distinction reference labels. The Abstract's broad grading language therefore warrants tighter alignment with the narrower task established in the Methods and Limitations.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.4±0.1

Contribution
3.4

The study adds a matched cross-family comparison that reports directional bias and label-specific discrimination rather than aggregate agreement alone. The advance remains incremental because it evaluates 114 essays from one filtered corpus and two reference bands.

Introduction, research question; Related work; Results, Tables 2–4

The empirical task is therefore best understood as Merit–Distinction discrimination conducted within a four-band response space.

Methodological Rigour
3.2

The locked cohort and paired nonparametric analyses fit the repeated-comparison question, and the metric suite exposes error direction and grade-band performance. Confidence is limited by one generated output per configuration and by model-family comparisons that also vary parameter scale and reasoning setting.

Methodology, Model configurations and Statistical analysis; Table 1; Tables 2–5

Reasoning effort was not analysed as an independent experimental factor.

Reporting
3.6

The paper develops a followable chain from cohort construction through agreement, bias, discrimination, and temperature analyses, with limitations repeatedly stated. The Abstract's broad university-grading framing is less precise than the Merit/Distinction-only task established in the main text.

Abstract; Methodology, Measures; Limitations and future research

The findings should therefore be interpreted as Merit–Distinction discrimination within a four-band response space.

Positioning
3.8

The discussion engages both favourable and cautionary findings in the recent LLM-grading literature and positions the results as context dependent. The limitations section connects the restricted labels, corpus, prompt, reasoning settings, and lack of new moderation to the permissible interpretation.

Related work; Discussion; Limitations and future research

Results may differ for other disciplines, writing genres, marking rubrics, or prompt designs.

Lower confidence on Contribution, Methodological Rigour — domain match limited.

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The reported statistics, tables, and matched-cohort analysis are internally consistent. The Abstract nevertheless creates mild scope overstatement by describing general university-assignment grading before the main text narrows the task to Merit–Distinction discrimination.

Research ConductClean

The paper declares secondary use of de-identified corpus materials, reports no conflicts, and states that derived outputs and the locked-analysis script accompany the supplementary material. No supplied-text conduct concern was identified.

Flags: 4 declared / 5 total

Reference IntegrityClean

25 of 25 checkable references verified

26 references in manuscript 1 have no canonical index record — counted, but not index-checkable 1 reference confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 4
Type of advanceEmpirical
Route to useNamed
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The findings come from a single proof-of-concept benchmark of eight model configurations on 114 authentic essays in one corpus, with no independent replication or real-world deployment.

From the paper
“The findings support a restrained use case for LLMs in higher education assessment. They may assist as supervised support tools”

The paper names the setting (higher education assessment) and the concrete use (supervised grading and moderation support) for its findings, while warning against autonomous use.

Pathway

Universities and assessment policymakers would use these findings to select well-calibrated model configurations and mandate human supervision for any LLM-assisted marking or moderation.

Barrier

Even the best model achieved only moderate agreement on a single corpus with only Merit and Distinction labels, so institutions cannot yet trust these tools without further validation and fine-tuning.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Compositional processing in the recognition of Chinese compounds: Behavioural and computational studies

    Hsieh, Marelli, Rastle · 2025 · Psychology and cognitive sciences

    Primary claim

    These results suggest that a compositional process is involved in Chinese compound processing, even in tasks that do not explicitly require meaning combination.

    Tentative
    Evidence Score
    3.4±0.2
    Reliability
    Caveats
  2. “I don’t need validation. I have a mirror, and i know myself”: Unravelling the tapestry of online dating experiences and self-construction in queer women

    Sienna Crawford, Matthew James Phillips, Scott Devenport · 2026 · Psychology and cognitive sciences

    Primary claim

    Dating apps simultaneously offer queer women unprecedented access to community while embedding the very structures that marginalise them into every swipe, every profile category, and every algorithmic decision.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats
  3. Exploring Lay Understandings of Romantic Chemistry Using Inductive and Deductive Content Analysis

    Devenport, Phillips, Mullan, et al. · 2025 · Psychology

    Primary claim

    Our findings suggest that romantic chemistry is only perceivable when multiple facets are experienced and that experiences of facets vary individually

    Tentative
    Evidence Score
    3.6±0.2
    Reliability
    Caveats

How the rubric works →