# Can large language models reproduce higher education grade bands? Cross-model study of calibration and grading bias in authentic student writing

- **DOI:** [10.1080/02602938.2026.2734796](https://doi.org/10.1080/02602938.2026.2734796)
- **Rating:** Sound (3.4 out of 5)
- **Verdict:** A useful matched-cohort benchmark reveals sharp cross-model grading bias, but single-run evidence from a two-band corpus limits broader grading conclusions.
- **Framework version:** v5.9
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-09-29
- **Evaluation page:** https://www.nabu.science/paper/10.1080%2F02602938.2026.2734796

## Key claims

1. performance differed significantly across model configurations with some models aligning more closely with the human-grades while others showed clear patterns of undergrading
   - Support: Well-supported
   - Type: Primary finding
   - Basis: paired tests and Tables 2–4 directly show substantial configuration differences and patterned undergrading
   - Location: Abstract; Results, Tables 2–4
2. In some cases, models predicted Pass or Fail despite the essays being human-labelled as Merit or Distinction, showing the risk of systematic grading bias under consistent assessment conditions
   - Support: Well-supported
   - Type: Secondary
   - Basis: Table 3 directly documents frequent below-reference-band predictions in several GPT-OSS configurations
   - Location: Abstract; Results, Table 3
3. large language models may be useful as supervised assessment-support tools, but they are not yet ready to be used independently for marking higher education student writing
   - Support: Supported
   - Type: Secondary
   - Basis: best-run performance remained modest, but one corpus and two reference bands limit the deployment conclusion
   - Location: Abstract; Discussion, Implications for supervised assessment support; Conclusion

## Strengths

- **Locked cohort enables direct comparisons** All eight configurations evaluate the same 114 essays, preventing shifting denominators and supporting paired Friedman, McNemar, and Wilcoxon analyses.
  - Dimension: methodological rigour
  - Location: Methodology, Analytic design and Statistical analysis
- **Bias metrics reveal consequential failures** Prediction distributions, grade-distance error, mean bias, and Distinction-specific metrics show that similar aggregate accuracy can conceal systematic undergrading.
  - Dimension: contribution
  - Location: Results, Tables 2–4 and Figures 2–3
- **Limitations constrain the interpretation** The paper explicitly reframes the task as Merit–Distinction discrimination and explains how the single corpus, common prompt, incomplete reasoning comparison, and absent moderation limit generalisation.
  - Dimension: positioning
  - Location: Limitations and future research

## Limitations

- **Single runs leave stochastic uncertainty unmeasured** Each model-temperature cell is represented by one output per essay, so run-to-run variability is not quantified, including at temperature 0.5.
  - Dimension: methodological rigour
  - Location: Methodology, Model configurations and Analytic design; Table 1
- **Model-family attribution remains partly confounded** The compared families differ simultaneously in parameter scale and reported reasoning settings, while reasoning effort is not analysed as an independent factor.
  - Dimension: methodological rigour
  - Location: Methodology, Model configurations; Table 1
- **Evidence transfers poorly beyond this cohort** The reference cohort contains only Merit and Distinction essays from one filtered corpus under one shared prompt, limiting conclusions about full-spectrum grading and other contexts.
  - Dimension: application pathway
  - Location: Methodology, Data source and essay selection; Limitations and future research

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. The reported statistics, tables, and matched-cohort analysis are internally consistent. The Abstract nevertheless creates mild scope overstatement by describing general university-assignment grading before the main text narrows the task to Merit–Distinction discrimination.
- **Research conduct:** Clean. The paper declares secondary use of de-identified corpus materials, reports no conflicts, and states that derived outputs and the locked-analysis script accompany the supplementary material. No supplied-text conduct concern was identified.
- **Reference integrity:** Clean. 25 of 25 checkable references verified
  - 26 references in manuscript
  - 1 have no canonical index record — counted, but not index-checkable
  - 1 reference confirmed by manual review
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
