# The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis

> **Reliability concern:** This paper was retracted on 2026-04-22. See notice: 10.1057/s41599-026-07310-z.

- **DOI:** [10.1057/s41599-025-04787-y](https://doi.org/10.1057/s41599-025-04787-y)
- **Rating:** Sound (3.3 out of 5)
- **Verdict:** A timely ChatGPT-in-education meta-analysis with strict inclusion criteria, materially limited by the absence of formal risk of bias assessment.
- **Framework version:** v5.0
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-05-21
- **Evaluation page:** https://www.nabu.science/paper/10.1057%2Fs41599-025-04787-y

## Strengths

- **Strict and well-justified inclusion criteria** The requirement that both experimental and control groups use the same learning method, combined with the exclusion of studies where control groups used emerging technologies, improves the internal validity of comparisons over prior meta-analyses such as Deng et al. (2025).
  - Dimension: methodological rigour
  - Location: Methods, Inclusion and exclusion criteria; Table 1
- **Comprehensive moderator analysis across three outcomes** Six moderating variables are examined across all three outcome domains with high inter-rater reliability (α = 0.96) and a detailed coding scheme, providing a granular map of where ChatGPT effects are largest and under what conditions.
  - Dimension: contribution
  - Location: Tables 2, 5–7; Methods, Coding of study characteristics
- **Clear structure and transparent reporting** The paper follows PRISMA guidelines with a flow diagram, provides detailed study characteristics in Table 3, and uses established effect size benchmarks for interpretation. The argument progresses logically from identified gaps in prior work through methods, results, and practical implications.
  - Dimension: reporting
  - Location: Figure 1; Tables 3–7; Methods and Results sections

## Limitations

- **No formal risk of bias assessment** No risk of bias tool (e.g., Cochrane RoB 2, ROBINS-I) was applied to the 51 included studies, which are predominantly quasi-experimental. Without systematic quality assessment of individual studies, the pooled effect estimates cannot be interpreted with confidence regarding the reliability of the underlying evidence.
  - Dimension: methodological rigour
  - Location: Methods section (absent); not addressed in Discussion limitations
- **No evidence certainty framework applied** No GRADE or equivalent assessment was used to characterize the certainty of the pooled evidence, limiting the actionability of the practical recommendations for policymakers and practitioners who need to gauge how provisional the findings are.
  - Dimension: methodological rigour
  - Location: Discussion; Conclusion and implications
- **Small subgroup samples in moderator analyses** Several moderator categories contain only k = 2 studies (e.g., contextual learning for learning performance, intelligent tutor for higher-order thinking), making those effect estimates unreliable. The paper does not consistently flag this limitation when interpreting subgroup results or deriving specific recommendations such as the 4–8 week duration.
  - Dimension: positioning
  - Location: Tables 5–7; Conclusion and implications

## Reliability checks

Concerns (4 of 4 checks).

- **Internal coherence:** Noted. The paper is largely internally consistent across abstract, methods, tables, and discussion. However, a numerical discrepancy exists between the text-reported Q-value for learning performance (339.725) and the Table 4 value (399.725), a difference of 60 units likely reflecting a transcription error. The sensitivity analysis CI range for the random-effects model (0.683–1.052) matches the overall CI in Table 4, raising questions about the precision of the sensitivity analysis description.
- **Research conduct:** Noted. The paper describes PRISMA-guided selection with independent screening and high inter-rater reliability (α = 0.96). However, no preregistration identifier (e.g., PROSPERO) is provided, no explicit ethics statement is present (though this is standard for meta-analyses of published data), and no public data or code repository is referenced.
- **Reference integrity:** Clean. All 134 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Concern
- **Summary:** This paper was retracted on 2026-04-22. See notice: 10.1057/s41599-026-07310-z.
- **Last checked:** 2026-10-17