Reliability concern: This paper was retracted on 2026-04-22. See notice: 10.1057/s41599-026-07310-z.
As a new type of artificial intelligence, ChatGPT is becoming widely used in learning. However, academic consensus regarding its efficacy remains elusive. This study aimed to assess the effectiveness of ChatGPT in improving students’ learning performance, learning perception, and higher-order thinking through a meta-analysis of 51 research studies published between November 2022 and February 2025. The results indicate that ChatGPT has a large positive impact on improving learning performance (g = 0.867) and a moderately positive impact on enhancing learning perception (g = 0.456) and fostering higher-order thinking (g = 0.457). The impact of ChatGPT on learning performance was moderated by type of course (QB = 64.249, P < 0.001), learning model (QB = 76.220, P < 0.001), and duration (QB = 55.998, P < 0.001); its effect on learning perception was moderated by duration (QB = 19.839, P < 0.001); and its influence on the development of higher-order thinking was moderated by type of course (QB = 7.811, P < 0.05) and the role played by ChatGPT (QB = 4.872, P < 0.05). This study suggests that: (1) appropriate learning scaffolds or educational frameworks (e.g., Bloom’s taxonomy) should be provided when using ChatGPT to develop students’ higher-order thinking; (2) the broad use of ChatGPT at various grade levels and in different types of courses should be encouraged to support diverse learning needs; (3) ChatGPT should be actively integrated into different learning modes to enhance student learning, especially in problem-based learning; (4) continuous use of ChatGPT should be ensured to support student learning, with a recommended duration of 4–8 weeks for more stable effects; (5) ChatGPT should be flexibly integrated into teaching as an intelligent tutor, learning partner, and educational tool. Finally, due to the limited sample size for learning perception and higher-order thinking, and the moderately positive effect, future studies with expanded scope should further explore how to use ChatGPT more effectively to cultivate students’ learning perception and higher-order thinking.
Strengths
The requirement that both experimental and control groups use the same learning method, combined with the exclusion of studies where control groups used emerging technologies, improves the internal validity of comparisons over prior meta-analyses such as Deng et al. (2025).
↳ Methods, Inclusion and exclusion criteria; Table 1
Six moderating variables are examined across all three outcome domains with high inter-rater reliability (α = 0.96) and a detailed coding scheme, providing a granular map of where ChatGPT effects are largest and under what conditions.
↳ Tables 2, 5–7; Methods, Coding of study characteristics
The paper follows PRISMA guidelines with a flow diagram, provides detailed study characteristics in Table 3, and uses established effect size benchmarks for interpretation. The argument progresses logically from identified gaps in prior work through methods, results, and practical implications.
↳ Figure 1; Tables 3–7; Methods and Results sections
Limitations
No risk of bias tool (e.g., Cochrane RoB 2, ROBINS-I) was applied to the 51 included studies, which are predominantly quasi-experimental. Without systematic quality assessment of individual studies, the pooled effect estimates cannot be interpreted with confidence regarding the reliability of the underlying evidence.
↳ Methods section (absent); not addressed in Discussion limitations
No GRADE or equivalent assessment was used to characterize the certainty of the pooled evidence, limiting the actionability of the practical recommendations for policymakers and practitioners who need to gauge how provisional the findings are.
↳ Discussion; Conclusion and implications
Several moderator categories contain only k = 2 studies (e.g., contextual learning for learning performance, intelligent tutor for higher-order thinking), making those effect estimates unreliable. The paper does not consistently flag this limitation when interpreting subgroup results or deriving specific recommendations such as the 4–8 week duration.
↳ Tables 5–7; Conclusion and implications
Withdrawn — shown for reference only
The meta-analysis competently executes core procedures — strict study selection, independent coding with high reliability, appropriate random-effects modeling — and addresses a topic of genuine urgency in education. The primary tension in the evaluation is between this competent execution and two significant methodological omissions: the lack of formal risk of bias assessment for included studies and the absence of an evidence certainty framework such as GRADE. These are standard expectations for meta-analyses published in 2025 and their absence limits how much confidence practitioners can place in the specific recommendations. The Methodological Rigour score reflects this tension, falling in the range where sound fundamentals are partially offset by meaningful gaps. A numerical discrepancy between the text-reported and table-reported Q-value for learning performance (339.725 vs. 399.725) constitutes a minor but notable internal inconsistency. The paper's practical recommendations, while specific and actionable, occasionally extend beyond what the moderator analyses with small subgroup sizes can robustly support.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.3
Confidence highThe paper synthesizes 51 studies across three outcome domains with stricter inclusion criteria than prior meta-analyses and examines six moderating variables, representing a genuine incremental advance over existing work. The contribution is bounded by the rapidly growing literature on this topic and the relatively small k for learning perception (k=19) and higher-order thinking (k=9).
Core meta-analytic procedures are competently executed: PRISMA-guided selection, independent coding with high reliability (α = 0.96), random-effects modeling with Hedges's g, and moderator analyses across six variables. However, the absence of formal risk of bias assessment (e.g., Cochrane RoB 2) and evidence certainty grading (e.g., GRADE), combined with reliance on the dated fail-safe N for publication bias, represent meaningful methodological gaps for a 2025 meta-analysis that limit interpretability of findings.
The paper follows a logical structure from research questions through methods, results, and discussion, with clearly formatted tables (Tables 3–7), a PRISMA flow diagram (Figure 1), and appropriate use of effect size benchmarks. Minor clarity issues include a Q-value discrepancy between text and Table 4 and potentially imprecise sensitivity analysis reporting.
The literature review engages substantively with prior meta-analyses and positions the contribution against their specific limitations. The discussion connects findings to constructivist learning theory and the technology acceptance model. However, the limitation discussion does not address the absent risk of bias assessment or discuss confounding in quasi-experimental designs (e.g., novelty effects).
Concerns4 of 4 checks
The paper is largely internally consistent across abstract, methods, tables, and discussion. However, a numerical discrepancy exists between the text-reported Q-value for learning performance (339.725) and the Table 4 value (399.725), a difference of 60 units likely reflecting a transcription error. The sensitivity analysis CI range for the random-effects model (0.683–1.052) matches the overall CI in Table 4, raising questions about the precision of the sensitivity analysis description.
The paper describes PRISMA-guided selection with independent screening and high inter-rater reliability (α = 0.96). However, no preregistration identifier (e.g., PROSPERO) is provided, no explicit ethics statement is present (though this is standard for meta-analyses of published data), and no public data or code repository is referenced.
Flags: 0 declared / 5 total
All 134 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.
References not verified automatically are resolved manually within 24 hours.
This paper was retracted on 2026-04-22. See notice: 10.1057/s41599-026-07310-z.
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The findings provide aggregated evidence with specific moderator-based recommendations, but the absence of risk of bias assessment and evidence certainty grading means practitioners cannot easily calibrate confidence in the recommendations. The work sits at the level of provisional synthesis rather than deployment-ready guidance.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Paul Sheals, Tim de Rosen · 2026 · Economics and business
such an expert [a seller-side expert that visiting AI agents consult] cannot be built from product knowledge alone
TentativeKubicek, Caciolai, Peng, et al. · 2026 · Computer and information sciences
At 10M tokens, one attention layer of D-LSA prefiles 22.7× and decodes 4.6× faster than sparse attention in GLM-5.3-FLASH, and 72.4× and 42.5× faster than Full Attention in QWEN-3.8-27B.
SupportedLiu, Zhao, Xie, et al. · 2026 · Computer and information sciences
CLAIMNOV catches 97% of disguised papers at a 1.5% false-alarm rate
Supported