Loading evaluation data…
Relative to baseline (no System 3 access), accuracy significantly rose when AI was accurate and fell when it erred (+25/-15 percentage points; Study 1)
directly identified by randomized AI accuracy and replicated in the trial-level synthesis
Participants chose to consult an AI assistant on a majority of trials (>50%).
directly measured chat-use rates exceed half of trials in the reported studies
A key prediction of the theory is “cognitive surrender”—adopting AI outputs with minimal scrutiny, overriding intuition (System 1) and deliberation (System 2).
consultation and following are measured, but minimal scrutiny and absent deliberation are inferred
Engaging System 3 also increased confidence, even following errors.
AI access increased confidence, though Study 3 shows confidence did not vary by Trial Type
Across studies, participants with higher trust in AI and lower need for cognition and fluid intelligence showed greater surrender to System 3.
trial-level synthesis supports the direction with some study-level variation in component effects
Derived from the full evaluation — not a separate score.
Strengths
The experiments randomize whether the embedded GPT-4o assistant returns correct or confidently wrong answers while preserving participant choice to consult or ignore it. This design directly supports the empirical claim that performance rises or falls with AI output quality.
↳ Empirical Overview; Common Procedures; Study 1 Methods
The core pattern appears in Study 1 and is replicated under time pressure and incentives plus feedback. The trial-level synthesis reports large AI-Accurate versus AI-Faulty differences across N = 1,372 and 9,593 trials.
↳ Cognitive Surrender Effect Size; Figure 10
The paper links greater susceptibility to trust in AI and lower need for cognition or fluid intelligence, while showing that incentives and feedback increase faulty-AI override. These findings provide plausible design levers for follow-up applied work.
↳ Study 3 Results; Individual Differences in Susceptibility to Cognitive Surrender
Limitations
The construct of cognitive surrender is defined as adoption with minimal scrutiny or no deliberation, but the primary measures are consultation, follow/override behavior, accuracy, and confidence. This supports reliance on AI outputs more directly than it proves absence of internal deliberation.
↳ Cognitive Surrender section; Key Variables; General Discussion
All experiments use seven adapted CRT items with known correct answers and seeded AI responses. Transfer to expert, ambiguous, longitudinal, or high-stakes decision contexts remains an open empirical question.
↳ Empirical Overview; Common Procedures; Societal Implications
Claims about a third cognitive system and a triadic cognitive ecology are broader than evidence from a single reasoning-task paradigm can establish. The framework is promising, but the data more securely demonstrate AI advice reliance than a structural revision of cognition.
↳ System 3 (Artificial): A New Cognitive Model; Table 2; General Discussion
The article’s strongest evidence comes from the randomized AI-accuracy manipulation and repeated finding that participant accuracy improves when AI is correct and declines when AI is faulty. The mixed-effects analyses, confidence intervals, and trial-level synthesis make the empirical reliance pattern credible within the CRT paradigm. Scores are constrained by construct validity: the article defines cognitive surrender as minimal scrutiny or bypassed deliberation, but mostly observes consultation, adoption, accuracy, and confidence. The practical implications are plausible because incentives and feedback improve calibration, but translation to real-world decision support requires studies beyond seeded CRT items.
Strong3.9/5.0
The contribution score reflects a substantive empirical and theoretical package: a Tri-System framework paired with three preregistered experiments and a trial-level synthesis. The score is held below the highest range because the strongest architectural and minimal-scrutiny claims extend beyond the direct behavioral measures.
“We introduce Tri-System Theory”
The craft score reflects a well-aligned experimental design using baseline comparisons, within-subject AI-accuracy manipulation, mixed-effects models, and replication across three studies. Limitations come from construct validity for minimal scrutiny, a narrow CRT task domain, post-hoc thinking-profile classifications, and key implementation details deferred to web appendices.
“randomized AI accuracy via hidden seed prompts”
The paper is organized clearly from theory to hypotheses, common procedures, three studies, and synthesis, with figures and tables anchoring the empirical story. The score is tempered because terms such as cognitive surrender, bypass, and inflated confidence sometimes compress more nuance than the results support.
“confidence did not vary significantly by Trial Type”
The context score reflects substantial engagement with dual-process theory and adjacent constructs including cognitive offloading, extended mind, automation bias, and algorithmic aversion/appreciation. It is not higher because the paper does not decisively separate Tri-System Theory from alternative advice-taking or automation-bias explanations in the empirical design.
“extends prior models of distributed cognition”
Watch-outs3 of 4 checks
The numerical reporting and main results are internally consistent, but some construct and scope language is stronger than the behavioral measures directly establish. The concern is overinterpretation rather than statistical impossibility or methods-results contradiction.
The paper declares IRB approval, informed consent/debriefing, preregistration, and OSF availability, with no text-grounded conduct concern identified. Repository contents, appendices, and preregistration conformity remain assessability gaps rather than conduct concerns.
Flags: 3 declared / 5 total
Reference integrity check could not run: no references section detected in extracted text.
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓PubPeer (coming soon)
Medium3.4/5.0
The traction score reflects a timely problem with identifiable stakeholders in AI design, education, policy, and decision-support deployment. The article’s relevance is broad and current, though not tied to a single operational workflow.
“designers and policymakers concerned with AI’s ethical and practical integration”
The translation score reflects proof-of-concept evidence in controlled lab and online experiments rather than field validation. The paper suggests mechanisms such as feedback and incentives, but implementation guidance for real-world systems remains preliminary.
“giving feedback and aligning incentives may help”
The transferability score reflects replication across three studies and lab/online samples, but all studies use a seven-item adapted CRT paradigm with optional GPT-4o consultation. Generalization to ambiguous, expert, longitudinal, or high-stakes settings remains uncertain.
“seven Cognitive Reflection Test (CRT-7) items”
The trajectory score reflects a plausible cumulative path linking cognitive theory, human-AI interaction, and intervention design. The framework can guide future studies of feedback, incentives, interface friction, and calibrated AI reliance.
“ethical design principles that preserve System 2 engagement”
Nabu’s assessment, alongside the field’s view.
Evaluation Metadata
AI-powered, human-governed. Something look off? Contact us to request a review.