Answer withholding adds a distinct outcome
The study separates the decision to answer from answer correctness, addressing an outcome not measured in the closest cognitive-surrender comparison.
↳ Introduction, pp. 2–3; Results, Studies 1a–4
Abstract
Knowing when to say "I don't know" is fundamental to human judgment, yet AI assistants offer a fluent answer to almost any question. In five experiments (N = 3,132; four preregistered, one direct replication), participants answered difficult questions and could always decline to respond. We engineered the questions so that AI advice was wrong, separating AI use from its accuracy. Merely having access to AI nearly eliminated participants' willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed. Consequently, participants answered more questions but were correct about a third as often as when AI was unavailable-yet their confidence nearly doubled. Incentivizing accuracy and penalizing inaccuracy led participants to seek and follow AI advice less, answer more accurately, and suspend judgment more often, though still far less than when AI was unavailable. As AI suggestions grow ubiquitous and unsolicited, they may not simply affect answer accuracy; they may even alter the metacognitive threshold at which people decide whether they know enough to answer.
Key Claims
Primary findingWell-supported
Merely having access to AI nearly eliminated participants’ willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed.
Likely to hold — the claim's weakest requirement is that it holds beyond the sample studied. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
large randomised suspension contrasts replicated across five experiments within the fixed film-question paradigm
Abstract
SecondaryTentative
Consequently, participants answered more questions but were correct about a third as often as when AI was unavailable—yet their confidence nearly doubled.
confidence was measured in only one study, while incompatible accuracy analyses qualify the compound quantitative claim
Abstract; Discussion, opening paragraph
SecondaryTentative
Incentivizing accuracy and penalizing inaccuracy led participants to seek and follow AI advice less, answer more accurately, and suspend judgment more often, though still far less than when AI was unavailable.
following advice was not directly measured, suspension effects varied, and conflicting accuracy analyses qualify the incentive inference
Abstract; Results, Studies 2–4
SecondaryTentative
AI availability and stakes acted largely independently: AI sharply reduced suspension regardless of stakes, while stakes modestly increased suspension
non-significant interactions on near-floor suspension outcomes do not establish independence
Results, Study 4, p. 10
SecondaryTentative
Because we engineered the questions so that the AI typically erred, this collapse cannot be read as sensible delegation to a reliable tool.
participants' knowledge of advice unreliability was not established, leaving rational delegation unresolved
Discussion, opening paragraph
Derived from the full evaluation — not a separate score.
Strengths · Limitations
Strengths
Randomised programme reports failed predictions openly
Four experiments are preregistered, one provides a direct replication, and the predicted AI-by-stakes interaction is reported as unsupported across Studies 2–4.
↳ Methods, Studies 1a–4; Results, Studies 2–4
Withholding effects persist under automatic advice
Study 4 extends the availability result to advice displayed without a participant request, connecting the experimental question to unsolicited AI answer surfaces.
↳ Results, Study 4; Figure 4a; Supplementary Tables S13–S14
Limitations
Accuracy analyses contain unresolved statistical conflicts
Tables S11–S12 disagree on the Study 3 interaction's sample size, coefficient, degrees of freedom and p-value. Table S16 also reports a sample size inconsistent with Table S15 and the Methods, qualifying the accuracy-and-incentive conclusions.
↳ Supplementary Tables S11–S12 and S15–S16; Methods, Studies 3–4
Mechanistic interpretation exceeds the experimental contrast
Participants were not shown to know that the advice was unreliable, so reduced withholding does not distinguish irrational deference from reliance on an apparently competent source. Near-floor suspension under AI also limits what null interaction tests establish.
↳ Introduction, p. 3; Methods, Studies 1a–4; Results, Studies 2–4; Discussion, pp. 11–14
Narrow task limits real world transfer
All studies use the same six film-detail questions and one model with deliberately unreliable advice. The discussion's broader domain-general interpretation is not tested by that repeated design.
↳ Methods, Questions common across studies; Discussion, limitations, p. 13
Nabu Commentary
The replicated randomised contrasts support a substantial effect of AI availability on answer withholding within this task. Four preregistrations and explicit reporting of failed interaction predictions strengthen the experimental record. However, Tables S11–S12 contain incompatible accuracy-interaction analyses, and Table S16 reports an inconsistent sample size, requiring reconciliation before the incentive findings can be relied upon quantitatively. The design does not establish a metacognitive mechanism, rule out rational reliance on an apparently competent source, or demonstrate transfer beyond the six film-detail questions.
Check another paper
Author Note / Field Feedback
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Dimension breakdown
Study Quality
Sound3.1
Confidence mediumContribution
3.8
The paper adds answer withholding as a distinct outcome and replicates a large AI-availability effect across five experiments. Its contribution is behavioural rather than a demonstrated change in metacognitive mechanism.
Introduction, pp. 2–3; Results, Studies 1a–4; Discussion, pp. 11–14
Methodological Rigour
2.5
Random assignment, four preregistrations, power calculations and a direct replication support the experimental programme. Incompatible accuracy analyses in the supplementary tables trigger the serious failure-class ceiling, while the primary suspension result remains separately supported.
Methods, Studies 1a–4; Supplementary Tables S11–S12 and S15–S16
Reporting
3.6
The experimental progression and failed preregistered interaction are presented explicitly. The reversed Study 4 accuracy sentence, misreported interaction p-value and inconsistent confidence multiplier reduce reporting precision.
Results, Studies 2–4, especially p. 10; Abstract; Discussion, opening paragraph
Positioning
3.5
The paper situates withholding within advice-taking, automation-bias and metacognitive research and identifies several boundaries. It does not adequately resolve rational delegation as an alternative interpretation or substantiate generalisation beyond the film-detail task.
Introduction, pp. 2–3; Discussion, pp. 11–14
Reliability
Concerns4 of 4 checks
Internal CoherenceConcern
Incompatible sample sizes and interaction statistics leave the provenance of parts of the accuracy analysis unresolved. The primary judgment-suspension result is not directly contradicted by these discrepancies.
Research ConductNoted
Ethics approval, consent and debriefing are not declared in the supplied human-participant study report. Preregistration, resource availability and competing-interest statements are provided.
Flags: 4 declared / 5 total
Reference IntegrityClean
9 references in manuscript 9 of 9 checkable references found in an index
Post-Publication RecordClean
No retraction notice found in Retraction Watch.
Last checked: 16 Oct 2026
Sources: Retraction Watch ✓
Readiness
Where this paper’s evidence sits on the path from initial observation to real-world use.
Demonstrated
The experiments characterise a behavioural effect under controlled conditions. They do not test an interface remedy or an AI-literacy intervention ready for implementation.
Methods, Studies 1a–4; Discussion, pp. 13–14
AI-generated, human-governed. Something look off? Contact us to request a review.