nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-B838AA77·Framework v5.10·Pipeline v0.4·Evaluated 04 Nov 2026·OECD: 5.1 Psychology and cognitive sciences·Basis: Full text·Type: Field experiment

Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls

Amaç Herdağdelen et al.Amaç Herdağdelen, Hailey Joren, Alex Tamkin·10.0000/anthropic-whos-aligned-to-whom-ai-agents ↗

Primary claim

On the analytical decisions, participants went with the agent's lean 86% of the time (50% by chance)

Well-supported · randomized preferences and consistent primary estimates support the observed alignment rate within this task
In brief

Randomized agent preferences reveal substantial influence on practitioners' analytical choices, but one synthetic task limits conclusions about real expert workflows.

ReliabilityCaveats4 of 4 checks completeRead the entry ↓
Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.
Abstract not available — view at DOI ↗

Key Claims

Primary finding · Well-supported

On the analytical decisions, participants went with the agent's lean 86% of the time (50% by chance)

Possible — the claim's weakest requirement is that the effect exists at all. Based on how the study was designed and conducted, not on statistics extracted from the paper. Experimental. Every point below is traceable to a quoted passage, but this forecast has not yet been validated against known replication outcomes.

randomized preferences and consistent primary estimates support the observed alignment rate within this task

Abstract
Quote verified
Secondary · Supported

the four intermediate leans alone correctly predict 82% of delivered recommendations among participants who did not choose an extension (61% majority baseline; AUC 0.84)

exploratory in-sample regression in a small single-task sample rather than independent predictive validation

Section 1.1, Summary of Our Results; Section 4.1
Quote verified
Secondary · Supported

Yet when asked to justify their choices, 77% repeated the agent's talking points, sometimes nearly verbatim, and only 15% mentioned the agent in any answer.

descriptive coding supports repetition and attribution rates, but the shared briefing prevents identifying unrecognized deferral

Abstract; Section 4.2.2
Quote verified
Secondary · Supported

more than half [of participants] rated the recommendation as more their own than the agent's

direct ownership self-reports describe perceived authorship rather than independently measured control

Abstract; Section 4.2.2, Figure 3
Quote verified
Secondary · Supported

The shift on the policy recommendation was smaller and not statistically significant under the registered primary estimator: 17% of the distance from discontinuing to rolling out (22% unadjusted).

randomized comparison supports the qualified estimate, but the small sample leaves a wide confidence interval

Section 1.1, Summary of Our Results; Table 2
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Randomized preferences identify influence on analytical choices

Hidden, randomly assigned agent preferences allow a causal comparison without labeling analytical options right or wrong. The primary pooled arm contrast is 0.72 with a 95% confidence interval of 0.60–0.83.

↳ Sections 3.3, 3.5, and 4.1; Table 2

#methodological rigour

Inference and hypothesis handling match the design

The primary analysis uses participant-clustered CR2 inference for repeated decisions. H3 is explicitly downgraded after its falsification check fails, and the non-significant registered policy estimate is distinguished from the significant unadjusted result.

↳ Section 3.5; Table 2

#positioning

Limitations are connected to their interpretive consequences

The discussion explains how amplified preferences, a single setting, and average-only identification constrain the findings. It also acknowledges that agents can curb analytical error rather than treating all expert discretion as beneficial.

↳ Sections 2 and 5

Limitations

#methodological rigour

Nonuser exclusions leave selection uncertainty after assignment

Eleven of 61 valid completers never used the agent and were excluded from the main analysis. The absence of arm-wise take-up comparisons leaves uncertainty about whether the included users remain comparable across assigned configurations.

↳ Sections 3.1 and 3.3

#reporting

Exploratory measures do not establish unrecognized deferral

Talking-point repetition can reflect the shared task briefing, and participants were asked to justify choices rather than attribute their sources. The 82% recommendation-prediction result is also in-sample rather than an independently validated predictive result.

↳ Abstract; Sections 4.1 and 4.2.2; Appendix A.5

#application pathway

One amplified setting limits real workflow generalization

A single synthetic task and pinned agent cannot establish how frequently comparable influence occurs in professional settings with stronger domain knowledge and stakeholder input. No proposed countermeasure is tested.

↳ Sections 5–6

Does this match your read? Add your assessment ↓

Nabu Commentary

Randomized agent preferences and participant-clustered inference provide a clear test of influence on analytical choices without requiring uniquely correct answers. The registered hypothesis structure is handled transparently, including the downgrade of H3 and separation of the adjusted policy estimate from the unadjusted result. Excluding nonusers without arm-wise take-up analysis leaves a selection concern, while registration after data collection began warrants a transparency note. Talking-point repetition and in-sample prediction are informative exploratory observations, but neither establishes the broader interpretation of unrecognized deferral on its own. The discussion traces the main limits appropriately, leaving replication across real workflows and evaluation of countermeasures as the principal next steps.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
4.0±0.1

Contribution
4.2

Randomizing defensible agent preferences supplies causal evidence about influence on underdetermined analytical choices, extending reliance research beyond tasks with correct answers. The advance remains bounded by one synthetic task and one deliberately amplified agent configuration.

Sections 1–2, 3.3, 4.1, and 5
Methodological Rigour
3.9

Balanced randomized assignment, participant-clustered CR2 inference, and the explicit downgrade of H3 support the main analysis. Excluding 11 nonusers without arm-wise take-up analysis leaves selection uncertainty, alongside disclosed late registration and limited independent validation of behavioral coding.

Sections 3.1, 3.3, 3.5–3.6; Table 2
Reporting
4.1

The text distinguishes the primary causal estimate, the non-significant adjusted policy result, conditional switching evidence, and exploratory prediction. The abstract compresses the limits of talking-point repetition as evidence of unrecognized influence, and does not foreground the assumption behind interpreting net influence as a flip rate.

Abstract; Sections 1.1, 3.5, 4.1, and 4.2.2
Positioning
4.0

The background engages reliance, automation bias, researcher freedom, cognitive offloading, and disempowerment, while the discussion traces the three main limitations to interpretation. Human advice-taking receives less direct treatment, and implications for institutional and alignment workflows extend beyond the tested setting.

Sections 2, 5, and 6

Lower confidence on Methodological Rigour, Positioning — domain match limited.

Reliability

Caveats4 of 4 checks


Results checkClean

The reported numbers and statistics are consistent with the stated results.

Reporting and transparencyNoted

The pre-analysis plan was registered after data collection began, with the timing and exceptions to the analysis freeze disclosed. Ethics review and participant consent are declared.

Ethics approval: declared · Preregistration: declared

Reference IntegrityClean

47 references in manuscript 43 of 43 checkable references found in an index 4 have no canonical index record — counted, but not index-checkable 1 reference confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 04 Nov 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 9
Type of advanceEmpirical
Route to useNamed
AdvanceAdvances
CitationsNot available
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessPiloted
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim is demonstrated in one controlled randomized trial with 50 practitioners on a single task and single agent, with no independent replication in other settings.

From the paper
“They will need to decide deliberately which decisions need a person's own informed judgment, and give people the time and resources to form one.”

The paper names institutions and agent designers as actors who must change oversight practices and agent design, but does not specify a concrete decision procedure or deployment setting.

Pathway

AI developers and research or oversight institutions would redesign agent behavior and sign-off procedures so judgment calls are surfaced and experts articulate their own assumptions.

Barrier

The evidence comes from one amplified-lean setting with a single agent and task, and no validated method yet exists to measure deferral in real-world work.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Compositional processing in the recognition of Chinese compounds: Behavioural and computational studies

    Hsieh, Marelli, Rastle · 2025 · Psychology and cognitive sciences

    Primary claim

    These results suggest that a compositional process is involved in Chinese compound processing, even in tasks that do not explicitly require meaning combination.

    Tentative
    Evidence Score
    3.4±0.2
    Reliability
    Caveats
  2. Pain as an assault on the self: An interpretative phenomenological analysis of the psychological impact of chronic benign low back pain

    Jonathan Ashley Smith, Mike Osborn · 2006 · Psychology and cognitive sciences

    Primary claim

    The notion of the self emerged in this study as an important aspect of the participants’ experience of chronic pain sensation, distress and disability.

    Supported
    Evidence Score
    3.5±0.1
    Reliability
    No concerns
  3. “I don’t need validation. I have a mirror, and i know myself”: Unravelling the tapestry of online dating experiences and self-construction in queer women

    Sienna Crawford, Matthew James Phillips, Scott Devenport · 2026 · Psychology and cognitive sciences

    Primary claim

    Dating apps simultaneously offer queer women unprecedented access to community while embedding the very structures that marginalise them into every swipe, every profile category, and every algorithmic decision.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Caveats

How the rubric works →