nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-BB4F3057·Framework v5.0·Pipeline v0.4·Evaluated 16 Jul 2026·OECD: 2.3 Mechanical engineering·Subfield: Information Systems·Basis: Full text·Type: Observational cohort study

Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

Mert Demirer et al.Mert Demirer, Leon Musolff, Liyuan Yang·2026·10.3386/w35275 ↗

In brief

Large-scale GitHub evidence links AI coding gains to weak final-output translation, constrained by observational identification and proprietary telemetry.

ReliabilityCaveats3 of 4 checks completeRead the 2 entries ↓
Abstract not available — view at DOI ↗

Strengths · Limitations

Strengths

#contribution

Large multi-generation tool comparison

The study compares autocomplete, sync agents, and async agents in a single empirical framework using more than 100,000 GitHub developers. That scope lets the article address whether productivity effects grow across tool generations rather than measuring a single tool snapshot.

↳ Abstract; Introduction; §5 Data and Summary Statistics

#methodological rigour

Design targets AI adoption contamination

The matched event-study design uses control outcomes from the same calendar week one year earlier to reduce contamination from unobserved private AI use among contemporaneous non-adopters. The article also reports activity matching, flat pre-trends, placebo tests with non-AI tools, and comparison to prior field-experimental autocomplete estimates.

↳ §6.1 Empirical Strategy; Figure 5; Table 3

#application pathway

Connects task gains to final output

The article does not stop at commits; it traces attenuation through files, pull requests, repositories, releases, and four app marketplaces. This makes the weak-link claim empirically meaningful by showing that large coding-activity gains correspond to smaller shipped-output and usage changes.

↳ Figure 1; Introduction marketplace summary; §4 Model of Software Production

Limitations

#methodological rigour

Proprietary telemetry limits reproducibility

Key adoption measures for GitHub Copilot autocomplete and GitHub Sync Agent rely on confidential Microsoft usage or subscription data. The paper discloses this data source, but independent readers cannot inspect the treatment definitions or reproduce those portions of the analysis from public materials.

↳ §5.1 Microsoft Data; §5.2 Construction of Adoption Events

#methodological rigour

Observational design leaves residual selection

The matching design addresses activity bias and contamination, but adoption of different tools may still reflect unobserved developer characteristics and changing platform trends. The article itself notes that cross-tool differences could partly reflect adopter heterogeneity rather than tool effects.

↳ §5.2 Construction of Adoption Events; §6.1 Empirical Strategy

#application pathway

External validity remains software-specific

The evidence covers public GitHub activity and four app marketplaces but excludes important private, enterprise, and internal-only software production. The article also cautions that software is unusually advanced for generative AI adoption and that quantitative estimates may not extrapolate to other domains.

↳ §5.1 GitHub Data; Introduction limitations paragraph

Does this match your read? Add your assessment ↓

Nabu Commentary

The scores are driven by a strong match between the research question and a large matched event-study design, especially the attempt to avoid contaminated contemporaneous controls in §6.1. The contribution is strongest where the article links task-level coding gains to attenuation across the production hierarchy in Figure 1 and §4, rather than treating commits as final productivity. Methodological Rigour is held below the highest range because confidential Microsoft telemetry in §5.1–§5.2, public-repository-only measurement, and residual adopter heterogeneity limit independent verification and causal interpretation. Impact potential is high for expectation-setting in software and policy debates, but the article’s own limitations prevent broad deployment or cross-sector generalization.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.9±0.2

Contribution
4.0

The score reflects a timely empirical contribution comparing autocomplete, sync agents, and async agents across a large GitHub developer sample, combined with a weak-link production interpretation. The contribution is bounded because the framework applies established theory to a new setting and some headline causal and structural claims rest on observational evidence.

Introduction; §4 Model of Software Production; Figure 1

provide one of the first empirical tests of this hypothesis

Methodological Rigour
3.7

The matched event-study design, year-lagged controls, pre-trend checks, placebo logic, and robustness comparisons are appropriate for the observational platform-data setting. The score is limited by residual adoption selection, untested stationarity assumptions in one-year-lag controls, public-repository scope, and confidential telemetry for key exposure definitions.

§5.1 Data Sources; §5.2 Construction of Adoption Events; §6.1 Empirical Strategy

control group outcomes from exactly one year before

Reporting
4.1

The article presents a clear sequence from research questions to production model, data construction, event-study design, and headline results. Precision is reduced by recurring productivity-effect language for an observational design and by the supplied text ending before all later empirical sections and appendices are available.

Abstract; Introduction; Figure 1; §6.1 Empirical Strategy

The remainder of the paper is organized as follows

Positioning
4.1

The paper situates itself across AI productivity, task-based automation, weak-link production, micro-to-aggregate productivity translation, and digital marketplace entry. It also explicitly discusses limitations around software quality, enterprise/internal software coverage, and extrapolation beyond software.

§2 Literature; Introduction limitations paragraph

Our work relates to four strands of literature

Reliability

Caveats3 of 4 checks


Internal CoherenceNoted

The main quantitative narrative is internally coherent, with headline attenuation patterns consistently distinguished across commits, repositories, releases, and marketplace usage. A notation is warranted because the Abstract and empirical-strategy sections use productivity-effect language for a matched observational event study, and some referenced supporting material is outside the supplied text.

Research ConductNoted

The study is transparent that key adoption measures rely on confidential Microsoft telemetry and public GitHub traces, but those proprietary inputs and the lack of visible code/data availability limit independent reproducibility. Conflict disclosure is only partially inspectable in the text, with details deferred to an external NBER page.

Flags: 1 declared / 5 total

Reference IntegrityNot checked

Reference integrity check could not run: no references section detected in extracted text.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaEconomy · SDG 8
Type of advanceEmpirical
Route to useNot stated
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The primary claim is supported by matched event studies on over 100,000 real-world developers and corroborated across four independent app marketplaces, though the design remains observational.

The paper frames its findings as informing understanding of AI's economic impact and future mechanisms, but names no concrete real-world use, actor, or decision for its own results.

Pathway

Economists, forecasters, and policymakers assessing AI's aggregate productivity effects would use these attenuation estimates and the elasticity of substitution to calibrate expectations.

Barrier

The observational design, inability to distinguish low app quality from consumer-side bottlenecks, and the paper's own caveat that software may not extrapolate to other domains limit broader application.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, et al. · 2023 · Computer and information sciences

    Evidence Score
    3.5±0.2
    Reliability
    Caveats
  2. Artificial intelligence in language learning: biometric feedback and adaptive reading for improved comprehension and reduced anxiety

    Hang Yuan · 2025 · Educational sciences

    Primary claim

    Quantitative results revealed significant improvements in reading comprehension, motivation, anxiety reduction, and cognitive load management among participants using the AI platform compared to the control group.

    Tentative
    Evidence Score
    2.2±0.2
    Reliability
    Concerns
  3. Blind spots in AI-assisted healthcare evidence search: multiplatform evaluation of clinical retrieval gaps and risk-of-bias

    Farrag, Soliman, Azubuike, et al. · 2026 · Computer and information sciences

    Primary claim

    Median formulation-level recall ranged from 7.2% to 42.2%, while pooled platform recall ranged from 45.8% to 72.3%.

    Well-supported
    Evidence Score
    3.9±0.2
    Reliability
    No concerns

How the rubric works →