# Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

- **DOI:** [10.3386/w35275](https://doi.org/10.3386/w35275)
- **Rating:** Strong (3.9 out of 5)
- **Verdict:** Large-scale GitHub evidence links AI coding gains to weak final-output translation, constrained by observational identification and proprietary telemetry.
- **Framework version:** v5.0
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-07-16
- **Evaluation page:** https://www.nabu.science/paper/10.3386%2Fw35275

## Strengths

- **Large multi-generation tool comparison** The study compares autocomplete, sync agents, and async agents in a single empirical framework using more than 100,000 GitHub developers. That scope lets the article address whether productivity effects grow across tool generations rather than measuring a single tool snapshot.
  - Dimension: contribution
  - Location: Abstract; Introduction; §5 Data and Summary Statistics
- **Design targets AI adoption contamination** The matched event-study design uses control outcomes from the same calendar week one year earlier to reduce contamination from unobserved private AI use among contemporaneous non-adopters. The article also reports activity matching, flat pre-trends, placebo tests with non-AI tools, and comparison to prior field-experimental autocomplete estimates.
  - Dimension: methodological rigour
  - Location: §6.1 Empirical Strategy; Figure 5; Table 3
- **Connects task gains to final output** The article does not stop at commits; it traces attenuation through files, pull requests, repositories, releases, and four app marketplaces. This makes the weak-link claim empirically meaningful by showing that large coding-activity gains correspond to smaller shipped-output and usage changes.
  - Dimension: application pathway
  - Location: Figure 1; Introduction marketplace summary; §4 Model of Software Production

## Limitations

- **Proprietary telemetry limits reproducibility** Key adoption measures for GitHub Copilot autocomplete and GitHub Sync Agent rely on confidential Microsoft usage or subscription data. The paper discloses this data source, but independent readers cannot inspect the treatment definitions or reproduce those portions of the analysis from public materials.
  - Dimension: methodological rigour
  - Location: §5.1 Microsoft Data; §5.2 Construction of Adoption Events
- **Observational design leaves residual selection** The matching design addresses activity bias and contamination, but adoption of different tools may still reflect unobserved developer characteristics and changing platform trends. The article itself notes that cross-tool differences could partly reflect adopter heterogeneity rather than tool effects.
  - Dimension: methodological rigour
  - Location: §5.2 Construction of Adoption Events; §6.1 Empirical Strategy
- **External validity remains software-specific** The evidence covers public GitHub activity and four app marketplaces but excludes important private, enterprise, and internal-only software production. The article also cautions that software is unusually advanced for generative AI adoption and that quantitative estimates may not extrapolate to other domains.
  - Dimension: application pathway
  - Location: §5.1 GitHub Data; Introduction limitations paragraph

## Reliability checks

Caveats (3 of 4 checks).

- **Internal coherence:** Noted. The main quantitative narrative is internally coherent, with headline attenuation patterns consistently distinguished across commits, repositories, releases, and marketplace usage. A notation is warranted because the Abstract and empirical-strategy sections use productivity-effect language for a matched observational event study, and some referenced supporting material is outside the supplied text.
- **Research conduct:** Noted. The study is transparent that key adoption measures rely on confidential Microsoft telemetry and public GitHub traces, but those proprietary inputs and the lack of visible code/data availability limit independent reproducibility. Conflict disclosure is only partially inspectable in the text, with details deferred to an external NBER page.
- **Reference integrity:** Not checked. Reference integrity check could not run: no references section detected in extracted text.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
