nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-BE957790·Framework v5.0·Pipeline v0.4·Evaluated 18 Jul 2026·OECD: 2.3 Electrical engineering, electronic engineering, information engineering·Subfield: Health Informatics·Basis: Full text·Type: Machine learning or AI study

Who is using AI to code? Global diffusion and impact of generative AI

Simone Daniotti et al.Simone Daniotti, Johannes Wachs, Xiangnan Feng et al.·2026·Science·10.1126/science.adz9311 ↗

Primary claim

Today, AI writes an estimated 29% of Python functions in the US

Supported · large-scale detector estimate with misclassification correction, limited by detector generalization and GitHub-Python scope
In brief

A strong behavioral measurement study of AI coding diffusion, constrained by observational productivity identification in open-source Python.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

Generative coding tools promise big productivity gains, but uneven uptake could widen skill and income gaps. We train a neural classifier to spot artificial intelligence (AI)-generated Python functions in more than 30 million GitHub commits by 160,097 software developers, tracking how fast, and where, these tools take hold. Currently, AI writes an estimated 29% of Python functions in the US-a shrinking lead over other countries. We estimate that quarterly output, measured in online code contributions, consequently increased by 3.6%. AI seems to benefit experienced, senior-level developers: They increased productivity and more readily expanded into new domains of software development. By contrast, early-career developers showed no significant benefits from AI adoption. This may widen skill gaps and reshape future career ladders in software development.

Key Claims

Primary finding · Supported

Today, AI writes an estimated 29% of Python functions in the US

large-scale detector estimate with misclassification correction, limited by detector generalization and GitHub-Python scope

Abstract; Results, Figure 2A; SI S1.3
Secondary · Tentative

quarterly output, measured in online code contributions, has increased by 3.6% because of this

fixed-effects association supports direction, but causal wording exceeds the observational design

Abstract; Results, Figure 3C; Table S7; SI S1.5
Secondary · Supported

AI seems to benefit experienced, senior-level developers

interaction estimates consistently favor experienced users across commit and library outcomes

Abstract; Results, Figure 3D; SI S4.3; Table S9
Secondary · Supported

early-career developers showed no significant benefits from AI adoption

subgroup estimates are non-significant, though null effects depend on power and model specification

Abstract; Results, Figure 3D; SI S4.3; Table S9
Secondary · Supported

genAI helps programmers expand their capabilities to new domains of software development

library-combination proxies and robustness checks support domain expansion, but the construct is indirect

Results, Figure 3C; SI S4.1; Table S7

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Behavioral measurement at unusual scale

The study applies a custom AI-code detector to 5 million functions from 31 million Python commits, moving beyond self-reported AI use or tool-access proxies. This gives the paper a distinctive empirical basis for tracking diffusion over time and across countries.

↳ Figure 1; Results, Detecting AI generated code; Figure 2

#methodological rigour

Extensive validation and robustness checks

The classifier is evaluated on held-out data, WildChat interactions, newer LLM-generated code, and retraining scenarios, while the productivity regressions are accompanied by placebo tests and measurement-error analysis. These checks directly address several plausible threats to the adoption and productivity estimates.

↳ SI S2.1–S2.2; SI S4.2–S4.5; Tables S7–S9; Figure S6

#application pathway

Policy-relevant heterogeneity in benefits

The paper finds that productivity and exploration gains are concentrated among experienced developers, while less experienced developers show no statistically significant productivity benefit. This result gives the work practical relevance for workforce development and inequality debates.

↳ Results, Figure 3D; SI S4.3; Discussion

Limitations

#methodological rigour

Observational design under causal wording

The productivity estimates come from user and quarter fixed-effects regressions with lagged AI-use measures, not from randomized or quasi-random assignment. The article acknowledges this in the SI, but phrases such as 'because of this' and 'substantially increases output' overstate the causal identification.

↳ Abstract; Results, Figure 3C; Discussion; SI S1.5

#methodological rigour

Detector performance remains context-dependent

The detector performs strongly on held-out synthetic data, but true-positive probabilities are lower on WildChat code and weaker for some newer models before retraining. Because adoption and productivity estimates depend on this detector, residual measurement uncertainty remains material.

↳ SI S2.1–S2.2; Figure S4; Figure S5; Table S4

#application pathway

External validity is materially bounded

The study focuses on open-source Python contributions on GitHub and covers six major countries, leaving uncertainty about other languages, proprietary enterprise codebases, lower-income countries, and China-specific platform substitution. The authors discuss these limits, but they constrain direct generalization.

↳ Discussion, Limitations; Figure 2B

#methodological rigour

Replication link is not verifiable

The paper states that code and data are available, but the supplied article text provides only 'this link' rather than a working repository or persistent identifier. This prevents independent verification of the replication package from the provided document.

↳ Data and materials availability; SI Materials and Methods, Data

Does this match your read? Add your assessment ↓

Nabu Commentary

The article’s strongest feature is its behavioral measurement strategy: a custom AI-code classifier is applied at scale and checked against held-out, real-world, and newer-model code. The regression design is competent for panel observational data, using user and quarter fixed effects, lagged AI adoption, clustered standard errors, placebo tests, and measurement-error analysis. The main score-limiting issue is that the 3.6% productivity estimate is framed more causally in the abstract and discussion than the design can establish. External validity is also bounded by the open-source Python GitHub setting, even though the discussion compares the estimates to firm reports and explicitly identifies scope limits.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.7±0.2

Contribution
3.9

The paper contributes a behavioral measure of generative AI use in coding, applied to more than 30 million GitHub commits and used to study diffusion, demographics, and productivity. Its contribution is reduced by causal phrasing around productivity and by scope limits to open-source Python on GitHub.

Introduction; Results, Figures 2–3; Discussion, Limitations

"measure the adoption and use of genAI in another way"

Methodological Rigour
3.5

The methods include a custom GraphCodeBERT-based detector, held-out validation, real-world WildChat validation, country-specific misclassification correction, fixed-effects regressions, placebo tests, and measurement-error analysis. The score reflects meaningful uncertainty from the 8,000-function training set, false-positive correction, weaker performance on some newer models, and observational productivity estimation.

Figure 1; SI S1.1–S1.3; SI S2.1–S2.2; SI S4.1–S4.5; Tables S2, S7–S9

"our estimation design is not optimized for identifying causal effects"

Reporting
4.0

The article presents a clear progression from the motivation for behavioral measurement to classifier construction, adoption results, productivity regressions, and limitations. The main clarity limitation is that the abstract and discussion sometimes compress regression associations into causal-sounding language.

Abstract; Results; Discussion; SI S1.5

"Moving from 0 to 29% genAI usage...is associated with a 3.6% increase"

Positioning
3.8

The paper situates its evidence against surveys, RCTs, natural experiments, firm reports, and general-purpose technology debates, and it explicitly discusses limits in platform, language, geography, spillovers, and code quality. Positioning is somewhat limited by optimistic monetary-value extrapolation and only modest engagement with AI-code-detection limitations as a literature.

Introduction; Discussion; Limitations; SI S1.5–S1.7

"There are several limitations to our study."

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

Core numerical claims are broadly consistent, including the 29% adoption estimate and 3.6% productivity calculation, but the paper does not reconcile different developer-count figures in one place. The 28.6% versus 29% distinction is explicitly explained in SI S1.5.

Research ConductNoted

The article states that code and data are available, but the supplied text gives only 'this link' rather than a verifiable repository. Public GitHub data, self-reported location, and name-based gender inference are described, with limitations discussed, but no ethics or COI declaration is evident in the supplied text.

Flags: 1 declared / 5 total

Reference IntegrityClean

All 49 cited references resolved to real publications via PubMed, Crossref, or OpenAlex.

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaEconomy · SDG 8
Type of advanceEmpirical
Route to useSuggested
AdvanceExtends
CitationsToo early
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The adoption estimate is derived from real-world population data on millions of commits, with the classifier validated out-of-sample and on external real-world LLM interactions, and corroborated by independent firm-level adoption reports.

From the paper
“The findings in this study provide better evidence of how genAI is used in a large, important, and highly exposed sector of the economy”

The paper offers its measurements as evidence and a real-time monitoring approach for adoption and productivity effects, but does not name a specific actor or decision that would apply them.

Pathway

Policymakers, firms and researchers would use these measured adoption rates and productivity estimates to inform AI regulation, workforce training and economic planning decisions.

Barrier

The productivity estimates are non-causal fixed-effects associations limited to open-source Python code, and the classifier's accuracy degrades on newer LLMs, limiting generalisation.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Large language models detect methodological errors that human peer review misses in clinical trials

    Dominikus Brian, Xavier Isidro Pereira Hernández, Hao Yin, et al. · 2026 · Computer and information sciences

    Primary claim

    Our findings suggest that LLMs can identify methodological errors in reports of randomized clinical trials that human peer review may miss.

    Tentative
    Evidence Score
    3.7±0.2
    Reliability
    Caveats
  2. Comparing AI chat tools and PubMed for medical literature searches

    Ettien, Downing, Drouhard, et al. · 2026 · Media and communications

    Primary claim

    Our results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for use in any in-depth literature searches

    Tentative
    Evidence Score
    3.0±0.2
    Reliability
    Caveats
  3. Generative AI bias against scientific novelty: a cautionary tale from a small-sample evaluation of research proposals

    Diogo Machado · 2026 · Other social sciences

    Primary claim

    Novelty is the only evaluation dimension that systematically predicts disagreement between the two evaluation systems.

    Supported
    Evidence Score
    3.6±0.1
    Reliability
    Caveats

How the rubric works →