nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-C445E5E1·Framework v5.8·Pipeline v0.4·Evaluated 14 Aug 2026·OECD: 5.8 Other social sciences·Subfield: Statistics, Probability and Uncertainty·Basis: Full text·Type: Dataset or benchmark paper

Predicting replicability-Analysis of survey and prediction market data from large-scale forecasting projects.

Gordon et al.Gordon, Viganola, Dreber et al.·2021·PloS one·10.1371/journal.pone.0248780 ↗

Primary claim

prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103)

Well-supported · The pooled count directly yields 75 correct predictions among 103 findings
In brief

A solid pooled benchmark strengthens evidence that expert forecasts predict replication outcomes, but cross-project heterogeneity limits generalization.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package "pooledmaRket" and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.

Key Claims

Primary finding · Well-supported

prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103)

The pooled count directly yields 75 correct predictions among 103 findings

Abstract; Results §3.2
Quote verified
Secondary · Well-supported

Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001)

Pooled correlations are reported with confidence intervals and matching significance tests

Abstract; Results §3.1
Quote verified
Secondary · Well-supported

We also found a significant relationship between p-values of the original findings and replication outcomes

The categorized p-value analysis and study-level meta-analysis both support the association

Abstract; Results §3.5
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Pooling enables higher-powered cumulative analysis

Pooling the four forecasting studies raises the analysis to 103 findings and supports analyses that were underpowered or unavailable in the individual projects, including meta-analysis and market-dynamics assessment.

↳ Introduction, final paragraph; Results §§3.1–3.5

#methodological rigour

Multiple metrics reduce selective interpretation

The paper reports binary accuracy, correlation, and mean absolute error, including the nonsignificant market-versus-survey accuracy comparison and the case where survey error was lower.

↳ Results §§3.1–3.2

#contribution

Reusable dataset supports future research

The pooled tables, codebook, trades, survey responses, and aggregation examples are released through the named pooledmaRket R package.

↳ Methods §2.3

Limitations

#methodological rigour

Pooling robustness remains incompletely tested

The four projects differ in replication criteria, power, and forecasting conditions, and estimated heterogeneity is substantial. The headline pooled estimates are not accompanied by a formal leave-one-study-out sensitivity analysis.

↳ Methods §2.1; Results §3.1

#positioning

Forecaster selection limits broader inference

Participants were recruited through blogs, mailing lists, and Twitter, but the implications of this self-selected group for claims about the scientific community are not fully traced in the Discussion.

↳ Methods §2.2; Discussion

#reporting

Minor inconsistencies require correction

Section 3.3 reports the variance-weighted mean as M = 58 and SD = 17 rather than proportions, while RPP accuracy appears as 70% in §2.2.1 and 71% in §3.2.

↳ Methods §2.2.1; Results §§3.2–3.3

Does this match your read? Add your assessment ↓

Nabu Commentary

The pooled analysis meaningfully increases statistical power and supports its principal descriptive and correlational claims across 103 findings. Results §§3.1–3.2 use complementary metrics and transparently report overestimation, heterogeneity, and a nonsignificant binary-accuracy difference between markets and surveys. The main methodological constraint is limited sensitivity analysis around pooling four projects with differing designs and substantial estimated heterogeneity. The Discussion appropriately narrows several conclusions, though participant self-selection is not fully integrated into the scope of claims about scientific-community beliefs.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.6±0.1

Contribution
3.6

Pooling four forecasting studies enables higher-powered estimates, meta-analysis, aggregation comparisons, and market-dynamics analyses while producing a reusable dataset. The main findings largely confirm earlier study-level results, making the advance genuine but incremental.

Introduction, final paragraph; Methods §2.3; Results §§3.1–3.5

testing hypotheses with substantially higher statistical power

Methodological Rigour
3.6

The analysis uses suitable correlations, paired comparisons, multiple accuracy metrics, and a conservative random-effects method, with data and example code released. Cross-study robustness is not extensively tested through leave-one-study-out or comparable sensitivity analyses.

Methods §2.3; Results §§3.1–3.5

using the conservative Hartung-Knapp-Sidik-Jonkman method

Reporting
3.7

The paper follows a clear progression from pooling rationale through methods, results, and implications, while reporting unfavorable and nonsignificant findings. The word “determine” and the missing decimals in §3.3 introduce minor imprecision without materially misleading the reader.

Abstract; Results §§3.1–3.3
Positioning
3.5

The paper engages with the p-value debate and explicitly connects observed heterogeneity to the need for further evidence. Recruitment self-selection and its implications for claims about the scientific community receive less sustained treatment.

Methods §2.2; Results §3.1; Discussion, paragraphs 2–3

with only a small number of studies used in our meta-analyses, further data are required

Lower confidence on Positioning — domain match limited.

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

Two minor numerical presentation inconsistencies are locatable in the text. Neither affects the pooled headline results or their interpretation.

Research ConductNoted

Ethics procedures are transparently declared, including the absence of formal review and documented consent for three studies and the advisory opinion for SSRP. Data and code availability are claimed through a named GitHub-hosted R package.

Flags: 3 declared / 5 total

Reference IntegrityClean

40 of 40 checkable references verified

40 references in manuscript citation details diverge from the cited records for 1 reference

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaPolicy and practice · SDG 9
Type of advanceEmpirical
Route to useNot stated
AdvanceConsolidates
CitationsFWCI 4.5
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The 73% accuracy claim is tested against actual replication outcomes pooled across four independent real replication projects covering 103 findings, which constitutes independent real-world confirmation.

The only stated uses of the findings are for future forecasting projects, benchmarking against other methods, and further research on replicability, which is further research rather than a real-world use.

Pathway

Journals, funders and replication project teams could run prediction markets and surveys to prioritise which published findings to replicate or treat as credible.

Barrier

Forecasts are only modestly accurate and systematically overestimate replication success, and the method has not been shown to improve any real editorial, funding or replication decision.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study

    Alexander D Gibson, David N Borg, Nicole M White, et al. · 2026 · Health sciences

    Primary claim

    not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice

    Tentative
    Evidence Score
    3.1±0.2
    Reliability
    Caveats
  2. Examining a citation advantage for publicly accessible articles in special education journals

    Jesse I. Fleming, Bryan G. Cook · 2026 · Education

    Primary claim

    publicly accessible articles were associated with 26% (β = 0.26, p < 0.001) and 41% (β = 0.41, p < 0.001) more citations than closed articles in WoS and GS, respectively, after controlling for other variables

    Supported
    Evidence Score
    3.3±0.1
    Reliability
    Caveats
  3. Human-AI Collaboration for Estimating Scientific Replicability

    Tatiana Chakravorti, Robert Fraleigh, Timothy Fritton, et al. · 2026 · Computer and Information Sciences

    Primary claim

    hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts

    Tentative
    Evidence Score
    3.3±0.2
    Reliability
    Caveats

How the rubric works →