Loading evaluation data…
The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package "pooledmaRket" and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.
prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103)
The pooled count directly yields 75 correct predictions among 103 findings
Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001)
Pooled correlations are reported with confidence intervals and matching significance tests
We also found a significant relationship between p-values of the original findings and replication outcomes
The categorized p-value analysis and study-level meta-analysis both support the association
Derived from the full evaluation — not a separate score.
Strengths
Pooling the four forecasting studies raises the analysis to 103 findings and supports analyses that were underpowered or unavailable in the individual projects, including meta-analysis and market-dynamics assessment.
↳ Introduction, final paragraph; Results §§3.1–3.5
The paper reports binary accuracy, correlation, and mean absolute error, including the nonsignificant market-versus-survey accuracy comparison and the case where survey error was lower.
↳ Results §§3.1–3.2
The pooled tables, codebook, trades, survey responses, and aggregation examples are released through the named pooledmaRket R package.
↳ Methods §2.3
Limitations
The four projects differ in replication criteria, power, and forecasting conditions, and estimated heterogeneity is substantial. The headline pooled estimates are not accompanied by a formal leave-one-study-out sensitivity analysis.
↳ Methods §2.1; Results §3.1
Participants were recruited through blogs, mailing lists, and Twitter, but the implications of this self-selected group for claims about the scientific community are not fully traced in the Discussion.
↳ Methods §2.2; Discussion
Section 3.3 reports the variance-weighted mean as M = 58 and SD = 17 rather than proportions, while RPP accuracy appears as 70% in §2.2.1 and 71% in §3.2.
↳ Methods §2.2.1; Results §§3.2–3.3
The pooled analysis meaningfully increases statistical power and supports its principal descriptive and correlational claims across 103 findings. Results §§3.1–3.2 use complementary metrics and transparently report overestimation, heterogeneity, and a nonsignificant binary-accuracy difference between markets and surveys. The main methodological constraint is limited sensitivity analysis around pooling four projects with differing designs and substantial estimated heterogeneity. The Discussion appropriately narrows several conclusions, though participant self-selection is not fully integrated into the scope of claims about scientific-community beliefs.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.6
Pooling four forecasting studies enables higher-powered estimates, meta-analysis, aggregation comparisons, and market-dynamics analyses while producing a reusable dataset. The main findings largely confirm earlier study-level results, making the advance genuine but incremental.
testing hypotheses with substantially higher statistical power
The analysis uses suitable correlations, paired comparisons, multiple accuracy metrics, and a conservative random-effects method, with data and example code released. Cross-study robustness is not extensively tested through leave-one-study-out or comparable sensitivity analyses.
using the conservative Hartung-Knapp-Sidik-Jonkman method
The paper follows a clear progression from pooling rationale through methods, results, and implications, while reporting unfavorable and nonsignificant findings. The word “determine” and the missing decimals in §3.3 introduce minor imprecision without materially misleading the reader.
The paper engages with the p-value debate and explicitly connects observed heterogeneity to the need for further evidence. Recruitment self-selection and its implications for claims about the scientific community receive less sustained treatment.
with only a small number of studies used in our meta-analyses, further data are required
Lower confidence on Positioning — domain match limited.
Caveats4 of 4 checks
Two minor numerical presentation inconsistencies are locatable in the text. Neither affects the pooled headline results or their interpretation.
Ethics procedures are transparently declared, including the absence of formal review and documented consent for three studies and the advisory opinion for SSRP. Data and code availability are claimed through a named GitHub-hosted R package.
Flags: 3 declared / 5 total
40 of 40 checkable references verified
40 references in manuscript citation details diverge from the cited records for 1 reference
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Medium3.3
The work addresses a recognized need among replication-project organizers and meta-science researchers. Its connection to specific journals, funders, or operational decision processes remains general.
used for future forecasting projects that are either planned or in progress
Prediction markets and surveys were used with actual replication projects, and the pooled data and code are available for reuse. The paper does not demonstrate adoption within a standing journal or funding workflow.
The pooled data can be downloaded within the R package
The four projects span psychology, experimental economics, and broader social science. Substantial heterogeneity and the absence of evidence outside social and behavioral science limit broader transferability.
The analysis directly extends four prior forecasting studies and releases a benchmark intended for cumulative research. Its contribution to that trajectory is clear, although principally incremental.
providing a benchmark for any further replication-based markets
AI-generated, human-governed. Something look off? Contact us to request a review.