# Predicting replicability-Analysis of survey and prediction market data from large-scale forecasting projects.

- **DOI:** [10.1371/journal.pone.0248780](https://doi.org/10.1371/journal.pone.0248780)
- **Rating:** Sound (3.6 out of 5)
- **Verdict:** A solid pooled benchmark strengthens evidence that expert forecasts predict replication outcomes, but cross-project heterogeneity limits generalization.
- **Framework version:** v5.8
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-08-14
- **Evaluation page:** https://www.nabu.science/paper/10.1371%2Fjournal.pone.0248780

## Key claims

1. prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103)
   - Support: Well-supported
   - Type: Primary finding
   - Basis: The pooled count directly yields 75 correct predictions among 103 findings
   - Location: Abstract; Results §3.2
2. Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001)
   - Support: Well-supported
   - Type: Secondary
   - Basis: Pooled correlations are reported with confidence intervals and matching significance tests
   - Location: Abstract; Results §3.1
3. We also found a significant relationship between p-values of the original findings and replication outcomes
   - Support: Well-supported
   - Type: Secondary
   - Basis: The categorized p-value analysis and study-level meta-analysis both support the association
   - Location: Abstract; Results §3.5

## Strengths

- **Pooling enables higher-powered cumulative analysis** Pooling the four forecasting studies raises the analysis to 103 findings and supports analyses that were underpowered or unavailable in the individual projects, including meta-analysis and market-dynamics assessment.
  - Dimension: contribution
  - Location: Introduction, final paragraph; Results §§3.1–3.5
- **Multiple metrics reduce selective interpretation** The paper reports binary accuracy, correlation, and mean absolute error, including the nonsignificant market-versus-survey accuracy comparison and the case where survey error was lower.
  - Dimension: methodological rigour
  - Location: Results §§3.1–3.2
- **Reusable dataset supports future research** The pooled tables, codebook, trades, survey responses, and aggregation examples are released through the named pooledmaRket R package.
  - Dimension: contribution
  - Location: Methods §2.3

## Limitations

- **Pooling robustness remains incompletely tested** The four projects differ in replication criteria, power, and forecasting conditions, and estimated heterogeneity is substantial. The headline pooled estimates are not accompanied by a formal leave-one-study-out sensitivity analysis.
  - Dimension: methodological rigour
  - Location: Methods §2.1; Results §3.1
- **Forecaster selection limits broader inference** Participants were recruited through blogs, mailing lists, and Twitter, but the implications of this self-selected group for claims about the scientific community are not fully traced in the Discussion.
  - Dimension: positioning
  - Location: Methods §2.2; Discussion
- **Minor inconsistencies require correction** Section 3.3 reports the variance-weighted mean as M = 58 and SD = 17 rather than proportions, while RPP accuracy appears as 70% in §2.2.1 and 71% in §3.2.
  - Dimension: reporting
  - Location: Methods §2.2.1; Results §§3.2–3.3

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. Two minor numerical presentation inconsistencies are locatable in the text. Neither affects the pooled headline results or their interpretation.
- **Research conduct:** Noted. Ethics procedures are transparently declared, including the absence of formal review and documented consent for three studies and the advisory opinion for SSRP. Data and code availability are claimed through a named GitHub-hosted R package.
- **Reference integrity:** Clean. 40 of 40 checkable references verified
  - 40 references in manuscript
  - citation details diverge from the cited records for 1 reference
  - Citation details diverge for 1 reference:
    - #25 Forsell E, Viganola D, Pfeiffer T, Almenberg J, Wilson B, Chen Y, et al. Predicting replication outcomes in the Many Labs 2 study. J Econ Psychol. 2018 [cited 22 Mar 2019]. https://doi.org/10.1016/j.joep. 2018.03.005 PMID: 30420788 (Differs from the cited DOI’s record on: title, authors)
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
