Predicting replicability-Analysis of survey and prediction market data from large-scale forecasting projects.

Publisher: Public Library of Science Country of Publication: United States NLM ID: 101285081 Publication Model: eCollection Cited Medium: Internet ISSN: 1932-6203 (Electronic) Linking ISSN: 19326203 NLM ISO Abbreviation: PLoS One Subsets: MEDLINE

Original Publication: San Francisco, CA : Public Library of Science

Reproducibility of Results*; Forecasting/*methods ; Research/*statistics & numerical data; Humans ; Research/trends ; Surveys and Questionnaires

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package "pooledmaRket" and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.
Competing Interests: The authors have declared that no competing interests exist.

Science. 2015 Aug 28;349(6251):aac4716. (PMID: 26315443)
Proc Natl Acad Sci U S A. 2020 May 19;117(20):10762-10768. (PMID: 32366645)
PLoS One. 2019 Dec 5;14(12):e0225826. (PMID: 31805105)
PLoS Med. 2005 Aug;2(8):e124. (PMID: 16060722)
Epidemiology. 2008 Sep;19(5):640-8. (PMID: 18633328)
Nat Hum Behav. 2018 Sep;2(9):637-644. (PMID: 31346273)
PLoS One. 2020 Apr 22;15(4):e0231416. (PMID: 32320420)
Proc Natl Acad Sci U S A. 2018 Mar 13;115(11):2628-2631. (PMID: 29531051)
BMC Med Res Methodol. 2014 Feb 18;14:25. (PMID: 24548571)
Environ Monit Assess. 2014 May;186(5):2729-40. (PMID: 24352867)
Perspect Psychol Sci. 2012 Nov;7(6):531-6. (PMID: 26168109)
Nature. 2016 May 25;533(7604):452-4. (PMID: 27225100)
R Soc Open Sci. 2020 Jul 22;7(7):200566. (PMID: 32874648)
Psychol Sci. 2012 May 1;23(5):524-32. (PMID: 22508865)
PLoS One. 2016 Feb 26;11(2):e0149794. (PMID: 26919473)
Perspect Psychol Sci. 2008 Jul;3(4):286-300. (PMID: 26158948)
Psychol Bull. 2020 May;146(5):451-479. (PMID: 31944796)
J Econ Psychol. 2018 Jun;66:13-21. (PMID: 30420788)
Proc Natl Acad Sci U S A. 2015 Dec 15;112(50):15343-7. (PMID: 26553988)
Clin Trials. 2005;2(4):282-90; discussion 301-4, 364-78. (PMID: 16281426)
PLoS One. 2011;6(11):e26828. (PMID: 22073203)
Science. 2008 May 16;320(5878):877-8. (PMID: 18487176)
Science. 2016 Mar 25;351(6280):1433-6. (PMID: 26940865)
Nat Hum Behav. 2018 Jan;2(1):6-10. (PMID: 30980045)

Date Created: 20210414 Date Completed: 20210922 Latest Revision: 20231103

20240513

PMC8046229

10.1371/journal.pone.0248780

33852589