We are in the process of searching and coding the entire history of lab and field research on human cooperation using social dilemmas. We have already coded approximately 3,000 studies from 1958 to 2017 published in English, Chinese, and Japanese. Each study has been coded for (a) 40 study characteristics that can vary across studies (e.g., sample characteristics, and structure of the social dilemma task), (b) the variables that have been measured and/or manipulated to predict cooperation, and (c) the corresponding quantitative results (e.g., mean levels of cooperation and effect sizes with cooperation). In collaboration with a team of computer scientists, we have developed an ontology that represents the knowledge contained in this literature on human cooperation. The ontology will form the basis to develop a semantic-based platform that can be used to search the databank and output on-demand meta-analyses. Our goal is to establish an open access database where researchers can go, perform a targeted search of the literature, and efficiently conduct their own meta-analyses on cooperation. We also aim to establish an institution where researchers can efficiently share their replications and null findings with the field (which are often never published).
We envision that this database and institution will provide several outstanding benefits.
Detection of Type I Errors. The databank will allow scientists to better monitor and compare the work of different labs, and in so doing provide a decentralized mechanism to detect Type I errors (i.e., an incorrect rejection of a null hypothesis).
Reducing Type II Errors: Estimating the True Effect Size for Power Analysis. Low statistical power is a pervasive problem in the social sciences. Statistical power is the probability that a researcher will correctly reject a false null hypothesis. In the behavioral sciences, on average power is less than 50 percent for a medium effect size (Cohen, 1962; Seldmeier & Gigerenzer, 1989). Not surprisingly, researchers are often failing to reject false null hypotheses (Type II errors). Power analysis will inform researchers how many participants are necessary to reach a specified level of power (recommended at .80, Cohen, 1992). However, a challenge for researchers when conducting power analyses is that they must estimate the true effect size at the level of the population. The databank can provide an estimate of the true effect size on several effects on cooperation allowing for a priori power analyses that can help to establish methodological standards in the field to increase a priori statistical power.
Promoting Replication. Researchers lack an incentive to replicate their own and other’s research findings, yet replication is the “backbone” of any scientific discipline. This institution will provide a platform for researchers to share their replications and null findings with the field and be recognized for that work. Researchers can cite this work as a peer-reviewed published finding and the results may be used in meta-analyses.
Reducing Publication Bias. Statistically significant findings are more likely to be published and this can have dramatic negative consequences for a field and the development of knowledge. For example, a few studies may be published in high impact journals which stimulate other labs to attempt to replicate (and extend) those findings, but when these replications are null results they tend not to be published. This institution will allow for scientists to share their null results with the field and so will address the long-standing problem of publication bias. Further, the databank can be used to observe and detect historical trends in publication.
Stimulating Quantitative Reviews. Meta-analysis has become increasingly recognized as a key tool to advance science in a mature scientific discipline. The cooperation databank will enable researchers to more easily monitor the development of the field, and increase accuracy and reduce the time and cost of subsequent meta-analyses.
The ACL has already hosted a workshop on the development of the databank that included members of the scientific advisory board (listed below). The development of the databank has been a large-scale effort including a trained team of international researchers (listed below). This project is funded by an ERC Starting Grant Awarded to Daniel Balliet. Our goal is to have a working prototype of the platform by June 2019, and the final open access version will be provided by August 2020.
Chris Aberson, Humboldt State University
Athena Aktipis, Arizona State University
Nancy Buchan, University of South Carolina
Carsten de Dreu, Leiden University
Andrew Delton, Stony Brook University
Susann Fiedler, Max Planck Institute for Research on Collective Goods
Simon Gächter, University of Nottingham
Nir Halevy, Stanford University
Paul van Lange, Vrije Universiteit Amsterdam
Caspar van Lissa, Utrecht University
Wolfgang Viechtbauer, Maastricht University
Junhui Wu, Beijing Normal University
Toshio Yamagishi, Hitotsubashi University
*We are currently searching for additional people to join our lab and contribute to developing the databank. Contact Daniel Balliet (d.p.balliet(at)vu.nl) if you are interested in joining this project.