Many labs 2: investigating variation in replicability across samples and settings

Klein, Richard A., Vianello, Michelangelo, Hasselman, Fred, Adams, Byron G., Adams, Reginald B., Alper, Sinan, Aveyard, Mark and Kappes, Heather Barry ORCID: 0000-0002-6335-3888 (2018) Many labs 2: investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1 (4). pp. 443-490. ISSN 2515-2459

Text - Accepted Version Download (4MB)

• Author

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (p < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (p < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s d effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.

Item Type:	Article
Official URL:	https://www.psychologicalscience.org/publications/...
Additional Information:	© 2018 Association for Psychological Science
Divisions:	Management
Subjects:	H Social Sciences > H Social Sciences (General)
Date Deposited:	10 Dec 2018 16:00
Last Modified:	26 Oct 2024 23:30
Funders:	Center for Open Science, Laura and John Arnold Foundation
URI:	http://eprints.lse.ac.uk/id/eprint/91159

Actions (login required)

View Item