Cookies?
Library Header Image
LSE Research Online LSE Library Services

Many labs 2: investigating variation in replicability across sample and setting

Klein, Richard A. and Vianello, Michelangelo and Hasselman, Fred and Adams, Byron G. and Adams, Reginald B. and Alper, Sinan and Aveyard, Mark and Kappes, Heather Barry (2018) Many labs 2: investigating variation in replicability across sample and setting. Advances in Methods and Practices in Psychological Science. (In Press)

[img] Text - Accepted Version
Download (4MB)

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (p < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (p < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s d effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.

Item Type: Article
Official URL: https://www.psychologicalscience.org/publications/...
Additional Information: © 2018 Association for Psychological Science
Divisions: Management
Subjects: H Social Sciences > H Social Sciences (General)
Sets: Departments > Management
Date Deposited: 10 Dec 2018 16:00
Last Modified: 10 Dec 2018 16:00
Funders: Center for Open Science, Laura and John Arnold Foundation
URI: http://eprints.lse.ac.uk/id/eprint/91159

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics