Relative feedback increases disparities in effort and performance in crowdsourcing contests: evidence from a quasi-experiment on Topcoder

Tsvetkova, Milena ORCID: 0000-0002-3552-108X, Mueller, Sebastian, Vuculescu, Oana, Ham, Haylee and Sergeev, Rinat (2022) Relative feedback increases disparities in effort and performance in crowdsourcing contests: evidence from a quasi-experiment on Topcoder. Proceedings of the ACM on Human-Computer Interaction, 6 (CSCW2). ISSN 2573-0142

Text (Tsvetkova__relative-feedback-increases--accepted) - Accepted Version Download (496kB)

• Author

Abstract

Rankings and leaderboards are often used in crowdsourcing contests and online communities to motivate individual contributions but feedback based on social comparison can also have negative effects. Here, we study the unequal effects of such feedback on individual effort and performance for individuals of different ability. We hypothesize that the effects of social comparison differ for top performers and bottom performers in a way that the inequality between the two increases. We use a quasi-experimental design to test our predictions with data from Topcoder, a large online crowdsourcing platform that publishes computer programming contests. We find that in contests where the submitted code is evaluated against others’ submissions, rather than using an absolute scale, top performers increase their effort while bottom performers decrease it. As a result, relative scoring leads to better outcomes for those at the top but lower engagement for bottom performers. Our findings expose an important but overlooked drawback from using gamified competitions, rankings, and relative evaluations, with potential implications for crowdsourcing markets, online learning environments, online communities, and organizations in general.

Item Type:	Article
Official URL:	https://dl.acm.org/journal/pacmhci
Additional Information:	© 2022 The Authors
Divisions:	Methodology
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science T Technology H Social Sciences > HM Sociology
Date Deposited:	15 Aug 2022 11:00
Last Modified:	20 Dec 2024 00:45
URI:	http://eprints.lse.ac.uk/id/eprint/115983

Actions (login required)

View Item