Scaling language model size yields diminishing returns for single-message political persuasion

Hackenburg, Kobi, Tappin, Ben ORCID: 0000-0002-3826-8016, Röttger, Paul, Hale, Scott, Bright, Jonathan and Margetts, Helen (2025) Scaling language model size yields diminishing returns for single-message political persuasion. Proceedings of the National Academy of Sciences of the United States of America. ISSN 0027-8424 (In Press)

Text (scaling_laws_persuasion) - Accepted Version Pending embargo until 1 January 2100. Available under License Creative Commons Attribution. Download (2MB)

Abstract

Large language models can now generate political messages as persuasive as those written by humans, raising concerns about how far this persuasiveness may continue to increase with model size. Here, we generate 720 persuasive messages on 10 U.S. political issues from 24 language models spanning several orders of magnitude in size. We then deploy these messages in a large-scale randomized survey experiment (N = 25,982) to estimate the persuasive capability of each model. Our findings are twofold. First, we find evidence that model persuasiveness is characterized by sharply diminishing returns, such that current frontier models are only slightly more persuasive than models smaller in size by an order of magnitude or more. Second, we find that the association between language model size and persuasiveness shrinks towards zero and is no longer statistically significant once we adjust for mere task completion (coherence, staying on topic); a pattern that highlights task completion as a potential mediator of larger models’ persuasive advantage. Given that current frontier models are already at ceiling on this task completion metric in our setting, taken together our results suggest that further scaling model size may not much increase the persuasiveness of static LLM-generated political messages.

Item Type:	Article
Additional Information:	© 2025 The Author(s)
Divisions:	Psychological and Behavioural Science
Subjects:	J Political Science B Philosophy. Psychology. Religion > BF Psychology P Language and Literature
Date Deposited:	06 Feb 2025 11:48
Last Modified:	06 Feb 2025 12:18
URI:	http://eprints.lse.ac.uk/id/eprint/127194

Actions (login required)

View Item