Validating the use of large language models for psychological text classification

Bunt, Hannah, Goddard, Alex, Reader, Tom W. ORCID: 0000-0002-3318-6388 and Gillespie, Alex ORCID: 0000-0002-0162-1269 (2025) Validating the use of large language models for psychological text classification. Frontiers in Social Psychology, 3. ISSN 2813-7876

Text (frsps-1-1460277) - Published Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

Large language models (LLMs) are being used to classify texts into categories informed by psychological theory (“psychological text classification”). However, the use of LLMs in psychological text classification requires validation, and it remains unclear exactly how psychologists should prompt and validate LLMs for this purpose. To address this gap, we examined the potential of using LLMs for psychological text classification, focusing on ways to ensure validity. We employed OpenAI's GPT-4o to classify (1) reported speech in online diaries, (2) other-initiations of conversational repair in Reddit dialogues, and (3) harm reported in healthcare complaints submitted to NHS hospitals and trusts. Employing a two-stage methodology, we developed and tested the validity of the prompts used to instruct GPT-4o using manually labeled data (N = 1,500 for each task). First, we iteratively developed three types of prompts using one-third of each manually coded dataset, examining their semantic validity, exploratory predictive validity, and content validity. Second, we performed a confirmatory predictive validity test on the final prompts using the remaining two-thirds of each dataset. Our findings contribute to the literature by demonstrating that LLMs can serve as valid coders of psychological phenomena in text, on the condition that researchers work with the LLM to secure semantic, predictive, and content validity. They also demonstrate the potential of using LLMs in rapid and cost-effective iterations over big qualitative datasets, enabling psychologists to explore and iteratively refine their concepts and operationalizations during manual coding and classifier development. Accordingly, as a secondary contribution, we demonstrate that LLMs enable an intellectual partnership with the researcher, defined by a synergistic and recursive text classification process where the LLM's generative nature facilitates validity checks. We argue that using LLMs for psychological text classification may signify a paradigm shift toward a novel, iterative approach that may improve the validity of psychological concepts and operationalizations.

Item Type:	Article
Additional Information:	© 2025 The Author(s)
Divisions:	Psychological and Behavioural Science
Subjects:	P Language and Literature H Social Sciences B Philosophy. Psychology. Religion > BF Psychology
Date Deposited:	28 Jan 2025 11:06
Last Modified:	10 Mar 2025 14:42
URI:	http://eprints.lse.ac.uk/id/eprint/127083

Actions (login required)

View Item