The repeated adjustment of measurement protocols method for developing high-validity text classifiers

Goddard, Alex ORCID: 0000-0003-1382-2700 and Gillespie, Alex ORCID: 0000-0002-0162-1269 (2025) The repeated adjustment of measurement protocols method for developing high-validity text classifiers. Psychological Methods. ISSN 1082-989X

Text (2026-72869-001) - Published Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

The development and evaluation of text classifiers in psychology depends on rigorous manual coding. Yet, the evaluation of manual coding and computational algorithms is usually considered separately. This is problematic because developing high-validity classifiers is a repeated process of identifying, explaining, and addressing conceptual and measurement issues during both the manual coding and classifier development stages. To address this problem, we introduce the Repeated Adjustment of Measurement Protocols (RAMP) method for developing high-validity text classifiers in psychology. The RAMP method has three stages: manual coding, classifier development, and integrative evaluation. These stages integrate the best practices of content analysis (manual coding), data science (classifier development), and psychology (integrative evaluation). Central to this integration is the concept of an inference loop, defined as the process of maximizing validity through repeated adjustments to concepts and constructs, guided by push-back from the empirical data. Inference loops operate both within each stage of the method and across related studies. We illustrate RAMP through a case study, where we manually coded 21,815 sentences for misunderstanding (Krippendorff’s α = .79), and developed a rule-based classifier (Matthews correlation coefficient [MCC] = 0.22), a supervised machine learning classifier (Bidirectional Encoder Representations From Transformers; MCC = 0.69) and a large language model classifier (GPT-4o; MCC = 0.47). By integrating manual coding and classifier development stages, we were able to identify and address a concept validity problem with misunderstandings. RAMP advances existing methods by operationalizing validity as an ongoing dynamic process, where concepts and constructs are repeatedly adjusted toward increasingly widespread intersubjective agreement on their utility.

Item Type:	Article
Additional Information:	© 2025 The Author(s)
Divisions:	Psychological and Behavioural Science
Subjects:	B Philosophy. Psychology. Religion > BF Psychology Q Science > QA Mathematics > QA76 Computer software
Date Deposited:	08 Jul 2025 15:27
Last Modified:	21 Oct 2025 11:33
URI:	http://eprints.lse.ac.uk/id/eprint/128730

Actions (login required)

View Item