Comparing chatbots to psychometric tests in hiring: reduced social desirability bias, but lower predictive validity

Dukanovic, Danilo and Krpan, Dario ORCID: 0000-0002-3420-4672 (2025) Comparing chatbots to psychometric tests in hiring: reduced social desirability bias, but lower predictive validity. Frontiers in Psychology, 16. ISSN 1664-1078

Text - Published Version Available under License Creative Commons Attribution. Download (703kB)

Abstract

This paper explores the efficacy of AI-driven chatbots in accurately inferring personality traits compared to traditional psychometric tests within a real-world professional hiring context. The study is driven by the increasing integration of AI tools in recruitment processes, which necessitates a deeper understanding of their reliability and validity. Using a quasi-experimental design with propensity score matching, we analysed data from 159 candidates and other professionals from Serbian and Montenegrin regions who completed both traditional psychometric assessments and AI-based personality evaluations based on the Big Five Personality model. A novel one-question-per-facet approach was employed in the chatbot assessments with a goal of enabling more granular analysis of the chatbot’s psychometric properties. The findings indicate that the chatbot demonstrated good structural, substantive, and convergent validity for certain traits, particularly Extraversion and Conscientiousness, but not for Neuroticism, Agreeableness, and Openness. While robust regression confirmed that AI-inferred scores are less susceptible to social desirability bias than traditional tests, they did not significantly predict real-world outcomes, indicating issues with external validity, particularly predictive validity. The results suggest that AI-driven chatbots show promise for identifying certain personality traits and demonstrate resistance to social desirability bias. This paper contributes to the emerging field of AI and psychometrics by offering insights into the potential and limitations of AI tools in professional selection, while developing an approach for refining psychometric properties of AI-driven assessments.

Item Type:	Article
Additional Information:	© 2025 The Authors
Divisions:	Psychological and Behavioural Science
Subjects:	B Philosophy. Psychology. Religion > BF Psychology H Social Sciences > HF Commerce
Date Deposited:	12 May 2025 08:39
Last Modified:	12 May 2025 08:39
URI:	http://eprints.lse.ac.uk/id/eprint/128106

Actions (login required)

View Item