Item pool quality control in educational testing: change point model, compound risk, and sequential detection

Chen, Yunxiao ORCID: 0000-0002-7215-2324, Lee, Yi-Hsuan and Li, Xiaoou (2022) Item pool quality control in educational testing: change point model, compound risk, and sequential detection. Journal of Educational and Behavioral Statistics, 47 (3). 322 - 352. ISSN 1076-9986

Text (10769986211059085) - Published Version Available under License Creative Commons Attribution Non-commercial. Download (396kB)

• Author
• Scopus publication

Abstract

In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this article, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric properties, where a change can be caused by, for example, leakage of the item or change of the corresponding curriculum. We propose a statistical framework for the detection of abrupt changes in individual items. This framework consists of (1) a multistream Bayesian change point model describing sequential changes in items, (2) a compound risk function quantifying the risk in sequential decisions, and (3) sequential decision rules that control the compound risk. Throughout the sequential decision process, the proposed decision rule balances the trade-off between two sources of errors, the false detection of prechange items, and the nondetection of postchange items. An item-specific monitoring statistic is proposed based on an item response theory model that eliminates the confounding from the examinee population which changes over time. Sequential decision rules and their theoretical properties are developed under two settings: the oracle setting where the Bayesian change point model is completely known and a more realistic setting where some parameters of the model are unknown. Simulation studies are conducted under settings that mimic real operational tests.

Item Type:	Article
Official URL:	https://journals.sagepub.com/home/jeb
Additional Information:	© 2021 The Authors
Divisions:	Statistics
Subjects:	H Social Sciences > HA Statistics L Education > LB Theory and practice of education B Philosophy. Psychology. Religion > BF Psychology
Date Deposited:	26 Oct 2021 08:27
Last Modified:	20 Dec 2024 00:42
URI:	http://eprints.lse.ac.uk/id/eprint/112498

Actions (login required)

View Item