How Tenable are Modeling Assumptions Around Rapid Guessing Behavior? Results From a Large Corpus of Low-Stakes Assessments

Abstract

Rapid guessing (RG) has been shown to distort psychometric information in low-stakes assessments if it is not properly accounted for. Recently, model-based approaches for mitigating the effects of RG have been proposed; however, these methods make strong behavioral assumptions about the nature of RG. To date, the tenability of these behavioral assumptions has not been extensively explored. Using data from 21 low-stakes assessments we investigate (a) the extent to which RG propensity is linearly related to ability and (b) to extent to which RG is predicted by expected response probability (ERP). We investigate these research questions via two model-based RG scoring approaches, the Holman-Glas (HG) and Mislevy-Wu multidimensional IRT models. Results indicated that RG propensity was inversely related to ability to varying degrees, with factor correlations ranging from -0.92 to -0.05. Additionally, we find evidence that RG behavior was predicted by ERP, favoring the MW model; however, item and ability parameter estimates were nearly identical to those from the HG model. Based on these findings, recommendations are provided for evaluating behavioral assumptions underlying RG behavior.

Alfonso J. Martinez

Assistant Professor of Psychometrics and Quantitative Psychology

My research interests include generalized latent variable modeling, Bayesian analysis, and computational statistics.