The Impact of Rapid Guessing on Model Fit and Factor-Analytic Reliability - An Exploratory Analysis

Abstract

Rapid guessing (RG) is form of non-effortful responding whereby examinees provide a response in a timeframe that is incommensurate with the amount of time needed to thoroughly engage with the item. Previous research has found that RG leads to biased parameter estimates, biased ability estimates, and measurement non-invariance if not properly accounted for. The consequences of RG on model fit and factor-analytic reliability (MF&R), however, are not well understood. To address this gap in the literature, the present study explores the effects of rapid guessing on model fit and reliability across a corpus of 20 diverse low-stakes assessments. Three RG scoring approaches (Naive, Penalized, Effort-moderated with imputation) were compared across four model fit indices (CFI, TLI, RMSEA, SRMR) and two reliability coefficients (McDonald’s omega and Cronbach’s alpha). We found evidence that model fit is influenced by choice of scoring procedure, with the effort-moderated with imputation method producing better model fit than naïve and penalized scoring procedures when RG rates were greater. RG was also found to differentially impact reliability indices, however no systematic trends across scoring method and RG rates emerged. Implications of the research, recommendations for practitioners, and a discussion of future directions are discussed.

Alfonso J. Martinez

Assistant Professor of Psychometrics and Quantitative Psychology

My research interests include generalized latent variable modeling, Bayesian analysis, and computational statistics.