Comparing Different Methods of Identifying Rapid-Guessing Thresholds

Abstract

Rapid-guessing behavior demonstrated by individuals in achievement tests is expressed as a factor that threatens validity. Response time data, which can be obtained through computer-based test applications, allows for the distinction between rapid-guess behavior and solving behavior. In the literature, there are various methods for determining the threshold value used to make this distinction. In this study, the response time data, which were binary-coded based on different threshold determination methods, were incorporated into the item response tree model, and the fit of response times determined by which method best suited the model was examined. The data of 3329 individuals who participated in the PISA 2018 implementation in English and in a computer-based environment were used. The response and response time data of these individuals for 18 mathematics items from booklet 2 (M02 – M03) were selected for analysis. Among the threshold determination methods, the normative threshold method (NT10), visual inspection of response time distributions (VI), graphical review methods based on response accuracy, the cumulative proportion (CUMP) method, median-based, and logarithmic transformation-based methods (logNT10) were selected. The individuals' responses to the items and their response times were analyzed using the "irtree" and "lme4" packages. Comparative results regarding the model-data fit of the response times obtained by different methods were provided. According to the results, the data obtained with the normative threshold method applied after the logarithmic transformations of the response times showed the best model fit. The logNT10 method can be a good alternative in cases where the RT distribution is non-bimodal and skewed. Furthermore, when compared to other evidence regarding the validity of the threshold values used, it was found to be consistent with the model-data fit results. Based on these results, existing methods and recommendations for determining threshold values for rapid-guess behavior were discussed.

Keywords

• Response time
• IRTree models
• Identifying thresholds
• Rapid-guessing behavior

References

1. Akbaş, U., & Koğar, H. (2020). Nicel araştırmalarda kayıp veriler ve uç değerler. Ankara: Pegem Akademi Yayıncılık.
2. Alfian, M., Yuhana, U. L., Pardede, E., & Bimantoro, A. N. P. (2023). Correction of threshold determination in rapid-guessing behaviour detection. Information, 14(7), 422. https://doi.org/10.3390/info14070422
3. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
4. Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665–678. https://doi.org/10.1037/a0028111
5. Bulut, O., Gorgun, G., Wongvorachan, T., & Tan, B. (2023). Rapid guessing in low-stakes assessments: finding the optimal response time threshold with random search and genetic algorithm. Algorithms, 16(2), 89. https://doi.org/10.3390/a16020089
6. De Boeck, P., & Partchev, I. (2012). Irtrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48(1), 1–28. https://doi.org/10.18637/jss.v048.c01
7. Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not-reached items using irtrees. Journal of Educational Measurement, 54(3), 333–363. https://doi.org/10.1111/jedm.12147
8. Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-scale Assessments in Education, 5(1). https://doi.org/10.1186/s40536-017-0051-9

9. Guo, H., Rios, J. A., Haberman, S., Liu, O. L., Wang, J., & Paek, I. (2016). A new procedure for detection of students’ rapid guessing responses using response time. Applied Measurement in Education, 29(3), 173–183. https://doi.org/10.1080/08957347.2016.1171766
10. Hadiana, D., Hayat, B., & Tola, B. (2021). A new method for setting response-time threshold to detect test takers’ rapid guessing behavior. Advances in Social Science, Education and Humanities Research, 545, 46–50. https://doi.org/10.2991/assehr.k.210423.062
11. Huang, H. Y. (2020). A mixture irtree model for performance decline and nonignorable missing data. Educational and Psychological Measurement, 80(6), 1168–1195. https://doi.org/10.1177/0013164420914711
12. Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48(3), 1070–1085. https://doi.org/10.3758/s13428-015-0631-y
13. Kong, X. J., Wise, S. L., & Bhola, D. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67(4), 606–619. https://doi.org/10.1177/0013164406294779
14. Kroehne, U., Deribo, T., & Goldhammer, F. (2020). Rapid guessing rates across administration mode and test setting. Psychological Test and Assessment Modeling, 62(2), 147–177. https://doi.org/10.25656/01:23630
15. Lee, Y.-H., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53(3), 359–379.
16. Lee, Y.-H., & Jia, Y. (2014). Using response time to investigate students' test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(1). https://doi.org/10.1186/s40536-014-0008-1
17. Leventhal, B. C., & Pastor, D. (2024). An illustration of an irtree model for disengagement. Educational and Psychological Measurement, 84(4), 810–834. https://doi.org/10.1177/00131644231185533
18. Michaelides, M. P., Ivanova, M., & Nicolaou, C. (2020). The relationship between response-time effort and accuracy in PISA science multiple choice items. International Journal of Testing, 20(3), 187–205. https://doi.org/10.1080/15305058.2019.1706529
19. Rios, J. A. (2022). Assessing the accuracy of parameter estimates in the presence of rapid guessing misclassifications. Educational and Psychological Measurement, 82(1), 122–150. https://doi.org/10.1177/00131644211003640
20. Rios, J. A., & Guo, H. (2020). Can culture be a salient predictor of test-taking engagement? An analysis of differential noneffortful responding on an international college-level assessment of critical thinking. Applied Measurement in Education, 33(4), 263–279. https://doi.org/10.1080/08957347.2020.1789141
21. Rios, J. A., & Soland, J. (2022). An investigation of item, examinee, and country correlates of rapid guessing in PISA. International Journal of Testing, 22(2), 154–184. https://doi.org/10.1080/15305058.2022.2036161
22. Rousselet, G. A., & Wilcox, R. R. (2020). Reaction times and other skewed distributions. Meta-psychology, 4. https://doi.org/10.15626/mp.2019.1630
23. Sahin, F., & Colvin, K. F. (2020). Enhancing response time thresholds with response behaviors for detecting disengaged examinees. Large-scale Assessments in Education, 8(1). https://doi.org/10.1186/s40536-020-00082-1
24. Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213–232. https://doi.org/10.1111/j.1745-3984.1997.tb00516.x
25. Schnipke, D. L., & Scrams, D. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C. N. Mills, M. T. Potenza, J. J. Fremer, & W. C. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237–266). Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410612250-20
26. Setzer, J. C., Wise, S. L., Van Den Heuvel, J. R., & Ling, G. (2013). An investigation of examinee test-taking effort on a large-scale assessment. Applied Measurement in Education, 26(1), 34–49. https://doi.org/10.1080/08957347.2013.739453
27. Soland, J., Kuhfeld, M., & Rios, J. (2021). Comparing different response time threshold setting methods to detect low effort on a large-scale assessment. Large-scale Assessments in Education, 9(1). https://doi.org/10.1186/s40536-021-00100-w
28. Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95–114. https://doi.org/10.1207/s15324818ame1902_2
29. Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61. https://doi.org/10.1111/emip.12165
30. Wise, S. L. (2019). An information-based approach to identifying rapid-guessing thresholds. Applied Measurement in Education, 32(4), 325–336. https://doi.org/10.1080/08957347.2019.1660350
31. Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38. https://doi.org/10.1111/jedm.2006.43.issue-1
32. Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2
33. Wise, S. L., & Ma, L. (2012, April 13–17). Setting response time thresholds for a CAT item pool: The normative threshold method. In Annual meeting of the National Council on Measurement in Education, Vancouver, Canada. https://www.nwea.org/resources/setting-response-time-thresholds-cat-item-pool-normative-threshold-method/
34. Wise, S. L., Kingsbury, G. G., Thomason, J., & Kong, X. (2004, April). An investigation of motivation filtering in a statewide achievement testing program. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego
35. Wise, S., & Kuhfeld, M. (2020). A cessation of measurement: Identifying test taker disengagement using response time. In M. Margolis, & R. Feinberg (Eds.), Integrating Timing Considerations to Improve Testing Practices, Routledge.