The Relation of Item Difficulty Between Classical Test Theory and Item Response Theory: Computerized Adaptive Test Perspective

Abstract

This study aims to transform the calculated item difficulty statistics according to Classical Test Theory (CTT) into the item difficulty parameter of Item Response Theory (IRT) by utilizing the normal distribution curve and to analyze the effectiveness of this transformation based on Rasch model. In this regard, 36 different data sets created with catR package were studied. For each data set, item difficulty parameters and transformed item difficulty parameters were calculated and the correlation coefficients between these parameters were analyzed. Then, Computerized Adaptive Test (CAT) simulations were performed using these parameters. According to the simulation results, the correlation coefficients between the estimated theta values with both methods were high. Furthermore, in CAT simulations in which both parameters were used, especially in the samples which were over 250, it was found to have similar bias, RMSE values, and the average number of administered items.

Keywords

• Item difficulty
• classical test theory
• item response theory
• Rasch model

References

1. Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29-51. https://doi.org/10.1007/BF02291411
2. Crocker, L., & Algina, J. (1986). Introduction to Classical and Modern Test Theory. Cengage Learning.
3. Çelen, Ü., & Aybek, E. C. (2013). Öğrenci başarısının öğretmen yapımı bir test ile klasik test kuramı ve madde tepki kuramı yöntemleriyle elde edilen puanlara göre karşılaştırılması. Journal of Measurement and Evaluation in Education and Psychology, 4(2), 64-75. https://dergipark.org.tr/tr/pub/epod/issue/5800/77213
4. De Ayala, R.J. (2009). The theory and practice of item response theory. The Guilford Press.
5. Hambleton, R., & Swaminathan, R. (1985). Fundementals of Item Response Theory. Sage Pub.
6. Hambleton, R., Swaminathan, R., & Rogers, H.J. (1991). Fundementals of Item Response Theory. Sage Pub.
7. Kohli, N., Koran, J., & Henn, L. (2015). Relationships among classical test theory and item response theory frameworks via factor analytic models. Educational and psychological measurement, 75(3), 389–405. https://doi.org/10.1177/0013164414559071
8. Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems. Routledge.

9. Magis, D. & Barrada, J.R. (2017). Computerized adaptive testing with R: Recent updates of the package catR. Journal of Statistical Software, Code Snippets, 76(1), 1-19. https://doi.org/10.18637/jss.v076.c01
10. Magis, D. & Raiche, G. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48(8), 1-31. https://doi.org/10.18637/jss.v048.i08
11. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174. https://doi.org/10.1007/BF02296272
12. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1), i-30. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
13. Pitman, J. (1993). Probability (6th Edition). Springer.
14. R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. URL https://www.R-project.org/.
15. Rasch, G. (1961). On general laws and the meaning of measurement in psychology. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability (pp. 321-333). University of California Press.
16. Raykov, T., & Marcoulides, G. A. (2016). On the relationship between classical test theory and item response theory: From one to the other and back. Educational and psychological measurement, 76(2), 325–338. https://doi.org/10.1177/0013164415576958
17. Reckase, D. (2009). Multidimensional Item Response Theory. Springer.
18. Samejima, F. (1996). Polychotomous responses and the test score. The University of Tennessee.
19. van der Linden, W. J. & Glas, G.A.W. (2022). Computerized Adaptive Testing: Theory and Practice. Kluwer Academic Publishers.