An Application of Multilevel Mixture Item Response Theory Model

Abstract

Although the mixture item response theory (IRT) models are useful for heterogeneous samples, they are not capable of handling a multilevel structure that is very common in education and causes dependency between hierarchies. Ignoring the hierarchical structure may yield less accurate results because of violation of the local independence assumption. This interdependency can be modeled straightforwardly in a multi-level framework. In this study, a large-scale data set, TEOG exam, was analyzed with a multilevel mixture IRT model to account for dependency and heterogeneity in the data set. Sixteen different multilevel models (different class solutions) were estimated using the eighth-grade mathematics data set. Model fit statistics for these 16 models suggested the CB1C4 model (one school-level and four student-level latent classes) was the best fit model. Based on CB1C4 model, the students were classified into four latent student groups and one latent school group. Parameter estimates obtained with maximum likelihood estimation were presented and interpreted. Several suggestions were made based on the results.

Keywords

• Item response theory
• mixture models
• multilevel mixture item response theory
• maximum likelihood estimation
• TEOG exam

References

1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. doi: 10.1109/TAC.1974.1100705
2. Asparouhov, T., & Muthén, B. (2008). Multilevel mixture models. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 27-51). Charlotte, NC: Information Age Publishing.
3. Bacci, S., & Gnaldi, M. (2015). A classification of university courses based on students’ satisfaction: An application of a two-level mixture item response model. Quality & Quantity, 49(3), 927-940. doi: 10.1007/s11135-014-0101-0
4. Bates, D. M., & DebRoy, S. (2004). Linear mixed models and penalized least squares. Journal of Multivariate Analysis, 91(1), 1-17. doi: 10.1016/j.jmva.2004.04.013
5. Bolt, D. M., Cohen, A. S., & Wollack, J. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331-348. doi: 10.1111/j.1745-3984.2002.tb01146.x
6. Cho, S. J., & Cohen, A. S. (2010). A multilevel mixture IRT model with an application to DIF. Journal of Educational and Behavioral Statistics, 35(3), 336-37. doi: 10.3102/1076998609353111
7. Choi, Y. J., Alexeev, N., & Cohen, A. S. (2015). Differential item functioning analysis using a mixture 3-parameter logistic model with a covariate on the TIMSS 2007 mathematics test. International Journal of Testing, 15(3), 239-253. doi: 10.1080/15305058.2015.1007241
8. Cohen, A. S., & Bolt, D. M. (2005). A mixture model analysis of differential item functioning. Journal of Educational Measurement, 42(2), 133-148. Retrieved from https://www.jstor.org/stable/20461782

9. Cohen, A. S., Gregg, N., & Deng, M. (2005). The role of extended time and item content on a high-stakes mathematics test. Learning Disabilities: Research and Practice, 20(4), 225-233. doi: 10.1111/j.1540-5826.2005.00138.x
10. de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Publications. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum Associates.
11. Finch, W. H., & Finch, M. E. H. (2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973-993. doi: 10.1177/0013164413494776
12. Jilke, S., Meuleman, B., & van de Walle, S. (2015). We need to compare, but how? Measurement equivalence in comparative public administration. Public Administration Review, 75(1), 36-48. doi: 10.1111/puar.12318
13. Lee, W. Y., Cho, S. J., & Sterba, S. K. (2018). Ignoring a multilevel structure in mixture item response models: Impact on parameter recovery and model selection. Applied Psychological Measurement, 42(2), 136-154. doi: 10.1177/0146621617711999
14. Li, F., Cohen, A. S., Kim, S. H., & Cho, S. J. (2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33(5), 499-518. doi: 10.1177/0146621608326422
15. Li, M., Liu, Y., & Liu, H. (2020). Analysis of the problem-solving strategies in computer-based dynamic assessment: The extension and application of multilevel mixture IRT model. Acta Psychologica Sinica, 52(4), 528-540. doi: 10.3724/SP.J.1041.2020.00528
16. Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, Article 1372. doi: 10.3389/fpsyg.2018.01372
17. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison- Wesley.
18. Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21-39. doi: 10.1037/1082-989X.10.1.21
19. Lukočienė, O., Varriale, R., & Vermunt, J. K. (2010). The simultaneous decision(s) about the number of lower-and higher-level classes in multilevel latent class analysis. Sociological Methodology, 40(1), 247-283. Retrieved from https://www.jstor.org/stable/41336886
20. McLachlan, G., &, Peel, D. (2000). Finite mixture models. New York: Wiley.
21. Mislevy, R. J., & Verhelst, N. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195-215. doi: 10.1007/BF02295283
22. Muthén, B. (2008). Latent variable hybrids: Overview of old and new models. In Hancock, G. R., & Samuelsen, K. M. (Eds.), Advances in latent variable mixture models (pp. 1-24). Charlotte, NC: Information Age Publishing, Inc.
23. Muthén, L. K., & Muthén, B. O. (1998-2018). Mplus users guide (7th ed.). Los Angeles, CA: Author.
24. Preinerstorfer, D., & Formann, A. K. (2011). Parameter recovery and model selection in mixed Rasch models. British Journal of Mathematical and Statistical Psychology, 65(2), 251-262. doi: 10.1111/j.2044-8317.2011.02020.x
25. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: SAGE.
26. Rost, J. (1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14(3), 271-282. doi: 10.1177/014662169001400305
27. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464. Retrieved from https://www.jstor.org/stable/2958889
28. Sclove, L. S. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333-343. doi: 10.1007/BF02294360
29. Sen, S., & Cohen A. S. (2020). The impact of test and sample characteristics on model selection and classification accuracy in the multilevel mixture IRT model. Frontiers in Psychology, 11, Article 197. doi: 10.3389/fpsyg.2020.00197
30. Sen, S., Cohen, A. S., & Kim, S. H. (2018). Model selection for multilevel mixture Rasch models. Applied Psychological Measurement, 43(4), 1-18. doi: 10.1177/0146621618779990
31. Şen, S., Cohen, A., & Kim, S.-H. (2020). A short note on obtaining item parameter estimates of IRT models with Bayesian estimation in Mplus. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 11(3), 266-282. doi: 10.21031/epod.693719
32. Tay, L., Diener, E., Drasgow, F., & Vermunt, J. K. (2011). Multilevel mixed-measurement IRT analysis: An explication and application to self-reported emotions across the world. Organizational Research Methods, 14(1), 177-207. doi: 10.1177/1094428110372674
33. van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Springer-Verlag.
34. Varriale, R., & Vermunt, J. K. (2012). Multilevel mixture factor models. Multivariate Behavioral Research, 47(2), 247-275. doi: 10.1080/00273171.2012.658337
35. Vermunt, J. K. (2008). Multilevel latent variable modeling: An application in education testing. Austrian Journal of Statistics, 37(3&4), 285-299.
36. Vermunt, J. K. (2011). Mixture models for multilevel data sets. In J. Hox & K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 59-81). New York: Routledge.