Determination of the Most Appropriate Reduced Model Defined at the Item Level in Cognitive Diagnostic Models

Abstract

This study aims to identify the most suitable reduced model at the item level for items requiring a minimum of two attributes in the Q-matrix utilizing the WALD, LR, 2LR, and LM test methods. Additionally, the study aims to compare the relative and absolute model fit values of the reduced model for each item with those of the G-DINA model. The research participants consists of 712 9th grade students attending seven different high schools in Ankara. Data were collected from a diagnostic mathematics test comprising 26 multiple-choice items developed by the researcher to assess five attributes. The analysis employed the "GDINA" package version 2.9.4 within RStudio software. The study's findings indicate that the LLM model serves as the most appropriate reduced model for items based on the Wald, LR, and 2LR test methods, while the DINA model provides the most suitable reduced model for items in the LM test method. All models exhibited good absolute model fit values; with the G-DINA model displaying the lowest RMSEA value, and the TMLM model showing the lowest SRMSR value. It was determined that TMwald and TMLM models did not exhibit as good a fit as the G-DINA model, whereas TMLR and TM2LR models displayed a model fit as good as that of the G-DINA model. Consequently, the study concludes that these models can be simplified at the item level and used as alternativesto the G-DINA model. It is recommended that researchers undertake further investigations into the methods for selecting item-level models that are both reliably and validly.

Keywords

• Cognitive Diagnostic Model
• G-DINA Model
• Item-Level Model
• Model Fit.

Bilişsel Tanı Modellerinde Madde Düzeyinde Tanımlanmış En Uygun İndirgenmiş Modelinin Belirlenmesi

Öz

Bu çalışmanın amacı, Q-matrisinde en az iki nitelik gerektiren maddeler için madde düzeyinde tanımlanan en uygun indirgenmiş modelin WALD, LR, 2LR ve LM test yöntemlerine göre belirlenmesidir. Ayrıca her bir madde için belirlenen indirgenmiş model kullanılarak tanımlanan modelin bağıl ve mutlak model uyum değerlerini G-DINA model uyum değerleriyle karşılaştırmak amaçlanmıştır. Araştırmanın çalışma grubu Ankara ilinde yedi farklı lisede 9.sınıfta öğrenim gören 712 öğrenciden oluşmaktadır. Çalışmada verileri araştırmacı tarafından geliştirilen beş niteliğin yoklandığı, çoktan seçmeli ve 26 maddeden oluşan tanılayıcı bir matematik testinden elde edilmiştir. Verilerin analizi RStudio yazılımında “GDINA” paket 2.9.4 versiyonu ile gerçekleştirilmiştir. Çalışma sonucunda, Wald, LR ve 2LR test yöntemlerinde en çok LLM modelinin, LM test yönteminde ise DINA modelin maddeler için en uygun indirgenmiş model olarak belirlendiği tespit edilmiştir. Tüm modellerde mutlak model uyum değerlerinin iyi olduğu, en düşük RMSEA değerinin G-DINA modelde, en düşük SRMSR değerinin ise TMLM modelinde olduğu belirlenmiştir. TMwald ve TMLM modellerinin G-DINA model kadar iyi uyuma sahip olmadığı, TMLR ve TM2LR modellerinin G-DINA model kadar iyi model uyumuna sahip olduğu belirlenmiştir. Bu modellerde madde düzeyinde sadeleştirmeye gidilebileceği ve G-DINA model yerine sadeleştirilmiş bu modellerin kullanılabileceği sonucuna ulaşılmıştır. Araştırmacılara madde düzeyinde hangi yöntem kullanılarak daha güvenilir ve geçerli bir şekilde model seçilebileceği konusunda çalışmalar yapması önerilmektedir.

Anahtar Kelimeler

• Bilişsel Tanı Modeli
• G-DINA Model
• Madde Düzeyinde Model
• Model Uyumu.

References

1. Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123-140. https://doi.org/10.1111/j.1745-3984.2012.00185.x
2. De Ayala, R. J. (2009). The theory and practice of item response theory. Methodology in the social sciences. Guildford Publications.
3. De Carlo, L. T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35(1), 8-26. https://doi.org/10.1177/0146621610377081
4. de La Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163-183. https://doi.org/10.1177/0146621608320523
5. de la Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item‐level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50(4), 355-373. https://doi.org/10.1111/jedm.12022
6. Hartz, S., Roussos, L., & Stout, W. (2002). Skills diagnosis: Theory and practice [User Manual for Arpeggio software]. Educational Testing Service.
7. Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6 (1), 1-55. https://doi.org/10.1080/10705519909540118
8. Li, X., & Wang, W. C. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28-54. https://doi.org/10.1111/jedm.12061

9. Liu, Y., Andersson, B., Xin, T., Zhang, H., & Wang, L. (2019). Improved Wald statistics for item-level model comparison in diagnostic classification models. Applied Psychological Measurement, 43(5), 402-414. https://doi.org/10.1177/0146621618798664
10. Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection and attribute classification. Applied Psychological Measurement, 40, 200-217. https://doi.org/10.1177/0146621615621717
11. Ma, W. (2017). A Sequential cognitive diagnosis model for graded response: Model development, Q-matrix validation, and model comparison. [Unpublished doctoral dissertation]. Rutgers University.
12. Ma, W., & de la Torre, J. (2019a). Category-level model selection for the sequential G-DINA model. Journal of Educational and Behavioral Statistics. 44, 61-82. https://doi.org/10.3102/1076998618792484
13. Ma, W. & de la Torre, J. (2019b). GDINA: The generalized DINA model framework (R package version 2.7.3.) https://CRAN.R-project.org/package=GDINA
14. Ma, W., de la Torre, J., Sorrel, M., & Jiang, Z. (2023). GDINA: The generalized DINA model framework (R package version 2.9.4.) https://cran.r-project.org/web/packages/GDINA/GDINA.pdf
15. Maydeu-Olivares, A., & Joe, H. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328. https://doi.org/10.1080/00273171.2014.911075
16. Rojas G., de la Torre J., Olea J. (2012, April 14-16). Choosing between general and specific cognitive diagnosis models when the sample size is small [Paper presentation]. National Council on Measurement in Education Annual Meeting, Vancouver, British Columbia, Canada.
17. Sorrel, M. A., de la Torre, J., Abad, F. J., & Olea, J. (2017a). Two-step likelihood ratio test for item-level model comparison in cognitive diagnosis models. Methodology, 13, 39-47. https://doi.org/10.1027/1614-2241/a000131
18. Sorrel, M. A., Abad, F. J., Olea, J., de la Torre, J., & Barrada, J. R. (2017b). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41, 614-631. https://doi.org/10.1177/0146621617707510