Improved Performance of Model Fit Indices with Small Sample Sizes in Cognitive Diagnostic Models

Abstract

Selecting an appropriate cognitive diagnostic model (CDM) for data analysis is always challenging. Studies have explored several model fit indices for CDMs. The common results of these studies indicate that Q-matrix misspecifications lead to poor performance of the model fit indices in the context of CDMs. Thus, this study explored whether model fit indices improve performance with a modified Q-matrix. The average class size has reduced to 23 students in Taiwan because of the low birth rate; therefore, the study sought the effect of sample size on the performance of model fit indices. The results showed that Akaike’s information criterion (AIC) was an excellent model fit index in small samples. Model fit indices with the modified Q-matrix presented superior performance.

Keywords

• RSS
• ζ2 index
• model fit indices
• cognitive diagnostic models
• Q-matrix

References

1. Akaike, H. (1974). A new look at the statistical identification model. IEEE Transactions on Automated Control, 19, 716-723.
2. DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 979-1030). Amsterdam, Netherlands: Elsevier.
3. Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnostic modeling. Journal of Educational Measurement, 50, 123-140.
4. Chiu, C.Y. (2013). Statistical refinement of the Q-Matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598-618.
5. Chiu, C.-Y., & Douglas, J. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30(2), 225-250.
6. de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362.
7. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
8. de la Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50(4), 355-373.

9. de la Torre, J., & Chiu, C. Y. (2016). A General Method of Empirical Q-matrix Validation. Psychometrika, 81(2), 253-273.
10. Hu, J., Miller, M. D., Huggins-Manley, A. C., & Chen, Y. (2016). Evaluation of Model Fit in Cognitive Diagnosis Models. International Journal of Testing, 16(2), 119-141.
11. Jiao, H. (2009). Diagnostic classification models: Which one should I use? Measurement: Interdisciplinary Research & Perspective, 7(1), 65-67.
12. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
13. Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.
14. Lei, P.-W., & Li, H. (2016). Choosing Correct Cognitive Diagnostic Models and Q-Matrices. Applied Psychological Measurement, 40(6), 1-12.
15. Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model Similarity, Model Selection, and Attribute Classification. Applied Psychological Measurement, 40(3), 200-217.
16. Ma, W., & de la Torre, J. (2018). GDINA: The generalized DINA model framework. R package version 2.3. Retrived from https://CRAN.R-project.org/package=GDINA
17. R Core Team (2017). R: A language and environment for statistical computing (Version 3.4.3) [Computing software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org
18. Rupp, A. A., & Templin, J. (2008). The effects of Q-matrix misspecification nonparameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96.
19. Schwarzer, G. (1976). Estimating the dimension of a model. Annals of Statistics, 6,461–464.
20. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354.
21. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.