A Comparison of the Classification Performances of the DINO Model, Artificial Neural Networks and Non-Parametric Cognitive Diagnosis

Abstract

The purpose of this study was to compare the attribute (ACR) and pattern-level (PCR) classification rates of the Deterministic-Input, Noisy-Or Gate (DINO) model, Artificial Neural Networks (ANNs), and Non-parametric Cognitive Diagnosis (NPCD) from datasets of simulation in various conditions such as the number of attributes, sample size, the number of items and missing data rate. A further purpose was to examine the similarities between the classification rates of the DINO model ANNs, and NPCD on the PISA 2015 collaborative problem-solving (CPS) datasets in various numbers of attributes and sample sizes. For the purpose of the study, simulation datasets were generated on the basis of the complex Q matrix structures and the DINO model. The conditions for the sample size factor for the real datasets were determined by simple random selection among the participants in the PISA 2015 administration. As a result, it was found that there was a similarity between the DINO model and NPCD classification rates in both simulation and real datasets. In addition, regarding the increase in sample size in both simulation and real datasets, no consistency was found in the increase or decrease of the classification rates of ANNs and NPCD and the similarities of these rates.

Keywords

• diagnostic assessment
• non-parametric cognitive diagnosis
• artificial neural network
• DINO
• PISA

References

1. Akbay, L. (2016). Relative efficiency of the nonparametric approach on attribute classification for small sample cases. Journal of European Education 6(1), 43-59.
2. Chiu, C.-Y., & Douglas, J. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30, 225-250. doi:10.1007/s00357-013-9132-9
3. Chiu, C.-Y., Douglas, J., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633-665.
4. Chiu, C.-Y., & Köhn, H.-F. (2019). Consistency theory for the general nonparametric classification method. Psychometrika, 84(3), 830-845.
5. Chiu, C.-Y., Sun, Y., & Bian, Y. (2017). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355-375.
6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Erlbaum.
7. Cui, Y., Gierl, M., & Guo, Q. (2016). Statistical classification for cognitive diagnostic assessment: An artificial neural network approach. Educational Psychology: An International Journal of Experimental Educational Psychology, 36(6), 1065-1082.
8. Cui, Y., Gierl, M., & Guo, Q. (2017). The rule space and attribute hierarchy methods. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment frameworks, methodologies, and applications (pp. 354-378). John Wiley & Sons.

9. Dai, S. (2017). Investigation of missing responses in implementation of cognitive diagnostic models. (Doctoral dissertation). Indiana University, Boston.
10. Dai, S., Wang, X., & Svetina, D. (2019, March). Test Data Imputation [software package in R]. https://cran.r-project.org/web/packages/TestDataImputation/index.html
11. de la Torre, J. (2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362.
12. de la Torre, J., & Chiu, C.-Y. (2015). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253-273.
13. de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595-624.
14. de la Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227-249.
15. Fritsch, S., Guenther, F., Wright, M. N., Suling, M., & Mueller, S. M. (2019, February). Neuralnet [software package in R]. https://cran.r-project.org/web/packages/neuralnet/neuralnet.pdf
16. Garson, G. D. (1998). Neural networks: An introductory guide for social scientists. Sage.
17. Gierl, M. J., Cui, Y., & Hunka, S. (2007, April). Using connectionist models to evaluate examinees’ response patterns on tests. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL, USA.
18. Gierl, M. J., Cui, Y., & Hunka, S. (2008). Using connectionist models to evaluate examinees’ response patterns on tests. Journal of Modern Applied Statistical Methods, 7, 234-245. doi:10.22237/jmasm/1209615480
19. Guo, Q., Cutumisu, M., Cui, Y. (2017). A neural network approach to estimate student skill mastery in cognitive diagnostic assessments. Poster presented at the 10th International Conference on Educational Data Mining, Wuhan, Hubei Province, in China.
20. He, Q., von Davier, M., Greiff, S., Steinhauer, E. W., & Borysewicz, P. B. (2017). Collaborative problem-solving measures in the Programme for International Student Assessment (PISA). In A. A. von Davier, M. Zhu & P. C. Kyllonen (Eds.), Methodology of educational measurement and assessment: Innovative assessment of collaboration (pp. 95-112). Springer.
21. IBM Corp. (2019). IBM SPSS Statistics for Windows (Version 22.0) [Computer software]. https://www.ibm.com/tr-tr/analytics/spss-statistics-software
22. Lim, Y. S., & Drasgow, F. (2017): Nonparametric calibration of ıtem-by-attribute matrix in cognitive diagnosis. Multivariate Behavioral Research, 52(5), 562-575.
23. Ma, C., de la Torre, J., & Xu, G. (2020). Bridging Parametric and Nonparametric Methods in Cognitive Diagnosis. Retrieved from arXiv:2006.15409
24. Ma, W., de la Torre, J., Sorrel, M. & Jiang, Z. (2020, May). GDINA [software package in R]. https://cran.r-project.org/web/packages/GDINA/index.html
25. McCoy, T., & Willse, J. (2014, April). Accuracy of neural network versus nonparametric approaches in diagnostic classification. Paper presented at the National Council on Measurement in Education, Washington, DC, USA.
26. No Child Left Behind Act of 2001, Pub. L. No. 107-110. Retrieved from http://thomas.loc.gov/
27. OECD (2017a). PISA 2015 collaborative problem-solving framework. France. https://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf
28. OECD (2017b). PISA 2015 results (Volume V): Collaborative problem solving. France. https://www.oecd.org/publications/pisa-2015-results-volume-v-9789264285521-en.htm#:~:text=PISA%202015%20Results%20(Volume%20V)%3A%20Collaborative%20Problem%20Solving%2C%20is,try%20to%20solve%20a%20problem
29. Paulsen, J. (2019). Examining cognitive diagnostic modeling in small sample contexts. (Doctoral dissertation). Indiana University, Boston.
30. Paulsen, J. & Valdivia, D. S. (2022). Examining cognitive diagnostic modeling in classroom assessment conditions. Journal of Experimental Education, 90(4), 916-933. doi:10.1080/00220973.2021.1891008
31. Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2019, September). CDM [software package in R]. https://cran.r-project.org/web/packages/CDM/index.html
32. Rupp, A. A., & Templin, J. (2008a). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96.
33. Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement. Guilford.
34. Shu, Z., Henson, R., & Willse, J. (2013). Using neural network analysis to define methods of DINA model estimation for small sample sizes. Journal of Classification, 30, 173-194. doi:10.1007/s00357-013-9134-7
35. Shuying, S. (2016). Nonparametric diagnostic classification analysis for testlet based tests. (Doctoral dissertation). The University of North Carolina at Greensboro, USA.
36. Stekhoven, D. J. (2016, August). missForest [software package in R]. https://cran.r-project.org/web/packages/missForest/missForest.pdf
37. Sünbül, S. (2018). The impact of different missing data handling methods on DINA model. International Journal of Evaluation and Research in Education, 7(1), 77-86.
38. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
39. Wang, M.-J. (2015). Computer-based assessment of collaborative problem-solving for intermediate elementary students with one on one, human-to-agent approach. (Master’s dissertation). National Taichung University, Taiwan.
40. Wang, S., & Douglas, J. (2015). Consistency of nonparametric classification in cognitive diagnosis. Psychometrika, 80(1), 85-100.
41. Yavuz, E. (2014). Determining the problem solving process skills of the primary education pre-service mathematics teachers as defined in PISA. Gazi University, Ankara.
42. Zheng, Y., Chiu, C.-Y., & Douglas, J. A. (2019, November). NPCD [software package in R]. https://cran.r-project.org/web/packages/NPCD/NPCD.pd