A Comparison of the Classification Performances of the DINO Model, Artificial Neural Networks and Non-Parametric Cognitive Diagnosis

Year 2023, Volume: 14 Issue: 4, 413 - 439, 31.12.2023

Emine Yavuz , Hakan Yavuz Atar

https://doi.org/10.21031/epod.1246752

Abstract

The purpose of this study was to compare the attribute (ACR) and pattern-level (PCR) classification rates of the Deterministic-Input, Noisy-Or Gate (DINO) model, Artificial Neural Networks (ANNs), and Non-parametric Cognitive Diagnosis (NPCD) from datasets of simulation in various conditions such as the number of attributes, sample size, the number of items and missing data rate. A further purpose was to examine the similarities between the classification rates of the DINO model ANNs, and NPCD on the PISA 2015 collaborative problem-solving (CPS) datasets in various numbers of attributes and sample sizes. For the purpose of the study, simulation datasets were generated on the basis of the complex Q matrix structures and the DINO model. The conditions for the sample size factor for the real datasets were determined by simple random selection among the participants in the PISA 2015 administration. As a result, it was found that there was a similarity between the DINO model and NPCD classification rates in both simulation and real datasets. In addition, regarding the increase in sample size in both simulation and real datasets, no consistency was found in the increase or decrease of the classification rates of ANNs and NPCD and the similarities of these rates.

Keywords

diagnostic assessment, non-parametric cognitive diagnosis, artificial neural network, DINO, PISA

Supporting Institution

This study was supported by the Scientific and Technological Research Council of Turkey under Grant 2228-B.

Project Number

TÜBİTAK 2228-B

Thanks

I would like to thank TÜBİTAK for its support.

References

• Akbay, L. (2016). Relative efficiency of the nonparametric approach on attribute classification for small sample cases. Journal of European Education 6(1), 43-59.
• Chiu, C.-Y., & Douglas, J. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30, 225-250. doi:10.1007/s00357-013-9132-9
• Chiu, C.-Y., Douglas, J., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633-665.
• Chiu, C.-Y., & Köhn, H.-F. (2019). Consistency theory for the general nonparametric classification method. Psychometrika, 84(3), 830-845.
• Chiu, C.-Y., Sun, Y., & Bian, Y. (2017). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355-375.
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Erlbaum.
• Cui, Y., Gierl, M., & Guo, Q. (2016). Statistical classification for cognitive diagnostic assessment: An artificial neural network approach. Educational Psychology: An International Journal of Experimental Educational Psychology, 36(6), 1065-1082.
• Cui, Y., Gierl, M., & Guo, Q. (2017). The rule space and attribute hierarchy methods. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment frameworks, methodologies, and applications (pp. 354-378). John Wiley & Sons.
• Dai, S. (2017). Investigation of missing responses in implementation of cognitive diagnostic models. (Doctoral dissertation). Indiana University, Boston.
• Dai, S., Wang, X., & Svetina, D. (2019, March). Test Data Imputation [software package in R]. https://cran.r-project.org/web/packages/TestDataImputation/index.html
• de la Torre, J. (2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362.
• de la Torre, J., & Chiu, C.-Y. (2015). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253-273.
• de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595-624.
• de la Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227-249.
• Fritsch, S., Guenther, F., Wright, M. N., Suling, M., & Mueller, S. M. (2019, February). Neuralnet [software package in R]. https://cran.r-project.org/web/packages/neuralnet/neuralnet.pdf
• Garson, G. D. (1998). Neural networks: An introductory guide for social scientists. Sage.
• Gierl, M. J., Cui, Y., & Hunka, S. (2007, April). Using connectionist models to evaluate examinees’ response patterns on tests. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL, USA.
• Gierl, M. J., Cui, Y., & Hunka, S. (2008). Using connectionist models to evaluate examinees’ response patterns on tests. Journal of Modern Applied Statistical Methods, 7, 234-245. doi:10.22237/jmasm/1209615480
• Guo, Q., Cutumisu, M., Cui, Y. (2017). A neural network approach to estimate student skill mastery in cognitive diagnostic assessments. Poster presented at the 10th International Conference on Educational Data Mining, Wuhan, Hubei Province, in China.
• He, Q., von Davier, M., Greiff, S., Steinhauer, E. W., & Borysewicz, P. B. (2017). Collaborative problem-solving measures in the Programme for International Student Assessment (PISA). In A. A. von Davier, M. Zhu & P. C. Kyllonen (Eds.), Methodology of educational measurement and assessment: Innovative assessment of collaboration (pp. 95-112). Springer.
• IBM Corp. (2019). IBM SPSS Statistics for Windows (Version 22.0) [Computer software]. https://www.ibm.com/tr-tr/analytics/spss-statistics-software
• Lim, Y. S., & Drasgow, F. (2017): Nonparametric calibration of ıtem-by-attribute matrix in cognitive diagnosis. Multivariate Behavioral Research, 52(5), 562-575.
• Ma, C., de la Torre, J., & Xu, G. (2020). Bridging Parametric and Nonparametric Methods in Cognitive Diagnosis. Retrieved from arXiv:2006.15409
• Ma, W., de la Torre, J., Sorrel, M. & Jiang, Z. (2020, May). GDINA [software package in R]. https://cran.r-project.org/web/packages/GDINA/index.html
• McCoy, T., & Willse, J. (2014, April). Accuracy of neural network versus nonparametric approaches in diagnostic classification. Paper presented at the National Council on Measurement in Education, Washington, DC, USA.
• No Child Left Behind Act of 2001, Pub. L. No. 107-110. Retrieved from http://thomas.loc.gov/
• OECD (2017a). PISA 2015 collaborative problem-solving framework. France. https://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf
• OECD (2017b). PISA 2015 results (Volume V): Collaborative problem solving. France. https://www.oecd.org/publications/pisa-2015-results-volume-v-9789264285521-en.htm#:~:text=PISA%202015%20Results%20(Volume%20V)%3A%20Collaborative%20Problem%20Solving%2C%20is,try%20to%20solve%20a%20problem
• Paulsen, J. (2019). Examining cognitive diagnostic modeling in small sample contexts. (Doctoral dissertation). Indiana University, Boston.
• Paulsen, J. & Valdivia, D. S. (2022). Examining cognitive diagnostic modeling in classroom assessment conditions. Journal of Experimental Education, 90(4), 916-933. doi:10.1080/00220973.2021.1891008
• Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2019, September). CDM [software package in R]. https://cran.r-project.org/web/packages/CDM/index.html
• Rupp, A. A., & Templin, J. (2008a). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96.
• Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement. Guilford.
• Shu, Z., Henson, R., & Willse, J. (2013). Using neural network analysis to define methods of DINA model estimation for small sample sizes. Journal of Classification, 30, 173-194. doi:10.1007/s00357-013-9134-7
• Shuying, S. (2016). Nonparametric diagnostic classification analysis for testlet based tests. (Doctoral dissertation). The University of North Carolina at Greensboro, USA.
• Stekhoven, D. J. (2016, August). missForest [software package in R]. https://cran.r-project.org/web/packages/missForest/missForest.pdf
• Sünbül, S. (2018). The impact of different missing data handling methods on DINA model. International Journal of Evaluation and Research in Education, 7(1), 77-86.
• Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
• Wang, M.-J. (2015). Computer-based assessment of collaborative problem-solving for intermediate elementary students with one on one, human-to-agent approach. (Master’s dissertation). National Taichung University, Taiwan.
• Wang, S., & Douglas, J. (2015). Consistency of nonparametric classification in cognitive diagnosis. Psychometrika, 80(1), 85-100.
• Yavuz, E. (2014). Determining the problem solving process skills of the primary education pre-service mathematics teachers as defined in PISA. Gazi University, Ankara.
• Zheng, Y., Chiu, C.-Y., & Douglas, J. A. (2019, November). NPCD [software package in R]. https://cran.r-project.org/web/packages/NPCD/NPCD.pd