Monitoring Student Achievement with Cognitive Diagnosis Model

Abstract

In this study, it is aimed to show how student achievement can be monitored by using the cognitive diagnosis models. For this purpose, responses of the 6th, 7th, and 8th grade Mathematics subtests of High School Placement Tests (HSPT) in 2009, 2010, and 2011, which provide longitudinal data, were used, respectively. There were 49933 examiners’ responses in data sets. The attributes examined by these tests were determined by the Mathematics experts, and the Q matrix consisting of five attributes was developed. As a result of the analysis, it was seen that the largest latent class for all three years consisted of those non-master for any attribute. It was observed that the probability of attribute mastery increased in the 7th grade and decreased in the 8th grade. The high classification accuracy seen as a result of the analysis applied to HSPT, which is not intended for the cognitive diagnosis, shows that the results can be used for monitoring student achievement.

Keywords

• Cognitive diagnosis
• student achievement
• g-dina
• attribute mastery probability
• longitudinal data

References

1. Akaike, H. (1974). A new look at the statistical identification model. IEEE Transactions on Automated Control, 19,716–723. doi:10.1109/TAC.1974.1100705
2. Akbay, L., Terzi, R., Kaplan, M., &Karaaslan, K. G. (2017). Expert-based attribute identification and validation on fraction subtraction: A cognitively diagnostic assessment application. Journal on Mathematics Education, 9(1), 103-120.
3. Chen, H., ve Chen, J., (2016). Retrofitting Non-cognitive-diagnostic Reading Assessment Under the Generalized DINA Model Framework, Language Assessment Quarterly,13(3), 218-230, DOI: 10.1080/15434303.2016.1210610
4. Cheng, Y. (2010). Improving cognitive diagnostic computerized adaptive testing by balancing attribute coverage: the modified maximum global discrimination index method. Educational and Psychological Measurement, 70 (6), 902-913.
5. Cui, Y., Gierl, M. J., & Chang, H. H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49(1), 19-38.
6. de la Torre, J. (2009). A cognitive diagnosis model for cognitively-based multiple-choice options. Applied Psychological Measurement, 33(3), 163–183.
7. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
8. de la Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227-249.

9. de la Torre, J., &Minchen N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. PsicologíaEducativa, 20(2), 89-97.
10. de la Torre, J., van der Ark, L. A., & Rossi, G. (2017). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 1-16.
11. DiBello, L. V., Stout, W. F., & Roussos, L. A. (2007). Cognitive diagnosis Part I. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26): Psychometrics (pp. 979-1029). Amsterdam: Elsevier
12. Embretson, S. E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380-396
13. Hartz, S. M. (2002). A bayesianframework for the unified model for assessing cognitive abilities: Blending theory with practicality, Unpublished PhD dissertation, University of Illinois at Urbana-Champaign, ABD.
14. Huang, H. (2017), Multilevel cognitive diagnosis models for assessing changes in latent attributes. Journal of Educational Measurement, 54: 440-480. doi: 10.1111/jedm.12156
15. Iaconangelo, C.(2017). Uses of classification error probabilities in the three-step approach to estimating cognitive diagnosis models. (Unpublished doctoral dissertation). New Brunswick, NJ: Rutgers University.
16. Junker, B. W., &Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
17. Kablan, Z., Baran, T. & Hazer, Ö. (2013). İlköğretim matematik 6-8 öğretim programında hedeflenen davranışların bilişsel süreçler açısından incelenmesi. Ahi Evran Üniversitesi Kırşehir Eğitim Fakültesi Dergisi, 14 (1), 347- 366.
18. Kaya, Y., &Leite, W. L. (2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 77(3), 369-388.
19. Kutlu, Ö., Doğan, C., & Karakaya, İ. (2010). Öğrenci başarısının belirlenmesi: Performansa ve portfolyoya dayalı durum belirleme. Pegem, Ankara
20. Li, L., Zhou, X., Huang, J., Tu, D., Gao, X., Yang, Z., & Li, M. (2020). Assessing kindergarteners’ mathematics problem solving: The development of a cognitive diagnostic test. Studies in Educational Evaluation, 66, 100879.
21. Liu, R., Huggins-Manley, A. C., &Bulut, O. (2018). Retrofitting diagnostic classification models to responses from irt-based assessment forms. Educational and Psychological Measurement, 78(3), 357–383. https://doi.org/10.1177/0013164416685599
22. Leighton, J. P. veGierl M. J. (2007). Why cognitive diagnostic assessment, Leighton, J. P. Gierl M. J. (Eds). Cognitive Diagnostic Assessment for Education. Cambridge University Press, New York, USA.
23. Lohman, D. F. (1999). Minding our p's and q's: On finding relationships between learning and intelligence. In P. L. Ackerman, P. C. Kyllonen, & R. D. Roberts (Eds.), The future of learning and individual differences: Process, traits, and content (55f72). Washington, DC: American Psychological Association.
24. Ma, W. & de la Torre, J. (2019). GDINA: The generalized DINA model framework. R package version 2.3. Retrived from https://CRAN.R-project.org/package=GDINA
25. Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200-217.
26. Madison, M. J., & Bradshaw, L. P. (2015). The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model. Educational and Psychological Measurement, 75(3), 491–511. https://doi.org/10.1177/0013164414539162
27. Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
28. Ministry of National Education (MoNE) (2018). Matematik dersi öğretim programı (İlkokul ve Ortaokul 1, 2, 3, 4, 5, 6, 7 ve 8. Sınıflar). Ankara: Talim ve Terbiye Kurulu Başkanlığı.
29. Ministry of National Education (MoNE), (2008), 64 Soruda Ortaöğretime Geçiş Sistemi ve Seviye Belirleme Sınavı Örnek Sorular. Ankara: MEB Yayınları
30. Ministry of National Education (MoNE), (2010), Seviye belirleme sınavının değerlendirilmesi. Turkish Ministry of National Education, Retrieved from https://www.meb.gov.tr/earged/earged/sbs_deger.pdf
31. Ministry of National Education (MoNE), (2017), Akademik becerilerin izlenmesi ve değerlendirilmesi: 8. Sınıflar raporu. Turkish Ministry of National Education, Retrieved from http://edirne.meb.gov.tr/meb_iys_dosyalar/2018_06/08104327_ABYDE_Turkiye.pdf.
32. Organisation for Economic Co-operation and Development (OECD), (2019), PISA 2018: Insights and Interpretations., Retrieved from https://www.oecd.org/pisa/PISA%202018%20Insights%20and%20Interpretations%20FINAL%20PDF..
33. Rupp, A. A., & Templin, J. (2008). The effects of q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96. https://doi.org/10.1177/0013164407301545
34. Schwarzer, G. (1976). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. doi:10.1214/aos/1176344136
35. Sezen Yüksel, N., Sağlam Kaya, Y., Urhan, S. & Şefik, Ö. (2019). Matematik Eğitiminde Modelleme Etkinlikleri (Ed: Şenol Dost). Ankara: Pegem Akademi
36. Smith, G.H., Wood, L.N., Coupland, M., Stephenson, B., Crawford, K. &Ball, G. (1996). Constructing mathematical examinations to assess a range of knowledge and skills. Int. J. Math. Educ. Sci. Technol., 27(1), 65-77.
37. Sorrel, M., Olea, J., Abad, F., de la Torre, J., Aguado, D., &Lievens, F. (2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods. 19(3), 506-532, doi: 10.1177/1094428116630065
38. Şen, S., &Arıcan, M. (2015). A diagnostic comparison of Turkish and Korean students’ mathematics performances on the TIMSS 2011 assessment. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 238-253.
39. Tatsuoka, K. K., Corter, J. E., &Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41(4), 901-926.
40. Templin, J., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.
41. Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Education Research Journal, 26, 237-255.
42. Uğurel, I., Moralı, H.S. &Kesgin, Ş. (2012). OKS, SBS ve TIMSS matematik sorularının ‘math taksonomi’ çerçevesinde karşılaştırmalı analizi. Gaziantep Üniversitesi Sosyal Bilimler Dergisi, 11 (2), 423- 444.
43. von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307. doi:10.1348/000711007X193957
44. Wang, S., Yang, Y., Culpepper, S. A., & Douglas, J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: a higher-order, hidden markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57-87.
45. Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52 , 457-476
46. Yakar, L. (2011). İlköğretim ikinci kademe öğrencilerinin SBS puanları ve akademik başarı puanları değişimlerinin izlenmesi ve SBS puanlarının kestirilmesi. Unpublished Master dissertation. Abant İzzet Baysal Üniversitesi /Eğitim Bilimleri Enstitüsü, Bolu, Turkey.
47. Zhan, P., Jiao, H., Liao, D., & Li, F. (2019). A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 44(3), 251-281.