inujfe İnönü University Journal of the Faculty of Education 1300-2899 2149-9683 Inonu University 10.17679/inuefd.1594482 Setting Standards and Norms Standart Belirleme ve Normlar Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study Matematik Sorularının Bilişsel Tanılama Modeliyle İncelenmesi: Q-Matris Çalışması https://orcid.org/0000-0003-3042-7585 Takır Aygil Atatürk Öğretmen Akademisi https://orcid.org/0000-0003-1094-3590 Özder Hasan Atatürk Öğretmen Akademisi https://orcid.org/0000-0002-4765-9297 Cankoy Osman ORTA DOĞU TEKNİK ÜNİVERSİTESİ 09 02 2025 26 2 759 796 12 05 2024 06 03 2025 Copyright © 2002, İnönü University Journal of the Faculty of Education 2002 İnönü University Journal of the Faculty of Education

The College Entrance Examination (CEE) is a curriculum-orientated performance examination administered to fifth-grade primary school students who wish to study at Maarif Colleges in Northern Cyprus, where the medium of instruction is English. This study was conducted to determine the psychometric characteristics of the CEE mathematics test administered to 1,833 fifth-grade primary school students in 2021. To achieve the purpose of the study, a cognitive diagnostic model, namely DINA, was used. The theoretical infrastructure of the DINA model is easy and strong compared to other models. The Q-matrix was constructed for 23 mathematics questions and 15 attributes were determined based on expert opinions. The results of the study showed that the mean values of the parameters g and s were 0.25 and 0.13, respectively. The mean of the δ values of the fifth-grade students was high (0.73). Skill patterns showed that 13% of fifth-grade students possessed all the attributes included in CEE. The construction of the Q-matrix is a very important stage in the use of CDMs. The cognitive attributes and the Q-matrix were determined by expert opinion in this study. The (1-s) values indicated the validity of the Q-matrix. For all of the items, 1-s values appeared to be quite high. This showed that the attainments and item-attainment relationship were done as accurately as possible. These results can be interpreted as the validity of the Q- matrix. The results of the study showed that the DINA model provides a detailed conclusion about students’ attributes.

Kolej Giriş Sınavı (KGS), Kuzey Kıbrıs'ta eğitim dili İngilizce olan Maarif Kolejleri'nde okumak isteyen beşinci sınıf ilkokul öğrencilerine uygulanan program odaklı bir performans sınavıdır. Bu çalışma, 2021 yılında 1.833 beşinci sınıf ilkokul öğrencisine uygulanan KGS matematik testinin psikometrik özelliklerini belirlemek amacıyla gerçekleştirilmiştir. Çalışmanın amacına ulaşmak için DINA adlı bilişsel tanı modeli kullanılmıştır. DINA modelinin teorik altyapısı, diğer modellere göre hem kolay hem de güçlüdür. 23 matematik sorusu için Q matrisi oluşturulmuş ve uzman görüşlerine dayanarak 15 özellik belirlenmiştir. Çalışmanın sonuçları, g ve s parametrelerinin ortalama değerlerinin sırasıyla 0,25 ve 0,13 olduğunu göstermiştir. Bulgular, beşinci sınıf öğrencilerinin δ değerlerinin ortalamasının yüksek olduğunu (0,73) ortaya koymuştur. Beceri örüntüleri, beşinci sınıf öğrencilerinin %13'ünün KGS'de yer alan tüm niteliklere sahip olduğunu göstermektedir. Q matrisinin oluşturulması, Bilişsel Tanı Modelleri'nin (BTM) kullanımında çok önemli bir aşamadır. Bu çalışmada bilişsel nitelikler ve Q matrisi, uzman görüşü ile belirlenmiştir. (1-s) değerleri, Q matrisinin geçerliliğini göstermektedir. Tüm maddeler için 1-s değerlerinin oldukça yüksek olduğu gözlemlenmiştir. Bu durum, kazanımların ve madde-başarı ilişkisinin mümkün olduğunca doğru bir şekilde yapıldığını göstermekte ve bu sonuçlar, Q matrisinin geçerliliği olarak yorumlanabilir. Çalışmanın sonuçları, DINA modelinin öğrencilerin nitelikleri hakkında ayrıntılı bir sonuç sağladığını göstermektedir.

 DINA Cognitive diagnosis model college entrance examination mathematics assessment DINA Bilişsel Tanı Modeli Kolej Giriş Sınavı Matematik Değerlendirmesi Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705 Birenbaum, M., Tatsuoka, C., & Yamada, T. (2004). Diagnostic assessment in TIMSS-R: Between countries and within-country comparisons of eighth graders’ mathematics performance. Studies in Educational Evaluation, 30(2), 151-173. https://doi.org/10.1016/j.stueduc.2004.06.004 Burkhardt, H., & Swan, M. (2012). Designing assessment of performance in mathematics. Educational Designer, 2(5), 1-40. Choi, K. M., Lee, Y. S., & Park, Y. S. (2015). What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 11(6), 1563-1577. https://doi.org/10.22034/EMES.2021.244398 Clements, D. H., Banse, H., Sarama, J., Tatsuoka, C., Joswick, C., Hudyma, A., ... & Tatsuoka, K. K. (2022). Young children’s actions on length measurement tasks: strategies and cognitive attributes. Mathematical thinking and learning, 24(3), 181-202. https://doi.org/10.1080/10986065.2020.1843231. DeCarlo, L. T. (2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54(3), 304–313. https://doi.org/10.1016/j. jmp.2010.01.001. DeCarlo, L.T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the q-matrix. Applied Psychological Measurement, 35(1), 8-26. https://doi.org/10.1177/0146621610377081 Delafontaine, A., Tang, C. Y., & De Boeck, P. (2022). Cross-cultural item analysis using cognitive diagnosis models: A Q-matrix approach. International Journal of Testing, 22(2), 145–165. https://doi.org/10.1080/15305058.2021.1972641 de La Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: development and applications. Journal of Educational Measurement, 45(4), 343–362. https://doi.org/10.1111/j.1745-3984.2008.00069.x de La Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163–183. https://doi.org/10.1177/0146621608320523 de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130. https://doi.org/10.3102/1076998607309474 de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7 de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333– 353. https://doi.org/10.1007/BF02295640 de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595-624. https://doi.org/10.1007/s11336-008-9063-2 de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89–97. https://doi.org/10.1016/j.pse.2014.11.001 de la Torre, J., Chiu, C.-Y., & Lee, Y.-S. (2022). Advances in cognitive diagnostic models: New techniques for applying the DINA model in large-scale assessments. Psychometrika, 87(4), 901–926. https://doi.org/10.1007/s11336-022-09863-3 Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68(3), 263-272. https://doi.org/10.1007/s10649-007-9099-8 Doust, A. R., Khan, W. A., & Alghafri, M. S. (2021). Application of DINA model to assess Omani and Iranian fourth-grade students’ mathematics’ attributes. North American Academic Research (Naar) Journal, 4(10), 220-237. https://doi.org/10.5281/zenodo.5639812 Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262-277. https://doi.org/10.1177/0146621604272623 Evran, D. (2019). An application of cognitive diagnosis modeling in TIMSS: a comparison of Intuitive definitions of Q-matrices. International Journal of Modern Education Studies, 3(1), p.04. https://doi.org/10.51383/ijonmes.2019.33. George, A. C., & Robitzsch, A. (2015). Cognitive diagnosis models in R: A didactic. The Quantitative Methods for Psychology, 11(3), 189–205. https://doi.org/10.20982/tqmp.11.3.p189 Gündüz, T., & Çakan, M. (2020). TIMSS 2015 Türkiye örnekleminde matematik başarı testine dayalı olarak öğrencilerin bilişsel tanı modelleri ile sınıflandırılması. Necatibey Eğitim Fakültesi Elektronik Fen ve Matematik Eğitimi Dergisi, 14(2), 1040-1079. Hidajat, F. A. (2021). Students creative thinking profile as a high order thinking in the improvement of mathematics learning. European Journal of Educational Research, 10(3), 1247-1258. Junker, B.W. & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064 Kalkan, Ö., & Toprak, M. (2022). The impact of Q-matrix refinement methods on the diagnostic accuracy of CDMs. Educational Measurement: Issues and Practice, 41(1), 52–64. https://doi.org/10.1111/emip.12464 Karagiannakis, G. N., Baccaglini-Frank, A. E., & Roussos, P. (2016). Detecting strengths and weaknesses in learning mathematics through a model classifying mathematical skills. Australian Journal of Learning Difficulties, 21(2), 115-141. https://doi.org/10.1080/19404158.2017.1289963 Lee, Y. S., Park, Y. S., & Taylan, D. (2011). Cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144-177. https://doi.org/10.1080/15305058.2010.534571 Leighton, J. & Gierl, M., (eds.). (2007). Cognitive diagnostic assessment for education: theory and applications. Cambridge, MA: Cambridge University Press. https://doi.org/10.1017/CBO9780511611186 Li, X. & Wang, W. C. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28-54. https://www.jstor.org/stable/43940553 Li, L., An, Y., Ren, J., & Wei, X. (2021). Research on the cognitive diagnosis of Chinese listening comprehension ability based on the G-DINA model. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.714568 Li, H. & Suen, H. K. (2013). Constructing and validating a q-matrix for cognitive diagnostic analyses of a reading test, Educational Assessment, 18(1), 1-25, https://doi.org/10.1080/10627197.2013.76152 Liu, J., & Kang, T. (2023). Advances in cognitive diagnostic models: The role of DINA in educational assessment. Educational Assessment, 29(1), 45–67. https://doi.org/10.1007/s11336-022-09762-9 Ma, W., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275. https://doi.org/10.1111/bmsp.12070 Parlak, B. (2018). Investigation of the TIMMS 2015 mathematics achievement in eight grade students with cognitive diagnostic model. (Publication No.494335) (Phd thesis, Hacettepe university] Council of Higher Education National Thesis Center). Parlak, B. (2023). Matematik performansının bilişsel tanı modeli ile değerlendirilmesi: TIMSS Türkiye ve Singapur örneği. Milli Eğitim Dergisi, 52(1), 413-436. Roussos, L. A., DiBello, L. V., Stout, W., Hartz, S. M., Henson, R. A., & Templin, J. L. (2007). The Fusion Model Skills Diagnosis system. In Cambridge University Press eBooks (pp. 275–318). https://doi.org/10.1017/cbo9780511611186.010 Rupp, A. A. & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state of the art. Measurement, 6(4), 219-262. https://doi.org/10.1080/15366360802490866 Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190–209. https://doi.org/10.1080/15434300902801917 Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464. Shi, Q., Ma, W., Robitzsch, A., Sorrel, M. A., & Man, K. (2021). Cognitively diagnostic analysis using the G-DINA model in R. Psych, 3(4), 812–835. https://doi.org/10.3390/psych3040052 Su, Y. L. (2013). Cognitive diagnostic analysis using hierarchically structured skills. The University of Iowa. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354. https://doi.org/10.1111/j.1745- 3984.1983.tb00212.x Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 10(1), 55–73. https://doi.org/10.2307/1164930 Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. Cognitively Diagnostic Assessment, 327-359. Tatsuoka, K.K. (2009). Cognitive assessment—an introduction to the rule space method. Routledge, New York. https://doi.org/10.4324/9780203883372 Tatsuoka, C., Varadi, F., & Jaeger, J. (2013). Latent partially ordered classification models and normal mixtures. Journal of Educational and Behavioral Statistics, 38(3), 267-294. https://doi.org/10.3102/1076998612458318 Tatsuoka, C., Clements, D. H., Sarama, J., Izsák, A., Orrill, C. H., de la Torre, J., ... & Tatsuoka, K. K. (2016). Chapter 4. Journal for Research in Mathematics Education. Monograph, 15, 73-96. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287 Terzi, R., & Sen, S. (2019). A nondiagnostic assessment for diagnostic purposes: Q-Matrix validation and Item-Based Model fit evaluation for the TIMSS 2011 assessment. SAGE Open, 9(1). https://doi.org/10.1177/2158244019832684 Yakar, L., Dogan, N. Senol, D., & Yuksel, N. S. (2021). Monitoring student achievement with cognitive diagnosis model. Journal of Measurement and Evaluation in Education and Psychology, 12(3), 303-320. https://doi.org/10.21031/epod.903084 Wang, L., Li, Y., & Zhang, J. (2023). A new approach to Q-matrix construction for cognitive diagnostic models: Combining expert knowledge with statistical methods. Psychometrika, 88(1), 123–145. Whitely, S. E., & Schneider, L.M. (1981). Information structure for geometric analogies: A test theory approach. Applied Psychological Measurement, 5, 383-397. https://doi.org/10.1177/014662168100500312 Wu, H., Liu, Y., & Zhang, X. (2020). The effects of attribute granularity on the estimation quality of cognitive diagnosis models. Applied Psychological Measurement, 44(7), 546–561. https://doi.org/10.1177/0146621620917447 Zhang, S., Liu, J., & Ying, Z. (2023). Statistical applications to cognitive diagnostic testing. Annual Review of Statistics and its Application, 10(1), 143–165. https://doi.org/10.1146/annurev-statistics-033021-111803