Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study

Year 2025, Volume: 26 Issue: 2, 759 - 796, 02.09.2025

Aygil Takır , Hasan Özder , Osman Cankoy

https://doi.org/10.17679/inuefd.1594482

Abstract

The College Entrance Examination (CEE) is a curriculum-orientated performance examination administered to fifth-grade primary school students who wish to study at Maarif Colleges in Northern Cyprus, where the medium of instruction is English. This study was conducted to determine the psychometric characteristics of the CEE mathematics test administered to 1,833 fifth-grade primary school students in 2021. To achieve the purpose of the study, a cognitive diagnostic model, namely DINA, was used. The theoretical infrastructure of the DINA model is easy and strong compared to other models. The Q-matrix was constructed for 23 mathematics questions and 15 attributes were determined based on expert opinions. The results of the study showed that the mean values of the parameters g and s were 0.25 and 0.13, respectively. The mean of the δ values of the fifth-grade students was high (0.73). Skill patterns showed that 13% of fifth-grade students possessed all the attributes included in CEE. The construction of the Q-matrix is a very important stage in the use of CDMs. The cognitive attributes and the Q-matrix were determined by expert opinion in this study. The (1-s) values indicated the validity of the Q-matrix. For all of the items, 1-s values appeared to be quite high. This showed that the attainments and item-attainment relationship were done as accurately as possible. These results can be interpreted as the validity of the Q- matrix. The results of the study showed that the DINA model provides a detailed conclusion about students’ attributes.

Keywords

DINA , Cognitive diagnosis model , college entrance examination , mathematics assessment

Matematik Sorularının Bilişsel Tanılama Modeliyle İncelenmesi: Q-Matris Çalışması

Abstract

Kolej Giriş Sınavı (KGS), Kuzey Kıbrıs'ta eğitim dili İngilizce olan Maarif Kolejleri'nde okumak isteyen beşinci sınıf ilkokul öğrencilerine uygulanan program odaklı bir performans sınavıdır. Bu çalışma, 2021 yılında 1.833 beşinci sınıf ilkokul öğrencisine uygulanan KGS matematik testinin psikometrik özelliklerini belirlemek amacıyla gerçekleştirilmiştir. Çalışmanın amacına ulaşmak için DINA adlı bilişsel tanı modeli kullanılmıştır. DINA modelinin teorik altyapısı, diğer modellere göre hem kolay hem de güçlüdür. 23 matematik sorusu için Q matrisi oluşturulmuş ve uzman görüşlerine dayanarak 15 özellik belirlenmiştir. Çalışmanın sonuçları, g ve s parametrelerinin ortalama değerlerinin sırasıyla 0,25 ve 0,13 olduğunu göstermiştir. Bulgular, beşinci sınıf öğrencilerinin δ değerlerinin ortalamasının yüksek olduğunu (0,73) ortaya koymuştur. Beceri örüntüleri, beşinci sınıf öğrencilerinin %13'ünün KGS'de yer alan tüm niteliklere sahip olduğunu göstermektedir. Q matrisinin oluşturulması, Bilişsel Tanı Modelleri'nin (BTM) kullanımında çok önemli bir aşamadır. Bu çalışmada bilişsel nitelikler ve Q matrisi, uzman görüşü ile belirlenmiştir. (1-s) değerleri, Q matrisinin geçerliliğini göstermektedir. Tüm maddeler için 1-s değerlerinin oldukça yüksek olduğu gözlemlenmiştir. Bu durum, kazanımların ve madde-başarı ilişkisinin mümkün olduğunca doğru bir şekilde yapıldığını göstermekte ve bu sonuçlar, Q matrisinin geçerliliği olarak yorumlanabilir. Çalışmanın sonuçları, DINA modelinin öğrencilerin nitelikleri hakkında ayrıntılı bir sonuç sağladığını göstermektedir.

Keywords

DINA , Bilişsel Tanı Modeli , Kolej Giriş Sınavı , Matematik Değerlendirmesi

References

• Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705
• Birenbaum, M., Tatsuoka, C., & Yamada, T. (2004). Diagnostic assessment in TIMSS-R: Between countries and within-country comparisons of eighth graders’ mathematics performance. Studies in Educational Evaluation, 30(2), 151-173. https://doi.org/10.1016/j.stueduc.2004.06.004
• Burkhardt, H., & Swan, M. (2012). Designing assessment of performance in mathematics. Educational Designer, 2(5), 1-40.
• Choi, K. M., Lee, Y. S., & Park, Y. S. (2015). What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 11(6), 1563-1577. https://doi.org/10.22034/EMES.2021.244398
• Clements, D. H., Banse, H., Sarama, J., Tatsuoka, C., Joswick, C., Hudyma, A., ... & Tatsuoka, K. K. (2022). Young children’s actions on length measurement tasks: strategies and cognitive attributes. Mathematical thinking and learning, 24(3), 181-202. https://doi.org/10.1080/10986065.2020.1843231.
• DeCarlo, L. T. (2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54(3), 304–313. https://doi.org/10.1016/j. jmp.2010.01.001.
• DeCarlo, L.T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the q-matrix. Applied Psychological Measurement, 35(1), 8-26. https://doi.org/10.1177/0146621610377081
• Delafontaine, A., Tang, C. Y., & De Boeck, P. (2022). Cross-cultural item analysis using cognitive diagnosis models: A Q-matrix approach. International Journal of Testing, 22(2), 145–165. https://doi.org/10.1080/15305058.2021.1972641
• de La Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: development and applications. Journal of Educational Measurement, 45(4), 343–362. https://doi.org/10.1111/j.1745-3984.2008.00069.x
• de La Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163–183. https://doi.org/10.1177/0146621608320523
• de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130. https://doi.org/10.3102/1076998607309474
• de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
• de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333– 353. https://doi.org/10.1007/BF02295640
• de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595-624. https://doi.org/10.1007/s11336-008-9063-2
• de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89–97. https://doi.org/10.1016/j.pse.2014.11.001
• de la Torre, J., Chiu, C.-Y., & Lee, Y.-S. (2022). Advances in cognitive diagnostic models: New techniques for applying the DINA model in large-scale assessments. Psychometrika, 87(4), 901–926. https://doi.org/10.1007/s11336-022-09863-3
• Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68(3), 263-272. https://doi.org/10.1007/s10649-007-9099-8
• Doust, A. R., Khan, W. A., & Alghafri, M. S. (2021). Application of DINA model to assess Omani and Iranian fourth-grade students’ mathematics’ attributes. North American Academic Research (Naar) Journal, 4(10), 220-237. https://doi.org/10.5281/zenodo.5639812
• Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262-277. https://doi.org/10.1177/0146621604272623
• Evran, D. (2019). An application of cognitive diagnosis modeling in TIMSS: a comparison of Intuitive definitions of Q-matrices. International Journal of Modern Education Studies, 3(1), p.04. https://doi.org/10.51383/ijonmes.2019.33.
• George, A. C., & Robitzsch, A. (2015). Cognitive diagnosis models in R: A didactic. The Quantitative Methods for Psychology, 11(3), 189–205. https://doi.org/10.20982/tqmp.11.3.p189
• Gündüz, T., & Çakan, M. (2020). TIMSS 2015 Türkiye örnekleminde matematik başarı testine dayalı olarak öğrencilerin bilişsel tanı modelleri ile sınıflandırılması. Necatibey Eğitim Fakültesi Elektronik Fen ve Matematik Eğitimi Dergisi, 14(2), 1040-1079.
• Hidajat, F. A. (2021). Students creative thinking profile as a high order thinking in the improvement of mathematics learning. European Journal of Educational Research, 10(3), 1247-1258.
• Junker, B.W. & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064
• Kalkan, Ö., & Toprak, M. (2022). The impact of Q-matrix refinement methods on the diagnostic accuracy of CDMs. Educational Measurement: Issues and Practice, 41(1), 52–64. https://doi.org/10.1111/emip.12464
• Karagiannakis, G. N., Baccaglini-Frank, A. E., & Roussos, P. (2016). Detecting strengths and weaknesses in learning mathematics through a model classifying mathematical skills. Australian Journal of Learning Difficulties, 21(2), 115-141. https://doi.org/10.1080/19404158.2017.1289963
• Lee, Y. S., Park, Y. S., & Taylan, D. (2011). Cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144-177. https://doi.org/10.1080/15305058.2010.534571
• Leighton, J. & Gierl, M., (eds.). (2007). Cognitive diagnostic assessment for education: theory and applications. Cambridge, MA: Cambridge University Press. https://doi.org/10.1017/CBO9780511611186
• Li, X. & Wang, W. C. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28-54. https://www.jstor.org/stable/43940553
• Li, L., An, Y., Ren, J., & Wei, X. (2021). Research on the cognitive diagnosis of Chinese listening comprehension ability based on the G-DINA model. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.714568
• Li, H. & Suen, H. K. (2013). Constructing and validating a q-matrix for cognitive diagnostic analyses of a reading test, Educational Assessment, 18(1), 1-25, https://doi.org/10.1080/10627197.2013.76152
• Liu, J., & Kang, T. (2023). Advances in cognitive diagnostic models: The role of DINA in educational assessment. Educational Assessment, 29(1), 45–67. https://doi.org/10.1007/s11336-022-09762-9
• Ma, W., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275. https://doi.org/10.1111/bmsp.12070
• Parlak, B. (2018). Investigation of the TIMMS 2015 mathematics achievement in eight grade students with cognitive diagnostic model. (Publication No.494335) (Phd thesis, Hacettepe university] Council of Higher Education National Thesis Center).
• Parlak, B. (2023). Matematik performansının bilişsel tanı modeli ile değerlendirilmesi: TIMSS Türkiye ve Singapur örneği. Milli Eğitim Dergisi, 52(1), 413-436.
• Roussos, L. A., DiBello, L. V., Stout, W., Hartz, S. M., Henson, R. A., & Templin, J. L. (2007). The Fusion Model Skills Diagnosis system. In Cambridge University Press eBooks (pp. 275–318). https://doi.org/10.1017/cbo9780511611186.010
• Rupp, A. A. & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state of the art. Measurement, 6(4), 219-262. https://doi.org/10.1080/15366360802490866
• Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190–209. https://doi.org/10.1080/15434300902801917
• Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464.
• Shi, Q., Ma, W., Robitzsch, A., Sorrel, M. A., & Man, K. (2021). Cognitively diagnostic analysis using the G-DINA model in R. Psych, 3(4), 812–835. https://doi.org/10.3390/psych3040052
• Su, Y. L. (2013). Cognitive diagnostic analysis using hierarchically structured skills. The University of Iowa.
• Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354. https://doi.org/10.1111/j.1745- 3984.1983.tb00212.x
• Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 10(1), 55–73. https://doi.org/10.2307/1164930
• Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. Cognitively Diagnostic Assessment, 327-359.
• Tatsuoka, K.K. (2009). Cognitive assessment—an introduction to the rule space method. Routledge, New York. https://doi.org/10.4324/9780203883372
• Tatsuoka, C., Varadi, F., & Jaeger, J. (2013). Latent partially ordered classification models and normal mixtures. Journal of Educational and Behavioral Statistics, 38(3), 267-294. https://doi.org/10.3102/1076998612458318
• Tatsuoka, C., Clements, D. H., Sarama, J., Izsák, A., Orrill, C. H., de la Torre, J., ... & Tatsuoka, K. K. (2016). Chapter 4. Journal for Research in Mathematics Education. Monograph, 15, 73-96.
• Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287
• Terzi, R., & Sen, S. (2019). A nondiagnostic assessment for diagnostic purposes: Q-Matrix validation and Item-Based Model fit evaluation for the TIMSS 2011 assessment. SAGE Open, 9(1). https://doi.org/10.1177/2158244019832684
• Yakar, L., Dogan, N. Senol, D., & Yuksel, N. S. (2021). Monitoring student achievement with cognitive diagnosis model. Journal of Measurement and Evaluation in Education and Psychology, 12(3), 303-320. https://doi.org/10.21031/epod.903084
• Wang, L., Li, Y., & Zhang, J. (2023). A new approach to Q-matrix construction for cognitive diagnostic models: Combining expert knowledge with statistical methods. Psychometrika, 88(1), 123–145.
• Whitely, S. E., & Schneider, L.M. (1981). Information structure for geometric analogies: A test theory approach. Applied Psychological Measurement, 5, 383-397. https://doi.org/10.1177/014662168100500312
• Wu, H., Liu, Y., & Zhang, X. (2020). The effects of attribute granularity on the estimation quality of cognitive diagnosis models. Applied Psychological Measurement, 44(7), 546–561. https://doi.org/10.1177/0146621620917447
• Zhang, S., Liu, J., & Ying, Z. (2023). Statistical applications to cognitive diagnostic testing. Annual Review of Statistics and its Application, 10(1), 143–165. https://doi.org/10.1146/annurev-statistics-033021-111803