Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study

Aygil Takır; Hasan Özder; Osman Cankoy

doi:10.17679/inuefd.1594482

EN TR

Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study

Abstract

The College Entrance Examination (CEE) is a curriculum-orientated performance examination administered to fifth-grade primary school students who wish to study at Maarif Colleges in Northern Cyprus, where the medium of instruction is English. This study was conducted to determine the psychometric characteristics of the CEE mathematics test administered to 1,833 fifth-grade primary school students in 2021. To achieve the purpose of the study, a cognitive diagnostic model, namely DINA, was used. The theoretical infrastructure of the DINA model is easy and strong compared to other models. The Q-matrix was constructed for 23 mathematics questions and 15 attributes were determined based on expert opinions. The results of the study showed that the mean values of the parameters g and s were 0.25 and 0.13, respectively. The mean of the δ values of the fifth-grade students was high (0.73). Skill patterns showed that 13% of fifth-grade students possessed all the attributes included in CEE. The construction of the Q-matrix is a very important stage in the use of CDMs. The cognitive attributes and the Q-matrix were determined by expert opinion in this study. The (1-s) values indicated the validity of the Q-matrix. For all of the items, 1-s values appeared to be quite high. This showed that the attainments and item-attainment relationship were done as accurately as possible. These results can be interpreted as the validity of the Q- matrix. The results of the study showed that the DINA model provides a detailed conclusion about students’ attributes.

Keywords

DINA, Cognitive diagnosis model, college entrance examination, mathematics assessment

Matematik Sorularının Bilişsel Tanılama Modeliyle İncelenmesi: Q-Matris Çalışması

Abstract

Kolej Giriş Sınavı (KGS), Kuzey Kıbrıs'ta eğitim dili İngilizce olan Maarif Kolejleri'nde okumak isteyen beşinci sınıf ilkokul öğrencilerine uygulanan program odaklı bir performans sınavıdır. Bu çalışma, 2021 yılında 1.833 beşinci sınıf ilkokul öğrencisine uygulanan KGS matematik testinin psikometrik özelliklerini belirlemek amacıyla gerçekleştirilmiştir. Çalışmanın amacına ulaşmak için DINA adlı bilişsel tanı modeli kullanılmıştır. DINA modelinin teorik altyapısı, diğer modellere göre hem kolay hem de güçlüdür. 23 matematik sorusu için Q matrisi oluşturulmuş ve uzman görüşlerine dayanarak 15 özellik belirlenmiştir. Çalışmanın sonuçları, g ve s parametrelerinin ortalama değerlerinin sırasıyla 0,25 ve 0,13 olduğunu göstermiştir. Bulgular, beşinci sınıf öğrencilerinin δ değerlerinin ortalamasının yüksek olduğunu (0,73) ortaya koymuştur. Beceri örüntüleri, beşinci sınıf öğrencilerinin %13'ünün KGS'de yer alan tüm niteliklere sahip olduğunu göstermektedir. Q matrisinin oluşturulması, Bilişsel Tanı Modelleri'nin (BTM) kullanımında çok önemli bir aşamadır. Bu çalışmada bilişsel nitelikler ve Q matrisi, uzman görüşü ile belirlenmiştir. (1-s) değerleri, Q matrisinin geçerliliğini göstermektedir. Tüm maddeler için 1-s değerlerinin oldukça yüksek olduğu gözlemlenmiştir. Bu durum, kazanımların ve madde-başarı ilişkisinin mümkün olduğunca doğru bir şekilde yapıldığını göstermekte ve bu sonuçlar, Q matrisinin geçerliliği olarak yorumlanabilir. Çalışmanın sonuçları, DINA modelinin öğrencilerin nitelikleri hakkında ayrıntılı bir sonuç sağladığını göstermektedir.

Keywords

DINA, Bilişsel Tanı Modeli, Kolej Giriş Sınavı, Matematik Değerlendirmesi

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705
Birenbaum, M., Tatsuoka, C., & Yamada, T. (2004). Diagnostic assessment in TIMSS-R: Between countries and within-country comparisons of eighth graders’ mathematics performance. Studies in Educational Evaluation, 30(2), 151-173. https://doi.org/10.1016/j.stueduc.2004.06.004
Burkhardt, H., & Swan, M. (2012). Designing assessment of performance in mathematics. Educational Designer, 2(5), 1-40.
Choi, K. M., Lee, Y. S., & Park, Y. S. (2015). What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 11(6), 1563-1577. https://doi.org/10.22034/EMES.2021.244398
Clements, D. H., Banse, H., Sarama, J., Tatsuoka, C., Joswick, C., Hudyma, A., ... & Tatsuoka, K. K. (2022). Young children’s actions on length measurement tasks: strategies and cognitive attributes. Mathematical thinking and learning, 24(3), 181-202. https://doi.org/10.1080/10986065.2020.1843231.
DeCarlo, L. T. (2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54(3), 304–313. https://doi.org/10.1016/j. jmp.2010.01.001.
DeCarlo, L.T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the q-matrix. Applied Psychological Measurement, 35(1), 8-26. https://doi.org/10.1177/0146621610377081
Delafontaine, A., Tang, C. Y., & De Boeck, P. (2022). Cross-cultural item analysis using cognitive diagnosis models: A Q-matrix approach. International Journal of Testing, 22(2), 145–165. https://doi.org/10.1080/15305058.2021.1972641
de La Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: development and applications. Journal of Educational Measurement, 45(4), 343–362. https://doi.org/10.1111/j.1745-3984.2008.00069.x
de La Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163–183. https://doi.org/10.1177/0146621608320523

de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130. https://doi.org/10.3102/1076998607309474
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333– 353. https://doi.org/10.1007/BF02295640
de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595-624. https://doi.org/10.1007/s11336-008-9063-2
de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89–97. https://doi.org/10.1016/j.pse.2014.11.001
de la Torre, J., Chiu, C.-Y., & Lee, Y.-S. (2022). Advances in cognitive diagnostic models: New techniques for applying the DINA model in large-scale assessments. Psychometrika, 87(4), 901–926. https://doi.org/10.1007/s11336-022-09863-3
Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68(3), 263-272. https://doi.org/10.1007/s10649-007-9099-8
Doust, A. R., Khan, W. A., & Alghafri, M. S. (2021). Application of DINA model to assess Omani and Iranian fourth-grade students’ mathematics’ attributes. North American Academic Research (Naar) Journal, 4(10), 220-237. https://doi.org/10.5281/zenodo.5639812
Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262-277. https://doi.org/10.1177/0146621604272623
Evran, D. (2019). An application of cognitive diagnosis modeling in TIMSS: a comparison of Intuitive definitions of Q-matrices. International Journal of Modern Education Studies, 3(1), p.04. https://doi.org/10.51383/ijonmes.2019.33.
George, A. C., & Robitzsch, A. (2015). Cognitive diagnosis models in R: A didactic. The Quantitative Methods for Psychology, 11(3), 189–205. https://doi.org/10.20982/tqmp.11.3.p189
Gündüz, T., & Çakan, M. (2020). TIMSS 2015 Türkiye örnekleminde matematik başarı testine dayalı olarak öğrencilerin bilişsel tanı modelleri ile sınıflandırılması. Necatibey Eğitim Fakültesi Elektronik Fen ve Matematik Eğitimi Dergisi, 14(2), 1040-1079.
Hidajat, F. A. (2021). Students creative thinking profile as a high order thinking in the improvement of mathematics learning. European Journal of Educational Research, 10(3), 1247-1258.
Junker, B.W. & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064
Kalkan, Ö., & Toprak, M. (2022). The impact of Q-matrix refinement methods on the diagnostic accuracy of CDMs. Educational Measurement: Issues and Practice, 41(1), 52–64. https://doi.org/10.1111/emip.12464
Karagiannakis, G. N., Baccaglini-Frank, A. E., & Roussos, P. (2016). Detecting strengths and weaknesses in learning mathematics through a model classifying mathematical skills. Australian Journal of Learning Difficulties, 21(2), 115-141. https://doi.org/10.1080/19404158.2017.1289963
Lee, Y. S., Park, Y. S., & Taylan, D. (2011). Cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144-177. https://doi.org/10.1080/15305058.2010.534571
Leighton, J. & Gierl, M., (eds.). (2007). Cognitive diagnostic assessment for education: theory and applications. Cambridge, MA: Cambridge University Press. https://doi.org/10.1017/CBO9780511611186
Li, X. & Wang, W. C. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28-54. https://www.jstor.org/stable/43940553
Li, L., An, Y., Ren, J., & Wei, X. (2021). Research on the cognitive diagnosis of Chinese listening comprehension ability based on the G-DINA model. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.714568
Li, H. & Suen, H. K. (2013). Constructing and validating a q-matrix for cognitive diagnostic analyses of a reading test, Educational Assessment, 18(1), 1-25, https://doi.org/10.1080/10627197.2013.76152
Liu, J., & Kang, T. (2023). Advances in cognitive diagnostic models: The role of DINA in educational assessment. Educational Assessment, 29(1), 45–67. https://doi.org/10.1007/s11336-022-09762-9
Ma, W., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275. https://doi.org/10.1111/bmsp.12070
Parlak, B. (2018). Investigation of the TIMMS 2015 mathematics achievement in eight grade students with cognitive diagnostic model. (Publication No.494335) (Phd thesis, Hacettepe university] Council of Higher Education National Thesis Center).
Parlak, B. (2023). Matematik performansının bilişsel tanı modeli ile değerlendirilmesi: TIMSS Türkiye ve Singapur örneği. Milli Eğitim Dergisi, 52(1), 413-436.
Roussos, L. A., DiBello, L. V., Stout, W., Hartz, S. M., Henson, R. A., & Templin, J. L. (2007). The Fusion Model Skills Diagnosis system. In Cambridge University Press eBooks (pp. 275–318). https://doi.org/10.1017/cbo9780511611186.010
Rupp, A. A. & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state of the art. Measurement, 6(4), 219-262. https://doi.org/10.1080/15366360802490866
Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-Matrix construction: Deﬁning the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190–209. https://doi.org/10.1080/15434300902801917
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464.
Shi, Q., Ma, W., Robitzsch, A., Sorrel, M. A., & Man, K. (2021). Cognitively diagnostic analysis using the G-DINA model in R. Psych, 3(4), 812–835. https://doi.org/10.3390/psych3040052
Su, Y. L. (2013). Cognitive diagnostic analysis using hierarchically structured skills. The University of Iowa.
Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354. https://doi.org/10.1111/j.1745- 3984.1983.tb00212.x
Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 10(1), 55–73. https://doi.org/10.2307/1164930
Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. Cognitively Diagnostic Assessment, 327-359.
Tatsuoka, K.K. (2009). Cognitive assessment—an introduction to the rule space method. Routledge, New York. https://doi.org/10.4324/9780203883372
Tatsuoka, C., Varadi, F., & Jaeger, J. (2013). Latent partially ordered classification models and normal mixtures. Journal of Educational and Behavioral Statistics, 38(3), 267-294. https://doi.org/10.3102/1076998612458318
Tatsuoka, C., Clements, D. H., Sarama, J., Izsák, A., Orrill, C. H., de la Torre, J., ... & Tatsuoka, K. K. (2016). Chapter 4. Journal for Research in Mathematics Education. Monograph, 15, 73-96.
Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287
Terzi, R., & Sen, S. (2019). A nondiagnostic assessment for diagnostic purposes: Q-Matrix validation and Item-Based Model fit evaluation for the TIMSS 2011 assessment. SAGE Open, 9(1). https://doi.org/10.1177/2158244019832684
Yakar, L., Dogan, N. Senol, D., & Yuksel, N. S. (2021). Monitoring student achievement with cognitive diagnosis model. Journal of Measurement and Evaluation in Education and Psychology, 12(3), 303-320. https://doi.org/10.21031/epod.903084
Wang, L., Li, Y., & Zhang, J. (2023). A new approach to Q-matrix construction for cognitive diagnostic models: Combining expert knowledge with statistical methods. Psychometrika, 88(1), 123–145.
Whitely, S. E., & Schneider, L.M. (1981). Information structure for geometric analogies: A test theory approach. Applied Psychological Measurement, 5, 383-397. https://doi.org/10.1177/014662168100500312
Wu, H., Liu, Y., & Zhang, X. (2020). The effects of attribute granularity on the estimation quality of cognitive diagnosis models. Applied Psychological Measurement, 44(7), 546–561. https://doi.org/10.1177/0146621620917447
Zhang, S., Liu, J., & Ying, Z. (2023). Statistical applications to cognitive diagnostic testing. Annual Review of Statistics and its Application, 10(1), 143–165. https://doi.org/10.1146/annurev-statistics-033021-111803

Details

Primary Language

English

Subjects

Setting Standards and Norms

Journal Section

Research Article

Authors

Aygil Takır
0000-0003-3042-7585
Cyprus

Hasan Özder ^*
0000-0003-1094-3590
Kuzey Kıbrıs Türk Cumhuriyeti

Osman Cankoy
0000-0002-4765-9297
Türkiye

Publication Date

September 2, 2025

Submission Date

December 5, 2024

Acceptance Date

June 3, 2025

Published in Issue

Year 2025 Volume: 26 Number: 2

DOI

https://doi.org/10.17679/inuefd.1594482

IZ

https://izlik.org/JA86UG46NT

APA

Takır, A., Özder, H., & Cankoy, O. (2025). Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study. İnönü University Journal of the Faculty of Education, 26(2), 759-796. https://doi.org/10.17679/inuefd.1594482

AMA

1.Takır A, Özder H, Cankoy O. Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study. INUJFE. 2025;26(2):759-796. doi:10.17679/inuefd.1594482

Chicago

Takır, Aygil, Hasan Özder, and Osman Cankoy. 2025. “Examining Mathematics Questions With a Cognitive Diagnostic Model: Q-Matrix Study”. İnönü University Journal of the Faculty of Education 26 (2): 759-96. https://doi.org/10.17679/inuefd.1594482.

EndNote

Takır A, Özder H, Cankoy O (September 1, 2025) Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study. İnönü University Journal of the Faculty of Education 26 2 759–796.

IEEE

[1]A. Takır, H. Özder, and O. Cankoy, “Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study”, INUJFE, vol. 26, no. 2, pp. 759–796, Sept. 2025, doi: 10.17679/inuefd.1594482.

ISNAD

Takır, Aygil - Özder, Hasan - Cankoy, Osman. “Examining Mathematics Questions With a Cognitive Diagnostic Model: Q-Matrix Study”. İnönü University Journal of the Faculty of Education 26/2 (September 1, 2025): 759-796. https://doi.org/10.17679/inuefd.1594482.

JAMA

1.Takır A, Özder H, Cankoy O. Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study. INUJFE. 2025;26:759–796.

MLA

Takır, Aygil, et al. “Examining Mathematics Questions With a Cognitive Diagnostic Model: Q-Matrix Study”. İnönü University Journal of the Faculty of Education, vol. 26, no. 2, Sept. 2025, pp. 759-96, doi:10.17679/inuefd.1594482.

Vancouver

1.Aygil Takır, Hasan Özder, Osman Cankoy. Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study. INUJFE. 2025 Sep. 1;26(2):759-96. doi:10.17679/inuefd.1594482

Examining mathematics questions with a cognitive diagnostic model: Q-Matrix study

Abstract

Keywords

Matematik Sorularının Bilişsel Tanılama Modeliyle İncelenmesi: Q-Matris Çalışması

Abstract

Keywords

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite