Analysis of the 2021 PIRLS Data for Türkiye Using Cognitive Diagnostic Modeling

Abstract

This study examines the Turkish PIRLS 2021 data through the lens of cognitive diagnostic modeling (CDM). Türkiye last participated in the PIRLS exam in 2001, prior to 2021. Türkiye’s participation in this international assessment after a twenty-year hiatus is of great significance. The analysis was restricted to responses to one publicly released PIRLS 2021 literary passage (“The Empty Pot”) administered to 667 students in the Turkish sample. Applying CDM to these assessment data makes it possible to identify students’ specific reading-related strengths and limitations and to generate more fine-grained feedback. Additionally, CDM can provide insights into the approach needed to better develop students’ reading skills during the educational process. In this study, the text “The Empty Pot” was selected for analysis, and data from 667 students in Türkiye who completed this text were included in the analysis. First, with input from subject matter experts, the PIRLS attributes “Interpreting and integrating ideas/information” was divided into “Interpretation” and “Using personal knowledge and experiences.” The final Q-matrix includes the attributes “Making inferences,” “Interpretation,” “Using personal knowledge and experiences,” and “Examination and Evaluation.” Additionally, a Q-matrix validation step was applied to correct possible classification errors. The analyses involved estimating and comparing six CDMs: G-DINA, DINA, DINO, A-CDM, LLM, and R-RUM. The fit evidence indicated that G-DINA provided the most adequate representation of the observed response data. The results indicated that 39 students (5.85%) were classified as experts (1111) in all attributes. Another notable result is that 69 students who took the exam were not proficient in any attributes (0000), indicating that under the selected model, nearly 10% of the students was assigned to the non-mastery profile. Results also showed that making inferences showed the highest estimated mastery probability in this analysis among the four, using personal knowledge and experience was identified as the most difficult. Because the study used a retrofitted CDM design based on a single released passage and a validated Q-matrix with some remaining interpretive limitations, the findings should be interpreted as passage-specific diagnostic evidence rather than nationally generalizable estimates of reading proficiency.

Keywords

• Reading Skills
• International Assessments
• PIRLS
• Cognitive Diagnostic Modeling

References

1. Bloom, B. S. (1995). İnsan nitelikleri ve okulda öğrenme. (Durmuş Ali Özçelik, Çev.). Milli Eğitim Bakanlığı Yayınları.
2. Boori, A. A., Ghazanfari, M., Ghonsooly, B., & Baghaei, P. (2023). The construction and validation of a Q-matrix for cognitive diagnostic analysis: The case of the reading comprehension section of the IAUEPT. International Journal of Language Testing, 13(Special Issue), 31–53.
3. Chen, J. S., & de la Torre, J. (2014). A procedure for diagnostically modeling extant large-scale assessment data: The case of the Programme for International Student Assessment in Reading. Psychology, 5(18), 1967–1978.
4. Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37, 598–618.
5. Christensen, K. B., Kreiner, S., & Mesbah, M. (Eds.). (2013). Rasch models in health. John Wiley & Sons.
6. Çiftçi, Ö. (2007). İlköğretim 5. sınıf öğrencilerinin Türkçe öğretim programında belirtilen okuduğunu anlamayla ilgili kazanımlara ulaşma düzeyinin belirlenmesi [Unpublished doctoral dissertation, Gazi University].
7. de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343–362.
8. de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163–183.

9. de la Torre, J., & Akbay, L. (2019). Implementation of cognitive diagnosis modeling using the GDINA R package. Eurasian Journal of Educational Research, 19(80), 171–192.
10. de la Torre, J., & Karelitz, T. (2009). Impact of diagnosticity on the adequacy of models for cognitive diagnosis under a linear attribute structure: A simulation study. Journal of Educational Measurement, 46, 450–469.
11. Duong Thi, D. T., & Loye, N. (2019). Cognitive diagnostic analyses of the progress in international reading literacy study (PIRLS) 2011 results. Mesure et évaluation en éducation, 42(spécial), 127–165.
12. Gao, M., Miller, M. D., & Liu, R. (2017). The impact of Q-matrix misspecification and model misuse on classification accuracy in the generalized DINA model. Journal of Measurement and Evaluation in Education and Psychology, 8, 391–403.
13. Goodrich, S. (2018). Application of a log linear diagnostic classification model to the 2011 Progress in International Reading Literacy Study [Unpublished doctoral dissertation, University of British Columbia].
14. Greene, B. (2001). Testing reading comprehension of theoretical discourse with cloze. Journal of Research in Reading, 24(1), 82–98.
15. Güneş, F. (2000). Uygulamalı okuma yazma öğretimi. Ocak.
16. Im, S., & Corter, J. E. (2011). Statistical consequences of attribute misspecification in the rule space method. Educational and Psychological Measurement, 71, 712–731.
17. Jang, E. E., Dunlop, M., Wagner, M., Kim, Y. H., & Gu, Z. (2013). Elementary school ELLs’ reading skill profiles using cognitive diagnosis modeling: Roles of length of residence and home language environment. Language Learning, 63(3), 400–436.
18. Kendeou, P., van den Broek, P., Helder, A., & Karlsson, J. (2014). A cognitive model of reading comprehension: Implications for reading difficulties. Learning Disabilities Research and Practice, 29, 10–16.
19. Kim, A.-Y. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 32(2), 227–258.
20. Lathrop, Q. N., & Cheng, Y. (2013). Two approaches to estimation of classification accuracy rate under item response theory. Applied Psychological Measurement, 37(3), 226–241.
21. Lee, W. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47, 1–17.
22. Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263.
23. Li, H., & Suen, H. K. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1–25.
24. Ma, W., de la Torre, J., Sorrel, M., & Jiang, Z. (2022). GDINA: The generalized DINA model framework [R package]. https://CRAN.R-project.org/package=GDINA
25. Martineau, J. A. (2007). An expansion and practical evaluation of expected classification accuracy. Applied Psychological Measurement, 31(3), 181–194.
26. Ministry of National Education. (2019a). Akademik becerilerin izlenmesi ve değerlendirilmesi: 8. sınıf raporu. MEB Yayınları. https://eskisehirodm.meb.gov.tr/meb_iys_dosyalar/2019_07/03111935_ABYDE_4_2018_Raporu.pdf
27. Ministry of National Education. (2019b). Türkçe-Matematik-Fen Bilimleri öğrenci başarı izleme araştırması (TMF-ÖBA)-I: 2019 4. sınıf seviyesi. Eğitim Analiz ve Değerlendirme Raporları Serisi No:9. MEB Yayınları. https://www.meb.gov.tr/meb_iys_dosyalar/2019_09/23150323_TMF-OBAI-4-2019_Rapor9.pdf
28. Ministry of National Education. (2023, May 16). Başkan Özer: Türkiye, PIRLS 2021 sonuçlarına göre puanını en çok artıran ikinci ülke oldu. https://www.meb.gov.tr/bakan-ozer-pirls-2021-sonuclarina-gore-turkiye-puanini-en-cok-artiran-ikinci-ulke/haber/29923/tr
29. Mullis, I. V. S., & Martin, M. O. (Eds.) (2019). PIRLS 2021 assessment frameworks. International Association for the Evaluation of Educational Achievement. https://pirls2021.org/frameworks/
30. Mullis, I. V. S., von Davier, M., Foy, P., Fishbein, B., Reynolds, K. A., & Wry, E. (2023). PIRLS 2021 international results in reading. Boston College, TIMSS & PIRLS International Study Center. https://doi.org/10.6017/lse.tpisc.tr2103.kb5342
31. Nallasamy, R., & Khairani, A. Z. (2022). Validating reading comprehension assessment under the GDINA model. Malaysian Journal of Social Sciences and Humanities (MJSSH), 7(11), e001877.
32. National Research Council. (2001). Knowing what students know: The science and design of educational assessment. National Academy Press.
33. Oakhill, J., & Cain, K. (2017). Children with specific text comprehension problems. In K. Cain, D. L. Compton, & R. K. Parrilla (Eds.), Theories of reading development (pp. 359–375). John Benjamins.
34. R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
35. Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34(8), 782–799.
36. Rupp, A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78–96.
37. Sen, S., & Cohen, A. S. (2021). Sample size requirements for applying diagnostic classification models. Frontiers in Psychology, 11, 621251.
38. Temizkan, M. (2007). İlköğretim ikinci kademe Türkçe derslerinde okuma stratejilerinin okuduğunu anlama üzerindeki etkisi [Unpublished doctoral dissertation, Gazi University].
39. Terzi, R., & Sen, S. (2019). A nondiagnostic assessment for diagnostic purposes: Q-matrix validation and item-based model fit evaluation for the TIMSS 2011 assessment. SAGE Open, 9(1), 1–11.
40. Toprak-Yıldız, T. E. (2021). An international comparison using cognitive diagnostic assessment: Fourth graders’ diagnostic profile of reading skills on PIRLS 2016. Studies in Educational Evaluation, 70, 101057.
41. Ungan, S. (2008). Okuma alışkanlığımızın kültürel altyapısı. Gaziantep Üniversitesi Sosyal Bilimler Dergisi, 7(1), 218–228.
42. Wang, D., Cai, Y., & Tu, D. (2021). Q-matrix estimation methods for cognitive diagnosis models: Based on partial known Q-matrix. Multivariate Behavioral Research, 56(3), 514–526. https://doi.org/10.1080/00273171.2020.1746901.
43. Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457–476.
44. Yi, Y. S. (2017). Probing the relative importance of different attributes in L2 reading and listening comprehension items: An application of cognitive diagnostic models. Language Testing, 34(3), 337–355.
45. Yun, J. (2017). Investigating structures of reading comprehension attributes at different proficiency levels: Applying cognitive diagnosis models and factor analyses [Unpublished doctoral dissertation, The Florida State University].