Investigation of Prediction Accuracy of Hierarchical Linear Modelling and Artificial Neural Networks Methods on PISA 2018 Reading Literacy

Year 2025, Volume: 45 Issue: 2, 543 - 568, 30.08.2025

Eda Akdoğdu Yıldız , Kübra Atalay Kabasakal

https://doi.org/10.17152/gefad.1700937

Abstract

In this research, it is aimed to compare hierarchical linear modelling and artificial neural network estimation methods in predicting students' reading comprehension success in the Program for International Student Assessment (PISA) 2018 application. In accordance with this purpose, it is planned to determine how students' PISA success status is estimated at student and school level, the data mining method used in estimation and the explained variance and error values of multilevel modelling. The type of study is, in a way, relational research because of the establishment of models in which there are relationships between dependent and independent variables. On the other hand, it is descriptive research in terms of performing analyses with two methods for each country sampled in the study and comparing the results obtained in terms of explained variance and error values. In this research, the performance of data mining techniques (artificial neural networks – ANN) and multilevel analysis methods (hierarchical linear modeling – HLM) in the field of education is evaluated. It has been determined that HLM carries out the estimation process with lower error and higher R^2 than ANN in the analysis of multi-level data. In addition, HLM provides more information about the predictive level of the variables and the variance that is not explained by the variables in the model compared to ANN. For this reason, HLM analysis was used to examine the variables that affect reading comprehension success in the study. As a result, it was seen that the student level and school level variables added to the model had a statistically significant effect on reading comprehension achievement. While teacher-directed instruction and lack of educational material at school cause negative effects on reading comprehension success, it has been determined that economic-social-cultural situation, metacognitive strategies, disciplinary climate in the classroom, teacher support, and staff shortage variables have positive effects. The results obtained are generally in agreement with similar studies in the literature.

Keywords

Hierarchical linear modelling , Data mining , Artificial neural networks , Reading comprehension , PISA

Hiyerarşik Doğrusal Modellemenin ve Yapay Sinir Ağları Yöntemlerinin PISA 2018 Okuma Okuryazarlığı Tahmin Doğruluğunun Araştırılması

Abstract

Bu araştırmada Uluslararası Öğrenci Değerlendirme Programı (PISA) 2018 uygulamasında öğrencilerin okuduğunu anlama başarısını tahmin etmede hiyerarşik lineer modelleme ve yapay sinir ağları tahmin yöntemlerinin karşılaştırılması amaçlanmaktadır. Bu amaç doğrultusunda; öğrencilerin PISA başarı durumlarının birey ve okul düzeyinde nasıl tahmin edildiği, tahmin etmede kullanılan veri madenciliği yöntemi ve çok düzeyli modellemenin açıklanan varyans ve hata değerlerinin belirlenmesi planlanmaktadır. Çalışmanın türü, bağımlı ve bağımsız değişkenler arasında ilişkilerin bulunduğu modellerin kurulmasından dolayı bir yönüyle ilişkisel araştırmadır. Diğer bir yönüyle ise çalışmada örnekleme alınan her ülke için iki yöntemle analizler gerçekleştirilip elde edilen sonuçların açıklanan varyans ve hata değerleri açısından karşılaştırılması bakımından betimsel araştırma niteliğindedir. Bu araştırmada eğitim alanında da kullanılmaya başlanan veri madenciliği (yapay sinir ağları-YSA) ve çok düzeyli analiz yöntemlerinin (hiyerarşik lineer model-HLM) nasıl performans gösterdiğine ilişkin bulgular elde edilmiştir. HLM’in çok düzeyleri verilerin analizinde YSA’ya göre daha düşük hata ve daha yüksek R^2 ile tahminleme sürecini yürüttüğü belirlenmiştir. Ayrıca HLM değişkenlerin yordama düzeyi ve modelde yer alan değişkenler tarafından açıklanmayan varyans hakkında YSA’ya göre daha fazla bilgi sunmaktadır. Bu sebeple çalışmada okuduğunu anlama başarısını etkileyen değişkenleri incelemek için HLM analizi kullanılmıştır. Sonuç olarak modele eklenen birey düzeyi (düzey 1) ve okul düzeyi (düzey 2) değişkenlerinin okuduğunu anlama başarısı üzerinde istatiksel olarak anlamlı etkisi olduğu görülmüştür. Okuduğunu anlama başarısı üzerinde öğretmen yönlendirmeli öğretim ve okuldaki eğitimsel materyal eksikliği negatif yönlü etkiye sebep olurken ekonomik-sosyal-kültürel durum, üstbiliş stratejileri, sınıftaki disiplin iklimi, öğretmen desteği, personel eksikliği değişkenleri pozitif yönlü etkisi olduğu tespit edilmiştir. Elde edilen sonuçlar genel olarak literatürde yer alan benzer çalışmalarla uyum göstermektedir.

Keywords

Hiyerarşik lineer modelleme , Veri madenciliği , Yapay sinir ağları , Okuduğunu anlama , PISA

References

• Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Mohamed, N. A., & Arshad, H. (2018). State-of-the-art in artificial neural network applications: A survey. Heliyon, 4(11), e00938. https://doi.org/10.1016/j.heliyon.2018.e00938
• Akkoyunlu, B. (2008, Mayıs). Bilgi okuryazarlığı ve yaşam boyu öğrenme. 8th International Educational Technology Conference (IECT) toplantısında sunulan bildiri, Anadolu Üniversitesi, Eskişehir.
• Anderson, R.C., Hiebert, E.H., Scott, J.A. & Wilkinson, I.A.G. (1988). Becoming a nation of readers: the report of the commission on reading. Education and Treatment of Children, 11(4), 389-396.
• Anderson, D., & McNeill, G. (1992). Artificial neural networks technology. Kaman Sciences Corporation
• Ali, O., Murray, P. A., Momin, M., Dwivedi, Y. K., & Malik, T. (2023). The effects of artificial intelligence applications in educational settings: Challenges and strategies. Technological Forecasting and Social Change, 199, 123076. https://doi.org/10.1016/j.techfore.2023.123076
• Baba, A. (2024). Neural networks from biological to artificial and vice versa. Biosystems, 235, 105110. https://doi.org/10.1016/j.biosystems.2023.105110
• Bennett, D. A. (2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25(5), 464-469.
• Botchkarev, A. (2019). A new typology design of performance metrics to measure errors in machine learning regression algorithms. Interdisciplinary Journal of Information Knowledge and Management, 14, 045–076. https://doi.org/10.28945/4184
• Chai, T., & Draxler, R. R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature. Geoscientific Model Development, 7(3), 1247–1250. https://doi.org/10.5194/gmd-7-1247-2014
• Chavez, H., Chavez-Arias, B., Contreras-Rosas, S., Alvarez-Rodríguez, J. M., & Raymundo, C. (2023). Artificial neural network model to predict student performance using nonpersonal information. Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.1106679
• Chen, J., Lin, C. H., & Chen, G. (2021). A cross-cultural perspective on the relationships among social media use, self-regulated learning and adolescents’ digital reading literacy. Computers & Education, 175, 104322.
• Chen, Q., Lei, Y., Wen, Z., Li, S., Li, J., & Kong, Y. (2019). Teacher support, reading strategy and reading literacy: A two-level mediation model. Best Evid Chin Edu, 2(1), 157-170. https://doi.org/10.15354/bece.19.ar1036
• Courtney, M., Karakus, M., Ersozlu, Z., & Nurumov, K. (2022). The influence of ICT use and related attitudes on students’ math and science performance: Multilevel analyses of the last decade’s PISA surveys. Large-Scale Assessments in Education, 10(1), 1-26. https://doi.org/10.1186/s40536-022-00128-6
• Çelenk, S. (2003). Okul aile iş birliği ile okuduğunu anlama başarısı arasındaki ilişki. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 24, 33-39.
• Dong, X., & Hu, J. (2019). An exploration of impact factors influencing students’ reading literacy in Singapore with machine learning approaches. International Journal of English Linguistics, 9(5), 52-65. https://doi.org/10.5539/ijel.v9n5p52
• Ercan, U. (2021). Ev dışı gıda tüketim sınıflarının yapay sinir ağları ile tahmin edilmesi. İşletme Araştırmaları Dergisi, 13(4), 3265-3277.
• Feng, Y., & Jones, K. (2015, July 8-10). Comparing multilevel modelling and artificial neural networks in house price prediction [Paper presentation]. 2015 2nd IEEE International Conference on Spatial Data Mining and Geographical Knowledge Services, Fuzhou, China.
• Fritsch, S., Guenther, F., & Wright, M. N. (2019). neuralnet: Training of Neural Networks. R package version 1.44.2.
• Gamazo, A., & Martínez-Abad, F. (2020). An exploration of factors linked to academic performance in PISA 2018 through data mining techniques. Frontiers in Psychology, 11, 575167. https://doi.org/10.3389/fpsyg.2020.575167
• Guthrie, J. T., & Wigfield, A. (2000). Engagement and motivation in reading. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. 3, pp. 403–422). Lawrence Erlbaum.
• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer Science & Business Media.
• Hawkins, D. M. (2003). The problem of overfitting. Journal of Chemical Information and Computer Sciences, 44(1), 1–12. https://doi.org/10.1021/ci0342472
• Haykin, S. (2008). Neural networks: a comprehensive foundation. New Jersey: Prentice-Hall.
• Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2. ed): New York: Routledge.
• Hu, J., & Wang, Y. (2022). Influence of students’ perceptions of instruction quality on their digital reading performance in 29 OECD countries: A multilevel analysis. Computers & Education, 189, 104591. https://doi.org/10.1016/j.compedu.2022.104591
• Kumova M., S. & Kışla, T. (2020). Yapay sinir ağları. In T. Güyer, H. Yurdugül & S. Yıldırım (Ed.), Eğitsel veri madenciliği ve öğrenme algoritmaları (pp. 127-146). Anı Publishing.
• Koyuncu, İ., & Fırat, T. (2021). Investigating Reading Literacy in PISA 2018 assessment. Lnternational Electronic Journal of Elementary Education, 13(2), 263–275. https://doi.org/10.26822/iejee.2021.189
• Kuhn, M. (2022). caret: classification and regression training. R package version 6.0-91.
• Kursa, M. B., & Rudnicki, W. R. (2010). Feature selection with the boruta package. Journal of Statistical Software, 36(11), 1-13.
• Leisch, F., & Dimitriadou, E. (2021). mlbench: Machine learning benchmark problems. R package version 2.1-3.
• Levy, J., Mussack, D., Brunner, M., Keller, U., Cardoso-Leite, P., & Fischbach, A. (2020). Contrasting classical and machine learning approaches in the estimation of Value-Added scores in Large-Scale Educational data. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.02190
• Liaw, A., & Wiener, M. (2002). Classification and Regression by randomForest. R News 2(3), 18-22.
• Lee, Y. H., & Wu, J. Y. (2013). The indirect effects of online social entertainment and information seeking activities on reading literacy. Computers & Education, 67, 168-177. https://doi.org/10.1016/j.compedu.2013.03.001
• Lin, M. B., Groves, W. A., Freivalds, A., Lee, E. G., & Harper, M. (2011). Comparison of artificial neural network (ANN) and partial least squares (PLS) regression models for predicting respiratory ventilation: an exploratory study. European Journal of Applied Physiology, 112(5), 1603–1611. https://doi.org/10.1007/s00421-011-2118-6
• Lim, H. J., & Jung, H. (2019). Factors related to digital reading achievement: A multi-level analysis using international large scale data. Computers & Education, 133, 82–93. https://doi.org/10.1016/j.compedu.2019.01.007
• Ma, L., Xiao, L., & Hau, K. T. (2022). Teacher feedback, disciplinary climate, student self-concept, and reading achievement: A multilevel moderated mediation model. Learning and Instruction, 79, 101602.
• Maas, C. J. M., & Hox, J. J. (2005). Sufficient Sample Sizes for Multilevel Modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1(3), 86–92. https://doi.org/10.1027/1614-2241.1.3.86
• McNeish, D. M., & Stapleton, L. M. (2016). The effect of small sample size on two-level model estimates: A review and illustration. Educational Psychology Review, 28(2), 295–314. https://doi.org/10.1007/s10648-014-9287-x
• Michela, J. L. (2006). Software review: HLM 6. Organizational Research Methods, 9(1), 119-122. https://doi.org/10.1177/10944281052812
• Munson, M.A., Caruana, R. (2009). On Feature Selection, Bias-Variance, and Bagging. Buntine. In W., Grobelnik, M., Mladenić, D., Shawe-Taylor, J. (Eds.), Machine Learning and Knowledge Discovery in Databases. ECML PKDD Lecture Notes in Computer Science, vol 5782. Springer, Berlin, Heidelberg.
• Muratkyzy, A. (2020). Equity and excellence in the Kazakhstani Education System: A multilevel analysis of the personal and contextual factors contributing to students’ reading literacy performance on PISA 2018 [Unpublished master’s thesis]. Nazarbayev University Institute of Educational Sciences, Astana.
• Ni, X. (2008). Research of data mining based on neural networks. World Academy of Science, Engineering and Technology, 39, 381-384.
• OECD. (2019). PISA 2018 results volume I: What students know and can do. Paris: OECD Publishing. https://doi.org/10.1787/5f07c754-en.
• Oreta, A. W. C. (2004). Simulating size effect on shear strength of RC beams without stirrups using neural networks. Engineering Structures, 26(5), 681-691. https://doi.org/10.1016/j.engstruct.2004.01.009
• Radoyevic, N. (2006). Exploring the use of effective learning strategies to increase students’ reading comprehension and test taking skills (Yüksek lisans tezi). Brock Üniversitesi Eğitim Fakültesi, St. Catharines, Ontario.
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models. Applications and Data Analysis Methods (2nd ed.). Thousand Oaks, CA: Sage Publications.
• Rose, D. S., Parks, M., Androes, K., & McMahon, S. D. (2000). Imagery-based learning: Improving elementary students' reading comprehension with drama techniques. The Journal of Educational Research, 94(1), 55-63.
• Shahini, A. (2021). Inequalities in Albanian education: Evidence from large-scale assessment studies. Kultura i Edukacja, 4(134), 40-70.
• Snow, C. E. (2002). Reading for Understanding: Toward an R&D Program in Reading Comprehension. RAND Corporation.
• Tang, F., & Ishwaran, H. (2017). Random forest missing data algorithms. Statistical Analysis and Data Mining the ASA Data Science Journal, 10(6), 363–377. https://doi.org/10.1002/sam.11348
• Weng, W., & Luo, W. (2023). A comparative analysis of data mining methods and hierarchical linear modeling using PISA 2018 data. International Journal of Database Management Systems, 15(2/3), 1–16. https://doi.org/10.5121/ijdms.2023.15301
• Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.
• Willmott, C., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30, 79–82. https://doi.org/10.3354/cr030079
• Yıldız Aybek, H. S., & Okur, M. R. (2018). Predicting achievement with artificial neural networks: The case of Anadolu University Open Education System. International Journal of Assessment Tools in Education, 5(3), 474-490.
• Yousafzai, B. K., Khan, S. A., Rahman, T., Khan, I., Ullah, I., Ur Rehman, A., Baz, M., Hamam, H., & Cheikhrouhou, O. (2021). Student-Performulator: Student Academic Performance Using Hybrid Deep Neural Network. Sustainability, 13(17), 9775. https://doi.org/10.3390/su13179775