Ranking the PISA Composite Performance of Countries Based on the PISA 2018 Survey Results

Abstract

Various measurement and evaluation studies at the national and international
levels are conducted by countries to reveal the extent of success in different
education levels. One such study is the Programme for International Student
Assessment (PISA) survey. The PISA survey results provide educators and decisionmakers with practical and relevant information about the education levels of their
countries. To this end, this study aimed to determine the composite PISA 2018
performance rankings of the participating countries. The mean scores of reading
skills, mathematics, and science literacies used in determining composite PISA
performance rankings were weighted through CRITIC and Entropy methods
allowing for objective criterion weighting. Two different composite PISA
performance rankings of countries were determined by applying the CRITIC- and
Entropy-based TOPSIS method, one of the multi-criteria decision-making (MCDM)
methods. The Spearman correlation coefficient was calculated to compare the
rankings determined through this method. A perfect positive correlation was found
between the two different composite PISA performance rankings. According to the
results of the study, when the PISA performance rankings of the 78 countries that
were participated in the PISA 2018 survey were examined, it was determined that
the composite PISA performance rankings of the first 5 and the last 5 countries,
and the rankings of 43 countries calculated by both methods remained the same
that calculated with the Entropy and CRITIC-based TOPSIS method

Keywords

• PISA
• international exams
• Entropy
• CRITIC
• TOPSIS

PISA 2018 Araştırma Sonuçlarına Göre Ülkelerin Bileşik PISA Performans Sıralaması

Abstract

Ülkeler farklı düzeylerde verilen eğitimlerin ne düzeyde başarılı olduğuna ilişkin
çeşitli ulusal ya da uluslararası alanda ölçme ve değerlendirme çalışmaları
yapmaktadır. Bu çalışmalardan biri de PISA araştırmasıdır. PISA araştırması
sonrasında yayınlanan raporlar, eğitimcilere ve karar vericilere ülkelerinin eğitim
düzeyleri hakkında işlevsel ve faydalı bilgiler sağlamaktadır. Bu çalışmada, 2018
PISA araştırmasına katılan ülkelerin bileşik PISA performans sıralamalarının
belirlenmesi amaçlanmıştır. Bileşik PISA performans sıralamalarının
belirlenmesinde kullanılan okuma becerileri, matematik ve fen okuryazarlığı
ortalama puanları; objektif yaklaşımla kriter ağırlıklandırmasına imkân veren
CRITIC ve Entropi yöntemleri ile ağırlıklandırılmıştır. Çok ölçütlü karar verme
metotlarından CRITIC ve Entropi tabanlı TOPSIS yöntemi uygulanarak ülkelerin
iki farklı bileşik PISA performans sırası belirlenmiştir. CRITIC ve Entropi tabanlı
TOPSIS yöntemiyle elde edilen sıralamaları karşılaştırmak için Spearman
korelasyon katsayısı hesaplanmıştır. CRITIC ve Entropi tabanlı TOPSIS yöntemiyle
hesaplanan iki farklı bileşik PISA performans sıralamaları arasında mükemmel
pozitif korelasyon saptanmıştır. Çalışmanın sonuçlarına göre PISA 2018
araştırmasına katılan 78 ülkenin PISA başarı sıralamaları incelendiğinde ilk 5 ve
son 5 ülkenin Entropi ve CRITIC tabanlı TOPSIS yöntemi ile hesaplanan bileşik
PISA performans (bileşik indeks) sıralamalarının ve 43 ülkenin her iki yöntem ile
hesaplanan sıralamasının aynı kaldığı gözlenmiştir.

Keywords

• PISA
• Uluslararası sınavlar
• Entropi
• CRITIC
• TOPSIS

References

1. Abdel-Basset, M., & Mohamed, R. (2020). A novel plithogenic TOPSIS-CRITIC model for sustainable supply chain risk management. Journal of Cleaner Production,247, 119586. https://doi.org/10.1016/j.jclepro.2019.119586
2. Acar, T., & Öğretmen, T. (2012). Çok düzeyli istatistiksel yöntemler ile 2006 PISA fen bilimleri performansının incelenmesi. Eğitim ve Bilim, 37(163), 178-189.
3. Akbaşlı, S., Şahin, M., & Yaykıran, Z. (2016). The Effect of Reading Comprehension on the Performance in Science and Mathematics. Journal of Education and Practice, 7(16), 108-121.
4. Aksu, G., & Güzeller, C. O. (2016). PISA 2012 matematik okuryazarlığı puanlarının karar ağacı yöntemiyle sınıflandırılması: Türkiye örneklemi. Eğitim ve Bilim, 41(185), 101-122. https://doi.org/10.15390/EB.2016.4766
5. Aydın, A., Sarıer, Y., & Uysal, Ş. (2012). Sosyoekonomik ve sosyokültürel değişkenler açısından PISA matematik sonuçlarının karşılaştırılması. Eğitim ve Bilim, 37(164), 20-30.
6. Aydoğdu İskenderoğlu, T., & Baki, A. (2011). İlköğretim 8. Sınıf Matematik Ders Kitabındaki Soruların PISA Matematik Yeterlik Düzeylerine Göre Sınıflandırılması. Education & Science/Eğitim ve Bilim, 36(161), 287-301.
7. Bloem, S. (2015). PISA for low-and middle-income countries. Compare: A Journal of Comparative and International Education, 45(3), 481-486. https://doi.org/10.1080/03057925.2015.1027513
8. Cordero, J. M., Polo, C., & Simancas, R. (2020). Assessing the efficiency of secondary schools: Evidence from OECD countries participating in PISA 2015. Socio-Economic Planning Sciences, 100927. https://doi.org/10.1016/j.seps.2020.100927

9. Çakır, S. (2017). Measuring logistics performance of OECD countries via fuzzy linear regression. Journal of Multi‐Criteria Decision Analysis, 24(3-4), 177-186. https://doi.org/10.1002/mcda.1601
10. Çalık, A., Çizmecioğlu, S., & Akpınar, A. (2019). An integrated AHP‐TOPSIS framework for foreign direct investment in Turkey. Journal of Multi‐Criteria Decision Analysis, 26(5-6), 296-307. https://doi.org/10.1002/mcda.1601
11. Diakoulaki, D., Mavrotas, G., & Papayannakis, L. (1995).: The critic method. Computers & Operations Research, 22(7), 763-770. https://doi.org/10.1016/0305-0548(94)00059-H
12. Dolu, A., & Ekinci, R. (2020). Eğitimde Etkinliğin Ölçülmesi: PISA Verileri İle Bootstrap Veri Zarflama Analizi Tahmini. Journal of Yaşar University, 15, 207-218.
13. Ehmke, T., van den Ham, A. K., Sälzer, C., Heine, J., & Prenzel, M. (2020). Measuring mathematics competence in international and national large-scale assessments: Linking PISA and the national educational panel study in Germany. Studies in Educational Evaluation, 65, 100847. https://doi.org/10.1016/j.stueduc.2020.100847
14. Ersoy, N. (2017). Measuring corporate sustainability performance in the rubber coating industry: an integrated multicriteria framework. The Online Journal of Science and Technology-October, 7(4).
15. Ertürk, R. (2020). İnsani Gelişim İndeksine Göre Farklı Gelişmişlik Düzeyinde Bulunan Ülkelerin PISA Sonuçlarının Karşılaştırılması. Adnan Menderes Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 7(1), 41-57.
16. Fagbote, E. O., Olanipekun, E. O., & Uyi, H. S. (2014). Water quality index of the ground water of bitumen deposit impacted farm settlements using entropy weighted method. International Journal of Environmental Science and Technology, 11(1), 127-138. https://doi.org/10.1007/s13762-012-0149-0
17. Feng, X., Zuo, W., Wang, J., & Feng, L. (2014). TOPSIS method for hesitant fuzzy multiple attribute decision making. Journal of Intelligent & Fuzzy Systems, 26(5), 2263-2269. https://doi.org/10.3233/IFS-130899
18. Fernandez-Cano, A. (2016). A methodological critique of the PISA evaluations. Relieve,22(1), 1-16.
19. Fu, C., & Xu, D. L. (2016). Determining attribute weights to improve solution reliability and its application to selecting leading industries. Annals of Operations Research, 245(1), 401-426. https://doi.org/10.1007/s10479-014-1657-8
20. Grey, S., & Morris, P. (2018). PISA: Multiple ‘truths and mediatised global governance. Comparative Education, 54(2), 109-131. https://doi.org/10.1080/03050068.2018.1425243
21. Gürsakal, S. (2012). PISA 2009 Öğrenci Başarı Düzeylerini Etkileyen Faktörlerin Değerlendirilmesi. Süleyman Demirel University Journal of Faculty of Economics & Administrative Sciences, 17(1).
22. Hashemkhani Zolfani, S., Yazdani, M., Ebadi Torkayesh, A., & Derakhti, A. (2020). Application of a gray-based decision support framework for location selection of a temporary hospital during COVID-19 pandemic. Symmetry, 12(6), 886. https://doi.org/10.3390/sym12060886
23. Hopfenbeck, T.N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.A. (2018). Lessons Learned from PISA: A Systematic Review of Peer-Reviewed Articles on the Programme for International Student Assessment, Scandinavian Journal of Educational Research, 62:3, 333-353. https://doi.org/10.1080/00313831.2016.1258726
24. Hsu, P. F., & Hsu, M. G. (2008). Optimizing the information outsourcing practices of primary care medical organizations using entropy and TOPSIS. Quality & Quantity, 42(2), 181-201. https://doi.org/10.1007/s11135-006-9040-8
25. Hu, X., Gong, Y., Lai, C., & Leung, F. K. (2018). The relationship between ICT and student literacy in mathematics, reading, and science across 44 countries: A multilevel analysis. Computers & Education, 125, 1-13. https://doi.org/10.1016/j.compedu.2018.05.021
26. Hwang, C. L., & Yoon, K. (1981). Methods for multiple attribute decision making. In Multiple attribute decision making (pp. 58-191). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-48318-9_3
27. Ishizaka, A., & Resce, G. (2021). Best-Worst PROMETHEE method for evaluating school performance in the OECD's PISA project. Socio-Economic Planning Sciences, 73, 100799. https://doi.org/10.1007/978-3-642-48318-9_3
28. Jahan, A., Mustapha, F., Sapuan, S. M., Ismail, M. Y., & Bahraminasab, M. (2012). A framework for weighting of criteria in ranking stage of material selection process. The International Journal of Advanced Manufacturing Technology, 58(1-4), 411-420. https://doi.org/10.1007/s00170-011-3366-7
29. Jerrim, J. (2016). PISA 2012: How do results for the paper and computer tests compare? Assessment in Education: Principles, Policy & Practice, 23(4), 495-518.https://doi.org/10.1080/0969594X.2016.1147420
30. Jerrim, J. (2021). PISA 2018 in England, Northern Ireland, Scotland and Wales: Is the data really representative of all four corners of the UK? Review of Education, 9(3), e3270. https://doi.org/10.1002/rev3.3270
31. Kasap, Y. , Doğan, N. & Koçak, C. (2021). PISA 2018’de Okuduğunu Anlama Başarısını Yordayan Değişkenlerin Veri Madenciliği İle Belirlenmesi . Manisa Celal Bayar Üniversitesi Sosyal Bilimler Dergisi , 19 (4) , 241-258. https://doi.org/10.18026/cbayarsos.959609
32. Kaynak, S., Altuntas, S., & Dereli, T. (2017). Comparing the innovation performance of EU candidate countries: an entropy-based TOPSIS approach. Economic research-Ekonomska istraživanja, 30(1), 31-54. https://doi.org/10.1080/1331677X.2016.1265895
33. Keleş, S. (2020). Seçilmiş OECD ülkelerinde eğitim harcamaları ve 2018 PISA performanslarının karşılaştırılmalı analizi. Maliye Çalışmaları Dergisi, (63), 57-75. https://doi.org/10.26650/mcd2020-772192
34. Kotte, D., Lietz, P., & Lopez, M. M. (2005). Factors Influencing Reading Achievement in Germany and Spain: Evidence from PISA 2000. International Education Journal, 6(1), 113-124.
35. Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika, 79(2), 210-231. https://doi.org/10.1007/s11336-013-9347-z
36. Li, X., Wang, K., Liu, L., Xin, J., Yang, H., & Gao, C. (2011). Application of the entropy weight and TOPSIS method in safety evaluation of coal mines. Procedia engineering, 26, 2085-2091. https://doi.org/10.1016/j.proeng.2011.11.2410
37. Liu, F., Zhao, S., Weng, M., & Liu, Y. (2017). Fire risk assessment for large-scale commercial buildings based on structure entropy weight method. Safety science, 94, 26-40. https://doi.org/10.1016/j.ssci.2016.12.009
38. Liu, X., Zhou, X., Zhu, B., He, K., & Wang, P. (2019). Measuring the maturity of carbon market in China: An entropy-based TOPSIS approach. Journal of Cleaner Production, 229, 94-103. https://doi.org/10.1016/j.jclepro.2019.04.380
39. Lynn, R., & Mikk, J. (2009). National IQs predict educational attainment in math, reading and science across 56 nations. Intelligence, 37(3), 305-310. https://doi.org/10.1016/j.intell.2009.01.002
40. Martens, Kerstin, and Dennis Niemann. 2010. “Governance by Comparison: How Ratings & Rankings Impact National Policy-Making in Education.” TranState Working PapersUR, https://www.econstor.eu/handle/10419/ 41595139. https://www.econstor.eu/bitstream-/10419/41595/1/639011268.pdf.
41. Min, J., & Peng, K. H. (2012). Ranking emotional intelligence training needs in tour leaders: an entropy-based TOPSIS approach. Current Issues in Tourism, 15(6), 563-576. https://doi.org/10.1080/13683500.2011.641946
42. Mohamadghasemi, A., Hadi‐Vencheh, A., & Hosseinzadeh Lotfi, F. (2020). The multiobjective stochastic CRITIC–TOPSIS approach for solving the shipboard crane selection problem. International Journal of Intelligent Systems, 35(10), 1570-1598. https://doi.org/10.1002/int.22265
43. Monjezi, M., Dehghani, H., Singh, T. N., Sayadi, A. R., & Gholinejad, A. (2012). Application of TOPSIS method for selecting the most appropriate blast design. Arabian journal of geosciences, 5(1), 95-101. https://doi.org/10.1007/s12517-010-0133-2
44. Navarro-Martinez, O., & Peña-Acuña, B. (2022). Technology Usage and Academic Performance in the Pisa 2018 Report. Journal of New Approaches in Educational Research, 11(1), 130-145. https://doi.org/10.7821/naer.2022.1.735
45. OECD (2018). PISA 2018 Database.https://doi.org/10.1787/888934029090 adresinden 16.05.2021tarihinde alınmıştır.
46. OECD (2019). PISA 2018 Results (Volume I) - © OECD 2019. https://doi.org/10.1787/5f07c754-en adresinden 16.05.2021tarihinde alınmıştır.
47. Okatan, Ö. ve Tomul, E. (2021). Uluslararası öğrenci başarılarını değerlendirme programı’na (PISA) göre Türkiye’deki öğrencilerin matematik başarıları ile ilişkili değişkenlerin incelenmesi. Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi, (57), 98-125.
48. Oluah, C., Akinlabi, E. T., & Njoku, H. O. (2020). Selection of phase change material for improved performance of Trombe wall systems using the entropy weight and TOPSIS methodology. Energy and Buildings, 217, 109967. https://doi.org/10.1016/j.enbuild.2020.109967
49. Özberk, E. H., Kabasakal, K. A., & Öztürk, N. B. (2017). Investigating the factors affecting Turkish students’ PISA 2012 mathematics achievement using hierarchical linear modeling PISA 2012. Hacettepe University Journal of Education, 32(3), 544–559. https://doi.org/10.16986/HUJE.2017026950
50. Özdemir, B., & Gelbal, S. (2014). PISA 2009 sonuçlarına göre öğrenci başarısını etkileyen faktörlerin kanonik ortak etki analizi ile incelenmesi. Eğitim ve Bilim, 39(175). https://doi.org/10.15390/EB.2014.3025
51. Phanden, R. K., Sindhwani, R., Kalsariya, V., & Salroo, F. (2019). Selection of material for electric arc spraying by using hierarchical entropy-TOPSIS approach. International Journal of Productivity and Quality Management, 26(3), 276-289. https://doi.org/10.1504/IJPQM.2019.098364
52. Polat, M., Toraman, Ç., & Turhan, N. S. (2022). Reliability analysis of PISA 2018 reading literacy student questionnaire based on item response theory (IRT): Turkey sample: Reliability analysis of PISA 2018 reading literacy. International Journal of Curriculum and Instruction, 14(1), 1004-1028.
53. Prais, S. J. (2004). Cautions on OECD's recent educational survey (PISA): rejoinder to OECD's response. Oxford Review of Education, 30(4), 569-573. https://doi.org/10.1080/0305498042000303017
54. Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational researcher, 39(2), 142-151. https://doi.org/10.3102/0013189X10363170
55. Saatçioğlu, Ö., & Gülleroğlu, H. D. (2017). PISA 2009 uygulamasına katılan ülkelerin okuma becerileri alt test sonuçlarının profil analizi ile değerlendirilmesi. Eğitim ve Bilim, 42(190).
56. Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal, 27(3), 379-423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
57. She, H. C., Stacey, K., & Schmidt, W. H. (2018). Science and mathematics literacy: PISA for better school education. International Journal of Science and Mathematics Education, 16(1), 1-5. https://doi.org/10.1007/s10763-018-9911-1
58. Shyur, H. J. (2006). COTS evaluation using modified TOPSIS and ANP. Applied mathematics and computation, 177(1), 251-259. https://doi.org/10.1016/j.amc.2005.11.006
59. Soh, K. (2014). Score-rank Inconsistency in International Ranking: An Example from PISA 2009-2012. International Journal, 1(1), 2-13.
60. Tang, H., Shi, Y., & Dong, P. (2019). Public blockchain evaluation using entropy and TOPSIS. Expert Systems with Applications, 117, 204-210. https://doi.org/10.1016/j.eswa.2018.09.048
61. Tienken, C. H. (2017). Understanding PISA results. Kappa Delta Pi Record, 53(1), 6-8. https://doi.org/10.1080/00228958.2017.1264806
62. Tienken, C. H. (2020). PISA Scores and Ranks Are Fundamentally Flawed. Kappa Delta Pi Record, 56(2), 55-57. https://doi.org/10.1080/00228958.2020.1729629
63. Tuş, A., & Aytaç Adalı, E. (2019). The new combination with CRITIC and WASPAS methods for the time and attendance software selection problem. Opsearch, 56(2), 528-538. https://doi.org/10.1007/s12597-019-00371-6
64. Türkan, A., S. S. Üner., & Alcı, B. (2015). 2012 PISA Matematik Testi Puanlarının Bazı Değişkenler Açısından İncelenmesi. Ege Eğitim Dergisi, 16(2), 358-372. https://doi.org/10.12984/eed.68351
65. Uçar, E., & Karsak, E. E. (2021). Educational Performance Assessment of OECD Countries Using PISA 2018 Data. Proceedings of IAC 2021 in Vienna, 1, 64.
66. Uğuz, E., Şahin, S., & Yılmaz, R. (2021). PISA 2018 fen bilimleri puanlarının değerlendirilmesinde eğitsel veri madenciliğinin kullanımı. Bilgi ve İletişim Teknolojileri Dergisi, 3(2), 212-227. https://doi.org/10.53694/bited.887425
67. Wang, X., He, L., Zhu, K., Zhang, S., Xin, L., Xu, W., & Guan, Y. (2019). An integrated model to evaluate the impact of social support on improving self-management of type 2 diabetes mellitus. BMC medical informatics and decision making, 19(1), 1-12. https://doi.org/10.1186/s12911-018-0723-6
68. Wang, Y. M., & Luo, Y. (2010). Integration of correlations with standard deviations for determining attribute weights in multiple attribute decision making. Mathematical and Computer Modelling, 51(1-2), 1-12. https://doi.org/10.1016/j.mcm.2009.07.016
69. Wang, Z., Parhi, S. S., Rangaiah, G. P., & Jana, A. K. (2020). Analysis of weighting and selection methods for pareto-optimal solutions of multiobjective optimization in chemical engineering applications. Industrial & Engineering Chemistry Research, 59(33), 14850-14867. https://doi.org/10.1021/acs.iecr.0c00969
70. Wittwer, J., & Senkbeil, M. (2008). Is students’ computer use at home related to their mathematical performance at school? Computers & Education, 50(4), 1558-1571. https://doi.org/10.1016/j.compedu.2007.03.001
71. Xu, J., Feng, P., & Yang, P. (2016). Research of development strategy on China’s rural drinking water supply based on SWOT–TOPSIS method combined with AHP-Entropy: a case in Hebei Province. Environmental Earth Sciences, 75(1), 1-11. https://doi.org/10.1007/s12665-015-4885-6
72. Yalçın, O. M., & Hanoğlu, E. T. (2020). OECD nin Uluslararası Öğrenci Değerlendirme Programında Başarılı Ülkeler ile Türkiye nin Eğitim Yönetimi ve Denetimi Açısından Karşılaştırılması. Yükseköğretim ve Bilim Dergisi, (1), 36-44. https://doi.org/10.14689/enad.27.10
73. Yıldız, D. (2021). Türkçe ve Türk dili-edebiyatı öğretmenlerinin gözünden PISA’daki okuma becerisi ve Türkiye’nin performansı: bir odak grup görüşmesi. Journal of Qualitative Research in Education, 27, 208-231. doi:10.14689/enad.27.10
74. Yore, L. D., & Van der Flier-Keller, E. (2011). Pacific Crystal Centre For Science, Mathematics, And Technology Literacy. In Pacific CRYSTAL Centre for Science, Mathematics, and Technology Literacy: Lessons Learned (pp. 3-22). Sense Publishers. https://doi.org/10.1007/978-94-6091-506-2
75. Yüksel, M., & Geban, Ö. (2018). Student performance task assessment using multiple criteria decision making (MCDM) techniques: An application for 9th grade chemistry course. Bartın University Journal of Faculty of Education, 7(3), 874-901. https://doi.org/10.14686/buefad.400787
76. Yüksel, M. (2021a). Ranking of Universities via Entropy and TOPSIS Method Based on Teacher Field Knowledge Test Results within the Field of Chemistry Teaching. Recent Studies of Education in Various Occasions (pp.115-149), Riga: LAP Lambert Academic Publishing.
77. Yüksel, Mehmet. (2021b). Kimya öğretmenliği programlarının taban puan bağlamında CRITIC ve TOPSIS yöntemi ile değerlendirilmesi. 3. Ulusal Başkent Disiplinler Arası Bilimsel Çalışmalar Kongresi. Ankara 14- 15 Mart. Türkiye.
78. Zardari, N. H., Ahmed, K., Shirazi, S. M., & Yusop, Z. B. (2015). Weighting methods and their effects on multi-criteria decision-making model outcomes in water resources management. Springer. https://doi.org/10.1007/978-3-319-12586-2
79. Zhao, X., Guo, H. T., Huang, C. L., & Zhong, J. S. (2017). Teaching evaluation system research based on structure entropy weight method. Journal of Discrete Mathematical Sciences and Cryptography, 20(1), 179-191. https://doi.org/10.1080/09720529.2016.1178915