TY  - JOUR
T1  - The Effect of Sample Weighting on Hierarchical Linear Modeling in the Large-Scale Assessment Data
TT  - Geniş Ölçekli Test Verilerinde Örneklem Ağırlıklandırmalarının Hiyerarşik Doğrusal Modellemeye Etkisi
AU  - Güngör, Metehan
AU  - Bekmezci, Sinan
AU  - Doğan, Nuri
PY  - 2024
DA  - September
Y2  - 2024
DO  - 10.17556/erziefd.1404346
JF  - Erzincan Üniversitesi Eğitim Fakültesi Dergisi
JO  - EUJEF
PB  - Erzincan Binali Yıldırım Üniversitesi
WT  - DergiPark
SN  - 2148-7758
SP  - 400
EP  - 413
VL  - 26
IS  - 3
LA  - en
AB  - This study examines how the different uses of sampling weights in the analysis of TIMSS 2019 data affect the ratio of variance in student achievement explained by schools and the estimation of standard errors. The research sample comprises 227,345 8th grade students from 7,636 schools in 39 countries. Mathematics achievement and science achievement are considered separately as dependent variables in all 39 countries. All plausible values are included in the analysis. Four weighting scenarios are examined: no weighting, weighting at only level 1, weighting at only level 2, and weighting at both levels. In total, 312 models are established and examined. According to the research results, the coefficients, standard errors, reliabilities, and χ^2 estimations change depending on how the weighting variable is handled in the models, and as a result, the ratio of variance in the dependent variable arising from the differences between schools also changes. The ratio attributable to between-school differences can reach up to 20% in some countries. Therefore, researchers modeling hierarchical data using HLM are suggested to plan how they handle the weighting variable prior to conducting the study.
KW  - large scale assessment
KW  - hierarchical linear modelling
KW  - weighting
KW  - TIMSS 2019
N2  - Bu çalışmada TIMSS 2019 verilerinin analizinde örneklem ağırlıklarının farklı şekilde kullanımlarının öğrenci başarısındaki varyansın okullar tarafından açıklanan kısmında ve standart hataların kestiriminde nasıl bir etkiye sahip olduğu incelenmiştir. Araştırmanın örneklemini TIMSS 2019 uygulamasına katılan 39 ülkeden toplam 7636 okuldaki 227345 8. sınıf öğrencisi oluşturmaktadır. Matematik başarısı ve fen başarısı tüm ülkelerde bağımlı değişkenler olarak ayrı ayrı ele alınmıştır. Tüm olası değerler analize dahil edilmiştir. Ağırlıklandırmanın olmadığı, yalnızca 1. düzeyde ağırlıklandırmanın olduğu, yalnızca 2. düzeyde ağırlıklandırmanın olduğu ve her iki düzeyde de ağırlıklandırmanın olduğu dört farklı durum incelenmiştir. Toplamda 312 model kurulmuş ve çözümlenmiştir. Araştırmanın sonuçlarına göre katsayıların, standart hataların, güvenirlik değerlerinin ve χ^2 istatistiklerinin, ağırlıklandırma değişkeninin kullanılma biçimine göre değiştiği gözlenmiştir. Bunun bir sonucu olarak da, çıktı değişkenindeki varyansın açıklanmasında okullar tarafından açıklanan kısım değişkenlik göstermektedir. Bu kısımdaki değişkenlik bazı ülkelerde %20’lere kadar çıkabilmektedir. Bu nedenle, geniş ölçekli testlerin verilerini HLM ile modelleyecek araştırmacıların ağırlıklandırma değişkenini ne şekilde ele alacaklarını araştırma öncesinde planlamaları önerilmektedir.
CR  - Ababneh, E., Al-Tweissi, A., & Abulibdeh, K. (2016). TIMSS and PISA impact – the case of Jordan. Research Papers in Education, 31(5), 542-555, http://dx.doi.org/10.1080/02671522.2016.1225350
CR  - Aksu, G., Güzeller, C. O., & Eser, M. T. (2017). Analysis of maths literacy performances of students with hierarchical linear modeling (HLM): The case of PISA 2012 Turkey. Education and Science, 42(191), 247-266. http://dx.doi.org/10.15390/EB.2017.6956
CR  - Arıkan, S., Özer, F., Şeker, V., & Ertaş, G. (2020). The importance of sample weights and plausible values in large-scale assessments. Journal of Measurement and Evaluation in Education and Psychology, 11(1), 43-60. https://doi.org/10.21031/epod.602765
CR  - Atar, H. Y., & Atar, B. (2012). Examining the effects of Turkish education reform on students’ TIMSS 2007 science achievements. Educational Sciences: Theory & Practice, 12(4), 2632-2636.
CR  - Barber, M., Chijioke, C., & Mourshed, M. (2010). How the World’s most improved school systems keep getting better. McKinsey and Company.
CR  - Bilican, S., & Yıldırım, Ö. (2013). The effects of approaches to learning on student's reflective and evaluative reading performance in Turkey: The results from PISA 2009. Procedia - Social and Behavioral Sciences, 116, 2437-2442. https://doi.org/10.1016/j.sbspro.2014.01.588
CR  - Boulifa, K., & Kaaouachi, A. (2022). The relationship between the school resources index, gender, age and mathematics achievement in TIMSS 2019 survey: Multilevel analysis. Procedia Computer Science, 201, 738-745. https://doi.org/10.1016/j.procs.2022.03.100
CR  - Carle, A. C. (2009). Fitting multilevel models in complex survey data with design weights: Recommendations. BMC Medical Research Methodology, 9, 1-13. http://doi.org/10.1186/1471-2288-9-49
CR  - Chu, M. W., Babenko, O., Cui, Y., & Leighton, J. P. (2014). Using HLM to explore effects of perceptions of learning environments and assessments on students’ test performance. International Journal of Testing, 14(2), 95-121. https://doi.org/10.1080/15305058.2013.841702
CR  - Fraenkel, J. R., & Wallen, N. E. (2006). How to design and evaluate research in education (6th Ed.). McGraw-Hill.
CR  - Gómez, R. L., & Suárez, A. M. (2020). Do inquiry-based teaching and school climate influence science achievement and critical thinking? Evidence from PISA 2015. International Journal of STEM Education, 7(43), 1-11. https://doi.org/10.1186/s40594-020-00240-5
CR  - Hox, J. J. (2010). Multilevel analysis: Techniques and applications. Routledge.
CR  - IBM Corp. (2013). IBM SPSS Statistics for Windows (Version 22.0). IBM Corp.
CR  - Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communications in Statistics - Theory and Methods, 46(22), 11341-11357. https://doi.org/10.1080/03610926.2016.1267764
CR  - Liang, X. (2010). Assessment use, self-efficacy and mathematics achievement: comparative analysis of PISA 2003 data of Finland, Canada and the USA. Evaluation & Research in Education, 23(3), 213-229. https://doi.org/10.1080/09500790.2010.490875
CR  - Liou, P. Y., & Hung, Y. C. (2015). Statistical techniques utilized in analyzing PISA and TIMSS data in science education from 1996 to 2013: A methodological review. International Journal of Science and Mathematics Education, 13, 1449-1468. https://doi.org/10.1007/s10763-014-9558-5
CR  - Mang, J., Küchenhoff, H., Meinck, S., & Prenzel, M. (2021). Sampling weights in multilevel modelling: An investigation using PISA sampling structures. Large-scale Assess Educ., 9(6), 1-39. https://doi.org/10.1186/s40536-021-00099-0
CR  - Manjunath, M. (2021). Using large-scale assessments to inform education policy and governance, Assessment resources, Azim Premji University. Retrieved from https://cdn.azimpremjiuniversity.edu.in/apuc3/media/resources/Assessments-and-Education-Policy.f1640171199.pdf
CR  - Meinck, S., & Vandenplas, C. (2012). Sample size requirements in HLM: An empirical study. IERI monograph series issues and methodologies in large-scale assessments. IER Institute, special issue 1, Educational Testing Service and International Association for the Evaluation of Educational Achievement.
CR  - Organisation for Economic Co-operation and Development [OECD]. (2017). PISA 2015 technical report. Paris: OECD Publishing.
CR  - Özdemir, C. (2016). A methodological review of research using OECD PISA Turkey data. Education Science Society Journal, 14(56), 10-27.
CR  - Pacheco Diaz, N., & Rocconi, L. M. (2021). Examining science achievement in Chile: A multilevel approach using PISA 2015 data. Journal of Research in STEM Education, 7(2), 93-116. https://doi.org/10.51355/jstem.2021.100
CR  - Pong, S. (2009). Grade level and achievement of immigrants' children: academic redshirting in Hong Kong. Educational Research and Evaluation: An International Journal on Theory and Practice, 15(4), 405-425. http://dx.doi.org/10.1080/13803610903087078
CR  - Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods 2. Sage Publications.
CR  - Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2007). HLM for Windows (Version 6.04). Scientific Software International.
CR  - Reinikainen, P. (2007). Sequential explanatory study of factors connected with science achievement in six countries: Finland, England, Hungary, Japan, Latvia and Russia: Study based on TIMSS 1999. Institute for Educational Research, University of Jyväskylä.
CR  - Ross, S. P. (2008). Motivation correlates of academic achievement: Exploring how motivation influences academic achievement in the PISA 2003 dataset [Doctoral Dissertation, University of Victoria]. UVicSpace. https://dspace.library.uvic.ca/handle/1828/3209
CR  - Rust, K. (2013). Sampling, weighting, and variance estimation in international large-scale assessments. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 117-154). Chapman and Hall/CRC Press.
CR  - Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142-151. https://doi.org/10.3102/0013189X10363170
CR  - Saal, P. E., van Ryneveld, L., & Graham, M. A. (2019). The relationship between using information and communication technology in education and the mathematics achievement of students. International Journal of Instruction, 12(3), 405-424. https://doi.org/10.29333/iji.2019.12325a
CR  - Sabudin, S., Mansor, A. N., Meerah, S. M., & Muhammad, A. (2018). Teacher-level factors that influence students’ science and technology culture: HLM analysis. International Journal of Academic Research in Business and Social Sciences, 8(5), 977-985. http://dx.doi.org/10.6007/IJARBSS/v8-i5/4243
CR  - Sahlberg, P., & Hargreaves, A. (2015). “The Tower of PISA is Badly Leaning. An Argument for Why It Should Be Saved”, The Washington Post. https://www.washingtonpost.com/news/answer-sheet/wp/2015/03/24/the-tower-of-pisa-is-badly-leaning-an-argument-for-why-it-should-be-saved/ (8 May 2023).
CR  - Schleicher A. (2015). “Attacks on PISA are Entirely Unjustified”, tes Magazine. https://www.tes.com/magazine/archive/attacks-pisa-are-entirely-unjustified-0 (8 May 2023).
CR  - Schmidt, W. H., McKnight, C. C., & Raizen, S. A. (1997). A splintered vision: An investigation of US science and mathematics education. U. S. National Research Center.
CR  - Sun, L., Bradley, K. D., & Akers, K. (2012). A multilevel modelling approach to investigating factors impacting science achievement for secondary school students: PISA Hong Kong sample. International Journal of Science Education, 34(14), 2107-2125. https://doi.org/10.1080/09500693.2012.708063
CR  - Takayama, K. (2015). “Has PISA Helped or Hindered?” https://headfoundation.org/2015/04/15/has-pisa-helped-or-hindered/ (8 May 2023).
CR  - Tat, O., Koyuncu, İ., & Gelbal, S. (2019). The influence of using plausible values and survey weights on multiple regression and hierarchical linear model parameters. Journal of Measurement and Evaluation in Education and Psychology, 10(3), 235-248. https://doi.org/10.21031/epod.486999
CR  - Thien, L. M., Darmawan, I. G. N., & Ong, M. Y. (2015). Affective characteristics and mathematics performance in Indonesia, Malaysia, and Thailand: What can PISA 2012 data tell us?. Large-Scale Assessments in Education, 3(3), 1-16. https://doi.org/10.1186/s40536-015-0013-z
CR  - Tobin, M., Lietz, P., Nugroho, D., Vivekanandan, R., & Nyamkhuu, T. (2015). Using large-scale assessments of students’ learning to inform education policy: Insights from the Asia-Pacific region. Melbourne: ACER and Bangkok: UNESCO.
CR  - Valente, M. O., Fonseca, J., & Conboy, J. (2011). Inquiry science teaching in Portugal and some other countries as measured by PISA 2006. Procedia - Social and Behavioral Sciences, 12, 255-262. https://doi.org/10.1016/j.sbspro.2011.02.034
CR  - Woltman, H., Feldstain, A., MacKay, J. C., & Rocchi, M. (2012). An introduction to hierarchical linear modeling. Tutorials in Quantitative Methods for Psychology, 8(1), 52-69. https://doi.org/10.20982/tqmp.08.1.p052
CR  - Woo, H., & Henfield, M. S. (2016). Student and teacher factors’ impact on fourth grade students’ mathematics achievement: An HLM analysis of TIMSS 2007. Journal of Mathematics Education, 9(1), 69-87.
UR  - https://doi.org/10.17556/erziefd.1404346
L1  - https://dergipark.org.tr/tr/download/article-file/3597186
ER  -