The Effect of Sample Weighting on Hierarchical Linear Modeling in the Large-Scale Assessment Data

Abstract

This study examines how the different uses of sampling weights in the analysis of TIMSS 2019 data affect the ratio of variance in student achievement explained by schools and the estimation of standard errors. The research sample comprises 227,345 8th grade students from 7,636 schools in 39 countries. Mathematics achievement and science achievement are considered separately as dependent variables in all 39 countries. All plausible values are included in the analysis. Four weighting scenarios are examined: no weighting, weighting at only level 1, weighting at only level 2, and weighting at both levels. In total, 312 models are established and examined. According to the research results, the coefficients, standard errors, reliabilities, and χ^2 estimations change depending on how the weighting variable is handled in the models, and as a result, the ratio of variance in the dependent variable arising from the differences between schools also changes. The ratio attributable to between-school differences can reach up to 20% in some countries. Therefore, researchers modeling hierarchical data using HLM are suggested to plan how they handle the weighting variable prior to conducting the study.

Keywords

• large scale assessment
• hierarchical linear modelling
• weighting
• TIMSS 2019

Geniş Ölçekli Test Verilerinde Örneklem Ağırlıklandırmalarının Hiyerarşik Doğrusal Modellemeye Etkisi

Öz

Bu çalışmada TIMSS 2019 verilerinin analizinde örneklem ağırlıklarının farklı şekilde kullanımlarının öğrenci başarısındaki varyansın okullar tarafından açıklanan kısmında ve standart hataların kestiriminde nasıl bir etkiye sahip olduğu incelenmiştir. Araştırmanın örneklemini TIMSS 2019 uygulamasına katılan 39 ülkeden toplam 7636 okuldaki 227345 8. sınıf öğrencisi oluşturmaktadır. Matematik başarısı ve fen başarısı tüm ülkelerde bağımlı değişkenler olarak ayrı ayrı ele alınmıştır. Tüm olası değerler analize dahil edilmiştir. Ağırlıklandırmanın olmadığı, yalnızca 1. düzeyde ağırlıklandırmanın olduğu, yalnızca 2. düzeyde ağırlıklandırmanın olduğu ve her iki düzeyde de ağırlıklandırmanın olduğu dört farklı durum incelenmiştir. Toplamda 312 model kurulmuş ve çözümlenmiştir. Araştırmanın sonuçlarına göre katsayıların, standart hataların, güvenirlik değerlerinin ve χ^2 istatistiklerinin, ağırlıklandırma değişkeninin kullanılma biçimine göre değiştiği gözlenmiştir. Bunun bir sonucu olarak da, çıktı değişkenindeki varyansın açıklanmasında okullar tarafından açıklanan kısım değişkenlik göstermektedir. Bu kısımdaki değişkenlik bazı ülkelerde %20’lere kadar çıkabilmektedir. Bu nedenle, geniş ölçekli testlerin verilerini HLM ile modelleyecek araştırmacıların ağırlıklandırma değişkenini ne şekilde ele alacaklarını araştırma öncesinde planlamaları önerilmektedir.

Anahtar Kelimeler

• geniş ölçekli test
• hiyerarşik doğrusal modelleme
• ağırlıklandırma
• TIMSS 2019

Kaynakça

1. Ababneh, E., Al-Tweissi, A., & Abulibdeh, K. (2016). TIMSS and PISA impact – the case of Jordan. Research Papers in Education, 31(5), 542-555, http://dx.doi.org/10.1080/02671522.2016.1225350
2. Aksu, G., Güzeller, C. O., & Eser, M. T. (2017). Analysis of maths literacy performances of students with hierarchical linear modeling (HLM): The case of PISA 2012 Turkey. Education and Science, 42(191), 247-266. http://dx.doi.org/10.15390/EB.2017.6956
3. Arıkan, S., Özer, F., Şeker, V., & Ertaş, G. (2020). The importance of sample weights and plausible values in large-scale assessments. Journal of Measurement and Evaluation in Education and Psychology, 11(1), 43-60. https://doi.org/10.21031/epod.602765
4. Atar, H. Y., & Atar, B. (2012). Examining the effects of Turkish education reform on students’ TIMSS 2007 science achievements. Educational Sciences: Theory & Practice, 12(4), 2632-2636.
5. Barber, M., Chijioke, C., & Mourshed, M. (2010). How the World’s most improved school systems keep getting better. McKinsey and Company.
6. Bilican, S., & Yıldırım, Ö. (2013). The effects of approaches to learning on student's reflective and evaluative reading performance in Turkey: The results from PISA 2009. Procedia - Social and Behavioral Sciences, 116, 2437-2442. https://doi.org/10.1016/j.sbspro.2014.01.588
7. Boulifa, K., & Kaaouachi, A. (2022). The relationship between the school resources index, gender, age and mathematics achievement in TIMSS 2019 survey: Multilevel analysis. Procedia Computer Science, 201, 738-745. https://doi.org/10.1016/j.procs.2022.03.100
8. Carle, A. C. (2009). Fitting multilevel models in complex survey data with design weights: Recommendations. BMC Medical Research Methodology, 9, 1-13. http://doi.org/10.1186/1471-2288-9-49

9. Chu, M. W., Babenko, O., Cui, Y., & Leighton, J. P. (2014). Using HLM to explore effects of perceptions of learning environments and assessments on students’ test performance. International Journal of Testing, 14(2), 95-121. https://doi.org/10.1080/15305058.2013.841702
10. Fraenkel, J. R., & Wallen, N. E. (2006). How to design and evaluate research in education (6th Ed.). McGraw-Hill.
11. Gómez, R. L., & Suárez, A. M. (2020). Do inquiry-based teaching and school climate influence science achievement and critical thinking? Evidence from PISA 2015. International Journal of STEM Education, 7(43), 1-11. https://doi.org/10.1186/s40594-020-00240-5
12. Hox, J. J. (2010). Multilevel analysis: Techniques and applications. Routledge.
13. IBM Corp. (2013). IBM SPSS Statistics for Windows (Version 22.0). IBM Corp.
14. Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communications in Statistics - Theory and Methods, 46(22), 11341-11357. https://doi.org/10.1080/03610926.2016.1267764
15. Liang, X. (2010). Assessment use, self-efficacy and mathematics achievement: comparative analysis of PISA 2003 data of Finland, Canada and the USA. Evaluation & Research in Education, 23(3), 213-229. https://doi.org/10.1080/09500790.2010.490875
16. Liou, P. Y., & Hung, Y. C. (2015). Statistical techniques utilized in analyzing PISA and TIMSS data in science education from 1996 to 2013: A methodological review. International Journal of Science and Mathematics Education, 13, 1449-1468. https://doi.org/10.1007/s10763-014-9558-5
17. Mang, J., Küchenhoff, H., Meinck, S., & Prenzel, M. (2021). Sampling weights in multilevel modelling: An investigation using PISA sampling structures. Large-scale Assess Educ., 9(6), 1-39. https://doi.org/10.1186/s40536-021-00099-0
18. Manjunath, M. (2021). Using large-scale assessments to inform education policy and governance, Assessment resources, Azim Premji University. Retrieved from https://cdn.azimpremjiuniversity.edu.in/apuc3/media/resources/Assessments-and-Education-Policy.f1640171199.pdf
19. Meinck, S., & Vandenplas, C. (2012). Sample size requirements in HLM: An empirical study. IERI monograph series issues and methodologies in large-scale assessments. IER Institute, special issue 1, Educational Testing Service and International Association for the Evaluation of Educational Achievement.
20. Organisation for Economic Co-operation and Development [OECD]. (2017). PISA 2015 technical report. Paris: OECD Publishing.
21. Özdemir, C. (2016). A methodological review of research using OECD PISA Turkey data. Education Science Society Journal, 14(56), 10-27.
22. Pacheco Diaz, N., & Rocconi, L. M. (2021). Examining science achievement in Chile: A multilevel approach using PISA 2015 data. Journal of Research in STEM Education, 7(2), 93-116. https://doi.org/10.51355/jstem.2021.100
23. Pong, S. (2009). Grade level and achievement of immigrants' children: academic redshirting in Hong Kong. Educational Research and Evaluation: An International Journal on Theory and Practice, 15(4), 405-425. http://dx.doi.org/10.1080/13803610903087078
24. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods 2. Sage Publications.
25. Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2007). HLM for Windows (Version 6.04). Scientific Software International.
26. Reinikainen, P. (2007). Sequential explanatory study of factors connected with science achievement in six countries: Finland, England, Hungary, Japan, Latvia and Russia: Study based on TIMSS 1999. Institute for Educational Research, University of Jyväskylä.
27. Ross, S. P. (2008). Motivation correlates of academic achievement: Exploring how motivation influences academic achievement in the PISA 2003 dataset [Doctoral Dissertation, University of Victoria]. UVicSpace. https://dspace.library.uvic.ca/handle/1828/3209
28. Rust, K. (2013). Sampling, weighting, and variance estimation in international large-scale assessments. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 117-154). Chapman and Hall/CRC Press.
29. Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142-151. https://doi.org/10.3102/0013189X10363170
30. Saal, P. E., van Ryneveld, L., & Graham, M. A. (2019). The relationship between using information and communication technology in education and the mathematics achievement of students. International Journal of Instruction, 12(3), 405-424. https://doi.org/10.29333/iji.2019.12325a
31. Sabudin, S., Mansor, A. N., Meerah, S. M., & Muhammad, A. (2018). Teacher-level factors that influence students’ science and technology culture: HLM analysis. International Journal of Academic Research in Business and Social Sciences, 8(5), 977-985. http://dx.doi.org/10.6007/IJARBSS/v8-i5/4243
32. Sahlberg, P., & Hargreaves, A. (2015). “The Tower of PISA is Badly Leaning. An Argument for Why It Should Be Saved”, The Washington Post. https://www.washingtonpost.com/news/answer-sheet/wp/2015/03/24/the-tower-of-pisa-is-badly-leaning-an-argument-for-why-it-should-be-saved/ (8 May 2023).
33. Schleicher A. (2015). “Attacks on PISA are Entirely Unjustified”, tes Magazine. https://www.tes.com/magazine/archive/attacks-pisa-are-entirely-unjustified-0 (8 May 2023).
34. Schmidt, W. H., McKnight, C. C., & Raizen, S. A. (1997). A splintered vision: An investigation of US science and mathematics education. U. S. National Research Center.
35. Sun, L., Bradley, K. D., & Akers, K. (2012). A multilevel modelling approach to investigating factors impacting science achievement for secondary school students: PISA Hong Kong sample. International Journal of Science Education, 34(14), 2107-2125. https://doi.org/10.1080/09500693.2012.708063
36. Takayama, K. (2015). “Has PISA Helped or Hindered?” https://headfoundation.org/2015/04/15/has-pisa-helped-or-hindered/ (8 May 2023).
37. Tat, O., Koyuncu, İ., & Gelbal, S. (2019). The influence of using plausible values and survey weights on multiple regression and hierarchical linear model parameters. Journal of Measurement and Evaluation in Education and Psychology, 10(3), 235-248. https://doi.org/10.21031/epod.486999
38. Thien, L. M., Darmawan, I. G. N., & Ong, M. Y. (2015). Affective characteristics and mathematics performance in Indonesia, Malaysia, and Thailand: What can PISA 2012 data tell us?. Large-Scale Assessments in Education, 3(3), 1-16. https://doi.org/10.1186/s40536-015-0013-z
39. Tobin, M., Lietz, P., Nugroho, D., Vivekanandan, R., & Nyamkhuu, T. (2015). Using large-scale assessments of students’ learning to inform education policy: Insights from the Asia-Pacific region. Melbourne: ACER and Bangkok: UNESCO.
40. Valente, M. O., Fonseca, J., & Conboy, J. (2011). Inquiry science teaching in Portugal and some other countries as measured by PISA 2006. Procedia - Social and Behavioral Sciences, 12, 255-262. https://doi.org/10.1016/j.sbspro.2011.02.034
41. Woltman, H., Feldstain, A., MacKay, J. C., & Rocchi, M. (2012). An introduction to hierarchical linear modeling. Tutorials in Quantitative Methods for Psychology, 8(1), 52-69. https://doi.org/10.20982/tqmp.08.1.p052
42. Woo, H., & Henfield, M. S. (2016). Student and teacher factors’ impact on fourth grade students’ mathematics achievement: An HLM analysis of TIMSS 2007. Journal of Mathematics Education, 9(1), 69-87.