A New Weighting Method in Meta-Analysis: The Weighting with Reliability Coefficient

Year 2023, Volume: 14 Issue: 3, 249 - 265, 30.09.2023

Yıldız Yıldırım , Şeref Tan

https://doi.org/10.21031/epod.1351485

Abstract

This study aimed to investigate the impact of various weighting methods for effect sizes on the outcomes of meta-analyses that examined the effects of the 5E teaching method on academic achievement in science education. Two effect size weighting methods were explored: one based on the inverse of the sampling error variance and the other utilizing the reliability of measures in primary studies. The study also assessed the influence of including gray literature on the meta-analysis results, considering factors such as high heterogeneity and publication bias. The research followed a basic research design and drew data from 112 studies, encompassing a total of 149 effect sizes. An exhaustive search of databases and archives, including Google Scholar, Dergipark, HEI Thesis Center, Proquest, Science Direct, ERIC, Taylor & Francis, EBSCOhost, Web of Science, and five journals was conducted to gather these studies. Analyses were performed by utilizing the CMA v2 software and employing the random effects model. The findings demonstrated divergent outcomes between the two weighting methods—weighting by reliability coefficient yielded higher overall effect sizes and standard errors compared to weighting by inverse variance. Ultimately, the inclusion of gray literature was found to not significantly impact any of the weighting methods employed.

Keywords

weighting methods, meta-analysis, reliability coefficient, gray literature

References

• Alacapınar, F. G., & Ok, M. (2020). Meta-analysis covering studies on problem-based learning. Research on Education and Psychology, 4(Special Issue), 53-73. https://dergipark.org.tr/tr/pub/rep/issue/54042/703272
• Altunoğlu, B.D., Bozdemir Yüzbaşıoğlu, H., Candan Helvacı, S., & Kurnaz, M.A. (2020). Genetik kavramlara ilişkin eğitim çalışmalarının meta analiz yöntemi ile incelenmesi. Batı Anadolu Eğitim Bilimleri Dergisi, 11(2), 643-661. https://dergipark.org.tr/en/pub/baebd/issue/58594/702868
• Arık, S., & Yılmaz, M. (2020). The effect of constructivist learning approach and active learning on environmental education: A meta-analysis study. International Electronic Journal of Environmental Education, 10(1), 44-84. https://dergipark.org.tr/tr/pub/iejeegreen/issue/49969/605746
• Basit, O. (2020). Türkiye’de yapılan okul öncesi dönem çocuklarının gelişim alanlarını destekleyici çalışmaların incelenmesi: Bir meta analiz çalışması. Doktora Tezi, Gazi Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
• Başpınar, M. M. (2021). Türkiye'de yapılan tez çalışmalarında sigara içiminin uyku kalitesi üzerine etkisinin değerlendirilmesi: meta-analiz. Journal of Turkish Sleep Medicine, 8(1), 7-15. https://doi.org/10.4274/jtsm.galenos.2021.98698
• Batdı, V., & Batdı, H. (2015). Effect of creative drama on academic achievement: A meta-analytic and thematic analysis. Educational Sciences: Theory & Practice, 15(6), 1459-1470. https://doi.org/10.12738/estp.2015.6.0156
• Becker, K. H., & Park, K. (2011). Integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students’ learning: A meta-analysis. Journal of STEM Education Innovation and Research, 12(5), 23-37. https://doi.org/10.12691/education-2-10-4
• Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088-1101. PMID: 7786990. https://doi.org/10.2307/2533446
• Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons.
• Bozdemir, H., Çevik, E. E., Altunoğlu, B. D., & Kurnaz, M. A. (2017). Astronomi konularının öğretiminde kullanılan farklı yöntemlerin akademik başarıya etkisi: Bir meta analiz çalışması. Alan Eğitimi Araştırmaları Dergisi, 3(1), 12-24. https://dergipark.org.tr/tr/download/article-file/266427
• Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003). Grey literature in meta analyses. Nursing Research, 52, 256–261. https://doi.org/10.1097/00006199-200307000-00008
• Corlett, R. T. (2011). Trouble with the gray literature. Biotropica, 43(1), 3-5. https://doi.org/10.1111/j.1744-7429.2010.00714.x
• Duval, S., & Tweedie, R. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95(449), 89-98. https://doi.org/10.1080/01621459.2000.10473905
• Duval, S., & Tweedie, R. (2000b). Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication bias in meta‐analysis. Biometrics, 56(2), 455-463. https://doi.org/10.1111/j.0006-341x.2000.00455.x.
• Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629-634. https://doi.org/10.1136/bmj.315.7109.629
• Englund, G., Sarnelle, O., & Cooper, S. D. (1999). The importance of data‐selection criteria: meta‐analyses of stream predation experiments. Ecology, 80(4), 1132-1141. https://www.jstor.org/stable/177060?seq=1#metadata_info_tab_contents
• Fabiano, G., Schatz, N., Aloe, A., Pelham, W., Smyth, A., Zhao, X., Merrill, B. M., Macphee, F., Ramos, M., Hong, N., Altszuler, A., Ward, L., Rodgers, D. B., Liu, Z., Karatoprak Ersen, R., & Coxe, S. (2021). Comprehensive meta-analysis of attention-deficit hyperactivity disorder psychosocial treatments investigated within between group studies. Review of Educational Research, 91(5), 718–760. https://doi.org/10.3102/00346543211025092
• Fuller, J. B., & Hester, K. (1999). Comparing the sample-weighted and unweighted meta-analysis: An applied perspective. Journal of Management, 25(6), 803-828. https://doi.org/10.1177/014920639902500602
• Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. https://doi.org/10.2307/1174772
• Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2017). Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology, 17(1), 1-11. https://doi.org/10.1186/s12874-017-0347-z
• Hedges, L. V. & Olkin, I.(1985). Statistical methods for meta-analysis. Academic.
• Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. https://doi.org/10.1037/1082-989X.3.4.486
• Higgins, J. P., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions. The Cochrane Collaboration and John Wiley & Sons.
• Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta analysis. Statistics in Medicine, 21(11), 1539-1558. https://doi.org/10.1002/sim.1186
• Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ: British Medical Journal, 327(7414), 557. https://doi.org/10.1136/bmj.327.7414.557
• Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Beverly Hills, CA: Sage.
• Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
• Karasar, N. (2013). Bilimsel araştırma yöntemleri (25th ed.). Nobel.
• Lazonder, A. W., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 86(3), 681-718. https://doi.org/10.3102/0034654315627366 Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology, 62(10), 1-34. https://doi.org/10.1136/bmj.b2700
• Marín-Martínez, F., & Sánchez-Meca, J. (2009). Weighting by inverse variance or by sample size in random-effects meta-analysis. Educational and Psychological Measurement, 70(1), 56-73. https://doi.org/10.1177/0013164409344534
• Moher, D., Fortin, P., Jadad, A. R., Jüni, P., Klassen, T., Le Lorier, J., ... & Linde, K. (1996). Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. The Lancet, 347(8998), 363-366. https://doi.org/10.1016/s0140-6736(96)90538-3
• Mutluer, C., Gündüz, T., Çelikten Demirel, S., & Çakan, M. (2020). Meta analiz çalışmasında sabit etki modeline karşı rastgele etki modeli. Presented at the VIIth International Eurasian Educational Research Congress, Eskişehir.
• National Research Council. (1992). Combining information: statistical issues and opportunities for research. National Academies. https://doi.org/10.17226/20865
• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. https://doi.org/10.1037/0033-2909.86.3.638
• Rosenthal, R. (1991). Quality-weighting of studies in meta-analytic research. Psychotherapy Research, 1(1), 25-28. https://doi.org/10.1080/10503309112331334031
• Saraç, H. (2018). Yapılandırmacı yaklaşım öğrenme halkası modellerinin öğrenilen bilgilerin kalıcılığına etkisi: Meta analiz çalışması. Kastamonu Eğitim Dergisi, 26(3), 753-764. https://doi.org/10.24106/kefdergi.413322
• Schmidt, F. L., Oh, I. S., & Hayes, T. L. (2009). Fixed‐versus random‐effects models in meta‐analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62(1), 97-128. https://doi.org/10.1348/000711007X255327
• Shuster, J. J. (2010). Empirical vs natural weighting in random effects meta‐analysis. Statistics in Medicine, 29(12), 1259-1265. https://doi.org/10.1002/sim.3607
• Simpson, R. J. S., & Pearson, K. (1904). Report on certain enteric fever inoculation statistics. The British Medical Journal, 2(2288), 1243-1246. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2355479/
• Sosa, G. W., Berger, D. E., Saw, A. T., & Mary, J. C. (2011). Effectiveness of computer-assisted instruction in statistics: A meta-analysis. Review of Educational Research, 81(1), 97-128. https://doi.org/10.3102/0034654310378174
• Toraman, Ç. , Çelik, Ö. C. & Çakmak, M. (2018). Oyun-Tabanlı Öğrenme Ortamlarının Akademik Başarıya Etkisi: Bir Meta–Analiz Çalışması . Kastamonu Eğitim Dergisi , 26 (6) , 1803-1811 . https://doi.org/10.24106/kefdergi.2074
• Özdemir, V. (2023). Okuma becerisi ile sosyoekonomik ve kültürel değişkenler arasındaki ilişkilerin incelenmesi meta-analiz çalışması. Presented at the X International Eurasian Educational Research Congress, Ankara.
• Warfa, A. R. M. (2016). Using cooperative learning to teach chemistry: A meta-analytic review. Journal of Chemical Education, 93(2), 248-255. https://doi.org/10.1021/acs.jchemed.5b00608
• Xie C., Wang, M., & Hu, H. (2018). Effects of constructivist and transmission instructional models on mathematics achievement in mainland China: A meta-analysis. Front. Psychol., 9(1923), 1-18. https://doi.org/10.3389/fpsyg.2018.01923
• Yeşilpınar Uyar, M. & Doğanay, A. (2018). Öğrenci merkezli strateji, yöntem ve tekniklerin akademik başarıya etkisi: Bir meta-analiz çalışması. Mersin Üniversitesi Eğitim Fakültesi Dergisi, 14(1), 186-209. https://doi.org/10.17860/mersinefd.334542
• Yıldırım, Y., & Şahin, M. G. (2023). How Do Different Weighting Methods Affect the Overall Effect Size in Meta-Analysis?: An Example of Science Attitude in Türkiye Sample. International Journal of Psychology and Educational Studies, 10(3), 744-757. https://doi.org/10.52380/ijpes.2023.10.3.1049