A New Weighting Method in Meta-Analysis: The Weighting with Reliability Coefficient

Abstract

This study aimed to investigate the impact of various weighting methods for effect sizes on the outcomes of meta-analyses that examined the effects of the 5E teaching method on academic achievement in science education. Two effect size weighting methods were explored: one based on the inverse of the sampling error variance and the other utilizing the reliability of measures in primary studies. The study also assessed the influence of including gray literature on the meta-analysis results, considering factors such as high heterogeneity and publication bias. The research followed a basic research design and drew data from 112 studies, encompassing a total of 149 effect sizes. An exhaustive search of databases and archives, including Google Scholar, Dergipark, HEI Thesis Center, Proquest, Science Direct, ERIC, Taylor & Francis, EBSCOhost, Web of Science, and five journals was conducted to gather these studies. Analyses were performed by utilizing the CMA v2 software and employing the random effects model. The findings demonstrated divergent outcomes between the two weighting methods—weighting by reliability coefficient yielded higher overall effect sizes and standard errors compared to weighting by inverse variance. Ultimately, the inclusion of gray literature was found to not significantly impact any of the weighting methods employed.

Keywords

• weighting methods
• meta-analysis
• reliability coefficient
• gray literature

References

1. Alacapınar, F. G., & Ok, M. (2020). Meta-analysis covering studies on problem-based learning. Research on Education and Psychology, 4(Special Issue), 53-73. https://dergipark.org.tr/tr/pub/rep/issue/54042/703272
2. Altunoğlu, B.D., Bozdemir Yüzbaşıoğlu, H., Candan Helvacı, S., & Kurnaz, M.A. (2020). Genetik kavramlara ilişkin eğitim çalışmalarının meta analiz yöntemi ile incelenmesi. Batı Anadolu Eğitim Bilimleri Dergisi, 11(2), 643-661. https://dergipark.org.tr/en/pub/baebd/issue/58594/702868
3. Arık, S., & Yılmaz, M. (2020). The effect of constructivist learning approach and active learning on environmental education: A meta-analysis study. International Electronic Journal of Environmental Education, 10(1), 44-84. https://dergipark.org.tr/tr/pub/iejeegreen/issue/49969/605746
4. Basit, O. (2020). Türkiye’de yapılan okul öncesi dönem çocuklarının gelişim alanlarını destekleyici çalışmaların incelenmesi: Bir meta analiz çalışması. Doktora Tezi, Gazi Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
5. Başpınar, M. M. (2021). Türkiye'de yapılan tez çalışmalarında sigara içiminin uyku kalitesi üzerine etkisinin değerlendirilmesi: meta-analiz. Journal of Turkish Sleep Medicine, 8(1), 7-15. https://doi.org/10.4274/jtsm.galenos.2021.98698
6. Batdı, V., & Batdı, H. (2015). Effect of creative drama on academic achievement: A meta-analytic and thematic analysis. Educational Sciences: Theory & Practice, 15(6), 1459-1470. https://doi.org/10.12738/estp.2015.6.0156
7. Becker, K. H., & Park, K. (2011). Integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students’ learning: A meta-analysis. Journal of STEM Education Innovation and Research, 12(5), 23-37. https://doi.org/10.12691/education-2-10-4
8. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 1088-1101. PMID: 7786990. https://doi.org/10.2307/2533446

9. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons.
10. Bozdemir, H., Çevik, E. E., Altunoğlu, B. D., & Kurnaz, M. A. (2017). Astronomi konularının öğretiminde kullanılan farklı yöntemlerin akademik başarıya etkisi: Bir meta analiz çalışması. Alan Eğitimi Araştırmaları Dergisi, 3(1), 12-24. https://dergipark.org.tr/tr/download/article-file/266427
11. Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003). Grey literature in meta analyses. Nursing Research, 52, 256–261. https://doi.org/10.1097/00006199-200307000-00008
12. Corlett, R. T. (2011). Trouble with the gray literature. Biotropica, 43(1), 3-5. https://doi.org/10.1111/j.1744-7429.2010.00714.x
13. Duval, S., & Tweedie, R. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95(449), 89-98. https://doi.org/10.1080/01621459.2000.10473905
14. Duval, S., & Tweedie, R. (2000b). Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication bias in meta‐analysis. Biometrics, 56(2), 455-463. https://doi.org/10.1111/j.0006-341x.2000.00455.x.
15. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. Bmj, 315(7109), 629-634. https://doi.org/10.1136/bmj.315.7109.629
16. Englund, G., Sarnelle, O., & Cooper, S. D. (1999). The importance of data‐selection criteria: meta‐analyses of stream predation experiments. Ecology, 80(4), 1132-1141. https://www.jstor.org/stable/177060?seq=1#metadata_info_tab_contents
17. Fabiano, G., Schatz, N., Aloe, A., Pelham, W., Smyth, A., Zhao, X., Merrill, B. M., Macphee, F., Ramos, M., Hong, N., Altszuler, A., Ward, L., Rodgers, D. B., Liu, Z., Karatoprak Ersen, R., & Coxe, S. (2021). Comprehensive meta-analysis of attention-deficit hyperactivity disorder psychosocial treatments investigated within between group studies. Review of Educational Research, 91(5), 718–760. https://doi.org/10.3102/00346543211025092
18. Fuller, J. B., & Hester, K. (1999). Comparing the sample-weighted and unweighted meta-analysis: An applied perspective. Journal of Management, 25(6), 803-828. https://doi.org/10.1177/014920639902500602
19. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. https://doi.org/10.2307/1174772
20. Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2017). Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology, 17(1), 1-11. https://doi.org/10.1186/s12874-017-0347-z
21. Hedges, L. V. & Olkin, I.(1985). Statistical methods for meta-analysis. Academic.
22. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. https://doi.org/10.1037/1082-989X.3.4.486
23. Higgins, J. P., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions. The Cochrane Collaboration and John Wiley & Sons.
24. Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta analysis. Statistics in Medicine, 21(11), 1539-1558. https://doi.org/10.1002/sim.1186
25. Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ: British Medical Journal, 327(7414), 557. https://doi.org/10.1136/bmj.327.7414.557
26. Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Beverly Hills, CA: Sage.
27. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
28. Karasar, N. (2013). Bilimsel araştırma yöntemleri (25th ed.). Nobel.
29. Lazonder, A. W., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 86(3), 681-718. https://doi.org/10.3102/0034654315627366 Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of Clinical Epidemiology, 62(10), 1-34. https://doi.org/10.1136/bmj.b2700
30. Marín-Martínez, F., & Sánchez-Meca, J. (2009). Weighting by inverse variance or by sample size in random-effects meta-analysis. Educational and Psychological Measurement, 70(1), 56-73. https://doi.org/10.1177/0013164409344534
31. Moher, D., Fortin, P., Jadad, A. R., Jüni, P., Klassen, T., Le Lorier, J., ... & Linde, K. (1996). Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. The Lancet, 347(8998), 363-366. https://doi.org/10.1016/s0140-6736(96)90538-3
32. Mutluer, C., Gündüz, T., Çelikten Demirel, S., & Çakan, M. (2020). Meta analiz çalışmasında sabit etki modeline karşı rastgele etki modeli. Presented at the VIIth International Eurasian Educational Research Congress, Eskişehir.
33. National Research Council. (1992). Combining information: statistical issues and opportunities for research. National Academies. https://doi.org/10.17226/20865
34. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. https://doi.org/10.1037/0033-2909.86.3.638
35. Rosenthal, R. (1991). Quality-weighting of studies in meta-analytic research. Psychotherapy Research, 1(1), 25-28. https://doi.org/10.1080/10503309112331334031
36. Saraç, H. (2018). Yapılandırmacı yaklaşım öğrenme halkası modellerinin öğrenilen bilgilerin kalıcılığına etkisi: Meta analiz çalışması. Kastamonu Eğitim Dergisi, 26(3), 753-764. https://doi.org/10.24106/kefdergi.413322
37. Schmidt, F. L., Oh, I. S., & Hayes, T. L. (2009). Fixed‐versus random‐effects models in meta‐analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62(1), 97-128. https://doi.org/10.1348/000711007X255327
38. Shuster, J. J. (2010). Empirical vs natural weighting in random effects meta‐analysis. Statistics in Medicine, 29(12), 1259-1265. https://doi.org/10.1002/sim.3607
39. Simpson, R. J. S., & Pearson, K. (1904). Report on certain enteric fever inoculation statistics. The British Medical Journal, 2(2288), 1243-1246. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2355479/
40. Sosa, G. W., Berger, D. E., Saw, A. T., & Mary, J. C. (2011). Effectiveness of computer-assisted instruction in statistics: A meta-analysis. Review of Educational Research, 81(1), 97-128. https://doi.org/10.3102/0034654310378174
41. Toraman, Ç. , Çelik, Ö. C. & Çakmak, M. (2018). Oyun-Tabanlı Öğrenme Ortamlarının Akademik Başarıya Etkisi: Bir Meta–Analiz Çalışması . Kastamonu Eğitim Dergisi , 26 (6) , 1803-1811 . https://doi.org/10.24106/kefdergi.2074
42. Özdemir, V. (2023). Okuma becerisi ile sosyoekonomik ve kültürel değişkenler arasındaki ilişkilerin incelenmesi meta-analiz çalışması. Presented at the X International Eurasian Educational Research Congress, Ankara.
43. Warfa, A. R. M. (2016). Using cooperative learning to teach chemistry: A meta-analytic review. Journal of Chemical Education, 93(2), 248-255. https://doi.org/10.1021/acs.jchemed.5b00608
44. Xie C., Wang, M., & Hu, H. (2018). Effects of constructivist and transmission instructional models on mathematics achievement in mainland China: A meta-analysis. Front. Psychol., 9(1923), 1-18. https://doi.org/10.3389/fpsyg.2018.01923
45. Yeşilpınar Uyar, M. & Doğanay, A. (2018). Öğrenci merkezli strateji, yöntem ve tekniklerin akademik başarıya etkisi: Bir meta-analiz çalışması. Mersin Üniversitesi Eğitim Fakültesi Dergisi, 14(1), 186-209. https://doi.org/10.17860/mersinefd.334542
46. Yıldırım, Y., & Şahin, M. G. (2023). How Do Different Weighting Methods Affect the Overall Effect Size in Meta-Analysis?: An Example of Science Attitude in Türkiye Sample. International Journal of Psychology and Educational Studies, 10(3), 744-757. https://doi.org/10.52380/ijpes.2023.10.3.1049