Eğitim Araştırmalarında Çok Düzeyli Meta-Analiz Modelleri: Örnek Uygulamalı Bir Rehber

Year 2024, Issue: 61, 2502 - 2530, 27.09.2024

Dilek Karaca , Burak Aydın , Hakan Atılgan

https://doi.org/10.53444/deubefd.1476011

Abstract

Meta-analiz belirli alanda yapılmış çalışmalardan sistematik şekilde elde edilen nicel verileri kullanarak o alanla ilgili genel durumu ortaya koymaya çalışan istatistiksel bir yöntemdir. Geleneksel meta-analiz yöntemleri, etki büyüklükleri arasında bağımlılık olmadığı varsayımına dayanmasına rağmen özellikle sosyal bilimlerde etki büyüklüğü bağımlılığına neden olabilecek çok sayıda durum söz konusudur. Araştırmacının elde ettiği etki büyüklüklerindeki bağımlılık veri setinde kümeli bir yapı oluşturur. Geleneksel meta-analiz uygulamalarındaki etki büyüklüğü bağımlılığı sorunu ile başa çıkmak ve kümeli veri yapısını dikkate almak için önerilen yöntemlerden biri çok düzeyli meta-analitik modellerin kullanılmasıdır. Çok düzeyli modeller diğer istatistiksel çerçevelerle birleştirilebilir ve kümeli veri yapılarının daha savunulabilir şekilde çözümlenebilmesini sağlayabilir. Bu sebeple çok düzeyli modellerin sosyal bilimlerde kullanım sıklığı her geçen gün artmaktadır. Bu çalışmanın amacı örnek bir veri seti üzerinden çok düzeyli meta-analitik modellerin nasıl uygulanabileceğini göstermektir. R yazılımı kullanılarak gerçekleştirilen analizlerde metafor paketinin rma.mv fonksiyonu kullanılmıştır. Bu uygulama okuyuculara veri dosyasının düzenlenmesi, R yazılımının hazırlanması, genel etkinin hesaplanması, çalışma içi ve çalışmalar arası varyans heterojenliğinin incelenmesi ile kategorik ve sürekli değişkenlere ait moderatör analizlerinin nasıl yapılacağını adım adım anlatan bir kılavuz niteliği taşımaktadır. Çalışmada kullanılan veri dosyası ve R betiği okuyucuların kullanımı için ekler kısmında sunulmuştur.

Keywords

Meta-analiz, çok düzeyli meta-analiz, çok düzeyli modeller, metafor, R yazılımı

Multilevel Meta-Analysis Models in Educational Research: A Practical Guide

Abstract

Meta-analysis is a statistical method that tries to reveal the general situation about that field by using quantitative data obtained systematically from studies conducted in a certain field. However, traditional meta-analysis methods assume effect size independence, which can be a significant and misleading limitation in social sciences research. The dependence in the effect sizes obtained by the researcher creates a clustered data structure, a problem that needs attention. One of the methods to address this problem is the use of multilevel meta-analytic models. Multilevel models can be combined with other statistical frameworks, providing a more defensible analysis of clustered data structures. For this reason, the frequency of multilevel models in social sciences is steadily increasing. This study demonstrates how multilevel meta-analytic models can be applied to a sample data set. In the R environment, the rma.mv function of the metafor package was utilized for the analyses. This application is a step-by-step guide for readers on organizing the data file, preparing the R software, calculating the overall effect, examining the heterogeneity of variance within and between studies, and conducting moderator analyses with categorical and continuous variables. The data file and R script used in the study are shared as supplementary material for the readers.

Keywords

Meta-analysis, multilevel meta-analysis, multilevel models, metafor, R software

References

• APA. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63(9), 839-851. https://doi.org/10.1037/0003-066X.63.9.839
• Assink, M., & Wibbelink, C. J. M. (2016). Fitting three-level meta-analytic models in R: A step-by-step tutorial. The Quantitative Methods for Psychology, 12(3), 154-174. https://doi.org/10.20982/tqmp.12.3.p154
• Becker, B. J. (2000). Multivariate meta-analysis. Handbook of applied multivariate statistics and mathematical modeling, 499-525. https://doi.org/10.1016/B978-012691360-6/50018-5
• Begg, C. B., & Mazumdar, M. (1994). Operating Characteristics of a Rank Correlation Test for Publication Bias. Biometrics, 50(4), 1088–1101. https://doi.org/10.2307/2533446
• Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons. http://doi.org/10.1002/9780470743386
• Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211-229. https://doi.org/10.1037/a0032968
• Cheung, M. W. L. (2015). Meta-Analysis: A Structural Equation Modeling Approach. John Wiley & Sons. https://doi.org/10.1080/10705510802561295
• Cheung, M. W. L. (2019). A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes. Neuropsychology Review, 29(4), 387-396. https://doi.org/10.1007/s11065-019-09415-6
• Cheung, S. F., & Chan, D. K. S. (2004). Dependent Effect Sizes in Meta-Analysis: Incorporating the Degree of Interdependence. Journal of Applied Psychology, 89(5), 780-791. https://doi.org/10.1037/0021-9010.89.5.780
• Cheung, S. F., & Chan, D. K. S. (2014). Meta-analyzing dependent correlations: An SPSS macro and an R script. Behavior Research Methods, 46(2), 331-345. https://doi.org/10.3758/s13428-013-0386-2
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2. edition). Lawrence Erlbaum Associates, Publishers. https://doi.org/10.4324/9780203771587
• Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2019). The handbook of research synthesis and meta-analysis (3.edition). Russell Sage Foundation.
• Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455-463. https://doi.org/10.1111/j.0006-341X.2000.00455.x
• Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. bmj, 315(7109), 629-634. https://doi.org/10.1136/bmj.315.7109.629
• Fisher, R. A. (1933). Statistical Methods for Research Workers. Nature, 131(3307), Article 3307. https://doi.org/10.1038/131383b0
• Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational researcher, 5(10), 3-8. https://doi.org/10.3102/0013189X005010003
• Gooty, J., Banks, G. C., Loignon, A. C., Tonidandel, S., & Williams, C. E. (2021). Meta-Analyses as a Multi-Level Model. Organizational Research Methods, 24(2), 389-411. https://doi.org/10.1177/1094428119857471
• Harrer, M., Cuijpers, P., Furukawa, T. A., & Ebert, D. D. (2021). Doing Meta-Analysis with R: A Hands-On Guide (1.st edition). Chapman and Hall/CRC. https://doi.org/10.1201/9781003107347
• Hedges, L. V. (1982). Fitting Categorical Models to Effect Sizes from a Series of Experiments. Journal of Educational Statistics, 7(2), 119-137. https://doi.org/10.3102/10769986007002119
• Hedges, L. V., & Olkin, I. (1988). Statistical Methods for Meta-Analysis. Journal of Educational Statistics, 13(1), 75. https://doi.org/10.2307/1164953
• Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203-217. https://doi.org/10.1037/1082-989X.6.3.203
• Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in Medicine, 21(11), 1539-1558. https://doi.org/10.1002/sim.1186
• Hox, J. J., & De Leeuw, E. D. (2003). Multilevel models for meta-analysis. In Multilevel modeling (pp. 87-104). Psychology Press.
• Hox, J. J., Moerbeek, M., & Schoot, R. van de. (2010). Multilevel Analysis: Techniques and Applications, Second Edition. Routledge. https://doi.org/10.4324/9780203852279
• Hunter, J. E., & Schmidt, F. L. (2004). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, Second Edition. SAGE.
• Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2(1), 61-76. https://doi.org/10.1002/jrsm.35
• Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis. SAGE Publications, Inc.
• Marín-Martínez, F., & Sánchez-Meca, J. (1999). Averaging Dependent Effect Sizes in Meta-Analysis: A Cautionary Note about Procedures. The Spanish Journal of Psychology, 2, 32-38. https://doi.org/10.1017/S1138741600005436
• Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009). Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches. Review of Educational Research, 79(3), 1290-1326. https://doi.org/10.3102/0034654309334143
• Midway, S. (2022). Data analysis in R. Retrieved April, 8, 2023. https://bookdown.org/steve_midway/DAR/
• Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999). Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. The Lancet, 354(9193), 1896-1900. https://doi.org/10.1016/S0140-6736(99)04149-5
• Morris, S. B. (2008). Estimating Effect Sizes From Pretest-Posttest-Control Group Designs. Organizational Research Methods, 11(2), 364-386. https://doi.org/10.1177/1094428106291059
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Journal of clinical epidemiology, 134, 103-112. https://doi.org/10.1136/bmj.n71
• Pastor, D. A., & Lazowski, R. A. (2018). On the Multilevel Nature of Meta-Analysis: A Tutorial, Comparison of Software Programs, and Discussion of Analytic Choices. Multivariate Behavioral Research, 53(1), 74-89. https://doi.org/10.1080/00273171.2017.1365684
• Pearson. (1904). Report On Certain Enteric Fever Inoculation Statistics. 1243-1246.
• Pustejovsky, J. (2023). clubSandwich: Cluster-Robust (Sandwich) Variance Estimators with Small-Sample Corrections (R package version 0.5.10). https://cran.r-project.org/web/packages/clubSandwich/index.html
• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological bulletin, 86(3), 638. https://doi.org/10.1037/0033-2909.86.3.638
• Rosenthal, R. (1991). Meta-Analytic Procedures for Social Research. SAGE Publications, Inc. https://doi.org/10.4135/9781412984997
• Rosenthal, R., & Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99(3), 400-406. https://doi.org/10.1037/0033-2909.99.3.400
• RStudio Team (2020). RStudio: Integrated Development for R. RStudio, PBC, Boston, MA URL http://www.rstudio.com/
• Scammacca, N., Roberts, G., & Stuebing, K. K. (2014). Meta-Analysis With Complex Research Designs: Dealing With Dependence From Multiple Measures and Multiple Group Comparisons. Review of Educational Research, 84(3), 328-364. https://doi.org/10.3102/0034654313500826
• Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., Moher, D., Becker, B. J., Sipe, T. A., & Thacker, S. B. (2000). Meta-analysis of observational studies in epidemiology: A proposal for reporting. Jama, 283(15), 2008-2012. https://doi.org/10.1001/jama.283.15.2008
• Şen, S., & Akbaş, N. (2016). Çok Düzeyli Meta-Analiz Yöntemleri Üzerine Bir Çalışma. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 7(1). https://doi.org/10.21031/epod.29995
• Şen, S., & Yıldırım, İ. (2020). CMA ile meta-analiz uygulamaları. Ankara: ANI Yayıncılık.
• Van Den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576-594. https://doi.org/10.3758/s13428-012-0261-6
• Viechtbauer, W. (2010). Conducting Meta-Analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1-48. https://doi.org/10.18637/jss.v036.i03