A Guide to Reliability Generalization Meta-Analyses: The REGEMA Checklist and Flowchart

Abstract

Reliability Generalization (RG) is a specific form of meta-analysis that examines how reliability estimates derived from test scores differ when the same measurement instrument is applied to different samples. Although various reporting guidelines such as PRISMA, AMSTAR, MOOSE, and MARS were developed for meta-analyses examining intervention effects or relationships between variables to aid in meta-analysis reporting, they are also frequently used for reporting reliability generalization meta-analyses. The Reliability Generalization Meta-Analysis (REGEMA) checklist and flowchart is a tool designed for the proper development and reporting of reliability generalization meta-analyses. REGEMA seems to be the only checklist specifically designed to guide the reporting of reliability generalization meta-analyses. In this adaptation study, the aim is to introduce the REGEMA Checklist and Flowchart to researchers in Türkiye and discuss its content. The study includes the REGEMA Checklist and corresponding explanations, and the Flowchart is presented and illustrated with examples. It is hoped that through this study, researchers in Türkiye can easily access and make use of REGEMA in their work. Researchers conducting meta-analyses can use REGEMA to follow the process and prevent any shortcomings in their reporting.

Keywords

• Reliability
• Reliability Generalization
• Meta-Analysis
• REGEMA

Güvenirlik Genelleme Meta-Analizlerine Bir Rehber: REGEMA Kontrol Listesi ve Akış Şeması

Abstract

Güvenirlik genellemesi [Reliability Generalization (RG)], bir testin farklı örneklemlere uygulandığında test puanlarına dayalı güvenirlik tahminlerinin nasıl değiştiğini araştırmayı amaçlayan özel bir meta-analiz türüdür. Meta-analiz raporlamasına yardımcı olmak amacıyla PRISMA, AMSTAR, MOOSE, MARS gibi çeşitli raporlama yönergeleri müdahale etkileri veya değişkenler arasındaki ilişkileri inceleyen meta-analizler için geliştirilmiş olmasına rağmen güvenirlik genelleme meta-analizleri için de sıklıkla raporlamada tercih edilmektedir. Güvenirlik Genellemesi Meta-Analizi [Reliability Generalization Meta-Analysis (REGEMA)] kontrol listesi ve akış şeması, güvenirlik genelleme meta-analizlerinin doğru bir şekilde geliştirilmesi ve raporlanması için tasarlanmış bir araçtır. Güvenirlik genellemesi meta-analizlerinin nasıl raporlanacağına ilişkin özel olarak odaklanan bir kontrol listesi sadece REGEMA’ya rastlanmıştır. Bu bağlamda temel araştırma türünde oluşturulan çalışmada REGEMA Kontrol Listesi ve Akış Şeması’nın Türkiye’deki araştırmacılara sunulması ve içeriğinin tartışılması amaçlanmıştır. Çalışmada REGEMA Kontrol Listesi’ne ve açıklamalarına yer verilmiş olup, Akış Şeması sunulup örnekler ile açıklanmıştır. Bu çalışma aracılığıyla Türkiye’deki araştırmacıların REGEMA’ya kolay bir şekilde erişerek çalışmalarında yararlanabilecekleri düşünülmektedir. REGEMA, meta-analiz uygulayan araştırmacılar tarafından süreci takip etmeleri ve raporlamalarında oluşacak herhangi bir eksikliğin önüne geçmeleri için kullanılabilir.

Keywords

• Güvenirlik
• Güvenirlik Genellemesi
• Meta-Analiz
• REGEMA

References

1. Aguinis, H., Gottfredson, R. K., & Joo, H. (2013). Best-practice recommendations for defining, identifying, and handling outliers. Organizational research methods, 16(2), 270-301. https://doi.org/10.1177/1094428112470848 American Psychological Association. Publications and Communications Board Working Group on Journal Article Reporting Standards (2008). Reporting standards for research in psychology: Why do we need them? What might they be. American Psychologist, 63(9), 839-851. https://doi.org/10.1037/0003-066X.63.9.839 Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3-25.
2. Badenes-Ribera, L., Duro-García, C., López-Ibáñez, C., Martí-Vilar, M., & Sánchez-Meca, J. (2023). The adult prosocialness behavior scale: A reliability generalization meta-analysis. International Journal of Behavioral Development, 47(1), 59-71. https://doi.org/10.1177/01650254221128280
3. Basu, A. (2017). How to conduct meta-analysis: A basic tutorial. PeerJ Preprints, 5, e2978v1. https://doi.org/10.7287/peerj.preprints.2978v1 Bernerth, J. B., & Aguinis, H. (2016). A critical review and best‐practice recommendations for control variable usage. Personnel Psychology, 69(1), 229-283. https://doi.org/10.1111/peps.12103
4. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley.
5. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2010). A basic introduction to fixed‐effect and random‐effects models for meta‐analysis. Research Synthesis Methods, 1(2), 97-111. https://doi.org/10.1002/jrsm.12
6. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Sage. https://doi.org/10.4135/9781412985642
7. Cheung, M. W. L., & Vijayakumar, R. (2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26, 121-128. https://doi.org/10.1007/s11065-016-9319-z
8. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Harcourt Brace Jovanovich.

9. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.
10. Deng, J., Wang, M. C., Zhang, X., Shou, Y., Gao, Y., & Luo, J. (2019). The inventory of callous unemotional traits: A reliability generalization meta-analysis. Psychological Assessment, 31(6), 765. https://doi.org/10.1037/pas0000698
11. Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research. Management Review Quarterly, 68, 103-106. https://doi.org/10.1007/s11301-018-0142-x
12. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. https://doi.org/10.3102/0013189X005010003
13. Greenland, S., & Lash, T. L. (2008). Modern epidemiology. Lippincott Williams & Wilkins. https://doi.org/10.1016/j.annemergmed.2008.06.461
14. Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181-217. https://doi.org/10.1002/jrsm.1378
15. Hansen, C., Steinmetz, H., & Block, J. (2022). How to conduct a meta-analysis in eight steps: a practical guide. Management Review Quarterly, 1-19. https://doi.org/10.1007/s11301-021-00247-4
16. Ho, R. (2014). Univariate and multivariate data analysis with IBM SPSS. CRC Press.
17. Hoon, C. (2013). Meta-synthesis of qualitative case studies: An approach to theory building. Organizational Research Methods, 16(4), 522-556. https://doi.org/10.1177/1094428113484969
18. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Sage.
19. Hür, G. (2021). PRISMA kontrol listesi 2020 güncellemesi. Online Turkish Journal of Health Sciences, 6(4), 603-605. https://doi.org/10.26453/otjhs.1001606 Ioannidis, J. P., Trikalinos, T. A., Ntzani, E. E., & Contopoulos-Ioannidis, D. G. (2003). Genetic associations in large versus small studies: An empirical assessment. The Lancet, 361(9357), 567-571. https://doi.org/10.1016/S0140-6736(03)12516-0
20. Lau, J., Ioannidis, J. P., & Schmid, C. H. (1997). Quantitative synthesis in systematic reviews. Annals of Internal Medicine, 127(9), 820-826. https://doi.org/10.7326/0003-4819-127-9-199711010-00008
21. McMillan, J. H. & Schumacher, S. (2001). Research in education: A conceptual introduction. Longman.
22. Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., ... & Prisma-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4, 1-9. https://doi.org/10.1186/2046-4053-4-1
23. Mor, E. (2024). Ölçme süreç ve sonuçlarının taşıması gereken temel nitelikler. In Ö. Yıldırım & K. S. Kartal (Eds.), Eğitimde ölçme ve değerlendirme (pp. 22-62). Lisans Yayıncılık.
24. Mosteller, F., & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review of Public Health, 17(1), 1-23. https://doi.org/10.1146/annurev.pu.17.050196.000245
25. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., ... & Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422-1425. https://doi.org/10.1126/science.aab2374
26. Osborne, J. W., & Fitzpatrick, D. C. (2012). Replication analysis in exploratory factor analysis: What it is and why it makes your analysis better. Practical Assessment, Research, and Evaluation, 17(15), 1-8.
27. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372. https://doi.org/10.1136/bmj.n71
28. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71.
29. Paul, J., & Criado, A. R. (2020). The art of writing literature review: What do we know and what do we need to know?. International Business Review, 29(4), 101717. https://doi.org/10.1016/j.ibusrev.2020.101717
30. Rothstein, H. R., Sutton, A.J., & Borenstein, M. (Eds.) (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Wiley. https://doi.org/10.1002/0470870168
31. Rudolph, C. W., Chang, C. K., Rauvola, R. S., & Zacher, H. (2020). Meta-analysis in vocational behavior: A systematic review and recommendations for best practices. Journal of Vocational Behavior, 118, 103397. https://doi.org/10.1016/j.jvb.2020.103397
32. Sánchez-Meca, J., Marín-Martínez, F., Núñez-Núñez, R. M., Rubio-Aparicio, M., López-López, J. A., & López-García, J. J. (2019, Temmuz). Reporting practices in reliability generalization meta-analyses: assessment with the REGEMA checklist. XVI Congress of Methodology of the Social and Health Sciences. Madrid, Spain.
33. Sánchez‐Meca, J., Marín‐Martínez, F., López‐López, J. A., Núñez‐Núñez, R. M., Rubio‐Aparicio, M., López‐García, J. J., ... & López‐Nicolás, R. (2021). Improving the reporting quality of reliability generalization meta‐analyses: The REGEMA checklist. Research Synthesis Methods, 12(4), 516-536. https://doi.org/10.1002/jrsm.1487
34. Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., ... & Bouter, L. M. (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(1), 10.
35. Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., ... & Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013-1020. https://doi.org/10.1016/j.jclinepi.2008.10.009
36. Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., ... & Thacker, S. B. (2000). Meta-analysis of observational studies in epidemiology: A proposal for reporting. Jama, 283(15), 2008-2012. https://doi.org/10.1001/jama.283.15.2008
37. Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., ... & Thacker, S. B. (2000). Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA, 283(15), 2008-2012.
38. Thompson, B. (2003). Score reliability: Contemporary thinking on reliability issues. Sage.
39. Traub, R. E. (1994). Reliability for the social sciences: Theory and applications. Sage Publications.
40. Tuttle, B. D., Isenburg, M. V., Schardt, C., & Powers, A. (2009). PubMed instruction for medical students: Searching for a better way. Medical Reference Services Quarterly, 28(3), 199-210. https://doi.org/10.1080/02763860903069839
41. Vacha Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58(1), 6-20. https://doi.org/10.1177/0013164498058001002
42. Whitehead, A. (2002). Meta-analysis of controlled clinical trials. John Wiley & Sons. https://doi.org/10.1002/0470854200
43. Wiernik, B. M., & Dahlke, J. A. (2020). Obtaining unbiased results in meta-analysis: The importance of correcting for statistical artifacts. Advances in Methods and Practices in Psychological Science, 3(1), 94–123. https://doi.org/10.1177/2515245919885611
44. Yörük, S., & Sen, S. (2023). A reliability generalization meta-analysis of the creative achievement questionnaire. Creativity Research Journal, 35(4), 714-729. https://doi.org/10.1080/10400419.2022.2148073