The Reporting of Effect Size in Educational Research

Abstract

The purpose of this study is to investigate effect size reporting and interpreting practices in Turkish academic journals in the field of education which are indexed in Social Sciences Citation Index. For this purpose, 990 articles published in four journals between the years 2007-2011 were selected for investigation. Among these articles, 480 of them were included in the study in accordance with reportable effect sizes. The results of the study showed that only in 35 of the articles (i. e. 7.2%) researchers reported the effect size values obtained. Besides, researchers interpreted the effect size only in 21 of them. The results of the study also revealed that there are no significant differences among journals with respect to effect size reporting and interpreting χ (3, n = 483) = 7. 30, p = .06, Cramer’s V = .12.

Keywords

• Education, research, effect size, practical significance, statistical significance.

Eğitim Araştırmalarında Etki Büyüklüğü Raporlanması

Öz

Bu araştırmanın amacı, eğitim alanında Türkiye kaynaklı yayımlanan ve SSCI’da taranan akademik bilimsel dergilerde yayımlanan makalelerde etki büyüklüğü raporlanması durumunu araştırmak ve okuyuculara etki büyüklüğü, hesaplanması ve raporlanması konularında bilgi sunmaktır. Araştırma kapsamına alınan dört dergide 2007-2011 yılları arasında yayımlanan toplam 990 makale incelenmiş; bunlardan etki büyüklüğü raporlanmaya uygun olan 480 tanesi araştırma kapsamına alınmıştır. Araştırma sonunda 480 makalenin sadece 35’inde (%7.2) etki büyüklüğü raporlandığı; bunların da 21’inde raporlanan etki büyüklüklerinin aynı zamanda yorumlandığı görülmüştür. Çalışmadan elde edilen sonuçlar incelendiğinde dergiler arasında etki büyüklüğü raporlama ve yorumlama açısından anlamlı bir farklılık olmadığı görülmüştür χ2 (3, n = 483) = 7. 30, p = .06, Cramer’s V = .12

Anahtar Kelimeler

• Eğitim, araştırma, etki büyüklüğü, pratik anlamlılık, istatistiksel anlamlılık.

Kaynakça

1. Alhija, F. N. & Levy, A. (2007, April). Effect size reporting practices in published articles. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
2. American Educational Research Association. (2006). Standards on reporting on emperical social science research in AERA publications. Educational Researcher, 35, 33-40.
3. American Psychological Association. (2001). Publication manual of the American Pyschological Association (5th ed.). Washington, DC: Author.
4. Balcı, A. (2009). Sosyal Bilimlerde Araştırma: Yöntem, Teknik ve İlkeler. Ankara: Pegem Akademi Yay. Brewerton, P. & Millward, L. (2001). Organizational Research Methods: A Guide for Students and Researchers, Sage Publications, London, GBR.
5. Cohen, J. (1988). Statistical power analysis fort he behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Cohen, J. (1990). Things I have learned so far. American Psychologist, 45, 1304-1312.
6. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
7. Davis, J. A. (1971). Elementary survey analysis. Englewood Cliffs, NJ: Prentice-Hall.
8. Daymon, C. & Holloway, I. (2002). Qualitative Research Methods in Public Relations and Marketing Communications, Routledge, Florence, KY, USA.

9. Dunleavy, E. M., Barr, C. D., Glenn, D. M., & Miller, K. R. (2006). Effect size reporting in applied psychology: How are we doing? The Industrial-Organizational Psychologist, 43, 29-37.
10. Falk, R. & Greenbaum, C. W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
11. Fan, X. (2001). Statistical significance and effect size in education research: Two sides of a coin. Journal of Educational Research, 94, 275-283.
12. Fraenkel, J. R. & Wallen, N. E. (2003). How to Design and Evaluate Research in Education, McGrawHill, New York.
13. Ghauri, P. & Gronhaug, K. (2002). Research Methods in Business Studies, Prentice Hall, London.
14. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
15. Hedges, L. V. (1981). Distributional theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.
16. Henson, R. K. (2006). Effect size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34,601-629.
17. Hinkle, D. E., Wiersma, W., & Jurs, S. G. (1979). Applied statistics fort he behavioral sciences. Chicago: Rand McNally College Publishing.
18. Hopkins, W. G: (1997). New view of statistics. Retrieved, September 12, 2011 from http://www.sportsci.org/resource/stats/effectmag.html
19. Huberty, C. J., & Pike, C. J. (1999). On some history regarding statistical testing. In B. Thompson (Ed.), Advances in social science methodology (Vol. 5, pp.1-23). Stamford, CT: JAI Press.
20. Hutchins, H, M., & Henson, R. K. (2002). In search of OZ: Effect size reporting and interpretation in communication research. Paper presented at the annual meting of the Southwest Educational Research Association, Austin, TX.
21. Ives, B. (2003). Effect size use in studies of learning disabilities. Journal of Learning Disabilities, 36,490-504.
22. İşman, A. Baytekin, Ç., Balkan, F., Horzum, B. & Kıyıcı, M. (2002). Fen Bilgisi Eğitimi ve Yapısalcı Yaklaşım. The Turkish Online Journal of Educational Technology, 1(1), 41-47.
23. Kabaca, T. ve Erdoğan, Y. (2007). Fen Bilimleri, Bilgisayar ve Matematik Eğitimi Alanlarındaki Tez Çalışmalarının İstatistiksel Açıdan İncelenmesi. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 22(2).
24. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
25. Kline, R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
26. Kotrlik, J. W. & Williams, H. A. (2003). The incorporation of effect size in information technology, learning, and performance research. Information Technology, Learning, and Performance Journal, 21(1), 1-7.
27. Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
28. Meline, T., & Schmitt, J. F. (1997). Case studies for evaluating statistical significance in group designs. American Journal of Speech-Language Pathology, 6, 33-41.
29. Meline, T., & Wang, B. (2004). Effect-size reporting practices in AJSLP and other ASHA journals, 1999200 American Journal of Speech Language Pathology, 13,202-207.
30. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301.
31. Olejnik, S. F. (1984). Planning educational research: Determining the necessary sample size. Journal of Experimental Education, 53, 40-48.
32. Ottenbacher, K. J., & Barrett, K. A. (1989). Measures of effect size in the reporting of rehabilation research. American Journal of Physical Medicine & Rehabilitation, 68, 52-58.
33. Özsoy, S., Keleş Ö. & Uzun, N. (2011). Methodological and Statistical Errors Found in Science Education Master’s Theses. The International Journal of Educational Researchers, 2(5), 35-46.
34. Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34, 1189-1208.
35. Paul, K. M., & Plucker, J. A. (2003). Two steps forward, one step back: Effect size reporting in gifted education research from 1995-2000. Roeper Review, 26, 68-72.
36. Plucker, J. A. (1997). Debunking the myth of the “highly significant” result: Effect sizes in gifted education research. Roeper Review, 2, 122-126.
37. Polonsky, M. J. & Waller, D. S. (2005). Designing and Managing a Research Project,Sage Publications, U.S.A.
38. Robinson, D. H., & Levin, J. R. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26, 21-27.
39. Sayın, S. (2008). Bilimsel araştırmalarda yapılan bazı istatistiksel ve yöntembilimsel Hatalar-III: Güvenirlik Kestirimlerine Yönelik Hatalar . Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi, 15.
40. Snyder, P. A. & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 331-349.
41. Snyder, P. A., & Thompson, B. (1998). Use of tests of statistical significance and other analytic choices in a school psychology journal. School Psychology Quarterly, 13, 335-348.
42. Spiegel, M. R. & Stephens, L. J. (1999). Schaum’s outline of theory and problems of statistics. 3rd edition. Schaum’s outline series, McGraw-Hill, New York.
43. Sönmez, V. (2005). Bilimsel Araştırmalarda Yapılan Yanlışlıklar. Eurasian Journal of Educational Research, 18, 150-170.
44. Sun, S., Pan W., & Wang, L. L. (2010). A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.Journal of Educational Psychology.Advance online publication. doi: 10.1037/a0019507
45. Thompson, B. (2008). Computing and interpreting effect sizes, confidence intervals, and confidence intervals for effect sizes. In J. W. Osborne (Ed.), Best practices in quantitative methods (pp. 246262). Thousand Oaks, CA: Sage.
46. Thompson, B. (2002). “Statistical”, “practical”, and “clinical”: How many kinds of significance do counselors need to consider? Journal of Counseling & Development, 80, 64-71.
47. Thompson, B. (1999). If statistical significance tests are broken/misused, what practices should supplement or replace them? Theory and Psychology, 82, 165-181. Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25, 26-30.
48. Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
49. Thompson, B., & Snyder, P. A. (1997). Statistical significance testing practices in the Journal of Experimental Education. Journal of Experimental Education, 66, 75-83.
50. Tonta, Y. (1999). Bilimsel Araştırmalarda İstatistik Tekniklerin Kullanımı ve Bulguların Sunumu Üzerine. Türk Kütüphaneciliği, 13 (2), 112-124.
51. Toy, Y. B., ve Tosunoğlu, G. N. (2007). Sosyal Bilimler Alanındaki Araştırmalarda Bilimsel Araştırma Süreci, İstatistiksel Teknikler ve Yapılan Hatalar, Ticaret ve Turizm Egitim Fakültesi Dergisi, Sayı: Vache-Haase, T., & Ness, C. M. (1999). Statistical significance testing as it relates to practice: Use within professional psychology: Research and practice. Professional Psychology: Research and Practice, 30, 104-105.
52. Vacha-Haase, T. & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.
53. Yates, F. (1951). The influence of “statistical methods for research workers” on the development of science of statistics. Journal of the American Statistical Association, 46, 19-34.
54. Yıldırım, H. H. & Yıldırım, S. (2011). Hipotez testi, güven aralığı, etki büyüklüğü ve merkezi olmayan olasılık dağılımları üzerine. İlköğretim Online, 10(3), 1112-1123.