Statistical power and precision of experimental studies originated in the Republic of Turkey from 2010 to 2020: Current practices and some recommendations

Year 2021, Volume: 8 Issue: 4, 24 - 43, 01.12.2021

Metin Bulus , İlhan Koyuncu

https://doi.org/10.17275/per.21.77.8.4

Cited By: 6

Abstract

This study systematically reviews randomly selected 155 experimental studies in education field originated in the Republic of Turkey between 2010 and 2020. Indiscriminate choice of sample size in recent publications prompted us to evaluate their statistical power and precision. First, above and beyond our review, we could not identify any large-scale experiments such as cluster-randomized or multisite randomized trials, which overcome shortcomings of small-scale experiments, better suit to the organizational structure of the education field, nevertheless require far greater effort and financial resources. Second, none of the small-scale experiments has reported or conducted ex-ante power analysis. Third, results indicate that studies are sufficiently powered to detect medium effects and above (Cohen’s d ≥ 0.50), however they are underpowered to detect small effects (Cohen’s d ≤ 0.20). Trends in the past ten years indicate precision remained unchanged. We made several recommendations to increase the precision of experimental designs and improve their evidential values: Determine sample size prior to an experiment with power analysis routine, randomize subjects / clusters to obtain unbiased estimates, collect pre-test information and other relevant covariates, adjust for baseline differences beyond covariate control, document attrition, report standardized treatment effect and standardized variance parameters. Findings should be interpreted considering minimum effects in education that are relevant to education policy and practice.

Keywords

experimental design, systematic review, minimum detectable effect size, statistical power, precision

References

• Arıcı, S., & Aslan-Tutak, F. (2015). The effect of origami-based instruction on spatial visualization, geometry achievement, and geometric reasoning. International Journal of Science and Mathematics Education, 13(1), 179-200. https://doi.org/10.1007/s10763-013-9487-8
• Arsal, Z. (2014). Microteaching and pre-service teachers’ sense of self-efficacy in teaching. European Journal of Teacher Education, 37(4), 453-464. https://doi.org/10.1080/02619768.2014.912627
• Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Handbook I. Cognitive domain. New York, NY: David McKay.
• Bloom, H. S. (1995). Minimum detectable effects a simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547-556. https://doi.org/10.1177/0193841X9501900504
• Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom (Ed.), Learning more from social experiments evolving analytic approaches (pp. 115–172). New York, NY: Russell Sage.
• Bloom, H. S. (2006). The core analytics of randomized experiments for social research. MDRC Working Papers on Research Methodology. New York, NY: MDRC. Retrieved from https://www.mdrc.org/sites/default/files/full_533.pdf
• Bloom, H. S., Bos, J. M., & Lee, S. W. (1999). Using cluster random assignment to measure program impacts: Statistical Implications for the evaluation of education programs. Evaluation Review, 23(4), 445–469. https://doi.org/10.1177%2F0193841X9902300405
• Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289-328. https://doi.org/10.1080/19345740802400072
• Boruch, R. F. (2005). Better evaluation for evidence based policy: Place randomized trials in education, criminology, welfare, and health. The Annals of American Academy of Political and Social Science, 599. https://doi.org/10.1177%2F0002716205275610
• Boruch, R. F., DeMoya, D., & Snyder, B. (2002). The importance of randomized field trials in education and related areas. In F. Mosteller & R. F. Boruch (Eds.), Evidence matters: Randomized fields trials in education research (pp. 50–79). Washington, DC: Brookings Institution Press.
• Boruch, R. F. & Foley, E. (2000). The honestly experimental society. In L. Bickman (Ed.), Validity and social experiments: Donald Campbell’s legacy (pp. 193–239). Thousand Oaks, CA: Sage.
• Bulus, M., Dong, N., Kelcey, B., & Spybrook, J. (2019). PowerUpR: Power analysis tools for multilevel randomized experiments. R package version 1.0.4. https://CRAN.R-project.org/package=PowerUpR
• Cengiz, E. (2020). A thematic content analysis of the qualitative studies on FATIH Project in Turkey. Journal of Theoretical Educational Science, 13(1), 251-276. https://doi.org/10.30831/akukeg.565421
• Cohen, J. (1973). Eta-squared and partial eta-squared in fixed factor ANOVA designs. Educational and psychological measurement, 33(1), 107-112. https://doi.org/10.1177%2F001316447303300111
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
• Cook, T. D. (2002). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24, 175–199. https://doi.org/10.3102%2F01623737024003175
• Cook, T. D. (2005). Emergent principles for the design, implementation, and analysis of cluster-based experiments in social science. The Annals of American Academy of Political and Social Science, 599. https://doi.org/10.1177%2F0002716205275738
• Cox, K., & Kelcey, B. (2019a). Optimal sample allocation in group-randomized mediation studies with a group-level mediator. The Journal of Experimental Education, 87(4), 616-640. https://doi.org/10.1080/00220973.2018.1496060
• Cox, K., & Kelcey, B. (2019b). Optimal design of cluster- and multisite-randomized studies using fallible outcome measures. Evaluation Review, 43(3-4), 189-225. https://doi.org/10.1177%2F0193841X19870878
• Çelik, H. C. (2018). The effects of activity based learning on sixth grade students’ achievement and attitudes towards mathematics activities. EURASIA Journal of Mathematics, Science and Technology Education, 14(5), 1963-1977. https://doi.org/10.29333/ejmste/85807
• Diken, İ. H., Cavkaytar, A., Abakay, A. M., Bozkurt, F., & Kurtılmaz, Y. (2011). Effectiveness of the Turkish version of' 'First Step to Success program'' in preventing antisocial behaviors. Education and Science, 36(161), 145-158. https://hdl.handle.net/11421/15128
• Dong, N., Kelcey, B., & Spybrook, J. (2017). Power analyses for moderator effects in three-level cluster randomized trials. The Journal of Experimental Education, 1-26. https://doi.org/10.1080/00220973.2017.1315714
• Dong, N., & Maynard, R. (2013). PowerUp!: A Tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24-67. https://doi.org/10.1080/19345747.2012.673143
• Fraenkel, J. R., Wallen, N. E., & Hyun, H. (2011). How to design and evaluate research in education (10th Ed.). New York, NY: McGraw-Hill.
• Göksün, D. O., & Gürsoy, G. (2019). Comparing success and engagement in gamified learning experiences via Kahoot and Quizizz. Computers & Education, 135, 15-29. https://doi.org/10.1016/j.compedu.2019.02.015
• Hedges, L. V., & Hedberg, E. C. (2013). Intraclass correlations and covariate outcome correlations for planning two-and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445-489. https://doi.org/10.1177/0193841X14529126
• Hedges, L. V., & Rhoads, C. (2010). Statistical power analysis in education research (NCSER 2010-3006). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. https://files.eric.ed.gov/fulltext/ED509387.pdf
• Hedges, L. V., & Vevea, J. L. (2005). Selection method approaches. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 145–174). Chichester, UK: Wiley.
• Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172-177. https://doi.org/10.1111/j.1750-8606.2008.00061.x
• Karaömerlioglu, M. A. (1998). The village institutes experience in Turkey. British Journal of Middle Eastern Studies, 25(1), 47-73. https://doi.org/10.1080/13530199808705654
• Kelcey B, Dong, N, Spybrook J, Cox K (2017a). Statistical power for causally defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42(5), 499–530. https://doi.org/10.3102/1076998617695506
• Kelcey B, Dong, N, Spybrook J, Shen Z (2017b). Experimental power for indirect effects in group-randomized studies with group-level mediators. Multivariate Behavioral Research, 52(6), 699–719. https://doi.org/10.1080/00273171.2017.1356212
• Kennedy, J. J. (1970). The eta coefficient in complex ANOVA designs. Educational and Psychological Measurement, 30(4), 885-889. https://doi.org/10.1177%2F001316447003000409
• Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: Handbook 2: Affective domain. New York, NY: David McKay.
• Konstantopoulos, S. (2009). Incorporating cost in power analysis for three-level cluster-randomized designs. Evaluation Review, 33(4), 335-357. https://doi.org/10.1177/0193841X09337991
• Konstantopoulos, S. (2011). Optimal sampling of units in three-level cluster randomized designs: An ANCOVA framework. Educational and Psychological Measurement, 71(5), 798-813. https://doi.org/10.1177/0013164410397186
• Konstantopoulos, S. (2013). Optimal design in three-level block randomized designs with two levels of nesting: An ANOVA framework with random effects. Educational and Psychological Measurement, 73(5), 784-802. https://doi.org/10.1177/0013164413485752
• Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863
• Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612-625. https://doi.org/10.1111/j.1468-2958.2002.tb00828.x
• Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.
• Moerbeek, M., & Safarkhani, M. (2018). The design of cluster randomized trials with random cross-classifications. Journal of Educational and Behavioral Statistics, 43(2), 159-181. https://doi.org/10.3102/1076998617730303
• Mosteller, F., & Boruch, R. F. (2002). Evidence matters: Randomized trials in education research. Washington, DC: Brookings Institution Press.
• Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. Oxford, UK: Blackwell.
• Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173. https://doi.org/10.1037/1082-989X.2.2.173
• Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite trials. Psychological Methods, 5(2), 199-213. https://doi.org/10.1037/1082-989X.5.2.199
• Rhoads, C. H. (2011). The implications of “contamination” for experimental design in education. Journal of Educational and Behavioral Statistics, 36(1), 76-104. https://doi.org/10.3102%2F1076998610379133
• Rickles, J., Zeiser, K., & West, B. (2018). Accounting for student attrition in power calculations: Benchmarks and guidance. Journal of Research on Educational Effectiveness, 11(4), 622-644. https://doi.org/10.1080/19345747.2018.1502384
• Sadi, Ö., & Cakiroglu, J. (2011). Effects of hands-on activity enriched instruction on students' achievement and attitudes towards science. Journal of Baltic Science Education, 10(2), 87-97. http://oaji.net/articles/2014/987-1410008481.pdf
• Slavin, R. E. (2008). Perspectives on evidence-based research in education: What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5-14. https://doi.org/10.3102%2F0013189X08314117
• Spybrook, J. (2008). Are power analyses reported with adequate detail? Evidence from the first wave of group randomized trials funded by the Institute of Education Sciences. Journal of Research on Educational Effectiveness, 1(3), 215-235. https://doi.org/10.1080/19345740802114616
• Spybrook, J., Congdon, R., Hill, C., Martinez, A., & Raudenbush, S. W. (2011). Optimal design plus empirical evidence: Documentation for the “Optimal Design” software (Version 3.0) [Software]. http://hlmsoft.net/od/
• Spybrook, J., Kelcey, B., & Dong, N. (2016). Power for detecting treatment by moderator effects in two- and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605-627. https://doi.org/10.3102/1076998616655442
• Spybrook, J., Puente, A. C., & Lininger, M. (2013). From planning to implementation: An examination of changes in the research design, sample size, and precision of group randomized trials launched by the Institute of Education Sciences. Journal of Research on Educational Effectiveness, 6(4), 396-420. https://doi.org/10.1080/19345747.2013.801544
• Spybrook, J., & Raudenbush, S. W. (2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298-318. https://doi.org/10.3102%2F0162373709339524
• Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the US Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255-267. https://doi.org/10.1080/1743727X.2016.1150454
• Spybrook, J., Westine, C. D., & Taylor, J. A. (2016). Design parameters for impact research in science education: A multistate analysis. AERA Open, 2(1). https://doi.org/10.1177/2332858415625975
• Stone, F. A. (1974). Rural revitalization and the Village Institutes in Turkey: Sponsors and critics. Comparative Education Review, 18(3), 419-429. https://doi.org/10.1086/445797
• Tok, Ş. (2013). Effects of the know-want-learn strategy on students’ mathematics achievement, anxiety and metacognitive skills. Metacognition and Learning, 8(2), 193-212. https://doi.org/10.1007/s11409-013-9101-z
• Vexliard, A., & Aytaç, K. (1964). The" Village Institutes" in Turkey. Comparative Education Review, 8(1), 41-47. https://doi.org/10.1086/445031