The Methodological Quality of Experimental STEM Education Articles Published in Scholarly Journals from 2014 to 2020

Öz

Experimental studies have a considerable impact on the educational policies and practices of many countries. In Turkey, policymakers are planning to initiate a STEM education reform in K-12 schools based on experimental studies. However, the methodological flaws in these studies may lead to biased outcomes and may mislead the STEM education community. Despite the importance of methodological quality, to the best of our knowledge, there are no studies that investigate the methodological quality of experimental STEM education articles published in scholarly journals. Therefore, in this study, we conducted a methodological review to examine the methodological quality of experimental STEM education articles published in refereed Turkish journals from 2014 to 2020. During the targeted period, we located 68 articles. We analyzed these articles by developing a coding framework. We found that the selected articles suffer seriously from various methodological flaws. We discuss the findings in light of the literature on methodological quality and suggest ways to improve the rigor of the experimental designs used. Ultimately, we discuss some implications for authors, journals editors, policymakers, and curriculum developers.

Anahtar Kelimeler

• STEM education
• Methodological quality
• Experimental research
• Scholarly journal articles

References

1. Ahern, K.J. (1999). Ten tips for reflexive bracketing. Qualitative Health Research, 9(3), 407–411. https://doi.org/10.1177/104973239900900309
2. Akgündüz, D., Aydeniz, M., Çakmakçı, G., Çavaş, B., Çorlu, M.S., Öner, T., & Özdemir, S. (2015). STEM eğitimi Türkiye raporu: Günün modası mı yoksa gereksinim mi? [A report on STEM education in Turkey: A provisional agenda or a necessity?]. Scala Press.
3. Aktemur, Ş. (2015). Review of aviation research: A content analysis of articles published in the Collegiate Aviation Review, 2007–2012 [Unpublished master’s thesis]. Florida Institute of Technology.
4. Aron, A., Coups, E.J., & Aron, E.N. (2019). Statistics for the behavioral and social sciences: A brief course (6th ed.). Pearson.
5. Ary, D., Jacobs, L.C., Sorensen, C.K., & Walker, D. (2014). Introduction to research in education (9th ed.). Wadsworth Cengage Learning.
6. Aydın Günbatar, S., & Tabar, V. (2019). Türkiye’de gerçekleştirilen STEM araştırmalarının içerik analizi [Content analysis of Science, Technology, Engineering and Mathematics (STEM) research conducted in Turkey]. Yüzüncü Yıl University Journal of Education, 16(1), 1054–1083. http://dx.doi.org/10.23891/efdyyu.2019.153
7. Baydaş, Ö., Küçük, S., Yılmaz, R. M., Aydemir, M., & Göktaş, Y. (2015). Educational technology research trends from 2002 to 2014. Scientometrics, 105, 709–725. https://doi.org/10.1007/s11192-015-1693-4
8. Borman, G.D., Slavin, R.E., Cheung, A., Chamberlain, A.M., Madden, N.A., & Chambers, B. (2005). Success for all: First-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27(1), 1 22. https://doi.org/10.3102/01623737027001001

9. Brown, J. (2012). The current status of STEM education research. Journal of STEM Education: Innovations and Research, 13(5), 7–11.
10. Brown, J.R., & Dant, R.P. (2008). On what makes a significant contribution to the retailing literature. Journal of Retailing, 84(2), 131 135. https://doi.org/10.1016/j.jretai.2008.05.002
11. Campbell, D.T., & Boruch, R.F. (1975). Making the case for randomized assignment to treatments by considering the alternatives: Six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects. In C. A. Bennett & A. A. Lumsdaine (Eds.), Evaluation and experiment: Some critical issues in assessing social programs (pp. 195–296). Academic Press.
12. Çavaş, P., Ayar, A., Bula Turuplu, S., & Gürcan, G. (2020). Türkiye’de STEM eğitimi üzerine yapılan araştırmaların durumu üzerine bir çalışma [A study on the status of STEM education research in Turkey]. Yüzüncü Yıl University Journal of Education, 17(1), 823–854. https://doi.org/10.33711/yyuefd.751853
13. Çevik, M. (2018). Impacts of the project based (PBL) science, technology, engineering and mathematics (STEM) education on academic achievement and career interests of vocational high school students. Pegem Journal of Education and Instruction, 8(2), 281–306. http://dx.doi.org/10.14527/pegegog.2018.012
14. Cheung, C.K., & Slavin, R.E. (2012). How features of educational technology applications affect student reading outcomes: A meta-analysis. Educational Research Review, 7(3), 198–215. https://doi.org/10.1016/j.edurev.2012.05.002
15. Coe, R.J. (2021). Effect size. In J. Arthur, M. Waring, R. Coe, & L. V. Hedges (Eds.), Research methods and methodologies in education (3rd ed., pp. 368–377). Sage.
16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
17. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. https://doi.org/10.1037/0003-066X.49.12.997
18. Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8th ed.). Routledge.
19. Creswell, J.W. (2015). Educational research: Planning, conducting and evaluating quantitative and qualitative research (5th ed.). Pearson Education.
20. Creswell, J.W., & Creswell, J.D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage.
21. D’Agostino, J. (2005). Measuring learning outcomes: Reliability and validity issues. In G. D. Phye, D. H. Robinson, & J. Levin (Eds.). Empirical methods for evaluating educational interventions (pp. 113–145). Elsevier Academic Press.
22. Daşdemir, İ., Cengiz, E., & Aksoy, G. (2018). Türkiye’de FeTeMM (STEM) eğitimi eğilim araştırması [An investigation of research trends in the field of STEM education in Turkey]. Yüzüncü Yıl University Journal of Education, 15(1), 1161–1183. http://dx.doi.org/10.23891/efdyyu.2018.100
23. Dochy, F. (2006). A guide for writing scholarly articles or reviews for the Educational Research Review. Retrieved March 15, 2021, from https://www.elsevier.com/__data/promis_misc/edurevReviewPaperWriting.pdf
24. Duman, G., Orhon, G., & Gedik, N. (2015). Research trends in mobile assisted language learning from 2000 to 2012. ReCALL, 27(2), 197 216. https://doi.org/10.1017/S0958344014000287
25. Elmalı, Ş., & Balkan Kıyıcı, F. (2017). Türkiye’de yayınlanmış FeTeMM eğitimi ile ilgili çalışmaların incelenmesi [Review of STEM studies published in Turkey]. Sakarya University Journal of Education, 7(3), 684–696. https://doi.org/10.19126/suje.322791
26. Erdoğan, M., Marcinkowski, T., & Ok, A. (2009). Content analysis of selected features of K–8 environmental education research studies in Turkey, 1997–2007. Environmental Education Research, 15(5), 525–548. https://doi.org/10.1080/13504620903085776
27. Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
28. Finn, J.D., & Achilles, C.M. (1999). Tennessee’s class size study: Findings, implications, misconceptions. Educational Evaluation and Policy Analysis, 21, 97–109. https://doi.org/10.3102/01623737021002097
29. Fraenkel, J.R. (1987). Toward improving research in social studies education. Theory & Research in Social Education, 15(3), 203 222. https://doi.org/10.1080/00933104.1987.10505546
30. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2012). How to design and evaluate research in education (8th ed.). McGraw-Hill.
31. Frey, B.B. (2018). The SAGE encyclopedia of educational research, measurement, and evaluation. Sage.
32. Gall, J.P., Gall, M.D., & Borg, W.R. (2014). Applying educational research: How to read, do, and use research to solve problems in practice (6th ed.). Pearson Education.
33. Gall, M., Gall, J., & Borg, R. (2007). Educational research: An introduction (8th ed.). Pearson Education.
34. Gravetter, F., Wallnau, L., Forzano, L., & Witnauer, J. (2021). Essentials of statistics for the behavioral sciences (10 ed.). Cengage Learning.
35. Hair, J.F., Black, W.C., Babin, B.J., & Anderson, R.E. (2019). Multivariate data analysis (8th ed.). Cengage Learning.
36. Hanel, P.H., & Mehler, D.M. (2019). Beyond reporting statistical significance: Identifying informative effect sizes to improve scientific communication. Public Understanding of Science, 28(4), 468–485. https://doi.org/10.1177/0963662519834193
37. Hedges, L.V., & Schauer, J. (2018). Randomised trials in education in the USA. Educational Research, 60(3), 265–275. https://doi.org/10.1080/00131881.2018.1493350
38. Horton, P.B., McConney, A.A., Woods, A.L., Barry, K., Krout, H.L., & Doyle, B.K. (1993). A content analysis of research published in the Journal of Research in Science Teaching from 1985 through 1989. Journal of Research in Science Teaching, 30(8), 857–869. https://doi.org/10.1002/tea.3660300805
39. Horváth, I. (2016). Theory building in experimental design research. In P. Cash, T. Stankovic, & M. Storga (Eds.), Experimental design research: Approaches, perspectives, applications (pp. 209–231). Springer International Publishing.
40. Howell, D.C. (2017). Fundamental statistics for the behavioral sciences (9th ed.). Cengage Learning.
41. Johnson, R.B., & Christensen, L.B. (2020). Educational research: Quantitative, qualitative, and mixed approaches (7th ed.). Sage.
42. Kaya, A., & Ayar, M.C. (2020). Türkiye örnekleminde STEM eğitimi alanında yapılan çalışmaların içerik analizi [Content analysis of STEM education studies in Turkey]. İstanbul Aydın University Journal of Education, 6(2), 275–306.
43. Kirk, R.E. (2013). Experimental design: Procedures for the behavioral sciences (4th ed.). Sage Publications.
44. LaCosse, J., Ainsworth, S.E., Shepherd, M.A., Ent, M., Klein, K.M., Holland-Carter, L.A., Moss, J.H., Licht, M., & Licht, B. (2017). An active-learning approach to fostering understanding of research methods in large classes. Teaching of Psychology, 44(2), 117–123. https://doi.org/10.1177/0098628317692614
45. Li, Y., Wang, K., Xiao, Y., & Froyd, J. E. (2020). Research and trends in STEM education: A systematic review of journal publications. International Journal of STEM Education, 7, 1. https://doi.org/10.1186/s40594-020-00207-6
46. Lincoln, Y.S., & Guba, E. (1985). Naturalistic inquiry. Sage.
47. Miles, M.B., Huberman, M.A., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Sage.
48. Mills, G.E., & Gay, L.R. (2016). Educational research: Competencies for analysis and applications (11th ed.). Pearson Education.
49. Mills, G.E., & Gay, L.R. (2019). Educational research: Competencies for analysis and applications (12th ed.). Pearson.
50. Ministry of National Education. (2016). STEM eğitimi raporu [STEM education report]. Innovation and Educational Technologies General Directorate.
51. Ministry of National Education. (2018). Küresel bağlamda STEM yaklaşımları [STEM approaches in a global context]. Innovation and Educational Technologies General Directorate.
52. Mizell, S., & Brown, S. (2016). The current status of STEM education research 2013-2015. Journal of STEM Education, 17(4), 52–56.
53. Nelson, J.L., & Shaver, J.P. (1985). On research in social education. In W. B. Stanley (Ed.), Review of research in social studies education: 1976-1983 (pp. 401–433). National Council for the Social Studies.
54. Orne, M.T. (1981). The why and how of a contribution to the literature: A brief communication. International Journal of Clinical and Experimental Hypnosis, 29(1), 1–4. https://doi.org/10.1080/00207148108409137
55. Özcan, H., & Koca, E. (2019). The impact of teaching the subject “pressure” with STEM approach on the academic achievements of the secondary school 7th grade students and their attitudes towards STEM. Education and Science, 44(198), 201–227. http://dx.doi.org/10.15390/EB.2019.7902
56. Pagano, R.R. (2013). Understanding statistics in the behavioral sciences (10th ed.). Wadsworth Cengage Learning.
57. Pallant, J. (2016). SPSS survival manual: A step by step guide to data analysis using the SPSS program (6th ed.). Open University Press.
58. Plano Clark, V.L., & Creswell, J.W. (2015). Understanding research: A consumer’s guide (2nd ed.). Pearson Education.
59. Privitera, G.J. (2019). Essential statistics for the behavioral sciences (2nd ed.). Sage.
60. Randolph, J.J., Griffin, A.E., Zeiger, S.R., Falbe, K.N., Freeman, N.A., Taylor, B.E., Westbrook, A.F., Lico, C.C., Cristy, N. S., Sprull, N. M., Holt, C., Smith, K., & McAnespie, H. (2013). A methodological review of the articles published in Georgia Educational Researcher from 2003-2010. Georgia Educational Researcher Online Edition, 10(1), Article 1. https://doi.org/10.20429/ger.2013.100101
61. Schulz, K.F., Altman, D.G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Trials, 11(1), Article 32. https://doi.org/10.1136/bmj.c332
62. Schweinhart, L.J., Barnes, H.V., & Weikart, D.P. (1993). Significant benefits: The High/Scope Perry Preschool Study through age 27. High/Scope Press.
63. Shapiro, G.M. (2008). Sample size. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (Vol. 2, pp. 781–783). Sage.
64. Shaver, J.P., & Norton, R.S. (1980). Populations, samples, randomness, and replication in two social studies journals. Theory & Research in Social Education, 8(2), 1–10. https://doi.org/10.1080/00933104.1980.10506078
65. Shukla, A. (2017). Literature review: An oblivious yet grounding task of research. Management Insight, 13(1), 7–15. https://doi.org/10.21844/mijia.v13i01.8363
66. Slavin, R.E. (2008). Perspectives on evidence-based research in education—What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5–14. https://doi.org/10.3102/0013189X08314117
67. Stebbins, L.F. (2006). Student guide to research in the digital age: How to locate and evaluate information sources. Libraries Unlimited.
68. Sung, Y.T., Lee, H.Y., Yang, J.M., & Chang, K.E. (2019). The quality of experimental designs in mobile learning research: A systemic review and self-improvement tool. Educational Research Review, 28, 100279. https://doi.org/10.1016/j.edurev.2019.05.001
69. Tabachnick, B.G., & Fidell, L.S. (2019). Using multivariate statistics (7th ed.). Pearson.
70. United States Department of Education. (2020). ED delivers historic investment in STEM. https://content.govdelivery.com/accounts/USED/bulletins/2ad85c3
71. Valentine, J.C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13(2), 130–149. https://doi.org/10.1037/1082-989X.13.2.130
72. Wallen, N.E., & Fraenkel, J.R. (1988). An analysis of social studies research over an eight year period. Theory & Research in Social Education, 16(1), 1 22. https://doi.org/10.1080/00933104.1988.10505553
73. Warner, R.M. (2013). Applied statistics: From bivariate through multivariate techniques (2nd ed.). Sage.
74. What Works Clearinghouse. (2020a). Standards handbook, version 4.1. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf
75. What Works Clearinghouse. (2020b). Procedures handbook, version 4.1. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf