Adaptation of STEM Competencies Assessment Framework for 4th-grade Level

Year 2024, Issue: 60, 201 - 225, 15.01.2024

Serkan Arıkan , Melek Pesen , Emine Erktin

https://doi.org/10.9779/pauefd.1249861

Cited By: 2

Abstract

As observed in the studies encountered in the literature, when students' STEM competencies are evaluated, scores from large-scale mathematics and/or science tests are used instead of scores from interdisciplinary measurement tools specifically developed to measure STEM competencies. In order to develop an assessment tool to determine students' STEM competency scores, firstly, a STEM competencies assessment framework is needed because no assessment framework has been found to assess STEM competencies at the primary school level. For this reason, the current study aimed to adapt the STEM competencies assessment framework to the 4th-grade level, which was developed by Arıkan et al. (2022) and presented validity evidence at the 8th-grade level. In this context, the reliability and validity evidence of the scores obtained from a measurement tool on the assessment framework adapted to the 4th-grade level are presented. The assessment framework presented in the current study was prepared by considering the multidimensional and integrated structure of STEM competencies- science, technology, engineering, and mathematics. The factor structure was evaluated by confirmatory factor analysis using data collected from 4th-grade students. The items in the developed measurement tool were analyzed at the item level using Item Response Theory. The findings show that the adapted STEM assessment framework can be used for 4th graders.

Keywords

STEM assessment framework, Non-routine problems, Confirmatory factor analysis, Item analysis

Project Number

2218

STEM Yeterlikleri Değerlendirme Çerçevesinin 4. sınıf Düzeyi için Uyarlanması

Abstract

Alan yazında karşılaşılan çalışmalarda görülmektedir ki öğrencilerin STEM yeterlikleri değerlendirilirken STEM yeterliklerini ölçmek amacı ile oluşturulmuş, disiplinler arası bir ölçme aracı ile elde edilen puanlar yerine, büyük ölçekli testlerden veya başka amaçlar için geliştirilmiş sınavlardan elde edilen matematik ve/veya fen puanları kullanılmaktadır. Öğrencilerin STEM yeterlik puanlarını belirlemek amacı ile bir ölçme aracı geliştirebilmek için öncelikle STEM yeterlikleri değerlendirme çerçevesine ihtiyaç duyulmaktadır. Özellikle ilkokul seviyesinde, STEM yeterliklerini değerlendirebilmek amacıyla kullanılabilecek bir değerlendirme çerçevesi bulunmamaktadır. Bu sebeple bu çalışmada, Arıkan ve diğerleri (2022) tarafından geliştirilmiş ve 8. sınıf düzeyinde geçerlik kanıtları sunulmuş olan STEM yeterlikleri değerlendirme çerçevesinin, 4. sınıf düzeyine uyarlanması amaçlanmıştır. Bu kapsamda, 4. sınıf düzeyine uyarlanan değerlendirme çerçevesi temel alınarak geliştirilen bir ölçme aracından elde edilen puanların güvenirlik ve geçerlik kanıtları sunulmuştur. Bu çalışmada ortaya konan değerlendirme çerçevesi, STEM yeterliklerinin çok boyutlu ve bütünleşik yapısı- fen, teknoloji, mühendislik ve matematik alanları- dikkate alınarak hazırlanmıştır. Ortaya konan faktör yapısı, 4. sınıf öğrencilerinden toplanan veriler kullanılarak doğrulayıcı faktör analizi ile değerlendirilmiştir. Geliştirilen ölçme aracında yer alan maddeler, madde tepki kuramı kullanılarak madde düzeyinde incelenmiştir. Bulgular, 8. sınıflar için geliştirilmiş olan STEM değerlendirme çerçevesinin 4. sınıflar için de kullanılabileceğini göstermektedir.

Keywords

STEM değerlendirme çerçevesi, rutin olmayan problemler, doğrulayıcı faktör analizi, madde analizi

Supporting Institution

TÜBİTAK

Project Number

2218

Thanks

Bu çalışma TÜBİTAK 2218 programı kapsamında desteklenmiştir.

References

• Arıkan, S., Erktin, E., & Pesen, M. (2022). Development and validation of a STEM competencies assessment framework. International Journal of Science and Mathematics Education, 20, 1-24. https://doi.org/10.1007/s10763-020-10132-3
• Bennett, C. A., & Ruchti, W. (2014). Bridging STEM with mathematical practices. Journal of STEM Teacher Education, 49(1), 17–28. https://doi.org/10.30707/JSTE49.1Bennett.
• Bicer, A., Capraro, R. M., & Capraro, M. M. (2017). Integrated STEM assessment model. Eurasia Journal of Mathematics Science and Technology Education, 13(7), 3959–3968. https://doi.org/10.12973/eurasia. 2017.00766a.
• Bicer, A., Navruz, B., Capraro, R. M., & Capraro, M. M. (2014). STEM schools vs. non-STEM schools: Comparing students mathematics state based test performance. International Journal of Global Education, 3(3), 8–18.
• Blanton, M., & Kaput, J. (2011). Functional thinking as a route into algebra in the elementary grades. In J. Cai & E. Knuth (Eds.), Early algebraization: A global dialogue from multiple perspectives (pp. 5–23). Berlin: Springer.
• Blum, W., & Ferri, R. B. (2009). Mathematical modelling: Can it be taught and learnt? Journal of Mathematical Modelling and Application, 1(1), 45–58.
• Broggy, J., O'Reilly, J., & Erduran, S. (2017). Interdisciplinarity and science education. In B. Akpan & K. Taber (Eds.), Science Education. New directions in mathematics and science education (pp. 81–90). Sense Publishers.
• Brown, R., Brown, J., Reardon, K., & Merrill, C. (2011). Understanding STEM: Current perceptions. Technology and Engineering Teacher, 70(6), 5–9.
• Brown, W. (2015). Introduction to algorithmic thinking. Retrieved from https://raptor.martincarlisle.com/Introduction%20to%20Algorithmic%20Thinking.doc.
• Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Lawrence Erlbaum Associates, Inc.
• Cai, L., du Toit, S. H. C., & Thissen, D. (2011). IRTPRO: User guide. Lincolnwood, IL: Scientific Software International.
• Capraro, R. M., & Corlu, M. S. (2013). Changing views on assessment for STEM project-based learning. In R. M. Capraro, M. M. Capraro, & J. Morgan (Eds.), STEM project-based learning: An integrated Science, Technology, Engineering, and Mathematics (STEM) approach (2nd ed., pp. 109–118). Rotterdam, The Netherlands: Sense Publishers.
• Carter, L. (2007). Globalization and science education: The implications of science in the new economy. Journal of Research in Science Teaching, 45(5), 617–633. https://doi.org/10.1002/tea.20189.
• Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. https://doi.org/10.1037/ 1040-3590.6.4.284.
• Crocker, L., & Algina, J. (2008). Introduction to classical and modern test theory. Mason, OH: Cengage Learning
• DiStefano, C., & Hess, B. (2005). Using confirmatory factor analysis for construct validation: An empirical review. Journal of Psychoeducational Assessment, 23, 225-241.
• Doerr, H., & English, L. D. (2003). A modeling perspective on students' mathematical reasoning about data. Journal for Research in Mathematics Education, 34(2), 110-136.
• Elia, I., van den Heuvel-Panhuizen, M., & Kolovou, A. (2009). Exploring strategy use and strategy flexibility in non-routine problem solving by primary school high achievers in mathematics. ZDM Mathematics Education, 41(5), 605–618. https://doi.org/10.1007/s11858-009-0184-6.
• Enderson, M. C., & Ritz, J. (2016). STEM in general education: Does mathematics competence influence course selection. Journal of Technology Studies, 42(1), 30–41. https://doi.org/10.21061/jots.v42i1.a.3.
• English, L. D. (2015). STEM: Challenges and opportunities for mathematics education. In K. Beswick, T. Muir, & J. Wells (Eds.), Proceedings of the 39th Conference of the International Group for the Psychology of Mathematics Education (Vol. 1, pp. 4–18). Hobart, Australia: PME.
• Feinstein, N. (2011). Salvaging science literacy. Science Education, 95(1), 168-185.
• Fitzallen, N. (2015). STEM education: What does mathematics have to offer? In M. Marshman (Ed.), Mathematics education in the margins. Proceedings of the 38th annual conference of the Mathematics Education Research Group of Australasia (pp. 237–244). Sydney, Australia: Mathematics Education Research Group of Australasia (MERGA).
• Freudenrich, C., & Boyd, R. (2001). How your brain works. Howstuffworks. http://www.howstuffworks.com/brainl.htm
• Guyer, R., & Thompson, N. A. (2014). User’s Manual for Xcalibre item response theory calibration software, version 4.2.2 and later. Woodbury MN: Assessment Systems Corporation.
• Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38–47. https://doi.org/10.1111/j.1745-3992.1993.tb00543.x.
• Han, S., Capraro, R., & Capraro, M. M. (2015). How science, technology, engineering, and mathematics (STEM) project-based learning (PBL) affects high, middle, and low achievers differently: The impact of student factors on achievement. International Journal of Science and Mathematics Education, 13(5), 1089–1113. https://doi.org/10.1007/s10763-014-9526-0.
• Harwell, M., Moreno, M., Phillips, A., Guzey, S. S., Moore, T. J., & Roehrig, G.H. (2015). A study of STEM assessments in engineering, science, and mathematics for elementary and middle school students. School Science and Mathematics, 115(2), 66–74. https://doi.org/10.1111/ssm.12105.
• Haudek, K. C., Kaplan, J. J., Knight, J., Long, T., Merrill, J., Munn, A., Nehm, R., Smith, M., & Urban- Lurain, M. (2011). Harnessing technology to improve formative assessment of student conceptions in STEM: Forging a national network. CBE—Life Sciences Education, 10(2), 149–155.
• Ing, M. (2014). Can parents influence children's mathematics achievement and persistence in STEM careers? Journal of Career Development, 41(2), 87–103. https://doi.org/10.1177/0894845313481672.
• Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(11), 1–11. https://doi.org/10.1186/s40594-016-0046-z.
• Kline, R. B. (1998). Principles and practice of structural equation modeling. The Guilford Press. Lee, N.H., Yeo, D. J. S., & Hong, S.E. (2014). A metacognitive-based instruction for primary four students to approach non-routine mathematical word problems. ZDM, 46(3), 465–480. https://doi.org/10.1007/s11858-014-0599-6.
• MEB. (2016). STEM Eğitimi Raporu. Ankara: Millî Eğitim Bakanlığı- Yenilik ve Eğitim Teknolojileri Genel Müdürlüğü (YEĞİTEK).
• Milgram, R. J. (2007). What is mathematical proficiency? In AH Schoenfeld (Ed.), Assessing mathematical proficiency (pp. 31–58). Cambridge University Press. https://doi.org/10.1017/CBO9780511755378.007.
• Miller, J. (2019). STEM education in the primary years to support mathematical thinking: Using coding to identify mathematical structures and patterns. ZDM, 51(6), 915-927.
• Moreno, N. P., Tharp, B. Z., Vogt, G., Newell, A. D., & Burnett, C. A. (2016). Preparing students for middle school through after-school STEM activities. Journal of Science Education and Technology, 25(6), 889– 897. https://doi.org/10.1007/s10956-016-9643-3.
• Mullis, I. V. S., & Martin, M. O. (Eds.). (2017). TIMSS 2019 assessment frameworks. Boston College, TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2019/frameworks/
• National Council of Teachers of Mathematics. (2018). Building STEM education on a sound Mathematical Foundation. Position statement. Retrieved from https://www.nctm.org/Standards-and-Positions/NCTM-Position Statements/.
• National Research Council. (2005). How students learn: Mathematics in the classroom. The National Academies Press.
• National Research Council. (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. The National Academies Press.
• National Science and Technology Council. (2011). The federal science, technology engineering, and math- ematics (STEM) education portfolio. Retrieved from https://www.whitehouse.gov/sites/default/files/microsites/ostp/costemfederal_stem_education_portfolio_report.pdf
• Nga, N. T., Quỳnh, T. T. X., Uyên, N. P, & Trung, T.T. (2022). Một số nghiên cứu về năng lực STEM trên thế giới và đề xuất khung năng lực STEM cho học sinh phổ thông tại Việtnam (An overview study on STEM competencies in the world and propose a STEM competency framework for high school students in Vietnam), Tạp chí Giáo dục, 22, 48-53.
• Pantziara, M., Gagatsis, A., & Elia, I. (2009). Using diagrams as tools for the solution of non-routine mathematical problems. Educational Studies in Mathematics, 72(1), 39–60. https://doi.org/10.1007/ s10649-009-9181-5.
• Rumsey, C., & Langrall, C. W. (2016). Promoting mathematical argumentation. Teaching Children Mathematics, 22(7), 412-419.
• Saxton, E., Burns, R., Holveck, S., Kelley, S., Prince, D., Rigelman, N., & Skinner, EA (2014). A common measurement system for K-12 STEM education: Adopting an educational evaluation methodology that elevates theoretical foundations and systems thinking. Studies in Educational Evaluation, 40, 18–35. https://doi.org/10.1016/j.stueduc.2013.11.005 .
• Schmidt, W. H., & Houang, R. T. (2007). Lack of focus in the mathematics curriculum: A symptom or a cause? In T. Loveless (Ed.), Lessons learned: What international assessments tell us about math achievement (pp. 65–84). Brookings Institution Press.
• Schumacker, R. E. & Lomax, R. G. (2004). A beginners guide to structural equation modeling. Lawrence Erlbaum Associates, Inc.
• Shulman, L. (2009). Assessment of teaching or assessment for teaching? In D.H. Gitomer (Ed.), Measurement issues and assessment for teaching quality. Sage Publications.
• Toulmin, S. E. (2003). The uses of argument. Cambridge University Press. Trung, T. T., Quỳnh, T. T. X., Uyên, N. P., & Nga, N. T. (2022). Xây dựng và chuẩn hóa công cụ đánh giá năng lực stem của học sinh trung học phổ thông tại thành Phố Hồ Chí Minh (Develop and standardize a stem competency assessment tool for high school students in Ho Chi Minh City). Ho Chi Minh City University of Education Journal Of Science, 19(8), 1255-1270. https://doi.org/10.54607/hcmue.js.19.8.3408(2022) Turner, E. E., Roth McDuffie, A., Bennett, A. B., Aguirre, J., Chen, M. K., Foote, M. Q., & Smith, J. E. (2022). Mathematical modeling in the elementary grades: Developing and testing an assessment. International Journal of Science and Mathematics Education, 20(7), 1387-1409.
• TÜSİAD. (2014). STEM Alanında eğitim almış işgücüne yönelik talep ve beklentiler araştırması. http://tusiad.org/tr/yayinlar/raporlar/item/download/7014_d28ffa2adda423c6d3852cc01c965993 Ullman, J. B. (2001). Structural equation modeling. In B. Tabachnick & L. S. Fidell (Eds.), Using multivariate statistics (4th ed., pp.653-771). Allyn & Bacon.
• van Borkulo, S., Chytas, C., Drijvers, P., Barendsen, E., & Tolboom, J. (2021, October). Computational thinking in the mathematics classroom: fostering algorithmic thinking and generalization skills using dynamic mathematics software. In The 16th Workshop in Primary and Secondary Computing Education (pp. 1-9).
• White, D. W. (2014). What is STEM education and why is it important. Florida Association of Teacher Educators Journal, 1(14), 1–9.
• Wong, G. K., Cheung, H. Y., Ching, E. C., & Huen, J. M. (2015, December). School perceptions of coding education in K-12: A large scale quantitative study to inform innovative practices. In 2015 IEEE international conference on teaching, Assessment, and learning for engineering (TALE) (pp. 5-10). IEEE.
• Woodward, J., Beckmann, S., Driscoll, M., Franke, M., Herzig, P., Jitendra, A., Koedinger, K.R., & Ogbuehi, P. (2012). Improving mathematical problem solving in grades 4 through 8: A practice guide (NCEE 2012-4055), National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education. Washington, DC. Retrieved from http://ies.ed.gov/ncee/wwc/ publications_reviews.aspx#pubsearch/.
• Yıldırım, H., & Gelmez-Burakgazi, S. (2020). Research on STEM education studies in Turkey: A qualitative meta-synthesis study. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 50, 291-314. https://doi.org/10.9779/pauefd.590319