Investigating the Effects of Prompt Engineering in STEM Activities Designed with ChatGPT-4o

Abstract

This study examines the use of ChatGPT-4o, an artificial intelligence-supported chatbot, in the design of STEM activities and investigates the effects of prompt engineering practices on this process. The aim of the study is to determine how pre-service mathematics teachers with no prior experience in designing STEM activities navigate ChatGPT-4o, how the activities they design fit into the STEM framework, and how prompt engineering techniques influence the quality of the activities. The research was conducted with the case study method with seven pre-service mathematics teachers enrolled in an elementary mathematics teacher education program at a university in Turkey. During the data collection process, the pre-service teachers designed STEM activities using ChatGPT-4o, first without knowledge of prompt engineering. Following a five-hour instructional session, they repeated the task using prompt engineering strategies. The designed activities were analyzed using the “STEM Activity Evaluation Form.” The findings showed that STEM activities designed without knowledge of prompt engineering were generally superficial in terms of thematic depth. However, after receiving the prompt engineering training, pre-service teachers produced more appropriate and interdisciplinary STEM activities. The results of the study reveal that ChatGPT-4o has strong potential as a guide in STEM education, but prompt engineering is critical for its effective use. The researchers recommend the development of prompt engineering skills into teacher education and the effective integration of artificial intelligence tools in STEM instruction.

Keywords

• STEM
• ChatGPT
• Prompt Engineering
• Artificial Intelligence
• Pre-service Teachers

ChatGPT-4o ile Tasarlanan STEM Etkinliklerinde İstem Mühendisliğinin Etkilerinin İncelenmesi

Abstract

Bu araştırma, yapay zekâ destekli sohbet robotu ChatGPT-4o’nun STEM etkinlikleri tasarımında kullanımını ve istem mühendisliği uygulamalarının bu süreçteki etkilerini incelemektedir. Çalışmanın amacı, STEM etkinlikleri tasarlama deneyimi olmayan matematik öğretmen adaylarının ChatGPT-4o’ü nasıl yönlendirdiğini, tasarladıkları etkinliklerin STEM çerçevesine uygunluğunu ve istem mühendisliği tekniklerinin etkinliklerin niteliğini nasıl etkilediğini belirlemektir. Araştırma, durum çalışması yöntemiyle yürütülmüş olup Türkiye'de bir üniversitenin ilköğretim matematik öğretmenliği programında öğrenim gören 7 öğretmen adayıyla gerçekleştirilmiştir. Veri toplama sürecinde, öğretmen adayları önce istem mühendisliği bilgisi olmadan ardından beş ders saati bir eğitim sonrasında istem mühendisliği bilgisiyle ChatGPT-4o kullanarak STEM etkinlikleri tasarlamışlardır. Tasarlanan etkinlikler, “STEM Etkinliği Değerlendirme Formu” kullanılarak analiz edilmiştir. Bulgular, istem mühendisliği bilgisi olmadan tasarlanan STEM etkinliklerinin temalar açısından genellikle yüzeysel kaldığını göstermiştir. Ancak, istem mühendisliği eğitimi sonrasında öğretmen adaylarının daha uygun ve disiplinler arası bağlantıları güçlü STEM etkinlikleri oluşturduğu belirlenmiştir. Çalışmanın sonuçları, ChatGPT-4o’nun STEM eğitimi için rehberlik potansiyeline sahip olduğunu ancak etkin kullanımı için istem mühendisliğinin kritik olduğunu ortaya koymaktadır. Araştırmacılar, öğretmen eğitiminde istem mühendisliği becerilerinin geliştirilmesini ve yapay zekâ araçlarının STEM öğretiminde etkin entegrasyonunu önermektedir.

Keywords

• STEM
• ChatGPT
• İstem Mühendisliği
• yapay zeka
• Öğretmen adayları

References

1. Alemdag, E. (2023). The effect of chatbots on learning: a meta-analysis of empirical research. Journal of Research on Technology in Education, 57(2), 459–481.. https://doi.org/10.1080/15391523.2023.2255698
2. Aydın, S. K., Kılıç, Ş., Özdemir, Z., & Geyik, M. (2023). Öğrenci Merkezli Eğitim: Öğrencilerin Başarıya Giden Yolu. Socrates Journal of Interdisciplinary Social Studies, 9(34), 135-144. https://doi.org/10.5281/zenodo.8430400
3. Bybee, R. W. (2010). Advancing STEM education: A 2020 vision. Technology and Engineering Teacher, 70(1), 30-35.
4. Caijun W., Xi J. & Zhenzhou Z. (2021)."Analysis of Systematic Reform of Future Teaching in the Age of Artificial Intelligence," 2021 2nd International Conference on Artificial Intelligence and Education (ICAIE), Dali, China, pp. 704-707, https://doi.org/10.1109/ICAIE53562.2021.00154
5. Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. Ieee Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
6. Chiu, T. K., Moorhouse, B. L., Chai, C. S., & Ismailov, M. (2023). Teacher support and student motivation to learn with Artificial Intelligence (AI) based chatbot. Interactive Learning Environments, 32(7), 3240–3256.https://doi.org/10.1080/10494820.2023.2172044
7. Chocarro, R., Cortiñas, M., & Marcos-Matás, G. (2021). Teachers’ attitudes towards chatbots in education: a technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics. Educational Studies, 49(2), 1–19. http://dx.doi.org/10.1080/03055698.2020.1850426
8. Clements, D.H. & Sarama, J. (2021). STEM or STEAM or STREAM? Integrated or Interdisciplinary?. In: Cohrssen, C., Garvis, S. (Eds) Embedding STEAM in Early Childhood Education and Care (pp.261-275). Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-65624-9_13

9. Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., & Lemma, S. (2018). Chatbot for e-learning: A case of study. International Journal of Mechanical Engineering and Robotics Research, 7(5), 528–533.
10. Collins, A. & Halverson, R. (2010). The second educational revolution: Rethinking education in the age of technology. Journal of Computer Assisted Learning, 26(1), 18–27. https://doi.org/10.1111/j.1365-2729.2009.00339.x
11. Creswell, J. W. (2021). Nitel araştirma yöntemleri. (M. Bütün & SB Demir, Çev. Ed.). Siyasal Kitapevi.
12. Dhinakaran, A. (Aralık, 2023) Evaluating prompts: A developer's guide. https://arize.com/blog-course/evaluating-prompt-playground/
13. Dimitriadou, E., & Lanitis, A. (2023). A critical evaluation, challenges, and future perspectives of using artificial intelligence and emerging technologies in smart classrooms. Smart Learning Environments, 10(12), 1–26. https://doi.org/10.1186/s40561-023-00231-3
14. Dinc, E. (2019). Prospective teachers’ perceptions of barriers to technology integration in education. Contemporary educational technology, 10(4), 381-398. https://doi.org/10.30935/cet.634187
15. Dowling, M., & Lucey, B. (2023). ChatGPT for (Finance) research: The Bananarama Conjecture. Finance Research Letters, 53, 103662. https://doi.org/10.1016/j.frl.2023.103662
16. Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846
17. Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in Human Behavior, 93, 279–289. https://doi.org/10.1016/j.chb.2018.12.023
18. Gasiewski, J. A., Eagan, M. K., Garcia, G. A., Hurtado, S., & Chang, M. J. (2012). From gatekeeping to engagement: A multicontextual, mixed method study of student academic engagement in introductory STEM courses. Research in Higher Education, 53(2), 229–261. https://doi.org/10.1007/s11162-011-9247-y
19. Gemini for Google Workspace (2024). Prompting Guide 101. Google Workspace; https://services.google.com/fh/files/misc/gemini-for-google-workspace-prompting-guide-101.pdf
20. Gupta, S., & Chen, Y. (2022). Supporting inclusive learning using chatbots? A chatbot-led interview study. Journal of Information Systems Education, 33(1), 98–108.
21. Hallström, J., & Schönborn, K. J. (2019). Models and modelling for authentic STEM education: Reinforcing the argument. International Journal of STEM Education. https://doi.org/10.1186/s40594-019-0178-z
22. Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning–Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
23. Huberman, M. (1994). Research utilization: The state of the art. Knowledge and Policy, 7(4), 13-33.
24. Jasin, J., Ng, H. T., Atmosukarto, I., Iyer, P., Osman, F., Wong, P. Y. K., Pua, C. Y., & Cheow, W. S. (2023). The implementation of chatbot-mediated immediacy for synchronous communication in an online chemistry course. Education and Information Technologies, 28, 10665–10690. https://doi.org/10.1007/s10639-023-11602-1
25. Jiao, W., Wang, W., Huang, J-T, Wang, X., & Tu, Z. (2023). Is ChatGPT A good translator? Yes With GPT-4 As The Engine. ArXiv. https://arxiv.org/abs/2301.08745
26. Johnson, C. C., & Czerniak, C. M. (2023). Interdisciplinary approaches and integrated STEM in science teaching. In N. G. Lederman, D. L. Zeidler, & J. S. Lederman (Eds.), Handbook of research on science education (pp. 559–585). Routledge.
27. Karadaş, Ö. F. (2021). Fen bilimleri öğretmenleri ve stem uygulamaları: tercih gerekçeleri, sorunlar ve çözüm önerileri [Yayınlanmamış Yüksek Lisans Tezi], Kırşehir Ahi Evran Üniversitesi, Kırşehir.
28. Kaya, G., Şahin, E. & Uz-Özkılıç, R. (2020). Fen bilimleri öğretmenlerinin STEM eğitimi ve uygulamaları hakkındaki görüşlerinin belirlenmesi (Bursa İli örneği). Uluslararası İnsan ve Sanat Araştırmaları Dergisi, 5(6), 100-113.
29. Kim, N. J., and Kim, M. K. (2022). Teacher's perceptions of using an artificial intelligence-based educational tool for scientific writing. Frontiers in Education. 7, 755914. https://doi.org/10.3389/feduc.2022.755914
30. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1
31. Knox, J. (2019). What does the ‘postdigital’ mean for education? Three critical perspectives on the digital, with implications for educational research and practice. Postdigital Science and Education, 1, 357–370. https://doi.org/10.1007/s42438-019-00045-y
32. Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973–1018. https://doi.org/10.1007/s10639-022-11177-3
33. Kumar, J. A. (2021). Educational chatbots for project-based learning: Investigating learning outcomes for a team-based design course. International Journal of Educational Technology in Higher Education, 18, 65-93. https://doi.org/10.1186/s41239-021-00302-w
34. Kurtuluş, A., Akçay, A. O., & Karahan, E. (2017). Ortaokul matematik derslerinde STEM uygulamalarına yönelik öğretmen görüşleri. Eğitim ve Öğretim Araştırmaları Dergisi, 6(4), 354-360.
35. Labadze, L., Grigolia, M. & Machaidze, L. (2023). Role of AI chatbots in education: systematic literature review. International Journal of Educational Technology in Higher Education, 20, 56-73. https://doi.org/10.1186/s41239-023-00426-1
36. Laksana, F. S. W., and Fiangga, S. (2022). The development of web-based chatbot as a mathematics learning media on system of linear equations in three variables. Jurnal Ilmiah Pendidikan Matematika 11, 145-154
37. Lau, W. W., & Jong, M. S. (2023). Typology of teachers’ stages of concern for STEM education. Research in Science & Technological Education, 41(4), 1560-1578. https://doi.org/10.1080/02635143.2022.2064447
38. Lee, A.V.Y., Tan, S.C. & Teo, C.L. (2023). Designs and practices using generative AI for sustainable student discourse and knowledge creation. Smart Learning Environments, 10, 59-77. https://doi.org/10.1186/s40561-023-00279-1
39. Leung, A. (2020). Boundary crossing pedagogy in STEM education. International Journal of STEM Education, 7, 15-26. https://doi.org/10.1186/s40594-020-00212-9
40. Li, Y., Schoenfeld, A. H., diSessa, A. A., Craesser, A. C., Benson, L. C., English, L. D., & Duschl, R. A. (2019). On thinking and STEM education. Journal for STEM Education Research, 2(4),1-13. https://doi.org/10.1007/s41979-019-00014-x.
41. Lin, C. H., Yu, C. C., Shih, P. K., & Wu, L. Y. (2021). STEM based artificial intelligence learning in general education for non-engineering undergraduate students. Educational Technology & Society, 24 (3), 224–237.
42. Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New directions for evaluation, (30), 73-84.
43. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1-35.
44. Loock, R., & Holt, B. (2024). Augmented linguistic analysis skills: machine translation and generative AI as pedagogical aids for analyzing complex English compounds. Technology in Language Teaching & Learning, 6(3), 1489-1489.
45. Luckin, R., & Holmes, W. (2016). Intelligence unleashed: An argument for AI in education.
46. Macdonald, C., Adeloye, D., Sheikh, A., & Rudan, I. (2023). Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis. Journal of Global Health, 13, 01003. https://doi.org/10.7189/jogh.13.01003
47. Majid, I. & Lakshmi, Y., V. (2022).Artıfıcıal Intellıgence In Educatıon. The Indian Journal of Technical Education, 45(3), 11-16.
48. Margot, K.C. & Kettler, T. (2019). Teachers’ perception of STEM integration and education: a systematic literature review. International Journal of STEM Education, 6, 2-18. https://doi.org/10.1186/s40594-018-0151-2
49. Millî Eğitim Bakanlığı (MEB). (2016). MEB STEM eğitimi raporu. Ankara: Yenilik ve Eğitim Teknolojileri Genel Müdürlüğü.
50. McComas, W.F. & Burgin, S.R. (2020). A Critique of “STEM” Education. Science & Education, 29, 805–829 https://doi.org/10.1007/s11191-020-00138-2
51. Mousavi, A., Schmidt, M., Squires, V., & Wilson, K. (2020). Assessing the effectiveness of student advice recommender agent (SARA): The case of automated personalized feedback. International Journal of Artificial Intelligence in Education, 31(2), 603–621. https://doi.org/10.1007/s40593-020-00210-6
52. Nurshatayeva, A., Page, L. C., White, C. C., & Gehlbach, H. (2020). Proactive student support using artificially intelligent conversational chatbots: The importance of targeting the technology (EdWorkingPaper No. 20-208). Annenberg Institute at Brown University. https://doi.org/10.26300/mp4q-4x12
53. Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
54. Okulu, H. Z. & Oguz-Unver, A. (2021). The Development and Evaluation of a Tool to Determine the Characteristics of STEM Activities. European Journal of STEM Education, 6(1), 06. https://doi.org/10.20897/ejsteme/10894
55. Patel, S. B., & Lam, K. (2023). ChatGPT: The future of discharge summaries? The Lancet Digital Health, 5(3), e107–e108. https://doi.org/10.1016/s2589-7500(23)00021-3
56. Patton, M. Q. (2018). Nitel araştırma ve değerlendirme yöntemleri. (M. Bürün ve S.B. Demir, Çev. Ed.). Ankara: Pegem Akademi.
57. Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systemat-ic literature review. Computer Applications in Engineering Education, 28(6), 1549–1565. https://doi.org/10.1002/cae.22326
58. Perez, S., Massey-Allard, J., Butler, D., Ives, J., Bonn, D., Yee, N., & Roll, I. (2017). Identifying productive inquiry in virtual labs using sequence mining. In E. André, R. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), Artificial intelligence in education, 10(331), 287–298. https://doi.org/10.1007/978-3-319-61425-0_24
59. Peters, M. A., Jackson, L., Papastephanou, M., Jandrić, P., Lazaroiu, G., Evers, C. W., ... & Fuller, S. (2023). AI and the future of humanity: ChatGPT-4, philosophy and education–Critical responses. Educational Philosophy and Theory, 1-35. https://doi.org/10.1080/00131857.2023.2213437
60. Priemer, B., Eilerts, K., Filler, A., Pinkwart, N., RöskenWinter, B., Tiemann, B., & Upmeier Zu Belzen, A. (2019). A framework to foster problem solving in STEM and computing education. Research in Science & Technological Education. https://doi.org/10.1080/02635143.2019.1600490.
61. Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023). Evaluating ChatGPT as an adjunct for radiologic decision-making [Preprint]. medRxiv,. https://doi.org/10.1101/2023.02.02.23285399
62. Salas-Pilco, S. Z., & Yang, Y. (2022). Artificial intelligence applications in Latin American higher education: A systematic review. International Journal of Educational Technology in Higher Education, 19(21), 1–20. https://doi.org/10.1186/s41239-022-00326-w
63. Salonen, A., Hartikainen-Ahia, A., Hense, J., Scheersoi, A., & Keinonen, T. (2017). Secondary school students’ perceptions of working life skills in science-related careers. International Journal of Science Education, 39(10), 1339–1352. https://doi.org/10.1080/09500693.2017.1330575
64. Shah, K., Xu, A. Y., Sharma, Y., Daher, M., McDonald, C., Diebo, B. G., & Daniels, A. H. (2024). Large language model prompting techniques for advancement in clinical medicine. Journal of Clinical Medicine, 13(17), 1-12. https://doi.org/10.3390/jcm13175101
65. Shahbazloo, F., & Mirzaie, R. A. (2023). Investigating the effect of 5E-based STEM education in solar energy context on creativity and academic achievement of female Junior high School students. Thinking Skills and Creativity, 49,1-13. https://doi.org/10.1016/j.tsc.2023.101336
66. Shernoff, D. J., Sinha, S., Bressler, D. M., & Ginsburg, L. (2017). Assessing teacher education and professional development needs for the implementation of integrated approaches to STEM education. International journal of STEM education, 4, 1-16. https://doi.org/10.1186/s40594-017-0068-1
67. Shieh, R. S. ve Chang, W. (2014). Fostering student‟s creative and problem solvingskills through hands on activity. Journal of Baltic Science Education, 13(5), 650-661
68. Sobania, D., Briesch, M., Hanna, C., & Petke, J.(2023). An analysis of the automatic bug fixing performance of ChatGPT. ArXiv. https://doi.org/10.48550/arXiv.2301.08653
69. Southgate, E., Blackmore, K., Pieschl, S., Grimes, S., McGuire, J., & Smithers, K. (2019). Artificial intelligence and emerging technologies in schools: Research report. University of Newcastle. https://www.dese.gov.au/supporting-family-school-community-partnerships-learning/resources/ai-schools-report
70. Struyf, A., Boeve-de Pauw, J., & Van Petegem, P. (2017). ‘Hard science’: A career option for socially and societally interested students? Grade 12 students’ vocational interest gap explored. International Journal of Science Education, 39(17), 2304–2320. http://dx.doi.org/10.1080/09500693.2017.1376259
71. Stojanov, A. (2023). Learning with ChatGPT 3.5 as a more knowledgeable other: an autoethnographic study. Int J Educ Technol High Educ 20, 35 https://doi.org/10.1186/s41239-023-00404-7
72. Şirin- Tezcan, G., Oğuz-Kaval, E., & Tüysüz, M. (2022). Ortaokul fen bilimleri ders kitaplarında yer alan etkinliklerin stem etkinlikleri açısından uygunluğunun incelenmesi. Ankara University Journal of Faculty of Educational Sciences (JFES), 55(1), 37-76. https://doi.org/10.30964/auebfd.863341
73. Ting, Y. L. (2016). STEM from the perspectives of engineering design and suggested tools and learning design. Journal of Research in STEM Education, 2(1), 59-71. https://doi.org/10.51355/jstem.2016.22
74. Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
75. Topal, A. D., Eren, C. D., and Geçer, A. K. (2021). Chatbot application in a 5th grade science course. Educ. Inf. Technol. 26, 6241–6265. https://doi.org/10.1007/s10639-021-10627-8
76. Tutak, F., Akaygün, S. ve Tezsezen , S., (2017) İşbirlikli fetemm eğitimi uygulaması: Kimya ve matematik öğretmen adaylarının FeTeMM farkındalıkları. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 32(4), 794-816.
77. Türk, Z. & Duran, M (2021). Uzaktan Eğitimde STEM Etkinliklerinin Uygulanmasına İlişkin Okul Öncesi Öğretmen Adaylarının Görüşlerinin İncelenmesi. Çocuk ve Gelişim Dergisi, 4(8), 46-73. https://doi.org/10.36731/cg.1020566 Wang S., Scells, H., Koopman, B. & Zuccon, G. (2023). Can ChatGPT write a good Boolean query for systematic review literature search? ArXiv. https://arxiv.org/abs/2302.03495
78. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., ... & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382. https://doi.org/10.48550/arXiv.2302.11382
79. Yao, X. (2022). Design and research of artificial intelligence in multimedia intelligent question answering system and self-test system. Advances in Multimedia. https://doi.org/10.1155/2022/2156111
80. Yin, J., Goh, T. T., Yang, B., & Xiaobin, Y. (2021). Conversation technology with micro-learning: The impact of chatbot-based learning on students’ learning motivation and performance. Journal of Educational Computing Research, 59(1), 154–177. https://doi.org/10.1177/0735633120952067
81. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022). Large language models are human-level prompt engineers [preprint]. arXiv. https://doi.org/10.48550/arXiv.2211.01910