Investigating the Process of Chatgpt as An Informal Learning Assistant to Guide Students in Problem Solving Process

Abstract

The aim of the study was to investigate the ChatGPT, an artifical intelligent chat bot, as an informal learning assistant in guiding students during problem solving. Students were asked to solve a given problem step by step according to Polya's problem solving steps and explain each step to ChatGPT. The students asked ChatGPT to provide guidance through feedback and questions for each step they explained. According to the answers given by ChatGPT, the questions it asked were classified according to three question categories (interrogative, guiding, factual questions). In addition, in order to examine the effect of prompt engineering on the type of guidance provided by ChatGPT, students realized their requests using different commands. When the answers given by ChatGPT to the students according to the questions asked by the students to ChatGPT regarding the solutions of the problems were taken into consideration, it was observed that ChatGPT was able to guide the students in the problem solving process by supporting informal learning. It was observed that ChatGPT generally asked factual and directive questions in the problem solving process. However, with the effect of prompt engineering, ChatGPT can be more oriented towards inquisitive questions. The study suggests that in order for ChatGPT to be an efficient informal learning assistant in the problem solving process, prompt engineering is necessary and students should be trained on prompt engineering by teachers.

Keywords

• Problem solving
• informal learning
• artificial intelligence
• ChatGPT
• prompt engineering.

Chatgpt’nin Bir Informal Öğrenme Asistanı Olarak Problem Çözme Sürecinde Öğrencilere Rehberlik Etme Sürecinin İncelenmesi

Abstract

Çalışmanın amacı bir yapay zekâ sohbet botu olan ChatGPT’nin informal öğrenme asistanı olarak problem çözme süresinde öğrencilere rehberlik etme durumlarının incelenmesidir. Öğrencilerden, Polya’nın problem çözme adımlarına göre verilen bir problemi adım adım çözmeleri ve her bir adımı ChatGPT’ye açıklamaları istenmiştir. Öğrenciler ChatGPT’den açıkladıkları her bir adım için kendilerine geri bildirim ve sorular aracılığıyla rehberlik sağlamasını istemişlerdir. ChatGPT’nin verdiği yanıtlara göre sorduğu sorular, üç soru kategorisine (sorgulayıcı, yönlendirici, olgusal sorular) göre sınıflandırılmıştır. Ayrıca ChatGPT’nin yaptığı rehberlik türüne istem mühendisliğinin etkisini incelemek amacıyla öğrenciler isteklerini farklı komutlar kullanarak gerçekleştirmişlerdir. Öğrencilerin problemlerin çözümlerine ilişkin ChatGPT’ye sordukları sorulara göre ChatGPT’nin öğrencilere verdiği yanıtlar dikkate alındığında ChatGPT’nin öğrencilerin problem çözme sürecinde informal öğrenmeyi destekleyici olarak rehberlik edebildiği gözlenmiştir. ChatGPT’nin problem çözme sürecinde genellikle olgusal ve yönlendirici sorular sorduğu görülmüştür. Ancak istem mühendisliğinin etkisiyle ChatGPT sorgulayıcı sorulara daha fazla yönelebilmektedir. Çalışma, ChatGPT’nin problem çözme sürecinde verimli bir informal öğrenme asistanı olabilmesi için istem mühendisliğinin gerekliliğini ve öğrencilerin öğretmenler tarafından istem mühendisliği konusunda eğitilmesini önermektedir.

Keywords

• problem çözme
• informal öğrenme
• yapay zeka
• ChatGPT
• istem mühendisliği

References

1. Anglin, K. L. (2004). Math Word Problem. USA: Wiley Publishing
2. Bayazıt, İ., ve Şenberber, H. (2021). Ortaokul öğrencilerinin rutin olmayan problemlerin çözümünde strateji kullanma ve öz-düzenleme becerilerinin incelenmesi. Başkent University Journal of Education, 8(2), 334-351.
3. Byun, H., Lee, J., ve Cerreto, F. A. (2014). Relative effects of three questioning strategies in ill-structured, small group problem solving. Instructional Science, 42, 229-250.
4. Callanan, M., Cervantes, C., ve Loomis, M. (2011). Informal learning. Wiley Interdisciplinary Reviews: Cognitive Science, 2(6), 646-655.
5. Cho, M. K., ve Kim, M. K. (2020). Investigating elementary students’ problem solving and teacher scaffolding in solving an ill-structured problem. International Journal of Education in Mathematics, Science and Technology (IJEMST), 8(4), 274-289.
6. Cooper, S. (2011). An exploration of the potential for mathematical experiences in informal learning environments. Visitor Studies, 14(1), 48-65. https://doi.org/10.1080/10645578.2011.557628
7. Creswell, J. W. (2021). Nitel araştirma yöntemleri. [Qualitative research methods]. M. Bütün ve SB Demir, Trans. (Eds.). İstanbul: Siyasal Kitapevi.
8. Cunningham-Nelson, S., Baktashmotlagh, M., ve Boles, W. (2019). Visualizing student opinionthroughtext analysis. IEEE Transactions on Education, 62(4), 305–311.

9. Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., ... ve Wright, R. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642.
10. D'zurilla, T. J., ve Goldfried, M. R. (1971). Problem solving and behavior modification. Journal of Abnormal Psychology, 78(1), 107.
11. Eager, B., ve Brunton, R. (2023). Prompting higher education towards AI-augmented teaching and learning practice. Journal of University Teaching ve Learning Practice, 20(5), 02.
12. Ellis, A. R., ve Slade, E. (2023). A new era of learning: considerations for ChatGPT as a tool to enhance statistics and data science education. Journal of Statistics and Data Science Education, 31(2), 128-133. https://doi.org/10.1080/26939169.2023.2223609
13. English, L., ve Gainsburg, J. (2016). Problem solving in a 21st-century mathematics curriculum. In L. English ve D. Kirshner (Eds.), Handbook of international research in mathematics education (3rd Ed., pp. 313–335). Routledge.
14. Farrokhnia, M., Banihashem, S. K., Noroozi, O., ve Wals, A. (2024). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460-474. https://doi.org/10.1080/14703297.2023.2195846
15. Foster, C. (2018). Developing mathematical fluency: Comparing exercises and rich tasks. Educational Studies in Mathematics, 97(2), 121– 141. https://doi.org/10.1007/s10649-017-9788-x
16. Foster, C. (2021). Problem solving and prior knowledge. Mathematics in School, 50(4), 6–8.
17. Foster, C. (2023). Problem solving in the mathematics curriculum: From domain‐general strategies to domain‐specific tactics. The Curriculum Journal, 34(4), 594-612. https://doi.org/10.1002/curj.213
18. Furlong, J., ve Davies, C. (2015). Young people, new technologies and learning at home: Taking context seriously. In Digital Technologies in the Lives of Young People (pp. 45-62). Routledge.
19. Giray, L. (2023). Prompt engineering with ChatGPT: a guide for academic writers. Annals of biomedical engineering, 51(12), 2629-2633. https://doi.org/10.1007/s10439-023-03272-4
20. Gonzales, N. A. (1994). Problem posing: A neglected component in mathematics courses for prospective elementary and middle school teachers, School Science and Mathematics, 94(2), 78-84. https://doi.org/10.1111/j.1949-8594.1994.tb12295.x
21. Gökkurt, B., Ornek, T., Hayat, F., ve Soylu, Y. (2015). Assessing students’ problem-solving and problem-posing skills. Bartın University Journal of Faculty of Education, 4(2), 751-774.
22. Grabb, D. (2023). The impact of prompt engineering in large language model performance: a psychiatric example. Journal of Medical Artificial Intelligence, 6.
23. Greenhow, C., ve Robelia, B. (2009). Informal learning and identity formation in online social networks. Learning, Media and Technology, 34, 119–140.
24. Guzman Gurat, M. (2018). Mathematical problem-solving heuristics among student teachers. Journal of Efficiency and Responsibility in Education and Science, 11(3), 53-64. https://doi.org/10.7160/eriesj.2018.110302]
25. Hidayat, A., ve Irawan, I. (2017). Pengembangan lks berbasis rme dengan pendekatan problem solving untuk memfasilitasi kemampuan pemecahan masalah matematis siswa. Jurnal Cendekia: Jurnal Pendidikan Matematika, 1(2), 51-63.
26. Hwang, J., ve Riccomini, P. J. (2016). Enhancing mathematical problem solving for secondary students with or at risk of learning disabilities: A literature review. Learning Disabilities Research ve Practice, 31(3), 169-181. https://doi.org/10.1111/ldrp.12105
27. Inoue, N., Asada, T., Maeda, N., ve Nakamura, S. (2019). Deconstructing teacher expertise for inquiry-based teaching: Looking into consensus building pedagogy in Japanese classrooms. Teaching and teacher education, 77, 366-377.
28. Ismail, M., ve Ade-Ibijola, A. (2019, November). Lecturer's apprentice: A chatbot for assisting novice programmers. In 2019 international multidisciplinary information technology and engineering conference (IMITEC) (pp. 1-8). IEEE.
29. Jiang, Z., Xu, F. F., Araki, J., ve Neubig, G. (2020). How can we know what language models know? Transactions of the Association for Computational Linguistics, 8, 423–438. https://doi.org/10.1162/tacl_a_00324
30. Jiao, W., Wang, W., Huang, J. T., Wang, X., ve Tu, Z. (2023). Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745, 1(10). https://doi.org/10.48550/arXiv.2301.08745
31. Kirschner, P. A., Sweller, J., ve Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75– 86. https://doi.org/10.1207/s15326985ep4102_1
32. Kojo, A., Laine, A., ve Näveri, L. (2018). How did you solve it?–Teachers’ approaches to guiding mathematics problem solving. LUMAT: International Journal on Math, Science and Technology Education, 6(1), 22-40.
33. Lai, K. W., Khaddage, F., ve Knezek, G. (2013). Blending student technology experiences in formal and informal learning. Journal of computer assisted learning, 29(5), 414-425.
34. Lancaster, T. (2023). Artificial intelligence, text generation tools and ChatGPT–does digital watermarking offer a solution?. International Journal for Educational Integrity, 19(1), 10. https://doi.org/10.1007/s40979-023-00131-6
35. Lee, E. Y., il, N. G. D., An, G. H., Lee, S., ve Lim, K. (2023, August). ChatGPT-based debate game application utilizing prompt engineering. In Proceedings of the 2023 International Conference on Research in Adaptive and Convergent Systems (pp. 1-6).
36. Lin, X. (2001). Designing metacognitive activities. Educational Technology Research and Development, 49(2), 23–40.
37. Liu, G., ve Ma, C. (2024). Measuring EFL learners’ use of ChatGPT in informal digital learning of English based on the technology acceptance model. Innovation in Language Learning and Teaching, 18(2), 125-138.
38. Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4), 102720.
39. Lock, S.: ‘What is AI chatbot phenomenon ChatGPT and could it replace humans?’, in Editor (Eds.): ‘Book What is AI chatbot phenomenon ChatGPT and could it replace humans?’ (2022, edn.) https://www.theguardian.com/technology/2022/dec/05/what-is-ai-chatbot-phenomenon-chatgpt-and-could-it-replace-humans
40. Mabunda, K., ve Ade-Ibijola, A. (2019, November). Pathbot: An intelligent chatbot for guiding visitors and locating venues. In 2019 6th international conference on soft computing ve machine intelligence (ISCMI) (pp. 160-168). IEEE.
41. Malogianni, C., Luo, T., Stefaniak, J., ve Eckhoff, A. (2021). An exploration of the relationship between argumentative prompts and depth to elicit alternative positions in ill-structured problem solving. Educational Technology Research and Development, 69, 2353-2375.
42. National Council of Teachers of Mathematics [NCTM], (2000). Principles and standards for school mathematics. Reston, VA: NCTM. https://doi.org/10.1109/ISCMI47871.2019.9004411
43. Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., ... and Sandison, C. (2023). ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity. European Journal of Engineering Education, 1-56. https://doi.org/10.1080/03043797.2023.2213169
44. Nurkaeti, N. (2018). Polya’s strategy: an analysis of mathematical problem solving difficulty in 5th grade elementary school. Edu Humanities| Journal of Basic Education Cibiru Campus, 10(2), 140. https://doi.org/10.17509/eh.v10i2.10868
45. Nygren, H., Nissinen, K., Hämäläinen, R., ve De Wever, B. (2019). Lifelong learning: Formal, non‐formal and informal learning in the context of the use of problem‐solving skills in technology‐rich environments. British Journal of Educational Technology, 50(4), 1759-1770.
46. Okonkwo, C.W., ve Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033.
47. OpenAI. (2023a). ChatGPT: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/
48. OpenAI. (2023b). How should AI systems behave, and who should decide? https://openai.com/blog/how-should-ai-systems-behave.
49. Pack, A., ve Maloney, J. (2023). Potential affordances of generative AI in language education: Demonstrations and an evaluative framework. Teaching English with Technology, 23(2), 4-24. https://doi.org/10.56297/BUKA4060/VRRO1747
50. Patton, M. Q. (2014). Nitel araştırma ve değerlendirme yöntemleri (M. Bütün ve S. B. Demir, çev.). Ankara: Pegem Akademi
51. Peranginangin, S. A., ve Surya, E. (2017). An analysis of students’ mathematics problem solving ability in VII grade at smp negeri 4 pancurbatu. International Journal of Sciences: Basic and Applied Research (IJSBAR), 33(2), 57-67.
52. Pimta, S., Tayruakham, S., ve Nuangchalerm, P. (2009). Factors influencing mathematic problem-solving ability of sixth grade students. Journal of School Sciences, 5(4), 381-385.
53. Pólya, G. (1945). How to solve it. Princeton University Press: Princeton
54. Polya, G. (1973). How to solve it. (2nd ed). Princeton, NJ: Princeton University Press
55. Ross, B. H., ve Kennedy, P. T. (1990). Generalizing from the use of earlier examples in problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(1), 42.
56. Rudolph, J., Tan, S., ve Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342- 363. https://doi.org/10.37074/jalt.2023.6.1.9
57. Sahin, A., ve Kulm, G. (2008). Sixth grade mathematics teachers’ intentions and use of probing, guiding, and factual questions. Journal of mathematics teacher education, 11, 221-241.
58. Schoenfeld, A. H. (1985). Mathematical problem solving, Academic.
59. Schroeder, T. L., ve Lester, F. K. (1989). Developing understanding in mathematics via problem solving. New directions for elementary school mathematics, 31, 42.
60. Sinha, S., Basak, S., Dey, Y., Mondal, A. (2020). An Educational Chatbot for Answering Queries. In: Mandal, J., Bhattacharya, D. (eds) Emerging Technology in Modelling and Graphics. Advances in Intelligent Systems and Computing, vol 937. Springer, Singapore. https://doi.org/10.1007/978-981-13-7403-6_7
61. Spasić, A. J., ve Janković, D. S. (2023, June). Using ChatGPT standard prompt engineering techniques in lesson preparation: Role, instructions and seed-word prompts. In 2023 58th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST) (pp. 47-50). IEEE.
62. Stigler, J. W., ve Hiebert, J. (2004). Improving mathematics teaching. Educational Leadership, 61(5), 12–17. Strzelecki, A. (2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments, 1-14.
63. Su, J., ve Yang, W. (2023). Unlocking the power of ChatGPT: A framework for applying generative AI in education. ECNU Review of Education, 20965311231168423. https://doi.org/10.1177/20965311231168423
64. Suharta, I. G. P. (2016). Kemampuan siswa sekolah dasar dalam pemecahan masalah matematika real. Jurnal Pendidikan dan Pengajaran, 49(3), 137-147.
65. Tabone, W., ve De Winter, J. (2023). Using ChatGPT for human–computer interaction research: a primer. Royal Society Open Science, 10(9), 231053. https://doi.org/10.1098/rsos.231053
66. Tricot, A., ve Sweller, J. (2014). Domain-specific knowledge and why teaching generic skills does not work. Educational Psychology Review, 26(2), 265– 283. https://doi.org/10.1007/s10648-013-9243-1
67. Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., ve Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart learning environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
68. Van De Walle, J. A., Karp, K. S., ve Bay-Williams, J. M. (2012). İlkokul ve ortaokul matematiği: Gelişimsel yaklaşımla öğretim (Çev. S. Durmuş). Nobel Yayıncılık.
69. Wang, L., Chen, X., Wang, C., Xu, L., Shadiev, R., ve Li, Y. (2023). ChatGPT's capabilities in providing feedback on undergraduate students’ argumentation: a case study. Thinking Skills and Creativity, 101440.
70. Wang, S., Scells, H., Koopman, B., ve Zuccon, G. (2023). Can ChatGPT write a good boolean query for systematic review literature search?. arXiv preprint arXiv:2302.03495. https://doi.org/10.1145/3539618.3591703
71. Zhai, X. (2022). ChatGPT user experience: Implications for education. Available at SSRN 4312418. https://dx.doi.org/10.2139/ssrn.4312418
72. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., ve Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382. https://doi.org/10.48550/arXiv.2302.11382