Understanding University Students' Intentions to Use Chatbots in Computer Programming Education: A Quantitative Study

Abstract

Recently, the use of artificial intelligence in education is one of the more frequently discussed issues by researchers. Especially the use of artificial intelligence applications called chatbots or artificial intelligence language models in education is increasing day by day. Although the use of chatbots in education is possible for every course, it is observed that students intend to use chatbots as an assistant, an instructor, or a guide, especially in computer programming courses, which are difficult to learn and have complex structures. In this context, the aim of this study is to examine the intention of university students taking computer programming courses to use chatbots in their education. The participants consisted of 413 university students studying at a state university and taking a "computer programming" course in the 2023-2024 academic year. Descriptive statistics, independent sample t-test and one-way analysis of variance (ANOVA) were used to analyse the data. Results have shown that university students indeed possess intentions to use chatbots in computer programming education and this intention is mostly motivated by the performance expectation towards the course. In addition, for the purposes of using chatbots in computer programming education; conceptual understanding, identifying errors in program code and looking up for correct syntactical rules were among the most prominent reasons. In terms of independent variables, while there was no difference in terms of department, income status, device using chatbots, and the role attributed to chatbot by the student, a significant difference was found in terms of gender, grade level, use experience and frequency of chatbots use. According to the study, university students studying programming plan to use chatbots powered by artificial intelligence, and the usage of chatbots in programming education is expected to grow over time in tandem with the advancement of AI technology.

Keywords

• Artificial Intelligence
• AI
• Chatbot
• Higher Education
• Survey Research

Ethical Statement

It was found ethically appropriate with the decision numbered 01/23 taken at the meeting of Trakya University Social Sciences and Humanities Research Ethics Committee dated 24.01.2024.(Number: E-29563864-050.04-589075)

References

1. Aktay, S., Gök. S., & Uzunoğlu, D. (2023). ChatGPT in Education. Türk Akademik Yayınlar Dergisi (TAY Journal), 7(2), 378-406.
2. Biswas, S. (2023). Role of ChatGPT in Computer Programming: ChatGPT in Computer Programming. Mesopotamian Journal of Computer Science, 2023, 8-16.
3. Buyukozturk, Ş., Kılıc-Cakmak, E., Akgun, O., Karadeniz, S., & Demirel, F. (2008). Eğitimde Bilimsel Araştırma Yöntemleri, (1. Basım), Pegem Akademi: Ankara.
4. Chan, C. K. Y., & Hu, W. (2023). Students' Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Education. arXiv preprint arXiv:2305.00290.
5. Clark, D. (2018, December 17). 10 uses for Chatbots in learning (with examples) [Blog post]. Retrieved from http://donaldclarkplanb.blogspot.com/2017/12/10-uses-for-chatbots-in-learning-with.html.
6. Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., & Lemma, S. (2018). Chatbot for e-learning: A case of study. International Journal of Mechanical Engineering and Robotics Research, 7(5), 528-533.
7. Draxler, F., Buschek, D., Tavast, M., Hämäläinen, P., Schmidt, A., Kulshrestha, J., & Welsch, R. (2023). Gender, Age, and Technology Education Influence the Adoption and Appropriation of LLMs. arXiv preprint arXiv:2310.06556.
8. Dwivedi, Y. K., Rana, N. P., Chen, H., & Williams, M. D. (2011). A Meta-analysis of the Unified Theory of Acceptance and Use of Technology (UTAUT). In Governance and Sustainability in Information Systems. Managing the Transfer and Diffusion of IT: IFIP WG 8.6 International Working Conference, Hamburg, Germany, September 22-24, 2011. Proceedings (pp. 155-170). Springer Berlin Heidelberg.

9. Fryer, L. K., Thompson, A., Nakao, K., Howarth, M., & Gallacher, A. (2020). Supporting self-efficacy beliefs and interest as educational inputs and outcomes: Framing AI and Human partnered task experiences. Learning and Individual Differences, 80, 101850.
10. Gill, S. S., & Kaur, R. (2023). ChatGPT: Vision and challenges. Internet of Things and Cyber-Physical Systems, 3, 262-271.
11. Holmes, W., Bialik, M., & Fadel, C. (2023). Artificial intelligence in education. Globethics Publications.
12. Huallpa, J. J. (2023). Exploring the ethical considerations of using Chat GPT in university education. Periodicals of Engineering and Natural Sciences, 11(4), 105-115.
13. Iqbal, S. M., Mathew, R., Tawafak, R. M.,& Alfarsi, G.(2021, July). A web-based model to enhance algorithmic thinking for novice programmers. E-Learning and Digital Media,18(6), 616–633.
14. Keles (2023, November). “Chat GPT Nedir? Chat GPT Nasıl Kullanılır?”. https://www.ticimax.com/blog/chat-gpt-nedir. Access date: 20 December 2023.
15. Kerlyl, A., Hall, P., & Bull, S. (2006, December). Bringing chatbots into education: Towards natural language negotiation of open learner models. In International conference on innovative techniques and applications of artificial intelligence (pp. 179-192). London: Springer London.
16. Kim, J. W., Jo, H. I., & Lee, B. G. (2019). The study on the factors influencing on the behavioral intention of chatbots service for the financial sector: Focusing on the UTAUT model. Journal of Digital Contents Society, 20(1), 41-50.
17. Luo, B., Lau, R. Y., Li, C., & Si, Y. W. (2022). A critical review of state‐of‐the‐art chatbots designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1), e1434.
18. Malik, S. I., Ashfque, M. W., Tawafak, R. M., Al-Farsi, G., Usmani, N. A., & Khudayer, B. H. (2022). A Chatbot to Facilitate Student Learning in a Programming 1 Course: A Gendered Analysis. International Journal of Virtual and Personal Learning Environments (IJVPLE), 12(1), 1-20.
19. McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B. D., ... & Wilusz, T. (2001). A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. Working group reports from ITiCSE on Innovation and technology in computer science education (pp. 125-180).
20. Mokmin, N. A. M., & Ibrahim, N. A. (2021). The evaluation of chatbots as a tool for health literacy education among undergraduate students. Education and Information Technologies, 26(5), 6033-6049.
21. Moon, J., Yang, R., Cha, S., & Kim, S. B. (2023, August). ChatGPT vs Mentor: Programming Language Learning Assistance System for Beginners. In 2023 IEEE 8th International Conference on Software Engineering and Computer Systems (ICSECS) (pp. 106-110). IEEE.
22. Neumann, A. T., Arndt, T., Köbis, L., Meissner, R., Martin, A., de Lange, P., ... & Wollersheim, H. W. (2021). Chatbots as a tool to scale mentoring processes: Individually supporting self-study in higher education. Frontiers in artificial intelligence, 4, 668220.
23. Okonkwo, C. W., & Ade-Ibijola, A. (2020). Python-Bot: A chatbots for teaching python programming. Engineering Letters, 29(1).
24. Qian, Y., & Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1), 1-24.
25. Philbin, C. A. (2023). Exploring the Potential of Artificial Intelligence Program Generators in Computer Programming Education for Students. ACM Inroads, 14(3), 30-38.
26. Raffaghelli, J. E., Rodríguez, M. E., Guerrero-Roldán, A. E., & Baneres, D. (2022). Applying the UTAUT model to explain the students' acceptance of an early warning system in Higher Education. Computers & Education, 182, 104468.
27. Ragheb, M. A., Tantawi, P., Farouk, N., & Hatata, A. (2022). Investigating the acceptance of applying chat-bot (Artificial intelligence) technology among higher education students in Egypt. International Journal of Higher Education Management, 8(2).
28. Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783.
29. Singh, R. (2018, May 2). AI and Chatbots in Education: What Does The Future Hold? [Blog post] Retrieved from https://chatbotssmagazine.com/ai-and-chatbots-in-education-what-does-the-futurehold-9772f5c13960.
30. Surameery, N. M. S., & Shakor, M. Y. (2023). Use chat gpt to solve programming bugs. International Journal of Information Technology & Computer Engineering (IJITC), 3(01), 17-22.
31. Şişman, B., & Küçük, S. (2018). Öğretmen adaylarının robotik programlamada akış, kaygı ve bilişsel yük seviyeleri. Eğitim teknolojisi kuram ve uygulama, 8(2), 125-156.
32. Teo, T., Doleck, T., Bazelais, P., & Lemay, D. J. (2019). Exploring the drivers of technology acceptance: a study of Nepali school students. Educational Technology Research and Development, 67, 495-517.
33. Topal, A. D., Eren, C. D., & Geçer, A. K. (2021). Chatbot application in a 5th grade science course. Education and Information Technologies, 26, 6241–6265.
34. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS quarterly, 425-478.
35. Verleger, M., & Pembridge, J. (2018, October). A pilot study integrating an AI-driven chatbots in an introductory programming course. In 2018 IEEE frontiers in education conference (FIE) (pp. 1-4). IEEE.
36. West M., Kraut, R. & Han, Ei. C. (2019). I’d blush if I could: closing gender divides in digital skills through education. Technical Report. UNESCO, unesdoc.unesco.org/ark:/48223/pf0000367416.
37. Vukojičić, M., & Krstić, J. (2023). ChatGPT in programming education: ChatGPT as a programming assistant. InspirED Teachers' Voice, 2023(1), 7-13.
38. Williams, M. D., Rana, N. P., & Dwivedi, Y. K. (2015). The unified theory of acceptance and use of technology (UTAUT): a literature review. Journal of enterprise information management, 28(3), 443-488.
39. Yildiz-Durak, H. (2023). Conversational agent-based guidance: examining the effect of frequency and satisfaction on visual design self-efficacy, engagement, satisfaction, and learner autonomy. Education and Information Technologies, 28(1), 471-488.
40. Yildiz-Durak, H., & Onan, A. (2023). Adaptation of Behavioral Intention to Use and Learn Chatbot in Education Scale into Turkish. Journal of Ahmet Keleşoğlu Education Faculty, 5(3), 1162-1172.
41. Yilmaz, R., & Yilmaz, F. G. K. (2023). Augmented intelligence in programming learning: Examining student views on the use of ChatGPT for programming learning. Computers in Human Behavior: Artificial Humans, 1(2), 100005.