GPT vs. Google: A comparative study of self-code learning in ODL students

Abstract

This research investigates the comparative effectiveness of the ChatGPT and the Google search engine in facilitating the self-learning of JavaScript functions among undergraduate open and distance learning students. The study employed a quasi-experimental post-test control group design to analyze the variables of disorientation, satisfaction, cognitive load, navigation performance, and test scores among 471 learners. Results indicated a marked advantage for ChatGPT users in reducing cognitive load and disorientation, improving navigation performance and increasing test scores. The study also explores the evolving dynamics between conventional keyword-based search engines and AI-driven conversational models, reflecting on their mutual influence in the domain of information retrieval. These findings suggest that AI chatbots like ChatGPT not only enhance the learning experience but also signify a paradigm shift in how information is accessed and processed in educational settings. The research contributes to understanding AI applications in education, particularly in self-directed learning scenarios in open and distance learning.

Keywords

• Open and distance learning
• artificial intelligence
• cognitive load
• disorientation
• navigation performance

References

1. Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. In IFIP international conference on artificial intelligence applications and innovations (pp. 373-383). Springer, Cham
2. Al Afnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). Chatgpt as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology, 3(2), 60-68.
3. Ali, J. K. M., Shamsan, M. A. A., Hezam, T. A., & Mohammed, A. A. (2023). Impact of ChatGPT on learning motivation: teachers and students’ voices. Journal of English Studies in Arabia Felix, 2(1), 41-49.
4. Alseddiqi, M., Anwar, A. M., Albalooshi, L., & Najam, O. (2023). Revolutionizing Online Learning: The Potential of ChatGPT in Massive Open Online Courses. European Journal of Education and Pedagogy, 4(4), 1-5.
5. Alshahrani, A. (2023). The impact of ChatGPT on blended learning: Current trends and future research directions. International Journal of Data and Network Science, 7(4), 2029-2040.
6. Amadieu, F., Van Gog, T., Paas, F., Tricot, A., & Mariné, C. (2009). Effects of prior knowledge and concept-map structure on disorientation, cognitive load, and learning. Learning and instruction, 19(5), 376-386.
7. Arcila, B. B. (2023). Is It a Platform? Is It a Search Engine? It’s ChatGPT! The European Liability Regime for Large Language Models. J. Free Speech L., 3, 455.
8. Ayoub, N. F., Lee, Y. J., Grimm, D., & Divi, V. (2023). Head‐to‐Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition. Otolaryngology–Head and Neck Surgery.

9. Balasubramanian, K. (2023). Transforming autodidactic experiences with ChatGPT – new challenges in teaching-learning. Thiagarajar College of Preceptors Edu Spectra, 5(1), 62-66. https://doi.org/10.34293/eduspectra.v5is1-may23.011
10. Barber, M., Bird, L., Fleming, J., Titterington-Giles, E., Edwards, E., & Leyland, C. (2021). Gravity assist: Propelling higher education towards a brighter future. Report of the digital learning and teaching review. London: Office for Students.
11. Biswas, S. (2023). Role of Chat GPT in Education. Available at SSRN 4369981.
12. Chen, E., & Tseng, Y. H. (2022, April). A Decision Model for Designing NLP Applications. In Companion Proceedings of the Web Conference 2022 (pp. 1206-1210).
13. Dan, O., & Davison, B. D. (2016). Measuring and predicting search engine users’ satisfaction. ACM Computing Surveys (CSUR), 49(1), 1-35.
14. Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2023). Impact of AI assistance on student agency. Computers & Education, 104967.
15. Firat, M., & Kabakçi, I. (2010). Use of visual metaphors for navigation in educational hypermedia: Effects on the navigational performance. Journal of Educational Multimedia and Hypermedia, 19(1), 5-22.
16. Firat, M., & Kabakci Yurdakul, I. (2016). The development of Hyper-MNP: Hyper-media navigational performance scale. Education and Information Technologies, 21, 149-169. https://doi.org/10.1007/s10639-014-9313-0
17. Firat, M. (2023). How Chat GPT Can Transform Autodidactic Experiences and Open Education?. https://doi.org/10.31219/osf.io/9ge8m
18. Firdous, F., Bashir, S., Rufai, S. Z., & Kumar, S. (2023, July). OpenAI ChatGPT as a Logical Interpreter of code. In 2023 2nd International Conference on Edge Computing and Applications (ICECAA) (pp. 1192-1197). IEEE.
19. Frederiksen, J. G., Sørensen, S. M. D., Konge, L., Svendsen, M. B. S., Nobel-Jørgensen, M., Bjerrum, F., & Andersen, S. A. W. (2020). Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: A randomized trial. Surgical Endoscopy, 34(3), 1244–1252.
20. Gatian, A. W. (1994). Is user satisfaction a valid measure of system effectiveness? Information & Management, 26(3), 119-131.
21. Guo, K., Zhong, Y., Li, D., & Chu, S. K. W. (2023). Effects of chatbot-assisted in-class debates on students’ argumentation skills and task motivation. Computers & Education, 104862.
22. Gwizdka, J. (2010). Distribution of cognitive load in web search. Journal of the American Society for Information Science and Technology, 61(11), 2167-2187.
23. Haleem, A., Javaid, M., & Singh, R. P. (2022). An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil transactions on benchmarks, standards and evaluations, 2(4), 100089.
24. HGS (2023). What is Google Gemini? How to use it? Features and Limitations. https://hgs.cx/blog/what-is-google-gemini-how-to-use-it-features-and-limitations/
25. Hristidis, V., Ruggiano, N., Brown, E. L., Ganta, S. R. R., & Stewart, S. (2023). ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results. Journal of Medical Internet Research, 25, e48966.
26. Kalla, D., & Smith, N. (2023). Study and Analysis of Chat GPT and its Impact on Different Fields of Study. International Journal of Innovative Science and Research Technology, 8(3).
27. Karadağ, N. (2023). The impact of artificial intelligence on online assessment: A preliminary review. Journal of Educational Technology and Online Learning, 6(4), 822-837.
28. Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and learning. Relc Journal, 54(2), 537-550.
29. Levene, M. (2011). Search engines: information retrieval in practice. The Computer Journal, 54(5), 831-832.
30. Lin, X. (2023). Exploring the Role of ChatGPT as a Facilitator for Motivating Self-Directed Learning Among Adult Learners. Adult Learning, 10451595231184928.
31. Liu, H. Y., Bonetti, M. A., Jeong, T., Pandya, S., Nguyen, V. T., & Egro, F. M. (2023). Dr. ChatGPT will see you now: How do Google and ChatGPT compare in answering patient questions on breast reconstruction?. Journal of Plastic, Reconstructive & Aesthetic Surgery, 85, 488-497.
32. Megahed, F. M., Chen, Y. J., Ferris, J. A., Knoth, S., & Jones-Farmer, L. A. (2023). How generative ai models such as chatgpt can be (mis) used in spc practice, education, and research? an exploratory study. Quality Engineering, 1-29. Doi: 10.1080/08982112.2023.2206479
33. Mracek, D. (2019, July). The Google search engine: A blended-learning tool for student empowerment. In 2019 International Symposium on Educational Technology (ISET) (pp. 224-229). IEEE.
34. Naidu, K., & Sevnarayan, K. (2023). ChatGPT: An ever-increasing encroachment of artificial intelligence in online assessment in distance education. Online Journal of Communication and Media Technologies, 13(3), e202336.
35. Naseer, S., Rashid, U., Saddal, M., Khan, A. R., Abbas, Q., & Daadaa, Y. (2023). WSREB Mechanism: Web Search Results Exploration Mechanism for Blind Users. Applied Sciences, 13(19), 11007.
36. OpenAI. (2023). Chat GPT. Retrieved from https://openai.com/blog/chatgpt/ on January 2 2023.
37. OpenAI. (2023). ChatGPT: Optimizing language models for dialogue. OpenAI. Retrieved January 14, 2023, from https:// openai.com/blog/chatgpt
38. Popovici, M. D. (2023). ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course. International Journal of Human–Computer Interaction, 1-12. https://doi.org/10.1080/10447318.2023.2269006
39. Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783. https://doi.org/10.3390/app13095783
40. Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law and human behavior, 29, 615-620.
41. Ruxton, G. D. (2006). The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behavioral Ecology, 17(4), 688-690.
42. Sakib, F. A., Khan, S. H., & Karim, A. H. M. (2023). Extending the frontier of chatgpt: Code generation and debugging. arXiv preprint arXiv:2307.08260.
43. Shoufan, A. (2023). Exploring Students’ Perceptions of CHATGPT: Thematic Analysis and Follow-Up Survey. IEEE Access. DOI: 10.1109/ACCESS.2023.3268224
44. Singh, S. K., Kumar, S., & Mehra, P. S. (2023, June). Chat GPT & Google Bard AI: A Review. In 2023 International Conference on IoT, Communication and Automation Technology (ICICAT) (pp. 1-6). IEEE.
45. Skulmowski, A., & Xu, K. M. (2022). Understanding cognitive load in digital and online learning: A new perspective on extraneous cognitive load. Educational psychology review, 34(1), 171-196.
46. Tonbuloğlu, B. (2023). An evaluation of the use of artificial intelligence applications in online education. Journal of Educational Technology and Online Learning, 6(4), 866-884.
47. Thu, C. H., Bang, H. C., & Cao, L. (2023). Integrating ChatGPT into Online Education System in Vietnam: Opportunities and Challenges.
48. Van Wyk, M. M., Adarkwah, M. A., & Amponsah, S. (2023). Why all the hype about chatGPT?: Academics’ views of a chat-based conversational learning strategy at an open distance e-learning institution. Open Praxis, 15(3), 214-225.
49. Varghese, R. M. (2020). Is ChatGPT a Game Changer for Higher Education in India?. Special Issue of ‘University News’, 61, 16.
50. Washif, J., Pagaduan, J., James, C., Dergaa, I., & Beaven, C. (2023). Artificial intelligence in sport: Exploring the potential of using ChatGPT in resistance training prescription. Biology of Sport, 41(2), 209-220.
51. Welch BL. 1947. The generalization of students problem when several different population variances are involved. Biometrika34:23–35
52. Wu, T. T., Lee, H. Y., Li, P. H., Huang, C. N., & Huang, Y. M. (2023). Promoting self-regulation progress and knowledge construction in blended learning via ChatGPT-based learning aid. Journal of Educational Computing Research, 07356331231191125.
53. Xu, R., Feng, Y., & Chen, H. (2023). Chatgpt vs. google: A comparative study of search performance and user experience. arXiv preprint arXiv:2307.01135.
54. Yilmaz, R., & Yilmaz, F. G. K. (2023). Augmented intelligence in programming learning: Examining student views on the use of ChatGPT for programming learning. Computers in Human Behavior: Artificial Humans, 1(2), 100005. https://doi.org/10.1016/j.chbah.2023.100005
55. Zhao, R., Li, X., Chia, Y. K., Ding, B., & Bing, L. (2023). Can chatgpt-like generative models guarantee factual accuracy? on the mistakes of new generation search engines. arXiv preprint arXiv:2304.11076.
56. Zhu, J. J., Jiang, J., Yang, M., & Ren, Z. J. (2023). ChatGPT and environmental research. Environmental Science & Technology.