The performance of artificial intelligence in the exams of tourist guidance

Year 2024, Volume: 9 Issue: 3, 231 - 243, 31.12.2024

Abdullah Ülkü

https://doi.org/10.31822/jomat.2024-9-3-231

Abstract

The aim of this study is to evaluate the efficiency of ChatGPT versions 3.5 and 4 for training tourist guides. The study followed a systematic approach by conducting assessments on undergraduate students from three institutions who are enrolled in tourist guide education programs and both ChatGPT versions. Competent academicians assessed a succession of questions in the form of open-ended and multiple-choice questions. The mean scores obtained on the multiple-choice test for ChatGPT-4 were better than those of both ChatGPT-3.5 and college students, thereby indicating that ChatGPT-4 has greatly improved. Nevertheless, when responding to open-ended queries, individuals with real-life experience as tour guides gave much more inclusive as well as convincing answers compared to ChatGPT-4. This underscores the importance of hands-on experiences in training tour guides, where AI technology is currently weak. This study contributes to better comprehension regarding the role played by artificial intelligence (AI) in education with reference to the tourism industry specifically. While at the same time emphasizing how critical human expertise is needed during practical learning sessions, this implies that AI has potential for disseminating theoretical knowledge. The results suggest that AI is a beneficial supplementary aid in educational environments, rather than a replacement for human-centered instructional approaches.

---

Keywords

Artificial intelligence, Large language models (LLMs), Tourism education, Tourist guidance, Exam

Ethical Statement

This article was approved by the ethics committee of Harran University. Number of ethics committee approval report: No. 2023/151.

References

• Abduljabbar, R., Dia, H., Liyanage, S., & Bagloee, S. A. (2019). Applications of artificial intelligence in transport: an overview. Sustainability, 11(1), 189. https://doi.org/10.3390/su11010189
• Agarwal, S., Agarwal, B., & Gupta, R. (2022). Chatbots and virtual assistants: A bibliometric analysis. Library Hi Tech, 40(4), 1013–1030. https://doi.org/10.1108/LHT-09-2021-0330
• Alawida, M., Mejri, S., Mehmood, A., Chikhaoui, B., & Isaac Abiodun, O. (2023). A comprehensive study of chatgpt: Advancements, limitations, and ethical considerations in natural language processing and cybersecurity. Information, 14(8), 462. https://doi.org/10.3390/info14080462
• Alazaizeh, M. M., Jamaliah, M. M., Mgonja, J. T., & Ababneh, A. (2019). Tour guide performance and sustainable visitor behavior at cultural heritage sites. Journal of Sustainable Tourism, 27(11), 1708–1724. https://doi.org/10.1080/09669582.2019.1658766
• Ali, K., Barhom, N., Marino, F. T., & Duggal, M. (2023). The thrills and chills of ChatGPT: Implications for assessments in undergraduate dental education [Preprint]. MEDICINE & PHARMACOLOGY. https://doi.org/10.20944/preprints202302.0513.v1
• Athavale, A., Baier, J., Ross, E., & Fukaya, E. (2023). The potential of chatbots in chronic venous disease patient management. JVS-Vascular Insights, 100019. https://doi.org/10.1016/j.jvsvi.2023.100019
• Atlas, S. (2023). ChatGPT for higher education and professional development: A guide to conversational AI. College of Business Faculty Publications. https://digitalcommons.uri.edu/cba_facpubs/548
• Aydın, Ö., & Karaarslan, E. (2023). Is ChatGPT leading generative AI? What is beyond expectations? SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4341500
• Barrot, J. S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57, 100745. https://doi.org/10.1016/j.asw.2023.100745
• Bommarito II, M., & Katz, D. M. (2022). GPT Takes the Bar Exam. arXiv. http://arxiv.org/abs/2212.14402
• Cao, Y., & Zhai, J. (2023). Bridging the gap – the impact of ChatGPT on financial research. Journal of Chinese Economic and Business Studies, 21(2), 177–191. https://doi.org/10.1080/14765284.2023.2212434
• Chang, C.-Y., Kuo, S.-Y., & Hwang, G.-H. (2022). Chatbot-facilitated nursing education: Incorporating a knowledge-based chatbot system into a nursing training program. Educational Technology & Society, 25(1), 15-27.
• Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018). Artificial Intelligence trends in education: A narrative overview. Procedia Computer Science, 136, 16–24. https://doi.org/10.1016/j.procs.2018.08.233
• Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in education: A Review. IEEE Access, 8, 75264–75278. https://doi.org/10.1109/ACCESS.2020.2988510
• Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023). ChatGPT Goes to law school. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4335905
• Chui, K. T., Lytras, M. D., & Visvizi, A. (2018). Energy sustainability in smart cities: Artificial intelligence, smart monitoring, and optimization of energy consumption. Energies, 11(11), 2869. https://doi.org/10.3390/en11112869
• Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20(1), 22. https://doi.org/10.1186/s41239-023-00392-8
• Currie, G., Singh, C., Nelson, T., Nabasenja, C., Al-Hayek, Y., & Spuur, K. (2023). ChatGPT in medical imaging higher education. Radiography, 29(4), 792–799. https://doi.org/10.1016/j.radi.2023.05.011
• Dalalah, D., & Dalalah, O. M. A. (2023). The false positives and false negatives of generative AI detection tools in education and academic research: The case of ChatGPT. The International Journal of Management Education, 21(2), 100822. https://doi.org/10.1016/j.ijme.2023.100822
• Dias, J., Kamdi, D., Gharat, N., & Chudhari, P. (2019). Chatbot for government examination using AI. Journal of Engineering, 9(4).
• Dinesh, A., Kamal, B., Akash, M., & Surthik, K. R. (2023). Implementation of artificial intelligence to the prediction of the mechanical properties of concrete: A review. Materials Today: Proceedings, S221478532302727X. https://doi.org/10.1016/j.matpr.2023.05.101
• Doborjeh, Z., Hemmington, N., Doborjeh, M., & Kasabov, N. (2022). Artificial intelligence: A systematic review of methods and applications in hospitality and tourism. International Journal of Contemporary Hospitality Management, 34(3), 1154–1176. https://doi.org/10.1108/IJCHM-06-2021-0767
• Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A. S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., … Wright, R. (2023). Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
• Essel, H. B., Vlachopoulos, D., Tachie-Menson, A., Johnson, E. E., & Baah, P. K. (2022). The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education. International Journal of Educational Technology in Higher Education, 19(1), 57. https://doi.org/10.1186/s41239-022-00362-6
• Fijacko, N., Gosak, L., Štiglic, G., Picard, C. T., & John Douma, M. (2023). Can ChatGPT pass the life support exams without entering the American heart association course? Resuscitation, 185, 109732. https://doi.org/10.1016/j.resuscitation.2023.109732
• Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). ChatGPT has mastered the principles of economics: Now What? SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4356034
• Ghosh, A., Maini Jindal, N., Gupta, V. K., Bansal, E., Kaur Bajwa, N., & Sett, A. (2023). Is ChatGPT’s knowledge and ınterpretative ability comparable to first professional mbbs (bachelor of medicine, bachelor of surgery) students of ındia in taking a medical biochemistry examination? Cureus. https://doi.org/10.7759/cureus.47329
• Giannos, P., & Delardas, O. (2023). Performance of ChatGPT on UK standardized admission tests: Insights from the BMAT, TMUA, LNAT, and TSA Examinations. JMIR Medical Education, 9, e47737. https://doi.org/10.2196/47737
• Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022). How does chatgpt perform on the medical licensing exams? the ımplications of large language models for medical education and knowledge assessment [Preprint]. Medical Education. https://doi.org/10.1101/2022.12.23.22283901
• Gimpel, H., Hall, K., Decker, S., Eymann, T., Lämmermann, L., Mädche, A., Röglinger, M., Ruiner, C., Schoch, M., Schoop, M., Urbach, N., & Vandirk, S. (2023). Unlocking the power of generative aı models and systems such as gpt-4 and chatgpt for higher education. Hohenheim Discussion Papers in Business, Economics and Social Sciences. https://doi.org/10.13140/RG.2.2.20710.09287/2
• Hartmann, J., Schwenzow, J., & Witte, M. (2023). The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4316084
• Ho, W. L. J., Koussayer, B., & Sujka, J. (2023). ChatGPT: Friend or foe in medical writing? An example of how ChatGPT can be utilized in writing case reports. Surgery in Practice and Science, 14, 100185. https://doi.org/10.1016/j.sipas.2023.100185
• Iqbal, M. Z., & Faiz, M. F. I. (2020). Active surveillance for Covıd-19 through artificial intelligence using real-time speech-recognition mobile application. 2020 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan), 1–2. https://doi.org/10.1109/ICCE-Taiwan49838.2020.9258276
• Ivanov, S., & Soliman, M. (2023). Game of algorithms: ChatGPT implications for the future of tourism education and research. Journal of Tourism Futures, 9(2), 214–221. https://doi.org/10.1108/JTF-02-2023-0038
• Jenneboer, L., Herrando, C., & Constantinides, E. (2022). The impact of chatbots on customer loyalty: a systematic literature review. Journal of Theoretical and Applied Electronic Commerce Research, 17(1), 212–229. https://doi.org/10.3390/jtaer17010011
• Jin, J. Q., & Dobry, A. S. (2023). ChatGPT for healthcare providers and patients: Practical implications within dermatology. Journal of the American Academy of Dermatology, S0190962223011064. https://doi.org/10.1016/j.jaad.2023.05.081
• Keiper, M. C., Fried, G., Lupinek, J., & Nordstrom, H. (2023). Artificial intelligence in sport management education: Playing the AI game with ChatGPT. Journal of Hospitality, Leisure, Sport & Tourism Education, 33, 100456. https://doi.org/10.1016/j.jhlste.2023.100456
• Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and learning. RELC Journal, 003368822311628. https://doi.org/10.1177/00336882231162868
• Kooli, C. (2023). Chatbots in education and research: A critical examination of ethical implications and solutions. Sustainability, 15(7), 5614. https://doi.org/10.3390/su15075614
• Kortemeyer, G. (2023). Could an Artificial-Intelligence agent pass an introductory physics course? Physical Review Physics Education Research, 19(1), 010132. https://doi.org/10.1103/PhysRevPhysEducRes.19.010132
• Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. https://doi.org/10.1371/journal.pdig.0000198
• Kutyauripo, I., Rushambwa, M., & Chiwazi, L. (2023). Artificial intelligence applications in the agrifood sectors. Journal of Agriculture and Food Research, 11, 100502. https://doi.org/10.1016/j.jafr.2023.100502
• Lee, D., & Yoon, S. N. (2021). Application of artificial ıntelligence-based technologies in the healthcare industry: Opportunities and Challenges. International Journal of Environmental Research and Public Health, 18(1), 271. https://doi.org/10.3390/ijerph18010271
• Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., & Eger, S. (2023). ChatGPT: A Meta-Analysis after 2.5 Months. arXiv. http://arxiv.org/abs/2302.13795
• Leung, X. Y., & Wen, H. (2020). Chatbot usage in restaurant takeout orders: A comparison study of three ordering methods. Journal of Hospitality and Tourism Management, 45, 377–386. https://doi.org/10.1016/j.jhtm.2020.09.004
• Li, L. (2023). Role of chatbots on gastroenterology: Let’s chat about the future. Gastroenterology & Endoscopy, S2949752323000341. https://doi.org/10.1016/j.gande.2023.06.002
• Li, M., & Wang, R. (2023). Chatbots in e-commerce: The effect of chatbot language style on customers’ continuance usage intention and attitude toward brand. Journal of Retailing and Consumer Services, 71, 103209. https://doi.org/10.1016/j.jretconser.2022.103209
• Li, W., Wang, J., Yan, Y., Yuan, P., Cao, C., Li, S., & Wu, Q. (2023). Potential applications of ChatGPT in endoscopy: Opportunities and limitations. Gastroenterology & Endoscopy, S294975232300033X. https://doi.org/10.1016/j.gande.2023.06.001
• Lin, J. C., Younessi, D. N., Kurapati, S. S., Tang, O. Y., & Scott, I. U. (2023). Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye. https://doi.org/10.1038/s41433-023-02564-2
• Lin, X., Shao, B., & Wang, X. (2022). Employees’ perceptions of chatbots in B2B marketing: Affordances vs. disaffordances. Industrial Marketing Management, 101, 45–56. https://doi.org/10.1016/j.indmarman.2021.11.016
• Lin, Y., & Yu, Z. (2023). A bibliometric analysis of artificial intelligence chatbots in educational contexts. Interactive Technology and Smart Education. https://doi.org/10.1108/ITSE-12-2022-0165
• Luo, B., Lau, R. Y. K., & Li, C. (2023). Emotion-regulatory chatbots for enhancing consumer servicing: An interpersonal emotion management approach. Information & Management, 60(5), 103794. https://doi.org/10.1016/j.im.2023.103794
• Ma, Y. (2023). The potential application of ChatGPT in gastrointestinal pathology. Gastroenterology & Endoscopy, 1(3), 130–131. https://doi.org/10.1016/j.gande.2023.05.002
• Melián-González, S., Gutiérrez-Taño, D., & Bulchand-Gidumal, J. (2021). Predicting the intentions to use chatbots for travel and tourism. Current Issues in Tourism, 24(2), 192–210. https://doi.org/10.1080/13683500.2019.1706457
• Michalon, B., & Camacho-Zuñiga, C. (2023). ChatGPT, a brand-new tool to strengthen timeless competencies. Frontiers in Education, 8, 1251163. https://doi.org/10.3389/feduc.2023.1251163
• Naser, M. Z., Ross, B., Ogle, J., Kodur, V., Hawileh, R., Abdalla, J., & Thai, H.-T. (2023). Can AI Chatbots Pass the Fundamentals of Engineering (FE) and Principles and Practice of Engineering (PE) Structural Exams? arXiv. https://doi.org/10.48550/ARXIV.2303.18149
• Newton, P. M., & Xiromeriti, M. (2023). Chatgpt performance on mcq exams in higher education. a pragmatic scoping review [Preprint]. EdArXiv. https://doi.org/10.35542/osf.io/sytu3
• Nunes, D., Primi, R., Pires, R., Lotufo, R., & Nogueira, R. (2023). Evaluating Gpt-3.5 and Gpt-4 models on brazilian university admission exams (arXiv:2303.17003). arXiv. http://arxiv.org/abs/2303.17003
• Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
• OpenAI. (n.d.). GPT-4. OpenAI Research. https://openai.com/research/gpt-4
• OpenAIb. (n.d.). About OpenAI. https://openai.com/about
• Orden-Mejía, M. A., & Huertas, A. (2022). Tourist interaction and satisfaction with the chatbot evokes pre-visit destination image formation? A case study. Anatolia, 1–15. https://doi.org/10.1080/13032917.2022.2072918
• Ouyang, F., & Jiao, P. (2021). Artificial intelligence in education: The three paradigms. Computers and Education: Artificial Intelligence, 2, 100020. https://doi.org/10.1016/j.caeai.2021.100020
• Owan, V. J., Abang, K. B., Idika, D. O., Etta, E. O., & Bassey, B. A. (2023). Exploring the potential of artificial intelligence tools in educational measurement and assessment. Eurasia Journal of Mathematics, Science and Technology Education, 19(8), em2307. https://doi.org/10.29333/ejmste/13428
• Parsons, H., Houge Mackenzie, S., & Filep, S. (2019). Facilitating self-development: How tour guides broker spiritual tourist experiences. Tourism Recreation Research, 44(2), 141–152. https://doi.org/10.1080/02508281.2019.1582159
• Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549–1565. https://doi.org/10.1002/cae.22326
• Peters, M. A., Jackson, L., Papastephanou, M., Jandric, P., Lazaroiu, G., Evers, C. W., Cope, B., Kalantzis, M., Araya, D., Tesar, M., Mika, C., Chen, L., Wang, C., Sturm, S., Rider, S., & Fuller, S. (2023). AI and the future of humanity: ChatGPT-4, philosophy and education – Critical responses. Educational Philosophy and Theory, 1–35. https://doi.org/10.1080/00131857.2023.2213437
• Pikhart, M. (2020). Intelligent information processing for language education: The use of artificial intelligence in language learning apps. Procedia Computer Science, 176, 1412–1419. https://doi.org/10.1016/j.procs.2020.09.151
• Pursnani, V., Sermet, Y., & Demir, I. (2023). Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice. ArXiv. https://doi.org/10.48550/ARXIV.2304.12198
• Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9
• Short, C. E., & Short, J. C. (2023). The artificially intelligent entrepreneur: ChatGPT, prompt engineering, and entrepreneurial rhetoric creation. Journal of Business Venturing Insights, 19, e00388. https://doi.org/10.1016/j.jbvi.2023.e00388
• Singh, S. V., & Hiran, K. K. (2022). The Impact of AI on Teaching and learning in higher education technology. Journal of Higher Education Theory and Practice, 22(13). https://doi.org/10.33423/jhetp.v22i13.5514
• Skavronskaya, L., Hadinejad, A. (Hana), & Cotterell, D. (2023). Reversing the threat of artificial intelligence to opportunity: A discussion of ChatGPT in tourism education. Journal of Teaching in Travel & Tourism, 23(2), 253–258. https://doi.org/10.1080/15313220.2023.2196658
• Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862. https://doi.org/10.1016/j.compedu.2020.103862
• Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752
• Susnjak, T. (2022). ChatGPT: The End of Online Exam Integrity? (arXiv:2212.09292). arXiv. http://arxiv.org/abs/2212.09292
• Tsaur, S.-H., & Teng, H.-Y. (2017). Exploring tour guiding styles: The perspective of tour leader roles. Tourism Management, 59, 438–448. https://doi.org/10.1016/j.tourman.2016.09.005
• TUREB. (n.d.-a). Şanlıurfa Regional Certificate Program General Culture Exam Questions. https://www.tureb.org.tr/tr/Page/Detail/93
• TUREB. (n.d.-b). Trabzon Regional Certificate Program General Culture Exam Questions. https://www.tureb.org.tr/tr/Page/Detail/92
• Tyson, M. M., & Sauers, N. J. (2021). School leaders’ adoption and implementation of artificial intelligence. Journal of Educational Administration, 59(3), 271–285. https://doi.org/10.1108/JEA-10-2020-0221
• Vasconcelos, M. A. R., & Dos Santos, R. P. (2023). Enhancing STEM learning with ChatGPT and Bing Chat as objects to think with: A case study. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), em2296. https://doi.org/10.29333/ejmste/13313
• Victor, B. G., Kubiak, S., Angell, B., & Perron, B. E. (2023). Time to move beyond the aswb licensing exams: Can generative artificial ıntelligence offer a way forward for social work? Research on Social Work Practice, 33(5), 511–517. https://doi.org/10.1177/10497315231166125
• Wang, C., Li, Y., Fu, W., & Jin, J. (2023). Whether to trust chatbots: Applying the event-related approach to understand consumers’ emotional experiences in interactions with chatbots in e-commerce. Journal of Retailing and Consumer Services, 73, 103325. https://doi.org/10.1016/j.jretconser.2023.103325
• Wang, F., Wong, W.-K., Reivan Ortiz, G. G., Shraah, A. A., Mabrouk, F., Li, J., & Li, Z. (2023). Economic analysis of sustainable exports value addition through natural resource management and artificial intelligence. Resources Policy, 82, 103541. https://doi.org/10.1016/j.resourpol.2023.103541
• Xu, Y., Niu, N., & Zhao, Z. (2023). Dissecting the mixed effects of human-customer service chatbot interaction on customer satisfaction: An explanation from temporal and conversational cues. Journal of Retailing and Consumer Services, 74, 103417. https://doi.org/10.1016/j.jretconser.2023.103417
• Zaitsu, W., & Jin, M. (2023). Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis.
• Zhang, C., & Lu, Y. (2021). Study on artificial intelligence: The state of the art and future prospects. Journal of Industrial Information Integration, 23, 100224. https://doi.org/10.1016/j.jii.2021.100224
• Zhu, D., & Xu, H. (2021). Guides’ handling of multiple roles in interactions with tourists: The prism of impression management. Tourism Management, 85, 104306. https://doi.org/10.1016/j.tourman.2021.104306