EkoBot: Intelligent Virtual Academic Advisor with Turkish Support

Abstract

In this study, EkoBot, an AI-powered software with Turkish language support, designed to assist university students as a virtual advisor, is presented. The goal of this software is to provide students with accurate and prompt answers to their questions. For this purpose, large language models supported by artificial intelligence were utilized. To enable the large language model to give responses based on university regulations, which were not included in its training, these documents were provided to the model externally, and a “retrieval-augmented generation” method was employed. To evaluate the performance of the proposed system, 100 questions with open-ended or positive and negative responses were generated. In the retrieval phase, 100% success was achieved with the five context texts most similar to the question. In the generation phase, the similarity between the responses and the context was found to be 0.82. Additionally, a prototype of the proposed solution was developed as a Web page and made available for the students.

Keywords

• Natural language processing
• cosine similarity
• prompt engineering
• large language models
• artificial intelligence
• retrieval augmented generation

EkoBot: Türkçe Destekli Akıllı Sanal Akademik Danışman

Abstract

Bu çalışmada üniversite öğrencilerine sanal danışman olarak yardımcı olabilecek, Türkçe destekli bir akıllı yazılım, EkoBot sunulmuştur. Bu yazılımla öğrencilerin sorularına doğru ve hızlı bir şekilde yanıt almaları hedeflenmiştir. Bunun için yapay zekâ destekli büyük dil modellerinden yararlanılmıştır. Büyük dil modelinin, eğitimi sırasında kullanılmamış, üniversite yönetmeliklerine dayalı yanıtlar verebilmesi için bu belgeler modele dışardan verilmiş ve “almayla artırılmış üretim” yöntemi kullanılmıştır. Önerilen sistemin performansını ölçmek için ucu açık ya da olumlu ve olumsuz yanıtlara sahip 100 adet soru üretilmiştir. Alma kısmında, soruya en çok benzeyen beş bağlam metni ile %100 başarım elde edilmiştir. Üretme kısmında, yanıt ile bağlam benzerlikleri 0,82 olarak bulunmuştur. Ayrıca önerilen çözümün bir Web sayfası olarak çalışan bir prototipi hazırlanmış ve öğrencilerin kullanımına sunulmuştur.

Keywords

• Doğal dil işleme
• kosinüs benzerliği
• istem mühendisliği
• büyük dil modelleri
• yapay zekâ
• almayla artırılmış üretim

References

1. Ayanouz, S., Abdelhakim, B. A., & Benhmed, M. (2020). A Smart Chatbot Architecture based NLP and Machine Learning for Health Care Assistance. Proceedings of the 3rd International Conference on Networking, Information Systems & Security, 1–6. https://doi.org/10.1145/3386723.3387897
2. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS ’20), 9459–9474.
3. Lee, K., Chang, M.-W., & Toutanova, K. (2019). Latent Retrieval for Weakly Supervised Open Domain Question Answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6086–6096. https://doi.org/10.18653/v1/P19-1612
4. Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). Retrieval-Augmented Language Model Pre-Training. Proceedings of Machine Learning Research, 3929–3938.
5. Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., & Zaharia, M. (2022). Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP.
6. Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023). In-Context Retrieval-Augmented Language Models.
7. Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., & Yih, W. (2023). REPLUG: Retrieval-Augmented Black-Box Language Models.
8. İEÜ Ön Lisans ve Lisans Eğitim-Öğretim ve Sınav Yönetmeliği. (2024). https://www.ieu.edu.tr/tr/bylaws/type/read/id/72.

9. Ranoliya, B. R., Raghuwanshi, N., & Singh, S. (2017). Chatbot for university related FAQs. 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 1525–1530. https://doi.org/10.1109/ICACCI.2017.8126057
10. Krassmann, A. L., Flach, J. M., Grando, A. R. C. da S., Tarouco, L. M. R., & Bercht, M. (2019). A Process for Extracting Knowledge Base for Chatbots from Text Corpora. 2019 IEEE Global Engineering Education Conference (EDUCON), 322–329. https://doi.org/10.1109/EDUCON.2019.8725064
11. Nwankwo, W. (2018). Interactive Advising with Bots: Improving Academic Excellence in Educational Establishments. American Journal of Operations Management and Information Systems, 3(1), 6. https://doi.org/10.11648/j.ajomis.20180301.12
12. Chandra, Y. W., & Suyanto, S. (2019). Indonesian Chatbot of University Admission Using a Question Answering System Based on Sequence-to-Sequence Model. Procedia Computer Science, 157, 367–374. https://doi.org/10.1016/j.procs.2019.08.179
13. Tommy, L., Kirana, C., & Riska, L. (2020). The Combination of Natural Language Processing and Entity Extraction for Academic Chatbot. 2020 8th International Conference on Cyber and IT Service Management (CITSM), 1–6. https://doi.org/10.1109/CITSM50537.2020.9268851
14. Guvindan Raju, K. R., Adams, C., & Srinivas, R. (2018). Cognitive Virtual Admissions Counselor. SMU Data Science Review, 1(1), 1–13.
15. Gbenga, O., Okedigba, T., & Oluwatobi, H. (2020). An Improved Rapid Response Model for University Admission Enquiry System Using Chatbot. International Journal of Computer (IJC), 38(1), 123–131.
16. Priadko, A. O., Osadcha, K. P., Kruhlyk. Vladyslav S., & Rakovych, V. A. (2019). Development of a chatbot for informing students of the schedule. Proceedings of the 2nd Student Workshop on Computer Science & Software Engineering (CS&SE@SW 2019), 128–137.
17. Shivam, K., Saud, K., Sharma, M., Vashishth, S., & Patil, S. (2018). Chatbot for College Website. International Journal of Computing and Technology, 5(6), 74–77.
18. Mendoza, S., Hernández-León, M., Sánchez-Adame, L. M., Rodríguez, J., Decouchant, D., & Meneses-Viveros, A. (2020). Supporting Student-Teacher Interaction Through a Chatbot. In Learning and Collaboration Technologies. Human and Technology Ecosystems (pp. 93–107). Springer International Publishing. https://doi.org/10.1007/978-3-030-50506-6_8
19. Nguyen, T. T., Le, A. D., Hoang, H. T., & Nguyen, T. (2021). NEU-chatbot: Chatbot for admission of National Economics University. Computers and Education: Artificial Intelligence, 2, 100036. https://doi.org/10.1016/j.caeai.2021.100036
20. Ula, M., Hardi, R., & Hipiny, I. (2023). An Improved Structure for Academic Information Services through AI Chatbots. Journal of Engineering Science and Technology Review, 16(5), 164–173. https://doi.org/10.25103/jestr.165.20
21. Verma, A., Kuntala, C., Khatri, P., . S., Kaur, S., Mohapatra, A. K., & Singhal, S. (2022). University Chatbot System Using NLP. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4255753
22. Oliveira, P. F., & Matos, P. (2023). Introducing a Chatbot to the Web Portal of a Higher Education Institution to Enhance Student Interaction. ASEC 2023, 128. https://doi.org/10.3390/ASEC2023-16621
23. Maryamah, M., Irfani, M. M., Tri Raharjo, E. B., Rahmi, N. A., Ghani, M., & Raharjana, I. K. (2024). Chatbots in Academia: A Retrieval-Augmented Generation Approach for Improved Efficient Information Access. 2024 16th International Conference on Knowledge and Smart Technology (KST), 259–264. https://doi.org/10.1109/KST61284.2024.10499652
24. Neupane, S., Hossain, E., Keith, J., Tripathi, H., Ghiasi, F., Golilarz, N. A., Amirlatifi, A., Mittal, S., & Rahimi, S. (2024). From Questions to Insightful Answers: Building an Informed Chatbot for University Resources. ArXiv.
25. Nguyen, H. D., Tran, T.-V., Pham, X.-T., Huynh, A. T., & Do, N. V. (2021). Ontology-based Integration of Knowledge Base for Building an Intelligent Searching Chatbot. Sensors and Materials, 33(9), 3101. https://doi.org/10.18494/SAM.2021.3264
26. Ait Baha, T., El Hajji, M., Es-Saady, Y., & Fadili, H. (2024). The impact of educational chatbot on student learning experience. Education and Information Technologies, 29(8), 10153–10176. https://doi.org/10.1007/s10639-023-12166-w
27. Aloqayli, A., & Abdelhafez, H. (2023). Intelligent Chatbot for Admission in Higher Education. International Journal of Information and Education Technology, 13(9), 1348–1357. https://doi.org/10.18178/ijiet.2023.13.9.1937
28. Alharethi, T. M. (2023). Autoresponder using Chatbot for Educational Services. 2023 1st International Conference on Advanced Innovations in Smart Cities (ICAISC), 1–5. https://doi.org/10.1109/ICAISC56366.2023.10084956
29. Lee, Y.-F., Hwang, G.-J., & Chen, P.-Y. (2022). Impacts of an AI-based chabot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation. Educational Technology Research and Development, 70(5), 1843–1865. https://doi.org/10.1007/s11423-022-10142-8
30. Martinez-Araneda, C., Gutiérrez, M., Maldonado, D., Gómez, P., Segura, A., & Vidal-Castro, C. (2024). Designing a chatbot to support problem-solving in a programming course. 966–975. https://doi.org/10.21125/inted.2024.0317
31. Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. ArXiv.
32. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81.
33. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001). BLEU. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, 311. https://doi.org/10.3115/1073083.1073135
34. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020, April 26). BERTScore: Evaluating Text Generation with BERT. 8th International Conference on Learning Representations.