Transformer-Based question answering systems for higher education: A comparative study of Turkish and multilingual models

Abstract

This study presents a question answering system developed for higher education using transformer-based models. Five pretrained models were evaluated including BERTurk Base cased/uncased, ELECTRA-Turk, mBERT and XLM-R. The models were fine-tuned on the THQuAD dataset and tested on a frequently asked questions dataset constructed from official university sources and student queries. In addition to standard evaluation metrics such as Exact Match and F1 score, an extended evaluation approach was applied to better capture semantically appropriate answers. ELECTRA-Turk achieved the highest F1 score of 0.8936 and an Exact Match score of 0.8478. The results show that transformer-based approaches can effectively support automated question answering in academic domains and improve information access for students.

Keywords

• Question-Answering System
• Transformer Models
• Information Retrieval
• Turkish Language Models
• Natural Language Processing

Yükseköğretim için transformer tabanlı soru-cevap sistemleri: Türkçe ve çok dilli modellerin karşılaştırmalı bir incelemesi

Abstract

Bu çalışma, yükseköğretimde kullanılmak üzere geliştirilen bir soru cevap sistemi sunmaktadır. BERTurk Base cased/uncased, ELECTRA-Turk, mBERT ve XLM-R olmak üzere beş önceden eğitilmiş model değerlendirilmiştir. Modeller, THQuAD veri kümesi üzerinde ince ayarlanarak eğitilmiş ve resmi üniversite kaynakları ile öğrenci sorularından oluşturulan bir sıkça sorulan sorular veri kümesi üzerinde test edilmiştir. Tam Eşleşme ve F1 gibi standart metriklere ek olarak anlam açısından doğru yanıtları da dikkate alan genişletilmiş bir değerlendirme yöntemi uygulanmıştır. En yüksek F1 skoru 0.8936 ve Tam Eşleşme skoru 0.8478 ile ELECTRA-Turk modeli tarafından elde edilmiştir. Bulgular, transformer tabanlı yaklaşımların akademik ortamda otomatik soru cevaplama görevlerinde etkili olduğunu ve öğrenci odaklı bilgiye erişimi artırabileceğini göstermektedir.

Keywords

• Soru-Cevap Sistemi
• Transformer Modeller
• Bilgi Edinme
• Türkçe Dil Modelleri
• Doğal Dil İşleme

References

1. [1] Guo X, Zhao B, Ning B. A Survey on Intelligent Question and Answer Systems. Editors: Chenggang Y, Honggang W, Yun L. Mobile Multimedia Communications. MobiMedia 2022. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 81-88, Cham, Switzerland, Springer,2022.
2. [2] Soygazi F, Çiftçi O, Kök U, Cengiz S. “THQuAD: Turkish historic question answering dataset for reading comprehension”. 6th international Conference on Computer Science and Engineering (UBMK), Ankara, Turkey, 15-17 September 2021.
3. [3] Okonkwo CW, Ade-Ibijola A. “Chatbots applications in education: A systematic review”. Computers and Education: Artificial Intelligence, 2, 100033, 2021.
4. [4] Smutny P, Schreiberova P. “Chatbots for learning: A review of educational chatbots for the facebook messenger”. Computers and Education, 151, 103862, 2020.
5. [5] Yousuf M, Jami SI. “An automated question-answering (q/a) system for academic environment”. 2022 Mohammad Ali Jinnah University International Conference on Computing (MAJICC), Karachi, Pakistan, 27-28 October 2022.
6. [6] Fulmal V, Moholkar KP, Patil SH. “The implementation of question answer system using deep learning”. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(1S), 176–182, 2021.
7. [7] Madabushi HT, Lee M, Barnden J. “Integrating question classification and deep learning for improved answer selection”. Proceeding of the 27th International Conference on Computational Linguistics, New Mexico, USA, 20-26 August 2018.
8. [8] Clarizia F, Colace F, Lombardi M, Pascale F, Santaniello D. “Chatbot: An education support system for student”. Cyberspace Safety and Security: 10th International Symposium, CSS 2018, Amalfi, Italy, 29–31 October 2018.

9. [9] Durmus E, He H, Diab M. “Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization”. arXiv, 2020. https://arxiv.org/abs/2005.03754
10. [10] Ardac HA, Erdogmus P. “Question answering system with text mining and deep networks”. Evolving Systems, 15(5), 1787-1799, 2024.
11. [11] Derici C, Aydin Y, Yenialaca N, Aydin NY, Kartal G, Ozgur A, Gungor T. “A closed-domain question answering framework using reliable resources to assist students”. Natural Language Engineering, 24(5), 725–762, 2018.
12. [12] Akyön FÇ, Cavuşoğlu ADE, Cengiz C, Altinuc SO, Temizel A. “Automated question generation and question answering from turkish texts”. Turkish Journal of Electrical Engineering and Computer Sciences, 30(5), 1931–1940, 2022.
13. [13] Kooli C. “Chatbots in education and research: A critical examination of ethical implications and solutions”. Sustainability, 15(7), 5614, 2023.
14. [14] Altintas V, Kilinc M. “Automated categorization of Turkish e-commerce product reviews using BERTurk”. 8th International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Turkiye, 21-22 September 2024.
15. [15] Incidelen M, Aydoğan M. “Developing question-answering models in low-resource languages: A case study on Turkish medical texts using transformer-based approaches”. 8th International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Türkiye, 21-22 September 2024.
16. [16] Ardic O, Ozturk MU, Demirtas I, Arslan S. “Information extraction from sustainability reports in Turkish through RAG approach”. 32nd Signal Processing and Communications Applications Conference (SIU), Mersin, Turkiye, 15-18 May 2024.
17. [17] Yildirim S. “Fine-tuning transformer-based encoder for turkish language understanding tasks”. arXiv, 2024. https://arxiv.org/abs/2401.17396
18. [18] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. “Attention is all you need”. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA, 4-9 December 2017.
19. [19] Kotan M. Duygu Analizi ve Dijital Dönüşüm Üzerine Etkileri. Editors: Tunacan T, Çallı L. Dijital Dönüşümler ve Sektörel Yansımalari II, 59-82, Ankara, Türkiye, Gazi Kitabevi, 2022.
20. [20] Devlin J, Chang MW, Lee K, Toutanova K. “Bert: Pre-training of deep bidirectional transformers for language understanding”. arXiv, 2018. https://arxiv.org/abs/1810.04805
21. [21] Alagöz NK, Küçüksille EU. “System of automatic scientific article summarization in Turkish”. Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, 30(4), 470–481, 2024.
22. [22] Amer E, Hazem A, Farouk O, Louca A, Mohamed Y, Ashraf M. “A proposed chatbot framework for COVID-19”. 2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), Cairo, Egypt, 26-27 May 2021.
23. [23] Koroteev MV. “Bert: a review of applications in natural language processing and understanding”. arXiv, 2021. https://arxiv.org/abs/2103.11943
24. [24] Gürbüz M, Kotan M. “Multi-Category E-Commerce Insights via Social Media Analysis using Machine Learning and BERT”. Acta Infologica, 9(1), 1-18, 2025.
25. [25] Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemover L, Stoyanov V. “Roberta: a robustly optimized bert pretraining approach”. arXiv, 2019. https://arxiv.org/abs/1907.11692
26. [26] Clark K, Luong MT, Le QV, Manning CD. “Electra: pre-training text encoders as discriminators rather than generators”. arXiv, 2020. https://arxiv.org/abs/2003.10555
27. [27] Schweter S. “BERTurk- BERT Models for Turkish”. Zenodo. https://doi.org/10.5281/zenodo.3770924 (08.10.2024).
28. [28] Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, Grave E, Ott M, Zettlemover L, Stoyanov V. “Unsupervised Cross-Lingual Representation Learning at Scale”. arXiv, 2019. https://arxiv.org/abs/1911.02116
29. [29] Tquad. “Turkish NLP Q&A Dataset”. https://github.com/TQuad/turkish-nlp-qa-dataset (08.10.2024).
30. [30] Saka SO, Cömert Z. “Sentiment analysis based on text with universal sentence encoder and CNN-LSTM models”. 8th International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Türkiye, 21-22 September 2024.
31. [31] Cömert Ö, Yücel N. “Müşteri duyarlılığını keşfetmek için yapay zeka destekli analiz ile çevrimiçi ürün incelemelerinden anlamlı bilgiler elde etme”. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 35(2), 679–690, 2023.