Transformer-Based question answering systems for higher education: A comparative study of turkish and multilingual models

Abstract

This study presents a question answering system developed for higher education using transformer-based models. Five pretrained models were evaluated including BERTurk Base cased/uncased, ELECTRA-Turk, mBERT and XLM-R. The models were fine-tuned on the THQuAD dataset and tested on a frequently asked questions dataset constructed from official university sources and student queries. In addition to standard evaluation metrics such as Exact Match and F1 score, an extended evaluation approach was applied to better capture semantically appropriate answers. ELECTRA-Turk achieved the highest F1 score of 0.8936 and an Exact Match score of 0.8478. The results show that transformer-based approaches can effectively support automated question answering in academic domains and improve information access for students.

Keywords

• Question-Answering System
• Transformer Models
• Information Retrieval
• Turkish Language Models
• Natural Language Processing

Yükseköğretim İçin Transformer Tabanlı Soru-Cevap Sistemleri: Türkçe ve Çok Dilli Modellerin Karşılaştırmalı Bir İncelemesi

Öz

Bu çalışma, yükseköğretimde kullanılmak üzere geliştirilen bir soru cevap sistemi sunmaktadır. BERTurk Base cased/uncased, ELECTRA-Turk, mBERT ve XLM-R olmak üzere beş önceden eğitilmiş model değerlendirilmiştir. Modeller, THQuAD veri kümesi üzerinde ince ayarlanarak eğitilmiş ve resmi üniversite kaynakları ile öğrenci sorularından oluşturulan bir sıkça sorulan sorular veri kümesi üzerinde test edilmiştir. Tam Eşleşme ve F1 gibi standart metriklere ek olarak anlam açısından doğru yanıtları da dikkate alan genişletilmiş bir değerlendirme yöntemi uygulanmıştır. En yüksek F1 skoru 0.8936 ve Tam Eşleşme skoru 0.8478 ile ELECTRA-Turk modeli tarafından elde edilmiştir. Bulgular, transformer tabanlı yaklaşımların akademik ortamda otomatik soru cevaplama görevlerinde etkili olduğunu ve öğrenci odaklı bilgiye erişimi artırabileceğini göstermektedir.

Anahtar Kelimeler

• Soru-Cevap Sistemi
• Transformer Modeller
• Bilgi Edinme
• Türkçe Dil Modelleri
• Doğal Dil İşleme

References

1. [1] Guo X, Zhao B, Ning B. “A survey on intelligent question and answer systems.” International Conference on Mobile Computing, Applications, and Services, 81–88, 2022.
2. [2] Soygazi F, Çiftçi O, Kök U, Cengiz S. “THQuAD: Turkish historic question answering dataset for reading comprehension.” 6th International Conference on Computer Science and Engineering (UBMK), 215–220, IEEE, 2021.
3. [3] Okonkwo CW, Ade-Ibijola A. “Chatbots applications in education: A systematic review.” Computers and Education: Artificial Intelligence, 2, 100033, 2021.
4. [4] Smutny P, Schreiberova P. “Chatbots for learning: A review of educational chatbots for the Facebook Messenger.” Computers & Education, 151, 103862, 2020.
5. [5] Yousuf M, Jami SI. “An automated question-answering (Q/A) system for academic environment.” Mohammad Ali Jinnah University International Conference on Computing (MAJICC), 1–6, 2022.
6. [6] Fulmal V, et al. “The implementation of question answer system using deep learning.” Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(1S), 176–182, 2021.
7. [7] Madabushi HT, Lee M, Barnden J. “Integrating question classification and deep learning for improved answer selection.” Proceedings of the 27th International Conference on Computational Linguistics, 3283–3294, 2018.
8. [8] Clarizia F, Colace F, Lombardi M, Pascale F, Santaniello D. “Chatbot: An education support system for students.” Cyberspace Safety and Security: 10th International Symposium (CSS 2018), Amalfi, Italy, 29–31 October 2018, Springer, 291–302, 2018.

9. [9] Durmus E, He H, Diab M. “FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.” arXiv preprint arXiv:2005.03754, 2020.
10. [10] Ardac HA, Erdogmus P. “Question answering system with text mining and deep networks.” Evolving Systems, 1–13, 2024.
11. [11] Derici C, Aydin Y, Yenialaca N, Aydin NY, Kartal G, Ozgur A, Gungor T. “A closed-domain question answering framework using reliable resources to assist students.” Natural Language Engineering, 24(5), 725–762, 2018.
12. [12] Akyön FÇ, Cavuşoğlu ADE, Cengiz C, Altinuc SO, Temizel A. “Automated question generation and question answering from Turkish texts.” Turkish Journal of Electrical Engineering and Computer Sciences, 30(5), 1931–1940, 2022.
13. [13] Kooli C. “Chatbots in education and research: A critical examination of ethical implications and solutions.” Sustainability, 15(7), 5614, 2023.
14. [14] Altintas V, Kilinc M. “Automated categorization of Turkish e-commerce product reviews using BERTurk.” 8th International Artificial Intelligence and Data Processing Symposium (IDAP), 1–6, September 2024, IEEE.
15. [15] Incidelen M, Aydoğan M. “Developing question-answering models in low-resource languages: A case study on Turkish medical texts using transformer-based approaches.” 8th International Artificial Intelligence and Data Processing Symposium (IDAP), 1–4, September 2024, IEEE.
16. [16] Ardic O, Ozturk MU, Demirtas I, Arslan S. “Information extraction from sustainability reports in Turkish through RAG approach.” 32nd Signal Processing and Communications Applications Conference (SIU), 1–4, May 2024, IEEE.
17. [17] Yildirim S. “Fine-tuning transformer-based encoder for Turkish language understanding tasks.” arXiv preprint arXiv:2401.17396, 2024.
18. [18] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. “Attention is all you need.” Advances in Neural Information Processing Systems, 30, 2017.
19. [19] Kotan M. “Duygu analizi ve dijital dönüşüm üzerine etkileri.” In: Dijital Dönüşümler ve Sektörel Yansımaları 2, 2022.
20. [20] Devlin J, Chang MW, Lee K, Toutanova K. “BERT: Pre-training of deep bidirectional transformers for language understanding.” arXiv preprint arXiv:1810.04805, 2018.
21. [21] Alagöz NK, Küçüksille EU. “System of automatic scientific article summarization in Turkish.” Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, 30(4), 470–481, 2024.
22. [22] Kenton JD, Chang MW, Toutanova K. “BERT: Pre-training of deep bidirectional transformers for language understanding.” Proceedings of NAACL-HLT, 1(2), 2019.
23. [23] Amer E, Hazem A, Farouk O, Louca A, Mohamed Y, Ashraf M. “A proposed chatbot framework for COVID-19.” International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), 263–268, 2021.
24. [24] Koroteev M. “BERT: A review of applications in natural language processing and understanding.” arXiv preprint arXiv:2103.11943, 2021.
25. [25] Gürbüz M, Kotan M. “Multi-category e-commerce insights via social media analysis using machine learning and BERT.” Acta Infologica, 10.26650/acin.1483488, 2025.
26. [26] Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. “RoBERTa: A robustly optimized BERT pretraining approach.” arXiv preprint arXiv:1907.11692, 2019.
27. [27] Clark K, Luong MT, Le QV, Manning CD. “ELECTRA: Pre-training text encoders as discriminators rather than generators.” arXiv preprint arXiv:2003.10555, 2020.
28. [28] Schweter S. “BERTurk – BERT models for Turkish.” Zenodo, 10.5281/zenodo.3770924, 2020.
29. [29] Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, et al. “Unsupervised cross-lingual representation learning at scale.” arXiv preprint arXiv:1911.02116, 2019.
30. [30] Turkish NLP Q&A Dataset, available at: https://github.com/TQuad/turkish-nlp-qa-dataset
31. [31] Saka SO, Cömert Z. “Sentiment analysis based on text with Universal Sentence Encoder and CNN-LSTM models.” 8th International Artificial Intelligence and Data Processing Symposium (IDAP), 1–4, September 2024, IEEE.
32. [32] Yücel N, Cömert Ö. “Müşteri duyarlılığını keşfetmek için yapay zeka destekli analiz ile çevrimiçi ürün incelemelerinden anlamlı bilgiler elde etme.” Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 35(2), 679–690, 2023.