Comparison of Transformer-Based Turkish Models for Question-Answering Task

Abstract

Question-answering systems facilitate information access processes by providing fast and accurate answers to questions that users express in natural language. Today, advances in Natural Language Processing (NLP) techniques increase the effectiveness of such systems and improve the user experience. However, for these systems to work effectively, an accurate understanding of the structural properties of language is required. Traditional rule-based and knowledge retrieval-based systems are not able to analyze the contextual meaning of questions and texts deeply enough and therefore cannot produce satisfactory answers to complex questions. For this reason, Transformer-based models that can better capture the contextual and semantic integrity of the language have been developed. In this study, within the scope of the developed models, the performances of BERTurk, ELECTRA Turkish and DistilBERTurk models for Turkish question-answer tasks were compared by fine-tuning under the same hyperparameters and the results obtained were evaluated. According to the findings, it was observed that higher Exact Match (EM) and F1 scores were obtained in models with case sensitivity; the best performance was obtained with 63.99 EM and 80.84 F1 scores in the BERTurk (Cased, 128k) model.

Keywords

• Natural Language Processing
• Question-Answering System
• Transformer
• BERTurk

References

1. [1] D. Khurana, A. Koli, K. Khatter, and S. Singh, ‘Natural language processing: state of the art, current trends and challenges’, Multimed. Tools Appl., vol. 82, no. 3, pp. 3713–3744, Jan. 2023, doi: 10.1007/s11042-022-13428-4.
2. [2] K. Crowston, E. E. Allen, and R. Heckman, ‘Using natural language processing technology for qualitative data analysis’, Int. J. Soc. Res. Methodol., vol. 15, no. 6, pp. 523–543, Nov. 2012, doi: 10.1080/13645579.2011.625764.
3. [3] M. Arzu and M. Aydoğan, ‘Türkçe Duygu Sınıflandırma İçin Transformers Tabanlı Mimarilerin Karşılaştırılmalı Analizi’, Comput. Sci., no. IDAP-2023, pp. 1–6, 2023.
4. [4] A. Allam and M. Haggag, ‘The Question Answering Systems: A Survey’, Int. J. Res. Rev. Inf. Sci., vol. 2, pp. 211–221, Sep. 2012.
5. [5] E. Mutabazi, J. Ni, G. Tang, and W. Cao, ‘A Review on Medical Textual Question Answering Systems Based on Deep Learning Approaches’, Appl. Sci., vol. 11, no. 12, Art. no. 12, Jan. 2021, doi: 10.3390/app11125456.
6. [6] V. Redhu, A. K. Singh, and M. Saravanan, ‘AI-Enhanced Learning Assistant Platform: An Advanced System for Q&A Generation from Provided Content, Answer Evaluation, Identification of Students’ Weak Areas, Recursive Testing for Strengthening Knowledge, Integrated Query Forum, and Expert Chat Support’, in 2024 2nd International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Internet of Things (AIMLA), Mar. 2024, pp. 1–6. doi: 10.1109/AIMLA59606.2024.10531533.
7. [7] K. Tohma and Y. Kutlu, ‘Challenges Encountered in Turkish Natural Language Processing Studies’, Nat. Eng. Sci., vol. 5, no. 3, Art. no. 3, Nov. 2020, doi: 10.28978/nesciences.833188.
8. [8] A. Vaswani et al., ‘Attention is All you Need’, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017. Accessed: May 22, 2024. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

9. [9] M. İncidelen and M. Aydoğan, ‘Developing Question-Answering Models in Low-Resource Languages: A Case Study on Turkish Medical Texts Using Transformer-Based Approaches’, in 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), Sep. 2024, pp. 1–4. doi: 10.1109/IDAP64064.2024.10711128.
10. [10] C. Özkurt, Comparative Analysis of State-of-the-Art Q\&A Models: BERT, RoBERTa, DistilBERT, and ALBERT on SQuAD v2 Dataset. 2024. doi: 10.21203/rs.3.rs-3956898/v1.
11. [11] F. Soygazi, O. Çiftçi, U. Kök, and S. Cengiz, ‘THQuAD: Turkish Historic Question Answering Dataset for Reading Comprehension’, in 2021 6th International Conference on Computer Science and Engineering (UBMK), Sep. 2021, pp. 215–220. doi: 10.1109/UBMK52708.2021.9559013.
12. [12] Y. Uğurlu, M. Karabulut, and İ. Mayda, ‘A Smart Virtual Assistant Answering Questions About COVID-19’, in 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Oct. 2020, pp. 1–6. doi: 10.1109/ISMSIT50672.2020.9254350.
13. [13] Ö. Ünlü and A. Çetin, ‘A Survey on Keyword and Key Phrase Extraction with Deep Learning’, in 2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Oct. 2019, pp. 1–6. doi: 10.1109/ISMSIT.2019.8932811.
14. [14] M. F. Amasyalı and B. Diri, ‘Bir Soru Cevaplama Sistemi: BayBilmiş’, Türkiye Bilişim Vakfı Bilgi. Bilim. Ve Mühendisliği Derg., vol. 1, no. 1, Art. no. 1, Jun. 2016.
15. [15] C. B. Gemirter and D. Goularas, ‘A Turkish Question Answering System Based on Deep Learning Neural Networks’, J. Intell. Syst. Theory Appl., vol. 4, no. 2, Art. no. 2, Sep. 2021, doi: 10.38016/jista.815823.
16. [16] A. Mukanova, A. Barlybayev, A. Nazyrova, L. Kussepova, B. Matkarimov, and G. Abdikalyk, ‘Development of a Geographical Question- Answering System in the Kazakh Language’, IEEE Access, vol. 12, pp. 105460–105469, 2024, doi: 10.1109/ACCESS.2024.3433426.
17. [17] J. Staš, D. Hládek, and T. Koctúr, ‘Slovak Question Answering Dataset Based on the Machine Translation of the Squad V2.0’, J. Linguist. Cas., vol. 74, no. 1, pp. 381–390, Jun. 2023, doi: 10.2478/jazcas-2023-0054.
18. [18] P. Rajpurkar, R. Jia, and P. Liang, ‘Know What You Don’t Know: Unanswerable Questions for SQuAD’, Jun. 11, 2018, arXiv: arXiv:1806.03822. doi: 10.48550/arXiv.1806.03822.
19. [19] N. Patwardhan, S. Marrone, and C. Sansone, ‘Transformers in the Real World: A Survey on NLP Applications’, Information, vol. 14, no. 4, Art. no. 4, Apr. 2023, doi: 10.3390/info14040242.
20. [20] Okan, okanvk/Turkish-Reading-Comprehension-Question-Answering-Dataset. (Oct. 26, 2024). Jupyter Notebook. Accessed: Oct. 30, 2024. [Online]. Available: https://github.com/okanvk/Turkish-Reading-Comprehension-Question-Answering-Dataset
21. [21] ‘TurQuest/turkish-bquad: Türkçe dilinde biyoloji soru/cevap veriseti’. Accessed: Jul. 20, 2024. [Online]. Available: https://github.com/TurQuest/turkish-bquad/tree/main
22. [22] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, ‘SQuAD: 100,000+ Questions for Machine Comprehension of Text’, presented at the Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Nov. 2016, pp. 2383–2392. doi: 10.18653/v1/D16-1264.
23. [23] S. Schweter, BERTurk - BERT models for Turkish. (Apr. 27, 2020). Zenodo. doi: 10.5281/zenodo.3770924.
24. [24] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding’, May 24, 2019, arXiv: arXiv:1810.04805. doi: 10.48550/arXiv.1810.04805.
25. [25] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, ‘ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators’, Mar. 23, 2020, arXiv: arXiv:2003.10555. doi: 10.48550/arXiv.2003.10555.
26. [26] ‘turkish-bert/electra/README.md at master · stefan-it/turkish-bert’, GitHub. Accessed: Oct. 30, 2024. [Online]. Available: https://github.com/stefan-it/turkish-bert/blob/master/electra/README.md
27. [27] ‘dbmdz/electra-small-turkish-cased-discriminator · Hugging Face’. Accessed: Oct. 30, 2024. [Online]. Available: https://huggingface.co/dbmdz/electra-small-turkish-cased-discriminator
28. [28] ‘dbmdz/electra-base-turkish-cased-discriminator · Hugging Face’. Accessed: Oct. 30, 2024. [Online]. Available: https://huggingface.co/dbmdz/electra-base-turkish-cased-discriminator
29. [29] ‘dbmdz/distilbert-base-turkish-cased · Hugging Face’. Accessed: Aug. 09, 2024. [Online]. Available: https://huggingface.co/dbmdz/distilbert-base-turkish-cased
30. [30] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, ‘DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter’, Feb. 29, 2020, arXiv: arXiv:1910.01108. doi: 10.48550/arXiv.1910.01108.
31. [31] P. Flach and M. Kull, ‘Precision-Recall-Gain Curves: PR Analysis Done Right’, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2015. Accessed: Aug. 09, 2024. [Online]. Available: https://papers.nips.cc/paper_files/paper/2015/hash/33e8075e9970de0cfea955afd4644bb2-Abstract.html