A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts

Uğur Dagtekin

doi:10.54287/gujsa.1966262

A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts

Abstract

Multi-class emotion classification in Turkish texts remains challenging due to the language’s agglutinative structure, morphological richness, and semantic overlap among emotion categories. To overcome these challenges, this research develops a hybrid framework that integrates lexical TF-IDF features with contextual representations generated by the pre-trained BERTurk model. Unigram and bigram TF-IDF features are combined with contextual embeddings to form a unified representation for emotion classification. The evaluation was performed on a publicly available Turkish emotion dataset consisting of 29,727 samples belonging to 13 emotion categories. Following data preparation, the corpus was partitioned into training and testing subsets through a stratified sampling strategy. Multinomial Naive Bayes, Logistic Regression, and Linear Support Vector Machine were evaluated as baseline models. The proposed hybrid architecture employed feature-level fusion and Logistic Regression as the classifier. A fine-tuned BERTurk model was also included as a transformer-based benchmark. Experimental findings showed that the hybrid framework achieved substantially better classification performance than approaches relying solely on lexical representations. While the strongest baseline model, Linear SVM, achieved a Macro F1-score of 92.79%, the hybrid model reached 95.66%, corresponding to an improvement of 2.87 percentage points. Five-fold cross-validation and statistical significance testing confirmed the robustness of the observed performance gains. Although fine-tuned BERTurk achieved the highest Macro F1-score (96.17%), the performance gap between the transformer and hybrid models was limited to 0.51 percentage points. The findings indicate that integrating lexical and contextual representations provides an effective and computationally efficient alternative to full transformer fine-tuning for Turkish emotion classification.

Keywords

Supporting Institution

No supporting institution.

Ethical Statement

Ethical approval was not required for this study because it was conducted using a publicly available dataset and did not involve human participants, animal subjects, or personal data.

References

Agrawal, S., Kirtani, Y., Girdhar, Y., & Aggarwal, S. (2018). Hindi sentence classification for expressive storytelling systems. In: S. Sundaram (Eds.), Proceedings of the 2018 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 2019–2025), 18-21 November 2018, Bangalore, India. https://doi.org/10.1109/ssci.2018.8628858
Altınel Girgin, A. B., & Şahin, S. (2023). Improving the Performance of Sentiment Analysis by Ensemble Hybrid Learning Algorithm With NLP And Cascaded Feature Extraction. International Journal of Advances in Engineering and Pure Sciences, 35(1), 125-141. https://doi.org/10.7240/jeps.1249586
Arzu, M., & Aydoğan, M. (2023). Comparative Analysis of Transformers Based Architectures for Turkish Sentiment Classification [Türkçe Duygu Sınıflandırma için Transformers Tabanlı Mimarilerin Karşılaştırılmalı Analizi]. Computer Science [Bilgisayar Bilimleri], IDAP-2023, 1–6. https://doi.org/10.53070/bbd.1350405
Çelikten, A., & Bulut, H. (2021). Turkish medical text classification using BERT. In: Proceedings of the 2021 29th Signal Processing and Communications Applications Conference (SIU), 09-11 June 2021, Istanbul, Turkey. https://doi.org/10.1109/SIU53274.2021.9477847
Çolakoğlu, E., Hızlısoy, S., & Arslan, R. S. (2021). A Detailed Survey on Speech Emotion Recognition: Features and Classification Methods [Konuşmadan Duygu Tanıma Üzerine Detaylı bir İnceleme: Özellikler ve Sınıflandırma Metotları]. European Journal of Science and Technology [Avrupa Bilim ve Teknoloji Dergisi], (32), 471–483. https://doi.org/10.31590/ejosat.1039403
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6), 391–407. https://doi.org/10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9
Dehghan, S., Şen, M. U., & Yanikoglu, B. (2025). Dealing with annotator disagreement in hate speech classification. https://doi.org/10.48550/arXiv.2502.08266

Dehghan, S., Uludoğan, G., Şen, M. U., Erol, E., Özgür, A., & Yanikoglu, B. (2026). Hate speech detection in Turkish and Arabic: A comprehensive study. https://doi.org/10.48550/arXiv.2607.00143
Demir, O., Chawai, A. I. B., & Doğan, B. (2019). Sentiment Analysis Using Dictionary Based Approach in Turkish Texts [Türkçe Metinlerde Sözlük Tabanlı Yaklaşımla Duygu Analizi ve Görselleştirme]. International Periodical of Recent Technologies in Applied Engineering, 1(2), 58–66. https://izlik.org/JA39TR59JY
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1–30.
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020). GoEmotions: A dataset of fine-grained emotions. In: D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, (pp. 4040–4054), 05-10 July 2020. https://doi.org/10.18653/v1/2020.acl-main.372
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In: J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (pp. 4171–4186), 02-07 June 2019, Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-1423
Dewi, C., Chen, R.-C., Christanto, H. J., & Cauteruccio, F. (2023). Multinomial Naive Bayes classifier for sentiment analysis of Internet Movie Database. Vietnam Journal of Computer Science, 10(04), 485–498. https://doi.org/10.1142/s2196888823500100
Ferhati, K., Burlea-Schiopoiu, A., & Nascu, A. G. (2025). A text-based project risk classification system using multi-model AI: Comparing SVM, Logistic Regression, Random Forests, Naive Bayes, and XGBoost. Systems, 13(12), 1078. https://doi.org/10.3390/systems13121078
Ferrario, A., & Nägelin, M. (2020). The art of natural language processing: Classical, modern and contemporary approaches to text document classification. SSRN. http://doi.org/10.2139/ssrn.3547887
Güngör, H. Ç. (2023). Türkçe cümleler ve duygular: Duygu analizi veri seti [Data set]. Kaggle. https://www.kaggle.com/datasets/aiwithcagri/trke-cmleler-ve-duygular-duygu-analizi/data
Huang, T., Shen, G., & Deng, Z.-H. (2019). Leap-LSTM: Enhancing long short-term memory for text categorization. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) (pp. 5017–5023), 10-16 August 2019, Macao, China. https://doi.org/10.24963/ijcai.2019/697
İncidelen, M., & Aydoğan, M. (2024). Performance evaluation of transformer-based pre-trained language models for Turkish question-answering. Black Sea Journal of Engineering and Science, 8(2), 323–329. https://doi.org/10.34248/bsengineering.1596832
Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In: C. Nédellec, & C. Rouveirol (Eds.), Proceedings of the 10th European Conference on Machine Learning (ECML-98) (pp 137–142), 21-23 April 1998, Chemnitz, Germany. https://doi.org/10.1007/BFb0026683
Kına, E. (2025). Analysis of Turkish Comments on Mental Health in Social Media Through Transformer-Based Emotion Classification [Transformer Tabanlı Duygu Sınıflandırması ile Sosyal Medyada Ruh Sağlığına Ilişkin Türkçe Yorumların Analizi]. Kahramanmaras Sutcu Imam University Journal of Engineering Sciences [Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi], 28(3), 1499–1511. https://doi.org/10.17780/ksujes.1694291
Kohavi, R. (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95) (pp. 1137–1145), 20-25 August 1995, Montreal, Quebec, Canada.
Köksal, A., & Özgür, A. (2021). Twitter dataset and evaluation of transformers for Turkish sentiment analysis. In: Proceedings of the 2021 29th Signal Processing and Communications Applications Conference (SIU), 09-11 June 2021, Istanbul, Turkey. https://doi.org/10.1109/SIU53274.2021.9477814
Larasati, U. I., Muslim, M. A., Arifudin, R., & Alamsyah, A. (2019). Improve the accuracy of support vector machine using chi-square statistic and TF-IDF. Scientific Journal of Informatics, 6(1), 138–149. https://doi.org/10.15294/sji.v6i1.14244
Liu, G., Ma, F., Fan, C., Xie, C., Li, Q., Ni, W., & Xu, H. (2023). CNN-LSTM integrated model with attention mechanism. In: Proceedings of the 2023 International Conference on Intelligent Sensing and Industrial Automation (ISIA’23), 9-10 December 2023, China. https://doi.org/10.1145/3632314.3632347
Ma, X. (2020). A novel hybrid model by using convolutional neural network and long short-term memory for text sentiment analysis. Dyna, 95, 527–533. https://doi.org/10.6036/9714
McCallum, A., & Nigam, K. (1998). A comparison of event models for Naive Bayes text classification. AAAI Workshop on Learning for Text Categorization, 752(1), 41-48
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. https://doi.org/10.48550/arXiv.1301.3781
Mittal, V., Gangodkar, D., & Pant, B. (2020). Exploring the dimension of DNN techniques for text categorization using NLP. In: Proceedings of the 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS) (pp. 497–501), 06-07 March 2020, Coimbatore, India. https://doi.org/10.1109/ICACCS48705.2020.9074228
Nazaretsky, T., Yolcu, H. H., Ariely, M., & Alexandron, G. (2023). Towards automated assessment of scientific explanations in Turkish using language transfer. https://doi.org/10.31219/osf.io/wuzy9
Nooruldeen, O. I., & Savaş, S. (2024). Using Deep Learning Techniques to Identify Depression in Social Media Interactions [Sosyal Medya Etkileşimlerinde Depresyonu Tanımlamak için Derin Öğrenme Tekniklerinin Kullanılması]. Sinop University Journal of Natural Sciences [Sinop Üniversitesi Fen Bilimleri Dergisi], 9(2), 449–466. https://doi.org/10.33484/sinopfbd.1456956
Özer, Ö., & Subaşı, N. (2025). Automatic classification of basic emotions using deep learning techniques. Journal of Artificial Intelligence and Data Science, 5(2), 75–88. https://izlik.org/JA43ZT75YD
Patel, A., & Meehan, K. (2021). Fake news detection on Reddit utilising TF-IDF and machine learning. In: Proceedings of the 2021 32nd Irish Signals and Systems Conference (ISSC), 10-11 June 2021, Athlone, Ireland. https://doi.org/10.1109/issc52156.2021.9467842
Peng, T., Liu, L., & Zuo, W. (2013). PU text classification enhanced by TF-IDF improved weighting. Concurrency and Computation: Practice and Experience, 26(3), 728–741. https://doi.org/10.1002/cpe.3040
Qaiser, S., & Ali, R. (2018). Text mining: Use of TF-IDF to examine the relevance of words to documents. International Journal of Computer Applications, 181(1), 25–29. https://doi.org/10.5120/ijca2018917395
Ramaswamy, M. P. A., & Palaniswamy, S. (2024). Multimodal emotion recognition: A comprehensive review, trends, and challenges. WIREs Data Mining and Knowledge Discovery, 14(6), e1563. https://doi.org/10.1002/widm.1563
Rani, M. S., & Subramanian, S. (2020). Attention Mechanism with Gated Recurrent Unit Using Convolutional Neural Network for Aspect Level Opinion Mining. Arabian Journal for Science and Engineering, 45(8), 6157–6169. https://doi.org/10.1007/s13369-020-04497-4
Rawat, S. S., Rawat, K. S., & Nijhawan, R. (2020). A novel Convolutional Neural Network-Gated Recurrent Unit approach for image captioning. In: Proceedings of the 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT) (pp. 704–708), 20-22 August 2020, Tirunelveli, India. https://doi.org/10.1109/ICSSIT48917.2020.9214109
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: K. Inui, J. Jiang, V. Ng, & X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982–3992), 03–07 November 2019, Hong Kong, China, https://doi.org/10.18653/v1/D19-1410
Safaya, A., Kurtuluş, E., Göktogan, A., & Yuret, D. (2022). Mukayese: Turkish NLP strikes back. In: S. Muresan, P. Nakov, & A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) (pp. 846–863), 22-27 May 2022, Dublin, Ireland. https://doi.org/10.18653/v1/2022.findings-acl.69
Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0
Saravanan, U. K. B., Vijay, M., Shreedhar, T., Rajasekar, G., R, Y., & Shakthipriya, P. (2023). Multinomial Naive Bayes based machine learning analysis of Twitter sentiment. In: Proceedings of the 2023 2nd International Conference on Edge Computing and Applications (ICECAA) (pp. 429–434), 19-21 July 2023, Namakkal, India. https://doi.org/10.1109/icecaa58104.2023.10212150
Schweter, S. (2020). BERTurk-BERT models for Turkish (Version 1.0.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.3770924
Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
Shah, S. M., Aljawarneh, M. M., Saleem, M. A., & Jawarneh, M. S. (2024). Mental illness detection through harvesting social media: A comprehensive literature review. PeerJ Computer Science, 10, e2296. https://doi.org/10.7717/peerj-cs.2296
Sharupa, N. A., Rahman, M., Alvi, N., Raihan, M., Islam, A., & Raihan, T. (2020). Emotion detection of Twitter posts using Multinomial Naive Bayes. In: Proceedings of the 2020 11th International Conference on Computing, Communication and Network Technologies (ICCCNT), 01-03 July 2020, Kharagpur, India. https://doi.org/10.1109/icccnt49239.2020.9225432
Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427–437. https://doi.org/10.1016/j.ipm.2009.03.002
Sun, X., Li, Y., Kang, H., & Shen, Y. (2019). Automatic document classification using convolutional neural network. Journal of Physics: Conference Series, 1176(3), 032029. https://doi.org/10.1088/1742-6596/1176/3/032029
Tang, Y., Huang, Y., Wu, Z., Meng, H., Xu, M., & Cai, L. (2016). Question detection from acoustic features using recurrent neural network with gated recurrent unit. In: Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6125–6129), 20-25 March 2016, Shanghai, China. https://doi.org/10.1109/ICASSP.2016.7472854
Temel, H., & Kına, E. (2025). A comparative analysis of transformer architectures for sentiment and emotion classification. Balkan Journal of Electrical and Computer Engineering, 13(4), 470–482. https://doi.org/10.17694/bajece.1802918
Thiengburanathum, P., & Charoenkwan, P. (2023). SETAR: Stacking ensemble learning for Thai sentiment analysis using RoBERTa and hybrid feature representation. IEEE Access, 11, 92822–92837. https://doi.org/10.1109/access.2023.3308951
Turkmen, H., Dikenelli, O., Eraslan, C., & Callı, M. C. (2022). BioBERTurk: Exploring Turkish Biomedical Language Model Development Strategies in Low Resource Setting. Research Square Platform LLC. https://doi.org/10.21203/rs.3.rs-2165226/v1
Ucan, A., Dörterler, M., & Akçapınar Sezer, E. (2022). A study of Turkish emotion classification with pretrained language models. Journal of Information Science, 48(6), 857–865. https://doi.org/10.1177/0165551520985507
Vinodhini, G., & Chandrasekaran, R. M. (2012). Sentiment analysis and opinion mining: A survey. International Journal of Advanced Research in Computer Science and Software Engineering, 2(6), 282–292.
Zhou, S., & Mohd, M. (2025). Mental health safety and depression detection in social media text data: A classification approach based on a deep learning model. IEEE Access, 13, 63284–63297. https://doi.org/10.1109/ACCESS.2025.3559170

Details

Primary Language

English

Subjects

Deep Learning, Natural Language Processing

Journal Section

Research Article

Authors

Uğur Dagtekin ^*
0000-0002-7298-2723
Türkiye

Early Pub Date

September 28, 2026

Publication Date

September 30, 2026

Submission Date

June 8, 2026

Acceptance Date

August 19, 2026

Published in Issue

Year 2026 Volume: 13 Number: 3

DOI

https://doi.org/10.54287/gujsa.1966262

IZ

https://izlik.org/JA64AD43HH

Cite

RIS / Bibtex

APA

Dagtekin, U. (2026). A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts. Gazi University Journal of Science Part A: Engineering and Innovation, 13(3), 1258-1288. https://doi.org/10.54287/gujsa.1966262

AMA

1.Dagtekin U. A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts. GU J Sci, Part A. 2026;13(3):1258-1288. doi:10.54287/gujsa.1966262

Chicago

Dagtekin, Uğur. 2026. “A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts”. Gazi University Journal of Science Part A: Engineering and Innovation 13 (3): 1258-88. https://doi.org/10.54287/gujsa.1966262.

EndNote

Dagtekin U (September 1, 2026) A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts. Gazi University Journal of Science Part A: Engineering and Innovation 13 3 1258–1288.

IEEE

[1]U. Dagtekin, “A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts”, GU J Sci, Part A, vol. 13, no. 3, pp. 1258–1288, Sept. 2026, doi: 10.54287/gujsa.1966262.

ISNAD

Dagtekin, Uğur. “A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts”. Gazi University Journal of Science Part A: Engineering and Innovation 13/3 (September 1, 2026): 1258-1288. https://doi.org/10.54287/gujsa.1966262.

JAMA

1.Dagtekin U. A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts. GU J Sci, Part A. 2026;13:1258–1288.

MLA

Dagtekin, Uğur. “A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts”. Gazi University Journal of Science Part A: Engineering and Innovation, vol. 13, no. 3, Sept. 2026, pp. 1258-8, doi:10.54287/gujsa.1966262.

Vancouver

1.Uğur Dagtekin. A Hybrid and Explainable Feature Fusion Framework for Multi-Class Emotion Classification in Turkish Texts. GU J Sci, Part A. 2026 Sep. 1;13(3):1258-8. doi:10.54287/gujsa.1966262