Konuşma Duygu Tanıma için Dikkat Tabanlı Derin Öğrenme ve Metasezgisel Optimizasyonun Hibrit Modeli

Abstract

İnsan-bilgisayar etkileşimi alanının önemli bir parçası haline gelen Konuşma Duygu Tanıma (SER) teknolojileri, insan konuşma sinyallerindeki duygusal durumları otomatik olarak algılayan yapay zekâ tabanlı sistemlerdir. Bu sistemlerin çalışma prensibi, ses sinyallerinden çıkarılan akustik özelliklerin makine öğrenimi yoluyla duygusal etiketlerle eşleştirilmesi sürecidir. Önemli ilerlemelere rağmen, mevcut SER teknolojileri hala bazı zorluklarla karşı karşıyadır. Bunların başında yerel ve küresel kalıpların yetersiz temsili gelmektedir. Bu çalışma, duygu temsilinde bu zorlukların üstesinden gelmek için yenilikçi bir hibrit model sunmaktadır. Model iki aşamadan oluşmaktadır. Birinci aşama, duyguların özellik temsillerinin çıkarılması ve bunların duygu etiketleriyle eşleştirilmesiyle uçtan uca bir öğrenme sürecini içeren temel modeli temsil eder. İkinci aşama, özellik optimizasyonunun gerçekleştiği süreçtir. Birinci aşama, yerel özellik çıkarımı, kanal düzeyinde özellik aktivasyonu, zamansal bağımlılık temsili ve son olarak önemli zaman dilimlerine odaklanmayı içeren adımlardan oluşmaktadır. İkinci aşama, altı meta-sezgisel algoritmanın üç makine öğrenimi paradigmasıyla entegrasyonundan türetilen on sekiz farklı optimizasyon konfigürasyonunu kullanarak özellikleri optimize etme sürecidir. Deneyler EMO-DB veri seti üzerinde gerçekleştirildi. 5 katlı çapraz doğrulama yönteminde, önerilen hibrit model, temel modele kıyasla ağırlıksız doğrulukta %7,8'lik bir artış gösterirken, özellik boyutluluğunu %54,8 oranında azaltarak %83,73'lük ağırlıksız doğruluk (UA) elde etti. Bu sonuçlar, önerilen hibrit modelin yüksek doğruluk ve verimli özellik temsilleri sağladığını doğrulamaktadır.

Keywords

• Konuşma duygu tanıma
• Dikkat mekanizması
• Metasezgisel optimizasyon

Hybrid Model of Attention-Guided Deep Learning and Metaheuristic Optimization for Speech Emotion Recognition

Abstract

Speech Emotion Recognition (SER) technologies, which have become a significant part of the human-computer interaction field, are artificial intelligence-based systems that automatically detect emotional states in human speech signals. The working principle of these systems is the process of matching acoustic features extracted from voice signals with emotional labels through machine learning. Despite significant advancements, current SER technologies still face some challenges. Chief among these is the inadequate representation of local and global patterns. This study presents an innovative hybrid model to overcome these challenges in emotion representation. The model consists of two phases. The first phase represents the basic model, which involves an end-to-end learning process with the extraction of feature representations of emotions and their matching with emotion labels. The second phase is the process where feature optimization takes place. The first phase consists of steps including local feature extraction, channel-level feature activation, temporal dependency representation, and finally, focusing on key time periods. The second phase is the process of optimizing features using eighteen different optimization configurations derived from the integration of six meta-heuristic algorithms with three machine learning paradigms. Experiments were conducted on the EMO-DB dataset. In a 5-fold cross-validation method, the proposed hybrid model achieved an unweighted accuracy (UA) of 83.73%, demonstrating a 7.8% increase in unweighted accuracy compared to the base model, while reducing feature dimensionality by 54.8%. These results confirm that the proposed hybrid model provides high accuracy and efficient feature representations.

Keywords

• Speech emotion recognition
• Attention mechanism
• Metaheuristic optimization

References

1. [1] J. Ye et al., “Multi-modal depression detection based on emotional audio and evaluation text”, J. Affect. Disord., vol. 295, pp. 904-913, 2021.
2. [2] F. Yin, J. Du, X. Xu, and L. Zhao, “Depression Detection in Speech Using Transformer and Parallel Convolutional Neural Networks”, Electronics 2023, vol. 12, no. 2, p. 328, 2023.
3. [3] E. W. McGinnis et al., “Giving Voice to Vulnerable Children: Machine Learning Analysis of Speech Detects Anxiety and Depression in Early Childhood”, IEEE J. Biomed. Health Inform., vol. 23, no. 6, pp. 2294-2301, 2019.
4. [4] W. Xian, “Speech Emotion Recognition Application for Education”, BCP Education & Psychology, vol. 7, pp. 378-383, 2022.
5. [5] M. Płaza et al., “Emotion Recognition Method for Call/Contact Centre Systems”, Applied Sciences 2022, vol. 12, no. 21, p. 10951, 2022.
6. [6] Y. Feng and L. Devillers, “End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations”, 2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, ACIIW 2023, 2023.
7. [7] I. Gurowiec and N. Nissim, “Speech emotion recognition systems and their security aspects”, Artif. Intell. Rev., vol. 57, no. 6, pp. 1-45, 2024.
8. [8] Z. Jia, J. Yu, H. Long, and D. Tang, “Coverage-Guaranteed Speech Emotion Recognition via Calibrated Uncertainty-Adaptive Prediction Sets”, Accessed: Jul. 29, 2025. [Online]. Available: https://arxiv.org/abs/2503.22712v4

9. [9] R. Mishra, A. Frye, M. M. Rayguru, and D. O. Popa, “Personalized Speech Emotion Recognition in Human-Robot Interaction using Vision Transformers”, Accessed: Jul. 29, 2025. [Online]. Available: http://arxiv.org/abs/2409.10687
10. [10] E. Lakomkin, M. A. Zamani, C. Weber, S. Magg, and S. Wermter, “On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks”, IEEE International Conference on Intelligent Robots and Systems, pp. 854-860, 2018.
11. [11] L. Matsouliadis, E. Siamtanidou, N. Vryzas, and C. Dimoulas, “Speech Emotion Recognition and Serious Games: An Entertaining Approach for Crowdsourcing Annotated Samples”, Information, vol. 16, no. 3, p. 238, 2025.
12. [12] S. Padi, S. O. Sadjadi, D. Manocha, and R. D. Sriram, “Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models”, pp. 407-414, 2022.
13. [13] M. Jain et al., “Speech Emotion Recognition using Support Vector Machine”, Lecture Notes in Networks and Systems, vol. 1003 LNNS, pp. 519-532, 2020.
14. [14] S. Yan, L. Ye, S. Han, T. Han, Y. Li, and E. Alasaarela, “Speech Interactive Emotion Recognition System Based on Random Forest”, 2020 International Wireless Communications and Mobile Computing, IWCMC 2020, pp. 1458-1462, 2020.
15. [15] M. Venkata Subbarao, S. K. Terlapu, N. Geethika, and K. D. Harika, “Speech Emotion Recognition Using K-Nearest Neighbor Classifiers”, pp. 123-131, 2022.
16. [16] I. Shahin, A. B. Nassif, and S. Hamsa, “Emotion Recognition Using Hybrid Gaussian Mixture Model and Deep Neural Network”, IEEE Access, vol. 7, pp. 26777-26787, 2019.
17. [17] S. Mao, D. Tao, G. Zhang, P. C. Ching, and T. Lee, “Revisiting Hidden Markov Models for Speech Emotion Recognition”, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6715-6719, 2019.
18. [18] R. Kingeski, E. Henning, and A. S. Paterno, “Fusion of PCA and ICA in Statistical Subset Analysis for Speech Emotion Recognition”, Sensors 2024, vol. 24, no. 17, p. 5704, 2024.
19. [19] J. T. Chien and B. C. Chen, “A new independent component analysis for speech recognition and separation”, IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1245-1254, Jul. 2006.
20. [20] P. Tiwari, H. Rathod, S. Thakkar, and A. D. Darji, “Multimodal emotion recognition using SDA-LDA algorithm in video clips”, Journal of Ambient Intelligence and Humanized Computing, vol. 14, pp. 6585-6602, 2023.
21. [21] J. Zhao, X. Mao, and L. Chen, “Speech emotion recognition using deep 1D & 2D CNN LSTM networks”, Biomed. Signal Process. Control, vol. 47, pp. 312-323, 2019.
22. [22] C. Sun, H. Li, and L. Ma, “Speech emotion recognition based on improved masking EMD and convolutional recurrent neural network”, Front. Psychol., vol. 13, p. 1075624, 2023.
23. [23] K. Deshpande, M. S. Sodhi, N. Raniyer, and M. Rao, “A Time-Distributed CNN-LSTM with Attention Model for Speech Based Emotion Recognition”, DMIP 2024 - Proceedings of 2024 7th International Conference on Digital Medicine and Image Processing, pp. 67-71, 2025.
24. [24] H. Kang, Y. Xu, G. Jin, J. Wang, and B. Miao, “FCAN : Speech emotion recognition network based on focused contrastive learning”, Biomed. Signal Process. Control, vol. 96, p. 106545, 2024.
25. [25] A. Vaswani et al., “Attention Is All You Need”, NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, p. 6000-6010, 2017.
26. [26] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, pp. 4171-4186, 2019.
27. [27] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”, NIPS'20: Proceedings of the 34th International Conference on Neural Information Processing Systems, p. 12449-12460, 2020.
28. [28] W. N. Hsu, B. Bolte, Y. H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”, IEEE/ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451-3460, 2021.
29. [29] C. Etienne, G. Fidanza, A. Petrovskii, L. Devillers, and B. Schmauch, “CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation”, Proc. Workshop on Speech, Music and Mind (SMM 2018), pp. 21–25, 2018.
30. [30] F. Makhmudov, A. Kutlimuratov, and Y. I. Cho, “Hybrid LSTM–Attention and CNN Model for Enhanced Speech Emotion Recognition”, Applied Sciences 2024, vol. 14, no. 23, p. 11342, 2024.
31. [31] S. S. Poorna, V. Menon, and S. Gopalan, “Hybrid CNN-BiLSTM architecture with multiple attention mechanisms to enhance speech emotion recognition”, Biomed. Signal Process. Control, vol. 100, p. 106967, 2025.
32. [32] S. Mishra, N. Bhatnagar, P. Prakasam, and S. T. R, “Speech emotion recognition and classification using hybrid deep CNN and BiLSTM model”, Multimed. Tools Appl., vol. 83, no. 13, pp. 37603-37620, 2024.
33. [33] L. T. Van, T. T. Le Dao, T. Le Xuan, and E. Castelli, “Emotional Speech Recognition Using Deep Neural Networks”, Sensors (Basel)., vol. 22, no. 4, p. 1414, 2022.
34. [34] A. P. Gopi, P. Deepika, G. A. Gayathri, C. Keerthana, and B. V. Mounika, “Building Emotion Identification System from Speech Using CNN-GRU Model”, Advances in Science, Technology and Innovation, pp. 53-59, 2025.
35. [35] R. Hasan, M. Nigar, N. Mamun, and S. Paul, “EmoFormer: A Text-Independent Speech Emotion Recognition using a Hybrid Transformer-CNN model”, 2024 27th International Conference on Computer and Information Technology (ICCIT), pp. 2881-2886, 2024.
36. [36] S. Akinpelu, S. Viriri, and A. Adegun, “An enhanced speech emotion recognition using vision transformer”, Scientific Reports 2024, vol. 14, 13126, 2024.
37. [37] R. Ullah et al., “Speech Emotion Recognition Using Convolution Neural Networks and Multi-Head Convolutional Transformer”, Sensors (Basel)., vol. 23, no. 13, p. 6212, 2023.
38. [38] X. Tang, J. Huang, Y. Lin, T. Dang, and J. Cheng, “Speech Emotion Recognition via CNN-Transformer and multidimensional attention mechanism”, Speech Commun., vol. 171, p. 103242, 2025.
39. [39] Y. Xia and L. Zhao, “CNN-BLSTM with Attention Model for Speech Emotion Recognition”, Oct. 2023, doi: 10.21203/RS.3.RS-3392008/V1.
40. [40] N. K. Kundu, S. Kobir, Md. R. Ahmed, T. Aktar, and N. Roy, “Enhanced Speech Emotion Recognition with Efficient Channel Attention Guided Deep CNN-BiLSTM Framework”, 2024, Accessed: Jul. 21, 2025. [Online]. Available: http://arxiv.org/abs/2412.10011
41. [41] C. Xu, Y. Liu, W. Song, Z. Liang, and X. Chen, “A New Network Structure for Speech Emotion Recognition Research”, Sensors 2024, vol. 24, no. 5, p. 1429, 2024.
42. [42] S. Lee, D. K. Han, and H. Ko, “Fusion-ConvBERT: Parallel Convolution and BERT Fusion for Speech Emotion Recognition”, Sensors (Basel)., vol. 20, no. 22, p. 6688, 2020.
43. [43] B. Maji, M. Swain, and Mustaqeem, “Advanced Fusion-Based Speech Emotion Recognition System Using a Dual-Attention Mechanism with Conv-Caps and Bi-GRU Features”, Electronics 2022, vol. 11, no. 9, p. 1328, 2022.
44. [44] C. Barhoumi and Y. BenAyed, “Real-time speech emotion recognition using deep learning and data augmentation”, Artif. Intell. Rev., vol. 58, 49, 2025.
45. [45] GitHub- rcantini/speech_emotion_recognition: How to detect emotions from speech using Bi-directional LSTM networks and attention mechanism in Keras.” Accessed: Aug. 11, 2025. [Online]. Available: https://github.com/rcantini/speech_emotion_recognition.
46. [46] F. Burkhardt, A. Paeschke, M. Rolfes, W. Sendlmeier, and B. Weiss, “A database of German emotional speech”, 9th European Conference on Speech Communication and Technology, pp. 1517-1520, 2005.
47. [47] S. B. Davis and P. Mermelstein, “Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences”, IEEE Trans. Acoust., vol. 28, no. 4, pp. 357-366, 1980.
48. [48] K. DONUK, “KONUŞMA DUYGU TANIMA İÇİN ÖZELLİK ANALİZİ”, Yapay Zekâ Tabanlı Sistemler: Teori, Uygulama ve Gelecek Perspektifleri-2, BİDGE Yayınları , Dec. 2025, 213-235.
49. [49] N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete Cosine Transform”, IEEE Transactions on Computers, vol. C–23, no. 1, pp. 90-93, 1974.
50. [50] J. Kennedy and R. Eberhart, “Particle swarm optimization”, Proceedings of ICNN’95 - International Conference on Neural Networks, vol. 4, pp. 1942-1948, 1995.
51. [51] K. Donuk, N. Özbey, M. Inan, C. Yeroǧlu, and D. Hanbay, “Investigation of PIDA Controller Parameters via PSO Algorithm”, 2018 International Conference on Artificial Intelligence and Data Processing, IDAP 2018, 2018.
52. [52] K. DONUK, A. ARI, M. F. ÖZDEMİR, and D. HANBAY, “Deep Feature Selection for Facial Emotion Recognition Based on BPSO and SVM”, Politeknik Dergisi, vol. 26, no. 1, pp. 131-142, 2023.
53. [53] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing”, Science (1979)., vol. 220, no. 4598, pp. 671-680, 1983.
54. [54] D. E. Goldberg and J. H. Holland, “Genetic Algorithms and Machine Learning”, Mach. Learn., vol. 3, no. 2, pp. 95-99, 1988.
55. [55] J. H. Holland, “Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence”, Adaptation in Natural and Artificial Systems, Apr. 1992.
56. [56] S. Mirjalili, “Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems”, Neural Comput. Appl., vol. 27, no. 4, pp. 1053-1073, 2016.
57. [57] D. Karaboga and B. Basturk, “Artificial Bee Colony (ABC) optimization algorithm for solving constrained optimization problems”, Lecture Notes in Computer Science, vol. 4529, pp. 789-798, 2007.
58. [58] M. Dorigo, V. Maniezzo, and A. Colorni, “Ant system: Optimization by a colony of cooperating agents”, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 26, no. 1, pp. 29-41, 1996.
59. [59] C. Cortes and V. Vapnik, “Support-vector networks”, Machine Learning 1995, vol. 20, no. 3, pp. 273-297, 1995.
60. [60] T. M. Cover and P. E. Hart, “Nearest Neighbor Pattern Classification”, IEEE Trans. Inf. Theory, vol. 13, no. 1, pp. 21-27, 1967.
61. [61] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors”, Nature 1986, vol. 323, no. 6088, pp. 533-536, 1986.
62. [62] Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks”, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 11531-11539, 2019.
63. [63] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation Networks”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 8, pp. 2011-2023, 2017.
64. [64] D. Bahdanau, K. H. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2014.
65. [65] K. Donuk et al., “Konuşma Duygu Tanıma için Akustik Özelliklere Dayalı LSTM Tabanlı Bir Yaklaşım”, Bilgisayar Bilimleri, vol. 7, no. 2, pp. 54-67, Dec. 2022.
66. [66] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks” IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 2673-2681, 1997.
67. [67] P. Singh, M. Sahidullah, and G. Saha, “Modulation spectral features for speech emotion recognition using deep neural networks”, Speech Commun., vol. 146, pp. 53-69, 2023.
68. [68] K. M. Roh, S. P. Lee, “Enhanced Speech Emotion Recognition Using Conditional-DCGAN-Based Data Augmentation”, Applied Sciences 2024, vol. 14, no. 21, p. 9890, 2024.
69. [69] Z. T. Liu, A. Rehman, M. Wu, W. H. Cao, and M. Hao, “Speech emotion recognition based on formant characteristics feature extraction and phoneme type convergence”, Inf. Sci. (N Y)., vol. 563, pp. 309-325, 2021.
70. [70] H. Li, X. Zhang, S. Duan, and H. Liang, “Speech emotion recognition based on bi-directional acoustic–articulatory conversion”, Knowl. Based. Syst., vol. 299, 112123, 2024.
71. [71] J. Ancilin and A. Milton, “Improved speech emotion recognition with Mel frequency magnitude coefficient”, Applied Acoustics, vol. 179, 108046, 2021.
72. [72] M. Hou, J. Li, and G. Lu, “A supervised non-negative matrix factorization model for speech emotion recognition”, Speech Commun., vol. 124, pp. 13-20, 2020.
73. [73] S. P. Mishra, P. Warule, and S. Deb, “Deep Learning Based Emotion Classification Using Mel Frequency Magnitude Coefficient”, 1st IEEE International Conference on Innovations in High Speed Communication and Signal Processing, IHCSP 2023, pp. 93-98, 2023.
74. [74] D. Issa, M. Fatih Demirci, and A. Yazici, “Speech emotion recognition with deep convolutional neural networks”, Biomed. Signal Process. Control, vol. 59, 101894, 2020.
75. [75] S. P. Mishra, P. Warule, and S. Deb, “Variational mode decomposition based acoustic and entropy features for speech emotion recognition”, Applied Acoustics, vol. 212, 109578, 2023.