MFCCFusionNet: MFCC temsilleri kullanılarak konuşma duygu tanıma için CNN–DNN tabanlı bir özellik birleştirme modeli

Abstract

Konuşma duygu tanıma, insan–makine etkileşimini geliştirme ve duygu farkındalığına sahip akıllı sistemlerin geliştirilmesini destekleme potansiyeli nedeniyle önemli bir araştırma alanı hâline gelmiştir. Bununla birlikte, aynı akustik özellik temsili kullanılarak eğitilen farklı derin öğrenme mimarilerinin nasıl performans gösterdiğine ilişkin daha fazla karşılaştırmalı kanıta ihtiyaç duyulmaktadır. Bu çalışmada, RAVDESS veri seti kullanılarak deneysel bir karşılaştırma yapılmıştır. Mel-frekansı kepstral katsayıları (MFCC’ler) tek özellik çıkarım yöntemi olarak kullanılmış ve üç derin öğrenme mimarisi tasarlanarak değerlendirilmiştir: Evrişimli Sinir Ağı (CNN), Derin Sinir Ağı (DNN) ve MFCCFusionNet olarak adlandırılan hibrit özellik birleştirme modeli. CNN ve DNN modelleri sırasıyla %81 ve %76 sınıflandırma doğruluğu elde ederken, önerilen MFCC-Fuse modeli %83 test doğruluğu ile değerlendirilen modeller arasında en yüksek başarıyı göstermiştir. Bulgular, MFCC tabanlı temsillerin farklı derin öğrenme mimarileri tarafından etkili biçimde işlenebildiğini ve özellik düzeyinde birleştirmenin konuşma duygu tanıma performansında sınırlı fakat olumlu bir iyileşme sağlayabileceğini göstermektedir. Genel olarak çalışma, MFCC özelliklerinden öğrenilen CNN ve DNN tabanlı temsillerin birleştirilmesinin konuşma tabanlı duygu tanıma modellerinin ayırt edici kapasitesini artırabileceğine ilişkin ön kanıt sunmaktadır.

Keywords

• Konuşma duygu tanıma
• MFCC
• derin öğrenme mimarileri
• özellik birleştirme
• CNN
• DNN

MFCCFusionnet: A CNN–DNN feature-fusion model for speech emotion recognition using MFCC representations

Abstract

Speech emotion recognition has become an increasingly important research area because of its potential to enhance human–machine interaction and support the development of emotion-aware systems. However, further comparative evidence is still needed on how different deep learning architectures perform when they are trained using the same acoustic feature representation. In this study, an experimental comparison was conducted using the RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) dataset. Mel-frequency cepstral coefficients (MFCCs) were used as the sole feature extraction method, and three deep learning architectures were designed and evaluated: a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), and a hybrid feature-fusion model named MFCCFusionNet. The CNN and DNN models achieved classification accuracies of 81% and 76%, respectively. The proposed MFCCFusionNet model combined the learned representations obtained from the CNN and DNN sub-models and achieved the highest accuracy among the evaluated models, with a test accuracy of 83%. The findings indicate that MFCC-based representations can be effectively processed by different deep learning architectures and that feature-level fusion may improve classification performance in speech-based emotion recognition. The study provides preliminary evidence that feature-level fusion of CNN- and DNN-based MFCC representations may improve classification performance in speech emotion recognition.

Keywords

• Speech emotion recognition
• MFCC
• deep learning architectures

References

1. Zhao, J., Mao, X., & Chen, L. (2019). Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control, 47, 312-323.
2. Makhmudov, F., Kutlimuratov, A., & Cho, Y. I. (2024). Hybrid LSTM– Attention and CNN Model for Enhanced Speech Emotion Recognition. Applied Sciences, 14(23), 11342.
3. Dinesh, P., Mishra, S. P., Warule, P., & Deb, S. (2024, December). Speech Emotion Recognition with DNN and Combination of CNN-LSTM. In TENCON 2024-2024 IEEE Region 10 Conference (TENCON) (pp. 1159-1162). IEEE.
4. Yang, Z., Li, Z., Zhou, S., Zhang, L., & Serikawa, S. (2024). Speech emotion recognition based on multi-feature speed rate and LSTM. Neurocomputing, 601, 128177.
5. Shetty, K. J., Shetty, S., & Shetty, M. (2024, April). Speech emotion recognition using lstm. In 2024 Third International Conference on Distributed Computing and Electrical Circuits and Electronics (ICDCECE) (pp. 1-6). IEEE.
6. Şeker, A., Diri, B., Balık, H. H. (2017). Derin öğrenme yöntemleri ve uygulamaları hakkında bir inceleme. Gazi Mühendislik Bilimleri Dergisi, 3(3), 47–64.
7. Ezz-Eldin, M., Khalaf, A. A. M., Hamed, H. F. A., & Hussein, A. I. (2021). Efficient feature-aware hybrid model of deep learning architectures for speech emotion recognition. IEEE Access, 9, 19999– 20011.
8. Jakubec, M., Lieskovska, E., Jarina, R., Spisiak, M., & Kasak, P. (2024). Speech emotion recognition using transfer learning: Integration of advanced speaker embeddings and image recognition models. Applied Sciences, 14(21), 9981.

9. Rajapakshe, T., Rana, R., Khalifa, S., Sisman, B., Schuller, B. W., & Busso, C. (2024). emodarts: Joint optimisation of cnn & sequential neural network architectures for superior speech emotion recognition. IEEE Access.
10. Ulgen, I. R., Du, Z., Busso, C., Sisman, B. (2024). Revealing emotional clusters in speaker embeddings: A contrastive learning strategy for speech emotion recognition. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 12081–12085.
11. Madan, P., Dixit, D., Shobana, J. (2024). Speech emotion recognition system using MLP Classifier Algorithm. 2024 International Conference on Communication, Computing and Internet of Things (IC3IoT), 1–6.
12. Şengül, F., Akkaya, S. (2024). A Modified MFCC-based deep Learning method for emotion classification from speech. International Advanced Researches and Engineering Journal, 8(1), 33–42.
13. Akinpelu, S., Viriri, S. (2024). Deep Learning Framework for Speech Emotion Classification: A Survey of the State-of-the-Art. IEEE Access.
14. Chowdhury, J. H., Ramanna, S., & Kotecha, K. (2025). Speech emotion recognition with light weight deep neural ensemble model using hand crafted features. Scientific Reports, 15(1), 11824.
15. Donuk, K., Hanbay, D. (2022). Konuşma Duygu Tanıma için Akustik Özelliklere Dayalı LSTM Tabanlı Bir Yaklaşım. Computer Science, 7(2), 54–67.
16. Yetkin, A. K., Köse, H. (2023, July). Speech-Based Emotion Analysis Using Log-Mel Spectrograms and MFCC Features. In 2023 31st Signal Processing and Communications Applications Conference (SIU) (pp. 1-4). IEEE.
17. Kılıç, F. R. (2023). Modlar Arası Transfer Öğrenimi İle Ses Sinyallerinden Duygu Tanıma.
18. Mountzouris, K., Perikos, I., & Hatzilygeroudis, I. (2023). Speech emotion recognition using convolutional neural networks with attention mechanism. Electronics, 12(20), 4376.
19. Middya, A. I., Nag, B., Roy, S. (2024). Effective MLP and CNN based ensemble learning for speech emotion recognition. Multimedia Tools and Applications, 83(36), 83963-83990.
20. Kumar, C. A., Maharana, A. D., Krishnan, S. M., Hanuma, S. S. S., Lal, G. J., Ravi, V. (2022, December). Speech emotion recognition using cnn-lstm and vision transformer. In International Conference on Innovations in Bio-Inspired Computing and Applications (pp. 86-97). Cham: Springer Nature Switzerland.
21. Zeng, Y., Mao, H., Peng, D. Yi, Z. Spectrogram based multi-task audio classification. Multimed. Tools Appl. 78, 3705–3722 (2019).
22. Issa, D., Demirci, M. F. & Yazici, A. Speech emotion recognition with deep convolutional neural networks. Biomed. Signal Process. Control 59, 101894 (2020).
23. Mustaqeem, M., Sajjad, M., & Kwon, S. (2020). Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM. IEEE Access, 8, 79861–79875.
24. Kim, S., Lee, S.-P. (2023). A BiLSTM–Transformer and 2D CNN Architecture for Emotion Recognition from Speech. Electronics, 12(19), 4034.
25. Akinpelu, S., Viriri, S., Adegun, A. (2023). Lightweight deep learning framework for speech emotion recognition. IEEE Access, 11, 77086- 77098.