Comparative Analysis of Machine Learning and Deep Learning Models for Speech Emotion Recognition

Abstract

Speech Emotion Recognition (SER) is a research topic of critical importance in areas such as human-computer interaction, health monitoring, and customer services. In this study, four benchmark datasets (RAVDESS, CREMA-D, TESS, SAVEE) were merged into a corpus of 12,162 recordings from 121 speakers, and seven machine learning and deep learning models were evaluated comparatively. Evaluation was carried out under a speaker-independent protocol using speaker-wise stratified group cross-validation (5 folds × 3 repeats), complemented by a leave-one-dataset-out protocol in which each corpus served in turn as the test set. The principal finding is that the evaluation protocol matters more than the model architecture: random splitting that permits speaker overlap produces an optimistic bias of 14.61 percentage points on average across all seven models, which exceeds the 11.24-point gap between the best and worst model. Under the speaker-independent protocol the highest balanced accuracy was obtained by the MLP (52.39%), with no significant difference from CNN-BiLSTM-Attention; since the performance ranges of traditional machine learning and deep learning models overlap, no claim of superiority at the model-family level is supported. In the cross-corpus evaluation the best model reached only 29.3% balanced accuracy on the six-class problem, approaching the chance level (16.67%) for two of the four corpora. When computational cost is measured end to end, including feature extraction time, the ranking reverses: Random Forest becomes the slowest model (211.6 ms per recording) and the 1D-CNN the fastest (37.0 ms). The findings indicate that both the evaluation protocol and end-to-end cost measurement must be reported explicitly in SER studies.

Keywords

• Speech emotion recognition
• deep learning
• machine learning
• MFCC
• CNN-BiLSTM
• attention mechanism

Konuşma Sinyallerinden Duygu Tanıma İçin Makine Öğrenmesi ve Derin Öğrenme Modellerinin Karşılaştırmalı Analizi

Abstract

Konuşma tabanlı duygu tanıma (Speech Emotion Recognition — SER), insan-bilgisayar etkileşimi, sağlık izleme ve müşteri hizmetleri gibi alanlarda kritik öneme sahip bir araştırma konusudur. Bu çalışmada dört referans veri seti (RAVDESS, CREMA-D, TESS, SAVEE) birleştirilerek 121 konuşmacıya ait 12.162 ses kaydı üzerinde yedi makine öğrenmesi ve derin öğrenme modeli karşılaştırmalı olarak değerlendirilmiştir. Değerlendirme, konuşmacıya göre gruplanmış katmanlı çapraz doğrulama (5 kat × 3 tekrar) ile konuşmacı-bağımsız olarak yürütülmüş; ayrıca her veri setinin sırayla test kümesi olarak kullanıldığı veri setleri arası protokol uygulanmıştır. Çalışmanın temel bulgusu, değerlendirme protokolünün model mimarisinden daha belirleyici olmasıdır: konuşmacı bağımlılığına izin veren rastgele bölme, yedi modelin tamamında ortalama 14,61 puanlık iyimser bir yanlılık üretmektedir ve bu, en iyi ile en kötü model arasındaki farktan (11,24 puan) büyüktür. Konuşmacı-bağımsız protokolde en yüksek dengeli doğruluğu MLP (%52,39) göstermiş, CNN-BiLSTM-Attention ile arasında anlamlı fark bulunmamıştır; geleneksel makine öğrenmesi ile derin öğrenme modellerinin başarım aralıkları örtüştüğünden model ailesi düzeyinde bir üstünlük iddiası desteklenmemektedir. Veri setleri arası değerlendirmede en iyi model altı sınıflı problemde %29,3 dengeli doğrulukta kalmış, iki veri seti için başarım şans düzeyine (%16,67) yaklaşmıştır. Hesaplama maliyeti özellik çıkarma süresi dâhil edilerek uçtan uca ölçüldüğünde sıralama tersine dönmekte, Random Forest en yavaş (211,6 ms/kayıt), 1D-CNN ise en hızlı (37,0 ms/kayıt) model olmaktadır. Bulgular, SER çalışmalarında protokol seçiminin ve uçtan uca maliyet ölçümünün açıkça raporlanması gerektiğini göstermektedir.

Keywords

• Konuşma duygu tanıma
• derin öğrenme
• makine öğrenmesi
• MFCC
• CNN-BiLSTM
• dikkat mekanizması

References

1. [1] El Ayadi, M., Kamel, M. S., & Karray, F. (2011). Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44(3), 572-587.
2. [2] Akçay, M. B., & Oğuz, K. (2020). Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication, 116, 56-76.
3. [3] Schuller, B. W. (2018). Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Communications of the ACM, 61(5), 90-99.
4. [4] Zhao, J., Mao, X., & Chen, L. (2019). Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control, 47, 312-323.
5. [5] Lian, Z., Liu, B., & Tao, J. (2021). CTNet: Conversational transformer network for emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 985-1000.
6. [6] Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003). Speech emotion recognition using hidden Markov models. Speech Communication, 41(4), 603-623.
7. [7] Jiang, W., Wang, Z., Jin, J. S., Han, X., & Li, C. (2019). Speech emotion recognition with heterogeneous feature unification of deep neural network. Sensors, 19(12), 2730.
8. [8] Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014). Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Transactions on Multimedia, 16(8), 2203-2213.

9. [9] Badshah, A. M., Ahmad, J., Rahim, N., & Baik, S. W. (2017). Speech emotion recognition from spectrograms with deep convolutional neural network. In International Conference on Platform Technology and Service (pp. 1-5).
10. [10] Mirsamadi, S., Barsoum, E., & Zhang, C. (2017). Automatic speech emotion recognition using recurrent neural networks with local attention. In IEEE ICASSP (pp. 2227-2231).
11. [11] Luo, D., Zou, Y., & Huang, D. (2018). Investigation on joint representation learning for robust feature extraction in speech emotion recognition. In INTERSPEECH (pp. 152-156).
12. [12] Li, Y., Zhao, T., & Kawahara, T. (2019). Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning. In INTERSPEECH (pp. 2803-2807).
13. [13] Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). PLoS ONE, 13(5), e0196391.
14. [14] Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014). CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4), 377-390.
15. [15] Dupuis, K., & Pichora-Fuller, M. K. (2010). Toronto Emotional Speech Set (TESS). University of Toronto, Psychology Department.
16. [16] Haq, S., & Jackson, P. J. B. (2008). Speaker-dependent audio-visual emotion recognition. In International Conference on Auditory-Visual Speech Processing (pp. 53-58).
17. [17] Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1-2), 227-256.
18. [18] Ahmed, M. R., Islam, S., Islam, A. K. M. M., & Shatabda, S. (2023). An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition. Expert Systems with Applications, 218, 119633.
19. [19] Ma, Z., Zheng, Z., Ye, J., Li, J., Gao, Z., Zhang, S., & Chen, X. (2024). emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 15747-15760).
20. [20] Gao, Y., Wang, L., Liu, J., Dang, J., & Okada, S. (2024). Adversarial domain generalized transformer for cross-corpus speech emotion recognition. IEEE Transactions on Affective Computing, 15(2), 697-708.
21. [21] Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460.
22. [22] Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451-3460.
23. [23] Schuller, B., Vlasenko, B., Eyben, F., Wöllmer, M., Stuhlsatz, A., Wendemuth, A., & Rigoll, G. (2010). Cross-corpus acoustic emotion recognition: Variances and strategies. IEEE Transactions on Affective Computing, 1(2), 119-131.
24. [24] Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1-30.
25. [25] Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895-1923.
26. [26] Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239-281.
27. [27] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.