The effects of sensor and feature level fusion methods in multimodal emotion analysis

Year 2024, Volume: 13 Issue: 4, 1093 - 1099, 15.10.2024

Bahar Hatipoğlu Yılmqz , Cemal Köse

https://doi.org/10.28948/ngumuh.1469289

Abstract

Fusion-based studies on multimodal emotion recognition (MER) are very popular nowadays. In this study, EEG signals and facial images are fused using Sensor Level Fusion (SLF) and Feature Level Fusion (FLF) methods for multimodal emotion recognition. The general procedure of the study is as follows. First, the EEG signals are converted into angle amplitude graph (AAG) images. Second, the most unique ones are automatically identified from all face images obtained from video recordings. Then, these modalities are fused separately using SLF and FLF methods. The fusion approaches were used to combine the obtained data and perform classification on the integrated data. The experiments were performed on the publicly available DEAP dataset. The highest accuracy was 82.14% with 5.26 standard deviations for SLF and 87.62% with 6.74 standard deviations for FLF. These results show that this study makes an important contribution to the field of emotion recognition by providing an effective method.

Keywords

Multimodal emotion recognition, Sensor level fusion, Feature level fusion, DEAP dataset

Project Number

121E002

Çok modlu duygu analizinde sensör ve özellik seviyesi füzyon yöntemlerinin etkileri

Abstract

Füzyon tabanlı çok modlu duygu tanıma (MER) çalışmaları günümüzde oldukça popülerdir. Bu çalışmada, çok modlu duygu tanıma için EEG sinyalleri ve yüz görüntüleri sensör seviyesinde füzyon (SLF) ve öznitelik seviyesinde füzyon (FLF) yöntemleri ile birleştirilmiştir. Çalışmanın genel akışı şu şekildedir. İlk olarak EEG sinyalleri açı genlik grafiği (AAG) görüntülerine dönüştürülmektedir. İkinci olarak, video kayıtlarından elde edilen tüm yüz görüntülerinden en benzersiz olanlar otomatik olarak belirlenmektedir. Daha sonra, bu modaliteler SLF ve FLF yöntemleri kullanılarak ayrı ayrı birleştirilmektedir. Elde edilen verileri birleştirmek ve bütünleşik veriler üzerinde sınıflandırma yapmak için füzyon yaklaşımlar kullanılmıştır. Deneyler halka açık DEAP veri kümesi üzerinde gerçekleştirilmiştir. En yüksek doğruluk SLF için 5,26 standart sapma ile %82,14 ve FLF için 6,74 standart sapma ile %87,62 olarak elde edilmiştir. Bu sonuçlar, bu çalışmanın etkili bir yöntem sunarak duygu tanıma alanına önemli bir katkı sağladığını göstermektedir.

Keywords

Çok modlu duygu tanıma, Sensör seviyesinde füzyon, Öznitelik seviyesinde füzyon, DEAP veri seti

Project Number

121E002

Thanks

This research was supported by the Turkish Scientific and Research Council (TUBITAK) through project 121E002 and 119E397.

References

• A. F. M. N. H. Nahin, J. M. Alam, H. Mahmud and K. Hasan, Identifying emotion by keystroke dynamics and text pattern analysis. Behaviour & Information Technology, 33(9), 987–996, 2014. https://doi.org/10.1080/0144929X.2014.907343.
• A. Sapra, N. Panwar, and S. Panwar, Emotion recognition from speech. International journal of emerging technology and advanced engineering, 3(2), 341-345, 2013.
• G. K. Verma, and U. S. Tiwary, Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage, 102, 162-172, 2014. https://doi.org/10.1016/j.neuroimage.2013.11.007.
• S. Luo, Y. T. Lan, D. Peng, Z. Li, W. L. Zheng, and B. L. Lu, Multimodal Emotion Recognition in Response to Oil Paintings. 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4167-4170, 2022. https://doi.org/10.1109/EMBC48229.2022.9871630.
• J. N. Njoku, A. C. Caliwag, W. Lim, S. Kim, H. Hwang, and J. Jung, Deep learning-based data fusion methods for multimodal emotion recognition. The Journal of Korean Institute of Communications and Information Sciences, 47(1), 79-87, 2022. https://doi.org/10.1109/ 10.7840/kics.2022.47.1.79.
• J. Pan, W. Fang, Z. Zhang, B. Chen, Z. Zhang, and S. Wang, Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG. IEEE Open Journal of Engineering in Medicine and Biology, 2023. https://doi.org/10.1109/10.7840/10.1109/OJEMB.2023.3240280.
• R. Li, Y. Liang, X. Liu, B. Wang, W. Huang, Z. Cai, and J. Pan, MindLink-eumpy: an open-source python toolbox for multimodal emotion recognition. Frontiers in Human Neuroscience, 15, 621493, 2021. https://doi.org/10.3389/fnhum.2021.621493.
• Y. Zhao and D. Chen, Expression eeg multimodal emotion recognition method based on the bidirectional lstm and attention mechanism. Computational and Mathematical Methods in Medicine, 1-12, 2021. https://doi.org/10.1155/2021/9967592.
• Y. Huang, J. Yang, S. Liu, J. Pan, Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition. Future Internet, 11(5):105, 2019. https://doi.org/10.3390/fi11050105.
• Z. Yin, M. Zhao, Y. Wang, J. Yang, and J. Zhang, Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer methods and programs in biomedicine, 140, 93-110, 2017. https://doi.org/10.1016/j.cmpb.2016.12.005.
• S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, and I. Patras, Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, 3(1), 18-31, 2011. https://doi.org/10.1109/T-AFFC.2011.15.
• B. Hatipoglu, C. M. Yilmaz and C. Kose, A signal-to-image transformation approach for EEG and MEG signal classification. Signal Image and Video Processing, 13, 483–490, 2019. https://doi.org/10.1007/s11760-018-1373-y.
• B. Hatipoglu Yilmaz, C. M. Yilmaz and C. Kose, Diversity in a signal-to-image transformation approach for EEG-based motor imagery task classification. Medical Biological Engineering Computing, 58, 443–459, 2020. https://doi.org/10.1007/s11517-019-02075-x.
• B. Hatipoglu Yilmaz and C. Kose, A novel signal to image transformation and feature level fusion for multimodal emotion recognition. Biomedical Engineering / Biomedizinische Technik, 66 (4), 353-362, 2021. https://doi.org/10.1515/bmt-2020-0229.
• S. Zhalehpour, Z. Akhtar, and C. Eroglu Erdem, Multimodal emotion recognition based on peak frame selection from video. Signal, Image and Video Processing, 10, 827-834, 2016. https://doi.org/10.1109/INISTA.2014.6873606.
• W. Yu, L. Gan, S. Yang, Y. Ding, P. Jiang, J. Wang and S. Li, An improved LBP algorithm for texture and face classification. Signal Image and Video Processing, 8, 155–161, 2014. https://doi.org/10.1007/s11760-014-0652-5.
• Matti Pietikäinen, Local Binary Patterns. http://www.scholarpedia.org/article/Local_Binary_Patterns, Accessed 13 March 2024.
• C. Turan, and K. M. Lam, Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study. Journal of visual communication and image representation, 55, 331-341, 2018. https://doi.org/10.1016/j.jvcir.2018.05.024.
• Behzad Javaheri, KNN with Examples in Python, https://domino.ai/blog/knn-with-examples-in-python, Accessed 14 March 2024.
• T. Fletcher, Support vector machines explained. Tutorial paper, 1-19, 2009.
• C. J. Burges, A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery, 2(2), 121-167, 1998.
• A. Yuexuan, D. Shifei, S. Songhui and L. Jingcan, Discrete space reinforcement learning algorithm based on support vector machine classification. Pattern Recognition Letters, 111, 30-35, 2018. https://doi.org/10.1016/j.patrec.2018.04.012.
• T. Q. Anh, P. Bao, T. T. Khanh, B. N. D. Thao, T. A. Tuan and N. T. Nhut, Video retrieval using histogram and sift combined with graph-based image segmentation. Journal of Computer Science, 8(6), 853, 2012. https://doi.org/10.3844/jcssp.2012.853.858.
• N. D. Anh, P.T. Bao, B. N. Nam, N.H. Hoang, A New CBIR System Using SIFT Combined with Neural Network and Graph-Based Segmentation. In: Nguyen, N.T., Le, M.T., Świątek, J. (eds) Intelligent Information and Database Systems. Lecture Notes in Computer Science, 5990. Springer, Berlin, Heidelberg, 2010. https://doi.org/10.1007/978-3-642-12145-6_30.
• J. Xu, K. Lu, X. Shi, S. Qin, H. Wang and J. Ma, A DenseUnet generative adversarial network for near-infrared face image colorization. Signal Processing, 183, 108007, 2021. https://doi.org/ 10.1016/j.sigpro.2021.108007.
• H. Lacheheb, and S. Aouat, SIMIR: new mean SIFT color multi-clustering image retrieval. Multimedia Tools and Applications, 76, 6333-6354, 2017. https://doi.org/10.1007/s11042-015-3167-3.