Araştırma Makalesi
BibTex RIS Kaynak Göster

The effects of sensor and feature level fusion methods in multimodal emotion analysis

Yıl 2024, Cilt: 13 Sayı: 4, 1093 - 1099, 15.10.2024
https://doi.org/10.28948/ngumuh.1469289

Öz

Fusion-based studies on multimodal emotion recognition (MER) are very popular nowadays. In this study, EEG signals and facial images are fused using Sensor Level Fusion (SLF) and Feature Level Fusion (FLF) methods for multimodal emotion recognition. The general procedure of the study is as follows. First, the EEG signals are converted into angle amplitude graph (AAG) images. Second, the most unique ones are automatically identified from all face images obtained from video recordings. Then, these modalities are fused separately using SLF and FLF methods. The fusion approaches were used to combine the obtained data and perform classification on the integrated data. The experiments were performed on the publicly available DEAP dataset. The highest accuracy was 82.14% with 5.26 standard deviations for SLF and 87.62% with 6.74 standard deviations for FLF. These results show that this study makes an important contribution to the field of emotion recognition by providing an effective method.

Proje Numarası

121E002

Kaynakça

  • A. F. M. N. H. Nahin, J. M. Alam, H. Mahmud and K. Hasan, Identifying emotion by keystroke dynamics and text pattern analysis. Behaviour & Information Technology, 33(9), 987–996, 2014. https://doi.org/10.1080/0144929X.2014.907343.
  • A. Sapra, N. Panwar, and S. Panwar, Emotion recognition from speech. International journal of emerging technology and advanced engineering, 3(2), 341-345, 2013.
  • G. K. Verma, and U. S. Tiwary, Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage, 102, 162-172, 2014. https://doi.org/10.1016/j.neuroimage.2013.11.007.
  • S. Luo, Y. T. Lan, D. Peng, Z. Li, W. L. Zheng, and B. L. Lu, Multimodal Emotion Recognition in Response to Oil Paintings. 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4167-4170, 2022. https://doi.org/10.1109/EMBC48229.2022.9871630.
  • J. N. Njoku, A. C. Caliwag, W. Lim, S. Kim, H. Hwang, and J. Jung, Deep learning-based data fusion methods for multimodal emotion recognition. The Journal of Korean Institute of Communications and Information Sciences, 47(1), 79-87, 2022. https://doi.org/10.1109/ 10.7840/kics.2022.47.1.79.
  • J. Pan, W. Fang, Z. Zhang, B. Chen, Z. Zhang, and S. Wang, Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG. IEEE Open Journal of Engineering in Medicine and Biology, 2023. https://doi.org/10.1109/10.7840/10.1109/OJEMB.2023.3240280.
  • R. Li, Y. Liang, X. Liu, B. Wang, W. Huang, Z. Cai, and J. Pan, MindLink-eumpy: an open-source python toolbox for multimodal emotion recognition. Frontiers in Human Neuroscience, 15, 621493, 2021. https://doi.org/10.3389/fnhum.2021.621493.
  • Y. Zhao and D. Chen, Expression eeg multimodal emotion recognition method based on the bidirectional lstm and attention mechanism. Computational and Mathematical Methods in Medicine, 1-12, 2021. https://doi.org/10.1155/2021/9967592.
  • Y. Huang, J. Yang, S. Liu, J. Pan, Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition. Future Internet, 11(5):105, 2019. https://doi.org/10.3390/fi11050105.
  • Z. Yin, M. Zhao, Y. Wang, J. Yang, and J. Zhang, Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer methods and programs in biomedicine, 140, 93-110, 2017. https://doi.org/10.1016/j.cmpb.2016.12.005.
  • S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, and I. Patras, Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, 3(1), 18-31, 2011. https://doi.org/10.1109/T-AFFC.2011.15.
  • B. Hatipoglu, C. M. Yilmaz and C. Kose, A signal-to-image transformation approach for EEG and MEG signal classification. Signal Image and Video Processing, 13, 483–490, 2019. https://doi.org/10.1007/s11760-018-1373-y.
  • B. Hatipoglu Yilmaz, C. M. Yilmaz and C. Kose, Diversity in a signal-to-image transformation approach for EEG-based motor imagery task classification. Medical Biological Engineering Computing, 58, 443–459, 2020. https://doi.org/10.1007/s11517-019-02075-x.
  • B. Hatipoglu Yilmaz and C. Kose, A novel signal to image transformation and feature level fusion for multimodal emotion recognition. Biomedical Engineering / Biomedizinische Technik, 66 (4), 353-362, 2021. https://doi.org/10.1515/bmt-2020-0229.
  • S. Zhalehpour, Z. Akhtar, and C. Eroglu Erdem, Multimodal emotion recognition based on peak frame selection from video. Signal, Image and Video Processing, 10, 827-834, 2016. https://doi.org/10.1109/INISTA.2014.6873606.
  • W. Yu, L. Gan, S. Yang, Y. Ding, P. Jiang, J. Wang and S. Li, An improved LBP algorithm for texture and face classification. Signal Image and Video Processing, 8, 155–161, 2014. https://doi.org/10.1007/s11760-014-0652-5.
  • Matti Pietikäinen, Local Binary Patterns. http://www.scholarpedia.org/article/Local_Binary_Patterns, Accessed 13 March 2024.
  • C. Turan, and K. M. Lam, Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study. Journal of visual communication and image representation, 55, 331-341, 2018. https://doi.org/10.1016/j.jvcir.2018.05.024.
  • Behzad Javaheri, KNN with Examples in Python, https://domino.ai/blog/knn-with-examples-in-python, Accessed 14 March 2024.
  • T. Fletcher, Support vector machines explained. Tutorial paper, 1-19, 2009.
  • C. J. Burges, A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery, 2(2), 121-167, 1998.
  • A. Yuexuan, D. Shifei, S. Songhui and L. Jingcan, Discrete space reinforcement learning algorithm based on support vector machine classification. Pattern Recognition Letters, 111, 30-35, 2018. https://doi.org/10.1016/j.patrec.2018.04.012.
  • T. Q. Anh, P. Bao, T. T. Khanh, B. N. D. Thao, T. A. Tuan and N. T. Nhut, Video retrieval using histogram and sift combined with graph-based image segmentation. Journal of Computer Science, 8(6), 853, 2012. https://doi.org/10.3844/jcssp.2012.853.858.
  • N. D. Anh, P.T. Bao, B. N. Nam, N.H. Hoang, A New CBIR System Using SIFT Combined with Neural Network and Graph-Based Segmentation. In: Nguyen, N.T., Le, M.T., Świątek, J. (eds) Intelligent Information and Database Systems. Lecture Notes in Computer Science, 5990. Springer, Berlin, Heidelberg, 2010. https://doi.org/10.1007/978-3-642-12145-6_30.
  • J. Xu, K. Lu, X. Shi, S. Qin, H. Wang and J. Ma, A DenseUnet generative adversarial network for near-infrared face image colorization. Signal Processing, 183, 108007, 2021. https://doi.org/ 10.1016/j.sigpro.2021.108007.
  • H. Lacheheb, and S. Aouat, SIMIR: new mean SIFT color multi-clustering image retrieval. Multimedia Tools and Applications, 76, 6333-6354, 2017. https://doi.org/10.1007/s11042-015-3167-3.

Çok modlu duygu analizinde sensör ve özellik seviyesi füzyon yöntemlerinin etkileri

Yıl 2024, Cilt: 13 Sayı: 4, 1093 - 1099, 15.10.2024
https://doi.org/10.28948/ngumuh.1469289

Öz

Füzyon tabanlı çok modlu duygu tanıma (MER) çalışmaları günümüzde oldukça popülerdir. Bu çalışmada, çok modlu duygu tanıma için EEG sinyalleri ve yüz görüntüleri sensör seviyesinde füzyon (SLF) ve öznitelik seviyesinde füzyon (FLF) yöntemleri ile birleştirilmiştir. Çalışmanın genel akışı şu şekildedir. İlk olarak EEG sinyalleri açı genlik grafiği (AAG) görüntülerine dönüştürülmektedir. İkinci olarak, video kayıtlarından elde edilen tüm yüz görüntülerinden en benzersiz olanlar otomatik olarak belirlenmektedir. Daha sonra, bu modaliteler SLF ve FLF yöntemleri kullanılarak ayrı ayrı birleştirilmektedir. Elde edilen verileri birleştirmek ve bütünleşik veriler üzerinde sınıflandırma yapmak için füzyon yaklaşımlar kullanılmıştır. Deneyler halka açık DEAP veri kümesi üzerinde gerçekleştirilmiştir. En yüksek doğruluk SLF için 5,26 standart sapma ile %82,14 ve FLF için 6,74 standart sapma ile %87,62 olarak elde edilmiştir. Bu sonuçlar, bu çalışmanın etkili bir yöntem sunarak duygu tanıma alanına önemli bir katkı sağladığını göstermektedir.

Proje Numarası

121E002

Teşekkür

This research was supported by the Turkish Scientific and Research Council (TUBITAK) through project 121E002 and 119E397.

Kaynakça

  • A. F. M. N. H. Nahin, J. M. Alam, H. Mahmud and K. Hasan, Identifying emotion by keystroke dynamics and text pattern analysis. Behaviour & Information Technology, 33(9), 987–996, 2014. https://doi.org/10.1080/0144929X.2014.907343.
  • A. Sapra, N. Panwar, and S. Panwar, Emotion recognition from speech. International journal of emerging technology and advanced engineering, 3(2), 341-345, 2013.
  • G. K. Verma, and U. S. Tiwary, Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage, 102, 162-172, 2014. https://doi.org/10.1016/j.neuroimage.2013.11.007.
  • S. Luo, Y. T. Lan, D. Peng, Z. Li, W. L. Zheng, and B. L. Lu, Multimodal Emotion Recognition in Response to Oil Paintings. 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4167-4170, 2022. https://doi.org/10.1109/EMBC48229.2022.9871630.
  • J. N. Njoku, A. C. Caliwag, W. Lim, S. Kim, H. Hwang, and J. Jung, Deep learning-based data fusion methods for multimodal emotion recognition. The Journal of Korean Institute of Communications and Information Sciences, 47(1), 79-87, 2022. https://doi.org/10.1109/ 10.7840/kics.2022.47.1.79.
  • J. Pan, W. Fang, Z. Zhang, B. Chen, Z. Zhang, and S. Wang, Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG. IEEE Open Journal of Engineering in Medicine and Biology, 2023. https://doi.org/10.1109/10.7840/10.1109/OJEMB.2023.3240280.
  • R. Li, Y. Liang, X. Liu, B. Wang, W. Huang, Z. Cai, and J. Pan, MindLink-eumpy: an open-source python toolbox for multimodal emotion recognition. Frontiers in Human Neuroscience, 15, 621493, 2021. https://doi.org/10.3389/fnhum.2021.621493.
  • Y. Zhao and D. Chen, Expression eeg multimodal emotion recognition method based on the bidirectional lstm and attention mechanism. Computational and Mathematical Methods in Medicine, 1-12, 2021. https://doi.org/10.1155/2021/9967592.
  • Y. Huang, J. Yang, S. Liu, J. Pan, Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition. Future Internet, 11(5):105, 2019. https://doi.org/10.3390/fi11050105.
  • Z. Yin, M. Zhao, Y. Wang, J. Yang, and J. Zhang, Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer methods and programs in biomedicine, 140, 93-110, 2017. https://doi.org/10.1016/j.cmpb.2016.12.005.
  • S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, and I. Patras, Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, 3(1), 18-31, 2011. https://doi.org/10.1109/T-AFFC.2011.15.
  • B. Hatipoglu, C. M. Yilmaz and C. Kose, A signal-to-image transformation approach for EEG and MEG signal classification. Signal Image and Video Processing, 13, 483–490, 2019. https://doi.org/10.1007/s11760-018-1373-y.
  • B. Hatipoglu Yilmaz, C. M. Yilmaz and C. Kose, Diversity in a signal-to-image transformation approach for EEG-based motor imagery task classification. Medical Biological Engineering Computing, 58, 443–459, 2020. https://doi.org/10.1007/s11517-019-02075-x.
  • B. Hatipoglu Yilmaz and C. Kose, A novel signal to image transformation and feature level fusion for multimodal emotion recognition. Biomedical Engineering / Biomedizinische Technik, 66 (4), 353-362, 2021. https://doi.org/10.1515/bmt-2020-0229.
  • S. Zhalehpour, Z. Akhtar, and C. Eroglu Erdem, Multimodal emotion recognition based on peak frame selection from video. Signal, Image and Video Processing, 10, 827-834, 2016. https://doi.org/10.1109/INISTA.2014.6873606.
  • W. Yu, L. Gan, S. Yang, Y. Ding, P. Jiang, J. Wang and S. Li, An improved LBP algorithm for texture and face classification. Signal Image and Video Processing, 8, 155–161, 2014. https://doi.org/10.1007/s11760-014-0652-5.
  • Matti Pietikäinen, Local Binary Patterns. http://www.scholarpedia.org/article/Local_Binary_Patterns, Accessed 13 March 2024.
  • C. Turan, and K. M. Lam, Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study. Journal of visual communication and image representation, 55, 331-341, 2018. https://doi.org/10.1016/j.jvcir.2018.05.024.
  • Behzad Javaheri, KNN with Examples in Python, https://domino.ai/blog/knn-with-examples-in-python, Accessed 14 March 2024.
  • T. Fletcher, Support vector machines explained. Tutorial paper, 1-19, 2009.
  • C. J. Burges, A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery, 2(2), 121-167, 1998.
  • A. Yuexuan, D. Shifei, S. Songhui and L. Jingcan, Discrete space reinforcement learning algorithm based on support vector machine classification. Pattern Recognition Letters, 111, 30-35, 2018. https://doi.org/10.1016/j.patrec.2018.04.012.
  • T. Q. Anh, P. Bao, T. T. Khanh, B. N. D. Thao, T. A. Tuan and N. T. Nhut, Video retrieval using histogram and sift combined with graph-based image segmentation. Journal of Computer Science, 8(6), 853, 2012. https://doi.org/10.3844/jcssp.2012.853.858.
  • N. D. Anh, P.T. Bao, B. N. Nam, N.H. Hoang, A New CBIR System Using SIFT Combined with Neural Network and Graph-Based Segmentation. In: Nguyen, N.T., Le, M.T., Świątek, J. (eds) Intelligent Information and Database Systems. Lecture Notes in Computer Science, 5990. Springer, Berlin, Heidelberg, 2010. https://doi.org/10.1007/978-3-642-12145-6_30.
  • J. Xu, K. Lu, X. Shi, S. Qin, H. Wang and J. Ma, A DenseUnet generative adversarial network for near-infrared face image colorization. Signal Processing, 183, 108007, 2021. https://doi.org/ 10.1016/j.sigpro.2021.108007.
  • H. Lacheheb, and S. Aouat, SIMIR: new mean SIFT color multi-clustering image retrieval. Multimedia Tools and Applications, 76, 6333-6354, 2017. https://doi.org/10.1007/s11042-015-3167-3.
Toplam 26 adet kaynakça vardır.

Ayrıntılar

Birincil Dil İngilizce
Konular Multimodal Analiz ve Sentez, Örüntü Tanıma, Makine Öğrenme (Diğer), Bilgi Çıkarma ve Füzyon
Bölüm Araştırma Makaleleri
Yazarlar

Bahar Hatipoğlu Yılmqz 0000-0002-7055-8945

Cemal Köse 0000-0002-5982-4771

Proje Numarası 121E002
Erken Görünüm Tarihi 2 Eylül 2024
Yayımlanma Tarihi 15 Ekim 2024
Gönderilme Tarihi 16 Nisan 2024
Kabul Tarihi 25 Haziran 2024
Yayımlandığı Sayı Yıl 2024 Cilt: 13 Sayı: 4

Kaynak Göster

APA Hatipoğlu Yılmqz, B., & Köse, C. (2024). The effects of sensor and feature level fusion methods in multimodal emotion analysis. Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi, 13(4), 1093-1099. https://doi.org/10.28948/ngumuh.1469289
AMA Hatipoğlu Yılmqz B, Köse C. The effects of sensor and feature level fusion methods in multimodal emotion analysis. NÖHÜ Müh. Bilim. Derg. Ekim 2024;13(4):1093-1099. doi:10.28948/ngumuh.1469289
Chicago Hatipoğlu Yılmqz, Bahar, ve Cemal Köse. “The Effects of Sensor and Feature Level Fusion Methods in Multimodal Emotion Analysis”. Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi 13, sy. 4 (Ekim 2024): 1093-99. https://doi.org/10.28948/ngumuh.1469289.
EndNote Hatipoğlu Yılmqz B, Köse C (01 Ekim 2024) The effects of sensor and feature level fusion methods in multimodal emotion analysis. Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi 13 4 1093–1099.
IEEE B. Hatipoğlu Yılmqz ve C. Köse, “The effects of sensor and feature level fusion methods in multimodal emotion analysis”, NÖHÜ Müh. Bilim. Derg., c. 13, sy. 4, ss. 1093–1099, 2024, doi: 10.28948/ngumuh.1469289.
ISNAD Hatipoğlu Yılmqz, Bahar - Köse, Cemal. “The Effects of Sensor and Feature Level Fusion Methods in Multimodal Emotion Analysis”. Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi 13/4 (Ekim 2024), 1093-1099. https://doi.org/10.28948/ngumuh.1469289.
JAMA Hatipoğlu Yılmqz B, Köse C. The effects of sensor and feature level fusion methods in multimodal emotion analysis. NÖHÜ Müh. Bilim. Derg. 2024;13:1093–1099.
MLA Hatipoğlu Yılmqz, Bahar ve Cemal Köse. “The Effects of Sensor and Feature Level Fusion Methods in Multimodal Emotion Analysis”. Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi, c. 13, sy. 4, 2024, ss. 1093-9, doi:10.28948/ngumuh.1469289.
Vancouver Hatipoğlu Yılmqz B, Köse C. The effects of sensor and feature level fusion methods in multimodal emotion analysis. NÖHÜ Müh. Bilim. Derg. 2024;13(4):1093-9.

download