Convolutional Neural Network Based Emotion Recognition from Facial Expressions Using Different Feature Engineering Methods

Yıl 2025, Cilt: 18 Sayı: 1, 73 - 97, 28.03.2025

Şengül Bayrak , Fatima Amiry , Anisah Kaso , Mina Çakır

https://doi.org/10.18185/erzifbed.1453842

Öz

Abstract
With the impact of advancing technology, the automatic detection of human emotions is of great interest in various industries. Emotion recognition systems from facial images are important to meet the needs of various industries in a wide range of application areas, such as security, marketing, advertising, and human-computer interaction. In this study, automatic facial expression detection of 7 different emotions (anger, disgust, fear, happy, neutral, sad, and surprised) from facial image data has been performed. The process steps of the study are as follows: (i) preprocessing the image data with image grayscale and image enhancement methods, (ii) feature extraction by applying Gradient Histogram, Haar Wavelet, and Gabor filter methods to the preprocessed image, (iii) modeling the feature sets obtained from three different feature extraction methods with Convolutional Neural Network method, (iv) calculating the most successful feature extraction method in the detection of 7 different emotions with Convolutional Neural Network. As a result of the experimental studies, it has been determined that the Gabor filter feature extraction method is thriving with an accuracy rate of 83.12%. When the results of these methods are compared with other studies, the model developed contributes to the literature by making a difference in recognition rate, dataset size, and feature engineering methods.

Anahtar Kelimeler

Gabor filter, Haar Wavelet, Gradient Histogram, emotion recognition from facial expression, Convolutional Neural Network

Etik Beyan

There are no ethical issues regarding the publication of this study.

Destekleyen Kurum

-

Teşekkür

-

Farklı Özellik Mühendisliği Yöntemleri ile Yüz İfadelerinden CNN-Tabanlı Duygu Tespiti

Öz

Gelişen teknolojinin etkisiyle, insan duygularının otomatik olarak algılanması çeşitli sektörlerde büyük ilgi görmektedir. Yüz görüntülerinden duygu tanıma sistemleri, güvenlik, pazarlama, reklamcılık ve insan-bilgisayar etkileşimi gibi çok çeşitli uygulama alanlarında çeşitli endüstrilerin ihtiyaçlarını karşılamak için önemlidir. Bu çalışmada, yüz görüntüsü verilerinden 7 farklı duygunun (kızma, iğrenme, korku, mutlu, nötr, üzgün ve şaşkın) otomatik ifade tespiti gerçekleştirilmiştir. Çalışmanın işlem adımları şöyledir: (i) görüntü verilerinin görüntü gri tonlama ve görüntü iyileştirme yöntemleri ile ön işleme uygulanması, (ii) ön işlem uygulanan görüntüye Gradient Histogram, Haar Wavelet ve Gabor filtre yöntemlerinin uygulanarak özellik çıkarımı yapılması, (iii) üç farklı özellik çıkarım yönteminden elde edilen özellik setlerinin Evrişimsel Sinir Ağı yöntemi ile modellenmesi, (iv) 7 farklı duygunun tespitinde en başarılı özellik çıkarım yönteminin Evrişimsel Sinir Ağı ile hesaplanmasıdır. Yapılan deneysel çalışmalar sonucunda Gabor filtresi özellik çıkarma yönteminin %83,12 doğruluk oranı ile başarılı olduğu tespit edilmiştir. Bu yöntemlerin sonuçları ile diğer çalışmaların sonuçları karşılaştırıldığında, geliştirilen model tanıma oranı, veri kümesi boyutu ve özellik mühendisliği yöntemleri açısından fark oluşturarak literatüre katkı sağlamaktadır.

Anahtar Kelimeler

Gabor filtresi, Haar Dalgacığı, Gradyan Histogramı, yüz ifadesinden duygu tanıma, Evrişimsel Sinir Ağı

Kaynakça

• [1] Ralph Adolphs, Leonard Mlodinow, and Lisa Feldman Barrett What is an emotion?, Current Biology Magazine, 2019.
• [2] Saraa Clemente Paul Ekman’a Gore Mikro-ifadeler, 2022.
• [3] Turetsky Goossens, B. I., Kohler, C. G., Indersmitten, T., Bhati, M. T., Charbonnier, D., Gur, R. C. Facial emotion recognition in schizophrenia: when and why does it go awry Schizophrenia research, 2007, pp. 94(1- 3), 253-263.
• [4] Madeline B. Harms, Alex Martin Gregory L. Wallace ”Facial Emotion Recognition in Autism Spectrum Disorders: A Review of Behavioral and Neuroimaging Studies”, 2010.
• [5] Ian M. Anderson, Clare Shippen,Gabriella Juhasz, Diana Chase, Emma Thomas, Darragh Downey, Zoltan G. Toth,Kathryn Lloyd-Williams, Rebecca Elliott and J. F. William Deakin "State-dependent alteration in face emotion recognition in depression.", 2018.
• [6] Neha Jain, Shishir Kumar, Amit Kumar, Pourya Shamsolmoal and iMa- soumeh Zareapoor "Hybrid deep neural networks for face emotion recogni- tion.", 2018.
• [7] Deepak Kumar,Jain Pourya, Shamsolmoali Paramjit and Sehdev "Extended deep neural network for facial emotion recognition.", 2019.
• [8] Hongli Zhang, Alireza Jolfaei, And Mamoun Alazab "A Face Emotion Recognition Method Using Convolutional Neural Network and Image Edge Computing.", 2019.
• [9] Liyanage C. De Silva, Tsutomu Miyasato, Ryohei Nakatsu "Facial Emotion Recognition Using Multi-modal Information.", 1997.
• [10] Shinde, S., Pande, S. (2012). A survey on: Emotion recognition with respect to database and various recognition techniques. International Journal of Computer Applications, 58(3), 9-12.
• [11] Kaburlasos, V. G., Papadakis, S. E., Papakostas, G. A. (2013). Lattice computing extension of the FAM neural classifier for human facial expression recognition. IEEE Transactions on Neural Networks and Learning Systems, 24(10), 1526-1538.
• [12] Piparsaniyan, Y., Sharma, V. K., Mahapatra, K. K. (2014, April). Robust facial expression recognition using Gabor feature and Bayesian discriminating classifier. In 2014 International Conference on Communication and Signal Processing (pp. 538-541). IEEE.
• [13] Burkert, P., Trier, F., Afzal, M. Z., Dengel, A., Liwicki, M. (2015). Dexpression: Deep convolutional neural network for expression recognition. arXiv preprint arXiv:1509.05371.
• [14] Yu, Z., Zhang, C. (2015, November). Image based static facial expression recognition with multiple deep network learning. In Proceedings of the 2015 ACM on international conference on multimodal interaction (pp. 435-442).
• [15] Li, X., Yu, J., Zhan, S. (2016, November). Spontaneous facial micro-expression detection based on deep learning. In 2016 IEEE 13th International Conference on Signal Processing (ICSP) (pp. 1130-1134). IEEE.
• [16] Matlovic, T., Gaspar, P., Moro, R., Simko, J., Bielikova, M. (2016, October). Emotions detection using facial expressions recognition and EEG. In 2016 11th international workshop on semantic and social media adaptation and personalization (SMAP) (pp. 18-23). IEEE.
• [17] Xiang, J., Zhu, G. (2017, July). Joint face detection and facial expression recognition with MTCNN. In 2017 4th international conference on information science and control engineering (ICISCE) (pp. 424-427). IEEE.
• [18] Greche, L., Jazouli, M., Es-Sbai, N., Majda, A., Zarghili, A. (2017, April). Comparison between Euclidean and Manhattan distance measure for facial expressions classification. In 2017 International conference on wireless technologies, embedded and intelligent systems (WITS) (pp. 1-4). IEEE.
• [19] Kumar, S., Singh, S., Kumar, J. (2018). Automatic live facial expression detection using genetic algorithm with haar wavelet features and SVM. Wireless Personal Communications, 103(3), 2435-2453. https://doi.org/10.1007/s11277-018-5923-y.
• [20] Chang, F. J., Tran, A. T., Hassner, T., Masi, I., Nevatia, R., Medioni, G. (2018, May). Expnet: Landmark-free, deep, 3d facial expressions. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (pp. 122-129). IEEE.
• [21] Jain, D. K., Shamsolmoali, P., Sehdev, P. (2019). Extended deep neural network for facial emotion recognition. Pattern Recognition Letters, 120, 69-74.
• [22] Xie, S., Hu, H., Wu, Y. (2019). Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern recognition, 92, 177-191. https://doi.org/10.1016/j.patcog.2019.03.019
• [23] Hammed, S. S., Sabanayagam, A., Ramakalaivani, E. (2020). A review on facial expression recognition systems. Journal of critical reviews, 7(4), 903-905.
• [24] Porcu, S., Floris, A., Atzori, L. (2020). Evaluation of data augmentation techniques for facial expression recognition systems. Electronics, 9(11), 1892. https://doi.org/10.3390/electronics9111892
• [25] Tsai, K. Y., Tsai, Y. W., Lee, Y. C., Ding, J. J., Chang, R. Y. (2021). Frontalization and adaptive exponential ensemble rule for deep-learning-based facial expression recognition system. Signal Processing: Image Communication, 96, 116321.
• [26] Almeida, J., Rodrigues, F. (2021, April). Facial Expression Recognition System for Stress Detection with Deep Learning. In ICEIS (1) (pp. 256-263).
• [27] Shabbir, N., Rout, R. K. (2023). Variation of deep features analysis for facial expression recognition system. Multimedia Tools and Applications, 82(8), 11507-11522. https://doi.org/10.1007/s11042-022-14054-w
• [28] Kadakia, R., Kalkotwar, P., Jhaveri, P., Patanwadia, R., Srivastava, K. (2022, November). Analysis of Micro Expressions using XAI. In 2022 3rd International Conference on Computing, Analytics and Networks (ICAN) (pp. 1-7). IEEE.
• [29] Lee, K. W., Lee, H. J., Hu, H., Kim, H. J. (2022). Analysis of facial ultrasonography images based on deep learning. Scientific reports, 12(1), 16480.
• [30] Yaddaden, Y. (2023). An efficient facial expression recognition system with appearance-based fused descriptors. Intelligent Systems with Applications, 17, 200166.
• [31] Bartlett, M. S., Littlewort, G., Frank, M., Lainsc-sek, C., Fasel, I., Movellan, J. Recognizing facial expression: machine learning and application to spontaneous behavior. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05);2005. pp. 568-573.
• [32] Sonmez, E., Albayrak, S. A facial component-based system for emotion classification. Turkish Journal of Electrical Engineering and Computer Sciences; 2016. 24(3): 1663-1673. https://doi.org/10.3906/elk-1401-18.
• [33] Farajzadeh, N., Hashemzadeh, M. Exemplar-based facial expression recognition. Information Sciences;2018. 460: 318-330. https://doi.org/10.1016/j.ins.2018.05.057
• [34] Baygin, M., Tuncer, I., Dogan, S., Barua, P. D., Tuncer, T., Cheong, K. H., Acharya, U. R. Automated facial expression recognition using exemplar hybrid deep feature generation technique. Soft Computing, 2023. 1- 17. 27:8721-8737. https://doi.org/10.1007/s00500-023- 08230-9.
• [35] Lu, S., Evans, F. Haar wavelet transform based facial emotion recognition. In 2017 7th international conference on education, management, computer and society, 2017. pp. 342-346. https://doi.org/10.2991/emcs17.2017.67
• [36] Chowdhury, J. H., Liu, Q., Ramanna, S. (2024). Simple Histogram Equalization Technique Improves Performance of VGG Models on Facial Emotion Recognition Datasets. Algorithms, 17(6), 238.
• [37] Haq, H. B. U., Akram, W., Irshad, M. N., Kosar, A., Abid, M. (2024). Enhanced real-time facial expression recognition using deep learning. Acadlore Trans. Mach. Learn, 3(1), 24-35.
• [38] Talaat, F. M., Ali, Z. H., Mostafa, R. R., El-Rashidy, N. (2024). Real-time facial emotion recognition model based on kernel autoencoder and convolutional neural network for autism children. Soft Computing, 1-14.
• [39] Meena, G., Mohbey, K. K., Indian, A., Khan, M. Z., Kumar, S. (2024). Identifying emotions from facial expressions using a deep convolutional neural network-based approach. Multimedia Tools and Applications, 83(6), 15711-15732.
• [40] Pacal, I. MaxCerVixT: A novel lightweight vision transformer-based Approach for precise cervical cancer detection. Knowledge-Based Systems, 2024, 289, 111482. https://doi.org/10.1016/j.knosys.2024.111482
• [41] Pacal, I. Enhancing crop productivity and sustainability through disease identification in maize leaves: Exploiting a large dataset with an advanced vision transformer model. Expert Systems with Applications, 2024, 238, 122099. https://doi.org/10.1016/j.eswa.2023.122099
• [42] Pacal, I. A novel Swin transformer approach utilizing residual multi-layer perceptron for diagnosing brain tumors in MRI images. International Journal of Machine Learning and Cybernetics, 2024, 15:3579–3597 https://doi.org/10.1007/s13042-024-02110-w
• [43]https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset (Access Time: 10 October 2023)
• [44] Yamini Piparsaniyan, Vijay K. Sharma, K. Mahapatra Robust facial expression recognition using Gabor feature and Bayesian discriminating classifier, 2014
• [45] Siddiqui, E. A., Chaurasia, V., Shandilya, M. (2023). Detection and classification of lung cancer computed to-mography images using a novel improved deep belief network with Gabor filters. Chemometrics and Intelligent Laboratory Systems, 235, 104763.
• [46] Sandeep Kumar, Sukhwinder Singh Jagdish Kumar, Au-tomatic Live Facial Expression Detection Using Genetic Algorithm with Haar Wavelet Features and SVM, 2018
• [47] Batziou, E., Ioannidis, K., Patras, I., Vrochidis, S., Kompatsiaris, I. (2023, January). Low-Light Image En-hancement Based on U-Net and Haar Wavelet Pooling. In International Conference on Multimedia Modeling (pp. 510-522). Cham: Springer Nature Switzerland.
• [48] Dixit, U. D., Shirdhonkar, M. S., Sinha, G. R. (2023). Automatic logo detection from document image using HOG features. Multimedia Tools and Applications, 82(1), 863-878.
• [49] Acar, Y. E., Saritas, I., Yaldiz, E. (2022). Com-parison of ML algorithms to distinguish between human or human-like targets using the HOG features of range-time and range-Doppler images in through-thewall applications. Turkish Journal of Electrical Engineering and Computer Sciences, 30(6), 2086-2096.
• [50] Bayrak, S. (2024). Unveiling intrusions: explainable SVM approaches for addressing encrypted Wi-Fi traffic in UAV networks. Knowledge and Information Systems, 1-21. https://doi.org/10.1007/s10115-024-02181-9
• [51] Phung, V. H., Rhee, E. J. (2019). A high-accuracy model average ensemble of convolutional neural networks for classification of cloud image patches on small datasets. Applied Sciences, 9(21), 4500. https://doi.org/10.3390/app9214500