Deep Feature Selection for Facial Emotion Recognition Based on BPSO and SVM

Öz

Facial expressions, which are important social communication tools in our daily life, provide important information about the mental state of people. Research is being done to obtain this information accurately. The importance of these researchs in the field of human-computer interaction is increasing. Many methods have been used for the recognition of universal facial expressions such as neutral, happiness, surprise, sadness, anger, disgust, and fear by intelligent systems with high accuracy. Emotion recognition is an example of difficult classification due to factors such as ambient light, age, race, gender, and facial position. In this article, a 3-stage system is proposed for emotion detection from facial images. In the first stage, the CNN-based network is trained with the Fer+ dataset. The Binary Particle Swarm Optimization algorithm is applied for feature selection to the feature vector in the fully connected layer of the CNN network trained in the second stage. Selected features are classified by Support Vector Machine. The performance of the proposed system has been tested with the Fer+ dataset. As a result of the test, 85.74% accuracy was measured. The results show that the combination of BPSO and SVM contributes to the classification accuracy and speed of the FER+ dataset.

Anahtar Kelimeler

• Facial emotion recognition
• convolutional neural network
• binary particle swarm optimization
• support vector machine

BPSO ve SVM'ye Dayalı Yüzde Duygu Tanıma için Derin Özellik Seçimi

Öz

Günlük hayatımızda önemli sosyal iletişim aracı olan yüz ifadeleri, insanların ruhsal durumu hakkında önemli bilgiler vermektedir. Bu bilgiyi doğru bir şekilde elde etmek için araştırmalar yapılmaktadır. Bu araştırmaların insan-bilgisayar etkileşimi alanındaki önemi giderek artmaktadır. Nötr, mutluluk, şaşkınlık, üzüntü, öfke, iğrenme, korku gibi evrensel yüz ifadelerinin akıllı sistemler tarafından yüksek doğrulukla tanınması için birçok yöntem kullanılmıştır. Duygu tanıma, ortam ışığı, yaş, ırk, cinsiyet ve yüz pozisyonu gibi faktörler nedeniyle zorlu bir sınıflandırma örneğidir. Bu makalede, yüz görüntülerinden duygu tanıma için 3 aşamalı bir sistem önerilmiştir. İlk aşamada, tasarlanan CNN tabanlı ağ Fer+ veri seti ile eğitiliyor. İkinci aşamada, eğitilmiş olan CNN ağının tam bağlı katmanındaki özellik vektörüne özellik seçimi için İkili Parçacık Sürü Optimizasyon algoritması uygulanıyor. Seçilen özellikler Destek Vektör Makinesi tarafından sınıflandırılır. Önerilen sistemin performansı Fer+ veri seti ile test edilmiştir. Test sonucunda %85,74 doğruluk ölçülmüştür. Elde edilen sonuçlar İkili Parçacık Sürü Optimizasyon algoritması ve Destek Vektör Makinesi birleşiminin FER+ veri setinin sınıflandırma doğruluğuna ve hızına katkısını ortaya koymuştur.

Anahtar Kelimeler

• Yüz duygu tanıma
• evrişimsel sinir ağı
• ikili parçacık sürü optimizasyonu
• destek vektör makinesi

Kaynakça

1. [1] Bouhlal M., Aarika K., AitAbdelouahid R., Elfilali S. and Benlahmar E., “Emotions recognition as innovative tool for improving students’ performance and learning approaches”, Procedia Computer Science, 175: 597-602, (2020).
2. [2] Simcock G., McLoughlin L. T., De Regt T., Broadhouse K. M., Beaudequin D., Lagopoulos J. and Hermens D. F., “Associations between facial emotion recognition and mental health in early adolescence”, International Journal of Environmental Research and Public Health, 17(1), (2020).
3. [3] Bouzakraoui M. S., Sadiq A. and Alaoui A. Y., “Appreciation of Customer Satisfaction Through Analysis Facial Expressions and Emotions Recognition”, Proceedings of 2019 IEEE World Conference on Complex Systems, WCCS 2019, 1-5, (2019).
4. [4] Owayjan M., Kashour A., Al Haddad N., Fadel M. and Al Souki G., “The design and development of a lie detection system using facial micro-expressions”, 2012 2nd International Conference on Advances in Computational Tools for Engineering Applications, ACTEA 2012, 33-38, (2012).
5. [5] Zloteanu M., “Reconsidering Facial Expressions and Deception Detection”, In Handbook of Facial Expression of Emotion, 3: 238-284, FEELab Science Books & Leya, (2020).
6. [6] Praditsangthong R., Slakkham B. and Bhattarakosol P., “A fear detection method based on palpebral fissure”, Journal of King Saud University - Computer and Information Sciences, (2019).
7. [7] Harms M. B., Martin A. and Wallace G. L., “Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies”, In Neuropsychology Review, 20(3): 290-322, (2010).
8. [8] Ekman P. and Friesen W., “Facial action coding system: a technique for the measurement of facial movement”, (1978).

9. [9] Fan Y., Lam J. C. K. and Li V. O. K., “Demographic effects on facial emotion expression: an interdisciplinary investigation of the facial action units of happiness”, Scientific Reports, 11(1): 5214, (2021).
10. [10] Ma J., Li X., Ren Y., Yang R. and Zhao Q., “Landmark-Based Facial Feature Construction and Action Unit Intensity Prediction”, Mathematical Problems in Engineering, (2021).
11. [11] Ko H., Kim K., Bae M., Seo M.-G., Nam G., Park S., Park S., Ihm J. and Lee J.-Y., “Changes in facial recognition and facial expressions with age”, (2021).
12. [12] Taubert J. and Japee S., “Using FACS to trace the neural specializations underlying the recognition of facial expressions: A commentary on Waller et al. (2020)”, Neuroscience and Biobehavioral Reviews, 120: 75–77, (2021).
13. [13] Nadeeshani M., Jayaweera A. and Samarasinghe P., “Facial emotion prediction through action units and deep learning”, ICAC 2020 - 2nd International Conference on Advancements in Computing, Proceedings, 293-298, (2020).
14. [14] Sari M., Moussaoui A. and Hadid A., “Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview”, In International Journal of Informatics and Applied Mathematics, 3(1): 39-53, (2020).
15. [15] Murugappan M. and Mutawa A., “Facial geometric feature extraction based emotional expression classification using machine learning algorithms”, PloS One, 16(2), (2021).
16. [16] S.Bavkar S., S. Rangole J. and U. Deshmukh V., “Geometric Approach for Human Emotion Recognition using Facial Expression”, International Journal of Computer Applications, 118(14): 17-22, (2015).
17. [17] Ghimire D. and Lee J., “Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines”, Sensors (Switzerland), 13(6): 7714-7734, (2013).
18. [18] Perez-Gomez V., Rios-Figueroa H. V., Rechy-Ramirez E. J., Mezura-Montes E. and Marin-Hernandez A., “Feature selection on 2d and 3d geometric features to improve facial expression recognition”, Sensors (Switzerland), 20(17), 1–20, (2020).
19. [19] Ounachad K., Oualla M. and Sadiq A., “Geometric feature based facial emotion recognition”, International Journal of Advanced Trends in Computer Science and Engineering, 9(3):3417-3425, (2020). [20] Liu X., Cheng X. and Lee K., “GA-SVM based Facial Emotion Recognition using Facial Geometric Features”, IEEE Sensors Journal, 1-1, (2020).
20. [21] Chouhayebi H., Riffi J., Mahraz M. A., Yahyaouy A., Tairi H. and Alioua N., “Facial expression recognition based on geometric features”, 2020 International Conference on Intelligent Systems and Computer Vision, ISCV 2020, 1-6, (2020).
21. [22] Ravi R., Yadhukrishna S. V. and Prithviraj R., “A Face Expression Recognition Using CNN LBP”, Proceedings of the 4th International Conference on Computing Methodologies and Communication, ICCMC 2020, 684–689, (2020).
22. [23] Niu B., Gao Z. and Guo B., “Facial Expression Recognition with LBP and ORB Features”, Computational Intelligence and Neuroscience, (2021).
23. [24] Lakshmi D. and Ponnusamy R., “Facial emotion recognition using modified HOG and LBP features with deep stacked autoencoders”, Microprocessors and Microsystems, 82: 103834, (2021).
24. [25] Jumani S. Z., Ali F., Guriro S., Kandhro I. A., Khan A. and Zaidi A., “Facial Expression Recognition with Histogram of Oriented Gradients using CNN”, Indian Journal of Science and Technology, 12(24): 1-8, (2019).
25. [26] Verma K. and Khunteta A., “Facial expression recognition using Gabor filter and multi-layer artificial neural network”, IEEE International Conference on Information, Communication, Instrumentation and Control, ICICIC 2017, 1-5, (2018).
26. [27] Mehta N. and Jadhav S., “Facial emotion recognition using log gabor filter and PCA”, Proceedings - 2nd International Conference on Computing, Communication, Control and Automation, ICCUBEA 2016, 1-5, (2017).
27. [28] Borui Z., Liu G. and Xie G., “Facial expression recognition using LBP and LPQ based on Gabor wavelet transform”, 2016 2nd IEEE International Conference on Computer and Communications, ICCC 2016 - Proceedings, 365–369, (2017).
28. [29] Georgescu M. I., Ionescu R. T. and Popescu M., “Local learning with deep and handcrafted features for facial expression recognition”, IEEE Access, 7: 64827-64836, (2019).
29. [30] Shi Y., Lv Z., Bi N. and Zhang C., “An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations”, Neural Computing and Applications, 32(13): 9267-9281, (2020).
30. [31] Hinton G. E. and Salakhutdinov R. R., “Reducing the dimensionality of data with neural networks”, Science, 313(5786):504-507, (2006).
31. [32] LeCun Y., Bottou L., Bengio Y. and Haffner P., “Gradient-based learning applied to document recognition”, Proceedings of the IEEE, 86(11): 2278-2323, (1998).
32. [33] Li M., Xu H., Huang X., Song Z., Liu X. and Li X., “Facial Expression Recognition with Identity and Emotion Joint Learning”, IEEE Transactions on Affective Computing, (2018).
33. [34] Ari A. and Hanbay D., “Deep learning based brain tumor classification and detection system”, Turkish Journal of Electrical Engineering and Computer Science, 26(5): 2275-2286, (2018).
34. [35] Türkoğlu M. and Hanbay D., “Plant disease and pest detection using deep learning-based features”, Turkish Journal of Electrical Engineering and Computer Science, 27(3): 1636-1651, (2019).
35. [36] Uzen H., Turkoglu M. and Hanbay D., “Texture defect classification with multiple pooling and filter ensemble based on deep neural network”, Expert Systems with Applications, 175: 114838, (2021).
36. [37] Liu S., Li D., Gao Q. and Song Y., “Facial Emotion Recognition Based on CNN”, Proceedings - 2020 Chinese Automation Congress, CAC 2020, 398-403, (2020).
37. [38] Chirra V. R. R., Uyyala S. R. and Kolli V. K. K., “Virtual facial expression recognition using deep CNN with ensemble learning”, Journal of Ambient Intelligence and Humanized Computing, 1: 3, (2021). [39] Miao S., Xu H., Han Z. and Zhu Y., “Recognizing facial expressions using a shallow convolutional neural network”, IEEE Access, 7: 78000-78011, (2019).
38. [40] Gupta R. and Vishwamitra L. K., “Facial expression recognition from videos using CNN and feature aggregation”, Materials Today: Proceedings, (2021).
39. [41] Bhandari A. and Pal N. R., “Can edges help convolution neural networks in emotion recognition?”, Neurocomputing, 433: 162-168, (2021).
40. [42] Liang D., Liang H., Yu Z. and Zhang Y., “Deep convolutional BiLSTM fusion network for facial expression recognition”, Visual Computer, 36(3): 499-508, (2020).
41. [43] Christou N. and Kanojiya N., “Human facial expression recognition with convolution neural networks”, In Advances in Intelligent Systems and Computing, 797: 539-545, (2019).
42. [44] Lian Z., Li Y., Tao J., Huang J. and Niu M., “Region Based Robust Facial Expression Analysis”, 2018 1st Asian Conference on Affective Computing and Intelligent Interaction, ACII Asia 2018, (2018).
43. [45] Mollahosseini A., Chan D. and Mahoor M. H., “Going deeper in facial expression recognition using deep neural networks”, 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016, 1-10, (2016).
44. [46] Lv Y., Feng Z. and Xu C., “Facial expression recognition via deep learning”, Proceedings of 2014 International Conference on Smart Computing, SMARTCOMP 2014, 303-308, (2014).
45. [47] Josephine Julina J. K. and Sharmila T. S., “Facial Emotion Recognition in Videos using HOG and LBP”, 2019 4th IEEE International Conference on Recent Trends on Electronics, Information, Communication and Technology, RTEICT 2019 - Proceedings, 56-60, (2019).
46. [48] Li B. and Lima D., “Facial expression recognition via ResNet-50”, International Journal of Cognitive Computing in Engineering, 2: 57-64, (2021).
47. [49] Bargal S. A., Barsoum E., Ferrer C. C. and Zhang C., “Emotion Recognition in the Wild from Videos using Images”, (2016).
48. [50] Simonyan K. and Zisserman A., “Very deep convolutional networks for large-scale image recognition”, 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, (2015).
49. [51] He K., Zhang X., Ren S. and Sun J., “Deep residual learning for image recognition”, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 770-778, (2016).
50. [52] Barsoum E., Zhang C., Ferrer C. C. and Zhang Z., “Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution”, ICMI 2016 - Proceedings of the 18th ACM International Conference on Multimodal Interaction, 279-283, (2016).
51. [53] Huang C., “Combining convolutional neural networks for emotion recognition”, 2017 IEEE MIT Undergraduate Research Technology Conference, URTC 2017, 1-4, (2018).
52. [54] Goodfellow I. J., Erhan D., Luc Carrier P., Courville A., Mirza M., Hamner B., Cukierski W., Tang Y., Thaler D., Lee D. H., Zhou Y., Ramaiah C., Feng F., Li R., Wang X., Athanasakis D., Shawe-Taylor J., Milakov M., Park J. and et al., “Challenges in representation learning: A report on three machine learning contests”, Neural Networks, 64: 59-63, (2015).
53. [55] Yan W. J., Wu Q., Liu Y. J., Wang S. J. and Fu X., “CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces”, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2013, (2013).
54. [56] Yan W. J., Li X., Wang S. J., Zhao G., Liu Y. J., Chen Y. H. and Fu X., “CASME II: An improved spontaneous micro-expression database and the baseline evaluation”, PLoS ONE, 9(1): e86041, (2014).
55. [57] Davison A. K., Lansley C., Costen N., Tan K. and Yap M. H., “SAMM: A Spontaneous Micro-Facial Movement Dataset”, IEEE Transactions on Affective Computing, 9(1): 116-129, (2018).
56. [58] Ma D. S., Correll J. and Wittenbrink B., “The Chicago face database: A free stimulus set of faces and norming data”, Behavior Research Methods, 47(4): 1122-1135, (2015).
57. [59] “Chicago Face Database.” Accessed May 19, 2021. https://chicagofaces.org/default/
58. [60] Li S., Deng W. and Du J. P., “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild”, Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2584-2593, (2017).
59. [61] Li S. and Deng W., “Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition”, IEEE Transactions on Image Processing, 28(1): 356-370, (2019).
60. [62] van der Schalk J., Hawk S. T., Fischer A. H. and Doosje B., “Moving Faces, Looking Places: Validation of the Amsterdam Dynamic Facial Expression Set (ADFES)”, Emotion, 11(4): 907-920, (2011).
61. [63] “Introduction - Amsterdam Interdisciplinary Centre for Emotion (AICE) - University of Amsterdam.” Accessed May 19, 2021. https://aice.uva.nl/research-tools/adfes-stimulus-set/adfes-stimulus-set.html?cb.
62. [64] Koelstra S., Mühl C., Soleymani M., Lee J. S., Yazdani A., Ebrahimi T., Pun T., Nijholt A. and Patras I., “DEAP: A database for emotion analysis; Using physiological signals”, IEEE Transactions on Affective Computing, 3(1): 18-31, (2012).
63. [65] Busso C., Bulut M., Lee C.C., Kazemzadeh A., Mower E., Kim S., Chang J.N., Lee S. and Narayanan S. S., “IEMOCAP: Interactive emotional dyadic motion capture database”, Journal of Language Resources and Evaluation, 42(4): 335-359, (2008).
64. [66] “IEMOCAP” Accessed May 19, 2021. https://sail.usc.edu/iemocap/iemocap_release.htm
65. [67] Lyons M., Kamachi M. and Gyoba J., “The Japanese Female Facial Expression (JAFFE) Dataset”, (1998).
66. [68] “(JAFFE) Dataset | Zenodo.” Accessed May 19, 2021.https://zenodo.org/record/3451524#.YKTd3bczZp8.
67. [69] “Google Facial Expression Comparison Dataset – Google Research.” Accessed May 19, 2021. https://research.google/tools/datasets/google-facial-expression/.
68. [70] Mollahosseini A., Hasani B. and Mahoor M. H., “AffectNet: A New Database for Facial Expression, Valence, and Arousal Computation in the Wild”, IEEE Transactions on Affective Computing, (2017).
69. [71] “AffectNet – Mohammad H. Mahoor, PhD.” Accessed May 19, 2021. http://mohammadmahoor.com/affectnet/.
70. [72] Lucey P., Cohn J. F., Kanade T., Saragih J., Ambadar Z. and Matthews I., “The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression”, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, 94-101, (2010).
71. [73] Dhall A., Goecke R., Lucey S. and Gedeon T., “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark”, Proceedings of the IEEE International Conference on Computer Vision, 2106-2112, (2011).
72. [74] Dhall A., Goecke R., Lucey S. and Gedeon T., “Collecting large, richly annotated facial-expression databases from movies”, IEEE Multimedia, 19(3): 34-41, (2012).
73. [75] Gross R., Matthews I., Cohn J., Kanade T. and Baker S., “Multi-PIE”, Image and Vision Computing, 28(5): 807-813, (2010).
74. [76] Mavadati S. M., Mahoor M. H., Bartlett K., Trinh P. and Cohn J. F., “DISFA: A spontaneous facial action intensity database”, IEEE Transactions on Affective Computing, 4(2): 151-160, (2013).
75. [77] Bänziger T. and Scherer K. R., “Introducing the Geneva Multimodal Emotion Portrayal (GEMEP) Corpus”, In Blueprint for affective computing: A sourcebook, 271–294, (2010).
76. [78] Pantic M., Valstar M., Rademaker R. and Maat L., “Web-based database for facial expression analysis”, IEEE International Conference on Multimedia and Expo, ICME 2005, 317–321, (2005).
77. [79] Zhao G., Huang X., Taini M., Li S. Z. and Pietikäinen M., “Facial expression recognition from near-infrared videos”, Image and Vision Computing, 29(9): 607-619, (2011).
78. [80] Zhalehpour S., Onder O., Akhtar Z. and Erdem C. E., “BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States”, IEEE Transactions on Affective Computing, 8(3): 300–313, (2017).
79. [81] Martin O., Kotsia I., Macq B. and Pitas I., “The eNTERFACE’05 Audio-Visual emotion database”, ICDEW 2006 - Proceedings of the 22nd International Conference on Data Engineering Workshops, (2006).
80. [82] Kingma D. P. and Lei Ba J., “ADAM: A method for stochastic optimization”, (2015).
81. [83] Fırat H. and Alpaslan N., “An effective approach to the two-dimensional rectangular packing problem in the manufacturing industry”, Computers and Industrial Engineering, 148:106687, (2020).
82. [84] Donuk K., Özbey N., Inan M., Yeroǧlu C. and Hanbay D., “Investigation of PIDA Controller Parameters via PSO Algorithm”, 2018 International Conference on Artificial Intelligence and Data Processing, IDAP 2018, (2019).
83. [85] Wulandhari L. A., Komsiyah S. and Wicaksono W., “Bat Algorithm Implementation on Economic Dispatch Optimization Problem”, Procedia Computer Science, 135: 275-282, (2018).
84. [86] Sarkar R., Barman D. and Chowdhury N., “Domain knowledge based genetic algorithms for mobile robot path planning having single and multiple targets”, Journal of King Saud University - Computer and Information Sciences, (2020).
85. [87] Sharma V. and Mir R. N., “An enhanced time efficient technique for image watermarking using ant colony optimization and light gradient boosting algorithm”, Journal of King Saud University - Computer and Information Sciences, (2019).
86. [88] Kennedy J. and Eberhart R., “Particle swarm optimization”, Proceedings of ICNN’95 - International Conference on Neural Networks, 4: 1942-1948, (1995).
87. [89] Kumari K., Singh J. P., Dwivedi Y. K. and Rana N. P., “Multi-modal aggression identification using Convolutional Neural Network and Binary Particle Swarm Optimization”, Future Generation Computer Systems, 118: 187-197, (2021).
88. [90] Dagar N. S. and Dahiya P. K., “Edge Detection Technique using Binary Particle Swarm Optimization”, Procedia Computer Science, 167: 1421-1436, (2020).
89. [91] Cortes C. and Vapnik V., “Support-vector networks”, Machine Learning, 20(3): 273–297, (1995).
90. [92] ”Scikit-Learn 0.24.2 Dokümantasyon.” Accessed May 21, 2021. https://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html.
91. [93] Vo T. H., Lee G. S., Yang H. J. and Kim S. H., “Pyramid with Super Resolution for In-the-Wild Facial Expression Recognition”, IEEE Access, 8: 131988-132001, (2020).
92. [94] Albanie S., Nagrani A., Vedaldi A. and Zisserman A., “Emotion recognition in speech using cross-modal transfer in the wild”, MM 2018 - Proceedings of the 2018 ACM Multimedia Conference, 292-301, (2018).
93. [95] Siqueira H., Magg S. and Wermter S., “Efficient facial feature learning with wide ensemble-based convolutional neural networks”, AAAI 2020 - 34th AAAI Conference on Artificial Intelligence, 5800-5809, (2020).