Sentiment Analysis from Face Expressions Based on Image Processing Using Deep Learning Methods

Abstract

In this study, the classification study of human facial expressions in real-time images is discussed. Implementing this work in software have some benefits for us. For example, analysis of mood in group photos is an interesting instance in this regard. The perception of people’s facial expressions in photographs taken during an event can provide quantitative data on how much fun these people have in general. Another example is context-aware image access, where only photos of people who are surprised can be accessed from a database. Seven different emotions related to facial expressions were classified in this context; these are listed as happiness, sadness, surprise, disgust, anger, fear and neutral. With the application written in Python programming language, classical machine learning methods such as k-Nearest Neighborhood and Support Vector Machines and deep learning methods such as AlexNet, ResNet, DenseNet, Inception architectures were applied to FER2013, JAFFE and CK+ datasets. In this study, while comparing classical machine learning methods and deep learning architectures, real-time and non-real-time applications were also compared with two different applications. This study conducted to demonstrate that real-time expression recognition systems based on deep learning techniques with the most appropriate architecture can be implemented with high accuracy via computer hardware with only one software. In addition, it is shown that high accuracy rate is achieved in real-time applications when Histograms of Oriented Gradients (HOG) is used as a feature extraction method and ResNet architecture is used for classification.

Keywords

• Image Processing
• classification
• Emotion analysis
• Convoultional Neural Network
• deep learning

References

1. Abdullah, S. M. S., and Abdulazeez, A. M. (2021). “Facial expression recognition based on deep learning convolution neural network: A review.” Journal of Soft Computing and Data Mining, 2(1), 53-65. Retrieved from: https://publisher.uthm.edu.my/ojs/index.php/jscdm/article/view/7906
2. Akinrotimi, A, O. (2018) “Facial Emotion Recognition Using Principal Component Analysis and Support Vector Machine” Retrieved from: https://www.researchgate.net/publication/336120020_Facial_Emotion_Recognition_Using_Principal_Component_Analysis_and_Support_Vector_Machine
3. Bhattacharya, S. (2021) “A Survey on: Facial Expression Recognition Using Various Deep Learning Techniques”, Advances in Intelligent Systems and Computing, Advanced Computational Paradigms and Hybrid Intelligent Computing pp 619–631. Retrieved from: https://doi.org/10.1007/978-981-16-4369-9_59
4. Bisogni, C., Castiglione, A., Hossain, S., Narducci, F, and Umer S. (2022), “Impact of Deep Learning Approaches on Facial Expression Recognition in Healthcare Industries”, IEEE Transactions on Industrial Informatics, vol. 18, no. 8. Retrieved from: https://ieeexplore.ieee.org/document/9674818
5. Buhari A. M., Ooi, C. P., Baskaran, V.M., Phan, R. CW, Wong, K. and Tan, W-H. (2022) “Invisible emotion magnification algorithm (IEMA) for real-time micro-expression recognition with graph-based features”, Advances in Soft Computing Techniques for Visual Information-based Systems, 81, pages 9151–9176. DOI https://doi.org/10.1007/s11042-021-11625-1
6. Chen, Y., Wu, H. (2018) "A Comparison of Methods of Facial Expression Recognition" Retrieved from: https://ieeexplore.ieee.org/document/8584202
7. Coşar, S., (2008) “Facial Feature Point Tracking Based On A Graphical Model Framework” Retrieved From: https://vpa.sabanciuniv.edu/vpadb/findfile_get_anonymous_paper.php?f=1954
8. Çınar, A., C., (2018) “Deep Learning” Retrieved From: https://www.ahmetcevahircinar.com.tr/2017/08/11/imagenet-classification-with-deep-convolutional-neural-networks/

9. Colmenarez, A., Frey, B. and Huang, T.S. (1999) “A probabilistic framework for embedded face and facial expression recognition.” Proceedings 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), 1999, pp. 592-597 Vol. 1, doi: 10.1109/CVPR.1999.786999. DOI: 10.1109/CVPR.1999.786999
10. Devrim, M., O. (2019) “Farklı Derin Evrişimsel Sinir Ağlarının Yüz İfadesi Tanıma İşleminde Karşılaştırılması” Retrieved from: https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=vLQ0gfV9vTaOpH16k6kjMA&no=ZgJjiVHnbVeqxh9M0L2OHg
11. Dhavalikar, A., Kulkarni, R., K., (2014) “Face detection and facial expression recognition system” Retrieved From: https://www.researchgate.net/publication/286592694_Face_detection_and_facial_expression_recognition_system
12. Engin, D., (2017) “Facial Expression Pair Matching” Retrieved from: https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=2bQ8F7EJQFFi10i_DbMMkQ&no=poooj2n8rQ268jwtzPwoqw
13. Frank, M., G., (2001) “Facial Expressions” Retrieved From: https://www.sciencedirect.com/topics/computer-science/facial-expression
14. Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning. Regularization for Deep learning pg 228-270, Cambridge, Massachusetts: MIT Press. Retrieved From: http://www.deeplearningbook.org
15. Güneş, T., Polat, E. (2009) "Feature Selection In Facial Expression Analysis And Its Effect On Multi-Svm Classifiers" Retrieved from: https://www.researchgate.net/publication/292375988_Feature_selection_in_facial_expression_analysis_and_its_effect_on_multi-svm_classifiers
16. Göngör, F., and Tutsoy, O. (2018). “Eigenface based emotion analysis algorithm and implementation to humanoid robot.” In International Science and Academic Congress.
17. Islam, K., Al-Murad, A. (2017) “Performance of SVM, CNN, and ANN with BoW, HOG, and Image Pixels in Face Recognition” Retrieved from: https://www.researchgate.net/publication/326682312_Performance_of_SVM_CNN_and_ANN_with_BoW_HOG_and_Image_Pixels_in_Face_Recognition
18. Jung H. et al., (2015) "Development of deep learning-based facial expression recognition system" 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), pp. 1-4, doi: 10.1109/FCV.2015.7103729. Retrieved from: https://ieeexplore.ieee.org/document/7103729
19. Karaboyacı, C., (2009) “Geometrical Feature Based Automated Facial Expression Analysis” Retrieved from: https://polen.itu.edu.tr/bitstream/11527/339/1/9597.pdf
20. Khoong, W., H., (2021) “When Do Support Vector Machines Fail?” Retrieved From: https://towardsdatascience.com/when-do-support-vector-machines-fail-3f23295ebef2
21. Kotsia, I., Pitas, I., (2008) “Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines” Retrieved From: https://www.researchgate.net/publication/6520512_Facial_Expression_Recognition_in_Image_Sequences_Using_Geometric_Deformation_Features_and_Support_Vector_Machines
22. Kuo, C., Lai, S. and Sarkis, M. (2018) "A Compact Deep Learning Model for Robust Facial Expression Recognition" 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2202-22028, doi: 10.1109/CVPRW.2018.00286. Retrieved from: https://ieeexplore.ieee.org/document/8575457
23. Lucey, P., Cohn, J., Kanade T. (2010) “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression” Retrieved From: https://ieeexplore.ieee.org/document/5543262
24. Li, B. and Lima, D. (2021) “Facial expression recognition via ResNet-50”, International Journal of Cognitive Computing in Engineering, 57-64. Retrieved From: https://doi.org/10.1016/j.ijcce.2021.02.002
25. Li, S. (2020) “Deep Facial Expression Recognition: A Survey”, IEEE Transactions on Affective Computing, pp 1:20. doi: 10.1109/TAFFC.2020.2981446
26. Mehrabian, A. (2016), https://en.wikipedia.org/wiki/Albert_Mehrabian, 01.06.2022.
27. Mena-Chalco, J.P., Macedo, I., Velho, L. and Cesar, R.M. (2009), “3D face computational photography using PCA spaces”, Visual Computer, 2009, 25:899-909. Retrieved From: https://doi.org/10.1007/s00371-009-0373-x
28. Mokhtari, N., I., (2021) “Which is Better for Your Machine Learning Task” Retrieved From: https://towardsdatascience.com/which-is-better-for-your-machine-learning-task-opencv-or-tensorflow-ed16403c5799
29. Nagaraj, P., Banala, R., (2021) “Real Time Face Recognition using Effective Supervised Machine Learning Algorithms” Retrieved From: https://www.researchgate.net/publication/354095467_Real_Time_Face_Recognition_using_Effective_Supervised_Machine_Learning_Algorithms
30. Ronthkrantz, J., M., Pantic, M., (2001) “Automatic Analysis of Facial Expressions: The State of the Art”. Retrieved From: https://www.researchgate.net/publication/3193199_Automatic_Analysis_of_Facial_Expressions_The_State_of_the_Art
31. Özdemir, M., A., Elagöz, B., Alaybeyoğlu, A., Akan, A. (2019) “Real Time Emotion Recognition from Facial Expressions Using CNN Architecture” Retrieved from: https://www.researchgate.net/publication/336287978_Real_Time_Emotion_Recognition_from_Facial_Expressions_Using_CNN_Architecture
32. Özdemir and Hanbay. (2022) “Deep feature selection for facial emotion recognition based on BPSO and SVM” Retrieved From: https://dergipark.org.tr/tr/pub/politeknik/issue/33364/992720
33. Pantic, M., Rothkrantz, J., (2001), “Automatic Analysis of Facial Expressions: The State of the Art” Retrieved from: https://www.researchgate.net/publication/3193199_Automatic_Analysis_of_Facial_Expressions_The_State_of_the_Art
34. Paleari, M., Velardo, C., Huet, B., Dugelay, J. (2009) “Face dynamics for biometric people recognition” Retrieved From: http://www.eurecom.fr/en/publication/2888/download/mm-publi-2888.pdf
35. Python Software Foundation (2012) Retrieved from: https://web.archive.org/web/20121024164224/http://docs.python.org/faq/general.html
36. Saurav, S., Gidde, P., Saini, R., and Singh, S. (2022) “Dual integrated convolutional neural network for real-time facial expression recognition in the wild” The Visual Computer volume 38, pages 1083–1096. Retrieved from: https://doi.org/10.1007/s00371-021-02069-7
37. Saxena, A., Khanna, A., Gupta, D. (2020) "Emotion Recognition and Detection Methods" Retrieved from:https://www.researchgate.net/publication/339119986_Emotion_Recognition_and_Detection_Methods_A_Comprehensive_Survey
38. Saraçbaşı, N., Zeynep. (2021) “Face Recognition Using Facial Dynamics of Emotional Expressions” Retrieved From: https://tez.yok.gov.tr/UlusalTezMerkezi/TezGoster?key=tqUiYt63sTQLTpozMJ92QpYVnXW_2-VG0SZiwpDZZlLMRrUm2f-TqRQA8TWiyPik
39. Sebe, N., Cohen, I., Huang, T.S (2004) "Multimodal Emotion Recognition" Retrieved from: https://www.researchgate.net/publication/228616884_Multimodal_emotion_recognition
40. Sharma, P. (2021) “Understanding Transfer Learning for Deep Learning ” Retrieved from: https://www.analyticsvidhya.com/blog/2021/10/understanding-transfer-learning-for-deep-learning/
41. Srivastava, T., (2018) “Introduction to k-Nearest Neighbors” Retrieved From: https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/
42. Terra, J., (2022) “Key Differences Among the Deep Learning Framework”, Retrieved From: https://www.simplilearn.com/keras-vs-tensorflow-vs-pytorch-article#:~:text=TensorFlow%20is%20an%20open%2Dsourced,because%20it's%20built%2Din%20Python.
43. Tutsoy, O., Güngör, F., Barkana, D. E., & Köse, H. (2017). An emotion analysis algorithm and implementation to NAO humanoid robot. The Eurasia Proceedings of Science Technology Engineering and Mathematics, (1), 316-330. Retrieved From: http://www.epstem.net/tr/download/article-file/381431
44. Umer, S., Rout, R.K., Pero, C., and Nappi, M. (2022) “Facial expression recognition with trade-offs between data augmentation and deep learning features”, Journal of Ambient Intelligence and Humanized Computing volume 13, pages721–735. Retrieved From: https://doi.org/10.1007/s12652-020-02845-8
45. Verma, G., Verma, H. (2020) “Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions” Retrieved from: https://www.researchgate.net/publication/343556711_Hybrid-Deep_Learning_Model_for_Emotion_Recognition_Using_Facial_Expressions
46. Yang, M., H., Kriegman, D., (2002) “Detecting Faces in Images: A Survey Retrieved from: https://www.researchgate.net/publication/3193340_Detecting_Faces_in_Images_A_Survey
47. Zhao, X., Shi X. and Zhang, S. (2015) “Facial Expression Recognition via Deep Learning” IETE Technical Review, 32:5, 347-355, DOI:10.1080/02564602.2015.1017542 Retrieved From: https://ieeexplore.ieee.org/document/7043872