Sentiment Analysis from Face Expressions Based on Image Processing Using Deep Learning Methods

Year 2022, Volume: 8 Issue: 4, 736 - 752, 15.12.2022

Orhan Emre Aksoy Selda Güney

https://doi.org/10.28979/jarnas.1056664

Abstract

In this study, the classification study of human facial expressions in real-time images is discussed. Implementing this work in software have some benefits for us. For example, analysis of mood in group photos is an interesting instance in this regard. The perception of people’s facial expressions in photographs taken during an event can provide quantitative data on how much fun these people have in general. Another example is context-aware image access, where only photos of people who are surprised can be accessed from a database. Seven different emotions related to facial expressions were classified in this context; these are listed as happiness, sadness, surprise, disgust, anger, fear and neutral. With the application written in Python programming language, classical machine learning methods such as k-Nearest Neighborhood and Support Vector Machines and deep learning methods such as AlexNet, ResNet, DenseNet, Inception architectures were applied to FER2013, JAFFE and CK+ datasets. In this study, while comparing classical machine learning methods and deep learning architectures, real-time and non-real-time applications were also compared with two different applications. This study conducted to demonstrate that real-time expression recognition systems based on deep learning techniques with the most appropriate architecture can be implemented with high accuracy via computer hardware with only one software. In addition, it is shown that high accuracy rate is achieved in real-time applications when Histograms of Oriented Gradients (HOG) is used as a feature extraction method and ResNet architecture is used for classification.

Keywords

Image Processing, classification, Emotion analysis, Convoultional Neural Network, deep learning

References

• Abdullah, S. M. S., and Abdulazeez, A. M. (2021). “Facial expression recognition based on deep learning convolution neural network: A review.” Journal of Soft Computing and Data Mining, 2(1), 53-65. Retrieved from: https://publisher.uthm.edu.my/ojs/index.php/jscdm/article/view/7906
• Akinrotimi, A, O. (2018) “Facial Emotion Recognition Using Principal Component Analysis and Support Vector Machine” Retrieved from: https://www.researchgate.net/publication/336120020_Facial_Emotion_Recognition_Using_Principal_Component_Analysis_and_Support_Vector_Machine
• Bhattacharya, S. (2021) “A Survey on: Facial Expression Recognition Using Various Deep Learning Techniques”, Advances in Intelligent Systems and Computing, Advanced Computational Paradigms and Hybrid Intelligent Computing pp 619–631. Retrieved from: https://doi.org/10.1007/978-981-16-4369-9_59
• Bisogni, C., Castiglione, A., Hossain, S., Narducci, F, and Umer S. (2022), “Impact of Deep Learning Approaches on Facial Expression Recognition in Healthcare Industries”, IEEE Transactions on Industrial Informatics, vol. 18, no. 8. Retrieved from: https://ieeexplore.ieee.org/document/9674818
• Buhari A. M., Ooi, C. P., Baskaran, V.M., Phan, R. CW, Wong, K. and Tan, W-H. (2022) “Invisible emotion magnification algorithm (IEMA) for real-time micro-expression recognition with graph-based features”, Advances in Soft Computing Techniques for Visual Information-based Systems, 81, pages 9151–9176. DOI https://doi.org/10.1007/s11042-021-11625-1
• Chen, Y., Wu, H. (2018) "A Comparison of Methods of Facial Expression Recognition" Retrieved from: https://ieeexplore.ieee.org/document/8584202
• Coşar, S., (2008) “Facial Feature Point Tracking Based On A Graphical Model Framework” Retrieved From: https://vpa.sabanciuniv.edu/vpadb/findfile_get_anonymous_paper.php?f=1954
• Çınar, A., C., (2018) “Deep Learning” Retrieved From: https://www.ahmetcevahircinar.com.tr/2017/08/11/imagenet-classification-with-deep-convolutional-neural-networks/
• Colmenarez, A., Frey, B. and Huang, T.S. (1999) “A probabilistic framework for embedded face and facial expression recognition.” Proceedings 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), 1999, pp. 592-597 Vol. 1, doi: 10.1109/CVPR.1999.786999. DOI: 10.1109/CVPR.1999.786999
• Devrim, M., O. (2019) “Farklı Derin Evrişimsel Sinir Ağlarının Yüz İfadesi Tanıma İşleminde Karşılaştırılması” Retrieved from: https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=vLQ0gfV9vTaOpH16k6kjMA&no=ZgJjiVHnbVeqxh9M0L2OHg
• Dhavalikar, A., Kulkarni, R., K., (2014) “Face detection and facial expression recognition system” Retrieved From: https://www.researchgate.net/publication/286592694_Face_detection_and_facial_expression_recognition_system
• Engin, D., (2017) “Facial Expression Pair Matching” Retrieved from: https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=2bQ8F7EJQFFi10i_DbMMkQ&no=poooj2n8rQ268jwtzPwoqw
• Frank, M., G., (2001) “Facial Expressions” Retrieved From: https://www.sciencedirect.com/topics/computer-science/facial-expression
• Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning. Regularization for Deep learning pg 228-270, Cambridge, Massachusetts: MIT Press. Retrieved From: http://www.deeplearningbook.org
• Güneş, T., Polat, E. (2009) "Feature Selection In Facial Expression Analysis And Its Effect On Multi-Svm Classifiers" Retrieved from: https://www.researchgate.net/publication/292375988_Feature_selection_in_facial_expression_analysis_and_its_effect_on_multi-svm_classifiers
• Göngör, F., and Tutsoy, O. (2018). “Eigenface based emotion analysis algorithm and implementation to humanoid robot.” In International Science and Academic Congress.
• Islam, K., Al-Murad, A. (2017) “Performance of SVM, CNN, and ANN with BoW, HOG, and Image Pixels in Face Recognition” Retrieved from: https://www.researchgate.net/publication/326682312_Performance_of_SVM_CNN_and_ANN_with_BoW_HOG_and_Image_Pixels_in_Face_Recognition
• Jung H. et al., (2015) "Development of deep learning-based facial expression recognition system" 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), pp. 1-4, doi: 10.1109/FCV.2015.7103729. Retrieved from: https://ieeexplore.ieee.org/document/7103729
• Karaboyacı, C., (2009) “Geometrical Feature Based Automated Facial Expression Analysis” Retrieved from: https://polen.itu.edu.tr/bitstream/11527/339/1/9597.pdf
• Khoong, W., H., (2021) “When Do Support Vector Machines Fail?” Retrieved From: https://towardsdatascience.com/when-do-support-vector-machines-fail-3f23295ebef2
• Kotsia, I., Pitas, I., (2008) “Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines” Retrieved From: https://www.researchgate.net/publication/6520512_Facial_Expression_Recognition_in_Image_Sequences_Using_Geometric_Deformation_Features_and_Support_Vector_Machines
• Kuo, C., Lai, S. and Sarkis, M. (2018) "A Compact Deep Learning Model for Robust Facial Expression Recognition" 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2202-22028, doi: 10.1109/CVPRW.2018.00286. Retrieved from: https://ieeexplore.ieee.org/document/8575457
• Lucey, P., Cohn, J., Kanade T. (2010) “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression” Retrieved From: https://ieeexplore.ieee.org/document/5543262
• Li, B. and Lima, D. (2021) “Facial expression recognition via ResNet-50”, International Journal of Cognitive Computing in Engineering, 57-64. Retrieved From: https://doi.org/10.1016/j.ijcce.2021.02.002
• Li, S. (2020) “Deep Facial Expression Recognition: A Survey”, IEEE Transactions on Affective Computing, pp 1:20. doi: 10.1109/TAFFC.2020.2981446
• Mehrabian, A. (2016), https://en.wikipedia.org/wiki/Albert_Mehrabian, 01.06.2022.
• Mena-Chalco, J.P., Macedo, I., Velho, L. and Cesar, R.M. (2009), “3D face computational photography using PCA spaces”, Visual Computer, 2009, 25:899-909. Retrieved From: https://doi.org/10.1007/s00371-009-0373-x
• Mokhtari, N., I., (2021) “Which is Better for Your Machine Learning Task” Retrieved From: https://towardsdatascience.com/which-is-better-for-your-machine-learning-task-opencv-or-tensorflow-ed16403c5799
• Nagaraj, P., Banala, R., (2021) “Real Time Face Recognition using Effective Supervised Machine Learning Algorithms” Retrieved From: https://www.researchgate.net/publication/354095467_Real_Time_Face_Recognition_using_Effective_Supervised_Machine_Learning_Algorithms
• Ronthkrantz, J., M., Pantic, M., (2001) “Automatic Analysis of Facial Expressions: The State of the Art”. Retrieved From: https://www.researchgate.net/publication/3193199_Automatic_Analysis_of_Facial_Expressions_The_State_of_the_Art
• Özdemir, M., A., Elagöz, B., Alaybeyoğlu, A., Akan, A. (2019) “Real Time Emotion Recognition from Facial Expressions Using CNN Architecture” Retrieved from: https://www.researchgate.net/publication/336287978_Real_Time_Emotion_Recognition_from_Facial_Expressions_Using_CNN_Architecture
• Özdemir and Hanbay. (2022) “Deep feature selection for facial emotion recognition based on BPSO and SVM” Retrieved From: https://dergipark.org.tr/tr/pub/politeknik/issue/33364/992720
• Pantic, M., Rothkrantz, J., (2001), “Automatic Analysis of Facial Expressions: The State of the Art” Retrieved from: https://www.researchgate.net/publication/3193199_Automatic_Analysis_of_Facial_Expressions_The_State_of_the_Art
• Paleari, M., Velardo, C., Huet, B., Dugelay, J. (2009) “Face dynamics for biometric people recognition” Retrieved From: http://www.eurecom.fr/en/publication/2888/download/mm-publi-2888.pdf
• Python Software Foundation (2012) Retrieved from: https://web.archive.org/web/20121024164224/http://docs.python.org/faq/general.html
• Saurav, S., Gidde, P., Saini, R., and Singh, S. (2022) “Dual integrated convolutional neural network for real-time facial expression recognition in the wild” The Visual Computer volume 38, pages 1083–1096. Retrieved from: https://doi.org/10.1007/s00371-021-02069-7
• Saxena, A., Khanna, A., Gupta, D. (2020) "Emotion Recognition and Detection Methods" Retrieved from:https://www.researchgate.net/publication/339119986_Emotion_Recognition_and_Detection_Methods_A_Comprehensive_Survey
• Saraçbaşı, N., Zeynep. (2021) “Face Recognition Using Facial Dynamics of Emotional Expressions” Retrieved From: https://tez.yok.gov.tr/UlusalTezMerkezi/TezGoster?key=tqUiYt63sTQLTpozMJ92QpYVnXW_2-VG0SZiwpDZZlLMRrUm2f-TqRQA8TWiyPik
• Sebe, N., Cohen, I., Huang, T.S (2004) "Multimodal Emotion Recognition" Retrieved from: https://www.researchgate.net/publication/228616884_Multimodal_emotion_recognition
• Sharma, P. (2021) “Understanding Transfer Learning for Deep Learning ” Retrieved from: https://www.analyticsvidhya.com/blog/2021/10/understanding-transfer-learning-for-deep-learning/
• Srivastava, T., (2018) “Introduction to k-Nearest Neighbors” Retrieved From: https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/
• Terra, J., (2022) “Key Differences Among the Deep Learning Framework”, Retrieved From: https://www.simplilearn.com/keras-vs-tensorflow-vs-pytorch-article#:~:text=TensorFlow%20is%20an%20open%2Dsourced,because%20it's%20built%2Din%20Python.
• Tutsoy, O., Güngör, F., Barkana, D. E., & Köse, H. (2017). An emotion analysis algorithm and implementation to NAO humanoid robot. The Eurasia Proceedings of Science Technology Engineering and Mathematics, (1), 316-330. Retrieved From: http://www.epstem.net/tr/download/article-file/381431
• Umer, S., Rout, R.K., Pero, C., and Nappi, M. (2022) “Facial expression recognition with trade-offs between data augmentation and deep learning features”, Journal of Ambient Intelligence and Humanized Computing volume 13, pages721–735. Retrieved From: https://doi.org/10.1007/s12652-020-02845-8
• Verma, G., Verma, H. (2020) “Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions” Retrieved from: https://www.researchgate.net/publication/343556711_Hybrid-Deep_Learning_Model_for_Emotion_Recognition_Using_Facial_Expressions
• Yang, M., H., Kriegman, D., (2002) “Detecting Faces in Images: A Survey Retrieved from: https://www.researchgate.net/publication/3193340_Detecting_Faces_in_Images_A_Survey
• Zhao, X., Shi X. and Zhang, S. (2015) “Facial Expression Recognition via Deep Learning” IETE Technical Review, 32:5, 347-355, DOI:10.1080/02564602.2015.1017542 Retrieved From: https://ieeexplore.ieee.org/document/7043872