Year 2020, Volume 3 , Issue 1, Pages 39 - 53 2020-06-01

Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview

Meriem SARİ [1] , Abdelouahab MOUSSAOUI [2] , Abdenour HADİD [3]


Facial expression recognition (FER) plays a key role in conveying human emotions and feelings. Automated FER systems enable different machines to recognize emotions without the help of humans; this is considered as a very challenging problem in machine learning. Over the years there has been a considerable progress in this field. In this paper we present a state of the art overview on the different concepts of a FER system and the different used methods; plus we studied the efficiency of using deep learning architectures specifically convolutional neural networks architectures (CNN) as a new solution for FER problems by investigating the most recent and cited works.
Facial Expression Recognition, Emotion Recognition, Machine Learning, Deep Learning, Convolutional Neural Network
  • 1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence (12), 2037{2041 (2006)
  • 2. Arriaga, O., Valdenegro-Toro, M., Ploger, P.: Real-time convolutional neural networks for emotion and gender classi cation. arXiv preprint arXiv:1710.07557 (2017)
  • 3. Chakraborty, A., Konar, A., Chakraborty, U.K., Chatterjee, A.: Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 39(4), 726{743 (2009)
  • 4. Chang, W.J., Schmelzer, M., Kopp, F., Hsu, C.H., Su, J.P., Chen, L.B., Chen, M.C.: A deep learning facial expression recognition based scoring system for restaurants. In: 2019 International Conference on Arti cial Intelligence in Information and Communication (ICAIIC). pp. 251{254. IEEE (2019)
  • 5. Christou, N., Kanojiya, N.: Human facial expression recognition with convolution neural networks. In: Third International Congress on Information and Communication Technology. pp. 539{545. Springer (2019)
  • 6. Cohen, I., Sebe, N., Gozman, F., Cirelo, M.C., Huang, T.S.: Learning bayesian network classi ers for facial expression recognition both labeled and unlabeled data. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. vol. 1, pp. I{I. IEEE (2003)
  • 7. Darwin, C., Prodger, P.: The expression of the emotions in man and animals. Oxford University Press, USA (1998)
  • 8. Dash, M., Liu, H.: Feature selection for classi cation. Intelligent data analysis 1(1- 4), 131{156 (1997)
  • 9. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. Journal of personality and social psychology 17(2), 124 (1971)
  • 10. Ekman, R.: What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA (1997)
  • 11. El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classi cation schemes, and databases. Pattern Recognition 44(3), 572{ 587 (2011)
  • 12. Eusebio, J.M.A.: Convolutional neural networks for facial expression recognition (2016)
  • 13. Friesen, E., Ekman, P.: Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3 (1978)
  • 14. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714{7734 (2013)
  • 15. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., et al.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing. pp. 117{124. Springer (2013)
  • 16. Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE transactions on A ective Computing 6(1), 1{12 (2014)
  • 17. Harms, M.B., Martin, A.,Wallace, G.L.: Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychology review 20(3), 290{322 (2010)
  • 18. Hasani, B., Mahoor, M.H.: Facial expression recognition using enhanced deep 3d convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 30{40 (2017)
  • 19. Heiselet, B., Serre, T., Pontil, M., Poggio, T.: Component-based face detection. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. vol. 1, pp. I{I. IEEE (2001)
  • 20. Jafri, R., Arabnia, H.R.: A survey of face recognition techniques. Jips 5(2), 41{68 (2009)
  • 21. Jain, D.K., Shamsolmoali, P., Sehdev, P.: Extended deep neural network for facial emotion recognition. Pattern Recognition Letters 120, 69{74 (2019)
  • 22. Karpathy, A., et al.: Cs231n convolutional neural networks for visual recognition. Neural networks 1 (2016)
  • 23. Kobayashi, H., Hara, F.: Recognition of six basic facial expression and their strength by neural network. In: [1992] Proceedings IEEE International Workshop on Robot and Human Communication. pp. 381{386. IEEE (1992)
  • 24. Koelstra, S., Pantic, M., Patras, I.: A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE transactions on pattern analysis and machine intelligence 32(11), 1940{1954 (2010)
  • 25. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1667{1675 (2017)
  • 26. Kristensen, R.L., Tan, Z.H., Ma, Z., Guo, J.: Binary pattern avored feature extractors for facial expression recognition: An overview. In: 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). pp. 1131{1137. IEEE (2015)
  • 27. Kumari, J., Rajesh, R., Pooja, K.: Facial expression recognition: A survey. Procedia Computer Science 58, 486{491 (2015)
  • 28. Lawrence, S., Giles, C.L., Tsoi, A.C., Back, A.D.: Face recognition: A convolutional neural-network approach. IEEE transactions on neural networks 8(1), 98{ 113 (1997)
  • 29. Lee, I., Jung, H., Ahn, C.H., Seo, J., Kim, J., Kwon, O.: Real-time personalized facial expression recognition system based on deep learning. In: 2016 IEEE International Conference on Consumer Electronics (ICCE). pp. 267{268. IEEE (2016)
  • 30. Levi, G., Hassner, T.: Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. pp. 503{510. ACM (2015)
  • 31. Li, S.Z., Zou, X., Hu, Y., Zhang, Z., Yan, S., Peng, X., Huang, L., Zhang, H.: Realtime multi-view face detection, tracking, pose estimation, alignment, and recognition. IEEE CVPR Demo Summary (2001)
  • 32. Liu, K., Zhang, M., Pan, Z.: Facial expression recognition with cnn ensemble. In: 2016 international conference on cyberworlds (CW). pp. 163{166. IEEE (2016)
  • 33. Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126{136 (2015)
  • 34. Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognition 61, 610{628 (2017)
  • 35. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotionspeci ed expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. pp. 94{101. IEEE (2010)
  • 36. Lucey, P., Cohn, J.F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., Prkachin, K.M.: Automatically detecting pain in video through facial action units. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41(3), 664{ 674 (2010)
  • 37. Majumder, A., Behera, L., Subramanian, V.K.: Automatic facial expression recognition system using deep network-based data fusion. IEEE transactions on cybernetics 48(1), 103{114 (2016)
  • 38. Mavani, V., Raman, S., Miyapuram, K.P.: Facial expression recognition using visual saliency and deep learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2783{2788 (2017)
  • 39. Michael, J., Lyons, M.K., Gyoba, J.: Japanese female facial expressions (ja e). Database of digital images (1997)
  • 40. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV). pp. 1{10. IEEE (2016)
  • 41. Mollahosseini, A., Hasani, B., Mahoor, M.H.: A ectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on A ective Computing 10(1), 18{31 (2017)
  • 42. Pentland, A., Moghaddam, B., Starner, T., et al.: View-based and modular eigenspaces for face recognition (1994)
  • 43. Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 23{38 (1998)
  • 44. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3d facial expression recognition: A comprehensive survey. Image and Vision Computing 30(10), 683{697 (2012)
  • 45. Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85{117 (2015)
  • 46. Schneiderman, H., Kanade, T.: A statistical approach to 3D object detection applied to faces and cars. Carnegie Mellon University, the Robotics Institute (2000)
  • 47. Shan, K., Guo, J., You, W., Lu, D., Bie, R.: Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA). pp. 123{128. IEEE (2017)
  • 48. Sprengelmeyer, R., Young, A., Mahn, K., Schroeder, U., Woitalla, D., Buttner, T., Kuhn, W., Przuntek, H.: Facial expression recognition in people with medicated and unmedicated parkinsons disease. Neuropsychologia 41(8), 1047{1057 (2003)
  • 49. Sung, K.K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 39{51 (1998)
  • 50. Tang, J., Zhou, X., Zheng, J.: Design of intelligent classroom facial recognition based on deep learning. In: Journal of Physics: Conference Series. vol. 1168, p. 022043. IOP Publishing (2019)
  • 51. Tian, Y., Kanade, T., Cohn, J.F.: Facial expression recognition. In: Handbook of face recognition, pp. 487{519. Springer (2011)
  • 52. Uddin Ahmed, T., Hossain, S., Hossain, M.S., Ul Islam, R., Andersson, K.: Facial expression recognition using convolutional neural network with data augmentation. In: Joint 2019 8th International Conference on Informatics, Electronics & Vision (ICIEV) (2019)
  • 53. Valstar, M.F., Almaev, T., Girard, J.M., McKeown, G., Mehu, M., Yin, L., Pantic, M., Cohn, J.F.: Fera 2015-second facial expression recognition and analysis challenge. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). vol. 6, pp. 1{8. IEEE (2015)
  • 54. Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The rst facial expression recognition and analysis challenge. In: Face and Gesture 2011. pp. 921{926. IEEE (2011)
  • 55. Viola, P., Jones, M.J.: Robust real-time face detection. International journal of computer vision 57(2), 137{154 (2004)
  • 56. Wu, Y., Hassner, T., Kim, K., Medioni, G., Natarajan, P.: Facial landmark detection with tweaked convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence 40(12), 3067{3074 (2017)
  • 57. Yang, J., Zhang, D., Frangi, A.F., Yang, J.y.: Two-dimensional pca: a new approach to appearance-based face representation and recognition. IEEE transactions on pattern analysis and machine intelligence 26(1), 131{137 (2004)
  • 58. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. pp. 435{442. ACM (2015)
  • 59. Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., Dobaie, A.M.: Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273, 643{649 (2018)
  • 60. Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Transactions on A ective Computing 2(4), 219{229 (2011)
  • 61. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural networkdriven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12), 2528{2536 (2016)
  • 62. Zhang, Y.D., Yang, Z.J., Lu, H.M., Zhou, X.X., Phillips, P., Liu, Q.M.,Wang, S.H.: Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and strati ed cross validation. IEEE Access 4, 8375{8385 (2016)
  • 63. Zhao, X., Shi, X., Zhang, S.: Facial expression recognition via deep learning. IETE technical review 32(5), 347{355 (2015)
  • 64. Zhao, X., Zhang, S.: A review on facial expression recognition: Feature extraction and classi cation. IETE Technical Review 33(5), 505{517 (2016)
Primary Language en
Subjects Computer Science, Interdisciplinary Application
Journal Section Articles
Authors

Author: Meriem SARİ (Primary Author)
Institution: University of Farhat Abbas Setif
Country: Algeria


Author: Abdelouahab MOUSSAOUI
Institution: University of Ferhat Abbas Setif1
Country: Algeria


Author: Abdenour HADİD
Institution: University of Oulu
Country: Finland


Dates

Publication Date : June 1, 2020

Bibtex @research article { ijiam657395, journal = {International Journal of Informatics and Applied Mathematics}, issn = {}, eissn = {2667-6990}, address = {}, publisher = {International Society of Academicians}, year = {2020}, volume = {3}, pages = {39 - 53}, doi = {}, title = {Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview}, key = {cite}, author = {Sari̇, Meriem and Moussaouı, Abdelouahab and Hadi̇d, Abdenour} }
APA Sari̇, M , Moussaouı, A , Hadi̇d, A . (2020). Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview . International Journal of Informatics and Applied Mathematics , 3 (1) , 39-53 . Retrieved from https://dergipark.org.tr/en/pub/ijiam/issue/54619/657395
MLA Sari̇, M , Moussaouı, A , Hadi̇d, A . "Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview" . International Journal of Informatics and Applied Mathematics 3 (2020 ): 39-53 <https://dergipark.org.tr/en/pub/ijiam/issue/54619/657395>
Chicago Sari̇, M , Moussaouı, A , Hadi̇d, A . "Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview". International Journal of Informatics and Applied Mathematics 3 (2020 ): 39-53
RIS TY - JOUR T1 - Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview AU - Meriem Sari̇ , Abdelouahab Moussaouı , Abdenour Hadi̇d Y1 - 2020 PY - 2020 N1 - DO - T2 - International Journal of Informatics and Applied Mathematics JF - Journal JO - JOR SP - 39 EP - 53 VL - 3 IS - 1 SN - -2667-6990 M3 - UR - Y2 - 2020 ER -
EndNote %0 International Journal of Informatics and Applied Mathematics Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview %A Meriem Sari̇ , Abdelouahab Moussaouı , Abdenour Hadi̇d %T Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview %D 2020 %J International Journal of Informatics and Applied Mathematics %P -2667-6990 %V 3 %N 1 %R %U
ISNAD Sari̇, Meriem , Moussaouı, Abdelouahab , Hadi̇d, Abdenour . "Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview". International Journal of Informatics and Applied Mathematics 3 / 1 (June 2020): 39-53 .
AMA Sari̇ M , Moussaouı A , Hadi̇d A . Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview. IJIAM. 2020; 3(1): 39-53.
Vancouver Sari̇ M , Moussaouı A , Hadi̇d A . Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview. International Journal of Informatics and Applied Mathematics. 2020; 3(1): 39-53.
IEEE M. Sari̇ , A. Moussaouı and A. Hadi̇d , "Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview", International Journal of Informatics and Applied Mathematics, vol. 3, no. 1, pp. 39-53, Jun. 2020