ijiam International Journal of Informatics and Applied Mathematics 2667-6990 International Society of Academicians Software Engineering (Other) Yazılım Mühendisliği (Diğer) Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview Sari Meriem University of Farhat Abbas Setif Moussaouı Abdelouahab University of Ferhat Abbas Setif1 Hadid Abdenour University of Oulu 06 01 2020 3 1 39 53 12 10 2019 02 19 2020 Copyright © 2018, International Journal of Informatics and Applied Mathematics 2018 International Journal of Informatics and Applied Mathematics

Facial expression recognition (FER) plays a key role in conveying human emotions and feelings. Automated FER systems enable different machines to recognize emotions without the help of humans; this is considered as a very challenging problem in machine learning. Over the years there has been a considerable progress in this field. In this paper we present a state of the art overview on the different concepts of a FER system and the different used methods; plus we studied the efficiency of using deep learning architectures specifically convolutional neural networks architectures (CNN) as a new solution for FER problems by investigating the most recent and cited works.

 Facial Expression Recognition Emotion Recognition Machine Learning Deep Learning Convolutional Neural Network 1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence (12), 2037{2041 (2006) 2. Arriaga, O., Valdenegro-Toro, M., Ploger, P.: Real-time convolutional neural networks for emotion and gender classi cation. arXiv preprint arXiv:1710.07557 (2017) 3. Chakraborty, A., Konar, A., Chakraborty, U.K., Chatterjee, A.: Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 39(4), 726{743 (2009) 4. Chang, W.J., Schmelzer, M., Kopp, F., Hsu, C.H., Su, J.P., Chen, L.B., Chen, M.C.: A deep learning facial expression recognition based scoring system for restaurants. In: 2019 International Conference on Arti cial Intelligence in Information and Communication (ICAIIC). pp. 251{254. IEEE (2019) 5. Christou, N., Kanojiya, N.: Human facial expression recognition with convolution neural networks. In: Third International Congress on Information and Communication Technology. pp. 539{545. Springer (2019) 6. Cohen, I., Sebe, N., Gozman, F., Cirelo, M.C., Huang, T.S.: Learning bayesian network classi ers for facial expression recognition both labeled and unlabeled data. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. vol. 1, pp. I{I. IEEE (2003) 7. Darwin, C., Prodger, P.: The expression of the emotions in man and animals. Oxford University Press, USA (1998) 8. Dash, M., Liu, H.: Feature selection for classi cation. Intelligent data analysis 1(1- 4), 131{156 (1997) 9. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. Journal of personality and social psychology 17(2), 124 (1971) 10. Ekman, R.: What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA (1997) 11. El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classi cation schemes, and databases. Pattern Recognition 44(3), 572{ 587 (2011) 12. Eusebio, J.M.A.: Convolutional neural networks for facial expression recognition (2016) 13. Friesen, E., Ekman, P.: Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3 (1978) 14. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714{7734 (2013) 15. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., et al.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing. pp. 117{124. Springer (2013) 16. Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE transactions on A ective Computing 6(1), 1{12 (2014) 17. Harms, M.B., Martin, A.,Wallace, G.L.: Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychology review 20(3), 290{322 (2010) 18. Hasani, B., Mahoor, M.H.: Facial expression recognition using enhanced deep 3d convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 30{40 (2017) 19. Heiselet, B., Serre, T., Pontil, M., Poggio, T.: Component-based face detection. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. vol. 1, pp. I{I. IEEE (2001) 20. Jafri, R., Arabnia, H.R.: A survey of face recognition techniques. Jips 5(2), 41{68 (2009) 21. Jain, D.K., Shamsolmoali, P., Sehdev, P.: Extended deep neural network for facial emotion recognition. Pattern Recognition Letters 120, 69{74 (2019) 22. Karpathy, A., et al.: Cs231n convolutional neural networks for visual recognition. Neural networks 1 (2016) 23. Kobayashi, H., Hara, F.: Recognition of six basic facial expression and their strength by neural network. In: [1992] Proceedings IEEE International Workshop on Robot and Human Communication. pp. 381{386. IEEE (1992) 24. Koelstra, S., Pantic, M., Patras, I.: A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE transactions on pattern analysis and machine intelligence 32(11), 1940{1954 (2010) 25. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1667{1675 (2017) 26. Kristensen, R.L., Tan, Z.H., Ma, Z., Guo, J.: Binary pattern avored feature extractors for facial expression recognition: An overview. In: 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). pp. 1131{1137. IEEE (2015) 27. Kumari, J., Rajesh, R., Pooja, K.: Facial expression recognition: A survey. Procedia Computer Science 58, 486{491 (2015) 28. Lawrence, S., Giles, C.L., Tsoi, A.C., Back, A.D.: Face recognition: A convolutional neural-network approach. IEEE transactions on neural networks 8(1), 98{ 113 (1997) 29. Lee, I., Jung, H., Ahn, C.H., Seo, J., Kim, J., Kwon, O.: Real-time personalized facial expression recognition system based on deep learning. In: 2016 IEEE International Conference on Consumer Electronics (ICCE). pp. 267{268. IEEE (2016) 30. Levi, G., Hassner, T.: Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. pp. 503{510. ACM (2015) 31. Li, S.Z., Zou, X., Hu, Y., Zhang, Z., Yan, S., Peng, X., Huang, L., Zhang, H.: Realtime multi-view face detection, tracking, pose estimation, alignment, and recognition. IEEE CVPR Demo Summary (2001) 32. Liu, K., Zhang, M., Pan, Z.: Facial expression recognition with cnn ensemble. In: 2016 international conference on cyberworlds (CW). pp. 163{166. IEEE (2016) 33. Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126{136 (2015) 34. Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognition 61, 610{628 (2017) 35. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotionspeci ed expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. pp. 94{101. IEEE (2010) 36. Lucey, P., Cohn, J.F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., Prkachin, K.M.: Automatically detecting pain in video through facial action units. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41(3), 664{ 674 (2010) 37. Majumder, A., Behera, L., Subramanian, V.K.: Automatic facial expression recognition system using deep network-based data fusion. IEEE transactions on cybernetics 48(1), 103{114 (2016) 38. Mavani, V., Raman, S., Miyapuram, K.P.: Facial expression recognition using visual saliency and deep learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2783{2788 (2017) 39. Michael, J., Lyons, M.K., Gyoba, J.: Japanese female facial expressions (ja e). Database of digital images (1997) 40. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV). pp. 1{10. IEEE (2016) 41. Mollahosseini, A., Hasani, B., Mahoor, M.H.: A ectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on A ective Computing 10(1), 18{31 (2017) 42. Pentland, A., Moghaddam, B., Starner, T., et al.: View-based and modular eigenspaces for face recognition (1994) 43. Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 23{38 (1998) 44. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3d facial expression recognition: A comprehensive survey. Image and Vision Computing 30(10), 683{697 (2012) 45. Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85{117 (2015) 46. Schneiderman, H., Kanade, T.: A statistical approach to 3D object detection applied to faces and cars. Carnegie Mellon University, the Robotics Institute (2000) 47. Shan, K., Guo, J., You, W., Lu, D., Bie, R.: Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA). pp. 123{128. IEEE (2017) 48. Sprengelmeyer, R., Young, A., Mahn, K., Schroeder, U., Woitalla, D., Buttner, T., Kuhn, W., Przuntek, H.: Facial expression recognition in people with medicated and unmedicated parkinsons disease. Neuropsychologia 41(8), 1047{1057 (2003) 49. Sung, K.K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 39{51 (1998) 50. Tang, J., Zhou, X., Zheng, J.: Design of intelligent classroom facial recognition based on deep learning. In: Journal of Physics: Conference Series. vol. 1168, p. 022043. IOP Publishing (2019) 51. Tian, Y., Kanade, T., Cohn, J.F.: Facial expression recognition. In: Handbook of face recognition, pp. 487{519. Springer (2011) 52. Uddin Ahmed, T., Hossain, S., Hossain, M.S., Ul Islam, R., Andersson, K.: Facial expression recognition using convolutional neural network with data augmentation. In: Joint 2019 8th International Conference on Informatics, Electronics & Vision (ICIEV) (2019) 53. Valstar, M.F., Almaev, T., Girard, J.M., McKeown, G., Mehu, M., Yin, L., Pantic, M., Cohn, J.F.: Fera 2015-second facial expression recognition and analysis challenge. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). vol. 6, pp. 1{8. IEEE (2015) 54. Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The rst facial expression recognition and analysis challenge. In: Face and Gesture 2011. pp. 921{926. IEEE (2011) 55. Viola, P., Jones, M.J.: Robust real-time face detection. International journal of computer vision 57(2), 137{154 (2004) 56. Wu, Y., Hassner, T., Kim, K., Medioni, G., Natarajan, P.: Facial landmark detection with tweaked convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence 40(12), 3067{3074 (2017) 57. Yang, J., Zhang, D., Frangi, A.F., Yang, J.y.: Two-dimensional pca: a new approach to appearance-based face representation and recognition. IEEE transactions on pattern analysis and machine intelligence 26(1), 131{137 (2004) 58. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. pp. 435{442. ACM (2015) 59. Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., Dobaie, A.M.: Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273, 643{649 (2018) 60. Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Transactions on A ective Computing 2(4), 219{229 (2011) 61. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural networkdriven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12), 2528{2536 (2016) 62. Zhang, Y.D., Yang, Z.J., Lu, H.M., Zhou, X.X., Phillips, P., Liu, Q.M.,Wang, S.H.: Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and strati ed cross validation. IEEE Access 4, 8375{8385 (2016) 63. Zhao, X., Shi, X., Zhang, S.: Facial expression recognition via deep learning. IETE technical review 32(5), 347{355 (2015) 64. Zhao, X., Zhang, S.: A review on facial expression recognition: Feature extraction and classi cation. IETE Technical Review 33(5), 505{517 (2016)