Automated Facial Expression Recognition Using Deep Learning Techniques: An Overview

Abstract

Facial expression recognition (FER) plays a key role in conveying human emotions and feelings. Automated FER systems enable different machines to recognize emotions without the help of humans; this is considered as a very challenging problem in machine learning. Over the years there has been a considerable progress in this field. In this paper we present a state of the art overview on the different concepts of a FER system and the different used methods; plus we studied the efficiency of using deep learning architectures specifically convolutional neural networks architectures (CNN) as a new solution for FER problems by investigating the most recent and cited works.

Keywords

• Facial Expression Recognition,Emotion Recognition,Machine Learning,Deep Learning,Convolutional Neural Network

References

1. 1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence (12), 2037{2041 (2006)
2. 2. Arriaga, O., Valdenegro-Toro, M., Ploger, P.: Real-time convolutional neural networks for emotion and gender classication. arXiv preprint arXiv:1710.07557 (2017)
3. 3. Chakraborty, A., Konar, A., Chakraborty, U.K., Chatterjee, A.: Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 39(4), 726{743 (2009)
4. 4. Chang, W.J., Schmelzer, M., Kopp, F., Hsu, C.H., Su, J.P., Chen, L.B., Chen, M.C.: A deep learning facial expression recognition based scoring system for restaurants. In: 2019 International Conference on Articial Intelligence in Information and Communication (ICAIIC). pp. 251{254. IEEE (2019)
5. 5. Christou, N., Kanojiya, N.: Human facial expression recognition with convolution neural networks. In: Third International Congress on Information and Communication Technology. pp. 539{545. Springer (2019)
6. 6. Cohen, I., Sebe, N., Gozman, F., Cirelo, M.C., Huang, T.S.: Learning bayesian network classiers for facial expression recognition both labeled and unlabeled data. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. vol. 1, pp. I{I. IEEE (2003)
7. 7. Darwin, C., Prodger, P.: The expression of the emotions in man and animals. Oxford University Press, USA (1998)
8. 8. Dash, M., Liu, H.: Feature selection for classication. Intelligent data analysis 1(1- 4), 131{156 (1997)

9. 9. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. Journal of personality and social psychology 17(2), 124 (1971)
10. 10. Ekman, R.: What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA (1997)
11. 11. El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classication schemes, and databases. Pattern Recognition 44(3), 572{ 587 (2011)
12. 12. Eusebio, J.M.A.: Convolutional neural networks for facial expression recognition (2016)
13. 13. Friesen, E., Ekman, P.: Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3 (1978)
14. 14. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714{7734 (2013)
15. 15. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., et al.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing. pp. 117{124. Springer (2013)
16. 16. Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE transactions on Aective Computing 6(1), 1{12 (2014)
17. 17. Harms, M.B., Martin, A.,Wallace, G.L.: Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychology review 20(3), 290{322 (2010)
18. 18. Hasani, B., Mahoor, M.H.: Facial expression recognition using enhanced deep 3d convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 30{40 (2017)
19. 19. Heiselet, B., Serre, T., Pontil, M., Poggio, T.: Component-based face detection. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. vol. 1, pp. I{I. IEEE (2001)
20. 20. Jafri, R., Arabnia, H.R.: A survey of face recognition techniques. Jips 5(2), 41{68 (2009)
21. 21. Jain, D.K., Shamsolmoali, P., Sehdev, P.: Extended deep neural network for facial emotion recognition. Pattern Recognition Letters 120, 69{74 (2019)
22. 22. Karpathy, A., et al.: Cs231n convolutional neural networks for visual recognition. Neural networks 1 (2016)
23. 23. Kobayashi, H., Hara, F.: Recognition of six basic facial expression and their strength by neural network. In: [1992] Proceedings IEEE International Workshop on Robot and Human Communication. pp. 381{386. IEEE (1992)
24. 24. Koelstra, S., Pantic, M., Patras, I.: A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE transactions on pattern analysis and machine intelligence 32(11), 1940{1954 (2010)
25. 25. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1667{1675 (2017)
26. 26. Kristensen, R.L., Tan, Z.H., Ma, Z., Guo, J.: Binary pattern avored feature extractors for facial expression recognition: An overview. In: 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). pp. 1131{1137. IEEE (2015)
27. 27. Kumari, J., Rajesh, R., Pooja, K.: Facial expression recognition: A survey. Procedia Computer Science 58, 486{491 (2015)
28. 28. Lawrence, S., Giles, C.L., Tsoi, A.C., Back, A.D.: Face recognition: A convolutional neural-network approach. IEEE transactions on neural networks 8(1), 98{ 113 (1997)
29. 29. Lee, I., Jung, H., Ahn, C.H., Seo, J., Kim, J., Kwon, O.: Real-time personalized facial expression recognition system based on deep learning. In: 2016 IEEE International Conference on Consumer Electronics (ICCE). pp. 267{268. IEEE (2016)
30. 30. Levi, G., Hassner, T.: Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. pp. 503{510. ACM (2015)
31. 31. Li, S.Z., Zou, X., Hu, Y., Zhang, Z., Yan, S., Peng, X., Huang, L., Zhang, H.: Realtime multi-view face detection, tracking, pose estimation, alignment, and recognition. IEEE CVPR Demo Summary (2001)
32. 32. Liu, K., Zhang, M., Pan, Z.: Facial expression recognition with cnn ensemble. In: 2016 international conference on cyberworlds (CW). pp. 163{166. IEEE (2016)
33. 33. Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126{136 (2015)
34. 34. Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognition 61, 610{628 (2017)
35. 35. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotionspeci ed expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. pp. 94{101. IEEE (2010)
36. 36. Lucey, P., Cohn, J.F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., Prkachin, K.M.: Automatically detecting pain in video through facial action units. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41(3), 664{ 674 (2010)
37. 37. Majumder, A., Behera, L., Subramanian, V.K.: Automatic facial expression recognition system using deep network-based data fusion. IEEE transactions on cybernetics 48(1), 103{114 (2016)
38. 38. Mavani, V., Raman, S., Miyapuram, K.P.: Facial expression recognition using visual saliency and deep learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2783{2788 (2017)
39. 39. Michael, J., Lyons, M.K., Gyoba, J.: Japanese female facial expressions (jae). Database of digital images (1997)
40. 40. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV). pp. 1{10. IEEE (2016)
41. 41. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Aectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Aective Computing 10(1), 18{31 (2017)
42. 42. Pentland, A., Moghaddam, B., Starner, T., et al.: View-based and modular eigenspaces for face recognition (1994)
43. 43. Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 23{38 (1998)
44. 44. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3d facial expression recognition: A comprehensive survey. Image and Vision Computing 30(10), 683{697 (2012)
45. 45. Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85{117 (2015)
46. 46. Schneiderman, H., Kanade, T.: A statistical approach to 3D object detection applied to faces and cars. Carnegie Mellon University, the Robotics Institute (2000)
47. 47. Shan, K., Guo, J., You, W., Lu, D., Bie, R.: Automatic facial expression recognition based on a deep convolutional-neural-network structure. In: 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA). pp. 123{128. IEEE (2017)
48. 48. Sprengelmeyer, R., Young, A., Mahn, K., Schroeder, U., Woitalla, D., Buttner, T., Kuhn, W., Przuntek, H.: Facial expression recognition in people with medicated and unmedicated parkinsons disease. Neuropsychologia 41(8), 1047{1057 (2003)
49. 49. Sung, K.K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 39{51 (1998)
50. 50. Tang, J., Zhou, X., Zheng, J.: Design of intelligent classroom facial recognition based on deep learning. In: Journal of Physics: Conference Series. vol. 1168, p. 022043. IOP Publishing (2019)
51. 51. Tian, Y., Kanade, T., Cohn, J.F.: Facial expression recognition. In: Handbook of face recognition, pp. 487{519. Springer (2011)
52. 52. Uddin Ahmed, T., Hossain, S., Hossain, M.S., Ul Islam, R., Andersson, K.: Facial expression recognition using convolutional neural network with data augmentation. In: Joint 2019 8th International Conference on Informatics, Electronics & Vision (ICIEV) (2019)
53. 53. Valstar, M.F., Almaev, T., Girard, J.M., McKeown, G., Mehu, M., Yin, L., Pantic, M., Cohn, J.F.: Fera 2015-second facial expression recognition and analysis challenge. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). vol. 6, pp. 1{8. IEEE (2015)
54. 54. Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The rst facial expression recognition and analysis challenge. In: Face and Gesture 2011. pp. 921{926. IEEE (2011)
55. 55. Viola, P., Jones, M.J.: Robust real-time face detection. International journal of computer vision 57(2), 137{154 (2004)
56. 56. Wu, Y., Hassner, T., Kim, K., Medioni, G., Natarajan, P.: Facial landmark detection with tweaked convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence 40(12), 3067{3074 (2017)
57. 57. Yang, J., Zhang, D., Frangi, A.F., Yang, J.y.: Two-dimensional pca: a new approach to appearance-based face representation and recognition. IEEE transactions on pattern analysis and machine intelligence 26(1), 131{137 (2004)
58. 58. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. pp. 435{442. ACM (2015)
59. 59. Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., Dobaie, A.M.: Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273, 643{649 (2018)
60. 60. Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Transactions on Aective Computing 2(4), 219{229 (2011)
61. 61. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural networkdriven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12), 2528{2536 (2016)
62. 62. Zhang, Y.D., Yang, Z.J., Lu, H.M., Zhou, X.X., Phillips, P., Liu, Q.M.,Wang, S.H.: Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratied cross validation. IEEE Access 4, 8375{8385 (2016)
63. 63. Zhao, X., Shi, X., Zhang, S.: Facial expression recognition via deep learning. IETE technical review 32(5), 347{355 (2015)
64. 64. Zhao, X., Zhang, S.: A review on facial expression recognition: Feature extraction and classication. IETE Technical Review 33(5), 505{517 (2016)