Enhancing Facial Expression Recognition in the Wild with Deep Learning Methods Using a New Dataset: RidNet

Abstract

In this study, emotion recognition process was performed by using deep learning methods for seven different facial expressions with the data set (RidNet) which was created by using images that are publicly accessible from internet. Afterwards, transfer learning over RidNet was done with well-known convolutional neural network architectures such as AlexNet, GoogLeNet and ResNet101. Compound Facial Expressions of Emotion (CE) and Static Facial Expressions in the Wild (SFEW) datasets were determined as test datasets. In the first experimental studies, convolutional neural network architecture with the best classification performance was determined. This convolutional neural network was trained with AffectNet, The Karolinska Directed Emotional Faces and RidNet. Similar classification performances were achieved when the AffectNet, KDEF, and RidNet-trained networks were tested with the data set (CE) generated in a controlled environment. In the test data set (SFEW) in an uncontrolled environment, RidNet-trained network gave a significant advantage over other networks.

Keywords

• Facial Expression Recognition,Deep Learning,Transfer Learning,Convolutional Neural Networks

Project Number

2019-01.BŞEÜ.03-05

Yeni Bir Veri Kümesi (RidNet) Kullanarak Kontrolsüz Ortamda Yüz İfadesi Tanımanın Derin Öğrenme Yöntemleri ile İyileştirilmesi

Abstract

Bu çalışmada, internetten genel erişime açık görüntüler kullanılarak oluşturulan veri kümesi (RidNet) ile yedi farklı yüz ifadesi için derin öğrenme yöntemleri kullanılarak duygu tanıma işlemi yapılmıştır. Daha sonra AlexNet, GoogLeNet ve ResNet101 gibi literatürdeki tanınmış evrişimli sinir ağları mimarileri ile RidNet üzerinden transfer öğrenimi yapılmıştır. Compound Facial Expressions of Emotion (CE) ve Static Facial Expressions in the Wild (SFEW) veri kümeleri test veri kümeleri olarak belirlenmiştir. İlk olarak yapılan deneysel çalışmalar ile en iyi sınıflandırma performansını gösteren evrişimli sinir ağı mimarisi belirlenmiştir. Bu evrişimli sinir ağı AffectNet, The Karolinska Directed Emotional Faces (KDEF) ve RidNet ile eğitilmiştir. AffectNet, KDEF ve RidNet ile eğitilmiş ağlar kontrollü ortamda oluşturulan veri kümesi (CE) ile test edildiğinde benzer sınıflandırma başarımları elde edilmiştir. Kontrolsüz ortamdaki test veri kümesinde (SFEW) ise RidNet ile eğitilen ağ diğer ağlara belirgin bir üstünlük sağlamıştır.

Keywords

• Yüz İfadesi Tanıma,Derin Öğrenme,transfer öğrenmesi

Supporting Institution

Bilecik Şeyh Edebali Üniversitei

Project Number

2019-01.BŞEÜ.03-05

References

1. [1] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014). Going deeper with convolutions. arXiv preprint arXiv:1409.4842.
2. [2] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., & Li, F.-F. (2014). Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575.
3. [3] Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv preprint arXiv:1312.4400.
4. [4] Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. 25th International Conference on Neural Information Processing Systems - Volume 1, 1097–1105.
5. [5] Liu, M., Li, S., Shan, S., & Chen, X. (2013) Au-aware deep networks for facial expression recognition. 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 1–6.
6. [6] Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gulcehre, C., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R.C., et al. (2013). Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 543–550.
7. [7] Susskind, J.M., Anderson, A.K., & Hinton, G.E. (2010). The toronto face database. Technical report, UTML TR 2010-001, University of Toronto.
8. [8] Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013). Emotion recognition in the wild challenge 2013. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 509–516.

9. [9] Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., & Chen, X. (2014) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In Proceedings of the 16th International Conference on Multimodal Interaction, 494–501.
10. [10] Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2014). Deeply learning deformable facial action parts model for dynamic expression analysis. In Computer Vision–ACCV 2014, 143–157.
11. [11] Jung, H., Lee, S., Yim, J., Park, S., Kim, J. (2015). Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2983–2991.
12. [12] Zhao, K., Chu, W.S. & Zhang, H. (2016). Deep region and multi-label learning for facial action unit detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 3391–3399.
13. [13] Hasani, B., Mahoor, M.H. (2017). Facial expression recognition using enhanced deep 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Hawaii, HI, USA, 1–11.
14. [14] Li, S. & Deng, W. (2018) Deep facial expression recognition: A survey. arXiv preprint arXiv:1804.08348.
15. [15] Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y. & Dobaie, A.M. (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 273, 643–649.
16. [16] Otberdout, N., Kacem, A., Daoudi, M., Ballihi, L. & Berretti, S. (2018) Deep covariance descriptors for facial expression recognition. 29th British Machine Vision Conferernce.
17. [17] Li, D., Li, Z., Luo, R., Deng, J. & Sun, S. (2019) Multi-Pose Facial Expression Recognition Based on Generative Adversarial Network. IEEE Access, 7, 143980-143989.
18. [18] Viola, P., & Jones, M.J. (2004). Robust Real-Time Face Detection. International Journal of Computer Vision, 57(2), 137-154.
19. [19] Du, S., Tao, Y., & Martinez, A.M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15), E1454–E1462.
20. [20] Dhall, A., Murthy, O.R., Goecke, R., Joshi, J., & Gedeon, T. (2015). Video and image based emotion recognition challenges in the wild: Emotiw 2015. International Conference on Multimodal Interaction, 423–426.
21. [21] Adobe Stock, https://stock.adobe.com/, (Erişim tarihi: 08.05.2019).
22. [22] Aydilek, İ.B. (2017). Approximate estimation of the nutritions of consumed food by deep learning. International Conference on Computer Science and Engineering (UBMK), 160-164.
23. [23] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition, 1–9.
24. [24] He, H., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
25. [25] Savoiu, A., & Wong, J. (2017). Recognizing Facial Expressions Using Deep Learning.