Derin Öğrenme Modeli ile Yüz İfadelerinden Duygu Tanıma

Abstract

Yüz ifadesinden duygu tanıma; insan-bilgisayar etkileşimi, duygusal hesaplama vb. gibi birçok bilgisayarla görme alanında uygulanabilen güncel bir araştırma konusudur. Bu çalışmada, KDEF ve PICS veri setleri kullanılarak derin öğrenme ile duygu tanımaya yönelik bir uygulama yapılmıştır. Öznitelik çıkarımı için derin öğrenme tekniklerinden olan ve yapay sinir ağları içeren bir yapay zekâ yaklaşımı olan Evrişimsel Sinir Ağı (ESA) kullanılarak yeni bir model geliştirilmiştir. Derin öğrenmenin yüksek başarımı için büyük veri setine ihtiyaç duyulmaktadır. KDEF veri setinde 4900, PICS veri setinde 322 görüntü bulunmaktadır. Bu amaçla öncelikle PICS veri setindeki görüntü sayısının az olmasından dolayı veri artırma yöntemi ile görüntü çoğaltma işlemi uygulanmıştır ve PICS veri seti 4830 görüntüye çıkarılmıştır. Daha sonra bu iki farklı veri seti üzerinde ayrı ayrı eğitim gerçekleştirilerek geliştirilen yeni model test edilmiştir. ESA modellerinden olan VGGNet temel alınarak geliştirilen yeni model ile gerçekleştirilen çalışmada, her bir veri setinde yedi farklı duygu sınıfı (korku, öfke, iğrenme, mutluluk, nötr, üzüntü, şaşırma) ele alınmıştır. Geliştirilen model ile KDEF veri setinin geçerleme kümesinde %97.44, PICS veri setinin geçerleme kümesinde %98.24 doğruluk değerleri elde edilerek yüksek bir başarı oranına ulaşılmıştır.

Keywords

• Derin öğrenme
• evrişimsel sinir ağı
• duygu analizi

Emotion Recognition from Facial Expressions by Deep Learning Model

Abstract

Emotion recognition from facial expression is a current research topic that can be applied in the many fields of computer vision, such as human-computer interaction, emotional computing, etc. In this study, an application for emotion recognition through deep learning was made using KDEF and PICS datasets. A new model was established using the convolutional neural network (CNN), an artificial intelligence approach that involves artificial neural networks, which is one of the deep learning techniques for attribute inference. Large datasets are needed for the high performance of deep learning. There are 4900 images in the KDEF dataset and 322 images in the PICS dataset. For this purpose, primarily due to the small number of images in the PICS dataset, image iteration was applied with the data augmentation method, and the PICS dataset was increased to 4830 images. Then, the new model developed by conducting separate training on these two different datasets was tested. Seven different classes of emotion (afraid, angry, disgusted, happy, neutral, sad, surprised) were covered in each dataset in the study conducted with a new model developed based on VGGNet which is one of the CNN models. With the developed model, a high success rate was achieved by obtaining 97.44% accuracy values in the validation set of the KDEF and 98.24% accuracy values in the validation set of the PICS dataset.

Keywords

• Deep learning
• convolutional neural network
• emotion recognition

References

1. Ahmad F, Najam A, Ahmed Z, 2013. Image-based Face Detection and Recognition: “State of the Art.”. ArXiv Preprint ArXiv: 1302.6379.
2. Altan G, 2019. DeepGraphNet: Grafiklerin Sınıflandırılmasında Derin Öğrenme Modelleri. European Journal of Science and Technology, October: 319–329. https://doi.org/10.31590/ejosat.638256
3. Calvo MG, Lundqvist D, 2008. Facial expressions of emotion (KDEF): Identification under different display-duration conditions. Behavior Research Methods, 40(1): 109–115. https://doi.org/10.3758/BRM.40.1.109
4. Chen M, Lu Z, Jan PA, 2015. Learning deep features for image emotion classification. 2015 IEEE International Conference on Image Processing (ICIP), 4491–4495.
5. Dandıl E, Özdemir R, 2019. Real-time Facial Emotion Classification Using Deep Learning. DATA SCIENCE AND APPLICATIONS, 2(1): 13–17.
6. Eng SK, Ali H, Cheah AY, Chong YF, 2019. Facial expression recognition in JAFFE and KDEF Datasets using histogram of oriented gradients and support vector machine. IOP Conference Series: Materials Science and Engineering, 705(1). https://doi.org/10.1088/1757-899X/705/1/012031
7. Fayek HM, Lech M, Cavedon L, 2017. Evaluating deep learning architectures for Speech Emotion Recognition. Neural Networks, 92: 60–68. https://doi.org/10.1016/j.neunet.2017.02.013
8. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J, Chen T, 2018. Recent advances in convolutional neural networks. Pattern Recognition, 77: 354–377. https://doi.org/10.1016/j.patcog.2017.10.013

9. Hancock P, 2008. Psychological image collection at stirling (pics). Web Address: Http://Pics. Psych. Stir. Ac. Uk.
10. He K, Zhang X, Ren S, Sun J, 2016. Deep Residual Learning for Image Recognition Kaiming. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1002/chin.200650130
11. Hossain MS, Muhammad G, 2019. Emotion recognition using deep learning approach from audio–visual emotional big data. Information Fusion, 49: 69–78. https://doi.org/10.1016/j.inffus.2018.09.008
12. Huang Y, Chen F, Lv S, Wang X, 2019. Facial expression recognition: A survey. Symmetry, 11(10). https://doi.org/10.3390/sym11101189
13. Irtija N, Sami M, Ahad MAR, 2018. Fatigue detection using facial landmarks. Joint Conference ISASE-MAICS 2018 - 4th International Symposium on Affective Science and Engineering 2018, and the 29th Modern Artificial Intelligence and Cognitive Science Conference, 1–6. https://doi.org/10.5057/isase.2018-c000041
14. Koç M, Özdemir R, 2019. Yeni Bir Veri Kümesi (RidNet) Kullanarak Kontrolsüz Ortamda Yüz İfadesi Tanımanın Derin Öğrenme Yöntemleri ile İyileştirilmesi. Bilecik Şeyh Edebali Üniversitesi Fen Bilimleri Dergisi, 6(2): 384–396. https://doi.org/10.35193/bseufbd.645138
15. Krizhevsky A, Sutskever I, Hinton GE, 2012. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, 1097–1105. https://doi.org/10.1201/9781420010749
16. Lecun Y, Bengio Y, Hinton G, 2015. Deep learning. Nature, 521(7553): 436–444. https://doi.org/10.1038/nature14539
17. LeCun Y, Bottou L, Bengio Y, Haffner P, 1998. Gradient-based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11): 2278–2323. https://doi.org/10.1109/5.726791
18. Li J, Jin K, Zhou D, Kubota N, Ju Z, 2020. Attention mechanism-based CNN for facial expression recognition. Neurocomputing, 411: 340–350. https://doi.org/10.1016/j.neucom.2020.06.014
19. Li S, Deng W, 2020. Deep Facial Expression Recognition: A Survey. IEEE Transactions on Affective Computing, 3045(c): 1–20. https://doi.org/10.1109/TAFFC.2020.2981446
20. Lu Y, Wang S, Zhao W, 2019. Facial expression recognition based on discrete separable shearlet transform and feature selection. Algorithms, 12(1). https://doi.org/10.3390/a12010011
21. Lundqvist D, Flykt A, Öhman A, 1998. The Karolinska directed emotional faces (KDEF). CD ROM from Department of Clinical Neuroscience, Psychology Section, Karolinska Institutet 91(630): 2–2.
22. Mehta D, Siddiqui MFH, Javaid AY, 2019. Recognition of emotion intensities using machine learning algorithms: A comparative study. Sensors (Switzerland), 19(8): 1–24. https://doi.org/10.3390/s19081897
23. Özdemir, D, Karaman, S, 2017. Investigating interactions between students with mild mental retardation and humanoid robot in terms of feedback types. Egitim ve Bilim, 42(191).
24. Ruiz-Garcia A, Elshaw M, Altahhan A, Palade V, 2016. Deep learning for emotion recognition in faces. International Conference on Artificial Neural Networks Springer, 38–46. https://doi.org/10.1007/978-3-319-44781-0_5
25. Shorten C, Khoshgoftaar TM, 2019. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1). https://doi.org/10.1186/s40537-019-0197-0
26. Simonyan K, Zisserman A, 2014. Very deep convolutional networks for large-scale image recognition. ArXiv Preprint ArXiv: 1409.1556.
27. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A, 2015. Going Deeper with Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1002/jctb.4820
28. Xu M, Cheng W, Zhao Q, Ma L, Xu F, 2015. Facial expression recognition based on transfer learning from deep convolutional networks. 2015 11th International Conference on Natural Computation (ICNC). IEEE, 702–708. https://doi.org/10.1109/ICNC.2015.7378076