FACIAL EXPRESSION CLASSIFICATION WITH HAAR FEATURES, GEOMETRIC FEATURES AND CUBIC BÉZIER CURVES

Abstract

Facial expressions are nonverbal communication channels to interact with other people.  Computer recognition of human emotions based on facial expression is  an interesting and difficult problem. In this study, images were analyzed based on facial expressions and tried to identify different emotions, such as smile, surprise, sadness, fear, disgust, anger and  neutral. In practice, it was used Viola-Jones face detector  used  AdaBoost  algorithm for  finding the location of the face. Haar filters were used in finding the eyes and mouth. In cases where erroneous detection of the mouth and eyes, facial geometric ratios were used. Cubic Bézier curves were used in determining emotion. FEEDTUM facial expression database were used for training and testing. The seven different emotions used for the study, the recognition success rates ranged from 97%  to 60%.

Keywords

• Emotion Recognition, Facial Expression Recognition, AdaBoost, Haar Features, Bézier curves.

-

Abstract

Yüz ifadeleri insanlar arası etkileşimde sözsüz iletişim kanallarıdır. Bilgisayarla insanın yüz ifadesine dayalı duyguları tanıma ilginç ve zor bir problemdir. Bu çalışmada resimlerdeki yüz ifadelerinden gülümseme, şaşkınlık, üzgünlük, korkma, iğrenme, kızgın ve nötr gibi farklı duygular tespit edilmeye çalışılmıştır. Uygulamada yüz yerinin bulunmasında AdaBoost algoritmasını kullanan Viola-Jones yüz detektöründen yararlanılmıştır. Gözlerin ve ağzın bulunmasında haar filtreleri kullanılmıştır. Ağız ve gözlerin hatalı tespit edildiği durumlarda, yüzdeki geometrik oranlarından faydalanılmıştır. Duygu tespitinde Kübik Bézier eğrileri kullanılmıştır. Eğitim ve test için FEEDTUM yüz ifadesi veritabanından yararlanılmıştır. Çalışma için belirlenen yedi farklı duygunun, tanıma başarı oranları %97 ile %60 arasında değişmektedir.

Keywords

• Duygu tanıma, Yüz ifadesi tanıma, AdaBoost, Haar özallikleri, Bézier eğrileri

References

1. Ekman, P., “About brows: emotional and conversational signals. In: Human ethology: claims and limits of a new discipline: contributions to the Colloquium”, pp. 169– 2 Cambridge University Press, England,1979
2. B. Fasel and J. Luettin, “Automatic facial expression analysis: a survey,” Pattern Recognition, vol. 1, no. 30, pp. 259–275, 2003.
3. Mehrabian A., “Communication without Words”, Psychology Today, vol. 2,no. 4, pp. 53-56, 1968.
4. L. Ma and K. Khorasani, “Facial Expression Recognition Using Constructive Feedforward and Neural Networks”, IEEE transactions on systems, man and cybernetics- part B: Cybernetics, vol.34, No. 3, June 2004.
5. Ekman, P., Rosenberg, E.L., “What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS)”, Oxford University Press, US, 2005
6. Russell, J.A., Fernez-Dols, J.M. (eds.), “The Psychology of Facial Expression”, Cambridge University Press, Cambridge, 1997
7. Darwin, C., “ The expression of the emotions in man and animals”, University of Chicago Press, Chicago, 1965
8. Faigin, G., “The Artist’s Complete Guide to Facial Expression”, Watson-Guptill, 2008

9. Bartlett M, Littlewort G, Frank M, et al., “ Automatic recognition of facial actions in spontaneous expressions [J]”, Journal of Multimedia, 1(6): 22–35, 2006.
10. R. Lienhart, J. Maydt. “An extended set of haar-like features for rapid object detection”, Proceedings of the IEEE International Conference on Image Processing, Rochester, New York, vol. 1,pp. 900-903, 2002.
11. Ryan, A., et al.,”Automated Facial Expression Recognition System”, Proc. Int. Carnahan Conf. on Security Technology, 2009, p. 172-177.
12. Ashraf, A.B., et al., “The painful face-Pain expression recognition using active appearance models”, Image and Vision Computing, 2009. 27(12): p. 1788-1796.
13. Vural, E., et al., “Automated Drowsiness Detection For Improved Driving Safety”, in Proc. of the Int. Conf. on Automotive Technologies ,2008.
14. P. Ekman, D. Matsumoto, and W.V. Friesen, “Facial Expression in Affective Disorders, What the Face Reveals”, P. Ekman and E.L. Rosenberg, eds., pp. 429-439, 2005.
15. OpenCv,http://www.cognotics.com/opencv/servo_2007_ series/part_2/sidebar.html#fig_1(Access: 13.07.2012)
16. Jones, M. ve Viola P., “Rapid Object Detection using a Boosted Cascade of Simple Features”, 2001
17. P. Viola and M. J. Jones, “Robust real-time object detection”, International Journal of Computer Vision, Vol. 57, No. 2, p.137–154, 2004.
18. Cohen, I., N. Sebe, F. G. Gozman, M. C. Cirelo, T. S. Huang, “Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I-595 - I-601, 2003
19. Khan, M., İ., Bhuiyan A., “Facial Features Approximation for Expression Detection in Human-Robot Interface”, Sixth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, 2010
20. Y.Wang, H.Ai, B.Wu, and C.Hung, “Real time facial expression recognition with adabost”, Proceengs of the 17 th International Conference on Pattern Recognition, vol 3, pp 926-929,2004
21. J.Whitehil, C.W.Omlin, “Haar Features for FACS AU Recognition”, Proceengs of the 7 th International Conference on Automatic Face and Gasture Recognition, 2006 http://www.mmk.ei.tum.de/~waf/fgnet/feedtum.pdf (Access: 02012)
22. S. U. Jung, D.H. Kim, K. H. An, M. J. Chung, “Efficient rectangle feature extraction for real-time facial expression recognition based on adaboost”, IEEE/RSJ International Conference on Intelligent Robotics and Systems (IROS), 200 C.Shan, S.Gong, P.W. McOwan, “Facial expression recognition based on Local Binary Patterns:A comprehensive study”, Image and Vision Computing 27 (2009) 803–816
23. N.Sarode, S.Bhatia, “Facial Expression Recognition”, (IJCSE) International Journal on Computer Science and Engineering ,Vol. 02, No. 05, 2010, 1552-1557
24. C. Papageorgiu, M. Oren, T. Poggio, “A general framework for Object Detection”, Proceedings of the International Conference on Computer Vision, Bombay, India, pp. 555562, 1998.
25. Kurt, B., “Bilgisayar ile psikolojik durum değerlendirmesi”, Yüksek Lisans Tezi, KTÜ Fen Bilimleri Enstitüsü Bilgisayar Mühendisliği Anabilim Dalı, 2007
26. G.Özmen, R.Kandemir, “Haar Dalgacıkları ve Kübik Bezier Eğrileri İle Yüz İfadesi Tespiti”, ELECO’2012, Bursa/Türkiye, 2012
27. Y. Tian, T. Kanade, J. Cohn, “Recognizing action units for facial expression analysis”, IEEE Transactions on Pattern Analysis and Machine Intelligence 23 (2) (2001) 97–
28. M. Pantic, L.J.M. Rothkrantz, “Facial action recognition for facial expression analysis from static face images”, IEEE Transactions on Systems, Man, and Cybernetics 34 (3) (2004) 1449–1461.
29. Z. Zhang, M.J. Lyons, M. Schuster, S. Akamatsu, “Comparison between geometry-based and Gaborwavelets-based facial expression recognition using multilayer perceptron”, in: IEEE International Conference on Automatic Face & Gesture Recognition (FG), 1998.
30. Z. Zeng, Y. Fu, G. I. Roisman, Z. Wen, Y. Hu and T. S. Huang, “Spontaneous Emotional Facial Expression Detection”, Journal of Multimedia, Vol. 1, No. 5, p. 1-8, 200 G.C. Littlewort, M.S. Bartlett, J. Chenu, I. Fasel, T. Kanda, H. Ishiguro, J.R. Movellan, “Towards social robots: Automatic evaluation of human-robot interaction by face detection and expression classification”, Advances in Neural Information Processing Systems, Vol 16, p. 1563-1570, 200
31. P. Visutsak, “Emotion Classification using Adaptive SVMs”, World Academy of Science, Engineering and Technology 63, 2012
32. K. Mase, “Recognition of facial expression from optical flow”, IEICE Transactions, pp. 3474-3483, 1991
33. J.F. Cohn, A.J. Zlochower, J.J. Lien, and T. Kanade, “Feature-Point Tracking by Optical Flow Discriminates Subtle Differences in Facial Expression”, Proc. Int'l Conf. Automatic Face and Gesture Recognition, p. 396401, 1998
34. I. Essa and A. Pentland, “Coding, Analysis Interpretation, Recognition of Facial Expressions”, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol.19, No. 7, p. 757-763, July 1997 Rembiye KANDEMİR, received the Ph.D. degree in Computer Engineering from the University of Trakya, Turkey. She is currently an assistant professor in the Department of Computer Engineering, University of Trakya. Her research interests include image analysis, computer vision and machine learning applied to face and body gesture recognition, multimodal human-computer interaction HCI). Gonca ÖZMEN, received B.S. degree in Computer Engineering Department at Kocaeli University, Turkey, and M.S. degree in Computer Engineering Department at Trakya University, Turkey. She is currently working in Computer Programming Department at Technical Science Vocational School, Kırklareli
35. University, Turkey, as a lecturer.