Video dosyaları üzerinde yüz ifade analizi için hızlandırılmış bir yaklaşım

Öz

Yüz ifadelerinin otomatik olarak analiz edilmesi ve sınıflandırılması; insan-bilgisayar etkileşimi, bilgisayarlı görme ve görüntü işleme gibi birçok alanda çalışılan zorlu bir problemdir. Son zamanlarda özellikle insan-bilgisayar etkileşiminde yaşanan gelişmelerle birlikte, insanlara ait duyguların bilgisayarlar tarafından anlaşılması elzem bir konu haline gelmiştir. Bunun yanında psikoloji, güvenlik, sağlık, oyun ve robotik gibi birçok çalışma alanında da yüz ifadelerinin analizine duyulan ihtiyaç giderek artmaktadır. Bu nedenlerden dolayı, yüz ifadelerinin hızlı ve doğru bir şekilde analiz edilmesi farklı uygulama alanlarında birçok yazılım sistemi için kritik bir rol oynamaktadır. Bu çalışmada, video dosyalarının hızlandırılmış yüz ifade analizi için bir yaklaşım önerilmiştir. Yüz ifadeleri mutluluk, normal, şaşkınlık ve üzüntü olmak üzere dört sınıfta ele alınmıştır. Analiz edilen toplam video kare sayısı azaltılarak ve paralel iş parçacıkları kullanılarak hızlandırılan ifade analizinin çok-çekirdekli bilgisayar üzerinde başarım değerlendirmesi sunulmuştur. Deneysel sonuçlar Hyper Threading teknolojisine sahip dört-çekirdekli işlemci kullanılarak elde edilmiştir. İşlemci üzerinde 2 iş parçacığı ile yaklaşık 1.8 kat, 4 iş parçacığı ile ise yaklaşık 2.9 kat hızlandırma elde edilirken; 8 iş parçacığı ile hızlandırma oranı yaklaşık olarak 3.5 kata çıkarılmıştır. Ayrıca istatiksel analiz sonuçları üzerinde hata analizi yapılarak, hatalı olduğu tespit edilen görüntü karelerine ait sonuçlar düzeltilmiştir.

Anahtar Kelimeler

• Çok-Çekirdekli hızlandırma,Haar özellikleri,Kübik Bézier eğrileri,Paralel hesaplama,Yüz ifade analizi

An accelerated approach for facial expression analysis on video files

Abstract

Automatic analysis and classification of facial expressions is a challenging problem which takes the attention of researchers studying in many areas such as human-computer interaction, computer vision and image processing. Currently, especially due to developments in human-computer interaction, the understanding of human emotions by computer has become an indispensable issue. Besides, analysis and recognition of facial expressions has prevailed in various areas like psychology, security, health, entertainment, and robotics. For these reasons, the analyzing of facial expressions quickly and correctly play a critical role for many software systems in different applications. In this study, an approach is proposed for the accelerated facial expression analysis of video files. Facial expressions are considered in four class; happy, normal, confused and sad. By reducing the total number of analyzed video frames and using parallel threads, performance evaluation of the expression analysis accelerated on multi-core computer was presented. Experimental results were obtained using quad-core processor with Hyper Threading technology. According to experimental results, quad core processor using two threads produced about 1.8 fold speed-up and four threads produced about 3  fold speed-up while eight threads produced about 3.5 fold speed-up has been obtained. Additionally, the results of image frames which were found to be incorrect were fixed by performing error analysis on the results of statistical analysis.

Keywords

• Multi-core acceleration,Haar features,Cubic Bézier curves,Parallel computing,Facial expression analysis

References

1. Mehriban A. “Communication without words”. Psychology Today, 2(4), 53-56, 1968.
2. Suwa M, Sugie N, Fujimora K. “A preliminary note on pattern recognition of human emotional expression”. 4th International Joint Conference on Pattern Recognition, Kyoto, Japan, 7-10 November 1978.
3. Murtaza M, Sharif M, Raza M, Shah JH. “Analysis of face recognition under varying facial expression: a survey”. The International Arab Journal of Information Technology, 10(4), 378-388, 2013.
4. Yang MH, Kriegman DJ, Ahuja N. “Detecting faces in images: a survey”. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(1), 34-58, 2002.
5. Tian YL, Kanade T, Cohn JF. Facial Expression Analysis. Editors: Li SZ, Jain AK. Handbook of Face Recognition, East Lansing, Michigan, USA, Springer, 2005.
6. Ekman P, Friesen WV. “Constants across cultures in the face and emotion”. Journal of Personality and Social Psychology, 17(2), 124-129, 1971.
7. Ekman P. “Universals and cultural differences in facial expressions of emotion”. Nebraska Symposium on Motivation, Nebraska, USA, 1972.
8. Akgun D. “A Practical parallel implementation for TDLMS image filter on multi-core processor”. Journal of Real-Time Image Processing, 13(2), 249-260, 2017.

9. Ekman P, Friesen WV. Facial Action Coding System: A Technique for the Measurement of Facial Movement. California, USA, Consulting Psychology Press, 1978.
10. Ekman P. Methods for Measuring Facial Action. Editors: Scherer K, Ekman P. Handbook of Methods in Nonverbal Behavior Research, 45-135, New York, USA, Cambridge University Press, 1982.
11. Yacoob Y, Davis LS. “Recognizing human facial expressions from long image sequences using optical flow”. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(6), 636-642, 1996.
12. Cohen I, Garg A, Huang TS. “Emotion recognition from facial expressions using multilevel HMM”. Neural Information Processing Systems, 2000.
13. Lajevardi SM, Lech M. “Facial expression recognition from image sequences using optimized feature selection”. IEEE 23rd International Conference Image and Vision Computing (IVCNZ), Christchurch, New Zealand, 26-28 November 2008.
14. Khan MI, Bhuiyan A. “Facial expression recognition for human-robot interface”. International Journal of Computer Science and Network Security, 9(4), 300-306, 2009.
15. Özmen G, Kandemir R. “Haar dalgacıkları ve kübik bezier eğrileri ile yüz ifadesi tespiti”. Elektrik-Elektronik ve Bilgisayar Mühendisliği Sempozyumu, Bursa, Türkiye, 29 Kasım-01 Aralık 2012.
16. Rai P, Dixit M. “Smile detection via bezier curve of mouth interest points”. International Journal of Advanced Research in Computer Science and Software Engineering, 3(7), 802-806, 2013.
17. Bao H, Ma T. “Feature extraction and facial expression recognition based on bezier curve”. IEEE International Conference on Computer and Information Technology, Xi'an, 11-13 September 2014.
18. Lopes AT, Aguiar E, Santos TO. “A facial expression recognition system using convolutional networks”. 28th Conference on Graphics, Patterns and Images, Salvador, Brazil, 26-29 September 2015.
19. Uddin Z. “A depth video-based facial expression recognition system utilizing generalized local directional deviation-based binary pattern feature discriminant analysis”. Multimedia Tools and Applications, 75(12), 6871-6886, 2016.
20. Viola P, Jones MJ. “Robust real-time face detection”. International Journal of Computer Vision, 57(2), 137-154, 2004.
21. Özmen G. Kübik Bezier Eğrileri ile Yüz İfadesi Tanıma. Yüksek Lisans Tezi, Trakya Üniversitesi, Edirne, Türkiye, 2012.
22. OpenCV. “OpenCV Library”. http://opencv.org (02.03.2016).
23. EmguCV. “EmguCV: OpenCV in.NET”. http://www.emgu.com (02.03.2016).
24. Cascade Classification. “Haar Özellikleri ve Adoboost”. http://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html (02.03.2016).
25. Castrillón M, Déniz O, Guerra C, Hernández M. “ENCARA2: real-time detection of multiple faces at different resolutions in video streams”. Journal of Visual Communication and Image Representation, 18(2), 130-140, 2007.
26. Castrillón M, Déniz O, Lorenzo J, Hernández D. “Using incremental principal component analysis to learn a gender classifier automatically”. Proceedings of the First Spanish Workshop on Biometrics, Girona, Spain, 2007.
27. Castrillón-Santana M, Déniz-Suárez O, Antón-Canalís L, Lorenzo-Navarro J. “Face and facial feature detection evaluation”. 3rd International Conference on Computer Vision Theory and Applications, Funchal, Madeira, Portugal, 22-25 January 2008.
28. Andersson F. Bezier and B-Spline Technology, Scientific Report, 58, 2003.
29. Joy KI. “Bernstein Polynomials”. University of California, Davis, On-Line Geometric Modeling Notes, 13, 2000.
30. Bayrakdar S. Video Üzerinde Yüz İfadelerinin Hızlandırılmış İstatiksel Analizi için Yeni Bir Algoritma. Yüksek Lisans Tezi, Düzce Üniversitesi, Düzce, Türkiye, 2016.
31. Microsoft Visual Studio. “Visual Studio IDE, Kod Düzenleyicisi, Team Services ve Mobile Center”. https://www.visualstudio.com/tr/ (02.03.2016).
32. Microsoft Visual Studio .Net. “.NET Geliştirme | Visual Studio”. https://www.visualstudio.com/vs/net-development/ (02.03.2016).
33. Zhang Z. “Feature-based facial expression recognition: sensitivity analysis and experiments with a multi-layer perceptron”. International Journal of Pattern Recognition and Artificial Intelligence, 13(6), 893-911, 1999.
34. Ghimire D, Lee J. “Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines”. Sensors, 13(1), 7714-7734, 2013.
35. Martin O, Kotsia I, Macq B, Pitas I. “The eNTERFACE'05 audio-visual emotion database”. 22nd International Conference on Data Engineering Workshops (ICDEW'06), Atlanta, Georgia, 3-7 April 2006.
36. Intel®. “Core™ i7-4700HQ Processor”. http://ark.intel.com/tr/products/75116/Intel-Core-i7-4700HQ-Processor-6M-Cache-up-to-3_40GHz#@specifications (05.03.2016).
37. Intel®. “Core™ i7-4700HQ Processor Architecture”. http://www.intel.com/pressroom/archive/releases/2010/20100330comp (05.03.2016).