Recognition of Microexpressions Using Xception Deep Learning Model and Gabor Filters with RFECV-SVM Algorithm

Year 2023, Volume: 13 Issue: 4, 2339 - 2352, 01.12.2023

Mehmet Zahit Uzun , Erdal Başaran , Yuksel Celık

https://doi.org/10.21597/jist.1252556

Abstract

Micro Expression (ME) is the leakage that occurs when people try to mask their involuntary and uncontrolled emotional response to an event in a risky environment. Because the person experiencing the emotion at risk tries to suppress it, its reflection on the face occurs in a low intensity, a specific region, and a very short time. Since the expression emerges involuntarily, it is not fake but completely natural. Thanks to the correct detection of these natural expressions, it can be used effectively in many fields such as forensics, clinical, and education. This study used preprocessing, feature extraction, feature selection, and classification tasks in the framework created for the ME recognition target. CASME-II, one of the literature's most widely used publicly available ME datasets, was used in the proposed framework. In the preprocessing stage, onset and apex Frames are taken from the image sequence of each video clip to be used in optical flow algorithms. These two frames obtained horizontal and vertical optical flow images of Farneback, TV-L1 Dual, and TV-L1. Then the features of these optical flow images were obtained using the convolutional neural network (CNN) model Xception and the traditional Gabor model. Recursive feature elimination with a cross-validation (RFECV) feature selection algorithm was used to filter the distinctive ones of these features. Finally, the SVC Linear classifier divided the filtered ME features into three classes: positive, negative, and surprise. The results obtained from the proposed ME framework showed an accuracy rate of 0.9248.

Keywords

Micro-expression, optical flow, xception, gabor, RFECV, SVM

Xception Derin Öğrenme Modeli ve Gabor Filtreleri ile ÇDÖÖE-DVM Algoritması Kullanılarak Mikro İfadelerin Tanınması

Abstract

Mikro ifade (Mİ), insanların riskli bir ortamda bir olaya karşı istemsiz ve kontrolsüz duygusal tepkilerini gizlemeye çalıştıklarında ortaya çıkan sızıntıdır. Duyguyu yaşayan kişi risk altında bunu bastırmaya çalıştığı için yüze yansıması düşük yoğunlukta, belirli bir bölgede ve çok kısa sürede gerçekleşir. İfade istemsizce ortaya çıktığı için sahte değil tamamen doğal olmaktadır. Bu doğal ifadelerin doğru tespiti sayesinde adli, klinik, eğitim gibi birçok alanda etkili bir şekilde kullanılması sağlanabilir. Bu çalışmada Mİ tanıma hedefi için oluşturulan model yapısında sırasıyla önişleme, öznitelik çıkarma, öznitelik seçme ve sınıflandırma görevleri kullanılmıştır. Önerilen model yapısında literatürde en çok kullanılan, kamuya açık Mİ veri setlerinden CASME-II kullanılmıştır. Ön işleme aşamasında Optik Akış algoritmalarında kullanılmak üzere her bir video klipin görüntü dizisinden başlangıç (onset) ve tepe (apex) kareleri seçilir. Bu iki kare kullanılarak Farneback, TV-L1 Dual ve TV-L1 e ait yatay ve dikey optik akış görüntüleri elde edilmiş, ardından bu optik akış görüntüleri evrişimsel sinir ağı (ESA) modeli olan Xception ve geleneksel model olan Gabor modelleri kullanılarak görüntülere ait öznitelikler elde edilmiştir. Elde edilen bu özniteliklere ait ayırt edici olanları filtrelemek için çapraz doğrulama ile özyinelemeli özellik eleme (ÇDÖÖE) öznitelik seçim algoritması kullanılmıştır. Son olarak doğrusal destek vektör sınıflandırıcısı (DVS), filtrelenmiş Mİ özniteliklerini pozitif, negatif ve sürpriz olmak üzere üç sınıfa ayırmıştır. Önerilen Mİ model yapısından elde edilen sonuçlar 0.9248 doğruluk oranı başarısı göstermiştir.

Keywords

Mikro ifade, Optik akış, Xception, Gabor, ÇDÖÖE, DVM

References

• Adegun, I. P., & Vadapalli, H. B. (2020). Facial micro-expression recognition: A machine learning approach. Scientific African, 8, e00465. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.sciaf.2020.e00465
• Ahadit, A. B., & Jatoth, R. K. (2022). A novel multi-feature fusion deep neural network using HOG and VGG-Face for facial expression classification. Machine Vision and Applications, 33(4), 55. Tarihinde adresinden erişildi https://doi.org/10.1007/s00138-022-01304-y
• Allaert, B, Ward, I. R., Bilasco, I. M., Djeraba, C., & Bennamoun, M. (2022). A comparative study on optical flow for facial expression analysis. Neurocomputing, 500, 434–448. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.neucom.2022.05.077
• Allaert, Benjamin, Ward, I. R., Bilasco, I.-M., Djeraba, C., & Bennamoun, M. (2019). Optical flow techniques for facial expression analysis: Performance evaluation and improvements.
• Başaran, E., Cömert, Z., & Çelik, Y. (2020). Convolutional neural network approach for automatic tympanic membrane detection and classification. Biomedical Signal Processing and Control, 56, 101734.
• Basaran, E., Cömert, Z., Çelik, Y., Budak, Ü., & Sengür, A. (2020). Otitis media diagnosis model for tympanic membrane images processed in two-stage processing blocks. IOP Sci, 14, 1–27.
• Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., & Liu, Y.-J. (2021). Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1. Tarihinde adresinden erişildi https://doi.org/10.1109/TPAMI.2021.3067464
• Bozkurt, F. Skin lesion classification on dermatoscopic images using effective data augmentation and pre-trained deep learning approach. Multimed Tools Appl 82, 18985–19003 (2023).
• Cai, L., Li, H., Dong, W., & Fang, H. (2022). Micro-expression recognition using 3D DenseNet fused Squeeze-and-Excitation Networks. Applied Soft Computing, 119, 108594. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.asoc.2022.108594
• Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. Içinde Proceedings of the IEEE conference on computer vision and pattern recognition (ss. 1251–1258).
• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. Tarihinde adresinden erişildi https://doi.org/10.1007/BF00994018
• Fan, L., He, J., Zheng ,Y., Nie, Y., Chen, T., & Zhang, H., “Facial micro-expression recognition impairment and its relationship with social anxiety in internet gaming disorder”, Curr. Psychol., 2022, doi: 10.1007/s12144-022-02958-7.
• Farnebäck, G. (2003). Two-frame motion estimation based on polynomial expansion. Içinde Scandinavian conference on Image analysis (ss. 363–370). Springer.
• Gan, Y S, See, J., Khor, H.-Q., Liu, K.-H., & Liong, S.-T. (2022). Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”. Neurocomputing, 503, 283–298. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.neucom.2022.06.101
• Gan, Yee Siang, Liong, S.-T., Yau, W.-C., Huang, Y.-C., & Tan, L.-K. (2019). OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication, 74, 129–139.
• Gao, T., Zhao, X. M., Chen, T., Liu, Z. W., & Ni, C. (2017). Face description based on adaptive local weighted Gabor comprehensive histogram feature. Multimedia Tools and Applications, 76(10), 12893–12916. Tarihinde adresinden erişildi https://doi.org/10.1007/s11042-016-3701-y
• Hurley, C. M., Anker, A. E., Frank, M. G., Matsumoto, D., & Hwang, H. C. (2014). Background factors predicting accuracy and improvement in micro expression recognition. Motivation and emotion, 38(5), 700–714.
• Jirik, M., Ryba, T., & Zelezny, M. (2011). Texture based segmentation using graph cut and Gabor filters. Pattern Recognition and Image Analysis, 21, 258–261.
• Karcioğlu, A. A., & Aydin, T. (2019, April). Sentiment analysis of Turkish and english twitter feeds using Word2Vec model. In 2019 27th Signal Processing and Communications Applications Conference (SIU) (pp. 1-4). IEEE
• Lee, Y.-C., & Chen, C.-H. (2009). Feature Extraction for Face Recognition Based on Gabor Filters and Two-Dimensional Locality Preserving Projections. Içinde 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (ss. 106–109). Tarihinde adresinden erişildi https://doi.org/10.1109/IIH-MSP.2009.210
• Li, Y., Huang, X., & Zhao, G. (2020). Joint Local and Global Information Learning With Single Apex Frame Detection for Micro-Expression Recognition. IEEE Transactions on Image Processing, 30, 249–263.
• Lin, C., Long, F., Huang, J., & Li, J. (2018). Micro-Expression Recognition Based on Spatiotemporal Gabor Filters. Içinde 2018 Eighth International Conference on Information Science and Technology (ICIST) (ss. 487–491). Tarihinde adresinden erişildi https://doi.org/10.1109/ICIST.2018.8426088
• Liong, S.-T., Gan, Y. S., See, J., Khor, H.-Q., & Huang, Y.-C. (2019). Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. Içinde 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (ss. 1–5). IEEE.
• Liu, K.-H., Jin, Q.-S., Xu, H.-C., Gan, Y.-S., & Liong, S.-T. (2021). Micro-expression recognition using advanced genetic algorithm. Signal Processing: Image Communication, 93, 116153. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.image.2021.116153
• Liu, N., Liu, X., Zhang, Z., Xu, X., & Chen, T. (2020). Offset or Onset Frame: A Multi-Stream Convolutional Neural Network with CapsuleNet Module for Micro-expression Recognition. Içinde 2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) (ss. 236–240). IEEE.
• Liu, Y., Du, H., Zheng, L., & Gedeon, T. (2019). A neural micro-expression recognizer. Içinde 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019) (ss. 1–4). IEEE.
• Mustaqim, A. Z., Adi, S., Pristyanto, Y., & Astuti, Y. (2021). The Effect of Recursive Feature Elimination with Cross-Validation (RFECV) Feature Selection Algorithm toward Classifier Performance on Credit Card Fraud Detection. Içinde 2021 International Conference on Artificial Intelligence and Computer Science Technology (ICAICST) (ss. 270–275). Tarihinde adresinden erişildi https://doi.org/10.1109/ICAICST53116.2021.9497842
• Ou, J., Bai, X.-B., Pei, Y., Ma, L., & Liu, W. (2010). Automatic Facial Expression Recognition Using Gabor Filter and Expression Analysis. Içinde 2010 Second International Conference on Computer Modeling and Simulation (C. 2, ss. 215–218). Tarihinde adresinden erişildi https://doi.org/10.1109/ICCMS.2010.45
• Peng, M., Wang, C., Chen, T., Liu, G., ve Fu, X., “Dual temporal scale convolutional neural network for micro-expression recognition”, Front. Psychol., c. 8, s. 1745, 2017.
• Porter, S., Ten Brinke, L., & Wallace, B. (2012). Secrets and lies: Involuntary leakage in deceptive facial expressions as a function of emotional intensity. Journal of Nonverbal Behavior, 36(1), 23–37.
• Rose, N. (2006). Facial Expression Classification using Gabor and Log-Gabor Filters. Içinde 7th International Conference on Automatic Face and Gesture Recognition (FGR06) (ss. 346–350). Tarihinde adresinden erişildi https://doi.org/10.1109/FGR.2006.49
• See, J., Yap, M. H., Li, J., Hong, X., & Wang, S. (2019). MEGC 2019 – The Second Facial Micro-Expressions Grand Challenge. Içinde 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (ss. 1–5). Tarihinde adresinden erişildi https://doi.org/10.1109/FG.2019.8756611
• Stanley, J. T., & Webster, B. A. (2019). A comparison of the effectiveness of two types of deceit detection training methods in older adults. Cognitive Research: Principles and Implications, 4(1), 26. Tarihinde adresinden erişildi https://doi.org/10.1186/s41235-019-0178-z
• Sun, M.-X., Liong, S.-T., Liu, K.-H., & Wu, Q.-Q. (2022). The heterogeneous ensemble of deep forest and deep neural networks for micro-expressions recognition. Applied Intelligence, 52(14), 16621–16639. Tarihinde adresinden erişildi https://doi.org/10.1007/s10489-022-03284-y
• Sun, Z., Hu, Z., Zhao, M., & Li, S. (2020). Multi-scale active patches fusion based on spatiotemporal LBP-TOP for micro-expression recognition. Journal of Visual Communication and Image Representation, 71, 102862. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.jvcir.2020.102862
• Takalkar, M., Xu, M., Wu, Q., & Chaczko, Z. (2018). A survey: facial micro-expression recognition. Multimedia Tools and Applications, 77(15), 19301–19325.
• Takalkar, M., Xu, M., ve Chaczko, Z. “Manifold feature integration for micro-expression recognition”, Multimed. Syst., c. 26, sayı 5, ss. 535–551, 2020, doi: 10.1007/s00530-020-00663-8.
• Tang, J., Li, L., Tang, M., & Xie, J. (2022). A novel micro-expression recognition algorithm using dual-stream combining optical flow and dynamic image convolutional neural networks. Signal, Image and Video Processing. Tarihinde adresinden erişildi https://doi.org/10.1007/s11760-022-02286-0
• Thuseethan, S., Rajasegarar, S., & Yearwood, J. (2022). Deep3DCANN: A Deep 3DCNN-ANN Framework for Spontaneous Micro-expression Recognition. Information Sciences. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.ins.2022.11.113
• Tonkal, Ö., Polat, H., Başaran, E., Cömert, Z., & Kocaoğlu, R. (2021). Machine Learning Approach Equipped with Neighbourhood Component Analysis for DDoS Attack Detection in Software-Defined Networking. Electronics, 10(11), 1227.
• Ukil, A. (2007). Support Vector Machine BT - Intelligent Systems and Signal Processing in Power Engineering. Içinde A. Ukil (Ed.) (ss. 161–226). Berlin, Heidelberg: Springer Berlin Heidelberg. Tarihinde adresinden erişildi https://doi.org/10.1007/978-3-540-73170-2_4
• Uzun, M. Z., Celik, Y., & Basaran, E. (y.y.). Micro-Expression Recognition by Using CNN Features with PSO Algorithm and SVM Methods. learning, 2(3), 5–8, (2022).
• Wang, C., Xiao, Z., & Wu, J. (2019). Functional connectivity-based classification of autism and control using SVM-RFECV on rs-fMRI data. Physica Medica, 65, 99–105. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.ejmp.2019.08.010
• Warren, G., Schertler, E., & Bull, P. (2009). Detecting deception from emotional and unemotional cues. Journal of Nonverbal Behavior, 33(1), 59–69.
• Xia, B., Wang, W., Wang, S., & Chen, E. (2020). Learning from Macro-expression: a Micro-expression Recognition Framework. Içinde Proceedings of the 28th ACM International Conference on Multimedia (ss. 2936–2944).
• Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014). CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PloS one, 9(1), e86041.
• Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., & Fu, X. (2013). How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions. Journal of Nonverbal Behavior, 37(4), 217–230. Tarihinde adresinden erişildi https://doi.org/10.1007/s10919-013-0159-8
• Yap, M. H., See, J., Hong, X., & Wang, S.-J. (2018). Facial Micro-Expressions Grand Challenge 2018 Summary. Içinde 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (ss. 675–678). Tarihinde adresinden erişildi https://doi.org/10.1109/FG.2018.00106
• Zhao, Y., & Xu, J. (2020). Compound Micro-Expression Recognition System. Içinde 2020 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS) (ss. 728–733). Tarihinde adresinden erişildi https://doi.org/10.1109/ICITBS49701.2020.00161
• Zhou, L., Mao, Q., Huang, X., Zhang, F., & Zhang, Z. (2022). Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition. Pattern Recognition, 122, 108275. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.patcog.2021.108275
• Zhou, L., Mao, Q., & Xue, L. (2019). Cross-database micro-expression recognition: a style aggregated and attention transfer approach. Içinde 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) (ss. 102–107). IEEE.
• Zhou, Y., Song, Y., Chen, L., Chen, Y., Ben, X., & Cao, Y. (2022). A novel micro-expression detection algorithm based on BERT and 3DCNN. Image and Vision Computing, 119, 104378. Tarihinde adresinden erişildi https://doi.org/https://doi.org/10.1016/j.imavis.2022.104378