AR and ML Solutions for Facial Expression Recognition on Mobile Phones

Abstract

The widespread use of smartphones has led to the development of a diverse range of personal and business applications. Especially, ever-improving computational power of smartphones has made real-time video processing operations feasible on mobile and portable devices. Consequently, real-time facial expression recognition (FER) on mobile phones has emerged as an efficient approach, despite certain limitations imposed by hardware and power storage limitations. An example of such implementation is the ubiquitous "Face Unlock" feature utilized in numerous smartphones and applications, including online banking. Facial expression recognition that only relies on Machine Learning (ML) faces challenges; such as the requirement for substantial datasets and expertise-level training to achieve accurate models. Accordingly, this study explores an alternative approach which involves Augmented Reality (AR) to achieve expression recognition. To achieve this goal, this study undertakes a comprehensive comparison of these two distinct techniques through the development of separate mobile applications for facial expression recognition. Experimental results indicate that the ML-based architecture achieves the highest classification accuracy of 91.63%, outperforming the AR-based approach (81.63%). However, from an operational efficiency viewpoint, AR-based approach demonstrates superior performance with an average inference latency of 37.8 ms, operating approximately 7.6 times faster than ML application (288.2 ms). Furthermore, environmental stress testing reveals that while RGB-based ML models suffer performance degradation in low light, the AR system maintains consistent tracking in total darkness due to active infrared depth sensing. The findings shed light on the advantages and limitations of each method, contributing to the advancement of FER technology in mobile applications.

Keywords

• ARKit
• augmented reality
• facial expression recognition
• machine learning
• mobile

Ethical Statement

There are no ethical issues after the publication of this manuscript.

References

1. [1]. Zhan, C., Li, W., Ogunbona, P. O., & Safaei, F. (2006). Facial expression recognition for multiplayer online games.
2. [2]. Tang, X. Y., Peng, W. Y., Liu, S. R., & Xiong, J. W. (2020, February). Classroom teaching evaluation based on facial expression recognition. In Proceedings of the 2020 9th International Conference on Educational and Information Technology (pp. 62-67).
3. [3]. Yolcu, G., Oztel, I., Kazan, S., Oz, C., Palaniappan, K., Lever, T. E., & Bunyak, F. (2017, November). Deep learning-based facial expression recognition for monitoring neurological disorders. In 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (pp. 1652-1657). IEEE.
4. [4]. Acquisti, A., Gross, R., & Stutzman, F. D. (2014). Face recognition and privacy in the age of augmented reality. Journal of Privacy and Confidentiality, 6(2), 1.
5. [5]. Alkawaz, M. H., Waili, T., & Adnan, S. M. (2020). Augmented reality for patient information using face recognition and cloud computing. International Journal on Perceptive and Cognitive Computing, 6(1), 24-27.
6. [6]. Golnari, A., Khosravi, H., & Sanei, S. (2020, February). Deepfacear: deep face recognition and displaying personal information via augmented reality. In 2020 International Conference on Machine Vision and Image Processing (MVIP) (pp. 1-7). IEEE.
7. [7]. Colmenarez, A., Frey, B., & Huang, T. S. (1999, June). A probabilistic framework for embedded face and facial expression recognition. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149) (Vol. 1, pp. 592-597). IEEE.
8. [8]. Li, Y., Wang, S., Zhao, Y., & Ji, Q. (2013). Simultaneous facial feature tracking and facial expression recognition. IEEE Transactions on image processing, 22(7), 2559-2573.

9. [9]. Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003). Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and image understanding, 91(1-2), 160-187.
10. [10]. Tong, Y., Chen, J., & Ji, Q. (2008). A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE transactions on pattern analysis and machine intelligence, 32(2), 258-273.
11. [11]. Liu, K. C., Hsu, C. C., Wang, W. Y., & Chiang, H. H. (2019, July). Real-Time facial expression recognition based on cnn. In 2019 International Conference on System Science and Engineering (ICSSE) (pp. 120-123). IEEE.
12. [12]. Devries, T., Biswaranjan, K., & Taylor, G. W. (2014, May). Multi-task learning of facial landmarks and expression. In 2014 Canadian conference on computer and robot vision (pp. 98-103). IEEE.
13. [13]. Puthanidam, R. V., & Moh, T. S. (2018, January). A hybrid approach for facial expression recognition. In Proceedings of the 12th International Conference on Ubiquitous Information Management and Communication (pp. 1-8).
14. [14]. Wang, X., Huang, J., Zhu, J., Yang, M., & Yang, F. (2018, August). Facial expression recognition with deep learning. In Proceedings of the 10th International Conference on Internet Multimedia Computing and Service (pp. 1-4).
15. [15]. Yang, B., Jianming, W., & Hattori, G. (2021, September). Face mask aware robust facial expression recognition during the COVID-19 pandemic. In 2021 IEEE International Conference on Image Processing (ICIP) (pp. 240-244). IEEE.
16. [16]. Yang, B., Wu, J., & Hattori, G. (2020, November). Facial expression recognition with the advent of face masks. In 19th International Conference on Mobile and Ubiquitous Multimedia (pp. 335-337).
17. [17]. Sarode, N., & Bhatia, S. (2010). Facial expression recognition. International Journal on computer science and Engineering, 2(5), 1552-1557.
18. [18]. Benedict, S. R., & Kumar, J. S. (2016, October). Geometric shaped facial feature extraction for face recognition. In 2016 IEEE International Conference on Advances in Computer Applications (ICACA) (pp. 275-278). IEEE.
19. [19]. Lemaire, P., Ben Amor, B., Ardabilian, M., Chen, L., & Daoudi, M. (2011, December). Fully automatic 3D facial expression recognition using a region-based approach. In Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding (pp. 53-58).
20. [20]. Ramanathan, S., Kassim, A., Venkatesh, Y. V., & Wah, W. S. (2006, October). Human facial expression recognition using a 3D morphable model. In 2006 International conference on image processing (pp. 661-664). IEEE.
21. [21]. Sarvadevabhatla, R. K., Benovoy, M., Musallam, S., & Ng-Thow-Hing, V. (2011, November). Adaptive facial expression recognition using inter-modal top-down context. In Proceedings of the 13th international conference on multimodal interfaces (pp. 27-34).
22. [22]. Hsu, R. L., & Jain, A. K. (2001, October). Face modeling for recognition. In Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205) (Vol. 2, pp. 693-696). IEEE.
23. [23]. Park, B. H., & Oh, S. Y. (2015, October). Facial Expression Training System using Bilinear Shape Model. In Proceedings of the 3rd International Conference on Human-Agent Interaction (pp. 239-241).
24. [24]. Alshamsi, H., Meng, H., & Li, M. (2016, August). Real time facial expression recognition app development on mobile phones. In 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD) (pp. 1750-1755). IEEE.
25. [25]. Lucey, Patrick, et al. "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression." 2010 ieee computer society conference on computer vision and pattern recognition-workshops. IEEE, 2010.
26. [26]. Yoon, C., & Kim, D. (2020, October). Mobile Convolutional Neural Networks for Facial Expression Recognition. In 2020 International Conference on Information and Communication Technology Convergence (ICTC) (pp. 1315-1317). IEEE.
27. [27]. LI, Shan; DENG, Weihong; DU, JunPing. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 2852-2861
28. [28]. OUFQIR, Zainab; EL ABDERRAHMANI, Abdellatif; SATORI, Khalid. ARKit and ARCore in serve to augmented reality. In: 2020 international conference on intelligent systems and computer vision (ISCV). IEEE, 2020. p. 1-7
29. [29]. WANG, Chien-Yao; BOCHKOVSKIY, Alexey; LIAO, Hong-Yuan Mark. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. p. 7464-7475.
30. [30]. Lyons, Michael, Miyuki Kamachi, and Jiro Gyoba. "The Japanese female facial expression (JAFFE) dataset." (No Title) (1998).
31. [31]. Khaireddin, Yousif, and Zhuofa Chen. "Facial emotion recognition: State of the art performance on FER2013." arXiv preprint arXiv:2105.03588 (2021).
32. [32]. Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2011). Static facial expression analysis in the wild: An evaluation of the SFEW database. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011) (pp. 819-824). IEEE.
33. [33]. Ekman, Paul, and Wallace V. Friesen. "Facial action coding system." Environmental Psychology & Nonverbal Behavior(1978).