Face Warping Deepfake Detection and Localization in a Digital Video using Transfer Learning Approach

Abstract

Generative AI (GenAI) can generate high-resolution and complex content mimicking the creativity of humans, thereby benefiting industries such as gaming, entertainment, and product design. In recent times, AI-generated fake videos, commonly referred to as deepfakes, have become more commonplace and convincing. An additional deepfake technique, face warping, uses digital processing to noticeably distort shapes on a face. Tracking such warping in images and videos is crucial and preventing its use for destructive purposes. A technique is proposed for detecting and localizing face warped areas in video. The input video is extracted to perform various image pre-processing techniques that refine the video into a format that is more likely to classify the classes efficiently. Transfer learning is employed, and the pre-trained model is adopted to train using Convolutional Neural Network (CNN) with the source videos to identify face warping. Based on the experimental results, it was determined that the proposed model detects and localizes the warped areas of the face satisfactorily with an accuracy of 89.25%.

Keywords

• Deepfake
• Face Warping
• Transfer Learning
• Convolutional Neural Network
• Generative Artificial Intelligence

References

1. Chan, C. K. Y., & Zhou, W. (2023). Deconstructing Student Perceptions of Generative AI (GenAI) through an Expectancy Value Theory (EVT)-based Instrument. arXiv preprint arXiv:2305.01186.
2. Younus, M. A., & Hasan, T. M. (2020, April). Effective and fast deepfake detection method based on haar wavelet transform. In 2020 International Conference on Computer Science and Software Engineering (CSASE) (pp. 186-190). IEEE.
3. Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., ... & Nguyen, C. M. (2022). Deep learning for deepfakes creation and detection: A survey. Computer Vision and Image Understanding, 223, 103525.
4. Guarnera, L., Giudice, O., Nastasi, C., & Battiato, S. (2020, September). Preliminary forensics analysis of deepfake images. In 2020 AEIT international annual conference (AEIT) (pp. 1-6). IEEE.
5. Gass, T., Pishchulin, L., Dreuw, P., & Ney, H. (2011, March). Warp that smile on your face: Optimal and smooth deformations for face recognition. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) (pp. 456-463). IEEE.
6. Pishchulin, L., Gass, T., Dreuw, P., & Ney, H. (2011). The fast and the flexible: Extended pseudo two-dimensional warping for face recognition. In Pattern Recognition and Image Analysis: 5th Iberian Conference, IbPRIA 2011, Las Palmas de Gran Canaria, Spain, June 8-10, 2011. Proceedings 5 (pp. 49-57). Springer Berlin Heidelberg.
7. Pishchulin, L., Gass, T., Dreuw, P., & Ney, H. (2012). Image warping for face recognition: From local optimality towards global optimization. Pattern Recognition, 45(9), 3131-3140.
8. Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., ... & Asari, V. K. (2019). A state-of-the-art survey on deep learning theory and architectures. electronics, 8(3), 292.

9. Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Nießner, M. (2019). Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 1-11).
10. Vasist, P. N., & Krishnan, S. (2022). Deepfakes: an integrative review of the literature and an agenda for future research. Communications of the Association for Information Systems, 51(1), 14.
11. Yang, X., Li, Y., & Lyu, S. (2019, May). Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 8261-8265). IEEE.
12. Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020). Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3207-3216).
13. Al-Dhabi, Y., & Zhang, S. (2021, August). Deepfake video detection by combining convolutional neural network (cnn) and recurrent neural network (rnn). In 2021 IEEE International Conference on Computer Science, Artificial Intelligence and Electronic Engineering (CSAIEE) (pp. 236-241). IEEE.
14. Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019, May). Capsule-forensics: Using capsule networks to detect forged images and videos. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2307-2311). IEEE.
15. Saikia, P., Dholaria, D., Yadav, P., Patel, V., & Roy, M. (2022, July). A hybrid CNN-LSTM model for video deepfake detection by leveraging optical flow features. In 2022 International Joint Conference on Neural Networks (IJCNN) (pp. 1-7). IEEE.
16. Kumar, M., & Sharma, H. K. (2023). A GAN-based model of deepfake detection in social media. Procedia Computer Science, 218, 2153-2162.
17. Li, Y., & Lyu, S. (2018). Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv:1811.00656.
18. Lin, Y. K., & Sun, H. L. (2023). Few-Shot Training GAN for Face Forgery Classification and Segmentation Based on the Fine-Tune Approach. Electronics, 12(6), 1417.
19. Olisah, C. C., & Smith, L. (2019). Understanding unconventional preprocessors in deep convolutional neural networks for face identification. SN Applied Sciences, 1(11), 1511.
20. Nirkin, Y., Masi, I., Tuan, A. T., Hassner, T., & Medioni, G. (2018, May). On face segmentation, face swapping, and face perception. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) (pp. 98-105). IEEE.
21. Guo, D., Fraichard, T., Xie, M., & Laugier, C. (2000, October). Color modeling by spherical influence field in sensing driving environment. In Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No. 00TH8511) (pp. 249-254). IEEE.
22. Yousefi, J. (2011). Image binarization using Otsu thresholding algorithm. Ontario, Canada: University of Guelph, 10.
23. Xie, X., Zheng, W. S., Lai, J., Yuen, P. C., & Suen, C. Y. (2010). Normalization of face illumination based on large-and small-scale features. IEEE Transactions on Image Processing, 20(7), 1807-1821.
24. Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1251-1258).
25. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
26. Deng, J. (2009). A large-scale hierarchical image database. Proc. of IEEE Computer Vision and Pattern Recognition, 2009.
27. Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How transferable are features in deep neural networks?. Advances in neural information processing systems, 27.
28. Dakin, S. C., & Watt, R. J. (2009). Biological “bar codes” in human faces. Journal of vision, 9(4), 2-2.
29. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826).
30. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
31. Hajian-Tilaki, K. (2013). Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian journal of internal medicine, 4(2), 627.