A YOLOV3-Based Method for Detecting Deepfake Manipulated Facial Images

Abstract

With the advancement of technology and the development of applications that make it easier to transfer images, sounds and videos to the virtual environment, it has become much easier to access people's personal information, videos and images. Deepfake technology produces fakes of authentic images or sounds using deep learning and artificial intelligence techniques. Today, in addition to being used in the entertainment and film industries, it is also used in situations such as creating fake news and discrediting people. Different studies have been conducted in the literature to detect deepfake images and videos to prevent these situations. In this study, a comprehensive literature review was conducted. Real and fake images were collected and labelled from different datasets or videos, and a dataset was created by applying the necessary pre-processing steps. With the created dataset, training was carried out with YOLOv3 technology, which calculates class probabilities differently from traditional methods using Convolutional Neural Networks (CNN) and handles all operations in a single regression problem, which can make fast and high-accurate detection, and the modelling process is explained. With the tests performed in the study, the model that can detect fake images produced with deepfake technology with 95% accuracy was obtained.

Keywords

• Convolutional neural networks
• deepfake image detection
• deep learning
• YOLOV3

Derin Sahte ile Manipüle Edilmiş Yüz Görüntülerin Tespiti için YOLOV3 Tabanlı Bir Yöntem

Abstract

Teknolojinin ilerlemesi ve görüntü, ses ve videoların sanal ortama aktarılmasını kolaylaştıran uygulamaların gelişmesiyle birlikte insanların kişisel bilgi, video ve görsellerine ulaşmak çok daha kolay hale gelmiştir. Derin sahte teknolojisi, derin öğrenme ve yapay zekâ tekniklerini kullanarak gerçek görüntü veya seslerin sahtelerini üretmek için kullanılmaktadır. Günümüzde eğlence ve film endüstrilerinde kullanılmasının yanı sıra, sahte haber oluşturma ve insanları itibarsızlaştırma gibi durumlarda da kullanılmaktadır. Bu durumların önüne geçmek için literatürde derin sahte görsel ve videoların tespitine yönelik farklı çalışmalar yapılmıştır. Bu çalışmada kapsamlı bir literatür taraması yapılmış ve farklı veri setlerinden veya videolardan gerçek ve sahte görseller toplanmış, etiketlenmiş ve gerekli ön işleme adımları uygulanarak bir veri seti oluşturulmuştur. Oluşturulan veri seti ile Evrişimli Sinir Ağlarını kullanarak geleneksel yöntemlerden farklı bir şekilde sınıf olasılıklarını hesaplayan ve tüm işlemleri tek bir regresyon probleminde ele alan hızlı ve yüksek doğrulukla tespit yapabilen YOLOv3 teknolojisi ile eğitim gerçekleştirilmiş ve modelleme süreci anlatılmıştır. Çalışmada yapılan testlerle derin sahte teknolojisiyle üretilmiş sahte görüntüleri %95 doğrulukla tespit edebilen bir model elde edilmiştir.

Keywords

• Evrişimli sinir ağları
• deepfake görüntü algılama
• derin öğrenme
• YOLOV3

References

1. Çeçen M, Karaköse M. A Deepfake Image Detection Approach Based on YOLOv3. In: 2th International Conference on Advances and Innovations in Engineering; 21-23 September 2023. pp. 10-18.
2. Franklin RJ, Mohona. Traffic Signal Violation Detection using Artificial Intelligence and Deep Learning. In: International Conference on Communication and Electronics Systems; 10-12 June 2020. pp. 839 - 844.
3. İlhan İ., Balı E., Karaköse M. An Improved DeepFake Detection Approach with NASNetLarge CNN. In: IEEE International Conference on Data Analytics for Business and Industry; 25-26 October 2022. pp. 598-602.
4. Seow JW, Lim MK, Phan R, Liu J. A comprehensive overview of Deepfake: Generation, detection, datasets, and opportunities. Elsevier Neurocomputing 2022; 513: 351–371.
5. İlhan İ, Karaköse M. Derin Sahte Videoların Tespiti ve Uygulamaları için Bir Karşılaştırma Çalışması. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi 2021; 8(14): 47-60.
6. John J, Sherif B. Comparative Analysis on Different DeepFakeDetection Methods and Semi Supervised GAN Architecture for DeepFake Detection. In: Proceedings of the Sixth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud); 10-12 November 2022.
7. Karras T, Laine S, Alia T. A Style-Based Generator Architecture for Generative Adversarial Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 2021; 43: 4217-4228.
8. Zhu JY, Park T, Isola P, Efros AA. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Image Translation Using Cycle-Consistent Adversarial Networks. In: IEEE/CVF International Conference on Computer Vision; 22-29 October 2017; Venice, Italy.

9. Choi Y, Choi M, Munyoung K, Ha JW, Kim S, Choo J. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Imageto-Image Translation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition; 18-23 June 2018. pp. 8789-8797.
10. Thies J, Zollhöfer M, Stamminger M, Theobalt C, Niebner M. Face2face: Real-Time Face Capture and Reenactment of RGB Videos. In: IEEE/CVF Conference on Compute Vision and Pattern Recognition; 27-30 June 2016. pp. 2387 – 2395.
11. Khatri N, Borar V, Garg R. A Comparative Study: Deepfake Detection Using Deep-learning. In: 13th International Conference on Cloud Computing, Data Science & Engineering; 19-20 January 2023.
12. Pipin SJ, Purba R, Pasha MF. Deepfake Video Detection Using Spatiotemporal Convolutional Network and Photo Response Non Uniformity. In: IEEE International Conference of Computer Science and Information Technology (ICOSNIKOM); 19-21 October 2022.
13. Zhang J, Cheng K, Sovernigo G, Lin X. A Heterogeneous Feature Ensemble Learning based Deepfake Detection Method. In: IEEE International Conference on Communications; 16-20 May 2022. pp: 2084 - 2089.
14. Budhiraja R, Kumar M, Das MK, Bafila AS, Singh S, MeDiFakeD: Medical Deepfake Detection using Convolutional Reservoir Networks. In: IEEE Global Conference on Computing, Power and Communication Technologies (GlobConPT); 23-25 September 2022.
15. Li Y, Zhang C, Sun P, Ke L, Ju Y, Qi H, Lyu S. DeepFake-o-meter: An Open Platform for DeepFake Detection. In: IEEE Security and Privacy Workshops (SPW); 27 May 2021; China. pp. 277-281.
16. Jia S, Li X,Siwei L. Model Attribution of Face-Swap Deepfake Videos. In: IEEE International Conference on Image Processing (ICIP), 16-19 October 2022. pp: 2356 - 2360.
17. Yang X, Li Y, Lyu S. Exposing deep fakes using inconsistent head poses. In: IEEE Int. Conf. Acoust., Speech and Signal Process. (ICASSP), 25-30 March 2012. pp. 8261 –8265.
18. Ataş S, İlhan İ, Karaköse M. An Efficient Deepfake Video Detection Approach with Combination of EfficientNet and Xception Models Using Deep Learning. In: 26th International Conference on Information Technology (IT); 13-15 December 2023.
19. Bar NF, Yetis H, Karakose M. An efficient and scalable variational quantum circuits approach for deep reinforcement learning. Quantum Information Processing 2023; 22(8): 300.
20. Srivastava N, Salakhutdinov RR. Multimodal Learning with Deep Boltzmann Machines. Advances in Neural Information Processing Systems 25 (NIPS 2012); 3-8 December 2012
21. Krizhevsky A, Sutskever I,Geoffrey EH. ImageNet Classification with Deep Convolutional Neural Networks. Adv.Neural Inf. Process. Syst. 2012; 25: 1–9.
22. Deng J, Dong W, Socher R, Li LJ, Li K, Fei LF. Imagenet: A large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition; 20-25 June 2009. pp. 248-255.
23. Rajput, SK, Patni JC, Alshamrani SS, Chaudhari V, Dumka A, Singh R, Rashid, M, Gehlot A, AlGhamdi AS. Automatic Vehicle Identification and Classification Model Using the YOLOv3 Algorithm for a Toll Management System. Sustainability 2022; 14(15): 9163.
24. Salih ZA, Thabit R, Zidan KA, Khoo Be. A new face image manipulation reveal scheme based on face detection and image watermarking. In: IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET); 13-15 September 2022.
25. Chang X, Wu J, Yang T, Feng G. DeepFake Face Image Detection based on Improved VGG Convolutional Neural Network. In: 39th Chinese Control Conference; 27-29 July 2020.
26. Belhi A, Gasmi H, Al-Ali AK, Bouras A, Foufou S, Yu X, Zhang H. Deep Learning and Cultural Heritage: The CEPROQHA Project Case Study. In: International Conference on Software, Knowledge Information, Industrial Management and Applications (SKIMA); 26-29 August 2019.
27. Concas S, Perelli G, Marcialis GL, Puglisi G. Tensor-Based Deepfake Detection in Scaled and Compressed Images. In: IEEE International Conference on Image Processing (ICIP); 16-19 October 2022. pp. 3121 – 3125.
28. Rahman A, Siddique N, Moon MJ, Tasnim T, Islam M, Shahiduzzaman Md, Ahmed S. Short and Low Resolution Deepfake Video Detection using CNN. In: IEEE Region 10 Humanitarian Technology Conference (R10-HTC) 16-19 September 2022. pp. 259 - 264.
29. Afchar D, Nozick V, Yamagishi J, Echizen I. MesoNet: a Compact Facial Video Forgery Detection Network. In: IEEE International Workshop on Information Forensics and Security (WIFS), 11-13 December 2018.