Derin Sahte Ses Manipülasyonu Tespit Sistemleri Üzerine Bir Derleme

Year 2024, Volume: 29 Issue: 1, 353 - 402, 30.04.2024

Gul Tahaoglu , Muhammed Kılıç , Beste Üstübioğlu , Güzin Ulutaş

https://doi.org/10.53433/yyufbed.1358880

Abstract

Gerçek kişilerin konuşmalarını içeren dijital ses dosyalarının kullanılması ile gerçekleştirilen derin sahte ses manipülasyonu, sesi taklit edilecek kişinin sesini klonlayarak kişinin söylemediği bir şeyi söylemiş gibi içerikte ses dosyalarını oluşturan bir sahtecilik türüdür. Konuşmacının kimliğini doğrulamak için güvenlik adımı olarak kabul edilen Otomatik Konuşmacı Doğrulama Sistemlerinin derin sahte ses sahtecilikleri saldırılarına karşı savunmasızlığı söz konusudur. Ayrıca mahkemelerde karar merciini etkileyecek delil olarak sunulan ses dosyalarının orijinal olup olmadığı kontrolü önemli bir ihtiyaç haline gelmiştir. Bu tür sahteciliklerin uzman sistemler tarafından tespit edilebilmesi günümüz çağı için oldukça önem arz etmektedir. Bu sahtecilik türündeki saldırıların tespit edilebilmesi için literatürde çeşitli yöntemler önerilmiştir. Literatürdeki çalışmalarda performans değerlendirmesinde kullanılan ücretsiz erişimli veri setleri de mevcut olup sonuç kıyaslamasında kullanabilmesi mümkündür. Bu çalışmada literatürdeki yöntemler ve verisetleri incelenmiş, yöntemlerin bu verisetleri üzerindeki performans değerlendirmeleri, avantaj ve dezavantajları vurgulanmıştır.

Keywords

Derin sahte ses manipülasyonu, Derin sahte ses tespiti, Ses doğrulama

Ethical Statement

Bu çalışmanın, özgün bir çalışma olduğunu; çalışmanın hazırlık, analiz ve bilgilerin sunumu olmak üzere tüm aşamalarından bilimsel etik ilke ve kurallarına uygun davrandığımı; bu çalışma kapsamında elde edilmeyen tüm veri ve bilgiler için kaynak gösterdiğimi ve bu kaynaklara kaynakçada yer verdiğimi beyan ederim.

Supporting Institution

Türkiye Bilimsel ve Teknolojik Araştırma Kurumu (TÜBİTAK)

Project Number

122E013

Thanks

Bu çalışma 1001 projesi kapsamında Türkiye Bilimsel ve Teknolojik Araştırma Kurumu (TÜBİTAK) tarafından desteklenmektedir. (Proje No: 122E013)

A Review of Deepfake Audio Manipulation Detection Systems

Abstract

Besides facilitating access to audio content on the Internet, developments in deep learning methods have made it possible to produce deep fake audio. Automatic Speaker Verification systems considered a security step to authenticate the speaker, are vulnerable to deep spoofing attacks. It is crucial for today's age that expert systems can detect such frauds. Deep fake audio spoofing is carried out to produce audio files in the content by cloning the speaker's voice that is planned to be changed as if he said something he did not say. Various methods are proposed in the literature to detect this type of forgery. There are free-access datasets used in performance evaluation in studies in the literature, and it is possible to use them in result comparison. The planned research aims to reduce or eliminate the noise that may exist in the audio file of the system by passing the preprocessing stage of the audio signal received as input. This paper examines the methods and datasets in the literature, and the advantages and disadvantages of the methods on these datasets are emphasized.

Keywords

Audio authentication, Deep fake audio manipulation, Deep fake audio detection

Project Number

122E013

References

• Abdzadeh, P., & Veisi, H. (2023). A comparison of CQT spectrogram with STFT-based acoustic features in Deep Learning-based synthetic speech detection. Journal of AI and Data Mining, 11(1), 119-129. doi:10.22044/jadm.2022.12373.2382
• Alluri, K. N. R. K., & Vuppala, A. K. (2019, September). IIIT-H spoofing countermeasures for automatic speaker verification spoofing and countermeasures challeng. Interspeech 2019, Graz, Austria. doi:10.21437/Interspeech.2019-1623
• Alzantot, M., Wang, Z., & Srivastava, M. B. (2019, September). Deep residual neural networks for audio spoofing detection. Interspeech 2019, Graz, Austria. doi:10.21437/Interspeech.2019-3174
• Balamurali, B. T., Lin, K. W. E., Lui, S., Chen, J. M., & Herremans, D. (2019). Toward robust audio spoofing detection: a detailed comparison of traditional and learned features. IEEE Access, 7, 84229-84241. doi:10.1109/ACCESS.2019.2923806
• Borrelli, C., Bestagini, P., Antonacci, F., Sarti, A., & Tubaro, S. (2021). Synthetic speech detection through short-term and long-term prediction traces. EURASIP Journal on Information Security, 2021, 2. doi:10.1186/s13635-021-00116-3
• Cai, W., Wu, H., Cai, D., & Li, M. (2019, September). The DKU replay detection system for the ASVspoof 2019 challenge: on data augmentation, feature representation, classification, and fusion. Interspeech 2019, Graz, Austria. doi:10.21437/Interspeech.2019-1230
• Chen, T., Kumar, A., Nagarsheth, P., Sivaraman, G., & Khoury, E. (2020, November). Generalization of audio deepfake detection. The Speaker and Language Recognition Workshop (Odyssey 2020), Tokyo, Japan. doi:10.21437/Odyssey.2020-19
• Chen, Z., Xie, Z., Zhang, W., & Xu, X. (2017, August). ResNet and model fusion for automatic spoofing detection. Interspeech 2017, Stockholm, Sweeden. doi:10.21437/Interspeech.2017-1085
• Cheng, X., Xu, M., & Zheng, T. F. (2019, March). Replay detection using CQT-based modified group delay feature and ResNeWt network in ASVspoof 2019. 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Lanzhou, China. doi:10.1109/APSIPAASC47483.2019.9023158
• Chettri, B., Stoller, D., Morfi, V., Ramírez, M. A. M., Benetos, E., Sturm, B. L. (2019, September). Ensemble models for spoofing detection in automatic speaker verification. Interspeech 2019, Graz, Austria. doi:10.21437/Interspeech.2019-2505
• Chintha, A., Thai, B., Sohrawardi, S. J., Bhatt, K., Hickerson, A., Wright, M., & Ptucha, R. (2020). Recurrent convolutional structures for audio spoof and video deepfake detection. Journal of Selected Topics in Signal Processing, 14(5), 1024-1037. doi:10.1109/JSTSP.2020.2999185
• Dua, M., Jain, C., & Kumar, S. (2022). LSTM and CNN based ensemble approach for spoof detection task in automatic speaker verification systems. Journal of Ambient Intelligence and Humanized Computing, 13, 1985-2000. doi:10.1007/s12652-021-02960-0
• Font, R., Espín, J. M., & Cano, M. J. (2017, August). Experimental analysis of features for replay attack detection — results on the ASVspoof 2017 challenge. Interspeech 2017, Stockholm, Sweeden. doi:10.21437/Interspeech.2017-450
• Gunendradasan, T., Wickramasinghe, B., Le, N. P., Ambikairajah, E., & Epps, J. (2018, September). Detection of replay-spoofing attacks using frequency modulation features. Interspeech 2018, Hyderabad, India. doi:10.21437/Interspeech.2018-1473
• Hua, G., Teoh, A. B. J., & Zhang, H. (2021). Towards end-to-end synthetic speech detection. IEEE Signal Processing Letters, 28, 1265-1269. doi:10.1109/LSP.2021.3089437
• Jiang, Z., Zhu, H., Peng, L., Ding, W., & Ren, Y. (2020, October). Self-supervised spoofing audio detection scheme. Interspeech 2020, Shangai, China. doi:10.21437/Interspeech.2020-1760
• Kinnunen, T., Delgado, H., Evans, N., Lee, K.A., Vestman, V., Nautsch, A., …, & Reynolds, D. A. (2020). t-DCF: a detection cost function for the tandem assessment of spoofing countermeasures and automatic speaker verification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, 2195-2210 2020.
• Korshunov, P., Marcel, S., Muckenhirn, H., Gonçalves, A. R., Souza Mello, A. G., Velloso, V. R. P., …, & Sahidullah, M. (2016, September). Overview of BTAS 2016 speaker anti-spoofing competition. 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS), Niagara Falls, NY, USA. doi:10.1109/BTAS.2016.7791200
• Kwak, Y., Kwag, S., Lee, J., Jeon, Y., Hwang, J., Choi, H.J., …, & Yoon, J. W. (2023). Voice spoofing detection through residual network, max feature map, and depthwise separable convolution. IEEE Access, 11, 49140-49152. doi:10.1109/ACCESS.2023.3275790
• Lai, CI, Chen, N., Villalba, J., & Dehak, N. (2019, September). ASSERT: Anti-spoofing with squeeze-excitation and residual networks. Interspeech 2019, Graz, Austria. doi:10.21437/Interspeech.2019-1794
• Lavrentyeva, G., Novoselov, S., Malykh, E., Kozlov, A., Kudashev, O., & Shchemelinin, V. (2017, August). Audio replay attack detection with deep learning frameworks. Interspeech 2017, Stockholm, Sweden. doi:10.21437/Interspeech.2017-360
• Mewada, H., Al-Asad, J. F., Almalki, F. A., Khan, A. H., Almujally, N. A., El-Nakla, S., & Naith, Q. (2023). Gaussian-filtered high-frequency-feature trained optimized BiLSTM network for spoofed-speech classification. Sensors, 23, 6637. doi:10.3390/s23146637
• Nagarsheth, P., Khoury, E., Patil, K., & Garland, M. (2017, August). Replay attack detection using DNN for channel discrimination. Interspeech 2017, Stockholm, Sweden. doi:10.21437/Interspeech.2017-1377
• Nautsch, A., Wang, X., Evans, N., Kinnunen, T. H., Vestman, V., Todisco, M., …, & Lee, K. A. (2021). ASVspoof 2019: Spoofing countermeasures for the detection of synthesized, converted and replayed speech. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(2), 252-265. doi:10.1109/TBIOM.2021.3059479
• Patel, T. B., & Patil, H. A. (2015, September). Combining evidences from mel cepstral, cochlear filter cepstral and instantaneous frequency features for detection of natural vs. spoofed speech. Interspeech 2015, Dresden, Germany. doi:10.21437/Interspeech.2015-467
• Paul, D., Sahidullah, M., & Saha, G. (2017, March). Generalization of spoofing countermeasures: A case study with ASVspoof 2015 and BTAS 2016 corpora. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA. doi:10.1109/ICASSP.2017.7952516
• Qian, Y., Chen, N., & Yu, K. (2016). Deep features for automatic spoofing detection. Speech Communication, 85, 43-52. doi:10.1016/j.specom.2016.10.007
• Rahul, T. P., Aravind, P. R., Ranjith, C., Usamath, N., & Paramparambath, N. (2020). Audio spoofing verification using deep convolutional neural networks by transfer learning. ArXiv, abs/2008.03464,2020. doi:10.48550/arXiv.2008.03464
• Reimao, R., & Tzerpos, V. (2019, October). FoR: A dataset for synthetic speech detection. 2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), Timisoara, Romania. doi:10.1109/SPED.2019.8906599
• Suthokumar, G., Sethu, V., Wijenayake, C., & Ambikairajah, E. (2018, September). Modulation dynamic features for the detection of replay attacks. Interspeech 2018, Hyderabad, India. doi:10.21437/Interspeech.2018-1846
• Sriskandaraja, K., Sethu, V., Ambikairajah, E., & Li, H. (2017). Front-end for antispoofing countermeasures in speaker verification: Scattering spectral decomposition. IEEE Journal of Selected Topics in Signal Processing, 11(4), 632-643. doi:10.1109/JSTSP.2016.2647202
• Sriskandaraja, K., Sethu, V., & Ambikairajah, E. (2018, September). Deep siamese architecture based replay detection for secure voice biometric. Interspeech 2018, Hyderabad, India. doi:10.21437/Interspeech.2018-1819
• Tak, H., Patino, J., Todisco, M., Nautsch, A., Evans, N., & Larcher, A. (2021, June). End-to-End anti-spoofing with RawNet2. ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada. doi:10.1109/ICASSP39728.2021.9414234
• Tan, C. B., Hijazi, M. H. A., & Nohuddin, P. N. E. (2023, September). A hybrid classification approach for artificial speech detection. 2023 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), Kota Kinabalu, Malaysia. doi:10.1109/IICAIET59451.2023.10291764
• Xiao, X., Tian, X., Du, S., Xu, H., Siong, C. E., & Li, H. (2015, September). Spoofing speech detection using high dimensional magnitude and phase features: the NTU approach for ASVspoof 2015 challenge. Interspeech 2015, Dresden, Germany. doi:10.21437/Interspeech.2015-465
• Wu, Z., Kinnunen, T., Evans, N., Yamagishi, J., Hanilçi, C., Sahidullah, M., & Sizov, A. (2015, September). ASVspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge. Interspeech 2015, Dresden, Germany. doi:10.21437/Interspeech.2015-462
• Wu, Z., Yamagishi, J., Kinnunen, T., Hanilçi, C., Sahidullah, M., Sizov, ..., & Delgado, H. (2017). ASVspoof: the automatic speaker verification spoofing and countermeasures challenge. IEEE Journal of Selected Topics in Signal Processing, 11(4), 588-604. doi:10.1109/JSTSP.2017.2671435
• Witkowski, M., Kacprzak, S., Żelasko, P., Kowalczyk, K., & Gałka, J. (2017, August). Audio replay attack detection using high-frequency features. Interspeech 2017, Stockholm, Sweden. doi:10.21437/Interspeech.2017-776
• Wang, L., Yoshida, Y., Kawakami, Y., & Nakagawa, S. (2015, September). Relative phase information for detecting human speech and spoofed speech. Interspeech 2015, Dresden, Germany. doi:10.21437/Interspeech.2015-473
• Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., …, & Delgado H. (2021, September). ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection. 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, France. doi:10.21437/ASVSPOOF.2021-8
• Yang, J., You, C., & He, Q. (2018, September). Feature with complementarity of statistics and principal information for spoofing detection. Interspeech 2018, Hyderabad, India. doi:10.21437/Interspeech.2018-1693
• Yu, H., Tan, Z. H., Zhang, Y., Ma, Z., Guo, J. (2017). DNN filter bank cepstral coefficients for spoofing detection. IEEE Access, 5, 4779-4787. doi:10.1109/ACCESS.2017.2687041
• Zarish, A., Javed, A., & Khalid, M. (2022). AEXANet: An end-to-end deep learning based voice anti-spoofing system. Workshop on Artificial Intelligence for Multimedia Forensics and Disinformation Detection (AI4MFDD).
• Zhang, Y., Jiang, F., & Duan, Z. (2021). One-class learning towards synthetic voice spoofing detection. IEEE Signal Processing Letters, 28, 937-941. https://doi.org/10.1109/LSP.2021.3076358
• Zhang, Z., Yi, X., & Zhao, X. (2021, June). Fake speech detection using residual network with transformer encoder. Proceedings of the 2021 ACM workshop on information hiding and multimedia security, Belgium. doi:10.1145/3437880.3460408
• Zhang, J., Tu, G., Liu, S., & Cai, Z. (2023). Audio anti-spoofing based on audio feature fusion. Algorithms, 16, 317. doi:10.3390/a16070317