Drone Detection Performance Evaluation via Real Experiments with Additional Synthetic Darkness

Year 2024, Volume: 11 Issue: 3, 546 - 562, 30.09.2024

Furkan Oruç , Hüseyin Birkan Yılmaz

https://doi.org/10.54287/gujsa.1526979

Abstract

Detecting drones is increasingly challenging, particularly when developing passive and low-cost defense systems capable of countering malicious attacks in environments with high levels of darkness and severe weather conditions. This research addresses the problem of drone detection under varying darkness levels by conducting an extensive study using deep learning models. Specifically, the study evaluates the performance of three advanced models: Yolov8, Vision Transformers (ViT), and Long Short-Term Memory (LSTM) networks. The primary focus is on how these models perform under synthetic darkness conditions, ranging from 20% to 80%, using a composite dataset (CONNECT-M) that simulates nighttime scenarios. The methodology involves applying transfer learning to enhance the base models, creating Yolov8-T, ViT-T, and LSTM-T variants. These models are then tested across multiple datasets with varying darkness levels. The results reveal that all models experience a decline in performance as darkness increases, as measured by Precision-Recall and ROC Curves. However, the transfer learning-enhanced models consistently outperform their original counterparts. Notably, Yolov8-T demonstrates the most robust performance, maintaining higher accuracy across all darkness levels. Despite the general decline in performance with increasing darkness, each model achieves an accuracy above 0.6 for data subjected to 60% or greater darkness. The findings highlight the challenges of drone detection under low-light conditions and emphasize the effectiveness of transfer learning in improving model resilience. The research suggests further exploration into multi-modal systems that combine audio and optical methods to enhance detection capabilities in diverse environmental settings.

Keywords

Drone Detection, Yolo, LSTM, Vision Transformers, Darkness

Supporting Institution

Boğaziçi University Scientific Research Projects

Project Number

BAP-SUP-17862

Thanks

This study has been supported by Boğaziçi University Scientific Research Projects Under Grant 17862 (BAP-SUP-17862)

References

• Adam, E. Y. (2020). Connectivity considerations for mission planning of a search and rescue drone team. Turkish Journal of Electrical Engineering and Computer Sciences, 28(4), 2228-2243. https://doi.org/10.3906/elk-1912-46
• Andraši, P., Radišić, T., Muštra, M., & Ivošević, J. (2017). Night-time detection of UAVs using thermal infrared camera. Transportation Research Procedia, 28, 183-190. https://doi.org/10.1016/j.trpro.2017.12.184
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
• Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
• Jamil, S., Abbas, M. S., & Roy, A. M. (2022). Distinguishing malicious drones using vision transformer. AI, 3(2), 260-273. https://doi.org/10.3390/ai3020016
• Khan, M. U., Misbah, M., Kaleem, Z., Deng, Y., & Jamalipour, A. (2023, June 20-23). GAANet: Ghost auto anchor network for detecting varying size drones in dark. In: Proceedings of the 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring) (pp. 1-5). Florence, Italy. https://doi.org/10.1109/VTC2023-Spring57618.2023.10200720
• Li, Y., Fan, Q., Huang, H., Han, Z., & Gu, Q. (2023). A modified YOLOv8 detection network for UAV aerial image recognition. Drones, 7(5), 304. https://doi.org/10.3390/drones7050304
• Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., & Houlsby, N. (2022, October 23-27). Simple Open-Vocabulary Object Detection. In: S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Proceedings of the 17th European Conference on Computer Vision (ECCV 2022) (pp. 728-755). Tel Aviv, Israel. https://doi.org/10.1007/978-3-031-20080-9_42
• Misbah, M., Khan, M. U., Yang, Z., & Kaleem, Z. (2023, March 12-13). Tf-net: Deep learning empowered tiny feature networks for night-time UAV detection. In: J. Zhao (Eds.), Proceedings of the 13th EAI International Conference on Wireless and Satellite Systems (pp. 3-18). Virtual Event, Singapore. https://doi.org/10.1007/978-3-031-34851-8_1
• Moustafa, N., & Jolfaei, A. (2020). Autonomous detection of malicious events using machine learning models in drone networks. In: Proceedings of the 2nd ACM MobiCom Workshop on Drone Assisted Wireless Communications for 5G and Beyond (pp. 61-66). https://doi.org/10.1145/3414045.3415951
• Moustafa, N., Slay, J., & Creech, G. (2017). Novel geometric area analysis technique for anomaly detection using trapezoidal area estimation on large-scale networks. IEEE Transactions on Big Data, 5(4), 481-494. https://doi.org/10.1109/TBDATA.2017.2715166
• Munir, A., Siddiqui, A. J., & Anwar, S. (2024, January 01-06). Investigation of UAV Detection in Images with Complex Backgrounds and Rainy Artifacts. In: Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) (pp. 221-230). Waikoloa, HI, USA. http://doi.org/10.1109/WACVW60836.2024.00031
• Ramadan, R. A., Emara, A. H., Al-Sarem, M., & Elhamahmy, M. (2021). Internet of drones intrusion detection using deep learning. Electronics, 10(21), 2633. https://doi.org/10.3390/electronics10212633
• Reis, D., Kupec, J., Hong, J., & Daoudi, A. (2023). Real-time flying object detection with YOLOv8. https://doi.org/10.48550/arXiv.2305.09972
• Svanström, F., Alonso-Fernandez, F., & Englund, C. (2022). Drone detection and tracking in real-time by fusion of different sensing modalities. Drones, 6(11), 317. https://doi.org/10.3390/drones6110317
• Yi, K. Y., Kyeong, D., & Seo, K. (2019). Deep learning-based drone detection and classification. The Transactions of the Korean Institute of Electrical Engineers, 68(2), 359-363. http://doi.org/10.5370/KIEE.2019.68.2.359
• Zhai, X., Huang, Z., Li, T., Liu, H., & Wang, S. (2023). YOLO-Drone: an optimized YOLOv8 network for tiny UAV object detection. Electronics, 12(17), 3664. https://doi.org/10.3390/electronics12173664