Digital twin of multi-model drone detection system on Airsim for RF and vision modalities

Abstract

Drones have become more prevalent in recent years and are used for both beneficial and malicious purposes. As a result, protecting restricted areas from unauthorized drone activities has become crucial. However, some researchers face challenges in developing drone detection systems due to the high costs of necessary equipment. This paper presents an innovative solution by creating an Airsim Graphical User Interface (GUI) tool compatible with the Unreal Engine. This tool enables the simulation of drone flights and creation of image and radio frequency (RF) datasets for drone detection in a simulation environment. Our approach involves modeling the measurement devices such as cameras to capture image data and software defined radio (SDR) receiver to capture RF signals as raw in-phase and quadrature (IQ) data. Moreover, users can manage automated route planning for drones, recording configurations, and different cameras and RF configurations. Researchers can now generate datasets with various images and RF configurations without the need for physical drones, cameras, or SDRs, enabling experimentation with different drone detection models. Furthermore, we proposed models for drone detection systems by using generated datasets from the proposed dataset generation system.

Keywords

• Drone detection
• Signal processing
• Image processing
• Machine learning
• Simulation

References

1. Şasi, A., & Yakar, M. (2017). Photogrammetric modelling of Sakahane Masjid using an unmanned aerial vehicle. Turkish Journal of Engineering, 1(2), 82-87. https://doi.org/10.31127/tuje.316675
2. Karataş, L., Alptekin, A., Kanun, E., & Yakar, M. (2022). Tarihi kârgir yapılarda taş malzeme bozulmalarının İHA fotogrametrisi kullanarak tespiti ve belgelenmesi: Mersin Kanlıdivane ören yeri vaka çalışması. İçel Dergisi, 2(2), 41-49.
3. Karataş, L., Alptekin, A., Karabacak, A., Yakar, M. (2022). Detection and documentation of stone material deterioration in historical masonry buildings using UAV photogrammetry: A case study of Mersin Sarisih Inn. Mersin Photogrammetry Journal, 4(2), 53-61. https://doi.org/10.53093/mephoj.1198605
4. Şasi, A., & Yakar, M. (2018). Photogrammetric modelling of Hasbey Dar'ülhuffaz (Masjid) using an unmanned aerial vehicle. International Journal of Engineering and Geosciences, 3(1), 6-11. https://doi.org/10.26833/ijeg.328919
5. Çolak, A., Aktan, N., & Yılmaz, H. M. (2022). Modelling of its surroundings and Selime Cadhetral by UAV data. Advanced UAV, 2(1), 24-28.
6. Banafaa, M., Pepeoğlu, Ö., Shayea, I., Alhammadi, A., Shamsan, Z., Razaz, M. A., ... & Al-Sowayan, S. (2024). A comprehensive survey on 5G-and-beyond networks with UAVs: Applications, emerging technologies, regulatory aspects, research trends and challenges. IEEE Access, 12, 7786 – 7826. https://doi.org/10.1109/ACCESS.2023.3349208
7. Jiang, N., Wang, K., Peng, X., Yu, X., Wang, Q., Xing, J., ... & Han, Z. (2021). Anti-uav: a large-scale benchmark for vision-based uav tracking. IEEE Transactions on Multimedia, 25, 486-500. https://doi.org/10.1109/TMM.2021.3128047
8. Zhao, J., Wang, G., Li, J., Jin, L., Fan, N., Wang, M., ... & Guo, Y. (2021). The 2nd anti-uav workshop & challenge: Methods and results. Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.2108.09909

9. Zhao, J., Li, J., Jin, L., Chu, J., Zhang, Z., Wang, J., ... & Shengmei, J. S. (2023). The 3rd anti-uav workshop & challenge: Methods and results. Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.2305.07290
10. Coluccia, A., Ghenescu, M., Piatrik, T., De Cubber, G., Schumann, A., Sommer, L., Klatte, J., Schuchert, T., Beyerer, J., Farhadi, M., Amandi, R., Aker, C., Kalkan, S., Saqib, M., Sharma, N., Daud, S., Makkah, K., & Blumenstein, M. (2017). Drone-vs-bird detection challenge at IEEE AVSS2017. IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) Drone-vs-Bird detection challenge, 1-6, Lecce, Italy.
11. Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Mantegh, I. (2017). Drone-vs-bird detection challenge at IEEE AVSS2017. 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-8. https://doi.org/ 10.1109/AVSS.2017.8078464
12. Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Rajashekar, S. (2021). Drone vs. bird detection: Deep learning algorithms and results from a grand challenge. Sensors, 21(8), 2824. https://doi.org/10.3390/s21082824
13. Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Mantegh, I. (2021). Drone-vs-bird detection challenge at IEEE AVSS2021. 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-8. https://doi.org/10.1109/AVSS52988.2021.9663844
14. Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Pavleski, D. (2022). Drone-vs-bird detection challenge at ICIAP 2021. In International Conference on Image Analysis and Processing, 410-421. https://doi.org/10.1007/978-3-031-13324-4_35
15. Li, J., Murray, J., Ismaili, D., Schindler, K., & Albl, C. (2020). Reconstruction of 3D flight trajectories from ad-hoc camera networks. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1621-1628. https://doi.org/10.1109/IROS45743.2020.9341479
16. Allahham, M. S., Al-Sa'd, M. F., Al-Ali, A., Mohamed, A., Khattab, T., & Erbad, A. (2019). DroneRF dataset: A dataset of drones for RF-based detection, classification and identification. Data in Brief, 26, 104313. https://doi.org/10.1016/j.dib.2019.104313
17. Swinney, C. J., & Woods, J. C. (2021). RF detection and classification of unmanned aerial vehicles in environments with wireless interference. International Conference on Unmanned Aircraft Systems (ICUAS), 1494-1498. https://doi.org/10.1109/ICUAS51884.2021.9476867
18. Glüge, S., Nyfeler, M., Ramagnano, N., Horn, C., & Schüpbach, C. (2023). Robust drone detection and classification from radio frequency signals using convolutional neural networks. 15th International Joint Conference on Computational Intelligence (IJCCI), 496-504. https://doi.org/10.5220/0012176800003595
19. Medaiyese, O., Ezuma, M., Lauf, A., & Adeniran, A. (2022). Cardinal RF (CardRF): An outdoor UAV/UAS/drone RF signals with Bluetooth and WiFi signals dataset.
20. Al-Sa’d, M. F., Al-Ali, A., Mohamed, A., Khattab, T., & Erbad, A. (2019). RF-based drone detection and identification using deep learning approaches: An initiative towards a large open source drone database. Future Generation Computer Systems, 100, 86-97. https://doi.org/10.1016/j.future.2019.05.007
21. Al-Emadi, S., & Al-Senaid, F. (2020). Drone detection approach based on radio-frequency using convolutional neural network. 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), 29-34. 10.1109/ICIoT48696.2020.9089489
22. Allahham, M. S., Khattab, T., & Mohamed, A. (2020). Deep learning for RF-based drone detection and identification: A multi-channel 1-D convolutional neural networks approach. IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), 112-117. https://doi.org/10.1109/ICIoT48696.2020.9089657
23. Medaiyese, O. O., Syed, A., & Lauf, A. P. (2021). Machine learning framework for RF-based drone detection and identification system. 2nd International Conference on Smart Cities, Automation & Intelligent Computing Systems (ICON-SONICS), 58-64. https://doi.org/10.1109/ICON-SONICS53103.2021.9617168
24. Nemer, I., Sheltami, T., Ahmad, I., Yasar, A. U. H., & Abdeen, M. A. (2021). RF-based UAV detection and identification using hierarchical learning approach. Sensors, 21(6), 1947. https://doi.org/10.3390/s21061947
25. Zhao, Z., Du, Q., Yao, X., Lu, L., & Zhang, S. (2023). A Two-Dimensional Deep Network for RF-based Drone Detection and Identification Towards Secure Coverage Extension. In 2023 IEEE 98th Vehicular Technology Conference (VTC2023-Fall), 1-5. https://doi.org/10.1109/VTC2023-Fall60731.2023.10333485
26. Nguyen, P., Kim, T., Miao, J., Hesselius, D., Kenneally, E., Massey, D., ... & Vu, T. (2019). Towards RF-based localization of a drone and its controller. Proceedings of the 5th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications, 21-26. https://doi.org/10.1145/3325421.3329766
27. Unlu, E., Zenou, E., Riviere, N., & Dupouy, P. E. (2019). Deep learning-based strategies for the detection and tracking of drones using several cameras. IPSJ Transactions on Computer Vision and Applications, 11, 1-13. https://doi.org/10.1186/s41074-019-0059-x
28. Singha, S., & Aydin, B. (2021). Automated drone detection using YOLOv4. Drones, 5(3), 95. https://doi.org/10.3390/drones5030095
29. Zhai, X., Huang, Z., Li, T., Liu, H., & Wang, S. (2023). YOLO-Drone: an optimized YOLOv8 network for tiny UAV object detection. Electronics, 12(17), 3664. https://doi.org/10.3390/electronics12173664
30. Seidaliyeva, U., Akhmetov, D., Ilipbayeva, L., & Matson, E. T. (2020). Real-time and accurate drone detection in a video with a static background. Sensors, 20(14), 3856. https://doi.org/10.3390/s20143856
31. Carrio, A., Vemprala, S., Ripoll, A., Saripalli, S., & Campoy, P. (2018). Drone detection using depth maps. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), 1034-1037. https://doi.org/10.1109/IROS.2018.8593405
32. Carrio, A., Tordesillas, J., Vemprala, S., Saripalli, S., Campoy, P., & How, J. P. (2020). Onboard detection and localization of drones using depth maps. IEEE Access, 8, 30480-30490. https://doi.org/10.1109/ACCESS.2020.2971938
33. Akyon, F. C., Eryuksel, O., Ozfuttu, K. A., & Altinuc, S. O. (2021). Track boosting and synthetic data aided drone detection. 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-5. https://doi.org/10.1109/AVSS52988.2021.9663759
34. Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014). Microsoft coco: Common objects in context. Computer Vision–ECCV 2014: 13th European Conference, Zurich, 13, 740-755. https://doi.org/10.1007/978-3-319-10602-1_48
35. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248-255. https://doi.org/10.1109/CVPR.2009.5206848