Digital twin of multi-model drone detection system on Airsim for RF and vision modalities

Year 2024, Volume: 8 Issue: 3, 572 - 582, 28.07.2024

Yusuf Özben , Süleyman Emre Demir , Hüseyin Birkan Yılmaz

https://doi.org/10.31127/tuje.1436757

Abstract

Drones have become more prevalent in recent years and are used for both beneficial and malicious purposes. As a result, protecting restricted areas from unauthorized drone activities has become crucial. However, some researchers face challenges in developing drone detection systems due to the high costs of necessary equipment. This paper presents an innovative solution by creating an Airsim Graphical User Interface (GUI) tool compatible with the Unreal Engine. This tool enables the simulation of drone flights and creation of image and radio frequency (RF) datasets for drone detection in a simulation environment. Our approach involves modeling the measurement devices such as cameras to capture image data and software defined radio (SDR) receiver to capture RF signals as raw in-phase and quadrature (IQ) data. Moreover, users can manage automated route planning for drones, recording configurations, and different cameras and RF configurations. Researchers can now generate datasets with various images and RF configurations without the need for physical drones, cameras, or SDRs, enabling experimentation with different drone detection models. Furthermore, we proposed models for drone detection systems by using generated datasets from the proposed dataset generation system.

Keywords

Drone detection, Signal processing, Image processing, Machine learning, Simulation

Supporting Institution

Bogazici University BAP

Project Number

BAP-SUP-17862

References

• Şasi, A., & Yakar, M. (2017). Photogrammetric modelling of Sakahane Masjid using an unmanned aerial vehicle. Turkish Journal of Engineering, 1(2), 82-87. https://doi.org/10.31127/tuje.316675
• Karataş, L., Alptekin, A., Kanun, E., & Yakar, M. (2022). Tarihi kârgir yapılarda taş malzeme bozulmalarının İHA fotogrametrisi kullanarak tespiti ve belgelenmesi: Mersin Kanlıdivane ören yeri vaka çalışması. İçel Dergisi, 2(2), 41-49.
• Karataş, L., Alptekin, A., Karabacak, A., Yakar, M. (2022). Detection and documentation of stone material deterioration in historical masonry buildings using UAV photogrammetry: A case study of Mersin Sarisih Inn. Mersin Photogrammetry Journal, 4(2), 53-61. https://doi.org/10.53093/mephoj.1198605
• Şasi, A., & Yakar, M. (2018). Photogrammetric modelling of Hasbey Dar'ülhuffaz (Masjid) using an unmanned aerial vehicle. International Journal of Engineering and Geosciences, 3(1), 6-11. https://doi.org/10.26833/ijeg.328919
• Çolak, A., Aktan, N., & Yılmaz, H. M. (2022). Modelling of its surroundings and Selime Cadhetral by UAV data. Advanced UAV, 2(1), 24-28.
• Banafaa, M., Pepeoğlu, Ö., Shayea, I., Alhammadi, A., Shamsan, Z., Razaz, M. A., ... & Al-Sowayan, S. (2024). A comprehensive survey on 5G-and-beyond networks with UAVs: Applications, emerging technologies, regulatory aspects, research trends and challenges. IEEE Access, 12, 7786 – 7826. https://doi.org/10.1109/ACCESS.2023.3349208
• Jiang, N., Wang, K., Peng, X., Yu, X., Wang, Q., Xing, J., ... & Han, Z. (2021). Anti-uav: a large-scale benchmark for vision-based uav tracking. IEEE Transactions on Multimedia, 25, 486-500. https://doi.org/10.1109/TMM.2021.3128047
• Zhao, J., Wang, G., Li, J., Jin, L., Fan, N., Wang, M., ... & Guo, Y. (2021). The 2nd anti-uav workshop & challenge: Methods and results. Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.2108.09909
• Zhao, J., Li, J., Jin, L., Chu, J., Zhang, Z., Wang, J., ... & Shengmei, J. S. (2023). The 3rd anti-uav workshop & challenge: Methods and results. Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.2305.07290
• Coluccia, A., Ghenescu, M., Piatrik, T., De Cubber, G., Schumann, A., Sommer, L., Klatte, J., Schuchert, T., Beyerer, J., Farhadi, M., Amandi, R., Aker, C., Kalkan, S., Saqib, M., Sharma, N., Daud, S., Makkah, K., & Blumenstein, M. (2017). Drone-vs-bird detection challenge at IEEE AVSS2017. IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) Drone-vs-Bird detection challenge, 1-6, Lecce, Italy.
• Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Mantegh, I. (2017). Drone-vs-bird detection challenge at IEEE AVSS2017. 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-8. https://doi.org/ 10.1109/AVSS.2017.8078464
• Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Rajashekar, S. (2021). Drone vs. bird detection: Deep learning algorithms and results from a grand challenge. Sensors, 21(8), 2824. https://doi.org/10.3390/s21082824
• Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Mantegh, I. (2021). Drone-vs-bird detection challenge at IEEE AVSS2021. 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-8. https://doi.org/10.1109/AVSS52988.2021.9663844
• Coluccia, A., Fascista, A., Schumann, A., Sommer, L., Dimou, A., Zarpalas, D., ... & Pavleski, D. (2022). Drone-vs-bird detection challenge at ICIAP 2021. In International Conference on Image Analysis and Processing, 410-421. https://doi.org/10.1007/978-3-031-13324-4_35
• Li, J., Murray, J., Ismaili, D., Schindler, K., & Albl, C. (2020). Reconstruction of 3D flight trajectories from ad-hoc camera networks. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1621-1628. https://doi.org/10.1109/IROS45743.2020.9341479
• Allahham, M. S., Al-Sa'd, M. F., Al-Ali, A., Mohamed, A., Khattab, T., & Erbad, A. (2019). DroneRF dataset: A dataset of drones for RF-based detection, classification and identification. Data in Brief, 26, 104313. https://doi.org/10.1016/j.dib.2019.104313
• Swinney, C. J., & Woods, J. C. (2021). RF detection and classification of unmanned aerial vehicles in environments with wireless interference. International Conference on Unmanned Aircraft Systems (ICUAS), 1494-1498. https://doi.org/10.1109/ICUAS51884.2021.9476867
• Glüge, S., Nyfeler, M., Ramagnano, N., Horn, C., & Schüpbach, C. (2023). Robust drone detection and classification from radio frequency signals using convolutional neural networks. 15th International Joint Conference on Computational Intelligence (IJCCI), 496-504. https://doi.org/10.5220/0012176800003595
• Medaiyese, O., Ezuma, M., Lauf, A., & Adeniran, A. (2022). Cardinal RF (CardRF): An outdoor UAV/UAS/drone RF signals with Bluetooth and WiFi signals dataset.
• Al-Sa’d, M. F., Al-Ali, A., Mohamed, A., Khattab, T., & Erbad, A. (2019). RF-based drone detection and identification using deep learning approaches: An initiative towards a large open source drone database. Future Generation Computer Systems, 100, 86-97. https://doi.org/10.1016/j.future.2019.05.007
• Al-Emadi, S., & Al-Senaid, F. (2020). Drone detection approach based on radio-frequency using convolutional neural network. 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), 29-34. 10.1109/ICIoT48696.2020.9089489
• Allahham, M. S., Khattab, T., & Mohamed, A. (2020). Deep learning for RF-based drone detection and identification: A multi-channel 1-D convolutional neural networks approach. IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), 112-117. https://doi.org/10.1109/ICIoT48696.2020.9089657
• Medaiyese, O. O., Syed, A., & Lauf, A. P. (2021). Machine learning framework for RF-based drone detection and identification system. 2nd International Conference on Smart Cities, Automation & Intelligent Computing Systems (ICON-SONICS), 58-64. https://doi.org/10.1109/ICON-SONICS53103.2021.9617168
• Nemer, I., Sheltami, T., Ahmad, I., Yasar, A. U. H., & Abdeen, M. A. (2021). RF-based UAV detection and identification using hierarchical learning approach. Sensors, 21(6), 1947. https://doi.org/10.3390/s21061947
• Zhao, Z., Du, Q., Yao, X., Lu, L., & Zhang, S. (2023). A Two-Dimensional Deep Network for RF-based Drone Detection and Identification Towards Secure Coverage Extension. In 2023 IEEE 98th Vehicular Technology Conference (VTC2023-Fall), 1-5. https://doi.org/10.1109/VTC2023-Fall60731.2023.10333485
• Nguyen, P., Kim, T., Miao, J., Hesselius, D., Kenneally, E., Massey, D., ... & Vu, T. (2019). Towards RF-based localization of a drone and its controller. Proceedings of the 5th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications, 21-26. https://doi.org/10.1145/3325421.3329766
• Unlu, E., Zenou, E., Riviere, N., & Dupouy, P. E. (2019). Deep learning-based strategies for the detection and tracking of drones using several cameras. IPSJ Transactions on Computer Vision and Applications, 11, 1-13. https://doi.org/10.1186/s41074-019-0059-x
• Singha, S., & Aydin, B. (2021). Automated drone detection using YOLOv4. Drones, 5(3), 95. https://doi.org/10.3390/drones5030095
• Zhai, X., Huang, Z., Li, T., Liu, H., & Wang, S. (2023). YOLO-Drone: an optimized YOLOv8 network for tiny UAV object detection. Electronics, 12(17), 3664. https://doi.org/10.3390/electronics12173664
• Seidaliyeva, U., Akhmetov, D., Ilipbayeva, L., & Matson, E. T. (2020). Real-time and accurate drone detection in a video with a static background. Sensors, 20(14), 3856. https://doi.org/10.3390/s20143856
• Carrio, A., Vemprala, S., Ripoll, A., Saripalli, S., & Campoy, P. (2018). Drone detection using depth maps. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), 1034-1037. https://doi.org/10.1109/IROS.2018.8593405
• Carrio, A., Tordesillas, J., Vemprala, S., Saripalli, S., Campoy, P., & How, J. P. (2020). Onboard detection and localization of drones using depth maps. IEEE Access, 8, 30480-30490. https://doi.org/10.1109/ACCESS.2020.2971938
• Akyon, F. C., Eryuksel, O., Ozfuttu, K. A., & Altinuc, S. O. (2021). Track boosting and synthetic data aided drone detection. 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-5. https://doi.org/10.1109/AVSS52988.2021.9663759
• Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014). Microsoft coco: Common objects in context. Computer Vision–ECCV 2014: 13th European Conference, Zurich, 13, 740-755. https://doi.org/10.1007/978-3-319-10602-1_48
• Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248-255. https://doi.org/10.1109/CVPR.2009.5206848