Deep Learning Based Object Detection with Unmanned Aerial Vehicle Equipped with Embedded System

Year 2024, , 15 - 25, 26.02.2024

Ertugrul Kırac , Sunullah Özbek

https://doi.org/10.30518/jav.1356997

Cited By: 2

Abstract

This study aims to introduce an Unmanned Aerial Vehicle (UAV) platform capable of performing real-time object detection and classification tasks using computer vision techniques in the field of artificial intelligence. Previous scientific research reveals the utilization of two different methods for object detection and classification via UAVs. One of these methods involves transmitting the acquired UAV images to a ground control center for processing, whereafter the processed data is relayed back to the UAV. The other approach entails transferring images over the internet to a cloud system, where image processing is conducted, and the resultant data is subsequently sent back to the UAV. This allows the UAV to autonomously perform predefined tasks. Enabling the UAV with autonomous decision-making capabilities and the ability to perform object detection and classification from recorded images requires an embedded artificial intelligence module. The ability of the UAV to utilize image processing technologies through embedded systems significantly enhances its object detection and classification capabilities, providing it with a significant advantage. This enables the UAV to be used more effectively and reliably in various tasks. In the proposed approach, image processing was achieved by mounting a Raspberry Pi 4 and camera on the UAV. Additionally, a Raspberry Pi-compatible 4G/LTE modem kit was used to provide remote intervention capability, and the Coral Edge TPU auxiliary processor was used to increase object detection speed. The TensorFlow Library and the SSD MobilNetV2 convolutional neural network model were used for image processing. During test flights, accuracy values of approximately 96.3% for car detection and 96.2% for human detection were achieved.

Keywords

UAV, Deep Learning, Computer Vision, Conventional Neural Network, Object Detection

References

• Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y. and Zheng, X. (2016). TensorFlow: A system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16), 2-4 November, Savannah, GA, USA, 265–283.
• Afzal, U. and Mahmood, T. (2013). Using predictive analytics to forecast drone attacks in Pakistan. 5th International Conference on Information and Communication Technologies, 17-18 November, Tianjin, China, 1-6.
• Akbari, Y., Almaadeed, N., Al-maadeed, S. and Elharrouss, O. (2021). Applications, databases and open computer vision research from drone videos and images: a survey. Artificial Intelligence Review, 54(5), 3887- 3938.
• Akhloufi, M. A., Couturier, A. and Castro, N. A. (2021). Unmanned Aerial Vehicles for Wildland Fires: Sensing, Perception, Cooperation and Assistance. Drones, 5(1), 15.
• Ariza-Sentís, M., Baja, H., Vélez, S. and Valente, J. (2023). Object detection and tracking on UAV RGB videos for early extraction of grape phenotypic traits. Computers and Electronics in Agriculture, 211, 108051.
• Atoev, S., Kwon, K. -R., Lee, S. -H. and Moon, K. -S. (2017). Data analysis of the MAVLink communication protocol. 2017 International Conference on Information Science and Communications Technologies (ICISCT), 2-4 November, Tashkent, Uzbekistan, 1-3.
• Bai, J. and Fei, J. (2020). Research and Implementation of Handwritten Numbers Recognition System Based on Neural Network and Tensorflow Framework. Journal of Physics: Conference Series, 1576(1), 012029.
• Bisong, E. (2019). Google Colaboratory. In: Building Machine Learning and Deep Learning Models on Google Cloud Platform. Apress, Berkeley, 59-64.
• Boudjit, K. and Ramzan, N. (2022). Human detection based on deep learning YOLO-v2 for real-time UAV applications. Journal of Experimental & Theoretical Artificial Intelligence, 34(3), 527-544.
• Bouguettaya, A., Zarzour, H., Kechida, A. and Taberkit, A. M. (2022). Deep learning techniques to classify agricultural crops through UAV imagery: a review. Neural Computing and Applications, 34(12), 9511-9536.
• Buric, M., Pobar, M. and Ivasic-Kos, M. (2018). Ball Detection Using Yolo and Mask R-CNN. International Conference on Computational Science and Computational Intelligence (CSCI), 12-14 December, Las Vegas, NV, USA, 319-323.
• Canedo, D. and Neves, A. J. (2019). Facial expression recognition using computer vision: A systematic review. Applied Sciences, 9(21), 4678.
• Dhillon, A., Verma, G.K. (2020). Convolutional neural network: a review of models, methodologies and applications to object detection. Progress Artif. Intell. 9(2), 85–112.
• Domozi , Z., Stojcsics , D., Benhamida , A., Kozlovszky, M. and Molnar, A. (2020). Real time object detection for aerial search and rescue missions for missing persons. 15th International Conference of System of Systems Engineering (SoSE), 2-4 June, Budapest, Hungary, 519-524.
• Dong K., Zhou, C., Ruan, Y. and Y. Li. (2020). MobileNetV2 Model for Image Classification. 2nd International Conference on Information Technology and Computer Application (ITCA), 18-20 December, Guangzhou, China, 476-480.
• Dong, K., Zhou, C., Ruan, Y. and Li, Y. (2020). MobileNetV2 Model for Image Classification. 2nd International Conference on Information Technology and Computer Application (ITCA), 18-20 December, Guangzhou, China, 476-480.
• Firmansyah, N. W., Arizal, F. W. and Sudarmanto, J. A. (2021). Use of FPV Drones for Sports Documentaries. ICADECS International Conference on Art, Design, Education and Cultural Studies (ICADECS), 29 July, Malang, Indonesia, 368-376.
• Goerzen, C., Kong, Z. and Mettle, B. (2010). Survey of Motion Planning Algorithms from the Perspective of Autonomous UAV Guidance. Journal of Intelligent and Robotic Systems, 57(1), 65-100.
• Greco, G., Lucianaz, C., Bertoldo, S. and Allegretti, M. (2015). A solution for monitoring operations in harsh environment: A RFID reader for small UAV. International Conference on Electromagnetics in Advanced Applications (ICEAA), 7-11 September, Torino, Italy, 859-862.
• Gupta, P., Pareek, B., Singal, G. and Vijay Rao, D. (2022) Edge device based Military Vehicle Detection and Classification from UAV. Multimedia Tools and Applications, 81(14), 19813–19834.
• Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
• Jafri, R., Ali, S. A., Arabnia, H. R. and Fatima, S. (2014). Computer vision-based object recognition for the visually impaired in an indoors environment: a survey. The Visual Computer, 30, 1197-1222.
• Jain, A., Ramaprasad, R., Narang, P., Mandal, M., Chamola, V., Yu, F. R. and Guizan, M. (2021). AI-enabled object detection in UAVs: challenges, design choices, and research directions. IEEE Network, 35(4), 129-135.
• Jalled, F. and Voronkov, I. (2016). Object Detection using Image Processing.
• Jindal, V., Narayan Singh, S., & Suvra Khan, S. (2022). Facial Recognition with Computer Vision. In Machine Intelligence and Data Science Applications: Proceedings of MIDAS, 313-330.
• Khdier, H. Y., Jasim, W. M. and Aliesawi , S. A. (2021). Deep Learning Algorithms based Voiceprint Recognition System in Noisy Environment. Journal of Physics: Conference Series, 1804(1), 012042.
• Kinaneva , D., Hristov , G., Raychev, J. and Zahariev, P. (2019). Early Forest Fire Detection Using Drones and Artificial Intelligence. 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 20-24 May, Ruse, Bulgaria, 1060-1065.
• Konaite, M., Owolawi, P. A., Mapayi, T., Malale, V., Odeyem, K., Aiyetoro, G. and Ojo, J. S. (2021). Smart Hat for the blind with Real-Time Object Detection using Raspberry Pi and TensorFlow Lite. International Conference on Artificial Intelligence and its Applications (ICARTI), 2-4 November, Bagatelle, Mauritius, 1-6.
• Kwak, J. and Sung, Y. (2018). Autonomous UAV Flight Control for GPS-Based Navigation, IEEE Access, 6, pp. 37947-37955.
• Lee, J., Wang, J., Crandall, D.J., Šabanović, S. and Fox, G.C. (2017). Real-Time, Cloud-Based Object Detection for Unmanned Aerial Vehicles. 2017 First IEEE International Conference on Robotic Computing (IRC), 36-43.
• Li, C., Sun, X. and Cai, J. (2019). Intelligent Mobile Drone System Based on Real-Time Object Detection. Journal on Artificial Intelligence, 1(1), 1-8.
• Li, Y., Liu, M. and Jiang, D. (2022). Application of Unmanned Aerial Vehicles in Logistics: A Literature Review. Sustainability, 14(21), 1-18.
• Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P. and Zitnick, C. L. (2014). Microsoft COCO: Common Objects in Context. Computer Vision – ECCV 2014 13th European Conference, 6-12 September, Zurich, Switzerland, 740-755.
• Liu, H., Yu, Y., Liu, S. and Wang, W. (2022). A Military Object Detection Model of UAV Reconnaissance Image and Feature Visualization. Applied Sciences, 12(23), 12236.
• Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. and Berg, A. C. (2016). Ssd: Single shot multibox detector. Computer Vision–ECCV 14th European Conference, 11-14 October, Amsterdam, Netherlands, 21-37.
• Meier, L., Tanskanen, P., Fraundorfer, F. and Pollefeys, M. (2011). PIXHAWK: A system for autonomous flight using onboard computer vision. 2011 IEEE International Conference on Robotics and Automation, 9-13 May, Shanghai, China, 2992-2997.
• Moffatt, A., Platt, E., Mondragon, B., Kwok, A., Uryeu, D. and Bhandari, S. (2020). Obstacle detection and avoidance system for small UAVs using a LiDAR. 2020 International Conference on Unmanned Aircraft Systems (ICUAS), 9-12 June, Athens, Greece, 633-640.
• Nex, F. and Remondino, F. (2014). UAV for 3D mapping applications: A review. Applied Geomatics, vol. 6(1), 1- 15.
• Nousi, P., Mademlis, I., Karakostas, I., Tefas, A. and Pitas, I. (2019, August). Embedded UAV real-time visual object detection and tracking. In 2019 IEEE International Conference on Real-time Computing and Robotics (RCAR), 4-9 August, Irkutsk, Russia, 708-713.
• O’Shea, K. and Nash, R. (2015). An introduction to convolutional neural networks. arXiv:1511.08458.
• Quiñonez, Y., Lizarraga, C., Peraza, J. and Zatarain, O. (2020). Image Recognition in UAV videos using Convolutional Neural Networks. IET Software, 14(2), 176-181.
• Radovic, M., Adarkwa, O. and Wang, Q. (2017). Object Recognition in Aerial Images Using Convolutional Neural Networks Journal of Imaging, 3(2), 21.
• Ragland, K. and Tharcis, P. (2014). A survey on object detection, classification and tracking methods. Int. J. Eng. Res. Technol, 3(11), 622-628.
• Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen L. C. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18-23 June, Utah, USA, 4510-4520.
• Schlegel, D. (2015). Deep machine learning on Gpu. University of Heidelber-Ziti, 12.
• Schnipke, E., Reidling, S., Meiring, J., Jeffers, W., Hashemi, M., Tan, R., Nemati, A. and Kumar, M., 2015. Autonomous Navigation of UAV through GPS-Denied Indoor Environment with Obstacles. AIAA Infotech at Aerospace, 5-9 January, Kissimmee, Florida, 0715.
• Singha, S. and Aydin B. 2021. Automated Drone Detection Using YOLOv4. Drones, 5(3), 95.
• Soekhoe, D., Van Der Putten, P. and Plaat, A. (2016). On the impact of data set size in transfer learning using deep neural networks. In Advances in Intelligent Data Analysis XV: 15th International Symposium, 13-15 October, Stockholm, Sweden, , 50-60.
• Sun, R. (2019). Optimization for deep learning: theory and algorithms. arXiv preprint arXiv:1912.08957.
• Sun, Y. and Kist, A.M. (2021). Deep learning on edge TPUs.
• Szolga, L. A. (2021). On Flight Real Time Image Processing by Drone Equipped with Raspberry Pi4. 27th International Symposium for Design and Technology in Electronic Packaging (SIITME), 27-30 October, Timișoara, Romania, 334-337.
• Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C. and Liu, C. (2018). A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, 4-7 October, Rhodes, Greece, 270-279.
• Wang, Y. E., Wei, G. Y. and Brooks, D. (2019). Benchmarking TPU, GPU, and CPU platforms for deep learning. arXiv preprint arXiv:1907.10701.
• Wiley, V. and Lucas, T. (2018). Computer vision and image processing: a paper review. International Journal of Artificial Intelligence Research, 2(1), 29-36.
• Xin, M. and Wang, Y. (2019). Research on image classification model based on deep convolution neural network. EURASIP Journal on Image and Video Processing, 2019(1), 1-13.
• Xu, W. (2021). Efficient Distributed Image Recognition Algorithm of Deep Learning Framework TensorFlow. Journal of Physics: Conference Series, 2066(1), 012070.
• Yong, S. -P. and Yeong, Y. -C. (2018). Human Object Detection in Forest with Deep Learning based on Drone’s Vision. 4th International Conference on Computer and Information Sciences (ICCOINS), 13-14 August, Lumpur, Malaysia, 1-5.
• Zela, A., Klein, A., Falkner, S., & Hutter, F. (2018). Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906.
• Zhang, C., Yang, T. and Yang, J. (2022). Image Recognition of Wind Turbine Blade Defects Using Attention- Based MobileNetv1-YOLOv4 and Transfer Learning. Sensors, vol. 22, 6009.
• Zhou, Z. H. (2021). Machine learning. Springer Nature.