ROS-Based Path Planning for Autonomous Underwater Vehicles Using Deep Q-Networks Algorithm

Abstract

This study focuses on addressing the limited maneuverability typically associated with underwater vehicles. The study investigates Reinforcement Learning (RL) algorithms on Tethys LRAUV (Long-Range Autonomous Underwater Vehicle), developed by the Monterey Bay Aquarium Research Institute [1]. Experiments were conducted in the Gazebo simulation environment [2]. The experiments utilized the Gazebo underwater simulation environment [3], which models the Tethys LRAUV developed by Paper et al. RL algorithms were examined in real-time, replacing traditional controllers. The Deep Q-Networks (DQN) algorithm was employed to determine the trajectory of LRAUV. Control of the Gazebo underwater vehicle was facilitated using the Robot Operating System (ROS). Results indicate the potential advantages of RL-based algorithms compared to traditional controllers. The study concludes that the Deep Q-Networks algorithm can be efficiently utilized for real-time control of LRAUV models, and the training environment required for Deep Q-Networks can be achieved in the simulation environment.

Keywords

• Robot Operating System
• Artificial Intelligence
• Reinforcement Learning
• Simulation
• Autonomous Underwater Vehicles

Su Altı Otonom Araçlarda Derin Q-Ağları Algoritması Kullanılarak ROS Tabanlı Yol Planlama

Öz

Su altı araçları genellikle sınırlı hareket kabiliyetine sahiptir. Bu çalışma, bu problemin çözümüne odaklanmaktadır. Çalışmada Monterey Körfezi Akvaryumu Araştırma Enstitüsü tarafından geliştirilen Tethys UMOSA (Uzun Menzilli Otonom Su Altı Aracı) [1] üzerinde Yeniden Güçlendirme Öğrenmesi (RL) algoritmasının kullanılması incelenmiştir. Deneyler Gazebo simülasyon ortamında [2] gerçekleştirilmiştir. Yapılan deneylerde, Paper ve arkadaşları tarafından geliştirilen Tethys UMOSA’nın modellendiği Gazebo su altı simülasyon ortamı [3] kullanılmıştır. Geleneksel denetleyicilerin yerine gerçek zamanlı olarak Yeniden Güçlendirme Öğrenmesi (RL) algoritmalarının kullanılması incelenmiştir. UMOSA’nın yörüngesini belirlemek için Derin Q-Ağları (DQN) algoritması kullanılmıştır. Gazebo simülasyon ortamındaki su altı aracının kontrolü Robot İşletim Sistemi (ROS) kullanılarak sağlanmıştır. Sonuçlar geleneksel denetleyicilere kıyasla RL tabanlı algoritmaların potansiyel avantajlarını göstermektedir. Çalışma sonucunda UMOSA modellerinde Derin Q-Ağları algoritmasının gerçek zamanlı kontrol için verimli olarak kullanılabileceği ve simülasyon ortamında Derin Q-Ağları için gereken eğitim ortamının gerçekleştirilebilecği gözlemlenmiştir.

Anahtar Kelimeler

• Robot İşletim Sistemi
• Yapay Zeka
• Pekiştirmeli Öğrenme
• Simülasyon
• Su Altı Otonom Araçları

Kaynakça

1. [1] [Watson, S.; Duecker, D.A.; Groves, K. Localisation of Unmanned Underwater Vehicles (UUVs) in Complex and Confined Environments: A Review. Sensors 2020
2. [2] Phillips, A.B., vd. (2023). "Autosub Long Range 1500: A continuous 2000 km field trial." Ocean Engineering, 280, 114626.
3. [3] Godin, M.A., vd. (2011). "Real-time sensing of upwelling from a moving autonomous platform." Limnology and Oceanography: Methods, 9(1), 1-13.
4. [4] Zhang, Y., vd. (2012). "Using AUVs to study frontal dynamics." Journal of Field Robotics, 29(6), 1035-1048.
5. [5] Kukulya, A., vd. (2016). "AUVs in the Arctic: A platform for interdisciplinary science." OCEANS 2016 MTS/IEEE Monterey.
6. [6] Qu, Xingru, et al. "A Deep Reinforcement Learning-Based Path-Following Control Scheme for an Uncertain Under-Actuated Autonomous Marine Vehicle." Journal of Marine Science and Engineering 11.9 (2023): 1762.
7. [7] Ma, Hui, Xiaokai Mu, and Bo He. "Adaptive navigation algorithm with deep learning for autonomous underwater vehicle." Sensors 21.19 (2021): 6406.
8. [8] Liu, Tao, Yuli Hu, and Hui Xu. "Deep reinforcement learning for vectored thruster autonomous underwater vehicle control." Complexity 2021 (2021): 1-25.

9. [9] Zhang, Jialei, et al. "Approach-angle-based three-dimensional indirect adaptive fuzzy path following of under-actuated AUV with input saturation." Applied Ocean Research 107 (2021): 102486.
10. [10] Ma, H.; Mu, X.; He, B. Adaptive Navigation Algorithm with Deep Learning for Autonomous Underwater Vehicle. Sensors 2021, 21, 6406
11. [11] Tian, Q.; Wang, T.; Song, Y.; Wang, Y.; Liu, B. Autonomous Underwater Vehicle Path Tracking Based on the Optimal Fuzzy Controller with Multiple Performance Indexes. J. Mar. Sci. Eng. 2023, 11, 463
12. [12] Fang, Ming-Chung, et al. "Applying the self-tuning fuzzy control with the image detection technique on the obstacle-avoidance for autonomous underwater vehicles." Ocean Engineering 93 (2015): 11-24.
13. [13] B. W. Hobson, J. G. Bellingham, B. Kieft, R. McEwen, M. Godin, and Y. Zhang, “Tethys-class long range AUVs - extending the endurance of propeller-driven cruising AUVs from days to weeks,” in 2012 IEEE/OES Autonomous Underwater Vehicles (AUV), 2012, pp. 1–8 [14] Open Source Robotics Foundation, “Gazebo.” [Çevrimiçi]. Erişim: https://gazebosim.org
14. [15] Player, T. R., Chakravarty, A., Zhang, M. M., Raanan, B. Y., Kieft, B., Zhang, Y., & Hobson, B. (2023, May). From Concept to Field Tests: Accelerated Development of Multi-AUV Missions Using a High-Fidelity Faster-than-Real-Time Simulator. In 2023 IEEE International Conference on Robotics and Automation (ICRA) (pp. 3102-3108). IEEE.
15. [16] Panda JP, Mitra A, Warrior HV. A review on the hydrodynamic characteristics of autonomous underwater vehicles. Proceedings of the Institution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment. 2021;235(1):15-29.
16. [17] Tonkal, Ö., & Polat, H. (2021). Traffic Classification and Comparative Analysis with Machine Learning Algorithms in Software Defined Networks. Gazi University Journal of Science Part C: Design and Technology, 9(1), 71-83.
17. [18] Dayan, Peter, and C. J. C. H. Watkins. "Q-learning." Machine learning 8.3 (1992): 279-292.
18. [19] Tesauro, Gerald. "Td-gammon: A self-teaching backgammon program." Applications of neural networks. Boston, MA: Springer US, 1995. 267-285.