Evaluation of the Deep Q-Learning Models for Mobile Robot Path Planning Problem

Abstract

Search algorithms such as A* or Dijkstra are generally used to solve the path planning problem for mobile robots. However, these approaches require a map and their performance decreases in dynamic environments. These drawbacks have led researchers to work on dynamic path planning algorithms. Deep reinforcement learning methods have been extensively studied for this purpose and their use is expanding day by day. However, these studies mostly focus on training performance of the models, but not on inference. In this study, we propose an approach to compare the performance of the models in terms of path length, path curvature and journey time. We implemented the approach by using Python programming language two steps: inference and evaluation. Inference step gathers information of path planning performance; evaluation step computes the metrics regarding the information. Our approach can be tailored to many studies to examine the performances of trained models.

Keywords

• Deep Q-Learning
• mobile robots
• model inference
• path planning

References

1. [1] H. Aydemir, M. Tekerek, and M. Gök, “Complete coverage planning with clustering method for autonomous mobile robots”, Concurr. Comput. Pract. Exp., 2023, doi:10.1002/cpe.7830
2. [2] M. Gök, Ö. Ş. Akçam, and, M. Tekerek, “Performance Analysis of Search Algorithms for Path Planning”, Kahramanmaraş Sütçü İmam University Journal of Engineering Sciences, 26 (2), 379-394., doi:10.17780/ksujes.1171461
3. [3] T. P. Lillicrap et al., “Continuous control with deep reinforcement learning”, in 4th International Conference on Learning Representations, 2016, pp. 1-14.
4. [4] Y. Kato, K. Kamiyama, and K. Morioka, “Autonomous robot navigation system with learning based on deep Q-network and topological maps”, in 2017 IEEE/SICE International Symposium on System Integration, 2018, pp. 1040-1046.
5. [5] A. I. Karoly, P. Galambos, J. Kuti, and I. J. Rudas, “Deep Learning in Robotics: Survey on Model Structures and Training Strategies”, IEEE Trans. on Systems, Man, and Cybernetics: Systems, vol. 51, no. 1, pp. 266–279, 2021.
6. [6] H. Van Hasselt, “Double Q-learning”, in 24th Annual Conference on Neural Information Processing Systems, 2010, pp. 1–9.
7. [7] A. Kamalova, S. G. Lee, and S. H. Kwon, “Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System”, Applied Sciences (Switzerland), vol. 12, no. 18, 2022.
8. [8] J. Gao, W. Ye, J. Guo, and Z. Li, “Deep reinforcement learning for indoor mobile robot path planning”, Sensors, vol. 20, no. 19, 2020, pp. 1–15.

9. [9] Turtlebot3 ROBOTIS e-Manual. https://emanual.robotis.com/docs/en/platform/turtlebot3/machine_learning/ (accessed Sept. 15, 2023).
10. [10] J. Tsai, C. C. Chang, Y. C. Ou, B. H. Sieh, and Y. M. Ooi, “Autonomous Driving Control Based on the Perception of a Lidar Sensor and Odometer”, Applied Sciences (Switzerland), vol. 12, no. 15, 2022.
11. [11] T. Ribeiro, F. Gonçalves, I. Garcia, G. Lopes, and A. F. Ribeiro, “Q-Learning for Autonomous Mobile Robot Obstacle Avoidance”, in 19th IEEE International Conference on Autonomous Robot Systems and Competitions, 2019.
12. [12] M. C. Bingöl, (2021). Investigation of the Standard Deviation of Ornstein - Uhlenbeck Noise in the DDPG Algorithm. Gazi University Journal of Science Part C: Design and Technology, 9(2), 200-210. https://doi.org/10.29109/gujsc.872646
13. [13] Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Frcitas, “Dueling Network Architectures for Deep Reinforcement Learning”, in 33rd International Conference on Machine Learning, vol. 4, no. 9, 2016, pp. 2939–2947.
14. [14] R. Van Hoa, L. K. Lai, and L. T. Hoan, “Mobile Robot Navigation Using Deep Reinforcement Learning in Unknown Environments”, International Journal of Electrical and Electronics Engineering (SSRG-IJEEE), vol. 7, no. 8, 2020, pp. 15–20.
15. [15] U. Orozco-Rosas, K. Picos, J. J. Pantrigo, A. S. Montemayor, and A. Cuesta-Infante, ‘Mobile Robot Path Planning Using a QAPF Learning Algorithm for Known and Unknown Environments’, IEEE Access, vol. 10, no. August, 2022, pp. 84648–84663.
16. [16] M. Wu, Y. Gao, A. Jung, Q. Zhang, and S. Du, “The actor-dueling-critic method for reinforcement learning”, Sensors, vol. 19, no. 7, 2019, pp. 1–20.
17. [17] H. Aydemir, M. Tekerek, and M. Gök, “Examining of the effect of geometric objects on slam performance using ROS and Gazebo”, El-Cezeri Journal of Science and Engineering, vol. 8, no. 3, 2021, pp. 1441–1454.
18. [18] M. Luong and C. Pham, “Incremental Learning for Autonomous Navigation of Mobile Robots based on Deep Reinforcement Learning”, Journal of Intelligent & Robotic Systems, vol. 101, no. 1, 2021, pp. 1–11.
19. [19] M. F. R. Lee and S. H. Yusuf, “Mobile Robot Navigation Using Deep Reinforcement Learning”, Processes, vol. 10, no. 12, 2022.