Autonomous Navigation in Search and Rescue Simulated Environment using Deep Reinforcement Learning

Abstract

Human assisted search and rescue (SAR) robots are increasingly being used in zones of natural disasters, industrial accidents, and civil wars. Due to complex terrains, obstacles, and uncertainties in time availability, there is a need for these robots to have a certain level of autonomy to act independently for approaching certain SAR tasks. One of these tasks is autonomous navigation. Previous approaches to develop autonomous or semi-autonomous SAR navigating robots use heuristics-based methods. These algorithms, however, require environment-related prior knowledge and enough sensing capabilities, which are hard to maintain due to restrictions of size and weight in highly unstructured environments such as collapsed buildings. This study approaches the problem of autonomous navigation using a modified version of the Deep Q-Network algorithm. Unlike the classical usage of the entire game screen images to train the agent, our approach uses only the images captured by the agent's low-resolution camera to train the agent for navigating through an arena avoiding obstacles and to reach a victim. This approach is a much more relevant way of decision making in complex, uncertain contexts; since in real-world SAR scenarios, it is almost impossible to have the area's full information to be used by SAR teams. We simulated a SAR scenario, which consists of an arena full of randomly generated obstacles, a victim, and an autonomous SAR robot. The simulation results show that the agent was able to reach the victim in 56% of the evaluation episodes after 400 episodes of training.

Keywords

• Deep Reinforcement Learning
• Autonomous Navigation
• Autonomous Search and Rescue
• Simulation

References

1. [1] C. Kenny, Why Do People Die In Earthquakes? The Costs, Benefits, And Institutions Of Disaster Risk Reduction In Developing Countries. The World Bank, 2009.
2. [2] “NFPA 1670, Standard on Operations and Training for Technical Rescue Incidents - National Fire Protection Association - Google Books.” (accessed Aug. 03, 2020).
3. [3] “Rescue: Technical Rescue Program Development Manual - Google Books.” https://books.google.com.tr/books/about/Rescue_Technical_Rescue_Program_Developm.html?id=2VNSswEACAAJ&redir_esc=y (accessed Aug. 03, 2020).
4. [4] R. R. Murphy et al., “Search and Rescue Robotics,” in Springer Handbook of Robotics, Springer Berlin Heidelberg, 2008, pp. 1151–1173.
5. [5] T. Bräunl, “Localization and Navigation,” in Embedded Robotics, Springer Berlin Heidelberg, 2008, pp. 241–269.
6. [6] K. N. McGuire, G. C. H. E. de Croon, and K. Tuyls, “A comparative study of bug algorithms for robot navigation,” Rob. Auton. Syst., vol. 121, p. 103261, Nov. 2019.
7. [7] S. Waharte and N. Trigoni, “Supporting search and rescue operations with UAVs,” in Proceedings - EST 2010 - 2010 International Conference on Emerging Security Technologies, ROBOSEC 2010 - Robots and Security, LAB-RS 2010 - Learning and Adaptive Behavior in Robotic Systems, 2010, pp. 142–147, DOI: 10.1109/EST.2010.31.
8. [8] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015, DOI: 10.1038/nature14236.

9. [9] D. Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, Jan. 2016, DOI: 10.1038/nature16961.
10. [10] J. Xiao, A. Owens, and A. Torralba, “SUN3D: A database of big spaces reconstructed using SfM and object labels,” in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 1625–1632, DOI: 10.1109/ICCV.2013.458.
11. [11] I. Toschi, P. Rodríguez-Gonzálvez, F. Remondino, S. Minto, S. Orlandini, and A. Fuller, “Accuracy evaluation of a mobile mapping system with advanced statistical methods,” in International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 2015, vol. 40, no. 5W4, pp. 245–253.
12. [12] P. Kim, J. Park, Y. K. Cho, and J. Kang, “UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments,” Autom. Constr., vol. 106, p. 102918, Oct. 2019, DOI: 10.1016/j.autcon.2019.102918.
13. [13] “ICARUS Project.” https://icarus2020.eu/ (accessed Aug. 03, 2020).
14. [14] L. Marconi et al., “The SHERPA project: smart collaboration between humans and ground-aerial robots for improving rescuing activities in alpine environments,” 2012, DOI: 10.1109/SSRR.2012.6523905.
15. [15] J. G. C. Zuluaga, J. P. Leidig, C. Trefftz, and G. Wolffe, “Deep Reinforcement Learning for Autonomous Search and Rescue,” in Proceedings of the IEEE National Aerospace Electronics Conference, NAECON, Dec. 2018, vol. 2018-July, pp. 521–524.
16. [16] O. Michel, “Cyberbotics Ltd. Webots TM : Professional Mobile Robot Simulation,” 2004. Accessed: Aug. 03, 2020. [Online]. Available: http://www.cyberbotics.com.