Thermal SLAM: Harnessing Temperature Variations to Enhance Object Detection and Tracking Performance

Öz

Thermal Simultaneous Localization and Mapping (SLAM) is a burgeoning field that collects robotics, computer vision, and thermal imaging. In this paper, we tried to present a thorough review of recent advancements in thermal SLAM, with a focus on its role in enhancing object detection and tracking. For better performance in low light, resistance to obstructions, and accuracy in bad weather, thermal SLAM systems work better with visual-based SLAM systems because they use changes in temperature in the environment. The review paper explains the fundamental principles of SLAM, including sensor technologies, data fusion techniques, and mapping algorithms. It then explores the methodologies used for object detection and tracking within the Thermal SLAM framework, encompassing classical approaches and deep learning techniques tailored for thermal imagery analysis. Additionally, the paper discusses challenges and limitations specific to thermal SLAM, such as thermal drift, sensor noise, and calibration issues, while also identifying potential areas for future research. The paper provides a comprehensive survey of applications that utilize thermal SLAM for object detection and tracking across various domains, including autonomous navigation, surveillance, search and rescue operations, and environmental monitoring. It synthesizes case studies and experimental results from relevant literature to demonstrate the effectiveness and practicality of thermal SLAM in complex scenarios where traditional visual-based methods struggle. Overall, this review emphasizes the role of thermal SLAM in advancing autonomous systems and enabling robust object detection and tracking in challenging environments. Examining recent developments, challenges, and applications, it sheds light on the progress made in this field.

Anahtar Kelimeler

• SLAM
• Thermal camera
• Object detection
• Robust localization
• Object tracking

Kaynakça

1. Bijelic M, Gruber T, Mannan F, Kraus F, Ritter W, Dietmayer K, Heide F. 2020. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 14-19, Seattle, WA, USA, pp: 11682-11692.
2. Borges PVK, Vidas S. 2016. Practical infrared visual odometry. IEEE Trans Intell Transp Syst, 17(8): 2205-2213.
3. Brunner C, Peynot T. 2010. Visual metrics for the evaluation of sensor data quality in outdoor perception. In: Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop, September 28-30, Baltimore, MD, USA, pp: 55-61.
4. Brunner C, Peynot T. 2014. Perception quality evaluation with visual and infrared cameras in challenging environmental conditions. In: Experimental Robotics: The 12th International Symposium on Experimental Robotics, March 17- 20, Tokyo, Japan, pp: 231-240.
5. Brunner C, Peynot T, Vidal-Calleja T, Underwood J. 2013. Selective combination of visual and thermal imaging for resilient localization in adverse conditions: Day and night, smoke and fire. J Field Robot, 30(4): 641-666.
6. Cadena C, Carlone L, Carrillo H, Latif Y, Scaramuzza D, Neira J, Leonard JJ. 2016. Past, present, and future of Simultaneous Localization and Mapping: Toward the robust-perception age. IEEE Trans Robot, 32(6): 1309-1332.
7. Calonder M, Lepetit V, Strecha C, Fua P. 2010. Brief: Binary robust independent elementary features. In: Proceedings of the Computer Vision– ECCV 2010:11th European Conference on Computer Vision. Springer- Verlag, September 5- 11, Heraklion, Crete, Greece, pp: 778-792.
8. Chen X, Liu L, Zhang J, Shao W. 2021. Infrared image denoising based on the variance-stabilizing transform and the dual-domain filter. Digit Signal Process, pp: 113-128.

9. Chen Z, Maffra F, Chli M. 2017. Only look once, mining distinctive landmarks from convnet for visual place recognition. In: Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 24-28, Vancouver, BC, Canada, pp: 6435-6441.
10. Debeunne C, Vivet D. 2020. A review of Visual-LiDAR fusion based Simultaneous Localization and Mapping. Sensors, 20(7): 20-68.
11. Delaune J, Hewitt R, Lytle L, Sorice C, Thakker R, Matthies L. 2019. Thermal-inertial odometry for autonomous flight throughout the night. In: Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November 4-8, Macau, China, pp: 1090- 1097.
12. Dellaert F, Kaess M. 2006. Square Root SLAM: Simultaneous Localization and Mapping via square root information smoothing. Int J Robot Res, 25(12): 1181-1203.
13. Geiger A, Ziegler J, Stiller C. 2011. Stereoscan: Dense 3D reconstruction in real-time. In: Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), June 5-9, Baden-Baden, Germany, pp: 963-968.
14. Grisetti G, Kümmerle R, Stachniss C, Burgard W. 2010. A tutorial on graph-based SLAM. IEEE Intell Transp Syst Mag, 2(4): 31-43.
15. Hoshi M, Hara Y, Nakamura S. 2024. Graph-based SLAM using wall detection and floor plan constraints without loop closure. ROBOMECH J, 11(1), 18-32.
16. Johansson J, Solli M, Maki A. 2016. An evaluation of local feature detectors and descriptors for infrared images. In: Proceedings of the Computer Vision–ECCV 2016 Workshops: October 8-10, Amsterdam, Netherlands, Proceedings, Part III, pp: 142.
17. Keil C, Gupta A, Kaveti P, Singh H. 2024. Towards Long-Term SLAM on Thermal Imagery. arXiv preprint arXiv: 2403.19885.
18. Khattak S, Papachristos C, Alexis K. 2019a. Keyframe-based direct thermal–inertial odometry. In: Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), May 20-24, Montreal, QC, Canada, pp: 6197-6203.
19. Khattak S, Papachristos C, Alexis K. 2019b. Visual- thermal landmarks and inertial fusion for navigation in degraded visual environments. In: Proceedings of the 2019 IEEE Aerospace Conference, March 2-9, Big Sky, MT, USA, pp: 1-9.
20. Khattak S, Papachristos C, Alexis K. 2020. Keyframe- based thermal–inertial odometry. J Field Robot, 37(4): 552-579.
21. Kim C, Sakthivel R, Chung WK. 2007. Unscented FastSLAM: A robust algorithm for the simulataneous Localization and Mapping problem. In: Proceedings of the 2007 IEEE International Conference on Robotics and Automation, April 10-14, Rome, Italy, pp: 2439-2445.
22. Leonard JJ, Durrant-Whyte HF. 1991. Mobile robot localization by tracking geometric beacons. In: Proceedings of the 1991 IEEE International Conference on Robotics and Automation, April 9- 11, Sacramento, CA, USA, pp: 376-382.
23. Li J, Li Z, Feng Y, Liu Y, Shi G. 2019. Development of a human–robot hybrid intelligent system based on brain teleoperation and deep learning SLAM. IEEE Trans Autom Sci Eng, 16(4): 1664-1674.
24. Lu C. 2020. Stripe non-uniformity correction of infrared images using parameter estimation. Infr Phys Tech, 107(1), pp: 1-7.
25. Lu Y, Lu G. 2021. Superthermal: Matching thermal as visible through thermal feature exploration. IEEE Robot Autom Lett, 6(2): 2690-2697.
26. Lucas BD, Kanade T. 1981. An iterative image registration technique with an application to stereo vision. In: Proceedings of the IJCAI'81: 7th International Joint Conference on Artificial Intelligence, August 24-28, Vancouver, BC, Canada, pp: 674-679.
27. Maddern W, and Vidas S. 2012. Towards robust night and day place recognition using visible and thermal imaging. In: Proceedings of the RSS 2012 Workshop: Beyond laser and vision: Alternative sensing techniques for robotic perception, July 9, Sydney, Australia, pp: 1-8.
28. Montemerlo M. 2002. FastSLAM: A factored solution to the Simultaneous Localization and Mapping problem. In: Proceedings of AAAI02, July 28- August 1, Edmonton, Canada, pp: 593-598.
29. Montemerlo M, Thrun S, Koller D, Wegbreit B. 2003. FastSLAM 2.0: An improved particle filtering algorithm for Simultaneous Localization and Mapping that provably converges. In: Proceedings of the IJCAI, August 9-15, Acapulco, Mexico, pp: 1151-1156.
30. Mouats T, Aouf N, Chermak L, Richardson MA. 2015. Thermal stereo odometry for UAVs. IEEE Sens J, 15(11): 6335-6347.
31. Mouats T, Aouf N, Nam D, Vidas S. 2018. Performance evaluation of feature detectors and descriptors beyond the visible. J Intell Robot Syst, 92: 33-63.
32. Mouats T, Aouf N, Sappa AD, Aguilera C, Toledo R. 2014. Multispectral stereo odometry. IEEE Trans Intell Transp Syst, 16(3): 1210-1224.
33. Mur-Artal R, Montiel JMM, Tardos JD. 2015. ORB- SLAM: A versatile and accurate monocular SLAM system. IEEE Trans Robot, 31(5): 1147-1163.
34. Papachristos C, Mascarich F, Alexis K. 2018. Thermal-inertial localization for autonomous navigation of aerial robots through obscurants. In: Proceedings of the 2018 International Conference on Unmanned Aircraft Systems (ICUAS), June 12-15, Dallas, TX, USA, pp: 395-401.
35. Poujol J, Aguilera CA, Danos E, Vintimilla BX, Toledo R, Sappa AD. 2016. A visible-thermal fusion-based monocular visual odometry. In: Proceedings of Robot 2015 Second Iberian Robotics Conference: Advances in Robotics, November 19-21, Lisbon, Portugal, pp: 213-225.
36. Rasmussen N. D, Morse B. S, Goodrich M. A, Eggett D. 2009. Fused visible and infrared video for use in wilderness search and rescue. In: Proceedings of the 2009 Workshop on Applications of Computer Vision (WACV), December 7-8, Snowbird, UT, USA, pp: 1-8.
37. Saputra MRU, De Gusmao PP, Lu CX, Almalioglu Y, Rosa S, Chen C, Trigoni N. 2020. Deeptio: A deep thermal-inertial odometry with visual hallucination. IEEE Robot Autom Lett, 5(2): 1672- 1679.
38. Saputra MRU, Lu CX, de Gusmao PPB, Wang B, Markham A, Trigoni N. 2021. Graph-based thermal-inertial SLAM with probabilistic neural networks. IEEE Trans Robot, 38(3): 1875-1893.
39. Shin YS, Kim A. 2019. Sparse depth enhanced direct thermal-infrared SLAM beyond the visible spectrum. IEEE Robot Autom Lett, 4(3): 2918- 2925.
40. Taketomi T, Uchiyama H, Ikeda S. 2017. Visual SLAM algorithms: A survey from 2010 to 2016. IPSJ Trans Comput Vis Appl, 9: 1-11.
41. Tang J, Ericson L, Folkesson J, Jensfelt P. 2019. GCNv2: Efficient correspondence prediction for real-time SLAM. IEEE Robot Autom Lett, 4(4): 3505-3512.
42. Tourani A, Bavle H, Sanchez-Lopez JL, Voos H. 2022. Visual slam: What are the current trends and what to expect? Sensors, 22(23): 9297-9326.
43. van Manen BR, Sluiter V, Mersha AY. 2023. FirebotSLAM: Thermal SLAM to increase situational awareness in smoke-filled environments. Sensors, 23(17): 7611-7636.
44. Vidas S, Sridharan S. 2012. Hand-held monocular SLAM in thermal-infrared. In: Proceedings of the 2012 12th International Conference on Control Automation Robotics and Vision (ICARCV), December 5-7, Guangzhou, China, pp: 500-506.
45. Wang S, Clark R, Wen H, Trigoni N. 2017. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. In: Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), May 29-June 3, Singapore, pp: 2043-2050.
46. Wu Y, Wang L, Zhang L, Bai Y, Cai Y, Wang S, Li Y. 2023. Improving autonomous detection in dynamic environments with robust monocular thermal SLAM system. ISPRS J Phot Remote Sens, 203: 265-284.
47. Zhang J, and Singh S. 2014. LOAM: Lidar odometry and mapping in real-time. In: Proceedings of Robotics: Science and Systems (RSS), July 12-16, Berkeley, CA, USA, pp: 109-117.
48. Zhao S, Singh D, Sun H, Jiang R, Gao Y, Wu T, Xu J. 2023. Subt-mrs: A subterranean, multi-robot, multi-spectral and multi-degraded dataset for robust SLAM. arXiv preprint arXiv:2307.07607.
49. Zhao S, Wang P, Zhang H, Fang Z, Scherer S. 2020. Tp-tio: A robust thermal-inertial odometry with deep thermal point. In: Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 25-29, Las Vegas, NV, USA, pp: 10321-10328.