Optimizing Visual Instruction Detection in Autonomous Mobile Robots Using Yolov8 and Tensorrt Acceleration

Abstract

In this study, the deep learning-based detection performance of instructions for the vehicle was examined through images obtained from a camera mounted on a mobile robotic system. The aim is to enhance the detection performance of a differential robot equipped with a robotic arm in recognizing various visual instructions it may encounter in the field. Traffic lights, direction signs, and speed limit signs were selected as the visual materials to be introduced to the robotic system. By utilizing the YOLOv8 object detection model on the embedded AI computer onboard the vehicle and leveraging the TensorRT accelerator, deep learning-based image processing achieved a high frame rate of 33 FPS and an mAP50 accuracy of 96.6%. This study highlights the advantages and challenges of integrating advanced detection models into autonomous robotic platforms, contributing to future improvements in reliability and efficiency.

Keywords

• Deep learning
• Differential robot
• Sign recognition
• TensorRT

References

1. Barba-Guaman L, Eugenio Naranjo J, Ortiz A. 2020. Deep learning framework for vehicle and pedestrian detection in rural roads on an embedded GPU. Electronics, 9(4): 589.
2. Cai ZX, Gu MQ. 2013. Traffic sign recognition algorithm based on shape signature and dual-tree complex wavelet transform. J Cent South Univ, 20(2): 433–439.
3. Chen J, Jia K, Chen W, Lv Z, Zhang R. 2022. A real-time and high-precision method for small traffic-signs recognition. Neural Comput Appl, 34(3): 2233–2245.
4. Çınarer G. 2024. Deep learning based traffic sign recognition using YOLO algorithm. Düzce Univ. J Sci Tech, 12(1): 219–229.
5. Dalal N, Triggs B. 2005. Histograms of oriented gradients for human detection. Soc Conf Comput Vision Pattern Recog (CVPR’05), San Diego, CA, USA, 1: 886–893.
6. Flores-Calero M, Astudillo CA, Guevara D, Maza J, Lita BS, Defaz B, Ante JS, Zabala-Blanco D, Armingol Moreno JM. 2024. Traffic sign detection and recognition using YOLO object detection algorithm: A Systematic Rev. Mathematics, 12(2): 1–31.
7. Girshick R, Donahue J, Darrell T, Jitendra M. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. IEEE Conf Comput Vision Pattern Recog., 580–587.
8. Gudigar A, Chokkadi S, Raghavendra U, Acharya UR. 2017. Multiple thresholding and subspace based approach for detection and recognition of traffic sign. Mult Tools Appl, 76(5), 6973–6991.

9. Guney E, Bayilmis C, Cakan B. 2022. An implementation of real-time traffic signs and road objects detection based on mobile GPU platforms. IEEE Access, 10: 86191–86203.
10. Han Y, Wang F, Wang W, Li X, Zhang J. 2024. YOLO-SG: Small traffic signs detection method in complex scene. J Supercomp, 80(2): 2025–2046.
11. Hassan IA, Abed IA, Al-Hussaibi WA. 2024. Path planning and trajectory tracking control for two-wheel mobile robot. J Robot Cont (JRC), 5(1): 1–15.
12. Hechri A, Mtibaa A. 2020. Two‐stage traffic sign detection and recognition based on SVM and convolutional neural networks. IET Image Process, 14(5): 939–946.
13. Houben S, Stallkamp J, Salmen J, Schlipsing M, Igel C. 2013. Detection of traffic signs in real-world images: The German traffic sign detection benchmark. Int Joint Conf Neural Networks, Dallas, TX, USA, pp: 1–8.
14. Jeong EJ, Kim J, Tan S, Lee J, Ha S. 2022. Deep learning inference parallelization on heterogeneous processors with TensorRT. IEEE Embed Syst Lett, 14(1): 15–18.
15. Jin Y, Fu Y, Wang W, Guo J, Ren C, Xiang X. 2020. Multi-feature fusion and enhancement single shot detector for traffic sign recognition. IEEE Access, 8: 38931–38940.
16. Jooshin HK, Nangir M, Seyedarabi H. 2024. Inception-YOLO: Computational cost and accuracy improvement of the YOLOv5 model based on employing modified CSP, SPPF, inception modules. IET Image Process, 18(8): 1985–1999.
17. Kounte MR, Shri CvA., Harshvardhan V, Kumari A, Dhruv S. 2022. Design and development of autonomous driving car using nvidiajetson nano developer kit. 8-9 Oct. 2022, 4th Int Conf Cybernetics, Cognition and Machine Learning Appl (ICCCMLA), Goa, India, pp: 486–489.
18. Krizhevsky A, Sutskever I, Hinton GE. 2017. ImageNet classification with deep convolutional neural networks. Commun ACM, 60(6): 84–90.
19. Kumar BA, Majji M, Marni HJ, Ateeq M, Koduru S, Maddi SM 2023. A deep transfer learning approach for enhanced traffic sign recognition in autonomous vehicles with NVIDIA jetson nano. Int. Conf Sustain. Emerg Innov Eng Technol (ICSEIET), Ghaziabad, India, pp: 692–698.
20. Lai G, Morris T. 2019. TensorRT inference with TensorFlow. GPU Tech Conf, pp: 75.
21. Li Y, Zhang Z, Yuan C, Hu J. 2024. Easily deployable real-time detection method for small traffic signs. J Intell Fuzzy Syst, 46: 8411–8424.
22. Lin Y, Chu L, Hu J, Hou Z, Li J, Jiang J, Zhang Y. 2024. Progress and summary of reinforcement learning on energy management of MPS-EV. Heliyon, 10(1): e23014.
23. Mahmoud MAB, Guo P. 2019. A novel method for traffic sign recognition based on DCGAN and MLP with PILAE algorithm. IEEE Access, 7: 74602–74611.
24. Maldonado-Bascón S, Lafuente-Arroyo S, Gil-Jiménez P, Gómez-Moreno H, López-Ferreras F. 2007. Road-sign detection and recognition based on support vector machines. IEEE Trans Intell Transp Syst, 8(2): 264–278.
25. Ozcan K, Sharma A, Knickerbocker S, Merickel J, Hawkins N, Rizzo M. 2020. Road weather condition estimation using fixed and mobile based cameras. Advances in Comput. Vision: Proc. of the 2019 Comput. Vision Conf (CVC), 1 (1): 192–204.
26. Ren S, He K, Girshick R, Sun J. 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 39(6): 1137–1149.
27. Sarvajcz K, Ari L, Menyhart J. 2024. AI on the road: NVIDIA jetson nano-powered computer vision-based system for real-time pedestrian and priority sign detection. Appl Sci, 14(4): 1440.
28. Satti SK, Rajareddy GNV, Mishra K, Gandomi AH. 2024. Potholes and traffic signs detection by classifier with vision transformers. Sci Rep, 14(1): 1–18.
29. Shamta I, Demir BE. 2024. Development of a deep learning-based surveillance system for forest fire detection and monitoring using UAV. PLoS ONE, 19(3): e0299058.
30. Shustanov A, Yakimov P. 2017. CNN design for real-time traffic sign recognition. Procedia Eng, 201: 718–725.
31. Stallkamp J, Schlipsing M, Salmen J, Igel C. 2011. The German traffic sign recognition benchmark: A multi-class classification competition. 31 July - 05 August 2011, Int. Joint Conf Neural Networks, San Jose, CA, USA, pp: 1453–1460.
32. Terakura K, Chang Q, Miyazaki J. 2024. Acceleration of neural network inference for embedded gpu systems. Int Conf Big Data Smart Comput, Bangkok, Thailand, pp: 361–362.
33. Terven J, Córdova-Esparza DM, Romero-González JA. 2023. A comprehensive review of YOLO architectures in computer vision: from YOLOv1 to YOLOv8 and YOLO-NAS. Machine Learn Knowl Extr, 5(4): 1680–1716.
34. Thasai Y, Kim P, Ynag Z. 2009. Generalized traffic sign detection model for developing a sign inventory. J Comp Civil Eng, 23(5): 266–276.
35. Wali SB, Abdullah MA, Hannan MA, Hussain A, Samad SA, Ker PJ, Mansor MB. 2019. Vision-based traffic sign detection and recognition systems: Current trends and challenges. Sensors, 19(9): 2093.
36. Wali SB, Hannan MA, Hussain A, Samad SA. 2015. An automatic traffic sign detection and recognition system based on colour segmentation, shape matching, and svm. Math Probl Eng, 2015(1): 250461.
37. You F, Zhang R, Lie G, Wang H, Wen H, Xu J. 2015. Trajectory planning and tracking control for autonomous lane change maneuver based on the cooperative vehicle infrastructure system. Expert Syst Appl, 42(14): 5932–5946.
38. Yuan X, Hao X, Chen H, Wei X. 2014. Robust traffic sign recognition based on color global and local oriented edge magnitude patterns. IEEE Trans Intell Transp Syst, 15(4): 1466–1474.
39. Zhang RH, He ZC, Wang HW, You F, Li KN. 2017. Study on self-tuning tyre friction control for developing main-servo loop integrated chassis control system. IEEE Access, 5: 6649–6660.