A Novel Solution to the Real-Time Lane Detection and Tracking Problem for Autonomous Vehicles by Using Faster R-CNN and Mask R-CNN

Abstract

Autonomous vehicle technology has advanced in the automobile sector. Autonomous vehicle technology aims to make driving safer and reduce driver-caused traffic accidents. Autonomous driving technology work toward this. Lane detection and tracking are crucial to autonomous driving systems. Mostly the image processing techniques are mainly utilized for the lane detection in the literature. But, while performing image processing techniques for lane detection and tracking, two basic problems are mainly encountered. First one is image also needs to work with a specific area on the image to reduce the processing load and to work for the correct area. The region of interest (ROI) process is often used to filter the area to be worked from the image. However, since fixed coordinates are provided for this operation, the vehicle restricts the oper-ation of the vehicle in areas where it must be rotated. Second one is the weather conditions are very effective in the detection of lanes by utilizing image processing techniques. There are serious problems with image processing and detection from cloudy, sunny or momentary changes in the air. This study uses deep learning methods against these two basic problems. Using the Mask R-CNN and faster R-CNN algorithms together, these two basic problems for lane detection and tracking are eliminated and successfully implemented. The problem solved by two algorithms has been tested experimentally on a developed tool. Both the originally developed da-taset and the KITTI dataset were used separately in the model training carried out for the experimental tests. Both systems work well for lane detection and tracking, according to tests.

Keywords

• Autonomous vehicle
• Faster R-CNN
• Lane detection
• Mask R-CNN.

References

1. [1] Moujahid A, Bencherif M, Alomari O, et al. Machine learn-ing techniques in ADAS: A review. 2018 International Con-ference on Advances in Computing and Communication En-gineering (ICACCE); 2018; 235-242. https://doi.org/10.1109/ICACCE.2018.8441758.
2. [2] Divakarla KP, Emadi A, Razavi S. A cognitive advanced driver assistance systems architecture for autonomous-capable electrified vehicles. IEEE Trans Transp Electr. 2019; 5(1):48-58. https://doi.org/10.1109/TTE.2018.2870819
3. [3] Kim I-H, Bong J-H, Park J, Park S. Prediction of driver’s intention of lane change by augmenting sensor information using machine learning techniques. Sensors. 2017; 17(6):1350. https://doi.org/10.3390/s17061350.
4. [4] Ahmed HU, Huang Y, Lu P, Bridgelall R. Technology de-velopments and impacts of connected and autonomous ve-hicles: An overview. Smart Cities. 2022;5 (1):382-404. https://doi.org/10.3390/smartcities5010022.
5. [5] ATKINS. Research on the impacts of connected and auton-omous vehicles (CAVs) on traffic flow, stage 2: Traffic modelling and analysis technical report. Department for Transport; 2016.
6. [6] So J, Hwangbo J, Kim SH, Yun I. Analysis on autonomous vehicle detection performance according to various road ge-ometry settings. J Intell Transp Syst. 2023; 27(3):3 84-395. https://doi.org/10.1080/15472450.2022.2042280.
7. [7] Zhan H, Chen L. Lane detection image processing algorithm based on FPGA for intelligent vehicle. 2019 Chinese Auto-mation Congress (CAC); 2019; 1190-1196. https://doi.org/10.1109/CAC48633.2019.8996283.
8. [8] Yi SC, Chen YC, Chang CH. A lane detection approach based on intelligent vision. Comput Electr Eng. 2015; 42:23-29. https://doi.org/10.1016/j.compeleceng.2015.01.002.

9. [9] Zhu D, Song R, Chen H, Klette R, Xu Y. Moment-based multi-lane detection and tracking. Signal Process Image Commun. 2021;95:116230. https://doi.org/10.1016/j.image.2021.116230.
10. [10] Muthalagu R, Bolimera A, Kalaichelvi V. Lane detection technique based on perspective transformation and histo-gram analysis for self-driving cars. Comput Electr Eng. 2020;85:106653. https://doi.org/10.1016/j.compeleceng.2020.106653.
11. [11] Marzougui M, Alasiry A, Kortli Y, Baili J. A lane tracking method based on progressive probabilistic Hough transform. IEEE Access. 2020; 8: 84893-84905. https://doi.org/10.1109/ACCESS.2020.2991930.
12. [12] Chen W, Wang W, Wang K, Li Z, Li H, Liu S. Lane depar-ture warning systems and lane line detection methods based on image processing and semantic segmentation: A review. J Traffic Transp Eng. 2020; 7(6):748-774. https://doi.org/10.1016/j.jtte.2020.10.002.
13. [13] Perng JW, Hsu YW, Yang YZ, Chen CY, Yin TK. Develop-ment of an embedded road boundary detection system based on deep learning. Image Vis Comput. 2020;100:103935. https://doi.org/10.1016/j.imavis.2020.103935.
14. [14] Tang J, Li S, Liu P. A review of lane detection methods based on deep learning. Pattern Recognit. 2021;111:107623. https://doi.org/10.1016/j.patcog.2020.107623.
15. [15] Almeida T, Lourenço B, Santos V. Road detection based on simultaneous deep learning approaches. Robot Auton Syst. 2020;133:103605. https://doi.org/10.1016/j.robot.2020.103605.
16. [16] Neven D, Brabandere BD, Georgoulis S, Proesmans M, V Gool L. Towards end-to-end lane detection: An instance segmentation approach. 2018 IEEE Intelligent Vehicles Symposium (IV); 2018. 286-291. https://doi.org/10.1109/IVS.2018.8500547.
17. [17] Olgun MC, Baytar Z, Akpolat KM, Sahingoz OK. Autono-mous vehicle control for lane and vehicle tracking by using deep learning via vision. 2018 6th International Conference on Control Engineering & Information Technology (CEIT); 2018;1-7. https://doi.org/10.1109/CEIT.2018.8751764.
18. [18] Zhao Z, Wang Q, Li X. Deep reinforcement learning based lane detection and localization. Neurocomputing. 2020;413:328-338. https://doi.org/10.1016/j.neucom.2020.06.094.
19. [19] Ojha A, Sahu SP, Dewangan DK. Vehicle detection through instance segmentation using Mask R-CNN for intelligent ve-hicle system. 2021 5th International Conference on Intelli-gent Computing and Control Systems (ICICCS); 2021; 954-959. https://doi.org/10.1109/ICICCS51141.2021.9432374.
20. [20] Anak Undit HJ, Abu Hassan MF, Zin ZM. Vision-based unmarked road detection with semantic segmentation using Mask R-CNN for lane departure warning system. 2021 4th International Symposium on Agents, Multi-Agent Systems and Robotics (ISAMSR); 2021; 1-6. https://doi.org/10.1109/ISAMSR53229.2021.9567892.
21. [21] Ortataş FN, Kaya M. Performance evaluation of YOLOv5, YOLOv7, and YOLOv8 models in traffic sign detection. 2023 8th International Conference on Computer Science and Engineering (UBMK); 2023; 151-156. https://doi.org/10.1109/UBMK59864.2023.10286611.
22. [22] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hier-archies for accurate object detection and semantic segmenta-tion. 2014 IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR); 2014; 580-587. https://doi.org/10.1109/CVPR.2014.81.
23. [23] Girshick R. Fast R-CNN. 2015 IEEE International Confer-ence on Computer Vision (ICCV); 2015; 1440-1448. https://doi.org/10.1109/ICCV.2015.169.
24. [24] Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell. 2017; 39(6):1137-1149. https://doi.org/10.1109/TPAMI.2016.2577031.
25. [25] He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV); 2017; 2980-8. https://doi.org/10.1109/ICCV.2017.322.
26. [26] Ortataş FN, Çetın E. Lane tracking with deep learning: Mask RCNN and Faster RCNN. 2022 Innovations in Intelligent Systems and Applications Conference (ASYU); 2022; 1-5. https://doi.org/10.1109/ASYU56188.2022.9925296.
27. [27] Feng J, Wu X, Zhang Y. Lane detection based on deep learn-ing. 2018 11th International Symposium on Computational Intelligence and Design (ISCID); 2018; 315-318. https://doi.org/10.1109/ISCID.2018.00078.
28. [28] Pizzati F, Allodi M, Barrera A, García F. Lane detection and classification using cascaded CNNs. In: Moreno-Díaz R, Pichler F, Quesada-Arencibia A, editors. Computer Aided Systems Theory – EUROCAST 2019. Springer, Cham; 2020; 451-460. https://doi.org/10.1007/978-3-030-45096-0_12.
29. [29] Tabelini L, Berriel R, Paixão TM, Badue C, De Souza AF, Oliveira-Santos T. Keep your eyes on the lane: Real-time at-tention-guided lane detection. In: 2021 IEEE/CVF Confer-ence on Computer Vision and Pattern Recognition (CVPR); 2021; 294-302. https://doi.org/10.1109/CVPR46437.2021.00036
30. [30] Liu R, Yuan Z, Liu T, Xiong Z. End-to-end lane shape pre-diction with transformers. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV); 2021;3693-3701. https://doi.org/10.1109/WACV48630.2021.00374
31. [31] Liu L, Chen X, Zhu S, Tan P. CondLaneNet: A top-to-down lane detection framework based on conditional convolution. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 2021;3753-3762. https:// doi.org/10.1109/ICCV48922.2021.00375.
32. [32] Baek SW, Kim MJ, Suddamalla U, et al. Real-time lane de-tection based on deep learning. J Electr Eng Technol. 2022;17(2):655-664. https://doi.org/10.1007/s42835-021-00902-6.
33. [33] Khan MA-M, Haque MF, Hasan KR, et al. LLDNet: A light-weight lane detection approach for autonomous cars using deep learning. Sensors. 2022; 22(15): 5595. https://doi.org/10.3390/s22155595.
34. [34] Oğuz E, Küçükmanisa A, Duvar R, Urhan O. A deep learn-ing-based fast lane detection approach. Chaos Solitons Frac-tals. 2022;155:111722. https://doi.org/10.1016/j.chaos.2021.111722.
35. [35] Tian Y, Gelernter J, Wang X, et al. Lane marking detection via deep convolutional neural network. Neurocomputing. 2018;280:46-55. https://doi.org/10.1016/j.neucom.2017.09.098.