Lane Detection for Autonomous Driving in a Video Game Environment

Abstract

To ensure comfortable and safe driving, the automotive industry has accelerated the development of autonomous vehicles in recent years. In the design of autonomous vehicles, challenging problems such as lane detection need to be solved. Convolutional neural networks, which show superior performance in many fields, have also been used in the lane detection problem. The datasets required to train CNN models are too large to be collected and labeled by manual effort. In this study, a method is proposed to automatically collect a labeled data set from the video game environment to be used in the detection of highway lanes. Different CNN models such as ResNet50, VGG16, Xception, and InceptionV3 networks are trained using the Transfer Learning method with 745,823 collected images. The images captured by the front vehicle camera are used as input, the coordinates of the points in the left and right lane and the center of the lane in the 2D plane in front of the vehicle and the angle of the vehicle are used as outputs. The performances of these models are tested and compared on the images collected from a road not used in the training set. According to the performance comparisons, ResNet50 performs best.

Keywords

• convolutional neural networks
• autonomous driving
• road lane detection

Project Number

2021.KB.FEN.005.

Video Oyunu Ortamında Otonom Sürüş İçin Şerit Tespiti

Abstract

Konforlu ve güvenli sürüşü sağlamak için otomotiv sektörü son yıllarda otonom araçların gelişimini hızlandırmıştır. Otonom araçların tasarımında şerit tespiti gibi zorlu problemlerin çözülmesi gerekmektedir. Birçok alanda üstün performans gösteren evrişimli sinir ağları şerit tespit problemlerinde de kullanılmıştır. CNN modellerini eğitmek için gerekli olan veri setleri, manuel çaba ile toplanıp etiketlenemeyecek kadar büyüktür. Bu çalışmada, otoyol şeritlerinin tespitinde kullanılacak etiketli bir veri setinin video oyunu ortamından otomatik olarak toplanması için bir yöntem önerilmiştir. ResNet50, VGG16, Xception ve InceptionV3 ağları gibi farklı CNN modelleri, toplanan 745,823 görsel ile Transfer Öğrenme yöntemi kullanılarak eğitilmiştir. Araç ön kamerası tarafından yakalanan görüntüler girdi olarak kullanılmış, aracın yol merkezine olan açısı ile birlikte aracın önündeki iki boyutlu düzlemde bulunan sol, sağ ve merkez şerit koordinatları çıktı olarak kullanılmıştır. Bu modellerin performansları eğitim setinde kullanılmayan bir otoyoldan toplanan görüntüler üzerinde test edilerek karşılaştırılmıştır. Performans karşılaştırmalarına göre en iyi performansı ResNet50 modeli vermektedir.

Keywords

• evrişimli sinir ağları
• otonom sürüş
• yol şerit tespiti

Supporting Institution

Dokuz Eylül Üniversitesi

Project Number

2021.KB.FEN.005.

References

1. Blade, A. (2022a). GTA V Native DB. Retrieved October 28, 2022 from http://www.dev-c.com/nativedb
2. Blade, A. (2022b). Script Hook V. Retrieved October 28, 2022 from https://www.dev-c.com/gtav/scripthookv
3. Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6), 679-698.
4. Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1251-1258).
5. Filipowicz, A. (2016). “Driving School II Video Games for Autonomous Driving,” M. Eng. thesis, Princeton University, New Jersey, USA.
6. Filipowicz, A., Liu, J., & Kornhauser, A. (2017). Learning to recognize distance to stop signs using the virtual world of Grand Theft Auto 5. In Transportation Research Board 96th Annual Meeting, 17-05456.
7. Gale, J. W. (2018). GTA Advanced Lane Finding. Retrieved October 28, 2022 from https://github.com/Will-J-Gale/GTA-Advanced-Lane-Finding
8. GTAV. (2022). Grand Theft Auto V. Retrieved October 28, 2022 from https://www.rockstargames.com/gta-v

9. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
10. Hough, P. V. (1962). U.S. Patent No. 3,069,654. Washington, DC: U.S. Patent and Trademark Office.
11. Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., ..... & Ng, A. Y. (2015). An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716.
12. Martinez, M., Sitawarin, C., Finch, K., Meincke, L., Yablonski, A., & Kornhauser, A. (2017). Beyond grand theft auto V for training, testing and enhancing deep learning in self driving cars. arXiv preprint arXiv:1712.01397.
13. Muller, U., Ben, J., Cosatto, E., Flepp, B., & Cun, Y. (2005). Off-road obstacle avoidance through end-to-end learning. Advances in neural information processing systems, 18.
14. Pakgohar, A., Tabrizi, R. S., Khalili, M., & Esmaeili, A. (2011). The role of human factor in incidence and severity of road crashes based on the CART and LR regression: a data mining approach. Procedia Computer Science, 3, 764-769.
15. Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016, October). Playing for data: Ground truth from computer games. In European conference on computer vision (pp. 102-118). Springer, Cham.
16. Shafaei, A., Little, J. J., & Schmidt, M. (2016). Play and learn: Using video games to train computer vision models. arXiv preprint arXiv:1608.01745.
17. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
18. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826).
19. Url-1 , date retrieved 28.10.2022.
20. Url-2 < https://tr.gta5-mods.com/scripts/hood-camera>, date retrieved 28.10.2022.
21. Url-3 < https://tr.gta5-mods.com/scripts/self-drive-plugin>, date retrieved 28.10.2022.
22. World Health Organization (WHO). (2018). Global Status Report on Road Safety 2018. Accessed: 28.10.2022. Retrieved from https://www.who.int/publications/i/item/9789241565684
23. Zou, Z., Shi, T., Li, W., Zhang, Z., & Shi, Z. (2020). Do game data generalize well for remote sensing image segmentation?. Remote Sensing, 12(2), 275.