Overtaking with deep reinforcement learning methods in autonomous vehicles

Öz

Deep reinforcement algorithms that combine reinforcement learning and deep learning approaches are used in challenging autonomous vehicle tasks. Passing the vehicle in front is one of the most challenging autonomous vehicle tasks due to the different types of subtasks involved. Recent studies in the literature use the curriculum learning approach with deep reinforcement learning to solve challenging tasks. In this study, 12 models, half of which have undergone a curriculum learning approach, are trained in a uniquely constructed environment with commonly used deep Q-networks, advantage actor critic and proximal policy optimization algorithms. The evaluation of the models is based on both the training process and the testing of the models in the environment. In the study, successful models were trained with deep Q-networks and proximal policy optimization methods, although not for all models. Among the successful models, the performance of a deep Q-network model was improved with curriculum learning, showing the positive impact of the approach.

Anahtar Kelimeler

• Deep reinforcement learning
• Curriculum learning
• Autonomous vehicles

Proje Numarası

FKB-2019-9388

Otonom araçlarda derin pekiştirmeli öğrenme yöntemleri ile sollama

Öz

Pekiştirmeli öğrenme ile derin öğrenme yaklaşımlarını birleştiren derin pekiştirmeli öğrenme algoritmaları zorlu otonom araç görevlerinde kullanılmaktadır. Öndeki aracı geçme, içerisinde barındırdığı farklı türden alt görevler nedeni ile en zorlu otonom araç görevlerinden biridir. Literatürdeki güncel çalışmalar zorlu görevleri çözmek için müfredat öğrenme yaklaşımını derin pekiştirmeli öğrenme ile kullanmaktadır. Bu çalışmada, özgün olarak oluşturulmuş ortamda, yaygın olarak kullanılan derin Q-ağları, avantaj aktör kritik ve proksimal politika optimizasyonu algoritmaları ile yarısı müfredat öğrenme yaklaşımına uğramış 12 model eğitilmiştir. Modellerin değerlendirilmesinde modellerin eğitim süreci ve modellerin ortamda test edilmesi birlikte kullanılmıştır. Çalışmada, tüm modellerde olmasa da derin Q-ağları ve proksimal politika optimizasyonu yöntemleri ile başarılı modeller eğitilmiştir. Başarılı modeller içerisinde müfredat öğrenimi ile bir derin Q-ağları modelinin performansı artırılarak yaklaşımın olumlu etkisi görülmüştür.

Anahtar Kelimeler

• Derin pekiştirmeli öğrenme
• Müfredat öğrenme
• Otonom araçlar

Destekleyen Kurum

Erciyes Üniversitesi Bilimsel Araştırma Projeleri Birimi

Proje Numarası

FKB-2019-9388

Teşekkür

Erciyes Üniversitesi Bilimsel Araştırma Projeleri Birimi tarafından FKB-2019-9388 kodu ile desteklenmiştir. Teşekkür ederiz.

Kaynakça

1. B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Peréz, Deep reinforcement learning for autonomous driving: a survey. IEEE Transactions on Intelligent Transportation Systems, 23(6), 4909-4926, 2021. https://doi.org/10.1109/TITS.2021.3054625.
2. Sürücü Eğitimi. https://www.taksimsurucukursu.com/assets/pdf/surucuegitimkitabi.pdf, Erişim: 11 Aralık 2023.
3. J. Janai, F. Güney, A. Behl, and A. Geiger, Computer vision for autonomous vehicles: Problems, Datasets and State of the Art. 2017, arXiv: 1704.05519. https://doi.org/10.48550/arXiv.1704.05519.
4. C. Chen, A. Seff, A. Kornhauser, and J. Xiao, Deepdriving: Learning affordance for direct perception in autonomous driving. Proceedings of the IEEE International Conference on Computer Vision, 2722-2730, Santiago, Chile, 2015.
5. D. Loiacono, A. Prete, P. L. Lanzi, and L. Cardamone, Learning to overtake in TORCS using simple reinforcement learning. IEEE Congress on Evolutionary Computation, pp. 1-8, Barcelona, Spain, 2010.
6. D. C. K. Ngai and N. H. C. Yung, A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers. IEEE Transactions on Intelligent Transportation Systems, 12(2), 509-522, 2011. https://doi.org/10.1109/TITS.2011.2106158.
7. X. Li, X. Xu, and L. Zuo, Reinforcement learning based overtaking decision-making for highway autonomous driving. 2015 Sixth International Conference on Intelligent Control and Information Processing (ICICIP), pp. 336-342, Wuhan, China, 2015.
8. T. Xia and Z. Han, Path planning using reinforcement learning and objective data. Master’s Thesis, University of Gothenburg, Gothenburg, Sweden, 2017.

9. M. Kaushik, V. Prasad, K. M. Krishna, and B. Ravindran, Overtaking maneuvers in simulated highway driving using deep reinforcement learning. 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1885-1890, Changshu, China, 2018.
10. X. Li, X. Qiu, J. Wang, and Y. Shen, A deep reinforcement learning based approach for autonomous overtaking. 2020 IEEE International Conference on Communication Workshops (ICC Workshops), pp. 1-5, Dublin, Ireland, 2020.
11. Y. Song, H. Lin, E. Kaufmann, P. Dürr, and D. Scaramuzza, Autonomous overtaking in gran turismo sport using curriculum reinforcement learning. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 9403-9409, Xi’an, China, 2021.
12. J. Liu, H. Li, Z.Yang, S. Dang, and Z. Huang, Deep dense network-based curriculum reinforcement learning for high-speed overtaking. IEEE Intelligent Transportation Systems Magazine, 15(1), 453-466, 2022. https://doi.org/10.1109/MITS.2022.3174410.
13. E. Aslan, M. A. Arserim, and A. Uçar, Development of Push-Recovery control system for humanoid robots using deep reinforcement learning. Ain Shams Engineering Journal, 14(10), 102167, 2023. https://doi.org/10.1016/j.asej.2023.102167.
14. V. Uc-Cetina, N. Navarro-Guerrero, A. Martin-Gonzalez, C. Weber, and S. Wermter, Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56(2), 1543-1575, 2023. https://doi.org/10.1007/s10462-022-10205-5.
15. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 2018.
16. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, Curriculum learning. International Conference on Machine Learning, pp. 41-48, Montreal, Canada, 2009.
17. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
18. D. Soydaner, A comparison of optimization algorithms for deep learning. International Journal of Pattern Recognition and Artificial Intelligence, 34(13), 2020. https://doi.org/10.1142/S0218001420520138.
19. S. S. Mousavi, M. Schukat, and Enda Howley, Deep reinforcement learning: an overview. In Proceedings of SAI Intelligent Systems Conference (IntelliSys), pp. 426-440, London, UK, 2016.
20. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278 – 2324, 1998. https://doi.org/10.1109/5.726791.
21. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing atari with deep reinforcement learning. 2013, arXiv: 1312.5602. https://doi.org/10.48550/arXiv.1312.5602.
22. C. J. Watkins and P. Dayan, Q-learning. Machine Learning, 8, 279-292, 1992. https://doi.org/10.1007/BF00992698.
23. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533, 2015. https://doi.org/10.1038/nature14236.
24. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928-1937, New York, USA, 2016.
25. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. 2017, arXiv: 1707.06347. https://doi.org/10.48550/arXiv.1707.06347.
26. Anaconda Software Distrubition. https://www.anaconda.org/, Erişim: 18 Temmuz 2023.
27. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, OpenAI Gym. 2016, arXiv: 1606.01540. https://doi.org/10.48550/arXiv.1606.01540.
28. Gymnasium. https://gymnasium.farama.org, Erişim: 18 Temmuz 2023.
29. OpenCV-Python. https://pypi.org/project/opencv-python. Erişim: 18 Temmuz 2023.
30. NumPy. https://numpy.org, Erişim: 18 Temmuz 2023.
31. Stable-Baselines3. https://www.ai4europe.eu/sites/default/files/2021-06/README_5.pdf, Erişim: 18 Temmuz 2023.
32. PyTorch Conv2d. https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html, Erişim: 18 Temmuz 2023.
33. Tensorflow TensorBoard. https://github.com/tensorflow/tensorboard, Erişim: 18 Temmuz 2023.