End-to-end, real time and robust behavioral prediction module with ROS for autonomous vehicles

Abstract

In the world where urbanization and population density are increasing, transportation methods are also diversifying and the use of unmanned vehicles is becoming widespread. In order for unmanned vehicles to perform their tasks autonomously, they need to be able to perceive their own position, the environment and predict the possible movements/routes of environmental factors, similar to living things. In autonomous vehicles, it is extremely important for the safety of the vehicle and the surrounding factors to be able to predict the future position of the objects around it with high performance so that the vehicle can plan correctly. Due to the stated reasons, the behavioral prediction module is a very important component for autonomous vehicles, especially in moving environments. In this study, fast and successful robotic behavioral prediction module has been developed to enable the autonomous vehicle to plan more safely and successfully.

Keywords

• Behavioral prediction
• trajectory prediction
• autonomous vehicles
• ROS

References

1. Koubaa, A., Robot Operating System (ROS): The Complete Reference (Volume 2), Springer, 2017, https://doi.org/10.1007/978-3-319-54927-9.
2. Hintjens, P., ZeroMQ: Messaging for Many Applications, O’REILLY, CA, 2013.
3. Macenski, S., Foote, T., Gerkey, B., Lalancette, C., Woodall, W., Robot operating system 2: Design, architecture, and uses in the wild, Sci. Robot., 7 (66) (2022), https://doi.org/10.48550/arXiv.2211.07752.
4. Salzmann, T., Ivanovic, B., Chakravarty, P., and Pavone, M., Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data, European Conference on Computer Vision, 12363 (2020), 683-700, https://doi.org/10.1007/978-3-030-58523-5_40.
5. Huang, Y., Du, J., Yang, Z., Zhou, Z., Zhang, L. and Chen, H., A survey on trajectory prediction methods for autonomous driving, IEEE Trans. Intell. Veh., 7 (3) (2022), https://doi.org/10.1109/TIV.2022.3167103.
6. Gulzar, M., Muhammad, Y. and Muhammad, N., A survey on motion prediction of pedestrians and vehicles for autonomous driving, IEEE Access, 9 (2021), 137957-137969, https://doi.org/10.1109/ACCESS.2021.3118224.
7. Lin, C. F. and Ulsoy, A. G., Vehicle dynamics and external disturbance estimation for vehicle path prediction, IEEE Trans. Control Syst. Technol., 8 (3) (2000), 508-518, https://doi.org/10.1109/87.845881.
8. Lefèvre, S., Vasquez, D. and Laugier, C., A survey on motion prediction and risk assessment for intelligent vehicles, ROBOMECH J., 1 (1) (2014), 1-14, https://doi.org/10.1186/s40648-014-0001-z.

9. Schöller, C. Aravantinos, Lay, V. F. and Knoll, A., What the constant velocity model can teach us about pedestrian motion prediction, IEEE Robot. Autom. Lett., 5 (2) (2020), 1696-1703, https://doi.org/10.48550/arXiv.1903.07933.
10. Ammoun, S. and Nashashibi, F., Real time trajectory prediction for collision risk estimation between vehicles, IEEE 5th International Conference on Intelligent Computer Communication and Processing, (2009), 417-422, https://doi.org/10.1109/ICCP.2009.5284727.
11. Schubert, R., Richter, E. and Wanielik, G., Comparison and evaluation of advanced motion models for vehicle tracking, 11th International Conference on Information Fusion, (2008), 1-6.
12. Lytrivis, P., Thomaidis, G. and Amditis, A., Cooperative path prediction in vehicular environments, 11th International IEEE Conference on Intelligent Transportation Systems, (2008), 803-808, https://doi.org/10.1109/ITSC.2008.4732629.
13. Batz, T., Watson, K. and Beyerer, J., Recognition of dangerous situ ations within a cooperative group of vehicles, IEEE Intelligent Vehicles Symposium, (2009), 907-912, https://doi.org/10.1109/IVS.2009.5164400.
14. Kumar, P., Perrollaz, M., Lefevre, S. and Laugier, C., Learning-based approach for online lane change intention prediction, IEEE Intelligent Vehicles Symposium (IV), (2013), 797-802, https://doi.org/10.1109/IVS.2013.6629564.
15. Qiao, S., Shen, D., Wang, X., Han, N. and Zhu, W., A self-adaptive parameter selection trajectory prediction approach via hidden markov models, IEEE Trans. Intell. Transp. Syst., 16 (1) (2015), 284-296, https://doi.org/10.1109/TITS.2014.2331758.
16. Deng, Q. and Soffker, D., Improved driving behaviors prediction based on fuzzy logic hidden markov model (fl-hmm), IEEE Intelligent Vehicles Symposium (IV), (2018), 2003-2008, https://doi.org/10.1109/IVS.2018.8500533.
17. Gindele, T., Brechtel, S. and Dillmann, R., A probabilistic model for estimating driver behaviors and vehicle trajectories in traffic environments, 13th Int. IEEE Conf. Intell. Transp. Syst., (2010), 1625-1631, https://doi.org/10.1109/ITSC.2010.5625262.
18. Lee, N., Choi, W., Vernaza, P., Chor, C. B., Torr, P. H. S., and Chandraker, M. K., DESIRE: distant future prediction in dynamic scenes with interacting agents, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 2165-2174, https://doi.ieeecomputersociety.org/10.1109/CVPR.2017.233.
19. Dai, S., Li, L. and Li, Z., Modeling vehicle interactions via modified LSTM models for trajectory prediction, IEEE Access, 7 (2019), 38287-38296, https://doi.org/10.1109/ACCESS.2019.2907000.
20. Sun, L., Zhan, W. and Tomizuka, M., Probabilistic prediction of interactive driving behavior via hierarchical inverse reinforcement learning, IEEE 21st International Conference on Intelligent Transportation Systems (ITSC), (2018), 2111-2117, https://doi.org/10.1109/ITSC.2018.8569453.
21. Kuefler, A., Morton, J., Wheeler, T. and Kochenderfer, M., Imitating driver behavior with generative adversarial networks, IEEE Intelligent Vehicles Symposium (IV), (2017), 204-211, https://doi.org/10.1109/IVS.2017.7995721.
22. Choi, S., Kim, J. and Yeo, H., Trajgail: Generating urban vehicle trajectories using generative adversarial imitation learning, Transp. Res. Part C Emerg. Technol., 128 (2021), 103091, https://doi.org/10.48550/arXiv.2007.14189.
23. Wulfmeier, M., Ondruska, P. and Posner, I., Maximum entropy deep in verse reinforcement learning, (2015), https://doi.org/10.48550/arXiv.1507.04888.
24. You, C., Lu, J., Filev, D. and Tsiotras, P., Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning, Robot. Auton. Syst., 114 (2019), 1-18, https://doi.org/10.1016/j.robot.2019.01.003.
25. Jung, C. and Shim, D. H., Incorporating multi-context into the traversability map for urban autonomous driving using deep inverse reinforcement learning, IEEE Robot. Autom. Lett., 6 (2) (2021), 1662-1669, https://doi.org/10.1109/LRA.2021.3059628.
26. Geiger, A., Lenz P. and Urtasun, R., Are we ready for autonomous driving? The KITTI vision benchmark suite, Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2012), 3354-3361, https://doi.org/10.1109/CVPR.2012.6248074.
27. Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G. and Beijbom, O., nuScenes: A multimodal dataset for autonomous driving, CVPR, (2020), 11618-11628, https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.01164.
28. Chang, M.-F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D. and Hays, J., Argoverse: 3D Tracking and Forecasting With Rich Maps, IEEE Conference on Computer Vision and Pattern Recognition, (2019), 8748-8757, https://doi.org/10.1109/CVPR.2019.00895.
29. Coifman, B. A critical evaluation of the next generation simulation (NGSIM) vehicle trajectory dataset, Trans. Res. B Methodol., 105 (2017), 362-377, https://doi.org/10.1016/j.trb.2017.09.018.
30. Caesar, H., Kabzan, J., Tan, K., nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles, CVPR ADP3 workshop, (2021), https://doi.org/10.48550/arXiv.2106.11810.
31. Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G. and Beijbom, O., nuScenes: A multimodal dataset for autonomous driving, CVPR, 2020.
32. Enyen, Deep Trajectory Prediction, (2017), https://github.com/enyen/Deep-Trajectory Prediction.
33. Deo, N. and Trivedi, M. M., Convolutional social pooling for vehicle trajectory prediction, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (2018), 1549-15498, https://doi.org/10.1109/ CVPRW.2018.00196.
34. Lefkopoulos, V., Menner, M., Domahidi, A. and Zeilinger, M. N., Interaction-aware motion prediction for autonomous driving: A multiple model kalman filtering scheme, IEEE Robotics and Automation Letters, 6 (1) (2021), 80-87, https://doi.org/10.1109/LRA.2020.3032079.
35. Deo, N., Rangesh, A. and Trivedi, M. M., How would surround vehicles move? a unified framework for maneuver classification and motion prediction, IEEE Trans. Intell. Veh., 3 (2) (2018), 129-140, https://doi.org/10.1109/TIV.2018.2804159.
36. Deo, N. and Trivedi, M. M., Multi-modal trajectory prediction of surrounding vehicles with maneuver based lstms, IEEE Intelligent Vehicles Symposium (IV), (2018), 1179- 1184, https://doi.org/10.1109/IVS.2018.8500493.
37. Tang C. and Salakhutdinov, R. R., Multiple futures prediction, Adv. Neural Inf. Process. Sys., 32 (2019), 15424-15434, https://doi.org/10.48550/arXiv.1911.00997.
38. Deo, N. and Trivedi, M. M., Convolutional social pooling for vehicle trajectory prediction, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (2018), 1468-1476.
39. Messaoud, K., Yahiaoui, I., Verroust-Blondet, A. and Nashashibi, F., Attention based vehicle trajectory prediction, IEEE Trans. Intell. Veh., 6 (1) (2020), 175-185, https://doi.org/10.1109/TIV.2020.2991952.
40. Li, X., Ying, X. and Chuah, M. C., Grip++: Enhanced graph-based interaction-aware trajectory prediction for autonomous driving, arXiv:1907.07792, (2019), https://doi.org/10.48550/arXiv.1907.07792.
41. Zhao, Z., Fang, H., Jin, Z. and Qiu, Q., Gisnet: Graph-based information sharing network for vehicle trajectory prediction, IEEE International Joint Conference on Neural Networks (IJCNN), (2020), 1-7, https://doi.org/10.48550/arXiv.2003.11973.
42. Zhao, T., Xu, Y., Monfort, M., Choi, W., Baker, C., Zhao, Y., Wang, Y. and Wu, Y. N., Multi-agent tensor fusion for contextual trajectory prediction, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), https://doi.org/10.48550/arXiv.1904.04776.
43. Wang, Y., Zhao, S., Zhang, R., Cheng, X. and Yang, L., Multi-vehicle collaborative learning for trajectory prediction with spatio-temporal tensor fusion, IEEE Transactions on Intelligent Transportation Systems, 23 (1) (2022), 236-248, https://doi.org/10.1109/TITS.2020.3009762.
44. Saleh, K., Hossny, M. and Nahavandi, S., Long-term recurrent predictive model for intent prediction of pedestrians via inverse reinforcement learning, Digital Image Computing: Techniques and Applications (DICTA), (2018), 1-8, https://doi.org/10.1109/DICTA.2018.8615854.
45. Kuefler, A., Morton, J., Wheeler, T. and Kochenderfer, M., Imitating driver behavior with generative adversarial networks, IEEE Intelligent Vehicles Symposium (IV), (2017), 204-211, https://doi.org/10.1109/IVS.2017.7995721.
46. Wulfmeier, M., Rao, D., Wang, D. Z., Ondruska, P. and Posner, I., Large-scale cost function learning for path planning using deep inverse reinforcement learning, Int. J. Rob. Res., 36 (10) (2017), 1073-1087, https://doi.org/10.1177/0278364917722396.
47. Fernando, T., Denman, S., Sridharan, S. and Fookes, C., Neighbourhood context embeddings in deep inverse reinforcement learning for predicting pedestrian motion over long time horizons, IEEE/CVF International Conference on Computer Vision Workshops, (2019), 1179-1187, https://doi.org/10.1109/ICCVW.2019.00149.
48. Chang, M.-F., Lambert, J., Sangkloy,P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D. et al., Argoverse: 3d tracking and forecasting with rich maps, IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2019), 8748-8757.
49. Mercat, J., Gilles, T., El Zoghby, N., Sandou, G., Beauvois, D. and Gil, G. P., Multihead attention for multi-modal joint vehicle motion forecasting, IEEE International Conference on Robotics and Automation (ICRA), (2020), 9638-9644, https://doi.org/10.1109/ICRA40945.2020.9197340.
50. Ngiam, J., Caine, B., Vasudevan, V., Zhang, Z., Chiang, H.-T. L., Ling, J., Roelofs, R., Bewley, A., Liu, C., Venugopal, A. et al., Scene transformer: A unified multi-task model for behavior prediction and planning, International Conference on Learning Representations (ICLR), (2021), https://doi.org/10.48550/arXiv.2106.08417.
51. Liu, Y., Zhang, J., Fang, L., Jiang, Q. and Zhou, B., Multimodal motion prediction with stacked transformers, IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021), 7577-7586, https://doi.org/10.1109/CVPR46437.2021.00749.
52. Liang, M., Yang, B., Hu, R., Chen, Y., Liao, R., Feng, S. and Urtasun, R., Learning lane graph representations for motion forecasting, European Conference on Computer Vision (ECCV), (2020), 541-556,https://doi.org/10.48550/arXiv.2007.13732.
53. Gu, J., Sun, C. and Zhao, H., Densetnt: End-to-end trajectory prediction from dense goal sets, IEEE/CVF International Conference on Computer Vision, (2021), 15303-15312, https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.01502.
54. Zeng, W., Liang, M., Liao, R. and Urtasun, R., Lanercnn: Distributed representations for graph-centric motion forecasting, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2021), 532-539, https://doi.org/10.1109/IROS51168.2021.9636035.
55. Song, H., Luan, D., Ding, W., Wang, M. Y. and Chen, Q., Learning to predict vehicle trajectories with model-based planning, arXiv:2103.04027, (2021), https://doi.org/10.48550/arXiv.2103.04027.