Performance comparison of machine learning models on a novel in-vehicle controller area network dataset

Abstract

The advancement of technology has significantly enhanced comfort and welfare across all aspects of life, particularly in the field of transportation. One notable development is the growing adoption of autonomous vehicles, driven by the integration of smart systems into automobiles. However, the sophisticated systems and networks within autonomous vehicles have also opened new avenues for cyberattacks. These attacks typically aim to achieve one of three objectives: gaining unauthorized control of system components, overloading the system network to slow its operation, or causing a system crash. The potentially severe consequences of such cyberattacks have underscored the urgent need for robust security measures to protect autonomous vehicles. This study focuses on detecting cyberattacks targeting in-vehicle networks of smart vehicles using machine learning models. A simulation environment was developed to generate cyberattack scenarios, resulting in the creation of a dataset. This dataset was then analyzed using classification algorithms, including XGBoost, LightGBM, Random Forest, and Decision Trees. Performance comparisons revealed that XGBoost achieved the highest accuracy at 86.22% and F1 Score at 79.7%, while the Decision Tree algorithm had the lowest accuracy at 80.7% and F1 Score at 72.5%. In addition, the LightGBM algorithm had an accuracy rate of 85.83% and the Random Forest algorithm had an accuracy rate of 85.84%. The findings of this study are expected to contribute to the efforts of smart vehicle security experts in mitigating cyber threats and raising awareness about the importance of cybersecurity in autonomous vehicles.

Keywords

• Classification algorithms
• Cyber attack
• Machine learning
• Smart vehicle

References

1. 1. Huang, T., J. Zhou, Y. Wang, and A. Cheng, On the security of in-vehicle hybrid network: Status and challenges, in Lecture Notes in Computer Science, 2017. Melbourne: p.621-637.
2. 2. Chakraborty, S., M. Lukasiewycz, C. Buckl, S. Fahmy, N.Chang and S. Park, Embedded systems and software challenges in electric vehicles, in 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2012. Dresden: p. 424–429.
3. 3. Du, X., S. Jiang, D. Zhou, A. B. Milhim, and H. Sadjadi, Ground Fault Diagnostics for Automotive Electronic Control Units, Int. J. Progn. Heal. Manag., 2023. 14(3): p. 1-13.
4. 4. Buttigieg, R., M. Farrugia, and C. Meli, Security issues in controller area networks in automobiles, in 2017 18th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA), IEEE, Dec. 2017. Monastir: p. 93–98.
5. 5. Shaw, R. and B. Jackman, An introduction to FlexRay as an industrial network, in IEEE International Symposium on Industrial Electronics, IEEE, Jun. 2008. Cambridge: p. 1849–1854.
6. 6. Wolf, M., A. Weimerskirch, and C. Paar, Security in Automotive Bus Systems, Proc. Work. Embed. Secur. Cars, 2004. 2004: p. 1–13.
7. 7. Hossain, M. D., H. Inoue, H. Ochiai, D. Fall, and Y. Kadobayashi, LSTM-Based Intrusion Detection System for In-Vehicle Can Bus Communications, IEEE Access, 2020. 8: p. 185489–185502.
8. 8. Hanselmann, M., T. Strauss, K. Dormann, and H. Ulmer, CANet: An Unsupervised Intrusion Detection System for High Dimensional CAN Bus Data, IEEE Access, 2020. 8: p. 58194–58205.

9. 9. Jin, S., J.-G. Chung, and Y. Xu, Signature-Based Intrusion Detection System (IDS) for In-Vehicle CAN Bus Network, in 2021 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, May 2021. Daegu: p. 1–5.
10. 10. Islam, R., M. K. Devnath, M. D. Samad, and S. M. Jaffrey Al Kadry, GGNB: Graph-based Gaussian naive Bayes intrusion detection system for CAN bus, Veh. Commun., 2022. 33: p. 100442.
11. 11. Deng, Z., J. Liu, Y. Xun, and J. Qin, IdentifierIDS: A Practical Voltage-Based Intrusion Detection System for Real In-Vehicle Networks, IEEE Trans. Inf. Forensics Secur., 2024. 19: p. 661–676,
12. 12. Khandelwal, S., E. Wadhwa, and S. Shreejith, Deep Learning-based Embedded Intrusion Detection System for Automotive CAN, in Proceedings of the International Conference on Application-Specific Systems, Architectures and Processors, 2022. Gothenburg: p. 88–92.
13. 13. Desta, A. K., S. Ohira, I. Arai, and K. Fujikawa, Rec-CNN: In-vehicle networks intrusion detection using convolutional neural networks trained on recurrence plots, Veh. Commun., 2022. 35: p. 100470.
14. 14. Yu, Z., Y. Liu, G. Xie, R. Li, S. Liu, and L. T. Yang, TCE-IDS: Time Interval Conditional Entropy- Based Intrusion Detection System for Automotive Controller Area Networks, IEEE Trans. Ind. Informatics, 2023. 19(2): p. 1185–1195.
15. 15. Tanksale, V., Intrusion detection system for controller area network, Cybersecurity, 2024. 7(1): p. 1-21.
16. 16. Ye, P., Y. Liang, Y. Bie, G. Qin, J. Song, Y. Wang and W. Liu, GDT-IDS: graph-based decision tree intrusion detection system for controller area network, J. Supercomput., 2025. 81(4): p. 591.
17. 17. Checkoway, S., D. McCoy, B. Kantor,D. Anderson, H. Shacham, S. Savage, K. Koscher, A. Czeskis, F. Roesner, and T. Kohno, Comprehensive experimental analyses of automotive attack surfaces, in Proceedings of the 20th USENIX Security Symposium, 2011. San Fransisco: p. 1-16.
18. 18. Petit, J., B. Stottelaar, M. Feiri, and F. Kargl, Remote Attacks on Automated Vehicles Sensors: Experiments on Camera and LiDAR, [cited 2025 14 May]; Available from: Blackhat.com, p. 1–13.
19. 19. Aliwa, E., O. Rana, C. Perera, and P. Burnap, Cyberattacks and Countermeasures for In-Vehicle Networks, ACM Comput. Surv., 2021. 54(1): p. 1–37.
20. 20. Deng, J., L. Yu, Y. Fu, O. Hambolu, and R. R. Brooks, Security and Data Privacy of Modern Automobiles, in Data Analytics for Intelligent Transportation Systems, Elsevier, 2017. 2017: p. 131–163.
21. 21. Bari, B. S., K. Yelamarthi, and S. Ghafoor, Intrusion Detection in Vehicle Controller Area Network (CAN) Bus Using Machine Learning: A Comparative Performance Study, Sensors, 2023. 23(7): p. 3610.
22. 22. Gao, S., L. Zhang, L. He, X. Deng, H. Yin, and H. Zhang, Attack Detection for Intelligent Vehicles via CAN- Bus: A Lightweight Image Network Approach, IEEE Trans. Veh. Technol., 2023. 72(12): p. 16624–16636.
23. 23. Lo, W., H. Alqahtani, K. Thakur, A. Almadhor, S. Chander, and G. Kumar, A hybrid deep learning based intrusion detection system using spatial-temporal representation of in-vehicle network traffic, Veh. Commun., 2022. 35: p. 100471.
24. 24. Wang, K., A. Zhang, H. Sun, and B. Wang, Analysis of Recent Deep-Learning-Based Intrusion Detection Methods for In-Vehicle Network, IEEE Trans. Intell. Transp. Syst., 2022. 24(2): p. 1–12.
25. 25. Lee, H., S. H. Jeong, and H. K. Kim, OTIDS: A novel intrusion detection system for in-vehicle network by using remote frame, in Proceedings - 2017 15th Annual Conference on Privacy, Security and Trust, PST 2017, IEEE. 2018. Calgary: p. 57–66.
26. 26. Verma, M. E., R. A. Bridges, M. D. Iannacone, S. C. Hollifield, P. Moriano, S. C. Hespeler, B. Kay and F. L. Combs, A comprehensive guide to CAN IDS data and introduction of the ROAD dataset, PLoS One, 2024. 19(1): p. e0296879.
27. 27. Seo, E., H. M. Song, and H. K. Kim, GIDS: GAN based Intrusion Detection System for In-Vehicle Network, in 2018 16th Annual Conference on Privacy, Security and Trust (PST), IEEE. 2018. Belfast: p. 1–6.
28. 28. Rajapaksha, S., H. Kalutarage, M. O. Al-Kadri, A. Petrovski, G. Madzudzo, and M. Cheah, AI-Based Intrusion Detection Systems for In-Vehicle Networks: A Survey, ACM Comput. Surv., 2023. 55(11): p. 1–40.
29. 29. Berger, I., R. Rieke, M. Kolomeets, A. Chechulin, and I. Kotenko, Comparative study of machine learning methods for in-vehicle intrusion detection, in Lecture Notes in Computer Science, 2018. Barcelona: p. 85-101.
30. 30. Han, M. L., B. Il Kwak, and H. K. Kim, Anomaly intrusion detection method for vehicular networks based on survival analysis, Veh. Commun., 2018. 14: p. 52–63.
31. 31. Lee, S., H. J. Jo, A. Cho, D. H. Lee, and W. Choi, TTIDS: Transmission-Resuming Time-Based Intrusion Detection System for Controller Area Network (CAN), IEEE Access, 10: pp. 52139–52153.
32. 32. GitHub - zombieCraig/ICSim: Instrument Cluster Simulator. Accessed: May 25, 2024. [Online]. Available: https://github.com/zombieCraig/ICSim.
33. 33. GitHub - collin80/SavvyCAN: QT based cross platform canbus tool. Accessed: May 26, 2024. [Online]. Available: https://github.com/collin80/SavvyCAN.
34. 34. Mercaldo, F., R. Casolare, G. Ciaramella, G. Iadarola, F. Martinelli, F. Ranieri and A. Santone, A Real-time Method for CAN Bus Intrusion Detection by Means of Supervised Machine Learning, in Proceedings of the International Conference on Security and Cryptography, 2022. Lisbon: p. 534-539.