Deep Learning and LSTM Integration for Analyzing Driver Behaviors

Abstract

Real-time detection of driver behaviors, fundamental for autonomous vehicles, is crucial for preventing accidents and enhancing traffic safety. Traditional methods, relying on manual observations or sensor-based monitoring, are increasingly being replaced by automated solutions using machine learning and computer vision technologies. This study aims to improve the classification of driver behaviors through the integration of deep learning models with LSTM layers. A multi-class driver behavior dataset, including images of safe driving, phone conversations, texting, turning, and other distractions, was used. Data processing involved cross-validation to ensure reliable performance evaluations. Various deep learning models such as VGG19, ResNet50, MobileNetV2, InceptionV3, DenseNet201, and InceptionResNetV2 were employed, each integrated with LSTM layers to create hybrid architecture. LSTM’s ability to capture temporal dependencies enabled more accurate behavior classification. Model performances were evaluated using accuracy, precision, recall, F1-Score, Log Loss, and ROC-AUC metrics. Experimental results demonstrated that LSTM integration significantly enhanced classification performance. InceptionResNetV2 and MobileNetV2 also achieved strong results with LSTM, while DenseNet201 was the most accurate at 94.77\%. Road safety applications and real-time monitoring systems can benefit from these findings. In addition, this study contributes to the development of driver monitoring systems based on machine learning, which has the potential to enhance safety in autonomous vehicles.

Keywords

• Deep Learning
• Driver Behavior
• Driving Scenarios
• Image Classification
• LSTM

Supporting Institution

No funding was received for this study.

Ethical Statement

The data used in this paper is a public dataset.

References

1. Al Ali, F.M., Alnuaimi, M.J., Alawadhi, S. A., Abdallah, S., Abnormal Driver Behavior Detection Using Deep Learning in 2024 7th International Conference on Signal Processing and Information Security (ICSPIS), IEEE, (2024), 1-4.
2. Al doori, S.K.S., Taspinar, Y.S., Koklu, M., Distracted driving detection with machine learning methods by cnn based feature extraction, International Journal of Applied Mathematics Electronics and Computers, 9(4)(2021), 116-121.
3. Alzebari, N.A.M., Becerikli, Y., Driver behavior detection using intelligent algorithms, Journal of Millimeterwave Communication, Optimization and Modelling, 4(2)(2024), 39-51.
4. Butt, F.M., Hussain, L., Mahmood, A., Lone, K.J., Artificial Intelligence based accurately load forecasting system to forecast short and medium-term load demands, Mathematical Biosciences and Engineering, 18(1)(2021), 400-425.
5. Cengel, T.A., Gencturk, B., Yasin, E.T., Yildiz, M.B., Cinar, I. et al., Apple (Malus domestica) Quality Evaluation Based on Analysis of Features Using Machine Learning Techniques, Applied Fruit Science, 66(2024), 2123-2133.
6. Cengel, T.A., Gencturk, B., Yasin, E.T., Yildiz, M.B., Cinar, I. et al., Automating egg damage detection for improved quality control in the food industry using deep learning, Journal of Food Science, 90(1)(2025), e17553.
7. Cengel, T.A., Gencturk, B., Yasin, E.T., Yildiz, M.B., Cinar, I. et al., Classification of Orange Features for Quality Assessment Using Machine Learning Methods, Selcuk Journal of Agriculture & Food Sciences/Selcuk Tarim ve Gida Bilimleri Dergisi, 38(3)(2024).
8. Chen, J.-C., Lee, C.-Y., Huang, P.-Y., Lin, C.-R., Driver behavior analysis via two-stream deep convolutional neural network, Applied Sciences, 10(6)(2020), 1-14.

9. Cinar, I., Kaya, F.F., Application of ConvNeXt Models for Indian Spices Classification, in Proceedings of International Conference, Abu Dhabi, BAE, (2024), 36-47.
10. Erdem, K., Yasin, E., Yıldız, M.B., Koklu, M., Classification of Heart Diseases with Ensemble Learning Algorithms, Sinop Üniversitesi Fen Bilimleri Dergisi, 9(2)(2024), 369-387.
11. Gencer, K., Gencer, G., Cizmeci, İ.H., Deep learning approaches for retinal image classification: a comparative study of GoogLeNet and ResNet architectures, International Scientific and Vocational Studies Journal, 8(2)(2024), 123-128.
12. Gong, Y., Lu, J., Liu, W., Li, Z., Jiang, X. et al., Sifdrivenet: Speed and image fusion for driving behavior classification network, IEEE Transactions on Computational Social Systems, 11(1)(2023).
13. He, K., Zhang, X., Ren, S.,d Sun, J., Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 770-778.
14. Howard, A.G., Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861, (2017).
15. Huang, C., Wang, X., Cao, J., Wang, S., Zhang, Y., HCF: A hybrid CNN framework for behavior detection of distracted drivers, IEEE access, 8(2020), 109335-109349.
16. Huang, G., Liu, Z., Pleiss, G., Van Der Maaten, L., Weinberger, K.Q., Convolutional networks with dense connectivity, IEEE transactions on pattern analysis and machine intelligence, 44(12)(2019), 8704-8716.
17. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., Densely connected convolutional networks, in Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), 4700-4708.
18. Huang, T., Fu, R., Chen, Y., Deep driver behavior detection model based on human brain consolidated learning for shared autonomy systems, Measurement, 179(2021), 109463.
19. Huang, W., Liu, X., Luo, M., Zhang, P., Wang, W., Wang, J., Video-based abnormal driving behavior detection via deep learning fusions, IEEE Access, 7(2019), 64571-64582.
20. Isik, H., Tasdemir, S., Taspinar, Y.S., Kursun, R., Cinar, I. et al., Maize seeds forecasting with hybrid directional and bi-directional long short-term memory models, Food Science & Nutrition, 12(2)(2024), 786-803.
21. Kang, H., Zhang, C., Jiang, H., Advancing Driver Behavior Recognition: An Intelligent Approach Utilizing ResNet, Automatic Control and Computer Sciences, 58(5)(2024), 555-568.
22. Koklu, M., Cinar, I., Taspinar, Y.S., CNN-based bi-directional and directional long-short term memory network for determination of face mask, Biomedical signal processing and control, 71(2022), 103216.
23. Koklu, M., Kursun, R., Taspinar, Y.S., Cinar, I., Classification of date fruits into genetic varieties using image analysis, Mathematical Problems in Engineering, 1(2021), 4793293.
24. Koklu, N., Sulak, S.A., The Systematic Analysis of Adults’ Environmental Sensory Tendencies Dataset, Data in Brief, 55(2024), 110640.
25. Koklu, N., Sulak, S.A., Using Artificial Intelligence Techniques for the Analysis of Obesity Status According to the Individuals’ Social and Physical Activities, Sinop U¨ niversitesi Fen Bilimleri Dergisi, 9(1)(2024), 217-239.
26. Lai, Z., Driver Behavior and Action Prediction in Human-Computer Interaction, in 2024 3rd International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC), IEEE, (2024), 97-101.
27. Sahin Afridi, A., Kafy, A., Nessa Moon, M. N., Shakil, M.S., Multi-Class Driver Behavior Image Dataset, Mendeley Data, (2024).
28. Seo, H., Hwang, J., Jeong, T., Shin, J., Comparison of deep learning models for cervical vertebral maturation stage classification on lateral cephalometric radiographs, Journal of Clinical Medicine, 10(16)(2021), 3591.
29. Simonyan, K., Zisserman, A., Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).
30. Sinha, D.,El-Sharkawy, M., Thin mobilenet: An enhanced mobilenet architecture, in 2019 IEEE 10th annual ubiquitous computing, electronics & mobile communication conference (UEMCON), IEEE, (2019) , 0280-0285.
31. Sulak, S.A., Koklu, N., Analysis of Depression, Anxiety, Stress Scale (DASS-42) With Methods of Data Mining, European Journal of Education, 59(4)(2024), e12778.
32. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A., Inception-v4, inception-resnet and the impact of residual connections on learning, in Proceedings of the AAAI conference on artificial intelligence, 31(1)(2017).
33. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. et al., Going deeper with convolutions, in Proceedings of the IEEE conference on computer vision and pattern recognition, (2015), 1-9.
34. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., Rethinking the inception architecture for computer vision, in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 2818-2826.
35. Taspinar, Y.S., Koklu, M., Altin, M., Acoustic-driven airflow flame extinguishing system design and analysis of capabilities of low frequency in different fuels, Fire technology, 58(3)(2022), 1579-1597.
36. Taspinar, Y.S., Selek, M., Object recognition with hybrid deep learning methods and testing on embedded systems, International Journal of Intelligent Systems and Applications in Engineering, (2020).
37. Tutuncu, K., Cinar, I., Kursun, R., Koklu, M., Edible and poisonous mushrooms classification by machine learning algorithms, in 2022 11th Mediterranean Conference on Embedded Computing (MECO), Budva, Montenegro, 07-10 June 2022: IEEE, (2022), 1-4.
38. Wen, L., Li, X., Li, X., Gao, L., A new transfer learning based on VGG-19 network for fault diagnosis, in 2019 IEEE 23rd international conference on computer supported cooperative work in design (CSCWD), IEEE, (2019), 205-209.
39. WHO. Road traffic injuries. https://www.who.int/news-room/fact-sheets/detail/ road-traffic-injuries (accessed 20.01.2025, 2025).
40. Xiao, W., Xie, G., Liu, H., Chen, W., Li, R., FDAN: Fuzzy deep attention networks for driver behavior recognition, Journal of Systems Architecture, 147(2024), 103063.
41. Xie, L., Xiang, X., Xu, H., Wang, L., Lin, L.et al., FFCNN: A deep neural network for surface defect detection of magnetic tile, IEEE Transactions on Industrial Electronics, 68(4)(2020), 3506-3516.
42. Yan, S., Teng, Y., Smith, J.S., Zhang, B., Driver behavior recognition based on deep convolutional neural networks, in 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), IEEE, (2016), 636-641.
43. Yasin, E., Koklu, M., A comparative analysis of machine learning algorithms for waste classification: inceptionv3 and chi-square features, International Journal of Environmental Science and Technology 22(2024), 9415–9428.
44. Yurttakal, A.H., Erbay, H., C¸ inarer, G., Bas, H., Classification of Diabetic Rat Histopathology Images Using Convolutional Neural Networks, International Journal of Computational Intelligence Systems, 14(1)(2021), 715-722.
45. Zhang, C., Li, R., Kim, W., Yoon, D., Patras, P., Driver behavior recognition via interwoven deep convolutional neural nets with multi-stream inputs, IEEE Access, 8(2020), 191138-191151.
46. Zhao, L., Yang, F., Bu, L., Han, S., Zhang, G. et al., Driver behavior detection via adaptive spatial attention mechanism, Advanced Engineering Informatics, 48(2021), 101280.