REAL-TIME AND DEEP LEARNING-BASED FATIGUE DETECTION FOR DRIVERS

Abstract

Among the causes of traffic accidents, driver errors are in the first place. Driver faults are generally considered to be situations such as drunk driving and excessive speeding. However, sleep-deprived and tired driving are also among the leading causes of driver faults. Driving while feeling sleepy and fatigued; Effects such as slow reaction time, decreased awareness, and inability to focus occur. Considering this situation, it is understood that driving while sleepy and tired is at least as dangerous as driving under the influence of alcohol. In this study, a system that works in real-time inside the vehicle constantly monitors the driver and works with high accuracy is proposed. This system is deep learning based and low cost. In the study, the driver's eye and mouth movements were analyzed to determine normal, yawning and fatigue. A data set has been created for this. The data set consists of videos taken at different times and in different ways from 129 volunteers. Videos shot in different formats, quality and sizes were collected, and turned into a single format. Grayscale, tilt addition, blurring, variability addition, noise addition, image brightness change, color vividness change, perspective change, sizing, and position change were added to the photographs that make up the data set. With these additions, the error that may occur due to any distortion that may occur from the camera is minimized. Thus, the accuracy rate in the detection process with images taken from the camera in real-time has been increased. At the same time, a new data set specific to the study was prepared. YOLOv5, YOLOv6, YOLOv7, and YOLOv8 architectures were used in the study. The newest and most used architectural results in the literature are compared. As a result of the study, a 98.15% accuracy rate was obtained in YOLOv8 architecture. It is aimed that the study will be highly effective in preventing traffic accidents.

Keywords

• Deep learning
• Driver fatigue detection
• Image processing
• YOLOv8 architecture

SÜRÜCÜLER İÇİN GERÇEK ZAMANLI VE DERİN ÖĞRENME TABANLI YORGUNLUK TESPİTİ

Abstract

Trafik kazaları nedenleri arasında sürücü kusurları ilk sırada yer almaktadır. Sürücü kusurları genel olarak alkollü iken araç kullanma, aşırı hız yapma gibi durumlar düşünülmektedir. Fakat uykusuz ve yorgun araç kullanmanın da sürücü kusurlarında ön sıralarda yer almaktadır. Uykusuz ve yorgun şekilde araç kullanımda; tepki süresinde yavaşlık, farkındalığın azalması, odaklanamama gibi etkiler ortaya çıkmaktadır. Bu durum göz önünde bulundurulduğunda uykusuz ve yorgun araç kullanmanın da en az alkollü araç kullanmak kadar tehlikeli olduğu anlaşılmaktadır. Yapılan bu çalışmada araç içeresinde gerçek zamanlı çalışan, sürekli olarak sürücüyü izleyen ve yüksek doğrulukta çalışan bir sistem önerilmiştir. Bu sistem derin öğrenme tabanlı ve düşük maliyetlidir. Yapılan çalışmada sürücünün göz ve ağız hareketleri analiz edilerek normal, esneme ve yorgunluk tespiti gerçekleştirilmiştir. Bunun için bir veri seti oluşturulmuştur. Veri seti 129 gönüllüden farklı saatlerde ve şekillerde çekilmiş videolardan oluşmaktadır. Farklı formatta, kalite veboyutta çekilen videolar toplanark tek bir format haline getirilmiştir. Veri setini oluşturan fotoğraflar üzerinde gri tonlama, eğim eklenmesi, bulanıklaştırma, değişkenlik eklenmesi, gürültü eklenmesi, görüntü parlaklığı değişikliği, renk canlılığı değişikliği, perspektif değişikliği, boyutlandırma ve konum değişikliği eklenmiştir. Bu eklemeler ile kameradan meydana gelebilecek herhangi bir bozulmaya karşı oluşacak hata en aza indirilmiştir. Böylelikle gerçek zamanlı olarak kameradan alınan görüntülerle yapılacak tespit işlemindeki doğruluk oranı arttırılmıştır. Aynı zamanda çalışmaya özgü ve yeni bir very seti hazırlanmıştır. Yapılan çalışmada YOLOv5, YOLOv6, YOLOv7 ve YOLOv8 mimarisi kullanılmıştır. Literatürde en yeni en çok kullaılan mimari sonuçları karşılaştırılmıştır. Çalışma sonucunda YOLOv8 mimarisinde %98.15 doğruluk oranı elde edilmiştir. Yapılan çalışma ile trafik kazalarının önlenmesinden yüksek oranda etkili olacağı hedeflenmektedir.

Keywords

• Derin öğrenme
• Sürücü yorgunluk tespiti
• Görüntü işleme
• YOLOv8 mimarisi

References

1. Hooda, R. Joshi, V. and Shah, M., "A comprehensive review of approaches to detect fatigue using machine learning techniques", In Chronic Diseases and Translational Medicine, 2022, Vol. 8.
2. Scaccabarozzi, D. Mazzola, L. Bocciolone, M. F. Resta, F. and Saggin, B., "Monitoring of train driver’s alertness: A feasibility study", IEEE International Instrumentation and Measurement Technology Conference, Proceedings. 2017, I2MTC 2017.
3. Weng, C. Lai, Y. and Lai, S., "Driver drowsiness detection via a hierarchical temporal deep belief network", ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016,117–133.
4. Jeong, J. H. Yu, B. W. Lee, D. H. and Lee, S. W., "Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional lstm network using electroencephalography signals", Brain Sciences, 2019, Vol 9(12).
5. Guragain, B. Rad, A. B. Wang, C. Verma, A. K. Archer, L. Wilson, N. and Tavakolian, K., "EEG-based Classification of Microsleep by Means of Feature Selection: An Application in Aviation", Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2019.
6. Wei, C. S. Wang, Y. Lin, C. T. and Jung, T. P., "Toward Drowsiness Detection Using Non-hair-Bearing EEG-Based Brain-Computer Interfaces", IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, Vol. 26(2).
7. Tateno, S. Guan, X. Cao, R. and Qu, Z., "Development of Drowsiness Detection System Based on Respiration Changes Using Heart Rate Monitoring", 57th Annual Conference of the Society of Instrument and Control Engineers of Japan, 2018.
8. Zhu, M., Chen, J., Li, H., Liang, F., Han, L., and Zhang, Z., "Vehicle driver drowsiness detection method using wearable EEG based on convolution neural network", Neural Computing and Applications, 2021, 33(20).

9. Gangadharan K, S. and Vinod, A. P., "Drowsiness detection using portable wireless EEG", Computer Methods and Programs in Biomedicine, 2022, 214.
10. Işık, Ş. and Anagün, Y., "A Deep Learnıng Based Sleepness and Wakefulness Detectıon For Drıvers". Eskişehir Osmangazi University Faculty of Engineering and Architecture Journal, 2021, 29(3).
11. Liu, C. C. Hosking, S. G. and Lenné, M. G., "Predicting driver drowsiness using vehicle measures: Recent insights and future challenges", Journal of Safety Research, 2022, 40(4).
12. Vasudevan, K. Das, A. P. Sandhya, B. and Subith, P., "Driver drowsiness monitoring by learning vehicle telemetry data" Proceedings 10th International Conference on Human System Interactions, 2017.
13. Zhenhai, G. Dinhdat, L. Hongyu, H. Ziwen, Y. and Xinyu, W., "Driver drowsiness detection based on time series analysis of steering wheel angular velocity", Proceedings - 9th International Conference on Measuring Technology and Mechatronics Automation, 2017.
14. Dehzangi, O. and Masilamani, S., "Unobtrusive Driver Drowsiness Prediction Using Driving Behavior from Vehicular Sensors. Proceedings", International Conference on Pattern Recognition, 2018.
15. Maior, C. B. S. Moura, M. J. C. Santana, J. M. M. and Lins, I. D., "Real-time classification for autonomous drowsiness detection using eye aspect ratio", Expert Systems with Applications, 2018, 158.
16. Liu, G. Yan, D. and Chen, Z., "Research on Early Warning of Driver Fatigue Status Based on Image Processing", Proceedings of the 33rd Chinese Control and Decision Conference, 2021.
17. Omidyeganeh, M. Shirmohammadi, S. Abtahi, S. Khurshid, A. Farhan, M. Scharcanski, J. Hariri, B. Laroche, D. and Martel, L., "Yawning Detection Using Embedded Smart Cameras", IEEE Transactions on Instrumentation and Measurement, 2016,65.
18. Tipprasert, W. Charoenpong, T. Chianrabutra, C. and Sukjamsri, C., "A Method of Driver’s Eyes Closure and Yawning Detection for Drowsiness Analysis by Infrared Camera", 1st International Symposium on Instrumentation, Control, Artificial Intelligence, and Robotics, ICA-SYMP, 2019.
19. Suganthi, M. and Sathiaseelan, J. G. R., "An Exploratory of Hybrid Techniques on Deep Learning for Image Classification", International Conference on Computer, Communication and Signal Processing, 2020.
20. Yu, C. Qin, X. Chen, Y. Wang, J. and Fan, C., "Drowsydet: A mobile application for real-time driver drowsiness detection", IEEE SmartWorld, Ubiquitous Intelligence and Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Internet of People and Smart City Innovation, 2019.
21. Galarza, E. E. Egas, F. D. Silva, F. M. and Velasco, P. M., "Real Time Driver Drowsiness Detection Based on Driver’s Face Image Behavior Using a System of Human Computer Interaction Implemented in a Smartphone" International Conference on Information Technology & Systems, 2018, 563-572.
22. Dwivedi, K. Biswaranjan, K. and Sethi, A., "Drowsy driver detection using representation learning", IEEE International Advance Computing Conference, 2018, 995–999.
23. Mehta S. Dadhich, S. Gumber, S. and Bhatt, A. J., "Real-Time Driver Drowsiness Detection System Using Eye Aspect Ratio and Eye Closure Ratio", International Conference on Sustainable Computing in Science, Technology and Management (SUSCOM), Amity University Rajasthan, Jaipur - India, 2022, 26-28.
24. Nagargoje, S. and Shilvant, D. S., "Drowsy Driver Warning System Using Image Processing" Internatıonal Journal of Engıneerıng Development and Research (IJEDR), 2015, vol. 3(3), 78-83.
25. Reddy, B. Kim, Y. H. Yun, S. Seo, C. and Jang, J.,"Real-time driver drowsiness detection for embedded system using model compression of deep neural networks", IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017,438–445.
26. Yan, J. J. Kuo H. H. Lin, Y. F. and Liao, T. L. "Real-Time Driver Drowsiness Detection System Based on PERCLOS and Grayscale Image Processing". International Symposium on Computer, Consumer and Control (IS3C), 2016, 75-85.
27. Omidyeganeh, M. Shirmohammadi, S. and Abtahi, S., "Yawning detection using embedded smart cameras", IEEE Trans. Instrum. Meas. 2016, 65 (3), 570–582.
28. Yarlagadda, V. Koolagudi, S. G. Kumar M V, M. and Donepudi, S. "Driver Drowsiness Detection Using Facial Parameters and RNNs with LSTM" 17th India Council International Conference, 2020.
29. Akrout, B. and Mahdi, W., "Yawning detection by the analysis of variational descriptor for monitoring driver drowsiness", 2nd International Image Processing, Applications and Systems Conference, 2016, 1–5.
30. Liu, G. Yan, D. and Chen, Z., "Research on Early Warning of Driver Fatigue Status Based on Image Processing", Proceedings of the 33rd Chinese Control and Decision Conference, CCDC, 2021.
31. Zhang, W. and Su, J., "Driver yawning detection based on long short term memory networks", IEEE Symposium Series on Computational Intelligence, 2018 Vol. 2018-Janua, pp. 1–5.
32. Flores-Monroy, J. Nakano-Miyatake, M. Sanchez-Perez, G. and Perez-Meana, H., "Visual-based Real Time Driver Drowsiness Detection System Using CNN. 2021 18th International Conference on Electrical Engineering, Computing Science and Automatic Control", 2021.
33. Zhang, W. Murphey, Y. L. Wang, T. and Xu, Q. "Driver yawning detection based on deep convolutional neural learning and robust nose tracking", International Joint Conference on Neural Networks, 2015, Vol. 2015-Septe, 1–8.
34. Jie, Z. M. and Stafford-Fraser, Q., "Analysis of yawning behaviour in spontaneous expressions of drowsy drivers" IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018, 2018, pp. 571–576,
35. Kassem, H.A, Chowdhury, M. U. Abawajy, J. and Al-Sudani, A.R., "Yawn based driver fatigue level", 2020 372–382.
36. Guo, J.M. and Markoni, H., "Driver drowsiness detection using hybrid convolutional neural network and long short-term memory", Multimedia Tools Appl. 2019, 78 (20) 29059–29087.
37. Geng, L. Liang, X. Xiao, Z. and Li, Y., "Real-time driver fatigue detection based on morphology infrared features and deep learning", Hongwai Yu Jiguang Gongcheng/Infrared Laser Eng. 2018, 47 (2).
38. You, F. Gong, Y. Tu, H, Liang, J. and Wang, H., "A fatigue driving detection algorithm based on facial motion information entropy", J. Adv. Transp. 2020, 1– 17.
39. Jabbar, R. Al-Khalifa, M. Kharbeche, W. Alhajyaseen, Jafari, M. and Jiang, S., "Real-time driver drowsiness detection for android application using deep neural networks techniques", Comput. Sci. 2018, 400–407.
40. Vu, T.H. Dang, A. and Wang, J.C., "A deep neural network for real-time driver drowsiness detection", IEICE Trans. Inf. Syst. 2019, 102 (12), 2637–2641.
41. Ayachi, R. Afif, M. Said, Y. and Abdelali, A.B., "Drivers fatigue detection using efficientdet in advanced driver assistance systems", International Multi-Conference on Systems, Signals & Devices, SSD, IEEE, 2021, 38–742.
42. Esra, C. and Ugur, Y., "Real-time driver fatigue detection system with deep learning on a low-cost embedded system", Microprocessors and Microsystems, 2023, 99,104851.
43. Emre, Ş., İbrahim A.D. Necaattin B, and Sinan Toklu., "IoT based mobile driver drowsiness detection using deep learning", Journal of the Faculty of Engineering and Architecture of Gazi University, 2022, 37:4, 1869-1881.