Design and implementation of a warning system for detection of sleepiness/drowsiness/sleep state in pilots

Öz

This study introduces a warning system designed for small training aircraft pilots flying below 10000 feet. The system detects the pilots' sleepiness/drowsiness/sleep states. If the system detects such an undesirable situation, it immediately transfers the image of that moment to the predetermined cloud service and activates the warning system consisting of audio and vibration equipment so that the pilot can escape from the current situation and recover. The warning system continues to operate until the pilot exits in that situation. The basic hardware used to implement the system is a Jetson developer kit, a Jetson Wi-Fi module, a vibration motor, a motor control circuit, and a camera. Tests of the designed system were carried out under two main headings: eye condition and head/neck position. In the tests performed according to eye condition, situations in which the eyes are constantly open (sleepiness) and situations in which the eyes are closed (drowsiness) are considered. In the tests performed with and without glasses, depending on the head/neck position, the cases of head tilting forward, backward, right, and left (sleep situation) were examined. It has been observed that the designed system successfully detects the relevant situations in all tests, both warning the pilot and sending the images of that moment to the cloud service.

Anahtar Kelimeler

• Aircraft
• Crash
• Sleepiness
• Drowsiness
• Sleep
• Pilot warning

Teşekkür

The authors would like to thank Dr. Mustafa KISA (Pilot), Ramazan GÖKSOY, and HAVELSAN R&D Laboratory for their contributions. This study was developed from İbrahim Halil Çoban's master's thesis, "Image processing-based pilot warning system design for small training planes.

Pilotlardaki dalgınlık/uyuklama/uyku durumlarının tespitine yönelik ikaz sistemi tasarımı ve gerçekleştirilmesi

Öz

Bu çalışmada 10000 feet altında uçan küçük eğitim uçağı pilotları için tasarlanmış bir uyarı sistemi tanıtılmaktadır. Sistem pilotların dalgınlık/uyuklama/uyku durumlarını tespit etmektedir. Sistem böyle bir istenmeyen durum tespit ederse o ana ait görüntüyü hemen önceden belirlenmiş bulut servisine aktarır ve pilotun mevcut durumdan kurtulup kendine gelebilmesi için ses ve titreşim ekipmanlarından oluşan uyarı sistemini devreye sokmaktadır. Pilot o durumdan çıkana kadar uyarı sistemi çalışmaya devam eder. Sistemi hayata geçirmek için kullanılan temel donanımlar Jetson geliştirici kiti, Jetson Wi-Fi modülü, titreşim motoru, motor kontrol devresi ve kameradır. Tasarlanan sistemin testleri göz durumu ve baş/boyun pozisyonu olmak üzere iki ana başlık altında gerçekleştirilmiştir. Göz durumuna göre yapılan testlerde gözlerin sürekli açık olduğu durumlar (dalgınlık hali) ve gözlerin kapalı olduğu durumlar (uyuklama hali) dikkate alınmıştır. Gözlüklü ve gözlüksüz olarak yapılan testlerde baş/boyun pozisyonuna bağlı olarak başın öne, arkaya, sağa ve sola eğilmesi (uyku durumu) durumları incelenmiştir. Tasarlanan sistemin tüm testlerde ilgili durumları başarılı bir şekilde tespit ederek hem pilotu uyardığı hem de o ana ait görüntüleri bulut servisine gönderdiği gözlemlenmiştir.

Anahtar Kelimeler

• Uçak
• Kaza
• Dalgınlık
• Uyuklama
• Uyku
• Pilot ikazı

Teşekkür

Yazarlar katkılarından dolayı Dr. Mustafa KISA (Pilot), Ramazan GÖKSOY ve HAVELSAN Ar-Ge Laboratuvarı'na teşekkürlerini sunarlar. Bu çalışma, İbrahim Halil Çoban'ın "Küçük eğitim uçakları için görüntü işleme tabanlı pilot uyarı sistemi tasarımı" adlı yüksek lisans tezinden geliştirilmiştir.

Kaynakça

1. İ.H. Çoban, and F.A. Kazan, Examination of Drowsiness Issue and Solution Proposals in Land and Air Vehicle Users. Proceedings of Selcuk University International Technology and Innovation Student Symposium (SUTIS), pp. 157-161, Konya, Türkiye, 07-10 December 2023.
2.    G. Salvendy, Handbook of human factors and ergonomics. John Wiley & Sons. 2012)
3.    X. Yan, and A. bin Abas. Advancements and Perspectives in Fatigue Driving Detection: A Comprehensive Review. IECE Transactions on Intelligent Unmanned Systems, 1 (1), 4-15, 2024. https://doi.org/10.62762/TIUS.2024.767724
4.    A. Sahayadhas, K. Sundaraj, and M. Murugappan. Detecting driver drowsiness based on sensors: a review. Sensors, 12 (12), 16937-16953, 2012. https://doi.org/ 10.3390/s121216937
5.    B. Mandal, L. Li, G.S. Wang, and J. Lin. Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Transactions on Intelligent Transportation Systems, 18 (3), 545-557, 2016. https://doi.org/10.1109/TITS.2016.2582900
6.    B. Cheng, W. Zhang, Y. Lin, R. Feng, and X. Zhang. Driver drowsiness detection based on multisource information. Human Factors and Ergonomics in Manufacturing & Service Industries, 22 (5), 450-467, 2012. https://doi.org/10.1002/hfm.20395
7.    Z. Lan, J. Zhao, P. Liu, C. Zhang, N. Lyu, and L. Guo. Driving fatigue detection based on fusion of EEG and vehicle motion information. Biomedical Signal Processing and Control, 92, 106031, 2024. https://doi.org/10.1016/j.bspc.2024.106031
8.    V. Häkkinen, K. Hirvonen, J. Hasan, M. Kataja, A. Värri, P. Loula, and H. Eskola. The effect of small differences in electrode position on EOG signals: application to vigilance studies. Electroencephalography and clinical neurophysiology, 86 (4), 294-300, 1993. https://doi.org/10.1016/0013-4694(93)90111-8

9.    N.R. Pal, C.-Y. Chuang, L.-W. Ko, C.-F. Chao, T.-P. Jung, S.-F. Liang, and C.-T. Lin. EEG-based subject-and session-independent drowsiness detection: an unsupervised approach. EURASIP Journal on Advances in Signal Processing, 2008, 1-11, 2008. https://doi.org/10.1155/2008/519480
10. A. Tsuchida, M.S. Bhuiyan, and K. Oguri, Estimation of drowsiness level based on eyelid closure and heart rate variability. Proceedings of 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2543-2546, Minneapolis, MN, USA, 03-06 September 2009.
11. S.J. Jung, H.S. Shin, and W.Y. Chung. Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intelligent Transport Systems, 8 (1), 43-50, 2014. https://doi.org/10.1049/iet-its.2012.0032
12. K. Azam, A. Shakoor, R.A. Shah, A. Khan, S.A. Shah, and M.S. Khalil. Comparison of fatigue related road traffic crashes on the national highways and motorways in Pakistan. Journal of Engineering and Applied Sciences, 33 (2) 2014
13. I. Belakhdar, W. Kaaniche, R. Djmel, and B. Ouni, Detecting driver drowsiness based on single electroencephalography channel. Proceedings of 2016 13th International Multi-Conference on Systems, Signals & Devices (SSD), pp. 16-21, Leipzig, Germany, 21-24 March 2016.
14. Z. Ma, B.C. Li, and Z. Yan, Wearable driver drowsiness detection using electrooculography signal. Proceedings of 2016 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNet), pp. 41-43, Austin, TX, USA, 24-27 January 2016.
15. X.-Q. Huo, W.-L. Zheng, and B.-L. Lu, Driving fatigue detection with fusion of EEG and forehead EOG. Proceedings of 2016 international joint conference on neural networks (IJCNN), pp. 897-904, Vancouver, BC, Canada, 24-29 July 2016.
16. S. Poorna, V. Arsha, P. Aparna, P. Gopal, and G. Nair, Drowsiness detection for safe driving using PCA EEG signals. Proceedings of Progress in Computing, Analytics and Networking: Proceedings of ICCAN 2017, pp. 419-428, Singapur, April 2018.
17. W.-L. Zheng, K. Gao, G. Li, W. Liu, C. Liu, J.-Q. Liu, G. Wang, and B.-L. Lu. Vigilance estimation using a wearable EOG device in real driving environment. IEEE Transactions on Intelligent Transportation Systems, 21 (1), 170-184, 2019. https://doi.org/ 10.1109/TITS.2018.2889962
18. M. Mahmoodi, and A. Nahvi. Driver drowsiness detection based on classification of surface electromyography features in a driving simulator. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine, 233 (4), 395-406, 2019. https://doi.org/10.1177/095441191983 1313
19. F. Trenta, S. Conoci, F. Rundo, and S. Battiato, Advanced motion-tracking system with multi-layers deep learning framework for innovative car-driver drowsiness monitoring. Proceedings of 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1-5, Lille, France, 14-18 May 2019.
20. J. Kim, and M. Shin. Utilizing HRV-derived respiration measures for driver drowsiness detection. Electronics, 8 (6), 669, 2019. https://doi.org/10.3390/electronics8 060669
21. L.-S. Lin, H.-Y. Tsai, Y.-J. Li, H.-H. Chen, and L.-L. Li, Driver Fatigue Detection Using TGAM EEG Signal Processing Module. Proceedings of 2023 IEEE 5th Eurasia Conference on IOT, Communication and Engineering (ECICE), pp. 984-986, Yunlin, Taiwan, 27-29 October 2023.
22. Y. Jiao, C. Zhang, X. Chen, L. Fu, C. Jiang, and C. Wen. Driver Fatigue Detection Using Measures of Heart Rate Variability and Electrodermal Activity. IEEE Transactions on Intelligent Transportation Systems 2023. https://doi.org/10.1109/TITS.2023.333 3252
23. S. Liu, Y. Wang, Q. Yu, J. Zhan, H. Liu, and J. Liu. A Driver Fatigue Detection Algorithm Based on Dynamic Tracking of Small Facial Targets Using YOLOv7. IEICE Transactions on Information and Systems, 106 (11), 1881-1890, 2023. https://doi.org/10.1587/transi nf.2023EDP7093
24. J.F. May, and C.L. Baldwin. Driver fatigue: The importance of identifying causal factors of fatigue when considering detection and countermeasure technologies. Transportation research part F: traffic psychology and behaviour, 12 (3), 218-224, 2009. https://doi.org/10.1016/j.trf.2008.11.005
25. C. Bougard, S. Espié, B. Larnaudie, S. Moussay, and D. Davenne. Effects of time of day and sleep deprivation on motorcycle-driving performance. PLoS One, 7 (6), e39735, 2012. https://doi.org/10.1371/jour nal.pone.0039735
26. M.-H. Sigari, M. Fathy, and M. Soryani. A driver face monitoring system for fatigue and distraction detection. International journal of vehicular technology, 2013 (1), 263983, 2013. https://doi.org/10.1155/2013/263983
27. R. Ahmed, K.E.K. Emon, and M.F. Hossain, Robust driver fatigue recognition using image processing. Proceedings of 2014 International Conference on Informatics, Electronics & Vision (ICIEV), pp. 1-6, Dhaka, Bangladesh, 23-24 May 2014.
28. B. Akrout, and W. Mahdi, Yawning detection by the analysis of variational descriptor for monitoring driver drowsiness. Proceedings of 2016 International Image Processing, Applications and Systems (IPAS), pp. 1-5, Hammamet, Tunisia, 05-07 November 2016.
29. S. Golgiyaz, A.F. Kocamaz, and F. Okumuş, Video Based Drowsy Driver Detection System. Proceedings of ELECO-2014, pp. 332-338, Bursa, Türkiye, 27-29 November 2014 2017.
30. L. Zhao, Z. Wang, X. Wang, and Q. Liu. Driver drowsiness detection using facial dynamic fusion information and a DBN. IET Intelligent Transport Systems, 12 (2), 127-133, 2018. https://doi.org/10.104 9/iet-its.2017.0183
31. V. Tümen, Ö. Yıldırım, and B. Ergen, Detection of driver drowsiness in driving environment using deep learning methods. Proceedings of 2018 Electric Electronics, Computer Science, Biomedical Engineerings' Meeting (EBBT), pp. 1-5, Istanbul, Türkiye, 18-19 April 2018.
32. R.A. Vural, M.Y. Sert, and B. Karaköse. Gerçek Zamanlı Sürücü Yorgunluk Tespit Sistemi. Marmara Fen Bilimleri Dergisi, 30 (3), 249-259, 2018. https://doi.org/10.7240/marufbd.417915
33. F. You, X. Li, Y. Gong, H. Wang, and H. Li. A real-time driving drowsiness detection algorithm with individual differences consideration. Ieee Access, 7, 179396-179408, 2019. https://doi.org/10.1109/ACCE SS.2019.2958667
34. J. Gielen, and J.-M. Aerts. Feature extraction and evaluation for driver drowsiness detection based on thermoregulation. Applied Sciences, 9 (17), 3555, 2019. https://doi.org/10.3390/app9173555
35. E. Çıvık, and U. Yüzgeç, Deep Learning Based Continuous Real-Time Driver Fatigue Detection for Embedded System. Proceedings of 2020 28th Signal Processing and Communications Applications Conference (SIU), pp. 1-4, Gaziantep, Türkiye, 05-07 October 2020.
36. M. Düman, E. Erdoğdu, F. Cogen, and T.Ç. Yıldız, Driver fatigue detection with image processing. Proceedings of 2020 12th International Conference on Electrical and Electronics Engineering (ELECO), pp. 246-250, Bursa, Türkiye, 26-28 November 2020.
37. I. Girish, A. Kumar, A. Kumar, and M. Anuradha, Driver fatigue detection. Proceedings of 2020 IEEE 17th India Council International Conference (INDICON), pp. 1-6, New Delhi, India, 10-13 December 2020.
38. M.F.A. Abdullah, M.H. Mohamad Hanafiah, S. Yogarayan, S.F. Abdul Razak, A. Azman, and M.S. Sayeed. Driver fatigue detection using Raspberry-Pi. Indonesian Journal of Electrical Engineering and Computer Science, 32 (2), 1142, 2023. https://doi.org/ 10.11591/ijeecs.v32.i2.pp1142-1149
39. K. Shen, R. Ramli, J.H. Chuah, and G.M.T. Chai, Driver Fatigue Detection Using OpenCV and Dlib Library. Proceedings of 2023 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), pp. 1-6, Nadi, Fiji, 04-06 December 2023.
40. S. Cichocka, and J. Ruminski, Driver fatigue detection method based on facial image analysis. Proceedings of 2024 16th International Conference on Human System Interaction (HSI), pp. 1-6, Paris, France, 08-11 July 2024.
41. P.V. Kumar, A. Ali, A.Z. Sha, and S. Rajesh, IoT based Intelligent Systems for Vehicle. Proceedings of 2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT), pp. 138-143, Bengaluru, India, 04-06 January 2024.
42. R. Schleicher, N. Galley, S. Briest, and L. Galley. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics, 51 (7), 982-1010, 2008. https://doi.org/10.1080/00140130701 817062
43. A.D. McDonald, C. Schwarz, J.D. Lee, and T.L. Brown, Real-time detection of drowsiness related lane departures using steering wheel angle. Proceedings of The Human Factors and Ergonomics Society Annual Meeting, pp. 2201-2205, Boston, USA, 22-26 October 2012.
44. A.A. Albousefi, H. Ying, D. Filev, F. Syed, K.O. Prakah-Asante, F. Tseng, and H.-H. Yang. A two-stage-training support vector machine approach to predicting unintentional vehicle lane departure. Journal of Intelligent Transportation Systems, 21 (1), 41-51, 2017. https://doi.org/10.1080/15472450.2016.1196141
45. Z. Li, S.E. Li, R. Li, B. Cheng, and J. Shi. Online detection of driver fatigue using steering wheel angles for real driving conditions. Sensors, 17 (3), 495, 2017. https://doi.org/10.3390/s17030495.
46. Z. Li, Q. Zhang, and X. Zhao. Performance analysis of K-nearest neighbor, support vector machine, and artificial neural network classifiers for driver drowsiness detection with different road geometries. International Journal of Distributed Sensor Networks, 13 (9), 1-12, 2017. https://doi.org/10.1177 /155014771 77333
47. J.-M. Guo, and H. Markoni. Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimedia tools and applications, 78, 29059-29087, 2019. https://doi.org/ 10.1007/s11042-018-6378-6
48. Q. Abbas. HybridFatigue: A real-time driver drowsiness detection using hybrid features and transfer learning. International Journal of Advanced Computer Science and Applications, 11 (1) 2020. https://doi.org/ 10.14569/IJACSA.2020.0110173
49. W. Sun, X. Zhang, S. Peeta, X. He, Y. Li, and S. Zhu. A self-adaptive dynamic recognition model for fatigue driving based on multi-source information and two levels of fusion. Sensors, 15 (9), 24191-24213, 2015. https://doi.org/10.3390/s150924191
50. B.-G. Lee, and W.-Y. Chung. Driver alertness monitoring using fusion of facial features and bio-signals. IEEE Sensors journal, 12 (7), 2416-2422, 2012. https://doi.org/10.1109/JSEN.2012.2190505
51. S. Samiee, S. Azadi, R. Kazemi, A. Nahvi, and A. Eichberger. Data fusion to develop a driver drowsiness detection system with robustness to signal loss. Sensors, 14 (9), 17832-17847, 2014. https://doi.org/ 10.3390/s140917832
52. Y. Pan, Z.S. Li, E. Zhang, and Z. Guo. A vigilance estimation method for high-speed rail drivers using physiological signals with a two-level fusion framework. Biomedical Signal Processing and Control, 84, 104831, 2023. https://doi.org/10.1016/j.bspc.20 23.104831
53. H. Zallen, J.E. Cochran Jr, and J.A. Bailey. Head‐tilt and pilot fatigue measured by flight simulation. Aircraft Engineering and Aerospace Technology, 84 (1), 51-57, 2012. https://doi.org/10.1108/000226612 11194979
54. S. Ruishan, and Y. Xingchen, Analysis of Influencing Factors of Pilot Fatigue Based on Structural Equation Model. Proceedings of 2021 5th International Conference on Vision, Image and Signal Processing (ICVISP), pp. 82-85, Kuala Lumpur, Malaysia, 18-20 December 2021.
55. H. Shuang, W. Chuanfeng, and W. Qi, Recognition of fatigue status of pilots based on deep sparse auto-encoding network. Proceedings of 2017 36th Chinese Control Conference (CCC), pp. 10945-10950, Dalian, China, 26-28 July 2017.
56. E.Q. Wu, X. Peng, C.Z. Zhang, J. Lin, and R.S. Sheng. Pilots’ fatigue status recognition using deep contractive autoencoder network. IEEE Transactions on Instrumentation and Measurement, 68 (10), 3907-3919, 2019. https://doi.org/10.1109/TIM.2018.2885608
57. P.-Y. Deng, X.-Y. Qiu, Z. Tang, W.-M. Zhang, L.-M. Zhu, H. Ren, G.-R. Zhou, and R.S. Sheng. Detecting fatigue status of pilots based on deep learning network using EEG signals. IEEE Transactions on Cognitive and Developmental Systems, 13 (3), 575-585, 2020. https://doi.org/10.1109/TCDS.2019.2963476
58. S.-Y. Han, J.-W. Kim, and S.-W. Lee, Recognition of pilot’s cognitive states based on combination of physiological signals. Proceedings of 2019 7th International Winter Conference on Brain-Computer Interface (BCI), pp. 1-5, Gangwon, Korea (South), 18-20 February 2019.
59. F. Dehais, A. Dupres, G. Di Flumeri, K. Verdiere, G. Borghini, F. Babiloni, and R. Roy, Monitoring pilot's cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI. Proceedings of 2018 IEEE international conference on systems, man, and cybernetics (SMC), pp. 544-549, Miyazaki, Japan, 07-10 October 2018.
60. A. Alaimo, A. Esposito, and C. Orlando, Cockpit pilot warning system: a preliminary study. Proceedings of 2018 IEEE 4th International Forum on Research and Technology for Society and Industry (RTSI), pp. 1-4, Palermo, Italy, 10-13 September 2018.
61. M. Luo, Y. Li, X. Liu, X. Wang, and C. Zhang, A Study on Civil Aviation Pilots Sleep Status Change on Ultra-Long-Range Routes. Proceedings of 2022 IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), pp. 370-373, Dali, China, 12-14 October 2022.
62. V. Peysakhovich, O. Lefrançois, F. Dehais, and M. Causse. The neuroergonomics of aircraft cockpits: the four stages of eye-tracking integration to enhance flight safety. Safety, 4 (1), 8, 2018. https://doi.org/ 10.3390/safety4010008
63. F. Dehais, M. Causse, and J. Pastor, Embedded eye tracker in a real aircraft: new perspectives on pilot/aircraft interaction monitoring. Proceedings of The 3rd International Conference on Research in Air Transportation. Fairfax, USA: Federal Aviation Administration, pp. 303-309, Fairfax, Virginia, USA, 1-4 June 2008.
64. W.B. Gartner, and M.R. Murphy, Pilot workload and fatigue: A critical survey of concepts and assessment techniques, NASA, TN D-8365, November 1976.
65. J.A. Caldwell. Fatigue in aviation. Travel medicine and infectious disease, 3 (2), 85-96, 2005. https://doi.org/ 10.1016/j.tmaid.2004.07.008
66. B. Şahinkaya, Evaluation of fatigue risk management practices in airline companies and modeling their effects in crew rostering problem. Ph.D. Thesis, Anadolu University, Eskişehir, Türkiye, 2020.
67. M.H. Van den Oord, J.K. Sluiter, and M.H. Frings-Dresen. Differences in physical workload between military helicopter pilots and cabin crew. International archives of occupational and environmental health, 87, 381-386, 2014. https://doi.org/10.1007/s00420-013-0876-7
68. M. Çelikkol, Askeri Havacılık olay ve kazalarında insan faktörünün yorgunluk yönüyle değerlendirilmesi. Proceedings of Hava Kuvvetleri Komutanlığı 2017 Havacılık Emniyeti Yönetim Sistemi (HEYS 2017) Sempozyumu, pp. 73-87, Ankara, Türkiye, 12-13 Nisan 2017.
69. C.A. de Vasconcelos, M.N. Vieira, G. Kecklund, and H.C. Yehia. Speech analysis for fatigue and sleepiness detection of a pilot. Aerospace medicine and human performance, 90 (4), 415-418, 2019
70. C.M. Hannah, The Effects of International Flight Schedules on Pilot Fatigue. Proceedings of 2007 International Symposium on Aviation Psychology, pp. 247-251, Dayton, Ohio, USA 2007.
71. S.A. Gomez, S. Vhaduri, M.D. Wilson, and J.C. Keller. Assessing perceived stress, sleep disturbance, and fatigue among pilot and non-pilot trainees. Smart Health, 32, 100472, 2024. https://doi.org/10.101 6/j.smhl.2024.100472
72. X. Sun, C. Lan, and X. Mao, Eye locating arithmetic in fatigue detection based on image processing. Proceedings of 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1-5, Shanghai, China, 14-16 October 2017.
73. A.R. Bentivoglio, S.B. Bressman, E. Cassetta, D. Carretta, P. Tonali, and A. Albanese. Analysis of blink rate patterns in normal subjects. Movement disorders, 12 (6), 1028-1034, 1997. https://doi.org/10.1002/ mds.870120629