Compliance Detection for Occupational Health and Safety of Employees: Fatigue Detection and Personalized Reporting with Image and Sound Processing

Abstract

Advancements in technology today have enabled the emergence of new systems aimed at increasing worker safety in the field of occupational health and safety. However, even with all these safety measures in place, it must not be forgotten that a worker's fatigue plays a critical role in safety. A tired worker may struggle to comply with safety protocols, regardless of how many are in place. Therefore, especially in industrial tasks that require attention and diligence, determining whether workers are fatigued is considered of vital importance. The study focused on detecting fatigue to ensure workers can sustain their work healthily in the long and short term. In this context, real-time video footage was utilized, and facial detection was performed using image processing techniques, mapping specific reference points on the face. Eye and mouth openness, along with the level of head tilt, were identified as signs of fatigue, and these parameters were evaluated against threshold values. Additionally, during fatigue detection, audio and visual questions related to occupational health and safety were asked to measure the worker's attention and knowledge. Thus, it was attempted to determine how prepared the worker was for certain work activities and equipment. Furthermore, individual fatigue reports were prepared using facial recognition in the proposed system. As a result of experimental studies, the performance of the proposed system on the subjective data set was calculated as accuracy 80%, precision 85%, recall 73% and F1 score 75%. On the YawDD dataset, accuracy was determined as 95.99%, precision as 96.83%, recall as 95.58% and F1 score as 95.59%s.

Keywords

• Image processing
• Occupational health and Safety
• Speech processing
• Fatigue detection
• Face recognition

Çalışanın İş Sağlığı ve Güvenliği için Uygunluk Tespiti: Görüntü ve Ses İşleme ile Yorgunluk Tespiti ve Kişiye Özel Raporlama

Abstract

Günümüzde teknolojinin ilerlemesi, iş sağlığı ve güvenliği alanında çalışanların güvenliğini artırmaya yönelik yeni sistemlerin ortaya çıkmasına olanak tanımıştır. Ancak, tüm bu güvenlik önlemleri alındığında bile, işçinin yorgunluğunun güvenlikte kritik bir rol oynadığı unutulmamalıdır. Yorgun bir işçi, ne kadar güvenlik protokolü olursa olsun, bu protokolleri uygulamakta zorlanabilir. Bu nedenle, özellikle dikkat ve özen gerektiren endüstriyel görevlerde, çalışanların yorgun olup olmadığını belirlemenin hayati öneme sahip olduğu kabul edilmektedir. Çalışmada, işçilerin uzun ve kısa vadede işlerini sağlıklı bir şekilde sürdürebilmeleri adına yorgunluk tespitine odaklanıldı. Bu bağlamda, gerçek zamanlı video görüntülerini kullanarak, görüntü işleme teknikleriyle yüz tespiti gerçekleştirildi ve yüzdeki belirli referans noktaları haritalandı. Göz ve ağız açıklığı ile başın eğiklik seviyesi, yorgunluk belirtileri olarak belirlendi ve bu parametreler eşik değerlere göre değerlendirildi. Ayrıca, işçinin dikkatini ve bilgisini ölçmek amacıyla yorgunluk tespit sırasında iş sağlığı ve güvenliğiyle ilgili sesli ve görsel sorular da soruldu. Böylelikle işçinin belirli iş aktiviteleri ve ekipmanlar için ne kadar hazır olduğu belirlenmeye çalışıldı. Ek olarak, önerilen sistemde kullanılan yüz tanıma ile bireysel yorgunluk raporları hazırlandı. Deneysel çalışmalar sonucunda, önerilen sistemin öznel veri setindeki performansı doğruluk %80, kesinlik %85, duyarlılık %73 ve F1 skoru %75 olarak hesaplanmıştır. YawDD veri seti üzerinde ise doğruluk %95.99, kesinliği %96.83, duyarlılığı %95.58 ve F1 skoru %95.59 olarak belirlenmiştir.

Keywords

• Görüntü İşleme
• İş sağlığı ve güvenliği
• Konuşma işleme
• Yorgunluk tespiti
• Yüz tanıma

References

1. Ricci, J.A., Chee, E., Lorandeau, A.L. and Berger, J., (2007). Fatigue in the US workforce: prevalence and implications for lost productive work time. Journal of Occupational and Environmental Medicine, 1-10.
2. Arıtan, A.E. and Ataman, M., (2017). Kaza oranları hesaplamalarıyla iş kazası analizi. Afyon Kocatepe Üniversitesi Fen ve Mühendislik Bilimleri Dergisi, 17(1), 239-246.
3. Çavdar, U., Manyaslı, M., Akkaya, E., Sevener, D. and Tüfekçi, Z., (2022). Yaşanan iş kazalarının kaza saatlerine ve cinsiyete göre istatistiki olarak değerlendirilmesi ve yorumlanması. Journal of Engineering Research and Development, 14(1), 360-368.
4. Ansari, S., Naghdy, F., Du, H. and Pahnwar, Y.N., (2021). Driver mental fatigue detection based on head posture using new modified relu-bilstm deep neural network. IEEE Transactions on Intelligent Transportation Systems, 28(8), 10957-10969.
5. Cui, Z., Sun, H.M., Yin, R.N., Gao, L., Sun, H.B. and Jia, R.S., (2021). Real-time detection method of driver fatigue state based on deep learning of face video. Multimedia Tools and Applications, 80, 25495-25515.
6. Adhinata, F.D., Rakhmadani, D.P. and Wijayanto, D., (2021). Fatigue detection on face image using facenet algorithm and k-nearest neighbor classifier. Journal of Information Systems Engineering and Business Intelligence, 7(1), 22-30.
7. Li, X., Luo, J., Duan, C., Zhi, Y and Yin, P., (2021). Real-time detection of fatigue driving based on face recognition. In Journal of Physics: Conference Series, 1802(2), 022044.
8. Sikander, G. and Anwar S., (2018). Driver fatigue detection systems: a review. IEEE Transactions on Intelligent Transportation Systems, 20(6), 2339-2352.

9. Ji, Q., Lan, P. and Looney, C., (2006). A probabilistic framework for modeling and real-time monitoring human fatigue. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 36(5), 862-875.
10. Fang, Y., Liu, C., Zhao, C., Zhang, H., Wang, W. and Zou, N., (2022). A study of the effects of different indoor lighting environments on computer work fatigue. International Journal of Environmental Research and Public Health, 19(11), 6866.
11. Vegso, S., Cantley, L., Slade, M., Taiwo, O., Sircar, K., Rabinowitz, P., Fiellin, M., Russi, M.B. and Cullen, M.R., (2007). Extended work hours and risk of acute occupational injury: a case‐crossover study of workers in manufacturing. American Journal of İndustrial Medicine, 50(8), 597-603.
12. Huang, G.B., Mattar, M., Berg, T. and Learned-Miller, E., (2008). Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Workshop on Faces in real-Life Images: Detection, Alignment, and Recognition, Marseille, France, 17-18 Ekim, ECCV.
13. Mohanty, S., Hegde, S.V., Prasad, S. and Manikandan, J., (2019). Design of real-time drowsiness detection system using dlib. 2019 IEEE International WIE Conference on Electrical and Computer Engineering, Bangalore, India, 15-16 Kasım 2019, IEEE.
14. Dalal, N. and Triggs, B., (2005). Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 20-25 Haziran 2005, IEEE.
15. Kazemi, V. and Sullivan, J., (2014). One millisecond face alignment with an ensemble of regression trees. Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23-28 Haziran 2014, IEEE.
16. Internet Related Technologies, Github Dlib Models, https://github.com/davisking/dlib-models, (Ağustos 2023)
17. Suwarno, S., and Kevin, K., (2020). Analysis of Face recognition algorithm: Dlib and OpenCV. Journal of Informatics and Telecommunication Engineering, 4(1), 173-184.
18. Van den Berg, J., (2006). Sleepiness and Head Movements. Industrial Health, 44(4), 564-576.
19. Thulasimani, L., Poojeevan, P. and Prithashasni, S.P., (2021). Real time driver drowsiness detection using OpenCV and facial landmarks. International Journal of Aquatic Science, 12(2), 4297-4314.
20. Mirunalini, K. and David, Dr.V.K., (2021). Drowsiness detection using deep neural network. Turkish Journal of Computer and Mathematics Education, 12(9), 317-326.
21. Satt, A., Rozenberg, S. and Hoory, R., (2017). Efficient emotion recognition from speech using deep learning on spectrograms. Interspeech.
22. Srinivasan, A., (2011). Speech recognition using hidden markov model. Applied Mathematical Sciences, 5(79), 3943-3948.
23. Nassif, A.B., Shahin, I., Attili, I, Azzeh, M. and Shaalan, K., (2019). Speech recognition using deep neural networks: a systematic review. IEEE Access, 7, 19143-19165.
24. Mathur, M., Samiulla, S., Bhat, V. and Jenitta, J., (2020). Design and development of writing robot using speech processing. 2020 IEEE Bangalore Humanitarian Technology Conference, Vijiyapur, India, 8-10 Ekim 2020, IEEE
25. Onaolapo, J.O., Idachaba, F.E., Badejo, J.A. and Odu, O.I., (2014). A simplified overview of text-to-speech synthesis. Proceedings of the World Congress on Engineering, London, U.K., 2-4 Temmuz, WCE.
26. Muthumari, M., Akash, V., Charan, K.P., Akhil, P., Deepak, V. and Praveen, S.P., (2022). Smart and Multi-Way Attendance Tracking System Using an Image-Processing Technique. 2022 4th International Conference on Smart Systems and Inventive Technology, Tirunelveli, India, 20-22 Ocak, IEEE.
27. Watson, N.F., Badr, M.S., Belenky, G., Bliwise, D.L., Buxton, O.M., Buysse, D., Dinges, D.F., Gangwisch, J., Grandner, M.A., Kushida, C., Malhotra, R.K., Martin, J.L., Patel, S.R., Quan, S.F. and Tasali, E., (2015). Recommended amount of sleep for a healthy adult: a joint consensus statement of the American Academy of Sleep Medicine and Sleep Research Society. Journal of Clinical Sleep Medicine: JCSM: Official Publication of The American Academy of Sleep Medicine, 11(6), 591–592.
28. Öztürk, M., Küçükmanisa, A. and Urhan, O., (2022). Drowsiness detection system based on machine learning using eye state. Balkan Journal of Electrical and Computer Engineering, 10(3), 258-263.
29. Dewi, C., Chen, R.C., Chang, C.W., Wu, S.H., Jiang X. and Yu, H., (2022). Eye aspect ratio for real-time drowsiness detection to improve driver safety. Electronics, 11(19), 3183.
30. Al-gawwam, S. and Benaissa, M., (2018). Robust eye blink detection based on Eye Landmarks and Savitzky–Golay Filtering. Information, 9(4), 93.
31. Chellappa, Y., Joshi, N.N. and Bharadwaj, V., (2016). Driver fatigue detection system. 2016 IEEE International Conference on Signal and Image Processing, Bejing, China, 13-15 Ağustos, IEEE.
32. Majeed, F., Shafique, U., Safran, M., Alfarhood, S. and Ashraf, I., (2023). Detection of Drowsiness among Drivers Using Novel Deep Convolutional Neural Network Model. Sensors, 23(21), 8741.
33. Gu, W.H., Zhu, Y., Chen X.D., He, L.F. and Zheng, B.B., (2018). Hierarchical cnn‐based real‐time fatigue detection system by visual‐based technologies using msp model. IET Image Processing, 12(12), 2319-2329.
34. Rajkar, A., Kulkarni, N. And Raut, A., (2022). Driver drowsiness detection using deep learning. Applied Information Processing Systems, Proceedings of ICCET 2021, Lonere, India, 30-31 Ocak 2021, Springer.
35. Salman, R.M., Rashid, M., Roy, R., Ahsan, M.M. and Siddique, Z., (2021). Driver drowsiness detection using ensemble convolutional neural networks on YawDD. arXiv 2021, arXiv:2112.10298.