The effect of image preprocessing methods in CNN-based eye state detection

Abstract

This research investigates how eye state detection is used to overcome simple problems including blinking, eyestrain, and driving drowsiness. The study utilized the ZJU dataset, and it suggested an eye state recognition approach based on image preprocessing methods and a deep learning-based convolutional neural network (CNN). First, multiple pooling layers were tested in the suggested CNN model, and it was discovered that the average pooling performed the best in the results. The image preprocessing methods applied on the ZJU dataset were then trained on the proposed CNN model and their results have been compared. The CNN model performed exceptionally well on the ZJU dataset, according to a comparison of the results obtained using the histogram equalization method, with 94.32% accuracy, 94.95% sensitivity, 92.42% specificity, 97.41% precision, and 96.16% F1 score performance metrics. The results of this investigation were compared to prior studies on the ZJU dataset, which had provided performance measures. When compared to the literature, it was obtained that the proposed technique has a high classification performance in detecting vision problems.

Keywords

• Deep learning
• convolutional neural network
• histogram equalization
• eye state detection

ESA tabanlı göz durumu tespitinde görüntü önişlem yöntemlerinin etkisi

Öz

Bu çalışma göz kırpma, göz yorgunluğu, sürücü uyuşukluğu gibi temel problemlerin çözümlenmesinde önemli olan göz durumu tespitine odaklanmaktadır. Bu çalışmada, göz durumu tespiti için görüntü önişlem yöntemleri ve derin öğrenme tabanlı evrişimsel sinir ağına (ESA) dayanan bir yöntem önerilmiş ve önerilen yöntem ZJU veri seti üzerinde performansı test edilmiştir. Ayrıca, önerilen ESA modelinde farklı havuzlama katmanları değerlendirilmiş ve ZJU veriseti üzerinde elde edilen bulgularda ortalama havuzlama kullanılan önerilen ESA modelinin en iyi performansı elde ettiği görülmüştür. Sonrasında, ZJU veri setine görüntü ön işlem yöntemleri uygulanmış ve işlenmiş ZJU veri seti, önerilen ESA modelinde eğitilerek performansları karşılaştırılmıştır. Elde edilen sonuçlara göre histogram eşitleme yöntemi kullanılarak eğitimi gerçekleştirilen ESA modelinin ZJU veri setinde %94.32 doğruluk, %94.95 duyarlılık, %92.42 özgüllük, %97.41 kesinlik ve %96.16 F1 skor performans metrikleri ile üstün bir başarı elde ettiği görülmüştür. Bu çalışmada elde edilen sonuçlar, ZJU veri setinde yapılan önceki çalışmalarda sunulan performans metrikleri ile karşılaştırılmıştır. Önerilen yöntemin literatür ile karşılaştırıldığında, göz durumu tespitinde güçlü sınıflandırma performansına sahip olduğu tespit edilmiştir.

Anahtar Kelimeler

• Derin öğrenme
• evrişimsel sinir ağı
• histogram eşitleme
• göz durumu tespiti

Kaynakça

1. M. H. Yang, D. J. Kriegman and N. Ahuja, Detecting faces in images: A survey. IEEE Transactions on pattern analysis and machine intelligence, 24(1), 34-58, 2002. https://doi.org/10.1109/34.982883
2. T. Soukupova and J. Cech, Eye blink detection using facial landmarks. In 21st computer vision winter workshop, Rimske Toplice, Slovenia, 3-5 February 2006.
3. R. Huang, Y. Wang and L. Guo, P-FDCN based eye state analysis for fatigue detection. In 2018 IEEE 18th International Conference on Communication Technology (ICCT), pp. 1174-1178, Chongqing, China, 8-11 October 2018. https://doi.org/10.1109/ICCT.2018.8599947
4. D. M. Joshi, N.K. Rana and V. Misra, Classification of brain cancer using artificial neural network. In 2010 2nd international conference on electronic computer technology, pp. 112-116, 19-21 March 2010. https://doi.org/10.1109/ICECTECH.2010.5479975M
5. W. Zhao, R. Chellappa, P.J. Phillips and A. Rosenfeld, Face recognition: A literature survey. ACM computing surveys (CSUR), 35(4), 399-458, 2003. https://doi.org/10.1145/954339.954342
6. M. Divjak and H. Bischof, Eye Blink Based Fatigue Detection for Prevention of Computer Vision Syndrome. In MVA, pp. 350-353, 2009.
7. C. L. Chou, Y. H. Huang and S. C. Ho, Blink Detection Using Facial Landmark Blink Detector and Multi-Layer Perceptron. In NCS 2019, pp. 542-545, 2019.
8. M. Lalonde, D. Byrns, L Gagnon, N. Teasdale and D. Laurendeau, Real-time eye blink detection with GPU-based SIFT tracking. In Fourth Canadian Conference on Computer and Robot Vision (CRV'07), Montreal, Quebec, Canada, pp. 481-487, Montreal, Quebec, Canada, 28-30 May 2007. https://doi.org/ 10.1109/CRV.2007.54

9. J. He, W. Choi, Y. Yang, J. Lu, X. Wu and K. Peng, Detection of driver drowsiness using wearable devices: A feasibility study of the proximity sensor. Applied ergonomics, 65, 473-480, 2017. https://doi.org/10.1016/j.apergo.2017.02.016
10. M. Su, C. Yeh, S. Lin, P. Wang and S. Hou, An implementation of an eye-blink-based communication aid for people with severe disabilities. In 2008 International Conference on Audio, Language and Image Processing, pp. 351-356, Shanghai, China, 7-9 July, 2008. https://doi.org/ 10.1109/ICALIP.2008.4590090
11. S. Soltani and A. Mahnam, A practical efficient human computer interface based on saccadic eye movements for people with disabilities. Computers in biology and medicine, 70, 163-173, 2016. https://doi.org/10.1016/j.compbiomed.2016.01.012
12. Y. Dong, Y. Zhang, J. Yue and Z. Hu, Comparison of random forest, random ferns and support vector machine for eye state classification. Multimedia Tools and Applications, 75(19), 11763-11783, 2016. https://doi.org/10.1007/s11042-015-2635-0
13. L. Zhao, Z. Wang, G. Zhang, Y. Qi and X. Wang, Eye state recognition based on deep integrated neural network and transfer learning. Multimedia Tools and Applications, 77(15), 19415-19438, 2018. https://doi.org/10.1007/s11042-017-5380-8
14. Y. LeCun, Y. Bengio and G. Hinton, Deep learning. nature, 521(7553), 436-444, 2015. https://doi.org/10.1038/nature14539
15. L. Pauly and D. Sankar, Non intrusive eye blink detection from low resolution images using HOG-SVM classifier. International Journal of Image, Graphics and Signal Processing, 8(10), 11, 2016. https://doi.org/ 10.5815/ijigsp.2016.10.02
16. L. Pauly and D. Sankar, A novel method for eye tracking and blink detection in video frames. In 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), pp. 252-257, Bhubaneswar, India, 2-3 November 2015. https://doi.org/ 10.1109/CGVIS.2015.7449931
17. Y. J. Han, W. Kim and J. S. Park, Efficient eye-blinking detection on smartphones: A hybrid approach based on deep learning. Mobile Information Systems, 2018, 2018. https://doi.org/10.1155/2018/6929762
18. W. O. Lee, E. C. Lee and K. R. Park, Blink detection robust to various facial poses. Journal of neuroscience methods, 193(2), 356-372, 2010. https://doi.org/10.1016/j.jneumeth.2010.08.034
19. Y. S. Wu, T. W. Lee, Q. Z. Wu and H. S. Liu, An eye state recognition method for drowsiness detection. In 2010 IEEE 71st Vehicular Technology Conference, pp. 1-5, Taipei, Taiwan, 16-19 May 2010. https://doi.org/ 10.1109/VETECS.2010.5493951
20. B. Wu and R. Nevatia, Cluster boosted tree classifier for multi-view, multi-pose object detection. In 2007 IEEE 11th International Conference on Computer Vision, pp. 1-8, Rio de Janeiro, Brazil, 14-21 October 2007. https://doi.org/10.1109/ICCV.2007.4409006
21. L. Cadena, A. Zotin, F. Cadena, A. Korneeva, A. Legalov and B. Morales, Noise reduction techniques for processing of medical images. In Proceedings of the World Congress on Engineering Vol. 1, pp. 5-9, London, U.K., 5-7 July, 2017.
22. C. Saravanan, Color image to grayscale image conversion. In 2010 Second International Conference on Computer Engineering and Applications Vol. 2, pp. 196-199, Bali Island, 19-21 March 2010. https://doi.org/ 10.1109/ICCEA.2010.192
23. C. Munteanu and V. Lazarescu, Evolutionary contrast stretching and detail enhancement of satellite images. In Proc. Mendel Vol. 99, pp. 94-99, 1999.
24. M. Abdullah-Al-Wadud, M. H. Kabir, M. A. A. Dewan and O. Chae, A dynamic histogram equalization for image contrast enhancement. IEEE Transactions on Consumer Electronics, 53(2), 593-600, 2007.
25. S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz, T. Greer and K. Zuiderveld, Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing, 39(3), 355-368, 1987. https://doi.org/10.1016/S0734-189X(87)80186-X
26. G. Deng and L. W. Cahill, An adaptive Gaussian filter for noise reduction and edge detection. In 1993 IEEE conference record nuclear science symposium and medical imaging conference, pp. 1615-1619, San Francisco, CA, USA, 31 Oct-6 Nov 1993. https://doi.org/ 10.1109/NSSMIC.1993.373563
27. T. Chen, K. K. Ma and L. H. Chen, Tri-state median filter for image denoising. IEEE Transactions on Image processing, 8(12), 1834-1838, 1999.
28. Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu and M.S. Lew, Deep learning for visual understanding: A review. Neurocomputing, 187, 27-48, 2016. https://doi.org/10.1016/j.neucom.2015.09.116
29. I. Goodfellow, Y. Bengio and A. Courville, Deep learning. MIT press, London, England, 2016.
30. V. Suárez-Paniagua and I. Segura-Bedmar, Evaluation of pooling operations in convolutional architectures for drug-drug interaction extraction. BMC bioinformatics, 19(8), 39-47, 2018. https://doi.org/10.1186/s12859-018-2195-1