Face Detection in near infrared images by ensembling different deep neural networks

Abstract

Face detection plays a crucial role in various areas such as security, healthcare, industry, biometrics, etc. It is essential to perform face detection accurately and efficiently in the mentioned areas regardless of the lighting conditions. However, detecting faces in poor lighting conditions can be challenging. To address this issue, this study presents an approach focusing on face detection on near infrared images, which can work efficiently even in insufficient lighting conditions. For this purpose, first of all, eight state-of-the-art face detection models in the literature were determined and compared visually and numerically to reveal their performance in near infrared images. Additionally, experiments were carried out in CPU and GPU environments to compare the running times of the face detection methods. The approach proposed in this study is based on the principle that a more successful and inclusive bounding-box can be produced by using a bounding-box ensemble generated by more than one face detection model. Based on this, two new bounding-box ensemble methods called Combinational Voting and Non-Maximum Averaging are proposed. The proposed methods are compared with bounding-box ensemble methods in the literature. Combinational Voting produces more successful results than the other methods in the literature, with an accuracy rate of 93.6%.

Keywords

• Face detection
• near-infrared images
• bounding box ensemble

Project Number

FYL-2023-12497

Farklı derin sinir ağı modelleri birleştirilerek yakın kızılötesi görüntülerde yüz tespiti

Abstract

Yüz tespiti, güvenlik, sağlık, endüstri, biyometri gibi alanlarda kritik bir rol oynamaktadır. Bahsi geçen alanlarda, aydınlatma durumundan bağımsız olarak yüz tespitinin doğru ve verimli bir şekilde gerçekleştirilebilmesi büyük öneme sahiptir. Ancak, yetersiz aydınlatma koşullarında verimli bir şekilde yüz tespiti yapmak zor olabilmektedir. Bu problem doğrultusunda bu çalışmada, yetersiz aydınlatma koşullarında dahi verimli çalışabilen, yakın kızılötesi görüntüler üzerinde yüz tespitine odaklanan bir yaklaşım sunulmaktadır. Bu amaçla öncelikle literatürde iyi bilinen sekiz yüz tespiti derin sinir ağı modeli belirlenmiş ve yakın kızılötesi görüntülerdeki başarımlarını ortaya koymak amacıyla görsel ve sayısal olarak karşılaştırılmıştır. Ayrıca ilgili yöntemlerin çalışma zamanlarının karşılaştırmasına yönelik deneyler CPU ve GPU ortamında gerçekleştirilmiştir. Çalışma kapsamında önerilen yaklaşım birden fazla yüz tespiti modelinin ürettiği sınırlayıcı kutu topluluklarının beraber analiz edilmesi ile daha başarılı ve daha kapsayıcı yeni sınırlayıcı kutular üretilebileceği ilkesine dayanmaktadır. Buradan hareketle, Kombinasyonel Oylama ve Maksimum Olmayan Ortalama adları verilen iki yeni sınırlayıcı kutu belirleme yöntemi önerilmektedir. Önerilen yöntemler literatürdeki sınırlayıcı kutu belirleme yöntemleri ile karşılaştırılmıştır. Kombinasyonel Oylama yönteminin, ürettiği %93,6 doğruluk değeriyle literatürdeki sınırlayıcı kutu belirleme yöntemlerinden daha başarılı sonuçlar ortaya koyduğu görülmüştür.

Keywords

• Yüz tespiti
• yakın kızılötesi görüntüler
• sınırlayıcı kutu topluluğu birleştirme

Supporting Institution

Erciyes Üniversitesi Bilimsel Araştırma Projeleri Birimi

Project Number

FYL-2023-12497

Ethical Statement

Gerçekleştirilen çalışma özgün bir içeriğe sahip olup etik davranışlar ve akademik kurallar çerçevesinde yapılmıştır.

Thanks

Bu çalışma, Erciyes Üniversitesi Bilimsel Araştırma Projeleri Birimi tarafından FYL-2023-12497 kodlu proje kapsamında desteklenmiştir.

References

1. Kumar, A., Kaur, A., & Kumar, M. Face detection techniques: a review. Artificial Intelligence Review, 52, 927-948. 2019.
2. Cho, S. W., Baek, N. R., Kim, M. C., Koo, J. H., Kim, J. H., & Park, K. R. Face detection in nighttime images using visible-light camera sensors with two-step faster region-based convolutional neural network. Sensors (Switzerland), 18(9). 2018. https://doi.org/10,3390/s18092995
3. Li, J., Zhang, D., Zhang, K., Hu, K., & Yang, L. Real-time face detection during the night. 2017 4th International Conference on Systems and Informatics, ICSAI 2017, 2018-January. https://doi.org/10,1109/ICSAI.2017.8248358
4. S Liao, A K Jain and S Z. Li, "A fast and accurate unconstrained face detector[J]", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 38, no. 2, pp. 211-223, 2016.
5. Wang, W., Wang, X., Yang, W., & Liu, J. Unsupervised face detection in the dark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 1250–1266. 2023. https://doi.org/10,1109/TPAMI.2022.3152562
6. Li, J., Wang, Y., Wang, C., Tai, Y., Qian, J., Yang, J., Wang, C., Li, J., & Huang, F. DSFD: Dual shot face detector. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019-June, 5055–5064. https://doi.org/10,1109/CVPR.2019.00520
7. Dash, P., Kisku, D. R., Gupta, P., & Sing, J. K. Fast face detection using a unified architecture for unconstrained and infrared face images. Cognitive Systems Research, 74, 18-38. 2022.
8. Gao, Z., Zhang, S., Fang, H., Li, L., & Huang, L. Multi-modal image fusion based improved face detection algorithm in poor lighting conditions. ACM International Conference Proceeding Series. 2021. https://doi.org/10,1145/3483207.3483213

9. King, D. E. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10, 2009.
10. Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., & Li, S. Z. FaceBoxes: A CPU real-time face detector with high accuracy. IEEE International Joint Conference on Biometrics, IJCB 2017, 2018-January,1–9.2018. https://doi.org/10,1109/BTAS.2017.8272675
11. Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., & Grundmann, M. BlazeFace: Sub-millisecond neural face detection on mobile GPUs. 2019. http://arxiv.org/abs/1907.05047
12. Xu, Y., Yan, W., Yang, G., Luo, J., Li, T., & He, J. CenterFace: Joint face detection and alignment using face as point. Scientific Programming, 2020, 1–8. https://doi.org/10,1155/2020/7845384
13. Deng, J., Guo, J., Ververas, E., Kotsia, I., & Zafeiriou, S. (2020). RetinaFace: Single-shot multi-level face localisation in the wild. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5202–5211. 2020, https://doi.org/10,1109/CVPR42600,2020,00525
14. Guo, J., Deng, J., Lattas, A., & Zafeiriou, S. Sample and computation redistribution for efficient face detection. ICLR 2022 - 10th International Conference on Learning Representations.2022.
15. Wu, W., Peng, H., & Yu, S. YuNet: A tiny millisecond-level face detector. Machine Intelligence Research. 2023. https://doi.org/10,1007/s11633-023-1423-y
16. Neubeck, A., & van Gool, L. Efficient non-maximum suppression. Proceedings - International Conference on Pattern Recognition, 3. 2006. https://doi.org/10,1109/ICPR.2006.479
17. Solovyev, R., Wang, W., & Gabruseva, T. Weighted boxes fusion: Ensembling boxes from different object detection models. 2019. https://doi.org/10,1016/j.imavis.2021.104117
18. C. Ning, H. Zhou, Y. Song, and J. Tang. Inception single shot multibox detector for object detection. In ICME, 2017.
19. Panetta, Karen, Qianwen Wan, Sos Agaian, Srijith Rajeev, Shreyas Kamath, Rahul Rajendran, Shishir Rao et al. "A comprehensive database for benchmarking imaging systems." IEEE Transactions on Pattern Analysis and Machine Intelligence. 2018.
20. TUFTS Face DB, TUFTS yüz görüntüsü veri seti depolama alanı, http://tdface.ece.tufts.edu/, son erişim 23 Kasım 2023.
21. Zou, Z., Chen, K., Shi, Z., Guo, Y., & Ye, J. Object detection in 20 years: A survey. Proceedings of the IEEE. 2023.
22. Mamieva, D., Abdusalomov, A. B., Mukhiddinov, M., & Whangbo, T. K. Improved face detection method via learning small faces on hard images based on a deep learning approach. Sensors, 23(1), 502. 2023.
23. Adouani, A., Henia, W. M. ben, & Lachiri, Z. A comparison of face detection methods using spontaneous videos. Multimedia Tools and Applications, 81(16), 23163–23191. 2022. https://doi.org/10,1007/s11042-022-12781-8
24. LabelImg, Tzutalin LabelImg Git code (2015). https://github.com/tzutalin/labelImg, son erişim 24 Kasım 2023.
25. R. Vaillant, C. Monrocq, and Y. Le Cun. Original approach for the localisation of objects in images. IEE Proceedings Vision, Image and Signal Processing, 1994.
26. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. SSD: Single shot multibox detector. Lecture Notes in Computer Science (Including subseries lecture notes in artificial ıntelligence and lecture notes in bioinformatics), 9905 LNCS. 2016. https://doi.org/10,1007/978-3-319-46448-0_2
27. Girshick, R. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 1440-1448). 2015.
28. Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 2015-January.
29. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2018. https://doi.org/10,1109/CVPR.2018.00474
30. Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. Feature pyramid networks for object detection. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January. https://doi.org/10,1109/CVPR.2017.106
31. He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December. https://doi.org/10,1109/CVPR.2016.90
32. Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollár, P. Designing network design spaces. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2020, https://doi.org/10,1109/CVPR42600,2020,01044
33. Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. Path Aggregation Network for Instance Segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2018. https://doi.org/10,1109/CVPR.2018.00913
34. Zhou, H., Li, Z., Ning, C., & Tang, J. CAD: Scale ınvariant framework for real-time object detection. proceedings - 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017, 2018-January. https://doi.org/10,1109/ICCVW.2017.95