A Hybrid Approach to Person Recognition and Tracking with Powerful Representation Methods

Year 2024, Volume: 24 Issue: 6, 1333 - 1345, 02.12.2024

Gizem Ortaç Koşun , Seçkin Yılmaz , Rüya Şamlı

https://doi.org/10.35414/akufemubid.1388032

Abstract

Tracking people is a difficult task in surveillance carried out in public and crowded areas and in manual monitoring without an automatic tracking system. The proposed study was developed to solve this problem and contribute to the literature. The hybrid system is designed for identification and tracking of individuals in surveillance systems. The person who is wanted to be followed in the videos recorded by the camera systems is segmented using the Mask R-CNN method and a stronger representation vector is created by combining the features extracted with different techniques. This representation vector makes it easier to search for people in videos and track them more effectively with an automated system. In the study, a powerful representation is created using feature extraction techniques consisting of color histograms, Gabor filters, directed gradient histogram and VGG16 architecture. This representation enables better identification of people, providing faster and more effective search performance in videos. The study provides an important infrastructure to facilitate the identification and tracking of usual suspects in situations where security personnel have difficulty in facial recognition. It provides an application example that can be used in the real world by ensuring that the interrogated person is detected and marked in each video frame. The results obtained were interpreted using precision, sensitivity and F score metrics.

Keywords

Person Recognition, Person Tracking, Surveillance video, Person reidentification

Güçlü Temsil Yöntemleri ile Kişi Tanıma ve Takibi için Hibrit Bir Yaklaşım

Abstract

Halka açık ve kalabalık alanlarda yapılan gözetimlerde, otomatik bir takip sistemi olmadan manuel olarak gerçekleştirilen izlemelerde kişilerin takibi zor bir görevdir. Önerilen çalışma, bu sorunu çözmek ve literatüre katkı sağlamak amacıyla geliştirilmiştir. Hibrit sistem, gözetim sistemlerinde kişilerin tanımlanması ve takibi için tasarlanmıştır. Kamera sistemlerinin kaydettiği videolarda takip edilmek istenen kişi, Mask R-CNN yöntemi kullanılarak segmente edilir ve farklı tekniklerle çıkarılan özellikleri birleştirilerek daha güçlü bir temsil vektörü oluşturulur. Bu temsil vektörü, kişilerin videolarda otomatik bir sistemle aranmasını ve daha etkili bir şekilde takip edilmesini kolaylaştırır. Çalışmada, renk histogramları, Gabor filtreleri, yönlendirilmiş gradyan histogramı ve VGG16 mimarisinden oluşan özellik çıkarım teknikleri kullanılarak güçlü bir temsil oluşturulmaktadır. Bu temsil, kişilerin daha iyi tanımlanmasını sağlayarak videolarda daha hızlı ve etkin bir arama performansı sunar. Çalışma, güvenlik personelinin yüz tanıma zorluğu yaşadığı durumlarda olağan şüpheli kişilerin tanımlanması ve takibini kolaylaştırmak için önemli bir altyapı sağlar. Sorgulanan kişinin her bir video karesinde tespit edilip işaretlenmesini sağlayarak gerçek dünyada kullanılabilir bir uygulama örneği sunar. Elde edilen sonuçlar, kesinlik, duyarlılık ve F skoru metrikleri kullanılarak yorumlanmıştır.

Keywords

Kişi Tanıma, Kişi Takibi, Gözetim Videoları, Kişi yeniden tanımlama

References

• Almasawa, M.O., Elrefaei, L.A., and Moria, K. 2019. A survey on deep learning-based person re-identification systems. IEEE Access, 7, 175228-175247. https://doi.org/10.1109/ACCESS.2019.2957336
• Battal, A. ve Tuncer, A. 2022. Detection of Face Mask Wearing Condition for COVID-19 Using Mask R-CNN. El-Cezeri, 9(3), 1051-1060. https://doi.org/10.31202/ecjse.1061270
• Bäuml, M. and Stiefelhagen, R,. 2011. Evaluation of Local Features for Person Re-Identification in Image Sequences. In Proceedings of the International Conference on Advanced Video and Signal-based Surveillance (AVSS), 291-296. https://doi.org/10.1109/AVSS.2011.6027339
• Chung, D., Tahboub, K. and Delp, E.J. 2017. A two stream siamese convolutional neural network for person re-identification. In Proceedings of the IEEE international conference on computer vision, 1983-1991.
• Gkelios, S., Sophokleous, A., Plakias, S., Boutalis, Y. and Chatzichristofis, S.A. 2021. Deep convolutional features for image retrieval. Expert Systems with Applications, 177, 114940. https://doi.org/10.1016/j.eswa.2021.114940 He, K., Gkioxari, G., Dollár, P. and Girshick, R. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2961-2969. https://doi.org/10.1109/ICCV.2017.322
• He, K., Zhang, X., Ren, S. and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778. https://doi.org/10.1109/CVPR.2016.90.
• Leng, Q., Ye, M. and Tian, Q. 2019. A survey of open-world person re-identification. IEEE Transactions on Circuits and Systems for Video Technology, 30(4), 1092-1108.
• Li, W., Mao, K., Zhang, H. and Chai, T. 2010. Designing compact Gabor filter banks for efficient texture feature extraction. In Proceedings of the IEEE International Conference on Control Automation Robotics and Vision, 1193-1197.
• Li, Y., Xie, S., Chen, X., Dollar, P., He, K. and Girshick, R. 2021. Benchmarking detection transfer learning with vision transformers. https://doi.org/10.48550/arXiv.2111.11429
• Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D. and Zitnick, C.L. 2014. Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, Proceedings, Part V 13. Springer International Publishing, 740-755. https://doi.org/10.1007/978-3-319-10602-1_48
• Liu, F., Wang, Y., Wang, F.C., Zhang, Y.Z. and Lin, J. 2019. Intelligent and secure content-based image retrieval for mobile users. IEEE Access, 7, 119209-119222. https://doi.org/10.1109/ACCESS.2019.2935222
• Luo, H., Gu, Y., Liao, X., Lai, S. and Jiang, W. 2019. Bag of tricks and a strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 0-0. https://doi.org/10.48550/arXiv.1903.07071
• Ma, L., Liu, H., Hu, L., Wang, C. and Sun, Q. 2016. Orientation driven bag of appearances for person re-identification. arXiv preprint arXiv:1605.02464. https://doi.org/10.48550/arXiv.1605.02464
• Miao, J., Zhu, W 2022. Precision–recall curve (PRC) classification trees. Evol. Intel. 15, 1545–1569 https://doi.org/10.1007/s12065-021-00565-2
• Rahutomo, F., Kitasuka, T. and Aritsugi, M. 2012. Semantic cosine similarity. In Proceedings of the International student conference on advanced science and technology-ICAST. South Korea: University of Seoul,1.
• Roshan, S., Srivathsan, G., Deepak, K. and Chandrakala, S. 2020. Violence detection in automated video surveillance: Recent trends and comparative studies. The Cognitive Approach in Cloud Computing and Internet of Things Technologies for Surveillance Tracking Systems, Ch.11, 157-171. https://doi.org/10.1016/B978-0-12-816385-6.00011-8
• Simonyan, K. and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. https://doi.org/10.48550/arXiv.1409.1556
• Singh, J. and Shekhar, S. 2018. Road damage detection and classification in smartphone captured images using mask r-cnn. arXiv preprint arXiv:1811.04535. https://doi.org/10.48550/arXiv.1811.04535
• Su, C., Zhang, S., Xing, J., Gao, W. and Tian, Q. 2016. Deep attributes driven multi-camera person re-identification. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer International Publishing, 475-491. https://doi.org/10.48550/arXiv.1605.03259
• Tharsanee, R.M., Soundariya, R.S., Kumar, A.S., Karthiga, M. and Sountharrajan, S. 2021. Deep convolutional neural network–based image classification for COVID-19 diagnosis. In Data Science for COVID-19. https://doi.org/10.1016/B978-0-12-824536-1.00012-5
• Velmurugan, K. and Baboo, S.S. 2011. Image retrieval using Harris corners and histogram of oriented gradients. International Journal of Computer Applications, 24(7), 6-10. https://doi.org/10.5120/2968-3968
• Wu, L., Wang, Y., Gao, J. and Li, X. 2018. Where-and-when to look: Deep siamese attention networks for video-based person re-identification. IEEE Transactions on Multimedia, 21(6), 1412-1424. https://doi.org/10.48550/arXiv.1808.01911
• Xiao, J., Li, S. and Xu, Q. 2019. Video-based evidence analysis and extraction in digital forensic investigation. IEEE Access, 7, 55432-55442. https://doi.org/10.1109/ACCESS.2019.2913648
• Yao, H., Zhang, S., Zhang, D., Zhang, Y., Li, J., Wang, Y. and Tian, Q. 2017. Large-scale person re-identification as retrieval. In Proceedings of the International Conference on Multimedia and Expo (ICME), 1440-1445. https://doi.org/10.1109/ICME.2017.8019485
• Yi, D., Lei, Z. and Li, S.Z. 2014. Deep metric learning for practical person re-identification, 34-39. ArXiv e-prints, 89. https://doi.org/10.48550/arXiv.1407.4979
• Zekri, K., Touzi, A.G. and Lachiri, Z. 2017. A comparative study of texture descriptor analysis for improving content based image retrieval. In Proceedings of the International International Conference on Control, Automation and Diagnosis (ICCAD), 247-253. https://doi.org/10.1109/CADIAG.2017.8075665.