Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis

Abstract

Due to the limitations of the hardware system, analysis of retail stores has caused problems such as excessive workload, incomplete analysis, slow analysis speed, difficult data collection, non-real-time data collection, passenger flow statistics, and density analysis. However, heatmaps are a viable solution to these problems and provide adaptable and effective analysis. In this paper, we propose to use the deep sequence tracking algorithm together with the YOLO object recognition algorithm to create heatmap visualizations. We will present key innovations of our customized YOLO-Deep SORT system to solve some fundamental problems in in-store customer behavior analysis. These innovations include our use of footpad targeting to make bounding boxes more precise and less noisy. Finally, we made a comprehensive evaluation and comparison to determine the success rate of our system and found that the success rate was higher than the systems we compared in the literature. The results show that our heatmap visualization enables accurate, timely, and detailed analysis.

Keywords

• Heatmap visualization
• retail analysis
• YOLO
• object detection
• object tracking
• machine learning
• deep learning

References

1. Liu, M., Lee, J., Kang, J., Liu, S., What we can learn from the data: a multiple-case study examining behavior patterns by students with different characteristics in using a serious game, Tech. Knowl. Learn., 21, (2016), 33-57, https://dx.doi.org/10.1007/s10758-015-9263-7.
2. Fernandez, N., Gundersen, G., Rahman, A., Grimes, M., Rikova, K., Hornbeck, P., Ma’ayan, A., Clustergrammer, A web-based heatmap visualization and analysis tool for high-dimensional biological data, Sci. Data, 4 (2017), 170151, https://dx.doi.org/10.1038/sdata.2017.151.
3. Gu, Z., Complex heatmap visualization, iMeta, 1 (3), (2022), https://doi.org/10.1002/imt2.43.
4. Deng, W., Wang, Y., Liu, Z., Cheng, H., Xue, Y., Hemi: a toolkit for illustrating heatmaps, PLoS ONE, 9 (11), (2014), https://doi.org/10.1371/journal.pone.0111988.
5. Mondal, S., Das, S., Musunuru, K., Dash, M., Study on the factors affecting customer purchase activity in retail stores by confirmatory factor analysis, ESPACIOS, 38 (61), 30 (2018).
6. Girgensohn, A., Shipman, F., Wilcox, L. D., Determining activity patterns in retail spaces through video analysis, Proc. ACM Conf. Multimedia (2008), 889-892, https://doi.org/10.1145/1459359.1459514.
7. Oliveira, K., RetailNet: A Deep Learning Approach for People Counting and Hot Spots Detection in Retail Stores, Rio de Janeiro, Brazil, 2019.
8. Onıga, F., Bacea, D., Single stage architecture for improved accuracy real-time object detection on mobile devices, Img. Vis. Comput., 130 (9), (2023), 104613, https://doi.org/10.1016/j.imavis.2022.104613.

9. Diwan, T., Anirudh, G., Tembhurne, J. V., Object detection using YOLO: challenges, architectural successors, datasets and applications, Multimed. Tools Appl., 82 (6), (2023), 9243-9275, https://doi.org/10.1007/s11042-022-13644-y.
10. Lakshmi Rishika, A., Aishwarya, Ch., Sahithi, A., Premchender, M., Real-time vehicle detection and tracking using yolo-based deep sort model: a computer vision application for traffic surveillance, Turkish J. Comp. Math. Edu., 14 (1), (2023), 255-264, https://doi.org/10.17762/turcomat.v14i1.13530.
11. Aich, S., Stavness, I., Improving object counting with heatmap regulation, (2018), https://doi.org/10.48550/arXiv.1803.05494.
12. Ilikci, B., Chen, L., Cho, H., Liu, O., Heat-map based emotion and face recognition from thermal images, Comput. Commun. IoT Appl., (2019), 449-453.
13. Bulat, A., Tzimiropoulos, G., Human Pose Estimation via Convolutional Part Heatmap Regression, Amsterdam, Netherlands, (2016).
14. Pharr, M., Humphreys, G., Bounding box, Physically Based Rendering, 3, (2017).
15. Huang, Z., Li, W., Xia, X.-G., Tao, R., A general Gaussian heatmap label assignment for arbitrary-oriented object detection, IEEE Transc. Img. Process., (2022), https://doi.org/10.1109/TIP.2022.3148874.
16. Salim, M. P., Ong, J. J., IS, E., Surhatono, D., Object detection for child learning media, Inter. Conf. Sci. Tech. (ICST), 8, Yogyakarta, Indonesia, (2022), 1-6.
17. He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X., Bounding box regression with uncertainty for accurate object detection, Proc. IEEE/CVF Conf. Comp.Vision Pattern Recog., (2019), 2888-2897, https://doi.org/10.48550/arXiv.1809.08545.
18. Hosang, J., Benenson, R., Schiele, B., Learning non-maximum suppression, Proc. IEEE Conf. Comp. Vision Pattern Recog. (CVPR), (2017), 4507-4515, https://doi.org/10.48550/arXiv.1705.02950.
19. Cordova-Esparza, M., Terven, J., A comprehensive review of yolo: from yolov1 to yolov8 and beyond, Mach. Learn. Knowl. Extr. 5, (2023), 1680-1716, https://doi.org/10.3390/make5040083.
20. Chandel, R., Gupta, G., Image filtering algorithms and techniques: a review, Int. J. Adv. Res. Comput. Sci. Softw. Eng., 3 (10), (2013).
21. Hicks, S. A., Strumke, I., Thambawita, V., Hammou, M., Riegler, M. A., Halvorsen, P., Parasa, S., On evaluation metrics for medical applications of artificial intelligence, Sci. Rep., 12 (1), (2022), 5979, https://doi.org/10.1038/s41598-022-09954-8.
22. Ajayi O. G. , Ashi J., Guda B., Performance evaluation of YOLO v5 model for automatic crop and weed classification on UAV images, Smart Agricult. Tech., 5, (2023), 100231.
23. Atik, M. E., Duran, Z, Ozgunluk, R., Comparison of YOLO versions for object detection from aerial images, Int. J. Environ. Geoinform, 9 (2), (2022), 87-93, https://doi.org/10.30897/ijegeo.1010741.
24. Karadağ, B., Arı, A., Akıllı mobil cihazlarda YOLOv7 modeli ile nesne tespiti, Politeknik J., 26 (3), (2023), 1207-1214, https://doi.org/10.2339/politeknik.1296541.
25. Özel, M. A., Baysal, S. S., Şahin, M., Derin öğrenme algoritması (YOLO) ile dinamik test süresince süspansiyon parçalarında çatlak tespiti, Eur. J. Sci. Technol, (26), (2021), 1-5, https://doi.org/10.31590/ejosat.952798.
26. Bayram, A. F., Nabiyev, V., Derin öğrenme tabanlı saklanan kamufle tankların tespiti: son teknoloji YOLO ağlarının karşılaştırmalı analizi, Gümüşhane Univ. J. Nat. Appl. Sci., 13 (4), (2023), 1082-1093, https://doi.org/10.17714/gumusfenbil.1271208.