Research Article
BibTex RIS Cite

Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis

Year 2024, , 118 - 131, 14.06.2024
https://doi.org/10.33769/aupse.1378578

Abstract

Due to the limitations of the hardware system, analysis of retail stores has caused problems such as excessive workload, incomplete analysis, slow analysis speed, difficult data collection, non-real-time data collection, passenger flow statistics, and density analysis. However, heatmaps are a viable solution to these problems and provide adaptable and effective analysis. In this paper, we propose to use the deep sequence tracking algorithm together with the YOLO object recognition algorithm to create heatmap visualizations. We will present key innovations of our customized YOLO-Deep SORT system to solve some fundamental problems in in-store customer behavior analysis. These innovations include our use of footpad targeting to make bounding boxes more precise and less noisy. Finally, we made a comprehensive evaluation and comparison to determine the success rate of our system and found that the success rate was higher than the systems we compared in the literature. The results show that our heatmap visualization enables accurate, timely, and detailed analysis.

References

  • Liu, M., Lee, J., Kang, J., Liu, S., What we can learn from the data: a multiple-case study examining behavior patterns by students with different characteristics in using a serious game, Tech. Knowl. Learn., 21, (2016), 33-57, https://dx.doi.org/10.1007/s10758-015-9263-7.
  • Fernandez, N., Gundersen, G., Rahman, A., Grimes, M., Rikova, K., Hornbeck, P., Ma’ayan, A., Clustergrammer, A web-based heatmap visualization and analysis tool for high-dimensional biological data, Sci. Data, 4 (2017), 170151, https://dx.doi.org/10.1038/sdata.2017.151.
  • Gu, Z., Complex heatmap visualization, iMeta, 1 (3), (2022), https://doi.org/10.1002/imt2.43.
  • Deng, W., Wang, Y., Liu, Z., Cheng, H., Xue, Y., Hemi: a toolkit for illustrating heatmaps, PLoS ONE, 9 (11), (2014), https://doi.org/10.1371/journal.pone.0111988.
  • Mondal, S., Das, S., Musunuru, K., Dash, M., Study on the factors affecting customer purchase activity in retail stores by confirmatory factor analysis, ESPACIOS, 38 (61), 30 (2018).
  • Girgensohn, A., Shipman, F., Wilcox, L. D., Determining activity patterns in retail spaces through video analysis, Proc. ACM Conf. Multimedia (2008), 889-892, https://doi.org/10.1145/1459359.1459514.
  • Oliveira, K., RetailNet: A Deep Learning Approach for People Counting and Hot Spots Detection in Retail Stores, Rio de Janeiro, Brazil, 2019.
  • Onıga, F., Bacea, D., Single stage architecture for improved accuracy real-time object detection on mobile devices, Img. Vis. Comput., 130 (9), (2023), 104613, https://doi.org/10.1016/j.imavis.2022.104613.
  • Diwan, T., Anirudh, G., Tembhurne, J. V., Object detection using YOLO: challenges, architectural successors, datasets and applications, Multimed. Tools Appl., 82 (6), (2023), 9243-9275, https://doi.org/10.1007/s11042-022-13644-y.
  • Lakshmi Rishika, A., Aishwarya, Ch., Sahithi, A., Premchender, M., Real-time vehicle detection and tracking using yolo-based deep sort model: a computer vision application for traffic surveillance, Turkish J. Comp. Math. Edu., 14 (1), (2023), 255-264, https://doi.org/10.17762/turcomat.v14i1.13530.
  • Aich, S., Stavness, I., Improving object counting with heatmap regulation, (2018), https://doi.org/10.48550/arXiv.1803.05494.
  • Ilikci, B., Chen, L., Cho, H., Liu, O., Heat-map based emotion and face recognition from thermal images, Comput. Commun. IoT Appl., (2019), 449-453.
  • Bulat, A., Tzimiropoulos, G., Human Pose Estimation via Convolutional Part Heatmap Regression, Amsterdam, Netherlands, (2016).
  • Pharr, M., Humphreys, G., Bounding box, Physically Based Rendering, 3, (2017).
  • Huang, Z., Li, W., Xia, X.-G., Tao, R., A general Gaussian heatmap label assignment for arbitrary-oriented object detection, IEEE Transc. Img. Process., (2022), https://doi.org/10.1109/TIP.2022.3148874.
  • Salim, M. P., Ong, J. J., IS, E., Surhatono, D., Object detection for child learning media, Inter. Conf. Sci. Tech. (ICST), 8, Yogyakarta, Indonesia, (2022), 1-6.
  • He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X., Bounding box regression with uncertainty for accurate object detection, Proc. IEEE/CVF Conf. Comp.Vision Pattern Recog., (2019), 2888-2897, https://doi.org/10.48550/arXiv.1809.08545.
  • Hosang, J., Benenson, R., Schiele, B., Learning non-maximum suppression, Proc. IEEE Conf. Comp. Vision Pattern Recog. (CVPR), (2017), 4507-4515, https://doi.org/10.48550/arXiv.1705.02950.
  • Cordova-Esparza, M., Terven, J., A comprehensive review of yolo: from yolov1 to yolov8 and beyond, Mach. Learn. Knowl. Extr. 5, (2023), 1680-1716, https://doi.org/10.3390/make5040083.
  • Chandel, R., Gupta, G., Image filtering algorithms and techniques: a review, Int. J. Adv. Res. Comput. Sci. Softw. Eng., 3 (10), (2013).
  • Hicks, S. A., Strumke, I., Thambawita, V., Hammou, M., Riegler, M. A., Halvorsen, P., Parasa, S., On evaluation metrics for medical applications of artificial intelligence, Sci. Rep., 12 (1), (2022), 5979, https://doi.org/10.1038/s41598-022-09954-8.
  • Ajayi O. G. , Ashi J., Guda B., Performance evaluation of YOLO v5 model for automatic crop and weed classification on UAV images, Smart Agricult. Tech., 5, (2023), 100231.
  • Atik, M. E., Duran, Z, Ozgunluk, R., Comparison of YOLO versions for object detection from aerial images, Int. J. Environ. Geoinform, 9 (2), (2022), 87-93, https://doi.org/10.30897/ijegeo.1010741.
  • Karadağ, B., Arı, A., Akıllı mobil cihazlarda YOLOv7 modeli ile nesne tespiti, Politeknik J., 26 (3), (2023), 1207-1214, https://doi.org/10.2339/politeknik.1296541.
  • Özel, M. A., Baysal, S. S., Şahin, M., Derin öğrenme algoritması (YOLO) ile dinamik test süresince süspansiyon parçalarında çatlak tespiti, Eur. J. Sci. Technol, (26), (2021), 1-5, https://doi.org/10.31590/ejosat.952798.
  • Bayram, A. F., Nabiyev, V., Derin öğrenme tabanlı saklanan kamufle tankların tespiti: son teknoloji YOLO ağlarının karşılaştırmalı analizi, Gümüşhane Univ. J. Nat. Appl. Sci., 13 (4), (2023), 1082-1093, https://doi.org/10.17714/gumusfenbil.1271208.
Year 2024, , 118 - 131, 14.06.2024
https://doi.org/10.33769/aupse.1378578

Abstract

References

  • Liu, M., Lee, J., Kang, J., Liu, S., What we can learn from the data: a multiple-case study examining behavior patterns by students with different characteristics in using a serious game, Tech. Knowl. Learn., 21, (2016), 33-57, https://dx.doi.org/10.1007/s10758-015-9263-7.
  • Fernandez, N., Gundersen, G., Rahman, A., Grimes, M., Rikova, K., Hornbeck, P., Ma’ayan, A., Clustergrammer, A web-based heatmap visualization and analysis tool for high-dimensional biological data, Sci. Data, 4 (2017), 170151, https://dx.doi.org/10.1038/sdata.2017.151.
  • Gu, Z., Complex heatmap visualization, iMeta, 1 (3), (2022), https://doi.org/10.1002/imt2.43.
  • Deng, W., Wang, Y., Liu, Z., Cheng, H., Xue, Y., Hemi: a toolkit for illustrating heatmaps, PLoS ONE, 9 (11), (2014), https://doi.org/10.1371/journal.pone.0111988.
  • Mondal, S., Das, S., Musunuru, K., Dash, M., Study on the factors affecting customer purchase activity in retail stores by confirmatory factor analysis, ESPACIOS, 38 (61), 30 (2018).
  • Girgensohn, A., Shipman, F., Wilcox, L. D., Determining activity patterns in retail spaces through video analysis, Proc. ACM Conf. Multimedia (2008), 889-892, https://doi.org/10.1145/1459359.1459514.
  • Oliveira, K., RetailNet: A Deep Learning Approach for People Counting and Hot Spots Detection in Retail Stores, Rio de Janeiro, Brazil, 2019.
  • Onıga, F., Bacea, D., Single stage architecture for improved accuracy real-time object detection on mobile devices, Img. Vis. Comput., 130 (9), (2023), 104613, https://doi.org/10.1016/j.imavis.2022.104613.
  • Diwan, T., Anirudh, G., Tembhurne, J. V., Object detection using YOLO: challenges, architectural successors, datasets and applications, Multimed. Tools Appl., 82 (6), (2023), 9243-9275, https://doi.org/10.1007/s11042-022-13644-y.
  • Lakshmi Rishika, A., Aishwarya, Ch., Sahithi, A., Premchender, M., Real-time vehicle detection and tracking using yolo-based deep sort model: a computer vision application for traffic surveillance, Turkish J. Comp. Math. Edu., 14 (1), (2023), 255-264, https://doi.org/10.17762/turcomat.v14i1.13530.
  • Aich, S., Stavness, I., Improving object counting with heatmap regulation, (2018), https://doi.org/10.48550/arXiv.1803.05494.
  • Ilikci, B., Chen, L., Cho, H., Liu, O., Heat-map based emotion and face recognition from thermal images, Comput. Commun. IoT Appl., (2019), 449-453.
  • Bulat, A., Tzimiropoulos, G., Human Pose Estimation via Convolutional Part Heatmap Regression, Amsterdam, Netherlands, (2016).
  • Pharr, M., Humphreys, G., Bounding box, Physically Based Rendering, 3, (2017).
  • Huang, Z., Li, W., Xia, X.-G., Tao, R., A general Gaussian heatmap label assignment for arbitrary-oriented object detection, IEEE Transc. Img. Process., (2022), https://doi.org/10.1109/TIP.2022.3148874.
  • Salim, M. P., Ong, J. J., IS, E., Surhatono, D., Object detection for child learning media, Inter. Conf. Sci. Tech. (ICST), 8, Yogyakarta, Indonesia, (2022), 1-6.
  • He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X., Bounding box regression with uncertainty for accurate object detection, Proc. IEEE/CVF Conf. Comp.Vision Pattern Recog., (2019), 2888-2897, https://doi.org/10.48550/arXiv.1809.08545.
  • Hosang, J., Benenson, R., Schiele, B., Learning non-maximum suppression, Proc. IEEE Conf. Comp. Vision Pattern Recog. (CVPR), (2017), 4507-4515, https://doi.org/10.48550/arXiv.1705.02950.
  • Cordova-Esparza, M., Terven, J., A comprehensive review of yolo: from yolov1 to yolov8 and beyond, Mach. Learn. Knowl. Extr. 5, (2023), 1680-1716, https://doi.org/10.3390/make5040083.
  • Chandel, R., Gupta, G., Image filtering algorithms and techniques: a review, Int. J. Adv. Res. Comput. Sci. Softw. Eng., 3 (10), (2013).
  • Hicks, S. A., Strumke, I., Thambawita, V., Hammou, M., Riegler, M. A., Halvorsen, P., Parasa, S., On evaluation metrics for medical applications of artificial intelligence, Sci. Rep., 12 (1), (2022), 5979, https://doi.org/10.1038/s41598-022-09954-8.
  • Ajayi O. G. , Ashi J., Guda B., Performance evaluation of YOLO v5 model for automatic crop and weed classification on UAV images, Smart Agricult. Tech., 5, (2023), 100231.
  • Atik, M. E., Duran, Z, Ozgunluk, R., Comparison of YOLO versions for object detection from aerial images, Int. J. Environ. Geoinform, 9 (2), (2022), 87-93, https://doi.org/10.30897/ijegeo.1010741.
  • Karadağ, B., Arı, A., Akıllı mobil cihazlarda YOLOv7 modeli ile nesne tespiti, Politeknik J., 26 (3), (2023), 1207-1214, https://doi.org/10.2339/politeknik.1296541.
  • Özel, M. A., Baysal, S. S., Şahin, M., Derin öğrenme algoritması (YOLO) ile dinamik test süresince süspansiyon parçalarında çatlak tespiti, Eur. J. Sci. Technol, (26), (2021), 1-5, https://doi.org/10.31590/ejosat.952798.
  • Bayram, A. F., Nabiyev, V., Derin öğrenme tabanlı saklanan kamufle tankların tespiti: son teknoloji YOLO ağlarının karşılaştırmalı analizi, Gümüşhane Univ. J. Nat. Appl. Sci., 13 (4), (2023), 1082-1093, https://doi.org/10.17714/gumusfenbil.1271208.
There are 26 citations in total.

Details

Primary Language English
Subjects Signal Processing
Journal Section Research Articles
Authors

Murat Şimşek 0000-0002-8648-3693

Mehmet Kemal Tekbaş 0009-0008-9877-6021

Early Pub Date April 7, 2024
Publication Date June 14, 2024
Submission Date October 19, 2023
Acceptance Date January 23, 2024
Published in Issue Year 2024

Cite

APA Şimşek, M., & Tekbaş, M. K. (2024). Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis. Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering, 66(1), 118-131. https://doi.org/10.33769/aupse.1378578
AMA Şimşek M, Tekbaş MK. Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis. Commun.Fac.Sci.Univ.Ank.Series A2-A3: Phys.Sci. and Eng. June 2024;66(1):118-131. doi:10.33769/aupse.1378578
Chicago Şimşek, Murat, and Mehmet Kemal Tekbaş. “Heatmap Creation With YOLO-Deep SORT System Customized for in-Store Customer Behavior Analysis”. Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering 66, no. 1 (June 2024): 118-31. https://doi.org/10.33769/aupse.1378578.
EndNote Şimşek M, Tekbaş MK (June 1, 2024) Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis. Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering 66 1 118–131.
IEEE M. Şimşek and M. K. Tekbaş, “Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis”, Commun.Fac.Sci.Univ.Ank.Series A2-A3: Phys.Sci. and Eng., vol. 66, no. 1, pp. 118–131, 2024, doi: 10.33769/aupse.1378578.
ISNAD Şimşek, Murat - Tekbaş, Mehmet Kemal. “Heatmap Creation With YOLO-Deep SORT System Customized for in-Store Customer Behavior Analysis”. Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering 66/1 (June 2024), 118-131. https://doi.org/10.33769/aupse.1378578.
JAMA Şimşek M, Tekbaş MK. Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis. Commun.Fac.Sci.Univ.Ank.Series A2-A3: Phys.Sci. and Eng. 2024;66:118–131.
MLA Şimşek, Murat and Mehmet Kemal Tekbaş. “Heatmap Creation With YOLO-Deep SORT System Customized for in-Store Customer Behavior Analysis”. Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering, vol. 66, no. 1, 2024, pp. 118-31, doi:10.33769/aupse.1378578.
Vancouver Şimşek M, Tekbaş MK. Heatmap creation with YOLO-Deep SORT system customized for in-store customer behavior analysis. Commun.Fac.Sci.Univ.Ank.Series A2-A3: Phys.Sci. and Eng. 2024;66(1):118-31.

Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.