A New Approach to Automatic Detection of Tactile Coating Surfaces with Deep Learning

Year 2024, Volume: 13 Issue: 4, 885 - 895, 31.12.2024

Abdil Karakan

https://doi.org/10.17798/bitlisfen.1432965

Abstract

In this study, tactile coating surfaces of visually impaired individuals were detected using the deep learning method. For this detection, 4 of the You Only Look Once (YOLO) architectures, one of the best deep learning methods, were used. No ready data set was used in the study. A unique and new data set was prepared for the study. For the data set, 6278 images were taken from tactile coating surfaces. Images for real-time applications were obtained from many different environments. The tactile coating surfaces in the pictures were labelled separately. A total of 9184 tags were made. The dataset was implemented in YOLOv5, YOLOv6, YOLOv7, and YOLOv8 architectures. The highest accuracy was achieved in the YOLOv8 architecture with an accuracy rate of 97%, F1-Score of 0.940, and mAP@.5 of 0.977. The model was applied with k-fold cross-validation to evaluate performance measurements. In order for the study to be used in real-time, the frame per second (FPS) was increased to 150.

Keywords

Tactile coating, Deep learning, Blind, Real-time detection, YOLO;

Ethical Statement

The study is complied with research and publication ethics.

References

• World Health Organization, Blindness and vision impairment, https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment. (30. 01. 2024).
• J. Lu, K. Siu, and P. Xu, "A comparative study of tactile paving design standards in different countries," 9th International Conference on Computer-Aided Industrial Design and Conceptual Design, Kumning, China, 753-758, 22-25 Nov. 2008. https://doi.org/10.1109/CAID/CD15004.2008
• A. Mancini, E. Frontoni, and P. Zingaretti, "Mechatronic System to Help Visually Impaired Users During Walking and Running," IEEE Transactions on Intelligent Transportation Systems, 2018, 19(2), 649-660. https://doi.org/10.1109/TITS.2017.2780621
• J. Redmon, and A. Farhadi, "Yolov3: An incremental improvement," Computer Vision and Pattern Recognition, 2018, 1-15. https://doi.org/10.48550/arXiv.1804.02767
• G. Huang, Z. Liu, and L. Maaten, "Weinberger, K. Q. Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii-USA, 2261-2269, 21-26 July 2017. https://doi.org/10.5555/2149960
• A. Garcia-Garcia, S. Orts, S. Oprea, V. Villena-Martinez, P. Martinez-Gonzalez, and J. Rodríguez, "A Survey on Deep Learning Techniques for Image and Video Semantic Segmentation," Applied Soft Computing, 2018, 70, 41-65. https://doi.org/10.1016/j.asoc.2018.05.018
• A. Krizhevsky, I. E. Sutskever, and G. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems, 2012, 25(2). https://doi.org/10.1145/3065386
• J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, "Ieee. ImageNet: A Large-Scale Hierarchical Image Database," IEEE-Computer-Society Conference on Computer Vision and Pattern Recognition Workshops, Miami Beach, FL, USA, 248-255, 20-25 Jun. 2009. https://doi.org/10.1109/CVPRWorkshops15504.2009
• O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, and F. F. Li, "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision, 2015, 115, 211-252. https://doi.org/10.48550/arXiv.1409.0575
• K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," Proceedings of the IEEE international conference on computer vision, Massachusetts, USA, 1026-1034, 7-13 Oct. 2015. https://doi.org/10.1109/ICCV33071.2015
• M. Tan, and Q. V. Le, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," 2019 International Conference on Machine Learning, California-USA, 6105-6114, 14-16 February 2019. https://doi.org/10.1109/COMITCon45641.2019
• R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, USA, 580-587, 25 Sept. 2014. https://doi.org/10.1109/CVPRW34339.2014
• W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A.C. Berg, "Ssd: Single shot multibox detector," European conference on computer vision(ECCV2016), Amsterdam,Netherlands, 21-37, 11-14 Oct. 2016. https://doi.org/10.1007/978-3-319-46466-4
• A. Kaya, A.S. Keçeli, and A.B Can, "Examination of various classification strategies in classification of lung nodule characteristics," Journal of the Faculty of Engineering and Architecture of Gazi University, 2019, 34(2), 709-725. https://doi.org/10.17341/gazimmfd.416530
• J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 779-788, 27-30 June 2016. https://doi.org/10.1109/CVPR33180.2016
• P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "Overfeat: Integrated recognition, localization and detection using convolutional networks", International Conference on Learning Representations, Scottsdale, Arizona, USA, 2-4 May 2013. https://doi.org/10.48550/arXiv.1312.6229
• Y. Du, N. Pan, Z. Xu. F. Deng, Y. Shen, and H. Kang, "Pavement distress detection and classification based on YOLO network," International Journal of Pavement Engineering, 2020, 1659-1672. https://doi.org/10.1080/10298436.2020.1714047
• J. Shen, N. Liu, H. Sun, X. Tao, and Q. Li, "Vehicle Detection in Aerial Photographs Based on Hyper Feature Map in Deep Convolutional Network," KSII Transactions on Internet & Information Systems, 2019, 13(4).No:4. https://doi.org/10.3837/tiis.2019.04.014
• K. Arai, Y. Ando, and M. Mizukawa, "Development of signal recognition improvement system in crosswalk", Jsme Robomec, 2008, pp.2A1- E12(1)-(2). https://doi.org/10.1299/jsmermd.2008._2A1-I03_1
• S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39,No:6. https://doi.org/10.1109/TPAMI.2012.205
• D. C. Einloft, M.C. Ghilardi, and I. H. Manssour, "Automatic Detection of Tactile Paving Surfaces in Indoor Environments", Workshop of Undergraduate Works (WUW) in the 29th Conference on Graphics, Patterns and Photographs, Sau Paul, Brazil, 4-7 Oct. 2016. https://doi.org/10.1109/SIBGRAPI.2016.001
• M. C. Ghilardi, R.C.O Macedo, and I. H. Manssour, "A New Approach for Automatic Detection of Tactile Paving Surfaces in Sidewalks", Procedia Computer Science, 2016, 80, 662-672. https://doi.org/10.1016/j.procs.2016.05.356
• X. Jie, W. Xiaochi, and F. Zhigang, "Research and implementation of blind sidewalk detection in portable eta system", International Forum on Information Technology and Applications, Kunming, China, 431-434, 16-18 July 2010. https://doi.org/10.1109/IFITA.2010.360
• T. Asami, and K. Ohnishi, "Crosswalk location, direction and pedestrian signal state extraction system for assisting the expedition of person with impaired vision", 10th France-Japan/8th Europe-Asia Congress on Mecatronics, Tokyo, Japan, 285-290, 27- 29 Nov. 2014. https://doi.org/10.1109/MECATRONICS33801.2014
• A. Kassim, T. Yasuno, M. S. Mohd, A. Hjshukor, H.I. Jaafar, F. Baharom, and F. Jafar, "Vision based of tactile paving detection method in navigation system for blind person", Jurnal Teknologi, 2015, 77, 20. https://doi.org/10.5772/intechopen.79886
• O. Yildiz, "Melanoma detection from dermoscopy photographs with deep learning methods: A comprehensive study", Journal of the Faculty of Engineering and Architecture of Gazi University, 2019, 34(4), 2241-2260. https://doi.org/10.17341/gazimmfd.435217
• E. Dandil, and R. Polattimur, "Dog Behavior Recognition and Tracking based on Faster R-CNN", Journal of the Faculty of Engineering and Architecture of Gazi University, 2019, 35(2), 819-834. https://doi.org/10.17341/gazimmfd.541677
• J.J. Lv, X.H. Shao, J.S. Huang, X.D. Zhou, and X. Zhou, "Data augmentation for face recognition". Neurocomputing, 2017, 230, 184-196. https://doi.org/10.1016/j.neucom.2016.12.025
• S. Longfei, Y. Zheng, Z. Zimu, Z. Xiaolong, W. Chenshu, and L. Yunhao, "Crossnavi: Enabling real-time crossroad navigation for the blind with commodity phones", In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 14, pages 787–798, New York, USA, 1-4 September 2014. https://doi.org/10.1145/2632048.2632083
• J. Wang, N. Wang, L. Li, and Z. Ren, "Real-time behavior detection and judgment of egg breeders based on YOLOv3", Neural Computing and Applications, 2019, 32, 2. https://doi.org/10.1007/s00521-019-04645-4
• W. Byung, Y. Sung, and J. Kang, "Brick path detection from shape pattern and texture feature", In System Integration (SII), 2011 IEEE/SICE International Symposium on, Kyoto, Japan, 78–83, 1 Dec 2011. https://doi.org/10.1109/SII.2011.6147423
• S. Shoval, J. Borenstein, and Y. Koren, "The NavBelt-a computerized travel aid for the blind based on mobile robotics technology", IEEE Transactions on Biomedical Engineering, 1998, 45(11), 1376-1386. https://doi.org/10.1109/10.725334
• J. Redmon, and A. Farhadi, "YOLO9000: Better, Faster, Stronger", 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii-USA, 6517- 6525, 21-26 July 2017. https://doi.org/10.1109/CVPR35066.2017
• C. Goutte, and E. Gaussier, "A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation", Proceedings of the 27th European conference on Advances in Information Retrieval Research, Spain, 345-359, 21-23 March 2005. https://doi.org/10.5555/2149960
• R. Redmon, and A. Farhadi, "Yolov3: An incremental improvement", Computer Vision and Pattern Recognition, 2018, 1-6. https://doi.org/10.1007/s11263-009-0275-4
• R. Pyun, Y. Kim, P. Wespe, and R. Gassert, "Advanced augmented white cane with obstacle height and distance feedback", IEEE Int. Conf. Rehabil. Robotics,Seattle, WA, USA, pp.1-6. 24-26 June 2013. https://doi.org/10.1109/ICORR21102.2013
• A. M. Kassim, M. H. Jamaluddin, M. R. Yaacob, N. S. N. Anwar, Z. M. Sani, and A. Noordin, "Design and development of MY 2nd EYE for person with impaired vision", IEEE Symposium on Computers & Informatics, Kuala Lumpur, Malaysia, pp.700-703. 20-23 March 2011. https://doi.org/10.1109/ISCI17961.2011
• D. Ahmetovic, C. Bernareggi, A. Gerino, and S. Mascetti, "Zebrarecognizer: Efficient and precise localization of pedestrian crossings", In Pattern Recognition (ICPR), International Conference on Pattern Recognition, Stockholm, Sweden, 2566–2571, 24-28 Aug 2014. https://doi.org/10.1109/ICPR33632.2014
• A. M. Kassim, H .İ. Jaafar, M. A. Azam, N. Abas, and T. Yasuno, "Design and development of navigation system by using rfid technology", In System Engineering and Technology (ICSET), IEEE International Conference on System Engineering and Technology, Shah Alam Malaysia, pages 258–262, 19-20 Aug. 2013. https://doi.org/10.1109/ICSEngT.2013.6650181
• K. W. M. Jiangyan, and X. Ping, "A comparative study of tactile paving design standards in different countries", International Conference on Computer-Aided Industrial Design and Conceptual Design, Kunming, China, pages 753–758, 22-25 Nov. 2008. https://doi.org/10.1109/CAID/CD15004.2008
• V. N. Murali, and J. M. Coughlan, "Smartphone-based crosswalk detection and localization for visually impaired pedestrians", In Multimedia and Expo Workshops (ICMEW), 2013 IEEE International Conference on, San Jose, California, USA, pages 1–7, 15-19 July 2013. https://doi.org 10.1109/ICMEW32311.2013
• H. Shuihua, P. Hangrong, Z. Chenyang, and T. Yingli, "Rgb-d image-based detection of stairs, pedestrian crosswalks and traffic signs", Journal of Visual Communication and Image Representation, 2014, 25(2): 263-272. https://doi.org/10.1016/j.jvcir.2013.11.005
• W. Shuihua, and T. Yingli, "Detecting stairs and pedestrian crosswalks for the blind by rgbd camera", In Bioinformatics and Biomedicine Workshops (BIBMW), 2012 IEEE International Conference on, Philidelphia, USA, 732–739, 4-7 Oct. 2012. https://doi.org/10.1109/BIBMW.2012.6470197
• C. Shorten, and T. M. Khoshgoftaar, "A survey on Image Data Augmentation for Deep Learning", Journal of Big Data, 2019, 6(1), No: 60. https://doi.org/10.1186/s40537-019-0197-0
• C. Y. Wang, A. Bochkovskiy, and H. Y. M. Liao, "YOLOv7: Trainable bag-of-freebies sets new stateof-the-art for real-time object detectors", Computer Vision and Pattern Recognition, 2022, 1-15. https://doi.org/10.48550/arXiv.2207.02696
• B. Hu, M. Zhu, L. Chen, L. Huang, P. Chen, and M. He, "Tree species identification method based on improved YOLOv7", IEEE 8th International Conference on Cloud Computing and Intelligent Systems, 622-627, Chengdu, China, 26-28 November 2022