A New Approach to Automatic Detection of Tactile Coating Surfaces with Deep Learning

Abstract

In this study, tactile coating surfaces of visually impaired individuals were detected using the deep learning method. For this detection, 4 of the You Only Look Once (YOLO) architectures, one of the best deep learning methods, were used. No ready data set was used in the study. A unique and new data set was prepared for the study. For the data set, 6278 images were taken from tactile coating surfaces. Images for real-time applications were obtained from many different environments. The tactile coating surfaces in the pictures were labelled separately. A total of 9184 tags were made. The dataset was implemented in YOLOv5, YOLOv6, YOLOv7, and YOLOv8 architectures. The highest accuracy was achieved in the YOLOv8 architecture with an accuracy rate of 97%, F1-Score of 0.940, and mAP@.5 of 0.977. The model was applied with k-fold cross-validation to evaluate performance measurements. In order for the study to be used in real-time, the frame per second (FPS) was increased to 150.

Keywords

• Tactile coating
• Deep learning
• Blind
• Real-time detection
• YOLO;

Ethical Statement

The study is complied with research and publication ethics.

References

1. World Health Organization, Blindness and vision impairment, https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment. (30. 01. 2024).
2. J. Lu, K. Siu, and P. Xu, "A comparative study of tactile paving design standards in different countries," 9th International Conference on Computer-Aided Industrial Design and Conceptual Design, Kumning, China, 753-758, 22-25 Nov. 2008. https://doi.org/10.1109/CAID/CD15004.2008
3. A. Mancini, E. Frontoni, and P. Zingaretti, "Mechatronic System to Help Visually Impaired Users During Walking and Running," IEEE Transactions on Intelligent Transportation Systems, 2018, 19(2), 649-660. https://doi.org/10.1109/TITS.2017.2780621
4. J. Redmon, and A. Farhadi, "Yolov3: An incremental improvement," Computer Vision and Pattern Recognition, 2018, 1-15. https://doi.org/10.48550/arXiv.1804.02767
5. G. Huang, Z. Liu, and L. Maaten, "Weinberger, K. Q. Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii-USA, 2261-2269, 21-26 July 2017. https://doi.org/10.5555/2149960
6. A. Garcia-Garcia, S. Orts, S. Oprea, V. Villena-Martinez, P. Martinez-Gonzalez, and J. Rodríguez, "A Survey on Deep Learning Techniques for Image and Video Semantic Segmentation," Applied Soft Computing, 2018, 70, 41-65. https://doi.org/10.1016/j.asoc.2018.05.018
7. A. Krizhevsky, I. E. Sutskever, and G. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems, 2012, 25(2). https://doi.org/10.1145/3065386
8. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, "Ieee. ImageNet: A Large-Scale Hierarchical Image Database," IEEE-Computer-Society Conference on Computer Vision and Pattern Recognition Workshops, Miami Beach, FL, USA, 248-255, 20-25 Jun. 2009. https://doi.org/10.1109/CVPRWorkshops15504.2009

9. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, and F. F. Li, "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision, 2015, 115, 211-252. https://doi.org/10.48550/arXiv.1409.0575
10. K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," Proceedings of the IEEE international conference on computer vision, Massachusetts, USA, 1026-1034, 7-13 Oct. 2015. https://doi.org/10.1109/ICCV33071.2015
11. M. Tan, and Q. V. Le, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," 2019 International Conference on Machine Learning, California-USA, 6105-6114, 14-16 February 2019. https://doi.org/10.1109/COMITCon45641.2019
12. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, USA, 580-587, 25 Sept. 2014. https://doi.org/10.1109/CVPRW34339.2014
13. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A.C. Berg, "Ssd: Single shot multibox detector," European conference on computer vision(ECCV2016), Amsterdam,Netherlands, 21-37, 11-14 Oct. 2016. https://doi.org/10.1007/978-3-319-46466-4
14. A. Kaya, A.S. Keçeli, and A.B Can, "Examination of various classification strategies in classification of lung nodule characteristics," Journal of the Faculty of Engineering and Architecture of Gazi University, 2019, 34(2), 709-725. https://doi.org/10.17341/gazimmfd.416530
15. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 779-788, 27-30 June 2016. https://doi.org/10.1109/CVPR33180.2016
16. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "Overfeat: Integrated recognition, localization and detection using convolutional networks", International Conference on Learning Representations, Scottsdale, Arizona, USA, 2-4 May 2013. https://doi.org/10.48550/arXiv.1312.6229
17. Y. Du, N. Pan, Z. Xu. F. Deng, Y. Shen, and H. Kang, "Pavement distress detection and classification based on YOLO network," International Journal of Pavement Engineering, 2020, 1659-1672. https://doi.org/10.1080/10298436.2020.1714047
18. J. Shen, N. Liu, H. Sun, X. Tao, and Q. Li, "Vehicle Detection in Aerial Photographs Based on Hyper Feature Map in Deep Convolutional Network," KSII Transactions on Internet & Information Systems, 2019, 13(4).No:4. https://doi.org/10.3837/tiis.2019.04.014
19. K. Arai, Y. Ando, and M. Mizukawa, "Development of signal recognition improvement system in crosswalk", Jsme Robomec, 2008, pp.2A1- E12(1)-(2). https://doi.org/10.1299/jsmermd.2008._2A1-I03_1
20. S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks", IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39,No:6. https://doi.org/10.1109/TPAMI.2012.205
21. D. C. Einloft, M.C. Ghilardi, and I. H. Manssour, "Automatic Detection of Tactile Paving Surfaces in Indoor Environments", Workshop of Undergraduate Works (WUW) in the 29th Conference on Graphics, Patterns and Photographs, Sau Paul, Brazil, 4-7 Oct. 2016. https://doi.org/10.1109/SIBGRAPI.2016.001
22. M. C. Ghilardi, R.C.O Macedo, and I. H. Manssour, "A New Approach for Automatic Detection of Tactile Paving Surfaces in Sidewalks", Procedia Computer Science, 2016, 80, 662-672. https://doi.org/10.1016/j.procs.2016.05.356
23. X. Jie, W. Xiaochi, and F. Zhigang, "Research and implementation of blind sidewalk detection in portable eta system", International Forum on Information Technology and Applications, Kunming, China, 431-434, 16-18 July 2010. https://doi.org/10.1109/IFITA.2010.360
24. T. Asami, and K. Ohnishi, "Crosswalk location, direction and pedestrian signal state extraction system for assisting the expedition of person with impaired vision", 10th France-Japan/8th Europe-Asia Congress on Mecatronics, Tokyo, Japan, 285-290, 27- 29 Nov. 2014. https://doi.org/10.1109/MECATRONICS33801.2014
25. A. Kassim, T. Yasuno, M. S. Mohd, A. Hjshukor, H.I. Jaafar, F. Baharom, and F. Jafar, "Vision based of tactile paving detection method in navigation system for blind person", Jurnal Teknologi, 2015, 77, 20. https://doi.org/10.5772/intechopen.79886
26. O. Yildiz, "Melanoma detection from dermoscopy photographs with deep learning methods: A comprehensive study", Journal of the Faculty of Engineering and Architecture of Gazi University, 2019, 34(4), 2241-2260. https://doi.org/10.17341/gazimmfd.435217
27. E. Dandil, and R. Polattimur, "Dog Behavior Recognition and Tracking based on Faster R-CNN", Journal of the Faculty of Engineering and Architecture of Gazi University, 2019, 35(2), 819-834. https://doi.org/10.17341/gazimmfd.541677
28. J.J. Lv, X.H. Shao, J.S. Huang, X.D. Zhou, and X. Zhou, "Data augmentation for face recognition". Neurocomputing, 2017, 230, 184-196. https://doi.org/10.1016/j.neucom.2016.12.025
29. S. Longfei, Y. Zheng, Z. Zimu, Z. Xiaolong, W. Chenshu, and L. Yunhao, "Crossnavi: Enabling real-time crossroad navigation for the blind with commodity phones", In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 14, pages 787–798, New York, USA, 1-4 September 2014. https://doi.org/10.1145/2632048.2632083
30. J. Wang, N. Wang, L. Li, and Z. Ren, "Real-time behavior detection and judgment of egg breeders based on YOLOv3", Neural Computing and Applications, 2019, 32, 2. https://doi.org/10.1007/s00521-019-04645-4
31. W. Byung, Y. Sung, and J. Kang, "Brick path detection from shape pattern and texture feature", In System Integration (SII), 2011 IEEE/SICE International Symposium on, Kyoto, Japan, 78–83, 1 Dec 2011. https://doi.org/10.1109/SII.2011.6147423
32. S. Shoval, J. Borenstein, and Y. Koren, "The NavBelt-a computerized travel aid for the blind based on mobile robotics technology", IEEE Transactions on Biomedical Engineering, 1998, 45(11), 1376-1386. https://doi.org/10.1109/10.725334
33. J. Redmon, and A. Farhadi, "YOLO9000: Better, Faster, Stronger", 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii-USA, 6517- 6525, 21-26 July 2017. https://doi.org/10.1109/CVPR35066.2017
34. C. Goutte, and E. Gaussier, "A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation", Proceedings of the 27th European conference on Advances in Information Retrieval Research, Spain, 345-359, 21-23 March 2005. https://doi.org/10.5555/2149960
35. R. Redmon, and A. Farhadi, "Yolov3: An incremental improvement", Computer Vision and Pattern Recognition, 2018, 1-6. https://doi.org/10.1007/s11263-009-0275-4
36. R. Pyun, Y. Kim, P. Wespe, and R. Gassert, "Advanced augmented white cane with obstacle height and distance feedback", IEEE Int. Conf. Rehabil. Robotics,Seattle, WA, USA, pp.1-6. 24-26 June 2013. https://doi.org/10.1109/ICORR21102.2013
37. A. M. Kassim, M. H. Jamaluddin, M. R. Yaacob, N. S. N. Anwar, Z. M. Sani, and A. Noordin, "Design and development of MY 2nd EYE for person with impaired vision", IEEE Symposium on Computers & Informatics, Kuala Lumpur, Malaysia, pp.700-703. 20-23 March 2011. https://doi.org/10.1109/ISCI17961.2011
38. D. Ahmetovic, C. Bernareggi, A. Gerino, and S. Mascetti, "Zebrarecognizer: Efficient and precise localization of pedestrian crossings", In Pattern Recognition (ICPR), International Conference on Pattern Recognition, Stockholm, Sweden, 2566–2571, 24-28 Aug 2014. https://doi.org/10.1109/ICPR33632.2014
39. A. M. Kassim, H .İ. Jaafar, M. A. Azam, N. Abas, and T. Yasuno, "Design and development of navigation system by using rfid technology", In System Engineering and Technology (ICSET), IEEE International Conference on System Engineering and Technology, Shah Alam Malaysia, pages 258–262, 19-20 Aug. 2013. https://doi.org/10.1109/ICSEngT.2013.6650181
40. K. W. M. Jiangyan, and X. Ping, "A comparative study of tactile paving design standards in different countries", International Conference on Computer-Aided Industrial Design and Conceptual Design, Kunming, China, pages 753–758, 22-25 Nov. 2008. https://doi.org/10.1109/CAID/CD15004.2008
41. V. N. Murali, and J. M. Coughlan, "Smartphone-based crosswalk detection and localization for visually impaired pedestrians", In Multimedia and Expo Workshops (ICMEW), 2013 IEEE International Conference on, San Jose, California, USA, pages 1–7, 15-19 July 2013. https://doi.org 10.1109/ICMEW32311.2013
42. H. Shuihua, P. Hangrong, Z. Chenyang, and T. Yingli, "Rgb-d image-based detection of stairs, pedestrian crosswalks and traffic signs", Journal of Visual Communication and Image Representation, 2014, 25(2): 263-272. https://doi.org/10.1016/j.jvcir.2013.11.005
43. W. Shuihua, and T. Yingli, "Detecting stairs and pedestrian crosswalks for the blind by rgbd camera", In Bioinformatics and Biomedicine Workshops (BIBMW), 2012 IEEE International Conference on, Philidelphia, USA, 732–739, 4-7 Oct. 2012. https://doi.org/10.1109/BIBMW.2012.6470197
44. C. Shorten, and T. M. Khoshgoftaar, "A survey on Image Data Augmentation for Deep Learning", Journal of Big Data, 2019, 6(1), No: 60. https://doi.org/10.1186/s40537-019-0197-0
45. C. Y. Wang, A. Bochkovskiy, and H. Y. M. Liao, "YOLOv7: Trainable bag-of-freebies sets new stateof-the-art for real-time object detectors", Computer Vision and Pattern Recognition, 2022, 1-15. https://doi.org/10.48550/arXiv.2207.02696
46. B. Hu, M. Zhu, L. Chen, L. Huang, P. Chen, and M. He, "Tree species identification method based on improved YOLOv7", IEEE 8th International Conference on Cloud Computing and Intelligent Systems, 622-627, Chengdu, China, 26-28 November 2022