Cross-Assist: Road Assistance Application for Visually Impaired People

Abstract

According to WHO (World Health Organization) 2.2 billion people in the world have visual impairment. About 40 million of them experience complete vision loss. This number is substantial for the world population. Lack of visual function is one factor that makes it difficult for the individual to participate in social life. Because a barrier-free life is aimed, studies have emerged due to the difficulties encountered. One of these difficulties is that they need help seeing pedestrian lights and roads to cross the street. In this study, a mobile application is designed to address this issue. The application provides visually impaired individuals with voice alerts about the status of crosswalks and traffic lights. This mobile application was developed using Flutter. The convolutional neural network model and YOLO (You Only Look Once) v2Tiny algorithm were used for real-time object recognition from the images taken from the mobile phone camera. Mobile application successfully recognizes red light, green light, and crosswalk with 89.52%, 89.1%, and 88.57% accuracies, respectively. The novelty of this study lies in incorporating both pedestrian traffic light detection and crosswalk identification within a mobile application.

Keywords

• Assistive Technology
• Crosswalk Detection
• Pedestrian Traffic Light Detection
• Visual Impairment
• YOLO Object Detector

Cross-Assist: Görme Engelli Kişiler için Yol Yardım Uygulaması

Abstract

DSÖ’ye (Dünya Sağlık Örgütü) göre dünyada 2.2 milyar kişinin görme engeli bulunmaktadır. Bu kişilerden yaklaşık 40 milyonu tamamen görme kaybı yaşamaktadır. Bu sayı dünya nüfusu için önemli bir rakamdır. Görme fonksiyonunun eksikliği, bireyin sosyal yaşama katılımını zorlaştıran bir faktördür. Engelsiz bir yaşam hedeflendiği için karşılaşılan zorluklar nedeniyle birçok çalışma ortaya çıkmıştır. Bu zorluklardan biri, görme engelli bireylerin yolda karşıya geçerken yaya ışıklarını ve yolları görmelerine yardımcı olmaya ihtiyaç duymalarıdır. Bu çalışmada bu soruna çözüm bulmak amacıyla tasarlanmış bir mobil uygulama geliştirilmiştir. Uygulama, görme engelli bireylere yaya yollarının ve trafik ışıklarının durumu hakkında sesli uyarılar sağlamaktadır. Bu mobil uygulama, Flutter kullanılarak geliştirilmiştir. Mobil telefon kamerasından alınan görüntüler üzerinden gerçek zamanlı nesne tanıma için konvolüsyonel sinir ağı modeli ve YOLO (You Only Look Once) v2 Tiny algoritması kullanılmıştır. Mobil uygulama, kırmızı ışık, yeşil ışık ve yaya geçidi tanımayı sırasıyla %89.52, %89.1 ve %88.57 doğruluk oranlarıyla başarıyla gerçekleştirmektedir. Bu çalışmanın yeniliği, bir mobil uygulama içinde hem yaya trafik ışığı tespiti hem de yaya geçidi tanımlamasını içermesidir.

Keywords

• Destekleyici Teknoloji
• Görme Engelli
• Yaya Geçidi Algılama
• Yaya Trafik Işıkları Algılama
• YOLO Nesne Algılayıcı

References

1. Arora, A., Grover, A., Chugh, R., & Reka, S. S. (2019). Real-time multi-object detection for the blind using single shot multibox detector. Wireless Personal Communications, 107(1), 651-661. doi.org/10.1007/s11277-019-06604-5
2. Ash, R., Ofri, D., Brokman, J., Friedman, I., & Moshe, Y. (2018). Real-time pedestrian traffic light detection. 2018 IEEE International Conference on the Science of Electrical Engineering in Israel (ICSEE). December 12-14, Eilat, Israel, 1-5. doi.org/10.1109/ICSEE.2018.8553696
3. Aydın, S., Samet, R., & Bay, Ö. F. (2020). A survey on parallel image processing studies using CUDA platform in GPU programming. Journal of Polytechnic, 23(3), 737-754. doi.org/10.2339/politeknik.576835
4. Chen, C., Min, H., Peng, Y., Yang, Y., & Wang, Z. (2022). An intelligent real-time object detection system on drones. Applied Sciences, 12(20), 10227.
5. Cheng, C.-C., & Tsai, C.-C. (2024). A visually assistive guidance system for visually impaired pedestrians passing crosswalks. 2024 International Conference of Control Systems, and Robotics (CDSR 2024). June 10-12, Toronto, Canada, Paper No. 112. doi.org/10.11159/cdsr24.112
6. Cheng, R., Wang, K., Yang, K., Long, N., & Hu, W. (2017). Crosswalk navigation for people with visual impairments on a wearable device. Journal of Electronic Imaging, 26(5), 053025. doi.org/10.1117/1.JEI.26.5.053025
7. Cheng, R., Wang, K., Yang, K., Long, N., & Liu, D. (2018). Real-time pedestrian crossing lights detection algorithm for the visually impaired. Multimedia Tools and Applications, 77(16), 20651-20671. doi.org/10.1007/s11042-018-6181-8
8. Dionisi, A., Sardini, E., & Serpelloni, M. (2012). Wearable object detection system for the blind. 2012 IEEE International Instrumentation and Measurement Technology Conference Proceedings. May 13-16, Graz, Austria, 1255-1258.

9. Francies, M. L., Mohamed, M. A., & Mohamed, A. M. (2022). A robust multiclass 3D object recognition based on modern YOLO deep learning algorithms. Concurrency and Computation: Practice and Experience, e6517. doi.org/10.1002/cpe.6517
10. Ghilardi, M. C., Simoes, G., Wehrmann, J., Manssour, I. H., & Barros, R. C. (2018). Real-time detection of pedestrian traffic lights for visually-impaired people. 2018 International Joint Conference on Neural Networks (IJCNN). Jul 08-13, Rio de Janeiro, Brazil, 1-8. doi.org/10.1109/IJCNN.2018.8489628
11. Huang, R., Pedoeem, J., & Chen, C. (2018). Yolo-Lite: A real-time object detection algorithm optimized for non-GPU computers. 2018 IEEE International Conference on Big Data. December 10-13, Seattle, WA, USA, 2503-2510. doi.org/10.1109/BigData.2018.8622624
12. Hwang, H., Kwon, S., Kim, Y., & Kim, D. (2024) Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing. arXiv. 2402.06794v2. doi.org/10.48550/arXiv.2402.06794
13. Kaggle (2022). Traffic Light Detection Dataset. Retrieved Jun 07, 2024, from https://www.kaggle.com/datasets/wjybuqi/traffic-light-detection-dataset.
14. Kaggle (2020). Crosswalk-Dataset. Retrieved Jun 07, 2024, from https://www.kaggle.com/datasets/davidsilvam/crosswalkdataset.
15. Khan, S., Rahmani, H., Shah, S. A. A., & Bennamoun, M. (2018). A guide to convolutional neural networks for computer vision. Synthesis Lectures on Computer Vision, 8(1), 1-207. doi.org/10.2200/S00839ED1V01Y201801COV014
16. Kuriakose, B., Shrestha, R., & Sandnes, F. E. (2023). DeepNAVI: A deep learning based smartphone navigation assistant for people with visual impairments, Expert Systems with Applications, 212. doi.org/10.1016/j.eswa.2022.118720.
17. Kuzmin, N., Ignatiev, K., & Grafov, D. (2020). Experience of developing a mobile application using Flutter. In Information Science and Applications. (pp. 571-575). doi.org/10.1007/978-981-15-6204-3_58
18. Li, X., Cui, H., Rizzo, J.-R., Wong, E., & Fang, Y. (2020). Cross-safe: A computer vision-based approach to make all intersection-related pedestrian signals accessible for the visually impaired. In Advances in Intelligent Systems and Computing. (pp. 132–146). doi.org/10.1007/978-3-030-40549-3_15
19. Li, Y., Li, J., & Meng, P. (2023). Attention-YOLOV4: A real-time and high-accurate traffic sign detection algorithm. Multimedia Tools and Applications, 82, 7567-7582. doi.org/10.1007/s11042-022-12150-2
20. Mahesh, T. Y., Parvathy, S. S., Thomas, S., Thomas, S. R., & Sebastian, T. (2021). Cicerone-A Real Time Object Detection for Visually Impaired People. IOP Conference Series: Materials Science and Engineering, 1085(1), 012006. doi.org/10.1088/1757-899X/1085/1/012006
21. Moura, R. S., Sanches, S. R. R., Bugatti, P. H., & Saito, P. T. (2022). Pedestrian traffic lights and crosswalk identification. Multimedia Tools and Applications, 81, 16497-16513. doi.org/10.1007/s11042-021-11817-8
22. Rajwani, R., Purswani, D., Kalinani, P., Ramchandani, D., & Dokare, I. (2018). Proposed system on object detection for visually impaired people. International Journal of Information Technology, 4(1), 1-6.
23. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. IEEE Conference On Computer Vision and Pattern Recognition, Jun 26-July 1, Las Vegas, USA, 779-788.
24. Shangguan, L., Yang, Z., Zhou, Z., Zheng, X., Wu, C., & Liu, Y. (2014). Crossnavi: Enabling real-time crossroad navigation for the blind with commodity phones. 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Sep 13-17, Seattle, WA, USA, 787-798.
25. Sinha, R. K., Pandey, R., & Pattnaik, R. (2018). Deep learning for computer vision tasks: A review. arXiv preprint arXiv:1804.03928.
26. Son, H., & Weiland, J. (2022). Wearable system to guide crosswalk navigation for people with visual impairment. Frontiers in Electronics, 2, 790081. doi.org/10.3389/felct.2021.790081
27. Srinivas, S., Sarvadevabhatla, R. K., Mopuri, K. R., Prabhu, N., Kruthiventi, S. S. S., & Babu, R. V. (2016). A taxonomy of deep convolutional neural nets for computer vision. Frontiers in Robotics and AI, 2, 36. doi.org/10.3389/frobt.2015.00036
28. Tosun, S., & Karaarslan, E. (2018). Real-time object detection application for visually impaired people: Third eye. 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), Sep 28-30, Malatya, Turkey, 1-6.
29. Wang, Y., Mao, K., Chen, T., Yin, Y., Chen, G., & He, S. (2021). Accelerating real-time object detection in high-resolution video surveillance. Concurrency and Computation: Practice and Experience, e6307. doi.org/10.1002/cpe.6307
30. Wang, J., Wang, S., & Zhang, Y. (2023) Artificial intelligence for visually impaired. Displays, 77, 102391, doi.org/10.1016/j.displa.2023.102391
31. Yu, S., Lee, H., & Kim, J. (2019). Lytnet: A convolutional neural network for real-time pedestrian traffic lights and zebra crossing recognition for the visually impaired. International Conference on Computer Analysis of Images and Patterns, Sep 28-30, Salerno, Italy, 259-270.