Real-Time Detection of Turkish Sign Language Letters and Numbers with Deep Learning

Abstract

The visual language that hearing or speech-impaired individuals communicate with through facial expressions and hand movements is called sign language. The rate of reading and writing sign language is very low. For this reason, hearing or speech-impaired individuals have great difficulty in communicating with other people, especially when benefiting from services such as hospitals and education. In this study, real-time sign language detection and display on the computer screen were performed with deep learning. The movements of hearing or speech-impaired individuals shown with their hands and fingers are detected in front of the camera. As a result of detection, the letter corresponding to the movement is recognized and displayed on the computer screen. YOLOv8 architecture was used in this method. First, a data set was created for the study. The data set consists of 29 letters and 10 numbers. Photographs of sign language movements from 100 different people were taken in the data set. Different changes were made to the photographs in the data set. With these additions, the error that may occur due to any distortion that may occur from the camera was minimized. With the changes made to the photographs, the number of photographs forming the data set increased to 11079. As a result of the study, average stability was 90.7%, mAP was 85.8%, and recall was 81.4%.

Keywords

• sign language
• sign language recognition
• deep learning
• YOLOv8
• Letter and Number Detection

References

1. T. Tao, Y. Zhao, J. Zhu, T. Liu, J. Kuang, “A survey on sign language recognition from perspectives of traditional and deep-learning methods”, Journal of Visual Communication and Image Representation, 6, pp. 104363, 2024, https://doi.org/10.1016/j.jvcir.2024.104363.
2. F. Nehal, T. Mohamed, F. S. Ahmed, A. M. Mahmoud “Efficient deep learning models based on tension techniques for sign language recognition”, Intelligent Systems with Applications, 20, pp. 200284, 2023. https://doi.org/10.1016/j.iswa.2023.200284.
3. J. Bora, S. Dehingia, A. Boruah, A. Chetia, D. Gogoi, “Real-time Assamese Sign Language Recognition using Media Pipe and Deep Learning”, Procedia Computer Science, 218, pp. 1384-1393, 2023. https://doi.org/10.1016/j.procs.2023.01.117.
4. Y. Zhang, X. Jiang, “Recent Advances on Deep Learning for Sign Language Recognition”, Computer Modeling in Engineering and Sciences, pp. 2399-2450, 2024. https://doi.org/10.32604/cmes.2023.045731.
5. Z. Katılmış, C. Karakuzu, “Double handed dynamic Turkish Sign Language recognition using Leap Motion with meta learning approach”, Expert Systems with Applications, 228, pp. 120453, 2023. https://doi.org/10.1016/j.eswa.2023.120453.
6. Y. Liu, X. Jiang, X. Yu, H. Ye, C. Ma, W. Wang, Y. Hu,“A wearable system for sign language recognition enabled by a convolutional neural network,” Nano Energy Volume 116, 108767, 2023. https://doi.org/10.1016/j.nanoen.2023.108767.
7. J. Hasanov, N. Alishzade, A. Nazimza, S. Dadashzad, T. Tahirov, “Development of a hybrid word recognition system and dataset for the Azerbaijani Sign Language dactyl alphabet,” Speech Communication, Volume 153, 102960, 2023, https://doi.org/10.1016/j.specom.2023.102960.
8. R. Sreemathy, MP. Turuk, S. Chaudhary, K. Lavate, A. Ushire, S. Khurana, “Continuous word level sign language recognition using an expert system based on machine learning,” International Journal of Cognitive Computing in Engineering, Volume 4, pp. 170-178, 2023. https://doi.org/10.1016/j.ijcce.2023.04.002.

9. Y. Qin, S. Pan, W. Zhou, D. Pan, Z. Li, “WiASL: American Sign Language writing recognition system using commercial WiFi devices,” Measurement, Volume 218, 113125, 2023. https://doi.org/10.1016/j.measurement.2023.113125.
10. Y. Obi, K.S. Claudio, V.M. Budiman, S. Achmad, A. Kurniawan, “Sign language recognition system for communicating to people with disabilities,” Procedia Computer Science, Volume 216, pp. 13-20, 2023. https://doi.org/10.1016/j.procs.2022.12.106.
11. L. R. Cerna, E. E. Cardenas, D. G. Miranda, D. Menotti, G. Camara-Chavez, “A multimodal LIBRAS-UFOP Brazilian sign language dataset of minimal pairs using a microsoft Kinect sensor”, Expert Systems with Applications, Volume 167, 1l4179, 2021. https://doi.org/10.1016/j.eswa.2020.114179.
12. S. Arooj, S. Altaf, Ş. Ahmed, H. Mahmud, A. Ş N. Muhammed, “Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language” Journal of King Saud University - Computer and Information Sciences, Volume 36, Issue 2, 101934 2024. https://doi.org/10.1016/j.jksuci.2024.101934.
13. B. Alsharif, E. Alalwany, M. Ilyas, “Transfer learning with YOLOV8 for real-time recognition system of American Sign Language Alphabet” Franklin Open, Volume 8, 100165, 2024. https://doi.org/10.1016/j.fraope.2024.100165.
14. S. Qahtan, H.A. Alsattar, A.A. Zaidan, M. Deveci, D. Pamucar, L. Martinez, “A comparative study of evaluating and benchmarking sign language recognition system-based wearable sensory devices using a single fuzzy set,” Knowledge-Based Systems, Volume 269, 110519, 2023. https://doi.org/10.1016/j.knosys.2023.110519.
15. N. Musthafa, C.G. Raji, “Real time Indian sign language recognition system,” Materials Today: Proceedings, Volume 58, pp.504–508, 2022. https://doi.org/10.1016/j.matpr.2022.03.011.
16. S. Katoch, V. Singh, U. S. Tiwary, “Indian Sign Language recognition system using SURF with SVM and CNN,” Array, Volume 14, 100141, 2022. https://doi.org/10.1016/j.array.2022.100141
17. K. Wangchuk, P. Riyamongkol, R. Waranusast, “Real-time Bhutanese Sign Language digits recognition system using Convolutional Neural Network,” ICT Express, Volume 7, Issue 2, pp. 215-220, June 2021. https://doi.org/10.1016/j.icte.2020.08.002.
18. M.A. Ahmed, B.B. Zaidan, A.A. Zaidan, M. Salih, Z.T. Al-qaysi, A.H. Alamoodi, “Based on wearable sensory device in 3D-printed humanoid: A new real-time sign language recognition system,” Measurement Volume 168, 108431, 2021. https://doi.org/10.1016/j.measurement.2020.108431.
19. R. Rastgoo, K. Kiania, and S. Escalerab, “Sign Language Recognition: A Deep Survey,” Expert Systems with Applications, 164, pp. 113794, 2021. htpss://doi.org/10.1016/j.eswa.2020.113794.
20. A. A. Mujeeb, A. H. Khan, S. Khalid, M. S. Mirza, S. J. Khan, “A neural-network based web application on real-time recognition of Pakistani sign language,” Engineering Applications of Artificial Intelligence, Volume 135, 108761, 2024. https://doi.org/10.1016/j.engappai.2024.108761.
21. A. O. Salau, N. K. Tamiru, B. T. Abeje, “Derived Amharic alphabet sign language recognition using machine learning methods,” Helijon, Volume 10, Issue 19, 15 2024, https://doi.org/10.1016/j.heliyon.2024.e38265.
22. Y. Nam, and C. Lee, “Cascaded convolutional neural network architecture for speech emotion recognition in noisy conditions,” Sensors, 21(13), pp. 4399, 2021. https://doi.org/10.3390/s21134399.
23. T. Tao, Y. Zhao, J. Zhu, T. Liu, J. Kuang, “A survey on sign language recognition from perspectives of traditional and deep-learning methods” Journal of Visual Communication and Image Representation, 104363, 2024. https://doi.org/10.1016/j.jvcir.2024.104363.
24. W. Hao, C. Hou, Z. Zhang, X, Zhai, L. Wang, G. Lv, “A sensing data and deep learning-based sign language recognition approach”, Computers and Electrical Engineering, Volume 118, 109339, Part A, 2024. https://doi.org/10.1016/j.compeleceng.2024.109339.
25. S. N. Koyineni, G. K. Sai, K. Anvesh, T. Anjali, “Silent Expressions Unveiled: Deep Learning for British and American Sign Language Detection” Procedia Computer Science, Volume 233, 2024, Pages 269-278, https://doi.org/10.1016/j.procs.2024.03.216.
26. S. Siddique, S. Islam, E. E. Neon, T. Sabbir, I. T. Naheen, and R. Khan, “Deep Learning-based Bangla Sign Language Detection with an Edge Device,” Intelligent Systems with Applications, 18, pp. 200224. 2023. https://doi.org/10.1016/j.iswa.2023.200224 30 March 2023.
27. Ö. Çelik, and A. Odabaş, “Sign2Text: Turkish sign Language Recognition Using Convolutional Neural Networks” European Journal of Science Technology. 19, pp. 923-934, Aug 2020.
28. M. Toğaçar, Z. Cömet and B. Ergen, “Recognition of the Digits in Turkish Sign Language Using Siamese Neural Networks” Dokuz Eylül University Faculty of Engineering Journal of Science and Engineering. 23(68), pp. 349-356, 2021. https://doi.org/10.35860/iarej.700564.
29. T. Özcan and A. Baştürk, “ERUSLR a new Turkish sign language dataset and its recognition using hyper parameter optimization aided convolution neural network,” Journal of Faculty Engineering Architecture of Gazi Universty, 36:1, pp. 527-542, 2021. htpss://doi.org/ 10.17341/gazimmfd.746793.
30. I. Pacal and M. Alaftekin, “CNN-Based approaches for automatic recognition of Turkish sign language,” Journal of the Instute of Science and Technology, 13(2), pp. 760-777, 2023. https://doi.org/10.55525/tjst.1073116.
31. J. Li, L. Cheng, J. Lei, W. Xiang, “Uyghur Sign Language Recognition Based on Improved YOLOv7,” Procedia Computer Science, Volume 242, pp.512-519, 2024. https://doi.org/10.1016/j.procs.2024.08.095.
32. L. Alzubaidi, J. Zhang, A. J. Humaidi, Y. D. Ayad Al-Dujaili, O. Al-Shamma, J. Santamaría, and L. Farhan, “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of big Data, 8(1), pp. 1-74, 2021. htpss://doir.org/10.1186/s40537-021-00444-8.
33. A. Halbouni, T. S. Gunawan, M. H. Habaebi, M. Halbouni, M. Kartiwi, and R. Ahmad, “Machine Learning and Deep Learning Approaches for CyberSecurity: A Review,” IEEE Access (10), pp. 19572 – 19585, 2022. htpss://doi.org/10.1109/ACCESS.2022.3151248.
34. S. Renjith, M. Rashmi, S. Suresh, “Sign Language Recognition by using Spatio-Temporal Features,” Procedia Computer Science, Volume 233, pp. 353-362, 2024. https://doi.org/10.1016/j.procs.2024.03.225.
35. A. Karaman, I. Pacal, A. Basturk, B. Akay, U. Nalbantoglu, S. Coskun, O. Sahin, and D. Karaboga, “Robust real-time polyp detection system design based on YOLO algorithms by optimizing activation functions and hyper-parameters with artificial bee colony (ABC),” Expert Systems with Applications, 221, pp. 119741, 2023. https://doi.org/10.1016/j.eswa.2023.119741.
36. S. Chaudhuri, U. Dayal, and V. Narasayya, “An overview of business intelligence technology,” Communications of the ACM, 54(8), pp. 88-98, 2021. htpss:/doi.org/ 10.1145/1978542.1978562.
37. D. Talukder, F. Jahara, S. Barua, and M. M. Haque, “OkkhorNama: BdSL image dataset for real time object detection algorithms,” IEEE Region 10 Symposium, pp. 1–6. Jeju, Korea, 23-25 August 2021. https://doi.org/10.1109/TENSYMP52854.2021.9550907.
38. O. B. Hoque, M. I. Jubair, A. F. Akash, and M. S. Islam, “BDSL36: A dataset for Bangladeshi sign letters recognition,” Asian Conference on Computer Vision, pp. 71–86, Kyoto, Japan, 30 November- 4 December 2020. https://doi.org/10.1007/978-3-030-69756-3_6.
39. K. A. Lipi, “Static-gesture word recognition in Bangla sign language using Convolutional Neural Network,” TELKOMNIKA (Telecommunication Computing Electronics and Control), 20, pp.1109–1116, 2022. https://doi.org/10.12928/telkomnika. v20i5.24096.
40. T. M. Angona, “Automated Bangla sign language translation system for alphabets by means of MobileNet,” TELKOMNIKA (Telecommunication Computing Electronics and Control), 18, pp. 1292–1301, 2020. https://doi.org/10.12928/telkomnika. v18i3.15311.
41. B. Shi, A. M. Del Rio, J. Keane, D. Brentari, G. Shakhnarovich, and K. Livescu, “Fingerspelling Recognition in the Wild with Iterative Visual Attention, 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27 October 2019 - 02 November 2019. htpss://doi.org/10.1109/ICCV.2019.00550.
42. A. Wadhawan, P. Kumar, “Deep Learning-Based Sign Language Recognition System for Static Signs,” Neural Comput & Applic, 32(2), pp. 1-12, 2020, https://doi.org/10.1007/s00521- 019-04691-y.
43. A. S. M. Miah, J. Shin, M. A. H. Hasan, and M. A. Rahim, “BenSignNet: Bengali sign language alphabet recognition using concatenated segmentation and convolutional neural network,” Applied Sciences, 12(8), pp. 3933, 2022. https://doi.org/10.3390/ app12083933.
44. M. S. Alam, M. Tanvir, D. K. Saha, and S. K. Das, “Two-Dimensional convolutional neural network approach for real-time bangla sign language characters recognition and translation,” SN Computer Science, 2(5), 2022. https://doi.org/10.1007/s42979-021- 00783-6.
45. T. M. Angona, “Automated Bangla sign language translation system for alphabets by means of MobileNet,” TELKOMNIKA (Telecommunication Computing Electronics and Control), 18, pp. 1292–1301, 2020. https://doi.org/10.12928/telkomnika. v18i3.15311.
46. F. M. Shamrat, “Bangla numerical sign language recognition using convolutional neural network CNNs,” Indonesian Journal of Electrical Engineering and Computer Science, 23, pp. 405–413, 2021. https://doi.org/10.11591/ijeecs.v23.i1.pp405-413.