CNN-Based Approaches for Automatic Recognition of Turkish Sign Language

Abstract

Sign language is a nonverbal communication tool used by deaf and dumb individuals to convey their feelings, thoughts and social identities to their environment. Sign language has a key role in communication between deaf and dumb individuals and the rest of the society. Many sign language recognition systems have been developed with the increase in human-computer interaction and the fact that sign language is not widely known among normal people. In this study, a new number-based data set for Turkish sign language is proposed for the first time in the literature. The most up-to-date deep learning approaches have been applied to the proposed data set in order to classify Turkish sign language autonomously and to enable computer-based communication of people who have difficulties in this regard. The most up-to-date and popular architectures such as CNN-based VGG, ResNet, MobileNet, DenseNet and EfficientNet were used in the study. In experimental studies, it has been observed that the ResNet152 model performs better than other models with 98.76% accuracy, 98.85% precision, 98.81% sensitivity and 98.80% F1-score. Additionally, the other models used in experimental studies all show a success rate above 90%, supporting the effectiveness of the proposed data set. This shows that CNN models can successfully detect Turkish sign language.

Keywords

• Turkish Sign Language
• CNN
• ResNet
• Classification

Türk İşaret Dilinin Sınıflandırılması için Derin Öğrenme Yaklaşımları

Abstract

İşaret dili, sağır ve dilsiz bireylerin duygularını, düşüncelerini ve sosyal kimliklerini çevrelerine aktarabilmek için kullandıkları sözsüz bir iletişim aracıdır. İşaret dili, sağır ve dilsiz bireyler ile toplumun geri kalan bireyleri arasındaki iletişimde kilit bir role sahiptir. Normal insanlar arasında işaret dilinin çok yaygın bilinmemesi ve insan-bilgisayar etkileşiminin artmasıyla birlikte birçok işaret dili tanıma sistemleri geliştirilmiştir. Bu çalışmada, Türk işaret dili için literatürde ilk kez rakam temelli yeni bir veri seti önerilmiştir. Türk işaret dilinin otonom bir şekilde sınıflandırılması ve bu konuda sıkıntı yaşayan insanların iletişimini bilgisayar temelli yapabilmesi için en güncel derin öğrenme yaklaşımları önerilen veri setine uygulanmıştır. Çalışmada özellikle CNN tabanlı VGG, ResNet, MobileNet, DenseNet ve EfficientNet gibi en güncel ve popüler mimariler kullanılmıştır. Deneysel çalışmalarda ResNet152 modeli, %98.76 doğruluk, %98.85 kesinlik, %98.81 duyarlılık ve %98.80 F1-skoru ile diğer modellere göre daha iyi performans gösterdiği gözlemlenmiştir. Ayrıca, deneysel çalışmalarda kullanılan diğer modellerin hepsi %90'ın üzerinde bir başarım oranı göstererek önerilen veri setinin etkililiğini desteklemektedir. Bu, CNN modellerinin Türk işaret dilini tanımayı başarılı bir şekilde tespit yapabildiğini göstermektedir.

Keywords

• Türk İşaret Dili
• CNN
• ResNet
• Sınıflandırma

References

1. Aiouez, S., Hamitouche, A., Belmadoui, M. S., (Belattar, K., & Souami, F. (2022). Real-time Arabic Sign Language Recognition based on YOLOv5. PROCEEDINGS OF THE 2ND INTERNATIONAL CONFERENCE ON IMAGE PROCESSING AND VISION ENGINEERING, (s. 17-25). doi:10.5220/0010979300003209
2. Alawwad, R. A., Bchir, O., & Ismail, M. M. (2021). Arabic Sign Language Recognition using Faster. International Journal of Advanced Computer Science and Applications, 12(3), 692-700.
3. Al-Hammadi, M., Muhammad, G., Abdul, W., Alsulaiman, M., Bencherif, M. A., & Mekhtiche, M. A. (2020). Hand Gesture Recognition for Sign Language Using 3DCNN. IEEE Access, 8, 79491 - 79509.
4. Alici-Karaca, D., Akay, B., Yay, A., Suna, P., Nalbantoglu, O. U., Karaboga, D., . . . Baran, M. (2022). A new lightweight convolutional neural network for radiation-induced liver disease classification. Biomedical Signal Processing and Control, 73. doi:10.1016/j.bspc.2021.103463
5. Almeida, S. G., Guimarães, F. G., & Ramírez, J. A. (2014). Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors. Expert Systems with Applications: An International Journal, 14(6), 7259–7271.
6. Alzubaidi, L., Zhang, J., Humaidi, A. J., Ayad Al-Dujaili, Y. D., Al-Shamma, O., Santamaría, J., . . . Farhan, L. (2021). Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. Journal of big Data, 8(1), 1-74.
7. Bhushan, S., Alshehri, M., Keshta, I., Chakraverti, A. K., Rajpurohit, J., & Abugabah, A. (2022). An Experimental Analysis of Various Machine Learning Algorithms for Hand Gesture Recognition. Electronics, 11(6). doi:10.3390/electronics11060968
8. Bordes, A., Glorot, X., Weston, J., & Bengio, Y. (2012). Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing. Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (s. 127-135). PMLR.

9. Burukanlı, M., Budak, Ü., & Çıbuk, M. (2019). Saldırı Tespit Sistemlerinde Makine Öğrenme Metotlarının Kullanımı. Uluslararası Bilim ve Mühendislik Sempozyumu, 20(22), 1052-1057. Chaudhuri, S., Dayal, U., & Narasayya, V. (2011). An overview of business intelligence technology. Communications of the ACM, 54(8), 88-98.
10. Deafness and hearing loss. (2021, Nisan 1). Word Health Orgnanization(WHO): https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss adresinden alındı
11. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. doi:10.1109/CVPR.2009.5206848
12. Fan, J., Ma, C., & Zhong, Y. (2019). A selective overview of deep learning. arXiv:1904.05526 . adresinden alındı
13. Gangrade, J. B. (2020). Vision-based hand gesture recognition for Indian sign language using convolution neural network. IETE Journal of Research, 1-10.
14. Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & S.Lew, M. (2016). Deep learning for visual understanding: A review. Neurocomputing, 187, 27-48.
15. Gschwend, D. (2020). Zynqnet: An fpga-accelerated embedded convolutional neural network. arXiv preprint arXiv:2005.06892.
16. Halbouni, A., Gunawan, T. S., Habaebi, M. H., Halbouni, M., Kartiwi, M., & Ahmad, R. (2022). Machine Learning and Deep Learning Approaches for CyberSecurity: A Review. IEEE Access (10), 19572 - 19585. doi:10.1109/ACCESS.2022.3151248
17. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (s. 770-778).
18. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., . . . Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. https://arxiv.org/abs/1704.04861 adresinden alındı
19. Huang, G., Liu, Z., Maaten, L. v., & Weinberger, K. Q. (2017). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (s. 4700-4708).
20. Justesen, N., Bontrager, P., Togelius, J., & Risi, S. (2020). Deep Learning for Video Game Playing. IEEE Transactions on Games, 12(1), 1 - 20.
21. Karaman, A., Karaboga, D., Pacal, I., Akay, B., Basturk, A., Nalbantoglu, U., Sahin, O. (2022). Hyper-parameter optimization of deep learning architectures using artificial bee colony (ABC) algorithm for high performance real-time automatic colorectal cancer (CRC) polyp detection. Applied Intelligence. https://doi.org/10.1007/s10489-022-04299-1
22. Karaman, A., Pacal, I., Basturk, A., Akay, B., Nalbantoglu, U., Coskun, S., Sahin, O., & Karaboga, D. (2023). Robust real-time polyp detection system design based on YOLO algorithms by optimizing activation functions and hyper-parameters with artificial bee colony (ABC). Expert Systems with Applications, 221. https://doi.org/10.1016/j.eswa.2023.119741
23. Karagoz, M. A., Akay, B., Basturk, A., Karaboga, D., & Nalbantoglu, O. U. (2023). An unsupervised transfer learning model based on convolutional auto encoder for non-alcoholic steatohepatitis activity scoring and fibrosis staging of liver histopathological images. Neural Computing and Applications, 1-15.
24. Khari, M., Garg, A., Crespo, R. G., & Verdú, E. (2019). Gesture Recognition of RGB and RGB-D static Images using Convolutional Neural Networks. International Journal of Interactive Multimedia and Artificial Intelligence, 5(7), 22-27.
25. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25.
26. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
27. Li, Y., Ding, L., & Gao, X. (2018). On the Decision Boundary of Deep Neural Networks. https://arxiv.org/abs/1808.05385 adresinden alındı
28. Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2021). A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Transactions on Neural Networks and Learning Systems, 33(12), 6999 - 7019.
29. LUQMAN, H., & ELALFY, E. (2022). Utilizing motion and spatial features for sign language gesture recognition using cascaded CNN and LSTM models. Turkish Journal of Electrical Engineering and Computer Sciences, 30(7), 2508-2525.
30. Ma, Y., Xu, T., & Kim, K. (2022). Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition. Sensors, 22(16), 5959.
31. Marais, M., Brown, D., Connan, J., & Boby, A. (2022). An Evaluation of Hand-Based Algorithms for Sign Language Recognition. 2022 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD). IEEE. doi:10.1109/icABCD54961.2022.9856310
32. Myagila, K., & Kilavo, H. (2021). A Comparative Study on Performance of SVM and CNN in Tanzania Sign Language Translation Using Image Recognition. Applied Artificial Intelligence, 1-16. doi:10.1080/08839514.2021.2005297
33. Naglot, D., & Kulkarni, M. (2016). Real time sign language recognition using the leap motion controller. International conference on inventive computation technologies (ICICT). 3, s. 1-5. IEEE.
34. Nam, Y., & Lee, C. (2021). Cascaded convolutional neural network architecture for speech emotion recognition in noisy conditions. Sensors, 21(13), 4399.
35. Núñez-Prieto, R., Gómez, P. C., & Liu, L. (2019, October). A real-time gesture recognition system with fpga accelerated zynqnet classification. In 2019 IEEE Nordic Circuits and Systems Conference (NORCAS): NORCHIP and International Symposium of System-on-Chip (SoC) (pp. 1-6). IEEE.
36. Ongsulee, P. (2017). Artificial intelligence, machine learning and deep learning. 2017 15th International Conference on ICT and Knowledge Engineering (ICT&KE) (s. 1-6). IEEE.
37. Ozkok, F. O., & Celik, M. (2023). Classification of High Resolution Melting Curves Using Recurrence Quantification Analysis and Data Mining Algorithms. In Smart Applications with Advanced Machine Learning and Human-Centred Problem Design (pp. 641-650). Cham: Springer International Publishing.
38. Özcan, T., & Baştürk, A. (2020). ERUSLR: A new Turkish sign language dataset and its recognition using hyperparameter. ournal of the Faculty of Engineering and Architecture of Gazi University, 36(1), 527-542.
39. PACAL, İ. (2022). Deep Learning Approaches for Classification of Breast Cancer in Ultrasound (US) Images. Journal of the Institute of Science and Technology, 1917–1927. https://doi.org/10.21597/jist.1183679
40. Pacal, I., & Karaboga, D. (2021). A robust real-time deep learning based automatic polyp detection system. Computers in Biology and Medicine, 134. https://doi.org/10.1016/J.COMPBIOMED.2021.104519
41. Pacal, I., Karaboga, D., Basturk, A., Akay, B., & Nalbantoglu, U. (2020). A comprehensive review of deep learning in colon cancer. Computers in Biology and Medicine, 126. https://doi.org/10.1016/J.COMPBIOMED.2020.104003
42. Pacal, I., Karaman, A., Karaboga, D., Akay, B., Basturk, A., Nalbantoglu, U., & Coskun, S. (2022). An efficient real-time colonic polyp detection with YOLO algorithms trained by using negative samples and large datasets. Computers in Biology and Medicine, 141. https://doi.org/10.1016/J.COMPBIOMED.2021.105031
43. Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 1345-1359.
44. Rao, G. A., Syamala, K., Kishore, P. V., & Sastry, A. S. (2018). Deep convolutional neural networks for sign language recognition. 2018 Conference on Signal Processing And Communication Engineering Systems (SPACES). doi:10.1109/SPACES.2018.8316344
45. Rastgoo, R., Kiania, K., & Escalerab, S. (2021). Sign Language Recognition: A Deep Survey. Expert Systems with Applications, 164, 113794. doi:10.1016/j.eswa.2020.113794
46. Ren, Z., Yuan, J., Meng, J., & Zhang, Z. (2013). Robust Part-Based Hand Gesture Recognition Using Kinect. IEEE Transactions on Multimedia, 15(5), 1110 – 1120.
47. Rezende, T. M., Almeida, S. G. M., & Guimarães, F. G. (2021). Development and validation of a Brazilian sign language database for human gesture recognition. Neural Computing and Applications, 33(16), 10449-10467.
48. Sajjanhar, A., Wu, Z., & Wen, Q. (2018). Deep learning models for facial expression recognition. 2018 digital image computing: Techniques and applications (dicta) (s. 1-6). IEEE.
49. Saqib, S., Ditta, A., Khan, M., Kazmi, S. A., & Alquhayz, H. (2021). Intelligent Dynamic Gesture Recognition Using CNN Empowered by Edit Distance. Computers, Materials and Continua, 66(2), 2061-2076.
50. Shukor, A. Z., Miskon, M. F., Jamaluddin, M. H., binAli@Ibrahim, F., FareedAsyraf, M., & binBahar, M. B. (2015). A new data glove approach for Malaysian sign language detection. Procedia Computer Science, 76, 60-67.
51. Simonyan, K., & Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. https://arxiv.org/abs/1409.1556 adresinden alındı
52. Suliman, W., Deriche, M., Luqman, H., & Mohandes, M. (2021). Arabic Sign Language Recognition Using Deep Machine Learning. (s. 4th International Symposium on Advanced Electrical and Communication Technologies (ISAECT)). IEEE. doi:10.1109/ISAECT53699.2021.9668405
53. Suri, K., & Gupta, R. (2019). Convolutional neural network array for sign language recognition using wearable IMUs. In 2019 6th International Conference on Signal Processing and Integrated Networks (SPIN) (pp. 483-488). IEEE.
54. Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International conference on machine learning (s. 6105-6114). PMLR.
55. Tasmere, D., & Ahmed, B. (2020). Hand Gesture Recognition for Bangla Sign Language Using Deep Convolution Neural Network. 2020 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI). IEEE. doi:10.1109/STI50764.2020.9350484
56. Wang, Z., Zhao, T., Ma, J., Chen, H., Liu, K., Shao, H., . . . Ren, J. (2022). Hear Sign Language: A Real-Time End-to-End Sign Language Recognition System. IEEE Transactions on Mobile Computing, 21(7), 2398 - 2410.
57. Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016). A survey of transfer learning. Journal of Big Data, 3(1), 1-40.
58. Wu, Y., & Huang, T. S. (1999). Vision-Based Gesture Recognition: A Review. In International gesture workshop (s. 103-115). Berlin Heidelberg: Springer.
59. Yu, S., Jia, S., & Xu, C. (2017). Convolutional neural networks for hyperspectral image classification. Neurocomputing, 219, 88-98.
60. Zhiqiang, W., & Jun, L. (2017). A review of object detection based on convolutional neural network. 2017 36th Chinese Control Conference (CCC) (s. 11104-11109). IEEE.