Development of an Artificial Intelligence Model for Classification of the Movements of Hearing Impaired Individuals

Abstract

Hearing impaired individuals utilize a crucial communication tool called sign language. There are numerous sign languages across different countries such as natural languages. This study proposes leveraging deep learning (DL) advancements to facilitate the conversion of sign language gestures into text. To this end, a novel dataset is curated under various environmental factors such as backgrounds, lighting conditions, and sign positions. Subsequently, Convolutional Neural Networks (CNNs) are employed to detect and classify twenty-three gestures of Turkish sign language alphabet. Furthermore, various hyperparameters are explored to optimize the performance of the developed models. The best model relies on a five-layer convolutional network coupled with the Adam optimization algorithm. This model approximately achieves a commendable accuracy of 98% after 80 epochs. As a result, the proposed models and dataset represent a significant advancement in the field of gestures recognition.

Keywords

• Artificial intelligence
• Gesture
• Recognition
• Hearing impaired

İşitme Engelli Bireylerin Hareketlerini Sınıflandırmaya Yönelik Yapay Zeka Modelinin Geliştirilmesi

Abstract

İşaret Dili, işitme engelli bireyler için hayati bir iletişim aracıdır. Farklı ülkelerde kendi ihtiyaçlarına geliştirilmiş birçok işaret dili vardır. Bu çalışma, Türk İşaret Dili (TİD) jestlerini derin öğrenme teknikleriyle metne dönüştürmeyi amaçlamaktadır. Bu amaçla, arka planlar, aydınlatma koşulları ve işaret pozisyonları gibi çeşitli çevresel faktörler açısından çeşitlilik gösteren yeni bir veri kümesi oluşturulmuştur. Daha sonra, TİD alfabesini algılamak ve sınıflandırmak için Evrişimli Sinir Ağları (CNN'ler) kullanılmıştır. Ayrıca, geliştirilen modellerin performansını optimize etmek için çeşitli hiperparametreler araştırılmıştır. En iyi CNN mimarisi, beş evrişimli katmanı içerir ve Adam öğrenme hızı optimizasyon yöntemini kullanır; 80 epoch'tan sonra yaklaşık %98'lik bir doğruluk (başarı) elde edilmiştir. Sonuç olarak, zorlu bir veri kümesi üzerinde eğitilen önerilen modeller, işaret dili tanıma alanında önemli bir ilerleme temsil etmektedir.

Keywords

• Yapay zeka
• Jest
• Tanımlama
• İşitme engelli

References

1. Addepalli N, Pabolu RK, GaneshChennuru S, Vissampalli VL, Madhumati GL. 2023. Conversion of American Sign Language to text using deep learning for feature extraction and naive bayes for classification. In: IEEE 8th International Conference for Convergence in Technology (I2CT), April 07-09, Lonavla, India, pp: 1.
2. Alshehri S. 2023. The Relationship between Language and Identity. Int J Linguist Lit Transl, 6(7): 156-161.
3. Alzubaidi L, Zhang J, Humaidi AJ, Al-Dujaili A, Duan Y, Al-Shamma O, Santamaría J, Fadhel MA, Al-Amidie M, Farhan L. 2021. Review of deep learning: concepts CNN architectures challenges applications future directions. J Big Data, 8(52): 1-74.
4. Anand K, Urolagin S, Mishra RK. 2021. How does hand gestures in videos impact social media engagement - Insights based on deep learning? Int J Inf Manag Data Insights, 1(2): 100036.
5. Arora S, Roy A. 2018. Recognition of sign language using image processing. Int J Bus Intell Data Min, 13(1-3): 163-176.
6. Bantupalli K, Xie Y. 2019. American Sign Language recognition using deep learning and computer vision. In: IEEE International Conference on Big Data, December 10-13, Seattle, WA, USA, pp: 4896.
7. Hurroo M, Walizad ME. 2020. Sign language recognition system using convolutional neural network and computer vision. Int J Eng Res Technol, 9(12): 59-64.
8. Jantunen T, Rousi R, Rainò P, Turunen M, MoeenValipoor M, García N. 2021. Is there any hope for developing automated translation technology for sign languages? In: Hämäläinen M, Partanen N, Alnajjar K. Editors. Multilingual Facilitation. University of Helsinki Rootroo, pp: 61-73.

9. Kaiming H, Xiangyu Z, Shaoqing R, Jian S. 2016. Deep residual learning for ımage recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, June 27-30, Las Vegas, NV, USA, pp: 770.
10. Karaca MF, Bayır Ş. 2018. Türk işaret dili incelemesi: İletişim ve dil bilgisi. Ulus Eğit Akad Derg, 2(2): 35-58.
11. Katılmış Z, Karakuzu C. 2020. Recognition of two-handed posture finger turkish sign language alphabet. In: 5th International Conference on Computer Science and Engineering (UBMK), September 09-11, Diyarbakir, Türkiye, pp: 5.
12. Kaya F, Tuncer AF, Yildiz Ş. K. 2018. Detection of the turkish sign language alphabet with strain sensor based data glove. In: 26th IEEE Signal Processing and Communications Applications Conference SIU, May 02-05, Izmir, Turkey, pp: 1.
13. Khan SU, Haq IU, Khan N, Muhammad K, Hijji M, Baik SW. 2022. Learning to rank: An intelligent system for person reidentification. Int J Intell Syst, 37(9): 5924-5948.
14. Khan SU, Khan N, Ullah FUM, Kim MJ, Lee MY, Baik SW. 2023. Towards intelligent building energy management: AI-based framework for power consumption and generation forecasting. Energy Build, 279(2023): 112705.
15. Lu D, Yu Y, Liu H. 2016. Gesture recognition using data glove: An extreme learning machine method. In: 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), December 03-07, Qingdao, China, pp: 1349.
16. MacHiraju S, Urolagin S, Mishra RK, Sharma V. 2021. Face mask detection using keras opencv and tensorflow by ımplementing mobilenetv2. In: 2021 3rd International Conference on Advances in Computing Communication Control and Networking (ICAC3N), December 17-18, Greater Noida, India, pp: 1485.
17. Nguyen HBD, Do HN. 2019. Deep learning for American Sign Language fingerspelling recognition system. In: 2019 26th International Conference on Telecommunications (ICT), April 08-10, Hanoi, Vietnam, pp: 314.
18. Oktekin B. 2018. Development of Turkish sign language recognition application. MSc thesis, Near East University the Graduate School Of Applied Sciences, Nicosia, Turkish Republic of Northern Cyprus, pp: 71.
19. Öztürk A, Karatekin M, Saylar İA, Bardakci NB. 2021. Recognition of sign language letters using ımage processing and deep learning methods. J Intell Syst Theory Appl, 4(1): 17-23.
20. Qi J, Jiang G, Li G, Sun Y, Tao B. 2020. Surface EMG hand gesture recognition system based on PCA and GRNN. Neural Comput. Appl, 32(10): 6343-6351.
21. Sabeenian RS, SaiBharathwaj S, MohamedAadhil M. 2020. Sign language recognition using deep learning and computer vision. J Adv Res Dyn Control Syst, 12(5 Special Issue): 964-968.
22. Sadeddine K, Chelali ZF, Djeradi R, Djeradi A, BenAbderrahmane S. 2021. Recognition of user-dependent and independent static hand gestures: Application to sign language. J Vis Commun Image Represent, 79(March): 103193.
23. Sevli O, Kemaloğlu N. 2020. Turkish sign language digits classification with CNN using different optimizers. Int Adv Res Eng J, 4(3): 200-207.
24. ShanmugaPriya G, NitishaSree V, Magisha K, Pooviga S. 2023. Gesture recognition using convolutional neural network. In: 2023 Second International Conference on Advances in Computational Intelligence and Communication (ICACIC), December 07-08, Puducherry, India, pp: 1.
25. Shokoori AF, Shinwari M, Popal JA, Meena J. 2022. Sign language recognition and translation into pashto language alphabets. In: 6th International Conference on Computing Methodologies and Communication (ICCMC), March 29-31, Erode, India, pp: 1401.
26. Singh P, Krishn Mishra R, Urolagin S, Sharma V. 2021. Enhancing Security by ıdentifying facial check-in using deep convolutional neural network. In: 3rd International Conference on Advances in Computing Communication Control and Networking (ICAC3N), December 17-18, Greater Noida, India, pp: 1006.
27. Tan YS, Lim KM, Tee C, Lee CP, Low C. Y. 2021. Convolutional neural network with spatial pyramid pooling for hand gesture recognition. Neural Comput Appl, 33(10): 5339-5351.
28. Thomas J, Mcdonagh D. 2013. Shared language: Towards more effective communication. Australas Med J, 6(1): 46-54.
29. Toğaçar M, Cömert Z, Ergen B. 2021. Recognition of the digits in Turkish sign language using siamese neural networks. Dokuz Eylul Uni Fac Eng J Sci Eng, 23(68): 349-356.
30. Unutmaz B, Karaca A. C, Güllü M. K. 2019. Kinect iskelet ve evrişimsel sinir ağları ile Türkçe işaret dili tanıma. In: 2019 27th Signal Processing and Communications Applications Conference (SIU), April 24-26, Sivas, Türkiye, pp: 2.
31. Vijayalakshmi P, Aarthi M. 2016. Sign language to speech conversion. In: 2016 International Conference on Recent Trends in Information Technology (ICRTIT 2016), April 08-09, Chennai, India, pp: 1.
32. Yalçin M, Ilgaz S, Özkul G, KumbayYildiz Ş. 2018. Türkçe işaret dili alfabesi çevirici / Turkish sign language alphabet translator. In: 26th Signal Processing and Communications Applications Conference (SIU): May 02-05 Izmir, Türkiye, pp: 4.
33. Yıldız Z, Yıldız S, Bozyer S. 2018. İşitme engelli turizmi̇ sessizturizm: dünya ve türkiye potansiyeline yönelik bir değerlendirme. Süleyman Demirel Üniv Vizyoner Derg, 9(20): 103-117.