Deep Learning Based Recognition of Turkish Sign Language Letters with Unique Data Set

Abstract

With its development, artificial intelligence has formed the basis for many studies aimed at facilitating people's lives. More successful results have been tried to be obtained with the increasing data and developing equipment in these studies. It is seen that these developments in artificial intelligence are reflected in the studies related to sign language conversion.
In this study, a data set belonging to the letters in the Turkish Sign Language Alphabet was created, and the classification process was carried out with both the deep learning model we created and VGG16, Inceptionv3, Resnet, and Mobilnet models, which are frequently used in image classification. In addition, an open-source data set containing the letters in the American Sign Language Alphabet was organized similar to the data set containing the letters in the Turkish Sign Language Alphabet we created, and Deep Learning models were used to classify the letters in the American Sign Language Alphabet by using this data set. Performance evaluations of the classifications made by Deep Learning Models using both data sets were made. With this study, the results obtained from training Deep Learning methods with different data sets were compared. In addition, it is thought that the study will be useful in determining both the data set and the deep learning method to be used for the studies on the recognition of Sign Language Letters. 

Keywords

• Turkish Sign Language Alphabet
• Deep Learning
• SSD Mobilenet
• EfficientNet
• YOLOv5

References

1. [1] Charayaphan, C., & Marble, A. E. (1992). Image processing system for interpreting motion in American Sign Language, Journal of Biomedical Engineering, 14(5), 419-425.
2. [2] Takahashi, T., & Kishino, F. (1991). Hand gesture coding based on experiments using a hand gesture interface device. Acm Sigchi Bulletin, 23(2), 67-74.
3. [3] Waldron, M. B., & Kim, S. (1995). Isolated ASL sign recognition system for deaf persons. IEEE Transactions on rehabilitation engineering, 3(3), 261-271.
4. [4] Allen, J. M., Asselin, P. K., & Foulds, R. (2003, March). American Sign Language finger spelling recognition system. In 2003 IEEE 29th Annual Proceedings of Bioengineering Conference (pp. 285- 286). IEEE.
5. [5] Wang, H. G., Sarawate, N. N., & Leu, M. C. (2004, July). Recognition of American sign language gestures with a sensory glove. In Japan USA Symposium on Flexible Automation, Denver, CO (pp. 102-109).
6. [6] M. Taskiran, M. Killioglu and N. Kahraman, (2018) "A Real-Time System for Recognition of American Sign Language by using Deep Learning," 2018 41st International Conference on Telecommunications and Signal Processing ( pp. 1-5 )
7. [7] Haberdar, H., & Albayrak, S. (2005, October). Real time isolated turkish sign language recognition from video using hidden markov models with global features. In International Symposium on Computer and Information Sciences (pp. 677-687). Springer, Berlin, Heidelberg.
8. [8] Işikdoğan F., Albayrak S., 2011, June, Automatic recognition of Turkish fingerspelling, In Innovations in Intelligent Systems and Applications (INISTA), 2011 International Symposium on (pp. 264-267), IEEE.

9. [9] Memiş, A., & Albayrak, S. (2013, April). Turkish Sign Language recognition using spatio-temporal features on Kinect RGB video sequences and depth maps. In 2013 21st Signal Processing and Communications Applications Conference (SIU) (pp. 1-4). IEEE.
10. [10] Demircioglu, Burcak & Bülbül, Güllü & Kose, Hatice. (2016). Leap Motion ile Türk İşaret Dili Tanıma / Turkish Sign Language Recognition with Leap Motion. 10.13140/RG.2.1.4923.3529.
11. [11] Ceber, Y. E., Karacaoğlan, E., Uysaf, F., & Tokmakçi, M. (2017, October). The design of glove that can translate sign language to Turkish language. In 2017 Medical Technologies National Congress (TIPTEKNO) (pp. 1-4). IEEE.
12. [12] Firat, Yelda & Uğurlu, Taşkın. (2018). LATİS TABANLI ANLAM ÇÖZÜMLENMESİ İLE TÜRKÇE İŞARET DİLİ TERCÜME SİSTEMİ. Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi. 10.28948/ngumuh.443157.
13. [13] Çelik, Ö , Odabas, A . (2020). Sign2Text: Konvolüsyonel Sinir Ağları Kullanarak Türk İşaret Dili Tanıma . Avrupa Bilim ve Teknoloji Dergisi , (19) , 923-934 .
14. [14] https://towardsdatascience.com/yolo-v4-optimal-speed-accuracy-for-object-detection-79896ed47b50 Erişim Tarihi 06/06/2021.