TURKISH SIGN LANGUAGE EXPRESSIONS RECOGNITION USING DEEP LEARNING AND LANDMARK DATA

Abstract

Sign language is a vital communication tool for hearing-impaired individuals to express their thoughts and emotions. Turkish Sign Language (TSL) is based on hand gestures, facial expressions, and body movements. In this study, deep learning models were developed to recognize 41 commonly used TSL expressions. An original dataset was created using the Media Pipe Holistic framework to capture the 3D landmarks of hand, face, and body movements. The study trained and evaluated GRU, LSTM, and Bi-LSTM models, as well as hybrid architectures such as CNN+GRU, GRU+LSTM, and GRU+Bi-LSTM. In the training of the models, a hold-out validation method was used. 80% of the dataset was allocated for training and 20% for testing. Additionally, 20% of the training data was used for validation. Among Deep Learning models, the CNN+GRU hybrid model achieved the highest accuracy rate of 96.72%, outperforming similar studies in the literature. Our results demonstrate that deep learning techniques can effectively classify TSL expressions, with the CNN+GRU combination showing particularly high performance. Future work will focus on expanding the dataset and developing real-time recognition systems that incorporate both skeleton images and landmarks.

Keywords

• Turkish sign language
• Sign language recognition
• Media pipe
• Deep learning
• Recurrent neural network

Supporting Institution

TUBİTAK 1002 A

Project Number

124E379

Thanks

This study was supported by Scientific and Technological Research Council of Turkey (TUBITAK) under the Grant Number 124E379. The authors thank to TUBITAK for their supports.

DERİN ÖĞRENME VE YER İŞARETİ VERİLERİNİ KULLANARAK TÜRK İŞARET DİLİ İFADELERİNİ TANIMA

Abstract

İşaret dili, işitme engelli bireylerin düşüncelerini ve duygularını ifade etmeleri için hayati bir iletişim aracıdır. Türk İşaret Dili (TİD), el hareketleri, yüz ifadeleri ve vücut hareketlerine dayanır. Bu çalışmada, yaygın olarak kullanılan 41 TİD ifadesini tanımak için derin öğrenme modelleri geliştirilmiştir. El, yüz ve vücut hareketlerinin 3D yer işaretlerini yakalamak için Media Pipe Holistic çerçevesi kullanılarak orijinal bir veri seti oluşturulmuştur. Çalışmada, GRU, LSTM, Bi-LSTM modelleri ve hibrit mimarilere sahip olan CNN+GRU, GRU+LSTM, GRU+Bi-LSTM modelleri eğitilmiş ve değerlendirilmiştir. Modellerin eğitiminde dışarda tutma doğrulama yöntemi kullanılmıştır. Veri setinin %80'i eğitim ve %20'si test için ayrılmıştır. Ayrıca eğitim için ayrılan verinin %20'si doğrulama için kullanılmıştır. Derin öğrenme modelleri arasında en yüksek doğruluk oranını %96,72 ile CNN+GRU hibrit modeli elde etmiştir ve literatürdeki benzer çalışmalardan daha yüksek performans göstermiştir. Sonuçlarımız, derin öğrenme tekniklerinin TİD ifadelerini etkili bir şekilde sınıflandırabileceğini ortaya koymaktadır ve özellikle CNN+GRU kombinasyonu yüksek performans sağlamıştır. Gelecek çalışmalar, veri setinin genişletilmesi ve iskelet görüntüleriyle birlikte yer işaretlerinin de kullanıldığı gerçek zamanlı tanıma sistemlerinin geliştirilmesine odaklanacaktır.

Keywords

• Türk işaret dili
• İşaret dili tanıma
• Media pipe
• Derin öğrenme
• Tekrarlayan sinir ağı

Project Number

124E379

References

1. Alaftekin, M., Pacal, I., and Cicek, K., “Real-Time Sign Language Recognition Based on YOLO Algorithm”, Neural Comput Appl, vol. 36, no. 14, 7609–7624, 2024.
2. Yirtici, T. and Yurtkan, K., “Regional-CNN-based Enhanced Turkish Sign Language Recognition”, Signal Image Video Process, vol. 16, no. 5, 1305–1311, 2022.
3. Katılmış, Z. and Karakuzu, C., “Double handed Dynamic Turkish Sign Language Recognition Using Leap Motion with Meta Learning Approach”, Expert Syst Appl, vol. 228, 120453, 2023.
4. Karacı, A., Akyol, K., and Turut, M. U., “Real-Time Turkish Sign Language Recognition Using Cascade Voting Approach with Handcrafted Features”, Applied Computer Systems, vol. 26, no. 1, 12-21, 2021.
5. Pacal, I. and Alaftekin, M., “Türk İşaret Dilinin Sınıflandırılması için Derin Öğrenme Yaklaşımları”, Iğdır Üniversitesi Fen Bilimleri Enstitüsü Dergisi, vol. 13, no. 2, 760-777, 2023.
6. Özcan, T. and Baştürk A., “ERUSLR: A new Turkish Sign Language Dataset and Its Recognition Using Hyperparameter Optimization Aided Convolutional Neural Network”, Journal of the Faculty of Engineering and Architecture of Gazi University, vol. 36, no. 1, 527-542, 2021.
7. Kirci, P., Durusan, B. B., and Özşahin, B., “El Hareketlerinden İşaret Dilini Algılayıp Yazıya Dönüştürme”, European Journal of Science and Technology, 32-35, 2022.
8. Çelik, Ö. and Odabas, A., “Sign2Text: Konvolüsyonel Sinir Ağları Kullanarak Türk İşaret Dili Tanıma”, European Journal of Science and Technology, no. 19, 923 – 934, 2020.

9. Demircioǧlu, Bülbül, B., G., and Köse, H., “Turkish Sign Language Recognition with Leap Motion”, in 2016 24th Signal Processing and Communication Application Conference, SIU 2016 - Proceedings, 2016, 24.
10. Haberdar, H. and Albayrak, S., “Real Time Isolated Turkish Sign Language Recognition from Video Using Hidden Markov models with Global Features”, in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 3733, 677–687, 2005.
11. Memiş, A. and Albayrak, S., “A Kinect Based Sign Language Recognition System Using Spatio-Temporal Features”, Proc. SPIE 9067, Sixth International Conference on Machine Vision (ICMV 2013), 2013, 6.
12. Martinez-Seis, B., Pichardo-Lagunas, O., Rodriguez-Aguilar, E., and Saucedo-Diaz, E.-R., “Identification of Static and Dynamic Signs of the Mexican Sign Language Alphabet for Smartphones using Deep Learning and Image Processing”, Research in Computing Science, vol. 148, no. 11, 199-211, 2019.
13. Aburass, S., Dorgham, O., and Al Shaqsi, J., “A hybrid Machine Learning Model For Classifying Gene Mutations in Cancer Using LSTM, BiLSTM, CNN, GRU, and GloVe”, Systems and Soft Computing, vol. 6, 200110, 2024.
14. Hochreiter, S. and Urgen Schmidhuber, J., “Long Shortterm Memory”, Neural Comput., vol. 9, no. 8, 1735–1780, 1997.
15. Cho, K., Merriënboer, B. van, Bahdanau, D., and Bengio, Y., “On The Properties of Neural Machine Translation: Encoder–Decoder Approaches”, in Proceedings of SSST 2014 - 8th Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014, 8.
16. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y., “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling,”, ArXiv, 2014. arXiv:1412.3555
17. Karacı, A. and Akyol, K., “YoDenBi-NET: YOLO + DenseNet + Bi-LSTM-Based Hybrid Deep Learning Model for Brain Tumor Classification”, Neural Comput Appl., vol. 35, no. 17, 12583–12598, 2023.
18. Karacı, A., “Predicting COVID-19 Cases on a Large Chest X-Ray Dataset Using Modified Pre-trained CNN Architectures”, Applied Computer Systems, vol. 28, no. 1, 44–57, 2023.
19. Maas et al., A. L., “Building DNN Acoustic Models for Large Vocabulary Speech Recognition”, Comput. Speech Lang., vol. 41, 195–213, 2017.