ERUSLR: a new Turkish sign language dataset and its recognition using hyperparameter optimization aided convolutional neural network

Öz

Sign language is one of the most important tools for the communication for deaf-and-dumb individuals who have lost their linguistic and auditory abilities. It is a very difficult process to learn the sign language, where communication involves using hand movements, mimic or lip movements. Significant problems may arise in situations where the sign language required to clearly understand deaf-and-dumb individuals is not known. More importantly, the failure to understand the disabled individuals who try to access emergency health services at a health institution may have fatal consequences. In this study, firstly, a new dataset was created with the frequently used words in the emergency department of hospitals. 25 words were repeated multiple times by 49 handicapped individuals where the videos were recorded from different angles. This dataset, named Erciyes University Sign Language Recognition (ERUSLR), contains 13186 samples. Using the developed ERUSLR dataset, classification experiments were performed. Sign language recognition can be realized by convolutional neural network (CNN), which is frequently used for classification problems. Rather than developing a new CNN model, transfer learning, an easier and more effective method, is preferred. Consequently, a GoogLeNet based CNN model was created by transfer learning from the GoogLeNet pre-trained model. Another factor that increases the performance of a CNN model is the optimization of its training parameters. Global and heuristic search methods are typically used in parameter optimization to save time. In this study, both grid search (GS), random search (RS), and genetic algorithm (GA) methods were used to optimize the training parameters of the GoogLeNet based CNN model. According to the experimental results, the GA supported GoogLeNet -based CNN model is more successful (with a success rate of 93.93%) than the other methods.

Anahtar Kelimeler

• Sign language recognition
• transfer learning
• GoogLeNet
• parameter optimization
• global search
• heuristic search

ERUSLR: Yeni bir Türkçe işaret dili veri seti ve hiperparametre optimizasyonu destekli evrişimli sinir ağı ile tanınması

Öz

İşaret dili, dilsel ve işitsel yetilerini kaybeden konuşma ve duyma engelli bireylerin iletişimini sağlayan en önemli araçtır. El hareketi, mimik veya dudak hareketi kullanılarak iletişimin sağlandığı işaret dilini öğrenmek oldukça zor bir süreçtir. Sağır ve dilsiz bireylerin anlaşılması için gerekli olan işaret dilinin bilinmediği ortamlarda ciddi sorunlar ortaya çıkabilir. Hastanelerin acil servislerine başvuran engelli bireylerin anlaşılamaması ise kritik sonuçlar doğurabilir. Bu çalışmada, öncelikle, hastanelerin acil servisinde sıklıkla kullanılan kelimelerle yeni bir veri seti oluşturulmuştur. 25 kelime, 49 engelli birey tarafından birden fazla tekrarlanmış ve farklı açılardan videoları kaydedilmiştir. Erciyes University Sign Language Recognition (ERUSLR) adı verilen bu veri seti 13186 örnek içermektedir. Geliştirilen ERUSLR veri seti kullanılarak bir sınıflandırma modeli oluşturmak istenmiştir. İşaret dilinin tanınması, son yıllarda sınıflandırma problemlerinde sıklıkla kullanılan evrişimli sinir ağı (CNN) ile gerçekleşebilmektedir. Yeni bir CNN modelinin geliştirilmesinden daha kolay ve etkili olan yöntem, transfer öğrenme ile CNN modeli oluşturmaktır. Dolayısıyla, GoogLeNet ön eğitimli modelinden transfer öğrenme gerçekleştirilerek GoogLeNet tabanlı bir CNN modeli oluşturulmuştur. CNN modelinin performansını artıran bir başka etken eğitim parametrelerinin optimize edilmesidir. Global ve sezgisel arama yöntemleri, parametre optimizasyonunda kullanılan ve zamansal kazanç sağlayan metotlardır. Bu çalışmada grid arama (GS), rastgele arama (RS) ve genetik algoritma (GA) yöntemleri, GoogLeNet tabanlı CNN modelinin eğitim parametrelerini optimize etmek için kullanılmıştır. Deneysel sonuçlara göre, GA destekli GoogLeNet tabanlı CNN modeli (%93,93 başarı oranıyla) diğer yöntemlerden daha başarılı sonuç vermiştir.

Anahtar Kelimeler

• : İşaret dili tanıma
• transfer öğrenme
• GoogLNet
• parametre optimizasyonu
• global arama
• sezgisel arama

Kaynakça

1. Ong E.J., Cooper H., Pugeault N., Bowden R., Sign language recognition using sequential pattern trees, Conference on Computer Vision and Pattern Recognition, Washington-USA, 2200–2207, 16-21 Haziran, 2012.
2. Ong E.J., Koller O., Pugeault N., Bowden R., Sign spotting using hierarchical sequential patterns with temporal intervals, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington-USA, 1923–1930, 23-28 Haziran, 2014.
3. Athitsos V., Neidle C., Sclaroff S., Nash J., Stefan A., Yuan Q., Thangali A., The american sign language lexicon video dataset, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Alaska-USA, 1–8, 23-28 Haziran, 2008.
4. Neidle C., Thangali A., Sclaroff S., Challenges in development of the american sign language lexicon video dataset(asllvd)corpus, Proc.5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, Language Resources and Evaluation Conference (LREC) 2012, İstanbul-Turkey, 1-8, 23-27 Mayıs 2012.
5. Kim J.H., Kim N., Park H., Park J.C., Enhanced sign language transcription system via hand tracking and pose estimation, Journal of Computing Science and Engineering, 10 (3), 95–101, 2016.
6. Metaxas D., Dilsizian M., Neidle C., Scalable ASL sign recognition using model-based machine learning and linguistically annotated corpora, 8th Workshop on the Representation & Processing of Sign Languages: Involving the Language Community, Language Resources and Evaluation Conference, Miyazaki-Japan, 1-5, 12 Mayıs, 2018.
7. Oszust M., Wysocki M., Polish sign language words recognition with Kinect, 2013 6th International Conference on Human System Interactions (HSI), Gdansk-Poland, 219–226, 6-8 Haziran, 2013.
8. Oszust M. ve Wysocki M., Some Approaches to Recognition of Sign Language Dynamic Expressions with Kinect, Advances in Intelligent Systems and Computing, vol 300, Hippe Zdzisaw S., Springer Cham, 75-86, 2014.

9. Kapuscinski T., Oszust M., Wysocki M., Warchol D., Recognition of hand gestures observed by depth cameras, International Journal of Advanced Robotic Systems,12 (4):36, 1-15, 2015.
10. Ronchetti F., Quiroga F., Estrebou C.A., Lanzarini L.C., Rosete A., LSA64: an Argentinian sign language dataset, CACIC 2016, Roma-Italy, 1-10, 3-7 Ekim, 2016.
11. Ronchetti F., Thesis Overview: Dynamic Gesture Recognition and its Application to Sign Language, Journal of Computer Science and Technology, 17, 1–10. 2017.
12. Konstantinidis D., Dimitropoulos K., Daras P., Sign Language Recognition based on Hand and Body Skeletal Data, 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), Haziran, 1–4, 2018.
13. Masood S., Srivastava A., Thuwal H.C., Ahmad M., Real-Time Sign Language Gesture (Word) Recognition from Video Sequences Using CNN and RNN, Intelligent Engineering Informatics, Springer Singapore, 623–632, 2018.
14. Chai X., Wang H., Chen X., The devisign large vocabulary of chinese sign language database and baseline evaluations, Technical report VIPL-TR-14-SLR-001. Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, 2014.
15. Zheng L., Liang B., Sign language recognition using depth images, 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), Phuket-Thailand, 1-6, 13-15 Kasım, 2016.
16. Yıldız O., Derin öğrenme yöntemleriyle dermoskopi görüntülerinden melanom tespiti: Kapsamlı bir çalışma, Journal of the Faculty of Engineering and Architecture of Gazi University, 34, 2241–2260. 2019.
17. Basturk, A., Sarikaya Basturk N., Qurbanov O., A comparative performance analysis of various classifiers for finger print recognition, Omer Halisdemir Universitesi Muhendislik Bilimleri Dergisi, 7, 504 – 513, 2018.
18. Badem H., Basturk A., Caliskan A., Yuksel M.E., A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited–memory BFGS optimization algorithms, Neurocomputing, 266, 506 – 526, 2017.
19. Arı A., Hanbay D., Bölgesel evris¸imsel sinir ağları tabanlı MR görüntülerinde tümör tespiti, Journal of the Faculty of Engineering and Architecture of Gazi University, 34, 1395 – 1408, 2019.
20. Yuksel M.E., Basturk N.S., Badem H., Caliskan A., Basturk A., Classification of high resolution hyperspectral remote sensing data using deep neural networks, Journal of Intelligent & Fuzzy Systems, 34, 2273–2285, 2018.
21. Badem H., Basturk A., Caliskan A., Yuksel M.E., A new hybrid optimization method combining artificial bee colony and limited-memory BFGS algorithms for efficient numerical optimization, Applied Soft Computing, 70, 826 – 844, 2018.
22. Maraqa M., Abu-Zaiter R., Recognition of Arabic Sign Language (ArSL) using recurrent neural networks, 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT), Ostrava-Czech Republic, 478–481, 4-6 Ağustos, 2008.
23. Flores C.J.L., Cutipa A.G., Enciso R.L., Application of convolutional neural networks for static hand gestures recognition under different invariant features, International Conference on Electronics, Electrical Engineering and Computing (INTERCON), Cuzco-Peru, 1–4, 15-18 Ağustos, 2017.
24. Alashhab S., Gallego A.J., Lozano M.Á., Hand Gesture Detection with Convolutional Neural Networks, International Symposium on Distributed Computing and Artificial Intelligence, 45–52, Springer, 2018.
25. Krizhevsky A., Sutskever I., Hinton G.E., ImageNet Classification with Deep Convolutional Neural Networks, NIPS, 1106–1114, 2012.
26. Cote-Allard U., Fall C.L., Campeau-Lecours A., Gosselin C., Laviolette F., Gosselin B., Transfer learning for sEMG hand gestures recognition using convolutional neural networks, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Banff-Canada, 1663–1668, 5-8 Ekim, 2017.
27. Sanchez-Illana A., Pérez-Guaita D., Cuesta-García D., Sanjuan-Herráez J.D., Vento, M. Ruiz-Cerdá J.L., Quintas G., Kuligowski J., Model selection for within-batch effect correction in UPLC-MS metabolomics using quality control-Support vector regression, Analyticachimicaacta, 1026, 62–68, 2018.
28. Ozcan, T., Basturk, A., Transfer learning-based convolutional neural networks with heuristic optimization for hand gesture recognition, Neural Computing and Applications, 31 (12), 8955–8970, 2019.
29. Chen P., Xie Y., Jin P., Zhang D., A wireless sensor data-based coalmine gas monitoring algorithm with least squares support vector machines optimized by swarm intelligence techniques, International Journal of Distributed Sensor Networks, 14 (5), 1–21, 2018.
30. Stoean R., Analysis on the potential of an EA–surrogate modelling tandem for deep learning parameter optimization: anexample for cancer classification from medical images, Neural Computing and Applications, 1-10, 2018.
31. Rubio G., Pomares H., Rojas I., Herrera L.J., A heuristic method for parameter selection in LS-SVM: Application to time series prediction, International Journal of Forecasting, 27 (3), 725 – 739, 2011.
32. Klein A., Falkner S., Bartels S., Hennig P., Hutter F., Fast bayesian optimization of machine learning hyperparameters on large datasets, arXiv, preprint arXiv:1605.07079, 1-9, 2016.
33. Hinz T., Navarro-Guerrero N., Magg S., Wermter S., Speeding up the hyperparameter optimization of deep convolutional neural networks, International Journal of Computational Intelligence and Applications, 17(02), 1-15, 2018.
34. Kousias K., Riegler M., Alay Ö., Argyriou A., HINDSIGHT: an R-based framework towards long short term memory (LSTM) optimization, Proceedings of the 9th ACM Multimedia Systems Conference, Amsterdam-Netherlands, 381–386, 12-15 Haziran, 2018.
35. Czuszynski K., Ruminski J., Kwasniewska A., Gesture recognition with the linear optical sensor and recurrent neural networks, IEEE Sensors Journal, 18 (13), 5429–5438, 2018.
36. Ng, A., 2018. Machine Learning Yearning, deeplearning.ai.
37. Holland, J.H., 1992. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence, A Bradford Book.
38. Ozcan T., Basturk, A., Static Image-Based Emotion Recognition Using Convolutional Neural Network, Signal Processing and Communications Applications Conference (SIU), Sivas-Turkey, 1–4, 24-26 Nisan, 2019.
39. Szegedy C., Liu W., Jia Y., Sermanet P., Reed S.E., Anguelov D., Erhan D., Vanhoucke V., Rabinovich A., Going Deeper with Convolutions, CoRR, 1409.4842, 1-12, 2014.
40. He K., Zhang X., Ren S., Sun J., Deep Residual Learning for Image Recognition, CoRR, 1512.03385, 1-12, 2015.
41. Ozcan T., Basturk A., Lip Reading Using Convolutional Neural Networks with and without Pre-Trained Models, Balkan Journal of Electrical and ComputerEngineering, 7 (2), 195–201, 2019.