Topluluk Öğrenmesi ve Bayes Optimizasyonu ile İşaret Dili Tanıma: Derin Öğrenme Temelli Bir Yaklaşım

Abstract

İletişim, insanlık tarihinin en temel ihtiyaçlarından biri olarak sürekli bir evrim geçirmiştir. İlk dönemlerde beden dili ve jestlerle kurulan iletişim, zamanla konuşma dilinin gelişmesiyle daha karmaşık bir yapıya bürünmüştür. Yazının icadı ise iletişimde devrim niteliğinde bir adım olmuştur. Ancak, bu hızlı gelişim iletişim sorunlarını da beraberinde getirmiştir. Günümüz dünyasında, bu problemler üzerine birçok çalışma yapılmakta ve çözüm yolları aranmaktadır. Teknolojik gelişmeler ve yapay zekâ, iletişim sorunlarına çözüm üretme potansiyeli taşımaktadır. Özellikle işitme engelli bireylerle iletişimde yaşanan zorluklar bu alanda öne çıkmaktadır. Bu çalışmada, işaret dili ile iletişimi ko-laylaştırmak amacıyla yapay zekâ algoritmaları kullanılarak Amerikan İşaret Dili'ni tespit eden bir model geliştirilmiştir. InceptionV3, DenseNet169 ve VGG16 gibi derin öğrenme mimarileri kullanılarak oluşturulan model, Kaggle veri seti üzerinde eğitilmiş ve sonuçlar ensemble learning yöntemiyle birleştirilmiştir. Model performansları, Bayes optimizasyonu ile optimize edilmiş ve karmaşıklık matrislerine dayanan metriklerle değerlendirilmiştir. Sonuçlar, ensemble learning modellerinin daha yüksek performans gösterdiğini ortaya koymuş, bu modelin işitme engelli bireylerle iletişimde kullanılabilecek etkili bir araç olabileceği sonucuna ulaşılmıştır.

Keywords

• Topluluk öğrenmesi
• bayes hiperparametre optimizasyonu
• InceptionV3
• DenseNet169
• VGG16

Sign Language Recognition with Ensemble Learning and Bayesian Optimization: A Deep Learning-Based Approach

Abstract

Communication has undergone a continuous evolution as one of the most fundamental needs in human history. Initially, communication was established through body language and gestures, but it became more complex over time with the development of spoken language. The invention of writing marked a revolutionary milestone in the history of communication. However, this rapid advancement also brought about communication challenges. In today’s world, numerous studies focus on addressing these issues and finding effective solutions. Technological advancements and artificial intelligence hold significant potential for solving communication problems. Notably, the difficulties in communicating with individuals who are hearing impaired have become a prominent area of focus. In this study, a model was developed using artificial intelligence algorithms to facilitate communication through sign language, specifically detecting American Sign Language. The model was created using deep learning architectures such as InceptionV3, DenseNet169, and VGG16, and trained on a dataset sourced from Kaggle. The results were combined using the ensemble learning method. The performance of the models was optimized through Bayesian search optimization algorithm and evaluated using metrics derived from confusion matrices. The findings revealed that ensemble learning models demonstrated superior performance, indicating that this model could serve as an effective tool in improving communication with hearing-impaired individuals.

Keywords

• Ensemble learning
• bayesian hyperparameter optimization
• InceptionV3
• Dense-Net169
• VGG16

References

1. Quinto-Pozos, D. (2011). Teaching American Sign Language to hearing adult learn-ers. Annual Review of Applied Linguistics, 31, 137-158.
2. Chowdhary, K., & Chowdhary, K. R. (2020). Natural language processing. Fundamentals of artificial intelligence, 603-649. https://doi.org/10.1007/978-81-322-3972-7_19.
3. Abu-Jamie, T. N., & Abu-Naser, S. S. (2022). Classification of sign-language using Mo-bileNet - deep learning. International Journal of Academic Information Systems Research, 6(7), 29–40.
4. Abu-Jamie, T. N., & Abu-Naser, S. S. (2022). Classification of sign-language using VGG16. International Journal of Academic Engineering Research, 6(6), 36–46.
5. Murali, R. S. L., Ramayya, L. D., & Santosh, V. A. (2020). Sign language recognition system using convolutional neural network and computer vision. Int J EngInnov Technol, 2582-1431.
6. Pigou, L., Dieleman, S., Kindermans, P. J., & Schrauwen, B. (2015). Sign language recog-nition using convolutional neural networks. In Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part I 13 (pp. 572-578). Springer Inter-national Publishing.
7. Daroya, R., Peralta, D., & Naval, P. (2018, October). Alphabet sign language image classi-fication using deep learning. In TENCON 2018-2018 IEEE Region 10 Conference (pp. 0646-0650). IEEE.
8. Nareshkumar, M. D., & Jaison, B. (2023). A Light-Weight Deep Learning-Based Architec-ture for Sign Language Classification. Intelligent Automation & Soft Computing, 35(3).

9. Bhattacharya, A., Zope, V., Kumbhar, K., Borwankar, P., & Mendes, A. (2019). Classifi-cation of sign language gestures using machine learning. Image, 8(12).
10. Hasan, M. M., Srizon, A. Y., Sayeed, A., & Hasan, M. A. M. (2020, November). Classifi-cation of sign language characters by applying a deep convolutional neural network. In 2020 2nd International Conference on Advanced Information and Communication Technology (ICAICT) (pp. 434-438). IEEE.
11. Barbhuiya, A. A., Karsh, R. K., & Jain, R. (2021). CNN based feature extraction and clas-sification for sign language. Multimedia Tools and Applications, 80(2), 3051-3069.
12. Amrutha, K., & Prabu, P. (2021, February). ML based sign language recognition system. In 2021 International Conference on Innovative Trends in Information Technology (ICITIIT) (pp. 1-6). IEEE.
13. Abdul, W., Alsulaiman, M., Amin, S. U., Faisal, M., Muhammad, G., Albogamy, F. R., Bencherif, M. A., & Ghaleb, H. (2021). Intelligent real-time Arabic sign language classification using attention-based Inception and BiLSTM. Computers and Electrical Engineering, 95, 107395. https://doi.org/10.1016/j.compeleceng.2021.107395.
14. Leth, P. G. (2023). Danish sign language recognition in virtual reality using written lan-guage ensemble learning (Master’s thesis, Aalborg University).
15. Kothadiya, D. R., Bhatt, C. M., Rehman, A., Alamri, F. S., & Saba, T. (2023). SignExplainer: an explainable AI-enabled framework for sign language recognition with ensemble learning. IEEE Access, 11, 47410-47419.
16. Öztürk, Ş., Yiğit, E., & Özkaya, U. (2020). Fused deep features based classification framework for COVID-19 classification with optimized MLP. Konya Journal of Engineering Sciences, 8, 15-27.
17. Benbakreti, S., Benbakreti, S., & Ozkaya, U. (2024). The classification of eye diseases from fundus images based on CNN and pretrained models.
18. Bredun, R. (n.d.). Sign Language: ENG Alphabet [Data set]. Kaggle. https://www.kaggle.com/datasets/ruslanbredun/sign-language-eng-alphabet.
19. Tammina, S. (2019). Transfer learning using vgg-16 with deep convolutional neural network for classifying images. International Journal of Scientific and Research Publications (IJSRP), 9(10), 143-150.
20. Younis, A., Qiang, L., Nyatega, C. O., Adamu, M. J., & Kawuwa, H. B. (2022). Brain tumor analysis using deep learning and VGG-16 ensembling learning approaches. Applied Scienc-es, 12(14), 7282.
21. Pan, Y., Liu, J., Cai, Y., Yang, X., Zhang, Z., Long, H., Zhao, K., Yu, X., Zeng, C., Duan, J., Xiao, P., Li, J., Cai, F., Yang, X., & Tan, Z. (2023). Fundus image classification using Inception V3 and ResNet 50 for the early diagnostics of fundus diseases. Frontiers in Physiology, 14, Article 1126780. https://doi.org/10.3389/fphys.2023.1126780.
22. Lin, C., Li, L., Luo, W., Wang, K. C., & Guo, J. (2019). Transfer learning based traffic sign recognition using inception-v3 model. Periodica Polytechnica Transportation Engineering, 47(3), 242-250.
23. Nair, K., Deshpande, A., Guntuka, R., & Patil, A. (2022). Analysing X-ray images to detect lung diseases using DenseNet-169 technique. Available at SSRN 4111864.
24. Frazier, P. I. (2018). A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811.
25. Wu, J., Chen, X. Y., Zhang, H., Xiong, L. D., Lei, H., & Deng, S. H. (2019). Hyperparameter optimization for machine learning models based on Bayesian optimization. Journal of Electronic Science and Technology, 17(1), 26-40.
26. Polikar, R. (2012). Ensemble learning. In C. Zhang & Y. Ma (Eds.), Ensemble Machine Learning: Methods and Applications (pp. 1–34). Springer US. https://doi.org/10.1007/978-1-4419-9326-7_1.
27. Dietterich, T. G. (2002). Ensemble learning. The handbook of brain theory and neural networks, 2(1), 110-125.
28. Rojarath, A., Songpan, W., & Pong-inwong, C. (2016, August). Improved ensemble learning for classification techniques based on majority voting. In 2016 7th IEEE international conference on software engineering and service science (ICSESS) (pp. 107-110). IEEE.
29. Manconi, A., Armano, G., Gnocchi, M., & Milanesi, L. (2022). A soft-voting ensemble classifier for detecting patients affected by COVID-19. Applied Sciences, 12(15), 7554.
30. Banda, J. M., Angryk, R. A., & Martens, P. C. H. (2013). Steps toward a large-scale solar image data analysis to differentiate solar phenomena. Solar Physics, 288, 435-462.
31. Hark, C. (2022). Sahte Haber Tespiti için Derin Bağlamsal Kelime Gömülmeleri ve Sinirsel Ağların Performans Değerlendirmesi. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 34(2), 733-742.
32. Powers, D. M. (2020). Evaluation: from precision, recall and F-measure to ROC, in-formedness, markedness and correlation. arXiv preprint arXiv:2010.16061.