A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language

Kübra Uçar; Uğur Yüzgeç

doi:10.35377/saucis...1942666

A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language

Abstract

This study introduces a vision-based mobile communication framework engineered to facilitate real-time, bidirectional interaction between deaf or hard-of-hearing and hearing individuals. To mitigate existing communication barriers, the proposed architecture seamlessly integrates deep learning algorithms with Automatic Speech Recognition (ASR) within a unified mobile environment. For the sign-to-text translation module, a Gated Recurrent Unit (GRU) network is utilized to recognize and classify 226 distinct Turkish Sign Language (TSL) words. The temporal model, trained and evaluated using the RGB videos of the AUTSL dataset, processes MediaPipe-based spatial features extracted from video frames and achieves a test accuracy of 76.77%. Predicted signs are subsequently rendered as instantaneous textual subtitles for both participants. Simultaneously, the hearing user’s vocal input is transcribed via a Speech-to-Text (STT) mechanism and displayed in real-time on both users’ screens. By prioritizing a computationally lightweight design and evaluating application-level latency, the developed system demonstrates the feasibility of responsive mobile-based interaction. Consequently, this approach offers a highly scalable solution to substantially enhance accessibility across social, educational, and professional domains.

Keywords

Ethical Statement

The study used the publicly available AUTSL dataset for model training and evaluation. No new biometric dataset was collected for training. The mobile application screenshots were obtained with the informed consent of the participant, and facial regions were anonymized before publication.

References

Z. Katılmış, “Makine öğrenmesi algoritmalarıyla Türk işaret dilinde harf ve dinamik sözcük tanıma,” Ph.D. dissertation, Graduate School of Natural and Applied Sciences, Bilecik Şeyh Edebali Univ., Bilecik, Türkiye, 2021. Accessed: Oct. 14, 2026. [Online]. Available: https://hdl.handle.net/11552/2422
O. Köpüklü, Y. Rong, and G. Rigoll, “Talking with your hands: Scaling hand gestures and recognition with CNNs,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshops (ICCVW), Seoul, South Korea, Oct. 2019, pp. 2836–2845, doi: 10.1109/ICCVW.2019.00345.
Ö. Çelik and A. Odabaş, “Sign2Text: Turkish sign language recognition using convolutional neural networks,” Eur. J. Sci. Technol., no. 19, pp. 923–934, 2020, doi: 10.31590/ejosat.747231.
A. Kumar, K. Thankachan, and M. M. Dominic, “Sign language recognition,” in Proc. 3rd Int. Conf. Recent Adv. Inf. Technol. (RAIT), Dhanbad, India, Mar. 3–5, 2016, pp. 422–428, doi: 10.1109/RAIT.2016.7507939.
World Health Organization, “Deafness and hearing loss,” Mar. 3, 2026. Accessed: Oct. 14, 2026. [Online]. Available: https://www.who.int/news-room/ fact-sheets/detail/deafness-and-hearing-loss
T. Özcan and A. Baştürk, “ERUSLR: A new Turkish sign language dataset and its recognition using hyperparameter optimization aided convolutional neural network,” J. Fac. Eng. Archit. Gazi Univ., vol. 36, no. 1, pp. 527–542, 2021, doi: 10.17341/gazimmfd.746793.
Z. Katılmış and C. Karakuzu, “Recognition of two-handed static fingerspelling in Turkish sign language alphabet,” in Proc. 5th Int. Conf. Comput. Sci. Eng. (UBMK), Diyarbakır, Türkiye, Sep. 9–11, 2020, pp. 181–186, doi: 10.1109/UBMK50275.2020.9219364.
S. Lang, “Sign language recognition with Kinect,” B.Sc. thesis, Institute of Computer Science, Freie Universität Berlin, Berlin, Germany, Sep. 2011.

Z. M. A. Burhani and J. Prasetyo, “The sign language interpreting gloves,” J. Appl. Sci. Adv. Eng., vol. 1, no. 1, pp. 18–27, 2023, doi: 10.59097/jasae.v1i1.10.
A. Wadhawan and P. Kumar, “Deep learning-based sign language recognition system for static signs,” Neural Comput. Appl., vol. 32, no. 12, pp. 7957–7968, 2020, doi: 10.1007/s00521-019-04691-y.
H. B. D. Nguyen and H. N. Do, “Deep learning for American sign language fingerspelling recognition system,” in Proc. 26th Int. Conf. Telecommun. (ICT), Hanoi, Vietnam, Apr. 2019, pp. 314–318, doi: 10.1109/ICT.2019.8798856.
M. De Coster, M. Van Herreweghe, and J. Dambre, “Isolated sign recognition from RGB video using pose flow and self-attention,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Nashville, TN, USA (online), Jun. 19–25, 2021, pp. 3441–3450, doi: 10.1109/CVPRW53098.2021.00383.
O. M. Sincan and H. Y. Keles, “AUTSL: A large scale multi-modal Turkish sign language dataset and baseline methods,” IEEE Access, vol. 8, pp. 181340–181355, 2020, doi: 10.1109/ACCESS.2020.3028072.
I. Grishchenko and V. Bazarevsky, “MediaPipe Holistic—Simultaneous face, hand and pose prediction, on device,” Google Research Blog, Dec. 10, 2020. Accessed: Apr. 16, 2026. [Online]. Available: https://research.google/blog/mediapipe-holistic-simultaneous-face-hand-and-pose-prediction-on-device/
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735.
F. Shahid, A. Zameer, and M. Muneeb, “A novel genetic LSTM model for wind power forecast,” Energy, vol. 223, p. 120069, May 2021, doi: 10.1016/j.energy.2021.120069.
K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder– decoder for statistical machine translation,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1724–1734, doi: 10.3115/v1/D14-1179.
Ultralytics, “Gated recurrent unit (GRU),” Ultralytics Glossary, n.d. Accessed: Oct. 14, 2026. [Online]. Available: https://www.ultralytics.com/tr/glossary/ gated-recurrent-unit-gru
L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324.
B. Eren and İ. Cesur, “Comparative analysis of machine learning models for CO emission prediction in engine performance,” Sakarya Univ. J. Comput. Inf. Sci., vol. 8, no. 1, pp. 1–11, Mar. 2025, doi: 10.35377/saucis...1564937.
T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, San Francisco, CA, USA, Aug. 13–17, 2016, pp. 785–794, doi: 10.1145/2939672.2939785.
M. Niazkar et al., “Applications of XGBoost in water resources engineering: A systematic literature review (Dec 2018–May 2023),” Environ. Model. Softw., vol. 174, p. 105971, Mar. 2024, doi: 10.1016/j.envsoft.2024.105971.
Agora, “Realtime communication overview,” Agora Documentation, n.d. Accessed: Oct. 14, 2026. [Online]. Available: https://docs.agora.io/en/ realtime-media/rtc
Android Developers, “SpeechRecognizer,” Android Developers Documentation, 2026. Accessed: Apr. 16, 2026. [Online]. Available: https://developer. android.com/reference/android/speech/SpeechRecognizer
K. M. Dafnis, E. Chroni, C. Neidle, and D. Metaxas, “Bidirectional skeleton-based isolated sign recognition using graph convolutional networks,” in Proc. 13th Lang. Resour. Eval. Conf. (LREC), Marseille, France: European Language Resources Association, Jun. 2022, pp. 7328–7338.
M. Boháček and M. Hr´uz, “Sign pose-based transformer for word-level sign language recognition,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. Workshops (WACVW), Waikoloa, HI, USA, Jan. 2022, pp. 182–191, doi: 10.1109/WACVW54805.2022.00024.
G. Wang, B. Wang, H. Zhai, Q. Tao, S. Jing, and P. Jin, “Pose-based isolated sign language recognition with semantic mapping and CorrFormer,” in Proc. 28th Int. Conf. Comput. Supported Cooperative Work Design (CSCWD), Compiègne, France, May 2025, pp. 2202–2208, doi: 10.1109/CSCWD64889.2025.11033397.

Details

Primary Language

English

Subjects

Artificial Intelligence (Other)

Journal Section

Research Article

Authors

Kübra Uçar
0009-0007-3496-2215
Türkiye

Uğur Yüzgeç ^*
0000-0002-5364-6265
Türkiye

Early Pub Date

October 27, 2026

Publication Date

October 30, 2026

Submission Date

May 2, 2026

Acceptance Date

July 12, 2026

Published in Issue

Year 2026 Volume: 9 Number: 5

DOI

https://doi.org/10.35377/saucis...1942666

IZ

https://izlik.org/JA73TC23XS

Cite

RIS / Bibtex

APA

Uçar, K., & Yüzgeç, U. (2026). A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language. Sakarya University Journal of Computer and Information Sciences, 9(5), 1451-1463. https://doi.org/10.35377/saucis...1942666

AMA

1.Uçar K, Yüzgeç U. A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language. SAUCIS. 2026;9(5):1451-1463. doi:10.35377/saucis.1942666

Chicago

Uçar, Kübra, and Uğur Yüzgeç. 2026. “A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language”. Sakarya University Journal of Computer and Information Sciences 9 (5): 1451-63. https://doi.org/10.35377/saucis. 1942666.

EndNote

Uçar K, Yüzgeç U (October 1, 2026) A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language. Sakarya University Journal of Computer and Information Sciences 9 5 1451–1463.

IEEE

[1]K. Uçar and U. Yüzgeç, “A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language”, SAUCIS, vol. 9, no. 5, pp. 1451–1463, Oct. 2026, doi: 10.35377/saucis...1942666.

ISNAD

Uçar, Kübra - Yüzgeç, Uğur. “A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language”. Sakarya University Journal of Computer and Information Sciences 9/5 (October 1, 2026): 1451-1463. https://doi.org/10.35377/saucis. 1942666.

JAMA

1.Uçar K, Yüzgeç U. A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language. SAUCIS. 2026;9:1451–1463.

MLA

Uçar, Kübra, and Uğur Yüzgeç. “A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language”. Sakarya University Journal of Computer and Information Sciences, vol. 9, no. 5, Oct. 2026, pp. 1451-63, doi:10.35377/saucis. 1942666.

Vancouver

1.Kübra Uçar, Uğur Yüzgeç. A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language. SAUCIS. 2026 Oct. 1;9(5):1451-63. doi:10.35377/saucis. 1942666