Classifiers and Image Processıng to Identify Sign Language Phonemes

Year 2024, Volume: 10 Issue: 2, 219 - 232

Ebru Efeoğlu , Ayşe Tuna

https://doi.org/10.34186/klujes.1546178

Abstract

One of the most visible symptoms of autism spectrum disorder is difficulty in speech and language. Difficulties in speech and language are generally very different for each child with autism spectrum disorder. Although some children with autism spectrum disorder can speak fluently, others will not be able to speak normally or will be even nonverbal. In all cases, parents try to communicate with, and understand their children’s needs, desires, and emotions. If a child with autism spectrum disorder cannot speak out loud, it is harder to communicate with him/her but there are other non-vocal methods for communication. In this paper, the benefits of teaching American sign language to children with autism spectrum disorder, the difficulties that families and children will experience while doing this, and technological solutions to these difficulties are presented. In parallel with advancements in technology, novel solutions to understand and use sign language have been proposed and these solutions are supposed to help parents who cannot understand sign language. Such solutions typically rely on image processing methods and classification algorithms to recognise sign language. Therefore, in this paper, the performance of various classification algorithms used to classify American Sign Language phonemes is compared. As the results show, when combined with image processing methods, classification algorithms can be used in various technological solutions aiming at helping to identify sign language phonemes.

Keywords

Autism spectrum disorder;, ; Classifiers, Image processing, Machine learning, Sign language

İşaret Dili Ses Birimlerini Belirlemek İçin Sınıflandırıcılar ve Görüntü İşleme

Abstract

Otizm spektrum bozukluğunun en belirgin semptomlarından biri konuşma ve dil zorluğudur. Konuşma ve dil zorlukları genellikle otizm spektrum bozukluğu olan her çocuk için çok farklıdır. Otizm spektrum bozukluğu olan bazı çocuklar akıcı bir şekilde konuşabilse de, diğerleri normal konuşamayacak veya hatta sözsüz konuşacaktır. Her durumda, ebeveynler çocuklarıyla iletişim kurmaya ve onların ihtiyaçlarını, isteklerini ve duygularını anlamaya çalışırlar. Otizm spektrum bozukluğu olan bir çocuk yüksek sesle konuşamıyorsa, onunla iletişim kurmak daha zordur ancak iletişim için başka sözsüz yöntemler de vardır. Bu makalede, otizm spektrum bozukluğu olan çocuklara Amerikan işaret dilini öğretmenin faydaları, ailelerin ve çocukların bunu yaparken yaşayacakları zorluklar ve bu zorluklara yönelik teknolojik çözümler sunulmaktadır. Teknolojideki gelişmelere paralel olarak, işaret dilini anlamak ve kullanmak için yeni çözümler önerilmiştir ve bu çözümlerin işaret dilini anlayamayan ebeveynlere yardımcı olması beklenmektedir. Bu tür çözümler genellikle işaret dilini tanımak için görüntü işleme yöntemlerine ve sınıflandırma algoritmalarına dayanmaktadır. Bu nedenle, bu makalede, Amerikan İşaret Dili fonemlerini sınıflandırmak için kullanılan çeşitli sınıflandırma algoritmalarının performansı karşılaştırılmıştır. Sonuçların gösterdiği gibi, görüntü işleme yöntemleriyle birleştirildiğinde, sınıflandırma algoritmaları işaret dili fonemlerini tanımlamaya yardımcı olmayı amaçlayan çeşitli teknolojik çözümlerde kullanılabilir.

Keywords

Otizm spektrum bozukluğu, sınıflandırıcılar, Görüntü işleme, Makine öğrenmesi, İşaret dili

References

• Abu Alfeilat, H. A., Hassanat, A., Lasassmeh, O., Tarawneh, A. S., Alhasanat, M. B., Eyal Salman, H. S., & Prasath, V. (2019). Effects of Distance Measure Choice on K-Nearest Neighbor Classifier Performance: A Review. Big Data, 7(4), 221-248. https://doi.org/10.1089/big.2018.0175
• Baker-Ramos, L. K. (2017). Gesture and Signing in Support of Expressive Language Development. i.e.: inquiry in education, 9(2), 2. Retrieved from: https://digitalcommons.nl.edu/ie/vol9/iss2/2
• Biçek, E. & Almalı, M. N. (2020). A Mobile Application That Allows People Who Do Not Know Sign Language to Teach Hearing-Impaired People by Using Speech-to-Text Procedures. International Journal of Applied Mathematics Electronics and Computers, 8(1), 27-33. https://doi.org/10.18100/ijamec.682806
• Boedeker, P., & Kearns, N. T. (2019). Linear Discriminant Analysis for Prediction of Group Membership: A User-Friendly Primer. Advances in Methods and Practices in Psychological Science, 2(3), 250-263. https://doi.org/10.1177/2515245919849378
• Bonvillian, J. D., & Nelson, K. E. (1976). Sign language acquisition in a mute autistic boy. Journal of Speech & Hearing Disorders, 41(3), 339-347. https://doi.org/10.1044/jshd.4103.339
• Bonvillian, J. D., Nelson, K. E., & Rhyne, J. M. (1981). Sign language and autism. Journal of Autism and Developmental Disorders, 11(1), 125-137. https://doi.org/10.1007/BF01531345
• Bowman-Smart, H., Gyngell, C., Morgan, A., & Savulescu, J. (2019). The moral case for sign language education. Monash Bioethics Review, 37(3-4), 94-110. https://doi.org/10.1007/s40592-019-00101-0
• Brown, R. S. (1978). Why Are Signed Languages Easier to Learn than Spoken Languages? Part Two. Bulletin of the American Academy of Arts and Sciences, 32(3), 25-44.
• Carr, E. G. (1979). Teaching autistic children to use sign language: some research issues. Journal of Autism and Developmental Disorders, 9(4), 345-359. https://doi.org/10.1007/BF01531444
• Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, 6. https://doi.org/10.1186/s12864-019-6413-7
• De Giacomo, A., Craig, F., Terenzio, V., Coppola, A., Campa, M. G., & Passeri, G. (2016). Aggressive Behaviors and Verbal Communication Skills in Autism Spectrum Disorders. Global Pediatric Health. https://doi.org/10.1177/2333794X16644360
• Delgado, R., & Tibau, X. A. (2019). Why Cohen's Kappa should be avoided as performance measure in classification. PloS One, 14(9), e0222916. https://doi.org/10.1371/journal.pone.0222916
• Denmark, T., Atkinson, J., Campbell, R., & Swettenham, J. (2019). Signing with the Face: Emotional Expression in Narrative Production in Deaf Children with Autism Spectrum Disorder. Journal of Autism and Developmental Disorders, 49(1), 294-306. https://doi.org/10.1007/s10803-018-3756-x
• Fawcett, T. (2006). An Introduction to ROC Analysis. Pattern Recognition Letters, 27(8), 861-874. https://doi.org/10.1016/j.patrec.2005.10.010
• Fitzpatrick, E. M., Stevens, A., Garritty, C., & Moher, D. (2013). The effects of sign language on spoken language acquisition in children with hearing loss: a systematic review protocol. Systematic Reviews, 2, 108. https://doi.org/10.1186/2046-4053-2-108
• Geurts, P., Ernst, D. & Wehenkel, L. (2006). Extremely randomized trees. Machine Learning, 63, 3-42. https://doi.org/10.1007/s10994-006-6226-1
• Gladfelter, A., Barron, K. L., & Johnson, E. (2019). Visual and verbal semantic productions in children with ASD, DLD, and typical language. Journal of Communication Disorders, 82, 105921. https://doi.org/10.1016/j.jcomdis.2019.105921
• Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. https://arxiv.org/abs/1704.04861
• Iacono, T., Trembath, D., & Erickson, S. (2016). The role of augmentative and alternative communication for children with autism: current status and future trends. Neuropsychiatric Disease and Treatment, 12, 2349-2361. https://doi.org/10.2147/NDT.S95967
• Jantzen, T. M. (2011). The Power of Sign: Enhancing Oral Communication with Young Children with Typical Hearing. Research Papers, Paper 171, Northern Illinois University. http://opensiuc.lib.siu.edu/gs_rp/171
• Kapp, S. K., Steward, R., Crane, L., Elliott, D., Elphick, C., Pellicano, E., & Russell, G. (2019). 'People should be allowed to do what they like': Autistic adults' views and experiences of stimming. Autism: the international journal of research and practice, 23(7), 1782-1792. https://doi.org/10.1177/1362361319829628
• Kasliwal, N. (2018). Natural Language Processing with Python Quick Start Guide: Going from a Python Developer to an Effective Natural Language Processing Engineer. Birmingham: Packt Publishing Ltd.
• Lal, R., & Sanghvi, D. (2015). Autism and Functional Language Development – An Experiment with AAC Intervention. In (Ed.), Autism Spectrum Disorder - Recent Advances. IntechOpen. https://doi.org/10.5772/59137
• McCleery, J. P., Elliott, N. A., Sampanis, D. S., & Stefanidou, C. A. (2013). Motor development and motor resonance difficulties in autism: relevance to early intervention for language and communication skills. Frontiers in integrative neuroscience, 7, 30. https://doi.org/10.3389/fnint.2013.00030
• McNeil, C. B., Quetsch, L. B., & Anderson, C. M. (2019). Handbook of Parent-Child Interaction Therapy for Children on the Autism Spectrum. Cham: Springer.
• National Academies of Sciences, Engineering, and Medicine. (2017). The Promise of Assistive Technology to Enhance Activity and Work Participation. Washington, DC: The National Academies Press. https://doi.org/10.17226/24740
• Proctor, L. A., & Wang, Y. (2016). Using iPads and Mobile Technology for Children with Developmental Disabilities: Facilitating Language and Literacy Development. In I. Management Association (Ed.), Special and Gifted Education: Concepts, Methodologies, Tools, and Applications (pp. 268-301). IGI Global. https://doi.org/10.4018/978-1-5225-0034-6.ch012
• Rathi, D. (2018). Optimization of Transfer Learning for Sign Language Recognition Targeting Mobile Platform. International Journal on Recent and Innovation Trends in Computing and Communication, 6(4), 198-203.
• Rose, V., Trembath, D., Keen, D., & Paynter, J. (2016). The proportion of minimally verbal children with autism spectrum disorder in a community-based early intervention programme. Journal of Intellectual Disability Research, 60, 464-477. https://doi.org/10.1111/jir.12284
• Sato, W., Sawada, R., Uono, S., Yoshimura, S., Kochiyama, T., Kubota, Y., Sakihama, M., & Toichi, M. (2017). Impaired detection of happy facial expressions in autism. Scientific Reports, 7, 13340. https://doi.org/10.1038/s41598-017-11900-y
• Schaeffer, B., Kollinzas, G., Musil, A., & McDowell, P. (1977). Spontaneous Verbal Language for Autistic Children Through Signed Speech (Vol. 17, p. 287-328). N.p.: Gallaudet University Press. https://doi.org/10.1353/sls.1977.0009
• Schioppo, J., Meyer, Z., Fabiano, D., & Canavan, S. (2020). Sign Language Recognition in Virtual Reality. Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 917-917, doi: 10.1109/FG47880.2020.00027
• Schneden, A. M. (2006). Sign to learn: sign language as a teaching tool in hearing classrooms. Graduate Research Papers, 1475, University of Northern Iowa. Retrieved from https://scholarworks.uni.edu/grp/1475
• Shield, A. & Meier, R. (2014). 4. The Acquisition of Sign Language by Deaf Children with Autism Spectrum Disorder. In D. Quinto-Pozos (Ed.), Multilingual Aspects of Signed Language Communication and Disorder (pp. 90-122). Bristol, Blue Ridge Summit: Multilingual Matters. https://doi.org/10.21832/9781783091317-007
• Simpson, C. G., & Lynch, S. A. (2007). Sign Language: Meeting Diverse Needs in the Classroom. Exchange: The Early Childhood Leaders' Magazine Since 1978, 176, 45-49.
• Szegedy, C., Vanhoucke, V., Ioffe, S., & Shlens, J. (2016). Rethinking the Inception Architecture for Computer Vision. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, pp. 2818-2826.
• Tarver, J., Pearson, E., Edwards, G., Shirazi, A., Potter, L., Malhi, P., & Waite, J. (2021). Anxiety in autistic individuals who speak few or no words: A qualitative study of parental experience and anxiety management. Autism: the international journal of research and practice, 25(2), 429–439. https://doi.org/10.1177/1362361320962366
• Thompson, R. H., Cotnoir-Bichelman, N. M., McKerchar, P. M., Tate, T. L., & Dancho, K. A. (2007). Enhancing early communication through infant sign training. Journal of applied behavior analysis, 40(1), 15-23. https://doi.org/10.1901/jaba.2007.23-06
• Thunberg, G., Ahlsén, E., & Sandberg, A. D. (2007). Children with autistic spectrum disorders and speech-generating devices: communication in different activities at home. Clinical Linguistics & Phonetics, 21(6), 457-479. https://doi.org/10.1080/02699200701314963
• Tomaszewski, P. (2001). Sign Language Development in Young Deaf Children. Psychology of Language and Communication, 5(1), 67-80.
• Toth, A. (2009). Bridge of signs: Can sign language empower non-deaf children to triumph over their communication disabilities? American Annals of the Deaf, 154(2), 85-95. https://doi.org/10.1353/aad.0.0084
• Trevisan, D. A., Roberts, N., Lin, C., & Birmingham, E. (2017). How do adults and teens with self-declared Autism Spectrum Disorder experience eye contact? A qualitative analysis of first-hand accounts. PloS One, 12(11), e0188446. https://doi.org/10.1371/journal.pone.0188446
• Trigueiros, P., Ribeiro, F., & Reis L. P. (2014) Vision-Based Portuguese Sign Language Recognition System. In: Rocha Á., Correia A., Tan F., Stroetmann K. (eds) New Perspectives in Information Systems and Technologies, Volume 1. Advances in Intelligent Systems and Computing, vol 275. Springer, Cham. https://doi.org/10.1007/978-3-319-05951-8_57
• U.S. Department of Health and Human Services. (n.d.). American sign language. National Institute of Deafness and Other Communication Disorders. Retrieved July 3, 2022, from https://www.nidcd.nih.gov/health/american-sign-language
• Valueva, M. V., Nagornov, N. N., Lyakhov, P. A., Valuev, G. V., & Chervyakov, N. I. (2020). Application of the residue number system to reduce hardware costs of the convolutional neural network implementation. Mathematics and Computers in Simulation, 177, 232-243. https://doi.org/10.1016/j.matcom.2020.04.031
• Weinberg, A. I., & Last, M. (2019). Selecting a representative decision tree from an ensemble of decision-tree models for fast big data classification. Journal of Big Data, 6, 23. https://doi.org/10.1186/s40537-019-0186-3
• White, E. N., Ayres, K. M., Snyder, S. K., Cagliani, R. R., & Ledford, J. R. (2021). Augmentative and Alternative Communication and Speech Production for Individuals with ASD: A Systematic Review. Journal of autism and developmental disorders, 51(11), 4199-4212. https://doi.org/10.1007/s10803-021-04868-2