Multimodal assistive wheelchair system operated by finger movement and capable of keyword extraction from images for visually and physically challenged people

Abstract

The Assistive Technology (AT) is a boon for the physically, mentally challenged and aged people. There are various research projects conducted around the world to facilitate the disadvantaged people to lead a normal life. A team of researchers from the University of Technology and Applied Sciences of Salalah campus with the support of the Ministry of Higher Education, Research and Innovation’s funding is working on a project on finger movement multimodal assistive system for the smart people (Disadvantaged people). The research team has identified people who have difficulty in speaking and locomotive disability. The team has designed a manual wheelchair with the visual and audio command options for the helper or the nearby people through the finger movement of the disabled people. The main objective is to convey the commands or instructions to others through simple finger movements which are not to be memorized. The machine learning component of this system will make the instruction personalized for the people and it will not be common for all.

Keywords

• Assistive technology
• Finger movement multi modal system
• Machine learning
• Smart gloves
• Wheel chair with AT

References

1. [1] United Nations. Convention the rights of persons with disabilities. United Nations, 2006. http://www.un.org/disabilities/convention/conventionfull.shtml (accessed on 10 Feb 2022).
2. [2] Kumar, A., Singh R.R., Thressiakutty, AT, & et al. Normalization vs social role valorization: similar or different? Int J Spec Educ, 2015 30:71-8.
3. [3] O’Doherty S, Linehan C, Tatlow-Golden M, et al. Perspectives of family members of people with an intellectual disability to a major reconfiguration of living arrangements for people with intellectual disability in Ireland. J. Intellect Disabil. 2016; 20: 137-51
4. [4] Said Jaboob, Balaji D, et al., 2022. A Study on the Disability Types and Assistive Technology Devices Prevalent Among the Population in Oman, 9th International Conference on Electrical and Electronics Engineering (ICEEE 2022). An IEEE Conference, Alanya, Turkey, during March 29-31, 2022
5. [5] Lecrosnier, L.; Khemmar, R.; Ragot, N.; Ertaud, J.-Y. Deep Learning-Based Object Detection, Localisation and Tracking for Smart Wheelchair Healthcare mobility. Int. J. Environ. Res. Public Health 2021, 18, 91. http://doi.org/10.3390/ijerph18010091
6. [6] Wald, M., "Captioning for Deaf and Hard of Hearing People by Editing Automatic Speech Recognition in Real Time", Proceedings of 10th International Conference on Computers Helping People with Special Needs ICCHP 2006, LNCS 4061, pp. 683-690
7. [7] Simone, L. K., Elovic, E., Kalambur, U., & Kamper, D., "A Low-Cost Method to Measure Finger Flexion in Individuals with Reduced Hand and Finger Range of Motion", 26th Annual International Conference of the IEEE Engineering in Medicine and Biology
8. [8] Zhao, Y., Xia, X. & Togneri, R., "Applications of Deep Learning to Audio Generation," in IEEE Circuits and Systems Magazine, 2019

9. [9] Patel, R., Hand Gesture Recognition System using Convolution Neural Networks,"4th International Conference on Computing Communication and Automation (ICCCA), 2018
10. [10] Petereit, J., Emtr, T., Frey, C.W., Kopfstedt, T., & Beutel, A. Application of Hybrid A* to an autonomous mobile robot for path planning in unstructured outdoor environments. In Proceedings of the 7th German Conference on Robotics, Munich, Germany, 21-22 May 2012.
11. [11] Shino, M.;Yamamoto, Y.; Mikata, T.; Inoue, T. Input device for electric wheelchairs considering physical functions of persons with severe Duchenne muscular dystrophy. Technol. Disabil. 2014, 26, 105-115.
12. [12] Xi, L; Yamamoto, Y.; Shino, M. One-dimensional input device of electric wheelchair for persons with severe Duchenne Muscular Dystrophy. Technol. Disabil. 2019, 31, 101-113.
13. [13] Dhawal L. Patel, Harshal S. Tapase,Praful A. Landge, Parmeshwar P. More, Prof. A. P. Bagade.”SMART HAND GLOVES FOR DISABLE PEOPLE”. International Research Journal of Engineering and Technology (IRJET).Volume: 05 Issue: 04, Apr-2018.
14. [14] Tushar Chouhan, Ankit Panse, Anvesh Kumar Voona and S. M. Sameer, “Smart Glove with Gesture Recognition Ability for Hearing and Speech Impaired,” IEEE Global Humanitarian Technology Conference- South Asia Satellite (GHTC-SAS), September 26-27, 2014.
15. [15] Priya Matnani, “Glove Based and Accelerometer Based Gesture Control: A Literature Review”, International Journal of Technical Research and Applications e-ISSN: 2320-8163, www.ijtra.com Volume 3, Issue 6 (November-December, 2015), PP.216-221.
16. [16] Zeng, Q.; Teo, C.L.; Rebsamen, B., & Burdet, E. Collaborative path planning for a robotic wheelchair. Disabil. Rehabil. Assit. Technol. 2008, 3, 315-324.
17. [17] Guerreiro, J., Sato, D., Asakawa, S., Dong, H., Kitani, K. M., & Asakawa, C. (2019). Cabot: Designing and evaluating an autonomous navigation robot for blind people. The 21st International ACM SIGACCESS Conference on Computers and Accessibility. DOI: 10.1145/3308561.3353771
18. [18] Said, J., Munesh, S.C., Balaji D., & Senthil K.N., Multimodal Assistive Technologies Prototype for the Disabled in Oman, 2022 6th international conference on computer, software and modeling, Rome, Italy, July 21-23, 2022
19. [19] Papineni, K., Roukos, S., Ward, T. & Zhu, W.J. “BLEU: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, 2001.
20. [20] Ghorbel, M., Pineau, J., Gourdeau, R., Javdani, S., & Srinivasa, S. (2018). A decision-theoretic approach for the collaborative control of a smart wheelchair. International Journal of Social Robotics,” 10(1): 131- 145.
21. [21] Sivakumar, B. G., & Sudhagar, K. (2018) “Progression of stair climbing wheelchair of the microcontroller of Global Positioning System (GPS) To Explore the Autonomous Robot. Mental.” 5(5.8): 5-4.
22. [22] Malhotra, R., Vanshika, & Neha, Construction and design of a device for obstacle detection, International Journal of Recent Technology and Engineering 2019; 8 (4):2312-2315.
23. [23] Malhotra R., & Taneja G. Stochastic analysis of a two-unit cold standby system wherein both the units may become operative depending upon the demand. Journal of Quality and Reliability Engineering 2014; Article ID 896379: 13 pages.
24. [24] Malhotra R., & Taneja G. (2015), Comparative study between a single unit system and a two-unit cold standby system with varying demand. Springer plus, 2015: 4:705
25. [25] Malhotra R., & Taneja G. (2013),“Reliability modelling of a cable manufacturing plant with variation in demand.” International Journal of Research In Mechanical engineering & technology 3(2):162-165
26. [26] Malhotra R., & Taneja G. (2013), "Cost-benefit analysis of a single unit system with scheduled maintenance and variation in demand," Journal of Mathematics and Statistics9 (3): 155-160, 2013
27. [27] Bourhis, G., Moumen, K., Pino, P., Rohmer, S., & Pruski, A., Assisted navigation for a powered wheelchair. syst. eng. in the service of humans. In Proc. IEEE Int. Conf. Syst., Man and Cybern., pages 553–558, Le Touquet, France, Oct. 1993.
28. [28] Connell, J. & Viola, P., Cooperative control of a semi-autonomous mobile robot. In Proc. IEEE Int. Conf. Robot. and auto., pages 1118–1121, Cincinnati, OH., May 1990.
29. [29] Simpson, R., Smart wheelchairs: A literature review. J. Rehabil. Research and Develop., 42(4):423–436, Aug. 2005.
30. [30] Faria, B. M., Reis, L.P., & Lau N., A survey on intelligent wheelchair prototypes and simulators. New Perspectives in Inform. Syst. And Technol., 1:545–557, 2014.
31. [31] Tyagi, V., Gupta, N.K., & Tyagi, P.K., Smart wheelchair using fuzzy inference system. In IEEE Global Humanitarian Technol. Conf.: South Asia Satellite, pages 175–180, Trivandrum, Aug. 2013.
32. [32] Reddy, K. B. P., & Kumar, K. K. (2016). A Smart Wheelchair System with Social Media update. Indian Journal of Science and Technology, 9(30):1-5.