Machine learning-based finger gesture classification using hybrid data

Abstract

Advances in biomedical engineering and robotic technology are increasing the importance of human-machine interaction through the detection of hand-finger movements used by traditional methods in applications ranging from prosthetics to rehabilitation. In this study, it is aimed to increase the accuracy performance by creating and filtering the necessary attribute dataset for supervised machine learning algorithms to create high-accuracy training and test performance by ensuring that camera images and sEMG sensor data are recorded simultaneously. Literature reviews of similar hybrid studies report accuracy rates ranging from 80.22% to 99.58%. In this study, after data preprocessing and feature extraction, the performance of Support Vector Machines (SVM), k-nearest Neighborhood (KNN) and Artificial Neural Networks (ANN) was evaluated using Matlab’s classification tutorial application, and test validation results of 92.70%, 92.26% and 90.41% were obtained, respectively. When the developed filtering process is applied to the dataset created during data preprocessing, 99.11%, 98.55% and 97.87% verification performance values are obtained, respectively. This article presents an integrated approach that combines sEMG sensors and camera image data, leveraging the strengths of both methods. The proposed hybrid method improves the accuracy of machine learning algorithms to classify hand-finger movements, providing a robust solution for applications that require precise motion detection and control. This research introduces a new approach in human-machine interaction technologies to the literature.

Keywords

• sEMG
• signal processing
• image processing
• machine learning

Project Number

KBUBAP-22-DR-127

Hibrit veri kullanarak makine öğrenme temelli parmak hareketi sınıflandırma

Abstract

Biyomedikal mühendisliği ve robotik teknolojideki gelişmeler, protezden rehabilitasyona kadar çeşitli uygulamalarda geleneksel yöntemlerle kullanılan el-parmak hareketlerinin algılanması yoluyla insan-makine etkileşiminin önemini artırmaktadır. Bu çalışmada, kamera görüntüleri ve sEMG sensör verilerinin eş zamanlı olarak kaydedilmesini sağlanarak denetimli makine öğrenme algoritmalarının yüksek doğrulukta eğitim ve test performansı oluşturabilmesi için gerekli öznitelik veri setinin oluşturulabilmesi ve filtrelenerek doğruluk performansının arttırılması amaçlanmaktadır. Benzer hibrit çalışmaların literatür taramaları, %80,22 ile %99,58 arasında değişen doğruluk oranları bildirmektedir. Bu çalışmada, veri önişleme ve öznitelik çıkarımından sonra, Destek Vektör Makineleri (SVM), k-en yakın Komşuluk (KNN) ve Yapay Sinir Ağlarının (ANN) performansı Matlab’ın sınıflandırma öğreticisi uygulaması kullanılarak değerlendirilmiş ve sırasıyla %92,70, %92,26 ve %90,41 test doğrulama sonuçları elde edilmiştir. Veri ön işleme sırasında oluşturulan verisetine, geliştirilen filtreleme işlemi uygulandığında sırasıyla %99,11, %98,55 ve %97,87 doğrulama performans değerleri elde edilmektedir. Bu makale, her iki yöntemin de güçlü yönlerinden yararlanarak sEMG sensörlerini ve kamera görüntü verilerini birleştiren bütünleşik bir yaklaşım sunmaktadır. Önerilen hibrit yöntem, el-parmak hareketlerini sınıflandırmak için makine öğrenimi algoritmalarının doğruluğunu artırarak, hassas hareket algılama ve kontrolü gerektiren uygulamalar için sağlam bir çözüm sunmaktadır. Bu araştırma, insan-makine etkileşimi teknolojilerinde yeni bir yaklaşımı literatüre kazandırmaktadır.

Keywords

• sEMG
• sinyal işleme
• görüntü işleme
• makine öğrenmesi.

Supporting Institution

Karabük Üniversitesi

Project Number

KBUBAP-22-DR-127

Ethical Statement

Veri elde etme işlemi, T.C. Karabük Üniversitesi Rektörlüğü Girişimsel Olmayan Klinik Araştırmalar Etik Kurulu’nun 2022/846 referans numaralı onayı ile gerçekleştirilmektedir.

Thanks

Bu çalışma Karabük Üniversitesi Bilimsel Araştırma Projeleri Birimi tarafından KBUBAP-22-DR-127 proje numarası ile desteklenmektedir.

References

1. [1] Kısa, D. H., Özdemir, M. A., Güren, O., Alaybeyoğlu Soy, A., El hareketi tahmini için EMG sinyalleri ve uyarlamalı sinirsel bulanık çıkarım sistemine (ANFIS) dayalı bir karar verme mekanizması, Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 38(3), 1417-1430, 2023.
2. [2] Liu, Y., Zhang, S., & Gowda, M., A practical system for 3-D hand pose tracking using EMG wearables with applications to prosthetics and user interfaces, IEEE Internet of Things Journal, 10(4), 3407-3427, 2022.
3. [3] Klotz, T., Gizzi, L., Yavuz, U. Ş., & Röhrle, O., Modelling the electrical activity of skeletal muscle tissue using a multi-domain approach, Biomechanics and modeling in mechanobiology, 19(1), 335-349, 2020.
4. [4] Biagetti, G., Crippa, P., Orcioni, S., & Turchetti, C., Homomorphic deconvolution for MUAP estimation from surface EMG signals, IEEE journal of biomedical and health informatics, 21(2), 328-338, 2016.
5. [5] İnternet: El tanıma kütüphanesi, Mediapipe, https://mediapipe-studio.webapps. google.com /demo/gesture_recognizer, 2023.
6. [6] Mersinkaya, I., & Kavsaoglu, A. R., A Data Acquisiton System with sEMG Signal and Camera Images for Finger Classification with Machine Learning Algorithms, Engineering, Technology & Applied Science Research, 14(2), 13554-13558, 2024.
7. [7] Raurale, S., McAllister, J., & del Rincon, J. M., Emg wrist-hand motion recognition system for real-time embedded platform, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1523-1527, 2019.
8. [8] Lobov, S., Krilova, N., Kastalskiy, I., Kazantsev, V., & Makarov, V. A., Latent factors limiting the performance of sEMG-interfaces, Sensors, 18(4), 1122, 2018.

9. [9] Zhang, F., Li, P., Hou, Z. G., Lu, Z., Chen, Y., Li, Q., & Tan, M., sEMG-based continuous estimation of joint angles of human legs by using BP neural network, Neurocomputing, 78(1), 139-148, 2012.
10. [10] Ju, Z., Ouyang, G., Wilamowska-Korsak, M., & Liu, H., Surface EMG based hand manipulation identification via nonlinear feature extraction and classificatio, IEEE Sensors Journal, 13(9), 3302-3311, 2013.
11. [11] Suzuki, M., Sasaki, M., Kamata, K., Nakayama, A., Shibamoto, I., & Tamada, Y., Swallowing pattern classification method using multichannel surface EMG signals of suprahyoid and infrahyoid muscles. Advanced Biomedical Engineering, 9, 10-20, 2020.
12. [12] Al-Timemy, A. H., Bugmann, G., Escudero, J., & Outram, N., Classification of finger movements for the dexterous hand prosthesis control with surface electromyography, IEEE journal of biomedical and health informatics, 17(3), 608-61, 2013.
13. [13] Faust, O. and Bairy, M. G., Nonlinear analysis of physiological signals: A review, Journal of Mechanics in Medicine and Biology, 12(4). ISSN 02195194, 2012.
14. [14] Vaiman, M., Standardization of surface electromyography utilized to evaluate patients with dysphagia, Head and Face Medicine, 3(1). ISSN 1716160X, 2007.
15. [15] Seçkin, A. Ç., Labeling System with EEG, EMG, and IMU for Visual Training of Autonomous Vehicles. Bilişim Teknolojileri Dergisi, 12(4), 299-305, 2019
16. [16] Molchanov, P., Gupta, S., Kim, K., and Pulli, K., Multi-sensor system for driver’s hand-gesture recognition, 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, ISBN 9781479960262, 2015.
17. [17] Sagayam, K. M. and Hemanth, D. J., Hand posture and gesture recognition techniques for virtual reality applications: a survey, Virtual Reality, 21(2):91–107. ISSN 14349957, 2017.
18. [18] Guo, W., Sheng, X., Liu, H., and Zhu, X., Toward an Enhanced Human-Machine Interface for Upper-Limb Prosthesis Control with Combined EMG and NIRS Signals, IEEE Transactions on Human-Machine Systems, 47(4):564–575. ISSN 21682291, 2017.
19. [19] Liang, H., Yuan, J., Thalmann, D., and Nadia, M. T., AR in hand: Egocentric palm pose tracking and gesture recognition for augmented reality applications, Proceedings of the 2015 ACM Multimedia Conference, Association for Computing Machinery, Inc. ISBN 9781450334594, 743–744, 2015.
20. [20] Mi, J., Sun, Y., Wang, Y., Deng, Z., Li, L., Zhang, J., and Xie, G., Gesture recognition based teleoperation framework of robotic fish, IEEE International Conference on Robotics and Biomimetics, ROBIO 2016, Institute of Electrical and Electronics Engineers Inc. ISBN 9781509043644, 137–142, 2016.
21. [21] Zhang, X., Chen, X., Li, Y., Lantz, V., Wang, K., and Yang, J., A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors, IEEE Transactions on Systems, Man, and Cybernetics- Part A: Systems and Humans, 41(6):1064–1076, 2011.
22. [22] Englehart, K., & Hudgins, B., “A robust, real-time control scheme for multifunction myoelectric control”, IEEE Transactions on Biomedical Engineering, 50(7), 848–854 (2003).
23. [23] Smith, L. H., Hargrove, L. J., Lock, B. A., & Kuiken, T. A., Determining the Optimal Window Length for Pattern Recognition-Based Myoelectric Control: Balancing the Competing Effects of Classification Error and Controller Delay, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 19(2), 186–192, 2011.
24. [24] Çalışkan, A., Emg Sinyalleri İçin Hibrid Öznitelik Çıkarma Yöntemi Geliştirilmesi, Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi, 8(2), 652-664, 2019.
25. [25] Xing, K., Yang, P., Huang, J., Wang, Y., & Zhu, Q., A real-time EMG pattern recognition method for virtual myoelectric hand control, Neurocomputing, 136, 345-355, 2014.
26. [26] Qi, J., Jiang, G., Li, G., Sun, Y., & Tao, B., Surface EMG hand gesture recognition system based on PCA and GRNN, Neural Computing and Applications, 32, 6343-6351, 2020.
27. [27] Akyel, C., Polat, H., & Ciylan, B., Öznitelik Seçimi ile Desteklenen Makine Öğrenmesine Dayalı Göğüs Kanserinin Erken Tespiti ve Teşhisi. Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım Ve Teknoloji, 12(2), 675-690, 2024.
28. [28] AbdelMaseeh, M., Chen, T. W., & Stashuk, D. W., Extraction and classification of multichannel electromyographic activation trajectories for hand movement recognition, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 24(6), 662-673, 2015.
29. [29] Yamanoi, Y., Ogiri, Y., & Kato, R., EMG-based posture classification using a convolutional neural network for a myoelectric hand, Biomedical Signal Processing and Control, 55, 101574, 2020.
30. [30] Li, W., Shi, P., & Yu, H., Gesture recognition using surface electromyography and deep learning for prostheses hand: State-of-the-art, challenges, and future, Frontiers in neuroscience, 15, 621885, 2021.
31. [31] Uğuz, S., Makine öğrenmesi teorik yönleri ve Python uygulamaları ile bir yapay zekâ ekolü, Nobel Yayıncılık, Ankara, 74, 201.
32. [32] Guo, S., Pang, M., Gao, B., Hirata, H., & Ishihara, H., Comparison of sEMG-based feature extraction and motion classification methods for upper-limb movement, Sensors, 15(4), 9022-9038, 2015.
33. [33] Narayan, Y., Mathew, L., & Chatterji, S., SEMG signal classification with novel feature extraction using different machine learning approaches, Journal of Intelligent & Fuzzy Systems, 35(5), 5099-5109, 2018.
34. [34] Lima, A. A. M., Araujo, R. M., dos Santos, F. A. G., Yoshizumi, V. H., de Barros, F. K., Spatti, D. H., ... & Dajer, M. E., Classification of hand movements from emg signals using optimized mlp, International Joint Conference on Neural Networks (IJCNN), 1-7, 2018.
35. [35] Fawcett, T., An introduction to ROC analysis, Pattern recognition letters, 27(8), 861-874, 2006.
36. [36] Schmidhuber, J., Annotated history of modern ai and deep learning, arXiv preprint arXiv, 2212.11279, 2022.
37. [37] Li, J., Zhong, J., & Wang, N., A multimodal human-robot sign language interaction framework applied in social robots, Frontiers in Neuroscience, 17, 1168888, 2023.
38. [38] Zhou, H., Zhang, Q., Zhang, M., Shahnewaz, S., Wei, S., Ruan, J., ... & Zhang, L., Toward hand pattern recognition in assistive and rehabilitation robotics using EMG and kinematics, Frontiers in Neurorobotics, 15, 659876, 2021.
39. [39] Jiang, X., Xiao, Z. G., & Menon, C., Virtual grasps recognition using fusion of Leap Motion and force myography, Virtual Reality, 22(4), 297-308, 2018.
40. [40] Chen, J., Liu, C., Cui, R., & Yang, C., Hand tracking accuracy enhancement by data fusion using leap motion and myo armband, IEEE International Conference on Unmanned Systems and Artificial Intelligence (ICUSAI), 256-261, 2019.
41. [41] Quivira, F., Koike-Akino, T., Wang, Y., & Erdogmus, D., Translating sEMG signals to continuous hand poses using recurrent neural networks, IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), 166-169, 2018.
42. [42] Su, H., Ovur, S. E., Zhou, X., Qi, W., Ferrigno, G., & De Momi, E., Depth vision guided hand gesture recognition using electromyographic signals, Advanced Robotics, 34(15), 985-997, 2020.
43. [43] Arafa, M. A., Awad, M. I., & Tolbah, F. A., Proportional myoelectric prosthetic hand control using multi-regression model estimator with pattern classifier selector, IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), 1-5, 2017.
44. [44] Amprimo, G., Ferraris, C., Masi, G., Pettiti, G., & Priano, L., Gmh-d: Combining google mediapipe and rgb-depth cameras for hand motor skills remote assessment, IEEE International Conference on Digital Health (ICDH),132-141, 2022.