Comparative analysis on real-time hand gesture and sign language recognition using convexity defects and YOLOv3

Abstract

The purpose of this paper is to help people with auditory and speech disabilities to com-municate with others and for controlling computers and machines. This paper proposes two different methods for identifying six distinctive hand gestures and sign language for diver-gent environmental conditions. The first method is based on the hand feature extraction i.e., convexity defects. For that, initially, the hand region is detected by HSV skin color conver-sion process. Contour and convex hull of hand are extracted from the hand region. Finally, convexity defects are determined to identify the hand gestures. The second method is deep learning based YOLOv3 model that uses DARKNET-53 convolutional neural network (CNN) as its backbone. The model is trained on a large annotated dataset. Experimental results reveal that the deep-leaning method outperforms the hand feature approach and obtain 98.92% and 95.57% accuracy for deep learning and hand feature-based model respectively.

Keywords

• Hand Gesture Recognition
• Convexity Defect
• Feature Extraction
• Deep Learning
• YOLO

References

1. REFERENCES
2. [1] Braffort A, Gherbi R, Gibet S, Richardson J, Teil D, editors. Gesture-Based Communication in Human-Computer Interaction: International Gesture Workshop, GW'99; 1999 Mar 17-19; Gif-sur-Yvette, France. Berlin: Springer; 2003. [CrossRef]
3. [2] Gupta S, Bagga S, Sharma DK. Hand Gesture Recognition for Human Computer Interaction and Its Applications in Virtual Reality. In: Advanced Computational Intelligence Techniques for Virtual Reality in Healthcare. Cham: Springer; 2020. p. 85105. [CrossRef]
4. [3] Jing Y, Bian Y, Hu Z, Wang L, Xie XQS. Deep learning for drug design: an artificial intelligence paradigm for drug discovery in the big data era. AAPS J 2018;20:58. [CrossRef]
5. [4] Redmon J, Farhadi A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767. 2018.
6. [5] Zhuang F, Qi Z, Duan K, Xi D, Zhu Y, Zhu H, et al. A comprehensive survey on transfer learning. Proc IEEE 2020;109:4376. [CrossRef]
7. [6] Singh S, Ahuja U, Kumar M, Kumar K, Sachdeva M. Face mask detection using YOLOv3 and faster R-CNN models: COVID-19 environment. Multimed Tools Appl 2021;80:1975319768. [CrossRef]
8. [7] Shi M, Ouyang P, Yin S, Liu L, Wei S. A fast and power-efficient hardware architecture for non-maximum suppression. IEEE Trans Circuits Syst II Express Briefs 2019;66:18701874. [CrossRef]

9. [8] Ren Z, Yuan J, Meng J, Zhang Z. Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 2013;15:11101120. [CrossRef]
10. [9] Li X. Gesture recognition based on fuzzy C-Means clustering algorithm [thesis]. Knoxville (TN): University of Tennessee; 2003.
11. [10] Sharma S, Jain S. A static hand gesture and face recognition system for blind people. In: 2019 6th International Conference on Signal Processing and Integrated Networks (SPIN); 2019 Mar; Noida, India. IEEE; 2019. p. 534539. [CrossRef]
12. [11] Yun L, Lifeng Z, Shujun Z. A hand gesture recognition method based on multi-feature fusion and template matching. Procedia Eng 2012;29:16781684. [CrossRef]
13. [12] Pradhan A, Ghose MK, Pradhan M, Qazi S, Moors T, EL-Arab IME, et al. A hand gesture recognition using feature extraction. Int J Curr Eng Technol 2012;2:323327.
14. [13] Prakash RM, Deepa T, Gunasundari T, Kasthuri N. Gesture recognition and finger tip detection for human computer interaction. In: 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS); 2017 Mar; Coimbatore, India. IEEE; 2017. p. 14. [CrossRef]
15. [14] Xu Y, Park DW, Pok G. Hand gesture recognition based on convex defect detection. Int J Appl Eng Res 2017;12:70757079.
16. [15] Hussain S, Saxena R, Han X, Khan JA, Shin H. Hand gesture recognition using deep learning. In: 2017 International SoC Design Conference (ISOCC); 2017 Nov; Seoul, South Korea. IEEE; 2017. p. 4849. [CrossRef]
17. [16] Zhang Q, Zhang Y, Liu Z. A dynamic hand gesture recognition algorithm based on CSI and YOLOv3. J Phys Conf Ser 2019;1267:012055. [CrossRef]
18. [17] Jiang D, Li G, Sun Y, Kong J, Tao B. Gesture recognition based on skeletonization algorithm and CNN with ASL database. Multimed Tools Appl 2019;78:2995329970. [CrossRef]
19. [18] Benjdira B, Khursheed T, Koubaa A, Ammar A, Ouni K. Car detection using unmanned aerial vehicles: Comparison between faster r-cnn and yolov3. In: 2019 1st International Conference on Unmanned Vehicle Systems-Oman (UVS); 2019; Muscat, Oman. IEEE; 2019. p. 16. [CrossRef]
20. [19] Tanmaie U, Rao CS. Hand posture detection and classification using you only look once (YOLO v2) object detector. JAC 2020;13:101–106.
21. [20] Ganapathyraju S. Hand gesture recognition using convexity hull defects to control an industrial robot. In: 2013 3rd International Conference on Instrumentation Control and Automation (ICA); 2013; Bali, Indonesia. IEEE; 2013. p. 6367. [CrossRef]
22. [21] Sayed U, Mofaddel MA, Bakheet S, El-Zohry Z. Human hand gesture recognition. Inf Sci Lett 2018;7:41–44. [CrossRef]
23. [22] Yang F, Chen H, Li J, Li F, Wang L, Yan X. Single shot multibox detector with kalman filter for online pedestrian detection in video. IEEE Access 2019;7:1547815488. [CrossRef]
24. [23] Sign Language for Numbers. Kaggle. 2012. Available from: https://kaggle.com/muhammadkhalid/sign-language-for-numbers Accessed on Feb 06, 2024.
25. [24] Liu J, Wang X. Tomato diseases and pests detection based on improved Yolo V3 convolutional neural network. Front Plant Sci 2020;11:898. [CrossRef]
26. [25] Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC. SSD: Single shot multibox detector. In: European Conference on Computer Vision; 2016; Amsterdam, Netherlands. Cham: Springer; 2016. p. 2137. [CrossRef]
27. [26] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016; Las Vegas, NV, USA. 2016. p. 779788. [CrossRef]
28. [27] Redmon J, Farhadi A. YOLO9000: better, faster, stronger. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017; Honolulu, HI, USA. 2017. p. 72637271. [CrossRef]
29. [28] Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S. Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017; Honolulu, HI, USA. 2017. p. 21172125. [CrossRef]