Comparison of CNN-based methods for yoga pose classification

Abstract

Yoga is an exercise developed in ancient India. People perform yoga in order to have mental, physical, and spiritual benefits. While yoga helps build strength in the mind and body, incorrect postures might result in serious injuries. Therefore, yoga exercisers need either an expert or a platform to receive feedback on their performance. Since access to experts is not an option for everyone, a system to provide feedback on the yoga poses is required. To this end, commercial products such as smart yoga mats and smart pants are produced; Kinect cameras, sensors, and wearable devices are used. However, these solutions are either uncomfortable to wear or not affordable for everyone. Nonetheless, a system that employs computer vision techniques is a requirement. In this paper, we propose a deep-learning model for yoga pose classification, which is the first step of a quality assessment and personalized feedback system. We introduce a wavelet-based model that first takes wavelet transform of input images. The acquired subbands, i.e., approximation, horizontal, vertical, and diagonal coefficients of the wavelet transform are then fed into separate convolutional neural networks (CNN). The obtained probability results for each group are fused to predict the final yoga class. A publicly available dataset with 5 yoga poses is used. Since the number of images in the dataset is not enough for a deep learning model, we also perform data augmentation to increase the number of images. We compare our results to a CNN model and the three models that employ the subbands separately. Results obtained using the proposed model outperforms the accuracy output achieved with the compared models. While the regular CNN model has 61% and 50% accuracy for the training and test data, the proposed model achieves 91% and 80%, respectively.

Keywords

• Image classification
• Wavelet transform
• Deep learning
• Yoga posture
• CNN

References

1. Chang, C. W., Da Nian, M., Chen, Y. F., Chi, C. H., & Tao, C. W. (2014, August). Design of a Kinect sensor based posture recognition system. In 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (pp. 856-859). IEEE. https://doi.org/10.1109/IIH-MSP.2014.216
2. Wang, J., Huang, Z., Zhang, W., Patil, A., Patil, K., Zhu, T., ... & Harris, T. B. (2016, December). Wearable sensor based human posture recognition. In 2016 IEEE International conference on big data (big data) (pp. 3432-3438). IEEE. https://doi.org/10.1109/BigData.2016.7841004
3. Gochoo, M., Tan, T. H., Huang, S. C., Batjargal, T., Hsieh, J. W., Alnajjar, F. S., & Chen, Y. F. (2019). Novel IoT-based privacy-preserving yoga posture recognition system using low-resolution infrared sensors and deep learning. IEEE Internet of Things Journal, 6(4), 7192-7200. https://doi.org/10.1109/JIOT.2019.2915095
4. Jain, S., Rustagi, A., Saurav, S., Saini, R., & Singh, S. (2021). Three-dimensional CNN-inspired deep learning architecture for Yoga pose recognition in the real-world environment. Neural Computing and Applications, 33, 6427-6441. https://doi.org/10.1007/s00521-020-05405-5
5. Gochoo, M., Tan, T. H., Alnajjar, F., Hsieh, J. W., & Chen, P. Y. (2020, October). Lownet: Privacy preserved ultra-low resolution posture image classification. In 2020 IEEE International Conference on Image Processing (ICIP) (pp. 663-667). IEEE. https://doi.org/10.1109/ICIP40778.2020.9190922
6. Anand Thoutam, V., Srivastava, A., Badal, T., Kumar Mishra, V., Sinha, G. R., Sakalle, A., ... & Raj, M. (2022). Yoga pose estimation and feedback generation using deep learning. Computational Intelligence and Neuroscience, 4311350. https://doi.org/10.1155/2022/4311350
7. Kumar, D., & Sinha, A. (2020). Yoga pose detection and classification using deep learning. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 6 (6), 160-184. https://doi.org/10.32628/CSEIT206623
8. Dittakavi, B., Bavikadi, D., Desai, S. V., Chakraborty, S., Reddy, N., Balasubramanian, V. N., ... & Sharma, A. (2022). Pose tutor: an explainable system for pose correction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3540-3549).

9. Wu, Y., Lin, Q., Yang, M., Liu, J., Tian, J., Kapil, D., & Vanderbloemen, L. (2021, December). A computer vision-based yoga pose grading approach using contrastive skeleton feature representations. Healthcare, 10(1), 36. https://doi.org/10.3390/healthcare10010036
10. Garg, S., Saxena, A., & Gupta, R. (2022). Yoga pose classification: a CNN and MediaPipe inspired deep learning approach for real-world application. Journal of Ambient Intelligence and Humanized Computing, 1-12. https://doi.org/10.1007/s12652-022-03910-0
11. Swain, D., Satapathy, S., Acharya, B., Shukla, M., Gerogiannis, V. C., Kanavos, A., & Giakovis, D. (2022). Deep Learning Models for Yoga Pose Monitoring. Algorithms, 15(11), 403. https://doi.org/10.3390/a15110403
12. Rishan, F., De Silva, B., Alawathugoda, S., Nijabdeen, S., Rupasinghe, L., & Liyanapathirana, C. (2020, December). Infinity yoga tutor: Yoga posture detection and correction system. In 2020 5th International conference on information technology research (ICITR) (pp. 1-6). IEEE. https://doi.org/10.1109/ICITR51448.2020.9310832
13. Yadav, S. K., Singh, A., Gupta, A., & Raheja, J. L. (2019). Real-time Yoga recognition using deep learning. Neural Computing and Applications, 31, 9349-9361. https://doi.org/10.1007/s00521-019-04232-7
14. Long, C., Jo, E., & Nam, Y. (2022). Development of a yoga posture coaching system using an interactive display based on transfer learning. The Journal of Supercomputing, 78, 5269–5284. https://doi.org/10.1007/s11227-021-04076-w
15. Chasmai, M., Das, N., Bhardwaj, A., & Garg, R. (2022). A View Independent Classification Framework for Yoga Postures. SN computer science, 3(6), 476. https://doi.org/10.1007/s42979-022-01376-7
16. Verma, M., Kumawat, S., Nakashima, Y., & Raman, S. (2020). Yoga-82: a new dataset for fine-grained classification of human poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 1038-1039).
17. Yadav, S. K., Singh, G., Verma, M., Tiwari, K., Pandey, H. M., Akbar, S. A., & Corcoran, P. (2022). YogaTube: a video benchmark for Yoga action recognition. In 2022 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). IEEE. https://doi.org/10.1109/IJCNN55064.2022.9892122
18. Li, J., Hu, H., Li, J., & Zhao, X. (2022). 3D-Yoga: A 3D Yoga Dataset for Visual-Based Hierarchical Sports Action Analysis. In Proceedings of the Asian Conference on Computer Vision (pp. 434-450).
19. Fu, Y., Lei, Y., Wang, T., Curran, W. J., Liu, T., & Yang, X. (2020). Deep learning in medical image registration: a review. Physics in Medicine & Biology, 65(20), 20TR01. https://doi.org/10.1088/1361-6560/ab843e
20. Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J. H., & Liao, Q. (2019). Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia, 21(12), 3106-3121. https://doi.org/10.1109/TMM.2019.2919431
21. Lu, D., & Weng, Q. (2007). A survey of image classification methods and techniques for improving classification performance. International journal of Remote sensing, 28(5), 823-870. https://doi.org/10.1080/01431160600746456
22. Gülgün, O. D., & Hamza, E. R. O. L. (2020). Classification performance comparisons of deep learning models in pneumonia diagnosis using chest x-ray images. Turkish Journal of Engineering, 4(3), 129-141. https://doi.org/10.31127/tuje.652358
23. Zeybek, M. (2021). Classification of UAV point clouds by random forest machine learning algorithm. Turkish Journal of Engineering, 5(2), 48-57. https://doi.org/10.31127/tuje.669566
24. Öztürk, A., Allahverdi, N., & Saday, F. (2022). Application of artificial intelligence methods for bovine gender prediction. Turkish Journal of Engineering, 6(1), 54-62. https://doi.org/10.31127/tuje.807019
25. Rawat, W., & Wang, Z. (2017). Deep convolutional neural networks for image classification: A comprehensive review. Neural computation, 29(9), 2352-2449. https://doi.org/10.1162/neco_a_00990
26. Su, D., Zhang, H., Chen, H., Yi, J., Chen, P. Y., & Gao, Y. (2018). Is Robustness the Cost of Accuracy?--A Comprehensive Study on the Robustness of 18 Deep Image Classification Models. In Proceedings of the European conference on computer vision (ECCV), 631-648.
27. Aydin, V. A., & Foroosh, H. (2017, September). Motion compensation using critically sampled dwt subbands for low-bitrate video coding. In 2017 IEEE International Conference on Image Processing (ICIP), 21-25). https://doi.org/10.1109/ICIP.2017.8296235
28. Aydin, V. A., & Foroosh, H. (2017). In-band sub-pixel registration of wavelet-encoded images from sparse coefficients. Signal, Image and Video Processing, 11, 1527-1535. https://doi.org/10.1007/s11760-017-1116-5
29. Aydin, V. A., & Foroosh, H. (2018). A linear well-posed solution to recover high-frequency information for super resolution image reconstruction. Multidimensional Systems and Signal Processing, 29, 1309-1330. https://doi.org/10.1007/s11045-017-0499-3
30. Huang, H., He, R., Sun, Z., & Tan, T. (2017). Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In Proceedings of the IEEE international conference on computer vision (pp. 1689-1697).
31. Le Moigne, J., Campbell, W. J., & Cromp, R. F. (2002). An automated parallel image registration technique based on the correlation of wavelet features. IEEE Transactions on Geoscience and Remote Sensing, 40(8), 1849-1864. https://doi.org/10.1109/TGRS.2002.802501
32. Postalcıoğlu, S., Erkan, K., & Bolat, E. D. (2005). Comparison of wavenet and neuralnet for system modeling. In Knowledge-Based Intelligent Information and Engineering Systems: 9th International Conference, KES 2005, Melbourne, Australia, September 14-16, 2005, Proceedings, Part II 9 (pp. 100-107). Springer Berlin Heidelberg. https://doi.org/10.1007/11552451_14
33. Postalcioglu, S., & Becerikli, Y. (2007). Wavelet networks for nonlinear system modeling. Neural Computing and Applications, 16, 433-441. https://doi.org/10.1007/s00521-006-0069-3
34. Robinson, M. D., Toth, C. A., Lo, J. Y., & Farsiu, S. (2010). Efficient Fourier-wavelet super-resolution. IEEE Transactions on Image Processing, 19(10), 2669-2681. https://doi.org/10.1109/TIP.2010.2050107
35. Li, Q., Shen, L., Guo, S., & Lai, Z. (2020). Wavelet integrated CNNs for noise-robust image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7245-7254.
36. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248-255. https://doi.org/10.1109/CVPR.2009.5206848
37. Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261. https://doi.org/10.48550/arXiv.1903.12261
38. Mallick, P. K., Ryu, S. H., Satapathy, S. K., Mishra, S., Nguyen, G. N., & Tiwari, P. (2019). Brain MRI image classification for cancer detection using deep wavelet autoencoder-based deep neural network. IEEE Access, 7, 46278-46287. https://doi.org/10.1109/ACCESS.2019.2902252
39. Khatami, A., Nazari, A., Beheshti, A., Nguyen, T. T., Nahavandi, S., & Zieba, J. (2020, July). Convolutional neural network for medical image classification using wavelet features. In 2020 International Joint Conference on Neural Networks (IJCNN), 1-8. https://doi.org/10.1109/IJCNN48605.2020.9206791
40. Said, S., Jemai, O., Hassairi, S., Ejbali, R., Zaied, M., & Amar, C. B. (2016, October). Deep wavelet network for image classification. In 2016 IEEE International conference on systems, man, and cybernetics (SMC), 000922-000927. https://doi.org/10.1109/SMC.2016.7844359
41. Fujieda, S., Takayama, K., & Hachisuka, T. (2017). Wavelet convolutional neural networks for texture classification. arXiv preprint arXiv:1707.07394. https://doi.org/10.48550/arXiv.1707.07394
42. Postalcioglu, S. (2022). Design of Automatic Tool for Diagnosis of Pneumonia Using Boosting Techniques. Brazilian Archives of Biology and Technology, 65, e22210322.
43. Serte, S., & Demirel, H. (2019). Gabor wavelet-based deep learning for skin lesion classification. Computers in biology and medicine, 113, 103423. https://doi.org/10.1016/j.compbiomed.2019.103423
44. Serte, S., & Demirel, H. (2020). Wavelet‐based deep learning for skin lesion classification. IET Image Processing, 14(4), 720-726. https://doi.org/10.1049/iet-ipr.2019.0553
45. Aydin, V. A. (2022). CNN Tabanlı Yoga Pozu Sınıflandırmasında Aktivasyon Fonksiyonu Karşılaştırması. In Proceedings of IES’22 International Engineering Symposium, Engineering Applications in Industry, 117-122.
46. https://www.kaggle.com/datasets/niharika41298/yoga-poses-dataset
47. Chen, C. H., & Ramanan, D. (2017). 3d human pose estimation= 2d pose estimation+ matching. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7035-7043).