Comparison of CNN-based methods for yoga pose classification

Yıl 2024, Cilt: 8 Sayı: 1, 65 - 75, 19.01.2024

Vildan Atalay Aydın

https://doi.org/10.31127/tuje.1275826

Öz

Yoga is an exercise developed in ancient India. People perform yoga in order to have mental, physical, and spiritual benefits. While yoga helps build strength in the mind and body, incorrect postures might result in serious injuries. Therefore, yoga exercisers need either an expert or a platform to receive feedback on their performance. Since access to experts is not an option for everyone, a system to provide feedback on the yoga poses is required. To this end, commercial products such as smart yoga mats and smart pants are produced; Kinect cameras, sensors, and wearable devices are used. However, these solutions are either uncomfortable to wear or not affordable for everyone. Nonetheless, a system that employs computer vision techniques is a requirement. In this paper, we propose a deep-learning model for yoga pose classification, which is the first step of a quality assessment and personalized feedback system. We introduce a wavelet-based model that first takes wavelet transform of input images. The acquired subbands, i.e., approximation, horizontal, vertical, and diagonal coefficients of the wavelet transform are then fed into separate convolutional neural networks (CNN). The obtained probability results for each group are fused to predict the final yoga class. A publicly available dataset with 5 yoga poses is used. Since the number of images in the dataset is not enough for a deep learning model, we also perform data augmentation to increase the number of images. We compare our results to a CNN model and the three models that employ the subbands separately. Results obtained using the proposed model outperforms the accuracy output achieved with the compared models. While the regular CNN model has 61% and 50% accuracy for the training and test data, the proposed model achieves 91% and 80%, respectively.

Anahtar Kelimeler

Image classification, Wavelet transform, Deep learning, Yoga posture, CNN

Kaynakça

• Chang, C. W., Da Nian, M., Chen, Y. F., Chi, C. H., & Tao, C. W. (2014, August). Design of a Kinect sensor based posture recognition system. In 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (pp. 856-859). IEEE. https://doi.org/10.1109/IIH-MSP.2014.216
• Wang, J., Huang, Z., Zhang, W., Patil, A., Patil, K., Zhu, T., ... & Harris, T. B. (2016, December). Wearable sensor based human posture recognition. In 2016 IEEE International conference on big data (big data) (pp. 3432-3438). IEEE. https://doi.org/10.1109/BigData.2016.7841004
• Gochoo, M., Tan, T. H., Huang, S. C., Batjargal, T., Hsieh, J. W., Alnajjar, F. S., & Chen, Y. F. (2019). Novel IoT-based privacy-preserving yoga posture recognition system using low-resolution infrared sensors and deep learning. IEEE Internet of Things Journal, 6(4), 7192-7200. https://doi.org/10.1109/JIOT.2019.2915095
• Jain, S., Rustagi, A., Saurav, S., Saini, R., & Singh, S. (2021). Three-dimensional CNN-inspired deep learning architecture for Yoga pose recognition in the real-world environment. Neural Computing and Applications, 33, 6427-6441. https://doi.org/10.1007/s00521-020-05405-5
• Gochoo, M., Tan, T. H., Alnajjar, F., Hsieh, J. W., & Chen, P. Y. (2020, October). Lownet: Privacy preserved ultra-low resolution posture image classification. In 2020 IEEE International Conference on Image Processing (ICIP) (pp. 663-667). IEEE. https://doi.org/10.1109/ICIP40778.2020.9190922
• Anand Thoutam, V., Srivastava, A., Badal, T., Kumar Mishra, V., Sinha, G. R., Sakalle, A., ... & Raj, M. (2022). Yoga pose estimation and feedback generation using deep learning. Computational Intelligence and Neuroscience, 4311350. https://doi.org/10.1155/2022/4311350
• Kumar, D., & Sinha, A. (2020). Yoga pose detection and classification using deep learning. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 6 (6), 160-184. https://doi.org/10.32628/CSEIT206623
• Dittakavi, B., Bavikadi, D., Desai, S. V., Chakraborty, S., Reddy, N., Balasubramanian, V. N., ... & Sharma, A. (2022). Pose tutor: an explainable system for pose correction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3540-3549).
• Wu, Y., Lin, Q., Yang, M., Liu, J., Tian, J., Kapil, D., & Vanderbloemen, L. (2021, December). A computer vision-based yoga pose grading approach using contrastive skeleton feature representations. Healthcare, 10(1), 36. https://doi.org/10.3390/healthcare10010036
• Garg, S., Saxena, A., & Gupta, R. (2022). Yoga pose classification: a CNN and MediaPipe inspired deep learning approach for real-world application. Journal of Ambient Intelligence and Humanized Computing, 1-12. https://doi.org/10.1007/s12652-022-03910-0
• Swain, D., Satapathy, S., Acharya, B., Shukla, M., Gerogiannis, V. C., Kanavos, A., & Giakovis, D. (2022). Deep Learning Models for Yoga Pose Monitoring. Algorithms, 15(11), 403. https://doi.org/10.3390/a15110403
• Rishan, F., De Silva, B., Alawathugoda, S., Nijabdeen, S., Rupasinghe, L., & Liyanapathirana, C. (2020, December). Infinity yoga tutor: Yoga posture detection and correction system. In 2020 5th International conference on information technology research (ICITR) (pp. 1-6). IEEE. https://doi.org/10.1109/ICITR51448.2020.9310832
• Yadav, S. K., Singh, A., Gupta, A., & Raheja, J. L. (2019). Real-time Yoga recognition using deep learning. Neural Computing and Applications, 31, 9349-9361. https://doi.org/10.1007/s00521-019-04232-7
• Long, C., Jo, E., & Nam, Y. (2022). Development of a yoga posture coaching system using an interactive display based on transfer learning. The Journal of Supercomputing, 78, 5269–5284. https://doi.org/10.1007/s11227-021-04076-w
• Chasmai, M., Das, N., Bhardwaj, A., & Garg, R. (2022). A View Independent Classification Framework for Yoga Postures. SN computer science, 3(6), 476. https://doi.org/10.1007/s42979-022-01376-7
• Verma, M., Kumawat, S., Nakashima, Y., & Raman, S. (2020). Yoga-82: a new dataset for fine-grained classification of human poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 1038-1039).
• Yadav, S. K., Singh, G., Verma, M., Tiwari, K., Pandey, H. M., Akbar, S. A., & Corcoran, P. (2022). YogaTube: a video benchmark for Yoga action recognition. In 2022 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). IEEE. https://doi.org/10.1109/IJCNN55064.2022.9892122
• Li, J., Hu, H., Li, J., & Zhao, X. (2022). 3D-Yoga: A 3D Yoga Dataset for Visual-Based Hierarchical Sports Action Analysis. In Proceedings of the Asian Conference on Computer Vision (pp. 434-450).
• Fu, Y., Lei, Y., Wang, T., Curran, W. J., Liu, T., & Yang, X. (2020). Deep learning in medical image registration: a review. Physics in Medicine & Biology, 65(20), 20TR01. https://doi.org/10.1088/1361-6560/ab843e
• Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J. H., & Liao, Q. (2019). Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia, 21(12), 3106-3121. https://doi.org/10.1109/TMM.2019.2919431
• Lu, D., & Weng, Q. (2007). A survey of image classification methods and techniques for improving classification performance. International journal of Remote sensing, 28(5), 823-870. https://doi.org/10.1080/01431160600746456
• Gülgün, O. D., & Hamza, E. R. O. L. (2020). Classification performance comparisons of deep learning models in pneumonia diagnosis using chest x-ray images. Turkish Journal of Engineering, 4(3), 129-141. https://doi.org/10.31127/tuje.652358
• Zeybek, M. (2021). Classification of UAV point clouds by random forest machine learning algorithm. Turkish Journal of Engineering, 5(2), 48-57. https://doi.org/10.31127/tuje.669566
• Öztürk, A., Allahverdi, N., & Saday, F. (2022). Application of artificial intelligence methods for bovine gender prediction. Turkish Journal of Engineering, 6(1), 54-62. https://doi.org/10.31127/tuje.807019
• Rawat, W., & Wang, Z. (2017). Deep convolutional neural networks for image classification: A comprehensive review. Neural computation, 29(9), 2352-2449. https://doi.org/10.1162/neco_a_00990
• Su, D., Zhang, H., Chen, H., Yi, J., Chen, P. Y., & Gao, Y. (2018). Is Robustness the Cost of Accuracy?--A Comprehensive Study on the Robustness of 18 Deep Image Classification Models. In Proceedings of the European conference on computer vision (ECCV), 631-648.
• Aydin, V. A., & Foroosh, H. (2017, September). Motion compensation using critically sampled dwt subbands for low-bitrate video coding. In 2017 IEEE International Conference on Image Processing (ICIP), 21-25). https://doi.org/10.1109/ICIP.2017.8296235
• Aydin, V. A., & Foroosh, H. (2017). In-band sub-pixel registration of wavelet-encoded images from sparse coefficients. Signal, Image and Video Processing, 11, 1527-1535. https://doi.org/10.1007/s11760-017-1116-5
• Aydin, V. A., & Foroosh, H. (2018). A linear well-posed solution to recover high-frequency information for super resolution image reconstruction. Multidimensional Systems and Signal Processing, 29, 1309-1330. https://doi.org/10.1007/s11045-017-0499-3
• Huang, H., He, R., Sun, Z., & Tan, T. (2017). Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In Proceedings of the IEEE international conference on computer vision (pp. 1689-1697).
• Le Moigne, J., Campbell, W. J., & Cromp, R. F. (2002). An automated parallel image registration technique based on the correlation of wavelet features. IEEE Transactions on Geoscience and Remote Sensing, 40(8), 1849-1864. https://doi.org/10.1109/TGRS.2002.802501
• Postalcıoğlu, S., Erkan, K., & Bolat, E. D. (2005). Comparison of wavenet and neuralnet for system modeling. In Knowledge-Based Intelligent Information and Engineering Systems: 9th International Conference, KES 2005, Melbourne, Australia, September 14-16, 2005, Proceedings, Part II 9 (pp. 100-107). Springer Berlin Heidelberg. https://doi.org/10.1007/11552451_14
• Postalcioglu, S., & Becerikli, Y. (2007). Wavelet networks for nonlinear system modeling. Neural Computing and Applications, 16, 433-441. https://doi.org/10.1007/s00521-006-0069-3
• Robinson, M. D., Toth, C. A., Lo, J. Y., & Farsiu, S. (2010). Efficient Fourier-wavelet super-resolution. IEEE Transactions on Image Processing, 19(10), 2669-2681. https://doi.org/10.1109/TIP.2010.2050107
• Li, Q., Shen, L., Guo, S., & Lai, Z. (2020). Wavelet integrated CNNs for noise-robust image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7245-7254.
• Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248-255. https://doi.org/10.1109/CVPR.2009.5206848
• Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261. https://doi.org/10.48550/arXiv.1903.12261
• Mallick, P. K., Ryu, S. H., Satapathy, S. K., Mishra, S., Nguyen, G. N., & Tiwari, P. (2019). Brain MRI image classification for cancer detection using deep wavelet autoencoder-based deep neural network. IEEE Access, 7, 46278-46287. https://doi.org/10.1109/ACCESS.2019.2902252
• Khatami, A., Nazari, A., Beheshti, A., Nguyen, T. T., Nahavandi, S., & Zieba, J. (2020, July). Convolutional neural network for medical image classification using wavelet features. In 2020 International Joint Conference on Neural Networks (IJCNN), 1-8. https://doi.org/10.1109/IJCNN48605.2020.9206791
• Said, S., Jemai, O., Hassairi, S., Ejbali, R., Zaied, M., & Amar, C. B. (2016, October). Deep wavelet network for image classification. In 2016 IEEE International conference on systems, man, and cybernetics (SMC), 000922-000927. https://doi.org/10.1109/SMC.2016.7844359
• Fujieda, S., Takayama, K., & Hachisuka, T. (2017). Wavelet convolutional neural networks for texture classification. arXiv preprint arXiv:1707.07394. https://doi.org/10.48550/arXiv.1707.07394
• Postalcioglu, S. (2022). Design of Automatic Tool for Diagnosis of Pneumonia Using Boosting Techniques. Brazilian Archives of Biology and Technology, 65, e22210322.
• Serte, S., & Demirel, H. (2019). Gabor wavelet-based deep learning for skin lesion classification. Computers in biology and medicine, 113, 103423. https://doi.org/10.1016/j.compbiomed.2019.103423
• Serte, S., & Demirel, H. (2020). Wavelet‐based deep learning for skin lesion classification. IET Image Processing, 14(4), 720-726. https://doi.org/10.1049/iet-ipr.2019.0553
• Aydin, V. A. (2022). CNN Tabanlı Yoga Pozu Sınıflandırmasında Aktivasyon Fonksiyonu Karşılaştırması. In Proceedings of IES’22 International Engineering Symposium, Engineering Applications in Industry, 117-122.
• https://www.kaggle.com/datasets/niharika41298/yoga-poses-dataset
• Chen, C. H., & Ramanan, D. (2017). 3d human pose estimation= 2d pose estimation+ matching. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7035-7043).