Automated Body Postures Assessment from Still Images Using Mediapipe

Abstract

Human poses assessment was an exciting research trend in the last decades. It was used in sports, health care, and many other fields, to help people get better performance. Machine learning and artificial intelligence techniques are used for this purpose. This paper used Google Mediapipe as a part of a framework for automatic Human-body pose assessment in real time. The proposed framework is based on detecting reference image poses, finding pose landmarks, and extracting discriminative features for each pose. These same process stages are applied to each image frame taken for the trainee using a web camera. The last stage of the framework compares the extracted features for the learner pose image with the saved features of the reference. The comparator specifies the inexact pose for each related human body part frame by frame. The reference image was proposed to enable the system to be used for various applications. Google Mediapipe was used for landmarks detection via Python, which was also used for feature extraction, making comparisons, and giving assessment advice. This system acts like a smart mirror that detects differences between the user pose and the reference still image then gives correction information in real time. Experiments were performed on side view cases like standing and sitting activities and gave promising results. This system could be very helpful for automatically self-pose assessment at home, or as an auxiliary tool for a certain learning program.

Keywords

• body postures
• pose
• landmarks
• Mediapipe
• Machine Learning

References

1. Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., & Grundmann, M. (2020). BlazePose: On device Real-time Body Pose tracking. Retrieved from doi: https://doi.org/10.48550/arXiv.2006.10204
2. Chen, K. (2019). Sitting Posture Recognition Based on OpenPose. IOP Conference Series: Materials Science and Engineering, 677(3). doi:https://doi.org/10.1088/1757- 899X/677/3/032057
3. Chiang, J.-C., Lie, W.-N., Huang, H.-C., Chen, K.-T., Liang, J.-Y., Lo, Y.-C., & Huang, W.-H. (2022). Posture Monitoring for Health Care of Bedridden Elderly Patients Using 3D Human Skeleton Analysis via Machine Learning Approach. Applied Sciences, 12(6), 3087. doi: https://doi.org/10.3390/app12063087
4. Dawange, S., Chavan, A., & Dusane, A. (2021). Workout Analysis Using Mediapipe BlazePose and Machine Learning. International Journal of Creative Research Thoughts (IJCRT), 9(12), 294–297. Diraco, G., Leone, A., & Siciliano, P. (2013). Human posture recognition with a time-of-flight 3D sensor for in-home applications. Expert Systems with Applications, 40(2), 744–751. doi: https://doi.org/10.1016/j.eswa.2012.08.007
5. Dittakavi, B., Bavikadi, D., Desai, S. V., Chakraborty, S., Reddy, N., Balasubramanian, V. N., … Sharma, A. (2022). Pose Tutor: An Explainable System for Pose Correction in the Wild. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022-June, 3539–3548. doi: https://doi.org/10.1109/CVPRW56347.2022.00398
6. George, S., Dcouth, J., Jaimy, S., Daji, E., & Antony, A. (2022). COMPUTER-ASSISTED TRAINING SYSTEM FOR YOGA AND PILATES USING COMPUTER VISION METHODS. Journal of Tianjin University Science and Technology, 55(06), 176–186. doi: https://doi.org/10.17605/OSF.IO/FXSQE
7. Gupta, R. (2020). Activity Monitoring of Islamic Prayer (Salat) Postures using Deep Learning. Roceedings - 2020 6th Conference on Data Science and Machine Learning Applications, CDMA, (October), 106–111. doi: https://doi.org/10.1109/CDMA47397.2020.00024
8. Hammadi, Y., Grondin, F., Ferland, F., & Lebel, K. (2022). Evaluation of Various State of the Art Head Pose Estimation Algorithms for Clinical Scenarios. Sensors, 22(18), 1–12. doi: https://doi.org/10.3390/s22186850

9. Han, K., Yang, Q., & Huang, Z. (2020). A two-stage fall recognition algorithm based on human posture features. Sensors (Switzerland), 20(23), 1–21. doi: https://doi.org/10.3390/s20236966
10. Jesmeen, M. Z. H., Bhuvaneswari, T., Mazbah, A. H., Chin, Y. B., Siong, L. H., & Aziz, N. H. A. (2023). SleepCon: Sleeping Posture Recognition Model using Convolutional Neural Network. Emerging Science Journal, 7(1), 50–59. doi: https://doi.org/10.28991/ESJ-2023-07- 01-04
11. Johnston, S. H., Berg, M. F., Eikevåg, S. W., Ege, D. N., Kohtala, S., & Steinert, M. (2022). Pure Vision-Based Motion Tracking for Data-Driven Design - A Simple, Flexible, and CostEffective Approach for Capturing Static and Dynamic Interactions. Proceedings of the Design Society, 2(2015), 485–494. doi: https://doi.org/10.1017/pds.2022.50
12. Khan, T. (2021). An Intelligent Baby Monitor with Automatic Sleeping Posture Detection and
13. Notification. Ai, 2(2), 290–306. doi: https://doi.org/10.3390/ai2020018
14. Kothari, S. (2020). Yoga Pose Detection and Classification Using Deep Learning (San Jose State University). doi: https://doi.org/10.32628/cseit206623
15. Koubaa, A., Ammar, A., Benjdira, B., Al-Hadid, A., Kawaf, B., Al-Yahri, S. A., … Ba Ras, M. (2020). Activity Monitoring of Islamic Prayer (Salat) Postures using Deep Learning. Proceedings - 2020 6th Conference on Data Science and Machine Learning Applications, CDMA 2020, 106–111. doi: https://doi.org/10.1109/CDMA47397.2020.00024
16. Kwon, Y., & Kim, D. (2022). Real-Time Workout Posture Correction using OpenCV and MediaPipe. The Journal of Korean Institute of Information Technology, 20(1), 199–208. doi: https://doi.org/10.14801/jkiit.2022.20.1.199
17. Lawanont, W., Inoue, M., Mongkolnam, P., & Nukoolkit, C. (2018). Neck posture monitoring system based on image detection and smartphone sensors using the prolonged usage classification concept. IEEJ Transactions on Electrical and Electronic Engineering, 13(10), 1501–1510. doi: https://doi.org/10.1002/tee.22778
18. Leightley, D., & Yap, M. H. (2018). Digital analysis of sit-to-stand in masters athletes, healthy old people, and young adults using a depth sensor. Healthcare (Switzerland), 6(1). doi: https://doi.org/10.3390/healthcare6010021
19. Long, C., Jo, E., & Nam, Y. (2022). Development of a yoga posture coaching system using an interactive display based on transfer learning. Journal of Supercomputing, 78(4), 5269–5284. doi: https://doi.org/10.1007/s11227-021-04076-w
20. Munea, T. L., Jembre, Y. Z., Weldegebriel, H. T., Chen, L., Huang, C., & Yang, C. (2020). The Progress of Human Pose Estimation: A Survey and Taxonomy of Models Applied in 2D Human Pose Estimation. IEEE Access, 8, 133330–133348. doi: https://doi.org/10.1109/ACCESS.2020.3010248
21. Nguyen, H. T. P., Woo, Y., Huynh, N. N., & Jeong, H. (2022). Scoring of Human Body-Balance Ability on Wobble Board Based on the Geometric Solution. Applied Sciences (Switzerland), 12(12). doi: https://doi.org/10.3390/app12125967
22. Nicolae-Adrian, J., Claudiu, A., Ana-Maria, V., & Ciprian, G. (2021). A systematic review of integrated machine learning in posture recognition. Timisoara Physical Education and Rehabilitation Journal, 14(27), 15–20. doi: https://doi.org/10.2478/tperj-2021-0009 Piñero-Fuentes, E., Canas-Moreno, S., Rios-Navarro, A., Domínguez-Morales, M., Sevillano,
23. J. L., & Linares-Barranco, A. (2021). A Deep-Learning Based Posture Detection System for Preventing Telework-Related Musculoskeletal Disorders. Sensors, 21(15), 5236. doi: https://doi.org/10.3390/s21155236
24. Stenum, J., Cherry-Allen, K. M., Pyles, C. O., Reetzke, R. D., Vignos, M. F., & Roemmich, R.
25. T. (2021). Applications of pose estimation in human health and performance across the lifespan. Sensors, 21(21). doi: https://doi.org/10.3390/s21217315
26. Wu, Y., Lin, Q., Yang, M., Liu, J., Tian, J., Kapil, D., & Vanderbloemen, L. (2022). A computer vision-based yoga pose grading approach using contrastive skeleton feature representations. Healthcare (Switzerland), 10(1), 1–12. doi: https://doi.org/10.3390/healthcare10010036
27. Xu, Z., Qu, W., Cao, H., Dong, M., Li, D., & Qiu, Z. (2022). An Adaptive Human Posture Detection Algorithm Based on Generative Adversarial Network. Computational Intelligence 2(2), 240-246, 2023 and Neuroscience, 2022. doi: https://doi.org/10.1155/2022/7193234