Performance Assessment of Functional Upper Extremity Exercises with Deep Learning

Öz

Rehabilitation exercises are essential for recovery following surgery and for managing musculoskeletal conditions. However, regular in-person physiotherapy sessions can be costly and difficult to access, particularly in home-based or remote care settings. This study presents a deep learning-based approach for automatically evaluating rehabilitation exercise performance using RGB videos captured with standard, low-cost cameras. Unlike conventional systems requiring costly depth sensors or wearables, the proposed method extracts 3D joint positions from standard RGB videos to assess movement quality. The model is trained on expert physiotherapist scores to ensure clinically meaningful evaluations. Experimental results show that the model’s predictions closely match the scores given by physiotherapists, demonstrating the reliability and accuracy of the system. This framework offers a practical and scalable solution for remote monitoring of rehabilitation exercises, reducing dependence on clinical supervision while maintaining assessment quality. The findings highlight the potential of deep learning to support more accessible, flexible, and cost-effective rehabilitation, particularly for individuals with limited access to traditional care services.

Anahtar Kelimeler

• Convolutional neural networks
• Deep learning
• Exercise
• Long short-term memory
• Physiotherapy and rehabilitation

Fonksiyonel Üst Ekstremite Egzersizlerinin Derin Öğrenme ile Performans Değerlendirmesi

Öz

Rehabilitasyon egzersizleri, ameliyat sonrası iyileşme süreci ve kas-iskelet sistemi bozukluklarının yönetimi için oldukça önemlidir. Ancak, düzenli yüz yüze fizyoterapi seansları maliyetli olabilir ve özellikle evde ya da uzaktan bakım gerektiren durumlarda erişimi zorlaşabilir. Bu çalışma, düşük maliyetli ve standart kameralarla kaydedilen RGB videolar kullanılarak rehabilitasyon egzersizlerinin performansını otomatik olarak değerlendiren derin öğrenme tabanlı bir yaklaşım sunmaktadır. Geleneksel olarak pahalı derinlik sensörleri veya giyilebilir cihazlara dayanan sistemlerin aksine, önerilen yöntem sıradan video görüntülerinden 3B eklem konumlarını çıkararak hareket kalitesini analiz eder. Model, uzman fizyoterapistler tarafından verilen performans puanlarıyla eğitilerek, klinik olarak anlamlı değerlendirmeleri öğrenmektedir. Deneysel sonuçlar, modelin tahminlerinin fizyoterapistlerin puanlarıyla yüksek oranda örtüştüğünü göstermekte ve sistemin güvenilirliğini ortaya koymaktadır. Bu çerçeve, klinik gözetim ihtiyacını azaltarak uzaktan rehabilitasyon egzersizlerinin izlenmesi için pratik ve ölçeklenebilir bir çözüm sunmaktadır. Bulgular, derin öğrenmenin daha erişilebilir, esnek ve düşük maliyetli rehabilitasyon süreçlerini desteklemedeki potansiyelini ortaya koymaktadır.

Anahtar Kelimeler

• Evrişimli sinir ağları
• Derin öğrenme
• Egzersiz
• Uzun kısa süreli bellek
• Fizyoterapi ve rehabilitasyon

Kaynakça

1. J. Seco, L. C. Abecia, E. Echevarría, I. Barbero, J. Torres-Unda, V. Rodríguez, and J. I. Calvo, “A long-term physical activity training program increases strength and flexibility, and improves balance in older adults,” Rehabil. Nurs., vol. 38, no. 1, pp. 37–47, 2013.
2. S. J. Allison, K. Brooke-Wavell, and J. Folland, “High and odd impact exercise training improved physical function and fall risk factors in community-dwelling older men,” J. Musculoskelet. Neuronal Interact., vol. 18, no. 1, p. 100, 2018.
3. S. R. Machlin, J. Chevan, W. W. Yu, and M. W. Zodet, “Determinants of utilization and expenditures for episodes of ambulatory physical therapy among adults,” Phys. Ther., vol. 91, no. 7, pp. 1018–1029, 2011.
4. S. Abbate, M. Avvenuti, and J. Light, “Usability study of a wireless monitoring system among Alzheimer’s disease elderly population,” Int. J. Telemed. Appl., vol. 2014, p. 7, 2014.
5. R. Komatireddy, A. Chokshi, J. Basnett, M. Casale, D. Goble, and T. Shubert, “Quality and quantity of rehabilitation exercises delivered by a 3-D motion controlled camera: A pilot study,” Int. J. Phys. Med. Rehabil., vol. 2, no. 4, 2014.
6. Y. Liao, A. Vakanski, and M. Xian, “A deep learning framework for assessment of quality of rehabilitation exercises,” arXiv preprint arXiv:1901.10435, 2019.
7. K.P. Dowd, R. Szeklicki, M.A. Minetto, M.H. Murphy, A. Polito, E. Ghigo, H. van der Ploeg, U. Ekelund, J. Maciaszek, R. Stemplewski, M. Tomczak, and A.E. Donnelly, “A systematic literature review of reviews on techniques for physical activity measurement in adults: A DEDIPAC study,” Int. J. Behav. Nutr. Phys. Act., vol. 15, no. 1, p. 15, 2018.
8. Z. B. S. Frih, Y. Fendri, A. Jellad, S. Boudoukhane, and N. Rejeb, “Efficacy and treatment compliance of a home-based rehabilitation programme for chronic low back pain: A randomized, controlled study,” Ann. Phys. Rehabil. Med., vol. 52, no. 6, pp. 485–496, 2009.

9. K. K. Miller, R. E. Porter, E. DeBaun-Sprague, M. Van Puymbroeck, and A. A. Schmid, “Exercise after stroke: patient adherence and beliefs after discharge from rehabilitation,” Top. Stroke Rehabil., vol. 24, no. 2, pp. 142–148, 2017.
10. A. Turolla, G. Rossettini, A. Viceconti, A. Palese, and T. Geri, “Musculoskeletal physical therapy during the COVID-19 pandemic: Is telerehabilitation the answer?” Phys. Ther., 2020.
11. G. Burdea, V. Popescu, V. Hentz, and K. Colbert, “Virtual reality-based orthopedic telerehabilitation,” IEEE Trans. Rehabil. Eng., vol. 8, no. 3, pp. 430–432, 2000.
12. G. K. Aytutuldu, İ. Aytutuldu, T. B. Olgun, and Y. S. Akgül, “Technology in physiotherapy: A bibliometric analysis of artificial intelligence in physiotherapy and rehabilitation,” Online Turkish J. Health Sci., vol. 10, no. 2, pp. 145–152, 2025.
13. Y.-J. Chang, S.-F. Chen, and J.-D. Huang, “A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities,” Res. Dev. Disabil., vol. 32, no. 6, pp. 2566–2570, 2011.
14. B. Lange, S. Koenig, E. McConnell, C.Y. Chang, R. Juang, and E. Suma, “Interactive game-based rehabilitation using the Microsoft Kinect,” in Proc. 2012 IEEE Virtual Reality Workshops (VRW), 2012, pp. 171–172.
15. D. Antón, A. Goñi, A. Illarramendi, J. J. Torres-Unda, and J. Seco, “KIRES: A Kinect-based telerehabilitation system,” in Proc. 2013 IEEE 15th Int. Conf. e-Health Netw., Appl. Serv. (Healthcom), Lisbon, Portugal, 2013, pp. 444–448.
16. H. M. Hondori, M. Khademi, and C. V. Lopes, “Monitoring intake gestures using sensor fusion (Microsoft Kinect and inertial sensors) for smart home tele-rehab setting,” in Proc. 2012 1st Annu. IEEE Healthcare Innovation Conf., Houston, TX, USA, 2012.
17. H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu, “RMPE: Regional multi-person pose estimation,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2017.
18. R. Girshick, I. Radosavovic, G. Gkioxari, P. Dollár, and K. He, Detectron. [Online]. Available: https://github.com/facebookresearch/detectron, 2018.
19. Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose: Realtime multi-person 2D pose estimation using part affinity fields,” arXiv preprint arXiv:1812.08008, 2018.
20. F. G. Hunsaker, D. A. Cioffi, P. C. Amadio, J. G. Wright, and B. Caughlin, “The American Academy of Orthopaedic Surgeons outcomes instruments: Normative values from the general population,” J. Bone Joint Surg. Am., vol. 84, no. 2, pp. 208–215, 2002.
21. S. Raschka, and V. Mirjalili, Python Machine Learning: Machine Learning and Deep Learning with Python, Scikit-learn, and TensorFlow 2. Birmingham, UK: Packt Publishing Ltd., 2019.
22. F. A. Kondori, S. Yousefi, H. Li, S. Sonning, and S. Sonning, “3D head pose estimation using the Kinect,” in Proc. 2011 Int. Conf. Wireless Commun. Signal Process. (WCSP), Nanjing, China, 2011, pp. 1–4.
23. P. Plantard, E. Auvinet, A.-S. Pierres, and F. Multon, “Pose estimation with a Kinect for ergonomic studies: Evaluation of the accuracy using a virtual mannequin,” Sensors, vol. 15, no. 1, pp. 1785–1803, 2015.
24. S. Li, P. N. Pathirana, and T. Caelli, “Multi-Kinect skeleton fusion for physical rehabilitation monitoring,” in Proc. 2014 36th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Chicago, IL, USA, 2014, pp. 5060–5063.
25. D. González-Ortega, F. J. Díaz-Pernas, M. Martínez-Zarzuela, and M. Antón-Rodríguez, “A Kinect-based system for cognitive rehabilitation exercises monitoring,” Comput. Methods Programs Biomed., vol. 113, no. 2, pp. 620–631, 2014.
26. A. Bernardino, C. Vismara, S. B. i Badia, É. Gouveia, F. Baptista, and F. Carnide, “A dataset for the automatic assessment of functional senior fitness tests using Kinect and physiological sensors,” in Proc. 2016 1st Int. Conf. Technol. Innov. Sports, Health Wellbeing (TISHW), Vila Real, Portugal, 2016, pp. 1–6.
27. K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view RGB-D object dataset,” in Proc. 2011 IEEE Int. Conf. Robot. Autom., Shanghai, China, 2011, pp. 1817–1824.
28. E. Dolatabadi, Y. X. Zhi, B. Ye, M. Coahran, G. Lupinacci, A. Mihailidis, R. Wang, and B. Taati, “The Toronto Rehab Stroke Pose dataset to detect compensation during stroke rehabilitation therapy,” in Proc. 11th EAI Int. Conf. Pervasive Comput. Technol. Healthc., Barcelona, Spain, 2017, pp. 375–381.
29. A. Vakanski, H.-p. Jun, D. Paul, and R. Baker, “A data set of human body movements for physical rehabilitation exercises,” Data, vol. 3, no. 1, p. 2, 2018.
30. İ. Aytutuldu and T. Aydin, “Performance assessment of physiotherapy and rehabilitation exercises with deep learning,” in Proc. 2022 30th Signal Process. Commun. Appl. Conf. (SIU), May 2022, pp. 1–4.
31. T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C.L. Zitnick, and P. Dollá, “Microsoft COCO: Common objects in context,” in Proc. Eur. Conf. Comput. Vis., Zurich, Switzerland, 2014, pp. 740–755.
32. Z. Bilici, I. Aytutuldu, Y. Genc, and Y. S. Akgul, “mmWave frequency modulated continuous wave radar-based human action recognition,” in Proc. 2024 32nd Signal Process. Commun. Appl. Conf. (SIU), May 2024, pp. 1–4.
33. J. Liu, H. Wang, W. Zhou, K. Stawarz, P. Corcoran, and Y. Chen, “Adaptive spatiotemporal graph transformer network for action quality assessment,” IEEE Trans. Circuits Syst. Video Technol., vol. 35, pp. 6628–6639, 2025.
34. Y. Bai, D. Zhou, S. Zhang, J. Wang, E. Ding, Y. Guan, Y. Long, and J. Wang “Action quality assessment with temporal parsing transformer,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Tel Aviv, Israel, Oct. 2022, pp. 1–18.
35. A. Iyer, M. Alali, H. Bodala, and S. Vaidya, “Action quality assessment using transformers,” arXiv preprint arXiv:2207.12318, 2022.
36. H. Fang, W.-G. Zhou, and H. Li, “End-to-end action quality assessment with action parsing transformer,” in Proc. IEEE Int. Conf. Visual Commun. Image Process. (VCIP), Dec. 2023, pp. 1–5.
37. K. Gedamu, Y. Ji, Y. Yang, J. Shao, and H. T. Shen, “Fine-grained spatio-temporal parsing network for action quality assessment,” IEEE Trans. Image Process., vol. 32, pp. 6386–6400, 2023.
38. H. Zhou, T. Hou, and J. Li, “Prior knowledge-guided hierarchical action quality assessment with 3D convolution and attention mechanism,” J. Phys. Conf. Ser., vol. 2632, no. 1, p. 012027, 2023.
39. A. Miron, N. Sadawi, W. Ismail, H. Hussain, and C. Grosan, “IntelliRehabDS (IRDS)—A dataset of physical rehabilitation movements,” Data, vol. 6, no. 5, p. 46, 2021.