Videolardan Kalp Atış Hızı Kestirimi Üzerine Bir İnceleme

Öz

Kalp atış hızı; kişinin sağlığı, aktivite seviyesi, stres durumu, zindeliği ve benzeri fizyolojik durumları hakkında önemli ipuçları vermektedir. Kalp atış hızı, elektrokardiyogram (EKG) ve nabız oksimetreleriyle ölçülebilir olmakla birlikte, bu cihazlar sürekli temas gerektirdiğinden zamanla rahatsız edici olabilmektedir. Bilgisayarlı görü (computer vision) alanındaki son gelişmeler, bir kişiye elektrot veya nabız oksimetreleri takmanın mümkün veya uygun olmadığı durumlarda, videolardan kişinin kalp atış hızını tespit etmeye olanak sağlamıştır. Uzaktan fotopletismografi (rPPG), bir video kamera aracılığıyla derideki hassas renk değişikliklerini yakalayarak, yaşamsal belirtilerin tespit edilmesine imkân sağlayan bir teknolojidir. Son yıllarda yapılan çalışmalar, uzaktan kalp atış hızı tespiti için en uygun bölgenin yüz olduğunu göstermiştir. Bu çalışmada; videolar aracılığıyla kişilerin yüz bölgesinden kalp atışı hızı kestiriminin nasıl yapılabildiği, kalp atışı hızı kestirimi sürecindeki aşamaların nasıl iyileştirilebileceği ve nasıl daha yüksek doğrulukta kalp atışı hızı tespiti yapılabileceği hakkında literatürdeki mevcut yöntemler incelenerek kapsamlı bir analiz yapılmıştır.

Anahtar Kelimeler

• Kalp atış hızı
• uzaktan fotopletismografi (rPPG)
• yüz videoları
• derin öğrenme.

Heart Rate Estimation from Videos: A Review

Abstract

Heartbeat rate provides important signs about a person's health, activity level, stress level, vitality, and other related physiological conditions. Although electrocardiograms (ECG) and pulse oximeters can be exploited to measure the heart rate, these instruments might become uncomfortable with time due to the requirement for constant contact. Recent advances in computer vision have made it achievable to detect a person's heart rate from videos when it is impossible or impractical to attach electrodes or pulse oximeters to a person. Remote photoplethysmography (rPPG) is a technology that detects vital physiological information by capturing considerably precise color changes in the skin via a video camera. Recent studies have shown that the face is the most appropriate area to use for remote heart rate detection. In this study, a comprehensive analysis has been conducted by investigating the existing methods in the literature on how to estimate the heartbeat rate from a person's face through videos, how to improve the stages in the heart rate estimation process, and hence to improve the accuracy of the estimations.

Keywords

• Heartbeat rate
• remote photoplethysmography (rPPG)
• face videos
• deep learning

Kaynakça

1. Aydemir, B. (2019). Egzersiz esnasında toplanan fotopletismografi sinyallerinden kalp atım hızı ve ventilasyon hızı değişkenlerinin ayrıştırılması [MARMARA ÜNİVERSİTESİ]. https://acikbilim.yok.gov.tr/handle/20.500.12812/282457
2. Bian, M. (2019). Pattern Recognition and Computer Vision. In Fundamentals of Uncertainty Calculi with Applications to Fuzzy Inference. https://doi.org/10.1007/978-94-015-8449-4_9
3. Bousefsaf, F., Maaoui, C., & Pruski, A. (2013). Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomedical Signal Processing and Control, 8(6), 568–574. https://doi.org/10.1016/j.bspc.2013.05.010
4. Bush, I. (2016). Measuring Heart Rate from Video. https://web.stanford.edu/class/cs231a/prev_projects_2016/finalReport.pdf
5. Chen, W., & McDuff, D. (2018). DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11206 LNCS, 356–373. https://doi.org/10.1007/978-3-030-01216-8_22
6. Cheng, C. H., Wong, K. L., Chin, J. W., Chan, T. T., & So, R. H. Y. (2021). Deep learning methods for remote heart rate measurement: A review and future research agenda. In Sensors (Vol. 21, Issue 18). MDPI. https://doi.org/10.3390/s21186296
7. De Haan, G., & Jeanne, V. (2013). Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 60(10), 2878–2886. https://doi.org/10.1109/TBME.2013.2266196
8. Deng, Y., & Kumar, A. (2020). Standoff heart rate estimation from video – a review. 6. https://doi.org/10.1117/12.2560683

9. Djeldjli, D., Bousefsaf, F., Maaoui, C., & Bereksi-Reguig, F. (2019). Imaging Photoplethysmography: Signal Waveform Analysis. Proceedings of the 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS 2019, 2, 830–834. https://doi.org/10.1109/IDAACS.2019.8924239
10. Elgendi, M. (2012). On the Analysis of Fingertip Photoplethysmogram Signals. Current Cardiology Reviews, 8(1), 14–25. https://doi.org/10.2174/157340312801215782
11. ElMaghraby, A., Abdalla, M., Enany, O., & Y. El Nahas, M. (2014). Detect and Analyze Face Parts Information using Viola- Jones and Geometric Approaches. International Journal of Computer Applications, 101(3), 23–28. https://doi.org/10.5120/17667-8494
12. Hassan, M. A., Malik, A. S., Fofi, D., Saad, N., Karasfi, B., Ali, Y. S., & Meriaudeau, F. (2017). Heart rate estimation using facial video: A review. In Biomedical Signal Processing and Control (Vol. 38, pp. 346–360). Elsevier Ltd. https://doi.org/10.1016/j.bspc.2017.07.004
13. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90
14. Huang, B., Lin, C.-L., Chen, W., Juang, C.-F., & Wu, X. (2021). A novel one-stage framework for visual pulse rate estimation using deep neural networks. Biomedical Signal Processing and Control, 66(June 2020), 102387. https://doi.org/10.1016/j.bspc.2020.102387
15. Irani, R., Nasrollahi, K., & Moeslund, T. B. (2014). Improved pulse detection from head motions using DCT. VISAPP 2014 - Proceedings of the 9th International Conference on Computer Vision Theory and Applications, 3, 118–124. https://doi.org/10.5220/0004669001180124
16. Kazemi, V., & Sullivan, J. (2014). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1867–1874. https://doi.org/10.1109/CVPR.2014.241
17. Kwon, S., Kim, J., Lee, D., & Park, K. (2015). ROI analysis for remote photoplethysmography on facial video. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2015-Novem, 4938–4941. https://doi.org/10.1109/EMBC.2015.7319499
18. Li, X., Chen, J., Zhao, G., & Pietikäinen, M. (2014). Remote heart rate measurement from face videos under realistic situations. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 4264–4271. https://doi.org/10.1109/CVPR.2014.543
19. McDuff, D., Gontarek, S., & Picard, R. W. (2014a). Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Transactions on Biomedical Engineering, 61(10), 2593–2601. https://doi.org/10.1109/TBME.2014.2323695
20. McDuff, D., Gontarek, S., & Picard, R. W. (2014b). Remote detection of photoplethysmographic systolic and diastolic peaks using a digital camera. IEEE Transactions on Biomedical Engineering, 61(12), 2948–2954. https://doi.org/10.1109/TBME.2014.2340991
21. Meinzer, H. P., Deserno, T. M., Handels, H., & Tolxdorff, T. (2013). ROI Selection for Remote Photoplethysmography. Informatik Aktuell, 99–103. https://doi.org/10.1007/978-3-642-36480-8
22. Niu, X., Han, H., Shan, S., & Chen, X. (2018). SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. Proceedings - International Conference on Pattern Recognition, 2018-Augus(i), 3580–3585. https://doi.org/10.1109/ICPR.2018.8546321
23. Niu, X., Shan, S., Han, H., & Chen, X. (2020). RhythmNet: End-to-End Heart Rate Estimation from Face via Spatial-Temporal Representation. IEEE Transactions on Image Processing, 29, 2409–2423. https://doi.org/10.1109/TIP.2019.2947204
24. Pagano, T. P., Santos, V. R., Bonfim, Y. da S., Paranhos, J. V. D., Ortega, L. L., Sá, P. H. M., Nascimento, L. F. S., Winkler, I., & Nascimento, E. G. S. (2022). Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics (Switzerland), 11(9). https://doi.org/10.3390/electronics11091473
25. Poh, M., Mcduff, D. J., & Picard, R. W. (2010). Noncontact automated cardiac pulse measurements using video imaging and blind.pdf. Medical Optics and Biotechnology, 18(10), 795–805.
26. Poh, M. Z., McDuff, D. J., & Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1), 7–11. https://doi.org/10.1109/TBME.2010.2086456
27. Premkumar, S., & Hemanth, D. J. (2022). Intelligent Remote Photoplethysmography-Based Methods for Heart Rate Estimation from Face Videos: A Survey. In Informatics (Vol. 9, Issue 3). MDPI. https://doi.org/10.3390/informatics9030057
28. Rautaray, S. S., & Agrawal, A. (2012). R Eal T Ime H and G Esture R Ecognition. 3(1), 21–31.
29. Rouast, P. V., Adam, M. T. P., Chiong, R., Cornforth, D., & Lux, E. (2018). Remote heart rate measurement using low-cost RGB face video: a technical literature review. In Frontiers of Computer Science (Vol. 12, Issue 5, pp. 858–872). Higher Education Press. https://doi.org/10.1007/s11704-016-6243-6
30. Sabokrou, M., Pourreza, M., Li, X., Fathy, M., & Zhao, G. (2021). Deep-HR: Fast heart rate estimation from face video under realistic conditions. Expert Systems with Applications, 186. https://doi.org/10.1016/j.eswa.2021.115596
31. Shao, D., Liu, C., & Tsow, F. (2021). Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions. ACS Sensors, 6(2), 321–334. https://doi.org/10.1021/acssensors.0c02042
32. Sinhal, R., Singh, K., & Raghuwanshi, M. M. (2020). An Overview of Remote Photoplethysmography Methods for Vital Sign Monitoring. Advances in Intelligent Systems and Computing, 992, 21–31. https://doi.org/10.1007/978-981-13-8798-2_3
33. Song, R., Chen, H., Cheng, J., Li, C., Liu, Y., & Chen, X. (2021). PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography. IEEE Journal of Biomedical and Health Informatics, 25(5), 1373–1384. https://doi.org/10.1109/JBHI.2021.3051176
34. Spetlik, R., Franc, V., Cech, J., & Matas, J. (2018). Visual heart rate estimation with convolutional neural network. British Machine Vision Conference 2018, BMVC 2018, 1–12.
35. Sun, Y., & Thakor, N. (2016). Photoplethysmography Revisited: From Contact to Noncontact, from Point to Imaging. IEEE Transactions on Biomedical Engineering, 63(3), 463–477. https://doi.org/10.1109/TBME.2015.2476337
36. Swinehart, D. F. (1962). The Beer-Lambert law. Journal of Chemical Education, 39(7), 333–335. https://doi.org/10.1021/ed039p333
37. Tamura, T., Maeda, Y., Sekine, M., & Yoshida, M. (2014). Wearable photoplethysmographic sensors—past and present. Electronics , 3(2), 282–302. https://doi.org/10.3390/electronics3020282
38. Tang, C., Lu, J., & Liu, J. (2018). Non-contact heart rate monitoring by combining convolutional neural network skin detection and remote photoplethysmography via a low-cost camera. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2018-June, 1390–1396. https://doi.org/10.1109/CVPRW.2018.00178
39. Tsou, Y. Y., Lee, Y. A., Hsu, C. T., & Chang, S. H. (2020). Siamese-rPPG network: Remote photoplethysmography signal estimation from face videos. Proceedings of the ACM Symposium on Applied Computing, 2066–2073. https://doi.org/10.1145/3341105.3373905
40. Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434. https://doi.org/10.1364/oe.16.021434
41. Wang, W. (2017). Robust And Automatic Remote Photoplethysmography (Vol. 1, Issue 2017). https://pure.tue.nl/ws/files/78340965/20171023_Wang.pdf%0Ahttps://research.tue.nl/en/publications/robust-and-automatic-remote- photoplethysmography%0Ahttps://pure.tue.nl/ws/portalfiles/portal/78340965/20171023_Wang.pdf
42. Wedekind, D., Trumpp, A., Gaetjen, F., Rasche, S., Matschke, K., Malberg, H., & Zaunseder, S. (2017). Assessment of blind source separation techniques for video-based cardiac pulse extraction. Journal of Biomedical Optics, 22(3), 035002. https://doi.org/10.1117/1.jbo.22.3.035002
43. YAMAN, A. U. (2018). Yüz tanıma sistemlerinin yanıltılmasına karşı bir yöntem: yüz videolarında nabız tespiti ile canlılık doğrulaması [Ankara Üniversitesi]. https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=UwWffTuiMVwjS85blanc6Q&no=LlzkDJAuYHVHoM9tysjGIA
44. Yang, W., Li, X., & Zhang, B. (2018). Heart Rate Estimation from Facial Videos Based on Convolutional Neural Network. Proceedings of 2018 6th IEEE International Conference on Network Infrastructure and Digital Content, IC-NIDC 2018, 45–49. https://doi.org/10.1109/ICNIDC.2018.8525602
45. Yu, Z., Li, X., & Zhao, G. (2020). Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. 30th British Machine Vision Conference 2019, BMVC 2019.
46. Yu, Z., Peng, W., Li, X., Hong, X., & Zhao, G. (2019). Remote heart rate measurement from highly compressed facial videos: An end-to-end deep learning solution with video enhancement. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob, 151–160. https://doi.org/10.1109/ICCV.2019.00024
47. Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016). Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters, 23(10), 1499–1503. https://doi.org/10.1109/LSP.2016.2603342