Augmented Reality Aided Pre-Diagnosis Environment For Telemedicine: Superficial Vein Surveillance System

Öz

The proposed system creates a virtual pre-diagnosis environment that can detect narrowings in superficial veins by using the near-infrared video images. In the study, the near-infrared video recordings of the tissue to be followed are taken by the user in the home environment via the smart device. The images obtained by improving the discontinuous structures in the vein images undergone the image pre-processing phase are classified by using a hybrid decision-making algorithm that evaluates two separate convolutional neural network models together. According to the results of the hybrid decision-making algorithm, the imaged regions could be classified with Model-1 (Accuracy Rate (0.872), Misclassification Rate (0.128), Precision (0.372), Prevalence (0.500) and F-Score (0.496)) and Model-2 (Accuracy Rate (0.816), Misclassification Rate (0.184), Precision (0.407), Prevalence (0.500) ve F-Score (0.543)) without the need for large amounts of training dataset. In the study, the detected vein narrowings in the vein images are marked on the relevant location. The marked images are superimposed on the real images and the narrowing progress process is presented to the user and his/her physician as a telemedicine application in the form of a video-based indirect augmented reality environment representing a long time interval (week, month, year).

Anahtar Kelimeler

• Near Infrared Light
• Vein Narrowing Detection
• Convolutional Neural Network
• Yolov3
• Augmented Reality
• Telemedicine

Teletıp İçin Artırılmış Gerçeklik Destekli Ön-Teşhis Ortamı: Yüzeysel Damar Takip Sistemi

Öz

Önerilen sistem, yakın-kızılötesi video görüntülerini kullanarak yüzeysel damarlardaki daralmaları tespit edebilen sanal bir ön-teşhis ortamı oluşturmaktadır. Çalışmada, takip edilecek dokunun yakın kızıl-ötesi video kayıtları akıllı cihaz aracılığıyla kullanıcı tarafından ev ortamında alınmaktadır. Görüntü ön-işleme aşamasından geçirilen damar görüntülerindeki kesikli yapılar giderilerek elde edilen görüntüler, iki ayrı evrişimsel sinir ağı modelini birlikte değerlendiren hibrit karar verme algoritması kullanılarak sınıflandırılmaktadır. Hibrit karar verme algoritması sonuçlarına göre, görüntülenen bölgeler, Model-1 (Doğruluk Oranı (0.872), Yanlış Sınıflandırma Oranı (0.128), Kesinlik (0.372), Yaygınlık (0.500) ve F-Skoru (0.496)) ve Model-2 ile (Doğruluk Oranı (0.816), Yanlış Sınıflandırma Oranı (0.184), Kesinlik (0.407), Yaygınlık (0.500) ve F- Skoru (0.543)) büyük miktarda eğitim verisetine ihtiyaç duyulmadan sınıflandırılmıştır. Çalışmada, damar görüntülerinde tespit edilen damar daralmaları, ilgili konum üzerine işaretlenmektedir. İşaretli görüntüler, gerçek görüntüler üzerine bindirilmekte ve daralma gelişim süreci, uzun bir zaman aralığını (hafta, ay, yıl) temsil eden video-tabanlı dolaylı artırılmış gerçeklik ortamı şeklindeki bir uzaktıp uygulaması olarak kullanıcıya ve hekimine sunulmaktadır.

Anahtar Kelimeler

• Yakın Kızıl-Ötesi Işık
• Damar Daralma Tespiti
• Evrişimli Sinir Ağı
• Yolov3
• Artırılmış Gerçeklik
• Uzaktıp

Teşekkür

Yazarlar, çalışmanın donanım aşamasındaki katkılarından dolayı Dr.Bora UZUN'a (Dokuz Eylül Üniversitesi, Tıp Fakültesi Biyomekanik Anabilim Dalı, İzmir, Türkiye) ve makalenin son düzenlemelerini yapan İnş.Yük.Müh.Işıl ERDEM'e teşekkür ederler. / The authors would like to thank PhD Bora UZUN (Dokuz Eylul University, Department of Biomechanics, School of Medicine, Izmir, Turkey) for his contributions during the hardware phase of the study and to Civil Engineer MSc Işıl ERDEM for making the final edits of the paper.

Kaynakça

1. Abdulghani, A. M. A., & Menekşe Dalveren, G. G. (2022). Moving object detection in video under different weather conditions using YOLO and faster R-CNN algorithms. European Journal of Science and Technology. (33): 40-54.
2. Ai, D., Yang, J., Fan, J., Zhao, Y., Song, X., Shen, J., Shao, L., & Wang, Y. (2016). Augmented reality based real-time subcutaneous vein imaging system. Biomedical Optics Express. 7(7): 2565-2585.
3. Albawi, S., Mohammed, T. A., & Al-Zawi, S. (2017). Understanding of a convolutional neural network. In 2017 International Conference on Engineering and Technology (ICET), (pp. 1-6). IEEE.
4. Alwazzan, M. J. (2020). Low cost blood vein detection system based on near-infrared LEDs and image-processing techniques. Polish Journal of Medical Physics and Engineering. 26(2): 61-67.
5. Anzueto-Rios, A., Hernandez-Gomez, L. E., & Hernandez-Santiago, K. A. (2016). Forearm and hand vein detection system for an infrared image database. Res. Comput. Sci. 127(1): 137-147.
6. Chunara, R., Zhao, Y., Chen, J., Lawrence, K., Testa, P. A., Nov, O., & Mann, D. M. (2020). Telemedicine and healthcare disparities: A cohort study in a large healthcare system in New York City during COVID-19. Journal of the American Medical Informatics Association. 28(1): 33-41.
7. Craig, J., & Petterson, V. (2005). Introduction to the practice of telemedicine. Journal of Telemedicine and Telecare. 11(1): 3-9.
8. Crisan, S., Tarnovan, J. G., & Crisan, T.E. (2007). A low cost vein detection system using near infrared radiation. In 2007 IEEE Sensors Applications Symposium, (pp. 1-6). IEEE.

9. Dikbayır, H. S., & Bülbül, H. İ. (2020). Real-time vehicle detection by using deep learning methods. Tübav Bilim Dergisi. 13(3): 1-14. “(Article in Turkish with an abstract in English)”
10. Demir, A. G. (2019). Determination of vascular stenosis on angiography images using convolutional neural network method. [Master's thesis, Başkent University, Ankara, Turkey]. “(Thesis in Turkish with an abstract in English)”
11. Doğan, D., Erol, T., & Mendi, A. F. (2021). Sağlık alanında karma gerçeklik. Avrupa Bilim ve Teknoloji Dergisi. (29): 11-18. “(Article in Turkish with an abstract in English)”
12. Elnasir, S., & Shamsuddin, S. M. (2014). Palm vein recognition based on 2D-discrete wavelet transform and linear discrimination analysis. Int. J. Advance Soft Compu. Appl. 6(3): 43-59.
13. Farhadi, A., & Redmon, J. (2018). Yolov3: An incremental improvement. In Computer Vision and Pattern Recognition Berlin/Heidelberg/Germany, (pp.1804.1-6). Springer.
14. Ferrari, M., Mottola, L., & Quaresima, V. (2004). Principles, techniques, and limitations of near infrared spectroscopy. Canadian Journal of Applied Physiology. 29(4): 463-487.
15. Francis, M., Jose, A., & Avinashe, K. K. (2017). A novel technique for forearm blood vein detection and enhancement. Biomedical Research. 28(7): 2913-2919.
16. Francisco, M. D., Chen, W. F., Pan, C. T., Lin, M. C., Wen, Z. H., Liao, C. F., & Shiue, Y. L. (2021). Competitive real-time near infrared (NIR) vein finder imaging device to improve peripheral subcutaneous vein selection in venipuncture for clinical laboratory testing. Micromachines. 12(4): 373.
17. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 580-587). IEEE.
18. Huda, A. N., Goh, C. M., Lim, C. H., Aluwee, S. S., Bajuri, M. N., & Wahab, N. H. A. (2021). Development of a near-infrared (NIR) forearm subcutaneous vein extraction using deep residual U-Net. International Conference on Biomedical Engineering (ICoBE).
19. Mangold, K., Shaw, J. A., & Vollmer, M. (2013). The physics of near-infrared photography. European Journal of Physics. 34(6); 51-57.
20. MathWorks, Inc. (1996). MATLAB (R2017a): The language of technical computing, computation, visualization, programming, installation guide for UNIX version 5. Natwick, Massachusetts.
21. Meng, G. C., Shahzad, A., Saad, N. M., Malik, A. S., & Meriaudeau, F. (2015). Prototype design for wearable veins localization system using near infrared imaging technique. In 2015 IEEE 11th International Colloquium on Signal Processing and Its Applications (CSPA), (pp. 112-115). IEEE.
22. Mistry, D., & Banerjee, A. (2017). Comparison of feature detection and matching approaches: SIFT and SURF. GRD Journals Global Research and Development Journal for Engineering. 2(4): 7-13.
23. Park, K. R. (2011). Finger vein recognition by combining global and local features based on SVM. Computing and Informatics. 30(2): 295-309.
24. Rao, H., Zhang, P., & Sun C. (2017). Contrast enhancement for the infrared vein image of leg based on the optical angular spectrum theory. Signal, Image and Video Processing. 11(3): 423-429.
25. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 779-788). IEEE.
26. Ren, S., He, K., Girshick, R., & Sun, J. (2016). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 39(6): 1137-1149.
27. Shrotri, A., Rethrekar, S. C., Patil, M. H., & Kore, S. N. (2010). IR-webcam imaging and vascular pattern analysis towards hand vein authentication. In 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE), (Vol. 5, pp. 876-880). IEEE.
28. Sordillo, L. A., Pu, Y., Pratavieira, S., Budansky, Y., & Alfano, R. R. (2014). Deep optical imaging of tissue using the second and third near-infrared spectral windows. Journal of Biomedical Optics. 19(5): 056004.
29. Şayli, Ö., Akdemir, A., Ataklı, Y., Emir, U. E., Çıtlak, P. Ö., Cengiz, L. S., & Akın, A. (2004). Kaslardaki oksidatif metabolizma farkının işlevsel yakın-kızılötesi spektroskopi ile incelenmesi. BİYOMUT. İstanbul. “(Article in Turkish with an abstract in English)”
30. Şeker, K., & Engin, M. (2017). Deep tissue near-infrared imaging for vascular network analysis. Journal of Innovative Optical Health Sciences. 10(3): 1650051.1-12.
31. Tan, F. G., Yüksel, A. S., Aydemir, E., & Ersoy, M. (2021). Derin öğrenme teknikleri ile nesne tespiti ve takibi üzerine bir inceleme. Avrupa Bilim ve Teknoloji Dergisi. (25): 159-171. “(Article in Turkish with an abstract in English)”
32. Tien, T. V., Mien, P. T., Dung, P. T., & Linh, H. Q. (2015). Using near-infrared technique for vein imaging. In 5th International Conference on Biomedical Engineering in Vietnam, (pp. 190-193). Springer, Cham.
33. Vedaldi, A., & Lenc, K. (2015). Matconvnet: Convolutional neural networks for matlab. In Proceedings of the 23rd ACM International Conference on Multimedia, (pp. 689-692).
34. Wadhwani, M., Sharma, A. D., Pillai, A., Pisal, N., & Bhowmick, M. (2015). Vein detection system using infrared light. Int. J. Sci. Eng. Res. 6(12): 780-786.
35. Wang, L., & Leedham, G. (2006). Near-and far-infrared imaging for vein pattern biometrics. In 2006 IEEE International Conference on Video and Signal Based Surveillance, (pp. 52-57). IEEE.
36. Wu, W., Elliott, S. J., Lin, S., & Yuan, W. (2019). Low-cost biometric recognition system based on NIR palm vein image. IET Biometrics. 8(3): 206-214.
37. Wu, Y., Zhang, Z., & Wang, G. (2021). Training deep neural networks via branch-and-bound. ArXiv Preprint. arXiv:2104.01730.
38. Yılmaz, K. (2014). The design of a new portable ophthalmoscope and utilization of the device for diagnosis and identification. [Doctoral thesis, Ege University, Izmir, Turkey]. “(Thesis in Turkish with an abstract in English)”