ULTRA WIDEBAND RADAR-BASED HAND GESTURE RECOGNITION USING PRE-TRAINED DEEP NEURAL NETWORKS

Year 2024, Volume: 12 Issue: 1, 205 - 216, 25.03.2024

Nurullah Çalık

https://doi.org/10.21923/jesd.1418355

Abstract

Ultra-wideband (UWB) radar sensors play a pivotal role in recognizing human movements. They can be used to distinguish sensitive movements with their high frequency feature. In addition, this feature allows the sensor dimensions to be portable and easy to use in many areas. Although this process can be done via RGB cameras, problems are encountered, especially regarding privacy. Since people are anonymized in the data obtained from UWB sensors, only their movement patterns can be considered. Therefore, it has significant potential in issues such as in-home and elderly monitoring. In this study, hand movements are classified from radar data collected through three different sensors. A 3-channel image was obtained by combining radar images one after the other, right, left, and top. Then, pre-trained models were trained and tested on these images. When the training and testing ratio was 50:50, a performance rate of 97.93% (DenseNet201) was achieved, and when the ratio was 75:25, a performance rate of 97.65% (EfficientNetB0) was achieved. It has been shown that training models with the proposed strategy instead of using a single sensor makes a significant improvement.

Keywords

Hand Gesture Recognition, Ultra Wide-Band, RADAR, Deep Learning, Pre-trained Model

ÖN-EĞİTİLMİŞ DERİN SİNİR AĞLARI KULLANILARAK ULTRA GENİŞ BANT RADAR TABANLI EL HAREKETİ TANIMA

Abstract

Ultra geniş-bant (UGB) radar sensörleri insan hareketlerinin tanınmasında kritik bir öneme sahiptir. Sahip oldukları yüksek frekans özelliği ile hassas hareketlerin ayırt edilmesinde kullanılabilmektedirler. Ayrıca bu özellik, sensör boyutlarının portatif olmasına ve birçok alanda kolay kullanımına imkân tanır. Her ne kadar RGB kameralar üzerinden bu işlem yapılabilse de özellikle mahremiyet gizliliği konusunda problemler ile karşılaşılmaktadır. UGB sensörlerden elde edilen verilerde kişiler anonimleştiği için sadece hareket örüntüsü ele alınabilmektedir. Dolayısıyla, ev içi izleme ve yaşlı takibi gibi konularda önemli bir potansiyeli bulunmaktadır. Bu çalışmada, üç farklı sensör üzerinden toplanan radar verilerinden el hareketlerinin sınıflandırılması yapılmaktadır. Radar görselleri sağ, sol ve üst olacak şekilde arka arkaya bir araya getirilerek 3 kanallı bir görüntü elde edilmiştir. Ardından bu görüntüler üzerinden ön-eğitilmiş modeller eğitilmiş ve test edilmiştir. Eğitim ve test oranı 50:50 olduğu durumda %97.93 (DenseNet201) 75:25 oranında ise %97.65 (EfficientNetB0) oranında bir başarım elde edilmiştir. Tek sensör kullanımı yerine öne sürülen strateji ile modellerin eğitilmesinin önemli bir iyileştirme yaptığı ortaya koyulmuştur.

Keywords

El Hareketi Tanıma, Ultra Geniş-Bant, Radar, Derin Öğrenme, Ön-eğitilmiş Model

References

• Ahmed, S., & Cho, S. H. (2020). Hand gesture recognition using an IR-UWB radar with an inception module-based classifier. Sensors, 20(2), 564.
• Ahmed, S., Kallu, K. D., Ahmed, S., & Cho, S. H. (2021). Hand gestures recognition using radar sensors for human-computer-interaction: A review. Remote Sensing, 13(3), 527.
• Ahmed, S., Wang, D., Park, J., & Cho, S. H. (2021). UWB-gestures, a public dataset of dynamic hand gestures acquired using impulse radar sensors. Scientific Data, 8(1), 102.
• Ahmed, S., Yoon, S., & Cho, S. H. (2024). A public dataset of dogs vital signs recorded with ultra wideband radar and reference sensors. Scientific Data, 11(1), 107.
• Bouaafia, S., Messaoud, S., Maraoui, A., Ammari, A. C., Khriji, L., & Machhout, M. (2021, March). Deep pre-trained models for computer vision applications: traffic sign recognition. In 2021 18th International Multi-Conference on Systems, Signals & Devices (SSD) (pp. 23-28). IEEE.
• Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., ... & Gao, W. (2021). Pre-trained image processing transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12299-12310).
• Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., ... & Sun, M. (2023). Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3), 220-235.
• Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., ... & Zhu, J. (2021). Pre-trained models: Past, present and future. AI Open, 2, 225-250.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
• Hendy, N., Fayek, H. M., & Al-Hourani, A. (2022). Deep Learning Approaches for Air-Writing Using Single UWB Radar. IEEE Sensors Journal, 22(12), 11989-12001.
• Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708).
• Jiang, S., Skibniewski, M. J., Yuan, Y., Sun, C., & Lu, Y. (2011). Ultra-wide band applications in industry: a critical review. Journal of Civil Engineering and Management, 17(3), 437-444.
• Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
• Koziel, S., Çalık, N., Mahouti, P., & Belen, M. A. (2022). Reliable computationally efficient behavioral modeling of microwave passives using deep learning surrogates in confined domains. IEEE Transactions on Microwave Theory and Techniques, 71(3), 956-968.
• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.
• Lai, D.K., Zha, L., Leung, T.Y., Tam, A.Y., So, B.P., Lim, H., Cheung, D.S., Wong, D.W., & Cheung, J.S. (2023). Dual ultra-wideband (UWB) radar-based sleep posture recognition system: Towards ubiquitous sleep monitoring. Engineered Regeneration.
• Mahouti, P., Belen, M. A., Çalık, N., & Koziel, S. (2022). Computationally efficient surrogate-assisted design of pyramidal-shaped 3-D reflectarray antennas. IEEE Transactions on Antennas and Propagation, 70(11), 10777-10786.
• Mekruksavanich, S., Jantawong, P., Tancharoen, D., & Jitpattanakul, A. (2023, June). A Convolutional Neural Network for Ultra-Wideband Radar-Based Hand Gesture Recognition. In 2023 International Technical Conference on Circuits/Systems, Computers, and Communications (ITC-CSCC) (pp. 1-4). IEEE.
• Park, G., Chandrasegar, V. K., & Koh, J. (2023). Accuracy Enhancement of Hand Gesture Recognition Using CNN. IEEE Access, 11, 26496-26501.
• Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10), 1872-1897.
• Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
• Skaria, S., Al-Hourani, A., & Evans, R.J. (2020). Deep-Learning Methods for Hand-Gesture Recognition Using Ultra-Wideband Radar. IEEE Access, 8, 203580-203590.
• Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).
• Tan, M., & Le, Q. (2019, May). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (pp. 6105-6114). PMLR.
• Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of machine learning research, 9(11).
• Yuan, L., Chen, D., Chen, Y. L., Codella, N., Dai, X., Gao, J., ... & Zhang, P. (2021). Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432.