A Deep Learning-Based Super Resolution Approach for Thermal Face Images Using New Datasets

Öz

Although thermal camera systems can be used in any application that requires the detection of temperature change, thermal imaging systems are highly costly systems and this situation makes difficult the common use of thermal systems. In addition, blurry images of low quality can occur when thermal images are obtained. In this article, super resolution application has been carried out on a data set consisting of thermal face images obtained from two different thermal cameras. The specified data set was created differently from traditional methods, low resolution (LR) thermal images were obtained from a 160x120 thermal resolution camera, while high resolution (reference) images were obtained from a camera with a thermal resolution of 640x480. Later, unnecessary parts of these images were cropped and another study was carried out by focusing only on the face area. A deep learning model based on adversarial generative networks (GAN) has been developed for these applications. The success performance of the results was evaluated by the image quality metrics PSNR (peak signal to noise ratio) and SSIM (structural similarity index). It has been observed that the results of the application performed by focusing only on the facial areas are better than the results of the application with original images. In addition, this study gave positive results in terms of approximating the resolution of the thermal images obtained by the less costly thermal camera to the resolution of the thermal camera, which has a high cost and can obtain high quality images, especially visually.

Anahtar Kelimeler

• Thermal imaging
• Super resolution
• Deep learning
• Datasets

Termal Yüz Görüntülerinden Oluşan Yeni Bir Veri Seti için Derin Öğrenme Tabanlı Süper Çözünürlük Uygulaması

Öz

Termal kamera sistemleri, ısı değişiminin tespitini gerektiren her türlü uygulamada faydalanılabilmesine rağmen termal görüntüleme sistemleri oldukça yüksek maliyete sahip sistemlerdir ve bu durum termal sistemlerin yaygın bir şekilde kullanımını zorlaştırmaktadır. Ayrıca termal görüntüler elde edilirken düşük kalitede bulanık görüntüler meydana gelebilmektedir. Bu makalede, iki farklı termal kameradan elde edilen termal yüz görüntülerinden oluşan bir veri seti üzerinde süper çözünürlük uygulaması gerçekleştirilmiştir. Belirtilen veri seti geleneksel yöntemlerden farklı bir şekilde oluşturulmuş olup, düşük çözünürlüklü (LR) termal görüntüler 160x120 termal çözünürlüğe sahip kameradan elde edilirken yüksek çözünürlüklü(referans) görüntüler ise 640x480 termal çözünürlüğe sahip kameradan elde edilmiştir. Daha sonra bu görüntülerdeki gereksiz kısımlar kırpılarak sadece yüz bölgesine odaklanılarak başka bir çalışma daha gerçekleştirilmiştir. Bu uygulamalar için çekişmeli üretici ağlar (GAN) tabanlı bir derin öğrenme modeli geliştirilmiştir. Sonuçların başarı performansı görüntü kalite metrikleri PSNR (tepe sinyal gürültü oranı) ve SSIM (yapısal benzerlik endeksi) ile değerlendirmeye alınmıştır. Sadece yüz bölgelerine odaklanılarak gerçekleştirilen uygulamanın sonuçları orijinal görüntülerle yapılan uygulama sonuçlarına kıyasla daha iyi olduğu görülmüştür. Bunun yanı sıra bu çalışma, daha az maliyetli termal kamera tarafından elde edilen termal görüntülerin çözünürlüğünü, yüksek maliyete sahip olan ve yüksek kalitede görüntüler elde edilebilen termal kameranın çözünürlüğüne bilhassa görsel olarak yaklaştırma yönünden olumlu sonuçlar vermiştir. 

Anahtar Kelimeler

• Termal görüntüleme
• Süper çözünürlük
• Derin öğrenme
• Veri setleri

Kaynakça

1. [1] Zhang X., Li C., Meng Q., Liu S., Zhang Y. and Wang J., “Infrared image super resolution by combining compressive sensing and deep learning”, Sensors (Basel),18(8):2587(2018).
2. [2] Yue L., Shen H., Li J., Yuan Q., Zhang H. and Zhang L., “Image super-resolution:the techniques, applications, and future” Signal Processing,128:389-408, (2018).
3. [3] Toyran M., “Reconstructing super resolution images from low resolution images”, M.Sc. Thesis, Institute of Science, Istanbul, (2008).
4. [4] Lobanov A.P., “Resolution limits in astronomical images”, arXiv, preprint astro-ph/0503225, (2005).
5. [5] Dong C., Loy C.C., He K., Tan X., “Image super-resolution using deep convolutional networks”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:295-307, (2016).
6. [6] Dong W., Fu F., Shi G., Cao X., Wu J., Li G. and Li X., “Hyperspectral image super-resolution via non-negative structured sparse representation”, IEEE Transactions on Image Processing, 25(5):2337-2352, (2016).
7. [7] Guei A., Akhloufi M., “Deep learning enhancement of infrared face images using generative adversarial networks”, Applied Optics, 57(18): 98, (2018).
8. [8] Nguyen K., Fookes C., Sridharan S., Denman S., “Feature-domain super-resolution for iris recognition”, Computer Vision and Image Understanding, 117(10):1526-1535, (2013).

9. [9] Glasner D., Bagon S., Irani M., “Super-resolution from a single image”, IEEE 12th International Conference on Computer Vision, 349-356, (2009).
10. [10] Lillesand T., Kiefer R. W., and Chipman J., “Remote sensing and image interpretation”, John Wiley & Sons, Hoboken, (2014).
11. [11] Çiftçi S., Karaman M.,“Landsat Uydu Görüntülerinde Derin Öğrenme Tabanlı Tek Görüntülü Süper-Çözünürlük Deneyleri”, Harran Üniversitesi Mühendislik Dergisi, 5(3): 194-204, (2020).
12. [12] Singh K., Gupta A., Kapoor R., “Fingerprint image super-resolution via ridge orientation-based clustered coupled sparse dictionaries”, Journal of Electronic Imaging, 24(4):043015, (2015).
13. [13] Gu Y., et al., “MedSRGAN: medical images super-resolution using generative adversarial networks”. Multimed Tools Appl, 79:21815–21840, (2020).
14. [14] Kim J., Lee J. K., Lee K. M., “Accurate image super-resolution using very deep convolutional networks”, IEEE CVPR, 1646–1654, (2016).
15. [15] Lim B., Son S., Kim H., Nah S., Lee K.M., “Enhanced deep residual networks for single image super-resolutaion” IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1132–1140, (2017).
16. [16] Ledig C., et al., “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu 105-114, (2017).
17. [17] Choi Y., Kim N., Hwang S, Kweon I.S., “Thermal image enhancement using convolutional neural network”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 223–230, (2016).
18. [18] Rivadeneira R., Sappa A., Vintimilla B., “Thermal Image Super-resolution: A Novel Architecture and Dataset”, 15th International Conference on Computer Vision Theory and Applications,111-119, (2020).
19. [19] Mandanici E., Tavasci L., Corsini F., Gandolfi S., “A multi-image super-resolution algorithm applied to thermal imagery”, Applied Geomatics, 11(3):215–228, (2019).
20. [20] Chudasama V., et al., “TherISuRNet- A computationally efficient thermal image super-resolution network”, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA 388-397, (2020).
21. [21] Senalp F.M., Ceylan M., “Enhancement of low resolution thermal face image resolution using deep learning”, European Journal of Science & Technologhy, 131-135, (2020).
22. [22] Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A. and Bengio Y., “Generative adversarial networks”, In Advances in Neural Information Processing Systems (NIPS), 2672–2680, (2014).
23. [23] Johnson J., Alahi A., Li F., “Perceptual losses for real-time style transfer and super resolution”, European Conference on Computer Vision (ECCV), Springer:694–711, (2016).
24. [24] Dosovitskiy A., Brox T., “Generating images with perceptual similarity metrics based on deep networks”, In Advances in Neural Information Processing Systems (NIPS), 658–666 (2016).
25. [25] Anwar S., Khan S., Barnes N. A., “Deep Journey into Super-resolution: A Survey”, ACM Computing Surveys, 53:1-34, (2020).
26. [26] Senalp F. M., Ceylan M., “Deep learning based super resolution and classifcation applications for neonatal thermal images”, Traitement du Signal, 38:5, pp. 1361-1368, (2021).
27. [27] Ioffe S., Szegedy C., “Batch normalization: accelerating deep network training by reducing internal covariate shift”, Proceedings of The 32nd International Conference on Machine Learning (ICML), 448–456, (2015).
28. [28] Rasamoelina A. D., Adjailia F. ve Sinčák P., “A Review of Activation Function for Artificial Neural Network”, IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI), 281-286, (2020).
29. [29] Javaid H., Babar T.K., Rasool A., Saghir R.U., “Video colour variation detection and motion magnification to observe subtle changes. M.Sc. Thesis”, Blekinge Institute of Technology, Faisalabad, Pakistan, (2013).
30. [30] Senalp F.M., Ceylan M., “Effects of the deep learning-based super-resolution method on thermal image classification applications”, Multimed Tools Appl, (2022).
31. [31] Zhang Y., Li K., Li K., Wang L., Zhong B., Fu Y., “Image super-resolution using very deep residual channel attention networks”, Proceedings of the European Conference on Computer Vision (ECCV),286–301, (2018).