A New Approach to Improve Visual Quality for Single Image Super Resolution Applications

Öz

The single image super resolution problem has been extensively studied in the literature using various deep learning-based techniques. Super resolution based on deep convolutional networks has become a rapidly growing area of interest with many practical applications. However, the first studies based on deep learning were based on convolutional neural networks and peak signal-to-noise ratio (PSNR) oriented. Thanks to the models developed based on generative adversarial networks (GAN) in recent years, it has been determined as the main objective to increase the visual quality; however, this is not seen when the image quality metrics are examined. In this study, both mean square error and perceptual loss values were used for the network loss used during the training of the network. Also, the combination of three different training datasets was used as a new training dataset. As a result of these factors, both the visual quality has been increased and a significant increase has been achieved in the image quality metric values. In addition, batch normalization layers are not included in the network architecture and the training speed of the deep network architecture is increased by using the skip connection technique. The success performance of the proposed model was compared with the state-of-the-art models in the literature. Here, the peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) values were calculated and evaluated separately for three different test data sets commonly used in the literature. When the obtained results are evaluated, it is seen that the proposed model is more successful than other models and generates higher quality images. When all the findings are evaluated, it is seen that the proposed model is a more efficient model in terms of both success and training speed compared to state-of-the-art models.

Anahtar Kelimeler

• Super resolution
• Deep learning
• Image quality metrics
• New training dataset

Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım

Öz

Tek görüntü süper çözünürlük problemi, literatürde çeşitli derin öğrenme tabanlı teknikler kullanılarak kapsamlı çalışmalar yapılmıştır. Derin evrişimli ağlar tabanlı süper çözünürlük, çok sayıda pratik uygulama ile beraber hızla büyüyen bir ilgi alanı haline gelmiştir. Bununla birlikte derin öğrenme tabanlı ilk çalışmalar evrişimli sinir ağları tabanlı olup, tepe sinyal gürültü oranı odaklı çalışmalardır. Son yıllardaki çekişmeli üretici ağlar tabanlı geliştirilen modeller sayesinde görsel kaliteyi artırmak esas amaç olarak belirlenmiştir; fakat bu durum görüntü kalite metrikleri incelendiğinde görülmemektedir. Bu çalışmada ise ağın eğitimi sırasında kullanılan ağ kaybı için hem ortalama kare hata hem de algısal kayıp değerlerinden faydalanılmıştır. Ayrıca, üç farklı eğitim veri setinin birleşimi yeni bir eğitim veri seti olarak kullanılmıştır. Bu etmenlerin sonucunda hem görsel kalite artırılmış hem de görüntü kalite metrik değerlerinde ciddi bir artış yakalanmıştır. Ek olarak, yığın normalleştirme katmanları ağ mimarisine dahil edilmemiş ve bağlantı atlama tekniği kullanılarak derin ağ mimarisinin eğitim hızı artırılmıştır. Önerilen modelin başarı performansı literatürde yer alan önemli modeller ile karşılaştırılmıştır. Burada, tepe sinyal gürültü oranı ve yapısal benzerlik indeksi değerleri literatürde yaygın kullanılan üç farklı test veri seti için ayrı ayrı hesaplanmış ve değerlendirilmiştir. Elde edilen sonuçlar değerlendirildiğinde önerilen modelin diğer modellere göre daha başarılı olduğu ve daha kaliteli görüntüler oluşturduğu görülmektedir. Tüm bulgular değerlendirildiğinde önerilen modelin diğer modellere kıyasla hem başarı hem de eğitim hızı bakımından daha verimli bir model olduğu görülmektedir.

Anahtar Kelimeler

• Süper çözünürlük
• Derin öğrenme
• Görüntü kalite metrikleri
• Yeni eğitim veri seti

Kaynakça

1. Anwar, S., Khan, S., Barnes, N., 2020. A deep journey into super-resolution: A Survey. ACM Computing Surveys 53:1-34. http://dx.doi.org/10.1145/3390462
2. Chudasama, V., Patel, H., Prajapati, K., Upla, K., Ramachandra, R., Raja, K., Busch, C., 2020. TherISuRNet- A computationally efficient thermal image super-resolution network. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp 388-397. http://dx.doi.org/10.1109/CVPRW50498.2020.00051
3. Dong, C., Loy, C.C., He, K., Tan, X., 2015. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38:295-307. http://dx.doi.org/10.1109/TPAMI.2015.2439281
4. Dong, C., Loy, C.C., Tang, X., 2016. Accelerating the Super-Resolution Convolutional Neural Network. Computer Vision ECCV 2016 Lecture Notes in Computer Science Springer 9906. https://doi.org/10.1007/978-3-319-46475-6_25
5. Dosovitskiy, A., Brox, T., 2016. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS). pp 658–666. https://doi.org/10.48550/arXiv.1602.02644
6. Glasner, D., Bagon, S., Irani, M., 2009. Super-resolution from a single image. IEEE 12th International Conference on Computer Vision. pp 349-356. http://dx.doi.org/10.1109/ICCV.2009.5459271
7. Goodfellow, I., vd., 2014. Generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS). pp 2672–2680. http://dx.doi.org/10.1145/3422622
8. Gu, Y., vd., 2020. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimedia Tools and Applications 79:21815–21840. http://dx.doi.org/10.1007/s11042-020-08980-w

9. Javaid, H., Babar, T.K., Rasool, A., Saghir, R.U., 2013. Video colour variation detection and motion magnification to observe subtle changes. M.Sc. Thesis, Blekinge Institute of Technology, Faisalabad, Pakistan.
10. Johnson, J., Alahi, A., Li, F., 2016. Perceptual losses for real-time style transfer and super resolution. In European Conference on Computer Vision (ECCV) Springer. pp 694–711. http://dx.doi.org/10.1007/978-3-319-46475-6_43
11. Kim, J., Lee, J.K., Lee, K.M., 2016. Accurate image super-resolution using very deep convolutional networks. IEEE CVPR. pp 1646–1654. http://dx.doi.org/10.1109/CVPR.2016.182
12. Ledig, C., vd., 2017. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu. pp 105-114. http://dx.doi.org/10.1109/CVPR.2017.19
13. Radford, A., Metz, L., Chintala, S., 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
14. Senalp, F.M., Ceylan, M., 2020. Enhancement of low resolution thermal face image resolution using deep learning. European Journal of Science & Tech. pp 131-135. https://doi.org/10.31590/ejosat.802174
15. Senalp, F.M., Ceylan, M., 2021. Deep learning based super resolution and classification applications for neonatal thermal images. Traitement du Signal 38(5):1361-1368. https://doi.org/10.18280/ts.380511
16. Toyran, M., 2008. Reconstructing super resolution images from low resolution images. M.Sc. Thesis, Institute of science, Istanbul.
17. Wang, M., Chen, Z., Wu, Q.M.J., Jian, M., 2020. Improved face super-resolution generative adversarial networks. Machine Vision and Applications 31:22. https://doi.org/10.1007/s00138-020-01073-6
18. Yan, R., Yang, K., Wang, K., 2021. NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation across RGB-Depth, Polarization, and Thermal Images. 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 1129-1135. http://dx.doi.org/10.1109/ROBIO54168.2021.9739390
19. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L., 2018. Image super-resolution:the techniques, applications, and future. Signal Processing 128:389-408. http://dx.doi.org/10.1016/j.sigpro.2016.05.002
20. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y., 2018. Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision. pp 286–301. http://dx.doi.org/10.1007/978-3-030-01234-2_18