Süper Çözünürlük Yönteminin Uydu İmgelerinin Sınıflandırma Performansına Etkisi

Abstract

Görüntünün yüksek çözünürlüğü uygulamalar için çok önemlidir. Halka açık sunulan uydu görüntülerinin çözünürlükleri genellikle düşüktür. Düşük çözünürlük bilgi kaybına yol açtığından uzaktan algılama alanında çalışılan problemin türüne bağlı olarak istenilen başarım sağlanamamaktadır. Böyle bir durumda düşük çözünürlüklü görüntülerden yüksek çözünürlüklü görüntü elde etmek için süper-çözünürlük algoritmaları kullanılmaktadır. Uydu görüntüleri ile yapılan çalışmalarda süper çözünürlükle zenginleştirilmiş görüntülerin kullanılması önemlidir. Uydu görüntülerinin çözünürlükleri düşük olduğundan dolayı sınıflandırma işleminde başarım oranı düşük çıkmaktadır. Bu çalışmada, uydu görüntülerinin sınıflandırma başarımını artırmak için süper çözünürlük yöntemi önerilmiştir. Derin öğrenme mimarisinden AlexNet, ResNet50, Vgg19 kullanılarak uydu imgelerinin öznitelikleri çıkarılmıştır. Ardından çıkarılan öznitelikler, AlexNet-Softmax, ResNet50-Softmax, Vgg19-Softmax, Destek Vektör Makinesi, K- En Yakın Komşu ve Naive Bayes sınıflandırma algoritmalarının girişine verilerek 6 sınıfa ayrılmıştır. Süper çözünürlük öncesi ve süper çözünürlük sonrası özellik çıkarma ve sınıflandırma işlemleri ayrı ayrı yapılmıştır. Süper çözünürlükten önce ve sonra sınıflandırma sonuçları karşılaştırılmıştır. Süper çözünürlük kullanılarak sınıflandırma performansında iyileşme gözlemlenmiştir.

Keywords

• Süper Çözünürlük
• Evrişimsel Sinir Ağları
• Derin Öğrenme
• Sınıflandırma

The Effect of Super Resolution Method on Classification Performance of Satellite Images

Abstract

The high resolution of the image is very important for applications. Publicly available satellite images generally have low resolutions. Since low resolution causes loss of information, the desired performance cannot be achieved depending on the type of problem studied in the field of remote sensing. In such a case, super resolution algorithms are used to render low resolution images high resolution. Super resolution algorithms are used to obtain high resolution images from low resolution images. In studies with satellite images, the use of images enhanced with super resolution is important. Since the resolution of satellite images is low, the success rate in the classification process is low. In this study, super resolution method is proposed to increase the classification performance of satellite images. The attributes of satellite images were extracted using AlexNet, ResNet50, Vgg19 from deep learning architecture. Then the extracted features were then classified into 6 classes by giving input to AlexNet-Softmax, ResNet50-Softmax, Vgg19-Softmax, Support Vector Machine, K-Nearest Neighbor, decision trees and Naive Bayes classification algorithms. Without super resolution and with super resolution feature extraction and classification processes were performed separately. Classification results without super resolution and with super resolution were compared. Improvement in classification performance was observed using super resolution.

Keywords

• Super Resolution
• Convolutional Neural Networks
• Deep Learning
• Classification

References

1. Dong C, Loy C, He K, Tang X. Image super-resolution using deep convolutional networks. EEE Trans. Pattern Anal. Mach. Intell. 2015; 38(2): 295-307.
2. Chen H, He X, Qing L, & Teng Q. Single image super-resolution via adaptive transform-based nonlocal self-similarity modeling and learning-based gradient regularization. IEEE Trans. Multimedia 2017; 19(8): 1702-1717.
3. Chang K, Zhang X, Ding P. L. K, Li B. Data-adaptive low-rank modeling and external gradient prior for single image super-resolution. J. Signal Process. Syst. 2019; 161: 36-49.
4. Li T, Dong X, Chen H. Single image super-resolution incorporating example-based gradient profile estimation and weighted adaptive p-norm. Neurocomputing 2019; 355: 105-120.
5. Li J, Guan W. Adaptive lq-norm constrained general nonlocal self-similarity regularizer based sparse representation for single image super-resolution. Inf. Fusion 2020; 53: 88-102.
6. Huang J. J, Liu T, Luigi Dragotti P, Stathaki T. SRHRF+: Self-example enhanced single image super-resolution using hierarchical random forests. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops; 2017; London. (pp. 71-79).
7. Huang J. B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition; 2015; (pp. 5197-5206).
8. Xiong Z, Xu D, Sun X, Wu F. Example-based super-resolution with soft information and decision. IEEE Trans. Multimedia 2013; 15(6): 1458-1465.

9. Huang J. B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition; 2015; (pp. 5197-5206).
10. Luo J, Sun X, Yiu M. L, Jin L, Peng X. Piecewise linear regression-based single image super-resolution via Hadamard transform. Inf. Sci. 2018; 462: 315-330.
11. Zhang Y, Du Y, Ling F, Li X. Improvement of the example-regression-based super-resolution land cover mapping algorithm. IEEE Geosci. Remote Sens. Lett. 2015; 12(8): 1740-1744.
12. Liu T, De Haan K, Rivenson Y, Wei Z, Zeng X, Zhang Y, Ozcan A. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 2019; 9(1): 1-13.
13. Jiang J, Wang C, Liu X, Ma J. Deep learning-based face super-resolution: A survey. ACM Comput. Surv. 2021; 55(1): 1-36.
14. Lim B, Son S, Kim H, Nah S, Mu Lee K. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops; 2017; (pp. 136-144).
15. Hatvani J, Horváth A, Michetti J, Basarab A, Kouamé D, Gyöngy M. Deep learning-based super-resolution applied to dental computed tomography. IEEE Trans. Radiat. Plasma Med. Sci. 2018; 3(2): 120-128.
16. Singh A, Singh J. Content adaptive single image interpolation based Super Resolution of compressed images. Int. J. Electr. Comput. Syst. Eng. 2020; 10(3): 3014-3021.
17. Zhou F, Yang W, Liao Q. Interpolation-based image super-resolution using multisurface fitting. IEEE Trans. Image Process. 2012; 21(7): 3312-3318.
18. Mahmoudzadeh A. P, Kashou N. H. Interpolation-based super-resolution reconstruction: effects of slice thickness. J. Med. Imaging Health Inf. 2014; 1(3): 034007-034007.
19. Zhang L, Zhang W, Lu G, Yang P, Rao Z. Feature-level interpolation-based GAN for image super-resolution. Pers. Ubiquitous Comput. 2022; 26(4): 995-1010.
20. Gulzar S, Arora S. Optical Flow Video Frame Interpolation Based MRI Super-Resolution. In Machine Intelligence and Smart Systems; 2022; Springer, Singapore. (pp. 451-462).
21. Alao H, Kim J. S, Kim T. S, Oh J, Lee K. Interpolation based Single-path Sub-pixel Convolution for Super-Resolution Multi-Scale Networks. Multimedia Syst. 2021; 8(4): 203-210.
22. Nazeri K, Thasarathan H, Ebrahimi M. Edge-informed single image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops ; 2019; pp. 1-10.
23. Zope A, Inamdar V. Edge Enhancement for Image Super-Resolution using Deep Learning Approach. 2nd Global Conference for Advancement in Technology (GCAT); 2021; Bangalore, India. pp. 1-4.
24. Zhou W, Wang Z, Chen Z. Image super-resolution quality assessment: Structural fidelity versus statistical naturalness. 13th International Conference on Quality of Multimedia Experience (QoMEX); 2021; pp. 61-64.
25. Jia S, Han B, Kutz J. N. Example-based super-resolution fluorescence microscopy. Sci. Rep. 2018; 8(1): 1-8.
26. Robey A, Ganapati V. Optimal physical preprocessing for example-based super-resolution. Opt. Express 2018; 26(24): 31333-31350.
27. Yang Q, Zhang Y, Zhao T. Example-based image super-resolution via blur kernel estimation and variational reconstruction. Pattern Recognit. Lett. 2019; 117: 83-89.
28. Glasner D, Bagon S, Irani M. Super-resolution from a single image. 12th international conference on computer vision; 2009; Kyoto. (pp. 349-356).
29. Timofte R, De Smet V, Van Gool L. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE international conference on computer vision; 2013; Sydney, Australia. (pp. 1920-1927).
30. Gao X, Zhang K, Tao D, Li X. Joint learning for single-image super-resolution via a coupled constraint. IEEE Trans. Image Process. 2011; 21(2): 469-480.
31. Cheong J. Y, Park I. K. Deep CNN-based super-resolution using external and internal examples. IEEE Signal Process Lett. 2017; 24(8): 1252-1256.
32. Wang Z, Wang Z, Chang S, Yang J, Huang T. A joint perspective towards image super-resolution: Unifying external-and self-examples. In IEEE Winter Conference on Applications of Computer Vision ; 2014; USA. ( pp. 596-603).
33. Nasrollahi K, Moeslund T. B, Super-resolution: a comprehensive survey. Mach. Vision Appl. 2014; 25(6) :1423-1468.
34. Chaudhuri S. Super-resolution imaging. London: Kluwer Academics Publishers, 2001.
35. Yang W, Zhang X, Tian Y, Wang W, Xue J. H, Liao Q. Deep learning for single image super-resolution: A brief review. IEEE Trans. Multimedia 2019; 21(12): 3106-3121.
36. Wang Z, Chen J, Hoi S. C. Deep learning for image super-resolution: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2020; 43(10): 3365-3387.
37. Jiang J, Wang C, Liu X, Ma J. Deep learning-based face super-resolution: A survey. ACM Comput. Surv. 2021; 55(1): 1-36.
38. Li Y, Sixou B, Peyrin F. A review of the deep learning methods for medical images super resolution problems. IRBM 2021; 42(2): 120-133.
39. Coşkun M, Yıldırım Ö, Uçar A, Demir, Y. An overview of popular deep learning methods. European Journal of Technique 2017; 7(2): 165-176.
40. Dong C, Loy C. C, He K, Tang X. Learning a deep convolutional network for image super-resolution. In Computer Vision–ECCV 2014: 13th European Conference; 2014; Zurich. (pp. 184-199).
41. Kim J, Lee J. K, Lee K. M. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition; 2016; (pp. 1646-1654).
42. Goodfellow I, Bengio Y, Courville A. Deep learning.London: MIT press,2016.
43. Fu Y, Liang Z, You S. Bidirectional 3d quasi-recurrent neural network for hyperspectral image super-resolution. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021;14: 2674-2688.
44. Chang Y, Luo B. Bidirectional convolutional LSTM neural network for remote sensing image super-resolution. J. Remote Sens. 2019; 11(20): 2333.
45. Zhu H, Xie C, Fei Y, Tao H. Attention mechanisms in CNN-based single image super-resolution: A brief review and a new perspective. Electronics 2021; 10(10): 1187.
46. Fu K, Peng J, Zhang H, Wang X, Jiang F. Image super-resolution based on generative adversarial networks: a brief review. CMC-Comput. Mater. Continua CMC 2020; 64(3): 1977-1997.
47. Shi W, Caballero J, Huszár F, Totz J, Aitken A. P, Bishop R, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition; 2016; (pp. 1874-1883).
48. Yue Y, Cheng X, Zhang D, Wu Y, Zhao Y, Chen Y, Zhang Y. Deep recursive super resolution network with Laplacian Pyramid for better agricultural pest surveillance and detection. Comput. Electron. Agric. 2018;150: 26-32.
49. Goyal B, Lepcha D. C, Dogra A, Wang S. H. A weighted least squares optimisation strategy for medical image super resolution via multiscale convolutional neural networks for healthcare applications. Complex Intell. Syst. 2022; 8:3089-3104.
50. Zhang H, Wang P, Jiang Z. Nonpairwise-trained cycle convolutional neural network for single remote sensing image super-resolution. IEEE Trans. Geosci. Remote Sens. 2020; 59(5): 4250-4261.
51. Yang C. Y, Ma C, Yang M. H. Single-image super-resolution: A benchmark. In Computer Vision–ECCV 2014 13th European Conference; 2014; Zurich. pp. 372-386.
52. Chen H, He X, Qing L, Wu Y, Re C, Sheriff R. E, Zhu C. Real-world single image super-resolution: A brief review. Inf. Fusion 2022; 79:124-145.
53. Deng X. Enhancing image quality via style transfer for single image super-resolution. IEEE Signal Process Lett. 2018; 25(4): 571-575.
54. Zamzmi G, Rajaraman S, Antani S. Accelerating super-resolution and visual task analysis in medical images. Adv. Nat. Appl. Sci. 2020; 10(12): 1-16.
55. Wagner L, Liebel L, Körner M. Deep Residual Learning For Single-Image Super-Resolution Of Multi-Spectral Satellite Imagery. SPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2019;4:189-196.
56. Kadhim M. A, Abed M. H. Convolutional neural network for satellite image classification. Int. J. Intell. Inf. Database Syst. 2020; 11: 165-178.
57. Basu S, Ganguly S, Mukhopadhyay S, DiBiano R, Karki M, Nemani R.Deepsat: a learning framework for satellite imagery. In Proceedings of the 23rd SIGSPATIAL international conference on advances in geographic information systems; 2015; USA. pp. 1-10.
58. Albert A, Kaur J, Gonzalez M. C. Using convolutional networks and satellite imagery to identify patterns in urban environments at a large scale. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining; 2017; Canada. (pp. 1357-1366).
59. Robinson C, Hohman F, Dilkina B. A deep learning approach for population estimation from satellite imagery. In Proceedings of the 1st ACM SIGSPATIAL Workshop on Geospatial Humanities; 2017; USA. (pp. 47-54).
60. Unnikrishnan A, Sowmya V, Soman K. P. Deep learning architectures for land cover classification using red and near-infrared satellite images. Multimedia Tools Appl. 2019; 78: 18379-18394.
61. Özbay E, Yıldırım M. Classification of satellite images for ecology management using deep features obtained from convolutional neural network models. Iran J.Comput. Sci. 2023; 1-9.
62. Chen Z, Guo X, Woo P. Y, Yuan Y. Super-resolution enhanced medical image diagnosis with sample affinity interaction. IEEE Trans. Med. Imaging 2021; 40(5): 1377-1389.
63. Wang P, Bayram B, Sertel E. A comprehensive review on deep learning based remote sensing image super-resolution methods. Earth Sci. Rev. 2022; 232:1-25.
64. Nguyen N. L, Anger J, Davy A, Arias P, Facciolo G. Self-supervised multi-image super-resolution for push-frame satellite images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021; USA.(pp. 1121-1131).
65. He Z, Li J, Liu L, He D, Xiao M. Multiframe video satellite image super-resolution via attention-based residual learning. IEEE Trans. Geosci. Remote Sens. 2021; 60: 1-15.
66. Agarwal A, Ratha N, Vatsa M, Singh R. Impact of Super-Resolution and Human Identification in Drone Surveillance. In 2021 IEEE International Workshop on Information Forensics and Security (WIFS);2021; France. pp. 1-6.
67. Toan N. Q. Super-Resolution Method for Reconstructing Street Images from Surveillance System based on Real-ESRGAN. 8th Student Comouting Research Symposium; 2022; Slovenia. pp.13-16.
68. Farooq M, Dailey M. N, Mahmood A, Moonrinta J, Ekpanyapong M. Human face super-resolution on poor quality surveillance video footage. Neural Comput. Appl. 2021; 33(20): 13505-13523.
69. Dabbech A, Terris M, Jackson A, Ramatsoku M, Smirnov O. M, Wiaux Y. First AI for deep super-resolution wide-field imaging in radio astronomy: unveiling structure in ESO 137-006. Astrophys. J. Lett. 2022; 939(1): 1-22.
70. Karwowska K, Wierzbicki D.Using Super-Resolution Algorithms for Small Satellite Imagery: A Systematic Review. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2022; 15: 3292-3312.
71. Ndajah P, Kikuchi H, Yukawa M, Watanabe H, Muramatsu S. SSIM image quality metric for denoised images. In Proc. 3rd WSEAS Int. Conf. on Visualization, Imaging and Simulation; 2010; (pp. 53-58).
72. http://en.wikipedia.org/wiki/Structural_similarity, (Access date: 23.11.2022).
73. http://www.google.com/int/tr/earth, (Access date: 16.05.2022)