Comparative Analysis on Deep Learning based Pan-sharpening of Very High-Resolution Satellite Images

Year 2021, Volume: 8 Issue: 2, 150 - 165, 15.06.2021

Peijuan Wang Ugur Alganci , Elif Sertel

https://doi.org/10.30897/ijegeo.834760

Cited By: 2

Abstract

Pan-sharpening is a fundamental task of remote sensing, aiming to produce a synthetic image having high spatial and spectral resolution of original panchromatic and multispectral images. In recent years, as in other tasks of the remote sensing field, deep learning based approaches have been developed for this task. In this research, a detailed comparative analysis was conducted to evaluate the performance and visual quality of pan-sharpening results from traditional algorithms and deep learning-based models. For this purpose, the deep learning based methods that are CNN based pan-sharpening (PNN), Multiscale and multi-depth convolutional neural networks (MSDCNN) and Pan-sharpened Generative Adversarial Networks (PSGAN) and traditional methods that are Brovey, PCA, HIS, Indusion and PRACS were applied. Analysis were performed on regions with different land cover characteristics to evaluate the stability of the methods. In addition, effects of the filter size, spectral indices, activation and loss functions on the pan-sharpening were investigated. For the accuracy assessment, commonly used with-reference and without-reference quality metrics were computed in addition to visual quality evaluations. According to results, the deep learning-based methods provided promising results in both the reduced resolution and full resolution experiments, while PRACS method outperformed other traditional algorithms in most of the experimental configurations.

Keywords

Pan-sharpening, Deep Learning, Convolutional Neural Network, Satellite Image

Thanks

The authors acknowledge the support of the ITU Center for Satellite Communications and Remote Sensing (ITU-CSCRS) by providing Pleiades satellite images for this research.

References

• Aiazzi, B., Alparone, L., Baronti, S., & Garzelli, a. (2002). Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2002.803623
• Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A., & Selva, M. (2003). An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas. 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas, URBAN 2003. https://doi.org/10.1109/DFUA.2003.1219964
• AIRBUS. (2020). Pleiades Products. https://www.intelligence-airbusds.com/optical-and-radar-data/#pleiades
• Chavez Jr., P. S., & Yaw Kwarteng, A. (1989). Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering & Remote Sensing.
• Cheng, M., Wang, C., & Li, J. (2014). Sparse representation based pansharpening using trained dictionary. IEEE Geoscience and Remote Sensing Letters. https://doi.org/10.1109/LGRS.2013.2256875
• Choi, J., Yu, K., & Kim, Y. (2011). A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2010.2051674
• Christian Ledig Ferenc Huszar, L. T. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 19.
• DigitalGlobe. (2020). Tools and Resources. http://www.digitalglobe.com/resources.
• Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2015.2439281
• Gao, B. (1996). NDWI—A Normalized Difference Water Index for Remote Sensing of Vegetation Liquid Water From Space. Remote Sensing of Environment, 266(April), 257–266. https://doi.org/10.1016/S0034-4257(96)00067-3
• Gillespie, A. R., Kahle, A. B., & Walker, R. E. (1987). Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sensing of Environment. https://doi.org/10.1016/0034-4257(87)90088-5
• Goodfellow, I. J., & Pouget-Abadie, J. (2014). Generative Adversarial Nets. Veterinary Immunology and Immunopathology, 155(4), 270–275. https://doi.org/10.1016/j.vetimm.2013.08.005
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. https://doi.org/10.1109/CVPR.2016.90
• Kang, X., Li, S., & Benediktsson, J. A. (2014). Pansharpening with Matting Model. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2013.2286827
• Khan, M. M., Chanussot, J., Condat, L., & Montanvert, A. (2008). Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geoscience and Remote Sensing Letters, 5(1), 98–102. https://doi.org/10.1109/LGRS.2007.909934
• Laben, C. ., & Brower, B. . (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. United States Patent 6. https://doi.org/10.1074/JBC.274.42.30033.(51)
• Liu, J. G. (2000). Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing. https://doi.org/10.1080/014311600750037499
• Liu, J. G. (2010). Smoothing Filter-based Intensity Modulation : A spectral preserve image fusion technique for improving spatial details. 1161. https://doi.org/10.1080/014311600750037499
• Liu, X., Wang, Y., & Liu, Q. (2018a). PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening. 1–5. https://doi.org/10.1109/ICIP.2018.8451049
• Liu, X., Wang, Y., & Liu, Q. (2018b). Remote Sensing Image Fusion Based on Two-Stream Fusion Network. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10704 LNCS, 428–439. https://doi.org/10.1007/978-3-319-73603-7_35
• Masi, G., Cozzolino, D., Verdoliva, L., & Scarpa, G. (2016). Pansharpening by convolutional neural networks. Remote Sensing. https://doi.org/10.3390/rs8070594
• Nencini, F., Garzelli, A., Baronti, S., & Alparone, L. (2007). Remote sensing image fusion using the curvelet transform. Information Fusion. https://doi.org/10.1016/j.inffus.2006.02.001
• Nouri, H., Beecham, S., Anderson, S., & Nagler, P. (2013). High spatial resolution WorldView-2 imagery for mapping NDVI and its relationship to temporal urban landscape evapotranspiration factors. Remote Sensing, 6(1), 580–602. https://doi.org/10.3390/rs6010580
• Open Remote Sensing. (2016). A Critical Comparison among Pansharpening Algorithms. http://openremotesensing.net/
• Ozcelik, F., Alganci, U., Sertel, E., & Unal, G. (2020). Rethinking CNN-Based Pansharpening: Guided Colorization of Panchromatic Images via GANs. IEEE Transactions on Geoscience and Remote Sensing, 1–16. https://doi.org/10.1109/tgrs.2020.3010441
• Palsson, F., Ulfarsson, M. O., & Sveinsson, J. R. (2020). Model-Based Reduced-Rank Pansharpening. IEEE Geoscience and Remote Sensing Letters. https://doi.org/10.1109/LGRS.2019.2926681
• Panchal, S., & Thakker, R. (2015). Implementation and Comparative Quantitative Assessment of Different Multispectral Image Pansharpening Approaches. Signal & Image Processing : An International Journal, 6(5), 35–48. https://doi.org/10.5121/sipij.2015.6503
• Pohl, C., & Van Genderen, J. L. (1998). Review article Multisensor image fusion in remote sensing: Concepts, methods and applications. In International Journal of Remote Sensing (Vol. 19, Issue 5). https://doi.org/10.1080/014311698215748
• Rouse, J.W.; Haas, R.H.; Schell, J.A.; Deering, D.W.; Harlan., J. C. (1974). Monitoring the vernal advancement and retrogradation (green wave effect) of natural vegetation. 1–8.
• Scarpa, G., Vitale, S., & Cozzolino, D. (2018). Target-Adaptive CNN-Based Pansharpening. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2018.2817393
• Shettigara, V. K. (1992). A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set. Photogrammetric Engineering and Remote Sensing. https://doi.org/10.1038/050073a0
• Tensorflow. (2015). https://www.tensorflow.org/
• Tu, T.-M., Su, S.-C., Shyu, H.-C., & Huang, P. S. (2001). A new look at IHS-like image fusion methods. Information Fusion. https://doi.org/10.1016/S1566-2535(01)00036-7
• Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G. A., Restaino, R., & Wald, L. (2015). A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2014.2361734
• Wald, L., Ranchin, T., & Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions : Assessing the quality of resulting images. Photogrammetric Engineering & Remote Sensing.
• Wei, Y., & Yuan, Q. (2017). Deep residual learning for remote sensed imagery pansharpening. RSIP 2017 - International Workshop on Remote Sensing with Intelligent Processing, Proceedings. https://doi.org/10.1109/RSIP.2017.7958794
• Yang, X., Jian, L., Yan, B., Liu, K., Zhang, L., & Liu, Y. (2018). A sparse representation based pansharpening method. Future Generation Computer Systems, 88(June), 385–399. https://doi.org/10.1016/j.future.2018.04.096
• Yin, H. (2015). Sparse representation based pansharpening with details injection model. Signal Processing. https://doi.org/10.1016/j.sigpro.2014.12.017
• Yuan, Q., Wei, Y., Meng, X., Shen, H., & Zhang, L. (2018). A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. https://doi.org/10.1109/JSTARS.2018.2794888
• Yuhas, R., Goetz, A. F. H., & Boardman, J. W. (1992). Descrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. Summaries of the Third Annual JPL Airborne Geoscience Workshop, JPL Publ. 92–14, Vol. 1.
• Zhong, S., Zhang, Y., Chen, Y., & Wu, D. (2017). Combining component substitution and multiresolution analysis: A novel generalized BDSD Pansharpening algorithm. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. https://doi.org/10.1109/JSTARS.2017.2697445
• Zhou, W., & Bovik, a C. (2002). A universal image quality index. Signal Processing Letters, IEEE. https://doi.org/10.1109/97.995823
• Zhu, X. X., & Bamler, R. (2013). A sparse image fusion algorithm with application to pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2012.2213604