Comparative Analysis on Deep Learning based Pan-sharpening of Very High-Resolution Satellite Images

Abstract

Pan-sharpening is a fundamental task of remote sensing, aiming to produce a synthetic image having high spatial and spectral resolution of original panchromatic and multispectral images. In recent years, as in other tasks of the remote sensing field, deep learning based approaches have been developed for this task. In this research, a detailed comparative analysis was conducted to evaluate the performance and visual quality of pan-sharpening results from traditional algorithms and deep learning-based models. For this purpose, the deep learning based methods that are CNN based pan-sharpening (PNN), Multiscale and multi-depth convolutional neural networks (MSDCNN) and Pan-sharpened Generative Adversarial Networks (PSGAN) and traditional methods that are Brovey, PCA, HIS, Indusion and PRACS were applied. Analysis were performed on regions with different land cover characteristics to evaluate the stability of the methods. In addition, effects of the filter size, spectral indices, activation and loss functions on the pan-sharpening were investigated. For the accuracy assessment, commonly used with-reference and without-reference quality metrics were computed in addition to visual quality evaluations. According to results, the deep learning-based methods provided promising results in both the reduced resolution and full resolution experiments, while PRACS method outperformed other traditional algorithms in most of the experimental configurations.

Keywords

• Pan-sharpening
• Deep Learning
• Convolutional Neural Network
• Satellite Image

References

1. Aiazzi, B., Alparone, L., Baronti, S., & Garzelli, a. (2002). Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2002.803623
2. Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A., & Selva, M. (2003). An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas. 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas, URBAN 2003. https://doi.org/10.1109/DFUA.2003.1219964
3. AIRBUS. (2020). Pleiades Products. https://www.intelligence-airbusds.com/optical-and-radar-data/#pleiades
4. Chavez Jr., P. S., & Yaw Kwarteng, A. (1989). Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering & Remote Sensing.
5. Cheng, M., Wang, C., & Li, J. (2014). Sparse representation based pansharpening using trained dictionary. IEEE Geoscience and Remote Sensing Letters. https://doi.org/10.1109/LGRS.2013.2256875
6. Choi, J., Yu, K., & Kim, Y. (2011). A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2010.2051674
7. Christian Ledig Ferenc Huszar, L. T. (2017). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. 19.
8. DigitalGlobe. (2020). Tools and Resources. http://www.digitalglobe.com/resources.

9. Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2015.2439281
10. Gao, B. (1996). NDWI—A Normalized Difference Water Index for Remote Sensing of Vegetation Liquid Water From Space. Remote Sensing of Environment, 266(April), 257–266. https://doi.org/10.1016/S0034-4257(96)00067-3
11. Gillespie, A. R., Kahle, A. B., & Walker, R. E. (1987). Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sensing of Environment. https://doi.org/10.1016/0034-4257(87)90088-5
12. Goodfellow, I. J., & Pouget-Abadie, J. (2014). Generative Adversarial Nets. Veterinary Immunology and Immunopathology, 155(4), 270–275. https://doi.org/10.1016/j.vetimm.2013.08.005
13. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. https://doi.org/10.1109/CVPR.2016.90
14. Kang, X., Li, S., & Benediktsson, J. A. (2014). Pansharpening with Matting Model. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2013.2286827
15. Khan, M. M., Chanussot, J., Condat, L., & Montanvert, A. (2008). Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geoscience and Remote Sensing Letters, 5(1), 98–102. https://doi.org/10.1109/LGRS.2007.909934
16. Laben, C. ., & Brower, B. . (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. United States Patent 6. https://doi.org/10.1074/JBC.274.42.30033.(51)
17. Liu, J. G. (2000). Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing. https://doi.org/10.1080/014311600750037499
18. Liu, J. G. (2010). Smoothing Filter-based Intensity Modulation : A spectral preserve image fusion technique for improving spatial details. 1161. https://doi.org/10.1080/014311600750037499
19. Liu, X., Wang, Y., & Liu, Q. (2018a). PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening. 1–5. https://doi.org/10.1109/ICIP.2018.8451049
20. Liu, X., Wang, Y., & Liu, Q. (2018b). Remote Sensing Image Fusion Based on Two-Stream Fusion Network. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10704 LNCS, 428–439. https://doi.org/10.1007/978-3-319-73603-7_35
21. Masi, G., Cozzolino, D., Verdoliva, L., & Scarpa, G. (2016). Pansharpening by convolutional neural networks. Remote Sensing. https://doi.org/10.3390/rs8070594
22. Nencini, F., Garzelli, A., Baronti, S., & Alparone, L. (2007). Remote sensing image fusion using the curvelet transform. Information Fusion. https://doi.org/10.1016/j.inffus.2006.02.001
23. Nouri, H., Beecham, S., Anderson, S., & Nagler, P. (2013). High spatial resolution WorldView-2 imagery for mapping NDVI and its relationship to temporal urban landscape evapotranspiration factors. Remote Sensing, 6(1), 580–602. https://doi.org/10.3390/rs6010580
24. Open Remote Sensing. (2016). A Critical Comparison among Pansharpening Algorithms. http://openremotesensing.net/
25. Ozcelik, F., Alganci, U., Sertel, E., & Unal, G. (2020). Rethinking CNN-Based Pansharpening: Guided Colorization of Panchromatic Images via GANs. IEEE Transactions on Geoscience and Remote Sensing, 1–16. https://doi.org/10.1109/tgrs.2020.3010441
26. Palsson, F., Ulfarsson, M. O., & Sveinsson, J. R. (2020). Model-Based Reduced-Rank Pansharpening. IEEE Geoscience and Remote Sensing Letters. https://doi.org/10.1109/LGRS.2019.2926681
27. Panchal, S., & Thakker, R. (2015). Implementation and Comparative Quantitative Assessment of Different Multispectral Image Pansharpening Approaches. Signal & Image Processing : An International Journal, 6(5), 35–48. https://doi.org/10.5121/sipij.2015.6503
28. Pohl, C., & Van Genderen, J. L. (1998). Review article Multisensor image fusion in remote sensing: Concepts, methods and applications. In International Journal of Remote Sensing (Vol. 19, Issue 5). https://doi.org/10.1080/014311698215748
29. Rouse, J.W.; Haas, R.H.; Schell, J.A.; Deering, D.W.; Harlan., J. C. (1974). Monitoring the vernal advancement and retrogradation (green wave effect) of natural vegetation. 1–8.
30. Scarpa, G., Vitale, S., & Cozzolino, D. (2018). Target-Adaptive CNN-Based Pansharpening. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2018.2817393
31. Shettigara, V. K. (1992). A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set. Photogrammetric Engineering and Remote Sensing. https://doi.org/10.1038/050073a0
32. Tensorflow. (2015). https://www.tensorflow.org/
33. Tu, T.-M., Su, S.-C., Shyu, H.-C., & Huang, P. S. (2001). A new look at IHS-like image fusion methods. Information Fusion. https://doi.org/10.1016/S1566-2535(01)00036-7
34. Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G. A., Restaino, R., & Wald, L. (2015). A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2014.2361734
35. Wald, L., Ranchin, T., & Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions : Assessing the quality of resulting images. Photogrammetric Engineering & Remote Sensing.
36. Wei, Y., & Yuan, Q. (2017). Deep residual learning for remote sensed imagery pansharpening. RSIP 2017 - International Workshop on Remote Sensing with Intelligent Processing, Proceedings. https://doi.org/10.1109/RSIP.2017.7958794
37. Yang, X., Jian, L., Yan, B., Liu, K., Zhang, L., & Liu, Y. (2018). A sparse representation based pansharpening method. Future Generation Computer Systems, 88(June), 385–399. https://doi.org/10.1016/j.future.2018.04.096
38. Yin, H. (2015). Sparse representation based pansharpening with details injection model. Signal Processing. https://doi.org/10.1016/j.sigpro.2014.12.017
39. Yuan, Q., Wei, Y., Meng, X., Shen, H., & Zhang, L. (2018). A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. https://doi.org/10.1109/JSTARS.2018.2794888
40. Yuhas, R., Goetz, A. F. H., & Boardman, J. W. (1992). Descrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. Summaries of the Third Annual JPL Airborne Geoscience Workshop, JPL Publ. 92–14, Vol. 1.
41. Zhong, S., Zhang, Y., Chen, Y., & Wu, D. (2017). Combining component substitution and multiresolution analysis: A novel generalized BDSD Pansharpening algorithm. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. https://doi.org/10.1109/JSTARS.2017.2697445
42. Zhou, W., & Bovik, a C. (2002). A universal image quality index. Signal Processing Letters, IEEE. https://doi.org/10.1109/97.995823
43. Zhu, X. X., & Bamler, R. (2013). A sparse image fusion algorithm with application to pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2012.2213604