Çok bantlı görüntülerde pan-keskinleştirme üzerine bir inceleme
Year 2021,
Volume: 11 Issue: 4, 1340 - 1357, 15.10.2021
Çiğdem Şerifoğlu Yılmaz
,
Volkan Yılmaz
,
Oğuz Güngör
Abstract
Uzaktan algılama uyduları, algılayıcılarındaki teknik kısıtlamalardan dolayı hem uzamsal detay kalitesi hem de spektral kalitesi yüksek görüntüler üretememektedir. Bu durum, kullanıcıları yüksek uzamsal çözünürlüklü çok bantlı görüntüler elde edebilme konusunda yeni arayışlar içine sokmaktadır. Pan-keskinleştirme işlemi bu probleme etkin bir çözüm sunmaktadır. Pan-keskinleştirme, yüksek uzamsal çözünürlüklü bir pankromatik görüntünün uzamsal detaylarının, yüksek spektral çözünürlüklü çok bantlı bir görüntüye aktarılarak uzamsal çözünürlüğü yüksek çok bantlı bir görüntü üretilmesi işlemidir. Literatürde pan-keskinleştirme için oldukça fazla sayıda yöntem geliştirilmiştir. Bu yöntemlerin her birinin kendine has avantaj ve dezavantajları vardır. Bu durum, kullanıcıları hangi durumda hangi yöntemin kullanılması gerektiği hususunda tereddüte düşürmektedir. Genel amacı, literatürdeki çeşitli konvansiyonel ve gelişmiş pan-keskinleştirme yöntemleri hakkında teorik bilgiler vermek ve bu yöntemlerin hangi durumlarda kullanılabileceği hususunda analistlere yol göstermek olan bu çalışmanın, pan-keskinleştirme hakkında iyi bir rehber olacağı kanaatindeyiz. Çalışmada, ayrıca pan-keskinleştirilmiş görüntülerin spektral ve uzamsal detay kalitelerinin görsel ve sayısal olarak nasıl değerlendirilebileceği hakkında da bilgiler verilmiştir.
References
- Addesso, P., Conte, R., Longo, M., Restaino, R. and Vivone, G. (2012). A pansharpening algorithm based on genetic optimization of Morphological Filters. International Geoscience and Remote Sensing Symposium, Munich, Germany.
- Aiazzi, B., Alparone, L., Baronti, S., Carlà, R., Garzelli, A. and Santurri, L. (2014). Full-scale assessment of pansharpening methods and data products. Image and Signal Processing for Remote Sensing XX, vol. 9244, p. 924402, Amsterdam, Netherlands. https://doi.org/10.1117/12.2067770.
- Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A. and Selva, M. (2006). MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogrammetric Engineering & Remote Sensing, 72(5), 591-596. https://doi.org/10.14358/PERS.72.5.591.
- Aiazzi, B., Baronti, S. and Selva, M. (2007). Improving component substitution pansharpening through multivariate regression of MS + Pan data. IEEE Transactions on Geoscience and Remote Sensing, 45(10), 3230-3239. https://doi.org/10.1109/TGRS.2007.901007.
- Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A. and Nencini, F. (2006). Information-Theoretic Image Fusion Assessment without Reference. ESA-EUSC 2006: Image Information Mining for Security and Intelligence, Torrejon Air Base, Madrid, Spain.
- Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F. and Selva, M. (2008). Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering & Remote Sensing, 74(2), 193-200. https://doi.org/10.14358/PERS.74.2.193.
- Alparone, L., Baronti, S., Garzelli, A. and Nencini, F. (2004). A global quality measurement of pan-sharpened multispectral imagery. IEEE Geoscience and Remote Sensing Letters, 1(4), 313-317. https://doi.org/10.1109/LGRS.2004.836784.
- Amro, I., Mateos, J., Vega, M., Molina, R. and Katsaggelos, A. K. (2011). A survey of classical methods and new trends in pansharpening of multispectral images. EURASIP Journal on Advances in Signal Processing, 2011(1), 1-22. https://doi.org/10.1186/1687-6180-2011-79.
- Bai, X., Zhou, F. and Xue, B. (2011). Fusion of infrared and visual images through region extraction by using multi scale center-surround top-hat transform. Optics Express, 19(9), 8444-8457. https://doi.org/10.1364/OE.19.008444.
- Benzenati, T., Kallel, A. and Kessentini, Y. (2020). Two stages pan-sharpening details injection approach based on very deep residual networks. IEEE Transactions on Geoscience and Remote Sensing, 59(6), 4984-4992. https://doi.org/10.1109/TGRS.2020.3019835.
- Censor, Y. (1977). Pareto optimality in multiobjective problems. Applied Mathematics and Optimization, 4(1), 41-59. https://doi.org/10.1007/BF01442131.
- Cheng, M. Y. and Prayogo, D. (2014). Symbiotic organisms search: a new metaheuristic optimization algorithm. Computers & Structures, 139, 98-112. https://doi.org/10.1016/j.compstruc.2014.03.007.
- Crippen, R. E. (1989). A simple spatial filtering routine for the cosmetic removal of scan-line noise from Landsat TM P-tape imagery. Photogrammetric Engineering and Remote Sensing, 55(3), 327-331.
- de Béthune, S., Muller, F. and Binard, M. (1997). Adaptive intensity matching filters: a new tool for multiresolution data fusion. Multi-Sensor Systems and Data Fusion for Telecommunications, Remote Sensing and Radar, Lisbon, Portugal.
- Deb, K., Pratap, A., Agarwal, S. and Meyarivan, T. A. M. T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182-197. https://doi.org/10.1109/4235.996017.
- Delleji, T., Kallel, A. and Ben Hamida, A. (2016). Iterative scheme for MS image pansharpening based on the combination of multi-resolution decompositions. International Journal of Remote Sensing, 37(24), 6041-6075. https://doi.org/10.1080/01431161.2016.1249303.
- Dou, W. (2018). Image degradation for quality assessment of pan-sharpening methods. Remote Sensing, 10(1), 154. https://doi.org/10.3390/rs10010154.
- Ehlers, M. (2004). Spectral characteristics preserving image fusion based on Fourier domain filtering. In Remote sensing for environmental monitoring, GIS applications, and geology IV (vol. 5574, pp. 1-13). International Society for Optics and Photonics. https://doi.org/10.1117/12.565160.
- El-Samie, F. E. A., Hadhoud, M. M. and El-Khamy, S. E. (2012). Image super-resolution and applications. CRC press.
- Farwell, L. S., Gudex-Cross, D., Anise, I. E., Bosch, M. J., Olah, A. M., Radeloff, V. C., Razenkova, E., Rogova, N., Silviera, E. M. O., Smith, M. M. and Pidgeon, A. M. (2021). Satellite image texture captures vegetation heterogeneity and explains patterns of bird richness. Remote Sensing of Environment, 253, 112175. https://doi.org/10.1016/j.rse.2020.112175.
- Fei, R., Zhang, J., Liu, J., Du, F., Chang, P., & Hu, J. (2019). Convolutional sparse representation of injected details for pansharpening. IEEE Geoscience and Remote Sensing Letters, 16(10), 1595-1599. https://doi.org/10.1109/LGRS.2019.2904526.
- Garzelli, A. and Nencini, F. (2006). PAN‐sharpening of very high resolution multispectral images using genetic algorithms. International Journal of Remote Sensing, 27(15), 3273-3292. https://doi.org/10.1080/01431160600554991.
- Garzelli, A. and Nencini, F. (2009). Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geoscience and Remote Sensing Letters, 6(4), 662-665. https://doi.org/10.1109/LGRS.2009.2022650.
- Ghahremani, M., Liu, Y., Yuen, P. and Behera, A. (2019). Remote sensing image fusion via compressive sensing. ISPRS Journal of Photogrammetry and Remote Sensing, 152, 34-48. https://doi.org/10.1016/j.isprsjprs.2019.04.001.
- Ghassemian, H. (2016). A review of remote sensing image fusion methods. Information Fusion, 32, 75-89. https://doi.org/10.1016/j.inffus.2016.03.003.
- González-Audícana, M., Saleta, J. L., Catalán, R. G. and García, R. (2004). Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing, 42(6), 1291-1299. https://doi.org/10.1109/TGRS.2004.825593.
- Gungor, O. (2008). Multi sensor multi resolution image fusion. PhD Thesis, Purdue University, USA.
- Hallabia, H., Kallel, A. and Hamida, A. B. (2014). Image pansharpening: Comparison of methods based on multiresolution analysis and component substitution. IEEE 1st International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Sousse, Tunisia. https://doi.org/10.1109/ATSIP.2014.6834602.
- Hallada, W. A. and Cox, S. (1983). Image sharpening for mixed spatial and spectral resolution satellite systems. 17th International Symposium on Remote Sensing of Environment (pp. 1023-1032), Ann Arbor, MI.
- Haydn, R., Dalke, G. W., Henkel, J. and Bare, J. E. (1982). Application of the IHS color transform to the processing of multisensor data and image enhancement. International Symposium on Remote Sensing of Environment, First Thematic Conference: ‘Remote Sensing of Arid and Semi-arid Lands’ (pp. 599-616), Cairo, Egypt.
- Holland, H. (1975). Adaptation in Natural and Artificial Systems. Ann Arbor (MI), The University of Michigan Press.
- Jiang, C., Zhang, H., Shen, H. and Zhang, L. (2012). A practical compressed sensing-based pan-sharpening method. IEEE Geoscience and Remote Sensing Letters, 9(4), 629-633. https://doi.org/10.1109/LGRS.2011.2177063.
- Jiang, M., Shen, H., Li, J., Yuan, Q. and Zhang, L. (2020). A differential information residual convolutional neural network for pansharpening. ISPRS Journal of Photogrammetry and Remote Sensing, 163, 257-271. https://doi.org/10.1016/j.isprsjprs.2020.03.006.
- Jinju, J., Santhi, N., Ramar, K. and Bama, B. S. (2019). Spatial frequency discrete wavelet transform image fusion technique for remote sensing applications. Engineering Science and Technology, an International Journal, 22(3), 715-726. https://doi.org/10.1016/j.jestch.2019.01.004.
- Kamir, E., Waldner, F. and Hochman, Z. (2020). Estimating wheat yields in Australia using climate records, satellite image time series and machine learning methods. ISPRS Journal of Photogrammetry and Remote Sensing, 160, 124-135. https://doi.org/10.1016/j.isprsjprs.2019.11.008.
- Khan, M. M., Alparone, L. and Chanussot, J. (2009). Pansharpening quality assessment using the modulation transfer functions of instruments. IEEE Transactions on Geoscience and Remote Sensing, 47(11), 3880-3891. https://doi.org/10.1109/TGRS.2009.2029094.
- Khan, M. M., Chanussot, J., Condat, L. and Montanvert, A. (2008). Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geoscience and Remote Sensing Letters, 5(1), 98-102. https://doi.org/10.1109/LGRS.2007.909934.
- Khatancharoen, C., Tsuyuki, S., Bryanin, S. V., Sugiura, K., Seino, T., Lisovsky, V. V., Borisova, I. G. and Wada, N. (2021). Long-Time Interval Satellite Image Analysis on Forest-Cover Changes and Disturbances Around Protected Area, Zeya State Nature Reserve, in The Russian Far East. Remote Sensing, 13(7), 1285. https://doi.org/10.3390/rs13071285.
- Klonus, S. and Ehlers, M. (2007). Image fusion using the Ehlers spectral characteristics preservation algorithm. GIScience & Remote Sensing, 44(2), 93-116. https://doi.org/10.2747/1548-1603.44.2.93.
- Kwarteng, P. and Chavez, A. (1989). Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing, 55(1), 339-348.
- Laben, C. A. and Brower, B. V. (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S. Patent No. 6,011,875.
- Li, S. and Yang, B. (2011). A new pan-sharpening method using a compressed sensing technique. IEEE Transactions on Geoscience and Remote Sensing, 49(2), 738-746. https://doi.org/10.1109/TGRS.2010.2067219.
- Liu, J. G. (2000). Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing, 21(18), 3461-3472. https://doi.org/10.1080/014311600750037499.
- Liu, J., Huang, J., Liu, S., Li, H., Zhou, Q. and Liu, J. (2015). Human visual system consistent quality assessment for remote sensing image fusion. ISPRS Journal of Photogrammetry and Remote Sensing, 105, 79-90. https://doi.org/10.1016/j.isprsjprs.2014.12.018.
- Liu, X., Zhang, Q., Li, Y., Tan, Z. and Werner, A. D. (2020). Satellite image-based investigation of the seasonal variations in the hydrological connectivity of a large floodplain (Poyang Lake, China). Journal of Hydrology, 585, 124810. https://doi.org/10.1016/j.jhydrol.2020.124810.
- Masi, G., Cozzolino, D., Verdoliva, L. and Scarpa, G. (2016). Pansharpening by convolutional neural networks. Remote Sensing, 8(7), 594. https://doi.org/10.3390/rs8070594.
- Munechika, C. K., Warnick, J. S., Salvaggio, C. and Schott, J. R. (1993). Resolution enhancement of multispectral image data to improve classification accuracy. Photogrammetric Engineering and Remote Sensing, 59(1), 67-72.
- Núñez, J., Otazu, X., Fors, O., Prades, A., Pala, V. and Arbiol, R. (1999). Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing, 37(3), 1204-1211. https://doi.org/10.1109/36.763274.
- Otazu, X., González-Audícana, M., Fors, O. and Núñez, J. (2005). Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10), 2376-2385. https://doi.org/10.1109/TGRS.2005.856106.
- Ozcelik, F., Alganci, U., Sertel, E. and Unal, G. (2020). Rethinking CNN-based pansharpening: Guided colorization of panchromatic images via GANS. IEEE Transactions on Geoscience and Remote Sensing, 59(4), 3486-3501. https://doi.org/10.1109/TGRS.2020.3010441.
- Padwick, C., Deskevich, M., Pacifici, F. and Smallwood, S. (2010). WorldView-2 pan-sharpening. ASPRS 2010 Annual Conference (pp. 1-14), San Diego, CA, USA.
- Pandit, V. R. and Bhiwani, R. J. (2020). Multispectral to Panchromatic Image Fusion Based on Morphological Extended-Half-Gradient. Journal of the Indian Society of Remote Sensing, 48(6), 945-957. https://doi.org/10.1007/s12524-020-01127-2.
- Pohl, C. and van Genderen, J. (2016). Remote sensing image fusion: A practical guide. Crc Press.
- Pohl, C. and van Genderen, J. L. (1998). Review article multisensor image fusion in remote sensing: concepts, methods and applications. International Journal of Remote Sensing, 19(5), 823-854. https://doi.org/10.1080/014311698215748.
- Ranchin, T. and Wald, L. (2000). Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation. Photogrammetric Engineering and Remote Sensing, 66(1), 49-61.
- Restaino, R., Vivone, G., Dalla Mura, M. and Chanussot, J. (2015). A pansharpening algorithm based on morphological filters. In International Symposium on Mathematical Morphology and its Applications to Signal and Image Processing (pp. 98-109). Springer, Cham.
- Restaino, R., Vivone, G., Dalla Mura, M. and Chanussot, J. (2016). Fusion of multispectral and panchromatic images based on morphological operators. IEEE Transactions on Image Processing, 25(6), 2882-2895. https://doi.org/10.1109/TIP.2016.2556944.
- Schowengerdt, R. A. (1980). Reconstruction of multispatial, multispectral image data using spatial frequency content. Photogrammetric Engineering and Remote Sensing, 46(10), 1325-1334.
- Serifoglu Yilmaz, C. (2020). Metaheuristic pansharpening based on symbiotic organisms search optimization. PhD Thesis, Karadeniz Technical University, Trabzon.
- Serifoglu Yilmaz, C., Yilmaz, V. and Güngör, O. (2020). On the use of the SOS metaheuristic algorithm in hybrid image fusion methods to achieve optimum spectral fidelity. International Journal of Remote Sensing, 41(10), 3993-4021. https://doi.org/10.1080/01431161.2019.1711244.
- Serifoglu Yilmaz, C., Yilmaz, V., Gungor, O. and Shan, J. (2019). Metaheuristic pansharpening based on symbiotic organisms search optimization. ISPRS Journal of Photogrammetry and Remote Sensing, 158, 167-187. https://doi.org/10.1016/j.isprsjprs.2019.10.014.
- Shi, Y. (2018). A new pansharpening algorithm using morphological lifting transform. IEEE 3rd International Conference on Signal and Image Processing (ICSIP) (pp. 250-254), Shenzhen, China. https://doi.org/10.1109/SIPROCESS.2018.8600445.
- Siddiqui, Y. (2003). The modified IHS method for fusing satellite imagery. ASPRS 2003 Annual Conference (pp. 5-9), Anchorage, Alaska.
- Strait, M., Rahmani, S. and Markurjev, D. (2008). Evaluation of Pan-Sharpening Methods. UCLA Department of Mathematics.
- Vicinanza, M. R., Restaino, R., Vivone, G., Dalla Mura, M. and Chanussot, J. (2015). A pansharpening method based on the sparse representation of injected details. IEEE Geoscience and Remote Sensing Letters, 12(1), 180-184. https://doi.org/10.1109/LGRS.2014.2331291.
- Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G. A., Restaino, R. and Wald, L. (2015). A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5), 2565-2586. https://doi.org/10.1109/TGRS.2014.2361734.
- Vivone, G., Restaino, R., Dalla Mura, M., Licciardi, G. and Chanussot, J. (2013). Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geoscience and Remote Sensing Letters, 11(5), 930-934. https://doi.org/10.1109/LGRS.2013.2281996.
- Wald, L. (2000). Quality of High Resolution Synthesized Images: Is There a Simple Criterion? 3rd Conference Fusion Of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images (pp. 99-103), Sophia Antipolis, France.
- Wald, L. and Ranchin, T. (2002). Liu'Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details'. International Journal of Remote Sensing, 23(3), 593-597. https://doi.org/10.1080/01431160110088772.
- Wald, L., Ranchin, T. and Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6), 691-699.
- Wang, F., Lu, X., Mei, S., Su, Y., Zhen, Z., Zou, Z., Zhang, X., Yin, R., Duic, N., Shafie-khah, M. and Catalão, J. P. S. (2021). A Satellite Image Data based Ultra-short-term Solar PV Power Forecasting Method Considering Cloud Information from Neighboring Plant. Energy, 121946. https://doi.org/10.1016/j.energy.2021.121946.
- Wang, Z. and Bovik, A. C. (2002). A universal image quality index. IEEE Signal Processing Letters, 9(3), 81-84. https://doi.org/10.1109/97.995823.
- Wang, Z. and Li, Q. (2011). Information content weighting for perceptual image quality assessment. IEEE Transactions on Image Processing, 20(5), 1185-1198. https://doi.org/10.1109/TIP.2010.2092435.
- Wang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612. https://doi.org/10.1109/TIP.2003.819861.
- Wang, Z., Simoncelli, E. P. and Bovik, A. C. (2003). Multiscale structural similarity for image quality assessment. 37th IEEE Asilomar Conference on Signals, Systems & Computers (pp. 1398-1402), Pacific Grove, CA, USA. https://doi.org/10.1109/ACSSC.2003.1292216.
- Xing, Y., Wang, M., Yang, S. and Jiao, L. (2018). Pan-sharpening via deep metric learning. ISPRS Journal of Photogrammetry and Remote Sensing, 145, 165-183. https://doi.org/10.1016/j.isprsjprs.2018.01.016.
- Xing, Y., Wang, M., Yang, S. and Zhang, K. (2018). Pansharpening with multiscale geometric support tensor machine. IEEE Transactions on Geoscience and Remote Sensing, 56(5), 2503-2517. https://doi.org/10.1109/TGRS.2017.2742002.
- Yang, C., Zhan, Q., Liu, H. and Ma, R. (2018). An IHS-based pan-sharpening method for spectral fidelity improvement using ripplet transform and compressed sensing. Sensors, 18(11), 3624. https://doi.org/10.3390/s18113624.
- Yilmaz, V. (2021). A Non‐Dominated Sorting Genetic Algorithm‐II‐based approach to optimize the spectral and spatial quality of component substitution‐based pansharpened images. Concurrency and Computation: Practice and Experience, 33(5), e6030. https://doi.org/10.1002/cpe.6030.
- Yilmaz, V. and Gungor, O. (2016a). Determining the optimum image fusion method for better interpretation of the surface of the Earth. Norsk Geografisk Tidsskrift-Norwegian Journal of Geography, 70(2), 69-81. https://doi.org/10.1080/00291951.2015.1126761.
- Yilmaz, V. and Gungor, O. (2016b). Fusion of very high-resolution UAV images with criteria-based image fusion algorithm. Arabian Journal of Geosciences, 9(1), 1-16. https://doi.org/10.1007/s12517-015-2109-8.
- Yilmaz, V., Serifoglu Yilmaz, C. and Gungor, O. (2021). Genetic algorithm-based synthetic variable ratio image fusion. Geocarto International, 36(9), 989-1006. https://doi.org/10.1080/10106049.2019.1629649.
- Yilmaz, V., Serifoglu Yilmaz, C., Güngör, O. and Shan, J. (2020). A genetic algorithm solution to the gram-schmidt image fusion. International Journal of Remote Sensing, 41(4), 1458-1485. https://doi.org/10.1080/01431161.2019.1667553.
- Yin, H. (2015). Sparse representation based pansharpening with details injection model. Signal Processing, 113, 218-227. https://doi.org/10.1016/j.sigpro.2014.12.017.
- Yusuf, Y., Sri Sumantyo, J. T. and Kuze, H. (2013). Spectral information analysis of image fusion data for remote sensing applications. Geocarto International, 28(4), 291-310. https://doi.org/10.1080/10106049.2012.692396.
- Zhang, L. and Li, H. (2012). SR-SIM: A fast and high performance IQA index based on spectral residual. 19th IEEE International Conference on Image Processing (pp. 1473-1476), Orlando, FL, USA. https://doi.org/10.1109/ICIP.2012.6467149.
- Zhang, L., Li, W., Shen, L. and Lei, D. (2020). Multilevel dense neural network for pan-sharpening. International Journal of Remote Sensing, 41(18), 7217-7232. https://doi.org/10.1080/01431161.2020.1755474.
- Zhou, J., Civco, D. L. and Silander, J. A. (1998). A wavelet transform method to merge Landsat TM and SPOT panchromatic data. International Journal of Remote Sensing, 19(4), 743-757. https://doi.org/10.1080/014311698215973.
- Zhu, X. X. and Bamler, R. (2013). A sparse image fusion algorithm with application to pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 51(5), 2827-2836. https://doi.org/10.1109/TGRS.2012.2213604.
A review on pansharpening of multispectral images
Year 2021,
Volume: 11 Issue: 4, 1340 - 1357, 15.10.2021
Çiğdem Şerifoğlu Yılmaz
,
Volkan Yılmaz
,
Oğuz Güngör
Abstract
Remote sensing satellites cannot produce images of high spatial detail quality and spectral quality due to technical limitations in their sensors, which forces users to find alternative ways to produce such images. Pan-sharpening offers an effective solution to this problem. Pan-sharpening aims to transfer the spatial details of a high-resolution panchromatic image into a high spectral resolution image, producing a multispectral image of high spatial resolution. A wide variety of pansharpening methods have been proposed in the litreture. Each pansharpening method has its own advantages and disadvantages. This situation makes users hesitant about which method should be used under what situation. We believe that this study, whose primary objective is to provide theoretical information about various conventional and state-of-the-art pan-sharpening methods in the literature, and to guide the analysts as to which pansharpening methods should be used under what circumstances, will be a good pan-sharpening guide. This study also provides information on how the spatial and spectral quality of pan-sharpened images may be evaluated qualitatively and quantitatively.
References
- Addesso, P., Conte, R., Longo, M., Restaino, R. and Vivone, G. (2012). A pansharpening algorithm based on genetic optimization of Morphological Filters. International Geoscience and Remote Sensing Symposium, Munich, Germany.
- Aiazzi, B., Alparone, L., Baronti, S., Carlà, R., Garzelli, A. and Santurri, L. (2014). Full-scale assessment of pansharpening methods and data products. Image and Signal Processing for Remote Sensing XX, vol. 9244, p. 924402, Amsterdam, Netherlands. https://doi.org/10.1117/12.2067770.
- Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A. and Selva, M. (2006). MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogrammetric Engineering & Remote Sensing, 72(5), 591-596. https://doi.org/10.14358/PERS.72.5.591.
- Aiazzi, B., Baronti, S. and Selva, M. (2007). Improving component substitution pansharpening through multivariate regression of MS + Pan data. IEEE Transactions on Geoscience and Remote Sensing, 45(10), 3230-3239. https://doi.org/10.1109/TGRS.2007.901007.
- Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A. and Nencini, F. (2006). Information-Theoretic Image Fusion Assessment without Reference. ESA-EUSC 2006: Image Information Mining for Security and Intelligence, Torrejon Air Base, Madrid, Spain.
- Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F. and Selva, M. (2008). Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering & Remote Sensing, 74(2), 193-200. https://doi.org/10.14358/PERS.74.2.193.
- Alparone, L., Baronti, S., Garzelli, A. and Nencini, F. (2004). A global quality measurement of pan-sharpened multispectral imagery. IEEE Geoscience and Remote Sensing Letters, 1(4), 313-317. https://doi.org/10.1109/LGRS.2004.836784.
- Amro, I., Mateos, J., Vega, M., Molina, R. and Katsaggelos, A. K. (2011). A survey of classical methods and new trends in pansharpening of multispectral images. EURASIP Journal on Advances in Signal Processing, 2011(1), 1-22. https://doi.org/10.1186/1687-6180-2011-79.
- Bai, X., Zhou, F. and Xue, B. (2011). Fusion of infrared and visual images through region extraction by using multi scale center-surround top-hat transform. Optics Express, 19(9), 8444-8457. https://doi.org/10.1364/OE.19.008444.
- Benzenati, T., Kallel, A. and Kessentini, Y. (2020). Two stages pan-sharpening details injection approach based on very deep residual networks. IEEE Transactions on Geoscience and Remote Sensing, 59(6), 4984-4992. https://doi.org/10.1109/TGRS.2020.3019835.
- Censor, Y. (1977). Pareto optimality in multiobjective problems. Applied Mathematics and Optimization, 4(1), 41-59. https://doi.org/10.1007/BF01442131.
- Cheng, M. Y. and Prayogo, D. (2014). Symbiotic organisms search: a new metaheuristic optimization algorithm. Computers & Structures, 139, 98-112. https://doi.org/10.1016/j.compstruc.2014.03.007.
- Crippen, R. E. (1989). A simple spatial filtering routine for the cosmetic removal of scan-line noise from Landsat TM P-tape imagery. Photogrammetric Engineering and Remote Sensing, 55(3), 327-331.
- de Béthune, S., Muller, F. and Binard, M. (1997). Adaptive intensity matching filters: a new tool for multiresolution data fusion. Multi-Sensor Systems and Data Fusion for Telecommunications, Remote Sensing and Radar, Lisbon, Portugal.
- Deb, K., Pratap, A., Agarwal, S. and Meyarivan, T. A. M. T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182-197. https://doi.org/10.1109/4235.996017.
- Delleji, T., Kallel, A. and Ben Hamida, A. (2016). Iterative scheme for MS image pansharpening based on the combination of multi-resolution decompositions. International Journal of Remote Sensing, 37(24), 6041-6075. https://doi.org/10.1080/01431161.2016.1249303.
- Dou, W. (2018). Image degradation for quality assessment of pan-sharpening methods. Remote Sensing, 10(1), 154. https://doi.org/10.3390/rs10010154.
- Ehlers, M. (2004). Spectral characteristics preserving image fusion based on Fourier domain filtering. In Remote sensing for environmental monitoring, GIS applications, and geology IV (vol. 5574, pp. 1-13). International Society for Optics and Photonics. https://doi.org/10.1117/12.565160.
- El-Samie, F. E. A., Hadhoud, M. M. and El-Khamy, S. E. (2012). Image super-resolution and applications. CRC press.
- Farwell, L. S., Gudex-Cross, D., Anise, I. E., Bosch, M. J., Olah, A. M., Radeloff, V. C., Razenkova, E., Rogova, N., Silviera, E. M. O., Smith, M. M. and Pidgeon, A. M. (2021). Satellite image texture captures vegetation heterogeneity and explains patterns of bird richness. Remote Sensing of Environment, 253, 112175. https://doi.org/10.1016/j.rse.2020.112175.
- Fei, R., Zhang, J., Liu, J., Du, F., Chang, P., & Hu, J. (2019). Convolutional sparse representation of injected details for pansharpening. IEEE Geoscience and Remote Sensing Letters, 16(10), 1595-1599. https://doi.org/10.1109/LGRS.2019.2904526.
- Garzelli, A. and Nencini, F. (2006). PAN‐sharpening of very high resolution multispectral images using genetic algorithms. International Journal of Remote Sensing, 27(15), 3273-3292. https://doi.org/10.1080/01431160600554991.
- Garzelli, A. and Nencini, F. (2009). Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geoscience and Remote Sensing Letters, 6(4), 662-665. https://doi.org/10.1109/LGRS.2009.2022650.
- Ghahremani, M., Liu, Y., Yuen, P. and Behera, A. (2019). Remote sensing image fusion via compressive sensing. ISPRS Journal of Photogrammetry and Remote Sensing, 152, 34-48. https://doi.org/10.1016/j.isprsjprs.2019.04.001.
- Ghassemian, H. (2016). A review of remote sensing image fusion methods. Information Fusion, 32, 75-89. https://doi.org/10.1016/j.inffus.2016.03.003.
- González-Audícana, M., Saleta, J. L., Catalán, R. G. and García, R. (2004). Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing, 42(6), 1291-1299. https://doi.org/10.1109/TGRS.2004.825593.
- Gungor, O. (2008). Multi sensor multi resolution image fusion. PhD Thesis, Purdue University, USA.
- Hallabia, H., Kallel, A. and Hamida, A. B. (2014). Image pansharpening: Comparison of methods based on multiresolution analysis and component substitution. IEEE 1st International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Sousse, Tunisia. https://doi.org/10.1109/ATSIP.2014.6834602.
- Hallada, W. A. and Cox, S. (1983). Image sharpening for mixed spatial and spectral resolution satellite systems. 17th International Symposium on Remote Sensing of Environment (pp. 1023-1032), Ann Arbor, MI.
- Haydn, R., Dalke, G. W., Henkel, J. and Bare, J. E. (1982). Application of the IHS color transform to the processing of multisensor data and image enhancement. International Symposium on Remote Sensing of Environment, First Thematic Conference: ‘Remote Sensing of Arid and Semi-arid Lands’ (pp. 599-616), Cairo, Egypt.
- Holland, H. (1975). Adaptation in Natural and Artificial Systems. Ann Arbor (MI), The University of Michigan Press.
- Jiang, C., Zhang, H., Shen, H. and Zhang, L. (2012). A practical compressed sensing-based pan-sharpening method. IEEE Geoscience and Remote Sensing Letters, 9(4), 629-633. https://doi.org/10.1109/LGRS.2011.2177063.
- Jiang, M., Shen, H., Li, J., Yuan, Q. and Zhang, L. (2020). A differential information residual convolutional neural network for pansharpening. ISPRS Journal of Photogrammetry and Remote Sensing, 163, 257-271. https://doi.org/10.1016/j.isprsjprs.2020.03.006.
- Jinju, J., Santhi, N., Ramar, K. and Bama, B. S. (2019). Spatial frequency discrete wavelet transform image fusion technique for remote sensing applications. Engineering Science and Technology, an International Journal, 22(3), 715-726. https://doi.org/10.1016/j.jestch.2019.01.004.
- Kamir, E., Waldner, F. and Hochman, Z. (2020). Estimating wheat yields in Australia using climate records, satellite image time series and machine learning methods. ISPRS Journal of Photogrammetry and Remote Sensing, 160, 124-135. https://doi.org/10.1016/j.isprsjprs.2019.11.008.
- Khan, M. M., Alparone, L. and Chanussot, J. (2009). Pansharpening quality assessment using the modulation transfer functions of instruments. IEEE Transactions on Geoscience and Remote Sensing, 47(11), 3880-3891. https://doi.org/10.1109/TGRS.2009.2029094.
- Khan, M. M., Chanussot, J., Condat, L. and Montanvert, A. (2008). Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geoscience and Remote Sensing Letters, 5(1), 98-102. https://doi.org/10.1109/LGRS.2007.909934.
- Khatancharoen, C., Tsuyuki, S., Bryanin, S. V., Sugiura, K., Seino, T., Lisovsky, V. V., Borisova, I. G. and Wada, N. (2021). Long-Time Interval Satellite Image Analysis on Forest-Cover Changes and Disturbances Around Protected Area, Zeya State Nature Reserve, in The Russian Far East. Remote Sensing, 13(7), 1285. https://doi.org/10.3390/rs13071285.
- Klonus, S. and Ehlers, M. (2007). Image fusion using the Ehlers spectral characteristics preservation algorithm. GIScience & Remote Sensing, 44(2), 93-116. https://doi.org/10.2747/1548-1603.44.2.93.
- Kwarteng, P. and Chavez, A. (1989). Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing, 55(1), 339-348.
- Laben, C. A. and Brower, B. V. (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S. Patent No. 6,011,875.
- Li, S. and Yang, B. (2011). A new pan-sharpening method using a compressed sensing technique. IEEE Transactions on Geoscience and Remote Sensing, 49(2), 738-746. https://doi.org/10.1109/TGRS.2010.2067219.
- Liu, J. G. (2000). Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing, 21(18), 3461-3472. https://doi.org/10.1080/014311600750037499.
- Liu, J., Huang, J., Liu, S., Li, H., Zhou, Q. and Liu, J. (2015). Human visual system consistent quality assessment for remote sensing image fusion. ISPRS Journal of Photogrammetry and Remote Sensing, 105, 79-90. https://doi.org/10.1016/j.isprsjprs.2014.12.018.
- Liu, X., Zhang, Q., Li, Y., Tan, Z. and Werner, A. D. (2020). Satellite image-based investigation of the seasonal variations in the hydrological connectivity of a large floodplain (Poyang Lake, China). Journal of Hydrology, 585, 124810. https://doi.org/10.1016/j.jhydrol.2020.124810.
- Masi, G., Cozzolino, D., Verdoliva, L. and Scarpa, G. (2016). Pansharpening by convolutional neural networks. Remote Sensing, 8(7), 594. https://doi.org/10.3390/rs8070594.
- Munechika, C. K., Warnick, J. S., Salvaggio, C. and Schott, J. R. (1993). Resolution enhancement of multispectral image data to improve classification accuracy. Photogrammetric Engineering and Remote Sensing, 59(1), 67-72.
- Núñez, J., Otazu, X., Fors, O., Prades, A., Pala, V. and Arbiol, R. (1999). Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing, 37(3), 1204-1211. https://doi.org/10.1109/36.763274.
- Otazu, X., González-Audícana, M., Fors, O. and Núñez, J. (2005). Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10), 2376-2385. https://doi.org/10.1109/TGRS.2005.856106.
- Ozcelik, F., Alganci, U., Sertel, E. and Unal, G. (2020). Rethinking CNN-based pansharpening: Guided colorization of panchromatic images via GANS. IEEE Transactions on Geoscience and Remote Sensing, 59(4), 3486-3501. https://doi.org/10.1109/TGRS.2020.3010441.
- Padwick, C., Deskevich, M., Pacifici, F. and Smallwood, S. (2010). WorldView-2 pan-sharpening. ASPRS 2010 Annual Conference (pp. 1-14), San Diego, CA, USA.
- Pandit, V. R. and Bhiwani, R. J. (2020). Multispectral to Panchromatic Image Fusion Based on Morphological Extended-Half-Gradient. Journal of the Indian Society of Remote Sensing, 48(6), 945-957. https://doi.org/10.1007/s12524-020-01127-2.
- Pohl, C. and van Genderen, J. (2016). Remote sensing image fusion: A practical guide. Crc Press.
- Pohl, C. and van Genderen, J. L. (1998). Review article multisensor image fusion in remote sensing: concepts, methods and applications. International Journal of Remote Sensing, 19(5), 823-854. https://doi.org/10.1080/014311698215748.
- Ranchin, T. and Wald, L. (2000). Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation. Photogrammetric Engineering and Remote Sensing, 66(1), 49-61.
- Restaino, R., Vivone, G., Dalla Mura, M. and Chanussot, J. (2015). A pansharpening algorithm based on morphological filters. In International Symposium on Mathematical Morphology and its Applications to Signal and Image Processing (pp. 98-109). Springer, Cham.
- Restaino, R., Vivone, G., Dalla Mura, M. and Chanussot, J. (2016). Fusion of multispectral and panchromatic images based on morphological operators. IEEE Transactions on Image Processing, 25(6), 2882-2895. https://doi.org/10.1109/TIP.2016.2556944.
- Schowengerdt, R. A. (1980). Reconstruction of multispatial, multispectral image data using spatial frequency content. Photogrammetric Engineering and Remote Sensing, 46(10), 1325-1334.
- Serifoglu Yilmaz, C. (2020). Metaheuristic pansharpening based on symbiotic organisms search optimization. PhD Thesis, Karadeniz Technical University, Trabzon.
- Serifoglu Yilmaz, C., Yilmaz, V. and Güngör, O. (2020). On the use of the SOS metaheuristic algorithm in hybrid image fusion methods to achieve optimum spectral fidelity. International Journal of Remote Sensing, 41(10), 3993-4021. https://doi.org/10.1080/01431161.2019.1711244.
- Serifoglu Yilmaz, C., Yilmaz, V., Gungor, O. and Shan, J. (2019). Metaheuristic pansharpening based on symbiotic organisms search optimization. ISPRS Journal of Photogrammetry and Remote Sensing, 158, 167-187. https://doi.org/10.1016/j.isprsjprs.2019.10.014.
- Shi, Y. (2018). A new pansharpening algorithm using morphological lifting transform. IEEE 3rd International Conference on Signal and Image Processing (ICSIP) (pp. 250-254), Shenzhen, China. https://doi.org/10.1109/SIPROCESS.2018.8600445.
- Siddiqui, Y. (2003). The modified IHS method for fusing satellite imagery. ASPRS 2003 Annual Conference (pp. 5-9), Anchorage, Alaska.
- Strait, M., Rahmani, S. and Markurjev, D. (2008). Evaluation of Pan-Sharpening Methods. UCLA Department of Mathematics.
- Vicinanza, M. R., Restaino, R., Vivone, G., Dalla Mura, M. and Chanussot, J. (2015). A pansharpening method based on the sparse representation of injected details. IEEE Geoscience and Remote Sensing Letters, 12(1), 180-184. https://doi.org/10.1109/LGRS.2014.2331291.
- Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G. A., Restaino, R. and Wald, L. (2015). A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5), 2565-2586. https://doi.org/10.1109/TGRS.2014.2361734.
- Vivone, G., Restaino, R., Dalla Mura, M., Licciardi, G. and Chanussot, J. (2013). Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geoscience and Remote Sensing Letters, 11(5), 930-934. https://doi.org/10.1109/LGRS.2013.2281996.
- Wald, L. (2000). Quality of High Resolution Synthesized Images: Is There a Simple Criterion? 3rd Conference Fusion Of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images (pp. 99-103), Sophia Antipolis, France.
- Wald, L. and Ranchin, T. (2002). Liu'Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details'. International Journal of Remote Sensing, 23(3), 593-597. https://doi.org/10.1080/01431160110088772.
- Wald, L., Ranchin, T. and Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6), 691-699.
- Wang, F., Lu, X., Mei, S., Su, Y., Zhen, Z., Zou, Z., Zhang, X., Yin, R., Duic, N., Shafie-khah, M. and Catalão, J. P. S. (2021). A Satellite Image Data based Ultra-short-term Solar PV Power Forecasting Method Considering Cloud Information from Neighboring Plant. Energy, 121946. https://doi.org/10.1016/j.energy.2021.121946.
- Wang, Z. and Bovik, A. C. (2002). A universal image quality index. IEEE Signal Processing Letters, 9(3), 81-84. https://doi.org/10.1109/97.995823.
- Wang, Z. and Li, Q. (2011). Information content weighting for perceptual image quality assessment. IEEE Transactions on Image Processing, 20(5), 1185-1198. https://doi.org/10.1109/TIP.2010.2092435.
- Wang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612. https://doi.org/10.1109/TIP.2003.819861.
- Wang, Z., Simoncelli, E. P. and Bovik, A. C. (2003). Multiscale structural similarity for image quality assessment. 37th IEEE Asilomar Conference on Signals, Systems & Computers (pp. 1398-1402), Pacific Grove, CA, USA. https://doi.org/10.1109/ACSSC.2003.1292216.
- Xing, Y., Wang, M., Yang, S. and Jiao, L. (2018). Pan-sharpening via deep metric learning. ISPRS Journal of Photogrammetry and Remote Sensing, 145, 165-183. https://doi.org/10.1016/j.isprsjprs.2018.01.016.
- Xing, Y., Wang, M., Yang, S. and Zhang, K. (2018). Pansharpening with multiscale geometric support tensor machine. IEEE Transactions on Geoscience and Remote Sensing, 56(5), 2503-2517. https://doi.org/10.1109/TGRS.2017.2742002.
- Yang, C., Zhan, Q., Liu, H. and Ma, R. (2018). An IHS-based pan-sharpening method for spectral fidelity improvement using ripplet transform and compressed sensing. Sensors, 18(11), 3624. https://doi.org/10.3390/s18113624.
- Yilmaz, V. (2021). A Non‐Dominated Sorting Genetic Algorithm‐II‐based approach to optimize the spectral and spatial quality of component substitution‐based pansharpened images. Concurrency and Computation: Practice and Experience, 33(5), e6030. https://doi.org/10.1002/cpe.6030.
- Yilmaz, V. and Gungor, O. (2016a). Determining the optimum image fusion method for better interpretation of the surface of the Earth. Norsk Geografisk Tidsskrift-Norwegian Journal of Geography, 70(2), 69-81. https://doi.org/10.1080/00291951.2015.1126761.
- Yilmaz, V. and Gungor, O. (2016b). Fusion of very high-resolution UAV images with criteria-based image fusion algorithm. Arabian Journal of Geosciences, 9(1), 1-16. https://doi.org/10.1007/s12517-015-2109-8.
- Yilmaz, V., Serifoglu Yilmaz, C. and Gungor, O. (2021). Genetic algorithm-based synthetic variable ratio image fusion. Geocarto International, 36(9), 989-1006. https://doi.org/10.1080/10106049.2019.1629649.
- Yilmaz, V., Serifoglu Yilmaz, C., Güngör, O. and Shan, J. (2020). A genetic algorithm solution to the gram-schmidt image fusion. International Journal of Remote Sensing, 41(4), 1458-1485. https://doi.org/10.1080/01431161.2019.1667553.
- Yin, H. (2015). Sparse representation based pansharpening with details injection model. Signal Processing, 113, 218-227. https://doi.org/10.1016/j.sigpro.2014.12.017.
- Yusuf, Y., Sri Sumantyo, J. T. and Kuze, H. (2013). Spectral information analysis of image fusion data for remote sensing applications. Geocarto International, 28(4), 291-310. https://doi.org/10.1080/10106049.2012.692396.
- Zhang, L. and Li, H. (2012). SR-SIM: A fast and high performance IQA index based on spectral residual. 19th IEEE International Conference on Image Processing (pp. 1473-1476), Orlando, FL, USA. https://doi.org/10.1109/ICIP.2012.6467149.
- Zhang, L., Li, W., Shen, L. and Lei, D. (2020). Multilevel dense neural network for pan-sharpening. International Journal of Remote Sensing, 41(18), 7217-7232. https://doi.org/10.1080/01431161.2020.1755474.
- Zhou, J., Civco, D. L. and Silander, J. A. (1998). A wavelet transform method to merge Landsat TM and SPOT panchromatic data. International Journal of Remote Sensing, 19(4), 743-757. https://doi.org/10.1080/014311698215973.
- Zhu, X. X. and Bamler, R. (2013). A sparse image fusion algorithm with application to pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 51(5), 2827-2836. https://doi.org/10.1109/TGRS.2012.2213604.