Learning Based Super Resolution Application for Hyperspectral Images

Abstract

Due to its spectral properties, hyperspectral imaging is superior to other types of imaging tools in identifying, distinguishing and classifying objects. Hyperspectral imaging instruments can detect light reflected from certain wavelengths between infrared and ultraviolet, apart from the wavelength that the human eye can distinguish on the electromagnetic spectrum. While this feature provides detailed information about the spectral feature of the object under investigation, it causes its spatial resolution to be low due to the technical overlap between spatial resolution and spectral resolution. Today, applications of hyperspectral images are increasing in important fields such as agriculture, mining, medicine and pharmacy, especially for military purposes. In order for applications to produce more precise results, high spatial resolution is required, as well as high spectral information. Hardware solving of low spatial resolution problem is a difficult and costly method. Therefore, software solution is an interesting area in the field of image processing. In this thesis, a hybrid solution method based on deep learning and sparse representation is proposed to increase the low spatial resolution of hyperspectral images. The method obtains a super-resolution image from a single hyperspectral image with a low spatial image with a deep convolutional mesh. Later, the super-resolution image obtained and the original low-spatial-resolution hyperspectral image are fused with the dictionary learning method, resulting in a new super-resolution image with high spectral and spatial resolutions. The application results show that our method achieves successful results compared to many super resolution applications in the literature.

Keywords

• Remote Sensing
• Hyperspectral Imaging
• Super-Resolution
• Deep Learning
• Convolutional Neural Networks
• Sparse Representation
• Dictionary Learning

References

1. [1] D. Wallach, F. Lamare, G. Kontaxakis, and D. Visvikis, “Super-Resolution in Respiratory Synchronized Positron Emission Tomography,” IEEE Trans. Med. Imaging, vol. 31, no. 2, pp. 438–448, Feb. 2012, doi: 10.1109/TMI.2011.2171358.
2. [2] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” Proc. IEEE, vol. 105, no. 12, pp. 2295–2329, Dec. 2017, doi: 10.1109/JPROC.2017.2761740.
3. [3] G. Ciaburro, V. K. Ayyadevara, and A. Perrier, Hands-on machine learning on Google cloud platform: implementing smart and efficient analytics using Cloud ML Engine. 2018.
4. [4] J. Li, T. Qiu, C. Wen, K. Xie, and F.-Q. Wen, “Robust Face Recognition Using the Deep C2D-CNN Model Based on Decision-Level Fusion,” Sensors, vol. 18, no. 7, p. 2080, Jun. 2018, doi: 10.3390/s18072080.
5. [5] H. Yakura, S. Shinozaki, R. Nishimura, Y. Oyama, and J. Sakuma, “Malware Analysis of Imaged Binary Samples by Convolutional Neural Network with Attention Mechanism,” in Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy, Tempe AZ USA, Mar. 2018, pp. 127–134. doi: 10.1145/3176258.3176335.
6. [6] B. Park, W. R. Windham, K. C. Lawrence, and D. P. Smith, “Contaminant Classification of Poultry Hyperspectral Imagery using a Spectral Angle Mapper Algorithm,” Biosyst. Eng., vol. 96, no. 3, pp. 323–333, Mar. 2007, doi: 10.1016/j.biosystemseng.2006.11.012.
7. [7] B. Zhukov, D. Oertel, F. Lanzl, and G. Reinhackel, “Unmixing-based multisensor multiresolution image fusion,” IEEE Trans. Geosci. Remote Sens., vol. 37, no. 3, pp. 1212–1226, May 1999, doi: 10.1109/36.763276.
8. [8] R. B. Gomez, A. Jazaeri, and M. Kafatos, “Wavelet-based hyperspectral and multispectral image fusion,” Orlando, FL, Jun. 2001, pp. 36–42. doi: 10.1117/12.428249.

9. [9] M. T. Eismann and R. C. Hardie, “Application of the stochastic mixing model to hyperspectral resolution enhancement,” IEEE Trans. Geosci. Remote Sens., vol. 42, no. 9, pp. 1924–1933, Sep. 2004, doi: 10.1109/TGRS.2004.830644.
10. [10] Y. Zhang, S. De Backer, and P. Scheunders, “Bayesian Fusion of Multispectral and Hyperspectral Image in Wavelet Domain,” in IGARSS 2008 - 2008 IEEE International Geoscience and Remote Sensing Symposium, Boston, MA, USA, 2008, p. V-69-V–72. doi: 10.1109/IGARSS.2008.4780029.
11. [11] Yifan Zhang, S. De Backer, and P. Scheunders, “Noise-Resistant Wavelet-Based Bayesian Fusion of Multispectral and Hyperspectral Images,” IEEE Trans. Geosci. Remote Sens., vol. 47, no. 11, pp. 3834–3843, Nov. 2009, doi: 10.1109/TGRS.2009.2017737.
12. [12] Q. Wei, N. Dobigeon, and J.-Y. Tourneret, “Bayesian fusion of multispectral and hyperspectral images with unknown sensor spectral response,” in 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, Oct. 2014, pp. 698–702. doi: 10.1109/ICIP.2014.7025140.
13. [13] N. Akhtar, F. Shafait, and A. Mian, “Bayesian sparse representation for hyperspectral image super resolution,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, Jun. 2015, pp. 3631–3640. doi: 10.1109/CVPR.2015.7298986.
14. [14] M. A. Bendoumi, Mingyi He, and Shaohui Mei, “Hyperspectral Image Resolution Enhancement Using High-Resolution Multispectral Image Based on Spectral Unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 10, pp. 6574–6583, Oct. 2014, doi: 10.1109/TGRS.2014.2298056.
15. [15] K. Zhang, M. Wang, S. Yang, and L. Jiao, “Spatial–Spectral-Graph-Regularized Low-Rank Tensor Decomposition for Multispectral and Hyperspectral Image Fusion,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 11, no. 4, pp. 1030–1040, Apr. 2018, doi: 10.1109/JSTARS.2017.2785411.
16. [16] Wei, “Fusion of multispectral and hyperspectral images based on sparse representation,” 2014.
17. [17] Bo Huang, Huihui Song, Hengbin Cui, Jigen Peng, and Zongben Xu, “Spatial and Spectral Image Fusion Using Sparse Matrix Factorization,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 3, pp. 1693–1704, Mar. 2014, doi: 10.1109/TGRS.2013.2253612.
18. [18] F. Palsson, J. R. Sveinsson, and M. O. Ulfarsson, “Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network,” IEEE Geosci. Remote Sens. Lett., vol. 14, no. 5, pp. 639–643, May 2017, doi: 10.1109/LGRS.2017.2668299.
19. [19] F. Zhou, R. Hang, Q. Liu, and X. Yuan, “Pyramid Fully Convolutional Network for Hyperspectral and Multispectral Image Fusion,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 12, no. 5, pp. 1549–1558, May 2019, doi: 10.1109/JSTARS.2019.2910990.
20. [20] X.-H. Han and Y.-W. Chen, “Deep Residual Network of Spectral and Spatial Fusion for Hyperspectral Image Super-Resolution,” in 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), Singapore, Singapore, Sep. 2019, pp. 266–270. doi: 10.1109/BigMM.2019.00-13.
21. [21] X. Han, J. Yu, J. Luo, and W. Sun, “Hyperspectral and Multispectral Image Fusion Using Cluster-Based Multi-Branch BP Neural Networks,” Remote Sens., vol. 11, no. 10, p. 1173, May 2019, doi: 10.3390/rs11101173.
22. [22] F. A. Mianji, Y. Gu, Y. Zhang, and J. Zhang, “Enhanced Self-Training Superresolution Mapping Technique for Hyperspectral Imagery,” IEEE Geosci. Remote Sens. Lett., vol. 8, no. 4, pp. 671–675, Jul. 2011, doi: 10.1109/LGRS.2010.2102334.
23. [23] S. He, H. Zhou, Y. Wang, W. Cao, and Z. Han, “Super-resolution reconstruction of hyperspectral images via low rank tensor modeling and total variation regularization,” in 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, China, Jul. 2016, pp. 6962–6965. doi: 10.1109/IGARSS.2016.7730816.
24. [24] J. Hu, Y. Li, and W. Xie, “Hyperspectral Image Super-Resolution by Spectral Difference Learning and Spatial Error Correction,” IEEE Geosci. Remote Sens. Lett., vol. 14, no. 10, pp. 1825–1829, Oct. 2017, doi: 10.1109/LGRS.2017.2737637.
25. [25] Y. Li, J. Hu, X. Zhao, W. Xie, and J. Li, “Hyperspectral image super-resolution using deep convolutional neural network,” Neurocomputing, vol. 266, pp. 29–41, Nov. 2017, doi: 10.1016/j.neucom.2017.05.024.
26. [26] Y. Li, L. Zhang, C. Dingl, W. Wei, and Y. Zhang, “Single Hyperspectral Image Super-Resolution with Grouped Deep Recursive Residual Network,” in 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), Xi’an, Sep. 2018, pp. 1–4. doi: 10.1109/BigMM.2018.8499097.
27. [27] J. Jia, L. Ji, Y. Zhao, and X. Geng, “Hyperspectral image super-resolution with spectral–spatial network,” Int. J. Remote Sens., vol. 39, no. 22, pp. 7806–7829, Nov. 2018, doi: 10.1080/01431161.2018.1471546.
28. [28] W. Xie, X. Jia, Y. Li, and J. Lei, “Hyperspectral Image Super-Resolution Using Deep Feature Matrix Factorization,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 8, pp. 6055–6067, Aug. 2019, doi: 10.1109/TGRS.2019.2904108.
29. [29] Q. Wang, Q. Li, and X. Li, “Spatial-Spectral Residual Network for Hyperspectral Image Super-Resolution,” ArXiv200104609 Cs Eess, Jan. 2020, Accessed: Dec. 17, 2021. [Online]. Available: http://arxiv.org/abs/2001.04609
30. [30] J. Li et al., “Hyperspectral Image Super-Resolution by Band Attention Through Adversarial Learning,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 6, pp. 4304–4318, Jun. 2020, doi: 10.1109/TGRS.2019.2962713.
31. [31] Y. Fu, Z. Liang, and S. You, “Bidirectional 3D Quasi-Recurrent Neural Network for Hyperspectral Image Super-Resolution,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 14, pp. 2674–2688, 2021, doi: 10.1109/JSTARS.2021.3057936.
32. [32] S. Albawi, T. A. Mohammed, and S. Al-Zawi, “Understanding of a convolutional neural network,” in 2017 International Conference on Engineering and Technology (ICET), Antalya, Aug. 2017, pp. 1–6. doi: 10.1109/ICEngTechnol.2017.8308186.
33. [33] M. Elad, Sparse and redundant representations: from theory to applications in signal and image processing. New York, NY Heidelberg: Springer, 2010.
34. [34] J. M. Bioucas-Dias et al., “Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 5, no. 2, pp. 354–379, Apr. 2012, doi: 10.1109/JSTARS.2012.2194696.
35. [35] F. A. Kruse et al., “The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data,” Remote Sens. Environ., vol. 44, no. 2–3, pp. 145–163, May 1993, doi: 10.1016/0034-4257(93)90013-N.