Hybrid 3D Convolution and 2D Depthwise Separable Convolution Neural Network for Hyperspectral Image Classification

Abstract

Convolutional neural networks (CNNs) are one of the popular deep learning methods used to solve the hyperspectral image classification (HSIC) problem. CNN has a strong feature learning ability that can ensure more distinctive features for higher quality HSIC. The traditional CNN-based methods mainly use the 2D CNN for HSIC. However, with 2D CNN, only spatial features are extracted in HSI. Good feature maps cannot be extracted from spectral dimensions with the use of 2D CNN alone. By using 3D CNN, spatial-spectral features are extracted simultaneously. However, 3D CNN is computationally complex. In this study, a hybrid CNN method, which is a combination of 3D CNN and 2D CNN, is improved to solve the two problems described above. Using hybrid CNN decreases the complexity of the method compared to using only 3D CNN and can perform well against a limited number of training samples. On the other hand, in Hybrid CNN, depthwise separable convolution (DSC) is used, which decreases computational cost, prevents overfitting and enables more spatial feature extraction. By adding DSC to the developed hybrid CNN, a hybrid depthwise separable convolutional neural network is obtained. Extensive applications on frequently used HSI benchmark datasets show that the classification performance of the proposed network is better than compared methods.

Keywords

• 3D Convolutional Neural Network
• Depthwise Separable Convolution
• Deep Learning
• Hyperspectral Image Classificaiton

References

1. [1] H. Gao, Y. Yang, C. Li, L. Gao, and B. Zhang, “Multiscale Residual Network with Mixed Depthwise Convolution for Hyperspectral Image Classification,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 4, pp. 3396–3408, 2021, doi: 10.1109/TGRS.2020.3008286.
2. [2] H. Fırat and D. Hanbay, “4CF-Net: Hiperspektral uzaktan algılama görüntülerinin spektral uzamsal sınıflandırılması için yeni 3B evrişimli sinir ağı,” Gazi Üniversitesi Mühendislik-Mimarlık Fakültesi Derg., vol. 1, pp. 439–453, 2021, doi: 10.17341/gazimmfd.901291.
3. [3] H. Firat, M. Uçan, and D. Hanbay, “Hyperspectral Image Classification Using MiniVGGNet,” J. Comput. Sci., vol. IDAP-2021, no. Special, pp. 295–303, 2021.
4. [4] H. Firat and D. Hanbay, “3B ESA Tabanlı ResNet50 Kullanılarak Hiperspektral Görüntülerin Sınıflandırılması Classification of Hyperspectral Images Using 3D CNN Based ResNet50,” 2021 29th Signal Process. Commun. Appl. Conf., pp. 6–9, 2021, doi: 10.1109/SIU53274.2021.9477899.
5. [5] H. Firat, M. Uçan, and D. Hanbay, “Classification of Hyperspectral Remote Sensing Images Using Hybrid 3D-2D CNN Architecture,” J. Comput. Sci., vol. IDAP-2021, no. Special, pp. 132–140, 2021.
6. [6] Y. Wang, W. Yu, and Z. Fang, “Multiple Kernel-based SVM classification of hyperspectral images by combining spectral, spatial, and semantic information,” Remote Sens., vol. 12, no. 1, 2020, doi: 10.3390/RS12010120.
7. [7] M. Ahmad et al., “Spatial prior fuzziness pool-based interactive classification of hyperspectral images,” Remote Sens., vol. 11, no. 9, pp. 1–19, 2019, doi: 10.3390/rs11091136.
8. [8] A. Alcolea, M. E. Paoletti, J. M. Haut, J. Resano, and A. Plaza, “Inference in supervised spectral classifiers for on-board hyperspectral imaging: An overview,” Remote Sens., vol. 12, no. 3, pp. 1–29, 2020, doi: 10.3390/rs12030534.

9. [9] J. S. Ham, Y. Chen, M. M. Crawford, and J. Ghosh, “Investigation of the random forest framework for classification of hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 3, pp. 492–501, 2005, doi: 10.1109/TGRS.2004.842481.
10. [10] S. Ghaderizadeh, D. Abbasi-Moghadam, A. Sharifi, N. Zhao, and A. Tariq, “Hyperspectral Image Classification Using a Hybrid 3D-2D Convolutional Neural Networks,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 14, pp. 7570–7588, 2021, doi: 10.1109/JSTARS.2021.3099118.
11. [11] M. F. Özdemir and D. Hanbay, “Comparison of Optimization Algorithms for Multi-Object Tracking FairMOT Algorithm,” J. Comput. Sci., vol. IDAP-2021, no. Special, pp. 147–153, 2021.
12. [12] İ. Sel and D. Hanbay, “Creating a Parallel Corpora for Turkish-English Academic Translations,” J. Comput. Sci., vol. IDAP-2021, no. Special, pp. 335–340, 2021.
13. [13] H. Uzen, M. Turkoglu, and D. Hanbay, “Texture defect classification with multiple pooling and filter ensemble based on deep neural network,” Expert Syst. Appl., vol. 175, no. March, p. 114838, 2021, doi: 10.1016/j.eswa.2021.114838.
14. [14] H. Üzen, H. Fırat, A. Karcİ, and D. Hanbay, “Automatic Thresholding Method Developed With Entropy For Fabric Defect Detection,” in 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), 2019, pp. 14–17.
15. [15] C. Zhao, X. Wan, G. Zhao, B. Cui, W. Liu, and B. Qi, “Spectral-Spatial Classification of Hyperspectral Imagery Based on Stacked Sparse Autoencoder and Random Forest,” Eur. J. Remote Sens., vol. 50, no. 1, pp. 47–63, 2017, doi: 10.1080/22797254.2017.1274566.
16. [16] H. Data et al., “Deep Learning-Based Classi fi cation of Hyperspectral Data,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 7, no. 6, pp. 2094–2107, 2014, doi: 10.1109/JSTARS.2014.2329330.
17. [17] A. Mughees and L. Tao, “Efficient deep auto-encoder learning for the classification of hyperspectral images,” Proc. - 2016 Int. Conf. Virtual Real. Vis. ICVRV 2016, no. September, pp. 44–51, 2017, doi: 10.1109/ICVRV.2016.16.
18. [18] P. Zhong, Z. Gong, S. Li, and C. B. Schonlieb, “Learning to Diversify Deep Belief Networks for Hyperspectral Image Classification,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 6, pp. 3516–3530, 2017, doi: 10.1109/TGRS.2017.2675902.
19. [19] Y. Chen, X. Zhao, and X. Jia, “Spectral-Spatial Classification of Hyperspectral Data Based on Deep Belief Network,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 8, no. 6, pp. 2381–2392, 2015, doi: 10.1109/JSTARS.2015.2388577.
20. [20] J. Li, B. Xi, Y. Li, Q. Du, and K. Wang, “Hyperspectral classification based on texture feature enhancement and deep belief networks,” Remote Sens., vol. 10, no. 3, 2018, doi: 10.3390/rs10030396.
21. [21] Y. Li, H. Zhang, and Q. Shen, “Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network,” Remote Sens., vol. 9, no. 1, 2017, doi: 10.3390/rs9010067.
22. [22] C. Zhang et al., “Joint Deep Learning for land cover and land use classification,” Remote Sens. Environ., vol. 221, no. May 2018, pp. 173–187, 2019, doi: 10.1016/j.rse.2018.11.014.
23. [23] H. S. Nogay, T. C. Akinci, and M. Yilmaz, “Detection of invisible cracks in ceramic materials using by pre-trained deep convolutional neural network,” Neural Computings and Applications, vol. 0123456789, 2021, doi: 10.1007/s00521-021-06652-w.6232–6251, 2016, doi: 10.1049/iet-ipr.2019.1282.
24. [24] S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri, “HybridSN: Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification,” arXiv, vol. 17, no. 2, pp. 277–281, 2019.
25. [25] M. Ahmad, A. M. Khan, M. Mazzara, S. Distefano, M. Ali, and M. S. Sarfraz, “A Fast and Compact 3-D CNN for Hyperspectral Image Classification,” IEEE Geosci. Remote Sens. Lett., no. April, pp. 1–5, 2020, doi: 10.1109/LGRS.2020.3043710.
26. [26] Z. Ge, G. Cao, X. Li, and P. Fu, “Hyperspectral Image Classification Method Based on 2D-3D CNN and Multibranch Feature Fusion,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 13, pp. 5776–5788, 2020, doi: 10.1109/JSTARS.2020.3024841.
27. [27] H. C. Mingyi He, Bo Li, “Multi-scale 3D deep convolutional neural network for hyperspectral image classification,” 2017 IEEE Int. Conf. Image Process., pp. 3904–3908, 2017.
28. [28] C. Mu, Z. Guo, and Y. Liu, “A multi-scale and multi-level spectral-spatial feature fusion network for hyperspectral image classification,” Remote Sens., vol. 12, no. 1, 2020, doi: 10.3390/RS12010125.
29. [29] Z. Zhong, J. Li, Z. Luo, and M. Chapman, “Spectral-Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 2, pp. 847–858, 2018, doi: 10.1109/TGRS.2017.2755542.
30. [30] A. Mohan and M. Venkatesan, “HybridCNN based hyperspectral image classification using multiscale spatiospectral features,” Infrared Phys. Technol., vol. 108, no. March, 2020, doi: 10.1016/j.infrared.2020.103326.
31. [31] A. Mohan and V. Meenakshi Sundaram, “V3O2: hybrid deep learning model for hyperspectral image classification using vanilla-3D and octave-2D convolution,” J. Real-Time Image Process., no. 0123456789, 2020, doi: 10.1007/s11554-020-00966-z.
32. [32] F. Cao and W. Guo, “Deep hybrid dilated residual networks for hyperspectral image classification,” Neurocomputing, vol. 384, pp. 170–181, 2020, doi: 10.1016/j.neucom.2019.11.092.
33. [33] A. Ben Hamida, A. Benoit, P. Lambert, and C. Ben Amar, “3-D deep learning approach for remote sensing image classification,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 8, pp. 4420–4434, 2018, doi: 10.1109/TGRS.2018.2818945.
34. [34] M. Ahmad, S. Shabbir, R. A. Raza, M. Mazzara, S. Distefano, and A. M. Khan, “Hyperspectral Image Classification: Artifacts of Dimension Reduction on Hybrid CNN,” no. January, pp. 1–9, 2021.
35. [35] L. Jiang, B. Zhu, and Y. Tao, “Hyperspectral Image Classification Methods,” Hyperspectral Imaging Food Qual. Anal. Control, pp. 79–98, 2010, doi: 10.1016/B978-0-12-374753-2.10003-6.
36. [36] A. Wang, C. Liu, D. Xue, H. Wu, Y. Zhang, and M. Liu, “Depthwise separable relation network for small sample hyperspectral image classification,” Symmetry (Basel)., vol. 13, no. 9, 2021, doi: 10.3390/sym13091673.
37. [37] L. Dang, P. Pang, and J. Lee, “Depth-wise separable convolution neural network with residual connection for hyperspectral image classification,” Remote Sens., vol. 12, no. 20, pp. 1–20, 2020, doi: 10.3390/rs12203408.
38. [38] B. C. Kuo, H. H. Ho, C. H. Li, C. C. Hung, and J. S. Taur, “A kernel-based feature selection method for SVM with RBF kernel for hyperspectral image classification,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 7, no. 1, pp. 317–326, 2014, doi: 10.1109/JSTARS.2013.2262926.
39. [39] Q. Wang, J. Gao, and Y. Yuan, “A Joint Convolutional Neural Networks and Context Transfer for Street Scenes Labeling,” IEEE Trans. Intell. Transp. Syst., vol. 19, no. 5, pp. 1457–1470, 2018, doi: 10.1109/TITS.2017.2726546.