Semantic segmentation of very-high spatial resolution satellite images: A comparative analysis of 3D-CNN and traditional machine learning algorithms for automatic vineyard detection

Abstract

The Erzincan (Cimin) grape, which is an endemic product, plays a significant role in the economy of both the region it is cultivated in and the overall country. Therefore, it is crucial to closely monitor and promote this product. The objective of this study was to analyze the spatial distribution of vineyards by utilizing advanced machine learning and deep learning algorithms to classify high-resolution satellite images. A deep learning model based on a 3D Convolutional Neural Network (CNN) was developed for vineyard classification. The proposed model was compared with traditional machine learning algorithms, specifically Support Vector Machine (SVM), Random Forest (RF), and Rotation Forest (ROTF). The accuracy of the classifications was assessed through error matrices, kappa analysis, and McNemar tests. The best overall classification accuracies and kappa values were achieved by the 3D CNN and RF methods, with scores of 86.47% (0.8308) and 70.53% (0.6279) respectively. Notably, when Gabor texture features were incorporated, the accuracy of the RF method increased to 75.94% (0.6364). Nevertheless, the 3D CNN classifier outperformed all others, yielding the highest classification accuracy with an 11% advantage (86.47%). The statistical analysis using McNemar's test confirmed that the χ2 values for all classification outcomes exceeded 3.84 at the 95% confidence interval, indicating a significant enhancement in classification accuracy provided by the 3D CNN classifier. Additionally, the 3D CNN method demonstrated successful classification performance, as evidenced by the minimum-maximum F1-score (0.79-0.97), specificity (0.95-0.99), and accuracy (0.91-0.99) values.

Keywords

• Machine Learning
• Deep Learning
• Random Forest
• Gabor
• Image Classification

References

1. Weaver, R. J. (1976). Grape growing. John Wiley & Sons.
2. Akpınar, E., & Çelikoğlu, Ş. (2016). Karaerik (Cimin) üzümünün Erzincan ekonomisine ve tanıtımına katkıları. Uluslararası Erzincan Sempozyumu, 2, 15-23.
3. Bulut, İ. (2006). Genel tarım bilgileri ve tarımın coğrafi esasları (Ziraat Coğrafyası). Gündüz Eğitim ve Yayıncılık, Ankara, 255.
4. Republic of Turkey Ministry of Agriculture and Forestry. (2021). 2021-January Agricultural Products Markets Report: GRAPE, https://arastirma.tarimorman.gov.tr/tepge/Menu/27/Tarim-Urunleri-Piyasalari
5. Erzincan Directorate of Provincial Agriculture and Forestry (2022). https://erzincan.tarimorman.gov.tr/Menu/66/Tarimsal-Veriler
6. Christian, B., & Krishnayya, N. S. R. (2009). Classification of tropical trees growing in a sanctuary using Hyperion (EO-1) and SAM algorithm. Current Science, 96(12), 1601-1607.
7. Prins, A. J., & Van Niekerk, A. (2020). Regional Mapping of Vineyards Using Machine Learning and LiDAR Data. International Journal of Applied Geospatial Research (IJAGR), 11(4), 1-22. https://doi.org/10.4018/IJAGR.2020100101
8. Darra, N., Psomiadis, E., Kasimati, A., Anastasiou, A., Anastasiou, E., & Fountas, S. (2021). Remote and proximal sensing-derived spectral indices and biophysical variables for spatial variation determination in vineyards. Agronomy, 11(4), 741. https://doi.org/10.3390/agronomy11040741

9. Vélez, S., Ariza-Sentís, M., & Valente, J. (2023). Mapping the spatial variability of Botrytis bunch rot risk in vineyards using UAV multispectral imagery. European Journal of Agronomy, 142, 126691. https://doi.org/10.1016/j.eja.2022.126691
10. Gungor, O., Boz, Y., Gokalp, E., Comert, C., & Akar, A. (2010). Fusion of low and high resolution satellite images to monitor changes on costal zones. Scientific Research and Essays, 5(7), 654-662.
11. Chi, M. V., Thi, L. P., & Si, S. T. (2009, October). Monitoring urban space expansion using Remote sensing data in Ha Long city, Quang Ninh province in Vietnam. In 7th FIG Regional Conference Spatial Data Serving People: Land Governance and the Environment–Building the Capacity Hanoi, Vietnam, 19-22.
12. Kaya, Y., & Polat, N. (2023). A linear approach for wheat yield prediction by using different spectral vegetation indices. International Journal of Engineering and Geosciences, 8(1), 52-62. https://doi.org/10.26833/ijeg.1035037
13. Akar, A., & Gökalp, E. (2018). Designing a sustainable rangeland information system for Turkey. International Journal of Engineering and Geosciences, 3(3), 87-97. https://doi.org/10.26833/ijeg.412222
14. Zhang, W., Xue, X., Sun, Z., Guo, Y. F., Chi, M., & Lu, H. (2007). Efficient feature extraction for image classification. IEEE 11th International Conference on Computer Vision, 1-8. https://doi.org/10.1109/ICCV.2007.4409058
15. Huang, Y., Fipps, G., Lacey, R. E., & Thomson, S. J. (2011). Landsat satellite multi-spectral image classification of land cover and land use changes for GIS-based urbanization analysis in irrigation districts of Lower Rio Grande Valley of Texas. Journal of Applied Remote Sensing, 2(1), 27-36.
16. Akar, Ö., & Tunç Görmüş, E. (2019). Göktürk-2 ve Hyperion EO-1 uydu görüntülerinden rastgele orman sınıflandırıcısı ve destek vektör makineleri ile arazi kullanım haritalarının üretilmesi. Geomatik, 4(1), 68-81. https://doi.org/10.29128/geomatik.476668
17. Ahady, A. B., & Kaplan, G. (2022). Classification comparison of Landsat-8 and Sentinel-2 data in Google Earth Engine, study case of the city of Kabul. International Journal of Engineering and Geosciences, 7(1), 24-31. https://doi.org/10.26833/ijeg.860077
18. Sefercik, U. G., Kavzoğlu, T., Çölkesen, I., Nazar, M., Öztürk, M. Y., Adali, S., & Dinç, S. (2023). 3D positioning accuracy and land cover classification performance of multispectral RTK UAVs. International Journal of Engineering and Geosciences, 8(2), 119-128. https://doi.org/10.26833/ijeg.1074791
19. Cengiz, A. V. C. I., Budak, M., Yağmur, N., & Balçik, F. (2023). Comparison between random forest and support vector machine algorithms for LULC classification. International Journal of Engineering and Geosciences, 8(1), 1-10. https://doi.org/10.26833/ijeg.987605
20. Tirmanoğlu, B., Ismailoğlu, I., Kokal, A. T., & Musaoğlu, N. (2023). Yeni nesil multispektral ve hiperspektral uydu görüntülerinin arazi örtüsü/arazi kullanımı sınıflandırma performanslarının karşılaştırılması: Sentinel-2 ve PRISMA Uydusu. Geomatik, 8(1), 79-90. https://doi.org/10.29128/geomatik.1126685
21. Çömert, R., Matci, D. K., & Avdan, U. (2019). Object based burned area mapping with random forest algorithm. International Journal of Engineering and Geosciences, 4(2), 78-87. https://doi.org/10.26833/ijeg.455595
22. Sun, Z., Di, L., Fang, H., & Burgess, A. (2020). Deep learning classification for crop types in north dakota. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, 2200-2213. https://doi.org/10.1109/JSTARS.2020.2990104
23. Gao, J. (2009). Digital analysis of remotely sensed imagery. McGraw-Hill Education, New York. ISBN: 9780071604659
24. Jay, S., Lawrence, R., Repasky, K., & Keith, C. (2009). Invasive species mapping using low-cost hyperspectral imagery. In ASPRS Annual Conference.
25. Ok, A. O., Akar, O., & Gungor, O. (2012). Evaluation of random forest method for agricultural crop classification. European Journal of Remote Sensing, 45(1), 421-432. https://doi.org/10.5721/EuJRS20124535
26. Akar, Ö., & Güngör, O. (2015). Integrating multiple texture methods and NDVI to the Random Forest classification algorithm to detect tea and hazelnut plantation areas in northeast Turkey. International Journal of Remote Sensing, 36(2), 442-464. https://doi.org/10.1080/01431161.2014.995276
27. Ntouros, K. D., Gitas, I. Z., & Silleos, G. N. (2009, August). Mapping agricultural crops with EO-1 Hyperion data. In 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, 1-4. https://doi.org/10.1109/WHISPERS.2009.5289057
28. Kpienbaareh, D., Sun, X., Wang, J., Luginaah, I., Bezner Kerr, R., Lupafya, E., & Dakishoni, L. (2021). Crop type and land cover mapping in northern Malawi using the integration of sentinel-1, sentinel-2, and planetscope satellite data. Remote Sensing, 13(4), 700. https://doi.org/10.3390/rs13040700
29. Wang, S., Azzari, G., & Lobell, D. B. (2019). Crop type mapping without field-level labels: Random forest transfer and unsupervised clustering techniques. Remote sensing of environment, 222, 303-317. https://doi.org/10.1016/j.rse.2018.12.026
30. Akar, Ö., Saralıoğlu, E., Güngör, O., & Bayata, H. F. (2021). Determination of vineyards with support vector machine and deep learning-based Image classification. Intercontinental Geoinformation Days, 3, 26-29.
31. Grinblat, G. L., Uzal, L. C., Larese, M. G., & Granitto, P. M. (2016). Deep learning for plant identification using vein morphological patterns. Computers and Electronics in Agriculture, 127, 418-424. https://doi.org/10.1016/j.compag.2016.07.003
32. Ferentinos, K. P. (2018). Deep learning models for plant disease detection and diagnosis. Computers and electronics in agriculture, 145, 311-318. https://doi.org/10.1016/j.compag.2018.01.009
33. Chlingaryan, A., Sukkarieh, S., & Whelan, B. (2018). Machine learning approaches for crop yield prediction and nitrogen status estimation in precision agriculture: A review. Computers and Electronics in Agriculture, 151, 61-69. https://doi.org/10.1016/j.compag.2018.05.012
34. Abdullahi, H. S., Sheriff, R., & Mahieddine, F. (2017). Convolution neural network in precision agriculture for plant image recognition and classification. Seventh International Conference on Innovative Computing Technology (INTECH), 10, 256-272.
35. Zhao, H., Duan, S., Liu, J., Sun, L., & Reymondin, L. (2021). Evaluation of five deep learning models for crop type mapping using sentinel-2 time series images with missing information. Remote Sensing, 13(14), 2790. https://doi.org/10.3390/rs13142790
36. Zhong, L., Hu, L., & Zhou, H. (2019). Deep learning based multi-temporal crop classification. Remote Sensing of Environment, 221, 430-443. https://doi.org/10.1016/j.rse.2018.11.032
37. TR Erzincan Governorate. (2021). http://www.erzincan.gov.tr/erzincan-uzumu
38. Padwick, C., Deskevich, M., Pacifici, F., & Smallwood, S. (2010). WorldView-2 pan-sharpening. In Proceedings of the ASPRS 2010 Annual Conference, San Diego, CA, USA, 2630, 1-14.
39. Akar, Ö. (2019). Göktürk-2 ve Worldview-2 Uydu Görüntüleri için Görüntü Keskinleştirme Yöntemlerinin Değerlendirilmesi. Erzincan University Journal of Science and Technology, 12(2), 874-885.
40. Li, H., Jing, L., & Tang, Y. (2017). Assessment of pansharpening methods applied to WorldView-2 imagery fusion. Sensors, 17(1), 89. https://doi.org/10.3390/s17010089
41. Anshu, S. K., Pande, H., Tiwari, P. S., & Shukla, S. (2017). Evaluation of Fusion Techniques for High Resolution Data-A Worldview-2 Imagery. International Journal of Applied Remote Sensing and GIS, 4, 10-22.
42. Fu, L., Ma, J., Chen, Y., Larsson, R., & Zhao, J. (2019). Automatic detection of lung nodules using 3D deep convolutional neural networks. Journal of Shanghai Jiaotong University (Science), 24, 517-523. https://doi.org/10.1007/s12204-019-2084-4
43. Zhu, X. X., Tuia, D., Mou, L., Xia, G. S., Zhang, L., Xu, F., & Fraundorfer, F. (2017). Deep learning in remote sensing: A comprehensive review and list of resources. Geoscience and Remote Sensing Magazine, 5(4), 8-36. https://doi.org/10.1109/MGRS.2017.2762307
44. Ji, S., Xu, W., Yang, M., & Yu, K. (2012). 3D convolutional neural networks for human action recognition. Transactions on Pattern Analysis and Machine Intelligence, 35(1), 221-231. https://doi.org/10.1109/TPAMI.2012.59
45. Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, 4489-4497.
46. Xu, Z., Guan, K., Casler, N., Peng, B., & Wang, S. (2018). A 3D convolutional neural network method for land cover classification using LiDAR and multi-temporal Landsat imagery. ISPRS journal of photogrammetry and remote sensing, 144, 423-434. https://doi.org/10.1016/j.isprsjprs.2018.08.005
47. Ji, S., Zhang, C., Xu, A., Shi, Y., & Duan, Y. (2018). 3D convolutional neural networks for crop classification with multi-temporal remote sensing images. Remote Sensing, 10(1), 75. https://doi.org/10.3390/rs10010075
48. Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., & Du, Q. (2017). Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote Sensing, 9(11), 1139. https://doi.org/10.3390/rs9111139
49. Saralioglu, E., & Gungor, O. (2022). Semantic segmentation of land cover from high resolution multispectral satellite images by spectral-spatial convolutional neural network. Geocarto International, 37(2), 657-677. https://doi.org/10.1080/10106049.2020.1734871
50. Li, Y., Zhang, H., & Shen, Q. (2017). Spectral–spatial classification of hyperspectral imagery with 3D convolutional neural network. Remote Sensing, 9(1), 67. https://doi.org/10.3390/rs9010067
51. Pérez, F., & Granger, B. E. (2007). IPython: a system for interactive scientific computing. Computing in Science & Engineering, 9(3), 21-29. https://doi.org/10.1109/MCSE.2007.53
52. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.
53. Watts, J. D., & Lawrence, R. L. (2008). Merging random forest classification with an object-oriented approach for analysis of agricultural lands. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(B7), 579-582
54. Waske, B., Heinzel, V., Braun, M., & Menz, G. (2007). Random forests for classifying multi-temporal sar data. Envisat Symposium, 2007, 23-27.
55. Gislason, P. O., Benediktsson, J. A., & Sveinsson, J. R. (2004). Random forest classification of multisource remote sensing and geographic data. In IGARSS 2004. International Geoscience and Remote Sensing Symposium, 2, 1049-1052. https://doi.org/10.1109/IGARSS.2004.1368591
56. Pal, M. (2003, July). Random forests for land cover classification. In IGARSS 2003. 2003 IEEE International Geoscience and Remote Sensing Symposium. Proceedings (IEEE Cat. No. 03CH37477) 6, 3510-3512. https://doi.org/10.1109/IGARSS.2003.1294837
57. Rodriguez, J. J., Kuncheva, L. I., & Alonso, C. J. (2006). Rotation forest: A new classifier ensemble method. IEEE transactions on pattern analysis and machine intelligence, 28(10), 1619-1630. https://doi.org/10.1109/TPAMI.2006.211
58. Xia, J., Du, P., He, X., & Chanussot, J. (2013). Hyperspectral remote sensing image classification based on rotation forest. IEEE Geoscience and Remote Sensing Letters, 11(1), 239-243. https://doi.org/10.1109/LGRS.2013.2254108
59. Liu, K. H., & Huang, D. S. (2008). Cancer classification using rotation forest. Computers in biology and medicine, 38(5), 601-610. https://doi.org/10.1016/j.compbiomed.2008.02.007
60. Vapnik, V. (1999). The nature of statistical learning theory. Springer science & business media.
61. Özkan, Y. (2008). Veri Madenciliği Yöntemleri, Papatya Yayıncılık, İstanbul.
62. Mather, P., & Tso, B. (2016). Classification methods for remotely sensed data. CRC press.
63. Stephens, D., & Diesing, M. (2014). A comparison of supervised classification methods for the prediction of substrate type using multibeam acoustic and legacy grain-size data. PloS one, 9(4), e93950. https://doi.org/10.1371/journal.pone.0093950
64. Çölkesen, İ., & Yomralıoğlu, T. (2014). Arazi örtüsü ve kullanımının haritalanmasında WorldView-2 uydu görüntüsü ve yardımcı verilerin kullanımı. Harita Dergisi, 152(2), 12-24.
65. Thanh Noi, P., & Kappas, M. (2018). Comparison of random forest, k-nearest neighbor, and support vector machine classifiers for land cover classification using Sentinel-2 imagery. Sensors, 18(1), 18. https://doi.org/10.3390/s18010018
66. Kavzoglu, T., & Colkesen, I. (2009). A kernel functions analysis for support vector machines for land cover classification. International Journal of Applied Earth Observation and Geoinformation, 11(5), 352-359. https://doi.org/10.1016/j.jag.2009.06.002
67. Congalton, R. G., & Green, K. (2019). Assessing the accuracy of remotely sensed data: principles and practices. CRC Press.
68. Congalton, R. G., & Green, K. (2019). Assessing the accuracy of remotely sensed data: principles and practices. CRC Press.
69. Pontius Jr, R. G., & Millones, M. (2011). Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. International Journal of Remote Sensing, 32(15), 4407-4429. https://doi.org/10.1080/01431161.2011.552923
70. Akar, A. (2022). Improving the accuracy of random forest‐based land‐use classification using fused images and digital surface models produced via different interpolation methods. Concurrency and Computation: Practice and Experience, 34(6), e6787. https://doi.org/10.1002/cpe.6787
71. Foody, G. M. (2004). Thematic map comparison. Photogrammetric Engineering & Remote Sensing, 70(5), 627-633. https://doi.org/10.14358/PERS.70.5.627
72. Amini, S., Saber, M., Rabiei-Dastjerdi, H., & Homayouni, S. (2022). Urban land use and land cover change analysis using random forest classification of landsat time series. Remote Sensing, 14(11), 2654. https://doi.org/10.3390/rs14112654