Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data

Year 2024, , 173 - 188, 30.04.2024

Amer Bandar , Ahmet Coşkunçay

https://doi.org/10.53433/yyufbed.1335866

Abstract

In the modern era, remote sensing data has become increasingly useful for determining land use and coverage requirements. Remote sensing data can be used for a variety of purposes, including the classification of crops. It is possible to aggregate remote sensing data for a specific area over time in order to obtain a more complete picture based on the time series of this data. One example of these types of data is the Breizhcrop dataset, which was collected using satellite images acquired by Sentinel 2 over a period of time. This study aims to investigate a neural network based on attention mechanisms using the BI-LSTM layer in conjunction with Temporal-CNN for the classification of crops. The aim of the research is to find a model for corps classification in image-based time series. In line with this goal, in addition to finding features over time, the presented model also needs to produce high-accuracy features at each time step to increase classification. Utilizing the designed neural network, we seek to find local features with the attention mechanism and general features with a second layer. This neural network was validated on the BreizhCrop dataset and we conclude that it performs better than alternative approaches. The proposed method has been compared with Temporal CNN, Star RNN, and Vanilla LSTM networks and it has obtained better results than the mentioned neural networks. Taking advantage of these local and global features that extract with developed model obtained a high accuracy rate of 82%.

Keywords

Attention mechanism, Crop classification, Land use and coverage, Remote sensing, Time series

Uzaktan Algılama Breizhcrop Zaman Serisi Verileri için Dikkat Tabanlı BI-LSTM ve Zamansal Evrişimli Sinir Ağı Kombinasyonu ile Mahsul Sınıflandırması

Abstract

Modern çağda, uzaktan algılama verileri, arazi kullanımı ve kaplama gereksinimlerini belirlemede giderek daha fazla kullanışlı hale gelmiştir. Uzaktan algılama verileri, aralarında mahsul sınıflandırması da bulunan çeşitli amaçlar için kullanılabilir. Belirli bir alan için uzaktan algılama verilerini zaman içinde toplamak, bu verilerin zaman serisi temelinde daha kapsamlı bir görüntü elde etmeyi mümkün kılar. Bu tür verilere örnek olarak, bir süre boyunca Sentinel 2 tarafından elde edilen uydu görüntüleri kullanılarak toplanan Breizhcrop veri seti gösterilebilir. Bu çalışma, mahsullerin sınıflandırılması için BI-LSTM katmanı ile Zaman-İlişkili CNN'nin birleşiminde dayanan, dikkat mekanizmaları temelinde bir sinir ağı araştırmayı hedeflemektedir. Araştırmanın amacı, görüntü tabanlı zaman serilerinde mahsul sınıflandırması için bir model bulmaktır. Bu hedef doğrultusunda, zaman içinde özellikler bulmanın yanı sıra, sunulan modelin her zaman adımında yüksek doğrulukta özellikler üretmesi gerekmektedir ki bu da sınıflandırmayı artırsın. Tasarlanan sinir ağı ile yerel özellikleri dikkat mekanizması ile ve genel özellikleri ikinci bir katman ile bulmayı amaçlıyoruz. Bu sinir ağı, BreizhCrop veri seti üzerinde doğrulanmış ve alternatif yaklaşımlara göre daha iyi performans sergilediği sonucuna varılmıştır. Önerilen yöntem, Zaman-İlişkili CNN, Star RNN ve Vanilya LSTM ağları ile karşılaştırılmış ve bahsedilen sinir ağlarından daha iyi sonuçlar elde edilmiştir. Geliştirilen modelle çıkarılan bu yerel ve küresel özelliklerin avantajını kullanarak, %82 gibi yüksek bir doğruluk oranı elde edilmiştir.

Keywords

Arazi kullanımı ve kapsamı, Dikkat mekanizması, Mahsul sınıflandırması, Uzaktan algılama, Zaman serisi

References

• Baroud, S., Chokri, S., Belhaous, S., & Mestari, M. (2021). A brief review of graph convolutional neural network based learning for classifying remote sensing images. Procedia Computer Science, 191, 349-354. doi:10.1016/j.procs.2021.07.047
• Bozo, M., Aptoula, E., & Cataltepe, Z. (2020). A discriminative long short term memory network with metric learning applied to multispectral time series classification. Journal of Imaging, 6(7), 68. doi:10.3390/jimaging6070068
• BreizhCrops. (2022). BreizhCrops - Smart Agriculture. https://www.breizhcrops.fr/en/ Access date: 01.01.2024.
• Cheng, D., Xiang, S., Shang, C., Zhang, Y., Yang, F., & Zhang, L. (2020, April). Spatio-temporal attention-based neural network for credit card fraud detection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01), 362-369. doi:10.1609/aaai.v34i01.5371
• Devadas, R., Denham, R. J., & Pringle, M. (2012). Support vector machine classification of object-based data for crop mapping, using multi-temporal Landsat imagery. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 39, 185-190. doi:10.5194/isprsarchives-XXXIX-B7-185-2012
• Dwivedi, A. K., Singh, A. K., & Singh, D. (2022, July). An object based image analysis of multispectral satellite and drone images for precision agriculture monitoring. IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia. doi:10.1109/IGARSS46834.2022.9884674
• Huang, C., Davis, L. S., & Townshend, J. R. G. (2002). An assessment of support vector machines for land cover classification. International Journal of Remote Sensing, 23(4), 725-749. doi:10.1080/01431160110040323
• Immitzer, M., Vuolo, F., & Atzberger, C. (2016). First experience with Sentinel-2 data for crop and tree species classifications in central Europe. Remote Sensing, 8(3), 166. doi:10.3390/rs8030166
• Jia, D., Cheng, C., Shen, S., & Ning, L. (2022). Multitask deep learning framework for spatiotemporal fusion of NDVI. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-13. doi:10.1109/TGRS.2021.3140144
• Junsomboon, N., & Phienthrakul, T. (2017, February). Combining over-sampling and under-sampling techniques for imbalance dataset. Proceedings of the 9th International Conference on Machine Learning and Computing, Singapore. doi:10.1145/3055635.3056643
• Kussul, N., Lemoine, G., Gallego, F. J., Skakun, S. V., Lavreniuk, M., & Shelestov, A. Y. (2016). Parcel-based crop classification in Ukraine using Landsat-8 data and Sentinel-1A data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6), 2500-2508. doi:10.1109/JSTARS.2016.2560141
• Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016, October). Temporal convolutional networks: A unified approach to action segmentation. Computer Vision–ECCV 2016 Workshops, Amsterdam, Netherlands. Publishing. doi:10.1007/978-3-319-49409-8_7
• Lee, J. K., Acharya, T. D., & Lee, D. H. (2018). Exploring land cover classification accuracy of landsat 8 image using spectral index layer stacking in hilly region of South Korea. Sensors & Materials, 30(12), 2927-2941. doi:10.18494/SAM.2018.1934
• Lever, J. (2016). Classification evaluation: It is important to understand both what a classification metric expresses and what it hides. Nature Methods, 13(8), 603-605.
• Li, Y., Zhu, Z., Kong, D., Han, H., & Zhao, Y. (2019). EA-LSTM: Evolutionary attention-based LSTM for time series prediction. Knowledge-Based Systems, 181, 104785. doi:10.1016/j.knosys.2019.05.028
• Li, R., Xu, M., Chen, Z., Gao, B., Cai, J., Shen, F., ..., & Chen, D. (2021). Phenology-based classification of crop species and rotation types using fused MODIS and Landsat data: The comparison of a random-forest-based model and a decision-rule-based model. Soil and Tillage Research, 206, 104838. doi:10.1016/j.still.2020.104838
• Liu, C., Zeng, D., Wu, H., Wang, Y., Jia, S., & Xin, L. (2020). Urban land cover classification of high-resolution aerial imagery using a relation-enhanced multiscale convolutional network. Remote Sensing, 12(2), 311. doi:10.3390/rs12020311
• Mazzia, V., Khaliq, A., & Chiaberge, M. (2019). Improvement in land cover and crop classification based on temporal features learning from Sentinel-2 data using recurrent-convolutional neural network (R-CNN). Applied Sciences, 10(1), 238. doi :10.3390/app10010238
• Mei, X., Pan, E., Ma, Y., Dai, X., Huang, J., Fan, F., ..., & Ma, J. (2019). Spectral-spatial attention networks for hyperspectral image classification. Remote Sensing, 11(8), 963. doi:10.3390/rs11080963
• MohanRajan, S. N., Loganathan, A., & Manoharan, P. (2020). Survey on Land Use/Land Cover (LU/LC) change analysis in remote sensing and GIS environment: Techniques and Challenges. Environmental Science and Pollution Research, 27(24), 29900-29926. doi:10.1007/s11356-020-09091-7
• Navnath, N. N., Chandrasekaran, K., Stateczny, A., Sundaram, V. M., & Panneer, P. (2022). Spatiotemporal assessment of satellite image time series for land cover classification using deep learning techniques: a case study of Reunion Island, France. Remote Sensing, 14(20), 5232. doi:10.3390/rs14205232
• Ngoc Hai, P., Manh Tien, N., Trung Hieu, H., Quoc Chung, P., Thanh Son, N., Ngoc Ha, P., & Tung Son, N. (2020, October). An empirical research on the effectiveness of different LSTM architectures on Vietnamese stock market. Proceedings of the 2020 1st International Conference on Control, Robotics and Intelligent System, Xiamen, China. doi:10.1145/3437802.3437827
• Rußwurm, M., & Korner, M. (2017, July). Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA. doi:10.1109/CVPRW.2017.193
• Rußwurm, M., Pelletier, C., Zollner, M., Lefèvre, S., & Körner, M. (2019). Breizhcrops: A time series dataset for crop type mapping. arXiv preprint arXiv:1905.11893. doi:10.48550/arXiv.1905.11893
• Rußwurm, M., Courty, N., Emonet, R., Lefèvre, S., Tuia, D., & Tavenard, R. (2023). End-to-end learned early classification of time series for in-season crop type mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 196, 445-456. doi:10.1016/j.isprsjprs.2022.12.016
• Sykas, D., Papoutsis, I., & Zografakis, D. (2021, July). Sen4AgriNet: A harmonized multi-country, multi-temporal benchmark dataset for agricultural earth observation machine learning applications. IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium. doi:10.1109/IGARSS47720.2021.9553603
• Tatsumi, K., Yamashiki, Y., Torres, M. A. C., & Taipe, C. L. R. (2015). Crop classification of upland fields using Random forest of time-series Landsat 7 ETM+ data. Computers and Electronics in Agriculture, 115, 171-179. doi:10.1016/j.compag.2015.05.001
• Thenkabail, P. S., Gumma, M. K., Teluguntla, P., & Irshad, A. M. (2014). Hyperspectral remote sensing of vegetation and agricultural crops. Photogrammetric Engineering & Remote Sensing (TSI), 80(8), 695-723.
• Toh, F. A., Angwafo, T. E., Ndam, L. M., & Antoine, M. Z. (2018). The socio-economic impact of land use and land cover change on the inhabitants of Mount Bambouto Caldera of the Western Highlands of Cameroon. Advances in Remote Sensing, 7(1), 25-45. doi:10.4236/ars.2018.71003
• Tran, H. D., Choi, S. W., Yang, X., Yamaguchi, T., Hoxha, B., & Prokhorov, D. (2023, May). Verification of recurrent neural networks with star reachability. 26th ACM International Conference on Hybrid Systems: Computation and Control, San Antonio, TX, USA. doi:10.1145/3575870.3587128
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,Gomez, ..., & Polosukhin, I. (2017) Attention is all you need. arXiv preprint arXiv:1706.03762. doi:10.48550/arXiv.1706.03762
• Wang, X., Zhang, J., Xun, L., Wang, J., Wu, Z., Henchiri, M., ..., & Yu, X. (2022). Evaluating the effectiveness of machine learning and deep learning models combined time-series satellite data for multiple crop types classification over a large-scale region. Remote Sensing, 14(10), 2341. doi:10.3390/rs14102341
• Werner de Vargas, V., Schneider Aranda, J. A., dos Santos Costa, R., da Silva Pereira, P. R., & Victória Barbosa, J. L. (2023). Imbalanced data preprocessing techniques for machine learning: a systematic mapping study. Knowledge and Information Systems, 65(1), 31-57. doi:10.1007/s10115-022-01772-8
• Yan, J., Wang, L., Song, W., Chen, Y., Chen, X., & Deng, Z. (2019). A time-series classification approach based on change detection for rapid land cover mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 158, 249-262. doi:10.1016/j.isprsjprs.2019.10.003
• Yan, J., Liu, J., Wang, L., Liang, D., Cao, Q., Zhang, W., & Peng, J. (2022). Land-cover classification with time-series remote sensing images by complete extraction of multiscale timing dependence. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15, 1953-1967. doi:10.1109/JSTARS.2022.3150430
• Yuan, Y., Lin, L., Chen, J., Sahli, H., Chen, Y., Wang, C., & Wu, B. (2019). A new framework for modelling and monitoring the conversion of cultivated land to built-up land based on a hierarchical hidden semi-Markov model using satellite image time series. Remote Sensing, 11(2), 210. doi:10.3390/rs11020210
• Zhu, T., Luo, W., & Yu, F. (2020). Convolution-and attention-based neural network for automated sleep stage classification. International Journal of Environmental Research and Public Health, 17(11), 4152. doi:10.3390/ijerph17114152