Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data

Amer Bandar; Ahmet Coşkunçay

doi:10.53433/yyufbed.1335866

TR EN

Uzaktan Algılama Breizhcrop Zaman Serisi Verileri için Dikkat Tabanlı BI-LSTM ve Zamansal Evrişimli Sinir Ağı Kombinasyonu ile Mahsul Sınıflandırması

Abstract

Modern çağda, uzaktan algılama verileri, arazi kullanımı ve kaplama gereksinimlerini belirlemede giderek daha fazla kullanışlı hale gelmiştir. Uzaktan algılama verileri, aralarında mahsul sınıflandırması da bulunan çeşitli amaçlar için kullanılabilir. Belirli bir alan için uzaktan algılama verilerini zaman içinde toplamak, bu verilerin zaman serisi temelinde daha kapsamlı bir görüntü elde etmeyi mümkün kılar. Bu tür verilere örnek olarak, bir süre boyunca Sentinel 2 tarafından elde edilen uydu görüntüleri kullanılarak toplanan Breizhcrop veri seti gösterilebilir. Bu çalışma, mahsullerin sınıflandırılması için BI-LSTM katmanı ile Zaman-İlişkili CNN'nin birleşiminde dayanan, dikkat mekanizmaları temelinde bir sinir ağı araştırmayı hedeflemektedir. Araştırmanın amacı, görüntü tabanlı zaman serilerinde mahsul sınıflandırması için bir model bulmaktır. Bu hedef doğrultusunda, zaman içinde özellikler bulmanın yanı sıra, sunulan modelin her zaman adımında yüksek doğrulukta özellikler üretmesi gerekmektedir ki bu da sınıflandırmayı artırsın. Tasarlanan sinir ağı ile yerel özellikleri dikkat mekanizması ile ve genel özellikleri ikinci bir katman ile bulmayı amaçlıyoruz. Bu sinir ağı, BreizhCrop veri seti üzerinde doğrulanmış ve alternatif yaklaşımlara göre daha iyi performans sergilediği sonucuna varılmıştır. Önerilen yöntem, Zaman-İlişkili CNN, Star RNN ve Vanilya LSTM ağları ile karşılaştırılmış ve bahsedilen sinir ağlarından daha iyi sonuçlar elde edilmiştir. Geliştirilen modelle çıkarılan bu yerel ve küresel özelliklerin avantajını kullanarak, %82 gibi yüksek bir doğruluk oranı elde edilmiştir.

Keywords

Arazi kullanımı ve kapsamı, Dikkat mekanizması, Mahsul sınıflandırması, Uzaktan algılama, Zaman serisi

Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data

Abstract

In the modern era, remote sensing data has become increasingly useful for determining land use and coverage requirements. Remote sensing data can be used for a variety of purposes, including the classification of crops. It is possible to aggregate remote sensing data for a specific area over time in order to obtain a more complete picture based on the time series of this data. One example of these types of data is the Breizhcrop dataset, which was collected using satellite images acquired by Sentinel 2 over a period of time. This study aims to investigate a neural network based on attention mechanisms using the BI-LSTM layer in conjunction with Temporal-CNN for the classification of crops. The aim of the research is to find a model for corps classification in image-based time series. In line with this goal, in addition to finding features over time, the presented model also needs to produce high-accuracy features at each time step to increase classification. Utilizing the designed neural network, we seek to find local features with the attention mechanism and general features with a second layer. This neural network was validated on the BreizhCrop dataset and we conclude that it performs better than alternative approaches. The proposed method has been compared with Temporal CNN, Star RNN, and Vanilla LSTM networks and it has obtained better results than the mentioned neural networks. Taking advantage of these local and global features that extract with developed model obtained a high accuracy rate of 82%.

Keywords

Attention mechanism, Crop classification, Land use and coverage, Remote sensing, Time series

References

Baroud, S., Chokri, S., Belhaous, S., & Mestari, M. (2021). A brief review of graph convolutional neural network based learning for classifying remote sensing images. Procedia Computer Science, 191, 349-354. doi:10.1016/j.procs.2021.07.047
Bozo, M., Aptoula, E., & Cataltepe, Z. (2020). A discriminative long short term memory network with metric learning applied to multispectral time series classification. Journal of Imaging, 6(7), 68. doi:10.3390/jimaging6070068
BreizhCrops. (2022). BreizhCrops - Smart Agriculture. https://www.breizhcrops.fr/en/ Access date: 01.01.2024.
Cheng, D., Xiang, S., Shang, C., Zhang, Y., Yang, F., & Zhang, L. (2020, April). Spatio-temporal attention-based neural network for credit card fraud detection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01), 362-369. doi:10.1609/aaai.v34i01.5371
Devadas, R., Denham, R. J., & Pringle, M. (2012). Support vector machine classification of object-based data for crop mapping, using multi-temporal Landsat imagery. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 39, 185-190. doi:10.5194/isprsarchives-XXXIX-B7-185-2012
Dwivedi, A. K., Singh, A. K., & Singh, D. (2022, July). An object based image analysis of multispectral satellite and drone images for precision agriculture monitoring. IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia. doi:10.1109/IGARSS46834.2022.9884674
Huang, C., Davis, L. S., & Townshend, J. R. G. (2002). An assessment of support vector machines for land cover classification. International Journal of Remote Sensing, 23(4), 725-749. doi:10.1080/01431160110040323
Immitzer, M., Vuolo, F., & Atzberger, C. (2016). First experience with Sentinel-2 data for crop and tree species classifications in central Europe. Remote Sensing, 8(3), 166. doi:10.3390/rs8030166
Jia, D., Cheng, C., Shen, S., & Ning, L. (2022). Multitask deep learning framework for spatiotemporal fusion of NDVI. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-13. doi:10.1109/TGRS.2021.3140144
Junsomboon, N., & Phienthrakul, T. (2017, February). Combining over-sampling and under-sampling techniques for imbalance dataset. Proceedings of the 9th International Conference on Machine Learning and Computing, Singapore. doi:10.1145/3055635.3056643

Kussul, N., Lemoine, G., Gallego, F. J., Skakun, S. V., Lavreniuk, M., & Shelestov, A. Y. (2016). Parcel-based crop classification in Ukraine using Landsat-8 data and Sentinel-1A data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6), 2500-2508. doi:10.1109/JSTARS.2016.2560141
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016, October). Temporal convolutional networks: A unified approach to action segmentation. Computer Vision–ECCV 2016 Workshops, Amsterdam, Netherlands. Publishing. doi:10.1007/978-3-319-49409-8_7
Lee, J. K., Acharya, T. D., & Lee, D. H. (2018). Exploring land cover classification accuracy of landsat 8 image using spectral index layer stacking in hilly region of South Korea. Sensors & Materials, 30(12), 2927-2941. doi:10.18494/SAM.2018.1934
Lever, J. (2016). Classification evaluation: It is important to understand both what a classification metric expresses and what it hides. Nature Methods, 13(8), 603-605.
Li, Y., Zhu, Z., Kong, D., Han, H., & Zhao, Y. (2019). EA-LSTM: Evolutionary attention-based LSTM for time series prediction. Knowledge-Based Systems, 181, 104785. doi:10.1016/j.knosys.2019.05.028
Li, R., Xu, M., Chen, Z., Gao, B., Cai, J., Shen, F., ..., & Chen, D. (2021). Phenology-based classification of crop species and rotation types using fused MODIS and Landsat data: The comparison of a random-forest-based model and a decision-rule-based model. Soil and Tillage Research, 206, 104838. doi:10.1016/j.still.2020.104838
Liu, C., Zeng, D., Wu, H., Wang, Y., Jia, S., & Xin, L. (2020). Urban land cover classification of high-resolution aerial imagery using a relation-enhanced multiscale convolutional network. Remote Sensing, 12(2), 311. doi:10.3390/rs12020311
Mazzia, V., Khaliq, A., & Chiaberge, M. (2019). Improvement in land cover and crop classification based on temporal features learning from Sentinel-2 data using recurrent-convolutional neural network (R-CNN). Applied Sciences, 10(1), 238. doi :10.3390/app10010238
Mei, X., Pan, E., Ma, Y., Dai, X., Huang, J., Fan, F., ..., & Ma, J. (2019). Spectral-spatial attention networks for hyperspectral image classification. Remote Sensing, 11(8), 963. doi:10.3390/rs11080963
MohanRajan, S. N., Loganathan, A., & Manoharan, P. (2020). Survey on Land Use/Land Cover (LU/LC) change analysis in remote sensing and GIS environment: Techniques and Challenges. Environmental Science and Pollution Research, 27(24), 29900-29926. doi:10.1007/s11356-020-09091-7
Navnath, N. N., Chandrasekaran, K., Stateczny, A., Sundaram, V. M., & Panneer, P. (2022). Spatiotemporal assessment of satellite image time series for land cover classification using deep learning techniques: a case study of Reunion Island, France. Remote Sensing, 14(20), 5232. doi:10.3390/rs14205232
Ngoc Hai, P., Manh Tien, N., Trung Hieu, H., Quoc Chung, P., Thanh Son, N., Ngoc Ha, P., & Tung Son, N. (2020, October). An empirical research on the effectiveness of different LSTM architectures on Vietnamese stock market. Proceedings of the 2020 1st International Conference on Control, Robotics and Intelligent System, Xiamen, China. doi:10.1145/3437802.3437827
Rußwurm, M., & Korner, M. (2017, July). Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA. doi:10.1109/CVPRW.2017.193
Rußwurm, M., Pelletier, C., Zollner, M., Lefèvre, S., & Körner, M. (2019). Breizhcrops: A time series dataset for crop type mapping. arXiv preprint arXiv:1905.11893. doi:10.48550/arXiv.1905.11893
Rußwurm, M., Courty, N., Emonet, R., Lefèvre, S., Tuia, D., & Tavenard, R. (2023). End-to-end learned early classification of time series for in-season crop type mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 196, 445-456. doi:10.1016/j.isprsjprs.2022.12.016
Sykas, D., Papoutsis, I., & Zografakis, D. (2021, July). Sen4AgriNet: A harmonized multi-country, multi-temporal benchmark dataset for agricultural earth observation machine learning applications. IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium. doi:10.1109/IGARSS47720.2021.9553603
Tatsumi, K., Yamashiki, Y., Torres, M. A. C., & Taipe, C. L. R. (2015). Crop classification of upland fields using Random forest of time-series Landsat 7 ETM+ data. Computers and Electronics in Agriculture, 115, 171-179. doi:10.1016/j.compag.2015.05.001
Thenkabail, P. S., Gumma, M. K., Teluguntla, P., & Irshad, A. M. (2014). Hyperspectral remote sensing of vegetation and agricultural crops. Photogrammetric Engineering & Remote Sensing (TSI), 80(8), 695-723.
Toh, F. A., Angwafo, T. E., Ndam, L. M., & Antoine, M. Z. (2018). The socio-economic impact of land use and land cover change on the inhabitants of Mount Bambouto Caldera of the Western Highlands of Cameroon. Advances in Remote Sensing, 7(1), 25-45. doi:10.4236/ars.2018.71003
Tran, H. D., Choi, S. W., Yang, X., Yamaguchi, T., Hoxha, B., & Prokhorov, D. (2023, May). Verification of recurrent neural networks with star reachability. 26th ACM International Conference on Hybrid Systems: Computation and Control, San Antonio, TX, USA. doi:10.1145/3575870.3587128
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,Gomez, ..., & Polosukhin, I. (2017) Attention is all you need. arXiv preprint arXiv:1706.03762. doi:10.48550/arXiv.1706.03762
Wang, X., Zhang, J., Xun, L., Wang, J., Wu, Z., Henchiri, M., ..., & Yu, X. (2022). Evaluating the effectiveness of machine learning and deep learning models combined time-series satellite data for multiple crop types classification over a large-scale region. Remote Sensing, 14(10), 2341. doi:10.3390/rs14102341
Werner de Vargas, V., Schneider Aranda, J. A., dos Santos Costa, R., da Silva Pereira, P. R., & Victória Barbosa, J. L. (2023). Imbalanced data preprocessing techniques for machine learning: a systematic mapping study. Knowledge and Information Systems, 65(1), 31-57. doi:10.1007/s10115-022-01772-8
Yan, J., Wang, L., Song, W., Chen, Y., Chen, X., & Deng, Z. (2019). A time-series classification approach based on change detection for rapid land cover mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 158, 249-262. doi:10.1016/j.isprsjprs.2019.10.003
Yan, J., Liu, J., Wang, L., Liang, D., Cao, Q., Zhang, W., & Peng, J. (2022). Land-cover classification with time-series remote sensing images by complete extraction of multiscale timing dependence. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15, 1953-1967. doi:10.1109/JSTARS.2022.3150430
Yuan, Y., Lin, L., Chen, J., Sahli, H., Chen, Y., Wang, C., & Wu, B. (2019). A new framework for modelling and monitoring the conversion of cultivated land to built-up land based on a hierarchical hidden semi-Markov model using satellite image time series. Remote Sensing, 11(2), 210. doi:10.3390/rs11020210
Zhu, T., Luo, W., & Yu, F. (2020). Convolution-and attention-based neural network for automated sleep stage classification. International Journal of Environmental Research and Public Health, 17(11), 4152. doi:10.3390/ijerph17114152

Details

Primary Language

English

Subjects

Information Systems (Other)

Journal Section

Research Article

Authors

Amer Bandar ^*
0000-0001-7699-162X
Türkiye

Ahmet Coşkunçay
0000-0002-7411-310X
Türkiye

Publication Date

April 30, 2024

Submission Date

August 1, 2023

Acceptance Date

February 13, 2024

Published in Issue

Year 2024 Volume: 29 Number: 1

DOI

https://doi.org/10.53433/yyufbed.1335866

IZ

https://izlik.org/JA68NU63ZA

APA

Bandar, A., & Coşkunçay, A. (2024). Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data. Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi, 29(1), 173-188. https://doi.org/10.53433/yyufbed.1335866

AMA

1.Bandar A, Coşkunçay A. Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data. YYU JINAS. 2024;29(1):173-188. doi:10.53433/yyufbed.1335866

Chicago

Bandar, Amer, and Ahmet Coşkunçay. 2024. “Crop Classification With Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data”. Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi 29 (1): 173-88. https://doi.org/10.53433/yyufbed.1335866.

EndNote

Bandar A, Coşkunçay A (April 1, 2024) Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data. Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi 29 1 173–188.

IEEE

[1]A. Bandar and A. Coşkunçay, “Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data”, YYU JINAS, vol. 29, no. 1, pp. 173–188, Apr. 2024, doi: 10.53433/yyufbed.1335866.

ISNAD

Bandar, Amer - Coşkunçay, Ahmet. “Crop Classification With Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data”. Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi 29/1 (April 1, 2024): 173-188. https://doi.org/10.53433/yyufbed.1335866.

JAMA

1.Bandar A, Coşkunçay A. Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data. YYU JINAS. 2024;29:173–188.

MLA

Bandar, Amer, and Ahmet Coşkunçay. “Crop Classification With Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data”. Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi, vol. 29, no. 1, Apr. 2024, pp. 173-88, doi:10.53433/yyufbed.1335866.

Vancouver

1.Amer Bandar, Ahmet Coşkunçay. Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data. YYU JINAS. 2024 Apr. 1;29(1):173-88. doi:10.53433/yyufbed.1335866

Uzaktan Algılama Breizhcrop Zaman Serisi Verileri için Dikkat Tabanlı BI-LSTM ve Zamansal Evrişimli Sinir Ağı Kombinasyonu ile Mahsul Sınıflandırması

Abstract

Keywords

Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data

Abstract

Keywords

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite