yyu jinas Yüzüncü Yıl Üniversitesi Fen Bilimleri Enstitüsü Dergisi 1300-5413 2667-467X Van Yuzuncu Yıl University 10.53433/yyufbed.1335866 Information Systems (Other) Bilgi Sistemleri (Diğer) Uzaktan Algılama Breizhcrop Zaman Serisi Verileri için Dikkat Tabanlı BI-LSTM ve Zamansal Evrişimli Sinir Ağı Kombinasyonu ile Mahsul Sınıflandırması Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data https://orcid.org/0000-0001-7699-162X Bandar Amer Atatürk Üniversitesi Mühendislik Fakültesi https://orcid.org/0000-0002-7411-310X Coşkunçay Ahmet Atatürk Üniversitesi Mühendislik Fakültesi 04 30 2024 29 1 173 188 08 01 2023 02 13 2024 Copyright © 1995, Yuzuncu Yil University Journal of the Institute of Natural and Applied Sciences 1995 Yuzuncu Yil University Journal of the Institute of Natural and Applied Sciences

Modern çağda, uzaktan algılama verileri, arazi kullanımı ve kaplama gereksinimlerini belirlemede giderek daha fazla kullanışlı hale gelmiştir. Uzaktan algılama verileri, aralarında mahsul sınıflandırması da bulunan çeşitli amaçlar için kullanılabilir. Belirli bir alan için uzaktan algılama verilerini zaman içinde toplamak, bu verilerin zaman serisi temelinde daha kapsamlı bir görüntü elde etmeyi mümkün kılar. Bu tür verilere örnek olarak, bir süre boyunca Sentinel 2 tarafından elde edilen uydu görüntüleri kullanılarak toplanan Breizhcrop veri seti gösterilebilir. Bu çalışma, mahsullerin sınıflandırılması için BI-LSTM katmanı ile Zaman-İlişkili CNN'nin birleşiminde dayanan, dikkat mekanizmaları temelinde bir sinir ağı araştırmayı hedeflemektedir. Araştırmanın amacı, görüntü tabanlı zaman serilerinde mahsul sınıflandırması için bir model bulmaktır. Bu hedef doğrultusunda, zaman içinde özellikler bulmanın yanı sıra, sunulan modelin her zaman adımında yüksek doğrulukta özellikler üretmesi gerekmektedir ki bu da sınıflandırmayı artırsın. Tasarlanan sinir ağı ile yerel özellikleri dikkat mekanizması ile ve genel özellikleri ikinci bir katman ile bulmayı amaçlıyoruz. Bu sinir ağı, BreizhCrop veri seti üzerinde doğrulanmış ve alternatif yaklaşımlara göre daha iyi performans sergilediği sonucuna varılmıştır. Önerilen yöntem, Zaman-İlişkili CNN, Star RNN ve Vanilya LSTM ağları ile karşılaştırılmış ve bahsedilen sinir ağlarından daha iyi sonuçlar elde edilmiştir. Geliştirilen modelle çıkarılan bu yerel ve küresel özelliklerin avantajını kullanarak, %82 gibi yüksek bir doğruluk oranı elde edilmiştir.

In the modern era, remote sensing data has become increasingly useful for determining land use and coverage requirements. Remote sensing data can be used for a variety of purposes, including the classification of crops. It is possible to aggregate remote sensing data for a specific area over time in order to obtain a more complete picture based on the time series of this data. One example of these types of data is the Breizhcrop dataset, which was collected using satellite images acquired by Sentinel 2 over a period of time. This study aims to investigate a neural network based on attention mechanisms using the BI-LSTM layer in conjunction with Temporal-CNN for the classification of crops. The aim of the research is to find a model for corps classification in image-based time series. In line with this goal, in addition to finding features over time, the presented model also needs to produce high-accuracy features at each time step to increase classification. Utilizing the designed neural network, we seek to find local features with the attention mechanism and general features with a second layer. This neural network was validated on the BreizhCrop dataset and we conclude that it performs better than alternative approaches. The proposed method has been compared with Temporal CNN, Star RNN, and Vanilla LSTM networks and it has obtained better results than the mentioned neural networks. Taking advantage of these local and global features that extract with developed model obtained a high accuracy rate of 82%.

 Attention mechanism Crop classification Land use and coverage Remote sensing Time series Arazi kullanımı ve kapsamı Dikkat mekanizması Mahsul sınıflandırması Uzaktan algılama Zaman serisi Baroud, S., Chokri, S., Belhaous, S., & Mestari, M. (2021). A brief review of graph convolutional neural network based learning for classifying remote sensing images. Procedia Computer Science, 191, 349-354. doi:10.1016/j.procs.2021.07.047 Bozo, M., Aptoula, E., & Cataltepe, Z. (2020). A discriminative long short term memory network with metric learning applied to multispectral time series classification. Journal of Imaging, 6(7), 68. doi:10.3390/jimaging6070068 BreizhCrops. (2022). BreizhCrops - Smart Agriculture. https://www.breizhcrops.fr/en/ Access date: 01.01.2024. Cheng, D., Xiang, S., Shang, C., Zhang, Y., Yang, F., & Zhang, L. (2020, April). Spatio-temporal attention-based neural network for credit card fraud detection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01), 362-369. doi:10.1609/aaai.v34i01.5371 Devadas, R., Denham, R. J., & Pringle, M. (2012). Support vector machine classification of object-based data for crop mapping, using multi-temporal Landsat imagery. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 39, 185-190. doi:10.5194/isprsarchives-XXXIX-B7-185-2012 Dwivedi, A. K., Singh, A. K., & Singh, D. (2022, July). An object based image analysis of multispectral satellite and drone images for precision agriculture monitoring. IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia. doi:10.1109/IGARSS46834.2022.9884674 Huang, C., Davis, L. S., & Townshend, J. R. G. (2002). An assessment of support vector machines for land cover classification. International Journal of Remote Sensing, 23(4), 725-749. doi:10.1080/01431160110040323 Immitzer, M., Vuolo, F., & Atzberger, C. (2016). First experience with Sentinel-2 data for crop and tree species classifications in central Europe. Remote Sensing, 8(3), 166. doi:10.3390/rs8030166 Jia, D., Cheng, C., Shen, S., & Ning, L. (2022). Multitask deep learning framework for spatiotemporal fusion of NDVI. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-13. doi:10.1109/TGRS.2021.3140144 Junsomboon, N., & Phienthrakul, T. (2017, February). Combining over-sampling and under-sampling techniques for imbalance dataset. Proceedings of the 9th International Conference on Machine Learning and Computing, Singapore. doi:10.1145/3055635.3056643 Kussul, N., Lemoine, G., Gallego, F. J., Skakun, S. V., Lavreniuk, M., & Shelestov, A. Y. (2016). Parcel-based crop classification in Ukraine using Landsat-8 data and Sentinel-1A data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6), 2500-2508. doi:10.1109/JSTARS.2016.2560141 Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016, October). Temporal convolutional networks: A unified approach to action segmentation. Computer Vision–ECCV 2016 Workshops, Amsterdam, Netherlands. Publishing. doi:10.1007/978-3-319-49409-8_7 Lee, J. K., Acharya, T. D., & Lee, D. H. (2018). Exploring land cover classification accuracy of landsat 8 image using spectral index layer stacking in hilly region of South Korea. Sensors & Materials, 30(12), 2927-2941. doi:10.18494/SAM.2018.1934 Lever, J. (2016). Classification evaluation: It is important to understand both what a classification metric expresses and what it hides. Nature Methods, 13(8), 603-605. Li, Y., Zhu, Z., Kong, D., Han, H., & Zhao, Y. (2019). EA-LSTM: Evolutionary attention-based LSTM for time series prediction. Knowledge-Based Systems, 181, 104785. doi:10.1016/j.knosys.2019.05.028 Li, R., Xu, M., Chen, Z., Gao, B., Cai, J., Shen, F., ..., & Chen, D. (2021). Phenology-based classification of crop species and rotation types using fused MODIS and Landsat data: The comparison of a random-forest-based model and a decision-rule-based model. Soil and Tillage Research, 206, 104838. doi:10.1016/j.still.2020.104838 Liu, C., Zeng, D., Wu, H., Wang, Y., Jia, S., & Xin, L. (2020). Urban land cover classification of high-resolution aerial imagery using a relation-enhanced multiscale convolutional network. Remote Sensing, 12(2), 311. doi:10.3390/rs12020311 Mazzia, V., Khaliq, A., & Chiaberge, M. (2019). Improvement in land cover and crop classification based on temporal features learning from Sentinel-2 data using recurrent-convolutional neural network (R-CNN). Applied Sciences, 10(1), 238. doi :10.3390/app10010238 Mei, X., Pan, E., Ma, Y., Dai, X., Huang, J., Fan, F., ..., & Ma, J. (2019). Spectral-spatial attention networks for hyperspectral image classification. Remote Sensing, 11(8), 963. doi:10.3390/rs11080963 MohanRajan, S. N., Loganathan, A., & Manoharan, P. (2020). Survey on Land Use/Land Cover (LU/LC) change analysis in remote sensing and GIS environment: Techniques and Challenges. Environmental Science and Pollution Research, 27(24), 29900-29926. doi:10.1007/s11356-020-09091-7 Navnath, N. N., Chandrasekaran, K., Stateczny, A., Sundaram, V. M., & Panneer, P. (2022). Spatiotemporal assessment of satellite image time series for land cover classification using deep learning techniques: a case study of Reunion Island, France. Remote Sensing, 14(20), 5232. doi:10.3390/rs14205232 Ngoc Hai, P., Manh Tien, N., Trung Hieu, H., Quoc Chung, P., Thanh Son, N., Ngoc Ha, P., & Tung Son, N. (2020, October). An empirical research on the effectiveness of different LSTM architectures on Vietnamese stock market. Proceedings of the 2020 1st International Conference on Control, Robotics and Intelligent System, Xiamen, China. doi:10.1145/3437802.3437827 Rußwurm, M., & Korner, M. (2017, July). Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA. doi:10.1109/CVPRW.2017.193 Rußwurm, M., Pelletier, C., Zollner, M., Lefèvre, S., & Körner, M. (2019). Breizhcrops: A time series dataset for crop type mapping. arXiv preprint arXiv:1905.11893. doi:10.48550/arXiv.1905.11893 Rußwurm, M., Courty, N., Emonet, R., Lefèvre, S., Tuia, D., & Tavenard, R. (2023). End-to-end learned early classification of time series for in-season crop type mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 196, 445-456. doi:10.1016/j.isprsjprs.2022.12.016 Sykas, D., Papoutsis, I., & Zografakis, D. (2021, July). Sen4AgriNet: A harmonized multi-country, multi-temporal benchmark dataset for agricultural earth observation machine learning applications. IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium. doi:10.1109/IGARSS47720.2021.9553603 Tatsumi, K., Yamashiki, Y., Torres, M. A. C., & Taipe, C. L. R. (2015). Crop classification of upland fields using Random forest of time-series Landsat 7 ETM+ data. Computers and Electronics in Agriculture, 115, 171-179. doi:10.1016/j.compag.2015.05.001 Thenkabail, P. S., Gumma, M. K., Teluguntla, P., & Irshad, A. M. (2014). Hyperspectral remote sensing of vegetation and agricultural crops. Photogrammetric Engineering & Remote Sensing (TSI), 80(8), 695-723. Toh, F. A., Angwafo, T. E., Ndam, L. M., & Antoine, M. Z. (2018). The socio-economic impact of land use and land cover change on the inhabitants of Mount Bambouto Caldera of the Western Highlands of Cameroon. Advances in Remote Sensing, 7(1), 25-45. doi:10.4236/ars.2018.71003 Tran, H. D., Choi, S. W., Yang, X., Yamaguchi, T., Hoxha, B., & Prokhorov, D. (2023, May). Verification of recurrent neural networks with star reachability. 26th ACM International Conference on Hybrid Systems: Computation and Control, San Antonio, TX, USA. doi:10.1145/3575870.3587128 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,Gomez, ..., & Polosukhin, I. (2017) Attention is all you need. arXiv preprint arXiv:1706.03762. doi:10.48550/arXiv.1706.03762 Wang, X., Zhang, J., Xun, L., Wang, J., Wu, Z., Henchiri, M., ..., & Yu, X. (2022). Evaluating the effectiveness of machine learning and deep learning models combined time-series satellite data for multiple crop types classification over a large-scale region. Remote Sensing, 14(10), 2341. doi:10.3390/rs14102341 Werner de Vargas, V., Schneider Aranda, J. A., dos Santos Costa, R., da Silva Pereira, P. R., & Victória Barbosa, J. L. (2023). Imbalanced data preprocessing techniques for machine learning: a systematic mapping study. Knowledge and Information Systems, 65(1), 31-57. doi:10.1007/s10115-022-01772-8 Yan, J., Wang, L., Song, W., Chen, Y., Chen, X., & Deng, Z. (2019). A time-series classification approach based on change detection for rapid land cover mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 158, 249-262. doi:10.1016/j.isprsjprs.2019.10.003 Yan, J., Liu, J., Wang, L., Liang, D., Cao, Q., Zhang, W., & Peng, J. (2022). Land-cover classification with time-series remote sensing images by complete extraction of multiscale timing dependence. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15, 1953-1967. doi:10.1109/JSTARS.2022.3150430 Yuan, Y., Lin, L., Chen, J., Sahli, H., Chen, Y., Wang, C., & Wu, B. (2019). A new framework for modelling and monitoring the conversion of cultivated land to built-up land based on a hierarchical hidden semi-Markov model using satellite image time series. Remote Sensing, 11(2), 210. doi:10.3390/rs11020210 Zhu, T., Luo, W., & Yu, F. (2020). Convolution-and attention-based neural network for automated sleep stage classification. International Journal of Environmental Research and Public Health, 17(11), 4152. doi:10.3390/ijerph17114152