Building detection from high-resolution satellite images with faster regional based deep learning model

Year 2022, , 550 - 563, 15.04.2022

Ekrem Saralıoğlu , Oğuz Güngör

https://doi.org/10.17714/gumusfenbil.1012519

Abstract

Deep learning algorithms, which try to automatically learn features from a large data set to mimic the learning and analysis mechanism in the human brain, have sometimes started to be more successful than humans in solving problems that require high computation. The successful use of deep learning-based methods in various fields also increases its use in remote sensing. This study, it is aimed to make automatic building detection by deep learning from satellite images with high spatial resolution. First, a fused image with more spatial details was obtained by fusing the image to the high spatial resolution Worldview-2 satellite image. Then, the fused image of the study area was divided into parts, including the areas where building details are concentrated. The test and training data set was created by labeling the building objects in these image fragments. Finally, the Faster R-CNN model was trained with the prepared data set, enabling building detection from high spatial resolution satellite images. Building detection was performed with an average accuracy of 88.6% from high-resolution satellite image fragments containing fragments from different regions.

Keywords

Building detection, Deep learning, Faster R-CNN, Artificial intelligence

Yüksek çözünürlüklü uydu görüntülerinden daha hızlı bölge tabanlı derin öğrenme modeli ile bina tespiti

Abstract

İnsan beynindeki öğrenme ve analiz mekanizmasını taklit ederek geniş bir veri kümesinden özellikleri otomatik olarak öğrenmeye çalışan derin öğrenme algoritmaları, yüksek hesaplama gerektiren problemleri çözmede bazen insanlardan daha başarılı olabilmektedir. Derin öğrenme tabanlı yöntemlerin çeşitli alanlarda başarı ile kullanımı bu yöntemlerin uzaktan algılama alanında da kullanımını arttırmaktadır. Bu çalışmada, yüksek mekânsal çözünürlüğe sahip uydu görüntülerinden derin öğrenme ile otomatik bina tespitinin yapılması amaçlanmıştır. Bina tespiti için, ilk olarak yüksek mekânsal çözünürlüklü Worldview-2 uydu görüntüsüne görüntü kaynaştırma işlemi yapılarak mekânsal olarak detayların daha belirgin olduğu kaynaştırılmış bir görüntü elde edilmiştir. Daha sonra çalışma bölgesine ait kaynaştırılmış görüntü bina detaylarının yoğun olduğu bölgeleri içerecek şekilde parçalara ayrılmıştır. Bu görüntü parçalarındaki bina nesneleri etiketlenerek test ve eğitim veri seti oluşturulmuştur. Oluşturulan veri seti ile Faster R-CNN modeli üzerinde ince ayar yapılarak model eğitimi %94 F1 skor ve %88 doğruluk değenlerinde gerçekleştirilmiştir. Sonuç olarak Faster R-CNN modeli ile (genel kullanımının dışında) küçük nesne boyutlu uzaktan algılanmış görüntülerden bina tespiti ortalama %88.6 doğrulukta gerçekleştirilmiştir.

Keywords

Bina tespiti, Derin öğrenme, Faster R-CNN, Yapay zeka

References

• Arel, I., Rose, D. C., & Karnowski, T. P. (2010). Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE Computational Intelligence Magazine, 5(4), 13-18. https://doi.org/10.1109/MCI.2010.938364
• Arévalo, V., González, J., Valdes, J., & Ambrosio, G. (2006, May). Detecting shadows in QuickBird satellite images. In ISPRS Commission VII Mid-term Symposium Remote Sensing: From Pixels to Processes, (pp. 8-11), Enschede, Netherlands.
• Ammar, A., Koubaa, A., Ahmed, M., & Saad, A. (2019). Aerial images processing for car detection using convolutional neural networks: Comparison between faster r-cnn and yolov3. arXiv preprint arXiv:1910.07234. https://arxiv.org/abs/1910.07234
• Ammour, N., Alhichri, H., Bazi, Y., Benjdira, B., Alajlan, N., & Zuair, M. (2017). Deep learning approach for car detection in UAV imagery. Remote Sensing, 9(4), https://doi.org/10.3390/rs9040312
• Barsi, A., & Heipke, C. (2003). Artificial neural networks for the detection of road junctions in aerial images. International Archives of Photogrammetry Remote Sensing and Spatial Information Sciences, 34(3/W8), 113-118
• Cao, Y., Niu, X., & Dou, Y. (2016). Region-based convolutional neural networks for object detection in very high resolution remote sensing images. In 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD) (ss. 548-554). Changsha: IEEE. https://doi.org/10.1109/FSKD.2016.7603232
• Cao, L., Luo, F., Chen, L., Sheng, Y., Wang, H., Wang, C., & Ji, R. (2017). Weakly supervised vehicle detection in satellite images via multi-instance discriminative learning. Pattern Recognition, 64, 417-424. https://doi.org/10.1016/j.patcog.2016.10.033
• Chen, C., Gong, W., Hu, Y., Chen, Y., & Ding, Y. (2017). Learning oriented region-based convolutional neural networks for building detection in satellite remote sensing images. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 42, 461. https://doi.org/10.5194/isprs-archives-XLII-1-W1-461-2017
• Chen, C., Gong, W., Chen, Y., & Li, W. (2019). Learning a two-stage CNN model for multi-sized building detection in remote sensing images. Remote Sensing Letters, 10(2), 103-110. https://doi.org/10.1080/2150704X.2018.1528398
• Cheng, G., & Han, J. (2016). A survey on object detection in optical remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing, 117, 11-28. https://doi.org/10.1016/j.isprsjprs.2016.03.014
• Cheng, G., Han, J., Guo, L., Qian, X., Zhou, P., Yao, X., & Hu, X. (2013). Object detection in remote sensing imagery using a discriminatively trained mixture model. ISPRS Journal of Photogrammetry and Remote Sensing, 85, 32-43. https://doi.org/10.1016/j.isprsjprs.2013.08.001
• Common Objects in Context (COCO).(2021, April 12). https://cocodataset.org/#home.
• Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) (pp. 886-893). IEEE. https://doi.org/10.1109/CVPR.2005.177
• Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2009). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627-1645. https://doi.org/10.1109/TPAMI.2009.167
• Fischer, P., Dosovitskiy, A., & Brox, T. (2014). Descriptor matching with convolutional neural networks: a comparison to sift. arXiv preprint, arXiv:1405.5769
• Gamba, P., & Houshmand, B. (2000). Digital surface models and building extraction: A comparison of IFSAR and LIDAR data. IEEE Transactions on Geoscience and Remote Sensing, 38(4), 1959-1968. https://doi.org/10.1109/36.851777
• Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 1440-1448). https://cdn-sv1.deepsense.ai/wp-content/uploads/2017/02/1504.08083.pdf
• GitHub. Tensorflow detection model zoo. (2019, October 10). https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md https://github.com/tensorflow/models/tree/master/research/object_detection/models Guo, W., Yang, W., Zhang, H., & Hua, G. (2018). Geospatial object detection in high resolution satellite images based on multi-scale convolutional neural network. Remote Sensing, 10(1), 131, https://doi.org/10.3390/rs10010131
• Guo, L., Chehata, N., Mallet, C., & Boukir, S. (2011). Relevance of airborne lidar and multispectral image data for urban scene classification using Random Forests. ISPRS Journal of Photogrammetry and Remote Sensing, 66(1), 56-66. https://doi.org/10.1016/j.isprsjprs.2010.08.007
• Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (pp. 91–99), MIT Press Cambridge, MA, USA. Haala, N., & Brenner, C. (1999). Extraction of buildings and trees in urban environments. ISPRS Journal of Photogrammetry and Remote Sensing, 54(2-3), 130-137. https://doi.org/10.1016/S0924-2716(99)00010-6
• Han, Y., Ma, S., Zhang, F., & Li, C. (2020). Object Detection of Remote Sensing Airport Image Based on Improved Faster R-CNN. In Journal of Physics: Conference Series. IOP Publishing. https://doi.org/10.1088/1742-6596/1601/3/032010
• Han, Q., Yin, Q., Zheng, X., & Chen, Z. (2021). Remote sensing image building detection method based on Mask R-CNN. Complex & Intelligent Systems, https://link.springer.com/article/10.1007/s40747-021-00322-z
• Han, J., Zhang, D., Cheng, G., Guo, L., & Ren, J. (2014). Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning. IEEE Transactions on Geoscience and Remote Sensing, 53(6), 3325-3337. https://doi.org/10.1109/TGRS.2014.2374218
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html
• Karathanassi, V., Kolokousis, P., & Ioannidou, S. (2007). A comparison study on fusion methods using evaluation indicators. International Journal of Remote Sensing, 28(10), 2309–2341. https://doi.org/10.1080/01431160600606890
• Kembhavi, A., Harwood, D., & Davis, L. S. (2010). Vehicle detection using partial least squares. IEEE Transactions on Pattern Analysis & Machine Intelligence, 33(6), 1250-1265. https://doi.org/ 10.1109/TPAMI.2010.182
• Klonus, S., & Ehlers, M. (2009). Performance of evaluation methods in image fusion. In 2009 12th International Conference on Information Fusion (pp.1409–1416). Seattle, WA: IEEE.
• Lafarge, F., Descombes, X., Zerubia, J., & Pierrot-Deseilligny, M. (2008). Automatic building extraction from DEMs using an object approach and application to the 3D-city modeling. ISPRS Journal of Photogrammetry and Remote Sensing, 63(3), 365-381. https://doi.org/10.1016/j.isprsjprs.2007.09.003
• Li, C., Liu, L., Wang, J., Zhao, C., & Wang, R. (2004, September). Comparison of two methods of the fusion of remote sensing images with fidelity of spectral information. In IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium (Vol. 4, pp. 2561-2564). IEEE. https://doi.org/10.1109/IGARSS.2004.1369819
• Li, X., Shang, M., Qin, H., & Chen, L. (2015). Fast accurate fish detection and recognition of underwater images with fast r-cnn. In OCEANS 2015-MTS/IEEE Washington (pp. 1-5). IEEE. https://doi.org/10.23919/OCEANS.2015.7404464
• Li, Y., Zhang, H., Xue, X., Jiang, Y., & Shen, Q. (2018). Deep learning for remote sensing image classification: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(6), 1-17. https://doi.org/10.1002/widm.1264
• Liow, Y. T., & Pavlidis, T. (1990). Use of shadows for extracting buildings in aerial images. Computer Vision, Graphics, and Image Processing, 49(2), 242-277. https://doi.org/10.1016/0734-189X(90)90139-M
• Liu, X., Ghazali, K. H., Han, F., Mohamed, I. I., Zhao, Y., & Ji, Y. (2020). Oil palm tree detection and counting in aerial images based on Faster R-CNN. In InECCE2019 (pp. 475-482). Springer, Singapore.
• Maas, H. G., & Vosselman, G. (1999). Two algorithms for extracting building models from raw laser altimetry data. ISPRS Journal of Photogrammetry and Remote Sensing, 54(2-3), 153-163. https://doi.org/10.1016/S0924-2716(99)00004-0
• Nahhas, F. H., Shafri, H. Z., Sameen, M. I., Pradhan, B., & Mansor, S. (2018). Deep learning approach for building detection using lidar–orthophoto fusion. Journal of Sensors, 2018 (7212307), 1-12. https://doi.org/10.1155/2018/7212307 Nevatia, R., Huertas, A., & Kim, Z. (1999). The MURI Project for rapid feature extraction in urban areas, International Archives of Photogrammetry and Remote Sensing, 32 (3–2W5), 3–14. Erişim Adresi: http:// citeseerx.ist. psu.edu/viewdoc/download?doi=10.1.1.39.2460&rep=rep1&type=pdf
• Padwick, C., Deskevich, M., Pacifici, F., & Smallwood, S. (2010). WorldView-2 pan-sharpening. In Proceedings of the ASPRS 2010 Annual Conference, San Diego, CA, USA. Vol. 2630, pp. 1-14. http://www.asprs.org/wp-content/uploads/2013/08/Padwick.pdf
• Peng, J., Wang, D., Liao, X., Shao, Q., Sun, Z., Yue, H., & Ye, H. (2020). Wild animal survey using UAS imagery and deep learning: modified Faster R-CNN for kiang detection in Tibetan Plateau. ISPRS Journal of Photogrammetry and Remote Sensing, 169, 364-376. https://doi.org/10.1016/j.isprsjprs.2020.08.026
• Rottensteiner, F., Trinder, J., Clode, S., & Kubik, K. (2007). Building detection by fusion of airborne laser scanner data and multi-spectral images: Performance evaluation and sensitivity analysis. ISPRS Journal of Photogrammetry and Remote Sensing, 62(2), 135-149. https://doi.org/10.1016/j.isprsjprs.2007.03.001
• Saralioglu, E., & Gungor, O. (2020). Crowdsourcing in remote sensing: a review of applications and future directions. IEEE Geoscience and Remote Sensing Magazine, 8(4), 89-110. https://doi.org/10.1080/01431160600606890 10.1109/MGRS.2020.2975132
• Saralioglu E., & Gungor O. (2022) Semantic segmentation of land cover from high resolution multispectral satellite images by spectral-spatial convolutional neural network, Geocarto International, 37(2), 657-677, https://doi.org/10.1080/10106049.2020.1734871
• Sardoğan, M., Yunus, Ö., & Tuncer, A. (2020). Detection of Apple Leaf Diseases using Faster R-CNN. Düzce Üniversitesi Bilim ve Teknoloji Dergisi, 8(1), 1110-1117. https://doi.org/10.29130/dubited.648387
• Satari, M., Samadzadegan, F., Azizi, A., & Maas, H. G. (2012). A multi‐resolution hybrid approach for building model reconstruction from lidar data. The Photogrammetric Record, 27(139), 330-359. https://doi.org/10.1111/j.1477-9730.2012.00688.x
• Senaras, C., Ozay, M., & Vural, F. T. Y. (2013). Building detection with decision fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 6(3), 1295-1304. https://doi.org/10.1109/JSTARS.2013.2249498
• Shufelt, J. A. (1996). Exploiting photogrammetric methods for building extraction in aerial images. International Archives of Photogrammetry and Remote Sensing, 31(B6), 74-79.
• Stuparu, D. G., Ciobanu, R. I., & Dobre, C. (2020). Vehicle detection in overhead satellite images using a one-stage object detection model. Sensors, 20(22), 6485. https://doi.org/10.3390/s20226485
• Sun, L., Tang, Y., & Zhang, L. (2017). Rural building detection in high-resolution imagery based on a two-stage CNN model. IEEE Geoscience and Remote Sensing Letters, 14(11), 1998-2002. https://doi.org/10.1109/LGRS.2017.2745900
• Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017, February). Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence. https://pub-tools-public-publication-data.storage.googleapis.com/pdf/45169.pdf
• Stankov, K., & He, D.C. (2014). Detection of buildings in multispectral very high spatial resolution images using the percentage occupancy hit-or-miss transform. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2014 (7), 4069–4080. https://doi.org/10.1109/JSTARS.2014.2308301
• Tao, H., Runsheng, L., Qing, H., & Qingqing, K. (2021). Aircraft target recognition in optical remote sensing image with Faster R-CNN. In 6th International Symposium of Space Optical Instruments and Applications. Delft, the Netherlands, September 24–25, 2019 (pp. 55-63). Springer International Publishing. https://doi.org/10.1007/978-3-030-56488-9_6
• The PASCAL Visual Object Classes Challenge 2007 (2021, April 12). http://host.robots.ox.ac.uk/pascal/VOC/voc2007 Tzutalin. LabelImg. Git code (2020 August 12) https://github.com/tzutalin/labelImg
• Viola, P., & Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001 (Vol. 1, pp. I-I). IEEE. https://doi.org/10.1109/CVPR.2001.990517
• Vu, T. T., Yamazaki, F., & Matsuoka, M. (2009). Multi-scale solution for building extraction from LiDAR and image data. International Journal of Applied Earth Observation and Geoinformation, 11(4), 281-289. https://doi.org/10.1016/j.jag.2009.03.005
• Yu, B., Liu, H., Wu, J., Hu, Y., & Zhang, L. (2010). Automated derivation of urban building density information using airborne LiDAR data and object-based method. Landscape and Urban Planning, 98(3-4), 210-219. https://doi.org/10.1016/j.landurbplan.2010.08.004
• Yu, Y., Guan, H., Zai, D., & Ji, Z. (2016). Rotation-and-scale-invariant airplane detection in high-resolution satellite images based on deep-Hough-forests. ISPRS Journal of Photogrammetry and Remote Sensing, 112, 50-64. https://doi.org/10.1016/j.isprsjprs.2015.04.014
• Wan, S., & Goudos, S. (2020). Faster R-CNN for multi-class fruit detection using a robotic vision system. Computer Networks, 168 (107036). https://doi.org/10.1016/j.comnet.2019.107036
• Wang, Q., & Tenhunen, J. D. (2004). Vegetation mapping with multitemporal NDVI in North Eastern China transect (NECT). International Journal of Applied Earth Observation and Geoinformation, 6(1), 17-31. https://doi.org/10.1016/j.jag.2004.07.002
• Wu, X., Sahoo, D., & Hoi, S. C. (2020). Recent advances in deep learning for object detection. Neurocomputing, 396, 39-64. doi: 10.1016/j.neucom.2020.01.085
• Xiao, Y., Tian, Z., Yu, J., Zhang, Y., Liu, S., Du, S., & Lan, X. (2020). A review of object detection based on deep learning. Multimedia Tools and Applications, 79(33), 23729-23791. https://doi.org/10.1007/s11042-020-08976-6
• Xu, Y., Yu, G., Wang, Y., Wu, X., & Ma, Y. (2017). Car detection from low-altitude UAV imagery with the faster R-CNN. Journal of Advanced Transportation, https://doi.org/10.1155/2017/2823617
• Very Deep Convolutional Networks for Large-Scale Visual Recognition (2019 October, 15). http://www.robots.ox.ac.uk/~vgg/research/very_deep. Yang, S., Fang, B., Tang, W., Wu, X., Qian, J., & Yang, W. (2017). Faster R-CNN based microscopic cell detection. In 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC) (pp. 345-350). IEEE. https://doi.org/10.1109/SPAC.2017.8304302
• Yao, Y., Jiang, Z., Zhang, H., Cai, B., Meng, G., & Zuo, D. (2017). Chimney and condensing tower detection based on faster R-CNN in high resolution remote sensing images. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) (pp. 3329-3332). IEEE. https://doi.org/10.1109/IGARSS.2017.8127710
• Yin, S., Li, H., & Teng, L. (2020). Airport detection based on improved faster RCNN in large scale remote sensing images. Sensing and Imaging, 21(1), 1-13. https://doi.org/10.1007/s11220-020-00314-2
• Yuhendra, J., & Kuze, H. (2011). Performance analyzing of high resolution pan-sharpening techniques: Increasing image quality for classification using supervised kernel support vector machine. Research Journal of Information Technology, 8(1), 12–28.
• Zeren, M. T., Aytulun, S. K., & Kırelli, Y. (2020). Comparison of SSD and Faster R-CNN algorithms to detect the airports with data set which obtained from unmanned aerial vehicles and satellite images. Avrupa Bilim ve Teknoloji Dergisi, (19), 643-658. https://doi.org/10.31590/ejosat.742789
• Zhang, L., Shi, Z., & Wu, J. (2015). A hierarchical oil tank detector with deep surrounding features for high-resolution optical satellite imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(10), 4895-4909. https://doi.org/10.1109/JSTARS.2015.2467377
• Zhang, S., Wu, R., Xu, K., Wang, J., & Sun, W. (2019). R-CNN-based ship detection from high resolution remote sensing imagery. Remote Sensing, 11(6), 631. https://doi.org/10.3390/rs11060631
• Zhang, Q. X., Lin, G. H., Zhang, Y. M., Xu, G., & Wang, J. J. (2018). Wildland forest fire smoke detection based on faster R-CNN using synthetic smoke images. Procedia Engineering, 211, 441-446. https://doi.org/10.1016/j.proeng.2017.12.034
• Zhu, X. X., Tuia, D., Mou, L., Xia, G. S., Zhang, L., Xu, F., & Fraundorfer, F. (2017). Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5(4), 8-36. https://doi.org/10.1109/MGRS.2017.2762307
• Zhu, Z., & Woodcock, C. E. (2012). Object-based cloud and cloud shadow detection in Landsat imagery. Remote Sensing of Environment, 118, 83-94. https://doi.org/10.1016/j.rse.2011.10.028