The Role of Ensemble Deep Learning for Building Extraction from VHR Imagery

Abstract

In modern geographical applications, the demand for up-to-date and accurate building maps is increasing, driven by essential needs in sustainable urban planning, sprawl monitoring, natural hazard mitigation, crisis management, smart city initiatives, and the establishment of climate-resilient urban environments. The unregulated growth in urbanization and settlement patterns poses multifaceted challenges, including ecological imbalances, loss of arable land, and increasing risk of drought. Leveraging recent technologies in remote sensing and artificial intelligence, particularly in the fields of very high-resolution satellite imagery and aerial photography, presents promising solutions for rapidly acquiring precise building maps. This research aims to investigate the efficiency of an ensemble deep learning framework comprising DeepLabV3+, UNet++, Pix2pix, Feature Pyramid Network, and Pyramid Scene Parsing Network architectures for the semantic segmentation of buildings. By employing the Wuhan University Aerial Building Dataset, characterized by a spatial resolution of 0.3 meters, as the training and testing dataset, the study assesses the performance of the proposed ensemble model. The findings reveal notable accuracies, with intersection over union metrics reaching 90.22% for DeepLabV3+, 91.01% for UNet++, 83.50% for Pix2pix, 88.90% for FPN, 88.20% for PSPNet, and finally at 91.06% for the ensemble model. These results reveal the potential of integrating diverse deep learning architectures to enhance the precision of building semantic segmentation.

Keywords

• Remote Sensing
• Deep Learning
• Building Extraction
• VHR Imagery
• Ensemble Model

References

1. Cohen, J. E. (2003). Human population: the next half century. Science, 302(5648), 1172-1175. Rees, W. E. (1999). The built environment and the ecosphere: a global perspective. Building Research & Information, 27(4-5), 206-220.
2. McKinney, M. L. (2008). Effects of urbanization on species richness: a review of plants and animals. Urban ecosystems, 11, 161-176.
3. Marzluff, J. M. (2001). Worldwide urbanization and its effects on birds. Avian ecology conservation in an urbanizing world, 19-47.
4. Grimmond, S. (2007). Urbanization and global environmental change: local effects of urban warming. The Geographical Journal, 173(1), 83-88.
5. Chen, W., & Chi, G. (2022). Urbanization and ecosystem services: The multi-scale spatial spillover effects and spatial variations. Land Use Policy, 114, 105964.
6. Abdollahi, A., Pradhan, B., Gite, S., & Alamri, A. (2020). Building footprint extraction from high resolution aerial images using generative adversarial network (GAN) architecture. IEEE Access, 8, 209517-209527.
7. Mousa, Y. A., Helmholz, P., Belton, D., & Bulatov, D. (2019). Building detection and regularisation using DSM and imagery information. The Photogrammetric Record, 34(165), 85-107.
8. Habib, W., Mahmood, S., Noor, S., Saleem, A., Siraj, M., & Ahmad, H. (2023). A post earthquake damage assessment using GIS in district Mirpur, Pakistan. Advanced GIS, 3(2), 53-58.

9. Pehlivan, H. (2021). The Analysis Methodology of Robotic Total Station Data for Determination of Structural Displacements. Advanced Geomatics, 1(1), 1-7.
10. Pala, İ., & Algancı, U. Investigating the performance of super-resolved remote sensing images on coastline segmentation with deep learning based methods. International Journal of Engineering Geosciences, 10(1), 93-106.
11. Varul, Y. E., Adıyaman, H., Bakırman, T., Bayram, B., Alkan, E., Karaca, S. Z., & Topaloğlu, R. H. (2023). Preserving human privacy in real estate listing applications by deep learning methods. Mersin Photogrammetry Journal, 5(1), 10-17.
12. Bovkır, R. (2024). İstanbul'da kentsel yeşil altyapı için çatı tarımı potansiyelinin CBS tabanlı karar analizi ile değerlendirilmesi. Geomatik, 10(1), 45-58.
13. Jabareen, Y. (2013). Planning the resilient city: Concepts and strategies for coping with climate change and environmental risk. Cities, 31, 220-229.
14. Liu, R., Wang, M., Hou, G., Wu, W., Zhao, C., & Ge, Q. (2023). The classification of airborne LiDAR building point clouds based on multi‐scale and multi‐level cloth simulation. The Photogrammetric Record, 38(182), 118-136.
15. Akbulut, Z., Özdemir, S., Acar, H., Dihkan, M., & Karslı, F. (2018). Automatic extraction of building boundaries from high resolution images with active contour segmentation. International Journal of Engineering Geosciences, 3(1), 36-42.
16. Tabakoğlu, C. (2024). A Review: Detection types and systems in remote sensing. Advanced GIS, 4(2), 100-105.
17. Sariturk, B., Bayram, B., Duran, Z., & Seker, D. Z. (2020). Feature extraction from satellite images using segnet and fully convolutional networks (FCN). International Journal of Engineering Geosciences, 5(3), 138-143.
18. Şenol, H. İ., Kaya, Y., Yiğit, A. Y., & Yakar, M. (2024). Extraction and geospatial analysis of the Hersek Lagoon shoreline with Sentinel-2 satellite data. Survey Review, 56(397), 367-382.
19. Ünyılmaz, C. N., Kulavuz, B., Bakırman, T., & Bayram, B. (2024). Deep Learning Based Panchromatic2RGB Image Generation from VHR Images. Mersin Photogrammetry Journal, 6(2), 87-92.
20. Bayram, B., Kılıç, B., Özoğlu, F., Erdem, F., Bakirman, T., Sivri, S., Delen, A. (2020). A deep learning integrated mobile application for historic landmark recognition: A case study of Istanbul. Mersin Photogrammetry Journal, 2(2), 38-50.
21. Wang, H., & Miao, F. (2022). Building extraction from remote sensing images using deep residual U-Net. European Journal of Remote Sensing, 55(1), 71-85.
22. Li, J., Huang, X., Tu, L., Zhang, T., & Wang, L. (2022a). A review of building detection from very high resolution optical remote sensing images. GIScience Remote Sensing, 59(1), 1199-1225.
23. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84-90.
24. Bakirman, T., Komurcu, I., & Sertel, E. (2022). Comparative analysis of deep learning based building extraction methods with the new VHR Istanbul dataset. Expert Systems with Applications, 202, 117346.
25. Şengül, G. S., & Sertel, E. (2024). Automatic Building Extraction From VHR Remote Sensing Images Using Geoai Methods. IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium,
26. Chen, D.-Y., Peng, L., Li, W.-C., & Wang, Y.-D. (2021a). Building extraction and number statistics in WUI areas based on UNet structure and ensemble learning. Remote Sensing, 13(6), 1172.
27. Erdem, F., Bayram, B., Bakirman, T., Bayrak, O. C., & Akpinar, B. (2021). An ensemble deep learning based shoreline segmentation approach (WaterNet) from Landsat 8 OLI images. Advances in Space Research, 67(3), 964-974.
28. Marmanis, D., Schindler, K., Wegner, J. D., Galliani, S., Datcu, M., & Stilla, U. (2018). Classification with an edge: Improving semantic image segmentation with boundary detection. ISPRS Journal of Photogrammetry Remote Sensing, 135, 158-172.
29. Guo, Z., Chen, Q., Wu, G., Xu, Y., Shibasaki, R., & Shao, X. (2017). Village building identification based on ensemble convolutional neural networks. Sensors, 17(11), 2487.
30. Li, W., He, C., Fang, J., Zheng, J., Fu, H., & Yu, L. (2019). Semantic segmentation-based building footprint extraction using very high-resolution satellite images and multi-source GIS data. Remote Sensing, 11(4), 403.
31. Kaya, Y., Şenol, H. İ., Yiğit, A. Y., & Yakar, M. (2023). Car detection from very high-resolution UAV images using deep learning algorithms. Photogrammetric Engineering & Remote Sensing, 89(2), 117-123.
32. Abdollahi, A., Pradhan, B., & Alamri, A. (2022). An ensemble architecture of deep convolutional Segnet and Unet networks for building semantic segmentation from high-resolution aerial images. Geocarto International, 37(12), 3355-3370.
33. Wang, S., Zang, Q., Zhao, D., Fang, C., Quan, D., Wan, Y., Jiao, L. (2023). Select, purify, and exchange: A multisource unsupervised domain adaptation method for building extraction. IEEE Transactions on Neural Networks Learning Systems.
34. Ji, S., Wei, S., & Lu, M. (2019a). Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set. IEEE Transactions on geoscience remote sensing, 57(1), 574-586.
35. Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2015). Semantic image segmentation with deep convolutional nets and fully connected crfs. International conference on learning representations, San Diego, United States.
36. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV),
37. Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018). Unet++: A nested u-net architecture for medical image segmentation. Deep learning in medical image analysis and multimodal learning for clinical decision support: 4th international workshop, DLMIA 2018, and 8th international workshop, ML-CDS 2018, held in conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, proceedings 4,
38. Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition,
39. Akçay, Ö., Erenoğlu, R. C., & Avşar, E. Ö. (2017). The effect of jpeg compression in close range photogrammetry. International Journal of Engineering and Geosciences, 2(1), 35-40. https://doi.org/10.26833/ijeg.287308.
40. Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition,
41. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition,
42. Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017). Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition,
43. Bekçi, R. N., Zorlu, Ö., & Menekşe, E. (2022). Regression analysis and use of artificial neural networks in housing valuation forecasting: case example of Güvenevler neighbourhood in Mersin. Advanced GIS, 2(1), 24-32.
44. Hansen, L. K., & Salamon, P. (1990). Neural network ensembles. IEEE transactions on pattern analysis machine intelligence, 12(10), 993-1001.
45. Schapire, R. E. (1990). The strength of weak learnability. Machine learning, 5, 197-227.
46. Pleșoianu, A.-I., Stupariu, M.-S., Șandric, I., Pătru-Stupariu, I., & Drăguț, L. (2020). Individual tree-crown detection and species classification in very high-resolution remote sensing imagery using a deep learning ensemble model. Remote Sensing, 12(15), 2426.
47. Ben Jabra, M., Koubaa, A., Benjdira, B., Ammar, A., & Hamam, H. (2021). COVID-19 diagnosis in chest X-rays using deep learning and majority voting. Applied Sciences, 11(6), 2884.
48. Tandel, G. S., Tiwari, A., & Kakde, O. G. (2021). Performance optimisation of deep learning models using majority voting algorithm for brain tumour classification. Computers in Biology Medicine, 135, 104564.
49. Lam, L., & Suen, C. Y. (1995). Optimal combinations of pattern classifiers. Pattern Recognition Letters, 16(9), 945-954.
50. Ye, Z., Fu, Y., Gan, M., Deng, J., Comber, A., & Wang, K. (2019). Building extraction from very high resolution aerial imagery using joint attention deep neural network. Remote Sensing, 11(24), 2970.
51. Kang, W., Xiang, Y., Wang, F., & You, H. (2019). EU-Net: An efficient fully convolutional network for building extraction from optical remote sensing images. Remote Sensing, 11(23), 2813.
52. Liu, P., Liu, X., Liu, M., Shi, Q., Yang, J., Xu, X., & Zhang, Y. (2019b). Building footprint extraction from high-resolution images via spatial residual inception convolutional neural network. Remote Sensing, 11(7), 830.
53. Liu, H., Luo, J., Huang, B., Hu, X., Sun, Y., Yang, Y.,…Zhou, N. (2019a). DE-Net: Deep encoding network for building extraction from high-resolution remote sensing imagery. Remote Sensing, 11(20), 2380.
54. Lin, J., Jing, W., Song, H., & Chen, G. (2019). ESFNet: Efficient network for building extraction from high-resolution aerial images. IEEE Access, 7, 54285-54294.
55. Ji, S., Wei, S., & Lu, M. (2019b). A scale robust convolutional neural network for automatic building extraction from aerial and satellite imagery. International Journal of Remote Sensing, 40(9), 3308-3322.
56. He, N., Fang, L., & Plaza, A. (2020). Hybrid first and second order attention Unet for building segmentation in remote sensing images. Science China Information Sciences, 63, 1-12.
57. Wang, S., Hou, X., & Zhao, X. (2020). Automatic building extraction from high-resolution aerial imagery via fully convolutional encoder-decoder network with non-local block. IEEE Access, 8, 7313-7322.
58. Liu, Y., Zhou, J., Qi, W., Li, X., Gross, L., Shao, Q., Li, Z. (2020). ARC-Net: An efficient network for building extraction from high-resolution aerial images. IEEE Access, 8, 154997-155010.
59. Zhu, Q., Li, Z., Zhang, Y., & Guan, Q. (2020). Building extraction from high spatial resolution remote sensing images via multiscale-aware and segmentation-prior conditional random fields. Remote Sensing, 12(23), 3983.
60. Zhou, D., Wang, G., He, G., Long, T., Yin, R., Zhang, Z., Luo, B. (2020). Robust building extraction for high spatial resolution remote sensing images with self-attention network. Sensors, 20(24), 7241.
61. Jing, W., Lin, J., & Wang, H. (2020). Building NAS: Automatic designation of efficient neural architectures for building extraction in high-resolution aerial images. Image Vision Computing, 103, 104025.
62. Yang, G., Zhang, Q., & Zhang, G. (2020). EANet: Edge-aware network for the extraction of buildings from aerial images. Remote Sensing, 12(13), 2161.
63. Deng, W., Shi, Q., & Li, J. (2021). Attention-gate-based encoder–decoder network for automatical building extraction. IEEE Journal of Selected Topics in Applied Earth Observations Remote Sensing, 14, 2611-2620.
64. Chen, M., Wu, J., Liu, L., Zhao, W., Tian, F., Shen, Q., Du, R. (2021c). DR-Net: An improved network for building extraction from high resolution remote sensing image. Remote Sensing, 13(2), 294.
65. Chen, Z., Li, D., Fan, W., Guan, H., Wang, C., & Li, J. (2021d). Self-attention in reconstruction bias U-Net for semantic segmentation of building rooftops in optical remote sensing images. Remote Sensing, 13(13), 2524.
66. Chen, K., Zou, Z., & Shi, Z. (2021b). Building extraction from remote sensing images with sparse token transformers. Remote Sensing, 13(21), 4441.
67. Jung, H., Choi, H.-S., & Kang, M. (2022). Boundary enhancement semantic segmentation for building extraction from remote sensed image. IEEE Transactions on geoscience remote sensing, 60, 1-12.
68. Li, Z., Xin, Q., Sun, Y., & Cao, M. (2021). A deep learning-based framework for automated extraction of building footprint polygons from very high-resolution aerial imagery. Remote Sensing, 13(18), 3630.
69. Liu, S., Ye, H., Jin, K., & Cheng, H. (2021). CT-UNet: Context-transfer-UNet for building segmentation in remote sensing images. Neural Processing Letters, 53, 4257-4277.
70. Seong, S., & Choi, J. (2021). Semantic segmentation of urban buildings using a high-resolution network (HRNet) with channel and spatial attention gates. Remote Sensing, 13(16), 3087.
71. Feng, D., Chen, H., Xie, Y., Liu, Z., Liao, Z., Zhu, J., & Zhang, H. (2022). GCCINet: Global feature capture and cross-layer information interaction network for building extraction from remote sensing imagery. International Journal of Applied Earth Observation Geoinformation, 114, 103046.
72. Li, W., Sun, K., Zhao, H., Li, W., Wei, J., & Gao, S. (2022b). Extracting buildings from high-resolution remote sensing images by deep ConvNets equipped with structural-cue-guided feature alignment. International Journal of Applied Earth Observation Geoinformation, 113, 102970.
73. Huang, L., Zhu, J., Qiu, M., Li, X., & Zhu, S. (2022). CA-BASNet: A Building Extraction Network in High Spatial Resolution Remote Sensing Images. Sustainability, 14(18), 11633.
74. Qiu, Y., Wu, F., Yin, J., Liu, C., Gong, X., & Wang, A. (2022). MSL-Net: An efficient network for building extraction from aerial imagery. Remote Sensing, 14(16), 3914.
75. Zhao, H., Zhang, H., & Zheng, X. (2022). A multiscale attention-guided UNet++ with edge constraint for building extraction from high spatial resolution imagery. Applied Sciences, 12(12), 5960.
76. Xiao, X., Guo, W., Chen, R., Hui, Y., Wang, J., & Zhao, H. (2022). A swin transformer-based encoding booster integrated in u-shaped network for building extraction. Remote Sensing, 14(11), 2611.
77. Yang, S., He, Q., Lim, J. H., & Jeon, G. (2024). RETRACTED ARTICLE: Boundary-guided DCNN for building extraction from high-resolution remote sensing images. The International Journal of Advanced Manufacturing Technology, 132(9), 5171-5171.
78. Yu, M., Zhang, W., Chen, X., Liu, Y., & Niu, J. (2022). An End-to-End Atrous Spatial Pyramid Pooling and Skip-Connections Generative Adversarial Segmentation Network for Building Extraction from High-Resolution Aerial Images. Applied Sciences, 12(10), 5151.
79. Chen, J., Zhang, D., Wu, Y., Chen, Y., & Yan, X. (2022). A context feature enhancement network for building.