Assessing household damages using multi-model deep learning pipeline

Abstract

Since the beginning of the pandemic, the home insurance sector has suffered from various difficulties. One of the most important difficulties was assessing the damages in the insurance owners’ homes. Due to the current pandemic, letting the experts assess the damages in place is a life-threatening risk. Therefore, the idea of automatically assessing the damage is born. This study aims to create a full report for home damages using Convolutional Neural Network (CNN) and various large deep learning model architectures such as EfficientNet, ResNet, U-Net, or Feature Pyramid Network (FPN). Multiple models for tasks such as binary classification and instance segmentation were developed to create an end-to-end reporting pipeline. In more detail, the pipeline consists of two binary classification models and a segmentation model. Binary classification models are responsible for detecting if the picture is indoors and if there is a wall in the picture, whereas the instance segmentation model is responsible for segmenting the damaged parts of the wall class. The effectiveness of the pipeline was measured using different metrics for each task, including accuracy, F1, dice, and Intersection over Union (IoU) scores. The data for each task is labeled by hand and fed to models. The results show that the constructed pipeline can successfully classify and segment the given images according to the needs of our project. The project will affect the home insurance assessment procedure and time spent tremendously by automatizing these repetitive processes.

Keywords

• Deep learning
• convolutional neural networks
• semantic segmentation
• classification

References

1. [1] Di Crosta, A., Ceccato, I., Marchetti, D., La Malva, P., Maiella, R., Cannito, L., Di Domenico, A. (2021) “Psychological factors and consumer behavior during the COVID-19 pandemic”. PloS one, vol. 16, no. 8, e0256095.
2. [2] Perez, H., Tah, J. H., Mosavi, A. (2019) “Deep learning for detecting building defects using convolutional neural networks”. Sensors, vol. 19, no. 16, 3556.
3. [3] Yu, Y., Wang, C., Gu, X., Li, J. (2019) “A novel deep learning-based method for damage identification of smart building structures”. Structural Health Monitoring, vol. 18, no. 1, pp 143-163.
4. [4] Li, Z., Tian, K., Wang, F., Zheng, X., Wang, F. (2016) „Home damage estimation after disasters using crowdsourcing ideas and Convolutional Neural Networks”. In: 2016 5th International Conference on Measurement, Instrumentation and Automation.
5. [5] Feng, C., Zhang, H., Wang, S., Li, Y., Wang, H., Yan, F. (2019) „Structural damage detection using deep convolutional neural network and transfer learning”. KSCE Journal of Civil Engineering, vol. 23, no. 10, pp. 4493-4502.
6. [6] Naito, S., Tomozawa, H., Mori, Y., Nagata, T., Monma, N., Nakamura, H., Shoji, G. (2020) “Building-damage detection method based on machine learning utilizing aerial photographs of the Kumamoto earthquake”. Earthquake Spectra, vol. 36, no. 3, pp. 1166-1187.
7. [7] Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A. A. (2020) “Albumentations: fast and flexible image augmentations”. Information, vol. 11, no. 2, p. 125.
8. [8] Paszke, A. (2019). “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. Advances in Neural Information Processing Systems, vol. 32, pp. 8024-8035.

9. [9] Wightman, R. (2019) PyTorch Image Models. GitHub. doi:10.5281/zenodo.4414861
10. [10] Kingma, D. P., & Ba, J. (2014) “Adam: A method for stochastic optimization”. arXiv preprint arXiv:1412.6980.
11. [11] Tan, M., Le, Q. (2019). “Efficientnet: Rethinking model scaling for convolutional neural networks”. In: International Conference on Machine Learning, pp. 6105-6114.
12. [12] He, K., Zhang, X., Ren, S., & Sun, J. (2016) “Deep residual learning for image recognition”. In: Proc. of the IEEE conference on computer vision and pattern recognition, pp. 770-778.
13. [13] Ronneberger, O., Fischer, P., Brox, T. (2015) „U-net: Convolutional networks for biomedical image segmentation”. In: International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. [14] Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S. (2017) “Feature pyramid networks for object detection”. In: Proc. of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125.