Road Crack Detection Using Deep Neural Networks Developed via Cooperative Coevolution with Backpropagation

Abstract

Cracks in highway flexible pavements are primarily caused by traffic loads and weather conditions. Detecting these cracks before they expand and conducting necessary maintenance will not only ensure the continuity of road comfort but will also reduce maintenance costs. This study aims to detect cracks on the road in real-time and with high accuracy. In this context, Deep Neural Networks Developed via Cooperative Coevolution with Backpropagation and image processing methods were used together. Moreover, a new data set was obtained by using EdmCrack600, AsphaltCrack, CFD, and CrackSegmentation datasets containing cracked visual data in various numbers and resolutions, and Deep Neural Networks-based learning was performed on this dataset. The accuracy of the model was evaluated with Precision, Recall, and F1-Score using the CFD dataset. As a result of the evaluation, it has been observed that the proposed method can detect cracks on 48 images per second, while it can reach 92.74% Precision, 88.92% Recall, and 89.61% F1-Score success rates.

Keywords

• Road Crack Detection
• Cooperative Coevolution with Backpropagation
• Deep Learning
• Image Processing
• Flexible Pavement

GERİ YAYILIMLI BİRLİKTE EVRİM İLE İYİLEŞTİRİLMİŞ DERİN SİNİR AĞLARI KULLANILARAK YOL ÇATLAK TESPİTİ

Abstract

Karayolu esnek üstyapılarındaki çatlaklar genellikle trafik yükleri ve hava koşullarından kaynaklanır. Bu çatlakların genişlemeden tespit edilip gerekli bakımlarının yapılması, yol konforunun sürekliliğini sağlamanın yanı sıra bakım maliyetlerini de azaltacaktır. Bu çalışma, yoldaki çatlakları gerçek zamanlı ve yüksek doğrulukla tespit etmeyi amaçlamaktadır. Bu bağlamda, Geri Yayımlı Birlikte Evrim yaklaşımıyla İyileştirilmiş Derin Sinir Ağları ve görüntü işleme yöntemleri birlikte kullanılmıştır. Ayrıca, çeşitli sayı ve çözünürlüklerde çatlaklı görsel veriler içeren EdmCrack600, AsphaltCrack, CFD ve CrackSegmentation veri setleri kullanılarak yeni bir veri seti oluşturulmuş ve bu veri seti üzerinde Derin Sinir Ağları tabanlı öğrenme gerçekleştirilmiştir. Modelin doğruluğu, CFD veri seti kullanılarak Kesinlik, Duyarlılık ve F1-Skoru ile değerlendirilmiştir. Değerlendirme sonucunda, önerilen yöntemin saniyede 48 görsel üzerinde çatlak tespit edebildiği ve %92,74 Kesinlik, %88,92 Duyarlılık ve %89,61 F1 Skoru başarı oranlarına ulaştığı gözlemlenmiştir.

Keywords

• Geri Yayılımlı Birlikte Evrim
• Derin Sinir Ağları
• Çatlak Tespiti
• Görüntü İşleme
• Esnek Üstyapı

References

1. Amhaz, R., Chambon, S., Idier, J. ve Baltazart, V. (2016) Automatic Crack Detection on Two-Dimensional Pavement Images: An Algorithm Based on Minimal Path Selection, IEEE Transactions on Intelligent Transportation Systems, 17(10), 2718-2729. doi:10.1109/TITS.2015.2477675
2. Badrinarayanan, V., Kendall, A. ve Cipolla, R. (2017) SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481-2495. doi:10.1109/TPAMI.2016.2644615
3. Cha, Y., Choi, W., Suh, G., Mahmoudkhani, S. ve Büyüköztürk, O. (2018) AutonomousStructural Visual Inspection Using Region‐Based Deep Learning for Detecting Multiple Damage Types, Computer-Aided Civil and Infrastructure Engineering, 33(9), 731-747. doi:10.1111/mice.12334
4. Cha, Y.-J., You, K. ve Choi, W. (2016) Vision-based detection of loosened bolts using the Hough transform and support vector machines, Automation in Construction, 71, 181-188. doi:10.1016/j.autcon.2016.06.008
5. Chen, T., Cai, Z., Zhao, X., Chen, C., Liang, X., Zou, T. ve Wang, P. (2020) Pavement crack detection and recognition using the architecture of segNet, Journal of Industrial Information Integration, 18, 100144. doi:10.1016/j.jii.2020.100144
6. Cui, L., Qi, Z., Chen, Z., Meng, F. ve Shi, Y. (2015) Pavement Distress Detection Using Random Decision Forests (ss. 95-102). doi:10.1007/978-3-319-24474-7_14
7. Doğan, G. ve Ergen, B. (2022) Karayollarındaki Asfalt Çatlaklarının Tespiti İçin Yeni Bir Konvolüsyonel Sinir Ağı Tabanlı Yöntem, Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 34(2), 485-494. doi:10.35234/fumbd.1014951
8. Du, Y., Pan, N., Xu, Z., Deng, F., Shen, Y. ve Kang, H. (2021) Pavement distress detection and classification based on YOLO network, International Journal of Pavement Engineering, 22(13), 1659-1672. doi:10.1080/10298436.2020.1714047

9. Dung, C. V. ve Anh, L. D. (2019) Autonomous concrete crack detection using deep fully convolutional neural network, Automation in Construction, 99, 52-58. doi:10.1016/j.autcon.2018.11.028
10. Fan, J., Bocus, M. J., Wang, L. ve Fan, R. (2021) Deep Convolutional Neural Networks for Road Crack Detection: Qualitative and Quantitative Comparisons, 2021 IEEE International Conference on Imaging Systems and Techniques (IST) (ss. 1-6), IEEE. doi:10.1109/IST50367.2021.9651375
11. Fan, Z., Lin, H., Li, C., Su, J., Bruno, S. ve Loprencipe, G. (2022) Use of Parallel ResNet for High-Performance Pavement Crack Detection and Measurement, Sustainability (Switzerland), 14(3). doi:10.3390/su14031825
12. Fei, Y., Wang, K. C. P., Zhang, A., Chen, C., Li, J. Q., Liu, Y., … Li, B. (2020) Pixel-Level Cracking Detection on 3D Asphalt Pavement Images Through Deep-Learning- Based CrackNet-V, IEEE Transactions on Intelligent Transportation Systems, 21(1), 273-284. doi:10.1109/TITS.2019.2891167
13. Gao Z., Zhao X., Cao M., Li Z., Liu K. ve Chen B. M. (2023) Synergizing Low Rank Representation and Deep Learning for Automatic Pavement Crack Detection, IEEE Transactions on Intelligent Transportation Systems, 24(10), 10676-10690. doi: 10.1109/TITS.2023.3275570
14. Gavilán, M., Balcones, D., Marcos, O., Llorca, D. F., Sotelo, M. A., Parra, I., … Amírola, A. (2011) Adaptive Road Crack Detection System by Pavement Classification, Sensors, 11(10), 9628-9657. doi:10.3390/s111009628
15. Gong, M., Liu, J., Qin, A. K., Zhao, K. ve Tan, K. C. (2021) Evolving Deep Neural Networks via Cooperative Coevolution With Backpropagation, IEEE Transactions on Neural Networks and Learning Systems, 32(1), 420-434. doi:10.1109/TNNLS.2020.2978857
16. Huyan, J., Li, W., Tighe, S., Xu, Z. ve Zhai, J. (2020) CrackU‐net: A novel deep convolutional neural network for pixelwise pavement crack detection, Structural Control and Health Monitoring, 27(8). doi:10.1002/stc.2551
17. Jayanth Balaji, A., Thiru Balaji, G., Dinesh, M. S., Binoy, N. ve Harish Ram, D. S. (2019) Asphalt Crack Dataset, Mendeley Data. doi:10.17632/xnzhj3x8v4.2
18. Krizhevsky, A., Sutskever, I. ve Hinton, G. E. (2017) ImageNet classification with deep convolutional neural networks, Communications of the ACM, 60(6), 84-90. doi:10.1145/3065386
19. Lee, B. J. ve Lee, H. “David”. (2004) Position‐Invariant Neural Network for Digital Pavement Crack Analysis, Computer-Aided Civil and Infrastructure Engineering, 19(2), 105-118. doi:10.1111/j.1467-8667.2004.00341.x
20. Liu, J., Yang, X., Lau, S., Wang, X., Luo, S., Lee, V. C. ve Ding, L. (2020) Automated pavement crack detection and segmentation based on two‐step convolutional neural network, Computer-Aided Civil and Infrastructure Engineering, 35(11), 1291-1305. doi:10.1111/mice.12622
21. Liu, K. ve Chen, B. M. (2023) Industrial UAV-Based Unsupervised Domain Adaptive Crack Recognitions: From Database Towards Real-Site Infrastructural Inspections, IEEE Transactions on Industrial Electronics, 70(9), 9410-9420. doi:10.1109/TIE.2022.3204953
22. Liu, K., Han, X. ve Chen, B. M. (2019) Deep Learning Based Automatic Crack Detection and Segmentation for Unmanned Aerial Vehicle Inspections, 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO) (ss. 381-387), IEEE. doi:10.1109/ROBIO49542.2019.8961534
23. Ma, D., Fang, H., Wang, N., Xue, B., Dong, J. ve Wang, F. (2022) A real-time crack detection algorithm for pavement based on CNN with multiple feature layers, Road Materials and Pavement Design, 23(9), 2115-2131. doi:10.1080/14680629.2021.1925578
24. Mei, Q. ve Gül, M. (2020) A cost effective solution for pavement crack inspection using cameras and deep neural networks, Construction and Building Materials, 256, 119397. doi:10.1016/j.conbuildmat.2020.119397
25. Mei, Q., Gül, M. ve Azim, M. R. (2020) Densely connected deep neural network considering connectivity of pixels for automatic crack detection, Automation in Construction, 110, 103018. doi:10.1016/j.autcon.2019.103018
26. Moon, H.-G. ve Kim, J.-H. (2011) Inteligent Crack Detecting Algorithm on the Concrete Crack Image Using Neural Network, International Association for Automation and Robotics in Construction (IAARC). doi:10.22260/ISARC2011/0279
27. Munawar, H. S., Hammad, A. W. A., Haddad, A., Soares, C. A. P. ve Waller, S. T. (2021) Image-Based Crack Detection Methods: A Review, Infrastructures, 6(8), 115. doi:10.3390/infrastructures6080115
28. Naddaf-Sh, S., Naddaf-Sh, M.-M., Kashani, A. R. ve Zargarzadeh, H. (2020) An Efficient and Scalable Deep Learning Approach for Road Damage Detection, 2020 IEEE International Conference on Big Data (Big Data) (ss. 5602-5608), IEEE. doi:10.1109/BigData50022.2020.9377751
29. Nguyen, T. S., Begot, S., Duculty, F. ve Avila, M. (2011) Free-form anisotropy: A new method for crack detection on pavement surface images, 2011 18th IEEE International Conference on Image Processing (ss. 1069-1072), IEEE. doi:10.1109/ICIP.2011.6115610
30. O’Byrne, M., Schoefs, F., Ghosh, B. ve Pakrashi, V. (2013) Texture Analysis Based Damage Detection of Ageing Infrastructural Elements, Computer-Aided Civil and Infrastructure Engineering, 28(3), 162-177. doi:10.1111/j.1467-8667.2012.00790.x
31. Oliveira, H. ve Correia, P. (2009) Automatic road crack segmentation using entropy and image dynamic thresholding, 2009 17th European Signal Processing Conference, 622-626. https://ieeexplore.ieee.org/document/6302929 adresinden erişildi.
32. Oliveira, H. ve Correia, P. L. (2014) Automated Visual Inspection of Pavement Crack Detection and Characterization, https://api.semanticscholar.org/CorpusID:9409925 adresinden erişildi.
33. Peng, L., Chao, W., Shuangmiao, L. ve Baocai, F. (2015) Research on Crack Detection Method of Airport Runway Based on Twice-Threshold Segmentation, 2015 Fifth International Conference on Instrumentation and Measurement, Computer, Communication and Control (IMCCC) (ss. 1716-1720), IEEE. doi:10.1109/IMCCC.2015.364
34. Potter, M. A. ve Jong, K. A. (1994) A cooperative coevolutionary approach to function optimization (ss. 249-257). doi:10.1007/3-540-58484-6_269
35. Qin, A. K., Huang, V. L. ve Suganthan, P. N. (2009) Differential Evolution Algorithm With Strategy Adaptation for Global Numerical Optimization, IEEE Transactions on Evolutionary Computation, 13(2), 398-417. doi:10.1109/TEVC.2008.927706
36. Qu, Z., Chen, W., Wang, S.-Y., Yi, T.-M. ve Liu, L. (2022) A Crack Detection Algorithm for Concrete Pavement Based on Attention Mechanism and Multi-Features Fusion, IEEE Transactions on Intelligent Transportation Systems, 23(8), 11710-11719. doi:10.1109/TITS.2021.3106647
37. Rumelhart, D. E. ve McClelland, J. L. (1987) Learning Internal Representations by Error Propagation, Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations (ss. 318-362). https://ieeexplore.ieee.org/document/6302929 adresinden erişildi.
38. Santhi, B., Krishnamurthy, G., .S, S. ve Ramakrishnan, P. K. (2012) Automatic detection of cracks in pavements using edge detection operator, Journal of Theoretical and Applied Information Technology, 36, 199-205. https://www.researchgate.net/publication/289942204_Automatic_detection_of_cracks_in_p avements_using_edge_detection_operator adresinden erişildi.
39. Shi, Y., Cui, L., Qi, Z., Meng, F. ve Chen, Z. (2016) Automatic Road Crack Detection Using Random Structured Forests, IEEE Transactions on Intelligent Transportation Systems, 17(12), 3434-3445. doi:10.1109/TITS.2016.2552248
40. Staniek, M. (2017) Detection of cracks in asphalt pavement during road inspection processes, Scientific Journal of Silesian University of Technology, Series Transport, 96, 175-184. doi:10.20858/sjsutst.2017.96.16
41. Tsai, Y.-C., Kaul, V. ve Mersereau, R. M. (2010) Critical Assessment of Pavement Distress Segmentation Methods, Journal of Transportation Engineering, 136(1), 11-19. doi:10.1061/(ASCE)TE.1943-5436.0000051
42. Vyas, V., Singh, A. P. ve Srivastava, A. (2019) Entropy-based fuzzy SWOT decision-making for condition assessment of airfield pavements, International Journal of Pavement Engineering, 22(10), 1226–1237. doi:10.1080/10298436.2019.1671590
43. Wang, W. ve Su, C. (2021) Deep Learning-Based Real-Time Crack Segmentation for Pavement Images, KSCE Journal of Civil Engineering, 25(12), 4495-4506. doi:10.1007/s12205-021-0474-2
44. Xin Yao. (1999) Evolving artificial neural networks, Proceedings of the IEEE, 87(9), 1423- 1447. doi:10.1109/5.784219
45. Yang, M., Yu, K., Zhang, C., Li, Z. ve Yang, K. (2018) DenseASPP for Semantic Segmentation in Street Scenes, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (ss. 3684-3692), IEEE. doi:10.1109/CVPR.2018.00388
46. Yeum, C. M. ve Dyke, S. J. (2015) Vision‐Based Automated Crack Detection for Bridge Inspection, Computer-Aided Civil and Infrastructure Engineering, 30(10), 759-770. doi:10.1111/mice.12141
47. Zhang, A. A., Wang, K. C. P., Liu, Y., Zhan, Y., Yang, G., Wang, G., … Shang, J. (2022) Intelligent pixel‐level detection of multiple distresses and surface design features on asphalt pavements, Computer-Aided Civil and Infrastructure Engineering, 37(13), 1654-1673. doi:10.1111/mice.12909
48. Zhang T., Wang D. ve Lu Y. (2023) ECSNet: An Accelerated Real-Time Image Segmentation CNN Architecture for Pavement Crack Detection, IEEE Transactions on Intelligent Transportation Systems, 24(12), 15105-15112. doi: 10.1109/TITS.2023.3300312
49. Zhou, J. (2006) Wavelet-based pavement distress detection and evaluation, Optical Engineering, 45(2), 027007. doi:10.1117/1.2172917
50. Zhu, G., Liu, J., Fan, Z., Yuan, D., Ma, P., Wang, M., … Wang, K. C. P. (2023) A lightweight encoder–decoder network for automatic pavement crack detection, Computer-Aided Civil and Infrastructure Engineering. doi:10.1111/mice.13103
51. Zou, Q., Cao, Y., Li, Q., Mao, Q. ve Wang, S. (2012) CrackTree: Automatic crack detection from pavement images, Pattern Recognition Letters, 33(3), 227-238. doi:10.1016/j.patrec.2011.11.004
52. Zou, Q., Li, Q., Zhang, F., Xiong Qian Wang, Z. ve Wang, Q. (2016) Path voting based pavement crack detection from laser range images, 2016 IEEE International Conference on Digital Signal Processing (DSP) (ss. 432-436), IEEE. doi:10.1109/ICDSP.2016.7868594