A Fast and Adaptive Road Defect Detection Approach Using Computer Vision with Real Time Implementation

: Road defect is one of the most important factors for traffic accident. Therefore, these defects should be corrected as soon as possible. It usually occurs cracks, rutting, and potholes in road surface. These errors are based on the fact that people have recognized and fixed these errors in our day. But if these errors are not corrected in a short time, the size of the error grows day by day. There are various methods used to detect road errors in the literature. One of these methods is the use of computer vision. There are various types of roads in real life. Since the studies in the literature have been carried out only by taking into account one type of road, the accuracy rates decrease when these studies are used in different types of roads. In the study carried out, different roads have been made adaptive by the operations performed in the detection of road errors from the received images. Images taken from the camera on a vehicle are used for the study. The study applied is ensured to have high accuracy rates in different types of roads via customization. In the second stage, the image blurred by using median filter and the unprocessed images are collected, and the darkest parts of the image are brought into the forefront. The image is converted into a binary image and improved by mathematical morphological operations. As a result of the operations performed, which of the five classes including un-cracked roads, superficial crack, crocodile crack, linear crack and transverse crack the roads belong to is determined. In the study carried out, the fact that it is fast and that its accuracy rates are good indicate that it can be used in real life.


Introduction
Roads are the transportation method which is most widely used around the world at the present time.One of the most important causes of accidents on the roads is the road surface faults.The improper operations in the construction phase and the reasons such as pressure caused by heavy vehicles on the roads and weather condition lead to the deformations on the road surfaces over time.These deformations on the road surfaces are called Alligator Cracking, Block cracks, Linear Cracking, Transverse Cracking, Joint Reflection Cracks etc.The elimination of the road defects is both a big work load and an organizational operation [1].Today, road defect detection based on the fact that people see these defects.This method prevents the elimination of defects in a short time.Many researchers have proposed different methods about how to detect the defects in a faster way.The features such as being cheap, being fast and applicable determine the effectiveness of a method [2].Among these methods, one of the cheapest and fastest method is the algorithms developed using image processing methods.The crack in the image taken and the contrast between the road are reduced due to the light on the road surface and the changes in weather conditions [3].The quality of the camera also affects the operations performed.The quality of the camera is determined by lens, digital image sensor and microcontroller [27].It is difficult to perform feature extraction from the road surfaces using image processing methods and to detect feature due to the plurality of the colour changes of the road surfaces [4].These colour changes cause the current program to be successful just on a type of road and the performance on the other road surface to decrease.In order to avoid this problem, customization is performed for different roads in the study carried out.
Edge detection methods are very efficient for feature extraction in the other detection problems [5][6][7], but performing feature extraction from the road surface is not only edge detection or texture analysis problem [8].Colour changes and noise in received image make difficult to defect detection.In this case, improvement of the image becomes an important duty in order to make an accurate prediction [9].However, pre-processing continues to be a challenging task due to reasons such as the fact that noises and cracks have the same texture [15], the presence of traces such as tire tracks, oil spills etc. on the road surfaces [19], the fact that cracks do not have the same density, and low contrast [21] etc.The road cracks constitute the continuous and darkest region of the image [10].By taking into account these features in [18], it was effective in the use of existing minimal path approaches for automatic crack detection.In the study carried out, the cracked structure was brought to the forefront to improve the image by using a median filter and collecting with an image on which no processing was performed.In the literature, productive results were also obtained by using median filter [12,13,20] to improve image.In addition to median filter, histogram equalization [8], calculate the threshold value [10], etc. were used for image improving.Apart from these methods, various methods have been proposed to get rid of the factors reducing the accuracy rate of the system such as shadow [16].Feature extraction defines the relevant shape information in the image [11].In the literature, edge control [14], Histograms of oriented gradients (HOG) features [17] and various parameters such as slope, inertia [10] which can be removed through image were used for feature extraction.For instance, Guanqun Bao [12] proposed feature mapping method.This method was based on from vertical histogram and horizontal histogram values.
The usability of road defect detection applications depends on being able to achieve fast and good results.In accordance with this purpose, an attempt to achieve faster and better results was made in the study carried out.If deep cracks on the road are not repaired in a short time, the damage on the road is further increased.Therefore, it is necessary to make the distinction of superficial and deep cracks.In the study carried out, roads were classified as without crack, superficial crack, crocodile cracking, transverse cracking and longitudinal (linear) cracking.

Proposed Work
In this section, a method that gives good results for different roads is introduced.The flow diagram of the study is presented in The colour of the road surface is determined in the first stage of the study carried out.With this stage, an efficient method is developed for all road types by performing the customization of the process to be performed at a later stage.The subsequent stages are image enhancement, feature extraction and classification stages respectively.This operation is implemented with an adaptive subprogram.Real-time and feature extraction process is carried out in many computer vision systems [28].The study carried out is one of them.

Road surface colour detection
Various problems are encountered in the design of a method that can be used in daily life.One of these problems is that the colour of the road surface changes as the structure changes.This situation causes it to be a program that works well for a type but disrupts the characteristic of the cracked structure or incorrectly calculates non-defective segments in other road types when we try to detect road surface faults.In the study carried out, this problem was avoided by performing customization in later operations depending on the colour of the road taken.The images to be processed are obtained as RGB.The fact that the image is gray level image or binary image makes it difficult to distinguish some roads.Therefore, the road is calculated by calculating the mean of the RGB value of the first received image when the program is first run.The system increases the accuracy rate without slowing down by performing the process just for once.The flow diagram of the colour control is presented in Fig.

Image Improvement
It is difficult to extract data through the obtained images.Because the received image may be noise, low contrast etc.In previous studies, an attempt to detect the road cracks is made by the information that the road cracks in [10] constitute the continuous and darkest region of the image.In the recent study, [15] indicates that this situation is not that easy, noises with the same texture with the noises in the images obtained may exist, and will not be able to get the cracked structure exactly.However, in the classification process, it will be sufficient to improve the image so that this faulty condition can be eliminated.In other words, eliminating all or a large part of the noises in the image discussed will allow us to get accurate results in the classification process.Therefore, image improvement stage in the study is the most important stage.The flow diagram of the image improvement stage is presented in Fig. 3.The operations required for the detection of the darkest region of the image are obtained more comfortably through the gray scale image.Median filter can be generally used to minimize the angular blurred degradations, soften the flat region and protect the edges [22].The fact that the Median filter softens straight areas and protects edges enables cracked structure to further come to the forefront [29].The customization of the process of the median filter in different background colours on the received image increases the accuracy rate of the operations performed.More accurate results are obtained if there is more fuzzification as the calculated mean number decreases.The cracked structure was brought to the forefront by collecting the blurred image and the gray scale image on which no processing was performed.Not only fracture parts were taken to the forefront, reduce noise as well in the image obtained after the collecting process.The images obtained after adding operation for two different road images are presented in Fig. 4.  Morphological operations were carried out in order to correct the sections which were seen as if they were disconnected on the image and to get rid of some undesired defects.The image was transformed into binary image in order to perform mathematical morphological operations and to achieve the cracked structure in a more accurate way.Mathematical morphological operations are the commonly used methods in the operations of the geometrical structures.Morphological medians are hierarchically represented in similar dimensions among spatial areas [23].Morphological operations are frequently exposed to scientific thinking and researches like morphological profiles, because Morphological operations are known to be useful for the classification of panchromatic and hyperspectral images with very high spatial resolution [24].Erosion, dilation, opening and closing are the basic morphological operations.The input image and output image are generated in the same dimension.The pixel values in the processed image calculated by the neighbouring pixels.The mathematical morphological operations of erosion and dilation were applied in order to improve the images of the cracks.In erosion, it is performed by giving the smallest of the neighbouring pixels.If the pixel is equal to '0', it remains as is.In dilation, the biggest of the neighbouring pixel values is given.If the pixel is equal to '1', it remains as is.Erosion and dilation operations are facilitated the detection of thin fractures in the image.Customization can also be performed in morphological operations depending on the characteristics of the road surface.
The images obtained after morphological operations can be seen in Fig. 5.

Feature extraction and Classification
Feature extraction process is based on performing operation over pixels through the image.The total number of pixels in the image is small because the surfaces without crack are the points which are independent of each other.Therefore, the images below a defined value were accepted as without cracks.In the images with cracks, it was detected that there was a superficial crack if the value which was obtained by the division of the area of the cracked region to the entire area was below a certain value.For crocodile cracking in the images, whether there was crocodile image was detected by calculating the number of objects because more than one objects were obtained.Whether the crack was extended horizontally or vertically was found by detecting its direction.It was accepted horizontal if the obtained value was between -45 and 45, and it was accepted vertical in the other case.
The classification process was carried out with decision trees.Decision tree is performed by the hierarchical fragmentation of the object feature space [25].A Decision Tree provides open relationship between input features and estimates [26].Using decision tree for classification process is possible by obtaining multiple feature data.Road surface is usually without cracked structure.So in the study carried out, whether the road surface is cracked in the root node is detected.If the path is faulty, whether there is a crack in the child node is detected.If the fault is deep crack, whether there is crocodile cracking is detected.If it is not crocodile cracking, whether it is linear or transverse cracking is detected.The classification process steps of the study are presented in Fig. 6.

Experiments
The study was carried out on a computer with Intel(R) Core(TM) i5 CPU, 2.4 GHz, 3 GB RAM and 64-bit operating system.After the detection and classification algorithm of road surface defects was coded in Matlab on the computer, for the testing operation of the study carried out, the performance of the algorithm was evaluated using the required images which were obtained from both web environment and the camera with CMOS 13.0 MP resolution.The images were obtained by being positioned to be about 1 m above the road surface and 90° to the surface.Some examples of the study carried out and information about how long the operation was performed are given in Table I.
The mean value of the RGB image is calculated after the first image is obtained.The images processed at later stages were converted into grayscale images.After the image was obtained, it was transformed into gray scale image.The gray scale image was dimensioned and prepared for the mathematical operations to be performed later.The cracked structure was brought to the forefront after blurring the image by median filter and the addition operation.As it is described in the proposed method, the image was transformed into binary image for the mathematical morphological operations after obtaining the cracked structure in the image.Mathematical morphological operations were used to improve the obtained image.An attempt to achieve better results was made by defining two different matrices for the morphological operations performed.In the feature extraction operation, the classification was performed by calculating the total number of pixels obtained through the image, solidity value, total number of objects and the slope of the object for the images consisting of single object.In classification, we found without crack structure because road surface is without cracked in generally.this situation gain time for find cracked structure.Within the context of the application, to which of the five classes including without crack, superficial, crocodile cracking, linear cracking and transverse cracking the obtained image was belonged was detected.It was applied on various types of roads.The performance of the algorithm was evaluated by developing the algorithm developed on Matlab.

Conclusion
A system that can work in real time to detect and classify errors on different road surfaces has been designed.The detection and classification of road errors should have high accuracy rate and the same average accuracy rate while running on all road types so that they can be used in real life.However, road colour affects the accuracy rates of the operations performed in error detection with image processing methods.Therefore, detection of the road is performed in the first stage in the study carried out.The second most important issue in real time operations is time.Therefore, the mean value of the road is calculated in the first image received, and then image processing methods customized according to this value are used in the study carried out.
In the detection of errors, in the first stage, operations were performed to bring the cracked structure into the forefront.The mathematical operation performed further comes to the forefront since the cracked structure constitutes the darkest part of the image.The roads are divided into five classes as un-cracked roads, superficial crack, crocodile crack, linear crack and transverse crack.The sum of pixels was subjected to classification according to solidity value, the number of objects in the image and the resumption angles of the cracks for these operations in the feature extraction.Decision trees were used as the classification process.Experimental results show that it can be used in real life.The cracks thinner than 1mm can be detected.

FigFigure 1 .
Figure 1.General flow diagram of the proposed system.

Figure 2 .
Figure 2. Flow diagram for proposed color detection.

Figure 3 .
Figure 3. Flow diagram for proposed image enhancement.

Figure 4 .
Figure 4.The images to be processed (a) (c), the image obtained after the addition of the image subjected to the median filter and the unprocessed image (b) (d).

Figure 5 .
Figure 5.The image obtained after the morphological operations (a) (b).

Figure 6 .
Figure 6.Flow diagram of classification process.

Table 1 .
Examples of experimental results