Adaptive Vision Based Condition Monitoring and Fault Detection Method for Multi Robots at Production Lines in Industrial Systems

Continuity of production is a highly important in the days that manufacturing is becoming bigger and serial. The mistakes done while producing process cause fail on products and it may bring about even big losses for the facility. Furthermore, hitches on robots at production line may also cause crucial damages that may give rise to high repair costs and discontinuance of production. In this study, it is aimed to obtain alive bird's eye view map of production lines, which are big and impossible to be monitored with only one camera, by using multi cameras and stitching algorithms. Finding the similar scenes of input images, estimation of homography, warping and blending operations, which are the steps used in feature based image-stitching algorithms, are applied respectively on images that are taken by cameras. The assignment of second nearest neighbor distance rate adaptively makes the results more qualified. After obtaining single stitched image movement detection is actualized by using the difference of sequential frames, and anomaly movements are determined. As a result, the robots at the long production lines can be monitored in one screen, and with processing the obtained image, faults on robots that may cause damage at non-cheap machines can be handled before time.


Introduction
Cyber-physical systems (CPS), which refers to systems whose cyber and physical layers are diverse but tightly integrated, is getting popularity day by day in manufacturing.In today industry, manufacturing become big and serial as it never been before thanks to the CPS.Monitoring physical layer from cyber layer is necessary to link the layers and integrate the system more tightly.Monitoring of physical parts of system such as robot arm is important not only for CPS but also for conventional facilities.Monitoring the products or active parts of system eases the control of big facilities.Hitches, which may cause damage at non-cheap hardware, may bring about big losses for companies.Repair costs and the costs arising from discontinuance of production are first examples that comes to mind.Therefore, observation of the system and determination/prevention of possible accidents are required for deduction of the cost for facility.In this study, bird's eye view map of robots at production lines is utilized for fault diagnosis.However, because capturing of whole production line is not generally possible with only one camera, a camera array and image-stitching algorithm is used to obtain the full image.Thus, it is possible to monitor all robots at production facilities in one screen.With the created alive map, it is also possible to determine positions of all robots instant and extract information with more detail about the facility.For this aim, firstly, it is need that locate the cameras each of whose angel of vision contain the small piece of neighbour one's.Finding the similar scenes of input images, estimation of homography, warping and blending operations are applied respectively in order to mosaic the images by twos.After all steps done, only one stitched image is obtained from all these images taken by cameras at the same time.Thus the robots in the facility can be observed in one screen and machines which could lead to accidents can be noticed early.
Because of applications of image mosaicing is wide and its quality and performance need to be improved, image mosaicing was subjected to studies many times [1][2][3][4][5].In one of the studies, Divya et al. [1] implemented weighted average method in addition to the methods used in image mosaicing such as feature extraction, cluster analysing, dynamic programming to obtain wide angle panorama.Although the study gives effective results for well illuminated images, it is not useful for dark images.Another study about this subject is Lin's study [2].Lin et al. use image mosaicing in order to obtain cylindrical panoramic images.He used Scale Invariant Feature Transform (SIFT) to calculate feature vectors and Random Sample Consensus (RANSAC) to eliminate occurred mismatches during matching process.Patel et al. [3] employed Speeded-Up Robust Features (SURF) for feature extraction and Sum of Squared Differences (SSD) for feature matching.He also used RANSAC to eliminate incorrect matches.At the end of their study, they compare Harris and SURF and decide that SURF is more effective for multi-imaged image mosaicing problems.Huang et al. [4] effort to mosaic frames in videos.Instead of using all frames in a video, they used the Images that will be stitched Images that will be stitched Sort the images with the aid of extracted features Sort the images with the aid of extracted features Form the stitched image Form the stitched image Apply methods such as noise reduction and balancing in order to improve quality Apply methods such as noise reduction and balancing in order to improve quality Obtained image that has high resolution Obtained image that has high resolution

Figure 1. General Image Mosaicing
Steps in Literature [5] frames that contain small piece of other frame to enhance video mosaicing performance.In their study, Lee et al. [6]

Image Mosaicing
Image mosaicing is the process of obtaining single image from more than one images that contain common view with corresponding of the matching points.It helps to obtain qualified images with not expensive cameras [11].Beside obtaining the image of scenes that can not be viewed with one camera, it is used also for taking panoramic photos and deleting the moving objects in a video.It has a wide application area from smart systems to military systems.Regardless the count of images that will be stitched, the images are stitched by twos as seen in figure 2. There are stitching methods in space and frequency domains.The methods in space domain can be grouped as area-based methods and feature-based methods [12].In this paper, feature-based techniques are applied because area-based techniques are sensible for changes in size, angle and illumination.
In feature-based image mosaicing methods, interest points that are characteristic points in image and their features are found first (Fig. 2-b).After, the features belong the interest points are calculated and feature vectors are gained (Fig. 2-c).With the help of the feature vectors, matching operation is applied (Fig. 2-d).It is possible to find similar feature vectors that are non-related, so mismatches that may be occurred are eliminated and homography is estimated (Fig. 2-e).With the found homography matrices warping operation is actualized (Fig. 2-f).If necessary blending operation is applied to make the stitching seamless [13].And at the end of the processes, single stitched image is obtained from the two input images (Fig. 2-g).The first step of feature-based image mosaicing is feature detection.In order to stitch images, it is required to determine common areas/objects at beginning.Feature based methods use characteristic points called interest point instead of all pixels in an image in order to make algorithm more efficient.Interest points can be edges, corners or blobs.
There are several edge and corner detection methods to identify the interest points [14,15].In this paper, interest points are extracted by Difference of Gaussian (DoG) method which is a blob detection method.In order to link between images, interest points are used.Some features must be extracted for calculating the similarity of interest points.There are various feature extraction methods in literature [16].In this study, we used Scale Invariant Feature Transform method [17].After extracted SIFT features, it is searched corresponding interest points from second image to interest points from first image according to similarity [18].In this stage second nearest neighbour distance rate is used to prevent mismatches [19].

Reference Image
According to this method the distance between interest points from reference and target image are calculated.The nearest 2 corresponding interest points are selected for each interest point from reference image.If the first distance is smaller than the multiply of distance ratio and second distance, it means there is no confusion probably and the first one selected as matched point.Otherwise it means there is confusion and no matching is realized.
where A is distance of first nearest point, B is distance of second nearest point and α is tolerance of distance difference.
Although the second nearest neighbour distance method reduce the count of mismatches, it can not completely eliminate them.
When we try to superpose the matched interest points, the mismatches make a fuss.So, the count of mismatches must be minimized before homography estimation.In this study RANSAC is used to eliminate the mismatches.
According to RANSAC method [20], some of the data are selected randomly and a model is tried to fit the samples.After all data are tested by the model respectively.If the model gives the correct result within the tolerance defined before, the datum is marked as in-liner.If in-liner data is sufficient to verify the model, the model is added to solution set.This operation is actualized pre-determined iteration times.At the end of the iterations best model that fits the majority of data with minimum error is selected as solution.
We aimed to fit a model whose input is the pixel coordinate of target interest points and output is the pixel coordinate of corresponding reference interest point.The model that help us to calculate the new position of target image pixels' in order to superpose these images is called homography matrices.Homography matrices is 3x3 sized matrices and it is used to move the points in homogeneous coordinate system.The equation for calculating new position of point and and its extended version are given by respectively (2) and (3).

𝐻. 𝑃 = 𝑃′
where H is homography matrices, P and P' are former and new coordinates of pixel in homogenous coordinate system respectively.To transform 2 sized vector to homogenous coordinate system,  0 is selected as 1; to transform homogenous coordinate system to X and Y values (4) is used.
where X' is new value of X and Y' is new value of Y.After found of new coordinates of target image, target image is transformed and coordinates of corresponding points be the same.
With the displaying transformed image and reference image in one frame, image stitching process is done.Stitched image may have visible colour transition because of the cameras whose angel of vision is not illuminated as same as the other one's.At last, to extinguish the difference blending operations are used [21].In order to remove blur that may be derived from stitching operation or derived due to the high speed of robots at production line, some blur removal methods in literature can be used [22].

Proposed Method
In this paper, adaptive image stitching approach is applied images that are taken by cameras at top of the production line first.The system which is desired for grabbing images of production line and robots is given by Fig. 3.The images that are taken by such system are subjected to our adaptive image mosaicing algorithm.The general steps of feature based image mosaicing operation are given by section 2. Second nearest neighbour (SNN) distance rate, which is the α value in Eq. 1, is using for reducing mismatches count.The parameter provides us ignoring matching if there is any close corresponding point to nearest corresponding point.In this manner, mismatch count is minimized.The optimum value of this parameter is not the same for every problem.While selecting the parameter below its optimum value causes insufficient number of matches, selecting the parameter above its optimum value causes the more count mismatches.Therefore, it should be brought near its own optimum value instead of setting general constant value for better stitching quality.A graphic whose Y axis gives us the second nearest neighbour distance rate for given total matches counts is represented by Fig. 4.  Total 2054 interest points were detected on the reference image, which is 581x376 sized, in Fig. 5.For values of 0.3, 0.7 and 0.8 SNN distance rates; 73, 90 and 133 matches are actualized respectively.The chart in Fig. 4, which is formed by sorting after dividing nearest neighbour distance to SNN distance of each interest points in reference image, shows the SNN distance rate values for how many matches will be obtained.So, if it is aimed to get 1000 matches for sample 1, SNN distance rate should be set about 0.95.Nevertheless, this ratio should be selected as low as possible because the higher selection directly increases the false match rate.The effect of this ratio to mismatches can be seen in Fig. 5.If it is chosen too small, it can not provide the number of matches required for homography estimation, so the optimum value should be the minimum number that will give enough number of matches.
Increasing the common area of the images will increase the number of matching that will be obtained, so it is expected that the graph will approach to the right when the common area is increased.The image in Sample 1 contains a common area of about 10%, while the samples in Sample 2 and 3 contain about 20% of the common area.When the 20% common area is specified for the image in Sample 1, the resulting graph is very similar to Sample 2 up to 0.8 ratio.While 73 matches are obtained for the 0.3 ratio, 180 matches are obtained for the new case, which has 20% of common area.
As seen in sample 1, there are very few key point detection on the grass area, which does not show any significant variation.Therefore, no match is observed in these regions.Although, sample 2 and sample 3 have the same size (540x540), the number of interest points and count of total matches differs from each other.While 310 matches are occurred in sample 3 with 3116 interest points, only 191 matches occurred in sample 2 with 1419 interest points.These cases emphasize that a sufficient number of matching is directly related to the number of characteristic points contained in the common areas that are not related to the size of the image.And this is why we don't use size while calculating optimum rate.According to the information given above, the optimum ratio is calculated as given in (5).
Where opc() is the function that gives optimum total match count, t is total matches, reg() is the function that calculates integral zero to t (the area under the graphic), and gain is the pre-set minimum gain ratio.If the function can not find optimum count over min value, which is pre-set also, then opc return the min value.But it is not an expected situation for such problems that are well designed.In order to get optimum total count value, first, t is set to total interest points and then it is calculated by recursively.A block diagram for proposed method is shown in Fig. 6.Similar areas have similar characteristics, so the optimum distance ratio is the same for images belonging the same scene.Therefore, the optimum distance ratio will be calculated once when the system is installed.And it will do image mosaicing by using the calculated value at the rest of the runtime.In calculating the optimum nearest second neighbour distance ratio, the ratios of the nearest neighbour distances to the nearest second neighbour distances are used.The values are sorted after calculated.The sample scenes, optimum nearest second neighbour distance rate that belongs to scenes and the graphic of the ordered values are given by Fig. 4. To make a healthy homography estimate, it is necessary to find a sufficient number of matching pairs with the minimum number of false matches.Using the graph in Fig. 4, the optimum value of nearest second neighbourhood ratio to provide a sufficient number of matches is calculated as given by equation 5.At the end of mosaicking process single enhanced image, which will be used in anomaly movement detection, is obtained.Anomaly movement detection is actualized by getting differences of sequential frames.Because the constant parts of system are

Experimental Results
In this paper, proposed adaptive image mosaicking algorithm is performed on images that belongs to robots at production line.A computer with intel i7 6020M CPU, 6 GB ram, and Windows 10 64bit is used to test the performance of proposed approach on the sample images belong to production environment.The sample 4 images, which was taken synchrony, are given by Fig. 6.Although cameras are stable and the distance between them is nearly constant, possible changes on camera angle or shakes make it necessary to find optimal points that are used in process of image superposing.We suppose that cameras are positioned as one's visual angle contains 1/3 of the neighbor one's visual angle maximum, include deflection.So instead of searching interest points whole image, we did it only in interested area.Thus the lapsed time for stitching process is reduced.Furthermore, the elapsed times in table 2 can be reduced by parallelization.The results for elapsed times are the same with our previous work [23].But, the quality of stitched image, which is inverse of the difference of overlapped points after homography transformation, is enhanced with the adaptive SNN distance rate.

Conclusions
It is very crucial to monitor and control physical elements especially for such systems.In this paper, it is proposed that use image mosaicing to monitor conditions of multi robots at production lines and the proposed method is tested by experimental.In order to test the method 4 images that belongs the same science are used, and no one of them contains more than 1/3 of another one in spite of the possible deflections.With the knowledge of this, only a part of image is used to search interest points instead of full image.In order to make the stitched image more qualified second nearest neighbour rate is adaptively calculated.Finding interest points, feature point extraction, matching, warping and blending operations are used respectively to stitch images.Since the order of images that are stitched are certain, sorting of images is not required for such camera array applications.The images are stitched by order and by twos respectively.The effect of image order is demonstrated by results given by table 1 and table 2. The results show that stitching by twos outperforms stitching by order for the camera arrays which occurred from NxN camera, N is even number.Furthermore, if N is not even or the size of camera array is not NxN, a sub array that provides the constraints and the operation is applied for sub array.After the remains may be stitched by orders.Little time consumption of this approach make it possible to be used in camera array problems.The feature-based method makes the solution more reliable in case of changing in position, angle or illumination.Controlling of the system by autonomy robotics is important for industry 4.0 and cyber physical systems, which aimed totally unmanned systems.Thus after obtaining stitched image, we process it to determine anomaly movements.Consequently, it is possible to get high quality reliable image using low quality cameras thanks to the method.Monitoring the condition of robots at production line, which is a requirement for today's industry facilities, in a screen is achieved by the study.With processing the image and determination of anomaly movements, the aim of establishment unmanned facilities become closer.

Figure 2 .
Figure 2. General Steps of Feature-Based Image Stitching methods in literature [13].a) Getting input images, b) Detection of Interest Points, c) Extraction of Features, d) Matching According to Feature Similarity, e) Elimination of Mismatches and Homography Estimation f) Warping Operation, g) Obtained Single Output Image

Figure 3 .
Figure 3. Desired system for grabbing images.

Figure 7 .
Figure 7. Matches on Sample 3 with 0.5 of SNN distance rate.

Figure 6 .
Figure 6.Block Diagram of Proposed Method

(a) 1 Figure 6 .
Figure 6.Images that are stitchedIn order to test the effect of the images order, the four image given by Fig.6are stitched respectively first.Then the same images are subjected to the process of stitching by twos.The elapsed time are demonstrated by table 1 and table 2.

Figure 7 .
Figure 7.An example of stitching by twos.

Table 1 .
The Performances of Stitching by Order with Full

Table 2 .
The Performances of Stitching by Twos with Limited