A Deep Learning-Based Quality Control Application

The study at hand is an implementation of a deep learning strategy on a quality control scheme. The quality control process is a substantial part of product manufacturing. It fundamentally targets to detect and eliminate defective products so that the erroneous ones will not be delivered to the customers. Final product control has been usually performed by experts. Generally, those experts can easily distinguish defective and trouble-free products. On the other hand, growing product lines and human-based natural problems may affect the efficiency of that quality control process. Therefore, there is an increasing demand for computer-aided software that will take the place of those experts. This software or algorithm typically increases the product control rate. Besides, they make it possible to avoid from human-driven faults. The algorithms run at high speed and efficacy under conditional situations i.e. perfectly lightening environment. However, they may easily fail when small changes occur in the environment or the product for some duties that humans can easily achieve. These robustness problems make them not preferable, although they have numerous advantages. At this point, deep learning-based artificial intelligence algorithms have made a significant enhancement. The general development and achievable prices of GPUs pave the way for using numerous training examples so that better networks, meaning more robust, can be created for the applications. To this end, we carried on an experiment that could realize the deep learning strategy on the quality control scheme. For this purpose, the developed algorithms applied to the inverters conveying on a product line to confirm whether they are erroneous or not. Results show that developed strategy could detect defective products similar to the human being.


Introduction
The development of automation and its applications offers many important advantages in our daily life. Thanks to automation, large-scale productions can be made very quickly. However, the decrease in the share of people in the process prevents possible human mistakes and also production costs are lowered. Although automation systems are used in many mass productions, people are mostly needed in quality control processes. This requirement is one of the inherent superior features of human beings, some of which are classification and distinguishing the different ones. Even though computer software also has features such as superior classification and detection of differences thanks to developing artificial intelligence techniques and different algorithms, a semantic relationship extraction that people have is still difficult to reach. On the other hand, thanks to deep learning, an important phenomenon that has developed in recent years, significant developments have been experienced in the problem of semantic detection and classification.
Deep learning is a structure with many more sub-layers than conventional artificial intelligence and usually has different processing steps and filters. In classical artificial intelligence methods, feature extraction and classification are done respectively while it is directly applied in deep learning-based methods. But this requires substantial computational power. This requirement is much higher especially in applications where data sets are constituted from images. However, the developing GPU technology has been provided to overcome this bottleneck so that GPUs lead up to run on deep layers and big data. Owing to powerful computational abilities, training samples have increased and this makes it possible to train better networks. Therefore, the processes in which classically if and only a human can overcome could be replaced by machines that have mentioned type of artificial intelligence. Thus, the software or robot applications that usually have very limited boundaries and can perform certain tasks may have been expanded to superior jobs requiring human characteristics.
The idea of using automation almost goes back to the beginning of the era of mechanization. The works classically done by manpower have also been accomplished by machines through automation techniques. With the development of semi-conductive equipment and PLCs, these developments have increased over time and revealed the result of the automation of the machines. The ultimate point of this enhancement is defined nowadays as Industry 4.0, and it brings our life many new definitions such as dark factories, IoT, etc. (Butuza, Nascu, Giurgioiu, & Crisan, 2014) used PLCs for hydropower plants. The system they developed has a 120kWh capacity of electrical generation and runs 7/24. (Väänänen, Horelli, & Katajisto, 2010) have developed a virtual learning environment for PLC programming. The environment has frequently used automation tools such as artificial conveyors, robot cells. Employing the artificial program, experiments like real-time could be performed on a virtual platform. (Mazur, Quint, & Centeno, 2012) investigated the time synchronization problem of automation controllers on power applications. The efficacy of programmable automation controllers, also known as PAC, was sought. (Basile, Chiacchio, & Gerbasio, 2012) mentioned PLCs importance on industrial systems and some novelties on PLC programs handled.
Artificial intelligence is a phenomenon introduced in the 1950s and has been an important part of recent years despite the decreasing and increasing usage rates from time to time. It has formed the basis of many algorithms, from classic classification problems to regression-based estimates. As mentioned before, classical artificial intelligence techniques lack conditioned data preparation, not having big training sets. Therefore, the application of them has not gone beyond certain limits. On the other hand, the development of the GPUs classical schemes has led to evolving classical techniques to the deep ones. Techniques involving deep layers can make semantic comments. (Ruan, Ren, Zhu, & Huang, 2019) handled mobile robot navigation based on deep learning. They improved an integrated structure which solves the mapping, navigation, and localization problems together. (Truby, Della Santina, & Rus, 2020) tackled the deep learning techniques on the soft actuators, which recently have gained popularity in the robotics community. (Lehr, Schlüter, & Krüger, 2019) developed a CNN-based industrial classification application. The study emphasized the detection of the product on industrial scenarios and the CNN-based scheme used to detect screws. (Çimen et al., 2020) presented a micropower system that estimates the applicants' demand using deep learning.
In industrial applications, the quality control process is usually done by experts or algorithms in conditioned environments. Within the study, an alternative quality control structure is presented, and the stages of a full automation system are discussed.
The study at hand is a result of this type of examination. The answer to the problem of our quality control process, which is usually performed by human experts replaced by the mentioned algorithms, is sought and experiments, results, and the whole process of the study are evaluated within the following sections.

Materials
In this study, Mitsubishi 6 axis, 2kg payload robotic arm is utilized to evaluate the product suitability which flows on a conveyor. The inputs are obtained by a camera that is attached to the robot. To transfer the data to the software part and communicate all equipment, PLC, and HMI are used. The locations of the robotic arm and conveyor run status delivered from the HMI platform using the serial communication protocol.
The software installed on the Nvidia Jetson Nano card is used for the realization of the deep learning and image processing parts. The card allows camera inputs and communication with PLC. The established system and communication structure can be seen in Figure 1.

Methods
In this study, template matching and deep learning techniques were used for inverter detection and the quality control process. The program languages for the deep learning, PLC, and robotic arm respectively as Python, Ladder diagram, and Mitsubishi basic robotic. The flowchart of the whole quality control process is illustrated in Figure 2.  Pre-processing: After the detection of the inverter, the robot arm moved automatically to the region of interest (ROI) that will be classified by the trained net. When the robot completes moving, a 15x15 median filter is applied to the camera image. Then, the image is classified with the trained deep learning net. Figure 4 shows the sample images, which were used during the training of the net.

Fig. 4 The training images
Deep learning: Deep learning is a popular artificial intelligence method that emerged in the last ten years and has been used in fields such as image processing, face recognition, classification, and electricity. The deep learning has a different technique according to the traditional machine learning methods. The traditional methods extract the features of the data manually while the deep learning method extracts the features automatically thanks to its convolution layers. This study used 2000 images to create the classifier structure. TensorFlow library was used to create different CNN structures. The best CNN structure was found optimally after classification processes.
Robot arm movement: After inverter detection, x, y, and z coordinate information that were obtained from the camera were sent to the robot arm via PLC and HMI. The coordinates were converted into robot arm space by using Equation 1.

=
. (1) represents the obtained point, which was found according to the inverter position. represents the image acquisition position. is a homogeneous matrix that includes rotation and translational parts. This study did not use a rotation axis; therefore, this part was saved as a unit matrix. The translational part consists of differences in the axis of x, y, and z.
Quality Control: In this process, the camera reads one frame and classify with a deep learning net. The result shows that the braking resistor is connected or not. When the classification is over, the robot arm moves to the inverter detection position, again.

Results and Discussion
Different software and tools were used for the quality control experiment. This study used OpenCV and TensorFlow libraries for detection and classification processes. The system took images and examined the product on the conveyor belt. The conveyor belt was used to provide the product to move beneath the camera. For the control in the hardware part, Mitsubishi FX5U PLC was used to communicate between the robot arm and Jetson Nano via serial communication. Figure 5 presents the experimental environment.

Fig. 5 The experimental system
The system controlled the conveyor belt. The inverter is detected, the conveyor belt stops, and the classification process is done. Accuracy and loss results for the training process are presented in Figure 6. Fig. 6 The training and test results of the model Figure 6 shows that the test process achieved 100% classification accuracy. This CNN structure was tested in the experimental environment, and the quality control process was performed successfully in a bright environment. When the environment becomes dark, the success of the structure was decreased. The reason is that the training images were taken in a bright environment, not in a dark environment because the quality control processes are done in bright environments in industrial systems.

Conclusions and Recommendations
This study performed quality control of the inverter. The quality control system controlled if the braking resistor is connected or not. The human control process was changed with a deep learning control process. In the simulation environment, the training process achieved 99.90% accuracy, and the test process achieved 100% accuracy performances. Besides, the structure created was tested during a scenario. According to the scenario, the product on the conveyor belt was photographed by a camera from a determined position on a robot arm. Then the system operated and decided the quality control result. The product, which was produced by robot arms, again controlled by the robot arm with a software. As a result, the need for humans during the quality control process is reduced. In future studies, different types of products will be added to the system to process the quality control system. The system will know the type of the product and select the system in the software and move according to the system selected.