Development of an Automatic System to Detect and Spray Herbicides in Corn Fields

Weed control is vital in agricultural production. Chemical control methods are generally preferred in weed control as they (1) affect quickly and (2) reduce the labour requirement. However, in conventional applications chemicals are generally applied to whole field surface. Therefore, non-targeted areas are also sprayed. This increases 1) amount of herbicide used and (2) risk of off-target chemical movement. In this study, a patch spraying system was developed to automatically detect and spray herbicides on weeds in the corn field based on weed density. In order to determine the weed regions, a digital camera was fitted in front of the tractor. The images taken using the camera were then simultaneously processed using an algorithm written in Matlab TM software. The results of the field study showed that at 4, 6 and 8 km h -1 forward speeds, application volumes decrease by 30.21%, 28.82% and 32.28%, respectively, when it is compared to the conventional application methods. It was also determined that the application accuracy rates were 80%, 81.66% and 75% respectively for 4, 6 and 8 km h -1 speeds.


Introduction
Some weed species have gained resistance against regularly and intensively used chemicals. Therefore the amount of energy used for the weed control hence cost of weed control has increased. In corn production, product losses can be up to 20-30% in the first two months period from planting due to weeds. In addition, weeds in corn fields make harvesting difficult and cause work loss (Aydemir & Karaoğlu 2008). Weeds are not desired in the corn fields as they compete with corn for limited resources such as water, nutrients, light, and space. Weeds can also change the quality of light received by corn (Rajcan et al 2004).
Uncontrolled use of chemicals in weed control in agricultural production causes negative effects on the environment and human health. Therefore reducing the amount of chemicals used and increasing their effectiveness in agricultural production is vital to keep agricultural production sustainable. In order to overcome the negative economic and environmental risks of over application and reducing the amount of chemicals used, patch spraying has been suggested (Pajares 2015).
In order to apply patch spraying determination of the weeds in the field environment is essential. However, some weeds are randomly distributed on the field whereas some weeds might have patchy distributions. On the other hand crops are sown in rows with a constant spacing (about 70 cm for corn). On field images, these rows appear as parallel lines and crops are sown with a constant spacing between two plants in the same row (also called intra-row spacing) (Vioix et al 2006). The periodic distributions of these crops can provide significant advantages in terms of image processing applications as objects with specific shapes in an image can be identified easily.
In recent years, researchers have developed weed control systems based on various image processing techniques using different types of cameras and spectral sensors (Burgos-Artizzu et al 2011; Agrawal et al 2012;Hlaing & Khaing 2014). One of the methods used to determine the crop row is the Hough transform (HT) (Tang et al 2016). HT information proves to be a very good way to differentiate crop from weed pixels presenting similar spectral information (Ortiz et al 2015). On the other hand, the high computational time of HT's method is one of the disadvantages of this method for real-time applications. Sabzi & Gilandeh (2018) aimed to locate and identify potato plants and three common types of weeds using a hybrid classification approach, consisting of artificial neural networks (ANN) and particle swarm optimization algorithm (PSO). However, the speed of the developed method was too slow due to the excessive computation required to classify weeds. Gonzalez-de-Soto et al (2016) presented structure of a unmanned ground vehicle derived from the project RHEA (Robot fleet for highly effective agricultural and forestry management). For the control of the vehicle a hybrid architecture was implemented. The vehicle has a camera for real-time field vision, a GPS receiver to provide the position and orientation of the vehicle, a laser system placed in the front of the vehicle for obstacle detection and a smart spraying system for selective spraying application. Although the system can accurately detect and apply spraying, the system is quite costly and cannot be afforded by small scale farmers.
In this study, a small scale and cost effective patch spraying system was developed. Physical spraying applications were also performed using real time field images taken under uncontrolled outdoor lighting conditions. The aim of this study was to (1) determine real time crop-weed discrimination using morphological image processing techniques in corn field and (2) perform spraying application automatically to the desired area via a control system (if the weed density is greater than a determined critical level).

Material
In this study, a 400 liter capacity three-point hitch type field sprayer was used. A camera and a speed sensor were mounted on an adjustable platform ( Figure 1). The original sprayer regulator was removed from the system and replaced with a flow-based control unit ( Figure 2a). This system was used only to prevent ripples in the pressure line because each nozzle group might be active at different times during the application. Section solenoid valves of the flow-based control system and three-way solenoid valves which activate the nozzle groups were connected to each other. The three-way solenoid valves were controlled by PLC (Programmable Logic Controller). When the sprayer was not used the liquid was sent back to the tank through a return line ( Figure 2b). An IDS UI-1240ML-C-HQ camera; resolution of 1280 x 1024 pixels, 5.3 x 5.3 μm pixel-size and 6.784 mm x 5.427 mm optical size (IDS 2017) with an Azure model C-Mount, 1/2", 4 mm lens were used to acquire images. The camera was mounted on a custom-made height adjustable platform. To acquire 4.20 m. horizontal field of view (HFOV), different camera height (3-4 m) from the ground and different pitch angle (30°-50°) was used. In order to measure the forward speed, a radar speed sensor (Dickey-John Radar III) was mounted in the front of the tractor. The supply voltage of the speed sensor was 12V DC whereas the output signal was a 12V square wave. The periodic signal of the 12V square wave coming from the radar speed sensor was increased to 24V amplitude by using a NPN type transistor so that this signal can be read by the PLC fast counter unit.
A Siemens brand PLC (model no: 1214C AC/DC/Rly) was used in the system. The PLC had 8 digital inputs, 6 digital outputs and 2 analogue inputs with 10 bit resolution. 6 digital inputs can be assigned as a fast counter (HSC) input to read signals up to 100 kHz.

Methods
This study was carried out in four stages namely; (1) transferring field images to Matlab TM and processing them to classify corn plants and weeds, (2) transferring data from Matlab TM to PLC via OPC (Ole for Process Control) server, (3) calculation of the spraying delay times using the PLC program developed at 'Tia Portal' (Totally Integrated Automation Portal) in accordance with the information from the radar speed sensor, and (4) application of spraying to the required areas.
In order to check if the system is working, firstly, some preliminary tests were performed. After the preliminary tests, the system was then tested under the field conditions. To determine the accuracy of the field test applications, 20 times 0.05 x 0.06 m white papers were placed at different points on the land. To identify the droplets sprayed on to the papers some red food colouring was also added to the sprayer tank ( Figure 3). These reference papers were placed in high and low dense weed areas and the accuracy of spraying application was analysed in terms of droplets presence (only by visually observing the presence of droplets) on the papers. For example, if the patch spraying was successfully applied 16 of the 20 papers, the accuracy rate was accepted as 80%. Field tests were conducted to examine two different situations (which have been explained below); a. Conventional spraying application: In order to determine the application norms for traditional application at different tractor speed (Camera data was not used in this application).
b. Patch spraying application (Depending on the camera data): Unlike the conventional application, the patch spraying application was performed using camera data. After the image was taken, firstly, a region of interest (ROI, 1.5 x 4.20) which is equal to working width of the sprayer boom was determined. After that the ROI was divided into sub regions automatically. Subsequently, the weed regions in the sub-sections were compared to a pre-determined weed threshold value. If the sub-sections had more weed than the threshold value, spraying was applied. In the sub-regions where the weed amount is less than the specified threshold value, the spraying process was interrupted and the liquid was returned to the tank through solenoid valve. The block diagram of the automatic controlled field sprayer system and the general algorithm of the system were given in Figure 4a and 4b, respectively. Algorithm steps include processes from image acquisition to spraying application. Images with a resolution of 752 x 480 pixels obtained by the camera were transferred to Matlab TM environment via Image Acquisition Toolbox™. Image Acquisition Toolbox™ provides functions and blocks that enable to connect industrial and scientific cameras to Matlab TM . It includes Matlab TM applications that interactively detect and configure hardware properties (Matlab 2017). After transferring the images to Matlab TM environment, the RGB (Red, Green, and Blue) images were, firstly, converted into the grayscale images. In a colour image, a pixel value consists of different combinations of Red (G), Green (G), Blue (B) values. Converting of the colour image into grayscale was to make the brightness value of the green objects (crops and weeds) greater than the other objects in the image (soil, stone, etc.), which will increase the accuracy in the binary level conversion phase. An example of how an RGB image taken from the field was converted into a grayscale image was given in Equations 1, 2, 3 and 4. Firstly, each field image was converted to normalized red (R), green (G), and blue (B) channel images. Then the normalized RGB channels were converted to the normalized excessive green (NEG) images to emphasizing green channel (Jeon et al 2011). This transformation was firstly employed by Woebbecke et al (1995). Then similar equations were used in several studies (Sabancı 2013;Hlaing & Khaing 2014;Liu et al 2014). (1) Where; r, g, and b are a pixel value of red, green and blue channel of RGB image.
In order to convert NEG images into binary images thresholding method was used. Although there are different thresholding method available in the literature, such as histogram-shaped-based, clustering-based, entropy-based, attribute similarity methods, object attribute-based, spatial approaches and local methods (Sezgin & Sankur 2004). In this study "Otsu automatic threshold method", which chooses the threshold to minimize the intraclass variance of the black and white pixels, was used due to its simplicity (Otsu 1979). The pixels below the threshold value were considered as black (soil, stone and other materials) while pixels above the threshold value was considered as white (corn and weed). An example of converting of an RGB image to binary image was given in Figure 5. In order to establish the communication between Matlab TM and PLC, OPC Toolbox functions (available in Matlab TM ) were used. To do so, firstly, the OPC connection settings between the PLC and the PC were made using the OPC.Simatic.Net software. After that Matlab TM OPC Toolbox has been assigned as a client, and thus Matlab TM had information about the server's name and each OPC item stored in the server (Tekinalp et al 2013). Then OPC group object was created and added into OPC items which represent the PLC memory.
PLCs require special equipment to detect signals faster than its cycle time. Therefore one of the PLC inputs was assigned as a fast counter to read the information sent by radar speed sensor. The S7-1200 PLC used in this study had six high-speed counters and these channels can be used in the 'Tia Portal' software to read signals up to 100 kHz. The radar speed sensor was connected to the I0.0 input of the PLC and the speed information was transferred to the 'HSC_1' channel. Spraying application delay time was calculated using the tractor speed information and the constant distance between the image frame and the spray nozzles (4.60 m). When the boom came to the weed area, which was sensed by the camera, spraying application started and continued along 1.5 m.
While corn rows showed a regular arrangement in the vertical direction, there was no regular arrangement for weeds. In order to process the images, firstly, the noise pixels (smaller than 5 pixels) on the image were cleaned. Then, in order to merge the pixels of the crop rows, extending vertically in the binary image, each image was dilated using a 1 pixel wide and 9 pixels long structuring element. The reason of using only one pixel wide structuring element was to prevent the incorporation of crop rows with weeds. As corn crops were arranged in equal intervals in the vertical direction, the possibility of structuring element to merge the crop rows was greater than that of the weeds. However, it should be noted that the irregularities in corn plant rows (deviations from sowing errors) were neglected. The binary image and its dilated status by structuring element was shown in Figures  6a and 6b. (a) (b) Figure 6-a, Original binary image; b,

dilation of the binary image with a 9x1 structuring element
After the dilation process, there was a clear difference between the merged crop rows and the weeds areas. In order to eliminate crop rows from the image a threshold value, calculated using Equations 5, was applied.

= {
< 1200 ≥ 1200 (5) Figure 7 shows the location of region of interest (ROI) from the processed image where crop rows and weed pixel groups displayed in different colours. In the field tests, dimension of the ROI was determined as 4.2 m wide and 1.5 m vertical long. The width of the ROI area was determined considering the coverage of the nozzles. After generating the binary image (which contains only classified objects (crops and weeds)), the objects representing crop rows were removed from the binary image and the image containing only weeds was divided into three subsections (the same size as each nozzle group) vertically ( Figure 8). The spray nozzles were also divided into three sections, using the same measurements, as right (section 1 with 2 nozzles), middle (section 2 with 3 nozzle) and left (section 3 with 2 nozzle) and each section was independently controlled through three-way solenoid valves. Then spraying was applied to each region in terms of the amount of weed per m 2 .

Figure 8-Dividing the ROI into three sub-sections using binary image that only contains
According to TAGEM (2017), for species with known damage threshold, the weed density must be less than the least damage threshold, whereas in the case of species whose damage threshold is unknown, the damage threshold was determined as 10 pieces in per m² or 10% of the area. Üstüner & Güncan (2002) classified the weed densities as; A) very dense (average> 10 m -2 ), B) dense (average= 1-10 m -2 ), C) medium dense (average= 0.1-1 m -2 ). In this study, spraying was applied, if the amount of weed in m 2 in each image was equal to or greater than 10 (very dense).

Preliminary tests
In the first stage, preliminary tests were performed by placing green objects at different spacing on a flat concrete surface to investigate whether or not the spraying application was performed at the right time and to the right regions. Since the ROI has a width of 1.5 m in the direction of the tractor travel, if the distance between the objects is less than 1.5 m (X1), the spraying process continues without interrupting between the two objects. If the distance between objects is more than 1.5 m (X2), spraying is done only the areas where the objects are located (Figure 9a and 9b).

Figure 9-Conditions for spray application for preliminary test, (a) a preliminary test (b)
During the tests, OPC item values were read at intervals of 0.2 s by changing the tractor speed and the distance between the green objects. For example, when the green object in the section 2 of the image area is detected during the application, the reading process of the PLC memory area is shown below. Time difference of spraying between green objects are given in Table 1 for 4 km h -1 tractor speed and 2 m distance between the objects. As seen in Table 1, spraying application was performed at time intervals of about 1.5 s to the areas where the green objects were located in the display area. In other words, the areas between the objects were not sprayed for approximately 1.5 s.

Field tests
Field tests were performed within the critical period for weed control (for corn), which was between the 20 th and 55 th day after planting (Tursun et al 2015). Spray applications were performed using water at 4, 6 and 8 km h -1 forward speeds in a 250 m long and 20 m wide area ( Figure 10).

Figure 10-Field tests
Firstly, the accuracy of the field tests were determined using patch spraying application method depending on the camera data at 4, 6 and 8 km h -1 speeds. Accuracy test results obtained by looking at the presence of droplets on test papers placed on the application land. Results are given in Table 2. It was found from the results that the accuracy of the tests performed at 8 km h -1 was lower than those of the 4 and 6 km h -1 ones. This can be attributed to the vibration (which increased with the increase of the speed) which reduced the quality of the images hence the performance of the sprayer.

Table 2-Accuracy rates of applications with test papers
application volumes for both methods were given in Table 3. Results showed that when it is compared to conventional method 30.21%, 28.82% and 32.28% less water was used in the patch spraying application method based on camera data at 4, 6 and 8 km h -1 operating speeds, respectively. Data given in Tables 2 and 3 showed that a more effective spraying can be applied using patch spraying application method based on camera data at low operating speeds (4 and 6 km h -1 ). In recent years, there has been a notable increase in studies regarding to the control of weeds using different digital image processing software (i.e., Matlab, Open CV, C++) (Burgos-Artizzu et al 2011; Vikhram et al 2018). Some factors that negatively affect the performance of these systems and the current system were that (1) vibration (which distorts the image quality), (2) sun rays (particularly in sunny days, infrared radiation enters the sensor impacting the different spectral channels coming from different angles (Romeo at al 2013) and (3) radar speed sensor output frequency sensitivity (due to the dense vegetation). The efficiency of the systems can be increased by analysing smaller areas using two or more cameras at a lower distance (to reduce camera vibration in field conditions). It is thought that the efficiency of the system can be improved by using more comprehensive image processing algorithms and enhanced computational power.

Conclusions
In this study, a system was developed to automatically determine weeds in a corn field and perform spray application (if the weed density is greater than a critical level). Field tests were performed to evaluate the efficiency of the system and it was found that the accuracy of patch spraying application method using camera was at 80%, 81.33% and 75% for 4, 6 and 8 km h -1 operation speed, respectively. In order to improve the success of the system infrared-cut filters, which help to reduce the sun light reflected by the corn leaves, can be used (Romeo at al 2013). It is also thought that by using the proposed system (1) negative effects of the chemicals used in agriculture on environment and human health and (2) the production costs can be reduced. Future work will focus on improving (1) the algorithm to increase accuracy of the image analysis and (2) the system to improve its effectiveness.