Control and Monitor of IoT Devices using EOG and Voice Commands

This paper aims to deploy a machine to control and monitor home devices, and to assist people who suffer from spinal cord injuries to control devices, such injuries cause people to lose their ability to use their body movements, normal people may use voice commands as well. The prototype used electrooculography (EOG) system [1, 2, 3]. The patients who suffer from spinal cord injuries may use this system to control household appliances and use the voice system to control home devices. This prototype use internet of things (IoT) technology through Wi-Fi and Arduino microcontroller to capture eye muscle movement signals, that are taken from patients, or voice signals to compare them with pre-recorded voice commands. Many tests have been made to assure correctness and speed using different environment parameters and conditions. The error rate was 2.5% for EOG and 1% for voice commands in the best cases. The idea could be developed further, smartphones and mobile data can be used for controlling and monitoring homes remotely.


Introduction
Controlling smart homes using electrical signals generated from eyes movements, or voice commands help people who suffer from muscular and neurological disorders [4,5], such as spinal cord injuries, and assist people to interact with the surrounding environment using their eye movement or using voice commands to control devices at home. In this prototype, a new technology is used, Internet of Things (IoT) which aims to connect everything everywhere and communicate over the Internet either wired or wireless [6,7,8].
The goal of this work is to control home in unusual way to become simpler and more flexible using the eye movement (electrooculography system) or voice to provide the amenities for the target group of this system. A procedure was developed for controlling a smart home using Arduino microcontroller and other devices.

Related Work
In [1], the work has been focused on controlling Human-machine interface (HMI) system that is based on Electrooculography (EOG) signals for guiding wheelchair motion manually. The user looks up, down, left, right and blink to guide the wheelchair to move forward, backward, left, right and stop respectively. In [2], an efficient implementation for IoT used for monitoring and controlling home appliances via World Wide Web, the aim was to control home appliances via smartphone over Wi-Fi and Arduino as a server system. In [3], authors proposed an eye-movement tracking system, based on Electrooculography technology to detect the signal with different directions in eye-movements and then analyze the signals to understand horizontal direction or vertical direction. This enables people to control applications using bio-electric signals recorded from the body.

Research Methodology and Solution
The purpose of this prototype is mainly to allow users to interact with a smart home environment, this was done through vertical and horizontal eye movement or using voice to give commands to the microcontroller, to communicate with the appropriate relay to control devices. The following steps explain how the prototype works in general:  Choose to control either by EOG or voice commands.  In the case of EOG, the electrodes get reactions from eye movements, then make conversions and send it to the microcontroller by general purpose input output (GPIO) ports.  In the case of voice, the voice enter the microcontroller using microphone.  Microcontroller will select device according to the previous command and send the new state of device to the relay over Wi-Fi. Figure 1 illustrates the main parts of the prototype, to connect the Arduino with appliances and select one of the relays based on the selected device using Wi-Fi, client and server, the control signal is transferred through the IP address of the selected relay and control the appliance. Here, convertors are used to adapt the voltage level between the different components. The Bluetooth module is used to connect the wireless microphone.

Figure 1. System Components
The system works in steps according the user status and the required function:  Select the suitable mode whether voice or EOG.  According to the mode, and the selected device, the IP address is determined.  The data will be sent from EOG or voice into Arduino to be analyzed.  The device status will be determined, and then the decision will be made to turn the device on or off.  According to decisions and received data, the Arduino will transmit commands using Wi-Fi to control the special relay of the selected device.

Software And Configurations
The Arduino microcontroller was programmed to analyze the input signals, after selecting EOG mode by the user, the Arduino read the analog signal result from the eye movements. As long as the patient didn't make a decision for control device, the analog readings will stay within the reference range.
When the patient makes a decision to control a device, the Arduino compares the last reference value with the new value resulted from his decision, then the new reading is determined as a right or left movement. This process is repeated four times to select the device to be controlled.
When the patient incorrectly places the electrodes on his face, the input signal readings will be out of range, then system state is "Not Ready" and the user is informed.
If the patient selected the voice mode, the voice command will be sent to online voice recognition application, then sent as an analog signal to Arduino, the patient will be able to send voice commands according the commands that were programmed earlier.
The commands are sent via Bluetooth to Arduino after connected with the application, and to increase the accuracy of voice recognition, similar possible words were programmed for the same command by software.
Because the patient is not able to carry the smartphone, EVA Mouse application [9] is used to assist the patient to control his phone using face movement.
Bi-Directional Logic Level Converter (4 Channels) is used to convert from (5 to 3.3) volts between Arduino and client Wi-Fi or convert from (3.3 to 5) volts between server Wi-Fi and relays.

EOG experiments
To measure performance, multiple tests were conducted to assure system capabilities in order to improve accuracy and decrease delay. The first test addressed the first component of the system; EOG accuracy. Many design steps for EOG were made, the first design had some problems, it needs continuous calibration, and when patient move the eye from right direction to center it considered as a left movement, so making a wrong result, low efficiency performance and long delay between each movement.
Then, the design was changed by correcting the direction problem, delay problem, and accuracy increased but still need improvement because the range for amplitude was still small. The final EOG circuit shown in figure 2 increased accuracy and solved problem by calibration sensors [4] and setting up the environment, where electrode placements should be precise, and amplifiers require calibration at the beginning and end of signal recording to allow actual waveform frequency and amplitude measurements. We made tests for EOG circuit in different conditions, in the first test, EOG circuit was without calibration, and we obtained results as shown in table 1. Here the error percentage is measured by the number of incorrect trials over all trials, and the delay is measured by the time needed to enter one command sequence to turn on, then turn off a device.
EOG commands are sent as patterns of Left, Right, Up and Down movements, for example RRLR means TV set ON, UUDU means heater OFF. The user should move his eyes clearly to enable the system detect the command sequence correctly.
As seen, when the user is tired, he do not move his eyes correctly, and more time is needed to detect the command, and took the maximum delay for turned light on, then off, with the lowest accuracy. Also, it is noticed that intense light causes more errors than when the patient in dark environment. Figure 3 shows the effect of conditions on the percentage of error, so the results were better when the user is in relax mode and normal lighting.  The second test for the EOG circuit was done using calibration where electrode placements should be precise, and amplifiers require calibration at the beginning and end of signal recording to allow actual waveform frequency and amplitude measurements. It is noted that the maximum delay of turning the light on and off occurred in dark mode.
More errors were noticed in the intense light mode. But errors were fewer than the previous experiment, tables 2 and 3 shows the results of the second experiment when the user is in relax mode and in tired mode respectively.  Table 3. EOG design results with calibration and tired user Figure 4 describes the curves for both tired user (upper curve) and the relaxed user (lower curve), it shows the error rate at best case when the user is relaxed and the light is normal, it also shows the highest error rate when the user is tired and the lighting is intense, that is caused by eye movement incorrectly, that causes false interpretation of commands.
The EOG is used to assess the function of the pigment epithelium. During dark adaptation, resting potential decreases slightly and reaches a minimum after several minutes. When light is switched on, a substantial increase of the resting potential occurs, which drops off after few minutes when the retina adapts to the light [5].

Voice recognition testing and accuracy
In this section, an online application was used to listen to the human voice and send it through a Bluetooth connection. The connection between the mobile voice recognition application and the porotypes was through HC-05 Bluetooth module as shown in figure1. The online application voice recognition was tested with EVA mouse application [9]. To reduce the error, we used four to five sentences for each state of the device, and these are some samples of sentences. For the fan whose state is "OFF", four similar words. These sentences for the same object which was to turn fan off: value == "fan of" value == "send off" value == "fan off" value == "fa off" After many trials, and more similar words were taken in consideration, the error rate was reduced to 3% on the average. Since the environment affects voice detection and recognition, we considered some tests with outer voices, and others without outer voices. And we considered different number of similar words. Table 4 shows the results.
For the state of the light one off we increase number of similar words to five to have more accuracy, and the error reduced from 45% to 3%.
value == "light one-off" value == "light one of" value == "light one off" value == "light win off" value == "lights one off"  Table 5 shows the results of controlling light commands, it is noticed that the error rate is getting lower with more similar words for the same command, and with pure voice with no voices around. It was about 50% of errors with one word and voices around, and decreased significantly to 5% of errors with 5 similar words, due to more probability of matching with more words, and 1% if the voice is clear enough with no interference voices around.

Wi-Fi Devices
The connectivity between the IoT devices and the user was established using two NodeMcu Wi-Fi modules. For one user, one sender is required. At the receiver side, many senders can send and control the same devices over the IP network.
NodeMCU can handle many connections at the same time over IP networks [10], the receiver node in this prototype is connected to relays that switches ON/OFF the connected devices. It can be used furthermore to receive commands from different circuits and applications, and from different users to control the same devices, it also supports commands like fan speed control, TV channel selection, phone number dialing.
The circuit of the wireless network module is shown in figure 5, it supports different GPIOs, it is equipped with a wireless Wi-Fi adaptor chip, and can be configured as a client or a server, here both modes were used.

Conclusions
An IoT based system was designed to help people to communicate with the surrounding environment using eyes movement, or by voice to control devices at homes.
The surrounding environment effects the error rate, when increasing the light rate, and when the patient is tired, the error rate in EOG increased, and when increase the noise, the error rate in voice command increased. Many changes were made on the EOG system with calibration and increase the number of similar command voices to decrease the error rate and increase accuracy.
This system can be further developed, by adding more functions and requirements to handle more needs of the users.