DETERMINATION OF EMOTIONAL STATUS FROM EEG TIME SERIES BY USING EMD BASED LOCAL BINARY PATTERN METHOD

Although detecting emotional states from brain dynamics is an issue that has been studied for a long time, the desired level has not been reached yet. In this study, Empirical mode decomposition (EMD) based Local Binary Pattern (LBP) method is proposed to determine emotional states using (positive-neutral-negative) Electroencephalogram (EEG) signals. By using this method, a hybrid structure is created to obtain features from EEG signals. In the study, Seed EEG dataset including 15 positive subjects and positive-neutral-negative emotional state is used. In proposed method, a classification task is utilized with the basis of individuals by using 27 EEG channels from left hemisphere of each subject. Level 5 is separated by applying EMD to EEG segments that contain three emotional states. Features are obtained from the Intrinsic Mode Functions (IMFs) using LBP method. These features are classified with k Nearest Neighbor (k-NN) and Artificial Neural Network (ANN). The average classification accuracy for 15 participants is 83.77% with k-NN classifier and 84.50% with the ANN classifier. Furthermore, the highest classification performance is found to be 96.75% with the k-NN classifier. The results obtained in the study support related studies in the literature.


Introduction
One of the biggest goal of technology is to make life easier for human beings and to offer the best option for their needs. In this manner, many different studies have been carried out in different fields. One of the most important studies is about the research on the human brain. Many techniques are used to solve mysterious structure of human nerveous system. Techniques such as Electrocortigography, Magnetoencephalography and Functional Magnetic Resonance Imaging can be evaluated in this context. However, none of these techniques are as accessible and inexpensive as the Electroencephalogram (EEG) as used today.
EEG refers to electrical activities related to functional and mental activities of the brain. According to these activities, different oscillations can be observed in the EEG records. [1][2]. Data from these oscillations may also be used to control computerized devices and objects. Such control methods are defined as Brain Computer Interfaces (BCIs). In BCIs, disease detection, emotion, movement of body parts or imaginary states have been evaluated using EEG. In researches relating these areas, signal processing methods are used for the extraction of distinctive features. The path generally followed in the classification task of EEG signals consists of three stages: i) taking the EEG record and removing it from the artifacts to be processed (Pre-Processing), ii) obtaining the features, iii) classifying the features with the machine learning algorithms. EEG records shared in public databases for analysis are mostly in artifact-free form. Therefore, no action needs to be taken for the first stage of the mentioned 3 steps. The most crucial stage of the classification of EEG records is obtaining the feature vector with the proper method. It has great importance that these feature vectors represent the differences of EEG signals. EEG signals show potentials at different frequencies. Therefore, while extracting features from EEG signals, conversion methods that can detect dominant frequencies come to the forth [3]. One of them is the Empirical mode decomposition (EMD) method, which can separate the signal into its components at different frequencies. Signal content at different frequencies is obtained by using EMD method. After this transformation, feature methods will be used in the classification process and can represent the characteristics of the data set. In this study, Local Binary Pattern (LBP) method is proposed as the method of obtaining features. Because it is sensitive to the local changes of the signal, LBP can be used as a feature extraction method in signal processing studies [4][5]. Therefore, this method was preferred for emotion determination in this study.
There are many studies on the emotional state from the EEG signals in the literature. A few studies are given below that are similar with proposed study.
In the study [6], advanced features of EMD and multivariate extension (MEMD) are used for emotion recognition. A MEMD-based feature extraction method is proposed for processing multichannel EEG signals for emotion recognition. The obtained features are classified with k Nearest Neighbor (k-NN) and Artificial Neural Network (ANN). In the study, 75% classification success is achieved with ANN classifier and 67% with k-NN.
In another study [7], a method based on flexible analytical wavelet transform (FAWT) is proposed for emotion detection from EEG signals. By using the FAWT method, the EEG signal is separated into different subbands. Mel Frequency Cepstral Coefficients (MFCC) of the subbands are obtained. The obtained features are classified with Support Vector Machine (SVM) and Random Forest classifiers. In related study, 83.5% classification success is achieved with SVM classifier and 72.07% with Random Forest. It can be seen that extensive research has been conducted in the literature for the automatic detection of emotion states from EEG signals. In proposed study, EEG signals are converted into different frequency bands using EMD to detect the three emotional states from the left hemisphere (Positive-Nort-Negative) of the brain. Features are obtained by using LBP method from IMFs of EMD. These features are classified using k-NN and ANN. The steps followed in this study is shown in Figure 1.
The rest of the paper is organized as follows. In the second part, the data set used in the study is explained with details. In the third part, the methods used to obtain the feature from the EEG data set used are described. The fourth section describes the classifiers used in the classification of features. Finally, the success performances are given and evaluated in the results section.

EEG Data Set
In this study, the Seed EEG data set is used [8]. The EEG data set has a sampling frequency of 200 Hz. A 0-75 Hz bandpass filter is applied to EEG signals. EEG signals were recorded from 15 subjects (7 males and 8 females; mean: 23.27, std: 2.37) while they were watching an emotional movie clip. 15 Chinese movie clips (positive, neutral and negative emotions) were selected for the study. The approximate duration of each movie clip is 4 minutes. The EEG segments matching with the duration of each movie were extracted. Three experiments were performed with each subject approximately one week apart. There are a total of 45 experiments in this dataset.
Details of the movie clips used in the experiments are listed in Table 1:  Figure 2 shows the EEG channel locations according to the international 10-20 system with 62 channels during EEG records from subjects.
In the study, 27 channels are used for each subject from EEG recordings taken from 62 channels. These 27 channels are taken only from the left hemisphere of the brain. In this perspective, it is aimed to observe the difference that the emotional state creates in the left hemisphere of the brain.

Obtaining the Features
In order to obtain features from EEG signals, recordings from each subject are separated up to 5th IMFs using EMD. Features are extracted using LBP from the obtained IMFs.

Empirical Mode Decomposition (EMD)
EMD is the method that seperates signal into its components at different frequencies [9]. In the EMD method, each parsing function is obtained after the conditions are met with and that is called the Intrinsic Mode Functions (IMFs). The IMFs are obtained as in steps given below. The signal to be processed is x (t); I. All extremes of the x (t) signal are defined (maximum and minimum points are determined). II. Interim evaluation is made between minimum and maximum points, and confident (t) and emax (t) envelopes are obtained. III. Average m(t) = (emin(t) + emax(t))/2 is calculated. IV. The detail is subtracted: d(t) = x (t) -m(t) V. The operations are repeated until the number of extreme points and the number of zero crossing points are equal to each other or one unit difference at most. VI. The IMFs are obtained from extracted signal. VII. Operation continues to the remaining m (t).
In this study, the signal is decomposed to the 5th IMFs. IMFs obtained from one channel of the EEG are shown in Figure 3. LBP can also be applied to EEG signals in addition to pattern recognition problems [10][11]. In the LBP method, for each data sample in a signal, the value of the central data is considered as the threshold value. This threshold value is based on the method of generating binary code by subtracting the data values on the right and left side. The flow chart of these processes is shown in Figure 4.
In proposed study, local binary codes are calculated as given below with LBP method from EEG records containing mental activities: I. The number of neighboring points is determined. II. For each signal data, the number of right and left points is determined from the midpoint. III. Each of determined point has a difference from the middle value. IV. If the obtained difference values are greater than 0, then 1 value is assigned. V. The binary code obtained has a decimal base equivalent. The formula for the Local Binary Pattern method is as follows: In order to apply LBP method, each EEG data is divided into segments. Each of them contains 9 data. By taking the mean value of these segments, the mean value in the segment is subtracted from each EEG data (4 neighbors on the right, 4 neighbors on the left). If the obtained result is greater than or equal to 0, 1 is assigned to the value obtained, and 0 in other cases ( see in Figure 4).

Figure 4. Local Binary Code Obtaining Method
The EEG signal is first converted into an 8-bit binary base using the LBP method. Then the resulting 8-bit binary base value is converted to decimal.

Obtaining a Histogram Based Feature Vector
The histogram is used to obtain an appropriate distribution of the signal codes calculated by the LBP method. The histogram shows the range of transformation codes of the horizontal axis and the structural distribution in two-dimensional space where the vertical axis contains the frequency of the code. This distribution represents the feature vector. The obtained range of binary code is between 0-255. In this study, 256 feature vectors are obtained from each EEG record for each emotional state. The positive emotional state LBP Histogram obtained from a subject is shown in Figure 5.

Figure 5. LBP Histogram of EEG Cyanyline of a Participant's Positive Emotional State
The features obtained from the histogram are classified by applying classification algorithms.

Classifiers
In current study, a total of 15x135x256 feature vectors are obtained (for one emotion), 135x256 (5 EMD total 5x27 for each channel) of each participant. Classification of the proposed model is achieved by applying 10-fold cross-validation.

k-Nearest Neighbor (k-NN) Algorithm
The k-NN is widely used in pattern recognition and data mining because it outperforms even in simple and noisy data [12]. In this algorithm, the similarity is measured by the distance in the feature area. The k-NN algorithm decides that the ti point falls in this category if it is a specific category with examples most similar to a ti point in the property space. In this classifier, the label of the data set to be given to train should be known [12]. Then, for a ti data whose tag is unknown and revealed by a vector in the property field, the distance between each point in the training data set is calculated [12]. After sorting the results of calculating the distances, the class label of the ti test data is determined according to the closest k value [12]. The coordinates of each point in the n-dimensional space can be expressed as the p-vector (Equation 3) [12].
Euclidean distance is often used to calculate the distance between two points in the multidimensional property space. Calculation of Euclidean distance is given in Equation 4.
In this study, the k values are tried at different values and the best performance is determined to be 3.

Artificial Neural Networks (ANN)
In ANN networks, neurons are organized into layers. These layers consist of input, one or more hidden, and an output layer. The input layer ensures that the information is imported into ANN. The output layer is the layer where information is transmitted to the outside. There may be one or more than one hidden layers between the input and output layers. For the proposed ANN, the number of hidden layer neurons is 100, the learning rate is 0.7, the epoch 1500, and the output layer is 3 neurons.

Classification Results
In the study, the features obtained from the left hemisphere of the brain were classified into different classes for emotional state detection. Features extracted from 15 subjects are classified using ANN and k-NN classifiers. Three emotions for each subject are classified with 10 fold cross validations. The results obtained for each subject in the study are shown in Table 2. As it can be seen in Table 2, an average of 83.77% success is obtained in the classification task using k-NN for 15 participants, and an average of 84.50% is achieved in the classification success with ANN. In addition, it is observed that the best result is obtained from the 6th participant (96.75%) using k-NN, and the 8th participant (95.00%) with ANN. On the other hand, the lowest success performance using k-NN is obtained from the 1st participant (66.00%) and the 5th and 11th participants (72.50%) with ANN. In the study, it was observed that the same features related to EEG of some participants gave different results in different classifiers, and the classification results of some participants were very close to each other. When the classification results are evaluated in general, it can be said that the ANN classifier is relatively successful compared to the k-NN classification.
Overview of the obtained results and the literature findings are given in Table 3. It can be seen that there are many studies in the literature using the EEG data set used in this study. Some of these studies are feature-based and some are deep learning-based studies. In this study, a feature-based approach has been applied. As can be seen from Table 3 that the classification results obtained are more successful than feature-based studies. On the other hand, it has been observed that the classification results give very similar results to deep learning-based studies.

Conclusion
In proposed study, EMB-based LBP features obtained from EEG records taken from 15 subjects, containing positive, neutral and negative emotion, are classified with k-NN and ANN methods. The electrodes placed on the left hemisphere of the brain are only considered. The proposed method is compared with a similar study in literature. From the view of obtained results, it is deduced that the proposed method provides an acceptable success in representing the differences of EEG signals. Therefore, it has been observed that the proposed method can be used to detect emotional states from EEG signals.On the other hand, it has been seen that EEG recordings taken from only one hemisphere of the brain can be used to determine the emotion. On the other hand, it can be seen that different classification performances from person to person are obtained in detecting emotional state from EEG signals. Therefore, it can be said that EEG recordings taken from different people for the same situation or paradigm have different oscillation characteristics. For future study, it is thought that the success to be achieved by using EEG records taken from the right hemisphere will reveal similarities and differences in the hemispheres according to the emotion. Also, it is aimed to apply deep learning methods to EEG data related to emotional states in the future.