Detection of Consumer Preferences Using EEG Signals

In this study, a liking estimation system based on electroencephalogram (EEG) signals is developed for neuromarketing applications. The determination of the degree of appreciation of a product by consumers has become an important research topic using machine learning methods. Biological data is recorded while viewing product pictures or videos, then processed by signal processing methods. In this study, 32 channel EEG signals are recorded from subjects who watched two different car advertisement videos and the liking status is determined. After watching the advertisement videos, the participants were asked to vote for the rating of the different images (front view, dashboard, side view, rear view, taillight, logo and grille) of the products. The signals corresponding to these different video regions from the EEG recordings were segmented and analyzed by the Empirical Mode Decomposition (EMD) and Ensemble Empirical Mode Decomposition (EEMD). The statistical features were extracted from Intrinsic Mode Functions (IMF) and the liking status classifications were performed. The classification performance of EMD- and EEMD-based methods are 93.4% and 97.8% respectively on Brand1, and 93.5% and 97.4% respectively on Brand2. In addition, the classification accuracy on both brands combined are 85.1% and 85.7% respectively. The promising results obtained using Support Vector Machines (SVM) show that the proposed EEG-based method may be used in neuromarketing studies.


Introduction
Neuromarketing has emerged through a combination of different disciplines such as marketing, neuroscience and psychology [1]. It is a marketing technique utilizing neuroscience knowledge in order to understand the market behavior of consumers, where these techniques are applied to consumers in the field of advertisingmarketing, to understand the consumer's mind, whether they like the product, their buying tendencies, and wishes [2].
Neuromarketing technique is used by large global companies in the design, color, and development of a product, as well as in the advertisement and logo design to ensure the end user purchases [2].
Neuromarketing research is concerned with consumer behavior in purchasing products using neuroscientific methods. The most important factors affecting these consumer behaviors are needs, likes, and the price of the product [3], [4]. As for neuroscientific methods to measure the brain's structural or functional activities; functional Magnetic Resonance Imaging (fMRI), electroencephalography (EEG), and functional near-infrared spectroscopy (fNIRS) may be utilized [5]. These methods are used in the detection of various diseases, as well as in the field of cognitive studies, neuromarketing, and for the investigation of visual, auditory, audiovisual, haptic, focus, smell, etc. mechanism. They are also used to study the reaction to those stimuli affecting our emotions in the brain [6], [7]. Researchers investigate stimuli that cause changes in consumer's brain electrical activities and to determine which parts of the brain are active when deciding whether or not to buy a product [8], [9], [10].
Electroencephalogram (EEG) signals are obtained by recording the electrical activity of the brain. Signals from electrodes placed on the scalp are called surface EEG recordings. EEG signals are mostly recorded with reference to the international 10-20 electrode positioning system. EEG is highly preferred in neuromarketing and other clinical and engineering application because it is non-interventional, relatively low cost, provides high temporal resolution, and simple to use. In this study, we utilize multichannel EEG signals to estimate consumers liking state of a product using advanced signal decomposition methods, and machine learning algorithms. The liking state detection using EEG signals is a complex study due to the non-stationary behavior of signals caused by complex neuronal activity in the brain. Advanced signal processing methods should be used to extract this information hidden in EEG signals. In this study, the Empirical Mode Decomposition (EMD), and Ensemble Empirical Mode Decomposition (EEMD) are used to extract features from the EEG fragments corresponding to the stimulus video regions presented to the subjects.

Data Collection And Experiment Procedure
Within the scope of this study, experiments were carried out in Izmir Katip Çelebi University, Biomedical Engineering Department, Electrophysiological Signal Laboratory. Advertisement videos of 2 different brands ("Brand 1" and "Brand 2") of cars were viewed by 24 participants; 13 people watched "Brand 1" and 11 people watched "Brand 2" advertisement videos. Each advertising video have about 75 seconds duration. The presentations of audiovisual stimuli and the questionnaires to the subjects were carried out with the IMotion 8.0 software [11].
Brain Product brand, ActiCHamp model, 32 channel EEG device and EEG cap designed with 10-20 electrode connection system were used to collect the EEG data.
In the experiment design, both the presentation of the stimulating videos and the questionnaire evaluation questions were presented to the subjects with the IMotions software on the same screen. The experiment design consists of 3 slides of personal information, followed by a brand's video and car-related questions. Volunteer individuals participating in the experiment were provided to watch their advertising videos after entering their personal information on the screen.
The subjects were shown an advertisement video of a car brand and 7 different photos selected from the video. These 7 photos consist of the front view, dashboard, side view, rear view, taillight, logo, and grille sections of the car. The display time of each photo is 10 seconds. A 5second black screen is shown between the video and each photo. During the 10-second photo display period, the participants were asked to indicate their preferences in the form opened on the screen using a scale of; I hated (-2), I did not like (-1), Undecided (0), I liked (1), and I liked very much (2). In the EEG signals recorded synchronously while watching the advertisement video, the EEG responses corresponding to these seven sections were segmented and used in the signal processing stage.

Data Preprocessing
The raw EEG signal has been pretreated to remove unwanted noise and non-EEG artifacts. These necessary filters were designed and noises other than 0.5 -45 Hz were cleared [12]. Fp1, AF3, F3, F7, FC1, FC5, T7, C3, Fp2, AF4, F4, F8, FC2, Fc6, T8 and C4 channels located in the frontal lobe, which is frequently used in EEGbased emotion recognition studies due to the presence of the emotional center in the brain frontal lobe. 16 EEG channels received were used for liking detection analysis.

Empirical Mode Decomposition
The Empirical Mode Decomposition (EMD) has been proposed as an adaptive and data based signal processing method by Huang et al. [13], [14]. The EMD method is a data driven algorithm that expresses a signal as the sum of a finite number of oscillations. The given signal is decomposed into intrinsic mode functions (IMF) and a residual component with the help of sifting iteration [15]. Unlike wavelet decomposition, IMFs are obtained as signal-dependent and semi-orthogonal basis functions as a result of the sifting algorithm [16]. EMD and its derivatives are successfully applied to a wide variety of problems for the analysis of non-stationary signals.
In our study, EMD is used to analyze the EEG signals and obtain the first 5 IMFs corresponding to target stimulus video regions. Features extracted from those IMFs are used for the classification of EEG signals into different emotional ratings [17].

Ensemble Empirical Mode Decomposition
Many different signal processing methods have been developed for the analysis of non-stationary signals such as EEG. In the Empirical Mode Decomposition (EMD) method, which is frequently used as the noise removal method, the signals are decomposed into the zero-average IMFs that make up itself. However, when examining the IMFs obtained in this method, it was observed that oscillations of similar amplitude occur in different modes or oscillations of different amplitudes in the same mode. The major deficiency of EMD is the effect of mode mixing. To overcome this problem, Wu and Huang have proposed a noise-assisted EMD algorithm called Ensemble Empirical Mode Decomposition (EEMD), which allows for better scale separation than the standard EMD method [18].
In this method, different Gaussian white noises are added to the signal in several trials. Afterwards, IMFs are obtained for each signal by applying EMD method to these signals separately. The noise added to the signal in each trial is different due to randomness. As the IMFs obtained as a result of each experiment are compared with each other, no similarity or correlation is observed between the IMFs. When sufficient trials are reached, the added noise can be eliminated by the average of the IMFs obtained for different trials [19].
In our study, EEMD is used to analyze the EEG signals to eliminate the deficiencies of the standard EMD. Features are extracted from the first 8 IMFs, obtained at 100 trials, corresponding to target stimulus video regions. These features are used for the classification of EEG signals into different emotional ratings.

Windowing and Feature Extraction
EEG signals are segmented according to the video contents after the preprocessing step is performed. In the video watched during the experiment, the sections in which the liking status is measured have different lengths. For example, while the front view of the car takes half a second in the advertisement, the part where the logo appears in the same advertisement takes longer. For this reason, the analysis window length varies depending on the segmented signal length. The window length is determined as a quarter of the segmented signal. The windows are shifted with 25% overlap. Thus, the EEG signals we segmented were analyzed by dividing them into windows. With the help of IMFs obtained from EMD and EEMD analysis of these windows, the following features have been obtained.
There are many proposed approaches to obtain features for different applications utilizing physiological signals [20].
The minimum, maximum, range, mean, median, coefficient of variance in eq. (1), third moment in (2), variance, kurtosis (3), root mean square (4), energy (5), mean deviation (6), higher order frequency moment (7) attributes are used in our approach: (2) Here ( )denotes the power spectral estimate of the EEG segment x(n), ̅ shows the mean, and σ is the standard deviation. In order to reduce the dimension of the feature set, 1000 active features were selected using the Relief algorithm [21].

Classification
Classification is the process of estimating the class of an unknown label using a model derived from a data set called training data set. Features extracted from labeled EEG segments are used for the classification. In this study, the first 5 IMFs obtained from EMD analysis of EEG segments, and the first 8 IMFs extracted by the EEMD analysis are used for feature extraction and classification. The liking status estimation from the extracted features is classified as brand dependent and independent of the brand For example, the features extracted from the front view, dashboard, side view, rear view, taillight, logo and grille view of people watching the Brand 1 ad video are classified by Support Vector Machines (SVM) by labeling them according to the scores given during the survey stage.
While classifying, 10-fold cross verification was used. While labeling, the answers "I like" and "I like very much" were grouped as "I like (1)" and the other answers were grouped as "I did not like (0) ".
Polynomial (linear, quadratic, cubic) and Gaussian medium kernel functions were chosen for classification with SVM.

Experimental Results
The classification results obtained in the study are summarized in the following performance tables. In order to evaluate the effectiveness of the proposed method, the classification results of features obtained by proposed EMD-based, and EEMD-based methods, are compared with that of the features obtained directly from raw EEG signals. Table 1 and Table 2 show the classification results obtained using EMD-based and EEMD-based methods respectively. The classification results using the attributes obtained directly from the EEG signals are provided in Table 3. When Table 1, Table 2 and Table 3 are analyzed, it is seen that the EEMD-based method increases accuracy performance compared to the classification by EMD-based method, and EEG signal features.
It is seen in the tables that the results are evaluated in 3 categories; "Brand 1", "Brand 2", and "Both Brands". "Brand 1" and "Brand 2" show the "like and dislike" classification of the separate groups watching the commercial films belonging to two different brands, while "Both Brands" show the classification results of both groups together, independent of the brand.  Evaluating the results in Tables 1 -3, it may be seen that the classification performance of EMD based and EEMD based methods are quite high. It is also observed that the classification performance of "Both Brands" is lower than the classification performances of "Brand 1" and "Brand 2". The "Brand 2" commercial appears to be better classified than the "Brand 1" commercial using EEG signal itself.
When we compare Table 1 and Table 2, it seems that EEMD-based method has better classification success than EMD-based method except for "Brand 2 -SVM(Quadratic)" thanks to the elimination of the Mod mixing problem in the EEMD method.
For the proposed EMD-and EEMD-based liking detection methods, other performance metrics are calculated for SVM-Medium classifier, which results the highest performance, and shown in Table 4 and Table 5. Accuracy, Sensitivity, Kappa, and F1-score performance metrics are given for both EMD-and EEMD-based approaches. Evaluating the results in the tables, we may conclude that liking state of participants are successfully classified using the features extracted by EEMD method in terms of the accuracy, sensitivity, Kappa, and F1-score metrics.

Conclusions and Discussion
In this study, EMD-and EEMD-based feature extraction methods are proposed for the detection of liking status for use in neuromarketing applications. 24 volunteers were divided into 2 different groups of 13 and 11 people, allowing them to watch advertisement videos of 2 different brands of cars. The features extracted from the first 5 IMFs of EMD, and from the first 8 IMFs of the EEMD analysis were used for classification after dimension reduction.
The features obtained from the EEG signal itself, and the same features extracted by the proposed EMD and EEMD based approaches are classified by the SVM. Comparing Tables 1-3, we observe that, proposed EMD or EEMD based approaches improve the classification performance. SVM-Medium classifier provided the highest classification accuracy in both methods.
We propose to use the EMD as well as the EEMD method in this study, for the analysis of EEG signals for like/dislike estimation of consumers. Notice that, by overcoming the mode mixing problem of EMD, EEMD method yields better classification performance. However, Tables 1 and 3 show that EMD method provides higher classification performance than that of the plain EEG signals.
In our ongoing studies, proposed approach will be extended by including different attributes. Moreover, IMF selection methods will be utilized to choose the effective IMFs for feature extraction and classification.