A new Approach for Hybrid BCI speller based on P300 and SSVEP

P300 and steady state visual evoked potential (SSVEP) are type of electroencephalography (EEG) phenomena that widely used in brain computer interface (BCI) systems since both of them have high signal response and signal noise ratio. Classification accuracy rate of signal, and signal detection time affect overall performance of BCI systems. These both values are used for calculation information transfer rate (ITR) that is a key performance indicator for a BCI system. A P300 based BCI or a SSVEP based BCI have higher ITR values than other type of BCI systems. Thus, in this study our aim was to use together these both P300 and SSVEP phenomena in a BCI speller. We proposed a hybrid BCI speller based on P300 and SSVEP. Moreover, our proposed BCI speller interface allows to use only P300 stimuli, only SSVEP stimuli, or hybrid stimuli. In this BCI speller, there are numbers in 3 × 3 matrix form for elicitind P300 signal and also 9 white square flickering objects were placed near numbers for eliciting SSVEP. In this research, experiments were performed in two stage (training and online stages) with three sessions (only SSVEP stimuli session, only P300 stimuli session, and hybrid session). Five subjects participated experiments. We used support vector machine method for detection of P300 signal and SSVEP. According to experiment results, average classification accuracy values were 83.78%, 84.67%, and 90.89% with using only SSVEP stimuli, only P300 stimuli, and hybrid stimuli, respectively. Furhermore, average information transfer rate values were 6.81, 6.97, and, 8.19 bit/min with using only SSVEP stimuli, only P300 stimuli, and hybrid stimuli, respectively. Results showed that the proposed hybrid BCI speller based on P300 and SSVEP reached higher classification accuracy and ITR values than using only SSVEP stimuli or only P300 stimuli based BCI spellers.


Introduction
Brain-computer interface (BCI) systems provides a new alternative communication systems for both disabled people and healthy people (Chaudhary et al., 2016). The main point of using BCI doesn't require to use neuro-muscular system. BCIs rely on the magnetoencephalography (MEG), functional magnetic resonance (FMRI), near infrared spectroscopy (NIRS), and electroencephalogram (EEG) . The most suitable for real time application and practical way to measure brain signals is EEG. Moreover, EEG measurement is a non-invasive method. Thus, Most of BCIs are based on EEG measurement (Kauhanen et al., 2006). A BCI which is rely on EEG is named according to types of EEG signals used for system control. Steady-state visual evoked potential (SSVEP), event-related potential (ERP) such as P300, event-related synchronization/desynchronization (ERD/ERS), and slow cortical potentials are type of EEG phenomena that widely used in BCIs (Ramadan et al., 2017).
The SSVEP is a brain's response to visual stimulation which has higher frequency value of 3 Hz. The response has same frequency of visual stimulation as well as stimulation's harmonics. Structure of visual stimulus is flickering shape with a constant frequency. Previous researches shows that the flickering visual stimulus can be modulated at frequencies between 3 Hz and 80 Hz (İşcan & Nikulin, 2018). SSVEP occurs dominantly brain area of visual cortex. Maximum amplitude of SSVEP can be acquired from occipital region of brain. SSVEP is widely using in BCI applications, because it has high value signal response and signal to noise ratio (SNR) value (Marx et al., 2019). Moreover, it provides short training time for signal classification .
When a user of the SSVEP based BCI gazes to target visual stimulus, EEG data is acquired from occipital region of brain. When the BCI detects the brain signals with same frequency value of target visual stimulus or its harmonics, a BCI command can be produced. Thus, A target visual stimulus corresponds a command in a SSVEP based BCI. Sutter (1992) presented a BCI based on transient visual evoked potential that was named brain response interface in 1992. A user reached rate of communication 10 to 12 words in a minute with implanted electrodes. Information transfer rate (ITR) and classification accuracy have critical role to evaluate to a BCI's overall performance. After this research SSVEP based BCI systems became more popular. In recent years, there are many research for SSVEP based BCIs for improving overall performance. Researchers has been investigating types of SSVEP stimulator, signal processing and classification methods for reaching higher ITR values. Oralhan and Tokmaci (2016) researched about SSVEP stimulator structure which is about duty cycle and brightness variation of a stimulus with using Liquid Crystal Display (LCD). They showed that stimulus optimal brightness and duty cycle ratio. LCD refresh rate can be problem if a visual stimulus frequency cannot be fully divided to LCD refresh rate. Gao et al. (2003) used light emitting diode (LED) SSVEP stimulator with 48 BCI commands. Thus, a LED stimulator provides high range of visual stimulus frequency.
The P300 signal is one of most preferable type of EEG phenomena for a BCI application. The P300 signal is an endogenous signal type in EEG phenomena . That's mean is The P300 signal is not related to physical features of stimulus, it is a respond to stimulus. The P300 signal can be elicited when expected stimulus occurs around unexpected stimuli. It is important that expected (can be named as target) stimulus occurs rarely according to unexpected stimuli (Wang et al., 2016). The P300 signal is observed as a positive peak 300 ms after the expected stimulus occurs (Hoffman et al., 2008). The latency can be changed according to user, task, and stimulus type. Moreover, amplitude of P300 signal depends on the target's improbability (Halder et al., 2015). The P300 phenomena is used in BCI design as much as SSVEP based BCI applications. A P300 based BCI widely depends on oddball paradigm. According to oddball paradigm there are two types of stimuli that are target and standard. Standard stimuli occurs more frequently than target one. Moreover, stimuli can be chosen as voice, visual, or both visual and voice (Belitski et al., 2011). There are also other paradigms such as three stimulus and single stimulus which are used for eliciting P300 signal (Walsh et al., 2017). Sutton et al. (1965) presented P300 signal to literature for the first time with their research. After that time P300 signal helped to diagnosis some physiological disorders. In following years, The P300 signal has been a research subject to use it in control external devices. P300 based BCI design became popular research topic since the P300 signal has high signal response and less cognitive fatigue effect. Moreover, it provides higher ITR and classification accuracy rate. Farwell & Donchin (1988) introduced 6 × 6 speller in matrix structure. It is called row-column or Farwell -Donchin (FD) P300 speller. Each row and column has 6 different characters. A user counts by him/herself how many times his/her target flashes while each row or column is flashing randomly. The P300 based BCI system detects P300 signal in which row and column flashes. Hence, corresponded row and columns character is selected as target character. There are also other types P300 speller such as region, checkerboard phenomenon, and three dimensional based ones (Fazel-Rezai, R., & Ahmad, 2011).
In this research, we designed a BCI speller which based on P300 and SSVEP. Both signals are very useful for BCI application. Thus, our approach combine them in one speller. There are also some researches in literature with using P300 and SSVEP together.  proved that hybrid BCI with using P300 and SSVEP could improve overall performance of a BCI application. They used flickering cells with six different frequencies in order to elicit SSVEP. They also used numbers in cells and each column or row intensifies randomly to elicit P300 signal. Li et al. (2013) investigated a different approach for P300 and SSVEP hybrid speller to control wheelchair. They used four group buttons image on graphical user interface (GUI). Their P300 speller paradigm was region based. This hybrid BCI controlled to wheelchair with high performance. Wang et al. (2015) suggested shape and color changing paradigm to elicit P300 speller for hybrid spellers with P300 and SSVEP. Panicker et al. proposed asynchronous BCI system with combining P300 and SSVEP. The control state detection was successfully performed with using SSVEP. Edlinger et al. (2011) also applied P300 and SSVEP hybrid paradigm with existing ones to control smart home environment. There is not sufficient research for P300 and SSVEP hybrid paradigm although the paradigm has high overall performance (Fazel-Rezai et al., 2013). In our new approach P300 and SSVEP paradigms combined with separated stimuli. Experiments were carried out with only P300, only SSVEP, and hybrid structured spellers. Our aim was to prove with enhanced hybrid speller to reach higher classification accuracy and ITR value. We reached higher performance with our approach hybrid speller. The main difference of our hybrid speller with other existing hybrid spellers that P300 and SSVEP were elicited from different stimuli in same speller.

Materials and Methods
Hybrid speller which based on P300 and SSVEP paradigm were presented in this research. Support vector machine method (SVM) were used for P300 and SSVEP detection.
Experiments were carried out in three sessions. P300 speller, SSVEP speller, and hybrid speller with P300 and SSVEP paradigm were used in sessions, respectively. Four male and one female subjects with 20 ± 2 average age were participated experiments. Any of subject has chronic disorder. They also hadn't have an operation before. Moreover, subjects had normal or corrected sight.

BCI Speller Type
Our BCI speller was developed with using C# coding. In the speller there are numbers in 3 × 3 matrix form. Also nine white square flickering objects were placed near numbers. The speller can be adjusted via settings menu as shown in Figure 1. It allows to work with only P300 stimuli, only SSVEP stimuli or hybrid stimuli. Furthermore, P300 inter stimulus interval time (ISI) and flickering objects' frequencies can be settled. ISI is a time indicator of between activating of two stimuli.

Figure 1. Proposed BCI Speller Settings Menu
When BCI speller is working as P300 speller only 3 × 3 matrix is displaying on screen. Each row or column intensifies in a pseudorandom sequence. When BCI speller is working as SSVEP stimulator only 9 flickering white square objects are shown on screen. Our proposed hybrid BCI speller based on P300 and SSVEP is shown in Figure 2. The hybrid speller provides a user spell number from one to nine. In the hybrid speller an object and number which is nearer to the object is within the viewing angle of a user. Thus, when a user want to spell a number, he or she gazes to flickering object near the number. User intend to count how many times focused number is changed its color from light blue to dark blue (intensifying or flashing).
To use our proposed hybrid BCI speller is suitable both online and offline session of experimental procedures.

Experiment Data Collection
In experiments, The EEG data were recorded from the scalp with eight channels with reference placed at FPz-A1, Cz-A1, C3-A1, Pz-A1, P8-A1, Oz-A1, and O1-A1 according to international 10-20 EEG electrodes placement system. Ground channel place was between eyebrows that is named intercilium point. EEG data were sampled at 250 Hz with 50 Hz notch filter.

Figure 3. Proposed hybrid BCI speller graphical user interface
CleveMed BioRadio mobile EEG device with BioCapture software were used for acquiring EEG data. Great Lakes Neuro Tech. which is US manufacturer of the device.

Experimental Procedure
All experiments were carried out in a normal university office room in a day time. After environmental preparation of EEG recording, experiment participants were seated in a chair nearly 70 cm across from LCD monitor which had 60 Hz refresh rate with 1366x768 pixels resolution.
Experiments were performed in two stage with three sessions as shown in Figure 4. SSVEP stimulator mode of our BCI user interface was used in session-1. P300 speller mode was used in session-2. As a session-3, hybrid (SSVEP and P300) mode of BCI user interface was used. In training stage, each column or row intensified for 200 ms and when P300 speller was used in sessions. Also, after a stimulus intensified there was 100 ms ISI time as non-intensified duration. A sequence was defined when each column and row intensified. Thus a sequence was taken 1.8 seconds. 10 repeated sequences that named a trial was completed in sessions with P300 speller. Thus, a number selection duration was 18 seconds in training stage. Moreover, all subjects gazed to stimulus for 18 seconds in sessions with SSVEP stimulator in training stage. An Experiment subject was asked to spell "468521973" numbers sequence which consisted nine different numbers in training stage. In online stage of experiment "861973542" number sequence was desired to spell for 9 times by experiment subjects.

Data Analysis
In experiments we applied same data analysis in both online and training stages. We used support vector machine for P300 and SSVEP detection.

Support Vector Machine
In experiments EEG data, determining whether P300 or SSVEP existing or not is a binary classification problem. The problem with discriminant function can be defined as equation 1.

× − = 0
(1) Support vector machine (SVM) is used to determine hyper plane which maximizes separating line between the two classes. Thus, it is used for binary classification (Baverina et al., 2003).
Class labels were codded as , equation 1 could be reformulated as equation 2.
" " and " " denote the regularization parameter and number of training samples, respectively. Lagrangian multipliers can be used for the optimization problem. Equivalent maximizing was given in equation 4. The parameters of SVM were varied in order to each user, resulting in insignificant performance differences. So, = 10 was determined as the best given performance on training data.

Calculation of Overall Performance of BCI Spellers
Information transfer rate (ITR) is a key performance indicator for overall performance of a BCI speller. ITR can be formulated as in equation 6.
According to equation 6 " " denotes the EEG signal classification accuracy value, " " denotes total number stimuli, and " " denotes the number selection time by a BCI speller's user.

Results
EEG data was analyzed according to SVM method. In online stage of experiment nine different words were spelled for nine times in a trial. As shown in Table 1 classification accuracy values were given according to using mode of BCI user interface in online stage of experiment. According to Table 1, average classification accuracy values were 83.78%, 84.67%, and 90.89% with using SSVEP stimulator, P300 speller, and hybrid BCI speller, respectively. Moreover, subject-5 reached highest classification accuracy value as 96.67% with using hybrid BCI speller. The lowest classification accuracy value was 72.22% by subject-3 with using SSVEP stimulator mode of BCI speller. Table 2 depicts the ITR values according to using mode of BCI user interface in online stage of experiment. Average ITR values were 6.81, 6.97, and, 8.19 with using SSVEP stimulator, P300 speller, and hybrid BCI speller, respectively. ITR is the significant indicator to show overall performance of a BCI speller. Thus, the proposed hybrid BCI speller has ITR value significantly higher than other modes of BCI speller.