EEG sinyallerinden bakılan görselin üretilmesi

Abstract

EEG sinyalleri kullanılarak engelliler için kontrol edilebilir tekerlekli sandalyelerin üretildiği veya yapılması düşünülen aktivitenin tahmin edildiği çalışmalara literatürde sıklıkla rastlanmaktadır. Genel olarak bu çalışmalarda elektroensefalografi (EEG) sinyalinin önceden belirlenen sınıflara aktarımı gerçekleştirilir. Bu çalışmalar EEG sinyalinin sınıflandırmasından ibarettir. Ancak son yıllarda yapay öğrenme alanında yaşanan gelişmelerle sınıflandırmadan öteye gidildiği, EEG sinyalinden bakılan görselin üretilebildiği görülmektedir. Klasik çekişmeli üretici ağlar (Generative adversarial networks-GAN) ve otomatik kodlayıcı (Auto encoder-AE) yaklaşımlarının kullanıldığı sınırlı sayıdaki bu çalışmalar incelendiğinde, EEG sinyallerinden kabaca görsellerin üretilebildiği görülmektedir. Bu çalışmanın özgün yönü, görsel üretim kabiliyetini arttıracak matematiksel yaklaşımlar içermesidir. Klasik GAN mimarileri üretilen görüntülerin çeşitliliğini sağlayabilmek için rastgele vektör girişini kullanırlar. Bu yaklaşım ile EEG sinyalinden üretilen görsellerin düşük kalitede olduğu gözlemlenmiştir. Önerilen yöntemde giriş iki kısım (kodlanmış EEG ve rastgelelik) olarak düşünülmüştür. EEG’nin kodlanması için değişken oto kodlayıcı (Variational auto encoder-VAE) ve fourier dönüşümü (FD) kullanılırken, rastgelelik için iki farklı yaklaşım önerilmiştir. Bu özgün GAN kullanımı, EEG sinyallerinden daha kaliteli görsel üretilmesini sağlamıştır. Bu kalitenin sayısal olarak anlaşılabilmesi için önceden eğitilmiş evrişimsel sinir ağları (ESA) kullanılmıştır. Yapılan deneysel çalışmalar neticesinde, klasik GAN ile EEG’den üretilen görsellerin başarım seviyesi %93 civarındayken, önerilen yaklaşımda bu seviyenin %95-%100 aralığına çıktığı görülmektedir.

Keywords

• EEG sinyalleri
• Çekişmeli üretici ağlar
• Yardımcı sınıflandırıcı GAN (ACGAN)
• Otomatik kodlayıcı
• Fourier dönüşümü
• Değişken oto kodlayıcı

Generating the image viewed from EEG signals

Abstract

In the literature, it is encountered a vast amount of studies related to the production of controllable wheelchairs for people with disabilities or the prediction of activity thought to be performed. In general, the electroencephalography (EEG) signal is transferred to predetermined classes in these studies. These studies consist of the classification of the EEG signal. However, it has been observed that in the recent years, with the developments in the field of artificial learning, the classification has gone beyond, It can be seen that the visual viewed from the EEG signal can be produced. When the limited number of studies using classical generative adversarial networks (GAN) and autu encoder (AE) approaches are examined, it is seen that visuals from EEG signals can be produced roughly. The original aspect of this study is that it includes mathematical approaches to increase the visual production capability. Classical GAN architectures use random vector input to provide a variety of images produced. With this approach, it is observed that the visuals produced from the EEG signal are of low quality. In the proposed method, the input is considered as two parts (coded EEG and randomness). Variable auto encoder (VAE) and fourier transform (FT) are used to encode the EEG, while two different approaches are proposed for randomness. The use of this original GAN has enabled higher quality visuals to be produced than EEG signals. In order to understand this quality numerically, pre-trained convolutional neural networks (CNN) was used. As a result of experimental studies, While the performance level of the visuals produced from EEG signals with classical GAN is around 93%, it is seen that this level rises to 95% -100% in the proposed approach.

Keywords

• EEG signals
• Generative adversarial networks
• Auxiliary classifier GAN (ACGAN)
• Auto encoder
• Fourier transform
• Variational auto encoder (VAE)

References

1. [1] Sampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M. "Deep learning human mind for automated visual classification". 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21-26 July 2017.
2. [2] Greenfield LJ, Geyer JD, Carney PR. "Reading EEGs: A Practical Approach". https://books.google.com.tr/books?hl=tr&lr=&id=DuzmRe5p7cYC&oi=fnd&pg=PR7&ots=vohxi2Wmlt&sig=asw948ngWgNjhnRbkFT9oGM0ojo&redir_esc=y#v=onepage&q&f=false, (10.09.2019).
3. [3] Pfurtscheller G, Lopes Da Silva FH. "Event-Related EEG/MEG synchronization and desynchronization: Basic principles". Clinical Neurophysiology, 110(11), 1842-1857, 1999.
4. [4] Grosse-wentrup M, Member S, Buss M. "Multiclass Common Spatial Patterns and Information Theoretic Feature Extraction". IEEE Transactions on Biomedical Engineering, 55(8), 1991-2000, 2008.
5. [5] Amin SU, Alsulaiman M, Muhammad G, Mekhtiche MA, Hossain MS. "Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion". Future Generation Computer Systems, 101, 542-554, 2019.
6. [6] Bashivan P, Rish I, Yeasin M, Codella N. "Learning representations from EEG with deep recurrent-convolutional neural networks". 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2-4 May 2016.
7. [7] Chen H, Song Y, Li X. "A deep learning framework for identifying children with ADHD using an EEG-Based brain network". Neurocomputing, 356, 83-96, 2019.
8. [8] Tirupattur P, Spampinato C, Rawat YS, Shah M. "ThoughtViz: Visualizing human thoughts using generative adversarial network". 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, 22-26 October 2018.

9. [9] Hwang S, Hong K, Son G, Byun H. "EZSL-GAN: EEG-based Zero-Shot Learning approach using a Generative Adversarial Network". 2019 7th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, Korea (South), 18-20 February 2019.
10. [10] Luo Y, Lu BL. "EEG Data Augmentation for Emotion Recognition Using a Conditional Wasserstein GAN". 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, 18-21 July 2018.
11. [11] Fahimi F, Ang KK. "Towards EEG Generation Using GANs for BCI Applications". 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Chicago, IL, USA, USA, 19-22 May 2019.
12. [12] Jiao Y, Deng Y, Luo Y, Lu B. "Driver Sleepiness Detection from EEG and EOG signals Using GAN and LSTM Networks". Neurocomputing, 408, 100-111, 2019. doi: 10.1016/j.neucom.2019.05.108, 2020.
13. [13] Corley IA, Huang Y. "Deep EEG super-resolution: Upsampling EEG spatial resolution with Generative Adversarial Networks". 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Las Vegas, NV, USA, 4-7 March 2018.
14. [14] Hartmann KG, Schirrmeister RT, Ball T. "EEG-GAN: Generative adversarial networks for electroencephalograhic (EEG) brain signals". arXiv, 2018. https://arxiv.org/abs/1806.01875.
15. [15] Olson LC. "How Brain Death Works". https://science.howstuffworks.com/life/inside-the-mind/human-brain/brain-death1.htmhttps://science.howstuffworks.com/life/inside-the-mind/human-brain/brain-death1.htm (13.11.2019).
16. [16] Yi X, Walia E, Babyn P. "Generative adversarial network in medical imaging: A review". Medical Image Analysis, 58, 101552, 2019.
17. [17] Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde Farley D, Ozair S, Courville A, Bengio Y. "Generative Adversarial Networks". Veterinary Immunology and Immunopathology, 155(4), 270-275, 2014.
18. [18] Kim T, Cha M, Kim H, Lee JK, Kim J. "Learning to discover cross-domain relations with generative adversarial networks". ICML 2017: 34th International Conference on Machine Learning, Sydney, Australia, 6-11 August 2017.
19. [19] Zhu JY, Park T, Isola P, Efros AA. "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks". 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22-29 October 2017.
20. [20] Yi Z, Zhang H, Tan P, Gong M. "DualGAN: Unsupervised Dual Learning for Image-to-Image Translation". 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22-29 October 2017.
21. [21] Mao X, Wang S, Zheng L, Huang Q. "Semantic invariant cross-domain image generation with generative adversarial networks". Neurocomputing, 293, 55-63, 2018.
22. [22] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W. "Photo-realistic single image super-resolution using a generative adversarial network". 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21-26 July 2017.
23. [23] Li C, Wand M. Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks. Editors: Leibe B, Matas J, Sebe N, Welling M. Computer Vision-ECCV 2016, 702-716, Springer, Cham, 2016.
24. [24] Xu L, Zeng X, Li W, Huang Z. “Multi-granularity generative adversarial nets with reconstructive sampling for image inpainting”. Neurocomputing, 2020, https://doi.org/10.1016/j.neucom.2020.04.011.
25. [25] Antipov G, Baccouche M, Dugelay JL. "Face aging with conditional generative adversarial networks". 2017 IEEE International Conference on Image Processing (ICIP), Beijing, Chinai 17-20 September 2017.
26. [26] Zhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas D. "StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks". 2017 IEEE International Conference on Computer Vision, Venice, Italy, 22-29 October 2017.
27. [27] Duarte A. Roldan F, Tubau M, Escur J, Pascual S, Salvador A, Mohedano E, McGuinness K, Torres J, Giro-I-Nieto X. "Wav2Pix: Speech-Conditioned face generation using generative adversarial networks". ICASSP 2019, Brighton, UK, 12-17 May 2019.
28. [28] Odena A, Olah C, Shlens J. "Conditional ımage synthesis with auxiliary classifier GANs". ICML'17 Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia, 06-11 August 2017.
29. [29] Meziani A, Djouani K, Medkour T, Chibani A. "A Lasso quantile periodogram based feature extraction for EEG-based motor imagery". Journal of Neuroscience Methods, 2019. https://doi.org/10.1016/j.jneumeth.2019.108434.
30. [30] Gao J, Li L. "A robust geometric mean-based subspace discriminant analysis feature extraction approach for image set classification". Optik, 2019. https://doi.org/10.1016/j.ijleo.2019.163368.
31. [31] Luo X, Li X, Wang Z, Liang J. "Discriminant autoencoder for feature extraction in fault diagnosis". Chemometrics and Intelligent Laboratory Systems, 2019. https://doi.org/10.1016/j.chemolab.2019.103814.
32. [32] Zhao X, Jia M, Lin M. "Deep Laplacian Auto-encoder and its application into imbalanced fault diagnosis of rotating machinery". Measurement, 2020. https://doi.org/10.1016/j.measurement.2019.107320.
33. [33] Kingma DP, Welling M. "Auto-encoding variational bayes". 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, 14-16 April 2014.
34. [34] Fu X, Wei Y, Xu F, Wang T, Lu Y, Li J, Huang JZ. "Semi-supervised aspect-level sentiment classification model based on variational autoencoder". Knowledge-Based Systems, 171, 81-92, 2019.
35. [35] Wang X, Du Y, Lin S, Cui P, Shen Y, Yang Y. "adVAE: A self-adversarial variational autoencoder with Gaussian anomaly prior knowledge for anomaly detection". Knowledge-Based Systems, 2020. https://doi.org/10.1016/j.knosys.2019.105187.
36. [36] Merry RJE. "Wavelet theory and applications: a literature study". Department of Mechanical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands, 41, DCT Rapporten (2005.053), 2005.
37. [37] Öztürk H. Eeg Sinyallerinde Farklı Boyut İndirgeme ve Sınıflandırma Yöntemlerinin Karşılaştırılması. Yüksek Lisans Tezi, Adnan Menderes Üniversitesi, Aydın, Türkiye, 2016.
38. [38] Vivancos D. "MindBigData The “MNIST” of Brain Digits". http://www.mindbigdata.com/opendb/index.html (15.08.2019).
39. [39] LeCun Y, Cortes C, Burges CJC. “THE MNIST DATABASE of Handwritten Digits”. http://yann.lecun.com/exdb/mnist/ (22.11 2019).
40. [40] Xiao H, Rasul K, Vollgraf R. “Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms”. arXiv, 2017. https://arxiv.org/abs/1708.07747.
41. [41] Acı Çİ, Çırak A. “Türkçe haber metinlerinin konvolüsyonel sinir ağları ve Word2Vec kullanılarak sınıflandırılması”. Bilişim Teknol. Dergisi, 12(3), 219-228, 2019.