Evaluation of Time-Frequency Based Features in Emotion Classification with EEG Using Machine Learning Methods

Abstract

Electroencephalography (EEG) signals are a widely used neurophysiological method for analyzing human mental and emotional states. EEG-based emotion recognition systems have enabled significant advancements in human-computer interaction and healthcare technologies. However, there is a lack of studies in the literature that integrate different types of features and assess their generalizability across individuals. Therefore, this study aims to compare the performance of various machine learning models using different validation methods, by addressing feature extraction, selection, and evaluation in a comprehensive manner. In our study, we extracted time and frequency components from 14-channel EEG signals recorded from 28 participants in the GAMEEMO dataset. The most informative features were then selected using the ANOVA method and interpreted at both the time and frequency levels. During the classification phase, we addressed both binary (positive/negative emotion) and multiclass (boring, calm, scary, fun) emotion recognition tasks. Various machine learning algorithms were compared using 5-fold cross-validation (CV) and leave-one-subject-out cross-validation (LOSOCV) strategies. In the binary emotion recognition task, the highest classification accuracy achieved was 94.70% under 5-fold CV and 96.43% under the LOSOCV scenario. In the multiclass emotion recognition task, the highest accuracy reached 96.44% using 5-fold CV and 98.21% under the LOSOCV strategy. The findings indicate that combining time- and frequency-based features not only yields high classification accuracy but also ensures strong inter-subject generalizability in EEG-based emotion recognition systems.

Keywords

• • Emotion recognition
• • Machine learning
• • Time-frequency features

EEG ile Duygu Sınıflandırmasında Zaman-Frekans Tabanlı Özniteliklerin Makine Öğrenmesi Yöntemleriyle Değerlendirilmesi

Abstract

Elektroansefalografi (EEG) sinyalleri insanların zihinsel ve duygusal durumlarını analiz etmede yaygın olarak kullanılan nörofizyolojik bir ölçüm yöntemidir. EEG tabanlı duygu tanıma sistemleri sayesinde insan-bilgisayar etkileşimi ve sağlık teknolojileri alanlarında önemli bulguların elde edilmesi mümkün olmuştur. Fakat literatürde farklı türdeki özniteliklerin birlikte kullanıldığı ve bireyler arası genellenebilirliğin değerlendirildiği çalışmalar oldukça sınırlıdır. Bu nedenle bu çalışmada öznitelik çıkarımı, seçimi ve değerlendirmesi bütüncül biçimde ele alınarak, farklı doğrulama yöntemleriyle çeşitli makine öğrenmesi modellerinin performanslarının karşılaştırılması amaçlanmıştır. Çalışmamızda GAMEEMO veri setinde yer alan 28 katılımcıya ait 14 kanallı EEG sinyallerinden, zaman ve frekans bileşenlerinin çıkarımı gerçekleştirilmiştir. Sonrasında ise, ANOVA yöntemi ile en etkili öznitelikler seçilerek, özniteliklerin frekans ve zaman düzeyinde yorumlanması sağlanmıştır. Sınıflandırma aşamasında, hem ikili (pozitif/negatif duygu) hem de çok sınıflı (Sıkıcı – Sakin – Korkunç – Eğlenceli) duygu sınıflandırma görevleri ele alınmış; çeşitli makine öğrenmesi algoritmaları hem 5 katlı çapraz doğrulama (5-fold CV) hem de birey-dışı bırakmalı çapraz doğrulama (LOSOCV) stratejileri altında karşılaştırılmıştır. İkili duygu sınıflandırma görevinde, 5-fold CV altında en yüksek sınıflandırma doğruluğu %94.70, LOSOCV senaryosunda ise %96.43 olarak elde edilmiştir. Çoklu duygu sınıflandırma görevinde, 5-fold CV yöntemiyle en yüksek doğruluk %96.44, LOSOCV stratejisi altında ise %98.21 olarak ölçülmüştür. Çalışma sonunda, zaman ve frekans temelli özniteliklerin birlikte kullanılmasının EEG tabanlı duygu tanıma sistemlerinde yalnızca yüksek sınıflandırma doğruluğu değil, aynı zamanda güçlü bireyler arası genellenebilirlik sunduğu görülmüştür.

Keywords

• • Duygu tanıma
• • Makine öğrenmesi
• • Zaman-frekans özellikleri

References

1. Abdulrahman, A., Baykara, M., & Alakus, T. B. (2022). A novel approach for emotion recognition based on EEG signal using deep learning. Applied Sciences, 12(19), 10028. https://doi.org/10.3390/app121910028
2. Alakus, T. B., Gonen, M., & Turkoglu, I. (2020). Database for an emotion recognition system based on EEG signals and various computer games – GAMEEMO. Biomedical Signal Processing and Control, 60, 101951. https://doi.org/10.1016/j.bspc.2020.101951
3. Aslan, M., Baykara, M., & Alakuş, T. B. (2023). Analysis of brain areas in emotion recognition from EEG signals with deep learning methods. Multimedia Tools and Applications, 83(11), 32423-32452. https://doi.org/10.1007/s11042-023-16696-w
4. Assunção, G., Patrão, B., Castelo-Branco, M., & Menezes, P. (2022). An overview of emotion in artificial intelligence. IEEE Transactions on Artificial Intelligence, 3(6), 867-886. https://doi.org/10.1109/TAI.2022.3159614
5. Bayat, S., & Işık, G. (2022). Aras kuş türlerinin ses özellikleri bakımından derin öğrenme yöntemleriyle tanınması. Journal of the Institute of Science and Technology, 12(3), 1250-1263. https://doi.org/10.21597/jist.1124674
6. Bayat, S., & Işık, G. (2023). Evaluating the effectiveness of different machine learning approaches for sentiment classification. Journal of the Institute of Science and Technology, 13(3), 1496-1510. https://doi.org/10.21597/jist.1292050
7. Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb), 281-305.
8. Bhattacharyya, A., Tripathy, R. K., Garg, L., & Pachori, R. B. (2021). A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sensors Journal, 21(3), 3579-3591. https://doi.org/10.1109/JSEN.2020.3027181

9. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. https://doi.org/10.1038/nrn3475
10. Cheng, J., Chen, M., Li, C., Liu, Y., Song, R., Liu, A., & Chen, X. (2021). Emotion recognition from multi-channel EEG via deep forest. IEEE Journal of Biomedical and Health Informatics, 25(2), 453-464. https://doi.org/10.1109/JBHI.2020.2995767
11. Chicco, D., Warrens, M. J., & Jurman, G. (2021). The Matthews Correlation Coefficient (MCC) is more informative than Cohen’s Kappa and Brier Score in binary classification assessment. IEEE Access, 9, 78368-78381. https://doi.org/10.1109/ACCESS.2021.3084050
12. Coan, J. A., & Allen, J. J. B. (2004). Frontal EEG asymmetry as a moderator and mediator of emotion. Biological Psychology, 67(1), 7-50. https://doi.org/10.1016/j.biopsycho.2004.03.002
13. Davidson, R. J. (1998). Affective style and affective disorders: Perspectives from affective neuroscience. Cognition and Emotion, 12(3), 307-330. https://doi.org/10.1080/026999398379628
14. Davidson, R. J. (2003). Affective neuroscience and psychophysiology: Toward a synthesis. Psychophysiology, 40(5), 655-665. https://doi.org/10.1111/1469-8986.00067
15. Demaree, H. A., Everhart, D. E., Youngstrom, E. A., & Harrison, D. W. (2005). Brain lateralization of emotional processing: Historical roots and a future incorporating “dominance”. Behavioral and Cognitive Neuroscience Reviews, 4(1), 3-20. https://doi.org/10.1177/1534582305276837
16. Fatourechi, M., Mason, S. G., Birch, G. E., & Ward, R. K. (2004). A wavelet-based approach for the extraction of event related potentials from EEG. 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2. https://doi.org/10.1109/ICASSP.2004.1326363
17. Frank, M. J., Woroch, B. S., & Curran, T. (2005). Error-related negativity predicts reinforcement learning and conflict biases. Neuron, 47(4), 495-501. https://doi.org/10.1016/j.neuron.2005.06.020
18. Gkintoni, E., Aroutzidis, A., Antonopoulou, H., & Halkiopoulos, C. (2025). From neural networks to emotional networks: A systematic review of EEG-based emotion recognition in cognitive neuroscience and real-world applications. Brain Sciences, 15(3), 220. https://doi.org/10.3390/brainsci15030220
19. Harmon-Jones, E., Gable, P. A., & Peterson, C. K. (2010). The role of asymmetric frontal cortical activity in emotion-related phenomena: A review and update. Biological Psychology, 84(3), 451-462. https://doi.org/10.1016/j.biopsycho.2009.08.010
20. He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263-1284. https://doi.org/10.1109/TKDE.2008.239
21. Huang, D., Chen, S., Liu, C., Zheng, L., Tian, Z., & Jiang, D. (2021). Differences first in asymmetric brain: A bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition. Neurocomputing, 448, 140-151. https://doi.org/10.1016/j.neucom.2021.03.105
22. Jenke, R., Peer, A., & Buss, M. (2014). Feature extraction and selection for emotion recognition from EEG. IEEE Transactions on Affective Computing, 5(3), 327-339. https://doi.org/10.1109/TAFFC.2014.2339834
23. Jiang, W., Zhang, Y., Han, H., Liu, X., Gwak, J., Gu, W., Shankar, A., & Maple, C. (2025). Fuzzy ensemble-based federated learning for EEG-based emotion recognition in internet of medical things. Journal of Industrial Information Integration, 44, 100789. https://doi.org/10.1016/j.jii.2025.100789
24. Katsigiannis, S., & Ramzan, N. (2018). DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics, 22(1), 98-107. https://doi.org/10.1109/JBHI.2017.2688239
25. Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012). DEAP: A database for emotion analysis using physiological signals. IEEE Transactions on Affective Computing, 3(1), 18-31. https://doi.org/10.1109/T-AFFC.2011.15
26. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. https://doi.org/10.1038/nature14539
27. Li, S., & Luo, H. (2025). EEG-based user identification in gaming: Comparative analysis of emotion simulations. Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence, 412-417. https://doi.org/10.1145/3709026.3709084
28. Li, W., Huan, W., Hou, B., Tian, Y., Zhang, Z., & Song, A. (2022). Can emotion be transferred?—A review on transfer learning for EEG-based emotion recognition. IEEE Transactions on Cognitive and Developmental Systems, 14(3), 833-846. https://doi.org/10.1109/TCDS.2021.3098842
29. Li, Y., Zheng, W., Wang, L., Zong, Y., & Cui, Z. (2022). From regional to global brain: A novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Transactions on Affective Computing, 13(2), 568-578. https://doi.org/10.1109/TAFFC.2019.2922912
30. Liu, D., Dai, W., Zhang, H., Jin, X., Cao, J., & Kong, W. (2023). Brain-machine coupled learning method for facial emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 10703-10717. https://doi.org/10.1109/TPAMI.2023.3257846
31. Liu, H., Lou, T., Zhang, Y., Wu, Y., Xiao, Y., Jensen, C. S., & Zhang, D. (2024). EEG-based multimodal emotion recognition: A machine learning perspective. IEEE Transactions on Instrumentation and Measurement, 73, 1-29. https://doi.org/10.1109/TIM.2024.3369130
32. Liu, Y., Ding, Y., Li, C., Cheng, J., Song, R., Wan, F., & Chen, X. (2020). Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Computers in Biology and Medicine, 123, 103927. https://doi.org/10.1016/j.compbiomed.2020.103927
33. Luck, S. J. (2014). An Introduction to the Event-Related Potential Technique. Bradford Books.
34. MacNamara, A., Joyner, K., & Klawohn, J. (2022). Event-related potential studies of emotion regulation: A review of recent progress and future directions. International Journal of Psychophysiology, 176, 73-88. https://doi.org/10.1016/j.ijpsycho.2022.03.008
35. Maheshwari, D., Ghosh, S. K., Tripathy, R. K., Sharma, M., & Acharya, U. R. (2021). Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals. Computers in Biology and Medicine, 134, 104428. https://doi.org/10.1016/j.compbiomed.2021.104428
36. Peng, Y., Kong, W., Qin, F., Nie, F., Fang, J., Lu, B.-L., & Cichocki, A. (2021). Self-weighted semi-supervised classification for joint EEG-based emotion recognition and affective activation patterns mining. IEEE Transactions on Instrumentation and Measurement, 70, 1-11. https://doi.org/10.1109/TIM.2021.3124056
37. Raufi, B., & Longo, L. (2024). Comparing ANOVA and PowerShap feature selection methods via shapley additive explanations of models of mental workload built with the theta and alpha EEG band ratios. BioMedInformatics, 4(1), 853-876. https://doi.org/10.3390/biomedinformatics4010048
38. Reuderink, B., Mühl, C., & Poel, M. (2013). Valence, arousal and dominance in the EEG during game play. International Journal of Autonomous and Adaptive Communications Systems, 6(1), 45. https://doi.org/10.1504/IJAACS.2013.050691
39. Saganowski, S., Perz, B., Polak, A. G., & Kazienko, P. (2023). Emotion recognition for everyday life using physiological signals from wearables: A systematic literature review. IEEE Transactions on Affective Computing, 14(3), 1876-1897. https://doi.org/10.1109/TAFFC.2022.3176135
40. Sassenrath, C., Sassenberg, K., Ray, D. G., Scheiter, K., & Jarodzka, H. (2014). A motivational determinant of facial emotion recognition: Regulatory focus affects recognition of emotions in faces. PLoS ONE, 9(11), e112383. https://doi.org/10.1371/journal.pone.0112383
41. Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61(2015), 85-117. https://doi.org/10.1016/j.neunet.2014.09.003
42. Sharma, L. D., & Bhattacharyya, A. (2021). A computerized approach for automatic human emotion recognition using sliding mode singular spectrum analysis. IEEE Sensors Journal, 21(23), 26931-26940. https://doi.org/10.1109/JSEN.2021.3120787
43. Tao, W., Li, C., Song, R., Cheng, J., Liu, Y., Wan, F., & Chen, X. (2023). EEG-based emotion recognition via channel-wise attention and self attention. IEEE Transactions on Affective Computing, 14(1), 382-393. https://doi.org/10.1109/TAFFC.2020.3025777
44. Tuncer, T., Dogan, S., & Subasi, A. (2021). A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos, Solitons & Fractals, 144, 110671. https://doi.org/10.1016/j.chaos.2021.110671
45. Vempati, R., & Sharma, L. D. (2023). EEG rhythm based emotion recognition using multivariate decomposition and ensemble machine learning classifier. Journal of Neuroscience Methods, 393, 109879. https://doi.org/10.1016/j.jneumeth.2023.109879
46. Vempati, R., Sharma, L. D., & Tripathy, R. K. (2025). Cross-subject emotion recognition from multichannel EEG signals using multivariate decomposition and ensemble learning. IEEE Transactions on Cognitive and Developmental Systems, 17(1), 77-88. https://doi.org/10.1109/TCDS.2024.3417534
47. Wang, F., Wu, S., Zhang, W., Xu, Z., Zhang, Y., Wu, C., & Coleman, S. (2020). Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia, 146, 107506. https://doi.org/10.1016/j.neuropsychologia.2020.107506
48. Welch, P. (1967). The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms. IEEE Transactions on Audio and Electroacoustics, 15(2), 70-73. https://doi.org/10.1109/TAU.1967.1161901
49. Wioleta, S. (2013). Using physiological signals for emotion recognition. 2013 6th International Conference on Human System Interactions (HSI), 556-561. https://doi.org/10.1109/HSI.2013.6577880
50. Yang, Y., Wu, Q., Qiu, M., Wang, Y., & Chen, X. (2018). Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. 2018 International Joint Conference on Neural Networks (IJCNN), 1-7. https://doi.org/10.1109/IJCNN.2018.8489331
51. Zhang, Z., Fort, J. M., & Giménez Mateu, L. (2024). Mini review: Challenges in EEG emotion recognition. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1289816
52. Zheng, W.-L., & Lu, B.-L. (2015). Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3), 162-175. https://doi.org/10.1109/TAMD.2015.2431497