Determining the Reading Comprehension Status of a Foreign Language Text Automatically with Functional Near Infrared Spectroscopy Signals and Sequential Forward Intrinsic Mode Function Selection Approach

Yıl 2024, Cilt: 14 Sayı: 4, 2023 - 2035, 15.12.2024

Ural Akıncıoğlu , Önder Aydemir , Ahmet Çil , Muhammed Baydere

https://doi.org/10.31466/kfbd.1517668

Öz

Nowadays, the level of knowledge of a foreign language is mostly determined by exams. In this study, neural activity signals recorded by functional near-infrared spectroscopy (fNIRS) while participants read a text in English as a foreign language are used to automatically determine the comprehension of the text. In the proposed method, linear interpolation and improved complementary ensemble empirical mode decomposition with adaptive noise (ICEEMDAN) were applied to the signals as preprocessing. With ICEEMDAN, intrinsic mode function (IMF) components were derived. In the feature extraction stage, the symbolic aggregate approximation method and the combination of kurtosis, skewness, and standard deviation statistical features were used. The extracted features were classified by the k-nearest neighbor (k-NN) classifier. The optimal IMF combination was determined by a sequential forward IMF selection approach using the training dataset. The proposed sequential forward IMF selection method-based approach was applied to 16 healthy participants and a classification accuracy of 85,37% was obtained with the k-NN Classifier on the test dataset. The results show that the comprehension of the whole English text can be classified as understood/did not understand with high accuracy through brain signals for which the most efficient IMF combination is determined.

Anahtar Kelimeler

Machine learning, fNIRS, Reading comprehension, Sequential forward IMF selection

Proje Numarası

122E102

Yabancı Dilde Okunan Metnin Anlama Durumunun Fonksiyonel Yakın Kızılötesi Spektroskopisi Sinyalleri ve Ardışıl İleri İçsel Mod Fonksiyonu Seçme Yaklaşımı ile Otomatik Belirlenmesi

Öz

Günümüzde, insanların bir yabancı dili hangi seviyede bildikleri daha çok sınavlarla tespit edilmektedir. Bu çalışmada ise katılımcıların yabancı dil olarak İngilizce dilinde bir metni okurken fonksiyonel yakın kızılötesi spektroskopisi (fYKS) ile kaydedilen nöral aktivite sinyalleri kullanılarak ilgili metni anlayıp anlamadıkları otomatik olarak tespit edilmiştir. Önerilen metotta, ön işleme olarak doğrusal interpolasyon ve uyarlanabilir gürültüyle geliştirilmiş tamamlayıcı topluluk mod ayrıştırması (ICEEMDAN) sinyallere uygulanmıştır. ICEEMDAN ile sinyallerin içsel mod fonksiyon (IMF) bileşenleri türetilmiştir. Öznitelik çıkarma aşamasında ise sembolik toplam yaklaşım metodu ve basıklık, çarpıklık, standart sapma istatistiksel öznitelik kombinasyonu kullanılmıştır. Elde edilen öznitelikler, k-en yakın komşuluk (k-NN) sınıflandırıcısı ile sınıflandırılmıştır. Sınıflandırma sonucunda eğitim veri setinde ardışıl ileri IMF seçme yaklaşımı ile en uygun IMF kombinasyonu belirlenmiştir. Önerilen ardışıl ileri IMF seçim metodu esaslı yaklaşım 16 sağlıklı katılımcı üzerinde uygulanmış ve k-NN sınıflandırıcısı ile test verilerinde %85,37 sınıflandırma doğruluğu elde edilmiştir. Sonuçlar, İngilizce metnin tümünün anlaşılma durumunun en etkin IMF kombinasyonu belirlenen beyin sinyalleri aracılığıyla yüksek doğrulukla anladı/anlamadı olarak sınıflandırılabileceğini göstermektedir.

Anahtar Kelimeler

Makine öğrenmesi, fYKS, Okuduğunu anlama, Ardışıl ileri yönlü IMF seçme

Etik Beyan

Yapılan çalışmada araştırma ve yayın etiğine uyulmuştur.

Destekleyen Kurum

TÜBİTAK

Proje Numarası

122E102

Teşekkür

Gerçekleştirilen çalışma, Türkiye Bilimsel ve Teknik Araştırma Kurumu (TÜBİTAK), 1002 Hızlı Destek Programı kapsamında 122E102 proje numarası ile desteklenmiştir. Ural AKINCIOĞLU, TÜBİTAK 2211-C Yurt İçi Öncelikli Alanlar Doktora Burs Programı bursiyeridir.

Kaynakça

• Carlson, S.E., Seipel, B., McMaster, K. (2014). Development of a new reading comprehension assessment: identifying comprehension differences among readers. Learning and Individual Differences, 32, 40-53. https://doi.org/10.1016/j.lindif.2014.03.003.
• Cheng, S., Hu, Y., Fan J., Wei, Q. (2020). Reading comprehension based on visualization of eye tracking and EEG data. Science China Information Sciences, 63(11). https://doi.org/10.1007/s11432-019-1466-7.
• Colominas, M.A., Schlotthauer, G., Torres, M.E. (2014). Improved complete ensemble emd: a suitable tool for biomedical signal processing. Biomedical Signal Processing and Control, 14, 19–29. https: //doi.org/10.1016/j.bspc.2014.06.009.
• Frank, S.L., Aumeistere, A. (2023). An eye-tracking-with-EEG coregistration corpus of narrative sentences. Lang Resources and Evaluation. https://doi.org/10.1007/s10579-023-09684-x.
• Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K.M., Melis, G., Grefenstette, E. (2017). The narrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics. 6, 317-328. https://doi.org/10.48550/arXiv.1712.07040.
• Lawrence, R.J., Wiggins, I.M., Anderson, C.A., Davies-Thompson, J., Hartley, D.E. (2018). Cortical correlates of speech intelligibility measured using functional near-infrared spectroscopy (fnirs). Hearing Research, 370, 53–64. https://doi.org/10.1016/j.heares.2018.09.005.
• Lin, J., Keogh, E., Lonardi, S., Chiu, B. (2003). A symbolic representation of time series, with implications for streaming algorithms. In Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, 2-11. https://doi.org/10.1145/882082.882086.
• Mathur, P., Mittal, T., Manocha, D. (2021). Dynamic graph modelling of simultaneous EEG and eye-tracking data for reading task identification. IEEE International Conference on Acoustics, Speech and Signal Processing, 1250-1254. https://doi.org/10.1109/ICASSP39728.2021.9414343.
• Midha, S., Maior, H.A., Wilson, M.L., Sharples, S. (2021). Measuring mental workload variations in office work tasks using fnirs. International Journal of Human-Computer Studies, 147, 102580. https//doi.org/10.1016/j.ijhcs.2020.102580.
• Omata, M. ve Tanabe, S. (2016). A regression equation to estimate the degree of understanding in a reading using physiological indexes. Proceedings of the 28th Australian Conference on Computer-Human Interaction, 333-337. https://doi.org/10.1145/3010915.3010968.
• Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. Association for Computational Linguistics, 2383–2392. https://doi.org/ 10.18653/v1/D16-1264.
• Ren, H., Wang, M.Y., He, Y., Du, Z., Zhang, J., Zhang, J., Li, D., Yuan Z. (2019). A novel phase analysis method for examining fNIRS neuroimaging data associated with Chinese/English sight translation. Behavioural Brain Research, 361, 151-158. https://doi.org/10.1016/j.bbr.2018.12.032.
• Richardson, M., Burges, C.J.C., Renshaw, E. (2013). MCTest: a challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 193–203.
• Schneegass, C., Kosch, T., Baumann, A., Rusu, M., Hassib, M., Hussmann, H. (2020). Braincode: electroencephalography-based comprehension detection during reading and listening. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1-13. https://doi.org/10.1145/3313831.3376707.
• Shieh, J. ve Keogh, E. (2008). Isax: indexing and mining terabyte sized time series. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 623–631. https://doi.org/10.1145/1401890.1401966.
• Yuan, Y., Chang, K., Taylor, J.N., Mostow, J. (2014). Toward unobtrusive measurement of reading comprehension using low-cost EEG. Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, 54–58. https://doi.org/10.1145/2567574.2567624.
• Zhang, Y., Duan, L., Duan, M. (2019). A new feature extraction approach using improved symbolic aggregate approximation for machinery intelligent diagnosis. Measurement, 133, 468–478. https://doi.org/10.1016/j.measurement.2018.10.045.