Scene Classification via Attention-Guided Integration of Visual and Auditory Data Streams

Abstract

This study proposes a novel multi-source deep learning architecture, called the Gated Cross-Modal Fusion Transformer (GCM-FT), designed to more effectively integrate the complementary structure of visual and auditory information sources in scene classification. The proposed framework extracts deep representations from the visual stream using an EfficientNetV2 backbone, while processing the MFCC-based time–frequency features provided within the dataset for the auditory stream. The representation vectors obtained from both streams are dynamically unified through a gated attention mechanism. With its multi-headed loss function, auxiliary stream outputs, and attention-based fusion block, the model is able to learn the contributions of visual and auditory information in a stable and balanced manner. Extensive cross-validation experiments demonstrate that GCM-FT achieves higher accuracy, lower variance, and more consistent class-wise performance compared with single-stream models and existing fused-information approaches. These findings indicate that attention-guided fusion offers a powerful and generalizable information integration strategy for visual–auditory scene classification tasks.

Keywords

• Audio-Visual Scene Classification
• Multimodal Deep Learning
• Gated Attention Fusion

Dikkat Tabanlı Görsel–İşitsel Akış Füzyonu ile Sahne Sınıflandırma

Abstract

Bu çalışmada, görsel ve işitsel bilgi kaynaklarının tamamlayıcı yapısını etkili bir şekilde birleştirerek sahne sınıflandırma performansını artırmayı amaçlayan yeni bir çok akışlı derin öğrenme mimarisi olan Gated Cross-Modal Fusion Transformer (GCM-FT) önerilmektedir. Model, görüntü verilerini EfficientNetV2-B1 tabanlı bir omurga ağ ile işlerken, ses verilerine ait MFCC özelliklerini evrişimsel ve tekrarlayan katmanlardan oluşan bir akışta analiz etmektedir. Her iki akıştan elde edilen özellik temsilleri, dinamik olarak öğrenilen kapılı dikkat mekanizması ile birleştirilmekte ve böylece görsel–işitsel bilgiler arasındaki etkileşim dengeli bir biçimde modellenmektedir. Önerilen mimari, 10 katlı çapraz doğrulama protokolü kullanılarak genişletilmiş Kaggle sürümündeki 17.252 örnekten oluşan dokuz sınıflı Scene Classification – Images and Audio veri kümesi üzerinde değerlendirilmiştir. Deneysel sonuçlar, GCM-FT’nin hem görsel hem işitsel tek akışlı modellere göre daha yüksek doğruluk sağladığını ve %97,99 ± 0,37 ortalama doğruluk ile düşük varyanslı ve kararlı bir performans sergilediğini göstermektedir. Sınıf bazlı analizler, tüm sınıflarda yüksek F1-skorları elde edildiğini ve modelin benzer sahneler arasındaki ayrımı güvenilir biçimde gerçekleştirdiğini ortaya koymaktadır. Elde edilen bulgular, dikkat yönlendirmeli füzyonun sahne sınıflandırma problemlerinde etkili ve genellenebilir bir bilgi entegrasyon stratejisi sunduğunu doğrulamaktadır.

Keywords

• Ses-Görüntü Sahne Sınıflandırma
• Çoklu-Modlu Derin Öğrenme
• Kapılı Dikkat Tabanlı Füzyon

References

1. Çelik Y. Application of deep learning for voice command classification in Turkish language. Bitlis Eren University Journal of Science. 2024;13(3):701–708. doi:10.17798/bitlisfen.1477191.
2. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Communications of the ACM. 2017;60(6):84–90. doi: 10.1145/3065386.
3. Güneş H, Hark C, Akkaya AE. Comparison of Deep Learning Models and Optimization Algorithms in the Detection of Scoliosis and Spondylolisthesis from X-Ray Images. Sakarya University Journal of Science. 2024;28(2):438–451. doi:10.16984/saufenbilder.1246001.
4. Doğan F, Aktaş M, Gürsoy Mİ. Classification of Skin Diseases with Different Deep Learning Models and Comparison of the Performances of the Models. TDFD. 2024;13(3):117–123. doi:10.46810/tdfd.1502471.
5. Ceylan T, İnik Ö. Development of an Effective Deep Learning Model for COVID-19 Detection from CT Images. Tr. J. Nature Sci. 2025;14(1):156–166. doi:10.46810/tdfd.1472034.
6. Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv Preprint. 2013;arXiv:1301.3781.
7. Liu X, Wang H, Li Z, Qin L. Deep learning in ECG diagnosis: A review. Knowledge-Based Systems. 2021;227:107187 doi: 10.1016/j.knosys.2021.107187.
8. İnik Ö. Classification of Scenes in Aerial Images with Deep Learning Models. TDFD. 2023;12(1):37–43. doi:10.46810/tdfd.1225756.

9. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–444. doi: 10.1038/nature14539.
10. Uckan T, Aslan C, Hark C. A Comprehensive Hybrid Approach for Indoor Scene Recognition Combining CNNs and Text-Based Features. Sensors. 2025;25(17):5350. doi: 10.3390/s25175350.
11. Bird JJ, Ekárt A, Faria DR. Look and listen: A multi-modality late fusion approach to scene classification for autonomous machines. In: Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2020. p. 10555–10560. doi: 10.1109/IROS45743.2020.9341557.
12. Wang S, Heittola T, Mesaros A, Virtanen T. Audio-visual scene classification: Analysis of DCASE 2021 challenge submissions. arXiv Preprint. 2021;arXiv:2105.13675. doi: 10.48550/arXiv.2105.13675.
13. Lei H, Chen N. Audio-visual scene classification based on multi-modal graph fusion. In: Proceedings of INTERSPEECH 2022. 2022. p. 4157–4161. doi: 10.21437/interspeech.2022-741.
14. Chen C, Wang M, Zhang P. Audio-visual scene classification using a transfer-learning-based joint optimization strategy. arXiv Preprint. 2022;arXiv:2204.11420. doi: 10.48550/arXiv.2204.11420.
15. Pham L, Ngo D, Nguyen T, Nguyen P, Hoang T, Schindler A. An audio-visual dataset and deep learning frameworks for crowded scene classification. In: Proceedings of the 19th International Conference on Content-Based Multimedia Indexing (CBMI). ACM; 2022. p. 23–28. doi: 10.1145/3549555.3549568.
16. Zhou L, Zhou Y, Qi X, Hu J, Lam TL, Xu Y. Feature pyramid attention network for audio-visual scene classification. CAAI Transactions on Intelligence Technology. 2025;10(2):359–374. doi: 10.1049/cit2.12375.
17. Xu C, Xu L, Zhao S, Yu L, Zhang C. Complementary knowledge-augmented multimodal learning method for yarn quality soft sensing. Engineering Applications of Artificial Intelligence. 2024;133:108057. doi: 10.1016/j.engappai.2024.108057.
18. Liang X, Qian Y, Guo Q, Cheng H, Liang J. AF: An association-based fusion method for multi-modal classification. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021;44(12):9236–9254. doi:10.1109/TPAMI.2021.3125995.
19. Cantarini M, Gabrielli L, Mancini A, Squartini S, Longo R. A3CarScene: An audio-visual dataset for driving scene understanding. Data in Brief. 2023;48:109146. doi:10.1016/j.dib.2023.109146.
20. Sumathi K, Kumar P, Mahadevaswamy HR, Ujwala BS. Optimizing multimodal scene recognition through relevant feature selection approach for scene classification. MethodsX. 2025;14:103226. doi:10.1016/j.mex.2025.103226.
21. Bird JJ. Scene Classification – Images and Audio [Data set]. Kaggle. Available from: https://www.kaggle.com/datasets/birdy654/scene-classification-images-and-audio
22. Woodward K, Kanjo E. Parallel Feature Fusion for Multimodal Scene Recognition on Dual-Core MCUs. IEEE Pervasive Comput. 2025;24(3):54–62. doi: 10.1109/mprv.2025.3555334.