AN OVERVIEW OF MULTIMODAL LEARNING: CONCEPTS, CHALLENGES, APPLICATIONS AND DATASETS

Abstract

Given the multifaceted nature of reality, phenomena can be interpreted not only through singular perspectives but also by bringing together various dimensions. Meaning often emerges from the convergence of diverse perspectives, contexts, and forms of representation. The construction of systems capable of analyzing this multilayered structure requires integrating heterogeneous types of information within a holistic, interactive framework. For this multilayered structure to be processable by artificial intelligence systems, the synthesis of heterogeneous information types from various sources in a holistic structure is mandated. In response to this requirement, multimodal learning is an approach that aims to develop more contextual and generalizable artificial intelligence systems by combining heterogeneous data from different modalities (e.g., text, images, audio, sensor data) within an integrated structure. Based on recent literature, this review examines the conceptual foundations of multimodal learning and its key technical challenges, including representation learning, alignment, fusion, translation, missing modality, and co-learning. This study systematically compares and classifies more than 50 of the most prominent review articles published between 2010 and 2025 in a comprehensive table, summarizing the challenges they address, their application areas, and practical contributions. Attention has been drawn to areas often neglected in the literature, such as co-learning and missing modality, as well as to other critical gaps persisting in the field. Furthermore, the paper presents multimodal applications in healthcare, robotics, autonomous driving, remote sensing, and security, along with common multimodal datasets. By bridging theoretical foundations and real-world applications, this study provides a comprehensive reference for the field of multimodal learning.

Keywords

• Artificial Intelligence
• Multimodal Applications
• Multimodal Challenges
• Multimodal Datasets
• Multimodal Learning.

References

1. P. P. Shinde and S. Shah, “A Review of Machine Learning and Deep Learning Applications,” Proceedings - 2018 4th International Conference on Computing, Communication Control and Automation, ICCUBEA 2018, Jul. 2018, doi: 10.1109/ICCUBEA.2018.8697857.
2. S. Dong, P. Wang, and K. Abbas, “A survey on deep learning and its applications,” Comput. Sci. Rev., vol. 40, p. 100379, May 2021, doi: 10.1016/J.COSREV.2021.100379.
3. S. Sengupta et al., “A review of deep learning with special emphasis on architectures, applications and recent trends,” Knowl. Based. Syst., vol. 194, p. 105596, Apr. 2020, doi: 10.1016/J.KNOSYS.2020.105596.
4. J. Hestness et al., “Deep Learning Scaling is Predictable, Empirically,” Dec. 2017, Accessed: Mar. 06, 2025. [Online]. Available: https://arxiv.org/abs/1712.00409v1
5. D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” IEEE Signal Process. Mag., vol. 34, no. 6, pp. 96–108, Nov. 2017, doi: 10.1109/MSP.2017.2738401.
6. X. Wang, Y. Han, V. C. M. Leung, D. Niyato, X. Yan, and X. Chen, “Convergence of Edge Computing and Deep Learning: A Comprehensive Survey,” IEEE Communications Surveys and Tutorials, vol. 22, no. 2, pp. 869–904, Apr. 2020, doi: 10.1109/COMST.2020.2970550.
7. T. Baltrusaitis, C. Ahuja, and L. P. Morency, “Multimodal Machine Learning: A Survey and Taxonomy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443, Feb. 2019, doi: 10.1109/TPAMI.2018.2798607.
8. S. Jabeen, X. Li, M. S. Amin, O. Bourahla, S. Li, and A. Jabbar, “A Review on Methods and Applications in Multimodal Deep Learning,” ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 19, no. 2s, pp. 1–41, Jun. 2023, doi: 10.1145/3545572.

9. A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” Proc. Mach. Learn. Res., vol. 139, pp. 8748–8763, Feb. 2021, Accessed: Mar. 06, 2025. [Online]. Available: https://arxiv.org/abs/2103.00020v1
10. A. Singh et al., “FLAVA: A Foundational Language and Vision Alignment Model,” CVPR 2022. doi: 10.1109/CVPR52688.2022.01519.
11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis With Latent Diffusion Models,” 2022. doi: 10.48550/ARXIV.2204.11824.
12. H. D. Le, G. S. Lee, S. H. Kim, S. Kim, and H. J. Yang, “Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning,” IEEE Access, vol. 11, pp. 14742–14751, 2023, doi: 10.1109/ACCESS.2023.3244390.
13. P. Tzirakis, J. Chen, S. Zafeiriou, and B. Schuller, “End-to-end multimodal affect recognition in real-world environments,” Information Fusion, vol. 68, pp. 46–53, Apr. 2021, doi: 10.1016/J.INFFUS.2020.10.011.
14. S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp. 98–125, Sep. 2017, doi: 10.1016/J.INFFUS.2017.02.003.
15. M. Abdelaziz, T. Wang, and A. Elazab, “Fusing Multimodal and Anatomical Volumes of Interest Features Using Convolutional Auto-Encoder and Convolutional Neural Networks for Alzheimer’s Disease Diagnosis,” Front. Aging Neurosci., vol. 14, p. 812870, Apr. 2022, doi: 10.3389/FNAGI.2022.812870.
16. Y. Dai, Y. Gao, F. Liu, and J. Fu, “Mutual Attention-based Hybrid Dimensional Network for Multimodal Imaging Computer-aided Diagnosis,” Jan. 2022, Accessed: Mar. 06, 2025. [Online]. Available: https://arxiv.org/abs/2201.09421v1
17. Q. Zhou, S. Ye, M. Wen, Z. Huang, M. Ding, and X. Zhang, “Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer,” Neural Comput. Appl., vol. 34, no. 24, pp. 21741–21761, Dec. 2022, doi: 10.1007/S00521-022-07635-1.
18. J. Zhou et al., “Cohesive Multi-Modality Feature Learning and Fusion for COVID-19 Patient Severity Prediction,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 5, pp. 2535–2549, May 2022, doi: 10.1109/TCSVT.2021.3063952.
19. K. Mahato, T. Saha, S. Ding, S. S. Sandhu, A. Y. Chang, and J. Wang, “Hybrid multimodal wearable sensors for comprehensive health monitoring,” Nature Electronics 2024 7:9, vol. 7, no. 9, pp. 735–750, Sep. 2024, doi: 10.1038/s41928-024-01247-4.
20. D. Feng et al., “Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 3, pp. 1341–1360, Mar. 2021, doi: 10.1109/TITS.2020.2972974.
21. M. Person, M. Jensen, A. O. Smith, and H. Gutierrez, “Multimodal Fusion Object Detection System for Autonomous Vehicles,” Journal of Dynamic Systems, Measurement and Control, Transactions of the ASME, vol. 141, no. 7, Jul. 2019, doi: 10.1115/1.4043222/726548.
22. P. Zhang, X. Li, X. Lin, and L. He, “A New Literature Review of 3D Object Detection on Autonomous Driving,” Journal of Artificial Intelligence Research, vol. 82, pp. 973–1015, Feb. 2025, doi: 10.1613/JAIR.1.15961.
23. Y. Zheng, F. Liu, and H. P. Hsieh, “U-Air: When urban air quality inference meets big data,” Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. Part F128815, pp. 1436–1444, Aug. 2013, doi: 10.1145/2487575.2488188.
24. X. Zhang et al., “Multi-modal spatio-temporal meteorological forecasting with deep neural network,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 188, pp. 380–393, Jun. 2022, doi: 10.1016/J.ISPRSJPRS.2022.03.007.
25. G. M. Dimitri, “A Short Survey on Deep Learning for Multimodal Integration: Applications, Future Perspectives and Challenges,” Computers 2022, Vol. 11, Page 163, vol. 11, no. 11, p. 163, Nov. 2022, doi: 10.3390/COMPUTERS11110163.
26. M. A. Manzoor, S. Albarri, Z. Xian, Z. Meng, p. Nakov, and S. Liang, “Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 20, no. 3, Feb. 2023, doi: 10.1145/3617833.
27. E. Warner et al., “Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects,” Int. J. Comput. Vis., vol. 132, no. 9, pp. 3753–3769, Sep. 2024, doi: 10.1007/S11263-024-02032-8.
28. T. Jiao, C. Guo, X. Feng, Y. Chen, and J. Song, “A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications,” Computers, Materials & Continua, vol. 80, no. 1, pp. 1–35, Jul. 2024, doi: 10.32604/CMC.2024.053204.
29. p. p. Liang, A. Zadeh, and L. p. Morency, “Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions,” ACM Comput. Surv., vol. 56, no. 10, Sep. 2022, doi: 10.1145/3656580.
30. R. Wu, H. Wang, H.-T. Chen, and G. Carneiro, “Deep Multimodal Learning with Missing Modality: A Survey,” Sep. 2024, Accessed: Mar. 07, 2025. [Online]. Available: https://arxiv.org/abs/2409.07825v3
31. A. Rahate, R. Walambe, S. Ramanna, and K. Kotecha, “Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directions,” Information Fusion, vol. 81, pp. 203–239, May 2022, doi: 10.1016/J.INFFUS.2021.12.003.
32. A. Waqas et al., “SELF-NORMALIZING FOUNDATION MODEL FOR ENHANCED MULTI-OMICS DATA ANALYSIS IN ONCOLOGY,” 2024.
33. A. Waqas, A. Tripathi, A. Mukund, p. Stewart, M. Naeini, and G. Rasool, “BIO24-031: Hierarchical Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes,” Journal of the National Comprehensive Cancer Network, vol. 22, no. 2.5, Apr. 2024, doi: 10.6004/JNCCN.2023.7137.
34. Z. J. Cao and G. Gao, “Multi-omics single-cell data integration and regulatory inference with graph-linked embedding,” Nature Biotechnology 2022 40:10, vol. 40, no. 10, pp. 1458–1466, May 2022, doi: 10.1038/s41587-022-01284-4.
35. B. Salehi et al., “Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network,” IEEE Trans. Veh. Technol., vol. 71, no. 7, pp. 7639–7655, Jul. 2022, doi: 10.1109/TVT.2022.3170733.
36. M.具jelic et al., “Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 11679–11689, 2020, doi: 10.1109/CVPR42600.2020.01170.
37. p. Hu et al., “T-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous Driving,” IEEE Trans. Mob. Comput., 2024, doi: 10.1109/TMC.2024.3462437.
38. L. Zheng et al., “RCFusion: Fusing 4-D Radar and Camera with Bird’s-Eye View Features for 3-D Object Detection,” IEEE Trans. Instrum. Meas., vol. 72, 2023, doi: 10.1109/TIM.2023.3280525.
39. D. Khan, M. Baek, M. Y. Kim, and D. Seog Han, “Multimodal Object Detection and Ranging Based on Camera and Lidar Sensor Fusion for Autonomous Driving,” APCC 2022 - 27th Asia-Pacific Conference on Communications: Creating Innovative Communication Technologies for Post-Pandemic Era, pp. 342–343, 2022, doi: 10.1109/APCC55198.2022.9943618.
40. X. Yun, Y. Sun, X. Yang, and N. Lu, “Discriminative Fusion Correlation Learning for Visible and Infrared Tracking,” Math. Probl. Eng., vol. 2019, no. 1, p. 2437521, Jan. 2019, doi: 10.1155/2019/2437521.
41. S. Meng and Y. Liu, “Multimodal Feature Fusion YOLOv5 for RGB-T Object Detection,” Proceedings - 2022 Chinese Automation Congress, CAC 2022, vol. 2022-January, pp. 2333–2338, 2022, doi: 10.1109/CAC57257.2022.10055263.
42. M. Ehatisham-Ul-Haq et al., “Robust Human Activity Recognition Using Multimodal Feature-Level Fusion,” IEEE Access, vol. 7, pp. 60736–60751, 2019, doi: 10.1109/ACCESS.2019.2913393.
43. J. Li, J. Li, X. Wang, X. Zhan, and Z. Zeng, “A Domain Generalization and Residual Network Based Emotion Recognition from Physiological Signals,” Cyborg and Bionic Systems, Feb. 2023, doi: 10.34133/CBSYSTEMS.0074.
44. Q. Li, Y. Liu, F. Yan, Q. Zhang, and C. Liu, “Emotion recognition based on multiple physiological signals,” Biomed. Signal Process. Control, vol. 85, p. 104989, Aug. 2023, doi: 10.1016/J.BSPC.2023.104989.
45. H. Liu, T. Fang, T. Zhou, and L. Wang, “Towards Robust Human-Robot Collaborative Manufacturing: Multimodal Fusion,” IEEE Access, vol. 6, pp. 74762–74771, 2018, doi: 10.1109/ACCESS.2018.2884793.
46. W. Wang, R. Li, Y. Chen, Y. Sun, and Y. Jia, “Predicting Human Intentions in Human-Robot Hand-Over Tasks Through Multimodal Learning,” IEEE Transactions on Automation Science and Engineering, vol. 19, no. 3, pp. 2339–2353, Jul. 2022, doi: 10.1109/TASE.2021.3074873.
47. C. E. Lee, J. Baek, J. Son, and Y. G. Ha, “Deep AI military staff: cooperative battlefield situation awareness for commander’s decision making,” Journal of Supercomputing, vol. 79, no. 6, pp. 6040–6069, Apr. 2023, doi: 10.1007/S11227-022-04882-W.
48. W. Aero et al., “A review on multimodal zero-shot learning,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 13, no. 2, p. e1488, Mar. 2023, doi: 10.1002/WIDM.1488.
49. Z. Lin, S. Yu, Z. Kuang, D. Pathak, and D. Ramanan, “Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models,” pp. 19325–19337, Aug. 2023, doi: 10.1109/CVPR52729.2023.01852.
50. L. A. Kumar, D. K. Renuka, S. L. Rose, and M. C. Shunmugapriya, “Attention based Multi Modal Learning for Audio Visual Speech Recognition,” AIST 2022 - 4th International Conference on Artificial Intelligence and Speech Technology, 2022, doi: 10.1109/AIST55798.2022.10065019.
51. Q. Song, B. Sun, and S. Li, “Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 12, pp. 10028–10038, Dec. 2023, doi: 10.1109/TNNLS.2022.3163771.
52. W. Guo, J. Wang, and S. Wang, “Deep Multimodal Representation Learning: A Survey,” IEEE Access, vol. 7, pp. 63373–63394, 2019, doi: 10.1109/ACCESS.2019.2916887.
53. C. Zhang, Z. Yang, X. He, and L. Deng, “Multimodal Intelligence: Representation Learning, Information Fusion, and Applications,” IEEE Journal on Selected Topics in Signal Processing, vol. 14, no. 3, pp. 478–493, Mar. 2020, doi: 10.1109/JSTSP.2020.2987728.
54. S. F. Zhang, J. H. Zhai, B. J. Xie, Y. Zhan, and X. Wang, “Multimodal Representation Learning: Advances, Trends and Challenges,” Proc. Int. Conf. Mach. Learn. Cybern., vol. 2019-July, Jul. 2019, doi: 10.1109/ICMLC48188.2019.8949228.
55. A. Gaonkar, Y. Chukkapalli, p. J. Raman, S. Srikanth, and S. Gurugopinath, “A Comprehensive Survey on Multimodal Data Representation and Information Fusion Algorithms,” 2021 International Conference on Intelligent Technologies, CONIT 2021, Jun. 2021, doi: 10.1109/CONIT51480.2021.9498415.
56. K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa, “A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,” The Visual Computer 2021 38:8, vol. 38, no. 8, pp. 2939–2970, Jun. 2021, doi: 10.1007/S00371-021-02166-7.
57. p. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, “Multimodal fusion for multimedia analysis: A survey,” Multimed. Syst., vol. 16, no. 6, pp. 345–379, Nov. 2010, doi: 10.1007/S00530-010-0182-0.
58. D. Lahat, T. Adali, and C. Jutten, “Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects,” Proceedings of the IEEE, vol. 103, no. 9, pp. 1449–1477, Sep. 2015, doi: 10.1109/JPROC.2015.2460697.
59. Y. Li, M. Yang, and Z. Zhang, “A Survey of Multi-View Representation Learning,” IEEE Trans. Knowl. Data Eng., vol. 31, no. 10, pp. 1863–1883, Oct. 2019, doi: 10.1109/TKDE.2018.2872063.
60. J. Gao et al., “A Survey on Deep Learning for Multimodal Data Fusion,” Neural Comput., vol. 32, no. 5, pp. 829–864, May 2020, doi: 10.1162/NECO_A_01273.
61. Y. Zhang, D. Sidibé, O. Morel, and F. Mériaudeau, “Deep multimodal fusion for semantic image segmentation: A survey,” Image Vis. Comput., vol. 105, p. 104042, Jan. 2021, doi: 10.1016/J.IMAVIS.2020.104042.
62. G. Joshi, R. Walambe, and K. Kotecha, “A Review on Explainability in Multimodal Deep Neural Nets,” IEEE Access, vol. 9, pp. 59800–59821, 2021, doi: 10.1109/ACCESS.2021.3070212.
63. A. Mogadala, M. Kalimuthu, and D. Klakow, “Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods,” Journal of Artificial Intelligence Research, vol. 71, pp. 1183–1317, Aug. 2021, doi: 10.1613/JAIR.1.11688.
64. W. C. Sleeman, R. Kapoor, and p. Ghosh, “Multimodal Classification: Current Landscape, Taxonomy and Future Directions,” ACM Comput. Surv., vol. 55, no. 7, Jul. 2022, doi: 10.1145/3543848.
65. J. Li et al., “Deep learning in multimodal remote sensing data fusion: A comprehensive review,” International Journal of Applied Earth Observation and Geoinformation, vol. 112, p. 102926, Aug. 2022, doi: 10.1016/J.JAG.2022.102926.
66. Y. Ma, S. Zhao, W. Wang, Y. Li, and I. King, “Multimodality in meta-learning: A comprehensive survey,” Knowl. Based. Syst., vol. 250, p. 108976, Aug. 2022, doi: 10.1016/J.KNOSYS.2022.108976.
67. W. Chai and G. Wang, “Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend,” Applied Sciences 2022, Vol. 12, Page 6588, vol. 12, no. 13, p. 6588, Jun. 2022, doi: 10.3390/APP12136588.
68. S. Uppal et al., “Multimodal research in vision and language: A review of current and emerging trends,” Information Fusion, vol. 77, pp. 149–171, Jan. 2022, doi: 10.1016/J.INFFUS.2021.07.009.
69. Y. Zhu, Y. Wu, N. Sebe, and Y. Yan, “Vision+X: A Survey on Multimodal Learning in the Light of Data,” IEEE Trans. Pattern Anal. Mach. Intell., Oct. 2022, doi: 10.1109/TPAMI.2024.3420239.
70. E. Erdem et al., “Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning,” Journal of Artificial Intelligence Research, vol. 73, pp. 1131–1207, Apr. 2022, doi: 10.1613/JAIR.1.12918.
71. Y. Ektefaie, G. Dasoulas, A. Noori, M. Farhat, and M. Zitnik, “Multimodal learning with graphs,” Nat. Mach. Intell., vol. 5, no. 4, pp. 340–350, Sep. 2022, doi: 10.1038/s42256-023-00624-6.
72. X. Wang et al., “Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey,” Machine Intelligence Research, vol. 20, no. 4, pp. 447–482, Aug. 2023, doi: 10.1007/S11633-022-1410-8.
73. Q. Tang, J. Liang, and F. Zhu, “A comparative review on multi-modal sensors fusion based on deep learning,” Signal Processing, vol. 213, p. 109165, Dec. 2023, doi: 10.1016/J.SIGPRO.2023.109165.
74. J. Peng, X. Hu, W. Huang, and J. Yang, “What Is a Multi-Modal Knowledge Graph: A Survey,” Big Data Research, vol. 32, p. 100380, May 2023, doi: 10.1016/J.BDR.2023.100380.
75. A. Barua, M. U. Ahmed, and S. Begum, “A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions,” IEEE Access, vol. 11, pp. 14804–14831, 2023, doi: 10.1109/ACCESS.2023.3243854.
76. Y. M. Lin, Y. Gao, M. G. Gong, S. J. Zhang, Y. Q. Zhang, and Z. Y. Li, “Federated Learning on Multimodal Data: A Comprehensive Survey,” Machine Intelligence Research, vol. 20, no. 4, pp. 539–553, Aug. 2023, doi: 10.1007/S11633-022-1398-0.
77. S. Tabakhi, M. Naimul, I. Suvon, p. Ahadian, and H. Lu, “Multimodal Learning for Multi-omics: A Survey,” vol. 01, Feb. 2023, doi: 10.1142/S2811032322500047.
78. p. Xu, X. Zhu, and D. A. Clifton, “Multimodal Learning With Transformers: A Survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 10, pp. 12113–12132, Oct. 2023, doi: 10.1109/TPAMI.2023.3275156.
79. C. Akkus et al., “Multimodal Deep Learning,” Jan. 2023, Accessed: Mar. 17, 2025. [Online]. Available: https://arxiv.org/abs/2301.04856v1
80. K. Zhou, F. H. Hassan, and G. K. Hoon, “The State of the Art for Cross-Modal Retrieval: A Survey,” IEEE Access, vol. 11, pp. 138568–138589, 2023, doi: 10.1109/ACCESS.2023.3338548.
81. N. Rodis, C. Sardianos, G. Th. Papadopoulos, p. Radoglou-Grammatikis, p. Sarigiannidis, and I. Varlamis, “Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions,” Sep. 2024, doi: 10.1109/ACCESS.2024.3467062.
82. Z. Chen et al., “Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models,” Computers, Materials & Continua, vol. 80, no. 2, pp. 1753–1808, Aug. 2024, doi: 10.32604/CMC.2024.052618.
83. S. Raya, M. Orabi, I. Afyouni, and Z. Al Aghbari, “Multi-modal data clustering using deep learning: A systematic review,” Neurocomputing, vol. 607, p. 128348, Nov. 2024, doi: 10.1016/J.NEUCOM.2024.128348.
84. D. Yu et al., “Recent Advances of Multimodal Continual Learning: A Comprehensive Survey,” Oct. 2024, Accessed: Apr. 14, 2025. [Online]. Available: https://arxiv.org/abs/2410.05352v2
85. K. Bayoudh, “A survey of multimodal hybrid deep learning for computer vision: Architectures, applications, trends, and challenges,” Information Fusion, vol. 105, p. 102217, May 2024, doi: 10.1016/J.INFFUS.2023.102217.
86. M. Xu et al., “A Survey of Resource-efficient LLM and Multimodal Foundation Models,” Jan. 2024, Accessed: Apr. 14, 2025. [Online]. Available: https://arxiv.org/abs/2401.08092v2
87. A. Waqas, A. Tripathi, R. p. Ramachandran, p. A. Stewart, and G. Rasool, “Multimodal data integration for oncology in the era of deep neural networks: a review,” Front. Artif. Intell., vol. 7, p. 1408843, Jul. 2024, doi: 10.3389/FRAI.2024.1408843.
88. S. Li and H. Tang, “Multimodal Alignment and Fusion: A Survey,” Nov. 2024, Accessed: Nov. 21, 2025. [Online]. Available: https://arxiv.org/pdf/2411.17040
89. T. Li et al., “Deep learning in multi-modal breast cancer data fusion: a literature review,” Quant. Imaging Med. Surg., vol. 15, no. 11, pp. 11578–11610, Nov. 2025, doi: 10.21037/QIMS-2024-2903.
90. Y. Zhan, R. Yang, J. You, M. Huang, W. Liu, and X. Liu, “A systematic literature review on incomplete multimodal learning: techniques and challenges,” Systems Science and Control Engineering, vol. 13, no. 1, p. 2467083, Dec. 2025, doi: 10.1080/21642583.2025.2467083.
91. N. M. Nguyen, T. T. Nguyen, p.-N. Tran, C. p. Lim, N. T. Pham, and D. N. M. Dang, “Multimodal fusion in speech emotion recognition: A comprehensive review of methods and technologies,” Eng. Appl. Artif. Intell., vol. 163, p. 112624, Jan. 2026, doi: 10.1016/J.ENGAPPAI.2025.112624.
92. J. An, J. Lee, J. Lee, and Y. Son, “Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques,” Jun. 2025, Accessed: Nov. 21, 2025. [Online]. Available: https://arxiv.org/pdf/2506.04788
93. X. Liu, H. Liu, X. Liu, and H. Liu, “Symmetry-Aware Advances in Multimodal Large Language Models: Architectures, Training, and Evaluation,” Symmetry 2025, Vol. 17, vol. 17, no. 9, p. 1400, Aug. 2025, doi: 10.3390/SYM17091400.
94. Y. Zong, O. Mac Aodha, and T. M. Hospedales, “Self-Supervised Multimodal Learning: A Survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 7, pp. 5299–5318, 2025, doi: 10.1109/TPAMI.2024.3429301.
95. Y. Bengio, A. Courville, and p. Vincent, “Representation learning: A review and new perspectives,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798–1828, 2013, doi: 10.1109/TPAMI.2013.50.
96. L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales, “Self-Supervised Representation Learning: Introduction, advances, and challenges,” IEEE Signal Process. Mag., vol. 39, no. 3, pp. 42–62, May 2022, doi: 10.1109/MSP.2021.3134634.
97. K. Liu, Y. Li, N. Xu, and p. Natarajan, “Learn to Combine Modalities in Multimodal Deep Learning,” May 2018, Accessed: May 04, 2025. [Online]. Available: https://arxiv.org/pdf/1805.11730
98. A. Jain, K. Nandakumar, and A. Ross, “Score normalization in multimodal biometric systems,” Pattern Recognit., vol. 38, no. 12, pp. 2270–2285, Dec. 2005, doi: 10.1016/J.PATCOG.2005.01.012.
99. G. K. Verma and U. S. Tiwary, “Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals,” Neuroimage, vol. 102, no. p1, pp. 162–172, Nov. 2014, doi: 10.1016/J.NEUROIMAGE.2013.11.007.
100. Z. Liu, Y. Shen, V. B. Lakshminarasimhan, p. p. Liang, A. Zadeh, and L. p. Morency, “Efficient Low-rank Multimodal Fusion with Modality-Specific Factors,” ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), vol. 1, pp. 2247–2256, May 2018, doi: 10.18653/v1/p18-1209.
101. E. Morvant, A. Habrard, and S. Ayache, “Majority Vote of Diverse Classifiers for Late Fusion,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8621 LNCS, pp. 153–162, 2014, doi: 10.1007/978-3-662-44415-3_16.
102. S. Vora, A. H. Lang, B. Helou, and O. Beijbom, “PointPainting: Sequential Fusion for 3D Object Detection,” CVPR 2020.
103. A. Geiger, p. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, Sep. 2013, doi: 10.1177/0278364913491297.
104. H. Caesar et al., “nuScenes: A Multimodal Dataset for Autonomous Driving,” CVPR 2020.
105. J. Schlosser, C. K. Chow, and Z. Kira, “Fusing LIDAR and images for pedestrian detection using convolutional neural networks,” Proc. IEEE Int. Conf. Robot. Autom., vol. 2016-June, pp. 2198–2205, Jun. 2016, doi: 10.1109/ICRA.2016.7487370.
106. J.-J. Hwang et al., “EMMA: End-to-End Multimodal Model for Autonomous Driving,” Oct. 2024, Accessed: May 16, 2025. [Online]. Available: https://arxiv.org/pdf/2410.23262
107. B. Yang, R. Guo, M. Liang, S. Casas, and R. Urtasun, “RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12363 LNCS, pp. 496–512, 2020, doi: 10.1007/978-3-030-58523-5_29.
108. H. Cho, Y. W. Seo, B. V. K. V. Kumar, and R. R. Rajkumar, “A multi-sensor fusion system for moving object detection and tracking in urban driving environments,” Proc. IEEE Int. Conf. Robot. Autom., pp. 1836–1843, Sep. 2014, doi: 10.1109/ICRA.2014.6907100.
109. K. Chen et al., “WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecasting,” Proc. IEEE Int. Conf. Robot. Autom., pp. 4766–4773, 2024, doi: 10.1109/ICRA57147.2024.10610651.
110. D. Driess et al., “PaLM-E: An Embodied Multimodal Language Model,” Proc. Mach. Learn. Res., vol. 202, pp. 8469–8488, Mar. 2023, Accessed: May 17, 2025. [Online]. Available: https://arxiv.org/pdf/2303.03378
111. A. Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” Proc. Mach. Learn. Res., vol. 229, Jul. 2023, Accessed: May 17, 2025. [Online]. Available: https://arxiv.org/pdf/2307.15818
112. G. R. Team et al., “Gemini Robotics: Bringing AI into the Physical World,” Mar. 2025, Accessed: May 17, 2025. [Online]. Available: https://arxiv.org/pdf/2503.20020
113. A. Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” Apr. 2022, Accessed: May 17, 2025. [Online]. Available: https://arxiv.org/pdf/2204.02311
114. K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2019-June, pp. 3190–3199, May 2019, doi: 10.1109/CVPR.2019.00331.
115. N. Bien et al., “Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet,” PLoS Med., vol. 15, no. 11, p. e1002699, Nov. 2018, doi: 10.1371/JOURNAL.PMED.1002699.
116. S. G. Armato et al., “PROSTATEx Challenges for computerized classification of prostate lesions from multiparametric magnetic resonance images,” vol. 5, no. 4, p. 044501, Nov. 2018, doi: 10.1117/1.JMI.5.4.044501.
117. S. Kumar, S. Sharma, and K. T. Megra, “Transformer enabled multi-modal medical diagnosis for tuberculosis classification,” J. Big Data, vol. 12, no. 1, pp. 1–21, Dec. 2025, doi: 10.1186/S40537-024-01054-W.
118. M. Alrawis, S. Al-Ahmadi, and F. Mohammad, “Bridging Modalities: A Multimodal Machine Learning Approach for Parkinson’s Disease Diagnosis Using EEG and MRI Data,” Applied Sciences 2024, Vol. 14, Page 3883, vol. 14, no. 9, p. 3883, May 2024, doi: 10.3390/APP14093883.
119. G. Ke et al., “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” Adv. Neural Inf. Process. Syst., vol. 30, 2017, Accessed: May 18, 2025. [Online]. Available: https://github.com/Microsoft/LightGBM.
120. “ADNI | Alzheimer’s Disease Neuroimaging Initiative.” Accessed: May 18, 2025. [Online]. Available: https://adni.loni.usc.edu/
121. S. Cui, A. Ma, Y. Wan, Y. Zhong, B. Luo, and M. Xu, “Cross-Modality Image Matching Network with Modality-Invariant Feature Representation for Airborne-Ground Thermal Infrared and Visible Datasets,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, doi: 10.1109/TGRS.2021.3099506.
122. S. Dong, L. Wang, B. Du, and X. Meng, “ChangeCLIP: Remote sensing change detection with multimodal vision-language representation learning,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 208, pp. 53–69, Feb. 2024, doi: 10.1016/J.ISPRSJPRS.2024.01.004.
123. W. G. C. Bandara and V. M. Patel, “HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2022-June, pp. 1757–1767, Mar. 2022, doi: 10.1109/CVPR52688.2022.00181.
124. J. Huang, X. Zhang, Q. Xin, Y. Sun, and p. Zhang, “Automatic building extraction from high-resolution aerial images and LiDAR data using gated residual refinement network,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 151, pp. 91–105, May 2019, doi: 10.1016/J.ISPRSJPRS.2019.02.019.
125. Y. Wang and X. Yang, “Design and implementation of a distributed security threat detection system integrating federated learning and multimodal LLM,” Feb. 2025, Accessed: May 17, 2025. [Online]. Available: https://arxiv.org/pdf/2502.17763
126. N. Jaafar and Z. Lachiri, “Multimodal fusion methods with deep neural networks and meta-information for aggression detection in surveillance,” Expert Syst. Appl., vol. 211, p. 118523, Jan. 2023, doi: 10.1016/J.ESWA.2022.118523.
127. S. J. I. Ismail, Hendrawan, B. Rahardjo, T. Juhana, and Y. Musashi, “MIDALF—multimodal image and audio late fusion for malware detection,” EURASIP J. Inf. Secur., vol. 2025, no. 1, pp. 1–20, Dec. 2025, doi: 10.1186/S13635-025-00188-5.
128. p. Sun et al., “Scalability in Perception for Autonomous Driving: Waymo Open Dataset,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2443–2451, Dec. 2019, doi: 10.1109/CVPR42600.2020.00252.
129. K. T. Yu, M. Bauza, N. Fazeli, and A. Rodriguez, “More than a Million Ways to Be Pushed: A High-Fidelity Experimental Dataset of Planar Pushing,” IEEE International Conference on Intelligent Robots and Systems, vol. 2016-November, pp. 30–37, Apr. 2016, doi: 10.1109/IROS.2016.7758091.
130. M. A. Lee et al., “Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks,” Proc. IEEE Int. Conf. Robot. Autom., vol. 2019-May, pp. 8943–8950, Oct. 2018, doi: 10.1109/ICRA.2019.8793485.
131. M. Schmitt, L. H. Hughes, C. Qiu, and X. X. Zhu, “SEN12MS -- A CURATED DATASET of GEOREFERENCED MULTI-SPECTRAL SENTINEL-1/2 IMAGERY for DEEP LEARNING and DATA FUSION,” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 4, no. 2/W7, pp. 153–160, Sep. 2019, doi: 10.5194/ISPRS-ANNALS-IV-2-W7-153-2019.
132. H. H. Aumann et al., “AIRS/AMSU/HSB on the aqua mission: Design, science objectives, data products, and processing systems,” IEEE Transactions on Geoscience and Remote Sensing, vol. 41, no. 2 PART 1, pp. 253–263, Jul. 2003, doi: 10.1109/TGRS.2002.808356.
133. C. Wang, W. Lu, X. Li, J. Yang, and L. Luo, “M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for Optical-SAR Fusion Object Detection,” May 2025, Accessed: May 19, 2025. [Online]. Available: https://arxiv.org/pdf/2505.10931
134. J. Carreira, E. Noland, C. Hillier, and A. Zisserman, “A Short Note on the Kinetics-700 Human Action Dataset,” Jul. 2019, Accessed: May 18, 2025. [Online]. Available: https://arxiv.org/pdf/1907.06987
135. S. Zhang et al., “CASIA-SURF: A Large-Scale Multi-Modal Benchmark for Face Anti-Spoofing,” IEEE Trans. Biom. Behav. Identity Sci., vol. 2, no. 2, pp. 182–193, Apr. 2020, doi: 10.1109/TBIOM.2020.2973001.
136. Y. Majib, M. Alosaimi, and M. S. Islam, “Multimodal Deep Learning for Brain Tumor Segmentation: A Survey,” IEEE Access, vol. 11, pp. 12345–12365, 2023.