Multimodal Fusion of Electrocardiogram Signals and Demographic Characteristics with Cross-Attention Networks in the Classification of Cardiovascular Diseases

Öz

Cardiovascular diseases are the leading cause of mortality worldwide. In diagnosis, low-cost and non-invasive electrocardiography (ECG) is the primary tool. Since manual examination imposes a high workload on experts, numerous deep learning approaches have been proposed in the literature. In this study, the simultaneous classification of five disease classes (NORM, MI, STTC, CD, HYP) defined by SCP codes in the PTB-XL database was targeted. The proposed architecture consists of three components. In the first component, a second-order polynomial cross-feature transformation was applied to the variables of sex, age, height, and weight. The data were projected into a dmodel-dimensional space using a neural network with a GELU activation function. In the second component, 12-channel ECG recordings were decomposed into multi-scale representations through parallel convolution (1D-CNN), and temporal dependencies were modeled using xLSTM (mLSTM) layers. In the third component, predictions were generated using class-specific context vectors obtained by fusing clinical data as the “query” and ECG data as the “key/value” through cross-attention. In training, which prevented information leakage, the BCEWithLogitsLoss loss function compatible with multi label classification was used. The learning rate and AdamW weight decay were optimized through a 12-cell grid search, ablation studies were conducted, and the model was compared with ResNet and InceptionTime. The 0.9106 macro AUROC and 0.7151 macro F1 scores obtained within a 95% confidence interval across five independent seeds demonstrate the performance of the method. Heatmaps generated by overlaying attention weights onto the ECG signal provide temporal interpretability of the decision mechanism. 

Anahtar Kelimeler

• Deep Learning
• ECG
• Cardiovascular Disease
• Multimodal Fusion
• Cross-Attention
• 1D-CNN
• xLSTM
• Temporal Interpretability

Etik Beyan

The data used in this study were obtained from the open-access and publicly available PTB-XL electrocardiography database. As the data are fully anonymized and do not involve any direct clinical intervention on human subjects or primary data collection, Ethics Committee approval and Informed Consent were not required for this study.

Teşekkür

The authors would like to thank the creators of the PTB-XL electrocardiography database and the PhysioNet platform for their contribution to scientific research by providing open access to this valuable dataset.

Kardiyovasküler Hastalıkların Sınıflandırılmasında Elektrokardiyogram Sinyallerinin ve Demografik Özelliklerin Çapraz Dikkat Ağları ile Çok Modlu Birleştirilmesi

Öz

Kardiyovasküler hastalıklar dünya çapında mortalitenin başlıca nedenidir. Tanıda düşük maliyetli ve invaziv olmayan elektrokardiyografi (EKG) birincil araçtır. Manuel inceleme uzmanlara yüksek iş yükü getirdiğinden literatürde çok sayıda derin öğrenme yaklaşımı bulunmaktadır. Bu çalışmada, PTB-XL veri tabanındaki SCP kodlarıyla tanımlanan beş hastalık sınıfının (NORM, MI, STTC, CD, HYP) eş zamanlı sınıflandırılması hedeflenmiştir. Önerilen mimari üç bileşenlidir. İlk bileşende cinsiyet, yaş, boy ve kilo değişkenlerine ikinci dereceden polinom çapraz öznitelik dönüşümü uygulanmıştır. Veriler GELU fonksiyonlu sinir ağıyla dmodel boyutlu uzaya aktarılmıştır. İkinci bileşende 12 kanallı EKG kayıtları paralel evrişimle (1D-CNN) çok ölçekli temsillere ayrılmış, zamansal bağımlılıklar xLSTM (mLSTM) katmanlarıyla modellenmiştir. Üçüncü bileşende klinik veriler 'sorgu', EKG verileri 'anahtar/değer' olarak çapraz dikkatle harmanlanmış sınıfa özgü bağlam vektörleriyle tahmin üretilmiştir. Bilgi sızıntısını önleyen eğitimde çoklu etiketle uyumlu BCEWithLogitsLoss kaybı kullanılmıştır. Öğrenme oranı ve AdamW ağırlık bozumu 12 hücreli ızgara taramasıyla optimize edilmiş, ablasyon çalışmaları yürütülmüş, model ResNet ve InceptionTime ile kıyaslanmıştır. Beş bağımsız tohumda %95 güven aralığında elde edilen 0.9106 makro AUROC ve 0.7151 makro F1 skorları yöntemin başarımını ortaya koymaktadır. Dikkat ağırlıkları EKG sinyaline bindirilerek üretilen ısı haritaları karar mekanizmasına zamansal yorumlanabilirlik kazandırmaktadır.

Anahtar Kelimeler

• Derin Öğrenme
• EKG
• Kardiyovasküler Hastalık
• Çok Modlu Füzyon
• Çapraz Dikkat
• 1D-CNN
• xLSTM
• Zamansal Yorumlanabilirlik

Etik Beyan

Bu çalışmada kullanılan veriler, açık erişimli ve kamuya açık olan PTB-XL elektrokardiyografi veri tabanından elde edilmiştir. Veriler tamamen anonimize edilmiş olup, doğrudan insan denekler üzerinde klinik bir müdahale veya primer veri toplama işlemi içermediğinden, bu çalışma için Etik Kurul onayına ve Bilgilendirilmiş Gönüllü Olur (Informed Consent) alınmasına gerek duyulmamıştır.

Teşekkür

Yazarlar, PTB-XL elektrokardiyografi veri tabanını derleyen araştırmacılara ve bu verileri açık erişime sunan PhysioNet platformuna bilimsel araştırmalara sağladıkları katkılardan dolayı teşekkür eder.

Kaynakça

1. [1] World Health Organization, "Cardiovascular diseases (CVDs)," 2015. [Online]. Available: http://www.who.int/cardiovascular_diseases/about_cvd/en/. [Accessed: May 11, 2026].
2. [2] P. Kligfield et al., "Recommendations for the standardization and interpretation of the electrocardiogram: Part I: The electrocardiogram and its technology: A scientific statement from the American Heart Association Electrocardiography and Arrhythmias Committee, Council on Clinical Cardiology," Circulation, vol. 115, no. 10, pp. 1306–1324, Mar. 2007.
3. [3] A. Lyon, A. Mincholé, J. P. Martínez, P. Laguna, and B. Rodriguez, "Computational techniques for ECG analysis and interpretation in light of their contribution to medical advances," J. R. Soc. Interface, vol. 15, no. 138, 2018.
4. [4] P. Wagner, N. Strodthoff, R.-D. Bousseljot, et al., "PTB-XL, a large publicly available electrocardiography dataset," Sci. Data, vol. 7, no. 154, pp. 1-15, May 2020, doi: 10.1038/s41597-020-0495-6.
5. [5] J. Pan and W. J. Tompkins, "A real-time QRS detection algorithm," IEEE Transactions on Biomedical Engineering, vol. 32, no. 3, pp. 230-236, 1985.
6. [6] R. J. Martis et al., "Application of higher order cumulant features for cardiac health diagnosis using ECG signals," International Journal of Neural Systems, vol. 23, no. 04, p. 1350014, 2013.
7. [7] E. J. da S. Luz et al., "ECG-based heartbeat classification for arrhythmia detection: A survey," Computer Methods and Programs in Biomedicine, vol. 127, pp. 144-164, 2016.
8. [8] A. Y. Hannun et al., "Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep convolutional neural network," Nature Medicine, vol. 25, no. 1, pp. 65-69, 2019.

9. [9] A. H. Ribeiro et al., "Automatic diagnosis of the 12-lead ECG using a deep neural network," Nature Communications, vol. 11, no. 1, p. 1760, 2020.
10. [10] N. Strodthoff et al., "Deep learning for ECG analysis: Benchmarks and insights from PTB-XL," IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 5, pp. 1519-1528, 2020.
11. [11] S. Smigiel, K. Pałczyński, and D. Ledziński, "ECG signal classification using deep learning techniques based on the PTB-XL dataset," Entropy, vol. 23, no. 9, p. 1121, 2021.
12. [12] T. Mehari and N. Strodthoff, "Self-supervised representation learning from 12-lead ECG data," Computers in Biology and Medicine, vol. 141, p. 105114, 2022.
13. [13] Ö. Yıldırım, "A novel wavelet sequence based on deep bidirectional LSTM network model for ECG signal classification," Computers in Biology and Medicine, vol. 96, pp. 189-202, 2018.
14. [14] Z. Wang et al., "A deep learning framework for the detection of abnormal heart rhythms," IEEE Access, vol. 7, pp. 121543-121553, 2019.
15. [15] M. Beck et al., "xLSTM: Extended Long Short-Term Memory," arXiv preprint arXiv:2405.04517, 2024.
16. [16] A. Vaswani et al., "Attention is all you need," Advances in Neural Information Processing Systems, vol. 30, 2017.
17. [17] J. Song et al., "Transformer-Based Spatial-Temporal Feature Learning for ECG Arrhythmia Classification," IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 10, pp. 5001-5012, Oct. 2022.
18. [18] C. Che et al., "Constrained transformer network for ECG signal processing and arrhythmia classification," BMC Medical Informatics and Decision Making, vol. 21, 2021.
19. [19] S. C. Huang et al., "Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines," NPJ Digital Medicine, vol. 3, no. 136, 2020.
20. [20] C. Qu, X. Zhang, Y. Lu, Y. Wang, and C. Su, "MAF-Net: Multimodal cross-attention-based fusion network for cardiovascular disease classification," PLoS One, vol. 21, no. 4, Art. no. e0345238, Apr. 2026, doi: 10.1371/journal.pone.0345238.
21. [21] S. Kiranyaz, O. Avci, O. Abdeljaber, T. Ince, M. Gabbouj, and D. J. Inman, "1D convolutional neural networks and applications: A survey," Mechanical Systems and Signal Processing, vol. 151, p. 107398, 2021, doi: 10.1016/j.ymssp.2020.107398.
22. [22] J. Qu, Z. Lin, X. Liang, and J. Song, "Decomposing xLSTM: A comprehensive study on financial time series forecasting," Mathematics, vol. 14, no. 8, Art. no. 1282, Apr. 2026, doi: 10.3390/math14081282.
23. [23] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, "Multimodal machine learning: A survey and taxonomy," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423-443, Feb. 2019, doi: 10.1109/TPAMI.2018.2798607.
24. [24] P. Xu, X. Zhu, and D. A. Clifton, "Multimodal learning with transformers: A survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12113-12132, Oct. 2023, doi: 10.1109/TPAMI.2023.3275156.
25. [25] K. Han et al., "A Survey on Vision Transformer," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 87-110, Jan. 2023, doi: 10.1109/TPAMI.2022.3152247.