Kalp Hastalığı Sınıflandırmasında Veri Sızıntısının Bir Kaynağı Olarak Yinelenen Klinik Profiller: Doğrulama, Kalibrasyon ve Açıklanabilirlik Üzerine Bir Karşılaştırmalı Denetim

Abstract

Yinelenen klinik profiller, aynı gözlemi hem eğitim hem de test bölümlerine yerleştirebilir ve bu da bağımsız vakalara genelleme yapmayan, iyimser performans tahminlerine yol açabilir. Bu karşılaştırmalı inceleme, 1025 satır ve 13 öngörücü içeren, yaygın olarak kullanılan bir kalp hastalığı veri setini incelemiştir. Veri bütünlüğü analizi sonucunda 302 benzersiz klinik profil ve 723 yinelenen satır tespit edilmiştir. Lojistik Regresyon, Gauss Naif Bayes, k-En Yakın Komşu, Rastgele Orman, Ekstra Ağaçlar ve Histogram Tabanlı Gradyan Güçlendirme yöntemleri, naif satır düzeyinde, grup güvenli ve yinelenen verilerden arındırılmış beş katlı çapraz doğrulama altında değerlendirilmiştir. Performans, sınıflandırma, ayrım ve kalibrasyon ölçütleri kullanılarak profil düzeyinde değerlendirilmiştir. Naif doğrulama altında, k-En Yakın Komşu, Rastgele Orman ve Ekstra Ağaçlar, 1,000 doğruluk ve ROC-AUC değerlerine ulaştı. Tekilleştirme işleminden sonra, doğruluk değerleri 0,8245 ile 0,8377 arasında değişti. Rastgele Orman için, naif doğrulama doğruluğu 0,1722 ve ROC-AUC'yi 0,0950 artırdı. Sızıntısız değerlendirmede, Lojistik Regresyon 0,9128 ile en yüksek ROC-AUC değerine ve 0,1142 ile en düşük Brier puanına ulaştı. Sigmoid kalibrasyonu, Rastgele Orman’ın beklenen kalibrasyon hatasını 0,0563’ten 0,0384’e düşürdü. İç içe geçmiş çapraz doğrulama duyarlılık analizi, hiperparametre optimizasyonundan sonra mütevazı ve tutarsız değişiklikler gösterdi; dış katlamalar genelinde istatistiksel olarak anlamlı bir iyileşme gözlenmedi. Permütasyon önem analizi, thal, cp ve ca'yı en etkili öngörücü değişkenler olarak belirledi. Bu bulgular, neredeyse kusursuz klinik makine öğrenimi performansının algoritmik üstünlük olarak yorumlanmadan önce veri bütünlüğünün ve bağımsız gözlem biriminin tanımının incelenmesi gerektiğini göstermektedir.

Keywords

• Kalp hastalığı
• veri sızıntısı
• yinelenen kayıtlar
• çapraz doğrulama
• kalibrasyon
• açıklanabilir makine öğrenmesi.

Duplicate Clinical Profiles as a Source of Data Leakage in Heart Disease Classification: A Benchmark Audit of Validation, Calibration, and Explainability

Abstract

Duplicate clinical profiles can place the same observation in both training and test partitions, producing optimistic performance estimates that do not represent generalization to independent cases. This benchmark audit examined a widely used heart disease dataset containing 1025 rows and 13 predictors. Data-integrity analysis identified 302 unique clinical profiles and 723 redundant rows. Logistic Regression, Gaussian Naive Bayes, k-Nearest Neighbors, Random Forest, Extra Trees, and Histogram-Based Gradient Boosting were evaluated under naive row-level, group-safe, and deduplicated repeated five-fold cross-validation. Performance was assessed at the profile level using classification, discrimination, and calibration measures. Under naive validation, k-Nearest Neighbors, Random Forest, and Extra Trees achieved accuracy and ROC-AUC values of 1.000. After deduplication, their accuracy ranged from 0.8245 to 0.8377. For Random Forest, naive validation inflated accuracy by 0.1722 and ROC-AUC by 0.0950. In the leakage-free evaluation, Logistic Regression achieved the highest ROC-AUC of 0.9128 and the lowest Brier score of 0.1142. Sigmoid calibration reduced the Random Forest expected calibration error from 0.0563 to 0.0384. A nested cross-validation sensitivity analysis showed modest and inconsistent changes after hyperparameter optimization, with no statistically significant improvement across the outer folds. Permutation importance identified thal, cp, and ca as the most influential predictors. These findings demonstrate that data integrity and the definition of the independent observation unit should be examined before near-perfect clinical machine-learning performance is interpreted as algorithmic superiority.

Keywords

• Heart disease
• data leakage
• duplicate records
• cross-validation
• calibration
• explainable machine learning.

References

1. [1] World Health Organization, “Cardiovascular diseases (CVDs),” WHO, 2025.
2. [2] G. A. Mensah et al., “Global Burden of Cardiovascular Diseases and Risks, 1990-2022,” Journal of the American College of Cardiology, vol. 82, no. 25, pp. 2350–2473, Dec. 2023, doi: 10.1016/j.jacc.2023.11.007.
3. [3] K. Shameer, K. W. Johnson, B. S. Glicksberg, J. T. Dudley, and P. P. Sengupta, “Machine learning in cardiovascular medicine: are we there yet?,” Heart, vol. 104, no. 14, pp. 1156–1164, Jul. 2018, doi: 10.1136/heartjnl-2017-311198.
4. [4] G. Quer, R. Arnaout, M. Henne, and R. Arnaout, “Machine Learning and the Future of Cardiovascular Care: JACC State-of-the-Art Review,” Journal of the American College of Cardiology, vol. 77, no. 3, pp. 300–313, Jan. 2021, doi: 10.1016/j.jacc.2020.11.030.
5. [5] E. Christodoulou, J. Ma, G. S. Collins, E. W. Steyerberg, J. Y. Verbakel, and B. Van Calster, “A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models,” Journal of Clinical Epidemiology, vol. 110, pp. 12–22, Jun. 2019, doi: 10.1016/j.jclinepi.2019.02.004.
6. [6] R. Detrano et al., “International application of a new probability algorithm for the diagnosis of coronary artery disease,” The American Journal of Cardiology, vol. 64, no. 5, pp. 304–310, Aug. 1989, doi: 10.1016/0002-9149(89)90524-9.
7. [7] W. S. Andras Janosi, “Heart Disease.” UCI Machine Learning Repository, 1989. doi: 10.24432/C52P4X.
8. [8] G. S. Collins, J. B. Reitsma, D. G. Altman, and K. G. M. Moons, “Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement,” BMJ, vol. 350, no. jan07 4, pp. g7594–g7594, Jan. 2015, doi: 10.1136/bmj.g7594.

9. [9] C. L. Andaur Navarro et al., “Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review,” BMJ, p. n2281, Oct. 2021, doi: 10.1136/bmj.n2281.
10. [10] B. Van Calster, E. W. Steyerberg, L. Wynants, and M. Van Smeden, “There is no such thing as a validated prediction model,” BMC Med, vol. 21, no. 1, p. 70, Feb. 2023, doi: 10.1186/s12916-023-02779-w.
11. [11] On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, B. Van Calster, D. J. McLernon, M. Van Smeden, L. Wynants, and E. W. Steyerberg, “Calibration: the Achilles heel of predictive analytics,” BMC Med, vol. 17, no. 1, p. 230, Dec. 2019, doi: 10.1186/s12916-019-1466-7.
12. [12] J. Bernett et al., “Guiding questions to avoid data leakage in biological machine learning applications,” Nat Methods, vol. 21, no. 8, pp. 1444–1453, Aug. 2024, doi: 10.1038/s41592-024-02362-y.
13. [13] M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome-based machine learning models,” Nat Commun, vol. 15, no. 1, p. 1829, Feb. 2024, doi: 10.1038/s41467-024-46150-w.
14. [14] C. Krittanawong et al., “Machine learning prediction in cardiovascular diseases: a meta-analysis,” Sci Rep, vol. 10, no. 1, p. 16057, Sep. 2020, doi: 10.1038/s41598-020-72685-1.
15. [15] R. Khera et al., “Transforming Cardiovascular Care With Artificial Intelligence: From Discovery to Practice,” Journal of the American College of Cardiology, vol. 84, no. 1, pp. 97–114, Jul. 2024, doi: 10.1016/j.jacc.2024.05.003.
16. [16] S. F. Weng, J. Reps, J. Kai, J. M. Garibaldi, and N. Qureshi, “Can machine-learning improve cardiovascular risk prediction using routine clinical data?,” PLoS ONE, vol. 12, no. 4, p. e0174944, Apr. 2017, doi: 10.1371/journal.pone.0174944.
17. [17] S. S. Al-Zaiti et al., “A clinician’s guide to understanding and critically appraising machine learning studies: a checklist for Ruling Out Bias Using Standard Tools in Machine Learning (ROBUST-ML),” European Heart Journal - Digital Health, vol. 3, no. 2, pp. 125–140, Jul. 2022, doi: 10.1093/ehjdh/ztac016.
18. [18] R. F. Wolff et al., “PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies,” Ann Intern Med, vol. 170, no. 1, pp. 51–58, Jan. 2019, doi: 10.7326/M18-1376.
19. [19] C. L. Andaur Navarro et al., “Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review,” BMC Med Res Methodol, vol. 22, no. 1, p. 12, Dec. 2022, doi: 10.1186/s12874-021-01469-6.
20. [20] C. L. Andaur Navarro et al., “Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models,” Journal of Clinical Epidemiology, vol. 154, pp. 8–22, Feb. 2023, doi: 10.1016/j.jclinepi.2022.11.015.
21. [21] R. D. Riley et al., “Evaluation of clinical prediction models (part 2): how to undertake an external validation study,” BMJ, vol. 384, p. e074820, Jan. 2024, doi: 10.1136/bmj-2023-074820.
22. [22] E. W. Steyerberg et al., “Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures,” Epidemiology, vol. 21, no. 1, pp. 128–138, Jan. 2010, doi: 10.1097/EDE.0b013e3181c30fb2.
23. [23] B. Van Calster, D. Nieboer, Y. Vergouwe, B. De Cock, M. J. Pencina, and E. W. Steyerberg, “A calibration hierarchy for risk models was defined: from utopia to empirical data,” Journal of Clinical Epidemiology, vol. 74, pp. 167–176, Jun. 2016, doi: 10.1016/j.jclinepi.2015.12.005.
24. [24] J. Allgaier and R. Pryss, “Practical approaches in evaluating validation and biases of machine learning applied to mobile health studies,” Commun Med, vol. 4, no. 1, p. 76, Apr. 2024, doi: 10.1038/s43856-024-00468-0.
25. [25] L. Sasse et al., “Overview of leakage scenarios in supervised machine learning,” J Big Data, vol. 12, no. 1, p. 135, May 2025, doi: 10.1186/s40537-025-01193-8.
26. [26] S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection,” BMC Bioinformatics, vol. 7, no. 1, p. 91, Dec. 2006, doi: 10.1186/1471-2105-7-91.
27. [27] G. C. Cawley and N. L. C. Talbot, “On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation,” Journal of Machine Learning Research, vol. 11, pp. 2079–2107, 2010.
28. [28] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nat Mach Intell, vol. 1, no. 5, pp. 206–215, May 2019, doi: 10.1038/s42256-019-0048-x.
29. [29] M. Ghassemi, L. Oakden-Rayner, and A. L. Beam, “The false hope of current approaches to explainable artificial intelligence in health care,” The Lancet Digital Health, vol. 3, no. 11, pp. e745–e750, Nov. 2021, doi: 10.1016/S2589-7500(21)00208-9.
30. [30] O. Efthimiou, M. Seo, K. Chalkou, T. Debray, M. Egger, and G. Salanti, “Developing clinical prediction models: a step-by-step guide,” BMJ, vol. 386, p. e078276, Sep. 2024, doi: 10.1136/bmj-2023-078276.
31. [31] “Heart Disease Dataset.” Accessed: Jul. 17, 2026. [Online]. Available: https://www.kaggle.com/datasets/johnsmith88/heart-disease-dataset