Neutral Benchmarking of Survival Models in Health Sciences: Comparative Study of Classical and Machine Learning Techniques

Year 2024, Volume: 11 Issue: 3, 518 - 534, 30.09.2024

Sumaıla Abubakari , Filiz Karaman

https://doi.org/10.54287/gujsa.1505905

Abstract

Survival analysis plays a central role in diverse research fields, especially in health sciences. As an analytical tool, it can be used to help improve patients’ survival time, or at least, reduce the prospects of recurrence in cancer studies. However, approaches to the predictive performance of the current survival models mainly center on clinical data along with the classical survival methods. For censored “omics” data, the performance of survival models has not been thoroughly studied, either often due to their high dimensionality issues or reliance on binarizing the survival time for classification analysis. We aim to present a neural benchmark approach that analyzes and compares a broad range of classical and state-of-the-art machine learning survival models for “omics” and clinical datasets. All the methods considered in our study are evaluated using predictability as a performance measure. The study is systematically designed to make 36 comparisons (9 methods over 4 datasets, i.e., 2 clinical and 2 omics), and shows that, in practice, predictability of survival models does vary across real-world datasets, model choice, as well as the evaluation metric. From our results, we emphasize that performance criteria can play a key role in a balanced assessment of diverse survival models. Moreover, the Multitask Logistic Regression (MTLR) showed remarkable predictability for almost all the datasets. We believe this outstanding performance presents a unique opportunity for a wider use of MTLR for survival risk factors. For translational clinicians and scientists, we hope our findings provide practical guidance for benchmark studies of survival models, as well as highlight potential areas of research interest.

Keywords

Survival Modeling, Machine Learning, Benchmarking, Biomarkers, Oncology

Ethical Statement

We declare no competing interests.

Supporting Institution

This work received no financial support in any form.

Project Number

None

Thanks

We would like to thank the instructors in the Department of Statistics for their invaluable suggestions and constructive criticisms.

References

• Aivaliotis, G., Palczewski, J., Atkinson, R., Cade, J. E., & Morris, M. A. (2021). A comparison of time to event analysis methods, using weight status and breast cancer as a case study. Scientific Reports, 11(1), 14058. https://doi.org/10.1038/s41598-021-92944-z
• Akcay, M., Etiz, D., & Celik, O. (2020). Prediction of survival and recurrence patterns by machine learning in gastric cancer cases undergoing radiation therapy and chemotherapy. Advances in Radiation Oncology, 5(6), 1179-1187. https://doi.org/10.1016/j.adro.2020.07.007
• Arib, M. A. A. (2023). Survival analysis of students not graduated on time using cox proportional hazard regression method and random survival forest method. Journal of Statistics and Data Science, 13-21. https://doi.org/10.33369/jsds.v2i1.24312
• Bengio, Y., & Grandvalet, Y. (2003). No unbiased estimator of the variance of k-fold cross-validation. Advances in Neural Information Processing Systems, 16.
• Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
• Bhambhvani, H. P., Zamora, A., Shkolyar, E., Prado, K., Greenberg, D. R., Kasman, A. M., Liao, J., Shah, S., Srinivas, S., Skinner, E. C., & Shah, J. B. (2021). Development of robust artificial neural networks for prediction of 5-year survival in bladder cancer. Urologic Oncology: Seminars and Original Investigations, 39(3), 193.e7-193.e12. https://doi.org/10.1016/j.urolonc.2020.05.009
• Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32. https://doi.org/10.1023/A:1010933404324
• Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2
• Ching, T., Zhu, X., & Garmire, L. X. (2018). Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data. PLoS Computational Biology, 14(4), e1006076. https://doi.org/10.1371/journal.pcbi.1006076
• Cox, D. R. (1972). Regression Models and Life-Tables. Journal of the Royal Statistical Society: Series B (Methodological), 34(2), 187-202. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
• Cox, D. R. (1975). Partial likelihood. Biometrika, 62(2), 269-276. https://doi.org/10.1093/biomet/62.2.269
• Dai, B., & Breheny, P. (2019). Cross-validation approaches for penalized Cox regression. Statistical Methods in Medical Research, 33(4), 702-715. https://doi.org/10.1177/09622802241233770
• De Bin, R. (2016). Boosting in cox regression: A comparison between the likelihood-based and the model-based approaches with focus on the r-packages CoxBoost and mboost. Computational Statistics, 31, 513-531. https://doi.org/10.1007/s00180-015-0642-2
• Deepa, P., & Gunavathi, C. (2022). A systematic review on machine learning and deep learning techniques in cancer survival prediction. Progress in Biophysics and Molecular Biology. https://doi.org/10.1016/j.pbiomolbio.2022.07.004
• Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52-64. https://doi.org/10.1080/01621459.1961.10482090
• Fotso, S. (2018). Deep neural networks for survival analysis based on a multi-task framework. arXiv Preprint arXiv:1801.05512. https://doi.org/10.48550/arXiv.1801.05512
• Freund, Y. (1990). Boosting a weak learning algorithm by majority. In M. FULK & J. CASE (Eds.), Colt proceedings 1990 (pp. 202-216). Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-146-8.50019-9
• Ganzfried, B. F., Riester, M., Haibe-Kains, B., Risch, T., Tyekucheva, S., Jazic, I., Wang, X. V., Ahmadifar, M., Birrer, M. J., Parmigiani, G., Huttenhower, C., & Waldron, L. (2013). curatedOvarianData: Clinically annotated data for the ovarian cancer transcriptome. Database, 2013. https://doi.org/10.1093/database/bat013
• Gijbels, I. (2010). Censored data. Wiley Interdisciplinary Reviews: Computational Statistics, 2(2), 178-188. https://doi.org/10.1002/wics.80
• Graf, E., Schmoor, C., Sauerbrei, W., & Schumacher, M. (1999). Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, 18(17-18), 2529-2545. https://doi.org/10.1002/(SICI)1097-0258(19990915/30)18:17/18%3C2529::AID-SIM274%3E3.0.CO;2-5
• Harrell Jr, F. E., Lee, K. L., & Mark, D. B. (1996). Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15(4), 361-387. https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4%3C361::AID-SIM168%3E3.0.CO;2-4
• Hastie, T., Tibshirani, R., & Wainwright, M. (2015). Statistical learning with sparsity: The lasso and generalizations. CRC press. https://doi.org/10.1201/b18401
• Herrmann, M., Probst, P., Hornung, R., Jurinovic, V., & Boulesteix, A. (2021). Large-scale benchmark study of survival prediction methods using multi-omics data. Briefings in Bioinformatics, 22(3), bbaa167. https://doi.org/10.1093/bib/bbaa167
• Ishwaran, H., Kogalur, U. B., Blackstone, E. H., & Lauer, M. S. (2008). Random survival forests. https://doi.org/10.1214/08-AOAS169
• Ishwaran, H., Kogalur, U. B., Chen, X., & Minn, A. J. (2011). Random survival forests for high-dimensional data. Statistical Analysis and Data Mining: The ASA Data Science Journal, 4(1), 115-132. https://doi.org/10.1002/sam.10103
• Kalbfleisch, J. D., & Prentice, R. L. (2011). The Statistical Analysis of Failure Time Data. (2nd Ed.). John Wiley & Sons.
• Kaplan, E. L., & Meier, P. (1958). Nonparametric Estimation from Incomplete Observations. Journal of the American Statistical Association, 53(282), 457-481. https://doi.org/10.1080/01621459.1958.10501452
• Karim, Md. R., & Islam, M. A. (2019). Reliability and Survival Analysis. Springer Singapore. https://doi.org/10.1007/978-981-13-9776-9
• Khan, F. M., & Zubek, V. B. (2008). Support vector regression for censored data (SVRc): A novel tool for survival analysis. 2008 Eighth IEEE International Conference on Data Mining, 863-868. https://doi.org/10.1109/ICDM.2008.50
• Kim, H., Park, T., Jang, J., & Lee, S. (2022). Comparison of survival prediction models for pancreatic cancer: Cox model versus machine learning models. Genomics & Informatics, 20(2). https://doi.org/10.5808/gi.22036
• Klein, J. P., & Moeschberger, M. L. (2003). Survival analysis: Techniques for censored and truncated data (2nd ed). Springer.
• Liang, F., Hatcher, W. G., Liao, W., Gao, W., & Yu, W. (2019). Machine learning for security and the internet of things: The good, the bad, and the ugly. IEEE Access, 7, 158126-158147. https://doi.org/10.1109/ACCESS.2019.2948912
• Liu, C., Chen, Y., Deng, Y., Dong, Y., Jiang, J., Chen, S., Kang, W., Deng, J., & Sun, H. (2019). Survival-based bioinformatics analysis to identify hub genes and key pathways in non-small cell lung cancer. Translational Cancer Research, 8(4). https://tcr.amegroups.org/article/view/30209
• Loprinzi, C. L., Laurie, J. A., Wieand, H. S., Krook, J. E., Novotny, P. J., Kugler, J. W., Bartel, J., Law, M., Bateman, M., & Klatt, N. E. (1994). Prospective evaluation of prognostic variables from patient-completed questionnaires. North central cancer treatment group. Journal of Clinical Oncology, 12(3), 601-607. https://doi.org/10.1200/JCO.1994.12.3.601
• Lynch, C. M., Abdollahi, B., Fuqua, J. D., de Carlo, A. R., Bartholomai, J. A., Balgemann, R. N., van Berkel, V. H., & Frieboes, H. B. (2017). Prediction of lung cancer patient survival via supervised machine learning classification techniques. International Journal of Medical Informatics, 108, 1-8. https://doi.org/10.1016/j.ijmedinf.2017.09.013
• Moncada-Torres, A., Maaren, M. C. van, Hendriks, M. P., Siesling, S., & Geleijnse, G. (2021). Explainable machine learning can outperform cox regression predictions and provide insights in breast cancer survival. Scientific Reports, 11(1), 6968. https://doi.org/10.1038/s41598-021-86327-7
• Peto, R., & Peto, J. (1972). Asymptotically efficient rank invariant test procedures. Journal of the Royal Statistical Society: Series A (General), 135(2), 185-198. https://doi.org/10.2307/2344317
• Richter, A. N., & Khoshgoftaar, T. M. (2018). A review of statistical and machine learning methods for modeling cancer risk using structured clinical data. Artificial Intelligence in Medicine, 90, 1-14. https://doi.org/10.1016/j.artmed.2018.06.002
• Salerno, S., & Li, Y. (2023). High-dimensional survival analysis: Methods and applications. Annual Review of Statistics and Its Application, 10(1), 25-49. https://doi.org/10.1146/annurev-statistics-032921-022127
• Shih, J., & Emura, T. (2021). Penalized cox regression with a five-parameter spline model. Communications in Statistics-Theory and Methods, 50(16), 3749-3768. https://doi.org/10.1080/03610926.2020.1772305
• Smola, A. J., & Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14, 199-222. https://doi.org/10.1023/B:STCO.0000035301.49549.88
• Thackham, M. (2022). Survival analysis: Applications to credit risk default modelling [PhD thesis, Macquarie University]. https://doi.org/10.25949/19436723.v1
• Uno, H., Cai, T., Pencina, M. J., D’Agostino, R. B., & Wei, L.-J. (2011). On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine, 30(10), 1105-1117. https://doi.org/10.1002/sim.4154
• Van Belle, V., Pelckmans, K., Suykens, J. A., & Van Huffel, S. (2008). Survival SVM: A practical scalable algorithm. In: Proceedings of the 16th European Symposium on Artificial Neural Networks (pp. 89-94).
• Wang, P., Li, Y., & Reddy, C. K. (2019). Machine learning for survival analysis: A survey. 51(6). https://doi.org/10.1145/3214306
• Xiao, J., Mo, M., Wang, Z., Zhou, C., Shen, J., Yuan, J., He, Y., & Zheng, Y. (2022). The application and comparison of machine learning models for the prediction of breast cancer prognosis: Retrospective cohort study. JMIR Medical Informatics, 10(2), e33440. https://doi.org/10.2196/33440
• Ye, J., & Liu, J. (2012). Sparse methods for biomedical data. ACM Sigkdd Explorations Newsletter, 14(1), 4-15. https://doi.org/10.1145/2408736.2408739
• Yu, C., Greiner, R., Lin, H., & Baracos, V. (2011). Learning patient-specific cancer survival distributions as a sequence of dependent regressors. Advances in Neural Information Processing Systems, 24.
• Zhang, Y., Wong, G., Mann, G., Muller, S., & Yang, J. Y. (2022). SurvBenchmark: Comprehensive benchmarking study of survival analysis methods using both omics data and clinical data. GigaScience, 11, giac071. https://doi.org/10.1093/gigascience/giac071
• Zhao, L., & Feng, D. (2019). Dnnsurv: Deep neural networks for survival analysis using pseudo values. IEEE Journal of Biomedical and Health Informatics, 24(11), 3308-3314. https://doi.org/10.1109/JBHI.2020.2980204