Neutral Benchmarking of Survival Models in Health Sciences: Comparative Study of Classical and Machine Learning Techniques

Abstract

Survival analysis plays a central role in diverse research fields, especially in health sciences. As an analytical tool, it can be used to help improve patients’ survival time, or at least, reduce the prospects of recurrence in cancer studies. However, approaches to the predictive performance of the current survival models mainly center on clinical data along with the classical survival methods. For censored “omics” data, the performance of survival models has not been thoroughly studied, either often due to their high dimensionality issues or reliance on binarizing the survival time for classification analysis. We aim to present a neural benchmark approach that analyzes and compares a broad range of classical and state-of-the-art machine learning survival models for “omics” and clinical datasets. All the methods considered in our study are evaluated using predictability as a performance measure. The study is systematically designed to make 36 comparisons (9 methods over 4 datasets, i.e., 2 clinical and 2 omics), and shows that, in practice, predictability of survival models does vary across real-world datasets, model choice, as well as the evaluation metric. From our results, we emphasize that performance criteria can play a key role in a balanced assessment of diverse survival models. Moreover, the Multitask Logistic Regression (MTLR) showed remarkable predictability for almost all the datasets. We believe this outstanding performance presents a unique opportunity for a wider use of MTLR for survival risk factors. For translational clinicians and scientists, we hope our findings provide practical guidance for benchmark studies of survival models, as well as highlight potential areas of research interest.

Keywords

• Survival Modeling
• Machine Learning
• Benchmarking
• Biomarkers
• Oncology

References

1. Aivaliotis, G., Palczewski, J., Atkinson, R., Cade, J. E., & Morris, M. A. (2021). A comparison of time to event analysis methods, using weight status and breast cancer as a case study. Scientific Reports, 11(1), 14058. https://doi.org/10.1038/s41598-021-92944-z
2. Akcay, M., Etiz, D., & Celik, O. (2020). Prediction of survival and recurrence patterns by machine learning in gastric cancer cases undergoing radiation therapy and chemotherapy. Advances in Radiation Oncology, 5(6), 1179-1187. https://doi.org/10.1016/j.adro.2020.07.007
3. Arib, M. A. A. (2023). Survival analysis of students not graduated on time using cox proportional hazard regression method and random survival forest method. Journal of Statistics and Data Science, 13-21. https://doi.org/10.33369/jsds.v2i1.24312
4. Bengio, Y., & Grandvalet, Y. (2003). No unbiased estimator of the variance of k-fold cross-validation. Advances in Neural Information Processing Systems, 16.
5. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
6. Bhambhvani, H. P., Zamora, A., Shkolyar, E., Prado, K., Greenberg, D. R., Kasman, A. M., Liao, J., Shah, S., Srinivas, S., Skinner, E. C., & Shah, J. B. (2021). Development of robust artificial neural networks for prediction of 5-year survival in bladder cancer. Urologic Oncology: Seminars and Original Investigations, 39(3), 193.e7-193.e12. https://doi.org/10.1016/j.urolonc.2020.05.009
7. Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32. https://doi.org/10.1023/A:1010933404324
8. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2

9. Ching, T., Zhu, X., & Garmire, L. X. (2018). Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data. PLoS Computational Biology, 14(4), e1006076. https://doi.org/10.1371/journal.pcbi.1006076
10. Cox, D. R. (1972). Regression Models and Life-Tables. Journal of the Royal Statistical Society: Series B (Methodological), 34(2), 187-202. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
11. Cox, D. R. (1975). Partial likelihood. Biometrika, 62(2), 269-276. https://doi.org/10.1093/biomet/62.2.269
12. Dai, B., & Breheny, P. (2019). Cross-validation approaches for penalized Cox regression. Statistical Methods in Medical Research, 33(4), 702-715. https://doi.org/10.1177/09622802241233770
13. De Bin, R. (2016). Boosting in cox regression: A comparison between the likelihood-based and the model-based approaches with focus on the r-packages CoxBoost and mboost. Computational Statistics, 31, 513-531. https://doi.org/10.1007/s00180-015-0642-2
14. Deepa, P., & Gunavathi, C. (2022). A systematic review on machine learning and deep learning techniques in cancer survival prediction. Progress in Biophysics and Molecular Biology. https://doi.org/10.1016/j.pbiomolbio.2022.07.004
15. Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52-64. https://doi.org/10.1080/01621459.1961.10482090
16. Fotso, S. (2018). Deep neural networks for survival analysis based on a multi-task framework. arXiv Preprint arXiv:1801.05512. https://doi.org/10.48550/arXiv.1801.05512
17. Freund, Y. (1990). Boosting a weak learning algorithm by majority. In M. FULK & J. CASE (Eds.), Colt proceedings 1990 (pp. 202-216). Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-146-8.50019-9
18. Ganzfried, B. F., Riester, M., Haibe-Kains, B., Risch, T., Tyekucheva, S., Jazic, I., Wang, X. V., Ahmadifar, M., Birrer, M. J., Parmigiani, G., Huttenhower, C., & Waldron, L. (2013). curatedOvarianData: Clinically annotated data for the ovarian cancer transcriptome. Database, 2013. https://doi.org/10.1093/database/bat013
19. Gijbels, I. (2010). Censored data. Wiley Interdisciplinary Reviews: Computational Statistics, 2(2), 178-188. https://doi.org/10.1002/wics.80
20. Graf, E., Schmoor, C., Sauerbrei, W., & Schumacher, M. (1999). Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, 18(17-18), 2529-2545. https://doi.org/10.1002/(SICI)1097-0258(19990915/30)18:17/18%3C2529::AID-SIM274%3E3.0.CO;2-5
21. Harrell Jr, F. E., Lee, K. L., & Mark, D. B. (1996). Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15(4), 361-387. https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4%3C361::AID-SIM168%3E3.0.CO;2-4
22. Hastie, T., Tibshirani, R., & Wainwright, M. (2015). Statistical learning with sparsity: The lasso and generalizations. CRC press. https://doi.org/10.1201/b18401
23. Herrmann, M., Probst, P., Hornung, R., Jurinovic, V., & Boulesteix, A. (2021). Large-scale benchmark study of survival prediction methods using multi-omics data. Briefings in Bioinformatics, 22(3), bbaa167. https://doi.org/10.1093/bib/bbaa167
24. Ishwaran, H., Kogalur, U. B., Blackstone, E. H., & Lauer, M. S. (2008). Random survival forests. https://doi.org/10.1214/08-AOAS169
25. Ishwaran, H., Kogalur, U. B., Chen, X., & Minn, A. J. (2011). Random survival forests for high-dimensional data. Statistical Analysis and Data Mining: The ASA Data Science Journal, 4(1), 115-132. https://doi.org/10.1002/sam.10103
26. Kalbfleisch, J. D., & Prentice, R. L. (2011). The Statistical Analysis of Failure Time Data. (2nd Ed.). John Wiley & Sons.
27. Kaplan, E. L., & Meier, P. (1958). Nonparametric Estimation from Incomplete Observations. Journal of the American Statistical Association, 53(282), 457-481. https://doi.org/10.1080/01621459.1958.10501452
28. Karim, Md. R., & Islam, M. A. (2019). Reliability and Survival Analysis. Springer Singapore. https://doi.org/10.1007/978-981-13-9776-9
29. Khan, F. M., & Zubek, V. B. (2008). Support vector regression for censored data (SVRc): A novel tool for survival analysis. 2008 Eighth IEEE International Conference on Data Mining, 863-868. https://doi.org/10.1109/ICDM.2008.50
30. Kim, H., Park, T., Jang, J., & Lee, S. (2022). Comparison of survival prediction models for pancreatic cancer: Cox model versus machine learning models. Genomics & Informatics, 20(2). https://doi.org/10.5808/gi.22036
31. Klein, J. P., & Moeschberger, M. L. (2003). Survival analysis: Techniques for censored and truncated data (2nd ed). Springer.
32. Liang, F., Hatcher, W. G., Liao, W., Gao, W., & Yu, W. (2019). Machine learning for security and the internet of things: The good, the bad, and the ugly. IEEE Access, 7, 158126-158147. https://doi.org/10.1109/ACCESS.2019.2948912
33. Liu, C., Chen, Y., Deng, Y., Dong, Y., Jiang, J., Chen, S., Kang, W., Deng, J., & Sun, H. (2019). Survival-based bioinformatics analysis to identify hub genes and key pathways in non-small cell lung cancer. Translational Cancer Research, 8(4). https://tcr.amegroups.org/article/view/30209
34. Loprinzi, C. L., Laurie, J. A., Wieand, H. S., Krook, J. E., Novotny, P. J., Kugler, J. W., Bartel, J., Law, M., Bateman, M., & Klatt, N. E. (1994). Prospective evaluation of prognostic variables from patient-completed questionnaires. North central cancer treatment group. Journal of Clinical Oncology, 12(3), 601-607. https://doi.org/10.1200/JCO.1994.12.3.601
35. Lynch, C. M., Abdollahi, B., Fuqua, J. D., de Carlo, A. R., Bartholomai, J. A., Balgemann, R. N., van Berkel, V. H., & Frieboes, H. B. (2017). Prediction of lung cancer patient survival via supervised machine learning classification techniques. International Journal of Medical Informatics, 108, 1-8. https://doi.org/10.1016/j.ijmedinf.2017.09.013
36. Moncada-Torres, A., Maaren, M. C. van, Hendriks, M. P., Siesling, S., & Geleijnse, G. (2021). Explainable machine learning can outperform cox regression predictions and provide insights in breast cancer survival. Scientific Reports, 11(1), 6968. https://doi.org/10.1038/s41598-021-86327-7
37. Peto, R., & Peto, J. (1972). Asymptotically efficient rank invariant test procedures. Journal of the Royal Statistical Society: Series A (General), 135(2), 185-198. https://doi.org/10.2307/2344317
38. Richter, A. N., & Khoshgoftaar, T. M. (2018). A review of statistical and machine learning methods for modeling cancer risk using structured clinical data. Artificial Intelligence in Medicine, 90, 1-14. https://doi.org/10.1016/j.artmed.2018.06.002
39. Salerno, S., & Li, Y. (2023). High-dimensional survival analysis: Methods and applications. Annual Review of Statistics and Its Application, 10(1), 25-49. https://doi.org/10.1146/annurev-statistics-032921-022127
40. Shih, J., & Emura, T. (2021). Penalized cox regression with a five-parameter spline model. Communications in Statistics-Theory and Methods, 50(16), 3749-3768. https://doi.org/10.1080/03610926.2020.1772305
41. Smola, A. J., & Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14, 199-222. https://doi.org/10.1023/B:STCO.0000035301.49549.88
42. Thackham, M. (2022). Survival analysis: Applications to credit risk default modelling [PhD thesis, Macquarie University]. https://doi.org/10.25949/19436723.v1
43. Uno, H., Cai, T., Pencina, M. J., D’Agostino, R. B., & Wei, L.-J. (2011). On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine, 30(10), 1105-1117. https://doi.org/10.1002/sim.4154
44. Van Belle, V., Pelckmans, K., Suykens, J. A., & Van Huffel, S. (2008). Survival SVM: A practical scalable algorithm. In: Proceedings of the 16th European Symposium on Artificial Neural Networks (pp. 89-94).
45. Wang, P., Li, Y., & Reddy, C. K. (2019). Machine learning for survival analysis: A survey. 51(6). https://doi.org/10.1145/3214306
46. Xiao, J., Mo, M., Wang, Z., Zhou, C., Shen, J., Yuan, J., He, Y., & Zheng, Y. (2022). The application and comparison of machine learning models for the prediction of breast cancer prognosis: Retrospective cohort study. JMIR Medical Informatics, 10(2), e33440. https://doi.org/10.2196/33440
47. Ye, J., & Liu, J. (2012). Sparse methods for biomedical data. ACM Sigkdd Explorations Newsletter, 14(1), 4-15. https://doi.org/10.1145/2408736.2408739
48. Yu, C., Greiner, R., Lin, H., & Baracos, V. (2011). Learning patient-specific cancer survival distributions as a sequence of dependent regressors. Advances in Neural Information Processing Systems, 24.
49. Zhang, Y., Wong, G., Mann, G., Muller, S., & Yang, J. Y. (2022). SurvBenchmark: Comprehensive benchmarking study of survival analysis methods using both omics data and clinical data. GigaScience, 11, giac071. https://doi.org/10.1093/gigascience/giac071
50. Zhao, L., & Feng, D. (2019). Dnnsurv: Deep neural networks for survival analysis using pseudo values. IEEE Journal of Biomedical and Health Informatics, 24(11), 3308-3314. https://doi.org/10.1109/JBHI.2020.2980204